Wednesday, December 6, 2017

Quick Start ElasticSearch on Windows

What is ElasticSearch?
ElasticSearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. It is built on top of Apache Lucene. It is commonly used for log analytics, full-text search, and operational intelligence use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

What will you get from this article?
In this article, I'll guide you how to install it on Windows, publish it with your domain, secure it. I also introduce to you first steps to combined it with Kibana and Logstash, so you can create a ELK system which can analyze a huge log data near real time.

Let's go!

Install Java Runtime
ElasticSearch requires Java runtime to run. To check if having Java on your Windows, let open cmd and key:
java -version
If you have no Java or the version < 8.x, let go to Oracle Java website for downloading and installing the latest version or from 8.x.

After installing, let create / update JAVA_HOME system environment variable, set it to the new folder installed Java runtime. For example:


Install & Configure ElasticSearch
Go to https://www.elastic.co/downloads/elasticsearch and download the zip file. Unpack into a folder, e.g. C:\ES

By default ElasticSearch is configured with a 1 GB heap. For real enviroment, this number is not enough. To set new heap size, you must create a system environment variable ES_HEAP_SIZE and set its value, e.g. 4g. This value depends on the memory size (RAM) of your server, it should be less than half of RAM (reference).

After setting new heap size, open C:\ES\config\elasticsearch.yml, set values for important parameters as the following:
# Path to directory where to store the data (separate multiple locations by comma).
path.data: F:\data
# Path to log files.
path.logs: F:\logs
# Lock the memory on startup.
bootstrap.memory_lock: true
# Upper limit on the fielddata. Old data will be evicted to make space for the new values if it is over the limit size. Can be set to a percentage of the heap size, or a concrete value like 5gb.
indices.fielddata.cache.size: 40%
# Set the bind address to a specific IP (IPv4 or IPv6).
network.bind_host: ["192.168.2.12", "localhost"]
# For ReadonlyREST

rest.action.multi.allow_explicit_index: false
Change the values of parameters upon your server. In which, parameter rest.action.multi.allow_explicit_index is reserved for ReadonlyREST tool which will be used for securing queries.

Start ElasticSearch by running bin\elasticsearch.bat file (e.g. C:\ES\bin\elasticsearch.bat). To set up ElasticSearch as a Windows service, run bin\elasticsearch-service.bat install then go to Windows services manager , find ElasticSearch service and change its Startup Type to Automatic.

After starting, let open a browser and check URL http://localhost:9200/ to see if it is working.

Install Logstash
Go to https://www.elastic.co/downloads/logstash and download the zip file. Unpack into a folder, e.g. C:\LS

Create a simple config file (e.g. logstash-simple.conf) in bin folder (e.g. C:\LS\bin) with a content as the following example to run Logstash.
input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}
Then open cmd, go to bin folder & key below command for starting Logstash:
logstash.bat -f logstash-simple.conf
To make a Windows service for Logstash, you can use Non-Sucking Service Manager (NSSM) tool. Download latest NSSM from its download page, unzip it into a folder, e.g. F:\soft\nssm. Use cmd go to F:\soft\nssm\win64 and run the command:
nssm.exe install Logstash
Then you may fill in values as below and click Install service button:


After that, you can open Windows services manager to start Logstash service.

Install Kibana
Go to Kibana download page, download and unzip Kibana into a folder, e.g. C:\KB
Open config/kibana.yml file and configure some important parameters like:
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://localhost:9200"
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
elasticsearch.requestTimeout: 30000

Run bin\kibana.bat in the installation folder (e.g. C:\KB\bin\kibana.bat) to start Kibana. Check http://localhost:5601 to see if Kibana works.


You can use NSSM to create a Windows service for Kibana. The above picture means no data has been shipped to Kibana yet. For shipping sample data into Kibana, you can try Winlogbeat which sends Windows event logs such as application events, security events, system events, etc. to Logstash.

Setup nginx to access ElasticSearch via a domain
With above configuration, you can only access ElasticSearch via localhost or IP 192.168.2.12. To access ElasticSearch via a domain, we can use nginx forward request to localhost (you can read my article on this link for how to installing nginx). Of course you can configure ElasticSearch directly on file C:\ES\config\elasticsearch.yml to allow to access it via a domain / public IP without using nginx, but I want to use nginx for allowing read only request to my ElasticSearch, for writing request I will setup for IP 192.168.2.12. This security will be done on ReadonlyREST tool in next step.

Below the configuration for nginx:
server {
        listen      *:80;
        server_name  yourdomain.com  www.yourdomain.com;
        location / {
            root your_web_site_root_folder;
            index  index.html index.htm;
        }
        location /es {
            proxy_pass http://localhost:9200;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;
            rewrite ^/es/(.*) /$1 break;
        }
}
Now you can access ElasticSearch via yourdomain.com/es.

Setup ReadonlyREST tool
Go to https://readonlyrest.com/download.html, select your ElasticSearch version and download.
Open cmdcd to the ElasticSearch home and run the command:
bin/elasticsearch-plugin install file:///download-folder/readonlyrest-<version>.zip
For example, download-folder is C:\ES. After that, let create readonlyrest.yml file in the config folder of ElasticSearch (e.g. C:\ES\config/readonlyrest.yml). Then add rules in this file. For example, I want to allow anything from 192.168.2.12 and other hosts can read only, I can add the following rules:
readonlyrest:
    access_control_rules:

    - name: "Rule 1 - Allowing anything from localhost"
      hosts: [192.168.2.12]

    - name: "Rule 2 - Other hosts can only read certain indices"
      actions: ["indices:data/read/*"]
      indices: ["logstash-*"] # aliases are taken in account!

For more examples, you can read here to add configuration snippets into this file.

Finally, your ELK is ready for using in real life. Let enjoy it :)
Any comment is welcome!
See you next time.

Saturday, October 28, 2017

Setup chat bot (wrriten by MS Bot Builder NodeJS) with Nginx + HTTPS on Windows Server

Continuing with my series of articles on chat bot:
In this article, I will cover a solution to set up an production environment for your chat bot written by MS Bot Builder NodeJS with Nginx & HTTPS on Windows Server. In which, you can register your chat bot hosted on any server (not on Azure) with MS Bot Framework.

1. Install Nginx

Download the latest version for Windows from: http://nginx.org/en/download.html
Unpack it on a folder (e.g. C:\nginx ) and click nginx.exe to run nginx. Open your browser, run http://localhost/. If you see a screen like below, it means Nginx is running well.


2. Run your chat bot

Assuming that you've coded a wonderful chat bot, let run it.  For example, I have a super chat bot by echoing what user says, it is in app.js file as the following:
var restify = require('restify');
var builder = require('botbuilder');

// Create chat bot
var connector = new builder.ChatConnector({
    appId: 'app id of your bot on MS Bot Framework',
    appPassword: 'password of your bot on MS Bot Framework'
 });

// Receive messages from the user and respond
var bot = new builder.UniversalBot(connector, function(session) {
    session.send("You said: %s", session.message.text);
});

// Setup Restify Server
var server = restify.createServer();
server.listen(process.env.port || process.env.PORT || 5678, function() {
    console.log('%s listening to %s', server.name, server.url);
});

// Listen for messages from users
server.post('/api/messages', connector.listen());


Run app.js:
node app.js
Note that appId and appPassword will be replaced by your real values when you register your bot on MS Bot Framework.

3. Set up Nginx as proxy for your chat bot

Assuming that you want to run your chat bot on the link: yourdomain.com/bot. Open nginx.conf file (e.g. C:\nginx\conf\nginx.conf) and add a server configuration for your domain:
server {
        listen      *:80;
        server_name  yourdomain.com  www.yourdomain.com;
        location / {
            root your_web_site_root_folder;
            index  index.html index.htm;
        }
        location /bot {
            proxy_pass http://localhost:5678/api/messages;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;
        }
}
Now you can open the chat emulator for testing with the link http://yourdomain.com/bot to see if it works (see section 3 of Create a Luis Chat Bot on Azure Bot Service - Part 1 for setting the emulator with ngrok):


4. Set up HTTPS

MS Bot Framework just allows to register a bot with HTTPS end point message. So that's why we need to set up HTTPS. Luckily, we can setup HTTPS with free SSL Certificate from Let's Encrypt. You can read my article WAMP 64 Bits + Free SSL (section 5) for how to create a free SSL certificate.

Below is the configuration on nginx for a server with SSL:
server {
        listen       443 ssl;
        server_name  yourdomain.com  www.yourdomain.com;
        ssl_certificate      C:/ProgramData/letsencrypt-win-simple/httpsacme-v01.api.letsencrypt.org/www.yourdomain.com-crt.pem;
        ssl_certificate_key  C:/ProgramData/letsencrypt-win-simple/httpsacme-v01.api.letsencrypt.org/www.yourdomain.com-key.pem;
        ssl_trusted_certificate C:/ProgramData/letsencrypt-win-simple/httpsacme-v01.api.letsencrypt.org/ca-<hex number>-crt.pem;
        ssl_session_cache    shared:SSL:50m;
        ssl_session_timeout  1d;
        ssl_ciphers EECDH+AESGCM:EECDH+AES;
        ssl_prefer_server_ciphers  on;
        location / {
            root your_web_site_root_folder;
            index  index.html index.htm;
        }
        location /bot {
            proxy_pass http://localhost:5678/api/messages;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;
        }
}
Test again on the emulator with https://yourdomain.com/bot

5. Register your bot

Go to https://dev.botframework.com, sign in and click My bots menu for registering your bot. Remember to select Register an existing bot built using Bot Builder SDK when creating new bot.
It will open a page for you keying your bot info. Note that Messaging endpoint is https://yourdomain.com/bot.
Click Create Microsoft App ID and password to create app id & password for your bot. Paste your app ID to the required box then copy the app id & password into the source code of your bot (e.g. appIdappPassword in section 2). Save the setting.

Now you can click Test button on top right for testing your bot directly from the website https://dev.botframework.com. For example:


You can also connect your chat bot to other channel like Facebook (see Create a Luis Chat Bot on Azure Bot Service - Part 2 for how to connect with Facebook).

Alright, you have known how to set up a production environment for your chat bot which can be hosted any where.

Have fun! Any comment is welcome.

Sunday, October 15, 2017

MS SQL SMS export large data to Excel without breaking format

If you often work on MS SQL, surely sometimes you want to export large data (result a query) to Excel file. After querying on MS SQL SMS (Microsoft SQL Server Management Studio), you can use right click on the cell top left then select "Copy with Headers" and copy to Excel file or "Save Result As" and save to a CSV file.


However 2 these functions have their problems. "Copy with Headers" cannot copy large data.  "Save Result As" often breaks CSV format if your data contains some special characters.

Fortunately, there is another function for exporting large data without breaking its format. Right click on your database and choose Tasks >> Export Data...



For Data Source, choose SQL Server Native >> select Server name >> select your Database, see the following for example:


Next, select Destination as Microsoft Excel & specify Excel file path.


Next, select Write a query to specify the data to transfer.


In next step, paste your query into or select a file containing your query. In the step Review Data Type Mapping, let review again columns have been converted data (Source Type vs. Destination Type). If you want to fix, click Back to select again Destination Type for converting.

If they are ok, click Next then Finish (don't worry for warning signs). Waiting a moment and you will have your Excel file with correct format you wanted.

Yeah! This is a small tip for you, hope it is useful. Share it to your friend for helping him or her out 😍. Any comment is welcome!

Happy Halloween!

Friday, September 29, 2017

Ionic 3: Debugging on VS Code and Chrome

After long time, since from my first post Quick start your mobile app with Ionic,  now I have a project using Ionic. So I want to share this post to my team for how to debugging on latest Ionic 3. Because saving debug time means you will delivery the application faster.

As beginning, I want to brief important things for starting with latest Ionic 3:
npm install -g ionic cordova
  • Start an app with command (cutePuppyPics is your app name/project)
ionic start cutePuppyPics tutorial
OK, you are ready to code! But wait, let setup to debug for your app on Chrome firstly. Open your app on VS Code and do the following steps:

1. Press F5 >> select Chrome:


2. Edit file launch.json generated with content like below:
{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "chrome",
            "request": "launch",
            "name": "Launch Ionic on Chrome",
            "url": "http://localhost:8100",
            "sourceMaps": true,
            "webRoot": "${workspaceRoot}/src"
        },
        {
            "type": "chrome",
            "request": "attach",
            "name": "Attach Ionic on Chrome",
            "url": "http://localhost:8100",
            "port": 9222,
            "sourceMaps": true,
            "webRoot": "${workspaceRoot}/src"
        }
    ]
}
3. If you use Windows, let open Properties of Chrome's short cut, and add --remote-debugging-port=9222 in to Target box:


For Mac, let use same debug flag as Windows. This debug flag is used for attaching your Ionic app into existing Chrome running.

4. Press Ctrl+` to open VS Code terminal, key:
ionic serve -b
It will build the app and start the server (http://localhost:8100/) but don't launch browser (-b option).

5. Close all Chrome opened, click debugger icon >> select Launch Ionic on Chrome option >> open a file, for example cutePuppyPics\src\pages\list\list.ts, set break point and press F5. It will launch the app on Chrome and display a debug player on the top of VS Code. In this example, when you click My First List menu and hit an item, it will jump to the break point for debugging, see the following picture:


You can stop the debug by pressing Shift+F5 or stop buttonon debug player. Stopping the debugger doesn't stop the app, it is still running until you press Ctrl+C in terminal.

6. In case you want to debug dynamically, you can attach the debugger into the app is running on Chrome. To try, let open Chrome and key http://localhost:8100 to running the app. Then go to VS Code >> click debugger icon >> select Attach Ionic on Chrome option >> click run (or press F5). It will open a debug player like below, and when you click an item in My First List page, it will jump to the break point.


7. You can change source code and see this change updated to running app on Chrome. That's great point of Ionic which can help to speed up developing your application. I love it due to this live update feature.

Happy coding! Any comment is welcome.

Saturday, September 16, 2017

WAMP 64 Bits + Free SSL (Let's Encrypt)

WAMP likes XAMPP, they are free tools packaged Apache, MySQL / Maria DB & PHP together. XAMP also has other services / applications such as FileZilla, Mercury & Tomcat and it also can run on Linux & OS-X. While WAMP just runs on Windows. However for Windows' users, I recommend WAMP because it is designed for Windows only, so it has some advanced features for Windows but XAMPP doesn't have, especially it has Windows 64 bits version while XAMPP has only 32 bits version for Windows.

1. Install

To install Wamp 64 bits, you can download it from wampserver.aviatechno.net. You should install it as Administrator. You also must install all Visual C++ redistributable packages (x86 + x64) (32bits  + 64bits) for running Wamp 64.

After installing & running, if its system tray icon is green, it's ok. Let choose versions for Apache, PHP & MySQL which you want for your websites.


2. Configure auto start up

For live server, you need to configure Apache & MySQL auto start up when the server restarts.
Go to Windows Services, find wampapache64 & wampmysqld64 services then set them Automatic startup.


Next, right click on Wamp tray icon, select Wamp Settings menu >> select Wampserver Homepage at startup.


3. Enable SSL (HTTPS)

The latest Wamp 64 has already included Open SSL built-in, so you don't need to install Open SSL. Below are steps to enable it for Apache:

  • Open httpd.conf file, uncomment the following lines:
LoadModule ssl_module modules/mod_ssl.so
Include conf/extra/httpd-ssl.conf
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so

  • Open php.ini file, uncomment the following line:
extension=php_openssl.dll

4. Add virtual host for your website

Open httpd-vhosts.conf file and add a virtual host running on HTTP (port 80) for your website, below is an example:
<VirtualHost *:80>
 ServerName mydomain.com
 ServerAlias www.mydomain.com
 DocumentRoot "c:/mywebsites/www.mydomain.com"
 <Directory  "c:/mywebsites/www.mydomain.com/">
  Options Indexes FollowSymLinks
        AllowOverride all
        Order Deny,Allow
        Allow from all
        Require all granted
 </Directory>
 ErrorLog "logs/mydomain.com-error.log"
 CustomLog "logs/mydomain.com-access.log" common
 Alias /.well-known c:/mywebsites/www.mydomain.com/.well-known
</VirtualHost>
In which, .wellknow folder will be used to store a key to challenge with Let's Encrypt server for creating your free SSL certificate in next step.

To check if you made right syntax, open cmd tool, go to Apache bin folder and run:
httpd –t
If the syntax is correct, let restart Apache service, then check your website on a browser. You can read my article "Creating Multiple Virtual Sites on a WAMP Server" for more info on creating virtual hosts.

5. Create free SSL certificate with Let's Encrypt

You can read my article Free SSL Certificate with Let's Encrypt to know about Let's Encrypt and tools to create free SSL certificate. In this post, I will show you how to do this with letsencrypt-win-simple tool.

Download latest letsencrypt-win-simple tool from its release page. Unpack it into a folder, for example: C:\letsencrypt-win-simple. Open cmd tool as Administrator, cd to this folder and run letsencrypt.exe, its interface will show as below:


Key N then key 4, it will ask you enter host names (domain names) as the following screen:


Let key your domain name, for example: www.mydomain.com, then it will require you enter the root folder containing your website, for example: c:\mywebsites\www.mydomain.com. After that, the tool will do a process to create a key in the folder c:\mywebsites\www.mydomain.com\.well-known\acme-challenge, then challenge with letsencrypt.org to authorize and create certificates. These certificates are in the folder: C:\ProgramData\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.org. We will use them to create virtual host running on port 443 (HTTPS).

On finishing, the tool will ask you to create a scheduled task running on 9 am every day to renew your certificate automatically when it is used 60 days (30 days before expiry), see the following picture. With this scheduled task, you can sleep well :)


6. Add virtual host HTTPS for your website

Now you are ready to up & run your HTTPS website, let create a virtual host running on port 443 with SSL certificates for it, see the following example:
<VirtualHost *:443>
 ServerName mydomain.com
 ServerAlias www.mydomain.com
 DocumentRoot "c:/mywebsites/www.mydomain.com"
 <Directory  "c:/mywebsites/www.mydomain.com/">
  Options Indexes FollowSymLinks
        AllowOverride all
        Order Deny,Allow
        Allow from all
        Require all granted
 </Directory>
 ErrorLog "logs/mydomain.com-error.log"
 CustomLog "logs/mydomain.com-access.log" common

 SSLEngine on
    SSLCertificateFile "C:/ProgramData/letsencrypt-win-simple/httpsacme-v01.api.letsencrypt.org/www.mydomain.com-crt.pem"
    SSLCertificateKeyFile "C:/ProgramData/letsencrypt-win-simple/httpsacme-v01.api.letsencrypt.org/www.mydomain.com-key.pem"
    SSLCertificateChainFile "C:/ProgramData/letsencrypt-win-simple/httpsacme-v01.api.letsencrypt.org/ca-<hex code>-crt.pem"
</VirtualHost>
Restart Apache service then check https://www.mydomain.com on a browser (for example Chrome). If it has the symbol https with green, it's ok. Congratulation!

7. Make some securities for Wamp 

Remember to restart Wamp's services after configuration 😃

7.1 Hide server info to prevent bad guys mining it

Open httpd-default.conf file, find & change parameters the following:
ServerSignature Off
ServerTokens Prod
7.2 Make sure  directories / files outside of the document root (website) are not allowed to access

Open httpd.conf file, check if the content of directory tag is like the following:
<Directory />
     Order Deny,Allow
     Deny from all
     Options None
     AllowOverride None
</Directory>
or
<Directory />
     AllowOverride none
     Require all denied
</Directory>

7.3 Reduce time out to prevent DoS attacks

Open httpd-default.conf file, find & set the following parameter:
TimeOut 60
You can set it less than, for example 30.

7.4 Set password for MySQL
  • Left click Wamp's system tray icon.
  • Select MySQL > MySQL console menu.
  • Press Enter on the console opened.
  • Key SET PASSWORD FOR root@localhost=PASSWORD('your_password'); (change your_password to your private password). They press Enter.
7.5 Change permissions for folders in your website

Normally, I prefer to set all folders in my website to allow Read & Execute permission only except some folders having Write permission. To check what user runs Apache, let open Windows Task Manager and click Details tab, and see in User name column:


It often is SYSTEM user. To remove Write permission, right click on your web root folder >> select Properties menu >> click Security tab >> click Edit button >> select SYSTEM user >> on Write permission row, untick on Allow column, tick Deny column:


If you cannot edit on Allow column, back to your web root folder >> select Properties menu >> click Security tab >> click Advanced button >> click Disable inheritance button on SYSTEM user having Full control access inheriting from parent folder. Then you can edit in above step.

For folders which you need to write (upload), just enable Write permission for SYSTEM user on those folders.

That's all for tonight. Bye and see you next post. Any comment is welcome!
Good night!


Subscribe to RSS Feed Follow me on Twitter!