{"id":5693,"date":"2018-07-23T19:17:32","date_gmt":"2018-07-23T15:17:32","guid":{"rendered":"https:\/\/www.howtoforge.com\/tutorial\/ubuntu-elastic-stack\/"},"modified":"2018-07-23T19:17:32","modified_gmt":"2018-07-23T15:17:32","slug":"how-to-install-elastic-stack-on-ubuntu-18-04-lts","status":"publish","type":"post","link":"https:\/\/afaghhosting.net\/blog\/how-to-install-elastic-stack-on-ubuntu-18-04-lts\/","title":{"rendered":"How to Install Elastic Stack on Ubuntu 18.04 LTS"},"content":{"rendered":"<p>Elasticsearch is an open source search engine based on Lucene, developed in Java. It provides a distributed and multitenant full-text search engine with an HTTP Dashboard web-interface (Kibana). The data is queried, retrieved and stored in a JSON document scheme. Elasticsearch is a scalable search engine that can be used to search for all kind of text documents, including log files. Elasticsearch is the heart of the &#8216;Elastic Stack&#8217; or ELK Stack.<\/p>\n<p>Logstash is an open source tool for managing events and logs. It provides real-time pipelining for data collections. Logstash will collect your log data, convert the data into JSON documents, and store them in Elasticsearch.<\/p>\n<p>Kibana is an open source data visualization tool for Elasticsearch. Kibana provides a pretty dashboard web interface. It allows you to manage and visualize data from Elasticsearch. It&#8217;s not just beautiful, but also powerful.<\/p>\n<p>In this tutorial, I will show you how to install and configure Elastic Stack on an Ubuntu 18.04 server for monitoring of server logs. Then I&#8217;ll show you how to install and configure &#8216;Elastic beats&#8217; on an Ubuntu 18.04 and a CentOS 7 client server.<\/p>\n<h2 id=\"prerequisites\">Prerequisites<\/h2>\n<ul>\n<li>3 Servers<\/li>\n<ul>\n<li>Ubuntu 18.04 with 4GB Ram\/memory as &#8216;elk-master&#8217; &#8211; 10.0.15.10<\/li>\n<li>Ubuntu 18.04 with 512MB\/1GB Ram\/Memory as &#8216;elk-client01&#8217; &#8211; 10.0.15.21<\/li>\n<li>CentOS 7.5 with 512MB\/1GB Ram\/Memory as &#8216;elk-client02&#8217; &#8211; 10.0.15.22<\/li>\n<\/ul>\n<li>Root privileges<\/li>\n<\/ul>\n<h2 id=\"what-we-will-do\">What we will do?<\/h2>\n<ol>\n<li>Install Elastic Stack<\/li>\n<ol>\n<li>Install Java<\/li>\n<li>Install and Configure ElasticSearch<\/li>\n<li>Install and Configure Kibana<\/li>\n<li>Install and Configure Nginx as Reverse Proxy for Kibana<\/li>\n<li>Install and Configure Logstash<\/li>\n<\/ol>\n<li>Install and Configure Filebeat on Ubuntu 18.04<\/li>\n<li>Install and Configure Filebeat on CentOS 7.5<\/li>\n<li>Testing<\/li>\n<\/ol>\n<h2 id=\"step-install-elastic-stack\">Step 1 &#8211; Install Elastic Stack<\/h2>\n<p>In this first step, we will install and configure the &#8216;Elastic Stack&#8217; on the &#8216;elk-master&#8217; server, so run all commands and stages for this step on the &#8216;elk-master&#8217; server only. We will install and configure each component of the elastic stack, including Elasticsearch, Logstash shipper, and Kibana Dashboard with Nginx web server.<\/p>\n<h3 id=\"install-java\">Install Java<\/h3>\n<p>Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8, and we will install Java 8 from a PPA repository.<\/p>\n<p>Install the &#8216;software-properties-common&#8217; and &#8216;apt-transport-https&#8217; packages, and then add the PPA &#8216;webupd8team&#8217; Java repository. Run the &#8216;apt install&#8217; and &#8216;add-apt-repository&#8217; command below.<\/p>\n<p class=\"command\">sudo apt install software-properties-common apt-transport-https -y<br \/>sudo add-apt-repository ppa:webupd8team\/java -y<\/p>\n<p>Now install the java8-installer.<\/p>\n<p class=\"command\">sudo apt install oracle-java8-installer -y<\/p>\n<p>After the installation is complete, check the java version.<\/p>\n<p class=\"command\">java -version<\/p>\n<p><a class=\"fancybox\" id=\"img-1\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/1.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts.png\" alt=\"Test installed Java version\" width=\"500\" height=\"129\" title=\"\"><\/a><\/p>\n<p>Java 1.8 installed on the system.<\/p>\n<p>Next, we will configure the java environment. Check the java binary file using the command below.<\/p>\n<p class=\"command\">update-alternatives &#8211;config java<\/p>\n<p>And you will get the java binary file on the &#8216;<strong>\/usr\/lib\/jvm\/java-8-oracle<\/strong>&#8216; directory.<\/p>\n<p>Now create the profile file &#8216;java.sh&#8217; under the &#8216;profile.d&#8217; directory.<\/p>\n<p class=\"command\">vim \/etc\/profile.d\/java.sh<\/p>\n<p>Paste java environment configuration below.<\/p>\n<pre>#Set JAVA_HOME&#13;\nJAVA_HOME=\"\/usr\/lib\/jvm\/java-8-oracle\"&#13;\nexport JAVA_HOME&#13;\nPATH=$PATH:$JAVA_HOME&#13;\nexport PATH<\/pre>\n<p>Save and exit.<\/p>\n<p>Make the file executable and load the configuration file.<\/p>\n<p class=\"command\">chmod +x \/etc\/profile.d\/java.sh<br \/>source \/etc\/profile.d\/java.sh<\/p>\n<p>Now check the java environment using the command below.<\/p>\n<p class=\"command\">echo $JAVA_HOME<\/p>\n<p>And you will get the java directory is located at &#8216;<strong>\/usr\/lib\/jvm\/java-8-oracle<\/strong>&#8216; directory.<\/p>\n<p><a class=\"fancybox\" id=\"img-2\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/2.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-1.png\" alt=\"Configure Java\" width=\"500\" height=\"312\" title=\"\"><\/a><\/p>\n<h3 id=\"install-elasticsearch\">Install Elasticsearch<\/h3>\n<p>After installing Java, we will install the first component of the Elastic Stack, we will install the elasticsearch.<\/p>\n<p>Add the elastic stack key and add the elastic repository to the system.<\/p>\n<p class=\"command\">wget -qO &#8211; https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch | sudo apt-key add &#8211;<br \/>echo &#8220;deb https:\/\/artifacts.elastic.co\/packages\/6.x\/apt stable main&#8221; | sudo tee -a \/etc\/apt\/sources.list.d\/elastic-6.x.list<\/p>\n<p>Now update the repository and install the elasticsearch package using the command below.<\/p>\n<p class=\"command\">sudo apt update<br \/>sudo apt install elasticsearch -y<\/p>\n<p>After the installation is complete, go to the &#8216;\/etc\/elasticsearch&#8217; directory and edit the configuration file &#8216;elasticsearch.yml&#8217;.<\/p>\n<p class=\"command\">cd \/etc\/elasticsearch\/<br \/>vim elasticsearch.yml<\/p>\n<p>Uncomment the &#8216;network.host&#8217; line and change the value to &#8216;localhost&#8217;, and uncomment the &#8216;http.port&#8217; line for the elasticsearch port configuration.<\/p>\n<pre>network.host: localhost&#13;\nhttp.port: 9200<\/pre>\n<p>Save and exit.<\/p>\n<p>Now start the elasticsearch service and enable it to launch every time on system boot.<\/p>\n<p class=\"command\">systemctl start elasticsearch<br \/>systemctl enable elasticsearch<\/p>\n<p><a class=\"fancybox\" id=\"img-3\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/3.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-2.png\" alt=\"Install Elasticsearch\" width=\"500\" height=\"105\" title=\"\"><\/a><\/p>\n<p>The elasticsearch is now up and running, check it using netstat command netstat and curl commands below.<\/p>\n<p class=\"command\">netstat -plntu<br \/>curl -XGET &#8216;localhost:9200\/?pretty&#8217;<\/p>\n<p>Now you will get the elasticsearch version &#8216;6.2.4&#8217; is running on the default port &#8216;9200&#8217;.<\/p>\n<p><a class=\"fancybox\" id=\"img-4\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/4.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-3.png\" alt=\"Check elasticsearch port\" width=\"500\" height=\"386\" title=\"\"><\/a><\/p>\n<p>The elasticsearch installation has been completed.<\/p>\n<h3 id=\"install-and-configure-kibana-dashboard\">Install and Configure Kibana Dashboard<\/h3>\n<p>The second component is a kibana Dashboard. We will install the Kibana dashboard from the elastic repository, and configure the kibana service to run on the localhost address.<\/p>\n<p>Install Kibana dashboard using the apt command below.<\/p>\n<p class=\"command\">sudo apt install kibana -y<\/p>\n<p>Now go to the &#8216;\/etc\/kibana&#8217; directory and edit the configuration file &#8216;kibana.yml&#8217;.<\/p>\n<p class=\"command\">cd \/etc\/kibana\/<br \/>vim kibana.yml<\/p>\n<p>Uncomment those lines &#8216;server.port&#8217;, &#8216;server.host&#8217;, and &#8216;elasticsearch.url&#8217;.<\/p>\n<pre>server.port: 5601&#13;\nserver.host: \"localhost\"&#13;\nelasticsearch.url: \"http:\/\/localhost:9200\"<\/pre>\n<p>Save and exit.<\/p>\n<p>Now start the kibana service and enable it to launch everytime at system boot.<\/p>\n<p class=\"command\">sudo systemctl enable kibana<br \/>sudo systemctl start kibana<\/p>\n<p>The kibana dashboard is now up and running on the &#8216;localhost&#8217; address and the default port &#8216;5601&#8217;. Check it using netstat command below.<\/p>\n<p class=\"command\">netstat -plntu<\/p>\n<p><a class=\"fancybox\" id=\"img-5\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/5.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-4.png\" alt=\"Install Kibana\" width=\"500\" height=\"306\" title=\"\"><\/a><\/p>\n<p>Kibana dashboard installation has been completed.<\/p>\n<h3 id=\"install-and-configure-nginx-as-reverseproxy-for-kibana\">Install and Configure Nginx as Reverse-Proxy for Kibana<\/h3>\n<p>In this tutorial, we will be using the Nginx web server as a reverse proxy for the Kibana Dashboard.<\/p>\n<p>Install Nginx and the &#8216;apache2-utils&#8217; packages to the system.<\/p>\n<p class=\"command\">sudo apt install nginx apache2-utils -y<\/p>\n<p>After the installation is complete, go to the &#8216;\/etc\/nginx&#8217; configuration directory and create new virtual host file named &#8216;kibana&#8217;.<\/p>\n<p class=\"command\">cd \/etc\/nginx\/<br \/>vim sites-available\/kibana<\/p>\n<p>Paste Nginx virtual host configuration below.<\/p>\n<pre>server {&#13; listen 80;&#13; &#13; server_name elastic-stack.io;&#13; &#13; auth_basic \"Restricted Access\";&#13; auth_basic_user_file \/etc\/nginx\/.kibana-user;&#13; &#13; location \/ {&#13; proxy_pass http:\/\/localhost:5601;&#13; proxy_http_version 1.1;&#13; proxy_set_header Upgrade $http_upgrade;&#13; proxy_set_header Connection 'upgrade';&#13; proxy_set_header Host $host;&#13; proxy_cache_bypass $http_upgrade;&#13; }&#13;\n}<\/pre>\n<p>Save and exit.<\/p>\n<p>Next, we will create new basic authentication web server for accessing the Kibana dashboard. We will create the basic authentication using the htpasswd command as below.<\/p>\n<p class=\"command\">sudo htpasswd -c \/etc\/nginx\/.kibana-user elastic<br \/>Type the elastic user password<\/p>\n<p>Activate the kibana virtual host and test all nginx configuration.<\/p>\n<p class=\"command\">ln -s \/etc\/nginx\/sites-available\/kibana \/etc\/nginx\/sites-enabled\/<br \/>nginx -t<\/p>\n<p>Make sure there is no error, now start the Nginx service and enable it to launch everytime at system boot.<\/p>\n<p class=\"command\">systemctl enable nginx<br \/>systemctl restart nginx<\/p>\n<p>Nginx installation and configuration as a Reverse-proxy for the Kibana dashboard have been completed.<\/p>\n<p><a class=\"fancybox\" id=\"img-6\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/6.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-5.png\" alt=\"Configure Nginx\" width=\"500\" height=\"281\" title=\"\"><\/a><\/p>\n<h3 id=\"install-and-configure-logstash\">Install and Configure Logstash<\/h3>\n<p>The last component for the Elastic Stack for this guide is the &#8216;Logstash&#8217;. We will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).<\/p>\n<p>Before installing logstash, make sure you check the OpenSSL Version your server.<\/p>\n<p class=\"command\">openssl version -a<\/p>\n<p><a class=\"fancybox\" id=\"img-7\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/7.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-6.png\" alt=\"Install and Configure Logstash\" width=\"500\" height=\"94\" title=\"\"><\/a><\/p>\n<p>For this guide, we will be using the OpenSSL &#8216;1.0.2o&#8217;. If you&#8217;re still using the OpenSSL version 1.1.2, you will get an error at the logstash and filebeat SSL connection.<\/p>\n<p>Install logstash using the apt command below.<\/p>\n<p class=\"command\">sudo apt install logstash -y<\/p>\n<p>After the installation is complete, we will generate the SSL certificate key to secure the log data transfer from the client filebeat to the logstash server.<\/p>\n<p>Edit the &#8216;\/etc\/hosts&#8217; file using <a href=\"https:\/\/www.howtoforge.com\/vim-basics\" target=\"_blank\" rel=\"noopener noreferrer\">vim<\/a>.<\/p>\n<p class=\"command\">vim \/etc\/hosts<\/p>\n<p>Add the configuration below.<\/p>\n<pre>10.0.15.10 elk-master elk-master<\/pre>\n<p>Save and exit.<\/p>\n<p>Now create new SSL directory under the logstash configuration directory &#8216;\/etc\/logstash&#8217; and go to that directory.<\/p>\n<p class=\"command\">mkdir -p \/etc\/logstash\/ssl<br \/>cd \/etc\/logstash\/<\/p>\n<p>Generate the SSL certificate for Logstash using the openssl command as below.<\/p>\n<p class=\"command\">openssl req -subj &#8216;\/CN=elk-master\/&#8217; -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout ssl\/logstash-forwarder.key -out ssl\/logstash-forwarder.crt<\/p>\n<p>The SSL certificate files for Logstash has been created on the &#8216;\/etc\/logstash\/ssl&#8217; directory.<\/p>\n<p>Next, we will create new configuration files for logstash. We will create a configuration file &#8216;filebeat-input.conf&#8217; as input file from filebeat, &#8216;syslog-filter.conf&#8217; for syslog processing, and then a &#8216;output-elasticsearch.conf&#8217; file to define the Elasticsearch output.<\/p>\n<p>Go to the logstash configuration directory and create the new configuration files &#8216;filebeat-input.conf&#8217; in the &#8216;conf.d&#8217; directory.<\/p>\n<p class=\"command\">cd \/etc\/logstash\/<br \/>vim conf.d\/filebeat-input.conf<\/p>\n<p>Paste the following configuration there.<\/p>\n<pre>input {&#13; beats {&#13; port =&gt; 5443&#13; type =&gt; syslog&#13; ssl =&gt; true&#13; ssl_certificate =&gt; \"\/etc\/logstash\/ssl\/logstash-forwarder.crt\"&#13; ssl_key =&gt; \"\/etc\/logstash\/ssl\/logstash-forwarder.key\"&#13; }&#13;\n}<\/pre>\n<p>Save and exit.<\/p>\n<p>For the syslog processing log data, we are using the filter plugin named &#8216;grok&#8217; to parse the syslog files.<\/p>\n<p>Create a new configuration &#8216;syslog-filter.conf&#8217;.<\/p>\n<p class=\"command\">vim conf.d\/syslog-filter.conf<\/p>\n<p>Paste the following configuration there.<\/p>\n<pre>filter {&#13; if [type] == \"syslog\" {&#13; grok {&#13; match =&gt; { \"message\" =&gt; \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\" }&#13; add_field =&gt; [ \"received_at\", \"%{@timestamp}\" ]&#13; add_field =&gt; [ \"received_from\", \"%{host}\" ]&#13; }&#13; date {&#13; match =&gt; [ \"syslog_timestamp\", \"MMM d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]&#13; }&#13; }&#13;\n}<\/pre>\n<p>Save and exit.<\/p>\n<p>And for the elasticsearch output, we will create the configuration file named &#8216;output-elasticsearch.conf&#8217;.<\/p>\n<p class=\"command\">vim conf.d\/output-elasticsearch.conf<\/p>\n<p>Paste the following configuration there.<\/p>\n<pre>output {&#13; elasticsearch { hosts =&gt; [\"localhost:9200\"]&#13; hosts =&gt; \"localhost:9200\"&#13; manage_template =&gt; false&#13; index =&gt; \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"&#13; document_type =&gt; \"%{[@metadata][type]}\"&#13; }&#13;\n}<\/pre>\n<p>Save and exit.<\/p>\n<p>When this is done, start the logstash service and enable it to launch everytime at system boot.<\/p>\n<p class=\"command\">sudo systemctl enable logstash<br \/>sudo systemctl start logstash<\/p>\n<p><a class=\"fancybox\" id=\"img-8\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/8.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-7.png\" alt=\"Enable and start logstah\" width=\"500\" height=\"158\" title=\"\"><\/a><\/p>\n<p>Check the logstash service using netstat and systemctl commands below.<\/p>\n<p class=\"command\">netstat -plntu<br \/>systemctl status logstash<\/p>\n<p>And the logstash service is now up and running. Running on the public IP address with port &#8216;5443&#8217;.<\/p>\n<p><a class=\"fancybox\" id=\"img-9\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/9.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-8.png\" alt=\"Check ports\" width=\"500\" height=\"319\" title=\"\"><\/a><\/p>\n<p>The Elastic Stack installation has been completed.<\/p>\n<h2 id=\"step-install-and-configure-filebeat-on-ubuntu-\">Step 2 &#8211; Install and Configure Filebeat on Ubuntu 18.04<\/h2>\n<p>In this step, we will configure the Ubuntu 18.04 client &#8216;elk-client01&#8217; by installing the Elastic Beats data shippers &#8216;Filebeat&#8217; on it.<\/p>\n<p>Before installing the filebeat to the system, we need to edit the &#8216;\/etc\/hosts&#8217; and download the logstash certificate file &#8216;logstash-forwarder.crt&#8217; file to the &#8216;elk-client01&#8217; server.<\/p>\n<p>Edit the &#8216;\/etc\/hosts&#8217; file using vim editor.<\/p>\n<p class=\"command\">vim \/etc\/hosts<\/p>\n<p>Paste the following configuration there.<\/p>\n<pre>10.0.15.10 elk-master elk-master<\/pre>\n<p>Save and exit.<\/p>\n<p>Copy the logstash certificate file &#8216;logstash-forwarder.crt&#8217; using scp command.<\/p>\n<p class=\"command\">scp <a href=\"https:\/\/www.howtoforge.com\/cdn-cgi\/l\/email-protection\" class=\"__cf_email__\" data-cfemail=\"24564b4b506441484f09494557504156\" target=\"_blank\" rel=\"noopener\">[email\u00a0protected]<\/a>:\/etc\/logstash\/ssl\/logstash-forwarder.crt .<\/p>\n<p><a class=\"fancybox\" id=\"img-10\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/10.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-9.png\" alt=\"Install Filebeat\" width=\"500\" height=\"196\" title=\"\"><\/a><\/p>\n<p>Next, install the Elastic Beats &#8216;Filebeat&#8217; by adding the elastic key and add the elastic repository.<\/p>\n<p class=\"command\">wget -qO &#8211; https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch | sudo apt-key add &#8211;<br \/>echo &#8220;deb https:\/\/artifacts.elastic.co\/packages\/6.x\/apt stable main&#8221; | sudo tee -a \/etc\/apt\/sources.list.d\/elastic-6.x.list<\/p>\n<p>Update the repository and install the &#8216;filebeat&#8217; package using the apt command below.<\/p>\n<p class=\"command\">sudo apt update<br \/>sudo apt install filebeat -y<\/p>\n<p>After the installation is complete, go to the &#8216;\/etc\/filebeat&#8217; directory and edit the configuration file &#8216;filebeat.yml&#8217;.<\/p>\n<p class=\"command\">cd \/etc\/filebeat\/<br \/>vim filebeat.yml<\/p>\n<p>Now enable the filebeat prospectors by changing the &#8216;enabled&#8217; line value to &#8216;true&#8217;.<\/p>\n<pre> enabled: true<\/pre>\n<p>Define system log files to be sent to the logstash server. For this guide, we will add the ssh log file &#8216;auth.log&#8217; and the syslog file.<\/p>\n<pre> paths:&#13; - \/var\/log\/auth.log&#13; - \/var\/log\/syslog<\/pre>\n<p>Setup the output to logstash by commenting the default &#8216;elasticsearch&#8217; output and uncomment the logstash output line as below.<\/p>\n<pre>output.logstash:&#13; # The Logstash hosts&#13; hosts: [\"elk-master:5443\"]&#13; ssl.certificate_authorities: [\"\/etc\/filebeat\/logstash-forwarder.crt\"]<\/pre>\n<p>Save and exit.<\/p>\n<p>Next, we need to edit the &#8216;filebeat.reference.yml&#8217; file to enable filebeat modules, and we will enable the &#8216;syslog&#8217; module.<\/p>\n<p class=\"command\">vim filebeat.reference.yml<\/p>\n<p>Enable the syslog system module for filebeat as below.<\/p>\n<pre>- module: system&#13; # Syslog&#13; syslog:&#13; enabled: true<\/pre>\n<p>Save and exit.<\/p>\n<p>Copy the logstash certificate file &#8216;logstash-forwarder.crt&#8217; to the &#8216;\/etc\/filebeat&#8217; directory.<\/p>\n<p class=\"command\">cp ~\/logstash-forwarder.crt \/etc\/filebeat\/logstash-forwarder.crt<\/p>\n<p>Filebeat installation and configuration have been completed. Now start the filebeat service and enable it to launch every time at system boot.<\/p>\n<p class=\"command\">systemctl start filebeat<br \/>systemctl enable filebeat<\/p>\n<p><a class=\"fancybox\" id=\"img-11\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/11.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-10.png\" alt=\"Configure Filebeat\" width=\"500\" height=\"130\" title=\"\"><\/a><\/p>\n<p>Check the filebeat service using commands below.<\/p>\n<p class=\"command\">systemctl status filebeat<br \/>tail -f \/var\/log\/filebeat\/filebeat<\/p>\n<p>The filebeat shippers are up and running under the Ubuntu 18.04 server.<\/p>\n<p><a class=\"fancybox\" id=\"img-12\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/12.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-11.png\" alt=\"Filebeat is up and running\" width=\"500\" height=\"200\" title=\"\"><\/a><\/p>\n<h2 id=\"step-install-and-configure-filebeat-on-centos-\">Step 3 &#8211; Install and Configure Filebeat on CentOS 7.5<\/h2>\n<p>In this step, we will configure the CentOS 7.5 client &#8216;elk-client02&#8217; by installing the Elastic Beats data shippers &#8216;Filebeat&#8217; on it.<\/p>\n<p>Before installing the Filebeat to the system, we need to edit the &#8216;\/etc\/hosts&#8217; and download the logstash certificate file &#8216;logstash-forwarder.crt&#8217; file to the &#8216;elk-client02&#8217; server.<\/p>\n<p>Edit the &#8216;\/etc\/hosts&#8217; file using vim.<\/p>\n<p class=\"command\">vim \/etc\/hosts<\/p>\n<p>Paste configuration below.<\/p>\n<pre>10.0.15.10 elk-master elk-master<\/pre>\n<p>Save and exit.<\/p>\n<p>Copy the logstash certificate file &#8216;logstash-forwarder.crt&#8217; using scp command.<\/p>\n<p class=\"command\">scp <a href=\"https:\/\/www.howtoforge.com\/cdn-cgi\/l\/email-protection\" class=\"__cf_email__\" data-cfemail=\"addfc2c2d9edc8c1c680c0ccded9c8df\" target=\"_blank\" rel=\"noopener\">[email\u00a0protected]<\/a>:\/etc\/logstash\/ssl\/logstash-forwarder.crt .<\/p>\n<p><a class=\"fancybox\" id=\"img-13\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/13.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-12.png\" alt=\"Install Filebeat on CentOS\" width=\"500\" height=\"202\" title=\"\"><\/a><\/p>\n<p>Next, install the Elastic Beats &#8216;Filebeat&#8217; by adding the elastic key and add the elastic repository.<\/p>\n<p class=\"command\">rpm &#8211;import https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch<\/p>\n<p>cat &lt;&lt;EOF &gt; \/etc\/yum.repos.d\/elastic.repo<br \/>[elasticsearch-6.x]name=Elasticsearch repository for 6.x packages<br \/>baseurl=https:\/\/artifacts.elastic.co\/packages\/6.x\/yum<br \/>gpgcheck=1<br \/>gpgkey=https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch<br \/>enabled=1<br \/>autorefresh=1<br \/>type=rpm-md<br \/>EOF<\/p>\n<p>Install filebeat using the yum command below.<\/p>\n<p class=\"command\">yum install filebeat -y<\/p>\n<p>After the installation is complete, go to the &#8216;\/etc\/filebeat&#8217; directory and edit the configuration file &#8216;filebeat.yml&#8217;.<\/p>\n<p class=\"command\">cd \/etc\/filebeat\/<br \/>vim filebeat.yml<\/p>\n<p>Now enable the filebeat prospectors by change the &#8216;enabled&#8217; line value to &#8216;true&#8217;.<\/p>\n<pre> enabled: true<\/pre>\n<p>Define system log files to be sent to the logstash server. For this guide, we will add the ssh log file &#8216;auth.log&#8217; and the syslog file.<\/p>\n<pre> paths:&#13; - \/var\/log\/secure&#13; - \/var\/log\/messages<\/pre>\n<p>Setup the output to logstash by commenting the default &#8216;elasticsearch&#8217; output and uncomment the logstash output line as below.<\/p>\n<pre>output.logstash:&#13; # The Logstash hosts&#13; hosts: [\"elk-master:5443\"]&#13; ssl.certificate_authorities: [\"\/etc\/filebeat\/logstash-forwarder.crt\"]<\/pre>\n<p>Save and exit.<\/p>\n<p>Next, we need to edit the &#8216;filebeat.reference.yml&#8217; file to enable filebeat modules, and we will enable the &#8216;syslog&#8217; module.<\/p>\n<p class=\"command\">vim filebeat.reference.yml<\/p>\n<p>Enable the syslog system module for filebeat as below.<\/p>\n<pre>- module: system&#13; # Syslog&#13; syslog:&#13; enabled: true<\/pre>\n<p>Save and exit.<\/p>\n<p>Copy the logstash certificate file &#8216;logstash-forwarder.crt&#8217; to the &#8216;\/etc\/filebeat&#8217; directory.<\/p>\n<p class=\"command\">cp ~\/logstash-forwarder.crt \/etc\/filebeat\/logstash-forwarder.crt<\/p>\n<p>Filebeat installation and configuration have been completed. Now start the filebeat service and add it to the boot time.<\/p>\n<p class=\"command\">systemctl start filebeat<br \/>systemctl enable filebeat<\/p>\n<p><a class=\"fancybox\" id=\"img-14\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/14.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-13.png\" alt=\"Configure Filebeat\" width=\"500\" height=\"134\" title=\"\"><\/a><\/p>\n<p>Check the filebeat service using commands below.<\/p>\n<p class=\"command\">systemctl status filebeat<br \/>tail -f \/var\/log\/filebeat\/filebeat<\/p>\n<p>The filebeat shippers are up and running under the CentOS 7.5 server.<\/p>\n<p><a class=\"fancybox\" id=\"img-15\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/15.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-14.png\" alt=\"Filebeat is running\" width=\"500\" height=\"247\" title=\"\"><\/a><\/p>\n<h2 id=\"step-testing\">Step 4 &#8211; Testing<\/h2>\n<p>Open your web browser and type the elastic stack domain name, mine is: &#8216;elastic-stack.io&#8217;.<\/p>\n<p>You will be prompted the username and password from the basic authentication to the Kibana Dashboard.<\/p>\n<p><a class=\"fancybox\" id=\"img-16\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/16.png\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-15.png\" alt=\"\" title=\"\"><\/a><\/p>\n<p>Type the username &#8216;elastic&#8217; with your password.<\/p>\n<p>Now you will get the beautiful kibana dashboard, click the &#8216;Set up index patterns&#8217; button on the right.<\/p>\n<p><a class=\"fancybox\" id=\"img-17\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/17.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-16.png\" alt=\"Kibana dashboard\" width=\"500\" height=\"171\" title=\"\"><\/a><\/p>\n<p>Define the &#8216;filebeat-*&#8217; index pattern and click the &#8216;Next step&#8217; button.<\/p>\n<p><a class=\"fancybox\" id=\"img-18\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/18.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-17.png\" alt=\"Define a filebeat\" width=\"500\" height=\"236\" title=\"\"><\/a><\/p>\n<p>For the &#8216;time filter field name&#8217;, choose the &#8216;@timestamp&#8217; and click &#8216;Create index pattern&#8217;.<\/p>\n<p><a class=\"fancybox\" id=\"img-19\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/19.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-18.png\" alt=\"Create index\" width=\"500\" height=\"251\" title=\"\"><\/a><\/p>\n<p>And the filebeat index pattern has been created.<\/p>\n<p><a class=\"fancybox\" id=\"img-20\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/20.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-19.png\" alt=\"Filebeat index pattern created\" width=\"500\" height=\"290\" title=\"\"><\/a><\/p>\n<p>Next, we will try to get the log information for the SSH login failed on each client servers &#8216;elk-client01&#8217; Ubuntu system and &#8216;elk-client02&#8217; CentOS system.<\/p>\n<p>Inside the Kibana Dashboard, click the &#8216;Discover&#8217; menu to get all server logs.<\/p>\n<p>Set the &#8216;beat.hostname&#8217; to the &#8216;elk-client01&#8217; server, the &#8216;source&#8217; is the &#8216;\/var\/log\/auth.log&#8217; file, and you will get the result as shown below.<\/p>\n<p><a class=\"fancybox\" id=\"img-21\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/21.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-20.png\" alt=\"Log result\" width=\"500\" height=\"273\" title=\"\"><\/a><\/p>\n<p>And following is the sample log details for SSH failed password from the &#8216;auth.log&#8217; file.<\/p>\n<p><a class=\"fancybox\" id=\"img-22\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/22.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-21.png\" alt=\"SSH failed login\" width=\"500\" height=\"289\" title=\"\"><\/a><\/p>\n<p>For the &#8216;elk-client02&#8217; CentOS server, set the &#8216;beat.hostname&#8217; to the &#8216;elk-client02&#8217; server, the &#8216;source&#8217; is the &#8216;\/var\/log\/secure&#8217; file, and you will get the result as shown below.<\/p>\n<p><a class=\"fancybox\" id=\"img-23\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/23.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-22.png\" alt=\"Report for server 2\" width=\"500\" height=\"273\" title=\"\"><\/a><\/p>\n<p>And following is the sample log details for SSH failed password from the &#8216;secure&#8217; file.<\/p>\n<p><a class=\"fancybox\" id=\"img-24\" href=\"https:\/\/www.howtoforge.com\/images\/how_to_install_elastic_stack_ubuntu_1804\/big\/24.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-23.png\" alt=\"Login report\" width=\"500\" height=\"289\" title=\"\"><\/a><\/p>\n<p>The Elastic Stack and the Elastic Beat &#8216;Filebeat&#8217; installation and configuration have been completed successfully.<\/p>\n<h2 id=\"reference\">Reference<\/h2>\n<div>\n<p><b>Share this page:<\/b><\/p>\n<p>\n<a href=\"https:\/\/www.facebook.com\/sharer.php?u=https%3A%2F%2Fwww.howtoforge.com%2Ftutorial%2Fubuntu-elastic-stack%2F\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-24.png\" height=\"20\" alt=\"\" title=\"\"><\/a><br \/>\n<a href=\"https:\/\/twitter.com\/intent\/tweet?url=https%3A%2F%2Fwww.howtoforge.com%2Ftutorial%2Fubuntu-elastic-stack%2F&amp;text=How+to+Install+Elastic+Stack+on+Ubuntu+18.04+LTS&amp;via=howtoforgecom&amp;related=howtoforgecom\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-25.png\" height=\"20\" alt=\"\" title=\"\"><\/a><br \/>\n<a href=\"https:\/\/twitter.com\/howtoforgecom\/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-26.png\" height=\"20\" alt=\"\" title=\"\"><\/a><br \/>\n<a href=\"https:\/\/plus.google.com\/share?url=https%3A%2F%2Fwww.howtoforge.com%2Ftutorial%2Fubuntu-elastic-stack%2F\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/07\/how-to-install-elastic-stack-on-ubuntu-18-04-lts-27.png\" height=\"20\" alt=\"\" title=\"\"><\/a>\n<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Elasticsearch is an open source search engine based on Lucene, developed in Java. It provides a distributed and multitenant full-text search engine with an HTTP Dashboard web-interface (Kibana). The data is queried, retrieved and stored in a JSON document scheme. Elasticsearch is a scalable search engine that can be used to search for all kind &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[36],"tags":[],"class_list":["post-5693","post","type-post","status-publish","format-standard","hentry","category-36"],"_links":{"self":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/posts\/5693","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/comments?post=5693"}],"version-history":[{"count":0,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/posts\/5693\/revisions"}],"wp:attachment":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/media?parent=5693"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/categories?post=5693"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/tags?post=5693"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}