Blogs

• CentOS 7 64 bit with 4GB RAM
• Elastic preinstalled on a machine
• Unlock the ports for Elastic in the firewall:

1. sudo firewall-cmd --zone=public --add-port=9200/tcp -permanent
2. sudo firewall-cmd --zone=public --add-port=9300/tcp --permanent
3. sudo firewall-cmd -reload

Key

Meaning of the colors in this guide:

1. elasticsearch.repo → Text written in red is a free-form naming convention, whereas text written in black is not free-form.
2. code examples appear in gray boxes
3. an http link with curl command or import command is to be executed in the VM/Moba. Without curl in any web browser.
4. this manual has CentOS7 preinstalled. The directories and commands may vary from OS to OS. In the case with the directory/command is shown as blue text, otherwise as orange text.

If not working on CentOS7, these links will help:

https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-elastic-stack.html#install-elasticsearch

https://www.elastic.co/guide/en/logstash/7.6/dir-layout.html

https://www.elastic.co/guide/en/logstash/7.6/running-logstash.html

Step 1 - Install Java/OpenJDK

Java version 8 or 11 is required for the installation:

yum install java-1.8.0-openjdk-devel -y

Step 2 - Verify that was installed correctly

java -version

openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

Step 3 - Creating the LogStack repo

First:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch 

and create the repo

vi /etc/yum.repos.d/logstash.repo

Then copy the following into it:

[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Step 4 - Install and launch Logstack

The command to install is:

sudo yum install logstash -y

Start Logstack:

sudo systemctl enable logstash

sudo systemctl start logstash

Step 5 - Check if everything worked out

journalctl –unit logstash

localhost.localdomain systemd[1]: Started logstash.

Example - Simple Output

Note: From now on the IP 192.168.20.100 will be entered, this must always be adjusted to the respective own IP of the VM on which Elasticsearch is installed.

Execute the following command:

/usr/share/logstash/bin/logstash -e 'input { stdin { } } output {
elasticsearch { hosts => "192.168.20.100:9200" } }'

Now wait for the following output:

[Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}

After that you can just start typing. The output can be seen in the browser at http://192.168.20.100:9200/_search?pretty.

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 9,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "logstash-2020.05.11-000001",
        "_type" : "_doc",
        "_id" : "-hDWAnIB5PIPaJRIY6Ll",
        "_score" : 1.0,
        "_source" : {
          "message" : "Hallo",
          "@timestamp" : "2020-05-11T08:25:03.941Z",
          "host" : "localhost.localdomain",
          "@version" : "1"
        }
      },

Example - Simple output to index

After creating an index logs in Elasticsearch, you can also move the output to the index by:

/usr/share/logstash/bin/logstash -e 'input { stdin { } } output { elasticsearch { hosts => "192.168.20.100:9200" index => "logs" } }'

Now wait for the following output:

[Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}

After that you can just start typing. The output can then be seen in the browser at http://192.168.20.100:9200/logs/_search?pretty

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "logs",
        "_type" : "_doc",
        "_id" : "4usEA3IBWUPpcn4FpXBV",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost.localdomain",
          "@timestamp" : "2020-05-11T09:15:35.106Z",
          "@version" : "1",
          "message" : "Hi"
        }
      },

Example - Send file change to index

After creating an index logs in Elasticsearch, you can also move the output to the index by:

/usr/share/logstash/bin/logstash -e 'input { file { path => "/etc/resolv.conf"
start_position => beginning } } output { elasticsearch { hosts =>
"192.168.20.100:9200" index => "logs" } }'

Now wait for the following output:

[Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}

Now you can change something in the folder /etc/resolv.conf and see the change
in the browser at http://192.168.20.100:9200/logs/_search?pretty.

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 16,
      "relation" : "eq"
    },
      {
        "_index" : "logs",
        "_type" : "_doc",
        "_id" : "VcKTA3IByzccBleDawQ9",
        "_score" : 1.0,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-05-11T11:51:32.291Z",
          "path" : "/etc/resolv.conf",
          "message" : "# Generated by NetworkManager Gentian Rrafshi",
          "host" : "localhost.localdomain"
        }

Example - Sending data changes via configfile in index

After creating an index logs in Elasticsearch, create a configfile elastic.conf in the subfolder /etc/logstash/conf.d/elastic.conf,
which looks like this:

eht:

input {
    file {
        path => "/etc/resolv.conf" 
        start_position => beginning
    }
}
output {
    elasticsearch {
        hosts => ["192.168.20.100:9200"]
        index => "logs"
   }
}

Test if the config fits:

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

Now execute the following command:

/usr/share/logstash/bin/logstash -f elastic.conf --path.settings=/etc/logstash/

Now wait for the following output:

[logstasg.agent] Successfully started Logstash API endpoint {:port=>9600}

Now you can change something in the folder /etc/resolv.conf and save it.
The change can then be seen in the browser at http://192.168.20.100:9200/logs/_search?pretty

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "logs",
        "_type" : "_doc",
        "_id" : "bsITBHIByzccBleDHATP",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2020-05-11T14:11:00.834Z",
          "path" : "/etc/resolv.conf",
          "host" : "localhost.localdomain",
          "message" : "# Generated by Networkmanager Gentian Rrafshi",
          "@version" : "1"
        }

Important!

After each run systemctl restart logstash before!

You can display a list of all available indexes with this command: http://192.168.20.100:9200/_cat/indices