[Day 11] 2. LAB Logstash and Kibana

1. Install the File

Install the required package for Logstash or Kibana using the package file provided.

1. Download the file from the server using

   Copy

   wget

   .

2. Install the package:

   bashCopy
   sudo dpkg -i <package_name>
   

2. Enable Logstash

Ensure Logstash is enabled and running on system boot.

1. Start and enable Logstash:

   bashCopy
   sudo systemctl enable --now logstash
   sudo systemctl status logstash
   

3. Edit the Configuration File

Create or modify a Logstash configuration file to specify input and output behavior.

1. Open or create the configuration file:

   bashCopy
   vim pipe1.conf
   

   • You can name the file anything, but it must have the
     Copy

     .conf

     extension.

2. Add the following configuration:

   plaintextCopy
   input {
       stdin {}
   }
   
   output {
       stdout { codec => rubydebug }
   }
   

3. Test the configuration:

   bashCopy
   echo "Hello World" | sudo /usr/share/logstash/bin/logstash -f /home/shuhari/pipe1.conf
   

4. Writing the Transform Section

Modify the Logstash configuration file to transform input data.

1. Open the configuration file again:

   bashCopy
   vim pipe1.conf
   

2. Add a filter block to modify the input:

   plaintextCopy
   input {
       stdin {}
   }
   
   filter {
       mutate { uppercase => ["message"] }
   }
   
   output {
       stdout { codec => rubydebug }
   }
   

3. Run the command again to verify the transformed output:

   bashCopy
   echo "Hello World" | sudo /usr/share/logstash/bin/logstash -f /home/shuhari/pipe1.conf
   

5. Checking the Logs of

Copy

apt

View the history of package downloads using

apt

1. Run the following command to check the logs:

   bashCopy
   sudo tail /var/log/apt/history.log
   

6. Processing

Copy

apt

Logs with Logstash

Configure Logstash to process

apt

1. Navigate to the Logstash configuration directory and create a new file:

   bashCopy
   cd /etc/logstash/conf.d/
   sudo vim one.conf
   

2. Add the following configuration:

   plaintextCopy
   input {
       file {
           path => "/var/log/apt/history.log"
           start_position => "end"
           sincedb_path => "/var/lib/logstash/apt.sincedb"
       }
   }
   
   filter {
       if [message] =~ /Commandline:/ {
           mutate {
               add_field => { "installation status" => "installed" }
           }
       }
       else {
           drop { }
       }
   }
   
   output {
       elasticsearch {
           hosts => ["http://localhost:9200"]
           index => "apt_logs"
       }
       stdout { codec => rubydebug }
   }
   

3. Restart Logstash to apply the configuration:

   bashCopy
   sudo systemctl restart logstash
   

4. Monitor Logstash in a separate terminal:

   bashCopy
   sudo journalctl -f -n 0 -u logstash
   

5. Install a package to see output in the terminal.

6. Verify the Elasticsearch index:

   bashCopy
   curl -X GET "localhost:9200/_aliases?pretty=true"
   

7. Installing Kibana

Kibana is a visualization tool designed for Elasticsearch. It helps you create interactive and dynamic dashboards to visually represent data stored in Elasticsearch. With Kibana, you can create various charts, graphs, maps, and other visualizations that make it easier to understand and analyze your data. It's widely used for monitoring and data exploration. In short, Kibana turns your raw data into visual insights!

Install and configure Kibana to visualize data.

1. Download the Kibana installation file (provided by the instructor).

2. Install Kibana:

   bashCopy
   sudo dpkg -i <package_name>
   

3. Check the Kibana status:

   bashCopy
   sudo systemctl status kibana
   

4. If Kibana is not running, edit the configuration file:

   bashCopy
   sudo chmod 755 /etc/kibana
   sudo vim /etc/kibana/kibana.yml
   

   Add or modify the following lines:

   plaintextCopy
   server.port: 5601
   server.host: "0.0.0.0"
   

5. Enable and start Kibana:

   bashCopy
   sudo systemctl enable --now kibana
   sudo systemctl status kibana
   

6. Access Kibana on your browser:

   plaintextCopy
   http://<ip>:5601
   

8. Kibana on Browser

Explore Kibana features to analyze your data visually.

1. Open the Kibana dashboard in your browser.
2. Navigate through the interface and explore its tools for data visualization.