[Day 10] 4. Elasticsearch Logstash Kibana (ELK)

What is ELK?

ELK stands for Elasticsearch, Logstash, and Kibana. It is a set of tools used for searching, analyzing, and visualizing large volumes of data, primarily logs.

Elasticsearch: It is a distributed, RESTful search and analytics engine. It is designed for horizontal scalability and real-time search and analysis of large amounts of data. It forms the core of the ELK stack, providing powerful search, aggregation, and data visualization capabilities.
Logstash: This is a server-side data processing pipeline that ingests data from various sources simultaneously, transforms it, and then sends it to your preferred “stash,” such as Elasticsearch. It handles data parsing, enriching, and formatting before it's stored.
Kibana: This tool provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. It enables users to create and share dynamic dashboards that include charts, graphs, and other visualizations to analyze and gain insights from their data.

Threat Hunting using ELK:

In the context of security, threat hunting with ELK involves querying, analyzing, and visualizing logs from various sources (like network traffic, system events, etc.) to identify potential threats or unusual activity in your environment.

Lab Setup for Elasticsearch:

OS Requirements: A recommended 8 GB of RAM (minimum 4 GB).

Time Zone: Set the correct time zone since we will be checking logs.

bash
      sudo dpkg-reconfigure tzdata

Select

Asia

first and then

Kolkata

Elasticsearch Components:
- Index: Equivalent to a database or collection of documents.
- Documents: Individual data units inside an index.
- Shards: Indices are divided into shards, distributed across multiple nodes.
- Replicas: Copies of shards for fault tolerance and high availability.

Steps for Setting Up Elasticsearch:

1. Install Java Development Kit (JDK):

Elasticsearch requires Java. Install the JDK:

bash
java -version
sudo apt-get install -y default-jdk curl
java -version

2. Download and Install Elasticsearch:

Download the Elasticsearch package from the server and install it:

bash
sudo dpkg -i <package_name>

After installation, note down the password (important for later).

3. Check Elasticsearch Status:

Check if Elasticsearch is running:

bash
sudo systemctl status elasticsearch

4. Edit the Elasticsearch Configuration File:

Modify the configuration file to set up the cluster and node. But before that we need to give the edit permissions to that file.

bash
sudo chmod 755 /etc/elasticsearch
sudo vim /etc/elasticsearch/elasticsearch.yml

Uncomment and modify the following lines:

plaintext
cluster.name: my-application
node.name: node-1
network.host: 0.0.0.0

Uncomment and set:

plaintext
cluster.initial_master_nodes: ["node-1"]

Set all 4 occurrences of the following security options to

false

(search for

xpack.security

): Make these changes:

plaintext
xpack.security.enabled: false
xpack.security.enrollment.enabled: false

# below xpack.security.http.ssl:
enabled: false

# below xpack.security.transport.ssl
enabled: false

Comment this line:

plaintext
cluster.initial_master_nodes: ["debian"]

5. Reload the Daemon:

Reload the systemd service manager and ensure Elasticsearch starts automatically:

bash
sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch
sudo systemctl status elasticsearch

6. Run Elasticsearch:

Access Elasticsearch through the following URL:

plaintext
http://<ip>:9200

7. Check Cluster Health:

Use

curl

to check the health of your Elasticsearch cluster:

bash
curl -X GET "localhost:9200/_cluster/health?pretty=true"

8. List and Create an Index:

List all existing indices and create a new index:

bash
curl -X GET "localhost:9200/_aliases?pretty=true"
curl -X PUT "localhost:9200/test-index1"
curl -X GET "localhost:9200/_aliases?pretty=true"

9. Create a Document:

Add a document to the

test-index1

index:

bash
curl -X POST "localhost:9200/test-index1/_doc/1?pretty" -H 'Content-type: application/json' -d'
{
  "name": "Gabbar Singh",
  "age": 30,
  "occupation": "hacker"
}
'

To view the document:

bash
curl -X GET "localhost:9200/test-index1/_doc/1?pretty=true"