Elasticsearch ELK Docker

이전 페이지에서 docker-compose를 통해 elasticsearch와 kibana 8.5.2버전을 설치하고 동작하는 실습을 하였다.
Docker - ES, Kibana

이번에는 ELK중 나머지 Logstash도 연동하고 curl을 통해서 실제 색인까지 해보도록 한다.( 매번 되어 있는것만 보았으니 신규버전에 맞게 다시!! )
추후 filebeat로도 하는 테스트를 진행 해보자

docker-compose

기존에 만들었던 docker-compose.yml파일에 logstash 설정을 추가해 보자
docker-compose.yml

logstash:
    image: docker.elastic.co/logstash/logstash:8.5.2
    container_name: logstash
    ports:
      - 5000:5000
      - 9600:9600
    environment:
      - node.name=logstash
      - xpack.monitoring.enabled=false
      - xpack.monitoring.elasticsearch.hosts=["http://es01:9200"]
      - "LS_JAVA_OPTS=-Xms1g -Xmx1g"
    volumes:
      - ./logstash/curl_test.conf:/usr/share/logstash/pipeline/curl_test.conf
    depends_on:
      - es01
    networks:
      - elastic

• image : docker pull을 수행하는 이미지 경로(8.5.2)
• ports : 사용하려는 포트 바인딩
  • 5000 : 테스트를 진행하려는 CURL 포트
  • 9600 : Logstash 접속 포트
• volumes : 테스트를 진행하려는 input/output정보가 담긴 conf파일을 docker 컨테이너에 삽입
  • config/pipeline.yml의 기본경로에 넣음 ::: path.config: "/usr/share/logstash/pipeline"

curl_test.conf

input {
	http {
		port  => "5000"
		codec => "json"
	}
}

filter { }

output {
	elasticsearch {
		hosts => "http://es01:9200"
		index => "curl_index"
	}
}

docker-compose run

docker-compose -f ./docker-compose.yml up -d

Test

Logstash status

curl -X GET "http://localhost:9600/_node?pretty

# Response
{
  "host" : "ce0c96e78f5a",
  "version" : "8.5.2",
  "http_address" : "0.0.0.0:9600",
  "id" : "21146675-8f39-4381-828a-1ae2f40082cf",
  "name" : "logstash",
  "ephemeral_id" : "0ab789ea-4d6e-40bc-b536-2b76393e386f",
  "status" : "green",
  "snapshot" : false,
  "pipeline" : {
    "workers" : 5,
    "batch_size" : 125,
    "batch_delay" : 50
  },
  "pipelines" : {
    "main" : {
      "ephemeral_id" : "344b5044-9be0-48bc-a3bb-9440a99a186e",
      "hash" : "1b977cc569e3058b42773b3ad0d228d8e84f2604998315bd91d95d55dca002c2",
      "workers" : 5,
      "batch_size" : 125,
      "batch_delay" : 50,
      "config_reload_automatic" : false,
      "config_reload_interval" : 3000000000,
      "dead_letter_queue_enabled" : false
    }
  },
  "os" : {
    "name" : "Linux",
    "arch" : "aarch64",
    "version" : "5.10.124-linuxkit",
    "available_processors" : 5
  },
  "jvm" : {
    "pid" : 1,
    "version" : "17.0.5",
    "vm_version" : "17.0.5",
    "vm_vendor" : "Eclipse Adoptium",
    "vm_name" : "OpenJDK 64-Bit Server VM",
    "start_time_in_millis" : 1670905154407,
    "mem" : {
      "heap_init_in_bytes" : 1073741824,
      "heap_max_in_bytes" : 1073741824,
      "non_heap_init_in_bytes" : 7667712,
      "non_heap_max_in_bytes" : 0
    },
    "gc_collectors" : [ "G1 Young Generation", "G1 Old Generation" ]
  }
}

curl post

curl -X POST -H "User-Agent: linux bla bla" -H "Content-Type: application/json" \
-d " \
{\
    \"message\": \"Input Curl test - POST\",\
    \"fields\": {\
        \"field_1\": \"first_field\",\
        \"field_2\": \"second_field\"\
    }\
}\
" \
http://localhost:5000

logstash log

[2022-12-13T04:28:01,225][INFO ][logstash.codecs.json     ][main][a8352c0603c86e4b9deb84e472193aca292b61d5771cf47c61baac1aa6ab135b] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
{
        "fields" => {
        "field_2" => "second_field",
        "field_1" => "first_field"
    },
       "message" => "Input Curl test - POST",
    "user_agent" => {
        "original" => "linux bla bla"
    },
           "url" => {
          "path" => "/",
        "domain" => "localhost",
          "port" => 5000
    },
    "@timestamp" => 2022-12-13T04:28:01.256882470Z,
          "host" => {
        "ip" => "172.22.0.1"
    },
          "http" => {
        "request" => {
            "mime_type" => "application/json",
                 "body" => {
                "bytes" => "129"
            }
        },
         "method" => "POST",
        "version" => "HTTP/1.1"
    },
      "@version" => "1",
         "event" => {
        "original" => " {    \"message\": \"Input Curl test - POST\",    \"fields\": {        \"field_1\": \"first_field\",        \"field_2\": \"second_field\"    }}"
    }
}

Kibana result

{
  "_index": "curl_index",
  "_id": "Zwq9CYUBqRhxJ0tQVcvj",
  "_score": 1,
  "_source": {
    "fields": {
      "field_2": "second_field",
      "field_1": "first_field"
    },
    "message": "Input Curl test - POST",
    "user_agent": {
      "original": "linux bla bla"
    },
    "url": {
      "path": "/",
      "domain": "localhost",
      "port": 5000
    },
    "@timestamp": "2022-12-13T04:28:01.256882470Z",
    "host": {
      "ip": "172.22.0.1"
    },
    "http": {
      "request": {
        "mime_type": "application/json",
        "body": {
          "bytes": "129"
        }
      },
      "method": "POST",
      "version": "HTTP/1.1"
    },
    "@version": "1",
    "event": {
      "original": """ {    "message": "Input Curl test - POST",    "fields": {        "field_1": "first_field",        "field_2": "second_field"    }}"""
    }
  }
}