[ELK Stack] ELK Stack 설치 및 연동

개요

• Data -> Logstash -> Elasticsearch
• ELK Stack 개발

Set Up

• 자원

	CPU	MEM	Program
우분투	2	4	ElasticSearch, Kibana
우분투	1	1	Logstash

• ElasticSearch 설치

  • ES 포트번호 : 9200
  • JDK 설치

  apt  update
  apt  install  -y  openjdk-8-jdk

  

  • ES 설치

  wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
  echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
  apt update
  apt install elasticsearch

  

• 확인

  curl -XGET localhost:9200
  systemctl status elasticsearch
  # 키는법
  systemctl start elasticsearch

  

• Kibana 설치

  • Kibana 포트번호 : 5601
  • JDK 설치

  apt  update
  apt  install  -y  openjdk-8-jdk

  

  • Kibana 설치

  wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
  echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
  apt update
  apt install kibana

  

  • 설정

  /etc/kibana/kibana.yml 설정
  
  server.host: "키바나가 설치된 컴퓨터의 IP"
  
  설정 저장 후
  systemctl restart kibana

  

  • 확인

  웹브라우저로
  
  http://키바나가설치된컴퓨터IP:5601/

  

• Logstash

  • JDK 설치

  apt  update
  apt  install  -y  openjdk-8-jdk

  

  • Logstash 설치

    • Logstash 실행 전 실행파일 먼저 수행

      wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
      echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
      apt update
      apt install logstash

      

  • 설정파일 생성

  vi /etc/logstash/conf.d/logstash.conf

  input {
    tcp {
      port => 5000
      codec => json { charset => "UTF-8" }
    }
  }
  
  filter {
    mutate {
      remove_field => [ "port", "path", "message", "@version", "@timestamp", "host" ]
    }
  }
  
  output {
    elasticsearch {
      hosts => ["http://192.168.216.135:9200"]
      index => "coffee"
    }
  }

  

  • 실행
    

• 웹 브라우저 이동

  • DEV TOOL이동
    

PUT coffee/_mapping
{    
  "properties" : {
    "brand" : { "type" : "text" },
    "coffee_geo" : { "type" : "geo_point" }
  }
}

Install

• pycharm
  • logstash를 이어주는 API

  pip install python-logstash
  pip install logstash

Code

import json
import logging
from logging import StreamHandler
import logstash

host = '192.168.216.135'

test_logger = logging.getLogger('python-logstash-logger')

test_logger.setLevel(logging.INFO)
test_logger.addHandler(logstash.TCPLogstashHandler(host, 5000, version=1))
test_logger.addHandler(StreamHandler())

inout_f = open("starbucks_ediya.csv", 'r', encoding="UTF-8")
inout_lines = inout_f.readlines()

json_f = open("coffee.json", 'w', encoding="UTF-8")

for i in inout_lines:
    brand = i.split(',')[0]
    lon = i.split(',')[1]
    lat = i.split(',')[2]
    # print(brand)
    data = {
        "brand": brand,
        "coffee_geo": {"lat": lat, "lon": lon}
    }

    json_f.write(json.dumps(data, ensure_ascii=False) + "\n")
    test_logger.info('커피', extra=data)


inout_f.close()
json_f.close()

URL

Starbucks_Ediya