filebeat + logstash + JDBC

아무튼 간에·2022년 8월 8일
0
post-thumbnail

* 이 글은 윈도우 리눅스 우분투 환경 구축 전제 하에 진행됩니다.

-> Notion에서 편하게 보기(클릭)

개발환경
OS: Windows 11
VM: Ubuntu 18.04 (bionic)

목표 플로우

  1. 어플리케이션이 발생시키는 로그파일을 filebeat로 수집
  2. logstash로 필터링하여
  3. MariaDB에 insert

기본 구동 -> output -> filtering 순으로 단계별로 하나씩 적용해보면서 함

1. 기본 설치 및 구동 테스트

1-1. 어플리케이션(java 기반) 실행 환경 세팅

  1. 경로 생성 \home\dis\apple

  2. openjdk 설치(참고: ubuntu openjdk 설치)

    > sudo apt-get install openjdk-8-jdk

    !! E: Unable to locate package openjdk-8-jdk 오류 발생 시 아래 커맨드 실행 후 재실행

    > sudo apt-get update

  3. JAVA_HOME 환경변수 설정: JAVA_HOME 환경변수 잡기

  4. jar, logback.xml 파일 이동(/mnt/c 에서 호스트 디렉토리 접근 가능)

  5. jar 실행

1-2. filebeat 설치 및 실행

참고: Filebeat quick start: installation and configuration

  1. 설치(8.3.3 설치함)
> curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.3.3-linux-x86_64.tar.gz
> tar xzvf filebeat-8.3.3-linux-x86_64.tar.gz
letsj@LAPTOP-RNH6NT7P:~/dis$ curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.3.3-linux-x86_64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 58.1M  100 58.1M    0     0  10.2M      0  0:00:05  0:00:05 --:--:-- 10.7M
letsj@LAPTOP-RNH6NT7P:~/dis$ ls
apple filebeat-8.3.3-linux-x86_64.tar.gz
letsj@LAPTOP-RNH6NT7P:~/dis$ tar xzvf filebeat-8.3.3-linux-x86_64.tar.gz

디렉토리명 filebeat-8.xxx → filebeat로 변경함)

  1. filebeat.yml 수정
path: 
- \home\dis\apple\log\*\*.log

이것만 수정하면 안됨. 다른 것도 같이 수정해줬어야 했는데 이땐 몰랐음… (아래에서 제대로 수정함)

  1. 실행 파일 권한 설정
# sudo chown root modules.d/nginx.yml.disabled
# 가이드에서는 chown root로 하라고 했는데 그냥 755로 권한변경해줌

> chmod 755 bin/filebeat.yml
> chmod 755 modules.d/nginx.yml.disabled

# 가이드에선 nginx.yml에 권한을 부여하라고 했는데 nginx.yml.disabled 파일만 있어서 일단 저렇게 줌
  1. filebeat 실행
> ./filebeat -e
letsj@LAPTOP-RNH6NT7P:/home/dis/apple/filebeat$ ./filebeat -e
{"log.level":"info","@timestamp":"2022-08-04T17:33:14.121+0900","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/home/dis/apple/filebeat] Config path: [/home/dis/apple/filebeat] Data path: [/home/dis/apple/filebeat/data] Logs path: [/home/dis/apple/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-04T17:33:14.133+0900","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: 4855760f-de2c-4129-994b-5756e6122ef9","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-08-04T17:33:17.138+0900","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-04T17:33:17.140+0900","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-04T17:33:17.140+0900","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1076},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/home/dis/apple/filebeat","data":"/home/dis/apple/filebeat/data","home":"/home/dis/apple/filebeat","logs":"/home/dis/apple/filebeat/logs"},"type":"filebeat","uuid":"4855760f-de2c-4129-994b-5756e6122ef9"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-04T17:33:17.140+0900","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1085},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"1755b5dd3127bf755ee39deb25a802438bdac620","libbeat":"8.3.3","time":"2022-07-23T00:34:44.000Z","version":"8.3.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-04T17:33:17.140+0900","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1088},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":8,"version":"go1.18.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-04T17:33:17.142+0900","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-07-27T16:46:57+09:00","containerized":false,"name":"LAPTOP-RNH6NT7P","ip":["127.0.0.1/8","::1/128","172.19.148.90/20","fe80::215:5dff:fe3a:47b9/64"],"kernel_version":"5.10.16.3-microsoft-standard-WSL2","mac":["16:a9:ce:91:a3:26","32:e9:60:cb:3c:6f","00:15:5d:3a:47:b9"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.6 LTS (Bionic Beaver)","major":18,"minor":4,"patch":6,"codename":"bionic"},"timezone":"KST","timezone_offset_sec":32400},"ecs.version":"1.6.0"}}

오류는 안나는데 로그 메시지에 filebeat input disabled, skipping it 라는 메시지가 보임.

filebeat.yml 작성이 잘못됐기 때문에 지정해준 path의 log는 읽지 못하고 filesystem 로그만 읽어올 거임. (아래에서 맞게 수정함)

1-3. logstash 설치

참고: Installing Logstash

  • offical guide

    1. Public Signing Key 받기

      wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list

    2. 실행

      sudo apt-get update && sudo apt-get install logstash

      인데 설치가 어디 된건지 모르겠긔

나는 offcial guide 말고 gz 파일 받아서 압축 풀었음ㅎ(참고: [ElasticStack] logstash 설치및 실행)

(디렉토리명 logstash-8.xxx → logstash로 변경함)

  1. 실행 테스트 
> bin/logstash -e 'input { stdin { } } output { stdout {} }'  # 키보드 입출력
letsj@LAPTOP-RNH6NT7P:/home/dis/apple/logstash$ bin/logstash -e 'input { stdin { } } output { stdout {} }'
Using bundled JDK: /home/dis/apple/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /home/dis/apple/logstash/logs which is now configured via log4j2.properties
[2022-08-04T18:28:18,670][INFO ][logstash.runner          ] Log4j configuration path used is: /home/dis/apple/logstash/config/log4j2.properties
[2022-08-04T18:28:18,684][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-08-04T18:28:18,686][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.3.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-08-04T18:28:18,688][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-08-04T18:28:18,722][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/home/dis/apple/logstash/data/queue"}
[2022-08-04T18:28:18,735][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/dis/apple/logstash/data/dead_letter_queue"}
[2022-08-04T18:28:18,964][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-08-04T18:28:18,986][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"442420db-881e-4051-bec8-a3f69d557370", :path=>"/home/dis/apple/logstash/data/uuid"}
[2022-08-04T18:28:19,928][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-08-04T18:28:20,328][INFO ][org.reflections.Reflections] Reflections took 78 ms to scan 1 urls, producing 124 keys and 408 values
[2022-08-04T18:28:20,791][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-08-04T18:28:20,979][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x6b0eac55 run>"}
[2022-08-04T18:28:21,628][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.65}
[2022-08-04T18:28:21,695][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2022-08-04T18:28:21,753][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
hihi
{
         "event" => {
        "original" => "hihi"
    },
      "@version" => "1",
    "@timestamp" => 2022-08-04T09:28:26.194108Z,
       "message" => "hihi",
          "host" => {
        "hostname" => "LAPTOP-RNH6NT7P"
    }
}

hihi 입력해보고 잘 나오는지 확인 후 종료시킴(ctrl+C)

  1. 커스텀 구성파일 작성(conf/logstash-test.conf)

    1. logstash 하위에 conf 디렉토리 생성

    2. confing/logstash-example.conf를 conf 하위에 logstash-test.conf로 복사

    3. logstash-test.conf 수정

      input {
  beats {
    port => 5044 # filebeat의 output.logstash port와 동일(아래 나옴)
  }
}

output {
        stdout{}
}
  1. logstash 재실행
> bin/logstash -f conf/logstash-test.conf # 구성 파일 지정
letsj@LAPTOP-RNH6NT7P:/home/dis/apple/logstash$ bin/logstash -f conf/logstash-test.conf
Using bundled JDK: /home/dis/apple/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /home/dis/apple/logstash/logs which is now configured via log4j2.properties
[2022-08-04T18:53:59,625][INFO ][logstash.runner          ] Log4j configuration path used is: /home/dis/apple/logstash/config/log4j2.properties
[2022-08-04T18:53:59,637][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-08-04T18:53:59,639][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.3.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-08-04T18:53:59,642][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-08-04T18:53:59,947][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-08-04T18:54:00,931][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-08-04T18:54:01,369][INFO ][org.reflections.Reflections] Reflections took 67 ms to scan 1 urls, producing 124 keys and 408 values
[2022-08-04T18:54:01,932][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-08-04T18:54:02,025][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/home/dis/apple/logstash/conf/logstash-test.conf"], :thread=>"#<Thread:0x299ce82b run>"}
[2022-08-04T18:54:02,744][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.72}
[2022-08-04T18:54:02,774][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2022-08-04T18:54:02,793][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-08-04T18:54:02,886][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-08-04T18:54:02,906][INFO ][org.logstash.beats.Server][main][ae7ef898cedff3cfddac3de407504799b16f07c6c9c0fea178cc16e19d71e98d] Starting server on port: 5044

오류는 안남.

2. 실제 환경에 맞게 재구성하기

1-2. filebeat 재구성

  1. filebeat.yml 수정
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: log
  id: data-api-log # 맘대루..
  enabled: true
  paths:
     - /home/dis/apple/log/*/*.log

# ================================== Outputs ===================================
# ---------------------------- Elasticsearch Output ----------------------------
# output.elasticsearch:
  # Array of hosts to connect to.
  # hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["172.19.***.**:5044"]
  # 윈도우 기반 리눅스를 사용중이 때문에 리눅스에서 ip를 확인한 뒤 호스트를 설정해줌.

filebeat.yml 기본 세팅은 Elasticsearch로 내보내는 것인데

우리는 logstash로 내보낼 것이기 때문에 주석처리 하고

하단에 output.logstash를 살려준다.

※ 리눅스 ip 확인 방법

> ifconfig
  1. 재실행
> ./filebeat -e

정상 구동되면 로그가 징그럽게 많이 올라오는데

자세히 보면 

"log.logger":"monitoring"

이렇게 나오기 시작한 부분이 있음.

이전까지는 구동 로그로 보면 됨.

pretty 패턴으로 보기 좋게 출력하는 옵션이 있다고는 했는데 이전에 테스트 해봤을 땐 오류가 났었음

logstash도 계속 구동 중이기 때문에 로그 올라옴.. (아직 output{stdout{}})

2-2. logstash 재구성

  1. jdbc 드라이버(jar) 파일 넣기

    /home/dis/apple/logstash/vendor/jar/mariadb-java-client-2.4.2.jar

    (logstash/vendor/ 하위에 jar 디렉토리 생성 후 넣었음)

  1. logstash-test.conf 수정
input {
  beats {
    port => 5044
  }
}

output {
        stdout{} # 콘솔 출력도 하고

 jdbc {
    driver_jar_path => "/home/dis/apple/logstash/vendor/jar/mariadb-java-client-2.4.2.jar"
    connection_string => "jdbc:mariadb://192.168.11.29:3306/apple?user=root&password=1234"
    statement => [ "insert into mmsdb.service_history(full_text) values (?)", "message" ]
	} # db에도 넣을거임
}
  1. 재기동
> bin/logstash -f conf/logstash-test.conf

또 다시 마주한 그녀석

Unable to load plugin. {:type=>"output", :name=>"jdbc"}

  1. logstash-output-jdbc 플러그인 설치
> bin/logstash-plugin install logstash-output-jdbc
letsj@LAPTOP-RNH6NT7P:/home/dis/apple/logstash$ bin/logstash-plugin install logstash-output-jdbc
Using bundled JDK: /home/dis/apple/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-output-jdbc
Resolving mixin dependencies
Installing logstash-output-jdbc
Installation successful

드디어? 와… 땡스투갓 땡스투붓다 땡스투알라

  1. 다시 실행
> bin/logstash -f conf/logstash-test.conf
letsj@LAPTOP-RNH6NT7P:/home/dis/apple/logstash$ bin/logstash -f conf/logstash-test.conf
Using bundled JDK: /home/dis/apple/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /home/dis/apple/logstash/logs which is now configured via log4j2.properties
[2022-08-05T10:02:55,394][INFO ][logstash.runner          ] Log4j configuration path used is: /home/dis/apple/logstash/config/log4j2.properties
[2022-08-05T10:02:55,404][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-08-05T10:02:55,405][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.3.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-08-05T10:02:55,408][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-08-05T10:02:55,865][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-08-05T10:02:57,021][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-08-05T10:02:57,436][INFO ][org.reflections.Reflections] Reflections took 51 ms to scan 1 urls, producing 124 keys and 408 values
[2022-08-05T10:02:57,971][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-08-05T10:02:58,018][INFO ][logstash.outputs.jdbc    ][main] JDBC - Starting up
[2022-08-05T10:02:58,049][INFO ][com.zaxxer.hikari.HikariDataSource][main] HikariPool-1 - Starting...
[2022-08-05T10:02:58,127][INFO ][com.zaxxer.hikari.HikariDataSource][main] HikariPool-1 - Start completed.
[2022-08-05T10:02:58,190][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/home/dis/apple/logstash/conf/logstash-test.conf"], :thread=>"#<Thread:0x22bc3623 run>"}
[2022-08-05T10:02:58,793][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.6}
[2022-08-05T10:02:58,817][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2022-08-05T10:02:58,828][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-08-05T10:02:58,889][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-08-05T10:02:58,915][INFO ][org.logstash.beats.Server][main][8b7314c417aebb2fa1fc920382fe918f0a42d000737d7245fdbf6fa5835cdf82] Starting server on port: 5044

오 오류안나고 됨 오

  1. 로그 발생 시켜 봄
# 그냥 이렇게 나온다~ 참고
{
           "ecs" => {
        "version" => "8.0.0"
    },
       "message" => "10:04:40.220 [http-nio-6080-exec-7] INFO  c.n.a.d.a.j.s.JwtAuthenticationService - Start to authenticate JSON Web Token",
         "agent" => {
                  "id" => "4855760f-de2c-4129-994b-5756e6122ef9",
                "name" => "LAPTOP-RNH6NT7P",
        "ephemeral_id" => "0f7aebd2-7a5a-414d-883b-3479a20688aa",
                "type" => "filebeat",
             "version" => "8.3.3"
    },
      "@version" => "1",
    "@timestamp" => 2022-08-05T01:04:40.764Z,
         "event" => {
        "original" => "10:04:40.220 [http-nio-6080-exec-7] INFO  c.n.a.d.a.j.s.JwtAuthenticationService - Start to authenticate JSON Web Token"
    },
         "input" => {
        "type" => "log"
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "host" => {
                   "ip" => [
            [0] "172.19.148.90",
            [1] "fe80::215:5dff:fe3a:47b9"
        ],
                  "mac" => [
            [0] "00:15:5d:3a:47:b9",
            [1] "16:a9:ce:91:a3:26",
            [2] "32:e9:60:cb:3c:6f"
        ],
                   "os" => {
                "type" => "linux",
             "version" => "18.04.6 LTS (Bionic Beaver)",
              "kernel" => "5.10.16.3-microsoft-standard-WSL2",
              "family" => "debian",
                "name" => "Ubuntu",
            "codename" => "bionic",
            "platform" => "ubuntu"
        },
         "architecture" => "x86_64",
                 "name" => "LAPTOP-RNH6NT7P",
        "containerized" => false,
             "hostname" => "LAPTOP-RNH6NT7P"
    },
           "log" => {
          "file" => {
            "path" => "/home/dis/apple/log/20220805/apple.log"
        },
        "offset" => 7331
    }
}
[2022-08-05T10:04:43,277][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=81) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.265 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Start to parse the messgae\",\"agent\":{\"name\":\"LAPTOP-RNH6NT7P\",\"type\":\"filebeat\",\"version\":\"8.3.3\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.765Z\",\"event\":{\"original\":\"10:04:40.265 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Start to parse the messgae\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"host\":{\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"architecture\":\"x86_64\",\"containerized\":false,\"name\":\"LAPTOP-RNH6NT7P\",\"hostname\":\"LAPTOP-RNH6NT7P\"},\"log\":{\"offset\":7792,\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"}}}"}
[2022-08-05T10:04:43,277][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=80) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.269 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Succeed in parsing the messgae\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.765Z\",\"event\":{\"original\":\"10:04:40.269 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Succeed in parsing the messgae\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"host\":{\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"codename\":\"bionic\",\"name\":\"Ubuntu\",\"platform\":\"ubuntu\"},\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"},\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":7900}}"}
[2022-08-05T10:04:43,277][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=82) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.265 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Succeed in receiving queue messgae\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"type\":\"filebeat\",\"version\":\"8.3.3\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.765Z\",\"event\":{\"original\":\"10:04:40.265 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Succeed in receiving queue messgae\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":7676},\"host\":{\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"}}"}
[2022-08-05T10:04:43,277][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=78) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.195 [http-nio-6080-exec-7] INFO  c.n.a.d.api.contorller.ApiController - Check requseted service is valid...\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.764Z\",\"event\":{\"original\":\"10:04:40.195 [http-nio-6080-exec-7] INFO  c.n.a.d.api.contorller.ApiController - Check requseted service is valid...\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":7214},\"host\":{\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"}}"}
[2022-08-05T10:04:43,277][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=79) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.194 [http-nio-6080-exec-7] INFO  c.n.a.d.api.contorller.ApiController - Invoke Service API: M01-01\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.759Z\",\"event\":{\"original\":\"10:04:40.194 [http-nio-6080-exec-7] INFO  c.n.a.d.api.contorller.ApiController - Invoke Service API: M01-01\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"host\":{\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"},\"log\":{\"offset\":7106,\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"}}}"}
[2022-08-05T10:04:43,284][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=79) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.269 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Check messgae count\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.765Z\",\"event\":{\"original\":\"10:04:40.269 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Check messgae count\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":8012},\"host\":{\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"}}"}
[2022-08-05T10:04:43,284][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=80) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.233 [http-nio-6080-exec-7] INFO  c.n.a.d.a.j.s.JwtAuthenticationService - Finished authenticate JSON Web Token\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"type\":\"filebeat\",\"version\":\"8.3.3\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.764Z\",\"event\":{\"original\":\"10:04:40.233 [http-nio-6080-exec-7] INFO  c.n.a.d.a.j.s.JwtAuthenticationService - Finished authenticate JSON Web Token\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"log\":{\"offset\":7451,\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"}},\"host\":{\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"codename\":\"bionic\",\"name\":\"Ubuntu\",\"platform\":\"ubuntu\"},\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"}}"}
[2022-08-05T10:04:43,284][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=78) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.233 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Try to Receive messages\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.765Z\",\"event\":{\"original\":\"10:04:40.233 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Try to Receive messages\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":7571},\"host\":{\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"architecture\":\"x86_64\",\"containerized\":false,\"name\":\"LAPTOP-RNH6NT7P\",\"hostname\":\"LAPTOP-RNH6NT7P\"}}"}
[2022-08-05T10:04:43,285][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=82) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.220 [http-nio-6080-exec-7] INFO  c.n.a.d.a.j.s.JwtAuthenticationService - Start to authenticate JSON Web Token\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.764Z\",\"event\":{\"original\":\"10:04:40.220 [http-nio-6080-exec-7] INFO  c.n.a.d.a.j.s.JwtAuthenticationService - Start to authenticate JSON Web Token\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"host\":{\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"},\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":7331}}"}

filebeat에서 로그는 잘 받아오는 것 같고

아래에 jdbc 오류 발생

[2022-08-05T10:04:43,277][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=81) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.265 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Start to parse the messgae\",\"agent\":{\"name\":\"LAPTOP-RNH6NT7P\",\"type\":\"filebeat\",\"version\":\"8.3.3\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.765Z\",\"event\":{\"original\":\"10:04:40.265 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Start to parse the messgae\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"host\":{\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"architecture\":\"x86_64\",\"containerized\":false,\"name\":\"LAPTOP-RNH6NT7P\",\"hostname\":\"LAPTOP-RNH6NT7P\"},\"log\":{\"offset\":7792,\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"}}}"}
[2022-08-05T10:04:43,277][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=80) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.269 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Succeed in parsing the messgae\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.765Z\",\"event\":{\"original\":\"10:04:40.269 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Succeed in parsing the messgae\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"host\":{\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"codename\":\"bionic\",\"name\":\"Ubuntu\",\"platform\":\"ubuntu\"},\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"},\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":7900}}"}
[2022-08-05T10:04:43,277][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=82) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.265 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Succeed in receiving queue messgae\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"type\":\"filebeat\",\"version\":\"8.3.3\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.765Z\",\"event\":{\"original\":\"10:04:40.265 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Succeed in receiving queue messgae\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":7676},\"host\":{\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"}}"}
[2022-08-05T10:04:43,277][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=78) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.195 [http-nio-6080-exec-7] INFO  c.n.a.d.api.contorller.ApiController - Check requseted service is valid...\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.764Z\",\"event\":{\"original\":\"10:04:40.195 [http-nio-6080-exec-7] INFO  c.n.a.d.api.contorller.ApiController - Check requseted service is valid...\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":7214},\"host\":{\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"}}"}
[2022-08-05T10:04:43,277][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=79) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.194 [http-nio-6080-exec-7] INFO  c.n.a.d.api.contorller.ApiController - Invoke Service API: M01-01\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.759Z\",\"event\":{\"original\":\"10:04:40.194 [http-nio-6080-exec-7] INFO  c.n.a.d.api.contorller.ApiController - Invoke Service API: M01-01\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"host\":{\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"},\"log\":{\"offset\":7106,\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"}}}"}
[2022-08-05T10:04:43,284][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=79) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.269 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Check messgae count\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.765Z\",\"event\":{\"original\":\"10:04:40.269 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Check messgae count\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":8012},\"host\":{\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"}}"}
[2022-08-05T10:04:43,284][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=80) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.233 [http-nio-6080-exec-7] INFO  c.n.a.d.a.j.s.JwtAuthenticationService - Finished authenticate JSON Web Token\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"type\":\"filebeat\",\"version\":\"8.3.3\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.764Z\",\"event\":{\"original\":\"10:04:40.233 [http-nio-6080-exec-7] INFO  c.n.a.d.a.j.s.JwtAuthenticationService - Finished authenticate JSON Web Token\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"log\":{\"offset\":7451,\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"}},\"host\":{\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"codename\":\"bionic\",\"name\":\"Ubuntu\",\"platform\":\"ubuntu\"},\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"}}"}
[2022-08-05T10:04:43,284][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=78) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.233 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Try to Receive messages\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.765Z\",\"event\":{\"original\":\"10:04:40.233 [http-nio-6080-exec-7] DEBUG c.n.a.d.api.contorller.ApiController - Try to Receive messages\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":7571},\"host\":{\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"architecture\":\"x86_64\",\"containerized\":false,\"name\":\"LAPTOP-RNH6NT7P\",\"hostname\":\"LAPTOP-RNH6NT7P\"}}"}
[2022-08-05T10:04:43,285][ERROR][logstash.outputs.jdbc    ][main][bbc639382f30757e5131087328553dcf06c64f783b6381978cf07a04e6645a48] JDBC - Exception. Not retrying {:exception=>java.sql.SQLSyntaxErrorException: (conn=82) Table 'mmsdb.service_history' doesn't exist, :statement=>"insert into mmsdb.service_history(full_text) values (?)", :event=>"{\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"10:04:40.220 [http-nio-6080-exec-7] INFO  c.n.a.d.a.j.s.JwtAuthenticationService - Start to authenticate JSON Web Token\",\"agent\":{\"id\":\"4855760f-de2c-4129-994b-5756e6122ef9\",\"name\":\"LAPTOP-RNH6NT7P\",\"ephemeral_id\":\"0f7aebd2-7a5a-414d-883b-3479a20688aa\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"@version\":\"1\",\"@timestamp\":\"2022-08-05T01:04:40.764Z\",\"event\":{\"original\":\"10:04:40.220 [http-nio-6080-exec-7] INFO  c.n.a.d.a.j.s.JwtAuthenticationService - Start to authenticate JSON Web Token\"},\"input\":{\"type\":\"log\"},\"tags\":[\"beats_input_codec_plain_applied\"],\"host\":{\"ip\":[\"172.19.148.90\",\"fe80::215:5dff:fe3a:47b9\"],\"mac\":[\"00:15:5d:3a:47:b9\",\"16:a9:ce:91:a3:26\",\"32:e9:60:cb:3c:6f\"],\"os\":{\"type\":\"linux\",\"version\":\"18.04.6 LTS (Bionic Beaver)\",\"kernel\":\"5.10.16.3-microsoft-standard-WSL2\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"codename\":\"bionic\",\"platform\":\"ubuntu\"},\"architecture\":\"x86_64\",\"name\":\"LAPTOP-RNH6NT7P\",\"containerized\":false,\"hostname\":\"LAPTOP-RNH6NT7P\"},\"log\":{\"file\":{\"path\":\"/home/dis/apple/log/20220805/apple.log\"},\"offset\":7331}}"}

Exception. Not retrying … SQLSyntaxErrorException … Table 'mmsdb.service_history' doesn't exist

db명 안바꿈 에헷

바꾸고 다시 실행 후 로그 발생시켜 봄

오류 안나고

db에도 데이터 잘 들어감 오 대박 해냈다 오..

2-3. logstash filter 추가 설정

그럼 이제 필터링 걸어보기

grok이라는 플러그인을 이용할 거임

나의 log 패턴(logback 설정)

%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n : logback 패턴

레퍼런스 파일 참고해서 짜봄

(?m)\%{TIMESTAMP_ISO8601:timestamp}\ %{DATA:thread}\ %{LOGLEVEL:logLevel} %{DATA:class} - %{GREEDYDATA:message}

일단 filter.grok이 되는지부터 확인하기 위해

filter{
 grok{
        match =>{"message" => "(?m)\%{TIMESTAMP_ISO8601:timestamp}\ %{DATA:thread}\ %{LOGLEVEL:logLevel} %{DATA:class} - %{GREEDYDATA:message}"}
 }
}

아래처럼 input과 output 사이에 끼워넣고 저장

input {
  beats {
    port => 5044
  }
}
filter{
 grok{
        match =>{"message" => "(?m)\%{TIMESTAMP_ISO8601:timestamp}\ %{DATA:thread}\ %{LOGLEVEL:logLevel} %{DATA:class} - %{GREEDYDATA:message}"}
 }
}
output {
 stdout{} 
 jdbc {
    driver_jar_path => "/home/dis/apple/logstash/vendor/jar/mariadb-java-client-2.4.2.jar"
    connection_string => "jdbc:mariadb://192.168.11.29:3306/apple?user=root&password=1234"
    statement => [ "insert into mmsdb.service_history(full_text) values (?)", "message" ]
	} 
}

실행하면 

[2022-08-05T10:30:32,833][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2022-08-05T10:30:32,944][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<RegexpError: unmatched close parenthesis: /(?m)\(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01]?[0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)\ (?<DATA:thread>.*?)\ (?<LOGLEVEL:logLevel>([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo?(?:rmation)?|INFO?(?:RMATION)?|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)) (?<DATA:class>.*?) - (?<GREEDYDATA:message>.*)/m>, :backtrace=>["org/jruby/RubyRegexp.java:973:in `initialize'", "/home/dis/apple/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:127:in `compile'", "/home/dis/apple/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.2/lib/logstash/filters/grok.rb:282:in `block in register'", "org/jruby/RubyArray.java:1821:in `each'", "/home/dis/apple/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.2/lib/logstash/filters/grok.rb:276:in `block in register'", "org/jruby/RubyHash.java:1415:in `each'", "/home/dis/apple/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.2/lib/logstash/filters/grok.rb:271:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/home/dis/apple/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/home/dis/apple/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `register_plugins'", "/home/dis/apple/logstash/logstash-core/lib/logstash/java_pipeline.rb:599:in `maybe_setup_out_plugins'", "/home/dis/apple/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in `start_workers'", "/home/dis/apple/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in `run'", "/home/dis/apple/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"], "pipeline.sources"=>["/home/dis/apple/logstash/conf/logstash-test.conf"], :thread=>"#<Thread:0x39bf1ec3 run>"}
[2022-08-05T10:30:32,948][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-08-05T10:30:32,959][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2022-08-05T10:30:33,022][INFO ][logstash.runner          ] Logstash shut down.
[2022-08-05T10:30:33,031][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
        at home.dis.apple.logstash.lib.bootstrap.environment.<main>(/home/dis/apple/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

안되네

정규식 오류인듯

#<RegexpError: unmatched close parenthesis: /(?m)(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))T :?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|+-(?::?(?:(?:[0-5][0-9])))))?)\ (?<DATA:thread>.?)\ (?<LOGLEVEL:logLevel>([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo?(?:rmation)?|INFO?(?:RMATION)?|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)) (?<DATA:class>.?) - (?<GREEDYDATA:message>.)/m>*

……. 재수정 기릿

위에 적은 테스트 사이트에서 예제 대입하면서 해봄

나의 실제 로그 

10:08:12.359 [http-nio-6080-exec-9] INFO c.n.a.d.a.j.s.JwtAuthenticationService - Start to authenticate JSON Web Token
  • 10:08:12.359 → %{TIME}
  • [http-nio-6080-exec-9] → %{DATA}
  • INFO → %{LOGLEVEL}
  • c.n.a.d.a.j.s.JwtAuthenticationService → %{DATA}
  • Start to authenticate JSON Web Token → %{GREEDYDATA}

작성된 패턴

%{TIME:timestamp}\ %{DATA:thread}\ %{LOGLEVEL:logLevel}\ %{DATA:class}\ - %{GREEDYDATA:message}
  • \: 각 패턴 사이의 공백
  • :변수명: 패턴으로 구분시킨 단위(?)의 변수명
filter{
 grok{
        match =>{"message" => "%{TIME:timestamp}\ %{DATA:thread}\ %{LOGLEVEL:logLevel}\ %{DATA:class}\ - %{GREEDYDATA:message}"}
 }
}

수정 후 재실행하면 오류 없이 잘 구동된다.

2-3. logstash filtering 결과를 DB에 넣기

설정한 필터대로 데이터 쪼개서 db 넣어보자

  1. logstash-test.conf 수정
input {
  beats {
    port => 5044
  }
}
filter{
 grok{
        match =>{"message" => "%{TIME:timestamp}\ %{DATA:thread}\ %{LOGLEVEL:logLevel}\ %{DATA:class}\ - %{GREEDYDATA:*msg*}"}
 }
}

output {
        stdout{}
         jdbc {
                driver_jar_path => "/home/dis/apple/logstash/vendor/jar/mariadb-java-client-2.4.2.jar"
                #driver_class => "org.mariadb.jdbc.Driver"
                connection_string => "jdbc:mariadb://192.168.11.29:3306/apple?user=root&password=1234"
                statement => [ "insert into apple.service_history(***log_data***,full_text) values (***?***,?)",***"msg"***,"message" ]
        }
}

→ service_history라는 테이블의 log_data 컬럼에 filter.grok에서 필터링한 msg라는 값을 넣을거다~

저장 후 재실행시 오류 없이 로그 올라옴.

  1. DB 확인

데이터 잘 들어감 굿

profile
아무튼 간에
armton garnet
이전 포스트

윈도우 리눅스 우분투 환경 구축

다음 포스트

JPA WHERE 조건(equal, like, in, between), ORDER BY 넣기 및 페이징(Specification, CriteriaBuilder, Pageable, PageRequest)

0개의 댓글