엘라스틱서치는 검색엔진 DBMS 이다.
구조 json
저장된 데이터를 분석 -> 검색엔진
json 형태라 조회는 REST API로 진행한다. (get, post ...)
-
우리는 Elasticsearch 를 OpenSearch로 이용한다. 둘 다 클러스터
- 왜 클러스터? 분산
- 왜 분산?
- scale up
- 왜 분산?
- 왜 클러스터? 분산
-
Cluster
- Master Node -> 노드들 제어, 관리, 헬스체크
- Data node (여기까지가 기본) -> 빠른 검색을 위한 data
- UltraWarm node -> IA(Infrequent Access) 검색을 위한 data
- Cold node -> store data
- Master Node -> 노드들 제어, 관리, 헬스체크
-
Logstash
- 데이터 정제 -> memory 사용량이 많다. -> Fluentd로 대체 가능 !
- beat 패키지를 사용
- Logstash로 정제하지 않은 데이터가 Elasticsearch에 들어오면?
-> 저장되는 데이터가 무거워지긴 하겠지만 검색은 잘 된다.
-
vm 세팅
192.168.56.101 ip
255.255.255.0 넷마스크
192.168.56.1 | 192.168.56.2
💡 컴퓨터가 서로 통신하기 위해서는 모든 컴퓨터마다 유일한 IP 주소를 할당해야 하듯, 게이트웨이에도 중복되지 않는 IP 주소가 필요합니다. 이 IP 주소를 토대로 각 컴퓨터가 다른 네트워크와 연결됩니다. 일반적으로 게이트웨이의 IP 주소는 해당 네트워크 내 컴퓨터에 할당된 IP 주소 중 끝자리만 다른 형태입니다. 보통 1을 지정합니다.
📕 Elastic stack?
-
빅데이터 : 거대한 규모(volume), 빠른 속도(velocity), 높은 다양성(variety)을 특징으로 하는 데이터
- 종류(variety), 양(volume), 속도(velocity)
-
기존 RDBMS의 빅데이터 처리는 병목현상 발생 -> 서비스 지연과 고객 불편을 가중
-
해결: 실시간으로 수집되는 다양한 대량 정보의 저장 및 검색을 위한 더 빠른 방법 제시가 필요
(이왕이면 지연이 발생해도 고객이 알아채지 못하게 할 방법까지 !)
이러한 요구사항이 적용된 DB의 탄생 -> NoSQL !
NoSQL 계열의 Elasticsearch는 고가용성의 확장 가능한 검색엔진 DBMS이다.
-> 자료를 쌓아두고 필요한 자료를 빠르게 검색할 수 있도록 해주는 도구
-> 검색 서비스를 제공하는 검색 시스템 기반의 검색 엔진(search engine)
검색엔진이란?
💡 수집 -> 색인화(indexing) -> 저장 -> 검색
사용자가 필요로하는 정보를 수집하여 내용을 분석한 뒤
찾기 쉬운 형태로 조직하여(색인, index), 정보에 대한 요구가 발생할 때
해당 정보를 빠르게 찾아 제공(검색, search)하는 시스템이나 프로그램
💡 Elastic stack은
Beats 를 통해 데이터 수집
Logstash로 데이터 가공(정제)
가공된 Data를 Elasticsearch에 인덱싱하여 저장(조회, 분석)
Kibana를 통해 시각화(data 관리, 모니터링)
Elasticsearch
분산형 RESTful 검색 및 분석 엔진 = 전문 검색 엔진 (full text search engine)
💻 실습
수집(Beats) 가공(Logstash) 저장(Elasticsearch) 시각화(Kinaba)
/etc/profile -> 시스템 전역 환경설정
/bash_profile -> 유저 레벨 전역 환경설정
export -> 세션 레벨 전역 환경 설정
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre/
classpath -> library
[root@elk-master ~]# rpm -ql java-1.8.0-openjdk
⭐ ~~~jre 까지 복사
👻 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre👻/bin/policytool
...
# 시스템 전역
[root@elk-master ~]# vi /etc/profile
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH='.'
[root@elk-master ~]# source /etc/profile
[root@elk-master ~]# echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre
[root@elk-master ~]# cd /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre
[root@elk-master jre]#
# Elastic-7.x 버전이 떠있으면 성공
cd /etc/yum.repos.d
vim Elastic.repo
cat Elastic.repo
[Elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
[root@elk-master yum.repos.d]# yum repolist
Loaded plugins: fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Loading mirror speeds from cached hostfile
* base: mirror.navercorp.com
* extras: mirror.navercorp.com
* updates: mirror.navercorp.com
Elastic-7.x | 1.3 kB 00:00
Elastic-7.x/primary | 457 kB 00:00
Elastic-7.x 1432/1432
repo id repo name status
Elastic-7.x Elastic repository for 7.x packages 1,432
base/7/x86_64 CentOS-7 - Base 10,072
extras/7/x86_64 CentOS-7 - Extras 515
updates/7/x86_64 CentOS-7 - Updates 4,346
repolist: 16,365
# 한 줄 추가 -> yum update 목록에서 제거된다.
vi /etc/yum.conf
exclude=elasticsearch logstash kibana
# 설치 확인
[root@elk-master ~]# rpm -qa | grep logstash
logstash-7.10.2-1.x86_64
[root@elk-master ~]# rpm -qa | grep kibana
kibana-7.10.2-1.x86_64
[root@elk-master ~]# rpm -qa | grep elasticsearch
elasticsearch-7.10.2-1.x86_64
세팅
/etc/elasticsearch/elasticsearch.yml
vi /etc/elasticsearch/elasticsearch.yml
# ---------------------------------- Cluster ------------------------- ----------
14 #
15 # Use a descriptive name for your cluster:
16 #
17 cluster.name: kakao-elk-cluster
18 #
19 # ------------------------------------ Node -------------------------- ----------
20 #
21 # Use a descriptive name for the node:
22 #
23 node.name: elk-master
29 # ----------------------------------- Paths -------------------------- ----------
30 #
31 # Path to directory where to store the data (separate multiple locatio ns by comma):
32 # 데이터 경로
33 path.data: /var/lib/elasticsearch
34 # 로그 경로
35 # Path to log files:
36 #
37 path.logs: /var/log/elasticsearch
38 #
51 # ---------------------------------- Network ------------------------- ----------
52 #
53 # Set the bind address to a specific IP (IPv4 or IPv6):
54 # 어디서든 들어오세요
55 network.host: 0.0.0.0
56 #
57 # Set a custom port for HTTP:
58 #
59 http.port: 9200
60 #
61 # For more information, consult the network module documentation.
62 #
63 # -------
# --------------------------------- Discovery ------------------------ ----------
64 #
65 # Pass an initial list of hosts to perform discovery when this node is started:
66 # The default list of hosts is ["127.0.0.1", "[::1]"]
67 #
68 discovery.seed_hosts: ["192.168.56.101", "127.0.0.1", "[::1]"]
69 #
70 # Bootstrap the cluster using an initial set of master-eligible nodes:
71 #
72 cluster.initial_master_nodes: ["elk-master"]
73 #
74 # For more information, consult the discovery and cluster formation mo dule documentation.
89 node.master: true # 한 줄 추가. 애가 마스터야.
# 메모리 값 키우기
vi /etc/elasticsearch/jvm.options
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms2g # 1-> 2
-Xmx2g # 1-> 2
[root@elk-master ~]# vi /etc/logstash/jvm.options
1 ## JVM configuration
2
3 # Xms represents the initial size of total heap space
4 # Xmx represents the maximum size of total heap space
5
6 -Xms2g # 1-> 2
7 -Xmx2g # 1 -> 2
[root@elk-master ~]# vi /etc/kibana/kibana.yml
1 # Kibana is served by a back end server. This setting specifies the po rt to use.
2 server.port: 5601 # 주석 해제
3
4 # Specifies the address to which the Kibana server will bind. IP addre sses and host names are both valid values.
5 # The default is 'localhost', which usually means remote machines will not be able to connect.
6 # To allow connections from remote users, set this parameter to a non- loopback address.
7 server.host: "0.0.0.0" # 주석해제 & 변경
8
9 # Enables you to specify a path to mount Kibana at if you are running behind a proxy.
10 # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
11 # from requests it receives, and to prevent a deprecation warning at s tartup.
12 # This setting cannot end in a slash.
13 #server.basePath: ""
14
15 # Specifies whether Kibana should rewrite requests that are prefixed w ith
16 # `server.basePath` or require that they are rewritten by your reverse proxy.
17 # This setting was effectively always `false` before Kibana 6.3 and wi ll
18 # default to `true` starting in Kibana 7.0.
19 #server.rewriteBasePath: false
20
21 # The maximum payload size in bytes for incoming server requests.
22 #server.maxPayloadBytes: 1048576
23
24 # The Kibana server's name. This is used for display purposes.
25 server.name: "elk-master" # 주석 해제 & 변경
26
27 # The URLs of the Elasticsearch instances to use for all your queries.
28 elasticsearch.hosts: ["http://192.168.56.101:9200"] # 주석 해제 & 변경
29
[root@elk-master ~]# systemctl daemon-reload
[root@elk-master ~]# systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[root@elk-master ~]# curl -XGET localhost:9200
{
"name" : "elk-app",
"cluster_name" : "kakao-elk-cluster",
"cluster_uuid" : "_na_",
"version" : {
"number" : "7.10.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
"build_date" : "2021-01-13T00:42:12.435326Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
-
여기까지하면 윈도우에서는 접근 안됨
-
방화벽 열기
[root@elk-master ~]# systemctl start firewalld.service
firewall-cmd --add-port=9200/tcp --permanent --zone=public
firewall-cmd --add-port=9300/tcp --permanent --zone=public
firewall-cmd --add-port=5443/tcp --permanent --zone=public
firewall-cmd --add-port=5443/udp --permanent --zone=public
firewall-cmd --add-port=5601/tcp --permanent --zone=public
[root@elk-master ~]# firewall-cmd --reload
success
[root@elk-master ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: enp0s3 enp0s8
sources:
services: dhcpv6-client ssh
ports: 9200/tcp 9300/tcp 5443/tcp 5443/udp 5601/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
/etc/elasticsearch
/usr/share/elasticsearch
/var/log/elasticsearch/
/var/lib/elasticsearch
실습) 시스템에 접근한 세션의 실패 기록 정보를 정제하여 저장하고 시각화 한다.
- logstash에서 파이프라인 작업
- input 파이프라인
- filter 파이프라인 (정제)
- output 파이프라인
vi /etc/logstash/conf.d/sshd_fail.conf
cat /etc/logstash/conf.d/sshd_fail.conf
input {
file {
type => "secure_log"
path => "/var/log/secure"
}
}
filter {
grok{
add_tag => [ "sshd_fail" ]
match => { "message" => "Failed %{WORD:sshd_auth_type} for %{USERNAME:sshd_invalid_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{GREEDYDATA:sshd_protocol}"}
}
}
}
output {
elasticsearch {
index => "sshd_fail-%{+YYYY.MM}"
}
}
[root@elk-master ~]# chgrp logstash /var/log/secure
[root@elk-master ~]# chmod 640 /var/log/secure
[root@elk-master ~]# systemctl restart logstash.service
[root@elk-master ~]# systemctl status logstash.service
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2022-10-27 12:33:06 KST; 3s ago
--- terminal 2
[root@elk-master ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/sshd_fail.conf
--- terminal 1
[root@elk-master ~]# curl localhost:9200/sshd_fail-2022.10/_search?pretty/
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/sshd_fail-2022.10/_search] contains unrecognized parameter: [pretty/] -> did you mean [pretty]?"}],"type":"illegal_argument_exception","reason":"request [/sshd_fail-2022.10/_search] contains unrecognized parameter: [pretty/] -> did you mean [pretty]?"},"status":400}[root@elk-master ~]# curl localhost:9200/sshd_fail-2022.10/_search?pretty
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index [sshd_fail-2022.10]",
"resource.type" : "index_or_alias",
"resource.id" : "sshd_fail-2022.10",
"index_uuid" : "_na_",
"index" : "sshd_fail-2022.10"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index [sshd_fail-2022.10]",
"resource.type" : "index_or_alias",
"resource.id" : "sshd_fail-2022.10",
"index_uuid" : "_na_",
"index" : "sshd_fail-2022.10"
},
"status" : 404
}
/usr/share/elasticsearch/bin/elasticsearch-sql-clicurl localhost:9200/⭐sshd_fail-2022.10/_search?pretty
⭐-> 저기가 인덱스
SQL (Structure Query Language)
구조적 질의어 ▶ 구조(Table)에 질문하는 언어
Table : 행과 열의 집합(행열)
행(row = data) / 열(column = structure)
-> type은 생략해도 괜찮다.
[root@elk-master ~]# curl -XPUT http://localhost:9200/books/book/1 -H 'Content-Type: application/json' -d '
{
"title" : "Elasticsearch Beginning",
"author" : "Ji",
"date" : "2022-10-27",
"pages" : 300
}'
{"_index":"books","_type":"book","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
[root@elk-master ~]# curl -XGET localhost:9200/books/book/1?pretty
{
"_index" : "books",
"_type" : "book",
"_id" : "1",
"_version" : 1,
"_seq_no" : 0,
"_primary_term" : 1,
"found" : true,
"_source" : {
"title" : "Elasticsearch Beginning",
"author" : "Ji",
"date" : "2022-10-27",
"pages" : 300
}
}
[root@elk-master ~]# curl -XPOST localhost:9200/books/book/1 -H 'Content-Type: application/json' -d '
{
"title" : "Elasticsearch Beginnig",
"author" : ["Ji", "Jung"],
"date" : "2022-10-27",
"pages" : 500
}'
[root@elk-master ~]# curl -XGET localhost:9200/books/book/1?pretty
{
"_index" : "books",
"_type" : "book",
"_id" : "1",
"_version" : 2, 💡 POST 후 버전이 바뀌었다~
"_seq_no" : 1,
"_primary_term" : 1,
"found" : true,
"_source" : {
"title" : "Elasticsearch Beginnig",
"author" : [
"Ji",
"Jung"
],
"date" : "2022-10-27",
"pages" : 500
}
}
# _update를 통해 컬럼 추가
[root@elk-master ~]# curl -XPOST localhost:9200/books/book/1/_update -H 'Content-Type: application/json' -d '
> {
> "doc" : {
> "category" : "ICT"
> }
> }'
{"_index":"books","_type":"book","_id":"1","_version":3,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1}
[root@elk-master ~]# curl -XGET localhost:9200/books/book/1?pretty
{
"_index" : "books",
"_type" : "book",
"_id" : "1",
"_version" : 3,
"_seq_no" : 2,
"_primary_term" : 1,
"found" : true,
"_source" : {
"title" : "Elasticsearch Beginnig",
"author" : [
"Ji",
"Jung"
],
"date" : "2022-10-27",
"pages" : 500,
"category" : "ICT"
}
}
[root@elk-master ~]# curl -XPOST localhost:9200/books/book/1/_update -H 'Content-Type: application/json' -d '
{
"doc" : {
"author" : "YRJI"
> }
> }'
{"_index":"books","_type":"book","_id":"1","_version":4,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":3,"_primary_term":1}
[root@elk-master ~]# curl -XGET localhost:9200/books/book/1?pretty
{
"_index" : "books",
"_type" : "book",
"_id" : "1",
"_version" : 4,
"_seq_no" : 3,
"_primary_term" : 1,
"found" : true,
"_source" : {
"title" : "Elasticsearch Beginnig",
"author" : "YRJI",
"date" : "2022-10-27",
"pages" : 500,
"category" : "ICT"
}
}
# 파일로 집어넣기
[root@elk-master ~]# vim book_1
[root@elk-master ~]# cat book_1
{
"title" : "Elasticsearch Guide",
"author" : ["Kim", "Lee"],
"date" : "2022-10-27",
"pages" : 300,
"category" : "kakao"
}
[root@elk-master ~]# curl -XPUT localhost:9200/books/book/2 -H 'Content-Type: application/json' -d @book_1
{"_index":"books","_type":"book","_id":"2","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4,"_primary_term":1}
[root@elk-master ~]# curl -XGET localhost:9200/books/book/2?pretty
{
"_index" : "books",
"_type" : "book",
"_id" : "2",
"_version" : 1,
"_seq_no" : 4,
"_primary_term" : 1,
"found" : true,
"_source" : {
"title" : "Elasticsearch Guide",
"author" : [
"Kim",
"Lee"
],
"date" : "2022-10-27",
"pages" : 300,
"category" : "kakao"
}
}
# bulk 파일 넣기
curl -XPOST localhost:9200/_bulk -H 'Content-Type: application/json' --data-binary @ktbooks.json
curl -XPOST localhost:9200/_bulk -H 'Content-Type: application/json' --data-binary @ ktmagazines.json
[root@elk-master LABs]# curl localhost:9200/_cat/indices?v | grep kt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1397 100 1397 0 0 57366 0 --:--:-- --:--:-- --:--:-- 58208
yellow open ktmagazines p-jZ5XW0RXuJqXqOiAICxw 1 1 8 0 11.9kb 11.9kb
yellow open ktbooks HgVW4FGwTQOoqh6--6doog 1 1 10 0 21.8kb 21.8kb
[root@elk-master LABs]# curl -XGET localhost:9200/ktbooks/_count
{"count":10,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
[root@elk-master LABs]# curl -XGET localhost:9200/ktmagazines/_count
{"count":8,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
# ktbooks에서 hamlet을 찾아주세요
curl -XGET 'localhost:9200/ktbooks/_search?q=hamlet&pretty'
{
"took" : 9,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 3.510925,
"hits" : [
{
"_index" : "ktbooks",
"_type" : "book",
"_id" : "5",
"_score" : 3.510925,
"_source" : {
"title" : "Hamlet",
"author" : "William Shakespeare",
"category" : "Tragedies",
"written" : "1599-06-01T12:34:00",
"pages" : 172,
"sell" : 146100000,
"plot" : "The protagonist of Hamlet is Prince Hamlet of Denmark, son of the recently deceased King Hamlet, and nephew of King Claudius, his father's brother and successor. Claudius hastily married King Hamlet's widow, Gertrude, Hamlet's mother. Denmark has a long-standing feud with neighbouring Norway, and an invasion led by the Norwegian prince, Fortinbras, is expected."
}
}
]
}
}
[root@elk-master LABs]# curl -XPOST localhost:9200/_bulk -H 'Content-Type: application/json' --data-binary @shakespeare2.json
# 2개의 레포지토리를 읽어오겠다.
vim /etc/elasticsearch/elasticsearch.yml
path.repo: ["/ELK/data/agg_example", "/ELK/data/search_example"]
[root@elk-master data]# systemctl restart elasticsearch.service
[root@elk-master data]# curl -XGET localhost:9200/_snapshot/_all?pretty
{ }
[root@elk-master data]# curl -XPUT 'localhost:9200/_snapshot/agg_exam' -H 'Content-Type: application/json' -d '{
"type": "fs",
"settings": {
"location" : "/ELK/data/agg_example",
"compress": true
}
}'
{"acknowledged":true}[
[root@elk-master data]# curl -XPUT 'localhost:9200/_snapshot/search_exam' -H 'Content-Type: application/json' -d '{
"type": "fs",
"settings": {
"location" : "/ELK/data/search_example",
"compress": true
}
}'
{"acknowledged":true}
[root@elk-master data]# curl -XGET 'localhost:9200/_snapshot/_all?pretty'
{
"agg_exam" : {
"type" : "fs",
"settings" : {
"compress" : "true",
"location" : "/ELK/data/agg_example"
}
},
"search_exam" : {
"type" : "fs",
"settings" : {
"compress" : "true",
"location" : "/ELK/data/search_example"
}
}
}
[root@elk-master data]# curl -XPOST 'localhost:9200/_snapshot/search_exam/_verify'
{"nodes":{"gjN6pRfgR0yQCJus5FGzMQ":{"name":"elk-master"}}}[
[root@elk-master data]# curl -XPOST "http://localhost:9200/_snapshot/search_exam/movie-search/_restore" -H 'Content-Type: application/json' -d ' {
> "indices": "movie_search"
> ,
> "ignore_unavailable": "true",
> "include_global_state": false
> }'
{"accepted":true}
[root@elurl -XPOST "http://localhost:9200/_snapshot/agg_exam/default/_restore" -H 'Content-Type: application/json' -d ' {
"indices": "apache-web-log",
"ignore_unavailable": "true",
"include_global_state": false
}'
{"accepted":true}
[root@elk-master data]# curl -XPOST "http://localhost:9200/_snapshot/agg_exam/applied-mapping/_restore" -H 'Content-Type: application/json' -d ' {
"indices": "apache-web-log-applied-mapping",
"ignore_unavailable": "true",
"include_global_state": false
}'
{"accepted":true}
[root@elk-master data]# curl -XGET localhost:9200/apache-web-log/_count
{"count":7964,"_shards":{"total":5,"successful":4,"skipped":0,"failed":0}}
[root@elk-master data]# curl -XGET localhost:9200/apache-web-log-applied-mapping/_count
{"count":4037,"_shards":{"total":5,"successful":2,"skipped":0,"failed":0}}
curl -XGET localhost:9200/movie_search/_count
{"count":63069,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0}}
여기까지 백업한거야 ?
[export] ktbookos, -> 마이그레이션, 백업 작업
1) path.repo -> 경로 추가 -> /ELK/data/ktbooks
2) restart elasticsearch
3) snapshot 등록 -> ktbooks_snap
4)/ELK/data/ktbooks
이미 수집이 완료된 데이터를
curl -XGET localhost:9200/_cat/indices?v
/usr/share/elasticsearch/bin/elasticsearch-sql-cli
path.repo: ["/ELK/data/agg_example", "/ELK/data/search_example", "/ELK/data/ktbooks"]
추가
[root@elk-master data]# vim /etc/elasticsearch/elasticsearch.yml
[root@elk-master data]# mkdir -p /ELK/data/ktbooks
[root@elk-master data]# chown -R elasticsearch.elasticsearch /ELK/data/ktbooks/
[root@elk-master data]# grep path.repo /etc/elasticsearch/elasticsearch.yml
path.repo: ["/ELK/data/agg_example", "/ELK/data/search_example", "/ELK/data/ktbooks"]
# elasticsearch 재시작
systemctl restart elasticsearch.service
# 스냅샷으로 등록
curl -XPUT 'localhost:9200/_snapshot/ktbooks_snap' -H 'Content-Type: application/json' -d '{
"type": "fs",
"settings": {
"location" : "/ELK/data/ktbooks",
"compress": true
}
}'
{"acknowledged":true}
[root@elk-master data]# curl -XPUT 'localhost:9200/_snapshot/ktbooks_snap/ktbooks_221027' -H 'Content-Type: application/json' -d '
{
"indices": "ktbooks",
"ignore_unavailable": true,
"include_global_state": true
}'
{"accepted":true}
# 파일ㅇ ㅣ생겻다...!
[root@elk-master data]# ls -l /ELK/data/ktbooks
total 24
-rw-r--r--. 1 elasticsearch elasticsearch 441 Oct 27 16:40 index-0
-rw-r--r--. 1 elasticsearch elasticsearch 8 Oct 27 16:40 index.latest
drwxr-xr-x. 3 elasticsearch elasticsearch 36 Oct 27 16:40 indices
-rw-r--r--. 1 elasticsearch elasticsearch 10749 Oct 27 16:40 meta-ZoYmVH9YQ5uNElQ_hYW8DA.dat
-rw-r--r--. 1 elasticsearch elasticsearch 257 Oct 27 16:40 snap-ZoYmVH9YQ5uNElQ_hYW8DA.dat
[root@elk-master data]# tar cvzf ktbooks.tar.gz /ELK/data/ktbooks
처음에는 imoprt한거고 지금은 exporting 작업을 한것이라고함실습
vim /etc/elasticsearch/elasticsearch.yml
systemctl restart elasticsearch.service
[root@elk-master exam]# chown -R elasticsearch.elasticsearch /ELK/data/exam/
[root@elk-master exam]# curl -XPUT 'localhost:9200/_snapshot/shakespeare' -H 'Content-Type: application/json' -d '{
"type": "fs",
"settings": {
"location" : "/ELK/data/exam",
"compress": true
}
}'
curl -XPUT 'localhost:9200/_snapshot/shakespeare/shakespeare_221027' -H 'Content-Type: application/json' -d '
{
"indices": "shakespeare",
"ignore_unavailable": true,
"include_global_state": true
}'
tar cvzf ss.tar.gz /ELK/data/exam
## 압축 파일 받아서 넣기
curl -XDELETE localhost:9200/shakespeare
1. 파일 넣을 디렉토리 생성
2. chown 💡 path.repo 과 순서 상관 없다.
3. vim /etc/elasticsearch/elasticsearch.yml에 path.repo 추가
4. systemctl restart elasticsearch.service
5. 스냅샷영역에 이 경로를 알려주는거야
[root@elk-master yeji]# curl -XPUT 'localhost:9200/_snapshot/yeji' -H 'Content-Type: application/json' -d '{
"type": "fs",
"settings": {
"location" : "/ELK/data/yeji",
"compress": true
}
}'
{"acknowledged":true}
6.
curl -XPOST "localhost:9200/_snapshot/yeji/shakespeare_221027/_restore" -H 'Content-Type: application/json' -d '
{
"indices": "shakespeare",
"ignore_unavailable": "true",
"include_global_state": false
}'
메모장
⭐ 📘 📗 💭 🤔 📕 📔 🐳 ✍ 🥳 ⭐ 🐣 👻 💡 💻
- 엘라스틱 Dock
https://www.elastic.co/guide/index.html
기업에서 nginx 왜 써? -> 웹 서비스 제공 -> 이때 로그 수집 왜 해?
UDP -> VoIP -> NLB
DNS 몇 계층이양
[root@elk-master ~]# systemctl daemon-reload
You have mail in /var/spool/mail/root
?
스레드와 프로세스의 차이
안녕하세요 😄