Redis Master-Slave를 적용 및 failover를 확인하자!

Karim·2021년 12월 22일
redis
6

Redis 운영

목록 보기
6/7
post-thumbnail

1. Version

💬

  • OS : CentOS Linux release 7.9.2009 (Core)
  • Redis : 6.0.7

2. Master-Slave / failover 란

💬 구성 그림

💬 Master

  • Redis에서 두뇌 역할
  • 모든 명령어 처리를 한다.
  • Master만 있으면 failover 상황 시 Redis는 data loss가 발생 한다.

💬 Slave

  • Master의 복제본
  • Read only로 Master의 데이터를 설정(실시간/배치)에 따라 읽는다.
  • failover 상황 시 Master의 역할을 한다.

💬 Failover

  • master 서버가 죽으면 그 해당 slave 서버가 master가 된다.
  • 죽었던 master 서버가 올라오면 master 서버가 된 slave 서버는 계속 master고
    올라온 master 서버는 slave가 된다.

3. 구성 전 확인상황

💬 한마디

  • 저는 테스트 서버가 한정되어 있어서 한 서버에 포트 6개를 띄어서 3(master)-3(slave) 묶었어요~

💬 slave 서버 구축

  • slave 서버로 사용할 서버를 올려줘야 한다!

4. Slave로 이용할 서버 Cluster 로 묶기

💬 meet 옵션

./redis-cli -h 192.168.124.xxx -p 1111 cluster meet 192.168.124.xxx 2111

$ ./redis-cli -h 192.168.124.xxx -p 1111 cluster meet 192.168.124.xxx 2111
OK
$ ./redis-cli -h 192.168.124.xxx -p 1111 cluster meet 192.168.124.xxx 2112
OK
$ ./redis-cli -h 192.168.124.xxx -p 1111 cluster meet 192.168.124.xxx 2113
OK

💬 cluster 확인

  • 6대 다 마스터로 띄어진다.
  • 이제 이 중 3개는 slave로 설정을 해야한다.
    ./redis-cli -p 1111 -c cluster nodes

    $ ./redis-cli -p 1111 -c cluster nodes
    b623c43868017a4e52bb33d17d089b642c7a56e0 192.168.124.238:2112@12112 master - 0 1640155992000 4 connected
    00a8c78a20a3b627c8320419eccb2b17b768373d 192.168.124.238:2113@12113 master - 0 1640155993292 5 connected
    037e904c9ebd98612a42f6920610162fdbf2610b 192.168.124.238:2111@12111 master - 0 1640155992000 3 connected
    b054593fdd1b3183b70ed262b2525dbf0ccf333e 192.168.124.238:1111@11111 myself,master - 0 1640155991000 1 connected 0-5460
    3ba4750956e7c9d31ae0f81e8d59eda2f4d427ea 192.168.124.238:1113@11113 master - 0 1640155994294 2 connected 10923-16383
    c495801a7be18b251ade5acfe2e30a68cd352cb7 192.168.124.238:1112@11112 master - 0 1640155992291 0 connected 5461-10922

5. Slave로 설정

💬 cluster replicate 옵션

./redis-cli -h 192.168.124.xxx -p [slave port] cluster replicate [master ID]

$ ./redis-cli -h 192.168.124.xxx -p 2111 cluster replicate b054593fdd1b3183b70ed262b2525dbf0ccf333e
OK
$ ./redis-cli -h 192.168.124.xxx -p 2112 cluster replicate c495801a7be18b251ade5acfe2e30a68cd352cb7
OK
$ ./redis-cli -h 192.168.124.xxx -p 2113 cluster replicate 3ba4750956e7c9d31ae0f81e8d59eda2f4d427ea
OK

6. Slave 확인

💬 -c nodes

  • 확인
  • [자기 salve ID] 192.168.124.xxx:2112@12112 slave [설정된 master ID] 0 1640156302857 0 connected

    $ ./redis-cli -p 1111 -c cluster nodes
    b623c43868017a4e52bb33d17d089b642c7a56e0 192.168.124.xxx:2112@12112 slave c495801a7be18b251ade5acfe2e30a68cd352cb7 0 1640156302857 0 connected
    00a8c78a20a3b627c8320419eccb2b17b768373d 192.168.124.xxx:2113@12113 slave 3ba4750956e7c9d31ae0f81e8d59eda2f4d427ea 0 1640156304860 2 connected
    037e904c9ebd98612a42f6920610162fdbf2610b 192.168.124.xxx:2111@12111 slave b054593fdd1b3183b70ed262b2525dbf0ccf333e 0 1640156304000 1 connected
    b054593fdd1b3183b70ed262b2525dbf0ccf333e 192.168.124.xxx:1111@11111 myself,master - 0 1640156303000 1 connected 0-5460
    3ba4750956e7c9d31ae0f81e8d59eda2f4d427ea 192.168.124.xxx:1113@11113 master - 0 1640156302000 2 connected 10923-16383
    c495801a7be18b251ade5acfe2e30a68cd352cb7 192.168.124.xx:1112@11112 master - 0 1640156303859 0 connected 5461-10922

💬 info

  • 확인

    $ ./redis-cli -p 1111 info

  • role : 자기 자신 상태
  • connected_slaves : 설정된 slave의 수
  • slave0 : 설정된 slave 환경 내용
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.124.xxx,port=2111,state=online,offset=644,lag=1
master_replid:20cb0cc277057407e6ac34624a697718fd2063a2
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:644
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:644

7. Failover 확인

💬 failover test 시나리오

  • 현재 마스터 상태인 1111 포트의 redis를 죽인다.
  • 1111 포트의 slave로 설정된 2111 이 master로 변경되는지 확인한다.
  • 다시 1111 포트를 기동한다.

💬 failover 시 slave log

  • Cluster state changed: ok 로 slave에서 master로 변환된 것을 log로 확인할 수 있다.
5925:S 22 Dec 2021 16:08:41.415 # Connection with master lost.
5925:S 22 Dec 2021 16:08:41.415 * Caching the disconnected master state.
5925:S 22 Dec 2021 16:08:42.383 * Connecting to MASTER 192.168.124.xxx:1111
5925:S 22 Dec 2021 16:08:42.383 * MASTER <-> REPLICA sync started
5925:S 22 Dec 2021 16:08:42.383 # Error condition on socket for SYNC: Operation now in progress
5925:S 22 Dec 2021 16:08:43.386 * Connecting to MASTER 192.168.124.xxx:1111
5925:S 22 Dec 2021 16:08:54.417 * MASTER <-> REPLICA sync started
5925:S 22 Dec 2021 16:08:54.417 # Error condition on socket for SYNC: Operation now in progress
5925:S 22 Dec 2021 16:08:55.420 * Connecting to MASTER 192.168.124.xxx:1111
5925:S 22 Dec 2021 16:08:55.421 * MASTER <-> REPLICA sync started
5925:S 22 Dec 2021 16:08:55.421 # Error condition on socket for SYNC: Operation now in progress
5925:S 22 Dec 2021 16:08:56.423 * Connecting to MASTER 192.168.124.xxx:1111
5925:S 22 Dec 2021 16:08:56.423 * MASTER <-> REPLICA sync started
...
5925:S 22 Dec 2021 16:08:56.423 # Error condition on socket for SYNC: Operation now in progress
5925:S 22 Dec 2021 16:08:57.426 * Connecting to MASTER 192.168.124.xxx:1111
5925:S 22 Dec 2021 16:08:57.426 * MASTER <-> REPLICA sync started
5925:S 22 Dec 2021 16:08:57.427 # Error condition on socket for SYNC: Operation now in progress
5925:S 22 Dec 2021 16:08:58.429 * Connecting to MASTER 192.168.124.xxx:1111
5925:S 22 Dec 2021 16:08:58.430 * MASTER <-> REPLICA sync started
5925:S 22 Dec 2021 16:08:58.430 # Error condition on socket for SYNC: Operation now in progress
5925:S 22 Dec 2021 16:08:59.432 * Connecting to MASTER 192.168.124.xxx:1111
5925:S 22 Dec 2021 16:08:59.432 * MASTER <-> REPLICA sync started
5925:S 22 Dec 2021 16:08:59.432 # Error condition on socket for SYNC: Operation now in progress
5925:S 22 Dec 2021 16:09:00.425 * FAIL message received from 3ba4750956e7c9d31ae0f81e8d59eda2f4d427ea about b054593fdd1b3183b70ed262b2525dbf0ccf333e
5925:S 22 Dec 2021 16:09:00.425 # Cluster state changed: fail
5925:S 22 Dec 2021 16:09:00.435 * Connecting to MASTER 192.168.124.xxx:1111
5925:S 22 Dec 2021 16:09:00.436 * MASTER <-> REPLICA sync started
5925:S 22 Dec 2021 16:09:00.436 # Start of election delayed for 735 milliseconds (rank #0, offset 1050).
5925:S 22 Dec 2021 16:09:00.436 # Error condition on socket for SYNC: Operation now in progress
5925:S 22 Dec 2021 16:09:01.238 # Starting a failover election for epoch 6.
5925:S 22 Dec 2021 16:09:01.241 # Failover election won: I'm the new master.
5925:S 22 Dec 2021 16:09:01.241 # configEpoch set to 6 after successful failover
5925:M 22 Dec 2021 16:09:01.241 * Discarding previously cached master state.
5925:M 22 Dec 2021 16:09:01.241 # Setting secondary replication ID to 20cb0cc277057407e6ac34624a697718fd2063a2, valid up to offset: 1051. New replication ID is 9a86e1290367698013c708755671fa2a360221f9

5925:M 22 Dec 2021 16:09:01.241 # Cluster state changed: ok

💬 slave였던 redis info

  • 확인

    $ ./redis-cli -p 2111 info

  • role이 master로 바뀐것을 확인할 수 있다.
# Replication
role:master
connected_slaves:0
master_replid:9a86e1290367698013c708755671fa2a360221f9
master_replid2:20cb0cc277057407e6ac34624a697718fd2063a2
master_repl_offset:1050
second_repl_offset:1051
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:1050

💬 -c nodes

  • 확인
  • 기존에 master였던 1111은 상태가 fail / disconnected 되어있는 것을 확인할 수 있다.
  • 기존에 slave였던 2111은 상태가 master가 된 것을 확인할 수 있다.

    $ ./redis-cli -p 2111 -c cluster nodes
    3ba4750956e7c9d31ae0f81e8d59eda2f4d427ea 192.168.124.xxx:1113@11113 master - 0 1640157101816 2 connected 10923-16383
    037e904c9ebd98612a42f6920610162fdbf2610b 192.168.124.xxx:2111@12111 myself,master - 0 1640157099000 6 connected 0-5460
    c495801a7be18b251ade5acfe2e30a68cd352cb7 192.168.124.xxx:1112@11112 master - 0 1640157100813 0 connected 5461-10922
    b623c43868017a4e52bb33d17d089b642c7a56e0 192.168.124.xxx:2112@12112 slave c495801a7be18b251ade5acfe2e30a68cd352cb7 0 1640157102818 0 connected
    b054593fdd1b3183b70ed262b2525dbf0ccf333e 192.168.124.xxx:1111@11111 master,fail - 1640156925291 1640156920278 1 disconnected
    00a8c78a20a3b627c8320419eccb2b17b768373d 192.168.124.xxx:2113@12113 slave 3ba4750956e7c9d31ae0f81e8d59eda2f4d427ea 0 1640157101000 2 connected

8. Failover 후 기존 master 기동 확인

  • 기존 master redis server 기동

💬 기존 master 기동 시 slave log

  • Synchronization with replica 192.168.124.xxx:1111 succeeded 로 slave가 붙었다는 것을 확인할 수 있다.
5925:M 22 Dec 2021 16:16:03.473 * Clear FAIL state for node b054593fdd1b3183b70ed262b2525dbf0ccf333e: master without slots is reachable again.
5925:M 22 Dec 2021 16:16:04.475 * Replica 192.168.124.xxx:1111 asks for synchronization
5925:M 22 Dec 2021 16:16:04.475 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '546eef4a9bb0ad309e23e06c5b905896fa060b45', my replication IDs are '9a86e1290367698013c708755671fa2a360221f9' and '20cb0cc277057407e6ac34624a697718fd2063a2')
5925:M 22 Dec 2021 16:16:04.475 * Starting BGSAVE for SYNC with target: disk
5925:M 22 Dec 2021 16:16:04.476 * Background saving started by pid 15447
15447:C 22 Dec 2021 16:16:04.478 * DB saved on disk
15447:C 22 Dec 2021 16:16:04.479 * RDB: 0 MB of memory used by copy-on-write
5925:M 22 Dec 2021 16:16:04.577 * Background saving terminated with success
5925:M 22 Dec 2021 16:16:04.577 * Synchronization with replica 192.168.124.xxx:1111 succeeded

💬 -c nodes

  • 확인.
  • 기존에 slave였던 2111은 상태가 master로 유지된 것을 확인할 수 있다.
  • 기존에 master였던 1111은 상태가 2111의 slave로 된것을 확인할 수 있다.

    $ ./redis-cli -p 1111 -c cluster nodes
    b623c43868017a4e52bb33d17d089b642c7a56e0 192.168.124.xxx:2112@12112 slave c495801a7be18b251ade5acfe2e30a68cd352cb7 0 1640157560000 0 connected
    00a8c78a20a3b627c8320419eccb2b17b768373d 192.168.124.xxx:2113@12113 slave 3ba4750956e7c9d31ae0f81e8d59eda2f4d427ea 0 1640157562817 2 connected
    037e904c9ebd98612a42f6920610162fdbf2610b 192.168.124.xxx:2111@12111 master - 0 1640157561000 6 connected 0-5460
    b054593fdd1b3183b70ed262b2525dbf0ccf333e 192.168.124.xxx:1111@11111 myself,slave 037e904c9ebd98612a42f6920610162fdbf2610b 0 1640157559000 6 connected
    c495801a7be18b251ade5acfe2e30a68cd352cb7 192.168.124.xxx:1112@11112 master - 0 1640157560813 0 connected 5461-10922
    3ba4750956e7c9d31ae0f81e8d59eda2f4d427ea 192.168.124.xxx:1113@11113 master - 0 1640157561816 2 connected 10923-16383

📌 여담

  • redis의 꽃은 master-slvae 구조 같다~

📚 참고

profile
Karim
나도 보기 위해 정리해 놓은 벨로그
이전 포스트

Redis Cluster 및 Slot 를 구성해보자!

다음 포스트

Redis String type redis-cli command를 알아보자!

0개의 댓글