[Apache Flink] Job 추가시 Downtime에 관하여

개발하려는 서비스에서 Apache Flink 클러스터에 Job을 동적으로 추가하려고 한다. 이 때 과연 Job 추가시 Downtime이 발생하는지 의문이 생긴다. 이 글에선 Apache Flink의 Job과 관련된 downtime과 더 나아가 Flink의 신뢰성에 대해 알아보고자 한다.

Job 추가 시 downtime은 발생하는가?

아래 phind의 대답에 의하면, 각 Job은 독립적이기에 추가하는 것에는 downtime은 발생하지 않는다고 한다.

phind와의 대화

질문

query: Does adding a job in Apache Flink cause downtime

대답

Adding a new job to Apache Flink without deleting or stopping an existing job generally does not cause downtime. Each job in Flink runs in its own separate execution environment, and thus adding a new job does not interfere with the running of existing jobs Source 0.

In Flink, jobs are independent of each other. Therefore, you can submit multiple jobs to a Flink cluster, and they will execute concurrently without affecting each other Source 11.

Here is a simple example of how you can submit a new job to a running Flink cluster:

./bin/flink run -c com.example.MyJob /path/to/myjob.jar

In this command, com.example.MyJob is the fully qualified name of the job's main class, and /path/to/myjob.jar is the path to the JAR file containing the job Source 11.

However, if you want to update the code of an existing job, you would have to stop the current job (which may cause a brief period of downtime), update the code, and then restart the job from a savepoint Source 4.

But, if you're simply adding a new, separate job to the cluster, it should not cause any downtime for the existing jobs.

Apache Flink는 어떻게 신뢰성을 확보하는가? 내결함성 관련하여...

참고 자료

• Jobs and Scheduling
• Savepoints
• Fault Tolerance via State Snapshots