elastic search dead 해결

도커로 ELK를 구축하여 운영중인데 자꾸 elastic search만 죽는 현상이 발생했다.

로그를 확인해보니 out of memory라고 한다...

java.lang.OutOfMemoryError: Java heap space
	at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:364)
	at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:225)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476)
	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656)
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:605)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258)
	at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:158)
	at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:55)
	at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:29)
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
	at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:359)
	at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:341)
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
	at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1694)
	at org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1673)
	at org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:3506)
	at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:880)
	at org.elasticsearch.index.IndexService.access$200(IndexService.java:102)
	at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1012)
	at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:133)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/docker-cluster.log

해당 오류는 찾아보니 힙 메모리가 부족해서 발생할 수 있다고 한다.

힙 옵션을 수정해서 해결해주면 된다.

공식 가이드 문서에서는 전체 서버 메모리의 50%로 지정해주라고 하지만.. 다른 컨테이너의 엘라스틱서치가 해당 옵션인데 죽지 않아서 다음과 같이 변경하였다.

# 기존
ES_JAVA_OPTS: "-Xmx512m -Xms512m"


# 새 설정
ES_JAVA_OPTS: "-Xmx1024m -Xms512m"