[Trouble Shooting] Zeppelin 구동 시 Lucene 에러 해결

Chan hae OH·2023년 6월 7일
trouble shootingzeppelin데이터플랫폼운영
0

1. 문제 발단


서버실의 전기 공사 때문에 S/W 를 내려달라는 요청을 수행했다.

Hadoop ECO를 전부 내리고 공사를 마무리 한 후 다시금 올리는 상황에서 관리 툴을 활용하여 서비스를 All start 하는 도중 어떤 이유인지는 모르겠지만 도중에 멈췄다. 그래서 다시 All stop 을 하고 올리니 Zeppelin Web UI가 떴지만 LDAP 연동을 계속 실패했다.

무슨 이유인지는 모르지만 평소에 Zeppelin config 설정을 하기 위해 자주 내리고 올렸었고 LDAP 연동에는 문제가 없었으니 config 문제가 아님을 직감했다.



2. 문제 로그


1. 서버를 올렸을 때 나온 로그


ZEPPELIN_CLASSPATH: :/opt/apps/zeppelin-0.10.1/lib/*:/opt/apps/zeppelin-0.10.1/*::/opt/apps/zeppelin-0.10.1/conf
Exception in thread "main" MultiException stack 1 of 8
java.nio.file.NoSuchFileException: /opt/apps/zeppelin-0.10.1/conf/interpreter.json
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:447)
        at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
        at java.nio.file.Files.move(Files.java:1395)
        at org.apache.zeppelin.util.FileUtils.atomicWriteToFile(FileUtils.java:60)
        at org.apache.zeppelin.util.FileUtils.atomicWriteToFile(FileUtils.java:71)
        at org.apache.zeppelin.storage.LocalConfigStorage.save(LocalConfigStorage.java:54)
        at org.apache.zeppelin.interpreter.InterpreterSettingManager.saveToFile(InterpreterSettingManager.java:350)
        at org.apache.zeppelin.interpreter.InterpreterSettingManager.init(InterpreterSettingManager.java:380)
        at org.apache.zeppelin.interpreter.InterpreterSettingManager.<init>(InterpreterSettingManager.java:200)
        at org.apache.zeppelin.interpreter.InterpreterSettingManager.<init>(InterpreterSettingManager.java:160)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1356)
        at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:248)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:342)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:679)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:205)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:679)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:205)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
        at org.glassfish.hk2.utilities.ServiceLocatorUtilities.getService(ServiceLocatorUtilities.java:669)
        at org.apache.zeppelin.server.ZeppelinServer.main(ZeppelinServer.java:274)
...

2. 로그인을 시도 했을 때 나온 로그


Jun 07, 2023 6:49:23 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: Unknown HK2 failure detected:
MultiException stack 1 of 6
java.io.IOException: Failed to create new IndexWriter
        at org.apache.zeppelin.search.LuceneSearch.<init>(LuceneSearch.java:103)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1356)
        at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:248)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:342)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:679)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:205)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:666)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:205)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160)
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30)
        at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105)
        at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260)
        at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
        at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:245)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
        at org.apache.zeppelin.server.CorsFilter.doFilter(CorsFilter.java:64)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
        at io.micrometer.core.instrument.binder.jetty.TimedHandler.handle(TimedHandler.java:120)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:501)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/tmp/zeppelin-index/segments_2u")))
        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:293)
        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1037)
        at org.apache.zeppelin.search.LuceneSearch.<init>(LuceneSearch.java:101)
        ... 123 more
Caused by: java.nio.file.NoSuchFileException: /tmp/zeppelin-index/_21.si
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
        at java.nio.channels.FileChannel.open(FileChannel.java:287)
        at java.nio.channels.FileChannel.open(FileChannel.java:335)
        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
        at org.apache.lucene.codecs.lucene86.Lucene86SegmentInfoFormat.read(Lucene86SegmentInfoFormat.java:91)
        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:357)
        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291)
        ... 125 more
        Suppressed: org.apache.lucene.index.CorruptIndexException: checksum passed (6f58f5de). possibly transient resource issue, or a Lucene or JVM bug (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/tmp/zeppelin-index/segments_2u")))
                at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:466)
                at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:434)
                ... 126 more

...



3. 문제 원인 분석


1. shiro.ini (LDAP)


에러 로그를 봤을 때 당장 감이 잡히지는 않아서 가장 먼저 본 곳은 shiro.ini 설정이다.
하지만 매번 잘 사용하던 config 이 갑자기 node 전원이 내려갔다 올라왔다고 해서 잘 못 될 일은 없을 것이다.

역시나 설정에는 문제가 없었고, 관리 툴이 해당 shiro.ini 이 아닌 backup 을 보고 있지도 않았다.


2. interpreter.json 의 존재 유무


java.nio.file.NoSuchFileException: /opt/apps/zeppelin-0.10.1/conf/interpreter.json 해당 에러로 interperter.json 파일이 없나 확인 했지만 멀쩡히 존재하고 있었다.


3. 내부 검색 엔진의 깨짐


로그인을 했을 때 LDAP 을 못 찾겠다는 에러가 아닌 아래와 같은 에러가 계속 신경이 쓰였다.

java.io.IOException: Failed to create new IndexWriter
        at org.apache.zeppelin.search.LuceneSearch.<init>(LuceneSearch.java:103)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at

왜 이런 에러가 나는 것일까? LuceneSearch 라는 부분이 계속 신경이 쓰인다.



4. 문제 심층 분석


먼저 Zeppelin 의 LuceneSearch 코드를 뜯어보자.

GITHUB ZEPPELIN REPOSITORY > LuceneSearch.java 

...

/**
 * Search (both, indexing and query) the notebooks using Lucene. Query is thread-safe, as creates
 * new IndexReader every time. Index is thread-safe, as re-uses single IndexWriter, which is
 * thread-safe.
 */
public class LuceneSearch extends SearchService {
  private static final Logger LOGGER = LoggerFactory.getLogger(LuceneSearch.class);

  private static final String SEARCH_FIELD_TEXT = "contents";
  private static final String SEARCH_FIELD_TITLE = "header";
  private static final String PARAGRAPH = "paragraph";
  private static final String ID_FIELD = "id";

  private final Directory indexDirectory;
  private final IndexWriter indexWriter;
  private final Notebook notebook;

  @Inject
  public LuceneSearch(ZeppelinConfiguration conf, Notebook notebook) throws IOException {
    super("LuceneSearch");
    this.notebook = notebook;
    if (conf.isZeppelinSearchUseDisk()) {
      try {
        final Path indexPath = Paths.get(conf.getZeppelinSearchIndexPath());
        this.indexDirectory = FSDirectory.open(indexPath);
        LOGGER.info("Use {} for storing lucene search index", indexPath);
      } catch (IOException e) {
        throw new IOException("Failed to create index directory for search service.", e);
      }
    } else {
      this.indexDirectory = new ByteBuffersDirectory();
    }
    final Analyzer analyzer = new StandardAnalyzer();
    final IndexWriterConfig indexWriterConfig = new IndexWriterConfig(analyzer);
    try {
      this.indexWriter = new IndexWriter(indexDirectory, indexWriterConfig);
    } catch (IOException e) {
      throw new IOException("Failed to create new IndexWriter", e);
    }
    if (conf.isIndexRebuild()) {
      notebook.addInitConsumer(this::addNoteIndex);
    }
    this.notebook.addNotebookEventListener(this);
  }

...

먼저 에러가 init 에서 발생 했으니 생성 할 때를 보겠다.

해당 코드에서 에러 문구를 만들어주는 throw new IOException("Failed to create new IndexWriter", e); 코드를 발견할 수 있다.

그리고 아래 2코드를 통해 LuceneSearch 가 무언가를 쓸려고 할 때의 경로를 찾는 것을 알 수 있다.

  • this.indexDirectory = FSDirectory.open(indexPath);
  • this.indexWriter = new IndexWriter(indexDirectory, indexWriterConfig);

그렇다면 indexPath 는 어디서 온걸까?

final Path indexPath = Paths.get(conf.getZeppelinSearchIndexPath()); 이 코드에서 매개변수인 conf 즉 ZeppelinConfiguration 에서 index 경로를 가져오고 있는 것이다.


그렇다면 이 경로를 지정한 코드를 분석 해보자.

GITHUB ZEPPELIN REPOSITORY > ZeppelinConfiguration.java 

...
 ZEPPELIN_SEARCH_USE_DISK("zeppelin.search.use.disk", true),
 ZEPPELIN_SEARCH_INDEX_PATH("zeppelin.search.index.path", "/tmp/zeppelin-index"),
 ZEPPELIN_JOBMANAGER_ENABLE("zeppelin.jobmanager.enable", false),
 ...

/tmp/zeppelin-index 라는 경로의 인덱스를 바라 보고 있는 것이다. 그러나 입출력에 문제가 있어 Failed to create new IndexWriter 를 일으키는 것을 확인했다.



5. 문제 해결


/tmp/zeppelin-index 해당 경로의 Directory 의 권한 문제인가 해서 권한만 바꿔 줄려다가

  • java.nio.file.NoSuchFileException: /tmp/zeppelin-index/_21.si
  • Suppressed: org.apache.lucene.index.CorruptIndexException: checksum passed (6f58f5de)

이 2 개의 에러를 보고 권한 문제보다는 파일이 작성중이다가 깨진것이 아닌가 추측하여 /tmp/zeppelin-index_bak 으로 변경 후 다시 Zeppelin을 실행 시켰다.

보란 듯이 멀쩡하게 Zeppelin 이 동작했다.

profile
Chan hae OH
Data Engineer
이전 포스트

Kafka-connect 예제

다음 포스트

[데이터 플랫폼 운영 / 개발] - Hive Tez

0개의 댓글