Spark History Server 오류

Spark History Server 관련하여 발생하는 오류들을 별도로 모아 정리하고자 한다.

 

FileNotFoundException : File file:/tmp/spark-events does not exist

./start-history-server.sh 실행 시 아래와 같은 오류가 발생하게 되는 경우

failed to launch: nice -n 0 /usr/local/Cellar/apache-spark/3.3.0/libexec/bin/spark-class org.apache.spark.deploy.history.HistoryServer
  	at org.apache.spark.deploy.history.FsHistoryProvider.start(FsHistoryProvider.scala:421)
  	at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:311)
  	at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
  Caused by: java.io.FileNotFoundException: File file:/tmp/spark-events does not exist
  	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:779)
  	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1100)
  	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:769)
  	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462)
  	at org.apache.spark.deploy.history.FsHistoryProvider.startPolling(FsHistoryProvider.scala:289)
  	... 4 more

 

조치 방법으로는 spark-default.conf에 있는 spark.history.fs.logDirectory 설정을 추가한다.

spark.eventLog.enabled true
spark.eventLog.dir hdfs:///<hdfs 생성한 로그 디렉토리>
spark.history.fs.logDirectory hdfs:///<hdfs 생성한 로그 디렉토리>