Question

当尝试使用Powershell终端启动Spark历史记录服务器（从我的SPARK_HOME / sbin）

.\start-history-server.sh

使用以下消息启动Windows终端，然后关闭。

ps: unknown option -- o
Try `ps --help' for more information.
starting org.apache.spark.deploy.history.HistoryServer, logging to C:\Spark/logs/spark--org.apache.spark.deploy.history.HistoryServer-1-<my-machine>.out
ps: unknown option -- o
Try `ps --help' for more information.
ps: unknown option -- o
Try `ps --help' for more information.
ps: unknown option -- o
Try `ps --help' for more information.
ps: unknown option -- o
Try `ps --help' for more information.

此处在' C：\ Spark \ logs '

中生成的spark--org.apache.spark.deploy.history.HistoryServer-1-<my-machine>.out中输出

Spark Command: C:\Program Files (x86)\Java\jre1.8.0_161\bin\java -cp C:\Spark/conf\;C:\Spark\jars\* -Xmx1g org.apache.spark.deploy.history.HistoryServer C:\Spark\logs
========================================
"C:\Program Files (x86)\Java\jre1.8.0_161\bin\java" -cp "C:\Spark/conf\;C:\Spark\jars\*" -Xmx1g org.apache.spark.deploy.history.HistoryServer C:\Spark\logs 
C:\Spark/bin/spark-class: line 96: CMD: bad array subscript

我已经尝试过的方法：

更新了“ spark-defaults.conf”，如下所示：

spark.eventLog.enabled           true
spark.eventLog.dir               file:///C:\Spark\logs
spark.history.fs.logDirectory    file:///C:\Spark\logs

[也在此讨论之后]（cannot start spark history server）我已经尝试运行以下命令（来自SPARK_HOME / sbin）

spark-class org.apache.spark.deploy.history.HistoryServer

但是它导致FileNotFound异常如下：（这很奇怪，因为它以某种方式试图寻找C:Sparklogs而不是C:\Spark\logs）

PS C:\Spark\sbin> spark-class org.apache.spark.deploy.history.HistoryServer                                                                                                                  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/08/26 12:18:03 INFO HistoryServer: Started daemon with process name: 24364@<my-machine>
20/08/26 12:18:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/08/26 12:18:03 INFO SecurityManager: Changing view acls to: <USER>
20/08/26 12:18:03 INFO SecurityManager: Changing modify acls to: <USER>
20/08/26 12:18:03 INFO SecurityManager: Changing view acls groups to:
20/08/26 12:18:03 INFO SecurityManager: Changing modify acls groups to:
20/08/26 12:18:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(<USER>); groups with view permissions: Set(); users  with modify permissions: Set(<USER>); groups with modify permissions: Set()
20/08/26 12:18:04 INFO FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; groups with admin permissions
20/08/26 12:18:05 INFO Utils: Successfully started service on port 18080.
20/08/26 12:18:05 INFO HistoryServer: Bound HistoryServer to 0.0.0.0, and started at http://my-machine:18080
Exception in thread "main" java.io.FileNotFoundException: Log directory specified does not exist: file:///C:Sparklogs
        at org.apache.spark.deploy.history.FsHistoryProvider.startPolling(FsHistoryProvider.scala:279)
        at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:227)
        at org.apache.spark.deploy.history.FsHistoryProvider.start(FsHistoryProvider.scala:409)
        at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:303)
        at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: File file:/C:Sparklogs does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:428)
        at org.apache.spark.deploy.history.FsHistoryProvider.startPolling(FsHistoryProvider.scala:269)

任何人都可以提出其他建议，以解决此问题并启动Spark History服务器吗？

谢谢。

Answer 1

更新： 以下内容有效 ：

将我的日志“ spark.eventLog.dir”和“ spark.history.fs.logDirectory”更新为：“ file：/// C：/ Spark / eventlog”
从SPARKHOME / sbin中执行Spark-class org.apache.spark.deploy.history.HistoryServer
现在可以从以下位置访问历史记录服务器Web ui：http://localhost:18080

无法在Windows 10计算机中启动``火花历史记录服务器''

1 个答案: