Question

一直在尝试习惯新的结构化流式传输，但是一旦我开始.writeStream查询，它就会一直让我失误。

知道可能导致这种情况的原因是什么？如果你在Local和HDFS之间拆分检查点和元数据文件夹，我可以找到最接近的Spark bug，但是。在Windows 10，Spark 2.2和IntelliJ上运行。

17/08/29 21:47:39 ERROR StreamMetadata: Error writing stream metadata StreamMetadata(41dc9417-621c-40e1-a3cb-976737b83fb7) to C:/Users/jason/AppData/Local/Temp/temporary-b549ee73-6476-46c3-aaf8-23295bd6fa8c/metadata
ExitCodeException exitCode=-1073741515: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
    at org.apache.hadoop.util.Shell.run(Shell.java:479)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
    at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
    at org.apache.spark.sql.execution.streaming.StreamMetadata$.write(StreamMetadata.scala:76)
    at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$6.apply(StreamExecution.scala:116)
    at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$6.apply(StreamExecution.scala:114)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.execution.streaming.StreamExecution.<init>(StreamExecution.scala:114)
    at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:240)
    at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:278)
    at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:282)
    at FileStream$.main(FileStream.scala:157)
    at FileStream.main(FileStream.scala)
Exception in thread "main" ExitCodeException exitCode=-1073741515: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
    at org.apache.hadoop.util.Shell.run(Shell.java:479)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
    at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
    at org.apache.spark.sql.execution.streaming.StreamMetadata$.write(StreamMetadata.scala:76)
    at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$6.apply(StreamExecution.scala:116)
    at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$6.apply(StreamExecution.scala:114)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.execution.streaming.StreamExecution.<init>(StreamExecution.scala:114)
    at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:240)
    at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:278)
    at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:282)
    at FileStream$.main(FileStream.scala:157)
    at FileStream.main(FileStream.scala)
17/08/29 21:47:39 INFO SparkContext: Invoking stop() from shutdown hook
17/08/29 21:47:39 INFO SparkUI: Stopped Spark web UI at http://192.168.178.21:4040
17/08/29 21:47:39 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/08/29 21:47:39 INFO MemoryStore: MemoryStore cleared
17/08/29 21:47:39 INFO BlockManager: BlockManager stopped
17/08/29 21:47:39 INFO BlockManagerMaster: BlockManagerMaster stopped
17/08/29 21:47:39 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/08/29 21:47:39 INFO SparkContext: Successfully stopped SparkContext
17/08/29 21:47:39 INFO ShutdownHookManager: Shutdown hook called
17/08/29 21:47:39 INFO ShutdownHookManager: Deleting directory C:\Users\jason\AppData\Local\Temp\temporary-b549ee73-6476-46c3-aaf8-23295bd6fa8c
17/08/29 21:47:39 INFO ShutdownHookManager: Deleting directory C:\Users\jason\AppData\Local\Temp\spark-117ed625-a588-4dcb-988b-2055ec5fa7ec

Process finished with exit code 1

Answer 1

实际上，我在本地计算机上运行Spark单元测试时遇到了同样的问题。这是由％HADOOP_HOME％文件夹中 WinUtils.exe 失败引起的：

输入： %HADOOP_HOME%\bin\winutils.exe chmod 777 %SOME_TEMP_DIRECTORY%

输出

winutils.exe - 系统错误
代码执行无法继续，因为找不到MSVCR100.dll 重新安装该程序可能会解决此问题。

在网上冲浪后，我发现了Steve Loughran的winutils项目存在问题：Windows 10: winutils.exe doesn't work。
特别是它说安装VC ++可再发行组件应该可以解决问题（在我的情况下也是如此）：How do I fix this error "msvcp100.dll is missing"

Answer 2

这是一个Windows问题

程序无法启动，因为您的计算机缺少MSVCP100.dll。尝试重新安装程序以解决此问题

您需要安装VC ++可再发行软件包：

从中下载Microsoft Visual C ++ 2010 Redistributable Package（x86）官方Microsoft下载中心

http://www.microsoft.com/en-us/download/details.aspx?id=5555

安装vcredist_x86.exe

从中下载Microsoft Visual C ++ 2010 Redistributable Package（x64）官方Microsoft下载中心

http://www.microsoft.com/en-us/download/details.aspx?id=14632

安装vcredist_x64.exe

Answer 3

就我而言，我使用的是Windows 10，不得不更改环境变量-> 用户变量

TMP 和 TEMP 到其他卷（D：\ Temp或E：\ Temp等）中的“自定义位置”，而不是默认值

％USERPROFILE％\ AppData \ Local \ Temp

并设置HADOOP_HOME

System.setProperty（“ hadoop.home.dir”， “ $ HADOOP_HOME \ winutils-master \ hadoop-2.x.x”）

不要忘记将 hadoop.dll 复制到 C：\ Windows \ System32 。

您可以从此链接下载适当的版本。 DOWNLOAD WINUTILS.exe

对我来说 hadoop-2.7.1 版本已解决了该问题。

Answer 4

将DataFrame转换为Parquet文件时发生的问题。它将创建目录，但失败，并显示“ ExitCodeException exitCode = -1073741515”。

我在Windows 10（版本2004）上从Intellij 2020.2.2 x64运行了Spark。我在C驱动器的GitBash中安装了spark-3.0.1-bin-hadoop3.2。从这个仓库https://github.com/cdarlint/winutils重定向后，我从这个Github仓库https://github.com/steveloughran/winutils下载了Winutils。

我将它们安装在C：\ winutils目录（顶层）中，该目录包含一个名为/ bin /的子目录，该子目录包含winutil.exe和相关文件。此顶级路径已作为Windows环境变量添加到名为HADOOP_HOME的系统变量中。我也有一个变量SPARK_HOME到C：\ Users \ name \ spark-3.0.1-bin-hadoop3.2 \

我遇到了这个错误，直到我找到了上面的SO帖子和Moises Trelles的答案，以及本页https://answers.microsoft.com/en-us/insider/forum/insider_wintp-insider_repair/how-do-i-fix-this-error-msvcp100dll-is-missing/c167d686-044e-44ab-8e8f-968fac9525c5?auth=1

我有一个64位系统，因此按照Answers.microsoft.com的建议安装了xv和x64版本的msvcp100.dll。我没有重新启动，但是我确实关闭了Intellij并重新加载，并在重新运行时生成了正确的输出（镶木地板文件）。祝好运！非常感谢有帮助的人的stackoverflow / google / internet / community。

为什么启动流式查询会导致“ExitCodeException exitCode = -1073741515”？

4 个答案: