Question

我正在Azure HDInsight（Hadoop）上运行一个简单的Map / Reduce作业。映射器失败并出现以下错误（来自JobTracker日志）...

2014-03-27 16:42:50,330 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201403262112_0004_m_000000_0: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 255
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)
at org.apache.hadoop.mapred.Child.main(Child.java:260)

我使用Hadoop.Connect方法连接到群集......

// Connect to the HDInsight cluster
var hadoop = Hadoop.Connect(hadoopClusterUri,
                            hadoopUserName,
                            localHadoopUser,
                            hadoopPassword,
                            asvAccount,
                            asvKey,
                            asvContainer,
                            createContinerIfNotExist);

由于作业已启动并已记录，因此我认为连接成功。作为一个简单的测试，我发出的数据很少......

context.EmitKeyValue("This is a key", "This is a value");

完全难倒，比尔

更新这个MapReduce产生了两个作业。第一个成功完成，第二个失败，出现以下异常...

Unhandled Exception: Microsoft.Hadoop.MapReduce.StreamingException: The user type could not be loaded. DLL=MapReduce.exe, Type=PrescienceRx.MapReduce.BeneficiaryMapper ---> System.BadImageFormatException: Could not load file or assembly &apos;file:///c:\apps\temp\hdfs\mapred\local\taskTracker\admin\jobcache\job_201403262112_0006\attempt_201403262112_0006_m_000001_0\work\MapReduce.exe&apos; or one of its dependencies. An attempt was made to load a program with an incorrect format.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at Microsoft.Hadoop.MapReduce.MapperMain.Main()
--- End of inner exception stack trace ---

at Microsoft.Hadoop.MapReduce.MapperMain.Main（）在Microsoft.HdInsight.MapDriver.Program.Main（String [] args

Answer 1

看起来System.BadImageFormatException: Could not load file or assembly可能是因为加载程序的.Net版本与程序集之间不匹配。尝试将其设置回4.0，然后在NuGet包管理器窗口中执行update-package -reinstall Microsoft.Hadoop.MapReduce以确保您拥有所有正确的版本。

Answer 2

我在解决方案的配置管理器中将目标CPU设置为x64，这解决了问题。

pipemapred waitoutputthreads子进程失败，代码为255

2 个答案: