当我使用MLUtils saveAsLibSVMFile时出现上述错误。尝试了各种各样的方法,但没有任何效果。
/*
conf.set("spark.io.compression.codec","org.apache.spark.io.LZFCompressionCodec")
*/
/*
conf.set("spark.executor.extraClassPath","/usr/hdp/current/hadoop-client/lib/snappy-java-*.jar")
conf.set("spark.driver.extraClassPath","/usr/hdp/current/hadoop-client/lib/snappy-java-*.jar")
conf.set("spark.executor.extraLibraryPath","/usr/hdp/2.3.4.0-3485/hadoop/lib/native")
conf.set("spark.driver.extraLibraryPath","/usr/hdp/2.3.4.0-3485/hadoop/lib/native")
*/
最后我只有两种方法可以解决它。这在下面的答案中给出。
答案 0 :(得分:0)
一种方法是使用不同的hadoop编解码器,如下所示
sc.hadoopConfiguration.set("mapreduce.output.fileoutputformat.compress", "true")
sc.hadoopConfiguration.set("mapreduce.output.fileoutputformat.compress.type", CompressionType.BLOCK.toString)
sc.hadoopConfiguration.set("mapreduce.output.fileoutputformat.compress.codec", "org.apache.hadoop.io.compress.BZip2Codec")
sc.hadoopConfiguration.set("mapreduce.map.output.compress", "true")
sc.hadoopConfiguration.set("mapreduce.map.output.compress.codec", "org.apache.hadoop.io.compress.BZip2Codec")
第二种方法是提及--driver-library-path /usr/hdp/<whatever is your current version>/hadoop/lib/native/
作为我的spark-submit作业的参数(在命令行中)