Question

当我尝试读取CSV文件时遇到错误。我在这里使用spark 1.6.1是我的代码

 val reftable_df = sqlContext.read
    .format("com.databricks.spark.csv")
    .option("header", "true") 
    .option("inferSchema", "true") 
    .load("/home/hadoop1/Reference_Currencyctoff.csv")
    reftable_df.show()

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/csv/CSVFormat
        at com.databricks.spark.csv.package$.<init>(package.scala:27)
        at com.databricks.spark.csv.package$.<clinit>(package.scala)
        at com.databricks.spark.csv.CsvRelation.inferSchema(CsvRelation.scala:218)
        at com.databricks.spark.csv.CsvRelation.<init>(CsvRelation.scala:72)
        at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:157)
        at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:44)
        at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109)
        at scb.HBaseBroadcast$.main(HBaseBroadcast.scala:138)
        at scb.HBaseBroadcast.main(HBaseBroadcast.scala)

注意：我已经厌倦了以下CSV依赖

Spark Csv»1.3.0 Spark Csv»1.3.1 Spark Csv»1.4.0 Spark Csv»1.5.0

谢谢！

Answer 1

我面临同样的问题

jar -tvf commons-csv.jar | grep CSVFormat

解决了这个问题。

commons-csv.jar有这个类

您可以使用{{1}}

查看课程

Answer 2

在启动spark shell时尝试这个

bin/spark-shell --packages com.databricks:spark-csv_2.10:1.5.0

包含此套餐

spark csv读取火花1.6.1

2 个答案: