当我使用Spark读取CSV文件并将其转换为数据集时，出现以下错误。我不知道原因。我的代码在下面提供。另外http://eforexcel.com/wp/wp-content/uploads/2017/07/10000-Sales-Records.zip也可以下载CSV文件。
我正在使用Scala：2.12.3，Spark：2.4.0。
错误消息：
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`itemType`' given input columns: [Order ID, Total Profit, Country, Total Revenue, Ship Date, Unit Cost, Sales Channel, Unit Price, Total Cost, Units Sold, Order Date, Order Priority, Region, Item Type];
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:110)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:107)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:278)
...
...
这是我的代码：
import spark.implicits._
case class Sales(region: String,
country: String,
itemType: String,
salesChannel: String,
orderPriority: String,
orderDate: String,
orderId: Long,
shipDate: String,
unitsSold: Integer,
unitsPrice: Double,
unitCost: Double,
totalRevenue: Double,
totalCost: Double,
totalProfit: Double
)
val ds = spark.read
.option("header", "true")
.option("inferSchema", "true")
.csv("src/main/resources/datasets/10000 Sales Records.csv")
.as[Sales]
答案 0 :(得分：2)
csv标头列与case类之间不匹配。 从csv标头中，您需要对数据进行整理以使其与case类匹配。那就是您需要删除空格，将第二个单词大写。以下变通办法将为您服务。
请注意，我在您的案例类中将unitPrice：Double更改为unitPrice。
val ds = spark.read
.option("header", "true")
.option("inferSchema", "true")
.csv("in/10000_Sales_Records.csv")
ds.printSchema()
val sch1 = ds.columns.map( x=> x match { case a if a.contains(" ") => { val q=a.split(" ");q(0)+q(1).capitalize } case a => a.toLowerCase } )
val ds2 = ds.toDF(sch1:_*)
ds2.printSchema()
val ds3 = ds2.as[Sales]
ds3.show(false)
结果：
root
|-- Region: string (nullable = true)
|-- Country: string (nullable = true)
|-- Item Type: string (nullable = true)
|-- Sales Channel: string (nullable = true)
|-- Order Priority: string (nullable = true)
|-- Order Date: string (nullable = true)
|-- Order ID: integer (nullable = true)
|-- Ship Date: string (nullable = true)
|-- Units Sold: integer (nullable = true)
|-- Unit Price: double (nullable = true)
|-- Unit Cost: double (nullable = true)
|-- Total Revenue: double (nullable = true)
|-- Total Cost: double (nullable = true)
|-- Total Profit: double (nullable = true)
root
|-- region: string (nullable = true)
|-- country: string (nullable = true)
|-- ItemType: string (nullable = true)
|-- SalesChannel: string (nullable = true)
|-- OrderPriority: string (nullable = true)
|-- OrderDate: string (nullable = true)
|-- OrderID: integer (nullable = true)
|-- ShipDate: string (nullable = true)
|-- UnitsSold: integer (nullable = true)
|-- UnitPrice: double (nullable = true)
|-- UnitCost: double (nullable = true)
|-- TotalRevenue: double (nullable = true)
|-- TotalCost: double (nullable = true)
|-- TotalProfit: double (nullable = true)
+---------------------------------+--------------------------------+---------------+------------+-------------+----------+---------+----------+---------+---------+--------+------------+----------+-----------+
|region |country |ItemType |SalesChannel|OrderPriority|OrderDate |OrderID |ShipDate |UnitsSold|UnitPrice|UnitCost|TotalRevenue|TotalCost |TotalProfit|
+---------------------------------+--------------------------------+---------------+------------+-------------+----------+---------+----------+---------+---------+--------+------------+----------+-----------+
|Sub-Saharan Africa |Chad |Office Supplies|Online |L |1/27/2011 |292494523|2/12/2011 |4484 |651.21 |524.96 |2920025.64 |2353920.64|566105.0 |
|Europe |Latvia |Beverages |Online |C |12/28/2015|361825549|1/23/2016 |1075 |47.45 |31.79 |51008.75 |34174.25 |16834.5 |
|Middle East and North Africa |Pakistan |Vegetables |Offline |C |1/13/2011 |141515767|2/1/2011 |6515 |154.06 |90.93 |1003700.9 |592408.95 |411291.95 |
|Sub-Saharan Africa |Democratic Republic of the Congo|Household |Online |C |9/11/2012 |500364005|10/6/2012 |7683 |668.27 |502.54 |5134318.41 |3861014.82|1273303.59 |
|Europe |Czech Republic |Beverages |Online |C |10/27/2015|127481591|12/5/2015 |3491 |47.45 |31.79 |165647.95 |110978.89 |54669.06 |
|Sub-Saharan Africa |South Africa |Beverages |Offline |H |7/10/2012 |482292354|8/21/2012 |9880 |47.45 |31.79 |468806.0 |314085.2 |154720.8 |
|Asia |Laos |Vegetables |Online |L |2/20/2011 |844532620|3/20/2011 |4825 |154.06 |90.93 |743339.5 |438737.25 |304602.25 |
|Asia |China |Baby Food |Online |C |4/10/2017 |564251220|5/12/2017 |3330 |255.28 |159.42 |850082.4 |530868.6 |319213.8 |
|Sub-Saharan Africa |Eritrea |Meat |Online |L |11/21/2014|411809480|1/10/2015 |2431 |421.89 |364.69 |1025614.59 |886561.39 |139053.2 |
|Central America and the Caribbean|Haiti |Office Supplies|Online |C |7/4/2015 |327881228|7/20/2015 |6197 |651.21 |524.96 |4035548.37 |3253177.12|782371.25 |
|Sub-Saharan Africa |Zambia |Cereal |Offline |M |7/26/2016 |773452794|8/24/2016 |724 |205.7 |117.11 |148926.8 |84787.64 |64139.16 |
|Europe |Bosnia and Herzegovina |Baby Food |Offline |M |10/20/2012|479823005|11/15/2012|9145 |255.28 |159.42 |2334535.6 |1457895.9 |876639.7 |
|Europe |Germany |Office Supplies|Online |C |2/22/2015 |498603188|2/27/2015 |6618 |651.21 |524.96 |4309707.78 |3474185.28|835522.5 |
|Asia |India |Household |Online |C |8/27/2016 |151717174|9/2/2016 |5338 |668.27 |502.54 |3567225.26 |2682558.52|884666.74 |
|Middle East and North Africa |Algeria |Clothes |Offline |C |6/21/2011 |181401288|7/21/2011 |9527 |109.28 |35.84 |1041110.56 |341447.68 |699662.88 |
|Australia and Oceania |Palau |Snacks |Offline |L |9/19/2013 |500204360|10/4/2013 |441 |152.58 |97.44 |67287.78 |42971.04 |24316.74 |
|Central America and the Caribbean|Cuba |Beverages |Online |H |11/15/2015|640987718|11/30/2015|1365 |47.45 |31.79 |64769.25 |43393.35 |21375.9 |
|Europe |Vatican City |Beverages |Online |L |4/6/2015 |206925189|4/27/2015 |2617 |47.45 |31.79 |124176.65 |83194.43 |40982.22 |
|Middle East and North Africa |Lebanon |Personal Care |Offline |H |4/12/2010 |221503102|5/19/2010 |6545 |81.73 |56.67 |534922.85 |370905.15 |164017.7 |
|Europe |Lithuania |Snacks |Offline |H |9/26/2011 |878520286|10/2/2011 |2530 |152.58 |97.44 |386027.4 |246523.2 |139504.2 |
+---------------------------------+--------------------------------+---------------+------------+-------------+----------+---------+----------+---------+---------+--------+------------+----------+-----------+
only showing top 20 rows
答案 1 :(得分：0)
可能在输入文件中包含标题-地区，国家/地区，商品类型，销售渠道，订单优先级，订单日期，订单ID，发货日期，售出单位，单价，单价，总计收入，总成本，总利润
输入文件中的编辑标题或案例类
名称：项目类型（带空格），如无空格的案例类