Question

如何从此数据框中删除，因为我无法将数据加载到基于ability_id的分区的hive表中。

由于'u'

，始终会收到Illigalargument错误

>>schema = StructType([ StructField("ability_id", StringType(), True),
StructField("bid", StringType(), True),
StructField("bidtime", StringType(), True),
StructField("bidder", StringType(), True),
StructField("bidderrate", StringType(), True),
StructField("openbid", StringType(), True),
StructField("price", StringType(), True)])`  

>>df = sqlContext.createDataFrame(auction_data,schema)
>>df.registerTempTable("auction")
>>first_line = sqlContext.sql("select * from auction where auctionid=8211480551").collect()
>>for i in first_line:
>>   print i

>>Row(ability_id=u'8211480551', bid=u'52.99', bidtime=u'1.201505', bidder=u'hanna1104', bidderrate=u'94', openbid=u'49.99', price=u'311.6')
>>Row(ability_id=u'8211480551', bid=u'50.99', bidtime=u'1.203843', bidder=u'wrufai1', bidderrate=u'90', openbid=u'49.99', price=u'311.6')`enter code here`


sqlContext.sql(""" INSERT INTO TABLE dev_core_t1.PINO_KLANT_3 partition (abillity_id) SELECT bid,bidtime,bidder,bidderrate,openbid,price from temp """)

Answer 1

此问题已解决，似乎任何小于或等于2.0.0.x.x.x的spark版本都无效。

它仅适用于Spark版本2.1.x.x.x或更高版本。

如何在加载到hive分区表

1 个答案: