PySpark / HIVE:附加到现有表格

时间:2017-11-29 16:09:25

标签: python apache-spark hive pyspark

真正基本的问题pyspark / hive问题:

如何附加到现有表格?我的尝试低于

from pyspark import SparkContext, SparkConf
from pyspark.sql import HiveContext
conf_init = SparkConf().setAppName('pyspark2')
sc = SparkContext(conf = conf_init)
hive_cxt = HiveContext(sc)

import pandas as pd
df = pd.DataFrame({'a':[0,0], 'b':[0,0]})
sdf = hive_cxt.createDataFrame(df)
sdf.write.mode('overwrite').saveAsTable('database.table') #this line works

df = pd.DataFrame({'a':[1,1,1], 'b':[2,2,2]})
sdf = hive_cxt.createDataFrame(df)
sdf.write.mode('append').saveAsTable('database.table') #this line does not work
#sdf.write.insertInto('database.table',overwrite = False) #this line does not work

谢谢! 萨姆

2 个答案:

答案 0 :(得分:3)

似乎使用选项('overwrite')导致问题;它会删除表,然后重新创建一个新表。如果我执行以下操作,一切正常:

from pyspark import SparkContext, SparkConf
from pyspark.sql import HiveContext

conf_init = SparkConf().setAppName('pyspark2')
sc = SparkContext(conf = conf_init)
print(sc.version)
hive_cxt = HiveContext(sc)
hive_cxt.sql('USE database')

query = """
        CREATE TABLE IF NOT EXISTS table (a int, b int)
        STORED AS parquet
        """
hive_cxt.sql(query)

import pandas as pd
df = pd.DataFrame({'a':[0,0], 'b':[0,0]})
sdf = hive_cxt.createDataFrame(df)
sdf.write.mode('append').format('hive').saveAsTable('table')

query = """
        SELECT *
        FROM   table
        """
df = hive_cxt.sql(query)
df = df.toPandas()
print(df) # successfully pull the data in table

df = pd.DataFrame({'a':[1,1,1], 'b':[2,2,2]})
sdf = hive_cxt.createDataFrame(df)
sdf.write.mode('append').format('hive').saveAsTable('table')

答案 1 :(得分:0)

我认为以前您忘了使用format选项,这会在尝试追加而不是像上面提到的那样覆盖时导致问题。