Question

我正在读取JSON，并且有一个字典（dictd），其键告诉我应从JSON df中选择哪些列。

我正在尝试创建一个新的df，然后附加那些来自dictn的键在JSON中存在的列，但出现以下错误：非常感谢这一方面的任何帮助，因为我真的很新。

'运算符！Project [ip＃238 AS ip＃267]中缺少已解决的属性ip＃238。;; \ n！项目[ip＃238 AS ip＃267] \ n +-LogicalRDD假\ < / p>

from pyspark.sql.functions import lit
from pyspark.sql.types import StructType
import json
from pyspark.sql.functions import explode

jsn={"body":[{"ip":"177.284.10.91","sg_message_id":"YcbG1IBnQ1-626TaUVg2bQ.filter1049p1las1-18982-5C868E5A-20.0","hostname":"d2214390ce89","useragent":"Mozilla/5.0 (Linux; Android 7.1.2; E6810) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.105 Mobile Safari/537.36","method_name":"mass_gifting","email":"test@aol.com","timestamp":1554076768,"url":"https://engagement.test.com/b/genghisgrill","object_id":42813,"category":["42813-3-11-19-bottomless-is-back","713","mass_gifting"],"sg_event_id":"Krfn-yDfTG-CQ-o8zhTb0w","event":"click","klass":"3-11-19-bottomless-is-back","url_offset":{"index":3,"type":"html"},"rails_env":"production","user_id":78003906,"business_id":713}],"account":"testaccount"}

dictn={'ip':'string',
       'sg_message_id':'string',
       'hostname':'string',
       'method_name':'string',
       'email':'string',
       'timestamp':'bigint',
       'smtp-id':'string',
       'object_id':'bigint',
       'response':'string',
       'sg_event_id':'string',
       'tls':'string',
       'event':'string',
       'klass':'string',
       'user_id':'string',
       'rails_env':'string',
       'business_id':'bigint'}
schema = StructType([])
new_df = sqlContext.createDataFrame(sc.emptyRDD(), schema)
a=[json.dumps(jsn)]
jsonRDD = sc.parallelize(a)
df = spark.read.json(jsonRDD)
x=df.select("body")
df1=df.withColumn("foo",explode("body")).select("foo.*")
for k1,v1 in dictn.items():
  if k1 in df1.columns:
      new_df=new_df.withColumn(k1,df1[k1])
  else:
      new_df=new_df.withColumn(k1,lit(10))
new_df.show()

Answer 1

您收到该错误，是因为您试图通过引用另一个DataFrame中的列来添加新列，而Spark实际上并不支持该列。已经在这里提出并回答了这个问题：Add a column from another DataFrame

但是要在这里实现您想要的，您只需要使用select中的df1，这将为您提供新的DataFrame以及从dict中获取的列列表。

这对您应该很好：

select_expr = [col(c).alias(c) if c in df1.columns else lit(10).alias(c) for c, _ in dictn.items()]

new_df = df1.select(select_expr)

new_df.show()

通过从旧数据框pyspark中选择列将列追加到新创建的数据框

1 个答案: