Question

在Spark SQL的帮助下，我试图过滤掉属于特定组类别的所有业务项目。

数据从JSON文件加载：

businessJSON = os.path.join(targetDir, 'yelp_academic_dataset_business.json')
businessDF = sqlContext.read.json(businessJSON)

该文件的架构如下：

businessDF.printSchema()

root
  |-- business_id: string (nullable = true)
  |-- categories: array (nullable = true)
  |    |-- element: string (containsNull = true)
  ..
  |-- type: string (nullable = true)

我试图提取与餐馆业务相关的所有业务：

restaurants = businessDF[businessDF.categories.inSet("Restaurants")]

但它不起作用，因为据我所知，预期的列类型应该是一个字符串，但在我的情况下，这是数组。关于它告诉我一个例外：

Py4JJavaError: An error occurred while calling o1589.filter.
: org.apache.spark.sql.AnalysisException: invalid cast from string to array<string>;

你能否建议任何其他方式来获得我想要的东西？

Answer 1

UDF怎么样？

from pyspark.sql.functions import udf, col, lit
from pyspark.sql.types import BooleanType

contains = udf(lambda xs, val: val in xs, BooleanType())
df = sqlContext.createDataFrame([Row(categories=["foo", "bar"])])

df.select(contains(df.categories, lit("foo"))).show()
## +----------------------------------+
## |PythonUDF#<lambda>(categories,foo)|
## +----------------------------------+
## |                              true|
## +----------------------------------+

df.select(contains(df.categories, lit("foobar"))).show()
## +-------------------------------------+
## |PythonUDF#<lambda>(categories,foobar)|
## +-------------------------------------+
## |                                false|
## +-------------------------------------+

如何通过多值列过滤JSON数据

1 个答案: