Question

我有一个包含多列的数据框，我需要选择其中的2列并将其转储到列表中，并且我尝试了以下操作：

df.show()
+------------------------------------+---------------+---------------+
|email_address                       |topic          |user_id        |
+------------------------------------+---------------+---------------+
|xyz@test.com                        |hello_world    |xyz123         |
+------------------------------------+---------------+---------------+
|lmn@test.com                        |hello_kitty    |lmn456         |
+------------------------------------+---------------+---------------+

我需要的结果是元组列表：

[(xyz@test.com, xyz123), (lmn@test.com, lmn456)]

我尝试的方式：

tuples = df.select(col('email_address'), col('topic')).rdd.flatMap(lambda x, y: list(x, y)).collect()

它会引发错误：

Py4JJavaError  Traceback (most recent call last)
<command-4050677552755250> in <module>()

--> 114 tuples = df.select(col('email_address'), col('topic')).rdd.flatMap(lambda x, y: list(x, y)).collect()
    115 
    116 

/databricks/spark/python/pyspark/rdd.py in collect(self)
    829         # Default path used in OSS Spark / for non-credential passthrough clusters:
    830         with SCCallSiteSync(self.context) as css:
--> 831             sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
    832         return list(_load_from_socket(sock_info, self._jrdd_deserializer))
    833 

/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1255         answer = self.gateway_client.send_command(command)
   1256         return_value = get_return_value(
-> 1257             answer, self.gateway_client, self.target_id, self.name)

如何解决？

Answer 1

您应该为此使用map：

tuples = df.select(col('email_address'), col('topic')) \
           .rdd \
           .map(lambda x: (x[0], x[1])) \
           .collect()

print(tuples)

# output
[('xyz@test.com', 'hello_world'), ('lmn@test.com', 'hello_kitty')]

另一种方法是收集DataFrame的行，然后循环获取值：

rows = df.select(col('email_address'), col('topic')).collect()

tuples = [(r.email_address, r.topic) for r in rows]
print(tuples)

# output
[('xyz@test.com', 'hello_world'), ('lmn@test.com', 'hello_kitty')]

如何从数据框中选择多个列并转储到pyspark中的列表

1 个答案: