我正在尝试在pyspark中加入2个数据帧。我的问题是我想要我的"内部加入"无论NULL如何都给它一个通行证。我可以看到,在scala中,我有一个< => 的替代品。但是,< => 在pyspark中无效。
userLeft = sc.parallelize([
Row(id=u'1',
first_name=u'Steve',
last_name=u'Kent',
email=u's.kent@email.com'),
Row(id=u'2',
first_name=u'Margaret',
last_name=u'Peace',
email=u'marge.peace@email.com'),
Row(id=u'3',
first_name=None,
last_name=u'hh',
email=u'marge.hh@email.com')]).toDF()
userRight = sc.parallelize([
Row(id=u'2',
first_name=u'Margaret',
last_name=u'Peace',
email=u'marge.peace@email.com'),
Row(id=u'3',
first_name=None,
last_name=u'hh',
email=u'marge.hh@email.com')]).toDF()
目前的工作版本:
userLeft.join(userRight, (userLeft.last_name==userRight.last_name) & (userLeft.first_name==userRight.first_name)).show()
当前结果:
+--------------------+----------+---+---------+--------------------+----------+---+---------+
| email|first_name| id|last_name| email|first_name| id|last_name|
+--------------------+----------+---+---------+--------------------+----------+---+---------+
|marge.peace@email...| Margaret| 2| Peace|marge.peace@email...| Margaret| 2| Peace|
+--------------------+----------+---+---------+--------------------+----------+---+---------+
预期结果:
+--------------------+----------+---+---------+--------------------+----------+---+---------+
| email|first_name| id|last_name| email|first_name| id|last_name|
+--------------------+----------+---+---------+--------------------+----------+---+---------+
| marge.hh@email.com| null| 3| hh| marge.hh@email.com| null| 3| hh|
|marge.peace@email...| Margaret| 2| Peace|marge.peace@email...| Margaret| 2| Peace|
+--------------------+----------+---+---------+--------------------+----------+---+---------+
答案 0 :(得分:2)
对于 PYSPARK <2.3.0,您仍然可以使用这样的表达式列来构建 <=> 运算符:
import pyspark.sql.functions as F
df1.alias("df1").join(df2.alias("df2"), on = F.expr('df1.column <=> df2.column'))
对于 PYSPARK> = 2.3.0 ,您可以使用 Column.eqNullSafe 或 IS NOT DISTINCT FROM 作为回答here
答案 1 :(得分:0)
使用其他值而不是null
:
userLeft = userLeft.na.fill("unknown")
userRight = userRight.na.fill("unknown")
userLeft.join(userRight, ["last_name", "first_name"])
+---------+----------+--------------------+---+--------------------+---+
|last_name|first_name| email| id| email| id|
+---------+----------+--------------------+---+--------------------+---+
| Peace| Margaret|marge.peace@email...| 2|marge.peace@email...| 2|
| hh| unknown| marge.hh@email.com| 3| marge.hh@email.com| 3|
+---------+----------+--------------------+---+--------------------+---+