Question

嗨，
我在将非StringType用作加载csv文件以创建数据帧时所使用的架构的一部分时遇到了麻烦。

我期望给定的架构可用于在加载时将每个记录的每个字段即时转换为相应的数据类型。
相反，我得到的只是空值。

这是重现我的问题的简化方法。在此示例中，有一个小的csv文件，该文件包含四列，我想分别将其视为str，date，int和bool：

python
Python 3.6.5 (default, Jun 17 2018, 12:13:06) 
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyspark
>>> from pyspark import SparkContext
>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import *
>>> 
>>> data_flnm = 'four_cols.csv'

>>> lines = [ln.rstrip() for  ln in open(data_flnm).readlines()[:3]]
>>> lines
['zzzc7c09:66d7:47d6:9415:87e5010fe282|2019-04-08|0|f', 'zzz304fa:6fc0:4337:91d0:05ef4657a6db|2019-07-08|1|f', 'yy251cf0:aa11:44e9:88f4:f6f9c1899cee|2019-05-13|0|t']


>>> parts = [ln.split("|") for ln in lines]
>>> parts
[['zzzc7c09:66d7:47d6:9415:87e5010fe282', '2019-04-08', '0', 'f'], ['zzz304fa:6fc0:4337:91d0:05ef4657a6db', '2019-07-08', '1', 'f'], ['yy251cf0:aa11:44e9:88f4:f6f9c1899cee', '2019-05-13', '0', 't']]

>>> cols1 = [StructField('u_id', StringType(), True), StructField('week', StringType(), True), StructField('flag_0_1', StringType(), True), StructField('flag_t_f', StringType(), True)]
>>> cols2 = [StructField('u_id', StringType(), True), StructField('week', DateType(), True), StructField('flag_0_1', IntegerType(), True), StructField('flag_t_f', BooleanType(), True)]
>>> sch1 = StructType(cols1)
>>> sch2 = StructType(cols2)
>>> sch1
StructType(List(StructField(u_id,StringType,true),StructField(week,StringType,true),StructField(flag_0_1,StringType,true),StructField(flag_t_f,StringType,true)))
>>> sch2
StructType(List(StructField(u_id,StringType,true),StructField(week,DateType,true),StructField(flag_0_1,IntegerType,true),StructField(flag_t_f,BooleanType,true)))

>>> spark_sess = SparkSession.builder.appName("xyz").getOrCreate()
19/09/10 19:32:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

>>> df1 = spark_sess.read.format('csv').option("nullValue", "null").load([data_flnm], sep='|', schema = sch1)
>>> df2 = spark_sess.read.format('csv').option("nullValue", "null").load([data_flnm], sep='|', schema = sch2)

>>> df1.show(5)
+--------------------+----------+--------+--------+
|                u_id|      week|flag_0_1|flag_t_f|
+--------------------+----------+--------+--------+
|zzzc7c09:66d7:47d...|2019-04-08|       0|       f|
|zzz304fa:6fc0:433...|2019-07-08|       1|       f|
|yy251cf0:aa11:44e...|2019-05-13|       0|       t|
|yy1d2f8e:d8f0:4db...|2019-07-08|       1|       f|
|zzz5ccad:2cf6:44e...|2019-05-20|       1|       f|
+--------------------+----------+--------+--------+
only showing top 5 rows

>>> df2.show(5)
+----+----+--------+--------+
|u_id|week|flag_0_1|flag_t_f|
+----+----+--------+--------+
|null|null|    null|    null|
|null|null|    null|    null|
|null|null|    null|    null|
|null|null|    null|    null|
|null|null|    null|    null|
+----+----+--------+--------+
only showing top 5 rows

>>>

我尝试了几种不同版本的.read（...）.... load（...）代码。没有一个产生预期的结果。请指教。谢谢！

PS：无法添加标签“ structfield”和“ structtype”：信誉不足（__。

Answer 1

解析时，您需要将flag_t_f列读取为字符串。以下模式将起作用：

StructType(List(StructField(u_id,StringType,true),StructField(week,DateType,true),StructField(flag_0_1,IntegerType,true),StructField(flag_t_f,StringType,true)))

之后，您可以根据需要向数据框添加一个布尔列：

import pyspark.sql.functions as f
df = df.withColumn("flag_t_f", 
      f.when(f.col("flag_t_f") == 'f', 'False')
      .when(f.col("flag_t_f") == 't', 'True')          
     )

如果您有多个布尔列，其值为'f'和't'，则可以通过遍历所有列来转换所有这些列

cols = df.columns
for col in cols:
    df = df.withColumn(col, 
        f.when(f.col(col) == 'f', 'False')
         .when(f.col(col) == 't','True')
         .otherwise(f.col(col))
    )

带有模式的pyspark.sql SparkSession load（）：模式中的Non-StringType字段使所有值都为null

1 个答案: