Question

我的数据框看起来像-

id            date             
1    2018-08-23 11:48:22       
2    2019-05-03 06:22:01       
3    2019-05-13 10:12:15       
4    2019-01-22 16:13:29       
5    2018-11-27 11:17:19

我的预期输出是-

id            date             date1
1    2018-08-23 11:48:22       2018-08
2    2019-05-03 06:22:01       2019-05
3    2019-05-13 10:12:15       2019-05
4    2019-01-22 16:13:29       2019-01
5    2018-11-27 11:17:19       2018-11

如何在pyspark中做到这一点？

Answer 1

我认为您正在尝试删除日期和时间的详细信息，可以使用date_format函数

>>> df.show()
+---+-------------------+
| id|               date|
+---+-------------------+
|  1|2018-08-23 11:48:22|
|  2|2019-05-03 06:22:01|
|  3|2019-05-13 10:12:15|
|  4|2019-01-22 16:13:29|
|  5|2018-11-27 11:17:19|
+---+-------------------+
>>> import pyspark.sql.functions as F
>>>
>>> df.withColumn('date1',F.date_format(F.to_date('date','yyyy-MM-dd HH:mm:ss'),'yyyy-MM')).show()
+---+-------------------+-------+
| id|               date|  date1|
+---+-------------------+-------+
|  1|2018-08-23 11:48:22|2018-08|
|  2|2019-05-03 06:22:01|2019-05|
|  3|2019-05-13 10:12:15|2019-05|
|  4|2019-01-22 16:13:29|2019-01|
|  5|2018-11-27 11:17:19|2018-11|
+---+-------------------+-------+

Answer 2

通过to_date然后通过substr函数...示例：

import pyspark.sql.functions as F
import pyspark.sql.types as T

rawData = [(1, "2018-08-23 11:48:22"),
           (2, "2019-05-03 06:22:01"),
           (3, "2019-05-13 10:12:15")]

df = spark.createDataFrame(rawData).toDF("id","my_date")

df.withColumn("new_my_date",\
            F.substring(F.to_date(F.col("my_date")), 1,7))\
.show()

+---+-------------------+-----------+
| id|            my_date|new_my_date|
+---+-------------------+-----------+
|  1|2018-08-23 11:48:22|    2018-08|
|  2|2019-05-03 06:22:01|    2019-05|
|  3|2019-05-13 10:12:15|    2019-05|
+---+-------------------+-----------+

Answer 3

import pyspark.sql.functions as F

split_col = F.split(df['date'], '-')
df = df.withColumn('year', split_col.getItem(0)).withColumn('month', split_col.getItem(1))
df = df.select(F.concat(df['year'], F.lit('-'),df['month']).alias('year_month'))
df.show()
+----------+
|year_month|
+----------+
|   2018-08|
|   2019-05|
|   2019-05|
|   2019-01|
|   2018-11|
+----------+

pyspark中的日期格式

3 个答案: