如何处理包含newLine字符的表或csv文件列以及如何在具有hdfs文件系统的pyspark dataFrame中进行操作

Question

如何处理带有hdfs文件系统的pyspark df中包含换行符的table / csv文件的列。

我实际上需要对具有换行符的列数据进行操作，而换行符不能通过下面提到的步骤实现。

面临的挑战是使用换行符char的元组数据正在创建新记录，并且无法对其进行解析和在df中对其进行操作

>>> df=spark.read.csv("hdfs://cluster-04d4-m/user/veerayyakumar_g/Cleansdata_Input_Test.csv", header = True, inferSchema = True).show()
+--------------------+----------+--------------------+---------------+-------------+---------+
|                item|item_group|     item_group_desc|item_group_qlty|product_group|   run_dt|
+--------------------+----------+--------------------+---------------+-------------+---------+
|            I1229422|        G1|"<?xml version=""...|           null|         null|     null|
|      <Instructions>|      null|                null|           null|         null|     null|
|    <Instruction ...|      null|                null|           null|         null|     null|
|        Instructi...|      null|                null|           null|         null|     null|
|     </Instructions>|      null|                null|           null|         null|     null|
| ,,P130872,4/22/2019|      null|                null|           null|         null|     null|

>>> df=spark.read.csv("hdfs://cluster-04d4-m/user/veerayyakumar_g/Cleansdata_Input_Test.csv", header = True, inferSchema = True).show()

tried manipluate using lambda function but it's not working 

>>> newdf["item_group_desc"]=df["item_group_desc"].apply(lambda x: x.replace("\n",""))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object has no attribute '__getitem__'


Tried with column funtion still not able to achive that my scenario where i wanted to bring the tuple data in single line in the column "item_group_desc"
>>> newdf=df["item_group_desc"].withColumn('item_group_desc',regexp_replace('item_group_desc','[\\n]',''))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object has no attribute '__getitem__'

在删除item_group_desc文件系统的newLine中的pyspark字符之后，需要在一行中hdfs数据。