Question

我从DataFrame发出RDD时遇到错误。

from pyspark.ml.fpm import FPGrowth

sogou = sc.textFile("SogouQ.sample.utf8", use_unicode = False)

def parse(line):
    value = [ x for x in line.split(",") if x]
    return list(set(value))

rdd = sogou.map(parse)
df = sogou.toDF('items')

我收到以下错误：

pyspark.sql.utils.ParseException：u＆＃34; \ nmismatched input＆＃39;＆＃39;期待{＆＃39; SELECT＆＃39;，＆＃39; FROM＆＃39;，＆＃39; ADD＆＃39;，＆＃39; AS＆＃39;，＆＃39; ALL＆＃39;，＆＃39 ; DISTINCT＆＃39;，＆＃39; WHERE＆＃39;，＆＃39; GROUP＆＃39;，＆＃39; BY＆＃39;，＆＃39; GROUPING＆＃39;，＆＃39; SETS＆＃39; ，＆＃39; CUBE＆＃39;，＆＃39; ROLLUP＆＃39;，＆＃39; ORDER＆＃39;，＆＃39; HAVING＆＃39;，＆＃39; LIMIT＆＃39;，＆＃39; AT＆＃39;，＆＃39; OR＆＃39;，＆＃39; AND＆＃39;，＆＃39; IN＆＃39;，NOT，＆＃39; NO＆＃39;，＆＃39; EXISTS＆＃39 ，＆＃39; BETWEEN＆＃39;，＆＃39; LIKE＆＃39;，RLIKE，＆＃39;是＆＃39;，＆＃39; NULL＆＃39;，＆＃39; TRUE＆＃39;，＆＃39; FALSE＆＃39;，＆＃39; NULLS＆＃39;＆＃39; ASC＆＃39;，＆＃39; DESC＆＃39;＆＃39; FOR＆＃39;，＆＃39; INTERVAL＆＃ 39;，＆＃39; CASE＆＃39;，＆＃39; WHEN＆＃39;，＆＃39; THEN＆＃39;，＆＃39; ELSE＆＃39;，＆＃39; END＆＃39;，＆＃ 39;加入＆＃39;，＆＃39; CROSS＆＃39;，＆＃39; OUTER＆＃39;，＆＃39; INNER＆＃39;，＆＃39; LEFT＆＃39;＆＃39; SEMI＆＃39 ;，＆＃39; RIGHT＆＃39;＆＃39; FULL＆＃39;，＆＃39; NATURAL＆＃39;＆＃39; ON＆＃39;，＆＃39; LATERAL＆＃39;，＆＃39; ; WINDOW＆＃39;，＆＃39; OVER＆＃39;，＆＃39; PARTITION＆＃39;＆＃39; RANGE＆＃39;，＆＃39; ROWS＆＃39;＆＃39; UNBOUNDED＆＃39; ，＆＃39; PRECEDING＆＃39;，＆＃39;关注＆＃39;，＆＃39;当前＆＃39;，＆＃39; F IRST＆＃39;，＆＃39; AFTER＆＃39;，＆＃39; LAST＆＃39;＆＃39; ROW＆＃39;，＆＃39; WITH＆＃39;，＆＃39; VALUES＆＃39;，＆＃39;创建＆＃39;，＆＃39;表＆＃39;目录＆＃39;＆＃39; VIEW＆＃39;，＆＃39; REPLACE＆＃39;，＆＃39; INSERT＆＃39;，＆＃39;删除＆＃39;，＆＃39; INTO＆＃39;，＆＃39; DESCRIBE＆＃39;，＆＃39; EXPLAIN＆＃39;，＆＃39; FORMAT＆＃39;，＆＃39; LOGICAL＆＃39;，＆＃39; CODEGEN＆＃39;，＆＃39; COST＆＃39;＆＃39; CAST＆＃39;，＆＃39; SHOW＆＃39;，＆＃39; TABLES＆＃ 39;，＆＃39; COLUMNS＆＃39;，＆＃39; COLUMN＆＃39;，＆＃39; USE＆＃39;，＆＃39; PARTITIONS＆＃39;，＆＃39; FUNCTIONS＆＃39;，＆＃ 39; DROP＆＃39;，＆＃39; UNION＆＃39;，＆＃39; EXCEPT＆＃39;，＆＃39; MINUS＆＃39;，＆＃39; INTERSECT＆＃39;，＆＃39; TO＆＃39 ;，＆＃39; TABLESAMPLE＆＃39;，＆＃39; STRATIFY＆＃39;，＆＃39; ALTER＆＃39;，＆＃39; RENAME＆＃39;，＆＃39; ARRAY＆＃39;，＆＃39; ; MAP＆＃39;，＆＃39; STRUCT＆＃39;，＆＃39; COMMENT＆＃39;＆＃39; SET＆＃39;，＆＃39; RESET＆＃39;，＆＃39; DATA＆＃39; ，＆＃39; START＆＃39;＆＃39; TRANSACTION＆＃39;，＆＃39; COMMIT＆＃39;，＆＃39; ROLLBACK＆＃39;，＆＃39; MACRO＆＃39;，＆＃39; IGNORE＆＃39;，＆＃39; BOTH＆＃39;，＆＃39; LEADING＆＃39;，＆＃39; TRAILING＆＃39;，＆＃39; IF＆＃39;，＆＃39; POSITION＆＃39;，＆＃39; DIV ＆＃39;，＆＃39; PERCENT＆＃39;＆＃39; BUCKET＆＃39;＆＃39; OUT＆＃39;，＆＃39; OF＆＃39;，＆＃39; SORT＆＃39;，＆＃39; CLUSTER＆＃39;，＆＃39; DISTRIBUTE＆＃39;，＆＃39; OVERWRITE＆＃39;，＆＃39; TRANSFORM＆＃39;，＆＃39; REDUCE＆＃39;，＆＃39; SERDE＆＃39;，＆＃39; SERDEPROPERTIES＆＃39; RECORDREADER＆＃39;，＆＃39; RECORDWRITER＆＃39;，＆＃39; DELIMITED＆＃39;，＆＃39; FIELDS＆＃39;，＆＃39; TERMINATED＆＃39;，＆＃39; COLLECTION＆＃39;，＆＃39; ITEMS＆＃39;＆＃39; KEYS＆＃39;，＆＃39; ESCAPED＆＃39;，＆＃39; LINES＆＃ 39;，＆＃39;分离＆＃39;，＆＃39;功能＆＃39;，＆＃39;扩展＆＃39;，＆＃39;刷新＆＃39;，＆＃39;清除＆＃39;，＆＃ 39; CACHE＆＃39;，＆＃39; UNCACHE＆＃39;，＆＃39; LAZY＆＃39;，＆＃39; FORMATTED＆＃39;，＆＃39; GLOBAL＆＃39;，TEMPORARY，＆＃39; OPTIONS＆＃39;，＆＃39; UNSET＆＃39;，＆＃39; TBLPROPERTIES＆＃39;，＆＃39; DBPROPERTIES＆＃39;，＆＃39; BUCKETS＆＃39;＆＃39; SKEWED＆＃39;，＆＃39;已存储＆＃39;，＆＃39;目录＆＃39;，＆＃39;地理位置＆＃39;，＆＃39;交换＆＃39;，＆＃39;存档＆＃39;＆＃39; UNARCHIVE＆＃ 39;，＆＃39; FILEFORMAT＆＃39;，＆＃39; TOUCH＆＃39;，＆＃39; COMPACT＆＃39;，＆＃39; CONCATENATE＆＃39;，＆＃39; CHANGE＆＃39;，＆＃ 39; CASCADE＆＃39;，＆＃39; R. ESTRICT＆＃39;，＆＃39; CLUSTERED＆＃39;，＆＃39; SORTED＆＃39;，＆＃39; PURGE＆＃39;，＆＃39; INPUTFORMAT＆＃39;＆＃39; OUTPUTFORMAT＆＃39;， DATABASE，DATABASES，＆＃39; DFS＆＃39;＆＃39; TRUNCATE＆＃39;，＆＃39; ANALYZE＆＃39;，＆＃39; COMPUTE＆＃39;，＆＃39; LIST＆＃39;，＆＃39;ç计＆＃39;，＆＃39;分区＆＃39;，＆＃39;外部＆＃39;，＆＃39;定义＆＃39;，＆＃39; REVOKE＆＃39;，＆＃39; GRANT＆＃ 39;，＆＃39; LOCK＆＃39;，＆＃39; UNLOCK＆＃39;，＆＃39; MSCK＆＃39;，＆＃39; REPAIR＆＃39;，＆＃39; RECOVER＆＃39;，＆＃ 39;出口＆＃39;，＆＃39;进口＆＃39;，＆＃39; LOAD＆＃39;，＆＃39; ROLE＆＃39;，＆＃39; ROLES＆＃39;，＆＃39; COMPACTIONS＆＃39 ;，＆＃39; PRINCIPALS＆＃39;，＆＃39; TRANSACTIONS＆＃39;＆＃39; INDEX＆＃39;，＆＃39; INDEXES＆＃39;＆＃39; LOCKS＆＃39;，＆＃39; ;选项＆＃39;，＆＃39; ANTI＆＃39;，＆＃39; LOCAL＆＃39;，＆＃39; INPATH＆＃39;，IDENTIFIER，BACKQUOTED_IDENTIFIER}（第1行，第5行）\ n \ n == SQL == \ nitems \ n ----- ^^^ \ n＆＃34;

该文字包含Chinese。有关系吗？文字是这样的：

360,安全卫士，
123，123，范冰冰，

当我使用pyspark.mllib.fpgrowth时，rdd工作正常。如何将其转换为数据帧？

Answer 1

这里有两个不同的问题：

toDF致电。 RDD.toDF有以下签名：
```
Signature: rdd.toDF(schema=None, sampleRatio=None)
```
schema应该

param schema：pyspark.sql.types.StructType或列名列表

所以在你的情况下它应该是：
```
sogou.toDF(["items"])
```
parse方法：
createDataFrame调用的
df方法需要RDD[tuple]或等效项，可以映射到structs，除非提供了模式。如果您只想使用名称，则应返回tuple
```
def parse(line):
    value = [ x for x in line.split(",") if x]
    return list(set(value)),  
```

组合：

>>> def parse(line):
...     value = [ x for x in line.split(",") if x]
...     return list(set(value)),  
... 
... 
>>> rdd = sc.parallelize(["360,安全卫士,", "123，123，范冰冰，"])
>>> rdd.map(parse).toDF(["items"]).show()
+--------------+
|         items|
+--------------+
|   [安全卫士, 360]|
|[123，123，范冰冰，]|
+--------------+

替代（保持当前的解析实现）将是

>>> from pyspark.sql.types import ArrayType, StringType
>>> def parse(line):
...     value = [ x for x in line.split(",") if x]
...     return list(set(value))
    >>> rdd.map(parse).toDF(ArrayType(StringType())).toDF("items").show()
+--------------+     
|         items|
+--------------+
|   [安全卫士, 360]|
|[123，123，范冰冰，]|
+--------------+

在Pyspark的FPGrowth

1 个答案: