Python-Spark RDD似乎仅保存第一个值

Question

我有一个要通过Word2Vec运行的python代码数据集。

sc = SparkContext()
sqlCtx = SQLContext(sc)
code_lines = sqlCtx.read.option("multiLine", True).option("mode", "PERMISSIVE").json("jsons\hi4.json")
code_lines = code_lines.repartition(300)

def split_code(input):
    strs = ' '.join(input)
    patt = re.compile(r"[\w]+", re.UNICODE) 
    return patt.findall(strs)

words = code_lines\
    .rdd.map(
        lambda line: line[11].split(" ")
    )\
    .map(lambda line: [f.lower() for f in line])\
    .map(lambda line: split_code(line))\
    .filter(lambda line: line != [])

word2vec = Word2Vec()
word2vec.setMinCount(1)
word2vec.setVectorSize(50)

这是json文件的第一行

{"repo_name": "hi4", "commit_id": "b'58db731ed75dd25d132a07f4d4a3524eb41d200f", "author": "None", "author_mail": "None", "author_time": "None", "author_timezone": "None", "committer": "None", "committer_mail": "None", "committer_time": "None", "committer_timezone": "None", "comment": "None", "filename": "setup.py", "line_num": "1", "line": ""}

当我打印count（）时，得到的RDD大小仅为1。该模型为空字符串（第一个条目的值）生成一个向量。