火花表现比scikit更糟糕 - 学习MultiClass

Question

目前我有文本数据，我正在尝试预测课程。在我的情况下，我有60个课程可供选择。当我使用scikit-learn在随机森林中部署模型时，我的f1得分为78％。

但是，我尝试在pyspark中设置模型，只获得30％。太低了！到底是怎么回事？也许我没有正确设置它。此外，使用pyspark，随机森林只能预测多达12个标签，在我的情况下，我有60个。

Sci-kit学习代码：

rf_model = Pipeline([
    ('featextract',FeatureExtractor()),
    ('union', FeatureUnion(
        transformer_list=[

            # pipeline for tfidf
            ('text', Pipeline([
                    ('selector',ItemSelector(key='TEXT')),
                    ('count_vec',TfidfVectorizer(max_features=5000)),
                    ('tfidf', TfidfTransformer())])),
            # pipeline for ata
            ('ata', Pipeline([
                    ('selector', ItemSelector(key="ATA_SYS_NO")),
                    ('atas',convert2dict()),
                    ('vect',DictVectorizer())]))                   
            ])),
    ('model', OneVsRestClassifier(RandomForestClassifier(n_estimators=200,n_jobs=5))),
    ])

pySpark代码

Tokenizer1 = Tokenizer(inputCol="TEXT",outputCol="words")
hashingTF = HashingTF(inputCol="words",outputCol="rawFeatures",numFeatures=4000)
idf = IDF(inputCol="rawFeatures",outputCol="tfidffeatures")
rf = RF(labelCol="componentIndex",featuresCol='tfidffeatures',numTrees=500)
pipeline = Pipeline(stages=[Tokenizer1,hashingTF,idf,labelIndexer,rf])
(trainingData,testData) = df.randomSplit([0.8,0.2])