Question

我正在pyspark中实现如下文本分类器

tokenizer = RegexTokenizer(inputCol="documents", outputCol="tokens", pattern='\\W+')

remover = StopWordsRemover(inputCol='tokens', outputCol='nostops')

vectorizer = CountVectorizer(inputCol='nostops', outputCol='features', vocabSize=1000)

labelIndexer = StringIndexer(inputCol="label", outputCol="indexedLabel", handleInvalid='skip')
labelIndexer_model = labelIndexer.fit(countModel_df)

convertor = IndexToString(inputCol='prediction', outputCol='predictedLabel', labels=labelIndexer_model.labels)

rfc = RandomForestClassifier(featuresCol='features', labelCol='indexedLabel', numTrees=30)

evaluator = BinaryClassificationEvaluator(labelCol='indexedLabel', rawPredictionCol='prediction')

pipe_rfc = Pipeline(stages=[tokenizer, remover, labelIndexer, vectorizer, rfc, convertor])

train_df, test_df = df.randomSplit((0.8, 0.2), seed=42)

model = pipe_rfc.fit(train_df)

prediction_rfc_df = rfc_model.transform(test_df)

代码正在运行，prediction_rfc_df进行了预期的预测。但是当我想检查元数据时-元数据字典如下所示为空

prediction_rfc_df.schema['features'].metadata

Output : {}

prediction_rfc_df.schema['label'].metadata

Output: {}

任何想法为何DataFrame中缺少元数据？

我正在从Cassandra表中读取数据，如下所示：

df = spark.read \
     .format("org.apache.spark.sql.cassandra") \
     .options(table='table_name', keyspace='key_space_name') \
     .load()

Pyspark spark-ml元数据字典为空

0 个答案: