Question

Mahout seq2sparse会生成一堆完整描述的here的sequenceFiles。我想使用具有以下格式的矢量化文档：＆lt; Text，VectorWritable＆gt; （docID，TF-IDF Vector）并从TF-IDF Vector中创建JavaRDD<Vector>。有人可以指导我完成吗？

Answer 1

此信息随时可用[{3}}。

预处理：对于PATH_TO_SEQUENCE_FILES中的一组序列文件格式化文档，mahout seq2sparse命令执行 TF-IDF转换（-wt tfidf选项）和L2长度归一化（-n 2选项）如下：
$ mahout seq2sparse 
  -i ${PATH_TO_SEQUENCE_FILES} 
  -o ${PATH_TO_TFIDF_VECTORS} 
  -nv 
  -n 2
  -wt tfidf

Training: The model is then trained using mahout spark-trainnb. The default is to train a Bayes model. The -c option is given to train
CBayes模型：
$ mahout spark-trainnb
  -i ${PATH_TO_TFIDF_VECTORS} 
  -o ${PATH_TO_MODEL}
  -ow 
  -c

Label Assignment/Testing: Classification and testing on a holdout set can then be performed via mahout spark-testnb. Again, the -c
选项表示该模型是CBayes：
$ mahout spark-testnb 
  -i ${PATH_TO_TFIDF_TEST_VECTORS}
  -m ${PATH_TO_MODEL} 
  -ow 
  -c

查看mahout in the documentation，我们发现它实际上正在使用org.apache.mahout.drivers.TrainNBDriver类。我们对TFIDF类型为<Text, VectorWritable>的{{1}}向量感兴趣：

  /** Read the training set from inputPath/part-x-00000 sequence file of form <Text,VectorWritable> */
  private def readTrainingSet: DrmLike[_]= {
    val inputPath = parser.opts("input").asInstanceOf[String]
    val trainingSet= drm.drmDfsRead(inputPath)
    trainingSet
  }

  override def process(): Unit = {
    start()

    val complementary = parser.opts("trainComplementary").asInstanceOf[Boolean]
    val outputPath = parser.opts("output").asInstanceOf[String]

    val trainingSet = readTrainingSet
    val (labelIndex, aggregatedObservations) = SparkNaiveBayes.extractLabelsAndAggregateObservations(trainingSet)
    val model = NaiveBayes.train(aggregatedObservations, labelIndex)

    model.dfsWrite(outputPath)

    stop()
  }

如果我们仔细观察，我们会看到drm.drmDfsRead(inputPath)调用正在转换输入。然后将这样转换（例如来自command script）

  /**
   * Load DRM from hdfs (as in Mahout DRM format)
   *
   * @param path
   * @param sc spark context (wanted to make that implicit, doesn't work in current version of
   *           scala with the type bounds, sorry)
   *
   * @return DRM[Any] where Any is automatically translated to value type
   */
  def drmDfsRead (path: String, parMin:Int = 0)(implicit sc: DistributedContext): CheckpointedDrm[_] = {

    val drmMetadata = hdfsUtils.readDrmHeader(path)
    val k2vFunc = drmMetadata.keyW2ValFunc

    // Load RDD and convert all Writables to value types right away (due to reuse of writables in
    // Hadoop we must do it right after read operation).
    val rdd = sc.sequenceFile(path, classOf[Writable], classOf[VectorWritable], minPartitions = parMin)

        // Immediately convert keys and value writables into value types.
        .map { case (wKey, wVec) => k2vFunc(wKey) -> wVec.get()}

    // Wrap into a DRM type with correct matrix row key class tag evident.
    drmWrap(rdd = rdd, cacheHint = CacheHint.NONE)(drmMetadata.keyClassTag.asInstanceOf[ClassTag[Any]])
  }

如何在Apache Spark中使用Mahout seq2sparse的矢量化文档输出

1 个答案: