Question

我们有一个条件，我们想要在其中生成模型得分，其中随机值在0-1之间。

为此，我们希望有一个自定义转换器，它将生成不带任何输入字段的随机数。

那么我们可以生成一个不包含mleap中输入字段的转换器吗？

通常，我们会按如下方式创建：

import ml.combust.mleap.core.Model
import ml.combust.mleap.core.types._

case class RandomNumberModel() extends Model {
  private val rnd = scala.util.Random

  def apply(): Double = rnd.nextFloat

  override def inputSchema: StructType = StructType("input" -> ScalarType.String).get

  override def outputSchema: StructType = StructType("output" -> ScalarType.Double ).get

}

如何使其不必作为输入架构？

Answer 1

我从没有尝试过，但是考虑到我是如何实现具有多个输入字段的自定义转换器的...

package org.apache.spark.ml.feature.mleap

import ml.combust.mleap.core.Model
import ml.combust.mleap.core.types._
import org.apache.spark.ml.linalg._

case class PropertyGroupAggregatorBaseModel (props: Array[String],
                                        aggFunc: String) extends Model {
  val outputSize = props.size

  //having multiple inputs, you will have apply with a parameter Seq[Any]
  def apply(features: Seq[Any]): Vector = {
    val properties = features(0).asInstanceOf[Seq[String]]
    val values = features(1).asInstanceOf[Seq[Double]]
    val mapping = properties.zip(values)
    val histogram = props.foldLeft(Array.empty[Double]){
      (acc, property) =>
        val newValues = mapping.filter(x => x._1 == property).map(x => x._2)
        val newAggregate = aggFunc match {
          case "sum" => newValues.sum.toDouble
          case "count" => newValues.size.toDouble
          case "avg" => (newValues.sum / Math.max(newValues.size, 1)).toDouble
        }
        acc :+ newAggregate
    }

    Vectors.dense(histogram)
  }

  override def inputSchema: StructType =  {
    //here you define the input 
    val inputFields = Seq(
      StructField("input1" -> ListType(BasicType.String)),
      StructField("input2" -> ListType(BasicType.Double))
    )
    StructType(inputFields).get
  }

  override def outputSchema: StructType = StructType(StructField("output" -> TensorType.Double(outputSize))).get
}

我的建议是，该申请可能已经为您服务。我想如果您将inputSchema定义如下，则可能会起作用：

override def inputSchema: StructType =  {
    //here you define the input 
    val inputFields = Seq.empty[StructField]
    StructType(inputFields).get
  }

如何创建没有任何输入列的自定义转换器？

1 个答案: