Cloud ML Engine批量预测 - 如何简单地将返回的预测与输入数据匹配？

时间：2017-08-01 09:26:33

标签： tensorflow tensorflow-serving google-cloud-ml google-cloud-ml-engine

根据ML Engine documentation，需要一个实例键来使返回的预测与输入数据相匹配。为简单起见，我想使用DNNClassifier，但显然罐装估算器似乎还不支持实例键（只有自定义或张量流核心估算器）。

所以我查看了Custom / TensorflowCore Estimators的Census代码示例，但它们对于我想要实现的目标看起来非常复杂。

我更喜欢使用此stackoverflow answer中描述的类似方法（将DNNClassifier包装到自定义估算器中）但我可以 没有让它工作，我得到一个错误，说'DNNClassifier'对象没有属性'model_fn'...

如何以简单的方式实现这一目标？

3 个答案:

答案 0 :(得分：1)

在版本1.2中，贡献估计器（例如tf.contrib.learn.DNNClassifier）被更改为从核心估计器类tf.estimator.Estimator继承，与其前身不同，它将模型函数隐藏为私有类成员。

尝试estimator._model_fn而不是estimator.model_fn。你应该能够在我之前的答案中保留其他所有内容。

编辑：我在这里更新了我的原始答案：https://stackoverflow.com/a/44443380/3597868以反映版本1.2的必要更改

答案 1 :(得分：1)

我的代码根据Eli的例子：

def key_model_fn_gen(estimator):
   def _model_fn(feature_columns, labels, mode):
      key = feature_columns.pop(KEY)
      params = estimator.params

      model_fn_ops = estimator._model_fn(features=feature_columns,
                 labels=labels,
                 mode=mode,
                 params=params)
      model_fn_ops.predictions[KEY] = key        

     return model_fn_ops
return _model_fn

但仍未成功使用ML Engine批量预测在预测结果中显示实例键... 我需要在实验中改变什么（或者可能在出口策略中） 使它工作？

答案 2 :(得分：0)

系统/版本信息

  1. Canned census example在2017_06_22_15：06：37承诺。
  2. TensorFlow 1.2。
  3. Python 3
  4. GCP ML Engine 1.2

    5. 方法

    Fabrice ，我和你有同样的问题，我花了一段时间来解决这个问题（在 Eli 的慷慨帮助下）。我采取了略微不同的方法。 我没有尝试创建实例密钥，而是假设实例密钥位于数据（培训，评估和预测）中。

    在这里，我使用 gender 字段作为实例键。显然，我不会将性别字段在现实中用作实例密钥，我只是在这里用它来进行说明。

    除了这里描述的那些更改之外，我没有对原始脚本中的任何其他函数或常量进行任何更新，除了将某些内容从python 2更改为python 3，例如，将dict.iteritems（）更改为dict.items （）。

    以下是我修改后的model.py文件的要点。我没有对task.py文件进行任何更改。

    更新key_model_fn_gen()功能

    此代码依赖于我从Eli获得的指导。对我的见解是，我需要修改output_alternatives字典以便返回密钥，并且我不需要修改预测字典。 （另外，我了解到我可以从你的（Fabrice's）例子中将params作为估算器的一个属性，谢谢你。）

    KEY = 'gender'
def key_model_fn_gen(estimator):
    def _model_fn(features, labels, mode):
        key = features.pop(KEY)
        params = estimator.params
        model_fn_ops = estimator._model_fn(features=features, labels=labels, mode=mode, params=params)
        model_fn_ops.output_alternatives[None][1]['key'] = key
        return model_fn_ops
    return _model_fn

    更新build_estimator()功能

    1. 我从deep_columns列表和wide_columns列表中删除性别，以便它不会用作培训和评估功能。
    2. 我修改了return以包含每Eli's guidance的密钥包装器。
    3. 我将model_dir作为config的一个属性。

      4. 以下是完整代码：

      def build_estimator(config, embedding_size=8, hidden_units=None):

  (gender, race, education, marital_status, relationship,
   workclass, occupation, native_country, age,
   education_num, capital_gain, capital_loss, hours_per_week) = INPUT_COLUMNS
  """Build an estimator."""

  # Reused Transformations.
  # Continuous columns can be converted to categorical via bucketization
  age_buckets = tf.feature_column.bucketized_column(
      age, boundaries=[18, 25, 30, 35, 40, 45, 50, 55, 60, 65])

  # Wide columns and deep columns.
  wide_columns = [
      # Interactions between different categorical features can also
      # be added as new virtual features.
      tf.feature_column.crossed_column(
          ['education', 'occupation'], hash_bucket_size=int(1e4)),
      tf.feature_column.crossed_column(
          [age_buckets, race, 'occupation'], hash_bucket_size=int(1e6)),
      tf.feature_column.crossed_column(
          ['native_country', 'occupation'], hash_bucket_size=int(1e4)),
      native_country,
      education,
      occupation,
      workclass,
      marital_status,
      relationship,
      age_buckets,
  ]

  deep_columns = [
      # Use indicator columns for low dimensional vocabularies
      tf.feature_column.indicator_column(workclass),
      tf.feature_column.indicator_column(education),
      tf.feature_column.indicator_column(marital_status),
      tf.feature_column.indicator_column(relationship),
      tf.feature_column.indicator_column(race),

      # Use embedding columns for high dimensional vocabularies
      tf.feature_column.embedding_column(
          native_country, dimension=embedding_size),
      tf.feature_column.embedding_column(occupation, dimension=embedding_size),
      age,
      education_num,
      capital_gain,
      capital_loss,
      hours_per_week,
  ]

  return tf.contrib.learn.Estimator(
      model_fn=key_model_fn_gen(
          tf.contrib.learn.DNNLinearCombinedClassifier(
              config=config,
              linear_feature_columns=wide_columns,
              dnn_feature_columns=deep_columns,
              dnn_hidden_units=hidden_units or [100, 70, 50, 25],
              fix_global_step_increment_bug=True)
      ),
      model_dir=config.model_dir
  )

      批量预测的输入格式

      将版本上传到ML Engine后，预测输入采用以下形式：

      {"native_country":" United-States","race":" Black","age":"44","relationship":" Other-relative","gender":" Male","marital_status":" Never-married","hours_per_week":"32","capital_gain":"0","education_num":"9","education":" HS-grad","occupation":" Other-service","capital_loss":"0","workclass":" Private"}
{"native_country":" United-States","race":" White","age":"35","relationship":" Not-in-family","gender":" Male","marital_status":" Divorced","hours_per_week":"40","capital_gain":"0","education_num":"9","education":" HS-grad","occupation":" Craft-repair","capital_loss":"0","workclass":" Private"}
{"native_country":" United-States","race":" White","age":"20","relationship":" Husband","gender":" Male","marital_status":" Married-civ-spouse","hours_per_week":"40","capital_gain":"0","education_num":"10","education":" Some-college","occupation":" Craft-repair","capital_loss":"0","workclass":" Private"}
{"native_country":" United-States","race":" White","age":"43","relationship":" Husband","gender":" Male","marital_status":" Married-civ-spouse","hours_per_week":"50","capital_gain":"0","education_num":"10","education":" Some-college","occupation":" Farming-fishing","capital_loss":"0","workclass":" Self-emp-not-inc"}
{"native_country":" England","race":" White","age":"33","relationship":" Husband","gender":" Male","marital_status":" Married-civ-spouse","hours_per_week":"40","capital_gain":"0","education_num":"13","education":" Bachelors","occupation":" Farming-fishing","capital_loss":"0","workclass":" Private"}
{"native_country":" United-States","race":" White","age":"38","relationship":" Unmarried","gender":" Female","marital_status":" Divorced","hours_per_week":"56","capital_gain":"0","education_num":"13","education":" Bachelors","occupation":" Prof-specialty","capital_loss":"0","workclass":" Private"}
{"native_country":" United-States","race":" White","age":"53","relationship":" Not-in-family","gender":" Female","marital_status":" Never-married","hours_per_week":"35","capital_gain":"8614","education_num":"14","education":" Masters","occupation":" ?","capital_loss":"0","workclass":" ?"}
{"native_country":" China","race":" Asian-Pac-Islander","age":"64","relationship":" Husband","gender":" Male","marital_status":" Married-civ-spouse","hours_per_week":"60","capital_gain":"0","education_num":"14","education":" Masters","occupation":" Prof-specialty","capital_loss":"2057","workclass":" Private"}

      批量预测的输出格式

      完成批量预测作业后，我得到以下输出：

      {"probabilities": [0.9633187055587769, 0.036681365221738815], "classes": ["0", "1"], "key": [" Male"]}
{"probabilities": [0.9452069997787476, 0.05479296296834946], "classes": ["0", "1"], "key": [" Male"]}
{"probabilities": [0.8586776852607727, 0.1413223296403885], "classes": ["0", "1"], "key": [" Male"]}
{"probabilities": [0.7370017170906067, 0.2629982531070709], "classes": ["0", "1"], "key": [" Male"]}
{"probabilities": [0.48797568678855896, 0.5120242238044739], "classes": ["0", "1"], "key": [" Male"]}
{"probabilities": [0.8111950755119324, 0.18880495429039001], "classes": ["0", "1"], "key": [" Female"]}
{"probabilities": [0.5560402274131775, 0.4439597725868225], "classes": ["0", "1"], "key": [" Female"]}
{"probabilities": [0.3235422968864441, 0.6764576435089111], "classes": ["0", "1"], "key": [" Male"]}
相关问题
最新问题