SkFlow:将数字和文本数据输入模型

时间:2016-12-14 20:15:22

标签: python tensorflow skflow

我在学习SkFlow / TensorFlow的早期阶段,所以我会对我尝试做的事情进行理解,尽管不正确。

让我们想象一下,我试图建立一个模型来预测汽车是否会通过排放测试。

我的训练和测试csv可能看起来像这样

 make, fuel,   year, mileage, days since service, passed test
   vw, diesel, 2015, 10000,   20,                 0
honda, petrol, 2008, 1000000, 234,                1

因此pass/fail列为y,其他列为x。

到目前为止,巴尔的摩在我之前SO question的帮助下,我能够处理CSV文件中的Iris数据集。然而,该数据集是所有数字。

example on the TensorFlow网站使用分类和连续数据显示使用人口普查数据构建的模型。我试图使用SkFlow,因为我理解它简化了过程。

无论如何,到我的代码

x_train = genfromtxt('/Users/ben/Desktop/data.csv', dtype=None, delimiter=',' , usecols=(0, 1, 2, 3,4))
y_train = genfromtxt('/Users/ben/Desktop/data.csv', dtype='int', delimiter=',', usecols = (5))

feature_columns = [tf.contrib.layers.real_valued_column("", dimension=1)]

classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
                                    hidden_units=[10, 20, 10],
                                    n_classes=2,
                                    model_dir="./tmp/model1")

# Fit model. Add your train data here
classifier.fit(x=x_train,y=y_train,steps=2000)

因此,我已将我的csv数据读入我的x_trainy_train对象中。 CSV没有标题,但如果需要可以执行。

我相信我试图定义哪些列包含哪种数据,例如

make = tf.contrib.layers.sparse_column_with_hash_bucket("make", hash_bucket_size=1000)

fuel = tf.contrib.layers.sparse_column_with_keys(column_name="fuel", keys=["diesel", "petrol"])

如何构建传递到feature_columns的<{1}}对象?

1 个答案:

答案 0 :(得分:2)

这是我的镜头。 input_fn函数创建一个张量的dict,通过包装器传递给fit和evaluate方法。在创建模型时使用该dict。它定义了将要使用的数据。其他常数值张量是数据。它们是在使用feature_columns参数进行模型拟合时传递的:feature_columns = [gear,mpg,cyl ...]。

我遗漏了所有交叉的栏目,但可以放入。

我关闭了警告,但如果你想要,那就是那里的开关。这也会产生令人惊讶的日志数据量,因此请务必查看带有张量板的图表。

# an experiment with regression in Tensorflow using one categorical feature
# MTCARS - auto data. Is the car an Automatic or a Manual Shift?
# Data set location: https://vincentarelbundock.github.io/Rdatasets/datasets.html
# Below is a HIGHLY cut down version of the tensorflow wide tutorial at:
# https://www.tensorflow.org/tutorials/wide/

import tensorflow as tf
import numpy as np
import urllib.request
import tempfile
import pandas as pd
from sklearn.model_selection import train_test_split

LABEL_COLUMN = "label"
COLUMNS = ["mpg","cyl","disp","hp","drat","wt","qsec","vs","am","gear","carb"]
CONTINUOUS_COLUMNS = ["mpg","cyl","disp","hp","drat","wt","qsec","vs","carb"]
CATEGORICAL_COLUMNS = ["gear"]

# had to update the urllib stuff for 3.5.
# pull down csv file
# I am running on ubuntu 14.04, so I don't know how well the tempfile stuff    will work on Windows.
# NamedTemporaryFile might have problems
data_file = tempfile.NamedTemporaryFile()
urllib.request.urlretrieve("https://vincentarelbundock.github.io/Rdatasets/csv/datasets/mtcars.csv", data_file.name)

cars = pd.read_csv(data_file, names=COLUMNS, skipinitialspace=True,skiprows=1)
# I want the "am" column as my label, so rename it - not really necessary,
# just trying to stay in sync the wide tutorial
# am: 0 = Automatic 1 = Manual
cars.rename(columns={'am':LABEL_COLUMN}, inplace=True)

# turn gears into a categorical variable, again not really useful, but I want    some categorical data
# turn the numbers into strings. I sure there is a oneliner somewhere that can do this...
cars['gear'] = cars['gear'].astype(str)
cars['gear'] = cars['gear'].replace({'3': 'THREE'}, regex=True)
cars['gear'] = cars['gear'].replace({'4': 'FOUR'}, regex=True)
cars['gear'] = cars['gear'].replace({'5': 'FIVE'}, regex=True)

# split into train and tests set - there is a woefully small number of rows here. Need a bigger data set.
train, test = train_test_split(cars, test_size = 0.2)

# These methods are a copy of the input functions from the tensorflow wide tutorial updated for python 3.5
def input_fn(df):
  # Creates a dictionary mapping from each continuous feature column name (k) to
  # the values of that column stored in a constant Tensor.
  continuous_cols = {k: tf.constant(df[k].values)
                     for k in CONTINUOUS_COLUMNS}
  # Creates a dictionary mapping from each categorical feature column name (k)
  # to the values of that column stored in a tf.SparseTensor.
  categorical_cols = {k: tf.SparseTensor(
      indices=[[i, 0] for i in range(df[k].size)],
      values=df[k].values,
      shape=[df[k].size, 1])
                  for k in CATEGORICAL_COLUMNS}

  # Merges the two dictionaries into one.
  # Old CODE
  #feature_cols = dict(continuous_cols.items() + categorical_cols.items())
  # NEW CODE - python 3.5
  feature_cols = dict(continuous_cols)
  feature_cols.update(categorical_cols)

  # Converts the label column into a constant Tensor.
  label = tf.constant(df[LABEL_COLUMN].values)
  # Returns the feature columns and the label.
  return feature_cols, label

def train_input_fn():
  return input_fn(train)

def eval_input_fn():
  return input_fn(test)

# shut down WARNINGs
# You can adjust by using DEBUG, INFO, WARN, ERROR, or FATAL
tf.logging.set_verbosity(tf.logging.ERROR)

# set up the TF column for the categorical variable
gear = tf.contrib.layers.sparse_column_with_keys(column_name="gear", keys=["THREE", "FOUR", "FIVE"])
# if my categorical data had more than 10 keys, I would use:
#gear = tf.contrib.layers.sparse_column_with_hash_bucket("gear", hash_bucket_size=1000)

# set up the TF columns for the continous variables
mpg = tf.contrib.layers.real_valued_column("mpg")
cyl = tf.contrib.layers.real_valued_column("cyl")
disp = tf.contrib.layers.real_valued_column("disp")
hp = tf.contrib.layers.real_valued_column("hp")
drat = tf.contrib.layers.real_valued_column("drat")
wt = tf.contrib.layers.real_valued_column("wt")
qsec = tf.contrib.layers.real_valued_column("qsec")
vs = tf.contrib.layers.real_valued_column("vs")
carb = tf.contrib.layers.real_valued_column("carb")

# Build the model. Make sure the logs dir already exists.
model_dir = "./logs"
m = tf.contrib.learn.LinearClassifier(
    feature_columns=[gear,mpg,cyl,disp,hp,drat,wt,qsec,vs,carb],
    optimizer=tf.train.FtrlOptimizer(
    learning_rate=0.01,
    l1_regularization_strength=1.0,
    l2_regularization_strength=1.0),
    model_dir=model_dir)

m.fit(input_fn=train_input_fn,steps=200)

# Results were not bad for a very small data set, but the recall is suspect
# In reality, these numbers don't mean a thing with such small data

results = m.evaluate(input_fn=eval_input_fn, steps=1)
for key in sorted(results):
    print("%s: %s" % (key, results[key]))

`