Question

我想用sklearn构建一个文本分类器，然后使用coremltools包将其转换为iOS11机器学习文件。我使用Logistic Regression，Random Forest和Linear SVC构建了三个不同的分类器，所有这些分类器在Python中运行良好。问题是coremltools包以及将sklearn模型转换为iOS文件的方式。正如its documentation所说，它只支持这些模型：

线性和逻辑回归
LinearSVC和LinearSVR
SVC和SVR
NuSVC和NuSVR
Gradient Boosting Classifier and Regressor
决策树分类器和回归器
随机森林分类器和回归器
正规化
Imputer
标准缩放器
DictVectorizer
One Hot Encoder

所以它不允许我对我的文本数据集进行矢量化（我在分类器中使用了TfidfVectorizer包）：

import coremltools
coreml_model = coremltools.converters.sklearn.convert(model, input_features='text', output_feature_names='category')

Traceback (most recent call last):

File "<ipython-input-3-97beddbdad10>", line 1, in <module>
    coreml_model = coremltools.converters.sklearn.convert(pipeline, input_features='Message', output_feature_names='Label')

  File "/usr/local/lib/python2.7/dist-packages/coremltools/converters/sklearn/_converter.py", line 146, in convert
    sk_obj, input_features, output_feature_names, class_labels = None)

  File "/usr/local/lib/python2.7/dist-packages/coremltools/converters/sklearn/_converter_internal.py", line 147, in _convert_sklearn_model
    for sk_obj_name, sk_obj in sk_obj_list]

  File "/usr/local/lib/python2.7/dist-packages/coremltools/converters/sklearn/_converter_internal.py", line 97, in _get_converter_module
    ",".join(k.__name__ for k in _converter_module_list)))

ValueError: Transformer 'TfidfVectorizer(analyzer='word', binary=False, decode_error=u'strict',
        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',
        lowercase=True, max_df=1.0, max_features=None, min_df=3,
        ngram_range=(1, 2), norm=u'l2', preprocessor=None, smooth_idf=1,
        stop_words='english', strip_accents='unicode', sublinear_tf=1,
        token_pattern='\\w+', tokenizer=None, use_idf=1, vocabulary=None)' not supported; 
supported transformers are coremltools.converters.sklearn._dict_vectorizer,coremltools.converters.sklearn._one_hot_encoder,coremltools.converters.sklearn._normalizer,coremltools.converters.sklearn._standard_scaler,coremltools.converters.sklearn._imputer,coremltools.converters.sklearn._NuSVC,coremltools.converters.sklearn._NuSVR,coremltools.converters.sklearn._SVC,coremltools.converters.sklearn._SVR,coremltools.converters.sklearn._linear_regression,coremltools.converters.sklearn._LinearSVC,coremltools.converters.sklearn._LinearSVR,coremltools.converters.sklearn._logistic_regression,coremltools.converters.sklearn._random_forest_classifier,coremltools.converters.sklearn._random_forest_regressor,coremltools.converters.sklearn._decision_tree_classifier,coremltools.converters.sklearn._decision_tree_regressor,coremltools.converters.sklearn._gradient_boosting_classifier,coremltools.converters.sklearn._gradient_boosting_regressor.

有没有办法构建一个sklearn文本分类器而不使用TfidfVectorizer或CountVectorizer模型？

Answer 1

现在，如果要将其转换为.mlmodel格式，则无法在管道中包含tf-idf矢量化程序。解决这个问题的方法是分别对数据进行矢量化，然后使用矢量化数据训练模型（线性SVC，随机森林......）。然后，您需要计算设备上的tf-idf表示，然后可以将其插入模型中。这是我写的tf-idf函数的副本。

func tfidf(document: String) -> MLMultiArray{
    let wordsFile = Bundle.main.path(forResource: "words_ordered", ofType: "txt")
    let dataFile = Bundle.main.path(forResource: "data", ofType: "txt")
    do {
        let wordsFileText = try String(contentsOfFile: wordsFile!, encoding: String.Encoding.utf8)
        var wordsData = wordsFileText.components(separatedBy: .newlines)
        let dataFileText = try String(contentsOfFile: dataFile!, encoding: String.Encoding.utf8)
        var data = dataFileText.components(separatedBy: .newlines)
        let wordsInMessage = document.split(separator: " ")
        var vectorized = try MLMultiArray(shape: [NSNumber(integerLiteral: wordsData.count)], dataType: MLMultiArrayDataType.double)
        for i in 0..<wordsData.count{
            let word = wordsData[i]
            if document.contains(word){
                var wordCount = 0
                for substr in wordsInMessage{
                    if substr.elementsEqual(word){
                        wordCount += 1
                    }
                }
                let tf = Double(wordCount) / Double(wordsInMessage.count)
                var docCount = 0
                for line in data{
                    if line.contains(word) {
                        docCount += 1
                    }
                }
                let idf = log(Double(data.count) / Double(docCount))
                vectorized[i] = NSNumber(value: tf * idf)
            } else {
                vectorized[i] = 0.0
            }
        }
        return vectorized
    } catch {
        return MLMultiArray()
    }
}

编辑：在http://gokulswamy.me/imessage-spam-detection/撰写了关于如何执行此操作的完整帖子。

构建一个sklearn文本分类器并使用coremltools转换它

1 个答案: