Question

我是hyperopt包的新手。现在，我想优化我在gensim中实现的LDA模型。优化LDA模型以最大化剪影得分而不是训练数据。

现在，我的问题是＆＃34;如何将训练数据（numpy.ndarray）传递给从hyperopt调用的目标函数？＆＃34; 我查看了教程和一些example codes。他们将训练数据设置为全局变量。但在我的情况下，将训练数据设置为全局变量并不困难。

我编写了以下代码来使用hyoeropt优化LDA。我已经将培训数据传递给gensim_objective_function函数，因为我会将gensim_lda_optimaze放入调用gensim_lda_optimaze函数的系统中。

如何实现？

# I want to pass training data to this function!
# gensim_lda_tuning_training_corpus, gensim_lda_tuning_num_topic, gensim_lda_tuning_word2id is what I wanna pass
def gensim_objective_function(arg_dict):
    from .gensim_lda import evaluate_clustering
    from .gensim_lda import call_lda_single
    from .gensim_lda import get_topics_ids

    alpha = arg_dict['alpha']
    eta = arg_dict['eta']
    iteration= arg_dict['iteration']
    gamma_threshold= arg_dict['gamma_threshold']
    minimum_probability= arg_dict['minimum_probability']
    passes= arg_dict['passes']
    # train LDA model
    lda_model, gensim_corpus = call_lda_single(matrix=gensim_lda_tuning_training_corpus,
                                               num_topics=gensim_lda_tuning_num_topic,
                                               word2id_dict=gensim_lda_tuning_word2id,
                                               alpha=alpha, eta=eta,
                                               iteration=iteration,
                                               gamma_threshold=gamma_threshold,
                                               minimum_probability=minimum_probability,
                                               passes=passes)
    topic_ids = get_topics_ids(trained_lda_model=lda_model, gensim_corpus=gensim_corpus)
    labels = [t[0] for t in topic_ids]
    # get silhouette score with extracted label
    evaluation_score = evaluate_clustering(feature_matrix=gensim_lda_tuning_training_corpus, labels=numpy.array(labels))

    return -1 * evaluation_score


def gensim_lda_optimaze(feature_matrix, num_topics, word2id_dict):
    assert isinstance(feature_matrix, (ndarray, csr_matrix))
    assert isinstance(num_topics, int)
    assert isinstance(word2id_dict, dict)

    parameter_space = {
        'alpha': hp.loguniform("alpha", numpy.log(0.1), numpy.log(1)),
        'eta': hp.loguniform("eta", numpy.log(0.1), numpy.log(1)),
        'iteration': 100,
        'gamma_threshold': 0.001,
        'minimum_probability': 0.01,
        'passes': 10
    }
    trials = Trials()

    best = fmin(
        gensim_objective_function,
        parameter_space,
        algo=tpe.suggest,
        max_evals=100,
        trials=trials
    )

    return best

Answer 1

您始终可以在python中使用partial。

from functools import partial

def foo(params, data):
  return params, data

goo = partial(foo, data=[1,2,3])

print goo('ala')

给出

ala [1, 2, 3]

换句话说，你创建一个代理函数，它将数据作为给定参数加载，并且你要求hyperopt优化这个新函数，并且已经设置了数据。

因此，在您的情况下，您将gensim_objective_function更改为接受所有参数的内容：

def RAW_gensim_objective_function(arg_dict, gensim_lda_tuning_training_corpus, 
                                  gensim_lda_tuning_num_topic,
                                  gensim_lda_tuning_word2id):

通过将数据传递到代码的不同部分来创建实际的优化函数

gensim_objective_function = partial(RAW_gensim_objective_function, 
                gensim_lda_tuning_training_corpus = YOUR_CORPUS, 
                gensim_lda_tuning_num_topic = YOUR_NUM_TOPICS,
                gensim_lda_tuning_word2id = YOUR_IDs)

如何将对象传递给使用hyperopt优化的函数？

1 个答案: