Question

我试图运行带有进程池的mcmc（我必须使用Billiard，因为我的Celery中也有我的代码），但是我有mcmc调用的两个函数，它们访问数据库，一个函数编写信息到数据库中，另一方只是读取它。我只需要在没有其他进程正在执行这两个步骤的情况下完成这两个步骤，并且当一个进程正在执行任何一个操作时，则该进程必须等到完成为止，而所有其他进程都必须等到轮到完成为止。我似乎找不到与此相关的任何内容，但我也在努力在搜索中表达它的短语。这是我意思的模拟版本：

import emcee
import time
import billiard.pool as BilPool


# Both Check_Database and Store_Calculations can't be done at the same time. Only one 
# process at a time can do either of these functions or it locks the database
################################
def Check_Database(Input):
    #####
    # Code that searches database to see if this input was already run
    #####
    if Input in Database:
        # showing as if the database was a dictionary for simplicity
        Observation = Database[Input]
        return Observation
    else:
        return None


def Store_Calculations(Observation):
    #####
    # Code that stores the output in a sql database
    #####
    return None
################################

# This can be done in parallel with no issues
def ArbitraryFunction(Input):
    #####
    # In my actual code this is something that can take several minutes to run on its own
    # Here I am just doing a time.sleep() for simplicity
    #####
    time.sleep(Input)
    return 2/Input


# Mock likelihood for the actual mcmc
def Likelihood(Input)
    Observation = Check_Database(Input)
    if Observation == None:
        Observation = ArbitraryFunction(Input)
    Store_Calculations(Observation)
    return Observation
   
# Parameters for the mcmc, can be whatever not important to the problem.
################################
number_walkers = 10
StartingPositions = [0,1,2,3,4,5,6,7,8,9]
number_dimensions = 1
number_steps = 1000
################################

with BilPool.Pool(4) as pool:
     sampler = emcee.EnsembleSample(number_walkers, number_dimension, Likelihood, 
                                    pool=pool)
     sampler.run_mcmc(StartingPositions, number_steps)

很抱歉，如果这是重复的内容，但是在尝试搜索时我找不到它。我希望我的模拟代码有意义并解决问题。我是多处理事物的新手，所以我不知道我是否在要求不可能的事情。

并行处理，一个串行功能

0 个答案: