Question

我已经在SO上研究了很多pool.map，但仍然无法找到任何暗示我的问题的东西。

我在每个.py文件中都有if __name__ == '__main__'。我在每个包含freeze_support()的.py中都有import multiprocessing，我仍然对所发生的事情感到茫然。我已经在代码中移动了freeze_support()，但结果不一致。

脚本A调用脚本B，脚本B调用脚本C（多处理发生的地方）。在本地，这种情况非常有效，但是当我将其加载到Windows Server 2008计算机时，奇怪的事情就开始发生了。

在服务器上，我可以看到第一个可迭代打印到解释器，但它然后跳回到脚本B并继续处理。脚本C的列表中还有51个其他项目。

脚本B代码：

if not arcpy.Exists(MergedDataFC):
    ScriptC.intersect_main(input1, input2) 

if not arcpy.Exists(MergedDataSHP):
    shpList = arcpy.ListFields(*.shp) # output of multiprocess
    # Merge all shapefiles into single shapefile
    # Being executed before the multiprocess finishes all 52 items

脚本C代码：

import multiprocessing as mp

def intersect_main(input1,input2):      
try: 
    mp.freeze_support()
    # Create a list of states for input1 polygons  
    log.log("Creating Polygon State list...")  
    fldList = arcpy.ListFields(input1) 
    flds = [fld.name for fld in fldList]
    idList = []  
    with arcpy.da.SearchCursor(input1, flds) as cursor:  
        for row in cursor:  
            idSTATE = row[flds.index("STATE")]
            idList.append(idSTATE)  

    idList = set(idList)
    log.log("There are " + str(len(idList)) + " States (polygons) to process.")  

    log.log("Sending to pool")  
    # declare number of cores to use, use 1 less than the max  
    cpuNum = mp.cpu_count() -1

    # Create the pool object  
    pool = mp.Pool(processes=cpuNum)  

    # Fire off list to worker function.  
    # res is a list that is created with what ever the worker function is returning
    log.log ("Entering intersectWork")  
    res = pool.map((intersectWork(input1, input2, idSTATE)),idList)
    pool.close()  
    pool.join()  

    # If an error has occurred report it  
    if False in res:  
        log.log ("A worker failed!")  
        log.log (strftime('[%H:%M:%S]', localtime()))
        raise Exception
    else:
        log.log("Finished multiprocessing!")
        log.log (strftime('[%H:%M:%S]', localtime()))  
except Exception, e:
    tb = sys.exc_info()[2]   
    # Geoprocessor threw an error 
    log.log("An error occurred on line " + str(tb.tb_lineno)) 
    log.log (str(e))

def intersectWork(input1,input2, idSTATE):  
try:  
    if idSTATE == None:
        query = "STATE IS NULL"
        idSTATE = 'pr'
    else:
        query = "STATE = '" + idSTATE + "'"

    DEMOlayer = arcpy.MakeFeatureLayer_management(input1,"input1_" + idSTATE)

    log.log (query)
    arcpy.SelectLayerByAttribute_management(DEMOlayer,"NEW_SELECTION",query)  

    # Do the Intersect  
    outFC = r'C:/EclipseWorkspace' + '/INTER_' + idSTATE.upper() + '.shp'
    strIntersect = str(DEMOlayer) + ";" + str(input2)
    arcpy.Intersect_analysis(strIntersect, outFC, "ALL", "", "LINE")
    return True  
except:  
    # Some error occurred so return False 
    log.log(arcpy.GetMessage(2))
    return False

if __name__ == '__main__':
    intersect_main(input1, input2)

修改

服务器上的所有数据都存储在本地，而不是网络处理。

Answer 1

问题是数据的完整路径没有正确地传递到服务器上的pool.map（），而不是之前的模块。我必须在import语句下添加所有文件路径。看起来不是很优雅，但它正在发挥作用。

Python Pool.map（） - 本地工作，在服务器上失败

1 个答案: