Python多处理仅使用一个核心

时间:2011-08-01 22:30:47

标签: python ubuntu multiprocessing

我正在尝试从standard python documentation获取代码片段,以了解如何使用多处理模块。代码将粘贴在此消息的末尾。 我在Ubuntu 11.04上使用Python 2.7.1在四核机器上(根据系统监视器,由于超线程而给我八个核心)

问题:尽管已启动多个进程,但所有工作负载似乎只安排在一个核心上,接近100%的利用率。有时,所有工作负载都会迁移到另一个核心,但工作负载永远不会在它们之间分配。

为什么会出现这种情况?

致以最诚挚的问候,

#
# Simple example which uses a pool of workers to carry out some tasks.
#
# Notice that the results will probably not come out of the output
# queue in the same in the same order as the corresponding tasks were
# put on the input queue.  If it is important to get the results back
# in the original order then consider using `Pool.map()` or
# `Pool.imap()` (which will save on the amount of code needed anyway).
#
# Copyright (c) 2006-2008, R Oudkerk
# All rights reserved.
#

import time
import random

from multiprocessing import Process, Queue, current_process, freeze_support

#
# Function run by worker processes
#

def worker(input, output):
    for func, args in iter(input.get, 'STOP'):
        result = calculate(func, args)
        output.put(result)

#
# Function used to calculate result
#

def calculate(func, args):
    result = func(*args)
    return '%s says that %s%s = %s' % \
        (current_process().name, func.__name__, args, result)

#
# Functions referenced by tasks
#

def mul(a, b):
    time.sleep(0.5*random.random())
    return a * b

def plus(a, b):
    time.sleep(0.5*random.random())
    return a + b


def test():
    NUMBER_OF_PROCESSES = 4
    TASKS1 = [(mul, (i, 7)) for i in range(500)]
    TASKS2 = [(plus, (i, 8)) for i in range(250)]

    # Create queues
    task_queue = Queue()
    done_queue = Queue()

    # Submit tasks
    for task in TASKS1:
        task_queue.put(task)

    # Start worker processes
    for i in range(NUMBER_OF_PROCESSES):
        Process(target=worker, args=(task_queue, done_queue)).start()

    # Get and print results
    print 'Unordered results:'
    for i in range(len(TASKS1)):
       print '\t', done_queue.get()

    # Add more tasks using `put()`
    for task in TASKS2:
        task_queue.put(task)

    # Get and print some more results
    for i in range(len(TASKS2)):
        print '\t', done_queue.get()

    # Tell child processes to stop
    for i in range(NUMBER_OF_PROCESSES):
        task_queue.put('STOP')

test()

4 个答案:

答案 0 :(得分:2)

一些如何更改CPU亲和力。我之前遇到过numpy这个问题。我在这里找到了解决方案 http://bugs.python.org/issue17038#msg180663

答案 1 :(得分:2)

尝试用实际需要CPU的内容替换time.sleep,您会看到multiprocess工作正常!例如:

def mul(a, b):
    for i in xrange(100000):
        j = i**2
    return a * b

def plus(a, b):
    for i in xrange(100000):
        j = i**2
    return a + b

答案 2 :(得分:0)

多处理并不意味着你将使用处理器的所有核心,你只需要多个进程而不是多核进程,这将由操作系统处理并且不确定,question @Devraj发布在评论有完成你想要的答案。

答案 3 :(得分:-1)

我找到了一个使用Parallel Python的工作。我知道这不是使用基本Python库的解决方案,但代码很简单,就像魅力一样