Question

我有一个脚本可以检查哪个MX记录属于电子邮件地址。我有大约30万封电子邮件需要检查。因此单线程进程需要很长时间。

我有一个队列的beanstalkd，php正在通过文件向它发送电子邮件。但是我只有一个工作人员来执行队列。我目前正在为一个过程产生10多名工人。

我运行do_job_mx.php然后打开一个只包含电子邮件的文件并将它们传递给队列。

用于从文件中收集电子邮件并放入队列的php代码 - do_job_mx.php：

require_once('pheanstalk_init.php');

$pheanstalk = new Pheanstalk_Pheanstalk('127.0.0.1:11300');

$filename = '_blank.txt';
$filename = dirname(__FILE__) . '/in/' . $filename;

foreach (file($filename, FILE_SKIP_EMPTY_LINES) as $line) 
{
    $json = json_encode(array("email" => trim($line)));

    $pheanstalk
        ->useTube('process_mx')
        ->put($json);
}

worker的php代码 - do_worker_process_mx.php：

class Worker 
{
    public function __construct() 
    {
        $this->log('worker process - starting');

        require_once('pheanstalk_init.php');
        $this->pheanstalk = new Pheanstalk_Pheanstalk('127.0.0.1:11300');
    }

    public function __destruct() 
    {
        $this->log('worker process - ending');
    }

    public function run() 
    {
        $this->log('worker process - starting to run');

        while(1) 
        {
            $job = $this->pheanstalk
                ->watch('process_mx')
                ->ignore('default')
                ->reserve();

            $data = json_decode($job->getData(), true);

            $this->process_mx($data);

            $this->pheanstalk->delete($job);
        }
    }

    private function process_mx($data)
    {
        $domain = explode("@", $data['email']);

        dns_get_mx($domain[1], $mx_records);

        $mx_array = explode(".", strtolower($mx_records[0]));

        $mx = array_slice($mx_array, -2, count($mx_array));

        $mx_domain = implode(".", $mx);

        echo $data['email'] . "\n";

        $this->write_file($mx_domain, $data['email']);
    }       

    private function write_file($mx, $email)
    {
        $filename = fopen(dirname(__FILE__) . "/out/" . $mx . ".txt", 'ab+');

        fwrite($filename, $email . "\n");

        fclose($filename);
    }

    private function log($txt) 
    {
        echo $txt . "\n";
    }
}

$worker = new Worker();
$worker->run();

Supervisord conf：

[program:do_worker_process]
command=/usr/bin/php /srv/www/mydev/public_html/esp/do_worker_process_mx.php
numprocs=10
numprocs_start=10
autostart=true
autorestart=true
stopsignal=QUIT
log_stdout=true
logfile=/var/log/supervisor/worker_process_mx.log

我目前无法为一个过程产生10多名工人。

正在运行的进程数：

# supervisorctl status

do_worker_process RUNNING    pid 44343, uptime 1:46:11

Answer 1

Centos 6附带：

beanstalkd 1.4.6 主管2.1.8

我只需要升级到supervisor 3.0。

现在我有多个工人设施。

PHP / Beanstalkd：并行生成多个worker

1 个答案: