Question

function cronProcess() {
    # > 100,000 users 
    $users = $this->UserModel->getUsers();

    foreach ($users as $user) {
        # Do lots of database Insert/Update/Delete, HTTP request stuff
    }
}

当用户数量达到100,000时，就会发生此问题。

我通过CronTab通过CURL调用了该函数。

那么最好的解决方案是什么？

Answer 1

我在CakePHP中完成了很多批量任务，其中一些任务处理了数百万条记录。当然是可以做到的，正如其他人建议的那样，关键是循环中的小批处理。

如果您是从Cron那里打电话来的，那么使用Shell（Command类（v3.6 +）可能比cURL更容易。

一般来说，这是我对大批量进行分页的方式，包括一些有用的可选内容，例如进度条，关闭水合作用以加快速度，并显示脚本能够处理多少用户/秒：

<?php

namespace App\Command;

use Cake\Console\Arguments;
use Cake\Console\Command;
use Cake\Console\ConsoleIo;

class UsersCommand extends Command
{
    public function execute(Arguments $args, ConsoleIo $io)
    {
        // I'd guess a Finder would be a more Cake-y way of getting users than a custom "getUsers" function:
        // See https://book.cakephp.org/3.0/en/orm/retrieving-data-and-resultsets.html#custom-finder-methods
        $usersQuery = $this->UserModel->find('users');

        // Get a total so we know how many we're gonna have to process (optional)
        $total = $usersQuery->count();
        if ($total === 0) {
            $this->abort("No users found, stopping..");
        }

        // Hydration takes extra processing time & memory, which can add up in bulk. Optionally if able, skip it & work with $user as an array not an object:
        $usersQuery->enableHydration(false);

        $this->info("Grabbing $total users for processing");

        // Optionally show the progress so we can visually see how far we are in the process
        $progress = $io->helper('Progress')->init([
            'total' => 10
        ]);

        // Tune this page value to a size that solves your problem:
        $limit = 1000;
        $offset = 0;

        // Simply drawing the progress bar every loop can slow things down, optionally draw it only every n-loops,
        // this sets it to 1/5th the page size:
        $progressInterval = $limit / 5;

        // Optionally track the rate so we can evaluate the speed of the process, helpful tuning limit and evaluating enableHydration effects
        $startTime = microtime(true);
        do {
            $users = $usersQuery->offset($offset)->toArray();
            $count = count($users);
            $index = 0;

            foreach ($users as $user) {
                $progress->increment(1);

                // Only draw occasionally, for speed
                if ($index % $progressInterval === 0) {
                    $progress->draw();
                }

                ### WORK TIME
                # Do your lots of database Insert/Update/Delete, HTTP request stuff etc. here
                ###
            }

            $progress->draw();

            $offset += $limit; // Increment your offset to the next page
        } while ($count > 0);

        $totalTime = microtime(true) - $startTime;
        $this->out("\nProcessed an average " . ($total / $totalTime) . " Users/sec\n");
    }
}

在CakePHP文档中查看以下部分：


Console Commands

Command Helpers

Using Finders & Disabling Hydration

希望这会有所帮助！

解决方案，由Cron调用一个在其中做很多事情的函数？

1 个答案: