Question

我在Laravel平台上有一个处理数千条记录的脚本，以及它的超时问题。我想避免服务器上的更新设置，如php.ini文件。

有没有办法将所有记录分成小块并在进一步处理中使用它们直到所有记录都被使用。

我有数千条记录从其他服务器获得价格。

所以，

1-10记录工作正常。

1-100条记录正常。

超过500个记录脚本卡住并出现如下错误。

不允许使用405 nginx服务器

我的路线如下： - example.com/updateAllKeywords

以下代码是获取关键字以从eBay API获取关键字＆＃39;价格

        $keywordCollection = Keywords::all();

        Keywords::chunk(100, function ($keywordCollection) use($request,$keywordCollection,$defaultStartDate,$ignoredItems,$response) {
            foreach ($keywordCollection->toArray() as $keyword_result) {
                $response = $this->apirequest($request, $keyword_result['id'], $keyword_result, $defaultStartDate, $ignoredItems);
            }
        });

这是api请求功能：

 /**
 * Call Finding API for Active List items 
 *
 * @return API Response
 */
public function apirequest(Request $request, $id, $keyword_result, $defaultStartDate, $ignoredItems) {

    try {

        /* Request */
        $request = new Types\FindItemsAdvancedRequest();

        /* 1) default 100 items being displayed without pagination */
        $request->paginationInput = new Types\PaginationInput();
        $request->paginationInput->entriesPerPage = Config::get('constants.paginationoptions.entries_per_page');

        /* 2) ItemFilter for three conditions NEW ,NEWOTHER,USED */
        $itemFilterForConditions = new Types\ItemFilter();
        $itemFilterForConditions->name = 'Condition';
        $itemFilterForConditions->value = ['1000', '1500', '3000'];

        /* 3) ItemFilter for hide duplicate items */
        $itemFilterduplicate = new Types\ItemFilter();
        $itemFilterduplicate->name = 'HideDuplicateItems';
        $itemFilterduplicate->value[] = 'true';

        /* 4) pull items before given date / days */
        $itemFilterStartDateFrom = new Types\ItemFilter();
        $itemFilterStartDateFrom->name = 'StartTimeFrom';
        $itemFilterStartDateFrom->value[] = date('Y-m-d\TH:i:s\Z', mktime(0, 0, 0, date('m'), date('d') - $defaultStartDate['defaultStartDate'], date('y')));

        /* Exclude Keywords/MPN by set Priced Percentage */
        if (!empty($keyword_result['price']) && ($defaultStartDate['exclusionPercentage'] > 0)) {
            $percentageAmount = $keyword_result['price'] * ($defaultStartDate['exclusionPercentage'] / 100);
            $percentageAmount = floor($percentageAmount).substr($percentageAmount-floor($percentageAmount),1,2+1);
            $itemFilterPriced = new Types\ItemFilter();
            $itemFilterPriced->name = 'MinPrice';
            $itemFilterPriced->value[] = $percentageAmount;
        }

        $itemFilter = (isset($itemFilterPriced)) ? [$itemFilterForConditions, $itemFilterduplicate, $itemFilterStartDateFrom, $itemFilterPriced] : [$itemFilterForConditions, $itemFilterduplicate, $itemFilterStartDateFrom];

        $request->itemFilter = $itemFilter;

        /* 5) Request KEYWORD/MPN to filter Items Active Lists */
        $searchKeyword = (!empty($keyword_result['exclusions'])) ? $keyword_result['keyword'] . " -" . "(" . trim($keyword_result['exclusions']) . ")" : $keyword_result['keyword'];
        $request->keywords = $searchKeyword;

        /* 6) Request For sorting Items by Lowest To Highest Itemprice Price + Shipping Price */
        $request->sortOrder = 'PricePlusShippingLowest';

        /* 7) Request For Keyword Search Including the Item Descriptions,Title and Subtitle */
        $request->descriptionSearch = ($defaultStartDate['advancedExclusions'] == 1) ? true : false;

        $service = $this->index();

        /* Response returned by $request */
        $response = $this->apiresponse($request, $service, $id, $keyword_result['keyword'], $ignoredItems);

        return $response;

    } catch (\Exception $e) {
        echo json_encode(array('ack' => 'Failure', 'msg' => $e->getMessage()));
        exit;
    }
}


public function apiresponse($request, $service, $id, $keyword, $ignoredItems) {

            try {

                $response = $service->findItemsAdvanced($request);

                /* fetch all ignored Items */

                $newItems = array();
                $newOtherItems = array();
                $usedItems = array();
                $errors = "";

                if ($response->ack == "Success") {
                    $results = $response->searchResult->item;

                    ob_start();

                    foreach ($results as $item) {

                        if ($item->listingInfo->buyItNowAvailable == '1') {
                            $itemPrice = $item->listingInfo->buyItNowPrice->value;
                        } elseif (($item->listingInfo->listingType == 'Auction') && ($item->sellingStatus->currentPrice->value == 0)) {
                            $itemResponse = $this->getItemDetails($item->itemId);
                            $itemPrice = isset($itemResponse) ? $itemResponse : 0;
                        } else {
                            $itemPrice = $item->sellingStatus->currentPrice->value;
                        }

                        /* check for ignored items */
                        if (!in_array($item->itemId, $ignoredItems)) {

                            /* check for NEW Condition (conditionId = 1000) */
                            if ($item->condition->conditionId == '1000') {
                                $newItems[] = array(
                                    'keywordId' => $id,
                                    'conditionId' => (string) $item->condition->conditionId,
                                    'itemId' => $item->itemId,
                                    'title' => $item->title,
                                    'price' => $itemPrice,
                                    'itemURL' => $item->viewItemURL,
                                    'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
                                );
                            }
                            /* check for NEW OTHER Condition (conditionId = 1500) */ elseif ($item->condition->conditionId == '1500') {
                                $newOtherItems[] = array(
                                    'keywordId' => $id,
                                    'conditionId' => (string) $item->condition->conditionId,
                                    'itemId' => $item->itemId,
                                    'title' => $item->title,
                                    'price' => $itemPrice,
                                    'itemURL' => $item->viewItemURL,
                                    'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
                                );
                            }
                            /* check for USED Condition (conditionId = 3000) */ elseif ($item->condition->conditionId == '3000') {
                                $usedItems[] = array(
                                    'keywordId' => $id,
                                    'conditionId' => (string) $item->condition->conditionId,
                                    'itemId' => $item->itemId,
                                    'title' => $item->title,
                                    'price' => $itemPrice,
                                    'itemURL' => $item->viewItemURL,
                                    'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
                                );
                            }
                        }

                        ob_flush();
                    }

                    $newItems = array_slice($newItems, 0, 20);
                    $newOtherItems = array_slice($newOtherItems, 0, 20);
                    $usedItems = array_slice($usedItems, 0, 20);

                    $insertItems = array_merge($newItems, $newOtherItems, $usedItems);

                    /* clean up items */
                    Items::where('keywordId', $id)->delete();

                    /* save pulled items */
                    $items = Items::insert($insertItems);
                } elseif ($response->ack == "Failure") {
                    foreach ($response->errorMessage->error as $error) {
                        $errors .= "%s: %s\n\n" . $error->severity === Enums\ErrorSeverity::C_ERROR ? '<b>Error</b>&nbsp;' : '<b>Warning</b>&nbsp;' . $error->message;
                    }
                }

                /* save API log fot the keyword */
                $this->setapilog($request, $response, $keyword);

                if (!empty($errors)) {
                    echo json_encode(array('ack' => 'Failure', 'msg' => $errors));
                    exit;
                } else {
                    return true;
                }
            } catch (\Exception $e) {
                echo json_encode(array('ack' => 'Failure', 'msg' => $e->getMessage()));
                exit;
            }
        }

有关此的任何信息将不胜感激。谢谢！

Answer 1

@kinjal jethva

<强>问题：

您正在将数据库查询用于foreach循环。您正在将apirequest（）方法调用到foreach循环中。

当你的$ keywords数组有500条记录时。

您正在致电

    $keyword_result => 500 times. 

    $defaultStartDate = 500 times

您总是在一个http请求中调用1000个数据库查询。

<强>解决方案：

在apirequest（）函数中调用$ defaultStartDate是完全没用的。而是在外面调用它并将值作为参数传递。所以它会打电话一次。

$ keyword_result可以被称为更好的方式。

$defaultStartDate = Settings::select('defaultStartDate', 
'exclusionPercentage', 'advancedExclusions')->where('currentEnv', 1)->get()-
>first()->toArray();

$keywordValues = array_keys($keywords);

$keywordCollection = Keywords::whereIn('id', $keywordValues)-
>select('keyword', 'exclusions', 'price')->first()->toArray();

foreach ($keywordCollection as $keyword_result) {

$response = $this->apirequest($request, $keyword_result->id, 
$keyword_result, $defaultStartDate);

}

/**
  * Call Finding API for Active List items
  *
  * @return API Response
  */
function apirequest(Request $request, $id, $keyword_result, $defaultStartDate)
{
 // your code here.
}

这只是您可以使用的代码结构。

Answer 2

创建一个有工作Laravel Queues 将函数放在handle方法中。

$response = $this->apiRequest(...);

从说您的控制器上分派工作

foreach ($keywordCollection as $key => $keywordResult) {
   ProcessJob::dispatch($keywordResult)->delay($key * 2); 
}

通过这种方式，您可以实现指数退避，从而随着数组的增加，将增加等待下一个api请求的时间。

Laravel 504网关超时

2 个答案: