Question

我需要从首先要求登录的网站中删除一些数据，并且我确实使用curl进行登录，这是我的登录代码：

$login = 'https://example.com/login';
$ch = curl_init($login);

curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_COOKIEJAR => COOKIE_FILE,
    CURLOPT_COOKIEFILE => COOKIE_FILE
]);

$response = curl_exec($ch);     
$re = '/<input type="hidden" name="csrf" value="(.*?)" \/>/m';

preg_match_all($re, $response, $matches, PREG_SET_ORDER, 0);

$arr = array(
    'email' => 'email@example.com',
    'password' => 'Password123',
    'csrf' => $matches[0][1]
);

curl_setopt_array($ch, [
    CURLOPT_URL => $login,
    CURLOPT_USERAGENT => 'Mozilla/5.0',
    CURLOPT_POST => true,
    CURLOPT_POSTFIELDS => http_build_query($arr),
    CURLOPT_COOKIEJAR => COOKIE_FILE,
    CURLOPT_COOKIEFILE => COOKIE_FILE,
    CURLOPT_FOLLOWLOCATION => true
]);

curl_exec($ch);

现在，登录后我必须抓取70-100页，我设法通过使用foreach循环来做到这一点，但它要花一辈子。这是我的代码：

$arr = [
    [
        'id' => '1',
        'csrf' => $matches[0][1] //same csrf as in login
    ],[
        'id' => '2',
        'csrf' => $matches[0][1] //same csrf as in login
    ],[
        ...
    ],[
        'id' => '100',
        'csrf' => $matches[0][1] //same csrf as in login
    ]
];

foreach($arr as $v){
    curl_setopt_array($ch,[
        CURLOPT_URL => 'https://example.com/submit',
        CURLOPT_USERAGENT => 'Mozilla/5.0',
        CURLOPT_POST => true,
        CURLOPT_POSTFIELDS => http_build_query($v),
        CURLOPT_FOLLOWLOCATION => true
    ]);

    $return = curl_exec($ch);
    $info = curl_getinfo($ch);

    //do something with the returned data   
}

但是，如果我尝试使用multi_curl，则无法保持登录状态，并且受到405 http_code的欢迎。

是否存在使用curl进行登录并使用multi进行报废的解决方案？谢谢！

编辑这是我用于multi_curl的代码（在stackoverflow上找到了它）：

function multiRequest($data, $options = array()) {

  // array of curl handles
  $curly = array();
  // data to be returned
  $result = array();

  // multi handle
  $mh = curl_multi_init();

  // loop through $data and create curl handles
  // then add them to the multi-handle
  foreach ($data as $id => $d) {

    $curly[$id] = curl_init();

    $url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
    curl_setopt($curly[$id], CURLOPT_URL, $url);
    curl_setopt($curly[$id], CURLOPT_USERAGENT, 'Mozilla/5.0');
    curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);

    // post?
    if (is_array($d)) {
      if (!empty($d['post'])) {
        curl_setopt($curly[$id], CURLOPT_POST, true);
        curl_setopt($curly[$id], CURLOPT_POSTFIELDS, http_build_query($d['post']));
        curl_setopt($curly[$id], CURLOPT_FOLLOWLOCATION, true);
      }
    }

    // extra options?
    if (!empty($options)) {
      curl_setopt_array($curly[$id], $options);
    }

    curl_multi_add_handle($mh, $curly[$id]);
  }

  // execute the handles
  $running = null;
  do {
    curl_multi_exec($mh, $running);
  } while($running > 0);


  // get content and remove handles
  foreach($curly as $id => $c) {
    $result[$id] = curl_multi_getcontent($c);
    curl_multi_remove_handle($mh, $c);
  }

  // all done
  curl_multi_close($mh);

  return $result;
}

PHP curl登录一次，然后使用multi_curl抓取网站

0 个答案: