Question

我正在尝试获取有关杂货，标题，图片，价格等的信息。

所有其他网址都运行良好，cUrl响应完全符合预期。

我遇到的问题是，网址包含重音拉丁/非标准网址/非英文字符，例如ü或è。

我已经尝试了所有我能想到的东西，但可能有一个我想念的简单解决方案：

stringtest.php?url=http://www.sainsburys.co.uk/shop/gb/groceries/desserts/g%C3%BC-lemon-pots-3x45g
stringtest.php?url=http://www.sainsburys.co.uk/shop/gb/groceries/desserts/gü-lemon-pots-3x45g
stringtest.php?url=http%3A%2F%2Fwww.sainsburys.co.uk%2Fshop%2Fgb%2Fgroceries%2Fdesserts%2Fg%C3%BC-lemon-pots-3x45g

这是我测试cUrl的代码：

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
  </head>
  <body>
<?php
  $url = $_GET['url'];

  echo curlUrl($url);

  function curlUrl($url){
    $ch = curl_init();
    $timeout = 5;
    $cookie_file = "/tmp/cookie/cookie1.txt";
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    $html = curl_exec($ch);
    curl_close($ch);

    return $html;
  }

?>
  <form action="stringtest.php" method="get" id="process">
    <input type="text" name="url" placeholder="Url" autofocus>
    <input type="submit">
  </form>
  </body>
</html>

我从cUrl得到的结果是Sainsburys的404页面声称找不到该页面。从网址栏复制http://www.sainsburys.co.uk/shop/gb/groceries/desserts /gü-lemon-pots-3x45g会导致按照预期复制ü（％C3％BC）的URL编码版本。在浏览器中输入URL时，ü和％C3％BC都可用于到达实际的产品页面，那么为什么Sainsburys会在cUrl'd时返回404？

我尝试使用浏览器使用的确切标头等urldecode()等各种内容，但无济于事。

Answer 1

似乎是塞恩斯伯里网站本身的一个问题。

如果您没有发送有效的Cookie，服务器会返回404。

你试过重装吗？

我试过

stringtest.php?url=http://www.sainsburys.co.uk/shop/gb/groceries/desserts/gü-chocolate-ganache-pots-3x45g

并使用了有效的Cookie。

Answer 2

如果您尝试：

wget http://www.sainsburys.co.uk/shop/gb/groceries/desserts/g%C3%BC-lemon-pots-3x45g

回复是：

http://www.sainsburys.co.uk/shop/gb/groceries/bakery
Resolving www.sainsburys.co.uk (www.sainsburys.co.uk)... 109.94.142.1
Connecting to www.sainsburys.co.uk (www.sainsburys.co.uk)|109.94.142.1|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.sainsburys.co.uk/webapp/wcs/stores/servlet/gb/groceries/bakery?langId=44&storeId=10151&krypto=xbYM3SJja%2F1mDOxJIVlKl9vZN6zjdlTL4MSiHOKiUMQoum9OkLwoTv6wj27CjUXwqM4%2BsteXag0O%0AQOWiHuS8onFdmoVLWlJyZ7hXaMhcMW9MIMMAsnPdWTPEzSEnOP5a&ddkey=http:AjaxAutoCompleteDisplayView [following]
--2014-10-07 11:56:11-- http://www.sainsburys.co.uk/webapp/wcs/stores/servlet/gb/groceries/bakery?langId=44&storeId=10151&krypto=xbYM3SJja%2F1mDOxJIVlKl9vZN6zjdlTL4MSiHOKiUMQoum9OkLwoTv6wj27CjUXwqM4%2BsteXag0O%0AQOWiHuS8onFdmoVLWlJyZ7hXaMhcMW9MIMMAsnPdWTPEzSEnOP5a&ddkey=http:AjaxAutoCompleteDisplayView
Reusing existing connection to www.sainsburys.co.uk:80.
HTTP request sent, awaiting response... 200 OK

要遵循curl中的重定向，请使用-L标志：

curl -L http://www.sainsburys.co.uk/shop/gb/groceries/desserts/g%C3%BC-lemon-pots-3x45g

cUrl获取包含“ü”U + 00FC％c3％bc的url的内容

2 个答案: