Question

我正在开发一个使用Curl和PHP的小项目来刮取Google学术搜索的结果。它在我的开发模式下工作正常，但是当我在生产模式下尝试时，某些东西不起作用且没有结果......

这是我的代码：

// SCRAPING GOOGLE SCHOLAR
    if (isset($_POST['google'])){
        $googleURL = 'http://scholar.google.com/scholar?hl=fr&q=' . $url_subject;

        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $googleURL);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
        curl_setopt($ch, CURLOPT_USERAGENT, $random->random_user_agent());
        $result = curl_exec ($ch);
        curl_close($ch);

        $html = $this->container->get('simple_html_dom');
        $html->load($result);

感谢您的帮助

Answer 1

Google学术搜索对于抓取内容感到皱眉。这违反了他们的服务条款。命令行curl有助于解决此类问题：

$ curl -vv https://scholar.google.com/scholar?hl=en&q=neurotransmitters
> GET /scholar?hl=en HTTP/1.1
> User-Agent: curl/7.35.0
> Host: scholar.google.com
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
...
<html>...<title>Sorry...</title></head><body>
<h1>We're sorry...</h1>
<p>... but your computer or network may be sending automated queries.
To protect our users, we can't process your request right now.</p>
<div style="margin-left: 4em;">See
<a href="https://support.google.com/websearch/answer/86640">Google Help</a>
for more information.</div>
</body></html>

谷歌的卷曲刮擦在生产中不起作用

1 个答案: