Question

我正在建立一个数据库（Pandas Dataframe），以存储新闻报道（过去一周的报道）的新闻网络链接以获取公司列表。我已经编写了python代码，但是该代码执行了一段时间，有时没有执行，也不会产生任何错误。由于它不会产生任何日志或错误，因此我很难理解该问题的背景。

由于我使用的是Jupyter笔记本，并且尝试了Sypder等其他应用程序，因此我尝试从浏览器中删除缓存。我在Jupyter笔记本和其他应用程序中遇到相同的问题


links_output=[]

class Newspapr:
    def __init__(self,term):
        self.term=term
        self.url='https://www.google.com/search?q={0}&safe=active&tbs=qdr:w,sdb:1&tbm=nws&source=lnt&dpr=1'.format(self.term)

    def NewsArticlerun(self):
        response=requests.get(self.url)
        soup=BeautifulSoup(response.text,'html.parser')
        links=soup.select(".r a")

        numOpen = min(5, len(links))
        for i in range(numOpen):
            response_links = "https://www.google.com" + links[i].get("href")
            print(response_links)
            links_output.append({"Weblink":response_links})
        pd.DataFrame.from_dict(links_output)



list_of_companies=["Wipro","Reliance","icici bank","vedanta", "DHFL","yesbank","tata motors","tata steel","IL&FS","Jet airways","apollo tyres","ashok leyland","Larson & Turbo","Mindtree","Infosys","TCS","AxisBank","Mahindra & Mahindra"]

for i in list_of_companies:
    comp_list = str('"'+ i + '"')
    call_code=Newspapr(comp_list)
    call_code.NewsArticlerun()

我希望将网络链接打印为熊猫数据框

Answer 1

首先，函数的命名约定不正确，我已对其进行了更改。

您未在函数中返回任何内容。 return。

def newsArticlerun(self):
    response=requests.get(self.url)
    soup=BeautifulSoup(response.text,'html.parser')
    links=soup.select(".r a")

    numOpen = min(5, len(links))
    for i in range(numOpen):
        response_links = "https://www.google.com" + links[i].get("href")
        print(response_links)
        links_output.append({"Weblink":response_links})
    return pd.DataFrame.from_dict(links_output) # this will return your df

要打印结果，请添加print

for i in list_of_companies:
    comp_list = str('"'+ i + '"')
    call_code=Newspapr(comp_list)
    print(call_code.NewsArticlerun()) # here

注意：由于这个原因，您没有得到结果。

<div style="font-size:13px;">
<b>About this page</b><br/><br/>Our systems have detected unusual traffic from your computer network.  This page checks to see if it's really you sending the requests, and not a robot.  <a href="#" onclick="document.getElementById('infoDiv').style.display='block';">Why did this happen?</a><br/><br/>
<div id="infoDiv" style="display:none; background-color:#eee; padding:10px; margin:0 0 15px 0; line-height:1.4em;">
This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the <a href="//www.google.com/policies/terms/">Terms of Service</a>. The block will expire shortly after those requests stop.  In the meantime, solving the above CAPTCHA will let you continue to use our services.<br/><br/>This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests.  If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible.  <a href="//support.google.com/websearch/answer/86640">Learn more</a><br/><br/>Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly.
</div>

Answer 2

我想您正在触发Google搜索的反垃圾邮件对策。为您的请求添加延迟可能会有所帮助。

编辑：正如Yu Chen所说的here，使用官方的Google API https://developers.google.com/custom-search/docs/tutorial/creatingcse

Edit2 ：查看此帖子以获取深入的答案：Programmatically searching google in Python using custom search

Edit3 ：为使其更有用，您应在标题前加上“ Google搜索”一词，以阐明问题的性质

Python代码有时会被执行，有时却不会

2 个答案: