Question

我正在尝试使用python创建脚本，在其中执行多处理以从网页中获取不同用户的链接。尽管用户链接在其登录页面中可用，但我试图从他们的内页中找出他们。但是，当我在yield函数中使用get_links()并在print()中使用get_target_link()时，可以获得预期的结果。

我的问题是：如何在两个功能中使用yield来达到相同的目的？

我尝试过：

import requests
import concurrent.futures
from urllib.parse import urljoin
from bs4 import BeautifulSoup

def get_links(url):
    res = requests.get(url)
    soup = BeautifulSoup(res.text,"lxml")
    for item in soup.select(".summary .question-hyperlink"):
        yield urljoin(base,item.get("href"))

def get_target_link(targeturl):
    res = requests.get(targeturl)
    soup = BeautifulSoup(res.text,"lxml")
    name_link = urljoin(base,soup.select_one(".user-details > a").get("href"))
    yield name_link

if __name__ == '__main__':
    base = 'https://stackoverflow.com'
    mlink = "https://stackoverflow.com/questions/tagged/web-scraping"

    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        future_to_url = {executor.submit(get_target_link, url): url for url in get_links(mlink)}
        concurrent.futures.as_completed(future_to_url)

上面的脚本根本没有结果。

Answer 1

您的初始方法存在一些导致“完全没有结果” 的问题：

BeautifulSoup(res.text,"lxml")-将解析器更改为html.parser（您正在解析html网页）
将函数get_target_link用作生成器没有任何好处，因为它不应该成为迭代器，并且已经可以立即生成单个结果。
concurrent.futures.as_completed返回Future实例的迭代器，而不是最终结果

更正后的方法如下：

import requests
import concurrent.futures as futures
from urllib.parse import urljoin
from bs4 import BeautifulSoup


def get_links(url):
    res = requests.get(url)
    soup = BeautifulSoup(res.text, "html.parser")
    for link in soup.select(".summary .question-hyperlink"):
        yield urljoin(base, link.get("href"))


def get_target_link(target_url):
    res = requests.get(target_url)
    soup = BeautifulSoup(res.text, "html.parser")
    name_link = urljoin(base, soup.select_one(".user-details a").get("href"))
    return name_link


if __name__ == '__main__':
    base = 'https://stackoverflow.com'
    mlink = "https://stackoverflow.com/questions/tagged/web-scraping"

    with futures.ThreadPoolExecutor(max_workers=10) as executor:
        future_to_url = {executor.submit(get_target_link, url): url for url in get_links(mlink)}
        for future in futures.as_completed(future_to_url):
            url = future_to_url[future]
            try:
                data = future.result()
            except Exception as ex:
                print(f'Failed to extract user details from url: {url}')
            else:
                print(data)

输出：

https://stackoverflow.com/users/10035985/andrej-kesely
https://stackoverflow.com/users/11520568/rachit-gupta
https://stackoverflow.com/users/10568531/robots-txt
https://stackoverflow.com/users/10664939/logan-anderson
https://stackoverflow.com/users/688393/c%c3%a9sar
https://stackoverflow.com/users/903061/gregor
https://stackoverflow.com/users/9950503/saraherceg
https://stackoverflow.com/users/80851/gmile
https://stackoverflow.com/users/11793150/saurabh-rawat
https://stackoverflow.com/users/11793061/xzatar
https://stackoverflow.com/users/11759292/rachel9866
https://stackoverflow.com/users/2628114/user2628114
https://stackoverflow.com/users/9810397/bart
https://stackoverflow.com/users/838355/ir2pid
https://stackoverflow.com/users/10629482/shreya
https://stackoverflow.com/users/11669928/thor-is
https://stackoverflow.com/users/7660288/acro2142
https://stackoverflow.com/users/3342430/freddiev4
https://stackoverflow.com/users/11767045/k-%c3%96sterlund
https://stackoverflow.com/users/11781213/mohamed-shire
https://stackoverflow.com/users/5412619/a-nonymous
https://stackoverflow.com/users/4354477/forcebru
https://stackoverflow.com/users/10568531/robots-txt
https://stackoverflow.com/users/6622587/eyllanesc
https://stackoverflow.com/users/10568531/robots-txt
https://stackoverflow.com/users/3273177/casabonita
https://stackoverflow.com/users/1540328/dipesh-parmar
https://stackoverflow.com/users/6231957/perth
https://stackoverflow.com/users/11400264/workin-4weekend
https://stackoverflow.com/users/1000551/vadim-kotov
https://stackoverflow.com/users/331508/brock-adams
https://stackoverflow.com/users/11300154/helloworld1990
https://stackoverflow.com/users/11786268/mohsine-jirou
https://stackoverflow.com/users/9707561/fatima-tt
https://stackoverflow.com/users/11759292/rachel9866
https://stackoverflow.com/users/6622587/eyllanesc
https://stackoverflow.com/users/11485683/titan
https://stackoverflow.com/users/11593630/supek
https://stackoverflow.com/users/11717116/raja-kishore-patnayakuni
https://stackoverflow.com/users/975887/madushan
https://stackoverflow.com/users/10568531/robots-txt
https://stackoverflow.com/users/283366/phil
https://stackoverflow.com/users/8677101/bpdesilva
https://stackoverflow.com/users/3504096/programmerper
https://stackoverflow.com/users/6303216/akhlaq-ahmed
https://stackoverflow.com/users/11457578/sh-student
https://stackoverflow.com/users/11783947/alexis-cruz-cruz
https://stackoverflow.com/users/3579212/adnanmuttaleb
https://stackoverflow.com/users/1060350/anony-mousse
https://stackoverflow.com/users/8100732/khadija-saeed

在进行多处理时使用yield难以获取结果

1 个答案: