Question

所以我点击按钮时调用的功能，如下所示

var min_news_id = "68feb985-1d08-4f5d-8855-cb35ae6c3e93-1";
function loadMoreNews(){
  $("#load-more-btn").hide();
  $("#load-more-gif").show();
  $.post("/en/ajax/more_news",{'category':'','news_offset':min_news_id},function(data){
      data = JSON.parse(data);
      min_news_id = data.min_news_id||min_news_id;
      $(".card-stack").append(data.html);
  })
  .fail(function(){alert("Error : unable to load more news");})
  .always(function(){$("#load-more-btn").show();$("#load-more-gif").hide();});
}
jQuery.scrollDepth();

现在我对javascript没有多少经验，但我假设它从“en / ajax / more_news”中的某种api返回一些json数据。

有没有办法可以直接调用这个api并从我的python脚本中获取json数据。如果是，怎么做？

如果不是我如何抓取正在生成的内容？

Answer 1

您需要将脚本中显示的新闻ID发布到 https://www.inshorts.com/en/ajax/more_news ，这是使用requests的示例：

from bs4 import BeautifulSoup
import requests
import re

# pattern to extract min_news_id
patt = re.compile('var min_news_id\s+=\s+"(.*?)"')

with requests.Session() as s:
    soup = BeautifulSoup(s.get("https://www.inshorts.com/en/read").content)
    new_id_scr = soup.find("script", text=re.compile("var\s+min_news_id"))
    print(new_id_scr.text)
    news_id = patt.search(new_id_scr.text).group()
    js = s.post("https://www.inshorts.com/en/ajax/more_news", data={"news_offset":news_id})
    print(js.json())

js为您提供所有html，您只需访问js["html"]。

Answer 2

以下脚本将自动循环遍历 inshort.com 中的所有页面

from bs4 import BeautifulSoup
from newspaper import Article
import requests
import sys
import re
import json

patt = re.compile('var min_news_id\s+=\s+"(.*?)"')
i = 0
while(1):
    with requests.Session() as s:
        if(i==0):soup = BeautifulSoup(s.get("https://www.inshorts.com/en/read").content,"lxml")
  new_id_scr = soup.find("script", text=re.compile("var\s+min_news_id"))
   news_id = patt.search(new_id_scr.text).group(1)

    js = s.post("https://www.inshorts.com/en/ajax/more_news", data={"news_offset":news_id})
    jsn = json.dumps(js.json())
    jsonToPython = json.loads(jsn)
    news_id = jsonToPython["min_news_id"]
    data = jsonToPython["html"]
    i += 1
    soup = BeautifulSoup(data, "lxml")
    for tag in soup.find_all("div", {"class":"news-card"}):
        main_text = tag.find("div", {"itemprop":"articleBody"})
        summ_text = main_text.text
        summ_text = summ_text.replace("\n", " ")
        result = tag.find("a", {"class":"source"})
        art_url = result.get('href') 
        if 'www.youtube.com' in art_url:
            print("Nothing")
        else:
            art_url = art_url[:-1]
            #print("Hello", art_url)
            article = Article(art_url)
            article.download()
            if article.is_downloaded:
                article.parse()
                article_text = article.text
                article_text = article_text.replace("\n", " ")

                print(article_text+"\n")
                print(summ_text+"\n")

它提供来自新闻频道的 inshort.com 和完整新闻的摘要。

使用python刮擦AJAX加载的内容？

2 个答案: