Question

我正在编写一个python程序，作为其中的一部分，我需要从google ngram viewer中提取数据。例如，搜索：

https://books.google.com/ngrams/graph?content=The+Godfather&year_start=1972-&year_end=2008&corpus=15&smoothing=3

我需要此标记中的图表值：

  var data = [{"ngram": "The Godfather", "type": "NGRAM", "timeseries": [1.4183381225052472e-07, 1.4025288521679614e-07, 1.5749316872870622e-07, 1.618123600824869e-07, 1.7873649125834034e-07, 1.8325580697364785e-07, 1.838378673418057e-07, 1.7964884234191102e-07, 1.7921279850595185e-07, 1.8174738970953642e-07, 1.7919142944070439e-07, 1.7377866307859741e-07, 1.6968103417574249e-07, 1.6785447241675554e-07, 1.698323818085815e-07, 1.7567729011196726e-07, 1.7198440259237812e-07, 1.6793204338227952e-07, 1.7446684604952418e-07, 1.8290416343396438e-07, 1.8512590876136009e-07, 1.8604621183320497e-07, 1.9866517269357636e-07, 1.8994029866397405e-07, 2.0815326909736802e-07, 2.2254876138764042e-07, 2.2457939508058189e-07, 2.5743728875633156e-07, 2.7453169236326046e-07, 2.7866381507075335e-07, 3.0070588609630378e-07, 3.0204388273042629e-07, 2.9686912585345583e-07, 2.9399397760698776e-07, 2.7703356645740013e-07, 2.7225316614476467e-07, 2.6805425434872632e-07], "parent": ""}];
  if (data.length > 0) {
    ngrams.drawD3Chart(data, 1972, 2008, 1.0, "main");
  }

我尝试使用lxml将HTML转换为树，然后使用从网站复制的xpath，如下所示：

page = requests.get('https://books.google.com/ngrams/graph?content=The+Godfather&year_start=1972-&year_end=2008&corpus=15&smoothing=3')
tree = html.fromstring(page.content)

nGramData = tree.xpath('//*[@id="container"]/script[11]/text()')

然而，如果我然后尝试打印nGramData，我没有结果，我做错了什么或有更好的方法来做到这一点？

Answer 1

使用以下工作代码：

>>> page = requests.get('https://books.google.com/ngrams/graph?content=The+Godfather&year_start=1972-&year_end=2008&corpus=15&smoothing=3')
>>> tree = html.fromstring(page.content)
>>> nGramData = tree.xpath('//*[@id="container"]/script')
>>> nGramData[6].text_content()
'\n  var data = [{"ngram": "The Godfather", "type": "NGRAM", "timeseries": [1.4183381225052472e-07, 1.4025288521679614e-07, 1.5749316872870622e-07, 1.618123600824869e-07, 1.7873649125834034e-07, 1.8325580697364785e-07, 1.838378673418057e-07, 1.7964884234191102e-07, 1.7921279850595185e-07, 1.8174738970953642e-07, 1.7919142944070439e-07, 1.7377866307859741e-07, 1.6968103417574249e-07, 1.6785447241675554e-07, 1.698323818085815e-07, 1.7567729011196726e-07, 1.7198440259237812e-07, 1.6793204338227952e-07, 1.7446684604952418e-07, 1.8290416343396438e-07, 1.8512590876136009e-07, 1.8604621183320497e-07, 1.9866517269357636e-07, 1.8994029866397405e-07, 2.0815326909736802e-07, 2.2254876138764042e-07, 2.2457939508058189e-07, 2.5743728875633156e-07, 2.7453169236326046e-07, 2.7866381507075335e-07, 3.0070588609630378e-07, 3.0204388273042629e-07, 2.9686912585345583e-07, 2.9399397760698776e-07, 2.7703356645740013e-07, 2.7225316614476467e-07, 2.6805425434872632e-07], "parent": ""}];\n  if (data.length > 0) {\n    ngrams.drawD3Chart(data, 1972, 2008, 1.0, "main");\n  }\n'

Answer 2

你也可以使用BeautifulSoup和Json：

打开并阅读链接（可能是Python 3中的urllib2）：

link=urllib.urlopen("https://books.google.com/ngrams/graph?content=The+Godfather&year_start=1972-&year_end=2008&corpus=15&smoothing=3&direct_url=t1%3B%2CThe%20Godfather%3B%2Cc0")
content=link.read()

从bs4导入BS后将其变为BeautifulSoup元素：

soup=BeautifulSoup(content, "html.parser")
extract=soup.select('script[type="text/javascript"]')[4].string

使用正则表达式来读取JSON（需要导入），然后只提取时间序列：

data=re.findall('(\[.*\])', extract)

t=json.loads(data[0])
result= t[0]['timeseries']

无法使用xpath，lxml

2 个答案: