Question

我正在使用Python中的scrapy从网站上抓取数据。

所需数据位于脚本标记中，如下所示：

<script type="text/javascript">
getDetailsfrmBean("storePg","564","Berwyn, IL","7180 W CERMAK RD.","SPACE A1","","BERWYN","IL","US","60402","(708) 788-5097","{Monday-Saturday=10-9,sunday=11-6}","41.8507029","-87.8033709");
</script>

我可以使用xpath如下获取此内容：

item['lat'] = tree.xpath('//script[@type="text/javascript"]/text()'.extract()[0].encode('utf-8')
item['long'] = tree.xpath('//script[@type="text/javascript"]/text()'.extract()[0].encode('utf-8')

然后

item['lat'] = 'getDetailsfrmBean("storePg","564","Berwyn, IL","7180 W CERMAK RD.","SPACE A1","","BERWYN","IL","US","60402","(708) 788-5097","{Monday-Saturday=10-9,sunday=11-6}","41.8507029","-87.8033709");'

item['long'] = 'getDetailsfrmBean("storePg","564","Berwyn, IL","7180 W CERMAK RD.","SPACE A1","","BERWYN","IL","US","60402","(708) 788-5097","{Monday-Saturday=10-9,sunday=11-6}","41.8507029","-87.8033709");'

但是我该如何解析这些内容以便

item['lat'] is equal to "41.8507029"
item['long'] is equal to "-87.8033709"
item['city'] is equal to "BERWYN"
item['state'] is equal to "IL"

请问有什么建议可以解决这个问题。

Answer 1

由于此调用也是有效的Python语法，因此我们可以使用ast模块。加上参数都是字符串文字，这使事情变得更简单。

import ast

line = 'getDetailsfrmBean("storePg","564","Berwyn, IL","7180 W CERMAK RD.","SPACE A1","","BERWYN","IL","US","60402","(708) 788-5097","{Monday-Saturday=10-9,sunday=11-6}","41.8507029","-87.8033709");'

print([arg.s for arg in ast.parse(line).body[0].value.args])

输出：

['storePg', '564', 'Berwyn, IL', '7180 W CERMAK RD.', 'SPACE A1', '', 'BERWYN', 'IL', 'US', '60402', '(708) 788-5097', '{Monday-Saturday=10-9,sunday=11-6}', '41.8507029', '-87.8033709']

说明：

print([arg.s           # value of string literal
       for arg in
       ast.parse(line)
      .body            # module (list of statements)
       [0]             # first statement (an Expr node)
      .value           # expression (a Call)
      .args            # arguments to function call
       ])

Answer 2

尝试使用Col1 Col2 Sub1 Sub2 SubX SubY 4 N E 3 Z 5 N F 3 Z

re

应产生以下输出：

import re
temp_string = 'getDetailsfrmBean("storePg","564","Berwyn, IL","7180 W CERMAK RD.","SPACE A1","","BERWYN","IL","US","60402","(708) 788-5097","{Monday-Saturday=10-9,sunday=11-6}","41.8507029","-87.8033709");'
split_list = filter(None, re.split("[, \-!?:\"]+",temp_string))
print split_list

从这里的答案中摘录：https://stackoverflow.com/a/23720594/5907969

Answer 3

您可以使用一个简单的正则表达式来提取仅用逗号分隔的带引号的字符串部分：

import re

line = 'getDetailsfrmBean("storePg","564","Berwyn, IL","7180 W CERMAK RD.","SPACE A1","","BERWYN","IL","US","60402","(708) 788-5097","{Monday-Saturday=10-9,sunday=11-6}","41.8507029","-87.8033709");'

args_string = re.match(r'getDetailsfrmBean\((.+)\);$', line.strip()).group(1)
print(args_string)

输出：

"storePg","564","Berwyn, IL","7180 W CERMAK RD.","SPACE A1","","BERWYN","IL","US","60402","(708) 788-5097","{Monday-Saturday=10-9,sunday=11-6}","41.8507029","-87.8033709"

然后有多种方法可以从这种数据中解析字符串列表：

import ast
import json
import csv

args_array = '[%s]' % args_string

assert (json.loads(args_array)
        == ast.literal_eval(args_array)
        == next(csv.reader([args_string]))
        == ['storePg', '564', 'Berwyn, IL', '7180 W CERMAK RD.', 'SPACE A1', '', 'BERWYN', 'IL', 'US', '60402',
            '(708) 788-5097', '{Monday-Saturday=10-9,sunday=11-6}', '41.8507029', '-87.8033709'])

Answer 4

正要编写一个包含ast和re + json的方法的答案-但是@Alex Hall使用ast方法的速度更快，这是imho更喜欢-但是另一种方法将涉及一个简单的正则表达式和json模块，该模块还为您提供一个列表，并且可以扫描同一字符串中的多个函数调用：

import re
import json

fn_cutter = re.compile("getDetailsfrmBean\((.+?)\);")

for key in item:
  for i, match in enumerate(fn_cutter.findall(item[key])):
    print(key, i, ':', json.loads("[" + match + "]"))

Online demo here

在将JSON对象转换为Python结构并捕获相同值内的多个方法调用时，这将节省您一些时间-但它肯定无法处理JS方法中包含的anotherMethod(args)或...value电话。

如何从JavaScript函数调用中提取数据，例如Python中的语法

4 个答案: