Question

我试图使用LXML＆amp;请求从网站创建数据表。我需要标记中的文本和标记中包含的文本。这是HTML：

<div class="houses">
    <input type="hidden" class="houseNumber" value="107">
    <input type="hidden" class="houseState" value="MT">
    <input type="hidden" class="houseStatus" value="Occupied">
<div class="houseInfo">
    <div class="houseCity">Helena</div>
    <div class="houseArea">Helena Valley</div>
</div>
</div>
<div class="houses">
    <input type="hidden" class="houseNumber" value="237">
    <input type="hidden" class="houseState" value="MT">
    <input type="hidden" class="houseStatus" value="Occupied">
<div class="houseInfo">
    <div class="houseCity">East Helena</div>
    <div class="houseArea">Helena Valley</div>
</div>
</div>
<div class="houses">
    <input type="hidden" class="houseNumber" value="104">
    <input type="hidden" class="houseState" value="MT">
    <input type="hidden" class="houseStatus" value="Vacant">
<div class="houseInfo">
    <div class="houseCity">Helena</div>
    <div class="houseArea">Helena Valley</div>
</div>
</div>

由此，我想创建一个这样的表：

['107', 'MT', 'Occupied', 'Helena', 'Helena Valley']
['237', 'MT', 'Occupied', 'East Helena', 'Helena Valley']
['104', 'MT', 'Vacant', 'Helena', 'Helena Valley']

使用请求＆amp; LXML，我试图遍历div class="houses"以获得我需要的东西，但每次我尝试打印这些值时，都会打印出来：

['107', '237', '104']
['MT', 'MT', 'MT']
['Occupied', 'Occupied', 'Vacant']
['Helena', 'East Helena', 'Helena']
['Helena Valley', 'Helena Valley', 'Helena Valley']
['107', '237', '104']
['MT', 'MT', 'MT']
['Occupied', 'Occupied', 'Vacant']
['Helena', 'East Helena', 'Helena']
['Helena Valley', 'Helena Valley', 'Helena Valley']
['107', '237', '104']
['MT', 'MT', 'MT']
['Occupied', 'Occupied', 'Vacant']
['Helena', 'East Helena', 'Helena']
['Helena Valley', 'Helena Valley', 'Helena Valley']

这是我的部分代码：

link = "example.com"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
response = requests.get(link, headers=headers, allow_redirects=False) 
sourceCode = response.content

htmlElem = html.document_fromstring(sourceCode)
houses = htmlElem.find_class('houses')
for house in houses:
    houseNumber = house.xpath('//input[@class="houseNumber"]/@value')
    houseState = house.xpath('//input[@class="houseState"]/@value')
    houseStatus = house.xpath('//input[@class="houseStatus"]/@value')

如何如上图所示捕获表格中的数据？我可以用不同的方式遍历房屋对象吗？

更新：@efirvida我已将代码修改为以下内容：

link = "example.com"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
response = requests.get(link, headers=headers, allow_redirects=False) 
sourceCode = response.content

htmlElem = html.document_fromstring(sourceCode)
houses = htmlElem.find_class('houses')
houseNumber = []
houseState = []
houseStatus = []

for house in houses:
    houseNumber.append(house.xpath('//input[@class="houseNumber"]/@value'))
    print(houseNumber)
    houseState.append(house.xpath('//input[@class="houseState"]/@value'))
    houseStatus.append(house.xpath('//input[@class="houseStatus"]/@value'))

data = map(list, zip(*[houseNumber,houseState,houseStatus]))

当我这样做时，会打印以下内容：

[['107', '237', '104']]
[['107', '237', '104'], ['107', '237', '104']]
[['107', '237', '104']], ['107', '237', '104'], ['107', '237', '104']]

Answer 1

尝试转置结果，请参阅this thread以了解我的代码。

# create a list with elements
houseNumber = []
houseState = []
houseStatus = []

# append each element to it's list
for house in houses:
    houseNumber.append(house.xpath('//input[@class="houseNumber"]/@value'))
    houseState.append(house.xpath('//input[@class="houseState"]/@value'))
    houseStatus.append(house.xpath('//input[@class="houseStatus"]/@value'))


# transpose the lists, and turn into a list of list
data = map(list, zip(*[houseNumber,houseState,houseStatus]))

>>> list(data)
#[['107', 'MT', 'Occupied'], ['237', 'MT', 'Occupied'], ['104', 'MT', 'Vacant']]

如果您可以将其用作元组，只需删除地图

即可

#just transpose
data = zip(*[houseNumber,houseState,houseStatus])

>>> list(data)
#[('107', 'MT', 'Occupied'), ('237', 'MT', 'Occupied'), ('104', 'MT', 'Vacant') ]

Python：如何使用LXML / Requests迭代HTML元素对象？

1 个答案: