Question

我需要进行一些房地产市场研究，因此我需要新房的价格和其他价值。

因此，我的想法是访问获得信息的网站。转到主搜索页面，然后抓取所有RealEstateID，这些ID会将我直接带到每个房子的单独页面，我可以在其中提取所需的信息。我有两个问题：首先，该网站有大约60个ID，但我只有大约20个ID。其次，Excel电子表格中的格式将列标题放回每个条目的上方，并且每个条目仅在一个单元格中，而不是在不同的列中。

如果有人可以向我解释如何解决这些问题并告诉我做错了什么，我将很高兴听到这个消息

import requests
import json
from bs4 import BeautifulSoup as bs
import datetime as dt 
import os
import pandas as pd 
import pandas_datareader.data as web
import re
import time
import urllib.request
from urllib.request import urlopen
import csv

res = requests.get('https://www.immobilienscout24.de/Suche/S-T/Wohnung-Kauf/Nordrhein-Westfalen/Duesseldorf/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/true?enteredFrom=result_list') #This is the main search site where I get the IDs
soup = bs(res.content, 'lxml')
r = re.compile(r'resultListModel:(.*)')
data = soup.find('script', text=r).text
script = r.findall(data)[0].rstrip(',')
results = json.loads(script)
ids = [item['@id'] for item in results['searchResponseModel']['resultlist.resultlist']['resultlistEntries'][0]['resultlistEntry']] #This should get me the IDs and put it into a list so i can use it later
print(ids) #My test that I get all the IDs and not just a few

data = json.dumps(ids)
houseinfo = {} 

csvData = [['id','purchasePrice','Spacesize','District','Flattyp','Rooms']] #Dataformat for the Excel-Spreadsheet colums later

def get_house_info (House):                                                 #this is the function that inserts the IDs from the List into the URLs and than scraps the Values that I need into a new list
    for id in ids:
        try:
            sourceCode = urllib.request.urlopen('https://www.immobilienscout24.de/expose/' + str(id)).read()
            purchasePrice = str(sourceCode).split('"purchasePrice":')[1].split(',"geoCode"')[0]
            Spacesize = str(sourceCode).split('"area":')[1].split('},"details"')[0]
            District = str(sourceCode).split('"quarter":')[1].split('},')[0]
            Flattyp = str(sourceCode).split('"is24qa-typ grid-item three-fifths">')[1].split('</dd> </dl> <dl class')[0]
            Rooms = str(sourceCode).split('is24qa-zimmer grid-item three-fifths"> ')[1].split(' </dd> </dl> <dl class=')[0]

            with open('foo.csv', 'a') as csvfile:                           #appends the values to the file
                cols = ['id', 'price', 'size', 'district', 'flattyp', 'rooms']
                dict_result = {'id': id, 'price': purchasePrice, 'size': Spacesize, 'district': District, 'flattyp': Flattyp, 'rooms': Rooms}
                writer = csv.DictWriter(csvfile, fieldnames=cols)
                writer.writeheader()
                writer.writerow(dict_result)

            csvfile.close()



        except Exception as e:

            print("failed in the main loop", str(e))

get_house_info(ids)

Answer 1

我无法确切说明您的代码，但以下链接可能对您有所帮助： 1. Data School YouTube Channel 4部分视频中的“ Python中的Web抓取”。

如何从网站抓取数据并将其附加到Excel电子表格中

1 个答案: