Question

I am learning to scrape data from website through Python. Extracting weather information about San Francisco from this page中。将数据组合到Pandas Dataframe中时卡住了。是否可以创建一个数据行，其中每行的长度都不同？

我已经根据这里的答案尝试了两种方法，但是它们并不是我所寻找的。两个答案都将temps列的值上移。 Here is the screen what I try to explain.。

第一种方式：https://stackoverflow.com/a/40442094/10179259

第二种方式：https://stackoverflow.com/a/19736406/10179259

import requests
from bs4 import BeautifulSoup
import pandas as pd


page = requests.get("http://forecast.weather.gov/MapClick.php?lat=37.7772&lon=-122.4168")

soup = BeautifulSoup(page.content, 'html.parser')

seven_day = soup.find(id="seven-day-forecast")

forecast_items = seven_day.find_all(class_="tombstone-container")

periods=[pt.get_text() for pt in seven_day.select('.tombstone-container .period-name')]

short_descs=[sd.get_text() for sd in seven_day.select('.tombstone-container .short-desc')]

temps=[t.get_text() for t in seven_day.select('.tombstone-container .temp')]

descs = [d['alt'] for d in seven_day.select('.tombstone-container img')]


#print(len(periods), len(short_descs), len(temps), len(descs))

weather = pd.DataFrame({
        "period": periods, #length is 9
        "short_desc": short_descs, #length is 9
        "temp": temps, #problem here length is 8
        #"desc":descs #length is 9
    })


print(weather)

我希望temp列的第一行是Nan。谢谢。

Answer 1

如果为字典iter的值分配了不存在的值，则可以用next和NaN循环每个Forecast_items值以选择第一个值：

page = requests.get("http://forecast.weather.gov/MapClick.php?lat=37.7772&lon=-122.4168")

soup = BeautifulSoup(page.content, 'html.parser')

seven_day = soup.find(id="seven-day-forecast")

forecast_items = seven_day.find_all(class_="tombstone-container")

out = []
for x in forecast_items:
    periods = next(iter([t.get_text() for t in x.select('.period-name')]), np.nan)
    short_descs = next(iter([t.get_text() for t in x.select('.short-desc')]), np.nan)
    temps = next(iter([t.get_text() for t in x.select('.temp')]), np.nan)
    descs = next(iter([d['alt'] for d in x.select('img')]), np.nan)
    out.append({'period':periods, 'short_desc':short_descs, 'temp':temps, 'descs':descs})

weather = pd.DataFrame(out)
print (weather)
                                               descs               period  \
0                                                     NOW until4:00pm Sat   
1  Today: Showers, with thunderstorms also possib...                Today   
2  Tonight: Showers likely and possibly a thunder...              Tonight   
3  Sunday: A chance of showers before 11am, then ...               Sunday   
4  Sunday Night: Rain before 11pm, then a chance ...          SundayNight   
5  Monday: A 40 percent chance of showers.  Cloud...               Monday   
6  Monday Night: A 30 percent chance of showers. ...          MondayNight   
7  Tuesday: A 50 percent chance of rain.  Cloudy,...              Tuesday   
8  Tuesday Night: Rain.  Cloudy, with a low aroun...         TuesdayNight   

                               short_desc         temp  
0                           Wind Advisory          NaN  
1                       Showers andBreezy  High: 56 °F  
2                           ShowersLikely   Low: 49 °F  
3                     Heavy Rainand Windy  High: 56 °F  
4  Heavy Rainand Breezythen ChanceShowers   Low: 52 °F  
5                           ChanceShowers  High: 58 °F  
6                           ChanceShowers   Low: 53 °F  
7                             Chance Rain  High: 59 °F  
8                                    Rain   Low: 53 °F

Creating a dataframe where one of the arrays has a different length

1 个答案: