Web讨论板

Question

我需要从以下网站的前5页中抓取所有问题：https://www.dell.com/community/Inspiron/bd-p/Inspiron

我需要将讨论标题，作者，提问时间和最后答复者返回到一个csv文件中。

有人可以让我开始吗？到目前为止，我还没有任何代码。我也不知道从哪里开始。这个问题有太多的部分让我无法解决。

谢谢！它不必是完整的代码，但是开始会很有帮助。我知道我应该使用请求和BeautifulSoup

编辑：我没有助教来寻求帮助。这就是为什么我在这里发布。

Answer 1

当然，有一个简单得多的Web抓取解决方案...直接拉入RSS feed ：）。这是合法的方法，并且在技术上更简单。导入数据已经是您期望的格式，这是一种一线工作。

https://www.dell.com/community/s/vjauj58549/rss/boardmessages?board.id=Inspiron&count=100

示例代码：

final SurfaceView _surface = (SurfaceView) findViewById(R.id.surfaceView);

与使用webscraping相比，它更可取，因为静态html网站依赖于此。

Answer 2

您应该尝试Scrapy，这是一个免费的开源网络爬网框架。

您可以在此处查看：https://scrapy.org/

解决方案可能类似于下面的示例：

# -*- coding: utf-8 -*-
import scrapy


class ToScrapeSpiderXPath(scrapy.Spider):
    name = 'toscrape-xpath'
    start_urls = [
        'http://quotes.toscrape.com/',
    ]

    def parse(self, response):
        for quote in response.xpath('//div[@class="quote"]'):
            yield {
                'text': quote.xpath('./span[@class="text"]/text()').extract_first(),
                'author': quote.xpath('.//small[@class="author"]/text()').extract_first(),
                'tags': quote.xpath('.//div[@class="tags"]/a[@class="tag"]/text()').extract()
            }

        next_page_url = response.xpath('//li[@class="next"]/a/@href').extract_first()
        if next_page_url is not None:
            yield scrapy.Request(response.urljoin(next_page_url))

示例来自：https://github.com/scrapy/quotesbot

Answer 3

您可以使用requests来检索页面源，并使用BeautifulSoup来分析结果。要查找问题，请使用包含问题ID的类遍历每个tr：

from bs4 import BeautifulSoup as soup
import requests, re
def scrape_page(page_num):
   d = soup(requests.get('https://www.dell.com/community/Inspiron/bd-p/Inspiron/page/{}'.format(page_num)).text, 'html.parser')
   q = d.find_all('tr', {'class':re.compile('lia-js-data-messageUid-\d+')}) #find all question trs
   result = [[i.find('a', {'class':'page-link'}).text]+[c.text for c in i.find_all('a', {'class':'lia-user-name-link'})]+[i.find('span', {'class':'lia-component-common-widget-date'}).text] for i in q]
   return [[re.sub('\n|\s{2,}', '', c) for c in i] for i in result]

每个页面的结果是一个列表列表，其中包含标题，作者，最后评论发布者以及提出问题的日期。要查找前五页：

results = [scrape_page(i) for i in range(1, 6)]

输出：

[[['Inspiron gaming laptops, battery drain is normal', 'DELL-Chris M', '\u200e11-30-201808:36 AM'], ['Where are my Inspiron Laptop manuals?', 'DELL-Chris M', '\u200e05-02-201809:53 AM'], ['Where are my Inspiron Laptop drivers?', 'DELL-Chris M', '\u200e05-02-201809:50 AM'], ['New Inspiron Laptop Usage and Troubleshooting Guides (3/2/18)', 'DELL-Chris M', '\u200e03-02-201810:28 AM'], ['Can not install dell mobile connect aplication from windows store', 'GeoNeo77', 'Mary G', '\u200e04-11-201908:20 AM'], ['Inspiron 13 7000 (7370) Black / Blank Screen / Display', 'brianms', 'Erutan409', '\u200e11-14-201805:46 PM'], ['Boot From External USB Drive or USB Stick', 'borntodeal', '\u200e04-11-201910:01 AM'], ['WiFi suddenly disappeared', 'Anagno7', '\u200e04-11-201909:50 AM'], ['Task bar ikons move quickly back and forth', 'ralphspajka', 'ralphspajka', '\u200e04-11-201907:06 AM'], ['Does an Intel Dual Band Wireless-AC 8265 or 9260 will work on a Inspiron 5555', 'Cauchons', 'jphughan', '\u200e04-07-201910:43 AM'], ['Fan not working', 'Tunay', 'JOcean', '\u200e04-10-201903:35 PM'], ['Upgraded Wireless N 7260 WIFI Card', 'fadsarmy', 'fadsarmy', '\u200e03-30-201909:13 PM'], ['XPS 13-9365, battery issues!', 'rickmanhatan', 'DELL-Chris M', '\u200e04-11-201903:18 AM'], ['Inspiron 17 7779, turns black after playing games', 'Prime7105', 'Teetertotter', '\u200e04-09-201908:12 PM'], ['New 7559 BIOS is corrupted', 'Suncatcher_13', 'Teetertotter', '\u200e02-26-201906:25 PM'], ['Inspiron 15 7559 compatible con SSD Samsung 970 EVO Plus?', 'defto', 'Saltgrass', '\u200e04-10-201908:40 PM'], ['Laptop wont turn on...', 'TonicWhisper', 'JOcean', '\u200e04-10-201911:40 PM'], ['Why is my Dell G7 buzzing when charging', 'Andre Dolar', 'Jey_31', '\u200e09-24-201806:09 AM'], ['Clone only OS partition to SSD/Install windows 10 on new SSD', 'HemantSangle', 'hezi', '\u200e09-26-201803:02 AM'], ['ERROR Alienware Command Center en G7 17 7790', 'Matafleur', 'SERGIO.MAGNO', '\u200e03-16-201908:03 PM'], ['Inspiron 15-7567, M.2 NVMe SSD, failed boot device', 'JohnJakeLindert', 'Saltgrass', '\u200e03-31-201902:02 PM'], ['Dell adapter not detected. Plugged in, not charging.', 'spbone', 'acmoo1964', '\u200e01-23-201802:11 PM'], ['Inspiron 5482 keyboard skipping characters, missing strokes. HELP! :(', 'NajibOzy', 'NajibOzy', '\u200e04-10-201902:43 AM'], ['Bluetooth Inspiron 14 3000 Demora demais para conectar com o fone.', 'VitorVitalino', 'Dell-SreejithR', '\u200e04-08-201912:45 PM'], ['Inspiron 13-7000 7359', 'jjrapy', 'dell-robert p', '\u200e04-08-201903:49 PM'], ['DELL INSPIRON i5 5570 CPU OVERHEATING', 'ryanmcgrath', 'Dell-SreejithR', '\u200e04-05-201911:30 PM'], ['Inspiron 24 5000 problem', 'cm_a', 'DELL-Chris M', '\u200e04-04-201912:07 AM'], ['Inspiron G5 15-5587, WiFi drops, then connects itself', 'abhar22', 'abhar22', '\u200e04-02-201907:16 PM'], ['Dell G5 15 5587 Overheating Issue', 'eldarinn', 'Dell-SreejithR', '\u200e03-06-201902:38 AM'], ['Inspiron 15, WiFi not detected', 'savaman', 'DELL-Chris M', '\u200e04-05-201911:35 AM'], ['Dell Inspiron 13 7000 P83G Will Not Wake After Sleep', 'random0o', 'dell-robert p', '\u200e04-06-201905:51 PM'], ['Inspiron 14, 2.1 USB sound kit', 'GuilhermeBil', 'DELL-Chris M', '\u200e04-05-201902:32 PM'], ['Help with display', 'dannyetlv', 'Mary G', '\u200e04-09-201902:00 PM'], ['Inspiron 15 5567 battery drains when shutdown', 'Razaaque', 'Dell-SreejithR', '\u200e03-21-201903:28 AM']], [['If the charger is plugged in at 100% battery life will it harm my battery life in any way?', 'Burlex', 'dell-robert p', '\u200e04-05-201911:48 AM'], ['Audio problems after Dell update........ EVERY TIME', 'Skaboudles', 'Saltgrass', '\u200e04-08-201909:50 AM'], ['Dell Inspiron 7548 Ram', 'Foofuu', 'JOcean', '\u200e04-09-201905:27 PM'], ['Inspiron 17 5770 max size of RAM (memory)', 'leks12lk', 'JOcean', '\u200e04-10-201912:16 AM'], ['Inspiron G3 15-3579, randomly freezes', 'livictor', 'Mike_Space', '\u200e10-17-201806:59 PM'], ['audio renderer error', 'Shoun0305', 'anttisum', '\u200e01-24-201911:41 PM'], ['Dell G5 5587 Sound issues', 'NickBk', 'Cptntrips', '\u200e02-15-201911:49 AM'], ['Inspiron G7 15-7588, speaker is not plugged after updating drivers', 'Thai Hung', 'Thai Hung', '\u200e04-07-201908:30 PM'], ["Dell Inspiron 15 5000 fan on constantly when connected to power but won't boot", '996sleepless', '\u200e04-10-201901:22 AM'], ['Inspiron 17 7000 - Headset not an option?', 'Vildpear', 'Vildpear', '\u200e02-14-201902:07 AM'], ['Dell Inspiron 5575 ryzen 5 vega 8 keeps throttling', 'FIDERIKS', '\u200e04-10-201912:49 AM'], ['Inspiron 15 5575 AMD M.2 NVME slow speed', 'Billdag', 'jphughan', '\u200e04-09-201905:16 PM'], ['hdd not detected inspiron 7573 after BIOs upgrade', 'abylan', 'abylan', '\u200e04-06-201902:11 PM'], ['Dell Inspiron 14 3421-Dell Wireless 1703 Wlan Card compatibility', '5GHzstarve', '5GHzstarve', '\u200e04-09-201907:43 AM'], ['Dell Inspiron 15 5000 laptop super slow', 'DH9500', '\u200e04-09-201905:55 PM'], ['USB Issue with TB18DC', 'AmazingDell', '\u200e04-09-201905:50 PM'], ['Nvidia 1060 max-q temperature in Dell Inspiron 7577', 'Villn52', '\u200e04-09-201905:49 PM'], ['Wifi issues with dell Inspiron 15 7000', 'Desperate4help', 'Zcarcath', '\u200e11-15-201804:23 AM'], ['Inspiron 15 7559 - Battery light randomly flickering', 'AstralC', 'AstralC', '\u200e04-09-201910:37 AM'], ['Intermittent noises from hard drive', 'SV126CBC', 'nyc10036', '\u200e04-09-201902:03 PM'], ['Smartbyte reinstalls itself?! How to make it stop?', 'krgman101', 'RivetNetworks', '\u200e04-08-201905:21 AM'], ['Mouse/Keyboard input overlap issue', 'Goodnight guy', 'Sgtsparkles', '\u200e04-08-201904:56 PM'], ['Inspiron 15-7567, reed switch?', 'shawanish8', 'Furrydaus', '\u200e04-01-201911:29 PM'], ['inspiron 15-5580', 'GOWTHAMAN RAVI', '\u200e04-09-201910:29 AM'], ['Inspiron 7580, CPU temperature', 'pl66', 'pl66', '\u200e04-02-201910:50 AM'], ['Ranura m2 en dell inspiron 15 3000 3567', 'Nico.eguia', 'Dell-SreejithR', '\u200e04-03-201909:10 AM'], ['Dell PC keeps booting up to diagnostic boot test', 'a__biee', 'Tiburon66', '\u200e10-09-201804:17 AM'], ['How To Disable Adaptive Brightness in DELL Insipiron 7580 laptop', 'Addyzaf', 'pl66', '\u200e04-09-201910:18 AM'], ['G3 3579 Fan makes clicking noise at low rpm', 'a2375658', 'Dell-SreejithR', '\u200e02-20-201910:27 PM'], ["Which Inspiron's have DBC?", 'Vortex--5', 'DELL-Chris M', '\u200e04-03-201905:27 AM']], [['backlit keyboard assembly for Inspiron 11-3185', 'mrsbranj', 'dell-robert p', '\u200e04-03-201903:57 AM'], ['Re: Inspiron 7580 fan noise and heating issue on m', 'Addyzaf', 'dell-robert p', '\u200e04-07-201910:41 AM'], ['Inspiron G7 15-7588, second monitor not working', 'Movie1011', 'DELL-Chris M', '\u200e04-08-201911:40 AM'], ['The display is okay?', 'dannyetlv', '\u200e04-09-201910:45 AM'], ['bios update problem', 'yodas91', 'JLJM', '\u200e02-08-201808:13 AM'], ["Inspiron 13 7000 (7370) Display Won't Wake Up", 'brianms', 'brianms', '\u200e11-15-201809:27 AM'], ['2019 G7 7590 Fan Control Abysmal', 'msbtt94', '\u200e04-08-201905:41 PM'], ['Dell G5 Startup problem on battery', 'bob8me', 'Dell-SreejithR', '\u200e02-04-201903:02 PM'], ['Monitor going black', 'uffda123', 'Dell-SreejithR', '\u200e04-07-201912:10 PM'], ['how to system rewrite on Inspiron 3153', 'HD72', 'Philip_Yip', '\u200e04-08-201911:45 AM'], ['I would like to upgrade my Dell Inspiron 17R, any advice?', 'careb', 'nyc10036', '\u200e04-08-201912:32 PM'], ['dell G5 8875 lagging or freezing during playing a video or surfing the internet and fan noise', 'zeyadmourad', 'Dell-SreejithR', '\u200e04-07-201903:42 AM'], ['dell G5 5587 over heating', 'zeyadmourad', 'Dell-SreejithR', '\u200e04-08-201908:49 PM'], ['Took a gamble... WD15 USB-C dock works with Inspiron 17 5770', 'Frozen Mitchell', 'leks12lk', '\u200e03-22-201908:44 PM'], ['Inspiron 7572, Intel 620, Nvidia MX 150 4GB', 'Teetertotter', 'Teetertotter', '\u200e04-06-201909:45 AM'], ['BIOS update will not run on Inspiron 15-3542', 'ai4gk', 'ai4gk', '\u200e04-08-201910:26 PM'], ['Dell inspiron 3558 low wifi signal', 'Gaffar', 'JOcean', '\u200e04-08-201912:45 PM'], ['Inspiron 7359 no video and 2 beeps', 'mek33', 'JOcean', '\u200e04-08-201906:56 PM'], ['Memory and SDD Upgrade for dell N3542', 'bitsrdx', 'JOcean', '\u200e04-09-201904:30 AM'], ['are Dell Inspiron 14" 5458 and 5459 parts interchangable?', 'tristan012345', '\u200e04-09-201903:47 AM'], ['G5 5587 Latest Realtek audio driver breaks sound entirely', 'BarnyTrubble', 'Mark3434', '\u200e02-06-201911:01 PM'], ['G3 3579, IRST Drivers causing data loss on SATA SSD.', 'ariaperdana', '\u200e04-08-201910:28 PM'], ['Inspiron 15-5580.', 'GOWTHAMAN RAVI', '\u200e04-08-201910:04 PM'], ['Dell Inspiron 7370 sleep / wake. Lid closure issues.', 'Harish_Vanguard', 'Nambabwe', '\u200e05-23-201803:50 AM'], ['Inspiron 13 7000', 'jjrapy', 'nyc10036', '\u200e04-08-201903:54 PM'], ['Computer not connecting to network after getting static IP', 'MChe', 'irishgeoff', '\u200e03-04-201805:34 AM'], ['HID Complaint touch screen driver not available', 'VarunDell', 'shahrishit9', '\u200e08-10-201811:35 AM'], ['5559 Touchscreen driver', 'Natalie Settoon', 'kroppykat', '\u200e03-30-201806:55 PM'], ['Support Assist Not installing', 'Kannans243', 'PT8888', '\u200e05-18-201802:06 AM'], ['Keyboard issue', 'SteveTree', 'SteveTree', '\u200e04-07-201904:09 AM']], [['Please fix the C-State bug in Inspiron 11 3000.', 'saudiqbal', 'leemell', '\u200e08-26-201810:27 PM'], ['Dell 7577 overheating', 'qaiser96', 'Dell-SreejithR', '\u200e04-06-201903:57 AM'], ['Dell G3 (3579) BSOD when is plugged into power supply', 'Brssvk', 'Dell-SreejithR', '\u200e04-06-201902:22 AM'], ['Power surge on USB port', 'Lazysocks', 'Mary G', '\u200e04-08-201912:29 PM'], ['Laptop Warranty', 'sophie_ep', 'Mary G', '\u200e04-08-201910:01 AM'], ['Dell updates fail to install after warranty expires', 'JWall61', 'Mary G', '\u200e04-08-201907:31 AM'], ['Inspiron 13 5378 - Touchpad missing from hardware and gestures do not work - Windows 10', 'bjames8787', '\u200e04-08-201910:56 AM'], ['Inspiron 7580, thermals, cooling, fan control', 'pl66', 'pl66', '\u200e04-06-201902:33 AM'], ['RAM upgrade inspiron 5521-6527 my service tag', 'Cihat Eren', 'JOcean', '\u200e04-08-201904:09 AM'], ["New hard drive, dell recovery disk won't go to install menu", 'yt1300don', 'yt1300don', '\u200e04-07-201911:49 PM'], ['Dell Inspiron i5 7000 gaming turning on for a few seconds and immediadtly back off', 'Cmac123', 'speedstep', '\u200e04-07-201909:10 PM'], ['Recover BIOS for Inspiron 5755', 'laltobelli', 'SHColf57', '\u200e03-15-201807:08 PM'], ['Inspiron 7380, loud fan', 'ladrian', '\u200e04-07-201906:43 PM'], ['Power Adapter Error', 'GMMister', 'dell-robert p', '\u200e01-25-201908:35 PM'], ['Inspiron 17 3721 Four Beeps on booting', 'Eddythealien2003', 'fireberd', '\u200e04-08-201902:38 AM'], ['inspiron', 'cosmicdancer', 'evegreen', '\u200e03-11-201903:48 AM'], ['RAM upgrade for Dell Inspiron 5570', 'asadsultanawan', 'Sunray001', '\u200e08-13-201806:28 AM'], ['ssd upgrade dell inspiron 7559', 'Mr Wob', 'Ning-123', '\u200e04-06-201901:32 PM'], ['AC Power Adapter not reading', 'Monemi', '\u200e04-07-201910:46 PM'], ['Battery replacement for Inspiron 5558', 'Chxrlotte', 'ron564848', '\u200e01-02-201905:38 AM'], ['Can Inspiron 5570 boot from NVMe M.2 SSD?', 'valentin_b', 'jamquam', '\u200e07-11-201809:37 AM'], ['Using laptop on direct power without using battery power [built in battery ]', 'RAJR11', '\u200e04-07-201908:56 PM'], ['charger wire colors', 'Shaul G', 'nyc10036', '\u200e04-07-201904:29 AM'], ['Dell Inspiron 13 5378 pen support', 'Joeldamole', 'Joeldamole', '\u200e04-03-201903:49 PM'], ['Dell OS Recovery Tool', 'vedantjoshi', 'Philip_Yip', '\u200e04-07-201910:07 AM'], ['Inspiron N5050 Upgrade', 'Forkman', '\u200e04-07-201901:45 PM'], ['Inspiron N5050 Upgrade', 'Forkman', '\u200e04-07-201901:42 PM'], ['Include XPS thermal management system to G3 BIOS', 'Evsign', 'JeffersonX', '\u200e02-26-201908:37 AM'], ['keyboard stopped working after upgrading the RAM', 'OudMaster', 'OudMaster', '\u200e04-06-201901:02 PM'], ['G3 3579 temperature problem', 'TheLonelySqrt3', 'JeffersonX', '\u200e12-01-201808:55 PM']], [['Latitude E6530 graphics card problem', 'Anujpandey324', '\u200e04-07-201912:35 PM'], ['Inspiron n5010 3 beeps & black screen', 'stayaway.fgs', 'JOcean', '\u200e04-07-201907:48 AM'], ['INSPIRON 15 5000 SERIES, POSSIBLY OTHERS BEWARE', 'louie.bitweiser', '\u200e04-07-201911:30 AM'], ['Lap tops', 'oakfans', '\u200e04-07-201911:02 AM'], ['USB 3.0 Ethernet Adapter', 'winterboy8', 'jphughan', '\u200e04-06-201911:54 AM'], ['Dell g5 fans not spinning at all', 'Ryinix', 'Ryinix', '\u200e04-05-201903:16 PM'], ['upgrade wlan in Dell Inspiron 3567', 'Jesse__G', 'Jesse__G', '\u200e03-26-201910:14 AM'], ['UPS for Dell Inspiron 15R 5520 laptop', 'Hsyn', 'nyc10036', '\u200e04-07-201901:58 AM'], ['Pressing 8 shows 0. Replaced keyboard, worked for a few weeks, same issue returned?', 'AW2', 'AW2', '\u200e04-07-201903:13 AM'], ['7437 Keyboard issues after A13 BIOS upgrade. Downgrade fixes', 'AW2', 'AW2', '\u200e03-06-201902:18 AM'], ['Heat', 'Prestonwhhhite', 'Afflospark', '\u200e10-13-201805:38 PM'], ['Inspiron 7586 screen issues', 'lharmon1922', 'vamp4200', '\u200e11-29-201801:22 PM'], ['Insiroin 5570 battery time', 'jurek224466', 'Afflospark', '\u200e04-05-201912:31 AM'], ['Dell inspiron 7577 battery discharges while gamimg with charger pluged in', 'George7577', 'Afflospark', '\u200e03-10-201809:50 PM'], ['Inspiron 3000 Dead after 1 year??', 'Eastwaangel', '\u200e04-06-201908:52 PM'], ['True Color App', 'Jukebox8691', 'dsfingers', '\u200e03-09-201803:58 PM'], ['Inspiron 7577, Games refusing to use Dedicated Graphics', 'InternetInfantry', 'JOcean', '\u200e04-06-201912:55 AM'], ['Inspiron N 4010 1600MHz sticks', 'sree harsha', 'Mary G', '\u200e04-06-201905:23 PM'], ['Inspiron I7 7737 video small after driver upgrade', 'Irish2204', 'Irish2204', '\u200e04-06-201904:08 PM'], ["Inspirion 13 7348 Won't Turn On", 'packer96', 'robbb9119', '\u200e02-25-201807:34 PM'], ['Screen issues with Dell Inspiron 15 7000', 'GDMTCaboose', 'Mr Wob', '\u200e06-17-201806:20 AM'], ['Inspiron 15-7???, not detecting headphones', 'Tensins', 'Greenphase', '\u200e05-23-201802:49 PM'], ['Inspiron 15-3521, ram upgrade', 'AymaneHTF', 'Pipibrasci', '\u200e02-07-201807:53 AM'], ['Leaving my dell 15 7559 connected to the AC when using?', 'LeoTwixx', 'Teetertotter', '\u200e04-05-201905:14 PM'], ['G3 3779 Slow wifi download speed around 400 kb/s', 'Teodor Yanchev', 'fireberd', '\u200e04-05-201907:56 PM'], ['how can i change boot order from sata 0 to sata 1 ????', 'roami', 'Mary G', '\u200e04-05-201903:24 PM'], ['Need to replace only display cable or entire display?', 'dannyetlv', 'dannyetlv', '\u200e04-06-201912:44 AM'], ['Inspiron 17r 5737 BIOS update AC Power Adapter type cannot be determined', 'kozle', 'SheTriedIt', '\u200e03-24-201807:08 AM'], ['13 7378 2-in-1 Can I replace the 256 GB with a 1 TB SSD', 'Biker 2017', 'Mike61986', '\u200e11-25-201806:57 AM'], ['Inspiron 15 7000 series case separating', 'MikeD927', '\u200e04-06-201904:10 AM']]]

Answer 4

这是一种简单的方法，我希望它可以对您有所帮助。

<div class="first box">
</div>
<div class="second box">
</div>
<div class="last box">
</div>