我想要源代码的网页包含一些产品。
当我只是加载它时，它只给出了前几个产品的源代码。
单击显示140-250项后，我需要的是完整的源代码，该代码是多次生成的，当没有像显示x - y项<< / kbd>了。生成的按钮的id为see-more-products。
我尝试了urllib，requests模块，这些模块没有按预期完成工作。
链接到该特定页面的是评论。
感谢任何帮助。谢谢:)）
答案 0 :(得分：2)
有几种不同的方法可以帮助您入门
import requests
from bs4 import BeautifulSoup
for i in range(0, 3339, 48):
r = requests.get("http://www.snapdeal.com/acors/json/product/get/search/175/{}/48".format(i))
print([a["href"] for a in BeautifulSoup(r.content).select("div.product-desc-rating.title-section-expand a")])
这将为您提供所有电话链接，其中一个示例是：
['http://www.snapdeal.com/product/micromax-canvas-spark-2-4/658765871888', 'http://www.snapdeal.com/product/alpha-cod/639184667095', 'http://www.snapdeal.com/product/apple-iphone-5s-16-gb/1204769399', 'http://www.snapdeal.com/product/micromax-canvas-a1-aq4502-8/630310793485', 'http://www.snapdeal.com/product/infocus-m370-8gb/685769734109', 'http://www.snapdeal.com/product/lenovo-a1000-8gb-black/634810874852', 'http://www.snapdeal.com/product/infocus-m350-16gb/667944684160', 'http://www.snapdeal.com/product/motorola-moto-x-force-32gb/677683041597', 'http://www.snapdeal.com/product/infocus-m812i-16gb-silver/672166899922', 'http://www.snapdeal.com/product/xolo-black-1x-32gb/633488491545', 'http://www.snapdeal.com/product/meizu-m2-m578h-4g-16gb/632749055332', 'http://www.snapdeal.com/product/iphone-6s-16gb/663413326062', 'http://www.snapdeal.com/product/apple-iphone-6-16-gb/1270529654', 'http://www.snapdeal.com/product/apple-iphone-5s-16-gb/347830397', 'http://www.snapdeal.com/product/micromax-s301-4gb-black/668491526095', 'http://www.snapdeal.com/product/infocus-m260-zl2-8gb/619518605458', 'http://www.snapdeal.com/product/micromax-juice-2-aq5001-dual/625046032091', 'http://www.snapdeal.com/product/swipe-konnect-51-8gb-sand/626439596420', 'http://www.snapdeal.com/product/lenovo-a6000-black/670500124351', 'http://www.snapdeal.com/product/redmi-2-prime-16-gb/683138068892', 'http://www.snapdeal.com/product/asus-zenfone-go-8gb/646618265597', 'http://www.snapdeal.com/product/karbonn-aura-8gb-3g/627156357326', 'http://www.snapdeal.com/product/panasonic-eluga-turbo-32gb-marine/644981636428', 'http://www.snapdeal.com/product/asus-zenfone-2-laser-ze550kl/633082651487', 'http://www.snapdeal.com/product/redmi-2-8-gb-grey/628016512272', 'http://www.snapdeal.com/product/infocus-m680-16gb-4g/649492687203', 'http://www.snapdeal.com/product/lava-iris-atom-8gb-white/665532304471', 'http://www.snapdeal.com/product/micromax-x073-black/632400291797', 'http://www.snapdeal.com/product/intex-aqua-3g-pro-4gb/661217697971', 'http://www.snapdeal.com/product/micromax-x605-black/663717815370', 'http://www.snapdeal.com/product/micromax-x088-red/550645299', 'http://www.snapdeal.com/product/micromax-unite-3-q372/639750556852', 'http://www.snapdeal.com/product/micromax-canvas-selfie-2-q340/673685061115', 'http://www.snapdeal.com/product/samsung-j7-16gb-espresso-brown/661359071561', 'http://www.snapdeal.com/product/hpl-a50-white-dual-core/752819001', 'http://www.snapdeal.com/product/intex-intex-aqua-power-hd/677955090549', 'http://www.snapdeal.com/product/samsung-galaxy-core-2-4gb/95934411', 'http://www.snapdeal.com/product/samsung-galaxy-j2-8gb/637947695307', 'http://www.snapdeal.com/product/micromax-bolt-x088/1252626810', 'http://www.snapdeal.com/product/lenovo-a7000-turbo-16gb-matte/676482356343', 'http://www.snapdeal.com/product/nokia-105-dual-sim-black/625877378012', 'http://www.snapdeal.com/product/lenovo-vibe-x2ap/619606468183', 'http://www.snapdeal.com/product/xolo-hive-8x1000i-16-gb/620631303689', 'http://www.snapdeal.com/product/panasonic-p41-black-android-dual/1202820682', 'http://www.snapdeal.com/product/asus-zenfone-2-laser-ze500kl/686932521097', 'http://www.snapdeal.com/product/samsung-galaxy-grand-max/647437791381', 'http://www.snapdeal.com/product/panasonic-p55-novo-8-gb/630374183875', 'http://www.snapdeal.com/product/samsung-tizen-z1/214936653']
['http://www.snapdeal.com/product/micromax-canvas-juice-3-q392/675338203115', 'http://www.snapdeal.com/product/micromax-x853-256-mb-gray/678836590088', 'http://www.snapdeal.com/product/iball-prince-2-black/683878915532', 'http://www.snapdeal.com/product/intex-cloud-swift-4g-16gb/651487709065', 'http://www.snapdeal.com/product/htc-desire-820-white/2096877771', 'http://www.snapdeal.com/product/celkon-2gb-star-16gb/668257971604', 'http://www.snapdeal.com/product/lava-iris-atom-8gb-black/671938280548', 'http://www.snapdeal.com/product/asus-zenfone-max-zc550kl-16gb/659567821492', 'http://www.snapdeal.com/product/micromax-x084/1189906548', 'http://www.snapdeal.com/product/infocus-bingo-21-m430-8gb/685620250128', 'http://www.snapdeal.com/product/xolo-black-4g-16gb-with/670501820959', 'http://www.snapdeal.com/product/nokia-130-dual-sim-black/1238158129', 'http://www.snapdeal.com/product/phicomm-e670-energy-2-16gb/642536836073', 'http://www.snapdeal.com/product/lenovo-a7000-8-gbwhite/660909822842', 'http://www.snapdeal.com/product/infocus-m260-8gb-yellow-3g/658001307243', 'http://www.snapdeal.com/product/gionee-m5l-32gb-golden-4g/664586124833', 'http://www.snapdeal.com/product/spice-gaming-mobile-x2-red/673912591179', 'http://www.snapdeal.com/product/lenovo-lenovo-a1000-8gb-white/656199391476', 'http://www.snapdeal.com/product/xolo-cube-50-8gb-black/681508479062', 'http://www.snapdeal.com/product/microsoft-lumia-540-8-gb/641216010217', 'http://www.snapdeal.com/product/infocus-m810-16gb-gold-4g/683493537817', 'http://www.snapdeal.com/product/samsung-guru-music-2-sm/672045180352', 'http://www.snapdeal.com/product/samsung-guru-music-2-duos/197236301', 'http://www.snapdeal.com/product/intex-cloud-cube-8gb-gray/678043979304', 'http://www.snapdeal.com/product/samsung-tizen-z3-8gb/638555381978', 'http://www.snapdeal.com/product/micromax-canvas-spark/634407752835', 'http://www.snapdeal.com/product/moto-e-2nd-gen-3g/627496057276', 'http://www.snapdeal.com/product/intex-cloud-breeze-8gb-grey/671476050074', 'http://www.snapdeal.com/product/samsung-j5-8gb-espresso-brown/659785514599', 'http://www.snapdeal.com/product/lg-google-nexus-5-16/848745269', 'http://www.snapdeal.com/product/mi4i-16gb/654856488809', 'http://www.snapdeal.com/product/micromax-joy-1850-35-kb/673275470330', 'http://www.snapdeal.com/product/phicomm-energy-653-4g-8gb/621455892516', 'http://www.snapdeal.com/product/panasonic-eluga-icon-16gb/650547748150', 'http://www.snapdeal.com/product/lenovo-vibe-x2ap-32gb-white/660083187849', 'http://www.snapdeal.com/product/xolo-prime/683763827810', 'http://www.snapdeal.com/product/meizu-mx5-silver-black/618643364985', 'http://www.snapdeal.com/product/karbonn-titanium-s205-2gb-16gb/662465421207', 'http://www.snapdeal.com/product/micromax-canvas-nitro-4g-e455/667663492017', 'http://www.snapdeal.com/product/i-kall-k88-gray/659450396588', 'http://www.snapdeal.com/product/karbonn-mach-two-titanium-s360/653888615715', 'http://www.snapdeal.com/product/asus-zenfone-2-laser-55/681651791446', 'http://www.snapdeal.com/product/i-kall-k55-fruity-orange/621445557898', 'http://www.snapdeal.com/product/yu-yunique-8gb/621238835100', 'http://www.snapdeal.com/product/htc-desire-326g-dual-sim/673825142381', 'http://www.snapdeal.com/product/micromax-h375-below-256-mb/626813860848', 'http://www.snapdeal.com/product/intex-intex-aqua-star-2/677150759819', 'http://www.snapdeal.com/product/micromax-canvas-nitro-2/674813358269']
['http://www.snapdeal.com/product/karbonn-s-15-plus-8gb/678547994280', 'http://www.snapdeal.com/product/huawei-honor-bee-8gb-3g/623753457685', 'http://www.snapdeal.com/product/intex-cloud-m6-16-gb/652438065304', 'http://www.snapdeal.com/product/karbonn-s310/2046781351', 'http://www.snapdeal.com/product/htc-desire-826/673132800168', 'http://www.snapdeal.com/product/oneplus-one-64gb/683926070447', 'http://www.snapdeal.com/product/samsung-galaxy-grand-prime-4g/682950239423', 'http://www.snapdeal.com/product/panasonic-panasonic-eluga-switch-32gb/651502528631', 'http://www.snapdeal.com/product/infocus-f120-dual-sim-gsm/647132622326', 'http://www.snapdeal.com/product/samsung-j5-8gb-white/625632131578', 'http://www.snapdeal.com/product/micromax-canvas-juice-3-q394/647451126738', 'http://www.snapdeal.com/product/micromax-x610-black/677067716681', 'http://www.snapdeal.com/product/videocon-v1393-dual-sim-mobile/519151652', 'http://www.snapdeal.com/product/nokia-105/1397084', 'http://www.snapdeal.com/product/micromax-canvas-blaze-4g-q400/635578549162', 'http://www.snapdeal.com/product/lenovo-a-6000-shot-16gb/652479804547', 'http://www.snapdeal.com/product/micromax-x597-black/666857291079', 'http://www.snapdeal.com/product/lenovo-a2010-16gb-black/641561931090', 'http://www.snapdeal.com/product/micromax-x597-blue/635774497484', 'http://www.snapdeal.com/product/lenovo-a1000-8gb-white-3g/682465160925', 'http://www.snapdeal.com/product/oorie-gsm-4gb-gray/644950735714', 'http://www.snapdeal.com/product/videocon-v1429w-black/659876102140', 'http://www.snapdeal.com/product/wham-q4-8gb-black/683618702751', 'http://www.snapdeal.com/product/lenovo-s60-8gb-white/682858759745', 'http://www.snapdeal.com/product/lenovo-a6000-white-8gb-white/687038221568', 'http://www.snapdeal.com/product/k2-air-8gb-black/662606076269', 'http://www.snapdeal.com/product/infocus-m680-16gb-4g/623852692656', 'http://www.snapdeal.com/product/micromax-x707-gray/675166544662', 'http://www.snapdeal.com/product/micromax-bolt-a-24-champagne/318667695', 'http://www.snapdeal.com/product/htc-desire-620-g/2140463474', 'http://www.snapdeal.com/product/whitecherry-k9-gray/634412911071', 'http://www.snapdeal.com/product/intex-neo-v-plus-black/49904988', 'http://www.snapdeal.com/product/intex-aqua-power-hd-mobile/639191044460', 'http://www.snapdeal.com/product/i-kall-k88-black/624856889024', 'http://www.snapdeal.com/product/apple-iphone-6-64-gb/131798351', 'http://www.snapdeal.com/product/infocus-f135-dual-sim-gsm/662122536540', 'http://www.snapdeal.com/product/apple-iphone-5s-32-gb/936530455', 'http://www.snapdeal.com/product/micromax-bolt-q338-8gb-black/633509849227', 'http://www.snapdeal.com/product/micromax-x084-black/676316493911', 'http://www.snapdeal.com/product/infocus-f110-dual-sim-gsm/668147567472', 'http://www.snapdeal.com/product/kenxinda-watch-mobile-dual-sim/149800122', 'http://www.snapdeal.com/product/panasonic-eluga-s/1270286256', 'http://www.snapdeal.com/product/swipe-virtue-16gb-white-3g/633490528209', 'http://www.snapdeal.com/product/micromax-canvas-play-q355-dual/663710099854', 'http://www.snapdeal.com/product/samsung-tizen-z1-4gb-gold/683743955020', 'http://www.snapdeal.com/product/asus-zenfone-go-45-zc451tg/632893288226', 'http://www.snapdeal.com/product/karbonn-a6-turbo/677649317861', 'http://www.snapdeal.com/product/karbonn-titanium-delight-s22-blackgrey/679556902984']
您可能需要稍微调整一下，但这应该接近获得您想要的所有链接：
import requests
from bs4 import BeautifulSoup
def get_all_links():
r = requests.get("http://www.snapdeal.com/acors/json/product/get/search/175/0/48")
total = int((BeautifulSoup(r.content).select("div.jsNumberFound.hidden")[0].text))
yield from (a["href"] for a in BeautifulSoup(r.content).select("div.product-desc-rating.title-section-expand a"))
print("There are {} total items".format(total))
for pge in range(50, total+1, 48):
r = requests.get("http://www.snapdeal.com/acors/json/product/get/search/175/{}/48".format(pge))
yield from (a["href"] for a in BeautifulSoup(r.content).select("div.product-desc-rating.title-section-expand a"))
There are 3340 total items
http://www.snapdeal.com/product/micromax-canvas-spark/634407752835
http://www.snapdeal.com/product/iphone-6s-16gb/663413326062
http://www.snapdeal.com/product/apple-iphone-5s-16-gb/1204769399
http://www.snapdeal.com/product/moto-e-2nd-gen-3g/627496057276
http://www.snapdeal.com/product/redmi-2-prime-16-gb/683138068892
http://www.snapdeal.com/product/micromax-juice-2-aq5001-dual/625046032091
http://www.snapdeal.com/product/alpha-cod/639184667095
http://www.snapdeal.com/product/intex-cloud-breeze-8gb-grey/671476050074
http://www.snapdeal.com/product/micromax-canvas-spark-2-4/658765871888
http://www.snapdeal.com/product/infocus-m370-8gb/685769734109
http://www.snapdeal.com/product/infocus-m350-16gb/667944684160
http://www.snapdeal.com/product/lg-google-nexus-5-16/848745269
http://www.snapdeal.com/product/lenovo-a6000-black/670500124351
http://www.snapdeal.com/product/apple-iphone-6-16-gb/1270529654
http://www.snapdeal.com/product/meizu-m2-m578h-4g-16gb/632749055332
http://www.snapdeal.com/product/micromax-canvas-a1-aq4502-8/630310793485
http://www.snapdeal.com/product/apple-iphone-5s-16-gb/347830397
http://www.snapdeal.com/product/infocus-bingo-21-m430-8gb/685620250128
http://www.snapdeal.com/product/samsung-j7-16gb-espresso-brown/661359071561
http://www.snapdeal.com/product/apple-iphone-6-64-gb/131798351
http://www.snapdeal.com/product/panasonic-p55-novo-8-gb/630374183875
http://www.snapdeal.com/product/lenovo-a1000-8gb-black/634810874852
http://www.snapdeal.com/product/infocus-m260-zl2-8gb/619518605458
http://www.snapdeal.com/product/asus-zenfone-2-laser-ze500kl/686932521097
http://www.snapdeal.com/product/meizu-mx5-silver-black/618643364985
http://www.snapdeal.com/product/moto-e-2nd-gen-4g/682444504031
http://www.snapdeal.com/product/xolo-black-1x-32gb/633488491545
http://www.snapdeal.com/product/panasonic-eluga-turbo-32gb-marine/644981636428
http://www.snapdeal.com/product/mi4i-16gb/654856488809
http://www.snapdeal.com/product/samsung-j5-8gb-espresso-brown/659785514599
http://www.snapdeal.com/product/intex-cloud-swift-4g-16gb/651487709065
http://www.snapdeal.com/product/lenovo-vibe-x2ap-32gb-white/660083187849
http://www.snapdeal.com/product/panasonic-eluga-icon-16gb/650547748150
http://www.snapdeal.com/product/micromax-s301-4gb-black/668491526095
http://www.snapdeal.com/product/apple-iphone-5s-32-gb/936530455
http://www.snapdeal.com/product/lenovo-a7000-turbo-16gb-matte/676482356343
http://www.snapdeal.com/product/samsung-tizen-z3-8gb/638555381978
http://www.snapdeal.com/product/oneplus-one-64gb/683926070447
http://www.snapdeal.com/product/asus-zenfone-2-laser-55/681651791446
http://www.snapdeal.com/product/htc-desire-826/673132800168
http://www.snapdeal.com/product/gionee-m5l-32gb-golden-4g/664586124833
http://www.snapdeal.com/product/intex-intex-aqua-power-hd/677955090549
http://www.snapdeal.com/product/samsung-a7-2016-16gb-4g/632787468738
http://www.snapdeal.com/product/micromax-unite-3-q372/639750556852