Question

嘿家伙所以亚马逊主搜索栏有以下信息

<input type="submit" class="nav-input" value="Go" tabindex="7">

我正在考虑创建一个能够查看amazon.co.uk的功能并找到此标记，并搜索我提供的任何给定关键字

要做到这一点，我做到了这一点：

path = 'https://www.amazon.co.uk/'

values = {'s': 'what-I-want-to-search',
          'submit':'search'}
data = urllib.parse.urlencode(values)
data = data.encode('utf-8')


driver = webdriver.PhantomJS()
driver.get(path)
html = driver.page_source

driver = webdriver.PhantomJS()
driver.get(path, data)

html = driver.page_source

我的想法是在senddex教程之后，我编码搜索术语，然后将它们发送到html路径，并且我一直在使用selenium来对抗动态加载的网页，但在这种情况下我认为应该没问题，但要么方式，我需要知道如何让python在主站点上搜索某些东西并让它带我到搜索结果页面，任何帮助？

业力将成为朋友的方式

Answer 1

使用请求和 bs4 ，您只需要传递正确的参数，如果您查看chrome中的网络标签，就可以看到开发工具：

In [4]: from bs4 import BeautifulSoup    
In [5]: import requests    
In [6]: params = {"url": "search-alias=",
   ...:           "field-keywords": "python"}

In [7]: with requests.Session() as s:
   ...:         url = "https://www.amazon.co.uk/s"
   ...:         r = s.get(url, params=params)
   ...:         soup = BeautifulSoup(r.content,"lxml")
   ...:         for a in cont:
   ...:               print(a.select_one("a")["title"])
   ...:         
Python Programming for the Absolute Beginner
Python: The Ultimate Beginner's Guide!
Automate the Boring Stuff with Python: Practical Programming for Total Beginners
Python: Learn Python in One Day and Learn It Well. Python for Beginners with Hands-on Project. (Learn Coding Fast with Hands-On Project Book 1)
Python Crash Course: A Hands-On, Project-Based Introduction to Programming
Learning Python
Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython
Python Cookbook
Python for Informatics: Exploring Information
Fluent Python
Python Playground: Geeky Projects for the Curious Programmer
Python in easy steps
Learn Python the Hard Way: A Very Simple Introduction to the Terrifyingly Beautiful World of Computers and Code (Zed Shaw's Hard Way)
Python: The Ultimate Beginners Guide: Start Coding Today
Programming the Raspberry Pi, Second Edition: Getting Started with Python
Data Science from Scratch: First Principles with Python

将代码分解为函数并获取我们只需要循环的所有页面，直到具有id pagnNextLink 的锚点不可见：

from bs4 import BeautifulSoup
import requests
from urlparse import urljoin
# from urllib.parse import  urljoin -> python 3


def parse(soup):
    yield [a["title"] for a in soup.select("a.a-link-normal.s-access-detail-page.a-text-normal")]

def get(term):
    params = {"url": "search-alias=",
              "field-keywords": term}

    with requests.Session() as s:
        head = {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36"}
        url = "https://www.amazon.co.uk/s"
        r = s.get(url, params=params)
        soup = BeautifulSoup(r.content, "lxml")
        nxt = soup.select_one("#pagnNextLink")
        while nxt:
            cont = requests.get(urljoin("https://www.amazon.co.uk/", nxt["href"]), headers=head)
            soup = BeautifulSoup(cont.content,"lxml")
            for t in parse(soup):
                print(t)
            nxt = soup.select_one("#pagnNextLink")

如果我们运行代码进行几次迭代：

In [5]: get("python")
['Python Machine Learning', 'Effective Python: 59 Specific Ways to Write Better Python (Effective Software Development)', 'Black Hat Python: Python Programming for Hackers and Pentesters', 'Doing Math with Python: Use Programming to Explore Algebra, Statistics, Calculus, and More!', 'Think Python: How to Think Like a Computer Scientist', 'Python Basics, Level 1 (Coding Club) (Coding Club, Level 1)', 'Python for Finance: Analyze Big Financial Data', 'Violent Python: A Cookbook for Hackers, Forensic Analysts, Penetration Testers and Security Engineers', "Python Essential Reference (Developer's Library)", 'Learn Web Scraping With Python In A Day: The Ultimate Crash Course to Learning the Basics of Web Scraping With Python In No Time (Python, Python ... Python Books, Python for Beginners)', 'Programming Python', 'QPython - Python on Android', 'Coding Club Python: Next Steps  Level 2', "Python: Programming, Master's Handbook; A TRUE Beginner's Guide! Problem Solving, Code, Data Science,  Data Structures & Algorithms (Code like a PRO ... engineering, r programming, iOS development)", 'Python: Complete Crash Course for Becoming an Expert in Python Programming', 'Coding Club Python: Building Big Apps Level 3']
['High Performance Python: Practical Performant Programming for Humans', '25ft Python No Spill Clean And Fill', 'Learning Python with Raspberry Pi', 'Web Scraping with Python: Collecting Data from the Modern Web', 'Invent Your Own Computer Games with Python, 3rd Edition', 'More Python Programming for the Absolute Beginner', 'Python for Kids: A Playful Introduction to Programming', "Monty Python's Life of Brian", 'Python 3 Object-oriented Programming - Second Edition', 'Introduction to Computation and Programming Using Python', 'Evolution of The Silly Walks T Shirt - Funny TV Ministry - Various Colours and Sizes XS - 3XL', "Hacking Secret Ciphers with Python: A beginner's guide to cryptography and computer programming with Python", 'Monty Python Fluxx', 'MASTER LOCK 8417DPRO Python Cable 1.80 m x 5 mm 2 Keys', "Learn Python: A beginner's guide book to programming python, learning the basics and start coding easily", 'Master Lock Python Disc Cylinder Key Adjustable Braided Steel Cable Lock, 10 x 1800 mm - Black']

In [6]: get("c programming")
['C Programming', 'C# 6.0 in a Nutshell: The Definitive Reference', 'PIC microcontrollers Programming in C with examples', 'C++: The Ultimate Crash Course to Learning the Basics of C++ In No Time (c plus plus, C++ for beginners, programming computer, how to program) (HTML, Javascript, ... Java, C++ Course, C++ Development Book 3)', 'Java: The Best Guide to Master Java Programming Fast (Java for Beginners, Java for Dummies, how to program, java app, java programming): Volume 2 (C Programming, HTML, Javascript)', 'A Book on C.: Programming in C.', "Learn C the Hard Way: Practical Exercises on the Computational Subjects You Keep Avoiding (Like C) (Zed Shaw's Hard Way Series)", 'C++: C++ and Hacking for dummies. A smart way to learn C plus plus and beginners guide to computer hacking: Volume 10 (C Programming, HTML, Javascript, Programming, Coding, CSS, Java, PHP)', 'Introduction to Algorithms', 'Programming: Computer Programming for Beginners: Learn the Basics of Java, SQL & C++ - 2. Edition (Coding, C Programming, Java Programming, SQL Programming, JavaScript, Python, PHP)', '21st Century C: C Tips from the New School', 'C For Dummies', 'Learn C# Programming Training DVD - Tutorial Video', 'GT01-C30R2-6P Programming PLC Cable 2.5M for Mitsubishi Melsec A970', 'Programming In C', 'Get Coding!: Learn HTML, CSS & JavaScript & build a website, app & game']
['Hewlett Packard [HP] Calculator Financial Platinum RPN Algebraic Programmable Ref HP12C PLATINUM', 'C: Easy C Programming for Beginners, Your Step-By-Step Guide To Learning C Programming (C Programming Series)', '4.9M RS232 DB9 F/M PLC Programming Cable Adapter White for Omron CQM1 C200HE HG', 'KOREAN COSMETICS, LG Household & Health Care_ SUM37, Secret Programming Eye C...', 'C++: C++ and Python. C++ for Beginners and Python for Dummies to Learn Fast (C Programming, Programming for beginners, c plus plus, programming ... Developers, Coding, CSS, Java, PHP)', '1:8 Brushless Combo BLC-150C Plus + Ripper 2000KV motor + programming Board', 'Lonely Planet Italian Phrasebook & Audio', 'Full Forgiveness - Let Go of Hurt & Offense With Guided Imagery, Self Hypnosis and Neuro-linguistic Programming (NLP)', 'Accelerated C++: Practical Programming by Example (C++ in Depth Series)', 'Gardena Water Computer C1060plus 1864-20', 'Learning To Build Apps For iPhone and iPad - Training DVD', 'Practical C Programming (A Nutshell handbook)', 'Prince Brat and the Whipping Boy', 'English: Practice Test Papers (Letts Key Stage 2 Success) (Letts Key Stage 1 Success)', 'Arabic For Dummies: Audio Set', 'The Actor and the Text (Applause Acting Series)']

你可以在解析时做任何你喜欢的事情，我只是拉出标题，以便我们很容易看到我们正在获得正确的数据。我还会考虑在请求之间添加一个休眠。

在主要的亚马逊网站上搜索一个术语（解析）

1 个答案: