Python BeautifulSoup并请求网络抓取

Question

我正在尝试从以下网站获取生成的短语作为字符串。 https://randomwordgenerator.com/phrase.php 我已经浏览了html，并且相信已经确定了该短语在html结构中的位置。

这是附近的html。

<div id="loading_result" class="small-img-results">
    <ol id="result">
        <li> == $0
            <div>
                <span class="support-phrase">Generated Phrase </span>
                <span class="subtle">...</span>
            </div>
        </li>
    </ol>
</div>

在这种情况下，我要输入“生成的短语”

这是我目前正在做的

pageLink = "https://randomwordgenerator.com/phrase.php"
pageResponse = requests.get(pageLink, timeout=5)
pageContent = BeautifulSoup(pageResponse.content, "html.parser")

span = pageContent.find_all("span", {"class": "support-phrase"})

问题在于此运行之后的span值为空列表。 我刚接触过漂亮的汤，所以这可能是一个非常简单的问题，但是我还没有发现任何特别明确的解决方案。

谢谢！

编辑： 我现在想知道问题是否在于我要查找的特定跨度嵌套在主体的一系列div中。

Answer 1

您需要硒才能获得页面上显示的准确值。这样做的原因是，虽然总短语（134）是从xhr（https://randomwordgenerator.com/json/phrases.json）的数组中返回的，而xhr（{{3}}）返回的是json；从该数组中选择的实际索引（例如函数randomiseUniqueNumbers），该数组中各项的顺序（例如Array.prototype.shuffle = function()）以及处理我认为可能发生的冲突的规则（例如{{1 }}）都在js文件function getResults中定义。其中的前两个至少在数组大小的边界之间使用随机数生成。没有种子，尽管我认为您可以编写自己的版本，但不能保证获得与页面上相同的结果-实际上，您更有可能获得不同的词组。

硒概述

https://randomwordgenerator.com/assets/js-compress/f0351bd03da6dab13a24355fa7deeabd.js?v=1577899960:formatted

对于单个短语，只需使用

from selenium import webdriver

d = webdriver.Chrome()
d.get('https://randomwordgenerator.com/phrase.php')
print([i.text for i in d.find_elements_by_css_selector('.support-phrase')])

Answer 2

如果转到 XHR 下的 NetWork选项卡，您将获得 json 网址，该网址返回短语和json format中的细微之处。您无需 selenium和Beautifulsoup

即可实现目标

https://randomwordgenerator.com/json/phrases.json

---

此处代码：

import requests
url='https://randomwordgenerator.com/json/phrases.json'
response=requests.get(url).json()
print(response['data'])
#Print phrase
print(response['data'][0]['phrase'])
#Print meaning
print(response['data'][0]['meaning'])

---

此处输出：

[{'meaning': 'Being angry about something that happened in the past.', 'phrase': 'A Chip on Your Shoulder'}, {'meaning': 'Something that is extremely common.', 'phrase': 'A Dime a Dozen'}, {'meaning': "It's easy for a fool to lose his/her money.", 'phrase': 'A Fool and His Money are Soon Parted'}, {'meaning': 'A task that is simple to accomplish.', 'phrase': 'A Piece of Cake'}, {'meaning': 'Something that is extremely expensive.', 'phrase': 'An Arm and a Leg'}, {'meaning': 'When something is incomprehensible due to complexity; unintelligble.', 'phrase': 'All Greek To Me'}, {'meaning': 'To go back to the beginning; back to the drawing board.', 'phrase': 'Back to Square One'}, {'meaning': 'Starting over again on a new design from a previously failed attempt.', 'phrase': 'Back To the Drawing Board'}, {'meaning': 'To make a wrong assumption about something.', 'phrase': 'Barking Up The Wrong Tree'}, {'meaning': 'To bring up an issue that has already been resolved.', 'phrase': 'Beating a Dead Horse'}, {'meaning': 'Someone who is beating around the bush is someone who avoids the main point.', 'phrase': 'Beating Around the Bush'}, {'meaning': 'Being faced with two difficult choices.', 'phrase': 'Between a Rock and a Hard Place'}, {'meaning': 'People tend to associate with others who share similar interests or values.', 'phrase': 'Birds of a Feather Flock Together'}, {'meaning': 'Breaking down a social stiffness.', 'phrase': 'Break The Ice'}, {'meaning': "To ruin someone's happy moment.", 'phrase': 'Burst Your Bubble'}, {'meaning': 'Coming close to a successful outcome only to fall short at the end.', 'phrase': 'Close But No Cigar'}, {'meaning': "It's useless to worry about things that\xa0 already happened and cannot be changed.", 'phrase': 'Cry Over Spilt Milk'}, {'meaning': 'Someone that calls for help when it is not needed. Someone who is lying.', 'phrase': 'Cry Wolf'}, {'meaning': 'A cup of joe is an American nickname for a cup of coffee.', 'phrase': 'Cup Of Joe'}, {'meaning': 'Typically said to indicate that any further investigation into a situation may lead to harm.', 'phrase': 'Curiosity Killed The Cat'}, {'meaning': 'To cut the mustard is to meet a required standard, or to meet expectations.', 'phrase': 'Cut The Mustard'}, {'meaning': 'To get to the point, leaving out all of the unnecessary details.', 'phrase': 'Cut To The Chase'}, {'meaning': 'Do not rely on something you are not sure of.', 'phrase': "Don't Count Your Chickens Before They Hatch"}, {'meaning': 'When you receive a gift from someone, do not be ungrateful.', 'phrase': "Don't Look a Gift Horse In The Mouth"}, {'meaning': '(1) A term used in a boxing. (2) Someone who has become incapacitated.', 'phrase': 'Down And Out'}, {'meaning': 'Someone or something that looks to be defeated, or nearly so.', 'phrase': 'Down For The Count'}, {'meaning': 'Practical or humble; unpretentious.', 'phrase': 'Down To Earth'}, {'meaning': 'A tense situation where the outcome is decided only in the last few seconds.', 'phrase': 'Down To The Wire'}, {'meaning': 'Failing to recall a memory. Unable to remember something.', 'phrase': 'Drawing a Blank'}, {'meaning': 'To greatly frustrate someone. To drive someone crazy, insane, bonkers, or bananas.', 'phrase': 'Drive Me Nuts'}, {'meaning': 'To fall down ill or to die in large numbers.', 'phrase': 'Dropping Like Flies'}, {'meaning': 'Something that is easy.', 'phrase': 'Easy As Pie'}, {'meaning': 'Having confidence in a specific outcome; being almost sure about something.', 'phrase': 'Eat My Hat'}, {'meaning': 'Ignoring a large, obvious problem or failing to address an issue that stands out in a major way.', 'phrase': 'Elephant in the Room'}, {'meaning': 'Something that is all over.', 'phrase': 'Elvis Has Left The Building'}, {'meaning': 'To be optimistic, even in difficullt times.', 'phrase': 'Every Cloud Has a Silver Lining'}, {'meaning': 'Including nearly everything possible.', 'phrase': 'Everything But The Kitchen Sink'}, {'meaning': 'To retaliate with an attack that is similar to the attack used against you.', 'phrase': 'Fight Fire With Fire'}, {'meaning': 'Someone being in a situation that they are unfamiliar or unsuited for.', 'phrase': 'Fish Out Of Water'}, {'meaning': 'Being fit as a fiddle means to be in perfect health.', 'phrase': 'Fit as a Fiddle'}, {'meaning': 'A type of bazaar where inexpensive goods are sold or bartered.', 'phrase': 'Flea Market'}, {'meaning': 'To be enraged and show it.', 'phrase': 'Foaming At The Mouth'}, {'meaning': 'Iron pyrities is a worthless mineral that resembles gold.', 'phrase': "Fool's Gold"}, {'meaning': "It's better to teach a person how to do something than to do that something for them.", 'phrase': 'Give a Man a Fish'}, {'meaning': 'To risk it all, even if it means losing everything. To go all out.', 'phrase': 'Go For Broke'}, {'meaning': 'Putting yourself in a risky situation in order to help someone; or to hazard a guess.', 'phrase': 'Go Out On a Limb'}, {'meaning': 'A smugly virtuous person.', 'phrase': 'Goody Two-Shoes'}, {'meaning': 'Very fast or quick.', 'phrase': 'Greased Lightning'}, {'meaning': "Anything that's easy or has no difficulty; something that is a certainty.", 'phrase': 'Hands Down'}, {'meaning': 'The state of being happy; feeling delighted.', 'phrase': 'Happy as a Clam'}, {'meaning': "Something that's difficult to accept.", 'phrase': 'Hard Pill to Swallow'}, {'meaning': 'Falling deeply in love with another person.', 'phrase': 'Head Over Heels'}, {'meaning': 'Used as an advanced warning. To become keenly aware.', 'phrase': 'Heads Up'}, {'meaning': 'A shout of agreement, or to draw attention to a speaker.', 'phrase': 'Hear, Hear'}, {'meaning': 'To be left behind; abandoned. Being in a helpless situation without a way to recover.', 'phrase': 'High And Dry'}, {'meaning': 'A boxing term. Also often used to refer to inappropriate words, or comments that are too personal.', 'phrase': 'Hit Below The Belt'}, {'meaning': 'A feeling that something is not quite right, or awry.', 'phrase': 'I Smell a Rat'}, {'meaning': 'One should discontinue with a task if they are unable to cope with it due to pressure.', 'phrase': "If You Can't Stand the Heat, Get Out of the Kitchen"}, {'meaning': 'Being in a difficult predicament; a mess; an undesirable situation.', 'phrase': 'In a Pickle'}, {'meaning': 'Losing money. Being in debt.', 'phrase': 'In the Red'}, {'meaning': 'Failing to meet expectations; not being as good as people say.', 'phrase': "It's Not All It's Cracked Up To Be"}, {'meaning': "A task that's easy to accomplish, a thing lacking complexity.", 'phrase': "It's Not Brain Surgery"}, {'meaning': 'Having suitable skill in multiple things, but not being an expert in any of them.', 'phrase': 'Jack of All Trades Master of None'}, {'meaning': 'Being in a dangerous or very deadly situation.', 'phrase': 'Jaws of Death'}, {'meaning': 'Usually this references a tool used by rescuers when they pry or cut open a car to save the occupant.', 'phrase': 'Jaws of Life'}, {'meaning': 'For a ruse or trick to be discovered; to be caught.', 'phrase': 'Jig Is Up'}, {'meaning': 'Something that occurs too early before preparations are ready. Starting too soon.', 'phrase': 'Jumping the Gun'}, {'meaning': 'To keep going, pressing forward; never stopping.', 'phrase': "Keep On Truckin'"}, {'meaning': 'To be watchful; paying careful attention to something.', 'phrase': 'Keep Your Eyes Peeled'}, {'meaning': 'Keeping calm. Usually said by someone who is trying to avoid making others upset.', 'phrase': 'Keep Your Shirt On'}, {'meaning': 'To be taken by surprise.', 'phrase': 'Knock Your Socks Off'}, {'meaning': 'Having a familiarity or understanding of how something works.', 'phrase': 'Know the Ropes'}, {'meaning': 'Getting sincere about something; applying oneself seriously to a job.', 'phrase': 'Knuckle Down'}, {'meaning': "Permission to start, or it could mean 'go faster!'", 'phrase': 'Let Her Rip'}, {'meaning': 'To go at a quick pace; no delaying!', 'phrase': 'Lickety Split'}, {'meaning': "Resembling one's parents in terms of appearance or behavior.", 'phrase': 'Like Father Like Son'}, {'meaning': 'Old in age. Mainly used when referring to people or horses.', 'phrase': 'Long In The Tooth'}, {'meaning': 'A pair of people who have a shared love for each other.', 'phrase': 'Love Birds'}, {'meaning': 'The affectionate stuff that people do when they are in love, such as kissing and hugging.', 'phrase': 'Lovey Dovey'}, {'meaning': 'A person who does not speak a great deal; someone who talks with as few words as possible.', 'phrase': 'Man of Few Words'}, {'meaning': "Suggests that money is a resource that must be earned and is not one that's easily acquired.", 'phrase': "Money Doesn't Grow On Trees"}, {'meaning': 'One who escalates small things and turns them into big problems.', 'phrase': 'Mountain Out of a Molehill'}, {'meaning': 'Delicious; something that looks or tastes appetizing.', 'phrase': 'Mouth-watering'}, {'meaning': 'Someone or something that one finds to be agreeable or delightful.', 'phrase': 'My Cup of Tea'}, {'meaning': 'Something that is impossible or extremely difficult to find, especially because the area you have to search is too large.', 'phrase': 'Needle In a Haystack'}, {'meaning': 'Anything that requires minimal brain activity to accomplish.', 'phrase': 'No-Brainer'}, {'meaning': 'Finishing a task without making any excuses.', 'phrase': 'No Ifs, Ands, or Buts'}, {'meaning': "Someone who isn't witty or sharp, but rather, they are ignorant, unintelligent, or senseless.", 'phrase': 'Not the Sharpest Tool in the Shed'}, {'meaning': 'A person that is crazy or behaving in idiotic ways', 'phrase': "Off One's Base"}, {'meaning': 'Having strong feelings of happiness or satisfaction.\xa0', 'phrase': 'On Cloud Nine'}, {'meaning': 'Being in a situation that looks to be hopeless!', 'phrase': 'On the Ropes'}, {'meaning': 'Thinking alike or understanding something in a similar way with others.', 'phrase': 'On the Same Page'}, {'meaning': 'What you would expect to happen; something normal or common.', 'phrase': 'Par For the Course'}, {'meaning': 'Said when things are about to get serious.', 'phrase': 'Playing For Keeps'}, {'meaning': 'Pretending to be dead, or to be deceitful about something.', 'phrase': 'Playing Possum'}, {'meaning': 'A situation that has gotten way more serious or interesting due to recent complexities or developments.', 'phrase': 'Plot Thickens\xa0- The'}, {'meaning': 'Making fun of something or someone; ridicule.', 'phrase': 'Poke Fun At'}, {'meaning': 'Asking someone to be quiet or to shut up.', 'phrase': 'Put a Sock In It'}, {'meaning': 'Spending time with another to strengthen the relationship.', 'phrase': 'Quality Time'}, {'meaning': "Things that are fixed with great speed, but as a result, it's probably not going to work very well.", 'phrase': 'Quick and Dirty'}, {'meaning': 'Performing an action with the greatest of haste.', 'phrase': 'Quick On the Draw'}, {'meaning': "To spoil someone's fun or plans; ruining a pleasurable moment", 'phrase': 'Rain on Your Parade'}, {'meaning': 'When it is raining heavily.', 'phrase': 'Raining Cats and Dogs'}, {'meaning': "Often said by the winner in poker, as the others 'weep' over the loss.", 'phrase': "Read 'Em and Weep"}, {'meaning': 'A cheer people yell, usually at rodeos when cowboys are clinging to the backs of untamed horses.', 'phrase': 'Ride Him, Cowboy!'}, {'meaning': 'Immediately, done in a hurry; without delay.', 'phrase': 'Right Off the Bat'}, {'meaning': 'Right from the beginning; to do something from the start.', 'phrase': 'Right Out of the Gate'}, {'meaning': 'Recalling a memory; causing a person to remember something or someone.', 'phrase': 'Ring Any Bells?'}, {'meaning': 'To tolerate or endure through the unexpected mishappenings you may encounter from time to time.', 'phrase': 'Roll With the Punches'}, {'meaning': 'Getting away freely from custody, punishment, or any type of risky situation.', 'phrase': 'Scot-free'}, {'meaning': 'Getting the bad end of a deal, or receiving the least desirable outcome from something.', 'phrase': 'Short End of the Stick'}, {'meaning': 'An attempt that has little chance for success.', 'phrase': 'Shot In the Dark'}, {'meaning': 'A person, usually one who is behaving badly.', 'phrase': 'Son of a Gun'}, {'meaning': 'Giving something your all.', 'phrase': 'Swinging For the Fences'}, {'meaning': 'Supporting what you say, not just with words, but also through action or evidence.', 'phrase': 'Talk the Talk'}, {'meaning': 'To not work alone, but rather, together with others in order to achieve a certain goal.', 'phrase': "There's No I in Team"}, {'meaning': 'Giving up; to surrender.', 'phrase': 'Throw In the Towel'}, {'meaning': "High quality, exceptional; something that's very valuable.", 'phrase': 'Top Drawer'}, {'meaning': 'To remain resillient even in hard times; enduring.', 'phrase': 'Tough It Out'}, {'meaning': 'It can refer to the popular rope pulling game or it can mean a struggle for authority.', 'phrase': 'Tug of War'}, {'meaning': 'Two things have been completed, but there is one more that has yet to be finished.', 'phrase': 'Two Down, One to Go'}, {'meaning': 'One who may seem plain at first in appearance or capability, but later turns out to be beautiful or great.', 'phrase': 'Ugly Duckling'}, {'meaning': 'Not feeling well, in health or mood.', 'phrase': 'Under the Weather'}, {'meaning': 'Missing something that should be really obvious.', 'phrase': 'Under Your Nose'}, {'meaning': 'Angry; being roused to the point that you are ready to fight.', 'phrase': 'Up In Arms'}, {'meaning': "An occurance of sorts that brings a problem to somebody's attention and they realize it needs fixing.", 'phrase': 'Wake Up Call'}, {'meaning': 'Things that go up must eventually return to the earth due to gravity.', 'phrase': 'What Goes Up Must Come Down'}, {'meaning': 'A rhetorical question used by a person who feels they are being given less consideration than someone else.', 'phrase': 'What Am I, Chopped Liver?'}, {'meaning': 'When something is about to begin, get serious, or put to the test.', 'phrase': 'When the Rubber Hits the Road'}, {'meaning': 'Futilely pursuing something that will never be attainable.', 'phrase': 'Wild Goose Chase'}, {'meaning': 'Nonviolent; someone who is mild or gentle.', 'phrase': "Wouldn't Harm a Fly"}, {'meaning': "A way to notify a person that what they're saying is predictable or boring.", 'phrase': 'Yada Yada'}, {'meaning': "Don't judge someone or something only by the outward appearance.", 'phrase': "You Can't Judge a Book By Its Cover"}, {'meaning': 'It can be challenging to teach a person something new.', 'phrase': "You Can't Teach an Old Dog New Tricks"}]

短语

A Chip on Your Shoulder

含义

Being angry about something that happened in the past.

Answer 3

from lxml import html
import requests

page = requests.get('https://randomwordgenerator.com/phrase.php')
# get xpath of the string and at end '/text()'
text_xpath= '/html/body/div[2]/div/div[3]/div/div/div[1]/h1/text()' 
tree = html.fromstring(page.content)
page_text =tree.xpath(text_xpath)
print(page_text)

Answer 4

Apple页的随机词组通过XHR加载。因此，您无法使用BeautifulSoup查找词组。使用XHR请求获取短语列表。

import requests

pageLink = "https://randomwordgenerator.com/json/phrases.json"

pageResponse = requests.get(pageLink, timeout=5)
print pageResponse.status_code, pageResponse.content

输出：

{"data":[ {"phrase":"A Chip on Your Shoulder","meaning":"Being angry about something that happened in the past."}, {"phrase":"A Dime a Dozen","meaning":"Something that is extremely common."}, {"phrase":"A Fool and His Money are Soon Parted","meaning":"It's easy for a fool to lose his/her money."}, ... ]}