Question

我正在从Yelp平台抓取酒店客户的评论，但我正努力从每个评论中获取文字。

我尝试硒 find_element_by_xpath ，但是没有用：

u = 'https://www.yelp.com/biz/fairmont-san-francisco-san-francisco?sort_by=rating_desc'
driver = webdriver.Chrome(executable_path = r'C:\chromedriver_win32\chromedriver.exe')#, options=options) 
driver.get(u)
sleep(5)

p = driver.find_elements_by_xpath('//p[@lang="eng"]')
for i in p:
    print(i.text)

我尝试 lxml 和 xpath ，但也无法正常工作：

u = 'https://www.yelp.com/biz/fairmont-san-francisco-san-francisco?sort_by=rating_desc'
driver = webdriver.Chrome(executable_path = r'C:\chromedriver_win32\chromedriver.exe')#, options=options) 
driver.get(u)
t = html.fromstring(driver.page_source)
sleep(5)

for i in t.xpath('//p[@lang="eng"]'):
    print(i)

除了我无法获得每个评论的文字之外，我找不到从所有评论页面（下一页）中刮取评论的解决方案。在此酒店示例中，共有53页分配了1060条评论。

Answer 1

如果不需要selenium，则可以尝试将requests与Beautifulsoup一起使用。

import requests
from bs4 import BeautifulSoup

url = 'https://www.yelp.com/biz/fairmont-san-francisco-san-francisco? 
sort_by=rating_desc'
page = requests.get(url)
soup = BeautifulSoup(page.text,'html.parser')
reviews = soup.find_all('p',attrs={'lang':'en'})
for review in reviews:
    print(review.text)

要查找所有页面上的所有评论，请尝试

import requests
from bs4 import BeautifulSoup

url = 'https://www.yelp.com/biz/fairmont-san-francisco-san-francisco?sort_by=rating_desc'
while url:
    page = requests.get(url)
    soup = BeautifulSoup(page.text,'html.parser')
    reviews = soup.find_all('p',attrs={'lang':'en'})
    for review in reviews:
        print(review.text)

    next_page = soup.find('a',{'class':'next'})
    if next_page:
        url = next_page['href']
    else:
        url = None

Answer 2

似乎可以与BeautifulSoup一起使用，好吧，我使用硒来获取页面源代码...查看代码

from selenium import webdriver
from bs4 import BeautifulSoup


u = 'https://www.yelp.com/biz/fairmont-san-francisco-san-francisco?sort_by=rating_desc'
driver = webdriver.Chrome(executable_path = r'C:\chromedriver_win32\chromedriver.exe')#, options=options) 
driver.get(u)


soup = BeautifulSoup(driver.page_source,'html.parser')
reviews = soup.find_all('p',attrs={'lang':'en'})
for review in reviews:
    print(review.text)

Answer 3

您的XPath找不到元素。如果您打印列表的长度，它将返回零。

尝试一下

p = driver.find_elements_by_xpath("//div[@class='review-list']/ul/li//p[@lang='en']")
print(len(p))
for i in p:
    print(i.text)

您可以在chrome开发工具中测试XPath或CSS选择器。

Answer 4

使用以下代码通过request，BeautifulSoup和pandas数据框获取所有页面的评论。您可以根据需要将数据导入csv。

from bs4 import BeautifulSoup
import re
import requests
import pandas as pd

headers = {'User-Agent':
               'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

page = "https://www.yelp.com/biz/fairmont-san-francisco-san-francisco?sort_by=rating_desc"
page_num =0
session = requests.Session()
review_comments = []
while True:
    pageTree = session.get(page, headers=headers)
    pageSoup = BeautifulSoup(pageTree.content, 'html.parser')
    posts = pageSoup.select('p[lang="en"]')
    for post in posts:
        comments = post.get_text().replace('\n', '').strip()
        #print(comments)
        review_comments.append(comments)

    if pageSoup.find("span", text=re.compile("Next")):
     page = "https://www.yelp.com/biz/fairmont-san-francisco-san-francisco?start={}&sort_by=rating_desc".format(page_num)
     page_num += 20
    else:
      break

df = pd.DataFrame({"Review_Comments": review_comments})
print(df)
df.to_csv('filename.csv')

输出：

                                        Review_Comments
0     I haven't fallen in love with a hotel in SF un...
1     This hotel was a dream come true to me!! My hu...
2     This place is THE BEST OF THE BEST! If you hav...
3     This is by far my favorite hotel in SF. Hands ...
4     Wow - what a hotel! Yes, it's expensive but th...
5     Came here for work and was very impressed. PRO...
6     With some of the greatest preservation of hist...
7     LOVE this hotel. Everyone I came into contact ...
8     The Fairmont is my favorite series of hotels b...
9     Exceptional and responsive service, comfortabl...
10    Everything about this hotel was phenomenal. Gr...
11    The Fairmont is a classic: a lovely Grand Dame...
12    Nicest hotel. They were very helpful and accom...
13    This is more of a review on just the penthouse...
14    Could not have asked for a better stay!  From ...
15    I must say that my sweet suite weekend getaway...
16    We had our company party in San Francisco and ...
17    The staff and views of the Fairmont made the c...
18    This hotel is fantastic. Very friendly staff a...
19    Of all the years that... I have been living in...
20    I haven't fallen in love with a hotel in SF un...
21    This hotel was a dream come true to me!! My hu...
22    This place is THE BEST OF THE BEST! If you hav...
23    This is by far my favorite hotel in SF. Hands ...
24    Wow - what a hotel! Yes, it's expensive but th...
25    Came here for work and was very impressed. PRO...
26    With some of the greatest preservation of hist...
27    LOVE this hotel. Everyone I came into contact ...
28    The Fairmont is my favorite series of hotels b...
29    Exceptional and responsive service, comfortabl...
...                                                 ...
1049  Before I went here, I read these Yelp reviews ...
1050  Over hyped. This place has small rooms, medioc...
1051  Stayed for three nights, very disappointing. T...
1052  After hearing such wonderful things about the ...
1053  Time warp!!! And holy shit. I've stayed at pre...
1054  My wedding night should have ended with my hus...
1055  My brother was a guest in the Fairmont hotel t...
1056  Our party of 12 people celebrated my husband's...
1057  very rude. The valet guy tells us we are disob...
1058  This review is for the tea at Fairmont.Attende...
1059  I wish I could choose less than one starI purc...
1060  The hotel is fine........nice, well kept. The ...
1061  OVERALL: I am NEVER staying in a Fairmont hote...
1062  Beautiful decor with grand history but the ser...
1063  I recently stayed here for a conference and ha...
1064  This place is not worth it. It doesn't matter ...
1065  I had what was possibly the worst service expe...
1066  After giving a poor review of the hotel, "Davi...
1067  My husband bought a groupon for afternoon tea ...
1068  Had a nice time here until the person I was wi...
1069  Eek!  Can we say "so yesterday"?  Unstylish de...
1070  If you like stuffy, old and tired decor, then ...
1071  No doubt this is a beautiful hotel.  It's got ...
1072  This hotel is really weak. The breakfast and d...
1073  The happy hour drink was awesome, though overp...
1074  last year, when i graduated from college my pa...
1075  I love to bring people here from out of town. ...
1076  Wow, I was shocked at how awful the service is...
1077  This is a feedback specifically on their after...
1078  I didn't stay here...but I did get a good enou...

[1079 rows x 1 columns]

Answer 5

仅使用Selenium来抓取hotel's客户评论，您需要为visibility_of_all_elements_located()引入 WebDriverWait ，并且您可以使用以下任一{{3 }}：

使用CSS_SELECTOR：

print([element.text for element in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.review-content p")))])

使用XPATH：

print([element.text for element in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='review-content']//p")))])

控制台输出：

["I haven't fallen in love with a hotel in SF until I met the Fairmont. Situated atop a hilly walk from Union Square, the Fairmont is a beacon of elegance and architectural classiness of a different time. Guess what? It IS the original Fairmont, the one that started it all! So it's no surprise that the lobby is absolutely breathtaking; embellished and polished. You feel like a king or queen just walking in. The service at the Fairmont is great; the staff are all friendly and quick to respond to requests such as bringing a toothbrush. The food in the hotel is also pretty great; a fun fact that is that they have their own beehives to help conserve and promote bees and they make honey beer that you can get in the restaurant. And honey ice cream that you can and should definitely room service. In your extra comfy bed. If you're feeling fancy or are lucky enough to get upgraded, the views of SF are stellar. And please don't forget to check out the tiki bar on the bottom floor! A truly unforgettable bar with crazy decor and water features- but that's a review of its own. Highly highly recommend!", 'This hotel was a dream come true to me!! My husband used pretty much all the points he accumulated for many many months through Chase Ultimate reward to book two nights here. He knows that it was my dream since little when I lived in SF and passed by this hotel all the time, wondering when will I ever have a chance to live like a princess. Well, thanks for his thoughtfulness, I sure felt like a princess the moment I stepped in! The staff were nice and the room that we got was spectacular! Maybe because they saw us with three kids, they put us to a corner room, and that looked like a suite, with three windows facing downtown and bay bridge! They gave us a metal crib as well, although it looks like it belongs to a museum but very sturdy. The closet is literally a room that can fit a full size bed! You know how much that room can cost in SF just by itself??? My husband was lucky enough to see a super star--Olivia Wilde when he was in the lobby too. He pretended it was normal but I would have screamed if I saw her myself. It was such a wonderful stay and at the end I bid farewell to this gorgeous "palace" with a Royal wave.', "This place is THE BEST OF THE BEST! If you have the money to spend this hotel is worth it all! My husband and I stayed her for our anniversary and we didn't want to leave! The bed was super comfortable along with the sheets. We ordered a dessert later in the evening and the crepe Burke was to die for! They do charge a delivery fee for bring it up to your room but we were ok with that! I absolutely loved all the history of this hotel and hope to stay again soon!", "This is by far my favorite hotel in SF. Hands down. Now truth is im going to nerd out, why? Well, i think you will understand....\n\nIf there was such thing as a 7 Wonders of The World for Hotels, The Fairmont Hotel in San Francisco would be on this list. What makes this landmark is her story. On April 18, the city of San Francisco was leveled, by one of the strongest earthquakes to hit the Western United States. It didn't end there after what we could call now is the 1906 earthquake, fires irrupted throughout the city and eventually it reached Nob Hill (which is where she sits). Her birth was met with a rude awakening.\n\n1906 was truly a defining moment for all of San Francisco, but for her Majesty, this is where her real story begins. Millionaire infidelity and murder was associated with the rebuilding process, but a local architect Julia Morgan came to the rescue. Ms. Morgan is first woman graduate of the prestigious Ecole des Beaux Arts in Paris, France. Her love for architecture and vision of this magnificent Italian curtain is truly a masterpiece. Believe it or not, Ms. Morgan made a promise to finished this project in 365 days. Days and nights her crew worked through it together and define all odds. On April 18, 1907 The Fairmont Hotel risen from her ashes and open her doors for business.\n\nThrough out the years, many memories were created within those marble floors, high ceilings, and double wide hallways. In 1945 for 11 weeks, The United Nation occupied these walls to eventually change the world and what now has become the the birthplace for The United Nation Charter. If you ask historians what is the most important event or best feature of her Crown Jewels, many would mention the famous swimming pool, entertainers, or even politicians. If you had to ask me, hands down its the rich history resting on the 8th floor.\n\nThis penthouse was feature in the blockbuster action movie THE ROCK starring Sean Connery and Nicolas Cage. Never mind the beautiful iconic downtown views of SF that awaits you out on the balcony. A Moroccan inspired pool table room which by the way has over 100,000 tiles. A dining room design for up to 50 of your guest to enjoy a Michelin star dinner. The constellation is embedded in the double story high oval ceiling, which below is a 2 level library. Memories and history was created on this very floor.. As I was lucky enough to walk throughout this floor, I had goosebumps thinking of the decision, and actions of individuals who occupied these very walls. Now If only those walls could talk.", "Wow - what a hotel! Yes, it's expensive but this is one of the nicer hotels I've ever stayed in. The rooms are gorgeous with very comfortable beds, great toiletries, the staff is beyond friendly and helpful and the gym is very nice as well.\n\nThe hotel is located in a great part of San Francisco and you can hop on the trolley right in front of the hotel too!", "Came here for work and was very impressed.\n\nPROS:\n+ LE LABO IN THE BATHROOM. This is not a drill. They have very nice soaps/lotions for you to - ahem - use while you're there.\n+ Tonga Room is right underneath the hotel, so if you want a nightcap and to be entertained look no further\n+ Really great service\n+ Beautiful hotel overall\n+ Pretty quiet considering the location\n\nHonestly the only downside I could think of is that it's at the top of a steep hill and for people wanting to explore downtown it is a little bit removed.", 'With some of the greatest preservation of history, attention to detail and high end customer service.  This landmark hotel has seen it all.  By the way you can see it all from your room too.  My room was on the 21st floor facing the city and east bay.  I could see the Sales force building and far south west toward Daily City and the ocean.  I loved my view so much, I left the windows wide open all night so I could go to bed with the legendary view of the city and wake the same way.  We had a function in the crown room on top the tower with some of the best 360 city views anywhere in SF.  Go explore the movie, music, and TV history of the hotel as well as the history surrounding the sighing and preparation to sign the United Nations Charter.  Anyone who is anyone in politics, world leadership, or power has stayed at this hotel.', 'LOVE this hotel. Everyone I came into contact with was over and beyond helpful and sweet! The rooms are comfy and roomy, the gym is a little strangely set up but has everything you need and the coffee shop has great options! Will definitely be staying here again', 'The Fairmont is my favorite series of hotels by far, I\'m hesitant to use the word "chain" :) because of the thoughtfulness and uniqueness each individual location provides. The staff are incredibly friendly, and the picture I\'m posting is a bit unusual, while I could show off the gorgeous steak dinner I had, or the watermelon and strawberry mint salad at the (extremely overpriced) breakfast buffet, what keeps this review 5 stars for me is the level of detail the housekeeping maintains! I feel like a princess here, even though I don\'t visually look like the typical clientele.\n\nThe other guests at the hotel can be quite rude, making out in front of the elevator when you need it, or shoving their bags at me to carry them (so weird) but the staff are delightful despite that, god bless them. For some reason every guest thinks they\'re the only ones that have ever stayed here. Oh well.\n\nI highly recommend the food, great gym with a sauna AND steam room free of charge as a guest, and super easy and polite check in. Very comfortable rooms. The pillows (or at least mine) vary in firmness so you can embark on a Goldilocks situation to get comfortable! Yay. Your stay can add up here $$$ but you get what you pay for!', 'Exceptional and responsive service, comfortable beds, modern bathrooms, classic decor and beautiful architecture.\n\nBest:\n-location\n-concierge/guest services staff\n-room service\n-lounge and bar\n\nAreas for improvement:\n-long lines at check in\n\nIf I am able to stay at the Fairmont, I always do. Even the walks up the steep hill to the hotel are worth it for the level of service, luxury, and sophistication at this classic hotel. The staff are never uppity and are kindly focused on making my stay as personalized as possible and check in regularly without being annoying.', 'Everything about this hotel was phenomenal. Great customer service all around, everyone was very helpful. The Laurel Court Bar and Restaurant had great food and drinks! The rooms were very clean and homey. The guys over at Valet were very nice as well. Definitely worth staying at! Not to mention the location and views are beautiful!', 'The Fairmont is a classic: a lovely Grand Dame perched on the hill, promising adventure and romance to all who enter, strains of Tony Bennett singing, "I left my heart..."\nThe lobby and lounge are gorgeous spaces to stop for a glass of champagne and a moment of rest. Put this iconic hotel on your list of must-see stops in SF!', 'Nicest hotel. They were very helpful and accomidating during the fires in Santa Rosa. They really made things nice for everyone during that time. My wife and kids felt safe staying there and at home in their hotel during a difficult time. Thank you.', "This is more of a review on just the penthouse suite, it's such an amazing place but not exactly something you might yelp about.\n\nIf it were just another room in a nice hotel or even a suite  in a very classy top end place, this may be something you'd book. But chances are that you'll never book this suit or even step foot in this historic and, very, well, presidential or stately luxury apartment with butler, chef and house dedicated services.\n\nSo instead of reviewing it to entice you into booking, i'm just going to totally brag about the party of a lifetime a close friend had. and throw in a few super cool historical facts.\n\nFirst of all the bare facts are impressive! 6000 square feet, the entire eighth floor of the hotel is this stunning roaring 20's style decedent three bedroom penthouse. It boasts a very large living room with a grand piano, a room that easily accommodates large ultra-exclusive parties. For the wealthy or entrepreneurial gastro-entertainer the full professional kitchen will easily support the full and grand formal dinning room.\n\nNot here for a business meeting, negotiate a new world treaty or plan your global domination?   You just want to enjoy and play? The R&R pool room or city views from balcony have you covered. The Balcony views are some of the best in the city. Enjoy the evening with cocktails and watch the fog roll in or out or more of the same for breakfast with your private space.  Maybe that's what JFK and Marilyn where hoping for when they stayed here, a private moment.\n\nHere is a link to the hotel's info, totally worth a look.  \nbit.ly/2C8m9pE", 'Could not have asked for a better stay!  From check in to housekeeping, I\nwas always greeted with a smile. Blanca at the front desk was kind and accommodating with our room and requests.\nSecond stay at the newer tower with Spectacular views of Alcatraz, Golden Gate Bridge, and Coit Tower.\nVery clean, nice new carpeting in hallways, great service and nice breakfast Buffett. That adds to a wonderful location in SF., especially for walking around. Tonga room is a must!  Loved my recent stay here!  Accor Hotels is doing a nice job!', "I must say that my sweet suite weekend getaway was FAVULOSO!!! It was full of Bliss, Luster and Wonder.\n\nYou'll take note that there are no pics to observe. YOU MUST have the experience for yourself. YOU MUST enter, see it, feel it, touch it, try it.\n\nThe Fairmont San Francisco is Opulent, Beautiful, Luxurious, Indulgent and I LOVE IT!!! Its old world charm is just amazing.   Service is impeccable!!! BTW, I stayed in the Constantine Suite :D\n\nLet the Fairmont envelop you, swoon you. She will!", 'We had our company party in San Francisco and blocked a group of rooms at The Fairmont for the night and some people stayed the weekend.  The Fairmont was amazing to work with from arranging the block to our arrival - everything was top notch service and exceeded our expectations.  Tommy Fullove arranged our block and even though we had to make some changes along the way - he was always more than accommodating.  The staff upon arrival were gracious and welcoming.  The Fairmont was beautifully decorated for Christmas, we were blown away by the details.  After our dinner, we all danced late into the night at the Tonga Room. Perfect stay.  Thank you Fairmont', "The staff and views of the Fairmont made the celebration of my mom's 90th Birthday a wonderfully memorable event.  The service was top notch and professional, yet all the staff were friendly, patient, and kind.   From the moment we arrived in the valet to the moment we left, we were treated like longtime friends and provided excellent service!!  One can tell that the philosophy and training of the staff is of utmost importance to the hotel.\n\nThe furnishings, building, and even bathrooms were all well maintained, new, and kept clean.  Carpet, furniture, and paint were all clean.\n\nMy mom was born in the year of the depression and she lived below the poverty level through college.  But she always had drive and class.  She would visit the lobby of the Fairmont in her 20's and wonder what kind of people stayed there and never imagined in her wildest dreams that she would ever be able to do so.  We have now travelled the world and have experienced some grand hotels.\n\nShe literally cried when we pulled up to the entrance and the doorman opened her door and kindly and enthusiastically welcomed her.  The whole way to the room, she was stating over and over, how beautiful the place was and how she couldn't believe she was going to actually stay there.\n\nBut the most amazing part was THE VIEW!!!  When we opened the door, my sister's jaw literally dropped!!!  We stayed in a Golden Gate Suite, which is a corner room.  It had a view all the way from the Golden Gate Bridge, all the way around past the Bay Bridge to Portero Hill.  My mom says she will never forget the view!!  \n\nEven upon our departure, the staff were accommodating, pleasant, patient and friendly.  They knew it was my mom's 90th and they added extra touches to make it special.  A stay at the Fairmont will be a wonderful memory of a lifetime (90-years plus!!!).  Thank you Fairmont Hotel!", 'This hotel is fantastic. Very friendly staff and wonderful service. There was a little hiccup and they want above and beyond to make it right!', "Of all the years that... I have been living in the City Of San Francisco this is the one hotel that I have been to... Eventhough I have never stayed at this Fairmont Hotel which is the most luxious hotel in the City Of San Francisco!!!\n\nThe reason why...  I wanted to go and visit this 5 Star Fairmont Hotel Of San Francisco is a because a co worker had told me that this luxious hotel had one of the most nicest Christmas decorations!!! So from there my co worker told me go and visit the Fairmont Hotel on your day off from work!!!\n\nWhen that day came... I just had to go and visit the Fairmont Hotel which was my very 1st time ever!!! As soon as I walked through those revolving fancy doors, as I got inside to the  Fairmont Hotel... I was really fascinated to see so much lighted up Christmas Decorations through out the whole Fairmont Hotel!!!\n\nThe one thing that... I really like most of the lighted up Christmas Decorations is the Moet Champagne bottle set up, Ginger Bread House and also the minature train that are running by at the display glass window of the Ginger Bread house!!!\n\nAfter seeing the entire lighted up Christmas Decorations of the Fairmont Hotel Of San Francisco Ca for the very 1st time... All I can say is that this Fairmont Hotel has one of the most beautiful lighted up Christmas Decorations ever!!! And one more time I just have say this again in what I really like most of the Fairmont Hotel Christmas decorations is the Moet Champagne Bottle set up, Ginger Bread House...  And the minature train running by the display glass window of the Ginger Bread House!!!\n\nIf your a out of towner or a local like me go and visit the Fairmont Hotel Of San Francisco Ca... During the Happy Holiday Season for it's most beautiful lighted up Christmas Decorations through out the whole Fairmont Hotel!!!"]

注意：您必须添加以下导入：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

搜寻酒店评论的隐藏文字

5 个答案: