如何使用Selenium + Python加载动态内容

时间:2018-11-28 09:33:59

标签: python selenium selenium-webdriver web-scraping webdriverwait

在Quora用户个人资料页面上,我想通过单击“更多”来显示每个用户答案的​​全文,但我无法获得“更多”元素列表,而是得到了一个空列表,怎么办?

这是我的代码:

float

以下是用户个人资料的示例: https://www.quora.com/profile/Ashley-Isabella-3

3 个答案:

答案 0 :(得分:2)

那是因为您使用了错误的CSS选择器。 尝试:

more_buttons = browser.find_elements_by_class_name('ui_qtext_more_link')

通过在常规浏览器中右键单击元素并单击“检查”,可以找到正确的属性。

答案 1 :(得分:2)

实际上,您位于以下链接之后(单击时),因此您可以仅收集链接,然后对.get使用循环。使用return [...document.querySelectorAll ]语法

from selenium import webdriver

d = webdriver.Chrome()
d.get("https://www.quora.com/profile/Ashley-Isabella-3")
links = d.execute_script("return [...document.querySelectorAll('.ui_qtext_more_link')].map(link => link['href'])")
info = []
for link in links:
    d.get(link)
    pageInfo = d.execute_script("return [...document.querySelectorAll('.ui_qtext_para')].map(link => link.innerText)")
    info.append(pageInfo)
print(info)

答案 2 :(得分:1)

在Quora用户个人资料页面上,单击文本为(更多)的链接并检索用户答案,您可以使用以下解决方案:

  • 代码块:

    from selenium import webdriver
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    
    driver=webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
    driver.get("https://www.quora.com/profile/Ashley-Isabella-3")
    more_buttons = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.ui_qtext_more_link")))
    for more_button in more_buttons:
        more_button.click()
    my_text_elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.ui_qtext_expanded>span.ui_qtext_rendered_qtext>p>b")))
    for my_text_element in my_text_elements:
        print(my_text_element.text)
    
  • 控制台输出:

    11 mistakes you make when trying to lose belly fat
    You could be doing the completely wrong exercise.
    1. You're not getting enough sleep
    2. You’re doing the wrong type of workout
    3. You’re consuming too much sugar
    4. You're not eating enough protein
    5. You’re feeling stressed or anxious
    6. You’re expecting a quick fix
    7. You’re not tracking your progress
    8. You’re crash-dieting
    9. You’re doing too much
    10. You haven't got the right exercise balance
    11. You’ve lost motivation
    
    9 Ways To Burn Fat Fast
    Implement these 9 fat-burning tips that use exercise and diet and watch the body fat melt like the butter you're no longer using.
    1. Stay Off The Scale
    2. Reduce Your Calories Gradually
    3. Vary Your Caloric Intake
    4. Train With Weights
    5. Do High-Intensity Intervals (HIIT)
    6. Eat More Fat
    7. Cut Carbs
    8. Increase Your Protein
    9. Eat 6 Smaller Meals Per Day, Not 2-3 Feasts
    Bonus:
    
    Researchers Explain 6 Reasons Why A Man Falls In Love
    1. HE’S CAPTIVATED BY THE WHOLE PACKAGE
    2. HE FEELS HE CAN MAKE HER HAPPY
    3. WHEN SHE’S OPEN TO LOVE
    4. THE WAY HE FEELS AROUND HER
    5. SHE SAYS “YES” TO LIFE
    6. HE FEELS PURPOSE AND PASSION
    Must Watch: Secret
    For You in Under 20 Minutes...
    These are the 7 weirdest things men find attractive in women
    Who knew anyone would ever find stubbornness attractive in a woman?
    1. Weakness
    2. Applying makeup
    3. Stubbornness
    4. Stretch marks
    5. Wearing glasses
    6. No make up
    7. Clumsiness
    Secret
    For You in Under 20 Minutes...
    5 surprising ways to get great skin
    There are so many bad ideas out there about how to achieve great skin. Here are five ways to get on the right track, regardless of 'the rules.'
    1. You don't need to wash your face with hot water to get it clean.
    2. Oil is good for your skin, not bad.
    3. You don't need to scrub to exfoliate.
    4. What matters most for healthy skin is not what you put on it, but what you eat.
    5. Chocolate doesn't cause acne, but bread and pasta might
    Make Your Skin Naturally Glow
    1. Skip the long, steamy showers and opt for shorter, cooler sprays.
    2. Check the dryness of your skin by scratching a small area on your arm or leg with your fingernail.
    3. Treat your neck and chest like an extension of your face.
    4. Run a humidifier every night in the winter to moisturize the air in your bedroom.
    
    12 easy tips for glowing skin
    1: Discover these glowing skin home remedies:
    2: Sweat it out for healthy skin
    3: Practice Yoga for Face Glow
    4: Know who you are!
    5: Follow a natural diet for glowing skin
    6: Indulge yourself…weekly
    7: Make Sudarshan Kriya your beauty mantra
    8: Meditate everyday to get Glowing Skin
    9: Silence is truly golden
    10: Save your mind at any cost
    11: 18 till I …why not?
    12: Flex those facial muscles
    15 Reasons Yoga Is Better Than The Gym
    1. Yoga benefits the mind, body and spirit.
    2. Yoga benefits your full body—externally and internally.
    3. Yoga teaches acceptance.
    4. Yoga is about focusing on yourself.
    5. Yoga will make you lean.
    6. Yoga is more efficient.
    7. You can do yoga anywhere.
    8. Yoga is kinder to the body.
    9. Yoga eases your aches and pains.
    10. Yoga helps you breathe easier.
    11. Yoga is calming.
    12. Yoga reduces stress.
    13. Everyone can practice yoga.
    14. Yoga improves concentration.
    15. Yogis rock.
    
    9 Ways to Tell If You’re Exercising Enough
    The sign: You never feel sore or tired the day after a workout
    The sign: You're not losing any weight
    The sign: You barely break a sweat
    The sign: The last rep feels hard, but you could probably do another
    The sign: You can chat with your friend throughout your workout
    The sign: You're not getting any more flexible in your yoga classes
    The sign: You run several days a week but aren't getting faster
    The sign: You're making strides in boot camp but are always stiff and un-flexible
    The sign: You take the same class regularly but aren't getting better at the exercises
    
    How To Build Muscle Mass After 50
    Exercise Plan
    Strength-Training Exercises
    Weight, Reps and Sets
    Aerobics Workouts
    
    Savannah Guthrie Says The Keto Diet Is Making Her Mind Sharp—And Her Body Sluggish
    She has zero desire to work out.
    
    How to Lose Weight in 10 Days at Home
    Flight of Stairs / Treadmill
    Lots of Water Fresh –
    Fruits and Vegetables –
    Low-Fat Foods Fish / Meat –
    Avoid Junk Food –
    Moderate Exercise -
    Drink lots of water -
    Day 1
    Day 2 and 3
    Day 4 and 5
    Day 6 and 7
    Day 8 and 9
    Day 10
    
    What are natural ways to make my face look younger?
    Smoothing Exfoliator
    Tightening Toner
    Plumping Moisturizing Facial Mask
    
    10 Ways to Lose 25 Pounds
    Follow these steps and weight loss won’t seem impossible.
    1. Do some detective work.
    2. Get moving.
    3. Plan ahead.
    4. Fill up on fruits and veggies.
    5. Don’t drink your calories.
    6. Don’t do anything drastic.
    7. Set smart calorie targets.
    8. Keep track.
    9. Do not make weight the only measure of success.
    10. Just practice; don’t try to be perfect.