[网页上的示例代码] [1]
我正在将BeautifulSoup与Python结合使用，从下面的网站[Knappily] [2]中提取文章标题：
在使用BS4导入html数据时，
<ul id="articleview" class="clearfix" style="padding-left: 0">
::before
未导入。 如图所示，
<li class="col-sm-6 col-md-4 col-lg-4">
以下标记包含有关文章的详细信息。 但是我无法使用find / find_all找到这些
任何建议都会有所帮助。 [1]：https://i.stack.imgur.com/ncJDH.jpg [2]：https://knappily.com
答案 0 :(得分：0)
该页面是使用javascript动态加载的，因此您必须使用selenium：
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('https://knappily.com/')
time.sleep(3)
st = time.time()
while True:
if time.time() - st <= 120: #Keeps loading more articles for 2 mins. You can increase the time if u want.
try:
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="buttons"]'))).click()
time.sleep(1)
except:
break
else:
break
main_art = driver.find_element_by_class_name('impr-content').find_element_by_xpath('.//h1').text
print(f"{'-'*80}\nMain Article:\n{'-'*80}\n{main_art}\n{'-'*80}\nOther Articles:")
articles = driver.find_elements_by_class_name('article-content')
for article in articles:
print('-'*80)
print(article.find_element_by_xpath('.//h3').text)
输出：
--------------------------------------------------------------------------------
Main Article:
--------------------------------------------------------------------------------
Coronavirus has exposed India’s digital divide in education
--------------------------------------------------------------------------------
Other Articles:
--------------------------------------------------------------------------------
Live with no doubts
--------------------------------------------------------------------------------
October 30, 1917: British government approves Balfour Declaration
--------------------------------------------------------------------------------
Delhi is choking again
--------------------------------------------------------------------------------
US Election: the final countdown
--------------------------------------------------------------------------------
At least I tried.
--------------------------------------------------------------------------------
Scam 1992: The real Harshad Mehta story
--------------------------------------------------------------------------------
October 29, 1969: The Internet is born
--------------------------------------------------------------------------------
Terrorist attack in Nice as French cartoon row escalates
--------------------------------------------------------------------------------
Manual scavenging: India must now end the evil
--------------------------------------------------------------------------------
In solitude I find my answers.
--------------------------------------------------------------------------------
October 28, 1929: Black Monday begins the Great Depression
--------------------------------------------------------------------------------
Digital payments are skyrocketing in India
--------------------------------------------------------------------------------
COVID antibodies drop over time, but that’s alright
--------------------------------------------------------------------------------
Detach from what destroys you.
--------------------------------------------------------------------------------
US and India step up defence cooperation
--------------------------------------------------------------------------------
October 27, 1913: Woodrow Wilson swears by pacifism
--------------------------------------------------------------------------------
Cyber Crimes are rising in India
--------------------------------------------------------------------------------
France targets ‘radical Islam’, faces Muslim backlash
--------------------------------------------------------------------------------
You'll never know 'til you go.
--------------------------------------------------------------------------------
The never-ending crisis of IL&FS
--------------------------------------------------------------------------------
October 26, 1858: Theodore Roosevelt is born
--------------------------------------------------------------------------------
Reducing inequality: Chile shows how it’s done
--------------------------------------------------------------------------------
The Indus Valley Tech continues to amaze us
--------------------------------------------------------------------------------
And into the forest I go...
--------------------------------------------------------------------------------
NASA and asteroid Bennu: the problem of plenty
--------------------------------------------------------------------------------
October 25, 1760: George III becomes King
--------------------------------------------------------------------------------
Urban Floods: the deluge of Hyderabad
--------------------------------------------------------------------------------
I doubt, therefore I am.
--------------------------------------------------------------------------------
India: where pollution is deadlier than the pandemic
--------------------------------------------------------------------------------
October 24, 1945: The United Nations is founded
--------------------------------------------------------------------------------
Europe’s hospitals are filling up again
--------------------------------------------------------------------------------
Let go...or be dragged.
--------------------------------------------------------------------------------
Pakistan remains on the FATF grey list
--------------------------------------------------------------------------------
October 23, 1917: Lenin calls for October Revolution
--------------------------------------------------------------------------------
Coronavirus makes America sick again
--------------------------------------------------------------------------------
Courage, dear heart.
--------------------------------------------------------------------------------
Lagos is in total chaos
--------------------------------------------------------------------------------
October 22, 1859: Spain declares war on Morocco
--------------------------------------------------------------------------------
Mysterious deaths in South Korea raise flu vaccine safety concerns
--------------------------------------------------------------------------------
One life. Read more.
--------------------------------------------------------------------------------
The curious rise of ‘Robin Hood’ hackers
--------------------------------------------------------------------------------
October 21, 1854: Florence Nightingale leaves for the Crimea
--------------------------------------------------------------------------------
The United States sues Google
--------------------------------------------------------------------------------
Time flies, we don't.
--------------------------------------------------------------------------------
Neanderthals became ‘feminine’ by mating with humans
--------------------------------------------------------------------------------
October 20, 1911: Roald Amundsen sets out to South Pole
--------------------------------------------------------------------------------
Biden is winning, but Trump may still win
--------------------------------------------------------------------------------
Courage is a daily decision.
--------------------------------------------------------------------------------
Coronavirus cases cross 40 million
--------------------------------------------------------------------------------
October 19, 1910: Subrahmanyan Chandrasekhar is born
--------------------------------------------------------------------------------
Come grow old with me...
--------------------------------------------------------------------------------
Chinese economy continues to grow
--------------------------------------------------------------------------------
Antarctica’s Emperors have no future
--------------------------------------------------------------------------------
October 18, 1954: Transistor Radio is born
--------------------------------------------------------------------------------
The pandemic philosophy of Albert Camus
--------------------------------------------------------------------------------
One big adventure. Or nothing.
--------------------------------------------------------------------------------
India vs. Bangladesh, per capita basis
--------------------------------------------------------------------------------
October 17, 1973: The Oil Crisis begins
--------------------------------------------------------------------------------
French teacher beheaded for “insulting” Islam
--------------------------------------------------------------------------------
Collect memories, not things.
--------------------------------------------------------------------------------
The United Kingdom prepares for no-deal Brexit
--------------------------------------------------------------------------------
October 16, 1923: The Walt Disney Company is founded
--------------------------------------------------------------------------------
Three-finger salute: Thais protest for democracy
--------------------------------------------------------------------------------
But not every man really lives.
--------------------------------------------------------------------------------
Digital Inequality and the Coronavirus
--------------------------------------------------------------------------------
October 15, 1917: Mata Hari is executed
--------------------------------------------------------------------------------
Jacinda Ardern prefers to earn her victory
--------------------------------------------------------------------------------
One life. Complain less.
--------------------------------------------------------------------------------
Resignation Syndrome: And I hereby withdraw from the world
--------------------------------------------------------------------------------
October 14, 1933: Germany quits the League of Nations
--------------------------------------------------------------------------------
Europe leads the world in COVID cases, again
--------------------------------------------------------------------------------
Quiet men are deep & dangerous
--------------------------------------------------------------------------------
2020: the year MBA went online
--------------------------------------------------------------------------------
October 13, 1792: The construction of ‘The White House’ begins
--------------------------------------------------------------------------------
Donald Trump’s last great fight is on
--------------------------------------------------------------------------------
It is what it is.
--------------------------------------------------------------------------------
October 12, 1999: Pervez Musharraf scripts a coup in Pakistan
--------------------------------------------------------------------------------
Nobel Memorial Prize in Economics goes to auction theorists
--------------------------------------------------------------------------------
Coronavirus can survive for 28 days in cold and dark
--------------------------------------------------------------------------------
Be. Here. Now.
--------------------------------------------------------------------------------
Rafael Nadal wins French Open to tie Grand Slam record
--------------------------------------------------------------------------------
October 11, 1987: Operation Pawan begins in Sri Lanka
--------------------------------------------------------------------------------
North Korea flaunts monster missiles and zero COVID cases
--------------------------------------------------------------------------------
To seek nothing takes time
--------------------------------------------------------------------------------
Women political leaders don’t exist in China
--------------------------------------------------------------------------------
October 10, 1967: The Outer Space Treaty comes into force
--------------------------------------------------------------------------------
The speed limit of sound has been found
--------------------------------------------------------------------------------
Good travelers never arrive.
--------------------------------------------------------------------------------
October 9, 1941: The Manhattan Project begins
--------------------------------------------------------------------------------
World Food Program wins Nobel Peace Prize 2020
--------------------------------------------------------------------------------
Is China winning the vaccine race?
--------------------------------------------------------------------------------
Rise above the crowd.
--------------------------------------------------------------------------------
Louise Glück wins Nobel Prize in Literature
--------------------------------------------------------------------------------
October 8, 1912: First Balkan War begins
--------------------------------------------------------------------------------
Coronavirus: making the poor poorer
--------------------------------------------------------------------------------
Your library is your paradise.
--------------------------------------------------------------------------------
Gene editing pioneers win the Chemistry Nobel
--------------------------------------------------------------------------------
October 7, 1952: Vladimir Putin is born
--------------------------------------------------------------------------------
Facebook bans conspiracy group QAnon
--------------------------------------------------------------------------------
When I do good, I feel good
--------------------------------------------------------------------------------
Black Hole hunters win Nobel Prize in Physics
--------------------------------------------------------------------------------
October 6, 1973: Yom Kippur War begins
--------------------------------------------------------------------------------
Donald Trump is back in action
--------------------------------------------------------------------------------
Believing is Seeing.
--------------------------------------------------------------------------------
Hepatitis C virus hunters win Nobel Prize in Medicine
--------------------------------------------------------------------------------
October 5, 1910: Portugal becomes a Republic
--------------------------------------------------------------------------------
Europe is breaking coronavirus records again
--------------------------------------------------------------------------------
Keep it simply, silly.
--------------------------------------------------------------------------------
The President has been put on steroids
--------------------------------------------------------------------------------
October 4, 2006: WikiLeaks is founded
--------------------------------------------------------------------------------
Remdesivir: the President’s drug for COVID-19
--------------------------------------------------------------------------------
Write the first true word you know
--------------------------------------------------------------------------------
No place for nuclear waste in Germany
--------------------------------------------------------------------------------
October 3, 1952: Britain explodes an atomic bomb
--------------------------------------------------------------------------------
Donald Trump has been hospitalized
--------------------------------------------------------------------------------
Seek what's seeking you.
--------------------------------------------------------------------------------
Gandhi: because right is right and wrong is wrong
--------------------------------------------------------------------------------
October 2, 1904: Lal Bahadur Shastri is born
--------------------------------------------------------------------------------
Donald Trump tests positive for coronavirus
--------------------------------------------------------------------------------
On and on it goes...
--------------------------------------------------------------------------------
EU sues Britain over breaching Brexit deal
--------------------------------------------------------------------------------
October 1, 1988: Gorbachev is the head of Supreme Soviet
--------------------------------------------------------------------------------
Japan fears ‘suicide contagion’ after celebrity suicides
--------------------------------------------------------------------------------
Timothy Ray Brown: the ‘Berlin patient’ who conquered HIV
--------------------------------------------------------------------------------
What the hell, leap.
--------------------------------------------------------------------------------
September 30, 1938: Munich Agreement is signed to appease Hitler
--------------------------------------------------------------------------------
The never-ending Babri demolition case has delivered its first verdict
--------------------------------------------------------------------------------
First Presidential Debate: what a hot mess
--------------------------------------------------------------------------------
The absurd is the first truth.
--------------------------------------------------------------------------------
The BTS IPO is a smash hit
--------------------------------------------------------------------------------
September 29, 1913: Rudolf Diesel vanishes in thin air
--------------------------------------------------------------------------------
Coronavirus: one million deaths is not just a statistic
--------------------------------------------------------------------------------
Conviction needs Courage.
--------------------------------------------------------------------------------
Can gene therapy cure glaucoma?
--------------------------------------------------------------------------------
September 28, 1970: Gamal Abdel Nasser is no more
--------------------------------------------------------------------------------
Donald Trump’ tax returns have been outed
--------------------------------------------------------------------------------
Accept it. Endure it.
--------------------------------------------------------------------------------
September 27, 1940: Axis powers sign the Tripartite Pact
--------------------------------------------------------------------------------
Armenia and Azerbaijan are close to war
--------------------------------------------------------------------------------
India must prepare for a long coronavirus fight
--------------------------------------------------------------------------------
You got this.
--------------------------------------------------------------------------------
Coulrophobia: the irrational fear of clowns
--------------------------------------------------------------------------------
Children can fight coronavirus better
--------------------------------------------------------------------------------
September 26, 1820: Ishwar Chandra ‘Vidyasagar’ is born
--------------------------------------------------------------------------------
The battle for the soul of the Supreme Court
--------------------------------------------------------------------------------
Stop thinking. Start living.
--------------------------------------------------------------------------------
September 25, 1969: Organization of Islamic Cooperation is founded
--------------------------------------------------------------------------------
Vodafone wins international arbitration against India
--------------------------------------------------------------------------------
Divided nations clash at the United Nations
--------------------------------------------------------------------------------
When I do good, I feel good.
--------------------------------------------------------------------------------
September 24, 2014: Mangalyaan reaches the orbit of Mars
--------------------------------------------------------------------------------
Coronavirus may have a dominant mutant strain
--------------------------------------------------------------------------------
Will the silent majority rise again for Trump?
--------------------------------------------------------------------------------
Don't be a Maybe.
--------------------------------------------------------------------------------
September 23, 1908: Ramdhari Singh ‘Dinkar’ is born
--------------------------------------------------------------------------------
The unbearable tragedy of whale beaching
--------------------------------------------------------------------------------
US coronavirus death toll crosses 200,000
--------------------------------------------------------------------------------
Find comfort in the chaos
--------------------------------------------------------------------------------
Resilience: the obstacle is the path
--------------------------------------------------------------------------------
September 22, 1980: Iraq invades Iran
--------------------------------------------------------------------------------
Vaccine: arriving in November…or Never?
--------------------------------------------------------------------------------
Be a happy rebel.
--------------------------------------------------------------------------------
Who really owns TikTok now?
--------------------------------------------------------------------------------
September 21, 1743: Maharaja Jai Singh II is no more
--------------------------------------------------------------------------------
India debates agriculture marketing reforms
--------------------------------------------------------------------------------
It is what it is.
--------------------------------------------------------------------------------
We are overusing this planet
--------------------------------------------------------------------------------
September 20, 1933: Annie Besant is no more
--------------------------------------------------------------------------------
Tadej Pogacar: the impossible Tour de France winner
--------------------------------------------------------------------------------
The mountains are calling...
--------------------------------------------------------------------------------
India's contribution to the world: a big ZERO
--------------------------------------------------------------------------------
Europe prepares for a viral winter
--------------------------------------------------------------------------------
September 19, 1946: Winston Churchill proposes the United States of Europe
--------------------------------------------------------------------------------
Taiwan learns to play with fire
--------------------------------------------------------------------------------
Rise, little man.
--------------------------------------------------------------------------------
September 18, 2001: First Anthrax letters are mailed
--------------------------------------------------------------------------------
India’s weird relationship with weed
--------------------------------------------------------------------------------
Talk Less. Do More.
--------------------------------------------------------------------------------
September 17, 2011: Occupy Wall Street begins, the 99 percent rise
--------------------------------------------------------------------------------
Everyone is a prepper in 2020
--------------------------------------------------------------------------------
Be fearlessly you.
--------------------------------------------------------------------------------
Yoshihide Suga: the self-made Prime Minister of Japan
--------------------------------------------------------------------------------
September 16, 1987: Montreal Protocol is signed
--------------------------------------------------------------------------------
The emptiness of Trump’s Middle East ‘peace’ deal
--------------------------------------------------------------------------------
Let go...or be dragged.
--------------------------------------------------------------------------------
Do we need animals to make meat?
--------------------------------------------------------------------------------
September 15, 1861: M. Visvesvaraya is born
--------------------------------------------------------------------------------
Pakistan rises against ‘rape culture’
--------------------------------------------------------------------------------
It's not What, but How you do it
--------------------------------------------------------------------------------
Are they on Venus?
--------------------------------------------------------------------------------
September 14, 1960: OPEC is founded
--------------------------------------------------------------------------------
This virus just keeps breaking records
--------------------------------------------------------------------------------
He who is brave is free.
--------------------------------------------------------------------------------
Dominic Thiem wins US Open with a stunning comeback
--------------------------------------------------------------------------------
September 13, 1948: Indian Army enters Hyderabad
--------------------------------------------------------------------------------
Naomi Osaka wins US Open and rallies for social justice
--------------------------------------------------------------------------------
The fight is everything.
--------------------------------------------------------------------------------
September 12, 1990: Two Plus Four Agreement to reunify Germany is signed
--------------------------------------------------------------------------------
Facultative parthenogenesis: the strange world of virgin births
--------------------------------------------------------------------------------
Afghan peace talks begin to end the forever war
--------------------------------------------------------------------------------
If it's endurable, then endure it.
--------------------------------------------------------------------------------
The curious case of India’s low COVID-19 mortality rate
--------------------------------------------------------------------------------
September 11, 1965: Bashar al-Assad is born
--------------------------------------------------------------------------------
Shall we reopen the schools?
--------------------------------------------------------------------------------
One life. Live more.
--------------------------------------------------------------------------------
Brexit gets even more chaotic
--------------------------------------------------------------------------------
September 10, 2008: Acceleration begins at Large Hadron Collider
--------------------------------------------------------------------------------
Coronavirus did not stop Climate Change
--------------------------------------------------------------------------------
Let go...or be dragged.
--------------------------------------------------------------------------------
Caster Semenya: too fast to be a woman
--------------------------------------------------------------------------------
September 9, 1993: PLO recognizes Israel
--------------------------------------------------------------------------------
Oxford vaccine suffers a major setback
--------------------------------------------------------------------------------
Calm is a superpower.
--------------------------------------------------------------------------------
Are we inside a Matrioshka Brain?
--------------------------------------------------------------------------------
September 8, 1945: Hideki Tojo attempts suicide
--------------------------------------------------------------------------------
Remote workers of the world, unplug
--------------------------------------------------------------------------------
Let the dead leaves drop
--------------------------------------------------------------------------------
India: the economy can’t wait, the virus won’t stop
--------------------------------------------------------------------------------
September 7, 1923: The INTERPOL is founded
--------------------------------------------------------------------------------
The strange disqualification of Novak Djokovic
--------------------------------------------------------------------------------
Even anxieties have anxieties today
--------------------------------------------------------------------------------
Someone is mutilating horses in France
--------------------------------------------------------------------------------
...
--------------------------------------------------------------------------------
May 16, 1966: Cultural Revolution begins in China
--------------------------------------------------------------------------------
Coronavirus: 4.5 million cases, 300,000 deaths
--------------------------------------------------------------------------------
Above all, be kind.
>>> len(articles)
714