使用beautifulsoup进行网页抓取,输出保持不变

时间:2018-01-22 15:36:58

标签: python django web-scraping beautifulsoup

我目前想从网站foursquare.com上删除评论。我使用以下代码来做到这一点

admin/people/create

运行上面的代码时,我应该得到144个数据。有时我得到所有数据,有时我只得到56个数据 为什么输出会不断变化?它是代码或互联网连接的问题吗?

3 个答案:

答案 0 :(得分:0)

我认为问题出在代码的(review.decode('unicode_escape')部分。 您应该使用print(review.encode('ascii','ignore'))将文本编码为ASCII

答案 1 :(得分:0)

我认为问题是有多个页面。但是你只能先获得。 您可以通过这种方式获取所有页面中的所有数据。

如果您想要刮另一页,只需更改网址即可。你不必知道那里有多少页。

from bs4 import BeautifulSoup
import requests

#List of urls you want to scrape
url_list = ['pantai-klebang/4c7c12d22d3ba14318e595d0','porta-de-santiago-a-famosa-fortress/4d26d123ebacb1f77692e14f']

def scrape(url_to_scrape):
    i = 1
    while True:
        #If there is more than one page we can loop through all. You can get all data no matter how many page there is
        request = requests.get('https://foursquare.com/v/{0}?tipsPage={1}&tipsSort=recent'.format(url,i))
        soup = BeautifulSoup(request.text, 'lxml')

        reviews = soup.find_all("li", {"class": "tip"})

        #If there is no more page break loop.
        if len(reviews) != 0:
            for item in reviews:
                review = item.find("div", {"class": "tipText"}).text
                print(review)
            i += 1
        else:
            break

for url in url_list:
    scrape(url)
    #Leave some space between two pages
    print()

对不起,这有点长。输出是

From sunset too variety food u cant liked it..
Byk motor laju2
Good place to chill
If you wan to play bubble and take photo, here is the best place
Scenic view for weekend getaway... coconut shake not taste as original klebang coconut shake...
Getting dirtier
Weekend getaway!
Good for casual walk & watching sunset with loved ones since my last visit macam2 ade kat sini very packed during public holidays
Afternoon time quite dry..beach is normal. Maybe evening/night might be better. The coconut shake they add vanilla ice cream,hmmm
Pantai awesome beb
Nice place for picnic
Cannot mandi here. Good place for recreation.. Calm place for weekdays! Haha
Very bz place. Need to go there early if you want to lepak. If not, no parking for you
So many good attraction here, worth a visit
Beautiful place for sunset
New beach!  Like all beaches, awesome view & windy.  Some stretch got many small crabs.
There is bustel "hotel in a bus" can get coconut shake or fried seafood in the evening at 5pm. Bustel rate is from RM80. Bus cafe, bus toilet... Total bus transformation
Too crowded la
Muzium kapal selam closed since 1/3 until further notice..
If you are looking for public toilets, look for a red bus. An old bus was modified and transformed to operate as toilets. Cool.
Most of the shops closed after 12 midnight..helloo,this place should be the place for the late nighters..late night supposed to be the peak hour for business..live band bar maybe?? :-P
Overrated....nothing much. Kemasik beach is much better.
Chillax 
Good place to chill out w family and friend..
The desert view if you go in deep enough! Really nice but damn hot! Rmb to bring umbrella if not you will get sunburn~
Coconut shake! ❤️
Jom pekena coconut shake original.
Totally overrated. They shld hv more warnings against swimming here, the artificial beach hides a steep, sudden and muddy slope!
Anyone who loves a scenery of a desert, do come here 
indah sungguh sunset...
A change of air and scenery
Hang out here for a change of scenery. Better yet, watch the sun set.
Best for lepak
Not a best place anymore
Finally caught a glimpse of the sand dune. Be careful while driving your car or else the car will caught in sand.
Center of gastronomic adventure and endure the view especially the sunset
The cafe built from old bus is worth to visit.
This " beach " cum " desert " is beautiful!
nice coconut shake
Coconut shake babyyyy
Let's menyelam with kapal selam here ! Hehehe
Beaches in Malacca are awesome!
Byk kedai mkn.. Ada bustel.. Coconut shake.. Muzium kapal selam.. Kereta kuda..
Nice place... Enjoy together with family.. 
Family time, build sand castle
Good place to hang out.
Santai  dng family
lepak2 thun batu lalu..
Angin sepoi2....nyaman
Nice place
Holiday before exam!!!
Coconut shake brought me here.
Dragtimes2u event
Dragtimes2u event....
best coconut shake
It's okay. White sands and beautiful scenery.
Coconut shake a must ! Ade rasa mcm vanilla sikit.
Diner w/ coursemate ♡
Boooooring
scenic viewwww
Sea view, breeze and lovely day
coconut shake
Best to calm n peace ur mind
Coconut shake thorbaikk !!
Coconut shake is so yummy 
0530 morning also still got so many cars. Lol
Great place to bring your family for weekend activities
Great place to bring your family
A place like desert
Best lpk ptg2. .dpt tenangkn jiwa. .kiki
Jom main buai!!
angin sepoi2
Many family picnics
I still remember playing on the swings with all the trees. Skrg, padang jarak padang tekukur
Now its crowded and pantai tercemar ); it used to be very windy and peacefuk back in my younger daysss (10 years ago)
Coconut shake best!!
Great view esp when the sun rise.
Muzium kapal selam is cheap at rm1, likely to increase when proper museum is completed. Include visit inside sub...
Breathtaking....
Do stop at Klebang Original Coconut Shake!
A.K.A. Dataran Propaganda
Nice beach to chillex. Currently, nigger were happily singing and make some noice here.
Assignment + coconut shake :D
Lot's of kites. had durians :)
Bawak datang Coconut Shake original Klebang . Enjoy your day !
Tmpt melepet pagi2. :)
Family day! - First time playing kites together <3 great day!
The weather :)
santaiiii with Vanilla roti ice creammmmmm~auuu...
The best place to play kites..
Pantai Satu Malaysia
No more beach > LOL ;(
Main layang2 kt tambak
Coconut shake klebang
looking for the sea
Xda ap sgt yg menarik...sape suka main pasir...oklah
Jauhnya nk kena jalan drpd tempat parking. 
Family place tapi harga makanan over price
terbaik..
memang sedap dan beratur panjang lah kelapa goncang klebang
kena kelapa goncang klebang dulu
Cantik. Dinasihatkan time dekat senja tu baliklah woi. Jangan main jauh sangat. Petang2 especially time cuti jem gila woi
Last pegi kat sini ialah pada tahun 2014 ☺ Semalam pegi kat tempat ni, wowww!! Banyak kemudahan serta perubahan tempat ni. Moment paling best semalam aku rasa.
Nice place to breathtaking. Tapi kalau banyak pokok, mesti lagi nice.
Kalau pergi Melaka tapi tak pekena Coconut Shake Klebang, tak lengkap percutian korang.
Gerai depan pantai Klebang Over rated . Tak sedap , mahal dan Service slow . highly not recommended .
Nak pergi ke padang pasir dia memang susah 
Pergi pantai klebang sambil minum coconut shake
The best coconut shake ever. Must try 
Banyak gerai makanan yang unik which is you can dine your food in the bus.. Sini jgk ada bustel yang memuatkan dua bilik dalam satu bus.. Toilet dia pun dalam bus okay.. Seronok main bubble and kite..
Makan sedap, harga agak cekik darah..
againnnnnn ! 
Kalau kereta bukan 4x4 better jangan masuk dekat kawasan tambak sampai dalam, takut nak balik nanti tersangkut dengan pasir  Park kat tempat ramai orang. Exercise sikit! 
Tak complete trip ke Melaka kalau tak datang sini  Put your shades on & enjoy the view  Tangkap banyak banyak gambar!!! 
Boleh main layang2.
Antara pantai yang cantik. Satu hari nak try tido kat BUSTEL. Memang awesome. Cenggini haaa 
Ada 1 kedai keropok lekor power area sini....
Pantai yang tenang
Angin sepoi.. Boleh main buih belon
Time lepas sunset, mmg sesak gile. Kalau nak keluar dari pantai klebang, sblm maghrib atau slps isya. Pandai2 lah ngkorg
Persekitaran dia best tapi bila hujan . Makanan yg nak dihantar kt cstomer tak ditutup . Dia biar je mknan kena hujan . Haihh . Penambah perisa katanyaa . :(
Coconut shake yummeh, nasi goreng klebang pun sedap dan sangat banyak berbaloi dengan harga..
Kak jom lah tutup booth awal . Takde org sgt ni 
Main belon air disini..... aktiviti petang. ..
HOT Burger.. tepi jalan sebelum simpang masuk pantai Klebang. Special.. ada burger arnab, rusa, udang, oblong dan menu2 special lain
Gerai mkn tepi pantai yg 1st kekenat shake walls ni service too bad. Makanan cm bdk drjah 1 masak lg sedap. Nasi grg 4.50 tp dlm kosong. Big failed!
Tk syok mandi kat sini 
Pantai x mcm pantai ! Tu lah kenyataan nya.
Bas banyak banyak tepi pantai buat apa ehh?
Tiap mlm dok sorg2 kat klebang. Tegur la lau nampak ;) hehehe smlm JAIM dtggggg lari sume kete haha lawak
Angin kuat euy...
Bengong p tambak jauh2.. Xbest xnmpak pntai..save our planet pls.. Ini kalilah..
Kuey tiow kerang kat kedai yg hujung sekali memang terbaik!!
Byk org melakukan maksiat di sini...
dah lama~  .................  pegi mase lepas gi kenduri...
Pari terbaik
Lepak2..buat BBQ cantik tmpt nie...
Dah takde pantai
Pantai dah tak nampak. Dari main road, dlm 300meter daratan, bru nampak pantai. :(
main layang2 sambil angkat kain....
Beautiful Beach :) . Mau nya di kreasi lagi .. Biar ramai orang yang kesana..
mandi mandi
Pantai yang dulunya ada sekarang dah jadi dataran. Ruginya hilang satu lagi pantai.
Holiday before exam!!!
Coconut shake brought me here.
Dragtimes2u event
Dragtimes2u event....
best coconut shake
It's okay. White sands and beautiful scenery.
Coconut shake a must ! Ade rasa mcm vanilla sikit.
Diner w/ coursemate ♡
Boooooring
scenic viewwww
Sea view, breeze and lovely day
coconut shake
Best to calm n peace ur mind
Coconut shake thorbaikk !!
Coconut shake is so yummy 
0530 morning also still got so many cars. Lol
Great place to bring your family for weekend activities
Great place to bring your family
A place like desert
Best lpk ptg2. .dpt tenangkn jiwa. .kiki
Jom main buai!!
angin sepoi2
Many family picnics
I still remember playing on the swings with all the trees. Skrg, padang jarak padang tekukur
Now its crowded and pantai tercemar ); it used to be very windy and peacefuk back in my younger daysss (10 years ago)
Coconut shake best!!
Great view esp when the sun rise.
Muzium kapal selam is cheap at rm1, likely to increase when proper museum is completed. Include visit inside sub...
Breathtaking....
Do stop at Klebang Original Coconut Shake!
A.K.A. Dataran Propaganda
Nice beach to chillex. Currently, nigger were happily singing and make some noice here.
Assignment + coconut shake :D
Lot's of kites. had durians :)
Bawak datang Coconut Shake original Klebang . Enjoy your day !
Tmpt melepet pagi2. :)
Family day! - First time playing kites together <3 great day!
The weather :)
santaiiii with Vanilla roti ice creammmmmm~auuu...
The best place to play kites..
Pantai Satu Malaysia
No more beach > LOL ;(
Main layang2 kt tambak
Coconut shake klebang
looking for the sea
Xda ap sgt yg menarik...sape suka main pasir...oklah
Jauhnya nk kena jalan drpd tempat parking. 
Family place tapi harga makanan over price
terbaik..
memang sedap dan beratur panjang lah kelapa goncang klebang
kena kelapa goncang klebang dulu
Cantik. Dinasihatkan time dekat senja tu baliklah woi. Jangan main jauh sangat. Petang2 especially time cuti jem gila woi
Last pegi kat sini ialah pada tahun 2014 ☺ Semalam pegi kat tempat ni, wowww!! Banyak kemudahan serta perubahan tempat ni. Moment paling best semalam aku rasa.
Nice place to breathtaking. Tapi kalau banyak pokok, mesti lagi nice.
Kalau pergi Melaka tapi tak pekena Coconut Shake Klebang, tak lengkap percutian korang.
Gerai depan pantai Klebang Over rated . Tak sedap , mahal dan Service slow . highly not recommended .
Nak pergi ke padang pasir dia memang susah 
Pergi pantai klebang sambil minum coconut shake
The best coconut shake ever. Must try 
Banyak gerai makanan yang unik which is you can dine your food in the bus.. Sini jgk ada bustel yang memuatkan dua bilik dalam satu bus.. Toilet dia pun dalam bus okay.. Seronok main bubble and kite..
Makan sedap, harga agak cekik darah..
againnnnnn ! 
Kalau kereta bukan 4x4 better jangan masuk dekat kawasan tambak sampai dalam, takut nak balik nanti tersangkut dengan pasir  Park kat tempat ramai orang. Exercise sikit! 
Tak complete trip ke Melaka kalau tak datang sini  Put your shades on & enjoy the view  Tangkap banyak banyak gambar!!! 
Boleh main layang2.
Antara pantai yang cantik. Satu hari nak try tido kat BUSTEL. Memang awesome. Cenggini haaa 
Ada 1 kedai keropok lekor power area sini....
Pantai yang tenang
Angin sepoi.. Boleh main buih belon
Time lepas sunset, mmg sesak gile. Kalau nak keluar dari pantai klebang, sblm maghrib atau slps isya. Pandai2 lah ngkorg
Persekitaran dia best tapi bila hujan . Makanan yg nak dihantar kt cstomer tak ditutup . Dia biar je mknan kena hujan . Haihh . Penambah perisa katanyaa . :(
Coconut shake yummeh, nasi goreng klebang pun sedap dan sangat banyak berbaloi dengan harga..
Kak jom lah tutup booth awal . Takde org sgt ni 
Main belon air disini..... aktiviti petang. ..
HOT Burger.. tepi jalan sebelum simpang masuk pantai Klebang. Special.. ada burger arnab, rusa, udang, oblong dan menu2 special lain
Gerai mkn tepi pantai yg 1st kekenat shake walls ni service too bad. Makanan cm bdk drjah 1 masak lg sedap. Nasi grg 4.50 tp dlm kosong. Big failed!
Tk syok mandi kat sini 
Pantai x mcm pantai ! Tu lah kenyataan nya.
Bas banyak banyak tepi pantai buat apa ehh?
Tiap mlm dok sorg2 kat klebang. Tegur la lau nampak ;) hehehe smlm JAIM dtggggg lari sume kete haha lawak
Angin kuat euy...
Bengong p tambak jauh2.. Xbest xnmpak pntai..save our planet pls.. Ini kalilah..
Kuey tiow kerang kat kedai yg hujung sekali memang terbaik!!
Byk org melakukan maksiat di sini...
dah lama~  .................  pegi mase lepas gi kenduri...
Pari terbaik
Lepak2..buat BBQ cantik tmpt nie...
Dah takde pantai
Pantai dah tak nampak. Dari main road, dlm 300meter daratan, bru nampak pantai. :(
main layang2 sambil angkat kain....
Beautiful Beach :) . Mau nya di kreasi lagi .. Biar ramai orang yang kesana..
mandi mandi
Pantai yang dulunya ada sekarang dah jadi dataran. Ruginya hilang satu lagi pantai.

答案 2 :(得分:0)

您可以通过多种方式实现相同目标。这是另一种方式:

from bs4 import BeautifulSoup
import requests

URL = "https://foursquare.com/v/pantai-klebang/4c7c12d22d3ba14318e595d0?tipsPage={}&tipsSort=recent"

for link in [URL.format(page) for page in range(1,4)]: ##looping through all the pages to get all the reviews available
    res = requests.get(link)
    soup = BeautifulSoup(res.content, "lxml")
    for item in soup.select(".tipText"):
        print(item.text)