Question

我正在尝试从以下网址（http://www.ancient-hebrew.org/m/dictionary/1000.html）中抓取数据。因此，每个希伯来语单词部分均以img网址开头，后跟2个文本，即实际的希伯来语单词及其发音。例如，URL中的第一个条目是以下“ img1 img2 img3אֶלֶףe-leph”，希伯来语单词是使用wget下载html后的unicode

例如，下面的代码为我提供了<img src="../../files/heb-anc-sm-pey.jpg"/>和<font face="arial" size="+1"> unicode_hebrew_text </font>和<a href="audio/ 505 .mp3"><img border="0" height="25" src="../../files/icon_audio.gif" width="25"/></a>

我只想../../files/heb-anc-sm-pey.jpg 以及unicode_hebrew_text和audio/505.mp3 (without any spaces in between)

from bs4 import BeautifulSoup

raw_html = open('/Users/gansaikhanshur/TESTING/webScraping/1000.html').read()
html = BeautifulSoup(raw_html, 'html.parser')

# output: <img src="../../files/heb-anc-sm-pey.jpg"/>
imgs = html.findAll("img")
for image in imgs:
    # print image source
    if "jpg" in str(image):
        print(image)

# output: <font face="arial" size="+1"> unicode_hebrew_text </font>
font = html('font', face="arial", size="+1")
for f in font:
    continue

# output: <a href="audio/   505 .mp3"><img border="0" height="25" src="../../files/icon_audio.gif" width="25"/></a>
mp3file = html.findAll(href=True)
for mp3 in mp3file:
    if "mp3" in str(mp3):
        continue

您可以看到我的代码并没有真正完成这项工作。最后，我想获取URL中每个单词的信息，并将其另存为文本文件或json文件，以较容易的形式为准。

例如，图像：URLsOfImages，希伯来语单词：txt，发音：txt，URLtoAudio：txt

以及下一个单词，依此类推。

Answer 1

您希望抓取的网页有些棘手。

现在，为什么要这么说？事实是，网页并没有过多地使用div标签或类似的东西来将单个希伯来语单词及其相关数据放在一个块中（我称其为希伯来语单词及其相关数据，即此处的希伯来语块）。但是，如果您专注于页面源代码，那么您会注意到每个希伯来语块都是从锚标记开始的，该锚标记看起来像这样：<a name="505"></a>。值“ 505”将根据每个希伯来语块的强数而变化。因此，我尝试将其作为每个希伯来语块的起点。现在，在希伯来语块中，您可以使用扬声器图标<a href="audio/ 505 .mp3"><img src="../../files/icon_audio.gif" width="25" height="25" border="0"></a>来表示希伯来语块的终点。

牢记上述想法，我想到了这一点：

from bs4 import BeautifulSoup
from bs4 import NavigableString, Tag 
import requests
import re

def make_soup(url):
    html = requests.get(url, verify = False)
    soup = BeautifulSoup(html.text, 'html.parser')
    return soup


url = "http://www.ancient-hebrew.org/m/dictionary/1000.html"
soup = make_soup(url)

points = soup.find_all("a", attrs = {"name" : re.compile("[0-9]+")}) 
#This will help you locate the starting points of the Hebrew block

for point in points:
    point_name = point['name'] 
    images = [] #Will hold the links for the Old Hebrew images
    text = "" #Will hold the immediate next Unicode Hebrew text
    usage = "" #Will hold the usage/meaning data
    mp3 = "" #Will hold the link to audio file

    temp = point.next_sibling 
    #'temp' variable will be used as a navigator to extract the data within the Hebrew block 

    while(True):
        #If 'temp' is a HTML 'Tag' by nature, then it'll contain the Unicode Hebrew text and links for images& audio
        if(type(temp) is Tag): 
            if(temp.name == 'img'):        
                #If it's an 'img' tag, then put it to 'images' list 
                images.append(temp['src'])
            elif(temp.name == 'font'):
                #If it's a 'font' tag, then put it in 'text'
                text = temp.string
            elif(temp.name == 'a'):
                #If it's 'a' tag, then it contains the link to audio file
                mp3 = temp['href']  
                #Link to audio file will mark the end of the Hebrew block. So, 'break' out
                break
        elif(type(temp) is NavigableString):
            #If 'temp' acquires a 'NavigableString' instead of a 'Tag' then it's usage/meaning
            usage += temp
        #Move to the next item within the Hebrew block
        temp = temp.next_sibling

    print([point_name, images, mp3, text, usage])

Python BeautifulSoup获取图像，文本和URL

1 个答案: