Question

所以我试图为搜索引擎制作一个抓取工具。我的代码大部分都是完整的。它的工作方式是：

使用BeautifulSoup打开页面，
将url和docID保存到索引（在本例中只是一个文本文件），
获取一些干净的文本并将其保存到文本文件
从网页上获取所有网址，并将其添加到网页列表中（如果网页尚未包含在网页中）。

我遇到的问题是，有时它会多次索引网址。

我做的一个示例运行是使用初始pages = ["http://www.ece.udel.edu"]。在输出文件中，每行包含docID和url。格式为docID url\n。这会在对结果进行排名时引起问题，因为虽然它应该对这些结果进行排名相同，但我会将同一页面的多个（每个都有不同的docID）排名。

这是我到目前为止的代码。我试图对其进行彻底评论，以便更容易理解。

def cleanText(soup):
    #remove all unnecessary tags & punctuation leaving text of all lowercase letters & words
    texts = soup.findAll(text=True)
    visible_elements=[]
    for elem in texts:
        if elem.parent.name in ['style', 'script', '[document]', 'head', 'title']:
            visible_elements.append('')
        elem = elem.encode('utf-8','replace')
        result = re.sub('<!--.*-->|\r|\n', '', str(elem), flags=re.DOTALL)
        result = re.sub('\s{2,}|&nbsp;', ' ', result)
        visible_elements.append(result)
    visible_text = ' '.join(visible_elements)
    visible_text = re.sub('[^0-9a-zA-Z]+', ' ', visible_text)
    visible_text = visible_text.lower()
    return visible_text

def naming(pagename, pathToOutput):
    #used to create filename to save output text file
    cantBeInFilename = ("/", "\\",":","*","?","<",">","|")
    for k in range(len(cantBeInFilename)):
        pagename = pagename.replace(cantBeInFilename[k], "-")
    filename = "\\".join((pathToOutput, pagename))
    filename = ".".join((filename, "txt"))
    return filename

def crawl(pages, pathToOutput, pathToLinks):
    depth = len(pages)
    docID = 1
    for i in range(depth): #for each link on the page
        newpages=set() #create an empty set for gathered pages
        for page in pages: #each page in pages, initially atleast 1
            linksText = open(pathToLinks, 'a') #open a file
            linksText.write(str(docID) + " "); #write a docID to identify page
            docID = docID + 1 #increment docID
            linksText.write(str(page) + '\n') #append link into end of file
            linksText.close() #close file to update
            try:
                c = urllib2.urlopen(page) #open the page
                print ("Opening " + page)
            except:
                print ("Could not open " + page)
                continue
            soup = BeautifulSoup(c.read())
            clean_text = cleanText(soup)
            #get all links off page
            links = soup.findAll('a')
            #write web page to text file in pathToOutput directory
            filename = naming(page, pathToOutput)
            f = open(filename, 'w')
            f.write(clean_text)
            f.close()
            depth = 0
            for link in links:
                if('href' in dict(link.attrs)):
                    #set depth equal to the # of links in the pages
                    depth = depth+1
                    url = urljoin(page, link['href'])
                    #remove unnecessary portions of url
                    if url.find("'") != -1:
                        continue
                    url = url.split('#')[0]
                    #get each line (each link) from linksText
                    linksText = open(pathToLinks, 'r')
                    lines = linksText.readlines()
                    linksText.close()
                    #remove docIDs just to have 
                    for j in range(len(lines)):
                        lines[j-1] = lines[j-1].split()[1]
                    #FOR ALL LINKS
                    if url[0:4] == "http":
                        #check to see if the url ends with a /, if so remove it
                        if url[len(url)-1]=="/":
                            url = url[0:len(url)-1]
                        #check to see if the url is already in links
                        present = 0
                        for line in lines:
                            if (url == line):
                                present = 1
                            else:
                                present = 0
                        #if the url isn't present, add to pages to be cycled through
                        if (present == 0):
                            print ("Indexing " + url)
                            newpages.add(url)
                #add newpages to pages
                for newpage in newpages:
                    pages.append(newpage)
                newpages = set()
            #remove already read page from pages
            pages.remove(page)

我会附上我已经从这里得到的输出，但我刚开始在网站上，不能发布一个带有两个以上链接的问题:(对不起，如果这很长，令人困惑，只是简单不做感

Answer 1

此代码段没有按您的想法执行

present = 0
for line in lines:
    if (url == line):
        present = 1
    else:
        present = 0
#if the url isn't present, add to pages to be cycled through
if (present == 0):
    print ("Indexing " + url)
    newpages.add(url)

如果匹配项位于字符串的最后一项，则只会离开present = 1。一种更简单的测试方法是：

#if the url isn't present, add to pages to be cycled through
if url not in lines:
    print ("Indexing " + url)
    newpages.add(url)

您可能还需要更改

#get each line (each link) from linksText
linksText = open(pathToLinks, 'r')
lines = linksText.readlines()
linksText.close()

进入

#get each line (each link) from linksText
linksText = open(pathToLinks, 'r')
lines = linksText.read().split('\n')
linksText.close()

以便line中的每个lines都不会以\n结尾

使用Beautiful Soup创建一个爬虫，索引URL的问题

1 个答案: