Question

我使用本地html文件启动了一个小爬虫而没有使用bs4中的beautifulsoup，我面临的问题是：

在脚本完成100％完成后我得到了重复的结果而不知道为什么有重复的元素
当我尝试比较两个提取的列表和测试链接时，我只得到错误的结果（字符串比较）。

您可以找到完整的文件here。

html代码（链接部分）：

<h2> test #2 <a href="file:///home/godzilla/crawler-project/test-2.html">first website</a></h2>
<h2> test #3 <a href="file:///home/godzilla/crawler-project/test-3.html">a second website refer to index file</a></h2>
<h2> test #4 <a href="file:///home/godzilla/crawler-project/test-4.html"> a website contain many links , atree one of them refer to index file and another duplicated link </a></h2>

Python脚本：

import string
import urllib

links = []

def ex_link(y):
    result = []
    for i in y:
        if i != None and i.find("<h2>") != -1:
            result.append(i)
    for element in result:
        for i in range(len(result)):
           raw = result[result.index(element)].replace(" ","")
           first = raw.find("<ahref")
           end = raw.find(".html")
           links.append(raw[first+7:end+6])
    print links
    return links
def page(x):
    page = urllib.urlopen(x)
    result = []
    while True:
        page_line = page.readline()
        result.append(page_line)
        if not page_line : break
    return result
link_test =  page("file:///home/godzilla/crawler-project/test-1.html")

链接测试

"file:///home/godzilla/crawler-project/test-2.html"
"file:///home/godzilla/crawler-project/test-3.html"
"file:///home/godzilla/crawler-project/test-3.html"
"file:///home/godzilla/crawler-project/test-3.html"
"file:///home/godzilla/crawler-project/test-4.html"
"file:///home/godzilla/crawler-project/test-4.html"
"file:///home/godzilla/crawler-project/test-4.html"

，测试链接

"file:///home/godzilla/crawler-project/test-2.html"

file:///home/godzilla/crawler-project/test-3.html

file:///home/godzilla/crawler-project/test-4.htm

http://www.google.com

file:///home/godzilla/crawler-project/test-1.html

输出结果为：

"file:///home/godzilla/crawler-project/test-2.html" not found !

"file:///home/godzilla/crawler-project/test-3.html" not found !

"file:///home/godzilla/crawler-project/test-3.html" not found !

"file:///home/godzilla/crawler-project/test-3.html" not found !

"file:///home/godzilla/crawler-project/test-4.html" not found !

"file:///home/godzilla/crawler-project/test-4.html" not found !

"file:///home/godzilla/crawler-project/test-4.html" not found !

没有BeautifulSoup的python2.7爬虫

0 个答案: