找到网站中最常见的单词

时间:2013-07-28 02:14:14

标签: python counter

我是python的新手。我有一个简单的程序来查找某个单词在网站中的使用次数。

opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]

url = 'http://en.wikipedia.org/wiki/Albert_Einstein'
ourUrl = opener.open(url).read()
soup = BeautifulSoup(ourUrl)
dem = soup.findAll('p') #find paragraphs
for i in dem:    # loop for each para

    words = re.findall(r'\w+', i.text)   
    cap_words = [word.upper() for word in words]
    word_counts = Counter(cap_words)
    print word_counts

事情是这给了我一个字段计数para,而不是网站的总字数。需要进行哪些更改。此外,如果我想过滤掉像a,an这样的常见文章,我需要包含哪些代码。

1 个答案:

答案 0 :(得分:1)

假设您确实只想查找段落中包含的单词,并且对正则表达式感到满意,这是获取检索到的文档的总字数的最小变化:

soup = BeautifulSoup(ourUrl)
dem = soup.findAll('p') #find paragraphs
word_counts = Counter()
for i in dem:    # loop for each para
    words = re.findall(r'\w+', i.text)
    cap_words = [word.upper() for word in words]
    word_counts.update(cap_words)

print word_counts

要忽略常用词,一种方法是定义可忽略词的冻结集:

word_counts = Counter()
stopwords = frozenset(('A', 'AN', 'THE'))
for i in dem:    # loop for each para
    words = re.findall(r'\w+', i.text)
    cap_words = [word.upper() for word in words if not word.upper() in stopwords]
    word_counts.update(cap_words)