从python中的链接中提取文本

Question

我在python 2.7中有一个脚本来擦除这个页面中的表格： http://www.the-numbers.com/movie/budgets/all

我想提取每一列，问题是我的代码无法识别具有链接的列（第2列和第3列）。

budgeturl = "http://www.the-numbers.com/movie/budgets/all"
s = urllib.urlopen(budgeturl).read()
htmlpage = etree.HTML(s)
htmltable = htmlpage.xpath("//td[@class='data']/text()")

使用此代码，htmltable [0]为排名，htmltable [1]为生产预算，并从此开始继续。 从我缺少的那些，我需要文本而不是链接。

Answer 1

import urllib

budgeturl = "http://www.the-numbers.com/movie/budgets/all"
s = urllib.urlopen(budgeturl).read()

def find_between( s, first, last ):
    try:
        start = s.index( first ) + len( first )
        end = s.index( last, start )
        return s[start:end]
    except ValueError:
        return ""

s = find_between(s, '<table>', '</table>')

print s[:500]
print '.............................................................'
print s[-250:]

Find string between two substrings

返回：

>>>
<tr><th>&nbsp;</th><th>Release Date</th><th>Movie</th><th>Production Budget</th><th>Domestic Gross</th><th>Worldwide Gross</th></tr>
<tr><td class="data">1</td>
<td><a href="/box-office-chart/daily/2009/12/18">12/18/2009</a></td>
<td><b><a href="/movie/Avatar#tab=summary">Avatar</a></td>
<td class="data">$425,000,000</td>
<td class="data">$760,507,625</td>
<td class="data">$2,783,918,982</td>
<tr>
<tr><td class="data">2</td>
<td><a href="/box-office-chart/daily/2015/12/18">12/18/2015</a></td>
.............................................................
</td>
<td><a href="/box-office-chart/daily/2005/08/05">8/5/2005</a></td>
<td><b><a href="/movie/My-Date-With-Drew#tab=summary">My Date With Drew</a></td>
<td class="data">$1,100</td>
<td class="data">$181,041</td>
<td class="data">$181,041</td>
<tr>

.........................................

  

我需要的是文字而不是链接。

通过http://www.convertcsv.com/html-table-to-csv.htm

Release Date,Movie,Production Budget,Domestic Gross,Worldwide Gross
1,12/18/2009,Avatar,"$425,000,000","$760,507,625","$2,783,918,982"
8/5/2005,My Date With Drew,"$1,100","$181,041","$181,041"

你可以使用beautifulsoup来做同样的事情，见：

beautifulSoup html csv

Answer 2

您需要修改xpath，因为并非所有td元素都有class="data"。 试试这个xpath表达式：//td//text()。

import urllib
from lxml import etree

budgeturl = "http://www.the-numbers.com/movie/budgets/all"
s = urllib.urlopen(budgeturl).read()
htmlpage = etree.HTML(s)
htmltable = htmlpage.xpath("//td//text()")

输出：