Question

="" I got a html string(responseData) with lots of <li> .. </li> sets. I want to parse following data from each set of <li> ...</li> :

1: call.php?category=fruits&amp;fruitid=123456
2: mango season
3: http://imagehosting.com/images/fru_123456.png

After getting the above data i want to make http request for each url(call.php?category=fruits&fruitid=123456) found an get its result for further processing. Could any one show me how get all above data ?Thanks in advance.

Data sets to scrape :

<div id="Category"><span>lists</span><ul>
 <li><a id="FR123456" onclick="setFood(false);setSeasonFruitID('123456');getit('call.php?category=fruits&amp;fruitid=123456&amp;',detailFruit,false);">mango season</a><img src="http://imagehosting.com/images/fru_123456.png">
 </li>



        <li><a id="FR123457" onclick="setFood(false);setSeasonFruitID('123457');getit('call.php?category=fruits&amp;fruitid=123457&amp;',detailFruit,false);">cherry season</a><img src="http://imagehosting.com/images/fru_123457.png">
                    </li>
 </ul></div>

python:

url ='http://somewebsite.com/data.php';

req = urllib2.Request(url)
req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0')
response = urllib2.urlopen(req)
responseData = response.read()

Answer 1

看一下python附带的HTMLParser。我将很快发布一些代码。对于您的个人用例，这里有一些示例代码。这是一个漂亮的转储刮刀，但我认为你应该得到要点：

from HTMLParser import HTMLParser

# create a subclass and override the handler methods
class MyHTMLParser(HTMLParser):

    srces = []
    datas = []
    onclicks = []

    def handle_starttag(self, tag, attrs):
        for attr in attrs:
            if attr[0] == 'src':
                 self.srces.append(attr[1])
            if  attr[0] == 'onclick':
                  self.onclicks.append(attr[1].split("getit('")[1].split("',")[0])


    def handle_data(self, data):
        if data != " " and not data.startswith('\n'):
           self.datas.append(data)

parser = MyHTMLParser()
parser.feed(""" <li><a id="FR123456" onclick="setFood(false);setSeasonFruitID('123456');getit('call.php?category=fruits&amp;fruitid=123456&amp;',detailFruit,false);">mango season</a><img src="http://imagehosting.com/images/fru_123456.png">
                </li>

    <li><a id="FR123457" onclick="setFood(false);setSeasonFruitID('123457');getit('call.php?category=fruits&amp;fruitid=123457&amp;',detailFruit,false);">cherry season</a><img src="http://imagehosting.com/images/fru_123457.png">
                </li>"""
)


#To fit your output
for i in range(0,len(parser.srces)):
    print "1: " + parser.onclicks[i] + "\n" + "2: " + parser.datas[i] + "\n" + "3: " + parser.srces[i] + "\n"


#  Output
# 1: call.php?category=fruits&fruitid=123456&
# 2: mango season
# 3: http://imagehosting.com/images/fru_123456.png

# 1: call.php?category=fruits&fruitid=123457&
# 2: cherry season
# 3: http://imagehosting.com/images/fru_123457.png

修改

回答你的后续问题：

base_url ='http://somewebsite.com/' for i in range(0,len(parser.onclicks)): url = base_url + parser.onclicks[i][:-1] # use onclick without the last character (&) req = urllib2.Request(url) req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0') response = urllib2.urlopen(req) responseData = response.read()

如何刮取onclick值（<li> <a id="FR123456" onclick=")" and="" its="" name="" and="" image="" using="" python?

1 个答案: