Question

我想从这个网站获得手机的费用 http://www.univercell.in/buy/SMART

我试过测试它所以我用过：疤痕贝壳http://www.univercell.in/control/AjaxCategoryDetail?productCategoryId=PRO-SMART&category_id=PRO-SMART&attrName=&min=&max=&sortSearchPrice=&VIEW_INDEX=2&VIEW_SIZE=15&serachupload=&sortupload=

但我无法连接到此网站。当使用ajax加载页面时，我使用firebug找到了start_url。任何人都可以建议我出错的地方

Answer 1

如何编写JavaScript脚本来执行单击页码时已执行的操作，然后只转储从服务器返回的XML。我的意思是尝试调用服务器，就像站点托管在桌面上一样！

点击某个号码时调用的JavaScript函数为paginateList('numberOfPage')，其中numberOfPage是您要访问的页面。

该功能的主体是

function paginateList(viewIndex) {
        var productCategoryId = document.pageSelect.category_id.value;
        var viewSize = document.pageSelect.VIEW_SIZE.value;
        var min = "";
        if(document.pageSelect.min!=null)
            min = document.pageSelect.min.value;
        var max = "";
        if(document.pageSelect.max!=null)
            max = document.pageSelect.max.value;
        var attrName = "";
        if(document.pageSelect.attrName!=null)
        attrName = document.pageSelect.attrName.value;
        if(attrName==""){
         var commaAttr=document.getElementById('commaAttr'); 
          attrName=commaAttr.value;
          }
        var limitView = 'true';
        var sortSearchPrice = "";
        if(document.pageSelect.sortSearchPrice!=null)   
        sortSearchPrice = document.pageSelect.sortSearchPrice.value;
          var url2="/control/AjaxCategoryDetail?productCategoryId="+productCategoryId+"&category_id="+productCategoryId+"&attrName="+attrName+"&min="+min+"&max="+max+"&sortSearchPrice="+sortSearchPrice+"&VIEW_INDEX="+viewIndex+"&VIEW_SIZE="+viewSize+"&serachupload=&sortupload=";
            pleaseWait('Y');
            jQuery.ajax({url: url2,
             data: null,
             type: 'post',
             async: false,
             success: function(data) {
              $('#searchResult').html(data);  
              pleaseWait('N');   
             },
             error: function(data) {
                alert("Error during product searching");
             }
         });

使用它们递归地从每个页面获取数据。

希望它有所帮助！

Answer 2

这是你的蜘蛛：

from scrapy.item import Item, Field
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector


class UnivercellItem(Item):
    vendor = Field()
    model = Field()
    price = Field()

BASE_URL = "http://www.univercell.in/control/AjaxCategoryDetail?productCategoryId=PRO-SMART&category_id=PRO-SMART&attrName=&min=&max=&sortSearchPrice=&VIEW_INDEX=%s&VIEW_SIZE=15&serachupload=&sortupload="

class UnivercellSpider(BaseSpider):
    name = "univercell_spider"
    allowed_domains = ["www.univercell.in"]
    start_urls = [BASE_URL % index for index in range(1, 21)]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        mobiles = hxs.select("//div[@class='productsummary']")
        print mobiles
        for mobile in mobiles:
            item = UnivercellItem()
            item['vendor'] = mobile.select('.//div[1]/div/text()').extract()[0].strip()
            item['model'] = mobile.select('.//div[3]/div[1]/a/text()').extract()[0].strip()
            item['price'] = mobile.select('.//span[@class="regularPrice"]/span/text()').extract()[0].strip()
            yield item

将其保存到spider.py并通过scrapy runspider spider.py -o output.json运行。然后在output.json中，您会看到：

{"model": "T375", "vendor": "LG", "price": "Special Price Click Here"}
{"model": "P725 Optimus 3D Max", "vendor": "LG", "price": "Special Price Click Here"}
{"model": "P705 Optimus L7", "vendor": "LG", "price": "Special Price Click Here"}
{"model": "9320 Curve", "vendor": "Blackberry", "price": "Special Price Click Here"}
{"model": "Xperia Sola", "vendor": "Sony", "price": "Rs.14,500.00"}
{"model": "Xperia U", "vendor": "Sony", "price": "Special Price Click Here"}
{"model": "Lumia 610", "vendor": "Nokia", "price": "Special Price Click Here"}
...

希望有所帮助。

如何检索数据...使用ajax加载页面

2 个答案: