Question

I'm trying to use request-promise to scrape a the price of an item from Asos.com. When I attempt to run the code below, I get a 403 error. Is this possible that I get this error although the URL that I am attempting to scrape is publicly available, no key required?

http://www.asos.com/api/product/catalogue/v2/stockprice?productIds=10000496&currency=SEK&keyStoreDataversion=7jhdf34h-6&store=ROE

I know some site are against scraping in their ToS but I just want to be sure I am not just performing this wrong or if I am actually getting blocked by the site.

const rp = require('request-promise');

var url = 'http://www.asos.com/api/product/catalogue/v2/stockprice?productIds=10000496&currency=SEK&keyStoreDataversion=7jhdf34h-6&store=ROE';

rp({ url:url, json:true })
  .then(function (data) {
    console.log(data.productPrice.current.value);
  })
  .catch(function (reason) {
    console.error("%s; %s", reason.error.message, reason.options.url);
    console.log("%j", reason.response.statusCode);
  });

Answer 1

您应该添加“标头”参数：

rp({ 
  url:url, 
  headers: {
    'User-Agent': 'Request-Promise'
  }
  json:true 
})

Getting 403 Error using request-promise to scrape site for price

1 个答案: