Question

您好，我正在尝试使用R通过ProPublica NonProfit Explorer API下载pdf文件：https://projects.propublica.org/nonprofits/api

当我查询API时，它会返回指向pdfs的链接。但是，这些链接重定向到AWS，例如https://projects.propublica.org/nonprofits/download-filing?path=2015_06_T%2F13-1624100_990T_201406.pdf

我已尝试按照本讨论中的建议指定method = "curl", extra='-L'：R download file redirect。这将返回状态127。

我也尝试过使用CRAN的“Downloader”软件包。这会下载一个文件，但是当我尝试打开它时，Adobe会以某种方式损坏，因为Adobe说“内存不足”。

有没有人有任何建议？

Answer 1

只需使用httr（您也应该将其用于API访问）。 write_disk()是你的bff：

library(httr)

pp_doc_url <- "https://projects.propublica.org/nonprofits/download-filing?path=2015_06_T%2F13-1624100_990T_201406.pdf"

GET(
  url = pp_doc_url,
  write_disk("file.pdf"),
  verbose()
) -> res

这里显示的是详细输出，显示重定向：

## -> GET /nonprofits/download-filing?path=2015_06_T%2F13-1624100_990T_201406.pdf HTTP/1.1
## -> Host: projects.propublica.org
## -> User-Agent: libcurl/7.54.0 r-curl/3.0 httr/1.3.1
## -> Accept-Encoding: gzip, deflate
## -> Accept: application/json, text/xml, application/xml, */*
## -> 
## <- HTTP/1.1 302 Found
## <- Content-Type: text/html; charset=utf-8
## <- X-Frame-Options: SAMEORIGIN
## <- X-XSS-Protection: 1; mode=block
## <- X-Content-Type-Options: nosniff
## <- Location: https://pp-990.s3.amazonaws.com/2015_06_T/13-1624100_990T_201406.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI7C6X5GT42DHYZIA%2F20171202%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171202T002756Z&X-Amz-Expires=1800&X-Amz-SignedHeaders=host&X-Amz-Signature=f90caae6a793239be8342d0ecbd96ff6f80b1821921cfadae00f78129a38a79f
## <- Cache-Control: max-age=0, private, must-revalidate
## <- Content-Encoding: gzip
## <- Transfer-Encoding: chunked
## <- Accept-Ranges: bytes
## <- Date: Sat, 02 Dec 2017 00:27:57 GMT
## <- Via: 1.1 varnish
## <- Connection: keep-alive
## <- X-Served-By: cache-bos8228-BOS
## <- X-Cache: MISS
## <- X-Cache-Hits: 0
## <- X-Timer: S1512174477.810292,VS0,VE194
## <- Vary: Accept,Accept-Encoding,Content-Type
## <- 
## -> GET /2015_06_T/13-1624100_990T_201406.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI7C6X5GT42DHYZIA%2F20171202%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171202T002756Z&X-Amz-Expires=1800&X-Amz-SignedHeaders=host&X-Amz-Signature=f90caae6a793239be8342d0ecbd96ff6f80b1821921cfadae00f78129a38a79f HTTP/1.1
## -> Host: pp-990.s3.amazonaws.com
## -> User-Agent: libcurl/7.54.0 r-curl/3.0 httr/1.3.1
## -> Accept-Encoding: gzip, deflate
## -> Accept: application/json, text/xml, application/xml, */*
## -> 
## <- HTTP/1.1 200 OK
## <- x-amz-id-2: fycJGU5JQZ+o+aTOWFa86ZFyasv7XEH6RGsmXNo29+CtgDC8IZ438Ek61Bo/nUlRhk3fPKPXdMg=
## <- x-amz-request-id: AB2E8B3421A6B7BB
## <- Date: Sat, 02 Dec 2017 00:27:58 GMT
## <- Last-Modified: Thu, 13 Aug 2015 19:22:03 GMT
## <- ETag: "fd89377252531684bec1828db05c54e6"
## <- Cache-Control: no-cache, no-store
## <- Content-Language: en
## <- Accept-Ranges: bytes
## <- Content-Type: application/pdf
## <- Content-Length: 537542
## <- Server: AmazonS3
## <-

这里是响应对象的内容：

res
## Response [https://pp-990.s3.amazonaws.com/2015_06_T/13-1624100_990T_201406.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI7C6X5GT42DHYZIA%2F20171202%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171202T002756Z&X-Amz-Expires=1800&X-Amz-SignedHeaders=host&X-Amz-Signature=f90caae6a793239be8342d0ecbd96ff6f80b1821921cfadae00f78129a38a79f]
##   Date: 2017-12-02 00:27
##   Status: 200
##   Content-Type: application/pdf
##   Size: 538 kB
## <ON DISK>  file.pdf

并且，以下是该文件的下载证明：

file.info("file.pdf")
##            size isdir mode               mtime               ctime               atime uid gid    uname grname
## file.pdf 537542 FALSE  644 2017-12-01 19:27:57 2017-12-01 19:27:57 2017-12-01 19:27:58 xxx  xx xxxxxxxx  xxxxx

在＆＃34; production＆＃34;。

中关闭verbose()

R下载文件重定向错误

1 个答案: