Question

我有200,000个链接正在尝试下载，我曾经尝试一次全部下载，但是遇到了内存问题。

我正在尝试创建一个函数，一次可下载1000个链接并将其保存在文件夹中。

包装：

library(dplyr)
library(purrr)
library(edgarWebR)

一小部分数据如下：

数据1：

urls_to_parse <- c("https://www.sec.gov/Archives/edgar/data/1750/000104746918004978/a2236183z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746917004528/a2232622z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746916014299/a2228768z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746915006136/a2225345z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746914006243/a2220733z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746913007797/a2216052z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746912007300/a2210166z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746911006302/a2204709z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746910006500/a2199382z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746909006783/a2193700z10-k.htm"
)

然后我应用以下功能下载这10个链接

parsed_files <- map(urls_to_parse, possibly(parse_filing, otherwise = NA))

将其存储为一个不错的列表，然后我可以应用names(parsed_files) <- urls_to_parse来命名列表，将其作为链接的下载来源。我还可以使用output <- plyr::ldply(parsed_files, data.frame)将所有内容存储在一个不错的数据框中。

使用以下数据，我如何创建批处理以按10个批次下载数据？

我目前所拥有的：

start = 1
end = 100

output <- NULL
output_fin <- NULL

for(i in start:end){
  output[[i]] <- map(urls_to_parse[[i]], possibly(parse_filing, otherwise = NA))
  names(output) <- urls_to_parse[start:end]
  save(output_fin, file = paste0("C:/Users/Downloads/data/",i, "output.RData"))
}

我确信使用函数会有更好的方法，因为这段代码会破坏某些结果。

更多数据：-100个链接

urls_to_parse <- c("https://www.sec.gov/Archives/edgar/data/1750/000104746918004978/a2236183z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746917004528/a2232622z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746916014299/a2228768z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746915006136/a2225345z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746914006243/a2220733z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746913007797/a2216052z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746912007300/a2210166z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746911006302/a2204709z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746910006500/a2199382z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746909006783/a2193700z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746908008126/a2186742z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000110465907055173/a07-18543_110k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000110465906047248/a06-15961_110k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000110465905033688/a05-12324_110k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746904023905/a2140220z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746903028005/a2116671z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000091205702033450/a2087919z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/61478/000095012310108231/c61492e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/61478/000095015208010514/n48172e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/61478/000095013707018659/c22309e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/61478/000095013707000193/c11187e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/61478/000095013406000594/c01109e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/61478/000120677405000032/d16006.htm", 
"https://www.sec.gov/Archives/edgar/data/61478/000120677404000013/d13773.htm", 
"https://www.sec.gov/Archives/edgar/data/61478/000104746903001075/a2097401z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/61478/000091205702001614/a2067550z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/319126/000115752308008030/a5800571.htm", 
"https://www.sec.gov/Archives/edgar/data/319126/000115752307009801/a5515869.htm", 
"https://www.sec.gov/Archives/edgar/data/319126/000115752306009238/a5227919.htm", 
"https://www.sec.gov/Archives/edgar/data/730469/000073046908000102/alpharmainc_10k.htm", 
"https://www.sec.gov/Archives/edgar/data/730469/000073046907000017/alo10k2006.htm", 
"https://www.sec.gov/Archives/edgar/data/730469/000073046906000027/alo10k2005.htm", 
"https://www.sec.gov/Archives/edgar/data/730469/000073046905000021/alo10k2004final.htm", 
"https://www.sec.gov/Archives/edgar/data/730469/000073046904000058/alo10k2003master.htm", 
"https://www.sec.gov/Archives/edgar/data/730469/000073046903000001/alo10k.htm", 
"https://www.sec.gov/Archives/edgar/data/730469/000073046902000004/alo10k2001.htm", 
"https://www.sec.gov/Archives/edgar/data/730469/000073046901500003/alo.htm", 
"https://www.sec.gov/Archives/edgar/data/4515/000000620118000009/a10k123117.htm", 
"https://www.sec.gov/Archives/edgar/data/4515/000119312517051216/d286458d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/4515/000119312516474605/d78287d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/4515/000119312515061145/d829913d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/4515/000000620114000004/aagaa10k-20131231.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000000620113000023/amr-10kx20121231.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000119312512063516/d259681d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000095012311014726/d78201e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000000620110000006/ar123109.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000000620109000009/ar120810k.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000000451508000014/ar022010k.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000095013407003888/d43815e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000095013406003715/d33303e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000095013405003726/d22731e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000095013404002668/d12953e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/6201/000104746903013301/a2108197z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/65695/000095013407003823/h42902e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/65695/000095012906002343/h31028e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/65695/000095012905002955/h22337e10vk.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000156459018005085/cece-10k_20171231.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000156459017004264/cece-10k_20161231.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000156459016015157/cece-10k_20151231.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312515095828/d864880d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312514098407/d661608d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312513109153/d444138d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312512119293/d293768d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312511067373/d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312510069639/d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312509055504/d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312508058939/d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312507071909/d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312506068031/d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312505077739/d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/3197/000119312504052176/d10k.htm", 
"https://www.sec.gov/Archives/edgar/data/2601/000110465910047121/a10-16705_110k.htm", 
"https://www.sec.gov/Archives/edgar/data/2601/000114420409046933/v159572_10k.htm", 
"https://www.sec.gov/Archives/edgar/data/2601/000110465906060737/a06-19311_110k.htm", 
"https://www.sec.gov/Archives/edgar/data/2601/000104746905022854/a2162888z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/2601/000104746904028585/a2143353z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/2601/000104746903031974/a2119476z10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000143774918010388/avx20180331_10k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916317000028/avx-20170331x10k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916316000079/avx-20160331x10k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916315000024/avx-20150331x10k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916314000035/avx-20140331x10k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916313000022/avx-20130331x10k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916312000024/avxform10kfy12.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916311000013/avxform10kfy11.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916310000020/avxform10kfy10.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916309000117/form10kfy09.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916308000192/form10qq1fy09.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916308000101/form10kfy08.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916307000122/form10kfy07.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916306000102/avxfy06form10-k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916305000094/fy0510k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916304000091/fy0410k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916303000020/fy0310k.htm", 
"https://www.sec.gov/Archives/edgar/data/859163/000085916302000007/r10k-0302.htm", 
"https://www.sec.gov/Archives/edgar/data/7286/000076462218000018/pnw2017123110-k.htm", 
"https://www.sec.gov/Archives/edgar/data/7286/000076462217000010/pnw2016123110-k.htm", 
"https://www.sec.gov/Archives/edgar/data/7286/000076462216000087/pnw2015123110-k.htm", 
"https://www.sec.gov/Archives/edgar/data/7286/000076462215000013/pnw12311410-k.htm", 
"https://www.sec.gov/Archives/edgar/data/7286/000110465914012068/a13-25897_110k.htm"
)

Answer 1

像您展示的那样，忙着做批处理工作是一个坏主意。如果要下载1000个文件，如何从错误中恢复？

性能不仅取决于计算机的配置，而且网络性能至关重要。

以下是一些建议。

选项1

将所有URL划分为多个批次，以便能够并行下载它们。要下载的文件数可能等于您计算机中的内核数。看这个问题； reading multiple files quickly in R
将这些批次存储在队列对象中-例如：使用https://cran.r-project.org/web/packages/dequer/dequer.pdf
弹出队列，并在并行文件下载功能中使用这批URL。
使用可重试的文件下载功能，例如-HTTP error 400 in R, error handling, How to retry instead of forcing to stop?
队列完成后，移至下一个分区。
将整个操作包装在可重试的循环中。例如; How to retry a statement on error?

为什么要使用队列？因为您可以轻松地重试错误。

伪代码


file_url_partitions <- partion_as_batches(all_urls, batch_size) 
attempts = 3
while( file_url_partitions is not empty && attempt <= 3 ) {
  batch = file_url_partitions.pop()

  tryCatch({
   download_parallel(batch)
  }, some_exception = function(se) {
    file_url_partitions.push(batch)
    attemp = attempt+1 
  })
}

注意：我现在无法访问R Studio /环境，因此无法尝试。

选项2 使用下载管理器/类似工具分别下载文件，然后使用下载的文件。

一些有用的资源： https://www.r-bloggers.com/r-with-parallel-computing-from-user-perspectives/ http://adv-r.had.co.nz/beyond-exception-handling.html

批量下载数据并将数据保存到文件夹

1 个答案: