Question

在Haskell / Scalpel刮刀上，我已经完成了99％的工作，可以批量提取数千个.gz文件并将其本地存储在我的客户端上。

我现在正在尝试将成块的[[String]]放入IO，以便可以保存它们。

我正在使用import Network.HTTP.Req。

-- This is the argument for the downloadFiles function
chunkLinks :: IO [[String]]
chunkLinks = fmap (chunksOf 10) filterNonGz

-- THIS IS WHAT YOU EXECUTE 
downloadFile :: (MonadHttp m) => String -> m ()
downloadFile url = do
    -- Pull them into memory
    contents <- req GET (https (T.pack url)) NoReqBody bsResponse mempty

    -- Get the filename itself
    let fileName = head $ reverse $ T.splitOn "/" (T.pack url)

    -- Write them to the client (the filename will be the url)
    liftIO $ BSS.writeFile (dataDir ++ (T.unpack fileName)) (responseBody contents)


dataDir = "./Data/LODES/"

eatChunks :: (MonadHttp m) =>  m [()]
eatChunks = do
    (wtf :: [[String]]) <- liftIO $ (take (5) <$> chunkLinks)
    fmap concat $ liftIO $ mapM (liftIO . mapConcurrently (liftIO . downloadFile)) wtf

我在最后一行的downloadFile上遇到错误。

错误是：

• Could not deduce (MonadHttp IO)
    arising from a use of ‘downloadFile’
  from the context: MonadHttp m
    bound by the type signature for:
               eatChunks :: forall (m :: * -> *). MonadHttp m => m [()]
    at /private/var/folders/70/kchtzk4j0hs398f95ywd78x00000gn/T/ghc-mod39229/UsCensusDataLodesScraper39228-2281.hs:195:1-37

请原谅乱扔垃圾的liftIO。他们是拼命的尝试，以使它起作用...

有人知道下一步该怎么做吗？

作为参考，chunkLinks的输出如下所示：

[["https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2002.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2003.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2004.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2005.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2006.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2007.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2008.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2009.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2010.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2011.csv.gz"], ...etc.

使用Haskell的Req库和Scalpel将文件保存到客户端

0 个答案: