Haskell使用http-conduit问题进行抓取

时间:2013-09-23 18:59:59

标签: haskell

我一直在努力编写我的第一个“真正的”haskell程序,该程序旨在最终从http://www.boxofficemojo.com/weekly/chart/?yr=2012&wk=52&p=.htm形式的页面中删除有关电影的信息。我这样做的第一步是创建一个能够从两个日期之间查询每周信息的函数。我提出的代码不起作用,错误消息超出了我目前的haskell能力。

代码:

import Network.HTTP.Conduit
import Data.Time.Clock
import Data.Time.Calendar.WeekDate
import Data.Time.Calendar (Day, addDays, fromGregorian)
import Control.Monad.Trans.Resource (runResourceT)

import qualified Data.ByteString.Char8 as C
import qualified Data.ByteString.Lazy as L
import qualified Data.ByteString as S


curDate :: IO Day
curDate = fmap utctDay getCurrentTime

dayToWkYr :: Day -> (S.ByteString, S.ByteString)
dayToWkYr day = (C.pack (show year), C.pack (show week))
                where (year, week, _) = toWeekDate day

mkDateList :: Day -> Day -> [Day] -> [Day]
mkDateList start end lst
    | start == end = lst
    | otherwise    = mkDateList (addWk start) end (start:lst)
    where addWk = addDays 7

getMovies' :: Manager -> [Day] -> [Response L.ByteString] -> [Response L.ByteString]
getMovies' manager (d:ds) results = runResourceT $ do
    let (year, week) = dayToWkYr d
    initreq <- parseUrl "http://boxofficemojo.com/weekly/chart/"
    let request = initreq { queryString = "?yr=" `S.append` year `S.append`
                                            "&wk=" `S.append` week}
    response <- httpLbs request manager
    getMovies' manager ds (response:results)

getMovies' _ [] results = results

错误:

scraper.hs:27:37:
    Couldn't match type `[]' with `IO'
    When using functional dependencies to combine
      Control.Monad.Trans.Control.MonadBaseControl [] [],
        arising from the dependency `m -> b'
        in the instance declaration in `Control.Monad.Trans.Control'
      Control.Monad.Trans.Control.MonadBaseControl IO [],
        arising from a use of `runResourceT' at scraper.hs:27:37-48
    In the expression: runResourceT
    In the expression:
      runResourceT
      $ do { let (year, week) = dayToWkYr d;
             initreq <- parseUrl "http://boxofficemojo.com/weekly/chart/";
             let request = ...;
             response <- httpLbs request manager;
             .... }

scraper.hs:33:5:
    Couldn't match type `[]'
                  with `Control.Monad.Trans.Resource.ResourceT []'
    Expected type: Control.Monad.Trans.Resource.ResourceT
                     [] (Response L.ByteString)
      Actual type: [Response L.ByteString]
    In the return type of a call of getMovies'
    In a stmt of a 'do' block:
      getMovies' manager ds (response : results)
    In the second argument of `($)', namely
      `do { let (year, week) = dayToWkYr d;
            initreq <- parseUrl "http://boxofficemojo.com/weekly/chart/";
            let request = ...;
            response <- httpLbs request manager;
            .... }'

如果有人能说清楚我做错了什么,我将非常感激!

1 个答案:

答案 0 :(得分:2)

我绝不是一名哈斯克尔专家,但这是我改为编译的原因。

问题在于函数getMovies'。 首先,返回类型应为IO [Response L.ByteString]。第二个问题在于您对管道资源Monad的处理,函数 runResourceT 返回您在管道流中所做的任何事情,在您的情况下应该是 httpLbs请求管理器的返回值< / strong>即可。因此,您需要从Resource monad中将递归调用移出 getMovies'

getMovies' :: Manager -> [Day] -> [Response L.ByteString] -> IO [Response L.ByteString]
getMovies' manager (d:ds) results = do
  response <- runResourceT $ do  -- we get the response here instead
    let (year, week) = dayToWkYr d
    initreq <- parseUrl "http://boxofficemojo.com/weekly/chart/"
    let request = initreq { queryString = "?yr=" `S.append` year `S.append`
                                            "&wk=" `S.append` week}
    httpLbs request manager
  getMovies' manager ds (response:results)

getMovies' _ [] results = return results -- wrap results in the IO monad.