为什么写入文件时这个Haskell程序会挂起?

时间:2016-08-01 18:40:23

标签: multithreading haskell io

如果使用runhaskell或已编译但未使用-O2运行,则以下程序有效。如果使用-O2进行编译,它似乎会挂起。

我正在使用GHC 7.10.2。

我已将最小/最大迭代次数分别更改为10和20。它会 在文件test.out中生成20到100 MB的输出。 运行时间约为15-60秒。

程序说明

下面是一个多线程程序,它有一个工作池和一个管理器。工人生成用于绘制Buddhabrot的痕迹,将其放入队列中,并且管理员定期清空队列并将数据写入磁盘。当生成了一定数量的数据时,程序就会停止。

但是当程序运行时,管理器线程只进行一次检查,然后它就会卡住(工作线程仍在运行)。但是,如果我删除管理器线程写入文件的部分,那么一切似乎都有效。我只是不明白为什么......

import Control.Concurrent
import Control.Concurrent.Async
import Control.Concurrent.STM
import Control.Monad
  ( forever
  , unless
  )
import Control.Monad.Loops
import System.IO
import System.Random

import qualified Data.Binary as B
import qualified Data.ByteString.Lazy as BS

type Coord = (Double, Double)

type Trace = [Coord]

-- | Represents a rectangle in the complex plane, bounded by a lower left
-- coordinate and an upper right coordinate.
data Plane
  = Plane { ll :: Coord, ur :: Coord }
  deriving (Show)

-- | Adds two coordinates.
(+.) :: Coord -> Coord -> Coord
(r1, i1) +. (r2, i2) = (r1 + r2, i1 + i2)

-- | Multiplies two coordinates.
(*.) :: Coord -> Coord -> Coord
(r1, i1) *. (r2, i2) = (r1*r2 - i1*i2, r1*i2 + r2*i1)

-- | Computes the square of a coordinate.
square :: Coord -> Coord
square (r, i) = (r*r - i*i, 2*r*i)

-- | Distance from origin to a given coordinate.
distFromOrigin :: Coord -> Double
distFromOrigin (r, i) = r*r + i*i

-- | A structure for passing data to the worker threads.
data WorkerData
  = WorkerData { wdMinIt :: Int
               , wdMaxIt :: Int
               , wdTraceQueue :: TQueue Trace
                 -- ^ A queue of traces to be written to disk.
               }

-- | A structure for passing data to the manager thread.
data ManagerData
  = ManagerData { mdOutHandle :: Handle
                   -- ^ Handle to the output file.
                , mdNumTraces :: Integer
                  -- ^ Number of traces to gather.
                , mdTraceQueue :: TQueue Trace
                  -- ^ A queue of traces to be written to disk.
                }

-- | Encodes an entity to binary bytestring.
encode :: B.Binary a => a -> BS.ByteString
encode = B.encode

-- | Writes a lazy bytestring to file.
writeToFile :: Handle -> BS.ByteString -> IO ()
writeToFile = BS.hPut

mkManagerData :: TQueue Trace -> IO ManagerData
mkManagerData t_queue =
  do let out_f = "test.out"
     out_h <- openBinaryFile out_f WriteMode
     let num_t = 1000
     return $ ManagerData { mdOutHandle = out_h
                          , mdNumTraces = num_t
                          , mdTraceQueue = t_queue
                          }

mkWorkerData :: TQueue Trace -> IO WorkerData
mkWorkerData t_queue =
  do let min_it =  10 -- 1000
         max_it =  20 -- 10000
     return $ WorkerData { wdMinIt = min_it
                         , wdMaxIt = max_it
                         , wdTraceQueue = t_queue
                         }

-- | The actions to be performed by the manager thread.
runManager :: ManagerData -> IO ()
runManager m_data =
  do execute 0
     return ()
  where execute count =
          do new_traces <- purgeTQueue $ mdTraceQueue m_data
             let new_count = count + (toInteger $ length new_traces)
             putStrLn $ "Found " ++ (show $ new_count) ++ " traces so far. "
             if length new_traces > 0
             then do putStrLn $ "Writing new traces to file..."
                     _ <- mapM (writeToFile (mdOutHandle m_data))
                               (map encode new_traces)
                     putStr "Done"
             else return ()
             putStrLn ""
             unless (new_count >= mdNumTraces m_data) $
               do threadDelay (1000 * 1000) -- Sleep 1s
                  execute new_count

-- | The actions to be performed by a worker thread.
runWorker :: WorkerData -> IO ()
runWorker w_data =
  forever $
    do c <- randomCoord
       case computeTrace c (wdMinIt w_data) (wdMaxIt w_data) of
         Just t  -> atomically $ writeTQueue (wdTraceQueue w_data) t
         Nothing -> return ()

-- | Reads all values from a given 'TQueue'. If any other thread reads from the
-- same 'TQueue' during the execution of this function, then this function may
-- deadlock.
purgeTQueue :: Show a => TQueue a -> IO [a]
purgeTQueue q =
  whileJust (atomically $ tryReadTQueue q)
            (return . id)

-- | Generates a random coordinate to trace.
randomCoord :: IO Coord
randomCoord =
  do x <- randomRIO (-2.102613, 1.200613)
     y <- randomRIO (-1.237710, 1.239710)
     return (x, y)

-- | Computes a trace, using the classical Mandelbrot function, for a given
-- coordinate and minimum and maximum iteration count. If the length of the
-- trace is less than the minimum iteration count, or exceeds the maximum
-- iteration count, 'Nothing' is returned.
computeTrace
  :: Coord
  -> Int
     -- ^ Minimum iteration count.
  -> Int
     -- ^ Maximum iteration count.
  -> Maybe Trace
computeTrace c0 min_it max_it =
  if isUsefulCoord c0
  then let step c = square c +. c0
           computeIt c it = if it < max_it
                            then computeIt (step c) (it + 1)
                            else it
           computeTr [] = error "computeTr: empty list"
           computeTr (c:cs) = if length cs < max_it
                              then computeTr (step c:(c:cs))
                              else (c:cs)
           num_it = computeIt c0 0
       in if num_it >= min_it && num_it <= max_it
          then Just $ reverse $ computeTr [c0]
          else Nothing
  else Nothing

-- | Checks if a given coordinate is useful by checking if it belongs in the
-- cardioid or period-2 bulb of the Mandelbrot.
isUsefulCoord :: Coord -> Bool
isUsefulCoord (x, y) =
  let t1 = x - 1/4
      p = sqrt (t1*t1 + y*y)
      is_in_cardioid = x < p - 2*p*p + 1/4
      t2 = x + 1
      is_in_bulb = t2*t2 + y*y < 1/16
  in not is_in_cardioid && not is_in_bulb

main :: IO ()
main =
  do t_queue <- newTQueueIO
     m_data <- mkManagerData  t_queue
     w_data <- mkWorkerData t_queue
     let num_workers = 1
     workers <- mapM async (replicate num_workers (runWorker w_data))
     runManager m_data
     _ <- mapM cancel workers
     _ <- mapM waitCatch workers
     putStrLn "Tracing finished"

为什么失败

在回顾下面的答案后,我终于意识到为什么它没有按预期工作。程序不会挂起,但是管理器线程对单个跟踪进行编码所花费的时间大约为几十秒(并且在编码时它消耗几兆字节)!这意味着即使队列耗尽时仍有数十条痕迹 - 在我的机器上,工作人员在队列被管理器线程耗尽之前设法产生大约250条痕迹 - 它将在下一次耗尽之前永远消耗。 / p>

因此,除非管理员线程的工作大大减少,否则我选择的解决方案很重要。为此,我将不得不放弃将每个单独的跟踪转储到文件的想法,而是在生成后处理它。

1 个答案:

答案 0 :(得分:3)

问题是双重的:

(1)经理线程不处理任何 跟踪,直到它耗尽队列。

(2)工作线程可以非常快速地将元素添加到队列中。

这导致经理人线程很少获胜的竞赛。 [这也解释了-O2观察到的行为 - 优化只是使工作线程更快。 ]

添加一些调试代码表明worker可以添加 队列中的项目超过每秒100K Traces。 而且,即使经理只对此感兴趣 写出前1000个痕迹,工人没有 停在这个极限。所以,在某些情况下, 经理永远无法退出这个循环:

purgeTQueue q = whileJust (atomically $ tryReadTQueue q) (return . id)

修复代码的最简单方法是使用 管理器线程使用readTQueue来读取和处理一个 项目一次离开队列。这也将阻止 当队列我们空虚时,经理线程 需要经理线程定期睡觉。

purgeTQueue更改为:

purgeTQueue = do item <- atomically $ readTQueue (mdTraceQueue m_data)
                 return [item]

并从threadDelay中移除runManager可解决问题。

Lib4.hs模块中可用的示例代码:https://github.com/erantapaa/mandel