我正在编写一个程序,其中输入文件被分成多个文件(Shamir的秘密共享方案)。
这是我想象的管道:
bsl !! 0
将被写入文件0,bsl !! 1
将被写入文件1,依此类推)我发现了一个关于多个输入文件here的问题,但在他们的情况下,整个管道为每个输入文件运行一次,而对于我的程序,我正在写入中的多个输出文件管道。
我也正在查看Conduit源代码here以查看我是否可以自己实现multiSinkFile,但是我对sinkFile的Consumer类型稍微感到困惑,如果我尝试深入挖掘则更多......(我还是初学者)
所以,问题是,我应该如何实现像multiSinkFile这样的函数,它允许将多个文件作为接收器的一部分写入?
感谢任何提示!
澄清
假设我们想要在包含二进制值“ABCDEF”的文件中共享Shamir的秘密(分为3部分)。
(因此我们输入文件srcFile
和输出文件outFile0
,outFile1
和outFile2
)
我们首先从文件中读取“ABC”,并进行处理,这将为我们提供一个列表,例如["133", "426", "765"]
。因此"133"
将写入outFile0
,"426"
至outFile1
和"765"
至outFile2
。然后我们从srcFile
读取“DEF”,对其进行处理,并将相应的输出写入每个输出文件。
编辑:
感谢您的回答。我花了一些时间来了解ZipSinks等会发生什么,我编写了一个简单的测试程序,它接受源文件的输入并简单地将其写入3个输出文件。希望这将有助于其他人。
{-# LANGUAGE NoImplicitPrelude #-}
{-# LANGUAGE RankNTypes #-}
{-# LANGUAGE OverloadedStrings #-}
import ClassyPrelude.Conduit
import Safe (atMay)
import Text.Printf
import Filesystem.Path.CurrentOS (decodeString, encodeString)
import Control.Monad.Trans.Resource (runResourceT, ResourceT(..))
-- get the output file name given the base (file) path and the split number
getFileName :: FilePath -> Int -> FilePath
getFileName basePath splitNumber = decodeString $ encodeString basePath ++ "." ++ printf "%03d" splitNumber
-- Get the sink file, given a filepath generator (that takes an Int) and the split number
idxSinkFile :: MonadResource m
=> (Int -> FilePath)
-> Int
-> Consumer [ByteString] m ()
idxSinkFile mkFP splitNumber =
concatMapC (flip atMay splitNumber) =$= sinkFile (mkFP splitNumber)
sinkMultiFiles :: MonadResource m
=> (Int -> FilePath)
-> [Int]
-> Sink [ByteString] m ()
sinkMultiFiles mkFP splitNumbers = getZipSink $ otraverse_ (ZipSink . idxSinkFile mkFP) splitNumbers
simpleConduit :: Int -> Conduit ByteString (ResourceT IO) [ByteString]
simpleConduit num = mapC (replicate num)
main :: IO ()
main = do
let mkFP = getFileName "test.txt"
splitNumbers = [0..2]
runResourceT $ sourceFile "test.txt" $$ simpleConduit (length splitNumbers) =$ sinkMultiFiles mkFP splitNumbers
答案 0 :(得分:8)
一种可能性是让您的算法输出类似(Int, ByteString)
的内容,其中Int
是指定输出文件的索引(当然您可以使用任何其他类型作为键)。这样,管道可以决定它想要附加其输出的文件。
import Data.Conduit
import qualified Data.Conduit.List as C
import qualified Data.Foldable as F
-- | Filter only pairs tagged with the appropriate key.
filterInputC :: (Monad m, Eq k) => k -> Conduit (k, a) m a
filterInputC idx = C.filter ((idx ==) . fst) =$= C.map snd
-- | Prepend a given sink with a filter.
filterInput :: (Monad m, Eq k) => k -> Sink a m r -> Sink (k, a) m r
filterInput idx = (filterInputC idx =$)
-- | Given a list of sinks, create a single sink that directs received values
-- depending on the index.
multiSink_ :: (Monad m) => [Sink a m ()] -> Sink (Int, a) m ()
multiSink_ = getZipSink . F.sequenceA_ . fmap ZipSink
. zipWith filterInput [0..]
更新:以下示例显示了如何使用multiSink_
(测试接收器只是使用适当的前缀将所有内容打印到stdout,而不是写入文件)。
-- | A testing sink that just prints its input, marking it with
-- a given prefix.
testSink :: String -> Sink String IO ()
testSink prefix = C.mapM_ (putStrLn . (prefix ++))
-- | An example that produces indexed output.
testSource :: (Monad m) => Source m (Int, String)
testSource = do
yield (0, "abc")
yield (0, "def")
yield (1, "opq")
yield (0, "0")
yield (1, "1")
yield (2, "rest")
main :: IO ()
main = testSource $$ multiSink_ (map testSink ["1: ", "2: ", "3: "])
答案 1 :(得分:6)
有多种方法可以实现,具体取决于您是要动态增加要写入的文件数,还是只保留固定数量。这是一个固定文件列表的例子:
{-# LANGUAGE NoImplicitPrelude #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE ViewPatterns #-}
import ClassyPrelude.Conduit
import Safe (atMay)
idxSinkFile :: MonadResource m
=> (Int -> FilePath)
-> Int
-> Consumer [ByteString] m ()
idxSinkFile mkFP idx =
concatMapC (flip atMay idx) =$= sinkFile fp
where
fp = mkFP idx
sinkMultiFiles :: MonadResource m
=> (Int -> FilePath)
-> [Int]
-> Sink [ByteString] m ()
sinkMultiFiles mkFP indices = getZipSink $ otraverse_ (ZipSink . idxSinkFile mkFP) indices
someFunc :: ByteString -> [ByteString]
someFunc (decodeUtf8 -> x) = map encodeUtf8 [x, toUpper x, toLower x]
mkFP :: Int -> FilePath
mkFP 0 = "file0.txt"
mkFP 1 = "file1.txt"
mkFP 2 = "file2.txt"
src :: Monad m => Producer m ByteString
src = yieldMany $ map encodeUtf8 $ words "Hello There World!"
main :: IO ()
main = do
let indices = [0..2]
runResourceT $ src $$ mapC someFunc =$ sinkMultiFiles mkFP indices
forM_ indices $ \idx -> do
let fp = mkFP idx
bs <- readFile fp
print (fp, bs :: ByteString)