Question

我们正在对我们公司的一些Haskell代码进行基准测试，而我们只是遇到了一个非常奇怪的案例。这是一个代码，它对同一个东西进行了2次基准测试。前一个使用Criterion.env，它为所有测试创建一次，后者为每个测试创建env。这是唯一的区别，但是为每个工作台创建环境的运行速度提高了5倍。

有谁知道会导致什么？最小的例子：

module Main where

import Prelude
import Control.Monad
import qualified Data.Vector.Storable.Mutable as Vector
import qualified Data.Vector.Storable         as Vector
import           Data.Vector.Storable         (Vector)
import           Criterion.Main


testf :: Int -> Vector Int -> IO (Vector Int)
testf !i !v = do
    mv <- Vector.unsafeThaw v
    let go j = do
          x <- if j == 0 then return 0 else Vector.unsafeRead mv (j - 1)
          Vector.unsafeWrite mv j (x+1)
          when (j < i - 1) $ go (j + 1)
    go 0
    Vector.unsafeFreeze mv

mkVec :: Int -> IO (Vector Int)
mkVec !i = Vector.unsafeFreeze =<< Vector.new (i + 1)

main :: IO ()
main = do
    defaultMain
        [ bgroup "env per all runs"
            $ (\(i :: Int) -> env (mkVec (10 ^ i))
            $ \v -> bench ("10e" ++ show i)
            $ nfIO (testf (10 ^ i) v))  <$> [7..8]

        , bgroup "env per every run"
            $ (\(i :: Int) -> bench ("10e" ++ show i)
            $ perRunEnv (mkVec (10 ^ i))
            $ (testf (10 ^ i)))  <$> [7..8]
        ]

结果：

benchmarking env per all runs/10e7
time                 17.34 ms   (17.20 ms .. 17.41 ms)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 17.43 ms   (17.34 ms .. 17.67 ms)
std dev              321.5 μs   (142.1 μs .. 591.3 μs)

benchmarking env per all runs/10e8
time                 173.5 ms   (173.2 ms .. 173.8 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 173.8 ms   (173.6 ms .. 174.0 ms)
std dev              279.5 μs   (194.9 μs .. 355.6 μs)
variance introduced by outliers: 12% (moderately inflated)

benchmarking env per every run/10e7
time                 4.289 ms   (1.807 ms .. 5.771 ms)
                     0.924 R²   (0.696 R² .. 1.000 R²)
mean                 8.903 ms   (5.752 ms .. 14.20 ms)
std dev              5.029 ms   (249.0 μs .. 6.244 ms)
variance introduced by outliers: 79% (severely inflated)

benchmarking env per every run/10e8
time                 53.76 ms   (30.23 ms .. 98.51 ms)
                     0.940 R²   (0.920 R² .. 1.000 R²)
mean                 102.9 ms   (68.67 ms .. 127.1 ms)
std dev              36.55 ms   (0.0 s .. 41.99 ms)
variance introduced by outliers: 73% (severely inflated)

Answer 1

正如你可能是同事的问题一样，我无法重现这个问题。注意我使用的是GHC 8.2.2和标准1.3.0.0

benchmarking env per all runs/10e7
time                 18.94 ms   (18.71 ms .. 19.22 ms)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 19.59 ms   (19.39 ms .. 19.99 ms)
std dev              618.3 μs   (379.7 μs .. 952.8 μs)

benchmarking env per all runs/10e8
time                 192.0 ms   (189.5 ms .. 194.9 ms)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 191.8 ms   (190.3 ms .. 193.1 ms)
std dev              1.778 ms   (1.088 ms .. 2.457 ms)
variance introduced by outliers: 14% (moderately inflated)

benchmarking env per every run/10e7
time                 18.97 ms   (18.38 ms .. 19.62 ms)
                     0.999 R²   (0.996 R² .. 1.000 R²)
mean                 18.98 ms   (18.83 ms .. 19.35 ms)
std dev              298.8 μs   (25.39 μs .. 391.3 μs)
variance introduced by outliers: 14% (moderately inflated)

benchmarking env per every run/10e8
time                 194.0 ms   (182.0 ms .. 211.6 ms)
                     0.999 R²   (0.997 R² .. 1.000 R²)
mean                 192.0 ms   (189.0 ms .. 193.9 ms)
std dev              2.850 ms   (0.0 s .. 3.261 ms)
variance introduced by outliers: 19% (moderately inflated)

./p  18.32s user 1.16s system 99% cpu 19.531 total
% ghc-pkg list criterion
...
    criterion-1.3.0.0

我注意到我的版本标准是在@jberryman指出的错误修复后上传的，所以也许这就是区别。

全局/本地环境影响Haskell的Criterion基准测试结果

1 个答案: