先前的代码似乎会影响以后的函数调用时间

Question

我正在尝试对用C ++实现的一组较大算法的相对较小部分进行基准测试。为简化起见，可以说每种算法都是通过两个函数（分别称为foo()和bar()）实现的，这些函数可以按任意顺序重复调用，并且这些函数可以对某些内部算法进行更改数据结构。除其他外，我想通过分别测量在foo()和bar()中花费的总时间来比较算法的性能。

现在我有两种算法：算法A在foo()中有一些工作，但是在bar()中却很少，而算法B在foo()中却没有任何作用（foo()是实际上这里是一个空函数），但bar()中有很多工作。我在这里观察到的意外情况是，在许多情况下，算法B在foo()中花费的总时间大于算法A在foo()中花费的总时间。经过一些调试后，我发现对于算法B，在调用bar（）之后对foo（）的第一次调用会花费很多时间，而对foo（）的后续调用往往会更快。

为了确定这种效果，我想出了算法B的以下简化形式，它包括一个空函数（对应于foo()）和两个我试图模拟工作的函数（对应于bar()，“实际” bar()基本上也只是分配空间并在数据结构上迭代）：

b.h：

#ifndef B_H
#define B_H

void foo_emptyFunction(unsigned long long u); // foo()
void bar_expensiveFunction1();   // bar() - version 1
void bar_expensiveFunction2();   // bar() - version 2

#endif

b.cpp

#include "b.h"

#include <iostream>
#include <vector>
#include <math.h>


void foo_emptyFunction(unsigned long long )
{
    // nothing
}

void bar_expensiveFunction1() {
  std::vector<unsigned long> vec;
  for (auto i = 0UL; i < 1000000UL; i++) {
      vec.push_back(i);
  }
  std::cout << "Created and filled a vector with " << vec.size() << " elements." << std::endl;
}

void bar_expensiveFunction2() {
  std::vector<unsigned long> vec;
  for (auto i = 1UL; i <= 1000000UL; i++) {
      vec.push_back(i);
  }
        auto sum = 0ULL;
        auto sumSqrts = 0.0;
        for (auto i : vec) {
                sum += i;
                sumSqrts += sqrt(i);
        }
  std::cout << "Sum of elements from " << vec.front()
                << " to " << vec.back() << " is " << sum
                << ", the sum of their square roots is " << sumSqrts << "." << std::endl;
}

然后我尝试测量一次``昂贵''的调用之后多次调用空函数所需的时间：

main.cpp：

#include "b.h"

#include <chrono>
#include <thread>

#include <iostream>

#include <math.h>

typedef std::chrono::high_resolution_clock sclock;
typedef unsigned long long time_interval;
typedef std::chrono::duration<time_interval, std::chrono::nanoseconds::period> time_as;

void timeIt() {
        auto start = sclock::now();
        auto end = start;

        for (auto i = 0U; i < 10U; i++) {
                start = sclock::now();
                asm volatile("" ::: "memory");
                foo_emptyFunction(1000ULL);
                asm volatile("" ::: "memory");
                end = sclock::now();
                std::cout << "Call #" << i << " to empty function took " << std::chrono::duration_cast<time_as>(end - start).count() << "ns." << std::endl;
        }
}

int main()
{
        timeIt();

        bar_expensiveFunction1();

        timeIt();

        std::this_thread::sleep_for(std::chrono::milliseconds(100));
        std::cout << "Slept for 100ms." << std::endl;

        timeIt();

        bar_expensiveFunction2();

        timeIt();

        bar_expensiveFunction1();

        timeIt();

        return 0;
}

如果我编译（g++ -o test main.cpp b.cpp或g++ -O3 -o test main.cpp b.cpp并运行代码，则会得到类似以下的输出：

./ test

Call #0 to empty function took 79ns.
Call #1 to empty function took 57ns.
Call #2 to empty function took 55ns.
Call #3 to empty function took 31ns.
Call #4 to empty function took 35ns.
Call #5 to empty function took 26ns.
Call #6 to empty function took 26ns.
Call #7 to empty function took 36ns.
Call #8 to empty function took 24ns.
Call #9 to empty function took 26ns.
Created and filled a vector with 1000000 elements.
Call #0 to empty function took 84ns.
Call #1 to empty function took 27ns.
Call #2 to empty function took 28ns.
Call #3 to empty function took 27ns.
Call #4 to empty function took 29ns.
Call #5 to empty function took 27ns.
Call #6 to empty function took 29ns.
Call #7 to empty function took 33ns.
Call #8 to empty function took 28ns.
Call #9 to empty function took 23ns.
Slept for 100ms.
Call #0 to empty function took 238ns.
Call #1 to empty function took 106ns.
Call #2 to empty function took 102ns.
Call #3 to empty function took 118ns.
Call #4 to empty function took 199ns.
Call #5 to empty function took 92ns.
Call #6 to empty function took 216ns.
Call #7 to empty function took 118ns.
Call #8 to empty function took 113ns.
Call #9 to empty function took 107ns.
Sum of elements from 1 to 1000000 is 500000500000, the sum of their square roots is 6.66667e+08.
Call #0 to empty function took 126ns.
Call #1 to empty function took 35ns.
Call #2 to empty function took 31ns.
Call #3 to empty function took 30ns.
Call #4 to empty function took 38ns.
Call #5 to empty function took 54ns.
Call #6 to empty function took 29ns.
Call #7 to empty function took 35ns.
Call #8 to empty function took 30ns.
Call #9 to empty function took 29ns.
Created and filled a vector with 1000000 elements.
Call #0 to empty function took 112ns.
Call #1 to empty function took 23ns.
Call #2 to empty function took 23ns.
Call #3 to empty function took 23ns.
Call #4 to empty function took 23ns.
Call #5 to empty function took 22ns.
Call #6 to empty function took 23ns.
Call #7 to empty function took 23ns.
Call #8 to empty function took 24ns.
Call #9 to empty function took 23ns.

我怀疑运行时间的差异（尤其是第一次调用时的高峰）可能是由于某种缓存效果引起的，但是我真的很想了解这里发生了什么。

编辑：我在这里观察到的效果与实际代码中的效果非常相似。第一次通话几乎总是有一个巨大的峰值，从第三次通话开始它一直是恒定的。我怀疑这种效果在实际代码中甚至更加明显，因为B::bar()在现实中会做更多的工作（它遍历一个图形，而不仅仅是遍历整数列表）。不幸的是，真正的代码是一个相当大的项目的一部分，所以我不能在这里发布它。我上面发布的代码是原始代码的相当大的简化，但似乎显示出相同的效果。实际上，foo()和bar()都是虚拟的（我知道这会带来时间损失），并且位于不同的补充单元中，因此编译器无法优化函数调用。另外，我还检查了真实程序的汇编程序。我也意识到以下事实：我不可避免地还会测量调用now（）的时间-但我对算法A使用了相同的基准测试代码（至少在其foo()的实现中做了一些工作）和总时间A::foo()的度量值要小于... 优化级别似乎对该效果没有太大影响，而且我对clang也有相同的行为。

编辑2：我还在专用计算机（Linux，仅系统进程，cpu频率调节器设置为性能）上运行算法基准测试。

此外，我知道通常当您进行这种微基准测试时，您会执行诸如缓存预热和将要基准测试的代码部分重复多次的工作。不幸的是，每次对foo()或bar()的调用都可能会修改内部数据结构，因此我不能仅仅重复它们。如果有任何改进建议，我将不胜感激。

谢谢！

Answer 1

我注意到睡眠后基准测试的表现更差。这可能是由于CPU进入了较低的频率/功率模式。

在进行基准测试之前，请将CPU频率固定为最大，以便CPU在基准测试期间不会对其进行调整。

在Linux上：

$ sudo cpupower --cpu all frequency-set --related --governor performance

在Windows上，将电源计划设置为“高性能”。