How robust is repeating a piece of code to measure it's execution time?

Question

Let's say I want to measure how long MyFunction() executes.

Let's also say that MyFunction() executes fast (a few milliseconds) and that Stopwatch cannot measure the function time with a good enough precision.

I then will put MyFunction() inside of a loop which executes 1,000,000 times and measure that. I will time the execution of the loop and then divide the result by the number of times the loop executed.

The questions are:

• How accurate is this measurement? I basically want to know how long the FIRST execution of MyFunction() takes, not how long the average takes.

• Does the time executing the same function scale linearly? Does that depend on the properties of MyFunction() (so some kind of caching is used)? What are those properties?

Answer 1

Let's say you run MyFunction() 1,000,000 times in 3 seconds.

You now know:

1. You can at least sometimes run MyFunction() in a time less than or equal to 3μs, on similar data and with similar cache and jitter situations.

2. It probably runs faster than an equivalent method that takes 3.5 seconds, as long as the tests are truly equivalent. (But 3.01 is too close to be sure unless this repeats consistently).

3. It probably runs slower than an equivalent method that takes 2.5 seconds, as long as the tests are truly equivalent. (But 2.99 is too close to be sure unless this repeats consistently).

You do not know:

1. How much the time is affected on first call by jitting, static constructors, and memoisation of values that are either static or related to the particular values used.

2. How much the time is affected by the method still being accessible from the instruction cache (or indeed, whether it is).

3. How much the time is affected by the data used still being accessible from the data cache (or indeed, whether it is).

4. How consistent the time is subsequent to first run; is it a steady 3μs per run, or more often 1μs with some occasional large delays?

5. Just what portion of the time was spent on the looping rather than the running.

These five things you don't know (or four if you do a run prior to the loop to at least reduce the impact of the first one) mean you don't know an awful lot in an absolute sense.

But the three things you do know can be very useful relative to other possible approaches, or if you're clearly above or below a certain threshold (if you need to beat 2μs you're screwed, if you need to beat 500ms you're fine, if you need to consistently beat 4μs then you don't know if you have).