Measuring our AI speed-up

Many firms quote a percentage for how much faster AI makes them. Those figures are rarely measured in a way you can check. We do not quote one until we can show how we measured it.

What we measure

• Cycle time: from the day a story is started to the day it meets the definition of done
• Review time: how long a change waits for and spends in peer review
• Quality: defects found after a story is called done, and rework on the same story
• AI level used on each story, using our three levels of AI adoption

How a pilot works

1. 1

   1. Agree the pilot

   The customer agrees to the measurement and to which work is included. We do not measure in secret.

2. 2

   2. Record a baseline first

   Before AI is used on the pilot, we record how the same team delivered comparable stories. The baseline covers enough completed stories to show the normal range, agreed for each pilot.

3. 3

   3. Compare like with like

   We compare stories of similar size and kind. A faster result on easier work is not a speed-up.

4. 4

   4. Tag every story

   Each story records whether AI was used and at which level, so results can be split fairly.

5. 5

   5. Judge speed and quality together

   A change that is faster but produces more defects or rework does not count as a gain.

6. 6

   6. Publish with the evidence

   We publish the result with the details listed below, and review it with the customer first.

What we publish with every number

• The result as a range, not a single figure
• How many stories and how many people it covers, and over what period
• How the baseline was set
• What was left out and why
• The quality result next to the speed result
• What we could not control, such as a team still learning a tool

Limits we state up front

Pilots are small. Teams get faster as they learn the tools, and some measures rely on people recording their work honestly. For these reasons we say when a result is too small to rely on, and we do not extend a pilot result to all work.