Reasoning R&R™: What We Found When We Measured Our Own AI
Quality & AI | Wednesday, October 21, 2026
If AI is going to become part of a quality process, eventually we have to stop asking whether it "seems to work."
We have to measure it.
That was the premise behind our Reasoning R&R™ study.
Manufacturing already has a language for evaluating measurement systems.
We wanted to apply that discipline to AI-assisted engineering judgment.
We looked at three different questions.
Reasoning: Did the evaluation agree with the known standard? In other words: did the system reach the conclusion it was supposed to reach? This has to come first. A system that produces the same answer every time is not useful if the answer is wrong.
Repeatability: Does the system produce the same substantive evaluation when the same input is evaluated again under controlled conditions? This tells us whether the process is stable.
Reproducibility: Do different appraisers using the controlled method reach comparable conclusions? This matters because a quality process should not depend entirely on which qualified person happens to run it.
The order matters.
First, the reasoning has to be right.
Then we can ask whether it is repeatable.
Then we can ask whether different people can reproduce the result.
For our MAD-Ai 8D grading workflow, we evaluated 2,370 individual criterion judgments.
The results were:
93.2% repeatability
99% reproducibility between two appraisers
Those numbers are useful.
But the more important part of the study was what measurement exposed.
Twelve criteria flipped from run to run because wording in our own criteria was ambiguous.
That is exactly the kind of problem normal use can hide.
If an engineer runs the system once and gets a plausible scorecard, there may be no reason to question the wording behind the evaluation.
Measurement creates a different opportunity.
It shows where the process itself needs improvement.
We also found an input-format problem.
One input format was silently not reaching the model, while the system still produced a complete-looking scorecard.
That is a much more important finding than whether the interface looked good.
The output looked finished.
The measurement told us the process was not behaving the way we thought it was.
And that is the larger lesson.
Measurement found things normal use did not.
That is what R&R is supposed to do.
We are not presenting these results as proof that MAD-Ai is perfect. They are not.
The point of the study was almost the opposite.
If we are asking customers to trust AI-assisted quality work, we need to be willing to measure our own system, find weaknesses, and fix them.
AI should be treated as a process that can be tested and improved—not as a black box that gets a pass because the answer sounds convincing.
That is the mindset behind Reasoning R&R™.
Don't just ask whether the AI works. Measure how the reasoning behaves.
#QualityEngineering #ManufacturingAI #AIinManufacturing #ReasoningRR #MeasurementSystems #GageRR #QualityManagement #8D
Rick Heilshorn lays out the full case for Reasoning R&R™ in Quality Digest.
Talk with us about measuring AI in your quality process → mad-ai.com/book-a-demo
Comments