Consistency Means Nothing If It's Consistently Wrong
That's the line I'm opening with on stage today at the AIAG Quality Summit, and it's the whole argument in nine words.
Why Order Matters
It would be easy to build an AI system that gives the same answer every time — just turn the randomness down. That system would be perfectly repeatable and completely useless if the answer is wrong. Reasoning R&R™ checks whether the evaluation agrees with a known standard first, then whether it repeats, then whether it reproduces across reviewers. Skip the first question and the other two are just measuring how consistently you're fooling yourself.
What We Found, in Order
On our own 8D grading workflow: reasoning checked out against a known-good and known-bad document, matching the expert answer 100% of the time on the known-good case. Then 93.2% repeatability across 2,370 individual grading decisions. Then 99% reproducibility between two reviewers using the same method. Each number only means something because the one before it held up.
If You're in Novi Today
1:30PM, Platinum Ballroom. Or find the team running the Quality Suite live on real documents at Booth #240 through tomorrow.
Comments