Organizations Are Standardizing AI Interaction Instead of Standardizing Evaluation

Years ago, I attended a meeting where the guest speaker did a demonstration with two pieces of string. He stood at one end of a very large room and had two volunteers come and each take hold of one of the pieces of string. One volunteer was told to walk to the other end of the room facing the same direction in a straight line. The other was told to turn, just slightly, and do the same.

Although both individuals started walking from the same point, by the time they reached the other end of the room, they were several feet away from each other.

The demonstration was for a specific purpose, but it is still something I find myself thinking of in various situations to this day. The idea that one small shift, one small change in direction, can have profound implications in the future. The direction you intended to go, the point you intended to be at, is nowhere near where you end up.

I also thought of this today in conversation with a professional who works directly with organizations on understanding risk in relationship to AI. Something we both agreed a lot of companies do not understand yet.

That risk is when organizations start standardizing AI interaction instead of standardizing the way AI gets evaluated.

Companies do this for two reasons: first, this is something intrinsic to the corporate world; second, it is a result of not having a healthy understanding of how AI adoption should occur.

The idea that if an organization wants a specific result there must be one standardized method for getting that result is very prevalent in corporate America. Not without reason. Traditionally speaking, the way you did something, the method you used, typically determined the output and the quality of the output. The problem is that with AI this approach runs directly opposite to that reasoning.

When using AI, users are participating in a back-and-forth exchange shaped by each unique user interaction with the tool itself. Even the most structured and standardized prompts cannot guarantee that all users will get the same output.

By standardizing the interaction, we are forcing users into a path that could potentially be leading them to a different output from the start. If each interaction with AI is unique, then like our string example, each interaction, no matter how intentional our starting direction is, is, without course correction, leading to a different output anyway. If this is the case, then placing emphasis on the path makes no sense.

The second reason this happens is because organizations are chasing the result without investing the effort or time needed to get there. While I doubt most do this intentionally, OpenAI only became available to the public less than four years ago. Many people still have no idea how to use AI at a level above that of a search engine.

The problem with the race to get to the result is that it cuts out all the intentional interaction needed with AI to learn how to evaluate those results. Workflows should be something that develop naturally — not something forced.

Users running similar scenarios through AI multiple times begin to recognize patterns and effects that different interactions have on outputs. This is essential to what we call ‘operational literacy with AI.’ It’s this learning process that builds the foundation users need to manage and direct future systems intentionally.

When systems break, when workflows drift, the need to be able to evaluate output is essential. If you have no standard for evaluating output — or if you are unaware of how to evaluate output — these drifts and breaks in the system go unnoticed. When they are noticed, teams lack the know-how to make the necessary adjustments. Yet many companies are racing to this end.

Imagine an automaker who hires machine operators for the assembly line. The only training the automaker requires is that the operators know how and when to push a button, pull a lever, turn this dial, or turn that one. That’s it. No other training is required. What would happen?

You don’t have to know the automaking industry to know. We all know the answer. Without the ability to adjust the machines and evaluate the outputs at each stage of production, you risk the whole operation. The automaker would end up with cars that can’t pass inspection because they have a driver’s-side door installed two inches off from where it needs to be.

The realization needed to mitigate this risk? Intentionally train and build a culture around standardized thinking, NOT standardized interaction.

Start with things like, ‘What defines a perfect output for you and your organization?’ Establish standards for how that output should look. Allow teams to build workflows naturally. As their ability to think structurally with AI tools increases, workflows will naturally develop. This time, however, they will develop with teams who know how to guide them, how to detect and correct drift when it happens.

In each of these organizations there are hundreds of professionals who have been delivering quality results for years. They know what excellence looks like, what the output should be.

When we focus on scripting instead of thinking, we force these same professionals into a scenario where they must abandon all those years of experience for an output, a result they know is not what it needs to be, but they do not have the tools to fix it, and some of them do not even realize they can.

Now imagine those same organizations choosing to provide that intentional, direct instruction on evaluation. They choose to let their teams naturally develop workflows over time through repetition and continued structured thinking with AI. Now you have an organization full of professionals who not only know what outputs need to be, but who, coupled with the power of AI, can take those outputs further than before, faster, and more efficiently than ever.

This decision to have a ‘hands-off’ approach to the process while firmly establishing the standards for the end result has to be a top-down philosophy. It has to be something that becomes embedded in the organization so that each level takes personal ownership of their outputs, workflows, and whatever portion of the system they own.

The problem with the individual holding the string who ended up several feet away from the other person was not because he started in a slightly different direction. The problem was that he was given specific instructions on how to get to the end. He was not allowed to change direction, not allowed to account for the variance in the direction he started, not allowed to make whatever changes he needed in order to end up in the same spot.

If we fall into the trap of doing the same thing with our teams, they will end up the same way.