The Consistency Quadrant: A Visual Guide to LLM Reliability

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.

Just interested in the code for this project? Find it here. I do not think it’s an overstatement to say that the worlds of software engineering, data science, and analysis are in the process of rapid and dramatic transformation due to AI tools. This transformation crept up on the community over the course of 2025 and has really exploded since the beginning of 2026 with the widespread adoption of coding harnesses like Claude Code, Codex, and Antigravity.

These are agentic tools, making multiple Large Language Model (LLM) calls per request to reason through and solve complex problems. And they are amazing, if a little surreal, to watch as they go about their work. Similarly, the field of business analytics is seeing a shift towards AI analysts, which can explore complex datasets, write and issue SQL queries and generate polished reports complete with figures and recommendations.

In summary, AI in the technology workplace is clearly transitioning from a productivity enhancing add-on to an indispensable tool. This transformation is taking developers a step away from the inner workings of the software or queries that they write, ideally freeing up bandwidth for them to tackle higher level problems of design. For this to work well, we have to be able to trust that the AI is reliable — or at least have some way of measuring its reliability so that we can select an appropriate setting for the task at hand.

Why is reliability so important? AI agents are powered by LLMs, which are non-deterministic token generators. This non-determinism becomes unpredictable in systems with long reasoning chains, multiple LLM calls and/or usage of different models with settings that are not visible to the user. On the flip side, non-determinism is very useful — it's what gives AI creativity, ability to reason through problems and adaptability.

So it's not necessarily bad, it’s just important to understand and measure its effect on the consistency of responses. Consistency is defined as the ability of a system to generate reproducible outputs from identical inputs. Measuring consistency is especially critical for tasks where there is a correct answer and deviations from that could be misleading.

Business analytics is a good example, but coding is too — there may be many valid ways of reaching a solution, but that solution should actually work as designed, or solve the problem that it was intended for. In real-world deployments, we rarely have ground-truth unit tests to verify an agent’s work, and different LLM-problem combinations show different consistency characteristics. How, then, do we evaluate reliability when we don't know the answer ahead of time? In this article we explore this problem by building a command line tool called Coding Agent Consistency (cca), which allows us to set up experiments where we call models multiple times and analyze the distribution of results.

This is helpful because in the absence of ground truth, multi-sample consistency might be the only proxy for model reliability that we have. Here we focus on well constrained coding problems and small models to keep cost manageable, but the concept is extensible to many other use cases. The cca package also enables us to explore different foundation models via API thanks to litellm, and can also interface with local models via ollama as well as coding agents via Omnigent.

It therefore offers a tool to compare the consistency of multiple AI tools on a given problem or set of problems.