When people hear “self-learning system,” they usually picture software that watches everything, rewrites itself overnight, and gets smarter without anyone touching it. That version makes headlines. The version that works in a real business is less dramatic: people correct the system's output, AI helps make sense of all those corrections, and the team improves the algorithm one measured step at a time.
That's how we run one of our own projects, and the numbers below come from it.
Self-improving AI made headlines this year
Over the past few months, AI that improves AI has been one of the biggest stories in tech.
- September: OpenAI said it had reached its goal of an “automated research intern”, a system that carries out well-defined research tasks under human direction.
- October: TIME reported on a Gladstone AI study of how the leading labs are working to automate AI research itself: coming up with ideas, running experiments, and interpreting the results.
- November: The evaluator METR published an 11-week assessment inside Anthropic. It estimated that the company is improving its models at about twice its earlier rate, mostly thanks to AI.
Read the details, though, and there's still a person in the loop. OpenAI's intern works under human direction. METR was told the speed-up comes “primarily via AI assistance rather than fully autonomous contributions.” Even at the companies with the most compute and some of the best engineers in the field, AI makes people faster. It doesn't run the process on its own.
When a system learns straight from its users
The tempting shortcut is to let a system learn directly from how people react to it. Recent experience shows why that's risky.
People reward flattery
In April 2025, OpenAI shipped a GPT-4o update that added a training signal based on thumbs-up and thumbs-down ratings from ChatGPT users. The model became noticeably sycophantic: it flattered users and agreed with them too easily. OpenAI started rolling the update back three days later. Its explanation was that the new signal weakened the one that had kept flattery in check.
This spring, a study by Stanford researchers in Science showed how deep the problem goes. They tested 11 leading models and found that the models affirmed users' actions 49% more often than people did, even when the user had done something wrong. In experiments with 2,405 participants, people trusted the flattering models more and preferred them.
So if you train a system on what users like, you get a system that tells them what they want to hear. A thumbs-down usually does mean something went wrong, so the feedback isn't useless. It just needs someone in between to work out what it means.
A moving target can't be tested
There's a second problem. In January, Fazl Barez of Oxford's AI Governance Initiative pointed out that a system which keeps learning after launch is no longer the system you tested: “The model that passed your evaluations at deployment is not the model users encounter a month later.” For a business, that means the checks you ran before going live stop telling you much.
What works: a loop with a person in it
The systems that improve reliably follow a different pattern. Feedback is collected, studied, and turned into changes that are tested before they ship.
- Google Search has human raters score search results, but Google says those ratings “do not directly impact ranking.” They're used to check whether a proposed change makes results better before it launches.
- Anthropic Insights (called Clio until it was renamed in August) uses Claude to summarize and group huge numbers of conversations, so the team can see patterns without anyone reading them one by one. People then decide what to change.
- Google DeepMind's AlphaEvolve is probably the closest thing to an AI that improves algorithms by itself. In May, a year after launch, DeepMind reported that it had cut DNA variant detection errors in one of Google's genomics models by 30%. It works because automated evaluators run and score every candidate program. Most business tasks don't have one. Whether a reply is right, or a document looks good enough, is usually decided by a person.
That's the useful idea behind “self-learning.” In most businesses, the scoreboard is the corrections your people make every day. AI's job is to read thousands of them, which nobody has time to do by hand.
Our case: a pipeline that learns from its reviewers
One of our clients turns phone photos and scans of building plans and permit documents into clean PDFs. The photos are anything but tidy: large sheets shot on the floor, at an angle, with shadows and glare.
We built a pipeline that does the routine work, and a person checks every page before it goes out. Every fix they make is kept, and that's what the system learns from. It doesn't rewrite itself. We regularly go through those fixes with the help of AI and improve the pipeline wherever people keep correcting it.
In about six weeks, that added up to dozens of small improvements. Tricky photos now come out right more often on their own, doubtful pages go straight to a person, and steps nobody wanted were switched off. We also use an AI agent that studied the reviewers' own corrections and now prepares fixes the way they would make them. The final say always stays with a person.
The reviewers didn't change how they work. They kept doing their job, and the system kept getting better at its own.
What this means for a small business
You probably don't need an AI that improves itself. You need a system that gets a little better every month, and that's mostly a matter of design.
- The system says when it's unsure, so people check the right cases first.
- A person reviews and corrects the output before it reaches a customer.
- Every correction is saved, together with what the system did on its own.
- Someone regularly goes through the corrections. This is where AI changes the math. Reading a thousand corrections and finding the few patterns behind them used to be a job nobody had time for. With AI, it's routine work.
- Every change is tested on past cases before it ships, and can be rolled back.
The same loop works far beyond image processing. An agent that sorts incoming email can learn from the messages your staff move to another folder. Invoice data extraction can learn from the fields your accountant corrects. Draft replies to customers can learn from the edits made before sending. Each edit is a data point, as long as you keep it.
And the GPT-4o lesson applies at any scale: don't feed raw ratings straight back into the system. Read them first.
Why talk to a team like ours
- We build the review step in from day one. Corrections are the most valuable data your process produces, so we make sure they're captured.
- We measure before we change anything. Every threshold in our pipeline comes with the range we measured on real pages, and every change is replayed on approved work first.
- We use AI where it pays off. Sometimes that's a model. Often it's a simple rule based on what your people keep correcting.
- We keep it affordable to run. The models in the pipeline above run on an ordinary server, without a GPU or a per-page fee. Where it makes sense, we run agents locally instead of tying you to one cloud vendor.
- People stay in charge. Our agents propose and fix. Your team approves.
A system that learns isn't magic. It's a loop: do the work, let people correct it, study the corrections, and improve. Setting that loop up is the part we're good at.
If your team keeps fixing the same mistakes by hand, tell us about it.
