Building an AI Trading Experiment: Model Councils, News, and the Search for an Edge

I've been experimenting with something a little outside my usual software development work: using AI models to evaluate potential stock trades. The idea isn't necessarily to build some magical trading bot that prints money; I'm more interested in whether multiple AI models working together can produce better decisions than a single model, and whether adding real-world information, such as financial news, actually improves those decisions.

The entire experiment is running in a paper-trading environment, with no actual broker orders being placed. That gives me the freedom to test different approaches, compare results, and make changes without putting real money at risk. More importantly, it gives me an opportunity to separate what sounds like a good idea from what the data eventually tells me.

Getting Multiple Models to Agree

One of the more interesting parts of this project is what I've been calling a model council. I'm using OpenAI's Codex and Anthropic's Claude to evaluate potential trading opportunities, with Jev acting as a third model that breaks ties when the first two disagree. Rather than trusting a single model's interpretation of the market, I'm experimenting with whether independent evaluations can lead to better decisions.

Jev's role is particularly interesting because it receives context about the disagreement and uses that information to make the tie-breaking decision. The goal isn't simply to ask three models the same question and count votes; it's to see whether a third perspective, informed by the competing assessments, can help resolve disagreements more intelligently.

This change has already been implemented, and tests have reportedly passed. I haven't independently rerun those tests for this write-up, and even if they pass again, that only tells me the software behaves according to its requirements. It doesn't tell me whether the council makes better trading decisions.

There's also an interesting problem with using multiple models in the first place. Three models agreeing doesn't necessarily mean they're correct; they might share the same assumptions, rely on similar training data, or confidently arrive at the same bad conclusion. A majority vote is still just a majority vote, and confidence expressed by an AI model isn't automatically a meaningful probability of success.

Does Financial News Actually Help?

Another question I've been exploring is whether giving these models access to financial news improves their decisions. It seems logical that recent headlines, earnings announcements, or market developments could provide useful context that historical price data alone might miss. The problem is that something sounding logical doesn't make it profitable, and additional information can just as easily introduce noise.

Rather than adding news directly into the existing strategy and hoping for the best, I've implemented a matched comparison. One path is allowed to consider news information while the other operates without it; both can then be evaluated over the same period. Three news providers were also checked as part of the setup, although that doesn't establish that any particular provider offers a measurable advantage.

This is the part of the experiment I'm particularly interested in. If the news-enabled approach consistently performs better under comparable conditions, that's worth investigating further. If it performs the same or worse, that's equally useful information. There's no reason to add complexity, additional API calls, and more opportunities for failure unless the results justify doing so.

The Difference Between Working Software and a Working Strategy

Something I've become increasingly conscious of while building AI systems is the distinction between software correctness and real-world effectiveness. I can write tests that prove a tie-breaker is called under the right conditions, verify that news is only available to the appropriate strategy, and confirm that the system records decisions correctly. None of those tests prove that the decisions themselves are good.

As of this update, there hasn't been a newly processed trading session or a new fill to evaluate. The model council changes and the news comparison are implemented, although the forward results needed to assess their effectiveness simply don't exist yet. Claiming an improvement at this point would be confusing a completed engineering task with a successful financial experiment.

I'm also interested in how this relates to a broader problem with AI development. It's becoming remarkably easy to build sophisticated systems that appear intelligent, generate convincing explanations, and pass extensive automated tests. Determining whether those systems actually accomplish what we intended is an entirely different challenge; one that requires independent measurement rather than trusting the system's own assessment of its performance.

Where This Goes Next

For now, the next step is to let the experiment generate enough forward observations to make comparisons meaningful. I'll be looking at more than just whether a simulated trade makes money; consistency, risk, drawdowns, transaction costs, and the differences between the news-enabled and news-free approaches all matter. A handful of successful predictions wouldn't be enough to establish an advantage, especially in a market where randomness can make almost anything look promising over a short period.

I have no idea whether this experiment will ultimately produce a useful trading strategy, and that's perfectly fine. The interesting part is building a system that can challenge its own assumptions, measure competing approaches, and produce evidence that either supports or rejects the original idea. If nothing else, it's another opportunity to explore where AI genuinely adds value and where we're simply adding more sophisticated ways to be wrong.