Musings on the Future of LLM AI

Training new models is prohibitively expensive, and seems to be slowing down in terms of noticeable improvement. Or at least comparative to cost. The value of improvements is not equal to the cost of using frontier models from each company.

Every few months there's a new benchmark, a new leaderboard, a new "state of the art" model that costs more to run than the last one. And sure, on paper it's better. But when you're actually using it day to day in your editor, the jump from the previous generation to this one doesn't always feel like it matches the price increase.

I'm not saying models aren't getting better. They are. I'm saying the gap between "this model is technically smarter" and "this model meaningfully changes how I work" is getting wider. And the cost to close that gap keeps going up.

The Secret Sauce Isn't the Model

There's a huge noticeable difference between each AI code editor. Even though many of them can use the same models. Which means the real secret sauce is in the instructions between each request to the model that they add.

Cursor, Copilot, Windsurf, whatever you're using. You can often pick the same underlying model across them. Same GPT, same Claude, same whatever. And you'll still get different results depending on which tool you're in. Different code quality, different awareness of your project, different ability to actually make the right change in the right file.

That's not the model doing that. That's everything the platform wraps around the model. The system instructions. What files it decides to pull into context. How it formats your prompt. What tools it gives the model access to. How it handles follow-up requests. All the stuff that happens in the gap between you hitting enter and the model actually responding.

Two people using the same model through different editors aren't really using the same product. They're using the same engine with different everything else.

The Context Window Problem

Everyone's (understandably) concerned about how to improve (or pay for) context windows for AI. Tradeoffs like too large of a context window isn't just prohibitively expensive but also performs worse.

More context means more tokens. More tokens means more money per request. And past a certain point, the model doesn't actually get better at using all that extra information. It starts losing track of what matters. Important details get buried. The answer gets worse.

But how does an AI know how to perform the proper changes unless it has a large context window using other parts of the pre-existing codebase?

That's the tension. You need the AI to understand how your codebase works. Your patterns, your conventions, where things live, how things connect. But dumping the whole repo into context every time is expensive, unreliable, and like I said, doesn't even perform that well once you get big enough.

So everyone's trying to solve this with smarter retrieval. Semantic search, indexing, only pulling in relevant files. Which helps. But it's still a workaround for the fact that the model doesn't actually know your codebase. It's being briefed on the parts someone (or something) thought were relevant right before it has to make a decision.

Skills and Rules

So the solution, currently being leveraged today, but I think will be the future of AI work, is skills/rules. These are like the middle-man instructions code editor AI platforms are already doing, but tailored specifically for you.

Right now, already, you can improve the code AI writes by giving it a base set of instructions for your codebase, even more direct and specific than your repo's docs. Things like how you structure components, where tests go, what naming conventions you follow, what libraries you prefer over others. The stuff that's in your head but never quite made it into the README.

And then more specific rules depending on what it's doing. Different guidance for writing a new feature vs refactoring vs writing tests vs reviewing code. The same way you'd give different instructions to a junior dev depending on the task, except you're encoding it once and the AI gets it every time.

And these can become self-reinforcing where rules are updated based on what AI did or did not do each time. Honing its effectiveness over time. It used the wrong import style? Add a rule. It kept putting business logic in the wrong layer? Add a rule. It ignored your error handling pattern for the third time? Add a rule. Each correction becomes permanent instruction for the next request.

Over enough sessions, you end up with a set of instructions that knows how your team works better than most onboarding docs do.

So What Is the Future of AI?

Not training new specific models, but each company and even employee having specially honed skills/rules for working just right. For a worker this is transportable and proprietary. The same way your past experiences have lead up to how you work, an employer can hire you because your AI instructions work juuuuust right.

Think about it like this. When a company hires a senior developer, they're not just hiring someone who knows React. They're hiring someone who knows how to work. How to navigate a legacy codebase. How to make pragmatic tradeoffs. How to write code that fits in with what already exists. That knowledge is built up over years and it's part of why you're valuable.

Skills and rules are the same thing, but for AI-assisted work. Your accumulated instructions are a record of how you've learned to get good output from these tools in your specific context. And unlike model weights you don't control, these are yours.

Who needs a frontier model at 10X the cost when a company can run (locally/internally) an older model that does just fine but with their long fought for and built up skill/rule structure?

A model that's a generation or two behind the frontier, running on your own hardware or a cheaper API tier, with a skill/rule setup that's been refined over months of actual use? That might outperform a raw frontier model with zero project-specific guidance. And it costs a fraction of the price.

The Only Question Left

What will become of the excessive frontier model training expenditure and SaaS model for API access to said models when the real "model training" is everyone building their own skill/rule bumper rails?

The labs aren't going to stop training bigger models. There's too much money in it and the benchmark race isn't over. But I think the thing that actually determines whether AI is useful to you specifically is going to matter less and less which model you're on, and more and more what instructions you've built up around it.

The companies pouring billions into training the next frontier model are betting that smarter raw intelligence is the answer. I think the answer is already sitting in people's .cursor/rules folders and nobody's fully realized it yet.