AI chat app and model distinction

We’re a solid three-plus years into AI chat apps. ChatGPT, Claude, Gemini, Grok, Perplexity, etc. From afar, they all look pretty similar, but each creates a distinctly different user experience.

One thing I found important early on when working with Large Language Models (LLMs) is recognizing that there’s a LOT more going on than just the underlying model. For any of these flagship AI chat apps, you’re never engaging directly with the foundational model. The model matters, for sure, but not as much as the application wrapped around it.

Putting a finer point on it, there are two ingredients that create differentiated experiences in these chat apps:

• The model
• The chat application

A great model embedded in a so-so application will be at best only so-so. A great app, even with a less capable foundation model, can still be amazing. The best of the best—which is hard to achieve—is having a great model and a great app at the same time.

In my day job, I work with organizations on AI projects, and this distinction shapes how I think about building with AI.

What all does the app do?

• Conversation management - Managing multiple chats, state, and enabling users to revisit and continue previous conversations
• User experience and performance - Input formats (text, speech-to-text, streaming audio, attachments), progress updates, streaming output
• Knowledge sourcing - Integration with databases, search engines, and APIs
• Model flexibility - Allowing model swapping or even user selection of which model to use
• Orchestration and business logic - Routing requests to the correct model, agent, or tool; deciding when to call external APIs or tools; post-processing results
• A bunch of other stuff - Safety guardrails, logging, compliance, and many things that are less differentiated

It’s hard to have a great app and model at the same time

Foundation models don’t stay the most powerful for long. There’s huge focus among model developers to continue innovating, and we see leapfrogging every month or so.

Meanwhile, the apps continually need to adapt to new capabilities that models support, as well as the nuances of communicating with different models.

Isn’t this all obvious? Why are you writing about it?

In short, no—it’s not obvious. Even experts in the AI space rarely speak about this distinction. I’ve been frustrated with this since the beginning of ChatGPT, and it’s continued to bug me for three years now.

Recently, Lex Fridman had a four hour episode (#490) discussing AI, starting with chat apps. They spent significant time discussing these applications, yet the conversation never quite touched on this fundamental distinction between the model and the application layer.

They know their stuff—I’m not suggesting they don’t “get it,” because they do. That said, I was hoping to hear some of the conversation delve into this distinction and the larger implications for AI projects outside of flagship chat apps.

There’s more nuance to it. For me, a big part of ChatGPT’s success has been the application experience they create. They’ve been great at it, had a huge head start, and still maintain a lead there.

But for people and organizations who aren’t also creating their own models, the focus should be on creating a great application—one that’s ready to swap in other models, which are increasingly becoming commodities.

This is where I like Microsoft’s capabilities to support developers

Note: I work at Microsoft. This is my opinion and I’m not speaking for Microsoft.

Microsoft Foundry is geared toward helping developers build intelligent applications. Despite some naming challenges (what it’s called isn’t the important thing), it brings significant capabilities relevant to this conversation:

• Model access - Foundry provides consistent and secure access to leading models like OpenAI, Anthropic, Grok, Llama, Deepseek, and many others
• Tools - Features like “Foundry IQ” provide robust knowledge management and search capabilities without needing to build from the ground up
• Agent hosting - Most initiatives benefit from focusing less on hosting infrastructure and more on simply having agents available to use
• Agent framework - A development kit combining the strengths of multiple early frameworks, accelerating development with key agent orchestration and workflow capabilities

Final thoughts

A few things to keep in mind as you go about your merry way:

• As you use Claude, Copilot, Grok, ChatGPT, or whatever in the coming days, take some time to think about the things you like the things you don’t, and whether those might actually be the application instead of the model.
• When you’re embarking or continuing on AI projects, prioritize the things where you will differentiate.