Every organization building AI capability faces the same question: what should our AI stack actually look like?
The answer seems obvious: get the best AI models, plug in the best tools, integrate everything together. But the real challenge isn’t selecting components. It’s making them work together effectively.
Most organizations get this wrong.
The 5 Layers of the Modern AI Stack
Think of your AI capability as five distinct layers, each with different requirements:
Layer 1: Infrastructure. Cloud compute, GPUs, storage. The foundational resources that everything else runs on. For most organizations, the answer is simple: buy this. Use AWS, Azure, or Google Cloud. Don’t build your own data centers.
Layer 2: Foundation Models. Large language models, specialized models, fine-tuning capabilities. These are the pre-trained AI models that form the intelligence layer. For most organizations, the answer is also buy this. OpenAI, Anthropic, Google, Mistral — pick the model that fits your needs. You might fine-tune it for your specific domain, but you’re not training foundation models from scratch.
Layer 3: AI Platforms. Integration, deployment, monitoring. How does the AI model actually connect to your business? How do you handle edge cases? How do you monitor that things are working? You might build this, or buy an enterprise AI platform that handles these concerns. This is where the first real choice emerges.
Layer 4: Orchestration. Agent frameworks, workflow management. How do you orchestrate AI working on complex, multi-step tasks? How do you handle AI working alongside humans? This is rapidly evolving — tools like LangChain, LlamaIndex, and others are evolving fast. This is an area where you might build custom solutions if you have specialized needs.
Layer 5: Applications. Custom solutions for your use cases. This is where you differentiate. This is where you build the AI capabilities that create competitive advantage specific to your business. You build this. Not because it’s easier than buying it, but because what you build should be specific to your business.
The Integration Problem
Here’s what most organizations get wrong: they treat each layer as independent, when the real challenge is making them work together.
You pick a great foundation model. You pick an integration platform. You pick an orchestration tool. And then you discover that the integration platform doesn’t work well with the orchestration tool. Or the foundation model has capabilities that your integration platform can’t expose. Or the workflow you want to build requires custom development that none of the tools handle.
The patchwork approach: Use the best tool at each layer, accept that integration will be messy, and spend enormous resources on custom glue code. This is flexible but extremely expensive.
The monolith approach: Pick one vendor who owns multiple layers, use everything from them, avoid integration problems. But you’re locked into their direction, and if they don’t evolve in ways you need, you’re stuck.
The custom-everything approach: Build your own versions of each layer. Full control. Full responsibility. Extremely expensive. Usually not worth it unless you have specialized needs no vendor meets.
The Right Approach
For most organizations, the right approach is somewhere between patchwork and monolith:
• Buy infrastructure (no question)
• Buy foundation models (you’re not training from scratch)
• Buy or use open-source for platforms and orchestration, but pick tools that play well together and have clear integration patterns
• Build custom applications where you differentiate
• Minimize custom glue code by choosing tools that work together
This means evaluating tools not just on their individual capability, but on how well they integrate with each other. It means asking: “If I’m building on this tool, what’s my path to scaling this? What happens when I need to swap out a component?”
Real-World Example
A financial services company might build like this:
• Infrastructure: AWS (compute, storage)
• Foundation Model: Claude or GPT-4 depending on their needs
• Platform: LangChain or LlamaIndex for orchestration and integration
• Applications: Custom agents built on top, fine-tuned for their domain-specific needs
This gives them flexibility (they’re not locked into a single vendor), reasonable integration (the tools are designed to work together), and clear ability to evolve each layer as technology improves.
What This Means for Your Organization
As you build your AI stack, ask these questions:
1. Which layers do I need to build? (Probably just applications. Possibly orchestration if you have specialized needs.)
2. Which layers should I buy? (Infrastructure and foundation models, definitely. Probably platforms too, unless you have reasons not to.)
3. How do my chosen tools integrate? (This matters more than individual tool capability.)
4. What’s my path to evolution? (As technology changes, can I swap out components without rebuilding everything?)
5. Where do I differentiate? (Put most of your effort here — the layers that create business value unique to your organization.)
The best AI stack isn’t the one with the most impressive components. It’s the one that reliably delivers value for your specific use cases, that you can maintain and evolve over time, and that doesn’t require you to become an infrastructure company to build AI capability.
Choose accordingly.
