# Building A Generative AI Platform

DevFeed: [Building A Generative AI Platform](<https://devfeed.tech/articles/building-a-generative-ai-platform-31800.md>)

Original publisher: [Read original article](<https://huyenchip.com//2024/07/25/genai-platform.html>)

Author: Chip Huyen

Published: 2024-07-25T00:00:00Z

Content type: tutorial

Language: en

Sources: [Chip Huyen](<https://devfeed.tech/sources/chip-huyen.md>)

Topics: [AI Platform](<https://devfeed.tech/topics/ai-platform.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>), [API](<https://devfeed.tech/topics/api.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Caching](<https://devfeed.tech/topics/caching.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Security](<https://devfeed.tech/topics/security.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-platform](<https://devfeed.tech/tags/ai-platform.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [cache](<https://devfeed.tech/tags/cache.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [latency](<https://devfeed.tech/tags/latency.md>), [observability](<https://devfeed.tech/tags/observability.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [security](<https://devfeed.tech/tags/security.md>)

## AI overview

This post explains the common architecture of a generative AI platform, starting with a simple application that sends queries to a model and progressively adding components such as context construction, guardrails, model routers and gateways, caching, complex logic, observability, and orchestration. It notes that the architecture should be adapted to each system and that evaluation is needed throughout development.

## Source excerpt

After studying how companies deploy generative AI applications, I noticed many similarities in their platforms. This post outlines the common components of a generative AI platform, what they do, and how they are implemented. I try my best to keep the architecture general, but certain applications might deviate. This is what the overall architecture looks like. This is a pretty complex system. This post will start from the simplest architecture and progressively add more components. In its simplest form, your application receives a query and sends it to the model. The model generates a response, which is returned to the user. There are no guardrails, no augmented context, and no optimization. The Model API box refers to both third-party APIs (e.g., OpenAI, Google, Anthropic) and self-hosted APIs. From this, you can add more components as needs arise. The order discussed in this post is common, though you don't need to follow the exact same order. A component can be skipped if your system works well without it. Evaluation is necessary at every step of the development process. Enhance context input into a model by giving the model access to external data sources and tools for information gathering. Put in guardrails to protect your system and your users. Add model router and gateway to support complex pipelines and add more security. Optimize for latency and costs with cache. Add complex logic and write actions to maximize your system's capabilities. Observability, which allows you to gain visibility into your system for monitoring and debugging, and orchestration, which involves chaining all the components together, are two essential components of the platform. We will discuss them at the end of this post. " What this post is not " This post focuses on the overall architecture for deploying AI applications. It discusses what components are needed and considerations when building these components. It's not about how to build AI applications and, therefore, does NOT di