Detailed Design of Scalable Data Infrastructure for AI/ML

In this lesson, we will zoom in on each of the five layers discussed earlier: data ingestion, storage, processing, feature store, and model serving. We will look at their internal components, technology choices, and key design considerations.

1. Data ingestion layer

The data ingestion layer ingests two main categories of data, i.e., real-time events and batch datasets, each handled by dedicated services.

• Real-time events: Events like clicks, page views, and payment activity are published into a distributed message queue (e.g., Apache Kafka), which serves as a durable, high-throughput buffer. A stream processing engine, like Apache Flink or Spark Structured Streaming, consumes these events, validates them, applies lightweight transformations, and enriches them with metadata. The processed events are then written to the infrastructure’s object storage, where they form the foundation for real-time features and near-real-time analytics.

• Batch data: Large-volume data, including database snapshots and historical records, enters at scheduled intervals. A workflow orchestrator, such as Apache Airflow, manages these workflows, handling dependencies and retries to extract, clean, and format data before loading it into the data lake’s raw zone. This ensures reproducible and consistent datasets, critical for model training and analytics.

Despite the different patterns, both ingestion paths enforce the same foundational guarantees: reliable delivery, resilience to upstream failures, and compatibility with evolving schemas. Services such as schema registry (for Kafka) or built-in validation libraries (in Airflow and Flink) ensure that schema changes do not break downstream systems.

To support lineage and debugging, all ingested data is tagged with metadata like ingestion timestamps, Kafka offsets, or batch IDs, stored alongside the data in object storage or logged in metadata repositories. By ensuring that both high-velocity event streams and large batch imports flow into the system cleanly, consistently, and with accurate lineage, the data ingestion layer establishes a dependable foundation for downstream processing, storage, and machine learning workflows.

Once the data is reliably ingested into our raw data lake, it must be durably stored and cataloged. This is the role of the data storage layer.

2. Data storage layer

The data storage layer provides a durable, scalable foundation for all data produced by the ingestion layer. All incoming data, whether real-time or batch, lands directly in the raw zone of the data lake. We utilize low-cost object storage (e.g., Amazon S3, GCS) for this layer. This zone preserves data in its original form and supports schema-on-read, giving the platform flexibility to reinterpret historical data with new logic.

From the raw zone, cleaned datasets move into processed zones. These zones utilize modern table formats to support ...