The Space Station: Medallion Architecture at the Heart of the Lakehouse Mission
- Databricks
- Databricks
- 7 min reading time
Dr. Andreas Wagner
The Medallion architecture represents a prominent design pattern in modern data platform development and serves as a blueprint for the logical organization of data within a unified data lakehouse. The term was popularized by Databricks around 2019 and introduced shortly after the release of Delta Lake as a core component of the evolving “Lakehouse” vision. Since its introduction, the term has gained widespread acceptance across the industry.
Imagine a space station: it is not a monolithic block, but a precisely constructed system consisting of specialized modules—each with a clearly defined task, each dependent on the next. This is exactly how the Medallion architecture works. Its three layers—Bronze, Silver, and Gold—are the modules of the space station. Each layer increases the reliability, semantic richness, and business value of the data without losing the connection to the original raw data.
Table of contents
Just like in a real space station, each module performs a clearly defined task—and no module takes on the work of another. The Medallion architecture therefore requires a strict separation of technical responsibilities across its three main layers:
The Bronze tier serves as the space station’s docking station: this is where all incoming data streams land—from transactional databases, external SaaS applications, real-time message queues like Kafka, or file uploads. The basic principle: capture everything, transform nothing. Faithful reproduction, without exception. The data is written exactly as it appears in the source system, preserving original structures, nested semi-structured data, and schemas. To ensure full traceability and enable future reprocessing, system metadata—such as ingestion timestamps, process IDs, and source system identifiers—is added to each record upon insertion. Bronze tables are typically append-only and immutable, meaning that once data has been written, it cannot be changed. They serve as a single source of truth from which all downstream layers can be systematically recreated at any time without having to query the operational source systems again.
In the space station’s processing lab, the raw data that has been delivered is organized. The Silver layer transforms the stream of raw data into a cleansed, compliant, and consistent enterprise view. The transformations applied here are deterministic and standardized, and focus on:
- Harmonization and Further Development of Schemas
- Deduplication of redundant data records
- Standardization of naming conventions and data formats
- Handling/Correcting Missing or Incorrect Values
- Linking different data sets to provide an integrated view
In line with the modern data engineering paradigm, the Silver layer typically prioritizes an ELT approach. Only “just enough” cleansing is applied to present a consistent view of key business entities (such as consistent views of customers, transactions, or products). It frequently uses third normal form (3NF) or Data Vault architectures to enable the most comprehensive integration and auditability possible, as well as the implementation of initial quality assurance measures. This ensures that, when the data is reused, quality issues are not propagated across different use cases.
The Command Center is the heart of the space station—the place where decisions are made. The Gold tier aggregates the prepared data from the Silver module and makes it available for specific use cases. This tier is explicitly optimized for read access and tailored for reporting, self-service BI, and machine learning models. Complex business logic, calculations, and aggregations are applied in the Gold tables. To maximize query performance and minimize the need for table joins at runtime, the data is typically denormalized. The Gold tier often uses Kimball star schemas (with highly structured fact and dimension tables) or Inmon data marts optimized for execution speed.
How the modules work together: The multi-hop architecture as a principle
Data passes through the space station module by module—which is why this is also referred to as a multi-hop architecture. The advantage of this is that an explicit intermediate result is produced for each data flow, which can be used for further processes, thereby potentially preventing or reducing the duplication of business logic. In addition, a separation of concerns is established, whereby each layer assumes a significantly more homogeneous set of tasks.
With each layer or “hop,” the processing effort invested in the data increases, both in terms of computing resources and developer resources. For this reason, the effort required must be increasingly justified as the data flows through the system, ideally through long-term, profitable use cases and a broad user base for the data. While only a few specialized developers (e.g., data engineers) have access to the Bronze data, some (advanced) analysts can also access the Silver data.
Medallion: More Than Just a Marketing Term
The multi-tiered architecture underlying the term “Medallion” is not entirely new. Modern data platforms have adopted a multi-stage refinement approach that has been in use since at least the late 1990s. In the past, data warehouse architects created similar physical boundaries and referred to the successive stages as “raw data” (“integration” or “processed”) and “presentation” (“refined”) layers.
However, a common issue with this architecture was that a central team managed and developed it, while users were only able to—or permitted to—access the final outputs (data marts).
This created a bottleneck within the central team responsible for developing new requirements, resulting in long wait times for stakeholders. This encouraged the creation of separate data silos within the business units. After all, the business units often did not rely on 100% processed data, especially when proof-of-concepts (PoCs) or other experimental projects needed to be launched. Data that was slightly less processed was often far better than having no data at all. The Medallion architecture modernizes this long-standing practice and optimizes it specifically for decentralized scaling and the open storage formats of the cloud-native Lakehouse era.
2. Onboard Resource Planning: Storage Capacity and TCO
A common misconception in budgeting is: “Three layers mean a threefold increase in storage costs (Raw * 3).” This narrow view falls short and fails to account for the physical behavior of structured data streams as well as the overall cost-effectiveness of a modern cloud architecture.
Why data usage is decreasing over time
In fact, the sheer physical storage capacity decreases dramatically with each stage of processing.
- Bronze acts as an unprocessed buffer and stores the unfiltered flood of input data (e.g., IoT temperature signals recorded every second over a period of months).
- Silver consolidates the CDC stream. Instead of storing every historical change, duplicates are removed and states are summarized in a cleaned-up format.
- Gold stores highly aggregated metrics at the hourly, daily, or route level.
The storage profile thus resembles a pyramid: in practice, the high-purity gold tables often account for less than 5% of the original bronze volume. Therefore, the volume does not simply triple across the board.
TCO Analysis: Storage vs. Compute and Personnel
Focusing solely on the cost of hard drives overlooks the actual drivers of TCO (Total Cost of Ownership):
- Storage is inexpensive: Cloud object storage (e.g., Azure Blob Storage or AWS S3) costs approximately $0.02 per gigabyte per month, making it a negligible expense.
- Compute (computing power) is extremely expensive: If BI analysts and data scientists had to run their queries directly on unprocessed, raw Bronze data every time, each execution would incur enormous computing costs for temporary filtering, type conversion, and deduplication. The Medallion architecture saves money by performing computationally intensive standard transformations only once during the transition to the Silver layer and persisting these results.
- Conserve staff resources: When data is already available in a clean, structured format in Silver and Gold, there is no need to repeatedly write SQL cleansing logic in downstream systems. This saves valuable time for analytics engineers and business users.
- Resilience and security through redundancy: The deliberate retention of layers ensures robust reliability. If a transformation fails or a source system goes down, the pipeline can be reconstructed independently and historically accurately starting from the Bronze level, while downstream applications continue to operate uninterrupted based on the Silver and Gold levels.
Andreas & Yvonne's Databricks-Guide
Would you like all the important information at a glance?
Download the free guide to SAP Databricks now!
3. Mission Cycle: Batch Processing vs. Real-Time Data Streams
The Medallion architecture was originally designed for batch processing. Even in 2026, daily or hourly batch processing remains the undisputed standard in enterprises—it is entirely sufficient for over 90% of all business scenarios (such as financial reporting, inventory reconciliations, and SLA billing).
However, for use cases that absolutely require real-time processing (e.g., immediate GPS-based route alerts in the event of temperature deviations while driving), the architecture does not need to be discarded. Real-time processing is pragmatically integrated into the overall workflow as a targeted supplement:
- The IoT stream continuously writes data to the Bronze table via lean streaming pipelines.
- While standard tables are updated in batches, real-time requirements can be implemented directly on the same physical tables using incremental materialized views or structured micro-batches. This keeps the architecture consistent and maintainable without introducing unnecessary complexity to scenarios that do not require real-time processing.
4. Docking Allowed: Medallion Meets Data Mesh
In modern enterprise data platforms, the transition between the Medallion architecture and the data mesh paradigm(decentralized organization of data by business domain) is seamless. These concepts do not conflict with one another—they are fully compatible.
Historically, data warehouse data was typically structured according to logical responsibilities (e.g., separate storage areas for finance, marketing, HR, or logistics). This traditional structure aligns perfectly with the domain ownership principle of the data mesh.
Bronze Layer vs. Source-Aligned Data Product
If the bronze layer of a domain is viewed not merely as a passive data dump, but rather as a stage where fundamental, non-business-oriented basic operations are already performed (such as formal conversion to the delta format, technical deduplication, or the assignment of global metadata), the difference between it and a source-aligned data product effectively disappears. This is all the more true when one considers that a Medallion architecture does not necessarily have to be managed by a central IT team, but can be operated decentrally within the domains.
Gold Layer vs. Consumer-Aligned Data Product
The difference between traditional data marts, gold-layer outputs, and consumer-aligned data products is purely a matter of terminology. All of these terms refer to business-ready, semantically rich data that has been optimized for end-user consumption.
The fundamental difference lies not in the final output, but in the data release philosophy. In traditional data warehouse approaches, business users were only allowed to access the final “gold” data marts, which created the well-known development bottlenecks in IT. The Medallion architecture in the Lakehouse breaks down this silo by granting advanced users and data scientists targeted read access to the cleaned-up intermediate results (the Silver layer). This combines the flexibility of a decentralized data mesh with the structural reliability of a traditional data platform.
5. Conclusion and Architecture Recommendation
The Medallion architecture is far more than a theoretical classification system—it is the pragmatic foundation for scalability, trust, and cost-effectiveness in the modern data ecosystem.
Based on this, we have three recommendations for your architecture roadmap:
- No direct access to raw data: Deny business units access to the Bronze Layer to prevent the emergence of shadow IT and conflicting metrics.
- Unlock Silver as an innovation platform: Give your advanced analysts and data scientists access to the streamlined Silver layer for experimental PoCs, thereby finally eliminating the central IT bottleneck of the past.
- Holistic cost planning: Treat storage and compute costs as a single unit. The deliberate redundancy of persistent intermediate layers drastically reduces your cloud costs by eliminating duplicate calculations.
Your architecture is unique – Let’s talk CTA
The way you structure your data flows plays a key role in determining the performance and acceptance of your BI and AI platform. Together, we’ll analyze your existing SAP landscape and design a space station that can hold up—even if the mission takes longer than planned.
Your data strategy is unique—your consulting should be too.
The choice between these methods depends on countless factors: your existing system landscape, your business goals, and your data culture. There is no standard answer.
Let’s talk, with no obligation, about which path is right for you.
Please contact us to schedule a personal consultation.
