Zero-Copy Delta Share: The End of Data Plumbing
- Databricks
- Databricks
- 5 min reading time
Dr. Andreas Wagner
This wiki explains how to overcome one of the biggest hurdles in the modern data landscape: the often complex, slow, and costly exchange of data. From SAP to SAP Databricks.
The solution is Databricks Delta Sharing and the "zero-copy" principle. Andreas describes in the wiki how this allows live data to be shared securely and in real time without having to create a physical copy.
Table of contents
There was a time when data integration was, above all, one thing: tedious. Anyone who wanted to use SAP data in a modern platform like Databricks had to set up pipelines, program extractors, and watch as valuable business semantics evaporated into flat CSV files. We called it “data plumbing”—and it was the opposite of a fresh start.
However, with the introduction of the SAP Business Data Cloud (BDC) and the principle of zero-copy delta share, this era of redundant copies is finally coming to an end. The final preparations are complete. The countdown is on.
2. The Engine Room: Where the Data Really Lies
It all starts in the FOS (File Object Store), SAP’s cloud-native storage layer. Here, data is not stored in rigid database tables, but in the HDLF (High-performance Data Lake Format) —technically speaking, a highly optimized implementation of Apache Parquet.
What makes it special: These files are immutable. Every change creates new versions with metadata, enabling true ACID transactions (Atomicity, Consistency, Isolation, Durability) at the file level. In short: a foundation that holds up. And that’s no small matter—it’s the prerequisite for everything that follows.
3. The Benefits of Zero-Copy Delta Sharing at a Glance
Why is this technological leap so crucial? The answer lies in the massive reduction in complexity—less weight, more thrust:
- No data copies, no ETL pipelines: We eliminate the costs and sources of error associated with duplicating and moving data.
- Real-time data exchange: Teams always work with the most up-to-date information directly from the source—without having to wait for time-consuming batch runs.
- Security and Control: SAP retains control over the data. Access rights are managed centrally and can be revoked at any time.
- Open and flexible: As an open standard, Delta Sharing works across different platforms and cloud providers.
4. The Digital Handshake: The Cloud-Native Mechanics Behind Delta Sharing
But how does this actually work from a technical standpoint? Let’s take a look under the hood. Imagine that the Databricks systems and the SAP BDC are communicating with each other. Instead of SAP laboriously packaging and sending the data, an elegant REST-based exchange takes place—the strict separation of storage and compute in its purest form.
First comes the handshake: The Databricks client authenticates itself using a bearer token —a digital security key. The BDC server sends back a manifest: a structured list of files representing the current data state. Then comes the crucial step: The server generates a pre-signed URL for each of these files—a temporary link with built-in access rights and a short lifespan.
The actual data stream never passes through the SAP application server. The Databricks worker nodes take over: using byte-range requests, they load only the specific data blocks that are actually needed directly from SAP storage (columnar pruning). The analytical processing takes place entirely in Databricks—while the storage remains securely with SAP.
5. Practical Example: The Smart Cash Flow Forecast
What does this look like in practice? A sales team wants to use daily sales figures from S/4HANA to generate forecasts using a machine learning model in Databricks.
Without Delta Sharing:
A complex ETL process extracts the data daily. By the time the data scientist begins their analysis, the data is already 12–24 hours old. The process is expensive, error-prone—and is painfully reminiscent of a rocket that first has to laboriously haul its own fuel to the launch pad in canisters.
With Delta Sharing:
Sales data is published directly from S/4HANA as a data product. The data science team accesses it in real time. Forecasts are more accurate because they are based on actual, real-time data. There is no overhead involved in building the data pipeline.
6. The Art of Translation: Why We Need Metadata
However, simply accessing the data is not enough to ensure that it is understood by the target system. Data without context is just noise. That is why BDC Connect uses two protocols as “interpreters”:
- ORD (Open Resource Discovery): Describes, in a machine-readable format, where data comes from and how it is versioned.
- CSN (Core Schema Notation): Ensures that primary keys and table relationships are automatically recognized when mounting in the Unity Catalog.
7. Universal Governance: Who gets to come on board—and who doesn't?
But context alone isn’t enough—we also need clear rules about who is allowed to access this data. Security doesn’t stop at system boundaries. We’re building a seamless identity pipeline: Microsoft Entra ID serves as the anchor point. Via the IPS (Identity Provisioning System), we mirror identities in SAP IAS, while Databricks uses the same Entra identities. When the Unity Catalog enforces access controls such as row-level security, it does so based on a synchronized, consistent identity—regardless of which system a user is working on.
The result: Role changes or deactivations take effect immediately and consistently—from SAP Datasphere to Databricks Notebook.
8. The Dremio Turning Point & A Look Ahead to 2026
It’s also worth taking a look at what’s on the horizon: Choosing the right architecture today protects your investments for the future. In May 2026, SAP paved the way for Apache Iceberg with the acquisition of Dremio. The catalog is becoming decentralized, and the BDC format-agnostic. This means that a Catalog Orchestration Layer (e.g., Atlan) is becoming increasingly important for maintaining an overview in a multi-catalog environment (Unity, Polaris, Horizon).
For AI scenarios, the MCP (Model Context Protocol) also comes into play. It abstracts the technical layer so that an AI agent sees only the stable data contract—regardless of whether Delta or Iceberg is running in the background.
9. Conclusion: Ready for liftoff!
Zero-Copy isn't just an architectural decision. It's the moment when we stop laying pipes and start becoming true architects of business success.
"Data plumbing" has had its day. And the question is no longer whether—but when—you’ll take advantage of this opportunity.
The Lakehouse mission is taking off. Are you on board? 🚀
Andreas & Yvonne's Databricks-Guide
Would you like all the important information at a glance?
Download the free guide to SAP Databricks now!
Your data strategy is individual - your consulting should be too
The introduction of zero-copy delta sharing is more than just a technical implementation—it is a strategic step toward modernizing your data architecture. Whether and how you make the most of this technology depends on your specific system landscape, your security requirements, and your long-term goals for analytics and AI. There is no one-size-fits-all solution here.
Let's talk, with no obligation, about how Delta Sharing can revolutionize data exchange in your company and unlock the full potential of your SAP data. Contact us for a personal consultation.
