Delta Sharing for Apache Iceberg

Delta Sharing for Apache Iceberg

Data sharing between organizations has evolved from costly duplication processes and complex point-to-point integrations to open protocols that enable real-time information sharing. This transformation is especially relevant as modern companies need to collaborate with multiple partners, clients, and suppliers, each using different analytics tools and platforms.

The convergence of Delta Sharing and Apache Iceberg marks a milestone in this evolution, breaking down traditional barriers to enterprise data sharing. This integration allows organizations to share their Iceberg-format data assets through Unity Catalog, maintaining centralized governance while enabling access from any compatible analytics engine, from Spark and Trino to specialized BI tools.

What is Delta Sharing?

Delta Sharing is the first open protocol for securely sharing data with other organizations, regardless of the compute platform they use.

This means a team can expose their data directly in the cloud, and a consumer can analyze it from tools like Pandas, Spark, Tableau, or even external systems, without moving or duplicating the data.

General architecture of Delta Sharing

Main benefits of Delta Sharing

🕒 Real-Time Access

Data is automatically kept up to date, with no need for manual synchronization processes.

🛡️ Centralized Security

Unified access control with Unity Catalog, full auditing, and granular permissions.

🔄 Total Interoperability

Compatible with multiple languages and analytics engines, removing technological barriers.

💾 Elimination of Silos

Prevents the creation of copies and information silos, reducing costs and complexity.

What is Apache Iceberg?

Apache Iceberg is an open table format designed for large volumes of analytical data. Its goal is to provide reliability, scalability, and multi-language compatibility in modern data lakehouse environments.

Key features of Apache Iceberg

FeatureBenefitImpact
Versioned metadataEnables time travel and auditingHistorical analysis and data recovery
Evolving schemasStructural changes without downtimeFlexibility in development
Evolving partitionsAutomatic reorganizationContinuous performance optimization
Multi-engine compatibilitySpark, Flink, Trino, PrestoFreedom of tools
ACID transactionsGuaranteed data consistencyEnterprise reliability

info

Apache Iceberg has become the de facto standard for modern data lakehouse architectures, adopted by organizations like Netflix, Apple, Airbnb, and LinkedIn to manage petabytes of data.

What does Delta Sharing mean for Iceberg?

The integration of Delta Sharing with Apache Iceberg represents a revolutionary advance: data providers can share Iceberg tables in the same way as any other asset in Unity Catalog, removing traditional barriers to data sharing.

The simplified process

In practice, this means that if you have a table managed in Iceberg within Databricks:

  1. Enable metadata with UniForm, which generates compatibility for both Delta and Iceberg
  2. Include that table in a Unity Catalog share
  3. Consumers connect via Delta Sharing from external platforms that read Iceberg

Common mistakes and how to avoid them

Not registering the table in Unity Catalog

If the table is not registered in Unity Catalog, it cannot be shared or governed properly.


Solution: Make sure to create or migrate tables to Unity Catalog before sharing them.

Not enabling column mapping

Without column mapping enabled, Iceberg compatibility may fail and cause read errors.


Solution: Use ‘delta.columnMapping.mode’ = ‘name’ when creating or modifying the table.

Incompatible protocol versions

Using outdated versions of Delta Lake can prevent interoperability.


Solution: Ensure that the read and write protocols are compatible and up to date.

Sharing sensitive data without governance

Sharing data without defining permissions and auditing can create security risks.


Solution: Set up access rules and review audit logs in Unity Catalog.

Basic configuration

To enable Iceberg compatibility on a Delta table, you need to set the following properties:

-- Enable Iceberg compatibility on an existing table
ALTER TABLE my_table
SET TBLPROPERTIES (
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Important requirements

To successfully implement Delta Sharing with Iceberg, you must meet these requirements:

The table must be registered in Unity Catalog
Column mapping must be used

The Delta read and write protocols must be at minimum compatible versions

-- Create a new table with Iceberg compatibility from the start
CREATE TABLE catalog.schema.iceberg_compatible_table (
    id BIGINT,
    name STRING,
    created_at TIMESTAMP
) USING DELTA
TBLPROPERTIES (
    'delta.enableIcebergCompatV2' = 'true',
    'delta.universalFormat.enabledFormats' = 'iceberg',
    'delta.columnMapping.mode' = 'name'
);

-- Add the table to a share
CREATE SHARE analytics_share;
ALTER SHARE analytics_share ADD TABLE catalog.schema.iceberg_compatible_table;

tip

Once configured, the table can be added to a share like any other resource, and will be available via Delta Sharing for consumers working with Iceberg-compatible tools.

Real-world use cases

Do you want to go deeper, check the technical documentation, or see real-world usage examples? Explore the official open source repository of Delta Sharing, where you’ll find documentation, releases, and practical examples:

Loading...
Waiting
Waiting
Waiting
Waiting

Conclusion

Delta Sharing and Apache Iceberg are transforming the way organizations share, govern, and consume data at a global scale. Their adoption enables breaking down silos, accelerating collaboration, and ensuring security and traceability in increasingly complex and heterogeneous environments. As more companies seek interoperability and flexibility in their data architectures, these technologies are consolidating as fundamental pillars for the future of open, secure, and efficient data sharing.

tip

To get the most out of Delta Sharing and Apache Iceberg, be sure to define a clear data governance strategy and check the compatibility of your analytics tools. Interoperability and security will be key to the success of your data sharing architecture.

Resources

  • #Delta Sharing
  • #Apache Iceberg
  • #Data Governance
  • #Unity Catalog
Share:
Contáctanos