Dr. Tim Dörnemann’s Post

🚀 𝐄𝐱𝐜𝐢𝐭𝐢𝐧𝐠 𝐓𝐢𝐦𝐞𝐬 𝐢𝐧 𝐭𝐡𝐞 𝐃𝐚𝐭𝐚 𝐋𝐚𝐤𝐞𝐡𝐨𝐮𝐬𝐞 𝐖𝐨𝐫𝐥𝐝! 🚀 🆕 Its already Tuesday, and a suspicious absence looms: no new catalog has surfaced this week. But fear not❗ The 🧊 Iceberg Catalog landscape is evolving rapidly with significant announcements from Snowflake and Databricks. At HANSETAG, we're thrilled to introduce 𝐓𝐈𝐏 - our 🦀 #Rust-native #OSS #Iceberg #REST Catalog designed for superior data quality, governance, and flexibility. 🌟 𝐓𝐈𝐏 𝐩𝐫𝐨𝐯𝐢𝐝𝐞𝐬: • 𝐌𝐮𝐥𝐭𝐢-𝐭𝐞𝐧𝐚𝐧𝐜y: Manage multiple projects and warehouses effortlessly. • 𝐂𝐡𝐚𝐧𝐠𝐞 𝐄𝐯𝐞𝐧𝐭𝐬: Stay updated with every table change. • 𝐂𝐨𝐧𝐭𝐫𝐚𝐜𝐭 𝐕𝐚𝐥𝐢𝐝𝐚𝐭𝐢𝐨𝐧: Ensure seamless data operations. • 𝐂𝐮𝐬𝐭𝐨𝐦𝐢𝐳𝐚𝐭𝐢𝐨𝐧: Extend TIP with ease using our built-in and public interfaces. Explore more: https://lnkd.in/ev5N_WgG #DataLakehouse #DataGovernance #OpenSource #Rust #IcebergCatalog #DataContract #DataMesh #DataProduct #Iceberg #DataProduct

There is a lot of dynamics in the #Lakehouse stack these days. It is high time that the community comes to an agreement for a common open catalog standard that can replace the increasingly dated #hivemetastore. For instance, Databricks (#DeltaLake ecosystem) open sourced #UnityCatalog last week: https://lnkd.in/d48Qvm44. Now, this catalog here by HANSETAG is an open source implementation of the #Iceberg catalog REST API, which comes with many bells and whistles for #enterprise deployments such as secure credential mapping. These two are not necessarily just disjunct alternatives when you take into account #Databricks acquisition of Tabular (now part of Databricks) and the plans they shared last week for convergence of #DeltaLake and #Iceberg: https://lnkd.in/dSC2zEf7 At the end of the day people will want a catalog that supports all established table formats and comes with the critical enterprise bells and whistles!

For those you have doubts about Data Catalogs, the following article can help clarify them. With the current rise in the news about Snowflake's Polaris Catalog and Databricks' Unity Catalog, it is good to understand what these catalogs are and what are the types of catalogs available. There are three types of data catalogs: - Technical catalogs (which focus on metadata for individual tools). - Embedded catalogs (which integrate metadata within specific data products). - Universal catalogs (which centralize metadata across an entire data estate). Each type serves different purposes in managing and understanding complex data environments, but are they fit for the present and future of the business needs? The traditional catalogs, whether technical, embedded, or universal, may not fully meet the diverse needs of modern data ecosystems. As data tools and users become increasingly varied, and as metadata itself evolves, there's a need for a more advanced solution—one that goes beyond simple cataloging. What will be the natural evolution of data catalogs? #Datacatalog #Dataliteracy #Datagovernance

🚀 Exciting News! 🚀 📊 Databricks has acquired #Tabular! Explore the implications of this acquisition on the data and analytics landscape. 📑 Read More: https://lnkd.in/gjW8MBE3 #databricks #iceberg #datalakehouse #DQLabs #deltalake #dataquality #dataobservability

News: #snowflakeSummit. Open Data Revolution Continues My Analysis: Snowflake’s strategy around Iceberg is a nuanced topic with significant implications. While the simplicity of Snowflake’s compute services is enhanced by Iceberg, the real value lies in how Snowflake chooses to handle interoperability and external engine support. The Polaris news might crucial in defining its position in the ever-evolving data platform market. Polaris is another #opensource gift from Netflix. Implications: Many including us were asking..will Snowflake choose to embrace Iceberg and its open data potential, even if it means disrupting Snowflake’s current model? Looks like the team is going to disrupt themselves to bring in a new era in open data. The technology industry has a saying, especially true during major market shifts: “Better to eat your own before someone else does.” This means that companies should cannibalize their own advantages before competitors do. For Snowflake, this is happening as the fully embracing Iceberg and its open data capabilities before competitors leverage these advantages against them.

"Starburst not only broadens its product offerings but also positions itself as a strong rival to established proprietary data warehouse vendors such as Teradata and Snowflake by integrating real-time data capabilities and supporting Snowflake Iceberg Catalogs." Check out this The Futurum Group article by Paul Nashawaty and Sam Holschuh about Starburst's latest Icehouse announcements: https://okt.to/yh8XjL #Icehouse #trinodb #ApacheIceberg

Sam H. and I had a great briefing today with Starburst. Vishal Singh and Manveer Sahota provided an update on the company, its partnerships with Dell Technologies and Google, and a great in-depth demo of #StarbustGalaxy, their SaaS offering showcasing an end-to-end analytics platform for both cloud and on-premises workloads. Another area I have to highlight is open-source projects. Trino is a distributed SQL query engine built in Java and is open source. It's very cool to check out. It is an efficient tool for running analytic queries against various data sources, ranging in size from gigabytes to petabytes. Trino was created to provide a platform for interactive data analytics that can deliver the same speed as commercial data warehouses while scaling up to the size of the largest organizations in the world. Trino was initially developed at Facebook under the name Presto. It was designed to handle the scale and performance requirements for querying their Hive-based data warehouse. However, Trino has been expanded to connect to many other data sources. Trino’s architecture is designed to combine the best of both traditional shared-nothing and shared-disk systems. It is built on a distributed cluster model where each node in the cluster has compute and storage resources. This allows Trino to scale horizontally across multiple nodes, with each node capable of handling a portion of the workload. This architecture ensures that Trino can handle massive amounts of data with ease. Trino is optimized for interactive queries and supports a wide range of data formats and data sources. It can be used with popular databases like MySQL, PostgreSQL, and MongoDB, as well as cloud storage solutions like Amazon S3 and Microsoft Azure. Additionally, Trino supports a range of data formats, including CSV, JSON, Avro, and Parquet. Definitely worth checking out! The Futurum Group #opensource Meta Amazon Web Services (AWS) Microsoft #postgresql MongoDB Justin Borgman #multicloud #distributedcloud

Starburst CEO Justin Borgman recently outlined a vision for the future of data architecture with "The Icehouse Manifesto: Building an Open Lakehouse." I've written and been briefed by Starburst before, and am convinced that this approach could fundamentally change the data management landscape. The manifesto introduces a lakehouse architecture that leverages #Trino for querying and #Iceberg for table formatting, and Justin advocates for an open and flexible system that simplifies data operations and enhances scalability by using standard #SQL across a decentralized platform. For organizations looking to evolve their data strategy and embrace a more open, agile approach to data warehousing, his concept ought to make a lot of sense. Consider the potential of an open lakehouse as detailed in the manifesto, which could be the key to unlocking greater innovation and efficiency in your data ecosystem. Sam Holschuh Beth Winkowski Jay Chen Tobias Ternström The Futurum Group

With #SnowflakeSummit around the corner, we're on a mission to help you digest all that's possible in Snowflake—shining a light on the areas that help your organization get the most value out of your data. Our #SnowflakePanorama ❄️ series dives deep into functionality—ultimately giving you a big picture understanding of all the ways you can access and enrich data for innovation and competitive advantage. Part one of the series, written by Kwon K. Lee, focuses on Data Engineering.