There is a duality between data and code:
1. Business-facing data is the result of continually applying transformations on source data (functions)
2. The validity of those transformations is determined by whether data meets business needs (state)
Because of that duality, observing only code *or* data is insufficient. Code is incorrect when the data does not meet spec. When data is incorrect, we have to look at the code.
But there remains a gap between these two worlds, like two pieces of a puzzle that don't quite fit together.
At the Summit Platform Keynote, Snowflake announced a key bridge between data and code: Snowflake Trail. The Snowflake telemetry team (which includes folks who founded OpenTelemetry!) is bringing the technology and practices of distributed tracing to the data world.
Two weeks ago, I shared that Snowflake invested in Metaplane. Our goal is to make observability more powerful, ubiquitous, native to Snowflake — like Airpods to an iPhone.
A key part of that vision is extending data observability with metrics, traces, and logs from code execution: Data-Code Observability.
I'm excited that Metaplane is named alongside companies we admire like Datadog and Grafana to make that a reality.
#snowflakesummit #dataengineering #dataquality #dataobservability