Jason Snyder

Snowflake Polaris: POC Status 2. Branching, tagging, branch fast-forwarding, expiring snapshots, and all other maintenance procedures are working smoothly without any issues. I’m hopeful that Snowflake will assist with the issues I’m experiencing related to add_files and register_table procedures. Snowflake Dremio ambariCloud Today's Quote: Arrogance has toppled empires.

7

Discover the power of metadata management in Databricks with Apache Hive Metastore (HMS). In his latest blog, "Hive Metastore (HMS) and hive_metastore in Databricks", Gentrit Mehmeti explores how Hive Metastore supports large-scale data organization and enhances compatibility with Apache Spark. Gentrit also highlights the evolution of data governance with Unity Catalog, which offers advanced features like auditing and improved access control, shaping the future of data management. Learn how mastering metadata is key to handling today’s data complexity. 🖇 https://lnkd.in/dZrvDnnc #DataManagement #Databricks #ApacheHive #Metadata #UnityCatalog #CloudComputing #ApacheSpark

29

Great blog on Spark AQE!

12

"Databricks Ushers In New Era for Open #Data And #AI Governance With Unity Catalog", click to get the full story via the News Desk over on Elnion. https://lnkd.in/gTkxYDRB #dataaisummit #data #datalake #ai #ml #gpt #genai #cloud #analytics #bi #datascience #datascientist

1

Navigating the Databricks landscape can be challenging, even for seasoned data engineers. Our resident expert Robert Leach wrote about the common mistakes people make, and offers practical advice on how to: 👉 Tailor your clusters to drive for performance and cost efficiency 👉 Ensure consistency, compliance, and reliability with proper data governance policies 👉 Boost your productivity by mastering the Databricks Command Line Interface 👉 Implement robust access controls, encryption, and regular security reviews 👉 Make the most of the platform’s capabilities You’ll want to bookmark this post! 🔖 https://bit.ly/45DZqhY

15

🔒 Fine-Tune LLMs with Differential Privacy in Databricks   Databricks simplifies the specialization of foundation models and offers a range of tutorials to showcase their capabilities. However, standard fine-tuning doesn't guarantee #dataprivacy. This is where #DP fine-tuning comes into play, providing the necessary privacy assurances. In our latest project, we demonstrate the ease of using Sarus and Databricks to fine-tune an open-source model without worrying about training set leakage. We compared models fine-tuned with and without DP (Epsilon 3, Delta 1e-5). Using GPT-4 to judge the answers' correctness and similarity, we found that in 90% of cases, the models achieved the highest grades. While DP models had slightly lower mean utility, they offered guaranteed privacy! This approach ensures robust model fine-tuning while safeguarding sensitive information. 🚀 If you're a Databricks user looking to harness the power of LLMs without compromising on privacy, let's connect! #GenAI #Databricks #MachineLearning #LLM https://lnkd.in/euRPtYid

18

Understanding cost control is essential to maximizing your Databricks investment. Our latest blog dives into Databricks' "Budgets" and cost dashboard, exploring the pros, cons, and everything in between. 📊💡 Get insights on: - Granular tag-level tracking - Quick and easy budget setup - Comprehensive in-workspace reporting - Areas needing improvement 🔍 Read the full analysis done by Josue A. Bogran and see how you can better manage your Databricks costs. Read the Blog: https://lnkd.in/dy2tTkMa #SunnyData #Databricks #DataEngineering #CostManagement #Budgeting #CloudComputing

8

Snowflake announced its new feature, Cortex Analyst, last month. Here’s our Panda’s 🐼 take on it. In this blog Sumanth Donthula covers: the importance of self-service in analytics, why Cortex Analyst, and how Cortex Analyst outperforms other tools. Read the full blog here ⬇ https://lnkd.in/gSRtsVYK

7

Ready to improve forecasting by 20%? Quickly connect Snowflake’s new #Cortex Forecasting to PredictHQ’s Beam API and Features API to reduce mean absolute percentage error (MAPE) by 20%. Plus, get ready for even deeper explainability with our Events API coming soon to Cortex. Watch the demo where PredictHQ Chief Data Officer Dr. Xuxu Wang shows you how it works 👇 https://lnkd.in/ekR_xWZq #demandforecasting #predictiveAI

6

Unveiling Snowflake and Databricks governance frameworks 🌐 In the latest episode of #theCUBEpod, David Vellante and John Furrier dig into Snowflake and Databricks governance infrastructure and strategies. “I think [Databricks’ CEO] Ali Ghodsi is right that Snowflake doesn't want to open up, and [Snowflake co-founder] Benoit Dageville admits this. They got to go where the customers are. So, the market's going to stay very fragmented, and I think that long-term, to the extent that stays the case, that does favor Snowflake. It will force customers, if they want integration and tight governance, to stay with Snowflake,” Vellante shares. He continues, saying “the other thing is that Databricks bought Tabular for a couple billion dollars. They don't own Iceberg, which is open source, but they have talent. Ali's going to take that talent and he's going to try to bring Delta and Iceberg together. Those are the open table formats to create a unified governance framework that you can apply Unity to, and then sell the Databricks tooling behind that. That's their strategy.” “Very different strategies. If the open source community can deliver on that promise, that's an advantage for Databricks. But I do think that's going to take some time,” Vellante concludes. 📺 Get into the complete analysis: https://lnkd.in/gJmArZdh

9

Fully control your #Databricks environment directly from your #terminal with Databricks #CLI ! This powerful command line interface allows you to manage Databricks objects and resources—including #clusters, #notebooks, #jobs and #users — directly from your terminal. In this article, we have covered: 📟 What Databricks CLI is used for 📟 How to install Databricks CLI 📟 Step-by-step guide to install Databricks CLI on various #OperatingSystems (#macOS / #Linux / #Windows) 📟 How to set up #authentication for the Databricks CLI 📟 List of available Databricks CLI commands 📟 How to use Databricks CLI commands—practical examples ...and so much more! New to #DatabricksCLI or want to go deeper? Check out this article to learn more in-depth about it! Dive right in! https://lnkd.in/gXJdTXW2

5

A common misconception with AI and machine learning: “all of the resources, infrastructure and technology" it takes to get started. With Databricks ML Flow, that’s simply not the case. In the video ⬇️ Kenny Shaevel breaks down how Databricks makes it easier to manage and track your models while ensuring full transparency, traceability, and governance—without the headache. Kenny covers everything from: - Simplifying Machine Learning Implementation - Utilizing AutoML for Quick Data Evaluation - The Importance of Feature Engineering - Leveraging Feature Store for Reusability - Governance and Traceability with the Model Registry - Seamless Integration and Application of Models Excited for more? Kenny will be diving even deeper into Databricks in our upcoming live stream series. Keep an eye out for details coming soon! 👀 #machinelearning #databricks #dataanalytics #MLFlow #AutoML #featurestore #dataengineering #techinnovation

5

Yesterday we demonstrated how quickly you can move contextualize and normalized data from Flow to Snowflake during a webinar with our partners at Streamline Control. This strategy makes all of the historical data in your legacy or enterprise data historians and SQL databases immediately available to Snowflake, and it's EASY!

7

Get a deep-dive into one of our performance improvements: Hierarchical Selectivity Estimation, and how for one customer workload, query duration improved by 8x and by as high as more than 100x! This feature provides a more accurate and adaptive approach to query optimization, significantly improving performance and resource utilization by understanding and leveraging the detailed data distribution within each partition:​​ https://okt.to/IWUTaq

42

Our last poll was about data and transparency. Today, Databricks has created The State Reader API , a new capability to help users have access and analyze internal state data for full transparency and ease of use __ Will this new capability make a real difference from what we are used to use so far? #Databricks #Newcapability #datatransparency #accessibility #easeofuse #nuboTed

2

2 Comments

Databricks LakeFlow helps data teams meet growing demands of reliable data & #AI through: ✔️ LakeFlow Connect: Simple & scalable data ingestion ✔️ LakeFlow Pipelines: Efficient declarative data pipelines ✔️ LakeFlow Jobs: Reliable orchestration for every workload See how👇 https://dbricks.co/3KPdPyh

560

14 Comments

DVT recently hosted Dael Williamson, EMEA CTO of Databricks, in an Insights Webinar focused on the critical role of interoperability in AI-driven business strategies. The webinar offered practical steps for CIOs and IT executives aiming to enhance their organisation's interoperability. DVT, as a strategic partner of Databricks, is uniquely positioned to help businesses unlock the full potential of these technologies. Webinar link - https://lnkd.in/dtWtedQP. #ai #databricks #webinar #technology #cto Karen Heydenrych Dantes de Beer

5