Orchestra

AWS Lambda users - we have something for you💡 Have you ever wanted to get visibility of all your Lambda processes that form core parts of your data pipelines? Well now you can! Orchestra now supports the triggering and monitoring of AWS Lambda Functions as part of end-to-end data pipeline flows. Combined with our integrations with EC2, ECS, Redshift and Snowflake this is a game-changer who is ready to move on from Step!!! Read more here 😎 https://lnkd.in/dy_bJdjc #dataengineering #aws #awslambda #orchestra #pipinstall

Adopting a Modular Data Stack and leaning on a piece of infra like Orchestra to knit everything together is a revolutionary new approach to Data. Find out more in the Architecture Whitepaper below. https://lnkd.in/eba6Kzeh The Illusion of Free Choice #dataengineeinrg #orchestra #airflow #adf

As Orchestra joins the Snowflake partner network, CEO Hugo Lu dives into what this means for Orchestra customers (Present and Future!) https://lnkd.in/eGsx4Vr3 #snowflake #dataengineering #datawarehouse #orchestra

Ok it's another insane roundup and our subscriber list is almost reaching 4,000 🤯 Check it out here 😎 https://lnkd.in/eVEwRHez Medium 🧠 **Editors Pick** 🧠 Deciding on your Data Platform Philosophy Dylan Anderson 🧠 Visual Snowflkae — how to make Snowflake Look even better 🧠 Why we evaluated a ton of observability tools and ended up building our own 🧠 Graph RAG using Knowledge graphs and Vector DBs 🧠 Migrating from Azure Data Factory - everything you need to know 🧠 Snowflake advanced features; balancing performance and cost **Editors Pick** 🧠 The Illusion of free choice - why all OSS Workflow Orchestrators have the same fundamental problem Hugo Lu **Editors Pick** 🧠 Streamlining dbt CI/CD LinkedIn🕴 🕴How to remove the confusion and properly approach building a data platform 🕴Some fun satire RE Databricks and Sigma Josue A. Bogran 🕴The rise of Data and AI companies in YC and some other awesome stuff from Benjamin Rogojan and Joe Reis 🤓 HAPPY MONDAY for all those who celebrate🙌 #dataengineering #ainews #datanews #airoundup

Excited to present my latest piece of artwork The illusion of free choice What do all OSS Workflow Orchestrators have in common? ---- A framework you need to learn how to code ---- A framework you need to learn how to deploy ---- Encourage a monolithic approach (do everything in this app) vs. the more popular modular approach (leveraging best-in-class functionality like Fivetran Airbyte Portable dbt Labs Coalesce.io Hightouch Census etc) It's a bit like building your own computer. That might be best if you've got highly custom needs or you're one of the best gamers in the world, but it's probably overkill. It's 2024. There are better, faster, leaner ways of actioning BI and Analytics use-cases. #dataengineering #dataorchestration #freechoice #satire

Today we are excited and humbled to share that Orchestra has joined the Snowflake Partner network ❄ 🎉🎉🎉 We join an esteemed group of Technology Partners contributing to the best-in-class modular data stacks like Coalesce.io Fivetran Airbyte Hightouch dbt Labs and on the SI side Stratos Consulting We are one of only two official technology partners in the #dataorchestration category for Snowflake. This is testament to the incredible work the team have done to build the first and best-in-class unified control plane for Data Operations on Snowflake. Read more in the press release below 🎺 #snowflake #dataengineering #analytics

We need your help! If you like Hugo Lu's Medium Content please share this post so other folks in #data can keep getting this content FREE. https://lnkd.in/eQWckwvp Almost at 10k follows 🎉 #dataengineering #dataroundup #datanews

Leveraging Azure Data Factory, particularly in a VNET or using a metadata framework for dynamic DAG generation leads to so many issues --- Visibility and debugging --- DAG Factories in ADF lead to PIPELINES calling PIPELINES calling PIPELINES. These are SO HARD to debug. --- dbt-core / Coalesce.io support --- ADF does not do a godo job of surfacing dbt or Coalesce metadata! This means to debug this step you need to go into multiple tools --- Data Quality --- ADF is an unopinionated workflow orchestrator! You will need to "Build-your-own" data quality tables, aggregate and store artifacts, build dashboards and alerting systems, compounding the issue of.... -- MANAGED VNET PRICING --- Significantly (200 times the price of Azure Integration runtime) more expensive than another orchestration option! My experience with ADF made me want to give data teams a single, unified control plane that could work. If that's interesting please check-out Orchestra and see article below https://lnkd.in/eMYRSChZ #dataengineering #adf #azuredatafactory #microsoftfabric

Building a lakehouse on Iceberg? Thinking about it? Check out our latest piece with Hugo Lu below https://lnkd.in/ewa-nksv #dataengineeinrg #iceberg #apacheiceberg

Monday Morning ☕ calls for Orchestra's data roundup. It is also swiflty becoming a roundup of roundups too I guess as we like to include Christophe Blefari's newsletter in ours??? Check it out here ⚡ https://lnkd.in/ezV_wfVr 🧠 Stop being data-driven Cai Parry-Jones 🧠 Why did Databricks open-source Unity Catalog 🧠 7 surprising habits that make data engineers unstoppable Tim Webster 🧠 How to run Kafka on 100% object storage Vu Trinh 🎥 How to build a data infrastructure that lasts Benjamin Rogojan Seattle Data Guy and great content from Joe Reis 🤓 as always #dataengineering #dataroundup #datanews