Part 2 of our investigation into long-context #RAG is out! After the results we published in August, there was tremendous interest in how the latest releases from OpenAI and Google perform against our benchmarks. We evaluate: * OpenAI o1-preview and o1-mini * Google Gemini 1.5 Pro, Gemini 1.5 Flash (May release) After running these additional experiments, we found that: * OpenAI o1 models show a consistent improvement over Anthropic and Google models on our long context RAG Benchmark up to 128k tokens. * Despite lower performance than the SOTA OpenAI and Anthropic models, Google Gemini 1.5 models have consistent RAG performance at extreme context lengths of up to 2 million tokens. * Models fail on long context RAG in highly distinct ways Full details in our blog post: https://lnkd.in/gKj--uq3
Databricks Mosaic Research
Software Development
San Francisco, California 27,776 followers
We remove the barriers to state-of-the-art generative AI model development and make data + AI available to all.
About us
At Databricks Mosaic AI, we believe that all organizations should have access to state-of-the art data + AI capabilities. The Mosaic Research team is continually evaluating methods to optimize the model development process - from algorithms to systems to hardware - so you can get more accurate insights, faster. Our rigorous science leads to real results.
- Website
-
https://www.databricks.com/research/mosaic
External link for Databricks Mosaic Research
- Industry
- Software Development
- Company size
- 5,001-10,000 employees
- Headquarters
- San Francisco, California
- Type
- Privately Held
- Founded
- 2021
- Specialties
- machine learning, optimization, deep learning, natural language processing, computer vision, artificial intelligence, ML, AI, CNN, PyTorch, NLP, and CV
Locations
-
Primary
160 Spear St
15th Floor
San Francisco, California 94105, US
Employees at Databricks Mosaic Research
Updates
-
Join us in Mountain View for the next event in the Compound #AI Systems meetup series! Date/time: October 22, 2024, 6:30-9:00 PM Pacific Location: Databricks Mountain View office (see RSVP link for details) RSVP here: https://lu.ma/hfqn3lj3 We bring together people from research and industry to highlight Composable Data Systems, AI/ML integrations, and Open Source project you can’t miss. Speakers present tools, tips and tricks, and discussions will put a spotlight on challenges and opportunities in the field today. Co-sponsored by LanceDB and Databricks.
Compound AI Systems: October/MTV · Luma
lu.ma
-
Our research and engineering teams did great work making the latest Llama models from AI at Meta available to our Databricks Mosaic AI customers. These new, smaller models are excellent options for latency-sensitive use cases. Fine-tuning Llama 3.2 on your data in Databricks is just one simple command away; learn more in this blog post by Daniel King, Hanlin Tang and Patrick Wendell.
Excited to partner with AI at Meta to release the latest models in the Llama 3 series on Databricks! The Llama 3.2 release pushes the frontier of enterprise #GenAI with new smaller models for latency + cost-sensitive use cases and larger multimodal models for visual understanding. Customers can easily get started tuning the Llama 3.2 models securely and efficiently on their data. Available today in Databricks Mosaic AI. https://dbricks.co/3BhIRx3
-
If you're in San Francisco Bay Area, join us at the inaugural Compound AI Systems meetup at the Databricks SF office tomorrow evening, Tuesday, Sep 24, 5:30-8:30 PM! RSVP: https://lu.ma/roev9rc6 #AI engineers, researchers and practitioners are coming together to discuss the practical realities of composing these systems and services, with a focus on modularity and observability. Co-sponsored by LanceDB and Databricks. Speakers: Chang She, CEO/Cofounder LanceDB Erik Bernhardsson, Founder/CEO Modal Kobie Crawford, Developer Advocate, Databricks RSVP here: https://lu.ma/roev9rc6
Foundations of Compound AI Systems · Luma
lu.ma
-
🎉 🙌 Mosaic AI Model Training now supports the full context length of 131K tokens when fine-tuning Meta Llama 3.1 models. Databricks customers can build even higher-quality #RAG systems by using long context length enterprise data to create specialized LLMs: https://lnkd.in/gdum_XJS
-
📣 At Databricks, we want to help customers build more #inference-friendly #llms. Our MixAttention architecture demonstrates that reductions in KV cache size make it possible to maintain model quality while improving inference speed and reducing memory footprint. Read our latest blog post to learn more: https://lnkd.in/gcHxM4DX
-
Great work by our Mosaic AI researchers + engineers to develop + ship a new judge for our #LLM agent eval tool, used to improve the quality of agentic #GenAI applications. Kudos to Max Marion, Arnav Singvi, Samraj M., Avesh Singh, Michael Carbin and Alkis Polyzotis!
Announcing the launch of an improved answer-correctness judge in Agent Evaluation, offering significant accuracy gains, especially on customer-representative use cases. Details and our evaluation methodology https://dbricks.co/3TcMaLZ Developed by Mosaic AI's research and engineering teams, this update is now automatically available to all users.
Databricks announces significant improvements to the built-in LLM judges in Agent Evaluation
databricks.com
-
New blog post from one of our Mosaic AI customers: learn how Twelve Labs enables developers to create multimodal embeddings for advanced video understanding use cases by using Databricks Vector Search to index and query high-dimensional vectors. https://lnkd.in/guZEig9A
-
#DSPy is an excellent example of the approach that is needed to automate the improvement of modular applications that have generative AI at their core. Great to see the community connecting and discussing these next steps toward reliable, useful, sophisticated AI functionality. Kudos to Stanford University NLP researchers and our colleagues at Databricks!
Had a great time hosting the South Bay Generative AI meetup (https://lnkd.in/gsuyvEym) last night at Databricks Mountain View! Great presentations on #DSPy (https://dspy.ai) by four of the core authors: Omar Khattab, Arnav Singhvi, Krista Opsahl-Ong, and Dilara Soylu. Shoutout and thanks to the meetup organizers, Mark Kuczmarski and Shashank Bharadwaj!
-
We are looking forward to hosting this Thursday's South Bay GenAI meetup at our Mountain View office! Omar K. and Arnav Singhvi of Databricks, along with Stanford University colleagues Krista Opsahl-Ong and Dilara Soylu, will be presenting some of the latest innovations in #DSPy. This Python package helps you optimize your #LLM prompts programmatically, borrowing the patterns used in training and hyperparameter tuning to improve application performance without manual intervention. If you're an aficionado of GenAI in the South Bay, this is a session you don't want to miss! RSVP here: https://lnkd.in/gmTNJKwy
DSPy: Programming—not prompting—Foundation Models at Databricks, Thu, Aug 22, 2024, 6:30 PM | Meetup
meetup.com