Mike L.’s Post

Introducing Scale AI's SEAL Leaderboards -- the first private, expert-driven, trustworthy LLM contest. Our Safety, Evaluations, and Alignment Lab (SEAL) designed the leaderboards with three principles: 🔒 Private evaluation datasets that can't be gamed 🥇 Evolving competition with periodic leaderboard updates 🔍 Expert evaluations using domain-specific methodologies Initial domains covered: Coding, Instruction Following, Math, and Multilinguality To see where GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3 rank, visit https://lnkd.in/gaBTsK9P

Exciting update for the GenAI Testing & Evaluation world (which is much smaller than my LinkedIn feed would lead me to believe)! Check it out.

📣 Scale is excited to release the SEAL leaderboards today, kicking off the first truly expert-driven, trustworthy LLM contest open to all: https://lnkd.in/g32X8Dcz   Compared to existing benchmarks, these leaderboards developed by our Safety, Evaluations, and Alignment Lab (SEAL) are built on: ✅ Private datasets that can’t be gamed ✅ Evolving competition ✅ Expert evaluations The initial domains covered include: Coding, Instruction Following, Math (based on GSM1k), and Multilinguality. These leaderboards are regularly updated to include new models and capabilities. Our goal is to foster a culture of transparency and openness in the development and evaluation of frontier models. 👉 Finally, we are also announcing the general availability of Scale Evaluation: a platform to enable organizations to evaluate and iterate on their AI models and applications. Learn more: https://lnkd.in/dVwvAhmN 👈 Check out the leaderboard yourself here: https://lnkd.in/gghYicsm And learn more about the development and motivation behind the leaderboards: https://lnkd.in/gSfZYMkE

It is very insightful for those comparing LLms for various use cases.

My new goto model can't wait for the finetunes. before this I was using wizardlm I'm not convinced llama 3 is the best for me especially for processing arxiv papers I might pivot to llama 3 after reducing papers down to necessary components / manageable size, but for now I only rely on models finetuned to handle 32k plus

VoiceCraft - A new zero-shot speech editing and text-to-speech model. Just tried it out. Really amazing speed at inference. You can also do voice cloning and it only needs like 3 seconds of reference. At the moment, it only supports English though. Models are on Huggingface 🤗 Really cool to see how TTS research has progressed. Wasn't like that when we started it at Axel Springer. 🎙️ Demo: https://lnkd.in/gbxqmz_d 📝 Paper: https://lnkd.in/gbTNnF_Z

In this episode, we discuss Mixtral of Experts by Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed. Mixtral 8x7B is a Sparse Mixture of Experts (SMoE) language model, building on Mistral 7B's architecture with 8 experts per layer, among which two experts are selected per token for processing, allowing access to 47B parameters but using only 13B actively. It excels in benchmarks, surpassing Llama 2 70B and GPT-3.5, especially in areas like math, code generation, and multilingual tasks. A special instruction-following version called Mixtral 8x7B – Instruct also outperforms leading models, with both models being open-sourced under the Apache 2.0 license.

https://lnkd.in/eAsNT8GC congratulations to Corinna Hertweck for the deep thinking in, and super-fast and effective writing of, "What's Distributive Justice Got to Do With It?" the paper concluding and cristallizing our shared, tortuous, interdisciplinary journey on machine learning fairness. She'll have the opportunity to discuss this at #AIES '24, hopefully engaging philosophers and computer scientists in the same way. There no better way to end our project with Christoph Heitz (and for Corinna to conclude her doctorate)! (Spoiler alert: we're not offering a "solution to the fairness issue", but we've dug the problems deeper and discovered new levels of complexity.)

AIs can generate new knowledge now. It's not yet 2024. What does next year look like? and 2025? and 2030? We aren't prepared for the rate of change and the increase in AI capabilities.

Daniel, this workshop sounds incredibly insightful! I'm particularly interested in your comparisons between Triton and CUDA, as well as your deep dive into the Llama model and its bug fixes. Thanks for sharing these valuable resources and for your dedication to advancing our understanding of LLMs. Looking forward to more content like this! #AI #ArtificialIntelligence #MachineLearning #DevOps #SoftwareDevelopment #Programming #FutureOfWork #Automation #Innovation #Technology #CareerDevelopment #Upskilling #Reskilling #DigitalTransformation #ITJobs #TechJobs #DeveloperProductivity #FinTech #HealthTech #CyberSecurity #DataScience #CloudComputing #InternetOfThings (IoT)