Hanzi Mao

Can you simulate pretrained models from #PyTorch and #TensorFlow directly in #Simulink? Yes, with the new coexecution Simulink blocks! Check out this blog post from Maggie Oltarzewski to learn more. https://lnkd.in/e4RkNrgz

25

Just did a keynote in PyTorch conference, on LLM deployment. Thanks for the great work of PyTorch team, our partners, and the interests from the audience! Please try out our deployment ecosystem of torchchat, torch.compile, torch.export, AOT Inductor, torchao and ExecuTorch, and checkout all the amazing talks, posters and demos in the conference!

50

3 Comments

SSMs (Mamba, S4, etc) will overtake transformers as the primary model architecture seen in SOTA applications of multimodal LLMs, by Dec 31st 2025

3

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts https://lnkd.in/gb78Y4bn In the latest foray into the world of multimodal language models, researchers have unveiled an intriguing new approach they've dubbed "CuMo." This novel technique aims to harness the power of Mixture-of-Experts (MoE) architectures, incorporating Co-upcycled Top-K sparsely-gated MoE blocks into the vision encoder and the multi-layer perceptron connector of multimodal LLMs. The incorporation of these MoE components is a bold move, promising to enhance the capabilities of multimodal LLMs in a significant way. However, the researchers didn't stop there. They've also adopted a three-stage training approach that employs auxiliary losses to help stabilize the training process and ensure a balanced distribution of workload across the expert modules. One of the most noteworthy aspects of this project is that CuMo has been exclusively trained on open-source datasets, a commendable decision that promotes transparency and accessibility in the field of AI research. Overall, this project presents an exciting exploration of MoE architectures in the context of multimodal language models. While the true impact of CuMo remains to be seen, the innovative approaches employed and the promising early results make this a development worth keeping an eye on in the rapidly evolving field of AI.

3

Unlock the full potential of your Large Language Models (LLMs) with Intel® Extension for PyTorch (IPEX) and the Intel® LLM Library for PyTorch (IPEX-LLM). Download this whitepaper to explore the optimization of LLMs, including their performance, resource utilization, and response times in real-world applications. Link - https://intel.ly/3BBJ4ey

1

Transformers Transformers has been a major milestone in the field of NLP and heavily used in Generative AI. There are 3 variants of transformer based models and they are 1. Encoder-only 2. Decoder-only 3. Encoder-Decoder Encoder only models: These are also called as autoencoders and these are pretrained using a technique called masked language modeling. A text with a random masked token is sent to the model to predict the masked token. For example, consider the text "If you don't stop at the sign, you will get a ticket" The training input that will be passed on the encoder model is "If you don't _____ at the sign, you will get a ticket. The model is expected to predict the token (word) "stop" These models use bi-directional representations of the input to better understand the full context of a token. Examples of encoder models: BERT Family models Decoder only models: These models are called autoregressive models and are pretrained using a technique called causal language modeling. These models predict the next token using the previous tokens. For example, consider the text "If you don't stop at the sign, you will get a ticket" The training input that will be passed on the encoder model is "If you don't ______" The model will still try to predict the word "stop", but only based on previous tokens. These models are used for generative tasks, including question answering. Examples of decoder only models: GPT Family, Falcon, LLama models Encoder decoder models: These models are called sequence to sequence models and the pretraining objective varies from model to model. For example the popular FLAN - T5 uses a consecutive multitoken masking called span corruption. For example, consider the text "If you don't stop at the sign, you will get a ticket" The training input that will be passed on the encoder model is "If you don't _____ _______ the sign, you will get a ticket. The model will try to predict the tokens "stop at" These models are good at translation tasks. Examples: T5 family In all the above explanations we just took one sentence of text. The LLMs you see in the market are trained on huge volumes of text available over the internet. #LLM #encoder #decoder #transformer #genai

3

A very useful checklist to follow when you have to put in production Transformer based models

5

PyTorch has a new attention function for LLMs called FlexAttention that supports various attention variants. I just took it for a spin and compared it to other implementations of multi-head attention, and it seems to work great in combination with torch.compile. However, if you are not compiling your models (e.g., due to compatibility issues), it may be better not to use it for multi-head attention. The notebook with the benchmark code is available here: https://lnkd.in/gcKY-yMd

1,196

36 Comments

Our new technical blog post details the inner workings of Nuro's LAMBDA model, a real-time reasoning engine for autonomous driving! https://lnkd.in/eQxQfAUN We're proud to demonstrate the advanced capabilities of LAMBDA and to showcase its capacity to reason intelligently about the wide variety of challenging scenarios that our Nuro vehicles encounter and need to handle autonomously. LAMBDA has shown that it can explain complex interactions and describe safe and responsible behavior in response. What we've shared in this post is one of the many ways that we're building and utilizing foundational models for autonomous driving. The generality and sophistication of the LAMBDA model has led us to explore applying this technology to benefit Nuro's autonomy on-board in addition to improving developer efficiency and Nuro's safety case off-board. We look forward to sharing more of these advancements in the near future! Many thanks to all the contributors to this project. If you want to join us on our mission to advance the state of the art in autonomous robotics please apply here! https://lnkd.in/e9km-dV2 #nuro #ai #deeplearning #autonomousvehicles #robotics

81

3 Comments

Just Arrived in Vienna for #ICML2024, I will be presenting our work on improving GNNs' generalization ability with ensemble learning (https://lnkd.in/gimXe5fS) at Oral 5A (Thu 25 Jul 10:45 a.m. — 11 a.m.) and Poster 2407 (Thu 25 Jul 11:30 a.m. — 1 p.m.). Paper title: From Coarse to Fine: Enable Comprehensive Graph Self-supervised Learning with Multi-granular Semantic Ensemble. A heartfelt thank you to all my collaborators for their incredible support; this work would not have been possible without them. I am also looking forward to reconnecting with old friends and making new ones. If you are attending ICML or are interested in my research, please join me during my presentation session. Additionally, please feel free to reach out to exchange ideas if you are interested in topics such as (1) improving graph model generalization across different datasets, and (2) foundational models for time series forecasting. See you in Vienna! 🇦🇹 #ICML2024 #MachineLearning #ArtificialIntelligence

14

AI NEWS SERIES Groq Secures $640M for Hardware for AI Inference to Challenge Nvidia Product: - Groq develops AI inference technology through its GroqCloud platform enabling rapid execution of AI, machine learning and high-performance computing workloads. Key offerings include APIs for easy integration and real-time analytics capabilities. - The platform features Groq's proprietary LPU™ (Language Processing Unit) designed to accelerate generative AI model inference significantly enhancing speed and reducing energy consumption compared to traditional processors. Company: - Founded in 2016 and headquartered in Mountain View, California, Groq has attracted over $1 billion in funding. Investors include BlackRock and Global Brain Corporation, positioning it as a notable player in the AI and semiconductor sectors. Impact: - Groq's technology enables both small developers and large enterprises to deploy powerful AI applications affordably and efficiently. This shifts the competitive landscape in AI technology by providing low-cost high-speed inference solutions that challenge traditional hardware paradigms. sources: https://lnkd.in/eV8n9yy5 https://groq.com/ #technology #innovation #future #artificialintelligence #business

2

Exploring the Pros and Cons of Multi-token Prediction in Language Models : A recent study, "Better & Faster Large Language Models via Multi-token Prediction," by researchers from FAIR offers a fascinating look at how predicting multiple tokens simultaneously can significantly enhance the efficiency of large language models like GPT and Llama. The researchers of the paper claim that the models developed via the technique discussed in the paper solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models Key Highlights: Enhanced Efficiency: The multi-token prediction method boosts sample efficiency without additional training time or memory overhead, presenting a compelling case for its adoption in larger models. Improved Performance: This technique has shown to dramatically improve performance, particularly in complex coding tasks and natural language processing, consistently outperforming traditional models. Pros of Multi-Token Prediction: Scalability and Versatility: The method scales impressively with larger models and demonstrates benefits across various data types, including both code and natural language. Inference Speed: Models trained with this method can be up to 3× faster at inference, making them not only more intelligent but also quicker. Potential Challenges: Complexity in Training Dynamics: The method involves multiple output heads which could complicate model training dynamics and require careful tuning. Optimal for Large-Scale Models: While the benefits are clear for large models, smaller models may not see as significant improvements, possibly limiting its applicability in resource-constrained projects. Generalization across Tasks: The effectiveness across all types of tasks is not fully explored, which may limit its applicability in certain scenarios. Future Directions: The paper discusses promising future research paths, including automating the selection of the number of tokens to predict and optimizing vocabulary sizes for multi-token prediction. This approach represents an exciting advancement in AI training techniques. However, it also poses new challenges that necessitate further exploration to fully understand and mitigate potential drawbacks. Paper link : https://lnkd.in/d69Je8wU #AI #MachineLearning #LanguageModels #Innovation #Research

14

This quantisation technique will cut inference costs for both enterprise and consumers, and will put larger models within reach of more modest hardware! I'm a big fan of quantisation and the described VPTQ is designed to allow 2-bit quantisation with minimal loss using lookup tables. This will allow you to run large models on undersized hardware. I have previously run 2-bit versions of Llama-3-70b which took up about 23GB VRAM. This technique would double token/s and improve accuracy greatly. I expect to see this integrated more in the near future. Catchy name too. https://lnkd.in/egzxCn_G

4