Aston Zhang

Our paper "DropBP" has been accepted at NeurIPS 2024! A big congratulations to all the authors and collaborators! At NAVER Cloud, we are deeply committed to advancing high-quality AI research, and I am truly honored to be a part of these efforts. Our research introduces a method to drastically reduce training time in fine-tuning large language models. By randomly dropping layers during the computationally heavy backpropagation process, we ensure training stability while achieving even greater efficiency as sequence lengths increase. This novel approach, DropBP, outperforms previous methods by significantly reducing training time without compromising task performance. https://lnkd.in/gnMtK-9W Title: DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation Authors: Sunghyeon Woo (Seoul National University), Baeseong Park (NAVER Cloud), Byeongwook Kim (NAVER Cloud), Minjung Jo (NAVER Cloud), Se Jung Kwon (NAVER Cloud), Dongsuk Jeon (Seoul National University), Dongsoo Lee (NAVER Cloud) (: Work done as an intern) Abstract: Large language models (LLMs) have achieved significant success across various domains. However, training these LLMs typically involves substantial memory and computational costs during both forward and backward propagation. While parameter-efficient fine-tuning (PEFT) considerably reduces the training memory associated with parameters, it does not address the significant computational costs and activation memory. In this paper, we propose Dropping Backward Propagation (DropBP), a novel approach designed to reduce computational costs and activation memory while maintaining accuracy. DropBP randomly drops layers during backward propagation, which is essentially equivalent to training shallow submodules generated by undropped layers and residual connections. Additionally, DropBP calculates the sensitivity of each layer to assign an appropriate drop rate, thereby stabilizing the training process. DropBP is not only applicable to full fine-tuning but can also be orthogonally integrated with all types of PEFT by dropping layers during backward propagation. Specifically, by using DropBP when fine-tuning LLaMA2-70B through QLoRA, we can reduce training time by 44%, accelerate convergence to the same perplexity by 1.5X, and enable training with a sequence length 6.2X larger on a single NVIDIA-A100 80GiB GPU. #NeurIPS2024 #DropBP #NAVERCloud

113

2 Comments

🚀 In collab with Snap Inc., Wei Ting Chen, we’ve brought RobustSAM to Hugging Face! A robust version of SAM, it boosts performance on low-quality images while keeping zero-shot segmentation intact. Even better, it was theoretically possible for SAM to be text-promptable, but now, thanks to 𝐎𝐖𝐋𝐯2, it’s happening! 🦉 RobustSAM can handle text prompts for flexible, accurate segmentation even on degraded images. 🔗 GitHub: https://lnkd.in/daVfg5sa 🤗 Model: https://lnkd.in/duimsJjw 🖥️ Cool Gradio Demo (ZeroGPU): https://lnkd.in/dF5CapzM Try it out—just input an image + comma-separated labels to see the robust results! ⚡ Paper: https://lnkd.in/d99rbciW

151

4 Comments

We just released a blog on whether the use of multimodal feature transfer can detect deception (phenomena like sarcasm, irony, and condescension) in conversation. The short answer... Yes. This particular write-up profiles a subset of our work at Symbl.ai on multimodal large language models, and is research led by one of our summer interns, Benjamin R. (from Prof. Larry Heck's AVA Lab at Georgia Institute of Technology). We have a paper currently under submission on this work, and we can't wait to share the full details with all of you soon. Blog Link: https://lnkd.in/g6m_wFDi #multimodal #featuretransfer #LLMs #deception #conversation #sarcasm #irony #language #AI #ML #NLP #LargeLanguageModels #SummerInternship #ResearchPapers #ACL #Llama2 #SymblDotAI

16

[research update] use a Transformer's multi-head attention activations to fetch RAG style, to augment multi faceted problem breakdown for LLMs in ."Multi-Head RAG: Solving Multi-Aspect Problems with LLMs": 1. Enhances LLMs by integrating document retrieval (RAG) into the LLM context. 2. Challenge: Existing RAG solutions struggle with queries needing multiple documents with varied contents due to distant embeddings. 3. Multi-Head RAG (MRAG): Introduces a novel approach using Transformer's multi-head attention activations to fetch diverse documents. 4. Activation Utilization: Different attention heads capture varied data aspects, improving complex query retrieval. 5. Improved Relevance: MRAG shows up to 20% improvement in relevance over standard RAG baselines. 6. Evaluation Methodology: Provides metrics, synthetic datasets, and real-world use cases to demonstrate effectiveness. 7. Seamless Integration: MRAG can be integrated with existing RAG frameworks and benchmarking tools like RAGAS. 8. Multi-Aspect Problem Solving: Designed specifically to handle multi-aspect queries more effectively. 9. Data Stores Compatibility: Compatible with various data store classes. 10. Potential Applications: Benefits scenarios requiring diverse information retrieval, enhancing LLM responses. Experiment The paper introduces MRAG and evaluates it using synthetic datasets and real-world data. It compares MRAG's performance against standard RAG models. The experiment focuses on the model's ability to retrieve relevant documents across various query aspects, demonstrating up to a 20% improvement in relevance over baselines. Use Case MRAG is tested on real-world scenarios like customer support, where queries often require information from multiple documents with diverse contents. The model's capability to retrieve and integrate multi-aspect data enhances the quality of responses.

81

5 Comments

Collaborative AI is gaining momentum at both MNC and Startups, they are investing into this EvolutionNext Robotics AI Transformative Experiential Learning Journey. I am extremely glad to put Singapore on the World map, by anchoring Robotics AI companies here, creating Jobs of the FutureNow as Roboticist. Do support my Leadership Initiative for Singapore as National Interest; help me enable our next-generation Singaporeans to take on Science, Technology, Engineering and Mathematics STEM Education and longer-term Career Practice into Industry 5.0 as Machine Intelligence of Human-Robot Collaborative, Human-centric Autonomy. * My STEM OneCareer Pinnacle and Finale as a 55-years old, always just another very ordinary Singapore-born Singaporean Scientist Engineer in Singapore, essentially a Nobody here 🌸

13

2 Comments

Open-source AI for evaluating language models Researchers from KAIST AI, LG AI Research, CMU, MIT, Allen Institute for AI, and UIC have introduced Prometheus 2, a groundbreaking open-source evaluator model that assesses language model outputs with accuracy rivaling expensive proprietary solutions like GPT-4. A few key highlights: Prometheus 2 combines models trained on direct assessment and pairwise ranking, enabling it to excel at both common evaluation formats. It leverages the new Preference Collection dataset with 1,000 evaluation criteria to boost performance. Prometheus 2 achieved the highest correlation with human judgments and proprietary models across 8 benchmarks, in some cases reaching 0.90 Pearson correlation and 85%+ accuracy. This helps narrow the gap between open-source and closed-source AI evaluation without sacrificing transparency and control As an AI and Data Strategist, I like that this will make high-quality AI evaluation more accessible and scalable for researchers and practitioners. Some potential strategic impacts: Accelerating AI/ML system development by enabling rapid, reliable evaluation of generated language outputs at lower cost Establishing clearer standards and benchmarks for assessing language model performance as open alternatives mature #ai #machinelearning #nlp #opensource #research #aistrategy Paper - https://lnkd.in/gsnGDwJd

1

Just Arrived in Vienna for #ICML2024, I will be presenting our work on improving GNNs' generalization ability with ensemble learning (https://lnkd.in/gimXe5fS) at Oral 5A (Thu 25 Jul 10:45 a.m. — 11 a.m.) and Poster 2407 (Thu 25 Jul 11:30 a.m. — 1 p.m.). Paper title: From Coarse to Fine: Enable Comprehensive Graph Self-supervised Learning with Multi-granular Semantic Ensemble. A heartfelt thank you to all my collaborators for their incredible support; this work would not have been possible without them. I am also looking forward to reconnecting with old friends and making new ones. If you are attending ICML or are interested in my research, please join me during my presentation session. Additionally, please feel free to reach out to exchange ideas if you are interested in topics such as (1) improving graph model generalization across different datasets, and (2) foundational models for time series forecasting. See you in Vienna! 🇦🇹 #ICML2024 #MachineLearning #ArtificialIntelligence

14

💥💥💥 LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, Jie Tang Abstract Long-Context Question Answering (LCQA), a challenging task, aims to reason over long-context documents to yield accurate answers to questions. Existing long-context Large Language Models (LLMs) for LCQA often struggle with the "lost in the middle" issue. Retrieval-Augmented Generation (RAG) mitigates this issue by providing external factual evidence. However, its chunking strategy disrupts the global long-context information, and its low-quality retrieval in long contexts hinders LLMs from identifying effective factual details due to substantial noise. To this end, we propose LongRAG, a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to enhance RAG's understanding of complex long-context knowledge (i.e., global information and factual details). We design LongRAG as a plug-and-play paradigm, facilitating adaptation to various domains and LLMs. Extensive experiments on three multi-hop datasets demonstrate that LongRAG significantly outperforms long-context LLMs (up by 6.94%), advanced RAG (up by 6.16%), and Vanilla RAG (up by 17.25%). Furthermore, we conduct quantitative ablation studies and multi-dimensional analyses, highlighting the effectiveness of the system's components and fine-tuning strategies. 👉 https://lnkd.in/d7PaajBQ #machinelearning

1

A Survey of Generative Search and Recommendation in the Era of Large Language Models Yongqi Li, Xinyu Lin, Wenjie Wang, Fuli Feng, Liang Pang, Wenjie Li, Liqiang Nie, Xiangnan He, Tat-Seng Chua Abstract With the information explosion on the Web, search and recommendation are foundational infrastructures to satisfying users' information needs. As the two sides of the same coin, both revolve around the same core research problem, matching queries with documents or users with items. In the recent few decades, search and recommendation have experienced synchronous technological paradigm shifts, including machine learning-based and deep learning-based paradigms. Recently, the superintelligent generative large language models have sparked a new paradigm in search and recommendation, i.e., generative search (retrieval) and recommendation, which aims to address the matching problem in a generative manner. In this paper, we provide a comprehensive survey of the emerging paradigm in information systems and summarize the developments in generative search and recommendation from a unified perspective. Rather than simply categorizing existing works, we abstract a unified framework for the generative paradigm and break down the existing works into different stages within this framework to highlight the strengths and weaknesses. And then, we distinguish generative search and recommendation with their unique challenges, identify open problems and future directions, and envision the next information-seeking paradigm. 👉 https://lnkd.in/dPCmCG-t #machinelearning

1

Some great work led by Cheng-Yu with collaborators from Google and others on positional attention biases in LLMs & how to correct them: https://lnkd.in/gFdy8Dpu Near-term takeaways: - Prompting still finicky! The order of information in the prompt matters a lot. - Tuning RAG systems/re-rankers critical, to optimize this ordering - Long context models (LCMs) still have challenges to overcome (some via the methods Cheng-Yu proposes here!)

57

1 Comment

Excited to have presented our paper on Bayesian Prompt Ensembles at ACL 2024. In this paper, we explore the use of multiple prompt re-formulations to capture model uncertainty in LLMs. We derive the connection between prompt re-formulation and model uncertainty/Bayesian NNs and propose a method to learn the relative importance of different prompts for best performance. As part of this work, we open-sourced a code package and tutorial with which you can a) easily use LLMs to perform classification tasks and b) apply our method, BayesPE, to improve uncertainty estimation through the use of multiple prompts. Hope it is useful to folks out there! Paper: https://lnkd.in/etpRRwJC Code: https://lnkd.in/eDHdQ6k4

26

Unlock the full potential of your Large Language Models (LLMs) with Intel® Extension for PyTorch (IPEX) and the Intel® LLM Library for PyTorch (IPEX-LLM). Download this whitepaper to explore the optimization of LLMs, including their performance, resource utilization, and response times in real-world applications. Link - https://intel.ly/3BBJ4ey

1

🚀 LLM workflows for product managers 🚀 At ProductTank San Francisco, Nilou Salehi, Assistant Professor at UC Berkeley, dived into the world of LLM workflows and explored practical solutions for product managers. Read 3 key takeaways or watch the talk. #ProductManagement #AI #LLM #ProductTankSF https://bit.ly/3TziFo7

8

From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, Huaming Chen University of Sydney 2024 https://lnkd.in/gVZ3r8G7 "As LLMs have demonstrated significant capabilities in tasks such as code generation and vulnerability detection, researchers are increasingly interested in their potential applications. However, the study highlights the limitations of LLMs, including their lack of autonomy and self-improvement, which LLM-based agents aim to address. These agents represent a promising advancement towards achieving Artificial General Intelligence (AGI) by integrating LLMs for decision-making and action-taking. Despite the growing body of research on LLMs in software engineering, there remains a lack of clarity distinguishing between LLMs and LLM-based agents, as well as a need for standardized benchmarks to evaluate these technologies. The survey systematically investigates current practices and solutions related to both LLMs and LLM-based agents, focusing on six key areas: requirement engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance. By reviewing and contrasting the contributions of LLMs and LLM-based agents across these topics, the survey provides insights into their respective tasks, benchmarks, and evaluation metrics. It also discusses the models and benchmarks utilized in this domain, offering a comprehensive analysis of their effectiveness and applications in software engineering. Ultimately, the authors aim to illuminate the path forward for LLM-based agents in software engineering, paving the way for future research and development in this promising area."

2