Rui (Ray) P.

I'm excited to share two papers from the Retrieval team at Walmart that were presented at CIKM last week. They're both about improving the relevance of Embedding-Based Retrieval. 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗙𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴 𝗳𝗼𝗿 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹. https://lnkd.in/gNQe7QhA The first is about how to intelligently choose a query-dependent cutoff on the cosine score to remove irrelevant items from the retrieved set. Since cosine scores aren't comparable across queries, we introduce a "Cosine Adapter" to map the scores to interpretable scores. 𝗘𝗻𝗵𝗮𝗻𝗰𝗶𝗻𝗴 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗼𝗳 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗮𝘁 𝗪𝗮𝗹𝗺𝗮𝗿𝘁. https://lnkd.in/gdqkc2hR The second is about model training techniques we learned to improve the relevance of embedding models. In particular, we describe how to use a teacher model to provide relevance labels for training Juexin (Joy) Lin Nicholas Rossi, PhD Feng LIU Zhen Yang Sachin Yadav Praveen Reddy Suram Satya Chembolu Prijith Chandran Hrushikesh Mohapatra Alessandro Magnani Ciya Liao

140

1 Comment

I have been always fascinated by optimization algorithms used in ML and Deep Learning. From the simplicity of SGD, momentum etc. to more involved ones such as conjugate gradient, adagrad, adam etc. Among all of these I found one very interesting algorithm L-BFGS. Truth is L-BFGS is not actually a different algorithm but it is an optimization technique to 2nd order Newton approach. In fact L-BFGS has been widely applied across multiple different problems such as conditional random fields, language translation, speech recognition, LLM + Transformers, dynamic pricing and other optimization problems. I have been using it regularly in my work at Microsoft for dynamic pricing. Initially, found it very hard to understand L-BFGS basics from resources over the internet. In these 2 medium posts, I write about the basics of L-BFGS, implement from scratch in Python and improve it using automatic differentiation (again written from scratch). Hope people in ML and AI will find it useful. https://lnkd.in/g7ybeD3N https://lnkd.in/gCHWssv2 #machinelearning #artificialintelligence #optimization #dataengineering #algorithms

9

3 Comments

Hi LinkedIn Community,   Short version:  Grad student building low-code tool for ML data pipelines. Goal: Make ML dev work enjoyable again, not just endless debugging. Ask: Shadow sessions (see Calendly link below) Longer version: As ML devs, we often leave the office thinking, “Do I actually love doing my work?” because we had spent most of the day on inefficient whack-a-mole debugging and maintenance work. I want to get us back to where we can love coding for those reasons we originally chose to pick up coding in the first place.   A lot of AI tools are restrictive and end up mangling codebases, and I believe the current state of AI coupled with strong design principles can enable much better code native developer experiences. I’ve gone through prototypes/testing and have some competing hypotheses. I’d love to observe coders working on ML data pipelines (happy to sign NDAs if necessary, can also be personal projects instead). If you're open to sharing your expertise, please schedule a shadow session: https://lnkd.in/eBW6p6qE Let's make ML data development more efficient and actually enjoyable! #MachineLearning #DataPipelines #LowCode #DevTools #AI #SoftwareEngineering #GradStudent #Research #MLOps #DataScience #ArtificialIntelligence #ProductivityTools #DeveloperExperience #CodeQuality #TechInnovation #ComputerScience #DataEngineering #CloudComputing

84

8 Comments

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

More opportunities

12

🎉 Big congratulations to our incredible team (Haoyu Chen, Nick Greenquist,Yuxing Wang, Sridevi Maharaj, Ruiyang Wu and many more) for delivering several quarters-long impactful features on the Play Store! 🚀 We’re thrilled to have launched key updates like Search Home, Sports Moment, and WearOS recommendations, all designed to enhance personalization and help billions of users worldwide search and discover content they love. 🌍✨ Check out the Play Store Q3 updates: https://lnkd.in/dn5wx8Mi #GooglePlay #Personalization #Recommendation

72

1 Comment

I just published Vol. 59 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Unifying Context Ranking and Generation in Retrieval-Augmented LLMs, from NVIDIA 📚 A Systematic Approach to Identifying Best Practices in Retrieval-Augmented Generation, from Fudan University 📚 Exploring the Mutual Benefits of LLMs and Search Engine Integration, from IEEE Members 📚 Rethinking Ranking Algorithms for Map Interfaces in Airbnb Search, from Airbnb 📚 Reassessing Context Relevance in LLM Question Answering, from USC 📚 A Benchmark for Evaluating Long-Context Language Models and RAG Systems, from Salesforce AI Research 📚 A New Taxonomy for Long-Context Language Model Evaluation, from Bar-Ilan University 📚 A Multi-Stage Content-Centric Framework for Advancing Retrieval-Augmented Generation, from Tsinghua University 📚 Efficient Document Expansion for E-commerce Search, from Walmart 📚 A Comparative Study of RAG vs. Fine-Tuning in API-Driven Code Generation, from Microsoft #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

67

2 Comments

Just did a keynote in PyTorch conference, on LLM deployment. Thanks for the great work of PyTorch team, our partners, and the interests from the audience! Please try out our deployment ecosystem of torchchat, torch.compile, torch.export, AOT Inductor, torchao and ExecuTorch, and checkout all the amazing talks, posters and demos in the conference!

50

3 Comments

I just published Vol. 52 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Comprehensive Survey of Retrieval-Augmented Large Language Models: Architectures, Training, and Applications, from PolyU 📚 An Analysis Framework for Benchmarking Retrieval-Augmented Generation Systems, from Tu et al. 📚 Distilling Cross-Encoders from Large Language Models for Efficient Passage Re-ranking, from Schlatt et al. 📚 A Comprehensive Benchmark for Assessing Retrieval-Augmented Language Models in Biomedicine, from UMTC 📚 Improving Factual Accuracy of Large Language Models for Healthcare using Targeted Knowledge Graph Retrieval, from Huawei 📚 Identifying Shopping Product Questions in Voice Assistants via Graph Attention Networks, from Amazon 📚 A Learnable Tokenizer for Generative Recommendations, from NUS 📚 An Enhanced Retrieval-Augmented Generation Framework for Accurate, Efficient, and Personalized Language Models, from UTS 📚 Constructing Synthetic Test Collections Using Large Language Models for IR Evaluation, from Rahmani et al. 📚 Enhancing Sequential Recommendation with Soft Contrastive Learning, from Renmin University #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

70

2 Comments

I still remember the early days of my career as a Machine Learning engineer, navigating various sites looking for data -- Github, Kaggle, UCI, you name it. One particular project comes to mind—a predictive analytics tool for early diagnosis of diabetic retinopathy. Despite the importance and potential impact of the project, we hit a major roadblock: insufficient data. The patient data we had was not only limited but also fraught with privacy concerns. This challenge was an eye-opener, underscoring the critical need for synthetic data in healthcare. Now, that I see multiple services are offering synthetic data. Synthetic data is artificially generated rather than obtained by direct measurement. It mirrors the statistical properties of real-world data, making it a powerful tool in scenarios where data is scarce, sensitive, or difficult to collect. Today we have LLMs, especially GPT-4 that can generate domain-specific (textual) data. Many LLMs are even finetuned and benchmarked using synthetic data generated from GPT-4. But, how can you start and what are some of the best practices? Here are some of the key points that you must consider: 1. Define Clear Objectives: Understand the purpose of generating synthetic data. Whether it's for augmenting training datasets, finetuning like I mentioned preserving privacy, or creating data for underrepresented scenarios, clarity in objectives is crucial. 2. Choose the Right Model: Today, there are various LLMs, according to your budget choose them wisely. GPT-4, Claude, Mistral, LLaMA, etc all good but some are expensive and others are free. If you are low on budget and low on requirements then an open-source model with free Kaggle or Colab support will do. But if you are required to generate a huge amount of data then you might need to spend a little on storage. Keep in mind that running open-source LLMs might require you to spend money because they are very large and require extra storage and computing power. 3. Ensure Data Quality: This is where your model will be tested. Synthetic data should mirror the statistical properties of real-world data. Regular validation against real data ensures the synthetic data’s utility and realism. 4. Maintain Privacy: One of the primary benefits of synthetic data is enhanced privacy. Ensure that the generation process and the synthetic data itself do not inadvertently expose sensitive information. In any case research thoroughly and choose the model according to your requirements. I have attached a code snippet that might help you understand how to generate synthetic and also how to design a productive prompt. The link to the original code can be found in the comment section.

19

7 Comments

Calling all IT leaders, developers, and practitioners! Ready to explore the future of data in the age of GenAI? Join Google Cloud for Database Day: Sunnyvale on August 27th! **🔥 Here's what you can expect: ** Two tracks packed with insights: one for IT Leaders & Executives, and another for Developers & Database Practitioners. Expert-led sessions on topics like building GenAI apps with Google Cloud databases, maximizing PostgreSQL ROI, and mastering Cloud SQL. Hands-on labs and demos to level up your skills and connect with peers. Don't miss this chance to build your GenAI future. Register now - spots are filling up fast! #GoogleCloud #DatabaseDays #GenAI #DataInnovation

1

I just published Vol. 65 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction, from Sarmah et al. 📚 A Method for Enhancing Precision in Dense Retrieval for E-commerce Search, from Walmart 📚 Enhancing E-commerce Search with LLM-Powered Attribute Extraction, from Amazon 📚 Enhancing Ad Retrieval with Joint Optimization of Hierarchical Clustering and Neural Networks, from Meta 📚 Integrating Web Search and Knowledge Graphs in Retrieval-Augmented Generation, from Xie et al. 📚 Enhancing BM25 with Entropy-Weighted Similarity and Semantic Understanding, from Li et al. 📚 A Novel GNN Framework for Multimodal and Multilingual Product Recommendations in E-commerce, from Amazon 📚 Human-Guided Optimization of Embedding-Based Retrieval for E-commerce, from Walmart 📚 A Comprehensive Guide to Similarity Measures in Data Science, from Levy et al. 📚 Evaluating Mamba Architecture for Dense Retrieval Tasks, from Renmin University #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

74

2 Comments

Congrats the team on ICML acceptance of the Wukong paper: https://lnkd.in/g-MyahNH, which represents a breakthrough of recommendation model design and efficiency. Recommendation models traditionally suffer from not able to utilize the full computational power of GPU (only 5-10% of peak GPU FLOPS while LLMs can reach >50%), due to their sparsity. Wukong is first of its kind by pushing toward 40% GPU peak FLOPS though model-infra codesign. One of my teams consisting of GPU experts is building a high performance model component libary where we’ll provide highly optimized model components (eg sequence summarization, x-sequence interaction, x-GPU layout scheme) that could be composed - with the goal that models built from this library have high performance out of box to scale the model-infra codesign effort. We are also scaling training beyond the limits that are typical of recommendation models - from hundreds of GPUs to 10s of thousands. If this is the kind of work that excites you, come join us!

264

3 Comments

I just published Vol. 67 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Comprehensive Study of Noise Types in RAG, from Wu et al. 📚 Lessons from Adobe Express’s Iterative Enhancement of Multi-Modal Search, from Adobe 📚 Dynamic Hard Negative Mining for Enhanced Text Embeddings, from Tencent 📚 Tackling the Unique Challenges of Location-Based Search, from Airbnb 📚 Jina-ColBERT-v2: Enhancing ColBERT for Multilingual Retrieval, from Jha et al. 📚 Enhancing Music Recommendation through Psychology-Informed Session Embeddings, from Deezer 📚 Analyzing [MASK] Token Behavior in ColBERTv2, from RIT 📚 A Critical Evaluation of Traditional Models vs. LLMs in Sequential Recommendation, from Xu et al. 📚 Collaborative Discourse for Complex Information Discovery, from Jiang et al. 📚 A Large-Scale Synthetic Test Collection for Advancing Passage Retrieval Research, from Rahmani et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

34

1 Comment

Recommendation failures on Amazon (based on a Rice Cooker). Who needs a bundle of two rice cookers? It seems like it's just putting the top two items from a query altogether (or co-click). It is definitely not a norm to have to get two new rice cookers. Of all these, the leftmost one is even the most insensible one. #randomday #recommendations #ml

13

7 Comments

I just published Vol. 74 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Agentic Information Retrieval as the Next Generation of IR Systems, from SJTU 📚 A Novel Approach to Extracting Rich Embeddings from Mixture-of-Experts LLMs, from UMD 📚 Starbucks Method for Efficient and Effective Embedding Model Training, from Zhuang et al. 📚 Enhancing Semi-Structured Retrieval with Knowledge-Aware Query Expansion, from Xia et al. 📚 A Comparative Study of Semantic vs. Fixed-Size Strategies, from Qu et al. 📚 Enhancing Encoder-Only Transformers for Next-Item Prediction in Session-Based Recommenders, from Redjdal et al. 📚 A Framework for Integrating Visual, Textual, and Graph Data in LLM-Based Recommenders, from Walmart 📚 Layer-of-Thoughts Framework for Advanced Information Retrieval, from ROIS-DS 📚 Language Model Agents for a Healthier Digital Public Sphere, from Lazar et al. 📚 Adaptive-Note RAG for Improved Information Integration, from Wang et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

43

1 Comment

I just published Vol. 48 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Systematic Survey of Retrieval-Augmented Generation Techniques, from York University 📚 Navigating the Evaluation Landscape of Generative Information Retrieval, from Alaofi et al. 📚 Leveraging RAG to Enhance the Trustworthiness of Generative AI for Enterprise Workflow Automation, from ServiceNow 📚 Exploring the Interplay Between Large Language Model Priors and Externally Sourced Content in Retrieval-Augmented Systems, from Stanford University 📚 A Practical Roadmap for Efficient Fine-Tuning on Proprietary Enterprise Data, from HCLTech 📚 Leveraging Look-Alike Users and Recalled Items to Improve Click-Through Rate Prediction, from Huawei 📚 Leveraging Collaborative Knowledge in Large Language Models for Robust and Versatile Recommendation, from KAIST 📚 A Study on How LLM-Generated Content Infiltrates and Reshapes Retrieval-Augmented Generation Systems, from UCAS 📚 Leveraging Rotary Position Embeddings for Extended Context Windows in Text Embedding Models, from Microsoft 📚 Taxonomy and Analysis of Sensitive User Queries in Generative AI Search, from Naver #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

71

3 Comments

I just published Vol. 60 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Enhancing Retrieval Augmented Generation through Parallel Drafting and Verification, from Google 📚 A Framework for Building Secure and Efficient Enterprise Chatbots with RAG, from NVIDIA 📚 Evaluating RAG vs. Long-Context Models for Environmental Policy Document Comprehension, from Phan et al. 📚 Bridging Language Models and Collaborative Filtering for Enhanced Recommendations, from Sheng et al. 📚 Refining Retrieved Content to Boost RAG Effectiveness, from The University of Lisbon 📚 Optimal Methods for Crowd-Sourcing Realistic Multi-modal Dialogue Data in E-commerce, from Amazon 📚 Improving HNSW for Efficient Dynamic Indexing, from UESTC 📚 Balancing Factual Information and Token Efficiency in Black-Box RAG Systems, from Zhejiang University 📚 Enhancing RAG Systems with Unsupervised Document Refinement, from KAIST 📚 An Approach to Multi-Modal Information Retrieval and LLM Integration, from Zhejiang University #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

33

1 Comment