AI Agent for Information Retrieval: Generating and Ranking

Yongfeng Zhang Zhiwei Liu Qingsong Wen

Rutgers University Salesforce AI Research Squirrel Ai Learning
New Brunswick, New Jersey, USA Palo Alto, California, USA Seattle, Washington , USA
[email protected] [email protected] [email protected]

Linsey Pang Wei Liu Philip S. Yu

Salesforce University of Technology Sydney University of Illinois at Chicago
San Francisco, California, USA Sydney, NSW, Australia Chicago, Illinois, USA
[email protected] [email protected] [email protected]

Abstract In Proceedings of the 33rd ACM International Conference on Information and

The field of information retrieval has significantly transformed with Knowledge Management (CIKM ’24), October 21–25, 2024, Boise, ID, USA.
ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3627673.3680120
the integration of AI technologies. AI agents, especially those lever-
aging LLMs and vast computational power, have revolutionized in-
formation retrieval, processing, and presentation. LLM agents, with
advanced memory, reasoning, and planning capabilities, can per- 1 Workshop Themes and Topics
form complex tasks, engage in coherent conversations, and provide The advent of Large Language Model (LLM) Agents marks a sig-
personalized responses. Despite these advancements, challenges nificant milestone in AI and NLP. LLM Agents use large language
such as ensuring relevance and accuracy, mitigating biases, pro- models to perform tasks beyond text generation, such as conducting
viding real-time responses, and maintaining data security remain. conversations, completing tasks, reasoning, planning, and demon-
This workshop aims to explore these challenges, share innovative strating some degree of autonomous behavior. According to a recent
solutions, and discuss future directions. It will provide a platform to market report [1], the LLM Agent market is estimated at USD 4.35
bring together researchers, practitioners to discuss the latest theo- billion in 2023, with projections reaching USD 36.1 billion by 2030.
retical advancements and practical implementations of AI agents in The workshop aims to bridge the gap between information retrieval
information retrieval. Topics include AI in search, recommendation, and AI technologies. It will explore the potential of generative mod-
and personalization systems. By gathering a diverse group of ex- els and AI-generated content to diversify available information and
perts, the workshop seeks to deepen the understanding of AI agents meet varied user preferences and needs. A key focus will be on the
in information retrieval, advance the field, and enhance its societal transformative potential of AI agents in reshaping user interactions
impact. Participants will gain insights into cutting-edge research, with information retrieval systems, leveraging advancements in
emerging trends, and foster knowledge exchange and collaboration LLMs. The workshop will also address critical issues of trust and
within the community. ethics in AI-driven information retrieval. Topic of Interests include
but not limited to:
CCS Concepts
• Agentic Retrieval System
• Information systems → Information retrieval query pro- • Agentic Conversational Recommendation
cessing; Retrieval models and ranking; Search engine archi- • AI Agent for Personalization
tectures and scalability; Users and interactive retrieval; Evalua- • AI Agent for Sequential Recommendation
tion of retrieval results; Document representation; Special- • AI Agent for Contextual Information Retrieval
ized information retrieval; Information retrieval query pro- • AI Agents for Cross-lingual and Multimodal Retrieval
cessing; Retrieval models and ranking; Search engine archi- • Retrieval-Augmented Generation Searching System
tectures and scalability; Users and interactive retrieval. • Optimization of AI Agent Retrieval Models
• Bias Mitigation in AI-driven Information Retrieval
Keywords • Interpretable Generative Retrieval System
Information Retrieval; AI agent; Large Language models (LLMs) ; • Adversarial Attack and Defense in Agentic Retrieval System
Recommender Systems • Ethical Considerations in Agentic Information Retrieval
ACM Reference Format: • Human-AI Collaboration in Information Retrieval
Yongfeng Zhang, Zhiwei Liu, Qingsong Wen, Linsey Pang, Wei Liu, and Philip • Evaluation for Agentic Information Retrieval
S. Yu. 2024. AI Agent for Information Retrieval: Generating and Ranking. • Scalability and Efficiency in AI Agent Retrieval

This work is licensed under a Creative Commons Attribution

International 4.0 License. 2 Motivation, Goal and expected outcomes
Although research on LLM agent for information retrieval has pro-
CIKM ’24, October 21–25, 2024, Boise, ID, USA gressed significantly, more challenges still need to be addressed. For
© 2024 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0436-9/24/10 instance, one of the key issues is enhancing the relevance and ac-
https://doi.org/10.1145/3627673.3680120 curacy of retrieved information. Despite significant advancements,

5605
CIKM ’24, October 21–25, 2024, Boise, ID, USA Yongfeng Zhang et al.

ensuring that AI agents can consistently provide precise and con- 5 Past Workshops
textually appropriate results remains a challenge. This workshop Relevant workshops run by some of us were held at KDD 2023-2021,
address those critical challenges, share innovative solutions, and WSDM 2023-2022, and CIKM 2022.
explore future directions in the field. Our goal is to create a collabo-
rative platform for researchers, practitioners, and industry experts • Foundation and Applications in Large-scale AI models work-
to discuss the latest theoretical advancements and practical ap- shop was held at KDD 2023. It was one of the popular among
plications of AI agents in information retrieval. The workshop is all the workshops, with more than 100 attendees on-site.
expected to attract submissions from both academia and industry, We had 4 highly influential keynote speakers from Google,
encompassing various disciplines such as data science, machine Salesforce and IBM. We had around 30 submissions and 9
learning, and etc. Participants will gain insights into cutting-edge papers were accepted for presentation.
research, emerging trends, and innovative solutions, fostering a • Deep Learning for Search Recommendation workshop is
vibrant community dedicated to advancing AI-driven information a full-day workshop in conjunction with CIKM 2022. The
retrieval. workshop invited 5 keynote speakers from Google, Amazon
Encourage physical attendance and in-person activities: etc. There are 27 submissions, among which 12 papers were
accepted.
• The 1st, 2nd and 3rd International Workshop on Interactive
• Invite high-profile and famous senior academics or industry and scalable information retrieval methods for e-commerce
leaders to be keynote speakers was held in conjunction with WSDM 2022, WWW 2023 and
• Require all accepted workshop papers to be presented in the WSDM 2024. Invited speakers and submissions were from
workshop in person academia and industry, such as Nvidia, Amazon Search.
• Set up a Best Paper Award in the workshop and announce • The 2nd International Workshop on Industrial Recommen-
the winner in the workshop. dation Systems was held as a full-day workshop in con-
junction with KDD 2021. There were 23 paper submissions,
among which 11 papers were accepted. The workshop in-
3 Target Audience vited keynote speakers from Amazon, Netflix, Spotify, Nvidia.
The rapid advancements in AI-driven information retrieval, par-
ticularly with AI agents and large language models (LLMs), have 6 Organizer Profiles
captured significant interest from both research and industry com- • A/Prof. Yongfeng Zhang is an Associate Professor in the
munities. Addressing challenges such as improving information Department of Computer Science at Rutgers University. His
relevance and accuracy, managing vast data volumes, and ensuring research interest is in Machine Learning and Data Mining,
real-time processing is crucial. This workshop aims to attract re- Information Retrieval and Recommender Systems, Natural
searchers and practitioners from academia and industry interested Language Processing, Trustworthy AI and AI for Science. He
in the latest AI trends in information retrieval. With a focus on regularly serves as Senior PC Member or Area Chair for SI-
generative AI agents in search and recommendation, the workshop GIR, WWW, KDD, ICLR, RecSys, ACL, CIKM, WSDM, AAAI,
expects high participation, with 50 to 80 attendees. It aims to facili- etc. He is a Distinguished Editor for ACM Transactions on
tate knowledge exchange and foster collaboration among experts Information Systems (TOIS), and also serves as Associate Ed-
dedicated to advancing AI-driven information retrieval technolo- itor for ACM Transactions on Recommender Systems (TORS)
gies. Participants will benefit from discussions on cutting-edge as well as Frontiers in Big Data. His research won the 2024
research, innovative applications, and practical challenges in de- ACM SIGIR Test of Time Award. He is a Siebel Scholar of
ploying AI agents for information retrieval, including areas like the class 2015 and an NSF career awardee in 2021. He has
personalization, cross-lingual retrieval, and interpretability. extensive experience in organizing workshops on top-tier
conferences such as the 1st to 3rd Workshop on Explainable
Recommendation and Search at SIGIR, and Workshop on
4 Workshop Relevance Machine Reasoning at WSDM.
CIKM is a premier conference for Information and Knowledge Man- • Dr. Zhiwei Liu Dr. Zhiwei Liu is a senior research scientist
agement research. This workshop aligns perfectly with CIKM’s at Salesforce AI Research. He received his Ph.D. degree from
mission. As information and knowledge management evolve, in- University of Illinois at Chicago. Dr. Liu’s research interests
tegrating AI agents into information retrieval systems becomes lie in AI agents, Large Models, recommender systems, and
crucial. The workshop addresses fundamental challenges and ex- data mining. He serves as the Guest Editor for ACM Trans-
plores innovative solutions at the intersection of AI technologies actions on Recommender System (TORS) and PC members
and information retrieval. Focusing on generating and ranking for CIKM, KDD, WWW, SIGIR, WSDM, AAAI, RecSys, ICLR,
information, the workshop delves into theoretical advancements ACL, EMNLP, NeurIPS, and etc. He has published over 40
and practical applications of AI agents, including large language top-ranking AI conference and journal papers, including
models (LLMs). By emphasizing AI-driven approaches and Agentic CIKM, SIGIR, WWW, ICLR, EMNLP, WSDM, RecSys, ACM
Systems, the workshop contributes to shaping the future landscape TIST.
of information retrieval, making it a valuable addition to the CIKM • Dr. Qingsong Wen is the Head of AI Research & Chief Sci-
conference. entist at Squirrel Ai Learning Inc. Before that, he worked at

5606
AI Agent for Information Retrieval: Generating and Ranking CIKM ’24, October 21–25, 2024, Boise, ID, USA

Alibaba, Qualcomm, and Marvell, and received his Ph.D. de- lished over 100 papers in CORE A*/A and Q1 (i.e., top-prestigious)
grees in Electrical and Computer Engineering from Georgia journals and conferences. He has received 3 Best Paper
Institute of Technology, USA. His research interests include Awards. In addition, one of his first-authored papers received
machine learning and decision intelligence, especially AI for the Most Influential Paper Award in conference PAKDD
Time Series (AI4TS) & AI for Education (AI4EDU). He has 2021. Wei served as a tutorial chair at ICDM 2021. He is
published over 100 top-ranked AI conference and journal a co-organizer of the NeuroRec 2021 workshop collocated
papers, had multiple Oral/Spotlight Papers at NeurIPS, ICML, with ICDM 2021, the AdKDD 2022 workshop collocated with
and ICLR, had multiple Most Influential Papers at IJCAI, re- KDD 2022, and the DL4SR 2022 workshop collocated with
ceived multiple IAAI Deployed Application Awards at AAAI, CIKM 2022.
and won First Place of SP Grand Challenge at ICASSP. Cur- • Prof. Philip S. Yu is a Distinguished Professor in Computer
rently, he serves as Organizer/Co-Chair of Workshop on AI Science at the University of Illinois at Chicago and also holds
for Time Series (AI4TS @ KDD, ICDM, SDM, AAAI, IJCAI) the Wexler Chair in Information Technology. Before joining
and Workshop on AI for Education (AI4EDU @ CAI). He UIC, Dr. Yu was with IBM at the Watson Research Center.
also serves as Associate Editor for Neurocomputing, Asso- His research interest is on big data, and artificial intelligence.
ciate Editor for IEEE Signal Processing Letters, Guest Editor He has published more than 1,600 papers in refereed journals
for Applied Energy, and Guest Editor for IEEE Internet of and conferences. He holds or has applied for more than 300
Things Journal. In addition, he has regularly served as Area US patents. Dr. Yu is a Fellow of the ACM and the IEEE. He
Chair/PC of the AI conferences including NeurIPS, KDD, received the VLDB 2022 Test of Time Award, ACM SIGSPA-
AAAI, IJCAI, ICDM, ICASSP, etc. TIAL 2021 10-year Impact Award, WSDM 2020 Honorable
• Dr. Linsey Pang is a Principal Machine Learning Scientist Mentions of the Test of Time Award, ICDM 2013 10-year
at Salesforce, working on deep learning recommendation Highest-Impact Paper Award, and the EDBT Test of Time
modeling projects. Prior to this, she was a Principal Data Award (2014). He was the Editor-in-Chiefs of ACM Transac-
Scientist at Walmart Lab. She was leading the Pricing, One tions on Knowledge Discovery from Data and IEEE Transac-
demand data science projects. Prior to joining Walmart Lab, tions on Knowledge and Data Engineering. Dr. Yu is also the
she worked as an applied scientist at eBay Inc. She finished recipient of ACM SIGKDD 2016 Innovation Award for his
her PhD degree in machine learning research at the Uni- influential research and scientific contributions on mining,
versity of Sydney. She co-organizes multiple workshops at fusion and anonymization of big data, the IEEE Computer
CIKM, KDD, and WSDM. She also serves as SPC and PC for Society’s 2013 Technical Achievement Award for pioneering
multiple top conferences. and fundamentally innovative contributions to the scalable
• A/Prof. Wei Liu is an Associate Professor in Machine Learn- indexing, querying, searching, mining and anonymization of
ing, and the Director of Future Intelligence Research Lab, in big data, and the Research Contributions Award from IEEE
the School of Computer Science, the University of Technol- Intl. Conference on Data Mining (ICDM) in 2003 for his
ogy Sydney (UTS). He obtained his PhD degree in Machine pioneering contributions to the field of data mining.
Learning research at the University of Sydney. His current
research focuses are adversarial machine learning, cyberse- References
curity, game theory, multimodal machine learning, natural [1] M. Kumarashanmugam. 2024. The Current State, Progress, and Future of LLM
Agents. https://www.linkedin.com/pulse/current-state-progress-markets-future-
language processing, and intrusion detection. Wei has pub- llm-agents-kumarashanmugam-mj8wf/ Accessed: 2024-06-09.

5607