Stars
Benchmarks, environments, and toolkits for general computer agents
m&ms: A Benchmark to Evaluate Tool-Use for multi-step multi-modal tasks
[ACL 2024] ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages
MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning
A language model (LM)-based emulation framework for identifying the risks of LM agents with tool use
ToolQA, a new dataset to evaluate the capabilities of LLMs in answering challenging questions with external tools. It offers two levels (easy/hard) across eight real-life scenarios.
RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning
The source code and dataset mentioned in the paper Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark.
Companion code to https://arxiv.org/abs/2402.15491
ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
An LLM-based autonomous agent controlling real-world applications via RESTful APIs
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
ToolBench, an evaluation suite for LLM tool manipulation capabilities.
Tool Learning for Big Models, Open-Source Solutions of ChatGPT-Plugins
中文大模型能力评测榜单：目前已囊括115个大模型，覆盖chatgpt、gpt4o、百度文心一言、阿里通义千问、讯飞星火、商汤senseChat、minimax等商用模型， 以及百川、qwen2、glm4、yi、书生internLM2、llama3等开源大模型，多维度能力评测。不仅提供能力评分排行榜，也提供所有模型的原始输出结果！
⚡️🧪 Fast LLM Tool Calling Experimentation, big and smol
Evaluation code for various unsupervised automated metrics for Natural Language Generation.
Google Research
SWE-agent takes a GitHub issue and tries to automatically fix it, using GPT-4, or your LM of choice. It solves 12.47% of bugs in the SWE-bench evaluation set and takes just 1 minute to run.
[ICLR 2024] MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
A python module to repair invalid JSON, commonly used to parse the output of LLMs
An opinionated list of awesome Python frameworks, libraries, software and resources.
LangGPT: Empowering everyone to become a prompt expert!🚀 Structured Prompt，Language of GPT, 结构化提示词，结构化Prompt
🔥中文 prompt 精选🔥，ChatGPT 使用指南，提升 ChatGPT 可玩性和可用性！🚀
[ACL 2024] On the Multi-turn Instruction Following for Conversational Web Agents