arXiv CSopen access

Agentic Frameworks for Reasoning Tasks: An Empirical Study

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

softwarearchitecture softwareengineering testing

software engineering, software architecture, testing

Agentic Frameworks for Reasoning Tasks: An Empirical Study Zeeshan Rasheed, Abdul Malik Sami,, Muhammad Waseem, Kai-Kristian Kemell, Mika Saari, Pekka Abrahamsson

arXiv:2604.16646v1 [cs.AI] 17 Apr 2026

Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland.

*Corresponding author(s). E-mail(s): [email protected]; Contributing authors: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Abstract Recent advances in agentic frameworks have enabled the development of AI agents capable of complex reasoning and decision-making. However, systematic evidence comparing their reasoning performance, efficiency, and real-world suitability remains limited, making it difficult to select appropriate frameworks for practical use. To address this gap, we empirically evaluate 22 widely used agentic frameworks across three reasoning benchmarks: BBH, GSM8K, and ARC. We selected these frameworks from 1,200 GitHub repositories collected between January 2023 and July 2025, and developed a taxonomy based on their architectural designs. We then evaluated these frameworks under a unified experimental setting, measuring reasoning accuracy, execution time, computational cost, and the consistency of framework performance across the benchmarks. Our results show that 19 of the 22 frameworks completed all three benchmarks. Among these, 12 frameworks showed consistent performance, with a mean accuracy of 74.6–75.9%, execution time of 4–6 seconds per task, and cost of 0.14–0.18¢ per task. The remaining frameworks performed worse, primarily because of orchestration issues rather than limitations in reasoning. For instance, Camel failed to complete BBH after 11 days of runtime because of uncontrolled context growth; Upsonic consumed $1,434 in a single day because extraction failures led to multiple retries, while uncontrolled context growth sharply increased prompttoken usage; and AutoGen and Mastra exhausted API quotas through iterative agent interactions that increased prompt length without improving the answers. These are system-level failures in memory management, retry policy, and context handling rather than failures in reasoning. We further found that all frameworks,

including the high performers, degrade sharply regarding mathematical reasoning: among the frameworks that completed all benchmarks, mean accuracy on GSM8K was 44.35%, compared with 89.80% on BBH and 89.56% on ARC. This gap is consistent across architectures, reinforcing that current agentic designs inherit rather than overcome the base model’s weaknesses in multi-step numerical computation. Overall, this study provides a systematic comparison and architectural perspective on agentic frameworks to support framework selection for reasoning-intensive software engineering tasks. Our results suggest that such selection should be guided primarily by orchestration quality—particularly memory discipline, failure handling, and cost control—rather than by architectural category or claimed reasoning capabilities alone. We have released the dataset, source code, and per-task logs to support replication and further research. Keywords: Agentic frameworks, AI agents, reasoning benchmarks, large language models, empirical evaluation

1 Introduction In recent years, AI agents have received increasing attention in relation to performing complex tasks that require planning, reasoning, and tool usage [1, 2]. At the same time, recent developments in agentic frameworks have improved agents’ ability to design multi-agent workflows, coordinate specialized roles, and execute complex tasks autonomously [3]. According to Wang et al. [4], agentic frameworks are toolkits or libraries that provide reusable components, along with built-in coordination and memory capabilities, to support the design, deployment, and orchestration of autonomous or semi-autonomous agents. The growing adoption of these frameworks in both academia and industry demonstrates their increasing importance in agent-based system development [5]. Reasoning is a core capability of intelligent agents, enabling them to perform logical inference, solve problems, and make decisions in dynamic and interactive environments [6]. As a result, agentic frameworks have been explored for applications that require advanced reasoning capabilities [5, 7]. However, despite their widespread use, there is still a lack of comprehensive empirical studies that systematically evaluate and compare agentic frameworks in terms of reasoning performance, efficiency, and practical effectiveness in software engineering contexts [4, 6]. Addressing this gap is essential for understanding the strengths and limitations of existing frameworks and supporting more informed framework selection in practice. Motivation: This study is part of the MAISA project (2024–2026) and the ANSE project (2025–2027), both funded by Business Finland. These projects bring together academic researchers and major Finnish companies to explore the use of agentic AI and large language model (LLM)-based technologies to advance software engineering practices. This collaboration focuses on developing, assessing, and facilitating the adoption of innovative AI-driven approaches in real-world software engineering environments. This study aims to address industrial needs by systematically evaluating

the capabilities of agentic frameworks for reasoning tasks and examining their practical trade-offs in software engineering contexts. Recent studies [4, 8] highlight that, despite the rapid emergence of these frameworks, industry practitioners still lack clear guidance on which ones are most suitable for different application contexts, making it difficult to assess trade-offs related to reliability, scalability, and reasoning performance. As several studies [9–13] have conducted benchmark-based empirical studies to evaluate and compare LLM capabilities across various software engineering tasks, the growing practical adoption of LLM-based agents signals the need for more comprehensive benchmarking of agentic frameworks [6]. Although several studies [7, 14–16] have compared such frameworks for different software engineering tasks using specific benchmarks and evaluation metrics, these studies typically consider only a limited number of frameworks or focus on narrow aspects of performance. As a result, there remains a lack of broad empirical evidence that can support framework selection for reasoning-intensive software engineering applications. To this end, this study addresses the needs of industrial practitioners by providing a comprehensive empirical evaluation of agentic frameworks, enabling a structured comparison across key dimensions and supporting informed decision-making for their practical adoption. In this work, we evaluate 22 agentic frameworks (see Table 3), selected through a systematic process, and develop a taxonomy based on their architectural design. We categorize these frameworks into five architectural types: single-agent, role-based multi-agent, hierarchical, modular, and graph-based architectures. We also identify their adopted reasoning strategies and memory mechanisms. These 22 frameworks are empirically assessed in a unified experimental setting to compare their reasoning performance, trade-offs, and consistency across three widely recognized reasoning benchmarks: Big-Bench Hard (BBH) [10], GSM8K [17], and the AI2 Reasoning Challenge (ARC) [18]. These benchmarks were selected because they contain diverse datasets designed to evaluate the complex reasoning capabilities of AI systems. Contributions: The key contributions of this paper are summarized as follows:

• Systematic selection of agentic frameworks and creation of a taxonomy based on their architectural design, providing an overview of the current framework landscape. • Development of a unified experimental setting for comparing the reasoning accuracy, response time, operational cost, and consistency of agentic frameworks across benchmarks, thereby providing insights into the computational trade-offs associated with their deployment as well as their performance consistency. • Empirical insights into the capabilities of agentic frameworks for reasoning-intensive tasks, offering practical guidance for researchers and industry practitioners in realworld applications. • Public release of the dataset and source code to support the replication, validation, and extension of the study. The source code is available via GitHub [19] and the dataset, including the analyzed frameworks, benchmark results, and supporting analysis, is available on Zenodo [20], thereby promoting transparency, reproducibility, and future research on agentic frameworks.

Structure of the paper: The work related to this study is presented in Section 2, followed by the study design in Section 3. The results of the empirical analysis are reported in Section 4. The key findings and their implications are discussed in Section 5, while threats to validity are examined in Section 6. Finally, Section 7 summarizes the paper and outlines directions for future research.

2 Related Work This section first provides an overview of agentic frameworks in Section 2.1, outlining their coordination strategies and workflow management approaches. In Section 2.2 we then examine prior studies that evaluate agentic frameworks across various tasks.

2.1 Agentic Frameworks: Architectures and Capabilities LLMs have delivered strong performance across a broad range of tasks under direct prompting, establishing single-model interaction as an important baseline [21, 22]. However, complex reasoning and software engineering tasks often require iterative planning, task decomposition, tool use, memory, and coordination across multiple steps [23]. Intelligent agents address these challenges by enabling LLMs to operate through structured, goal-directed workflows rather than isolated prompt-response interactions [2]. Prior to the development of agentic frameworks, traditional agent development required the implementation of orchestration logic, state management, communication protocols, and execution pipelines from scratch, which demanded high levels of technical expertise [24]. Moreover, these approaches offered limited support for external tool integration and persistent memory, thereby constraining the development of scalable and context-aware intelligent agents [25, 26]. Agentic frameworks were introduced to address these limitations by providing reusable abstractions and built-in support for agent orchestration, memory management, tool integration, and multi-step execution, thereby simplifying the development and deployment of intelligent agent systems [27]. Since 2023, the development of agentic frameworks such as AutoGen [28], Camel [29], CrewAI [30], SuperAGI [31], TaskWeaver [32], MetaGPT [33], and ChatDev [34] has simplified the development of intelligent agent systems. These frameworks support complex functionalities, including state management, tool integration, and inter-agent communication [3]. By providing structured APIs for defining agent roles, behaviors, and collaborative workflows, they enable more efficient development of advanced single-agent and multi-agent applications [35]. These frameworks organize agentic systems in different ways to support collaboration and task execution. In particular, they adopt a variety of architectural designs for agent workflows, including single-agent architectures, role-based multi-agent architectures, hierarchical architectures, modular architectures, and graph-based architectures [36], [32], [3]. Single-agent architectures rely on a centralized agent responsible for planning, reasoning, and executing tasks [32]. While this approach simplifies coordination, it may face limitations when handling complex and large-scale tasks. Frameworks such as TaskWeaver [32], AutoGPT [37], LangChain [38], and LlamaIndex [39] adopt this architectural design [32]. In addition, frameworks such as LangChain and LlamaIndex further extend agentic systems by supporting interaction with external environments, 4

large document collections, and complex execution flows. These frameworks provide abstractions for chaining reasoning steps, integrating external knowledge through retrieval-augmented generation, and managing agent memory [40], [4]. In contrast, role-based multi-agent architectures structure collaboration by assigning specialized roles and responsibilities to different agents, enabling coordinated problem-solving through defined interaction protocols [41]. This approach is followed by frameworks such as AutoGen [28], Camel [29], CrewAI [30], and SuperAGI [31]. These frameworks often incorporate explicit memory management mechanisms—such as short-term conversational memory, shared context buffers, and persistent storage—to maintain state across multi-step interactions [6]. Hierarchical architectures introduce a layered organization of agents, typically consisting of high-level planning agents and lower-level execution agents [36]. This structure facilitates effective task decomposition, coordination, and management of complex workflows. Frameworks such as MetaGPT [33] and Google ADK [42] exemplify this architectural paradigm [3]. Modular architectures focus on decomposing agent systems into reusable and interchangeable components, such as planning, memory, and tool-use modules [36]. This design enhances the extensibility, maintainability, and customization of agentic workflows. Frameworks such as Mastra represent this architectural approach [3]. Graph-based architectures model agent workflows as directed graphs, where nodes represent agents or tasks and edges define dependencies and execution flow [36]. This enables dynamic, flexible, and stateful orchestration of complex processes. LangGraph is a prominent example of this architectural paradigm, which enables more reliable and flexible coordination of agent behavior across complex workflows [36].

2.2 Agentic Framework Evaluation In recent years, several studies have been conducted to evaluate the suitability of agentic frameworks for various software engineering tasks. As summarized in Table 1, existing studies compare agentic frameworks using five different approaches: (1) benchmark-based evaluation, (2) metrics-based evaluation, (3) architecture-based evaluation, (4) qualitative analysis of GitHub discussions, and (5) survey-based evaluation. Benchmark-based evaluation: Several recent studies evaluate the performance of agentic frameworks using benchmark-driven experiments. Yin et al. [14] compare seven agentic frameworks across three widely adopted benchmarks designed for software engineering tasks. These include the SRDD benchmark, the LLM-SmartAudit benchmark for vulnerability detection, and the SWE-bench Lite benchmark for program repair. Their study analyzes the capability of different frameworks in identifying code vulnerabilities and performing automated software maintenance tasks. Similarly, Liu et al. [7] propose the OctoTools framework for solving complex reasoning tasks and evaluate it against three existing frameworks—AutoGen, LangChain, and GPTFunction. The comparison is conducted across sixteen reasoning benchmarks, where the proposed OctoTools framework demonstrates comparatively higher accuracy on several tasks.

Table 1 Comparison of existing studies on agentic framework evaluation. S.No

Paper

Frameworks

Benchmark

Metrics

Architecture

Survey

Qualitative Analysis

Yin et al. [14]

7 frameworks

SRDD, LLM-SmartAudit, SWE-bench Lite

Liu et al. [7]

OctoTools, AutoGen, LangChain, GPT-Function

16 reasoning benchmarks

Accuracy

Barbarroxa et al. [15]

AutoGen, CrewAI, TaskWeaver

RMSE

Serafim and Mariama [16]

AutoGen, AutoGPT, Dify, Semantic-kernel

ROUGE

Wei et al. [6]

6 frameworks

✓

Aratchige and Ilmini [21]

AutoGen, Camel, CrewAI, MetaGPT, LangGraph

✓

Guo et al. [43]

MetaGPT, Camel, AutoGen

✓

Ferrag et al. [44]

LangChain, LlamaIndex, CrewAI, Swarm

✓

Bandi et al. [25]

Multiple frameworks

✓

Vaidhyanathan et al. [3]

10 frameworks

✓

Patel et al. [36]

6 frameworks

✓

Shi et al. [45]

8 frameworks

✓

Hasan et al. [41]

39 frameworks

✓

Sun et al. [23]

10 frameworks

✓

Wang et al. [4]

10 frameworks

✓

In addition, Barbarroxa et al. [15] compare three multi-agent frameworks: AutoGen, CrewAI, and TaskWeaver, using a machine learning code-generation case study. In their evaluation, each framework generates code to build energy forecasting models from a shared dataset, and the resulting models are assessed on a held-out dataset using the RMSE metric. This study provides a direct task-level comparison of agentic frameworks and complements research that focuses primarily on development practices rather than empirical performance. Metrics-based evaluation: Two studies compare agentic frameworks using task-specific evaluation metrics to assess their effectiveness. For instance, Serafim and Mariama [16] compare four frameworks: AutoGen, AutoGPT, Dify, and Semantic-kernel, on an optimization task based on the Metagente pipeline, evaluating summarization quality using ROUGE scores. Their results show that no single framework performs best across all dimensions: Dify achieves the highest ROUGE scores, whereas AutoGen and Semantic-kernel demonstrate stronger orchestration flexibility. In contrast, AutoGPT exhibits higher execution time and less efficient agent communication. Similarly, Barbarroxa et al. [15] utilize the RMSE metric to compare three multi-agent frameworks: AutoGen, CrewAI, and TaskWeaver, in a machine learning code-generation case study, where the generated models are evaluated based on their predictive accuracy. Survey-based evaluation: Several studies compare agentic frameworks through survey-based analyses that examine their architectural characteristics, design principles, and application domains. Wei et al. [6] provide a comparative overview of multiple frameworks, including AgentOrchestra, OWL, SE-Agent, Trae, GPTSwarm, OpenHands, and SWE-Agent. Their study analyzes architectural designs, coordination mechanisms, and application domains, highlighting how these frameworks support agent orchestration, task decomposition, tool integration, and inter-agent communication. Similarly, Aratchige and Ilmini [21] review frameworks such as AutoGen, Camel, CrewAI, MetaGPT, and LangGraph, outlining the technological landscape of agentic systems while identifying challenges such as coordination overhead and scalability. Guo et al. [43] present an overview of multi-agent frameworks and their domain applications, identifying key components of AI agents such as environment interfaces, profiling, communication, and capability acquisition. Their study discusses representative frameworks including MetaGPT, Camel, and AutoGen. Likewise, Ferrag et al. [44] analyze LLM-based agent frameworks across different domains, comparing systems such as LangChain, LlamaIndex, CrewAI, and Swarm in terms of workflows,

components, and core design ideas. Furthermore, Bandi et al. [25] conduct a comprehensive survey of agentic systems, summarizing framework architectures, evaluation methods, application domains, and open research challenges. Architecture-based evaluation: Several studies compare agentic frameworks by analyzing their architectural designs and system components. Vaidhyanathan et al. [3] provide an overview of ten agentic frameworks, examining their architectural patterns, planning and reasoning mechanisms, and memory management strategies used to support agent workflows. Similarly, Patel et al. [36] compare six popular frameworks: AutoGen, Google ADK, CrewAI, LlamaIndex, LangGraph, and Semantic-kernel, with a focus on their architectural characteristics. Their study proposes a taxonomy of framework design patterns, demonstrates agentic workflows for academic tasks, and discusses open challenges for future research. Shi et al. [45] conducted a comparative analysis of eight agentic frameworks, including LangGraph, CrewAI, OpenAI Swarm, AutoGen, IBM Watsonx.ai, NVIDIA NIM, Hugging Face Smolagents, and Pydantic-AI. The study evaluates these frameworks across several dimensions, such as developer experience, scalability, system performance, and multi-agent collaboration, by analyzing technical documentation and developer resources to understand their architectural capabilities. Qualitative analysis of GitHub discussions: Three studies evaluate agentic frameworks by analyzing GitHub developer discussions. For example, Hasan et al. [41] conducted a study analyzing 39 open-source agentic frameworks and 439 agentic applications to investigate common testing practices in the agent ecosystem. Their study examines how developers implement tests within these systems by systematically mining GitHub repositories to extract and categorize test functions. As a result, the authors identify 19 recurring testing patterns, including rule-based, input-driven, and application-level testing. Similarly, Sun et al. [23] analyze ten different agentic frameworks, focusing on the datasets used, the underlying LLM architectures, and the roles of agents in the decision-making process, such as communication, planning, and coordination. Furthermore, Wang et al. [4] examine ten agentic frameworks by mining GitHub repositories and developer discussions. Their work investigates widely used frameworks to identify practical challenges in real-world adoption and industrial implementation. The study compares these frameworks across several dimensions, including development efficiency, functional abstraction, learning cost, performance optimization, and maintainability. Summary: Despite the rapid growth of agentic frameworks and their increasing adoption in both research and industry, there remains a lack of comprehensive empirical evaluations focusing on reasoning-intensive tasks. As summarized in Table 1, only two studies have utilized benchmarks to compare a limited number of frameworks, while three studies employ various metrics to evaluate framework performance across different tasks. In contrast, several survey studies have been conducted to compare these frameworks, along with empirical studies that analyze GitHub developer discussions. However, there is a lack of a comprehensive benchmark-based empirical study that systematically selects leading agentic frameworks and evaluates their performance on reasoning tasks using multiple widely adopted benchmarks that encompass a large number of datasets. To address this gap, we have conducted a comprehensive empirical

study by systematically selecting widely used agentic frameworks and evaluating their performance on reasoning tasks. In total, we analyzed 22 representative frameworks and assessed their reasoning capabilities across three widely adopted benchmarks. Our evaluation provides insights into the reasoning performance of these frameworks and highlights the trade-offs in terms of accuracy, efficiency, and consistency across different tasks.

3 Study Design This study adopts an empirical research methodology using benchmark-based evaluation to assess the reasoning capabilities of agentic frameworks. As shown in Figure 1, the study is structured around three phases: (i) selecting agentic frameworks and benchmarks, (ii) evaluating the frameworks across multiple benchmarks to measure their reasoning capabilities, and (iii) conducting quantitative and qualitative data analysis.

3.1 Research Questions Based on our study goal, we formulated the following three research questions (RQs).

RQ1. How effectively do agentic frameworks perform on reasoning tasks across selected benchmarks? Objective. This research question evaluates the effectiveness of multiple agentic frameworks in solving reasoning tasks across selected benchmarks by analyzing both quantitative (e.g., accuracy, success rate) and qualitative (e.g., failure case analysis) performance. The objective is to identify the relative strengths and weaknesses of these frameworks in order to inform future improvements in reasoning methods and benchmark design. RQ2. How do different agentic frameworks compare in terms of cost and time efficiency when performing reasoning tasks across selected benchmarks? Objective. This question investigates the trade-offs between cost and time efficiency across different agentic frameworks when executing reasoning tasks on standardized benchmarks. The objective is to identify the cost- and time-optimal framework for practical deployment. RQ3. To what extent do agentic frameworks demonstrate consistent reasoning performance across the selected benchmarks? Objective. This research question examines the consistency of different agentic frameworks across multiple benchmarks. The objective is to determine whether these frameworks maintain consistent reasoning accuracy across diverse task types, indicating their reliability and generalizability. 8

Fig. 1 Overview of the research design and evaluation workflow.

3.2 Agentic Framework Selection In this study, we collected open-source agentic frameworks from GitHub. The following section describes the selection process and criteria used to identify the frameworks included in our analysis.

3.2.1 Search String Initially, we considered adopting the PICO (Population, Intervention, Comparison, Outcome) framework [46] to guide the development of our search string. However, applying PICO resulted in an extensive set of keywords and boolean operators, which did not produce relevant results when used as search queries on GitHub. As a result, we developed the final search string based on domain expertise and iterative trial searches conducted collaboratively by the co-authors. This approach allowed us to refine the search strategy in line with the study’s objectives while maintaining a broad scope within practical limits. In developing our search string, we combined domain-specific terms to identify relevant agentic frameworks. Specifically, we used combinations of the phrases “agent framework”, “multi-agent framework”, “agent-based framework”, “autonomous agent framework”, “agentic workflow framework”, and “LLM agent framework” to ensure broad coverage of repositories describing agent-based or agentic architectures. 9

ID I1 I2 I3 I4

Type Inclusion Inclusion Inclusion Inclusion

Criteria Description The repository must have more than 10 stars and 10 forks. It must contain at least one open issue. The project should have more than 5 contributors. The project must genuinely be a framework for agent-based systems, not just contain the keywords. I5 Inclusion Project documentation and content must be in English. E1 Exclusion The repository is still under development. E2 Exclusion The repository is set to read-only mode. E3 Exclusion The repository is no longer active or maintained. E4 Exclusion The repository content is not in English. Table 2 Inclusion and exclusion criteria for agentic framework selection

These terms were searched within both repository README files and description fields (in:readme in:description) to increase the likelihood of retrieving relevant projects that explicitly reference these concepts.

(”agent framework” OR ”multi-agent framework” OR ”agentbased framework” OR ”autonomous agent framework” OR ”agentic workflow framework” OR ”LLM agent framework”) in:readme in:description stars:> 10 The initial search produced 1,200 repositories as of 26 July 2025. In the next phase, a manual analysis of the retrieved repositories was conducted. Based on the predefined inclusion and exclusion criteria presented in Table 2, a final set of agentic frameworks was selected.

3.2.2 Framework Selection Initially, we manually analyzed a total of 1,200 repositories retrieved from GitHub to identify potential agentic frameworks. In the first phase, based on the inclusion and exclusion criteria described in Table 2, we shortlisted 55 repositories that met the minimum quality and relevance requirements. More details about the frameworks can be found in [20].

3.2.3 Finalized Frameworks For further experimentation, we selected 22 agentic frameworks from the 55 shortlisted frameworks based on the following criteria:

• The number of GitHub stars and forks, used to select the top 22 frameworks with the most number of GitHub stars and forks. • The number of contributors and recent commits, used to select the top 22 frameworks with the most active development. • A healthy number of open and closed issues.

Applying these criteria resulted in a final set of 22 agentic frameworks for empirical analysis and evaluation. The selected frameworks are presented in Table 3.

3.3 Classification of Agentic Frameworks To analyze the agentic frameworks systematically, we classified the 22 selected frameworks into five categories based on their primary architectural focus and core functionality: (i) single-agent architecture, (ii) role-based multi-agent architecture, (iii) hierarchical architecture, (iv) modular component architecture, and (v) graph-based workflow frameworks. To categorize the selected frameworks into these five categories, we followed previous studies such as [32], [3], and [36].

3.3.1 Single-Agent Architecture Single-agent architectures focus on planning, observation, and action within the capabilities of a single autonomous agent [32]. As illustrated in Table 3, four frameworks fall into this category: AutoGPT, TaskWeaver, Semantic-kernel, and LangChain.

3.3.2 Role-Based Multi-Agent Architecture Role-based multi-agent architecture frameworks focus on collaboration among multiple autonomous agents that interact through structured, role-based communication, often combining predefined coordination rules with LLM-driven decision-making [47]. As shown in Table 3, 13 frameworks were classified under this category, i.e., BabyAGI, AutoGen, Camel, CrewAI, SuperAGI, Swarm, Agency-Swarm, OpenAIAgents-Python, Agent-zero, PraisonAI, Qwen-Agent, Pydantic-AI, and ANUS. These frameworks are designed around agent-to-agent communication and collaboration as the primary mechanisms for task execution.

3.3.3 Hierarchical Architecture Hierarchical architectures organize agents in a multi-level structure where higher-level agents are responsible for planning, coordination, and task decomposition, while lowerlevel agents execute specific subtasks [3]. As shown in Table 3, two frameworks are classified in this category: MetaGPT and Google ADK. These frameworks highlight structured task management, where responsibilities are distributed across different layers of agents to improve coordination and scalability.

3.3.4 Modular Component Architecture Modular component architectures are designed around the integration of specialized and reusable components that can be combined to extend agent capabilities and facilitate interaction with external systems [36]. In our taxonomy, two frameworks fall under this category: Mastra and Upsonic. These frameworks highlight the importance of modularity, allowing developers to build flexible agent systems by assembling independent functional components.

Primary Category

Agentic Framework

Reasoning

GitHub Stars

Forks

Company Name

GitHub Link

Single-Agent Architecture

AutoGPT TaskWeaver Semantic-kernel LangChain

CoT TF TF CoT

182k 6.1k 27.4k 110k

46.2k 759 4.5k 21.1k

Significant-Gravitas Microsoft Microsoft langchain-ai

https://github.com/Significant-Gravitas/AutoGPT https://github.com/microsoft/TaskWeaver https://github.com/microsoft/semantic-kernel https://github.com/langchain-ai/langchain

Role-Based Multi-Agent Architecture

BabyAGI Autogen Camel CrewAI SuperAGI Swarm Agency-swarm OpenAI-Agents-Python Agent-zero PraisonAI Qwen-Agent Pydantic-AI ANUS

CoT Hybrid CoT Hybrid Hybrid Hybrid Hybrid Hybrid CoT Hybrid Hybrid TF Hybrid

22.2k 55.3k 16.2k 45.5k 17.2k 21.1k 4k 19.4k 15.9k 5.6k 15.2k 15.3k 6.3k

2.9k 8.3k 1.8k 6.1k 2.2k 2.2k 1k 3.2k 3.3k 773 1.5k 1.7k 921

yoheinakajima Microsoft camel-ai crewAIInc TransformerOptimus OpenAI VRSEN OpenAI agent0ai MervinPraison QwenLM pydantic anus-dev

https://github.com/yoheinakajima/babyagi https://github.com/microsoft/autogen https://github.com/camel-ai/camel https://github.com/crewAIInc/crewAI https://github.com/TransformerOptimus/SuperAGI https://github.com/openai/swarm https://github.com/VRSEN/agency-swarm https://github.com/openai/openai-agents-python https://github.com/agent0ai/agent-zero https://github.com/MervinPraison/PraisonAI https://github.com/QwenLM/Qwen-Agent https://github.com/pydantic/pydantic-ai https://github.com/anus-dev/ANUS

Hierarchical Architecture

MetaGPT Google ADK

Hybrid Hybrid

64.9k 18.2k

8.2k 3k

FoundationAgents Google

https://github.com/FoundationAgents/MetaGPT https://github.com/google/adk-python

Modular Component Architecture

Upsonic Mastra

TF TF

7.8k 21.8k

722 1.7k

Upsonic mastra-ai

https://github.com/Upsonic/Upsonic https://github.com/mastra-ai/mastra

Graph-Based Architecture

LangGraph

Hybrid

25.9k

4.5k

langchain-ai

https://github.com/langchain-ai/langgraph

Table 3 Taxonomy of selected agentic frameworks based on architectural design.

3.3.5 Graph-Based Workflow Frameworks Graph-based workflow frameworks represent the execution and coordination of agents as directed graphs, where nodes correspond to agents or tasks and edges capture control flow, state transitions, and dependencies [27], [36]. LangGraph is classified within this category due to its explicit graph-based execution model. As noted by Derouiche et al. [47], LangGraph introduces a novel graph-based approach for sequencing tasks among LLM agents. By supporting compositional workflows and stateful operations, it enables traceable and scalable agent design, particularly in research and analytics contexts.

3.4 Benchmark Selection We searched for existing benchmarks that satisfy the following specific selection criteria: (i) The benchmarks should be well-established within the research community and frequently used for evaluating reasoning performance, ensuring that our results are comparable and credible. (ii) The selected benchmarks should include a diverse set of reasoning tasks that cover multiple reasoning dimensions, such as logical, scientific, mathematical, commonsense, and abstract reasoning [48]. (iii) Each benchmark should provide publicly available datasets and evaluation protocols to facilitate reproducibility and transparency. (iv) The benchmark dataset must be suitable for agentic AI evaluation. Initially, we searched for suitable benchmarks based on the study conducted by Ferrag et al. [44]. This study presents a comprehensive comparison of benchmarks developed between 2018 and 2025 that evaluate language models and agents across multiple domains. Based on the selection criteria described above, we chose the BBH benchmark [10], which is widely recognized for its emphasis on multi-step and abstract reasoning [9]. BBH comprises 27 distinct reasoning tasks, including Boolean logic, causal judgment, arithmetic reasoning, and other forms of analytical and deductive reasoning.

To evaluate mathematical reasoning, we selected the GSM8K benchmark [17], which includes grade-school-level mathematical word problems. The dataset specifically focuses on word problems that involve real-world contexts such as calculating costs, determining quantities, working with rates and ratios, and solving problems involving time, distance, and other practical mathematical applications [49]. Additionally, to assess scientific reasoning, we utilized the ARC benchmark [18], which consists of multiple-choice science questions designed to test factual understanding and reasoning ability. This benchmark contains a comprehensive dataset containing grade-school-level science questions sourced from actual human standardized tests. The benchmark is divided into two partitions: Easy and Challenge, where the challenge set specifically contains questions that cannot be answered through simple information retrieval or statistical word co-occurrence methods, thus requiring genuine reasoning capabilities [18]. Table 4 presents an overview of the three selected benchmarks, highlighting their reasoning task types and the number of tasks included in each. S. No. 1 2 3

Benchmark BBH GSM8K ARC

Reasoning Task Type 27 challenges covering various reasoning types Mathematical reasoning (grade-school-level problem solving) Scientific reasoning (multiple-choice science questions)

Paper Reference Suzgun et al. [10] Cobbe et al. [17] Clark et al. [18]

Table 4 Overview of selected benchmarks and their reasoning tasks.

3.5 Experiment Setup In this section, we present the experimental details of the agentic framework development setup. We describe the models and external tools used for agent development and execution. We also detail the configuration, environment, and evaluation setup of the selected agentic frameworks, as well as the hardware platforms used in this project.

3.5.1 Language Model In this study, we utilized the GPT-5.2 model for all experiments. The same model was used across all agentic frameworks to maintain experimental control and ensure fair and valid framework comparisons. We selected GPT-5.2 as the base model for all agentic framework evaluations because of its strong reasoning capabilities and high performance on standardized benchmarks [50]. On the ARC-AGI-1 [51] and ARC-AGI-2 [52] benchmarks, GPT-5.2 achieves accuracies of 86.2% and 52.1%, respectively, outperforming Gemini 3 Pro and Opus 4.5 [53]. Similarly, on the CharXiv Reasoning benchmark [54], which focuses on complex reasoning tasks requiring information synthesis, GPT-5.2 achieves a leading score of 0.821, compared to 0.814 by Gemini 3 Pro. On FrontierMath (Tier 1–3) [55], a benchmark designed to evaluate advanced mathematical reasoning, GPT-5.2 achieves the highest accuracy of 40.3% among the models evaluated.

Collectively, these benchmarks address abstract reasoning, advanced mathematical reasoning, and scientific problem-solving tasks. In addition, GPT-5.2 demonstrates strong performance on other established benchmarks, including SWE-Bench (Verified), GPQA Diamond, and AIME, further supporting its suitability as a robust base model for comparing the reasoning capabilities of different agentic frameworks [50].

3.5.2 External Environment In this experiment, we used the GPT-provided external web search 1 tool to enable access to information beyond the model’s internal training data. The web search tool allows agents to query online sources when the required information cannot be obtained from the language model’s internal knowledge. This tool-augmented setup enables agents to retrieve up-to-date and task-specific information at runtime, which is particularly beneficial for knowledge-intensive and dynamically evolving tasks. By integrating web search while preserving the LLM as the core reasoning component, agentic frameworks can enhance response accuracy and contextual awareness without altering the underlying decision-making process of the model.

3.5.3 Framework Configuration Each agentic framework was configured under a standardized setup to ensure comparability and reproducibility across the evaluations. All frameworks were initialized within a unified execution environment that maintained identical model parameters, including a fixed few-shot configuration and chain-of-thought prompting for step-bystep reasoning. The temperature parameter was set to 0 to eliminate randomness [49]. The token limit for each benchmark was determined through a pilot experiment. Initially, we adopted token limits suggested by prior studies. For example, previous work reports an average limit of 224 tokens per task for BBH [56], while Zhang et al. [49] used a limit of 256 tokens for GSM8K to support step-by-step mathematical reasoning. For ARC, we initially selected a token limit of 224 tokens because the benchmark consists of multiple-choice questions with relatively short expected answers. Accordingly, we first applied these token limits in our pilot experiments. However, based on the initial results, we observed that these limits were insufficient for our multi-agent setup, as agents frequently required additional tokens to generate complete reasoning steps and final answers when using chain-of-thought reasoning. The previous studies used smaller token limits because their experiments did not involve multi-agent communication and extended reasoning steps. Therefore, based on the pilot results, we increased the token limits to avoid cutting off responses and to allow agents to produce full reasoning outputs. As a result, we set the maximum token limit to 1200 tokens for BBH and GSM8K. For the ARC benchmark, our pilot experiments indicated that a smaller limit was sufficient due to the shorter answer format. Therefore, we set the maximum token limit to 1024 tokens for ARC, which allows sufficient reasoning while keeping the responses concise. In addition, we set the API timeout to 500 seconds to ensure consistent response behavior across all benchmarks. 1

https://platform.openai.com/docs/guides/tools-web-search

3.5.4 Environment Setup and Dependencies All agentic frameworks were deployed within isolated virtual environments to ensure dependency integrity and consistency throughout the experimental process. Each environment was initialized using a unified dependency management system that synchronized library versions and Python configurations across all frameworks. Core dependencies, including libraries for model interaction, data processing, and evaluation management, were version-locked to ensure consistent behavior and stable execution across the experiments.

3.5.5 Multi-Agent Setup To evaluate the reasoning capabilities of different agentic frameworks, we designed a multi-agent setup and applied it consistently across 22 agentic frameworks. For each framework, we developed agents that execute reasoning tasks defined by three standardized benchmarks. In this project, we adopted a two-agent setup. We conducted a pilot study to experiment with different numbers of agents. For instance, a single-agent setup does not allow evaluation of agent communication, coordination, and other core characteristics of agentic frameworks that cannot be examined in single-agent settings. On the other hand, configurations with a larger number of agents (e.g., four agents) significantly increased the execution time and API cost due to more frequent interactions between agents. Finally, based on the results of the pilot experiments, we selected a two-agent setup as a balanced configuration that enables the evaluation of agent interaction while keeping the computational cost and execution time manageable. In this setup, the Task Execution Agent is responsible for performing benchmarkspecific reasoning tasks. It processes each input question from the benchmark dataset, systematically applies chain-of-thought prompting techniques to generate step-bystep reasoning, and produces candidate solutions. These outputs are then forwarded to the Verification Agent for consistency checking and refinement. The Verification Agent independently evaluates the outputs produced by the Task Execution Agent, by checking the correctness, consistency, and reasoning quality of the generated answers. To ensure consistent and controlled comparisons across frameworks, we have kept the agent roles, prompts, task descriptions, and verification criteria the same for all frameworks. Fixing the multi-agent configuration and prompt design ensures that performance differences can be attributed to the underlying agentic frameworks rather than variations in agent behavior or prompt engineering.

3.5.6 Hardware Platforms Experiments were conducted on a server running Debian GNU/Linux 12 (Bookworm). The system is equipped with an Intel Core i7-14700 processor (20 cores, 28 threads), 31 GB RAM, and a 952 GB NVMe SSD. The server includes two GPUs: an NVIDIA Tesla P40 with 24 GB VRAM and an NVIDIA L4 with 24 GB VRAM. The system uses NVIDIA driver version 535.216.01 with CUDA 12.2 support.

3.5.7 Evaluation Setup For the evaluation phase, each benchmark was executed through an automated evaluation pipeline that sequentially loaded the benchmark dataset, set up the corresponding agentic framework, and executed the reasoning tasks. Tasks within each benchmark were processed sequentially to ensure consistent execution and resource usage across frameworks. Model inference was performed with a predefined retry policy to handle transient failures, such as API timeouts or rate-limit errors. Failed requests were retried up to a fixed number of attempts before being marked as unsuccessful and logged accordingly. Tasks exceeding a predefined execution time limit were terminated and recorded as timeouts. The system supported both sample-mode evaluations, using a small subset of tasks for rapid testing and debugging, and full-run evaluations, covering the complete benchmark dataset for comprehensive analysis. All generated responses were validated against benchmark-specific answer formats and automatically scored using the official evaluation criteria provided by each benchmark. During execution, the pipeline logged detailed metadata for each task, including task identifiers, execution time, number of retries, success or failure status, and final scores. Performance metrics such as accuracy, task completion rate, and total runtime were computed and stored for each framework and benchmark combination. This evaluation setup enabled systematic and reproducible comparison of agentic frameworks across multiple reasoning benchmarks.

3.6 Data Collection In this study, experimental results were systematically collected for each evaluated agentic framework across all selected benchmarks. All experiments were executed in a standardized and controlled computational environment. This controlled setup ensures that the collected data accurately reflect framework-level behavior rather than external infrastructure effects. For each framework–benchmark execution, the generated outputs and associated metadata were automatically recorded and stored. The collected data include task results, accuracy scores, framework response times, computational cost, and other evaluation-related statistics. All results were saved in a structured JSON format to support reproducibility and downstream analysis. To maintain clear organization and traceability, results were stored separately for each benchmark and framework (e.g., FrameworkName BBH config.log.json, FrameworkName GSM8K config.log..json, FrameworkName ARC config.log.json), enabling modular analysis and facilitating comparison across benchmarks during the evaluation phase.

3.7 Data Analysis In this study, we analyzed multiple types of data using both quantitative and qualitative approaches. For the quantitative analysis, descriptive statistical methods [57] were employed. The collected results were examined using a combination of descriptive statistics (e.g., mean, median, standard deviation, standard error of the mean (SEM), F-statistics, P-value, mean absolute difference, and coefficient of variation

(CV)). For the qualitative data, we utilized thematic analysis [58] to analyze openended responses. To facilitate clearer interpretation of the experimental outcomes, various data visualizations—including bar charts, line graphs, and box plots—were used for effective presentation and comparison of the results. In the following sections, we discuss the data analysis methods employed for each research question (RQ) in detail.

3.7.1 Accuracy (RQ1) To address RQ1, we evaluated the reasoning performance of the agentic frameworks using three widely adopted reasoning benchmarks: BBH, GSM8K, and ARC. We first applied descriptive statistical analysis to compute the accuracy of each framework on individual benchmarks. To summarize the overall performance of each framework, the benchmark-level accuracy scores were combined by calculating their mean across the three benchmarks. This process produces a single representative accuracy score for each framework, enabling a clear and holistic comparison of reasoning performance across diverse reasoning tasks. Finally, the frameworks were ranked according to their mean accuracy scores to facilitate comparative analysis of their reasoning capabilities. n

1X Mean Accuracy = x̄ = xi n i=1

Framework-level performance variation analysis: To understand how the performance of each framework varies across the evaluated benchmarks, the SEM is reported as an additional measure in the performance analysis. SEM reflects the extent to which the benchmark accuracies of a framework vary around its mean accuracy. In this study, a smaller SEM indicates that the benchmark results are relatively close to one another, whereas a larger SEM reflects greater differences among the benchmark accuracies. Since the SEM is calculated from the standard deviation, the standard deviation is first defined as sP n 2 i=1 (xi − x̄) SD = , (1) n−1 where xi denotes the accuracy value of a framework on benchmark i, x̄ denotes the mean accuracy of that framework across the available benchmarks, and n denotes the number of available benchmark results. The SEM is then computed as SD SEM = √ , (2) n where SD denotes the standard deviation of the benchmark accuracies for a given framework and n denotes the number of available benchmark results used to compute the mean. In this study, n = 3 for frameworks with valid results on all three benchmarks, while smaller values of n are used for frameworks with incomplete benchmark results.

Architecture-based performance analysis: To understand how framework performance varies within each architectural category, we employed one-way analysis of variance (ANOVA). This method was used to test whether the benchmark accuracies of frameworks in the same category differ significantly across the three evaluated benchmarks. ANOVA is appropriate in this context because it enables the comparison of mean performance across multiple benchmark groups within a single statistical framework. The ANOVA test is based on the F-statistic, which is calculated as M Sbetween , (3) M Swithin where M Sbetween denotes the mean square between groups, representing the variation among benchmark means, and M Swithin denotes the mean square within groups, representing the variation within each benchmark group. A larger F value indicates that the variation between benchmark means is greater relative to the variation within groups. To determine whether the observed variation is statistically significant, the corresponding p-value was also computed. In this study, statistical significance was determined using a threshold of p < 0.05. A p-value below this threshold indicates that the observed performance differences across benchmarks are unlikely to have occurred by chance, whereas a p-value above this threshold suggests that the variation is not statistically significant. Failure modes: We also analyzed the data to identify failure modes in which the frameworks show differing behaviors. We applied thematic analysis to qualitatively analyze and characterize the observed failure patterns. F =

3.7.2 Cost+Time (RQ2) To evaluate the computational cost of each agentic framework, we recorded the total number of tokens consumed for each benchmark task. The overall token consumption was then calculated by summing the tokens used across all tasks, as defined in Eq. (4): Total Tokens =

N X

(4)

i=1

To obtain a normalized measure of computational cost, we computed the average token consumption per task using the arithmetic mean, as shown in Eq. (5): N

Average Tokens per Task =

1 X Ti N i=1

(5)

Here, Ti represents the number of tokens consumed for the i-th task, and N denotes the total number of benchmark tasks. Finally, the cost associated with each framework was calculated based on the total and average token consumption. Specifically, both the total cost and the average cost per task were estimated using the current OpenAI token pricing. As the GPT-5.2

model was used in this study, the cost calculation was performed according to its pricing at the time of experimentation. Time: To evaluate the time efficiency of the agentic frameworks, we measured the execution time required to complete each benchmark task. For every task, the total response time was recorded as the elapsed wall-clock time between the start of execution and the generation of the final output. The overall time consumption was then calculated by summing the execution time across all tasks. In addition, the average time per task was computed using the arithmetic mean to provide a normalized measure of performance.

3.7.3 Consistency Analysis (RQ3) To address RQ3, we evaluated the consistency of agentic frameworks across multiple benchmarks by computing both absolute and relative (scale-adjusted) consistency measures. Specifically, we analyzed the consistency of each framework across the three benchmarks (BBH-ARC, BBH-GSM8K, and ARC-GSM8K). To measure absolute consistency, we employed an absolute difference-based approach. This method computes the pairwise absolute differences between benchmark accuracies and aggregates them to obtain a total deviation score for each framework. A lower total deviation indicates higher consistency across benchmarks. The total deviation is calculated as shown in Eq. (6): Total Deviation = |A1 − A2 | + |A1 − A3 | + |A2 − A3 |

(6)

To provide a normalized measure of variation, we further computed the mean deviation across benchmark pairs, as defined in Eq. (7):

|A1 − A2 | + |A1 − A3 | + |A2 − A3 | (7) 3 To measure relative (scale-adjusted) consistency, we utilized the CV method, which quantifies variability relative to the mean performance across benchmarks. The CV is defined in Eq. (8): Mean Deviation =

σ , (8) µ where σ represents the standard deviation and µ denotes the mean accuracy across the three benchmarks. In this study, a lower CV value indicates higher consistency in framework performance. CV =

4 Experimental Results In this section, we present the outcome of our benchmark-based evaluation of 22 agentic frameworks. As shown in Table 5, the agentic frameworks are divided into five categories based on their architecture. The results section is divided according to the three research questions presented in Section 3.1.

4.1 Agentic Framework Performance (RQ1) We evaluated the performance of agentic frameworks on reasoning tasks using three benchmarks, namely BBH, GSM8K, and ARC. Table 5 reports the accuracy of each agentic framework on the three benchmarks, together with its mean score, SEM, and completion. The mean score serves as an overall basis for comparing framework performance across the benchmarks. A more detailed comparison based on these mean scores is presented below and further illustrated in Figure 2. In addition to mean accuracy, the SEM is reported to reflect the variation in each framework’s accuracy across the benchmarks. A smaller SEM indicates that the accuracy levels of the framework across the three benchmarks are relatively close to each other, whereas a larger SEM reflects greater differences in framework performance across benchmarks. For example, TaskWeaver and Agency-Swarm show relatively low SEM values of 3.12 and 2.98, respectively, indicating limited variation across benchmarks, although their accuracy remains consistently low. In contrast, frameworks such as Mastra, MetaGPT, AutoGen, and PraisonAI exhibit high SEM values of 26.55, 27.93, 26.16, and 25.08, respectively, owing to substantial fluctuations in performance across the benchmarks. Overall, most frameworks show SEM values ranging approximately from 12 to 16. We have also reported completion to indicate the number of benchmarks for which valid results were obtained for each framework. Most frameworks completed all three benchmarks, whereas Camel completed only one benchmark, and Upsonic and Mastra completed two. To further illustrate comparative performance, Figure 2 presents the ranking of the agentic frameworks based on their overall mean scores across the three benchmarks. Overall, 12 frameworks achieved mean accuracies ranging from 74.57% to 75.94%, with only a 1.4 percentage point difference between the lowest and highest values. Among these frameworks, those in the role-based multi-agent category tended to achieve slightly higher accuracy. For example, OpenAI-Agents-Python, Pydantic-AI, QwenAgent, Agent-zero, ANUS, and BabyAGI ranked first, second, third, fourth, fifth, and sixth, respectively. However, some well-known frameworks in this category, such as AutoGen and Camel, showed poorer performance, ranking 18th and 22nd, respectively. The reasons for the failure of these frameworks are provided in Section 4.1.2. The frameworks in the single-agent architecture category exhibit notable variation in performance. Of these frameworks, AutoGPT achieved consistently high accuracy across the evaluated benchmarks, securing an overall rank of seventh. Similarly, Semantic-kernel and LangChain achieved moderate accuracy, ranking 11th and 12th, respectively. In contrast, TaskWeaver ranked 21st overall, reflecting lower average accuracy compared to other frameworks. Overall, the results indicate that, while certain single-agent frameworks can maintain competitive performance, others may struggle to generalize effectively across diverse benchmark tasks. Similarly, within the hierarchical architecture category, performance varies across frameworks. For example, MetaGPT ranked 15th overall, while Google ADK ranked 9th, indicating comparatively stronger performance across the evaluated benchmarks. Within the modular component architecture category, both frameworks show relatively lower rankings. For example, the Upsonic framework ranked 20th and also exhibited the highest token consumption during execution. This high computational

Primary Category

Agentic Framework AutoGPT TaskWeaver Semantic-kernel LangChain BabyAGI AutoGen Camel CrewAI SuperAGI Swarm Agency-Swarm OpenAI-Agents-Python Agent-zero PraisonAI Qwen-Agent Pydantic-AI ANUS MetaGPT Google ADK Upsonic Mastra LangGraph

Single-Agent Architecture

Role-Based Multi-Agent Architecture

Hierarchical Architecture Modular Component Architecture Graph-Based Architecture

BBH (%) 90.77 13.88 89.16 89.86 89.77 24.08 – 78.91 90.11 90.43 34.57 91.06 89.48 20.30 90.69 90.71 89.83 72.75 90.49 – – 89.80

GSM8K (%) 44.40 7.92 44.49 44.39 44.48 4.55 – 42.82 44.51 44.29 37.58 44.31 44.21 4.66 44.33 44.44 44.21 0.00 44.57 43.29 32.46 44.35

ARC (%) 90.76 18.70 90.94 90.05 91.78 90.94 8.75 91.51 90.49 90.72 27.51 92.45 92.58 86.49 92.54 92.67 92.18 91.60 90.15 67.61 90.77 89.56

Mean (%) 75.31 13.50 74.86 74.77 75.34 39.86 – 71.08 75.04 75.15 33.22 75.94 75.42 37.15 75.85 75.94 75.41 54.78 75.07 23.63 41.08 74.57

SEM 15.46 3.12 15.20 15.19 15.44 26.16 – 14.59 15.26 15.43 2.98 15.82 15.63 25.08 15.77 15.76 15.61 27.93 15.25 22.01 26.55 15.11

Completion 3/3 3/3 3/3 3/3 3/3 3/3 1/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3 2/3 2/3 3/3

Table 5 Accuracy of agentic frameworks across three benchmarks, mean accuracy (higher percentages indicate better results), SEM (lower values indicate less variation), and completion (valid benchmark results).

Framework ranking based on mean accuracy across three benchmarks

Role-Based

Hierarchical

Modular

Mastra

Langraph

Upsonic

MetaGPT

ANUS

Pydantic-AI

Agency-swarm

OpenAI-Agents

Swarm

CrewAI

SuperAGI

Autogen

Single-Agent

Camel-ai

Qwen-Agent

Agent-zero

Google ADK

19 14

PraisonAI

22 17

BabyAGI

LangChain

Semantic Kernel

AutoGPT

TaskWeaver

25 21 20 15 11 12 10 7 5 0

Graph

Fig. 2 Comparative ranking of agentic frameworks based on the mean accuracy across three benchmarks (Rank 1 is the highest).

cost, combined with relatively weak performance, highlights potential inefficiencies in task orchestration and reasoning control mechanisms. A detailed analysis of execution failures, runtime behavior, and cost-related challenges is provided in Section 4.1.2. Similarly, the Mastra framework ranked 16th overall, indicating lower mean performance. In the graph-based architecture category, LangGraph ranked 13th, indicating moderate performance compared to the other evaluated frameworks. More details about the dataset can be found in our replication package [20].

4.1.1 Performance Variation Across Categories As shown in Table 6, the statistical significance of performance differences across frameworks within each category was evaluated using the F-statistic and p-value approach. The results indicate that the role-based multi-agent architecture category exhibits a statistically significant variance (F (2, 33) = 16.61, p = 1.02 × 10−5 ). This highlights the fact that framework performance within this category varies considerably, suggesting that specific architectural designs and task characteristics strongly influence their effectiveness. For instance, although the highest-ranking frameworks belong to this category, it also includes lower-performing frameworks, resulting in significant differences in overall performance. In contrast, the single-agent architecture and hierarchical architecture categories do not show statistically significant differences across the frameworks, as indicated by their respective p-values of 0.231 and 0.071. This suggests a higher degree of performance consistency among frameworks within these groups. However, the limited number of frameworks in some categories, particularly in the hierarchical architecture category, may reduce the statistical power and affect the reliability of these comparisons. Category Single-Agent Architecture Role-Based Multi-Agent Architecture Hierarchical Architecture Modular Component Architecture Graph-Based Architecture

F-Statistic F (2, 9) = 1.73 F (2, 33) = 16.61 F (2, 3) = 7.22 – –

p-value 0.231 1.02 × 10−5 0.071 – –

Significant No Yes No No No

Table 6 ANOVA results for performance variance across agentic framework categories.

For the modular component architecture and graph-based architecture categories, statistical comparison was not feasible due to insufficient benchmark results or the presence of only a single framework. Overall, the findings suggest that, while role-based multi-agent architectures offer the highest performance potential, they also show significant internal variance, indicating that architectural complexity does not guarantee superior results. These results suggest that for complex reasoning and mathematical tasks, the choice of a specific multi-agent design is more effective than the broad selection of a category, as individual framework maturity remains a key factor in overall performance.

4.1.2 Observed Failures and Practical Constraints The Camel framework demonstrated limited performance across the evaluated benchmarks. It was only able to fully execute the ARC benchmark, where it achieved an accuracy of 8.75%. However, for the BBH and GSM8K benchmarks, Camel failed to complete the evaluation tasks. On the BBH benchmark, Camel ran continuously for 266 hours and 49 minutes (approximately 11 days) without successfully completing the evaluation. Due to the excessive execution time and the absence of meaningful 22

progress, the process was terminated manually. Similarly, for the GSM8K benchmark, Camel ran for approximately 73 hours and 23 minutes (around 3 days) but still failed to complete the evaluation tasks. Our analysis indicates that uncontrolled context growth was the primary cause of these failures. Specifically, the framework attempted to build extremely large memory contexts and repeatedly shorten or reduce them to fit within the model’s context limit. This process caused frequent context overflow handling and message chunking. As a result, the system generated a large number of API calls and increased prompt-token usage while producing minimal or incomplete outputs. The evaluation process became computationally inefficient and was unable to reach task completion within a reasonable time frame. As shown in Table 5, AutoGen achieved 90.94% accuracy on the ARC benchmark, possibly because it consists of relatively simple tasks. However, on more complex benchmarks such as BBH and GSM8K, its accuracy fell to 24.08% and 4.55%, respectively, while its runtime remained significantly higher. Our analysis indicates that AutoGen generated a large number of iterative agent interactions, resulting in repeated API calls for a single task. Complex benchmarks typically require multi-step reasoning, planning, and agent coordination, which increase prompt length and token consumption. As a result, the evaluation process produced a high volume of API requests, eventually exceeding the available API usage quota and causing the execution to terminate before completion. We observed a similar situation with the Mastra framework, which also generated a large number of API requests, and the execution failed due to exceeding the available API usage quota. The Upsonic framework presented a different limitation related to computational cost. When running BBH and GSM8K on the same day, Upsonic consumed approximately 4,780,000 tokens within a single day, resulting in a cost of $1,434.20. As a result of this unusually high cost, the evaluation process was stopped. During this period, the GSM8K benchmark was completed; however, BBH was not fully executed. The analysis indicates that the high computational cost is primarily due to very high prompt-token usage caused by repeated retries and uncontrolled context growth. The framework frequently caused extraction failures, which forced the system to repeatedly call the LLM until the output matched the expected format. In addition, the framework collects large contextual information through its memory and execution traces, where previous prompts, outputs, and task instructions are continuously added to the prompt context. As a result, the input context becomes significantly larger with each call. This behavior is reflected in the logs, which show extremely high input token counts per request, even though the final model responses remain relatively short. As a result, the combination of memory-based context growth, repeated extraction retries, and large prompt contexts considerably increased both token consumption and the overall API cost when running the Upsonic framework on the benchmark tasks.

Takeaway 1. Most frameworks showed mean accuracies between 74.57% and 75.94%, indicating a narrow performance plateau across architectures. Frameworks that fell outside this plateau underperformed, not because of weaker reasoning strategies but rather due to system-level orchestration issues such as poor memory management, unstable retry behavior, and uncontrolled context growth. This interpretation is further supported by the role-based multi-agent category, which contains both the best and some of the worst performers, with statistically significant within-category variance confirmed by ANOVA (F (2, 33) = 16.61, p = 1.02 × 10−5 ).

4.2 Performance Trade-Offs (RQ2) In this section, we present the trade-off results of the agentic frameworks, which show the computational cost and execution time required to complete each task. The benchmarks contain a total of 16,495 tasks. Across all experiments, the total number of input tokens consumed was 10.861 billion, while the total number of output tokens was 83,196. The large gap between input and output token consumption is due to the extensive input context and prompts required by the frameworks, whereas the outputs were usually very short, often limited to labels such as correct, incorrect, or failed. The overall experimental cost was $3,154.30, with a total of 685,443 API requests. The total execution time was approximately 24 days (575 hours 37 minutes). In the following sections, we present a detailed analysis of the trade-offs for each agentic framework. As shown in Table 7, we report the total token consumption, the average number of tokens used per task across the three benchmarks, and the corresponding average cost in cents. This allows us to examine the computational cost associated with each framework. In addition, execution time is analyzed by reporting the total runtime and average time per task for each framework. Figure 3 presents a visual representation of the performance–efficiency trade-off for the selected agentic frameworks by plotting their average mean time per task (yaxis) against their average mean cost per task (x-axis) across the three benchmarks. We computed the mean response time per task for each framework across the three benchmarks, as well as the mean token consumption per task, which reflects the overall cost of execution. This analysis provides an overview of the time and cost efficiency of each framework when performing the benchmark tasks. In Figure 3, different colors represent the categories of the agentic frameworks, while each data point corresponds to an individual framework. The figure also shows the mean accuracy of each framework, enabling a comprehensive comparison between reasoning performance, response time, and cost. The results show that the average cost of most frameworks ranges between 0.14¢ and 0.18¢ per benchmark task, while the average response time typically varies between 4 and 6 seconds per task. As shown in Figure 3, four frameworks: Pydantic-AI, ANUS, SuperAGI, and OpenAI-Agent, fall within the efficient frontier zone, indicating highly efficient and cost-optimal performance. Their average completion time is around 4 seconds, with the average cost ranging from 0.14¢ to 0.15¢. Notably, all four belong to the role-based multi-agent category. Similarly, other frameworks from the 24

same category, such as Swarm, Qwen-Agent, Agent-zero, and BabyAGI, also show efficient average completion times of between 3 and 4 seconds; however, their average cost is slightly higher, at around 0.17¢. CrewAI appears to be the most cost-efficient framework, with an average cost per task of 0.13¢, although its average completion time is slightly longer, at around 5 seconds. In contrast, frameworks such as AutoGen, Agency-Swarm, and PraisonAI exhibit slower performance, with average response times of 9 to 12 seconds per task, while their average cost consumption ranges between 0.14¢ and 0.17¢. In addition, frameworks in the single-agent category vary in terms of both time and cost. For instance, AutoGPT, LangChain, and Semantic Kernel show an average cost per task of between 0.15¢ and 0.16¢, with average completion times ranging from 4 to 6 seconds. However, TaskWeaver demonstrates an unusual pattern: although its average completion time is lower than that of all other frameworks, at approximately 2 seconds, its average cost per task is comparatively higher, at 0.22¢. Frameworks in the hierarchical architecture category, such as Google ADK and MetaGPT, also show variation in time and cost. Google ADK demonstrates more timeefficient performance, with an average completion time of around 4 seconds, although its average cost is slightly higher at 0.18¢. In contrast, MetaGPT records an average completion time of around 6 seconds, while its average cost is lower at 0.16¢. The modular architecture framework, Mastra, demonstrates a relatively low response time but results in the highest average cost per task (0.19¢) among the evaluated frameworks. Similarly, the graph-based framework, LangGraph, demonstrates an average response time of around 6 seconds per task, with an average cost of 0.18¢. Note: During the experiments, several exceptional cases were observed. For example, the Camel framework required an unusually long time to complete tasks and often failed to finish execution. In contrast, Upsonic consumed a significantly larger number of tokens than the other frameworks. The AutoGen and Mastra frameworks resulted in repeated API calls for complex tasks, which significantly increased prompt length and token consumption. A detailed analysis of these four frameworks is provided in Section 4.1.2, where we discuss the possible reasons for their failures based on an examination of their output datasets. Limitation: One limitation of the dataset is that when an execution results in an error or failure, the number of tokens consumed is not recorded. Token information is only available for successful executions (both correct and incorrect), where the input and output token counts are reported. Therefore, in cases of failures or errors, we relied on the OpenAI usage dashboard to estimate the token consumption of the frameworks across the three benchmarks.

Takeaway 2. Most agentic frameworks maintain a suitable balance between reasoning performance, execution time, and cost, with average response times of 4–6 seconds per task and costs between 0.14¢ and 0.18¢ per task. However, higher computational cost and longer execution time do not necessarily result in better reasoning performance, highlighting the importance of efficient framework design and orchestration strategies.

Frameworks

BBH Avg-Time (s)

Total-Time (min)

ARC

Avg-Tok/Cost

Total-Tok/Cost

Avg-Time (s)

Total-Time (min)

GSM8K

Avg-Tok/Cost

Total-Tok/Cost

Avg-Time (s)

Total-Time (min)

Avg-Tok/Cost

Total-Tok/Cost

173 min

696/0.12¢

1567793/$2.86

3.32s

413 min

917/0.16¢

6850974/$14.31

Single-Agent Architecture AutoGPT

4.86s

527 min

1191/0.20¢

7760406/$13.90

4.61s

TaskWeaver Semantic-kernel

0.91s

99 min

1,362/0.23¢

8873320/$15.84

2.05s

77 min

908/0.15¢

2045866/$3.69

1.38s

172 min

2,252/0.39¢

16838592/$29.84

5.28s

573 min

1,176/0.20¢

7659727/$13.72

5.86s

220 min

693/0.12¢

1560815/$2.85

3.85s

479 min

908/0.159¢

Langchain

6792003/$12.25

5.57s

604 min

916/0.16¢

7955670/$14.24

6.08s

228 min

693/0.12¢

1560114/$2.84

3.58s

446 min

916/0.16¢

6847193/$12.35

Role-Based Multi-Agent Architecture BabyAGI

4.39s

476 min

1,162/0.20¢

7572701/$13.57

3.42s

192 min

1,041/0.18¢

2345503/$4.22

3.28s

409 min

911/0.15¢

6813456/$13.11

Autogen

9.31s

1010 min

974/0.17¢

6346009/$11.42

10.34s

388 min

646/0.11¢

1454435/$2.66

9.47s

1179 min

865/0.15¢

6461109/$11.37

14.58s

547 min

781/0.13

1758031/$3.26

Camel

Work stopped due to time limits

CrewAI

6.01s

652 min

781/0.14¢

5085091/$9.21

5.22s

196 min

627/0.10¢

1412455/$2.59

6.53s

813 min

909/0.15¢

6793281/$12.26

SuperAGI

5.06s

549 min

1,167/0.20¢

7604764/$13.63

4.45s

167 min

693/0.12¢

1561377/$2.85

3.55s

442 min

916/0.16¢

6845253/$12.35

Swarm

4.39s

476 min

1,204/0.21¢

7840728/$14.04

4.64s

174 min

697/0.12¢

1569081/$2.86

3.67s

457 min

916/0.16¢

6847178/$12.35

Agency-swarm

16.85s

1829 min

979/0.17¢

6378294/$11.48

11.27s

633 min

901/0.15¢

2029749/$3.67

8.53s

1062 min

924/0.16¢

6910939/$12.46

OpenAI-Agents-Python

4.98s

570 min

1,156/0.20¢

7529003/$13.49

4.91s

184 min

702/0.12¢

1582025/$2.88

3.69s

460 min

921/0.16¢

6888398/$12.42

Agent-zero

2.99s

324 min

1,016/0.17¢

6618517/$11.90

3.42s

192 min

1,042/0.18¢

2346839/$4.22

4.11s

512 min

914/0.16¢

6837492/$12.33

PraisonAI

6.40s

695 min

932/0.16¢

6069847/$10.94

10.29s

386 min

673/0.23¢

1515319/$2.77

5.95s

741 min

808/0.14¢

6039478/$10.94

Qwen-Agent

4.52s

490 min

1,165/0.20¢

7590411/$13.60

4.82s

181 min

697/0.12¢

1570964/$2.86

3.72s

463 min

920/0.16¢

6880026/$12.41

Pydantic-AI

4.95s

537 min

1,141/0.19¢

7432928/$13.32

5.68s

213 min

696/0.12¢

1568631/$2.86

3.64s

453 min

914/0.16¢

6834467/$12.33

ANUS

4.32s

469 min

907/0.15¢

7629524/$13.67

4.59s

172 min

693/0.12¢

1561189/$2.85

3.41s

425 min

907/0.15¢

6782353/$12.24

Hierarchical Architecture MetaGPT

7.85s

852 min

1273/0.22¢

8288503/$14.82

4.42s

166 min

673/0.12¢

1488003/$2.71

5.75s

715 min

873/0.15¢

6523929/$12.64

Google ADK

4.77s

518 min

913/0.15¢

7588520/$13.59

3.85s

216 min

1,047/0.18¢

2357361/$4.24

3.39s

422 min

913/0.15¢

6829228/$12.32

1452/0.25¢

3270423/$5.84

8.79s

1094 min

746/0.13¢

5577395/$10.13

683/0.11¢

1538055/$2.81

3.57s

444 min

921/0.16¢

6889649/$12.42

Modular Component Architecture Upsonic

Work stopped due to high cost

Mastra

Failed

4.88s

274 min Graph-Based Architecture

Langraph

5.18s

562 min

921/0.16¢

7923272/$14.18

4.91s

184 min

Table 7 Trade-offs of agentic frameworks evaluated across the BBH, ARC, and GSM8K benchmarks, showing the average execution time per task, total execution time per benchmark, average cost per task, and total cost per benchmark for each framework.

Agentic frameworks trade-off comparison Benchmarking key frameworks by mean cost and processing speed. Agency-Swarm (33.22%)

Role-Based Multi-Agents Architecture Single-Agent Architecture

AutoGen (39.86%)

Average Mean Time Task (Second)

Hierarchical Architecture Modular Component Architecture

PraisonAI (37.05%)

8 MetaGPT (54.78%) 7

Semantic-Kernel (74.86%) CrewAI (71.08%)

PydanticAI (75.94%)

Mastra (41.08%)

LangChain (74.77%)

OpenAI-Agent (75.94%) SuperAGI (75.04%)

Swarm AutoGPT (75.15%) (75.31%)

Efficiency Frontier (Pareto)

Qwen-Agent (75.58%) Google ADK (75.07%)

BabyAGI (75.34%)

Graph-Based Framework

LangGraph (74.54%)

Agent-Zero (75.42%)

ANUS (75.41%)

TaskWeaver (13.50%)

2 Optimal Efficiency Zone

1 0.11

0.12

0.13

0.14

0.15

0.16

0.17

0.18

0.19

0.20

0.21

0.22

0.23

0.24

0.25

Average Mean Cost Per Task (c)

Fig. 3 Performance–efficiency trade-off between average inference time and average token cost for the evaluated agentic frameworks.

4.3 Multi-Benchmark Consistency (RQ3) Multi-benchmark consistency is examined in this section across all frameworks to evaluate how stable their performance remains in different reasoning tasks. First, we identify the absolute consistency in terms of benchmark performance, which measures the direct differences in accuracy across tasks. The results of this analysis are presented in Section 4.3.1. We then examine relative (scale-adjusted) consistency, which accounts for differences in benchmark difficulty by normalizing performance variations. The findings of this analysis are discussed in Section 4.3.2.

4.3.1 Absolute Consistency As shown in Table 8, we used the mean absolute difference to measure the absolute consistency of each agentic framework across the three benchmarks (BBH, ARC, and GSM8K). Specifically, we computed the pairwise absolute differences between benchmark accuracies (|BBH − ARC|, |BBH − GSM 8K|, and |ARC − GSM 8K|) and combined them to obtain a total deviation score per framework. Lower deviation values indicate greater cross-benchmark stability, whereas higher values suggest performance variability across reasoning domains. The overall results highlight the fact that BBH and ARC show the strongest absolute consistency, with a mean deviation of 10.12, which is much lower than the

Framework |BBH-ARC| |BBH-GSM8K| |ARC-GSM8K| Total Deviation AutoGPT 0.01 46.37 46.36 92.74 TaskWeaver 4.82 5.96 10.78 21.56 Semantic-kernel 1.78 44.67 46.45 92.90 Langchain 0.19 45.47 45.66 91.32 BabyAGI 2.01 45.29 47.30 94.60 Autogen 66.86 19.53 86.39 172.78 CrewAI 12.60 36.09 48.69 97.38 SuperAGI 0.38 45.60 45.98 91.96 Swarm 0.29 46.14 46.43 92.86 Agency-swarm 7.06 3.01 10.07 20.14 OpenAI-Agents-Python 1.39 46.75 48.14 96.28 Agent-zero 3.10 45.27 48.37 96.74 PraisonAI 66.19 15.64 81.83 163.66 Qwen-Agent 1.85 46.36 48.21 96.42 Pydantic-AI 1.96 46.27 48.23 96.46 ANUS 2.35 45.62 47.97 95.94 MetaGPT 18.85 72.75 91.60 183.20 Google ADK 0.34 45.92 45.58 91.84 Langraph 0.24 45.45 45.21 90.90 Mean Deviation 10.12 39.38 49.43 – Table 8 Mean absolute differences between benchmark pairs and total deviation per framework. Lower total deviation indicates higher cross-benchmark stability.

deviations observed between BBH–GSM8K (39.38) and ARC–GSM8K (49.43). This finding indicates that agentic frameworks that perform well on BBH usually achieve similar performance on ARC. Since BBH includes logical, commonsense, hyperbaton, abstract, and general reasoning tasks, and ARC includes scientific reasoning tasks, this suggests that the frameworks generally perform better on these types of reasoning problems. However, GSM8K shows larger deviations compared to both BBH and ARC, indicating that performance on mathematical word problems does not align as closely with performance on broader reasoning benchmarks. This pattern indicates that, while current agentic frameworks are reasonably effective in handling scientific, logical, commonsense, and abstract reasoning tasks, they struggle with mathematical reasoning tasks that require multi-step numerical computation and precise symbolic manipulation. This gap suggests that current frameworks lack mechanisms for reliable step-wise verification, numerical precision control, or specialized mathematical reasoning strategies. More details about the dataset can be found in our replication package [20]. Note: Camel, Upsonic, and Mastra were excluded from the absolute consistency analysis because valid results were not available for all three benchmarks. Since the calculation of mean deviation and total deviation requires complete benchmark results for each framework, these frameworks could not be included due to failure on one or two benchmarks.

# No Framework Mean Accuracy Std Dev CV 1 AutoGPT 75.31 26.77 0.355 2 TaskWeaver 13.50 5.40 0.400 3 Semantic-kernel 74.86 26.32 0.352 4 Langchain 74.77 26.31 0.352 5 BabyAGI 75.34 26.75 0.355 6 Autogen 39.86 45.30 1.137 7 CrewAI 71.08 25.27 0.356 8 SuperAGI 75.04 26.44 0.352 9 Swarm 75.15 26.72 0.356 10 Agency-swarm 33.22 5.17 0.156 11 OpenAI-Agents-Python 75.94 27.40 0.361 12 Agent-zero 75.42 27.08 0.359 13 PraisonAI 37.15 43.44 1.169 14 Qwen-Agent 75.85 27.32 0.360 15 Pydantic-AI 75.94 27.30 0.359 16 ANUS 75.41 27.04 0.359 17 MetaGPT 54.78 48.37 0.883 18 Google ADK 75.07 26.41 0.352 19 Langraph 74.57 26.17 0.351 Table 9 CV across BBH, ARC, and GSM8K. Lower CV indicates higher proportional cross-benchmark stability.

4.3.2 Relative (Scale-Adjusted) Consistency To further quantify the proportional consistency of each framework across the benchmarks, we computed the CV, defined as CV = σµ , where σ represents the standard deviation of accuracies across BBH, ARC, and GSM8K, and µ denotes their mean. As shown in Table 9, frameworks such as Agency-Swarm show the lowest CV (0.156), highlighting strong proportional stability across reasoning domains. In contrast, frameworks such as AutoGen (1.13), PraisonAI (1.16), and MetaGPT (0.88) exhibit substantially higher CV values, reflecting considerable performance variability across the benchmarks. Notably, the CV values for most frameworks fall approximately within the range of 0.35 to 0.36, suggesting that, although their absolute accuracies differ across the benchmarks, their proportional variability remains relatively similar. Overall, the CV analysis provides a scale-adjusted measure of cross-benchmark consistency, supporting the mean absolute deviation analysis by accounting for relative performance variation rather than absolute differences. Note: Although CV provides a measure of relative variability, it should be interpreted cautiously in this study. Frameworks with low mean accuracy may show comparatively high CV values even when their absolute variation across benchmarks is modest. Likewise, a low CV does not necessarily indicate strong overall performance; for example, Agency-Swarm shows a relatively low CV but also low accuracy across the benchmarks. Therefore, CV should be interpreted in conjunction with mean accuracy and standard deviation. Additionally, Camel, Mastra, and Upsonic were excluded from the CV analysis because complete results were not available for all three benchmarks.

Takeaway 3. The results indicate that agentic frameworks show lower and less consistent performance on mathematical reasoning tasks than on other reasoning tasks. BBH and ARC remain relatively close, with a mean deviation of 10.12, whereas GSM8K shows much larger deviations of 39.38 from BBH and 49.43 from ARC. Furthermore, our analysis also highlights that MetaGPT, AutoGen, and PraisonAI have relatively high CV values, indicating greater proportional variation across benchmarks. For most frameworks, however, CV values fall between 0.35 and 0.36.

5 Discussion This section discusses the key findings for each research question and outlines their implications for both academia and industrial practice. It highlights the performance and stability of agentic frameworks, the trade-offs between time efficiency and cost, and the challenges of achieving consistent accuracy across the evaluated benchmarks.

5.1 Performance (RQ1) The most notable pattern in Table 5 is not the identity of the top-ranked framework, but the limited differences in performance between most frameworks. Twelve frameworks, representing single-agent (AutoGPT, Semantic Kernel, LangChain), rolebased multi-agent (BabyAGI, SuperAGI, Swarm, OpenAI Agents Python, Agent Zero, Qwen-Agent, PydanticAI, ANUS), hierarchical (Google ADK), and graph-based (LangGraph) categories, are separated by only 1.4 percentage points in mean accuracy (74.57–75.94%). Their per-benchmark scores are highly similar: approximately 89–91% on BBH, 44% on GSM8K, and approximately 90–93% on ARC. This pattern reflects not a meaningful ranking, but a performance plateau. The frameworks that fall outside the plateau do not fail because of weaker reasoning strategies. Instead, the common thread across every observed failure mode is memory and controlflow discipline. Uncontrolled context growth (Camel), unbounded retry on extraction failure (Upsonic), iterative agent chatter without termination criteria (AutoGen, Mastra)—these are engineering failures in how the frameworks manage state and bind their own behavior, not failures in how they reason. These findings are broadly consistent with prior work. As noted by Vaidhyanathan et al. [3], agentic frameworks offer diverse approaches to solving complex tasks, but many remain at an early stage of maturity. This view is reinforced by the performance variation observed in our results, including the gap between high-performing specialized frameworks and lower-performing but widely adopted frameworks. In particular, limitations in memory mechanisms, error handling, security and privacy protection, and scalable deployment options may help explain why many frameworks still struggle to achieve reliable performance in real-world industrial settings [3, 8]. To support the development of more mature frameworks for software engineering applications, future systems should incorporate stronger memory mechanisms, hybrid reasoning strategies, cost-effective designs, improved error recovery, and greater scalability. Addressing

these limitations will be essential to making agentic AI frameworks more reliable and effective in real-world software engineering [6]. Implications: For researchers, these results point to several directions for improving the design and evaluation of agentic frameworks. (1) For the development of next-generation agentic frameworks, the observed convergence in performance should be interpreted as a challenge rather than rejection. These findings indicate that agent coordination strategies, role allocation mechanisms, and workflow orchestration techniques require further investigation to improve reasoning consistency and reliability. In addition, future frameworks may benefit from hybrid architectures that combine LLM-based reasoning with structured memory systems and symbolic or tool-based reasoning modules. Such developments could reduce computational overhead while enabling more reliable performance on complex real-world tasks. (2) The results also highlight the need for more comprehensive benchmarking methodologies and standardized evaluation frameworks for agentic systems. As LLM-based agent systems continue to evolve, large-scale empirical evaluations will remain essential for guiding framework design and improving reasoning reliability. (3) Finally, the public availability of the source code and replication package enables researchers to reproduce, validate, and extend this work. For example, future studies could explore the performance of agentic frameworks using open-source LLMs and compare their behavior with closedsource models. Future studies could also include newly developed frameworks and benchmarks to examine additional aspects such as privacy, security, robustness, and reliability. These factors are increasingly important for real-world deployment. For practitioners, these results have several practical implications. (1) Industry is increasingly adopting agentic frameworks to develop AI-driven systems and autonomous workflows [59]. In this context, this study provides practical guidance and evidence-based insights for practitioners in selecting suitable frameworks for real-world industrial deployment. (2) The proposed taxonomy enables practitioners to systematically identify frameworks that align with the specific requirements of their target application domain.

5.2 Trade-off (RQ2) The results of this study highlight the trade-off between reasoning capability, computational cost, and execution time required by agentic frameworks to complete reasoning tasks. Our analysis indicates that the majority of frameworks are able to maintain reasonable execution efficiency while achieving acceptable reasoning performance. However, several notable exceptions highlight important inefficiencies: (1) Frameworks such as AutoGen, Agency Swarm, TaskWeaver, and Mastra exhibit higher execution time and cost compared to others. These frameworks also demonstrate lower reasoning accuracy, indicating that higher computational cost does not necessarily lead to better performance. (2) The Camel framework showed high runtime due to uncontrolled context growth, where the framework attempted to construct extremely large memory contexts that repeatedly exceeded the model’s context limit. This led to frequent context overflow handling and message chunking, which increased the number of API calls and token usage while producing minimal outputs. (3) Upsonic consumed significantly higher costs due to repeated LLM calls when outputs failed to match the 31

required extraction format, and accumulated memory and execution traces expanded the prompt context. These observations indicate that framework design, orchestration strategies, and agent coordination mechanisms play a critical role in determining the efficiency and effectiveness of agentic systems. Implications: For researchers, these results point to several important avenues of research: (1) Although many frameworks demonstrate promising reasoning capabilities, the observed trade-offs between performance, cost, and execution time suggest that current agentic architectures are still far from optimal in terms of computational efficiency. Future research should therefore focus on developing more resource-efficient agent orchestration mechanisms, improved task decomposition strategies, and optimized reasoning pipelines that reduce computational overhead while maintaining strong reasoning performance. (2) It is also important to incorporate cost and efficiency metrics into the evaluation of agentic frameworks, as reasoning accuracy alone may not provide a complete understanding of framework performance in real-world scenarios. Overall, achieving a balance between reasoning capability, computational efficiency, and execution time remains a key challenge in designing agentic frameworks. For practitioners, the findings provide several important implications: (1) These findings offer empirical guidance for selecting frameworks that ensure reliable reasoning performance while minimizing computational cost, which is crucial for large-scale deployments where API usage, token consumption, and inference time directly affect operational expenses and system scalability. (2) The results indicate that higher computational cost and longer execution time do not necessarily guarantee better reasoning performance. Therefore, practitioners should carefully evaluate agentic frameworks before large-scale deployment to ensure efficient and reliable system performance. In real-world deployments, organizations must balance reasoning performance with computational efficiency and operational cost, particularly when agentic systems operate at scale. (3) Some frameworks, such as Agent-zero and ANUS, demonstrate better cost efficiency and lower execution time while maintaining competitive reasoning performance, whereas several widely used frameworks require higher execution time and computational resources. This implies that practitioners should carefully consider framework efficiency in addition to popularity when selecting agentic frameworks for real-world deployment. In summary, selecting frameworks that balance accuracy, execution speed, and cost efficiency is critical.

5.3 Consistency (RQ3) The results of this study highlight the consistency of agentic frameworks across different reasoning benchmarks. Specifically, the BBH and ARC benchmarks demonstrate relatively consistent performance, with a mean deviation of 10.12, indicating that frameworks tend to perform similarly across these two reasoning tasks. This suggests that many agentic frameworks are capable of maintaining stable reasoning performance when evaluated on benchmarks that focus on logical, commonsense, and abstract reasoning tasks. However, when comparing the outcomes of BBH and ARC with the GSM8K benchmark, a significantly higher deviation was observed. The mean deviation between BBH and GSM8K was 39.38, while the deviation between ARC

and GSM8K was 49.43. These larger deviations indicate that the reasoning performance of agentic frameworks becomes considerably less consistent when evaluated on mathematical reasoning tasks. This finding further reinforces the observation that current agentic frameworks are generally more reliable in knowledge-based and logical reasoning tasks, but struggle to maintain consistent performance in numerical and multi-step mathematical reasoning problems. Similar observations have been reported in previous studies, for instance [60–64]. These studies suggest that language models tend to perform more consistently on logical and commonsense reasoning tasks, while showing greater variability and lower reliability on mathematical reasoning benchmarks because of limitations in multi-step numerical reasoning and arithmetic generalization. These studies further highlight that LLMs struggle with mathematical reasoning due to their dependence on pattern recognition rather than symbolic reasoning. As a result, they often fail to generalize to unseen arithmetic tasks and frequently make errors in multi-step numerical calculations. Collectively, these findings suggest that current LLMs lack the robust algorithmic reasoning capabilities required for reliable mathematical problem solving [65]. Implications: For researchers, the significant deviation observed in mathematical reasoning benchmarks suggests the need for new agent architectures and reasoning mechanisms that better support structured and multi-step numerical problem solving. For practitioners, these findings suggest that agentic frameworks show stable and predictable performance in applications that primarily involve logical, commonsense, and knowledge-based reasoning tasks, making them suitable for domains such as information analysis, research assistance, and decision support systems. However, the findings also suggest that organizations, particularly in domains that require precise numerical reasoning, financial analysis, or complex quantitative decisionmaking, should apply additional safeguards such as verification mechanisms or hybrid architectures, due to the current limitations of agentic frameworks in mathematical reasoning tasks. Additionally, the findings indicate that organizations should consider task–domain alignment when selecting agentic frameworks, ensuring that the chosen framework demonstrates consistent performance on benchmarks that closely resemble the intended real-world application scenarios.

6 Threats to Validity We follow the guidelines proposed by Wohlin et al. [66] to identify and discuss potential threats to validity. This section deals with four categories: internal, external, construct, and conclusion validity. Internal validity refers to how well the experiments were conducted and the extent to which the resulting data can be trusted. First, a potential threat to internal validity arises from the hardware configuration used to conduct the experiments. All evaluations were executed on a fixed workstation with a specific CPU, memory capacity, storage configuration, and overall compute resources. These hardware characteristics may affect performance-related measurements such as execution time, latency,

throughput, and the occurrence of timeouts or execution instability during agent workflows. As a result, observed differences in efficiency- or cost-related metrics may be partially influenced by the execution environment rather than by the design of the agentic frameworks alone. To reduce this threat, all frameworks were executed within the same controlled hardware and software environment, ensuring consistent compute resources, operating system configuration, and runtime dependencies across experiments. To ensure methodological consistency and reduce other influencing factors, we applied identical framework configurations and decoding parameters across all evaluated agentic frameworks. While this controlled setup supports fair comparison, it may limit the exploration of how different frameworks behave under alternative decoding strategies. Investigating the stability of agentic framework performance under different decoding configurations will therefore be addressed in future work. Another potential threat to internal validity arises from the dependence of agentic frameworks on LLMs, which are known to exhibit non-deterministic behavior and output variability for the same input. Such inconsistency may influence reasoning outcomes and lead to changes in performance across repeated runs. To reduce this threat, each reasoning task was executed multiple times, and the same input was evaluated three times under identical settings. The final results were derived by aggregating the outcomes across these runs (e.g., averaging accuracy scores), reducing random variation. Finally, the last threat to internal validity arises from the use of static benchmarks in this study. Although existing reasoning benchmarks provide controlled and reproducible evaluation environments, they are primarily based on static datasets. Such benchmarks may not fully capture the emerging capabilities of agentic systems, such as adaptive reasoning, continuous learning, and long-horizon task execution. This limitation is also reflected in the choice of benchmark versions. For example, we used the BBH benchmark, whereas a more advanced version, BBEH, was introduced subsequently with more challenging reasoning tasks. However, at the time our experiments were conducted, BBEH was not yet available, and therefore BBH was the most suitable benchmark we could use. As a result, benchmark-based evaluation may only partially represent the behavior of agentic systems in dynamic real-world environments. This limitation is further reinforced by the current lack of standardized benchmarks specifically designed to evaluate agentic systems in complex real-world settings involving dynamic context, long task horizons, and iterative decision-making processes. Future work should therefore incorporate newer and more challenging benchmarks to provide a more comprehensive assessment of agentic frameworks. External validity concerns the extent to which the findings of this study can be generalized beyond the evaluated settings. A potential threat to external validity arises from the evolving nature of agentic frameworks. The selection of frameworks in this study was based on a systematic search conducted on GitHub starting in July 2025. As a result, some agentic frameworks that were released or gained prominence after this period were not included in the evaluation. Therefore, the set of analyzed frameworks may not fully represent the most recent developments in the rapidly evolving agentic framework landscape. In addition, we employed a few-shot prompting approach combined with CoT to evaluate the reasoning capabilities of the agentic

frameworks. However, this approach may not fully capture the external validity of realworld software engineering scenarios. In practice, reasoning is often an iterative and interactive process where developers provide incremental context, respond to model hallucinations, and offer corrective feedback—elements that are not fully represented by the static examples used in a few-shot setting. Therefore, although benchmarkbased experiments enable controlled and reproducible comparisons, they may only approximate the interactive reasoning processes that occur in real-world development environments. Construct validity addresses whether the evaluation accurately measures the intended outcomes of the benchmark-based comparison of agentic frameworks. In this study, a potential threat to construct validity arises from the use of a single LLM (GPT-5.2) to evaluate the agentic frameworks. Agentic frameworks may indicate different behaviors, interaction patterns, or performance characteristics when paired with different LLMs. Although evaluating multiple models could provide a broader understanding of framework behavior, this was not feasible in the current study due to cost constraints. Specifically, the API usage cost for GPT-5.2 in this project exceeded $3154.30. In future work, we plan to extend the evaluation to five additional models, including open-source alternatives, in order to assess whether the observed findings remain consistent across different underlying LLMs. Another potential threat to construct validity arises from the training data limitations of the selected LLM used by agentic frameworks. These models are trained on fixed and potentially outdated datasets, which may limit their ability to reason about recent or evolving information. To mitigate this threat, we enabled external environment support, such as web search tools, allowing agents to access up-to-date information from the internet during task execution. This setup helps reduce the impact of static training data and better reflects realistic agent usage scenarios; however, the extent to which external tools influence reasoning behavior may vary across frameworks. Conclusion validity refers to the extent to which the obtained results support the conclusions derived from the study. In this study, agentic framework executions may be affected by runtime issues such as timeouts, resource exhaustion, or partial failures, especially when complex reasoning tasks are executed or when frameworks involve multiple agent interactions. Such issues may lead to incomplete results for certain frameworks on specific benchmarks, which may introduce bias into the comparative analysis. To mitigate this risk, we applied consistent execution constraints across all frameworks, including limits on token generation and execution time where necessary. All failed or incomplete runs were systematically documented and excluded uniformly across frameworks to avoid skewing the comparisons. Another limitation relates to abnormal execution behavior observed in a small number of frameworks. During the experiments, some frameworks generated excessive context growth, repeated agent interactions, or repeated API retries, which resulted in unusually long runtimes or extremely high token consumption. For example, one framework consumed a very large number of tokens within a short execution period, resulting in substantial API cost before the experiment was manually terminated to prevent uncontrolled resource usage. Moreover, when a framework execution failed or terminated unexpectedly, detailed token usage information was not always recorded

by the logging pipeline. In such cases, token usage was estimated using the OpenAI usage dashboard rather than the experiment logs. As a result, the reported token statistics may slightly underestimate the true computational cost for frameworks that experienced execution failures or incomplete runs. These limitations should therefore be considered when interpreting the comparative efficiency and cost analysis.

7 Conclusion In this study, we systematically selected and evaluated 22 agentic frameworks (see Table 3) to examine their capabilities in reasoning tasks. The study was structured around the three research questions defined in Section 3.1. To address these questions, we conducted experiments on three widely used benchmarks: BBH, ARC, and GSM8K. Our evaluation focused on three key aspects. First, we compared the overall performance of the selected frameworks by analyzing their mean accuracy across the benchmarks. Second, we investigated the trade-offs associated with each framework by measuring inference time and computational cost in order to understand their resource requirements. Third, we analyzed the consistency of framework performance across different benchmark datasets to assess their stability in diverse reasoning scenarios. Based on the experimental results, the main findings of this study are summarized as follows:

• The findings highlight that most of the evaluated frameworks achieved relatively similar mean accuracy across the selected benchmarks. However, a smaller group of frameworks performed worse, primarily because of orchestration-related issues, such as uncontrolled context growth, retry loops, and inefficient agent interactions, rather than limitations in reasoning itself. • The results indicate a clear trade-off between performance, response time, and operational cost. While many frameworks converged within a narrow accuracy range, they still varied in latency and API cost, indicating that framework selection should not be based on accuracy alone but also on efficiency and cost considerations. • The findings further show that mathematical reasoning remains a major limitation across all evaluated frameworks. In particular, performance on GSM8K was lower than on BBH and ARC, suggesting that current agentic frameworks largely inherit the underlying model’s weaknesses in multi-step numerical reasoning rather than overcoming them. • Overall, the study indicates that the practical value of agentic frameworks for reasoning-intensive tasks depends more on orchestration robustness, including memory management, failure handling, and cost control, than on the architectural category alone. The findings of this study provide useful insights for both researchers and practitioners. For researchers, the results highlight several directions for future investigation into the design and evaluation of agentic frameworks. In addition, the source code and replication data are publicly available, enabling other researchers to reproduce our experiments and extend this work by evaluating agentic frameworks with multiple LLMs. Future studies could also incorporate newly developed agentic frameworks and

recently proposed benchmarks to examine additional aspects such as privacy, security, robustness, and reliability, which are becoming increasingly important for real-world deployment of agentic systems. For practitioners, this study offers empirical guidance for selecting appropriate frameworks for industrial applications by considering their reasoning performance, computational trade-offs, and overall consistency across different tasks.

Acknowledgment This project was co-funded by the BF/MAISA/SW kust project (2024–2026) and the ANSE-BF research project (2025–2027), both funded by Business Finland. These projects represent a collaboration between academia and eight leading Finnish companies. They managed all the API costs and other expenses required for the completion of this project.

Data Availability To support the replication and validation of this study, we have publicly released the source code on GitHub [19]. In addition, the complete benchmark results for all evaluated frameworks, including benchmark data files, execution logs, and the Excel file used for data analysis, are provided in Zenodo [20]. The released materials are publicly available to facilitate further research and future extensions of this work.

Declaration of AI Assistance During the preparation of this manuscript, the authors used language models to assist with grammar refinement, sentence restructuring, and formatting improvements. Following the use of these tools, the authors carefully reviewed and revised the content and assume full responsibility for the final version of the publication.

References [1] Liu, J., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., Lou, Y.: Large language model-based agents for software engineering: A survey. ACM Transactions on Software Engineering and Methodology (2024) [2] He, J., Treude, C., Lo, D.: Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology 34(5), 1–30 (2025) [3] Vaidhyanathan, K., Taibi, D.: Agentic ai frameworks under the microscope: What works, what doesn’t. IEEE Software 43(1), 133–138 (2025) [4] Wang, Y., Xu, X., Chen, J., Bi, T., Gu, W., Zheng, Z.: An empirical study of agent developer practices in AI agent frameworks. arXiv preprint arXiv:2512.01939 (2025) 37

[5] Zhao, B., Foo, L.G., Hu, P., Theobalt, C., Rahmani, H., Liu, J.: LLM-based agentic reasoning frameworks: A survey from methods to scenarios. arXiv preprint arXiv:2508.17692 (2025) [6] Wei, T., Li, T.-W., Liu, Z., Ning, X., Yang, Z., Zou, J., Zeng, Z., Qiu, R., Lin, X., Fu, D., et al.: Agentic reasoning for large language models. arXiv preprint arXiv:2601.12538 (2026) [7] Lu, P., Chen, B., Liu, S., Thapa, R., Boen, J., Zou, J.: Octotools: An agentic framework with extensible tools for complex reasoning. arXiv preprint arXiv:2502.11271 (2025) [8] Garg, V.: Designing the mind: How agentic frameworks are shaping the future of AI behavior. Journal of Computer Science and Technology Studies 7(5), 182–193 (2025) [9] Kazemi, M., Fatemi, B., Bansal, H., Palowitch, J., Anastasiou, C., Mehta, S.V., Jain, L.K., Aglietti, V., Jindal, D., Chen, P., et al.: Big-bench extra hard. arXiv preprint arXiv:2502.19187 (2025) [10] Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H.W., Chowdhery, A., Le, Q.V., Chi, E.H., Zhou, D., et al.: Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 (2022) [11] Yu, Z., Zhao, Y., Cohan, A., Zhang, X.-P.: Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation. arXiv preprint arXiv:2412.21199 (2024) [12] Frendi Gunawan, G., Amien, M.: Comprehensive evaluation of large language models on software engineering tasks: A multi-task benchmark. arXiv e-prints, 2602 (2026) [13] Hasan, M.M., Waseem, M., Kemell, K.-K., Rasku, J., Ala-Rantala, J., Abrahamsson, P.: Assessing small language models for code generation: An empirical study with benchmarks. Journal of Systems and Software, 112815 (2026) [14] Yin, Z., Gao, C., Fan, C., Yang, W., Xue, Y., Zhang, L.: A comprehensive empirical evaluation of agent frameworks on code-centric software engineering tasks. arXiv preprint arXiv:2511.00872 (2025) [15] Barbarroxa, R., Gomes, L., Vale, Z.: Benchmarking large language models for multi-agent systems: A comparative analysis of autogen, crewai, and taskweaver. In: International Conference on Practical Applications of Agents and Multi-Agent Systems, pp. 39–48 (2024). Springer [16] Oliveira, M.C.: A Comparative Analysis of LLM-Based Multi-Agent Frameworks. Unpublished manuscript (2025)

[17] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021) [18] Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., Tafjord, O.: Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018) [19] GPT-Laboratory: 22-Agentic-Framework-Comparison-for-Reasoning-Tasksacross-BBH-GSM8K-and-ARC-Benchmarks. https://github.com/GPT-Laborat ory/22-Agentic-Framework-Comparison-for-Reasoning-Tasks-across-BBH-G SM8K-and-ARC-Benchmarks. GitHub repository, accessed April 15, 2026 (2025) [20] Rasheed, Z.: Agentic Frameworks for Reasoning Tasks: An Empirical Study. https: //doi.org/10.5281/zenodo.19593242 . https://doi.org/10.5281/zenodo.19593242 [21] Aratchige, R., Ilmini, W.: Llms working in harmony: A survey on the technological aspects of building effective LLM-based multi agent systems. arXiv preprint arXiv:2504.01963 (2025) [22] Rasheed, Z., Muhammad, W., Kemell, K.-K., Saari, M., Abrahamsson, P.: Llmbased multi-agent systems for code generation: A multi-vocal literature review. Available at SSRN 6328882 (2026) [23] Sun, C., Huang, S., Pompili, D.: Llm-based multi-agent decision-making: Challenges and future directions. IEEE Robotics and Automation Letters (2025) [24] Abou Ali, M., Dornaika, F., Charafeddine, J.: Agentic AI: a comprehensive survey of architectures, applications, and future directions. Artificial Intelligence Review 59(1), 11 (2025) [25] Bandi, A., Kongari, B., Naguru, R., Pasnoor, S., Vilipala, S.V.: The rise of agentic AI: A review of definitions, frameworks, architectures, applications, evaluation metrics, and challenges. Future Internet 17(9), 404 (2025) [26] Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., Wen, J.R.: A survey on the memory mechanism of large language model-based agents. ACM Transactions on Information Systems 43(6), 1–47 (2025) [27] Wang, J., Duan, Z.: Agent AI with langgraph: A modular framework for enhancing machine translation using large language models. arXiv preprint arXiv:2412.03801 (2024) [28] Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., et al.: Autogen: Enabling next-gen LLM applications via multi-agent conversations. In: First Conference on Language Modeling (2024)

[29] Li, G., Hammoud, H., Itani, H., Khizbullin, D., Ghanem, B.: Camel: Communicative agents for mind exploration of LLM society. Advances in Neural Information Processing Systems 36, 51991–52008 (2023) [30] CrewAI Inc.: CrewAI: AI Agent Development Framework. GitHub repository. Available at: https://github.com/crewAIInc/crewAI (accessed February 2026) (2025) [31] TransformerOptimus: SuperAGI: An Open-Source Autonomous AI Agent Framework. GitHub repository. Available at: https://github.com/TransformerOptimus/SuperAGI (accessed February 2026) (2025) [32] Qiao, B., Li, L., Zhang, X., He, S., Kang, Y., Zhang, C., Yang, F., Dong, H., Zhang, J., Wang, L., et al.: Taskweaver: A code-first agent framework. arXiv preprint arXiv:2311.17541 (2023) [33] Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S.K.S., Lin, Z., et al.: Metagpt: Meta programming for a multi-agent collaborative framework. In: The Twelfth International Conference on Learning Representations (2023) [34] Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al.: Chatdev: Communicative agents for software development. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (volume 1: Long Papers), pp. 15174–15186 (2024) [35] Händler, T.: Balancing autonomy and alignment: a multi-dimensional taxonomy for autonomous llm-powered multi-agent architectures. arXiv preprint arXiv:2310.03659 (2023) [36] Patel, V.: Systematic comparison of agentic AI frameworks for scholarly literature processing. Available at SSRN 5484727 (2025) [37] Yang, H., Yue, S., He, Y.: Auto-GPT for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224 (2023) [38] LangChain AI: LangChain. GitHub repository. Available at: https://github.com/langchain-ai/langchain (accessed February 2026) (2023) [39] LlamaIndex Contributors: LlamaIndex. GitHub repository. Available at: https: //github.com/run-llama/llama index (accessed February 2026) (2023) [40] Topsakal, O., Akinci, T.C.: Creating large language model applications utilizing LangChain: A primer on developing LLM apps fast. In: International Conference on Applied Engineering and Natural Sciences, vol. 1, pp. 1050–1056 (2023)

[41] Hasan, M.M., Li, H., Fallahzadeh, E., Rajbahadur, G.K., Adams, B., Hassan, A.E.: An empirical study of testing practices in open source AI agent frameworks and agentic applications. arXiv preprint arXiv:2509.19185 (2025) [42] Google: ADK-Python: An Open-Source Python Toolkit for AI Agent Development. GitHub repository. Available at: https://github.com/google/adk-python (accessed February 2026) (2026) [43] Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N.V., Wiest, O., Zhang, X.: Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680 (2024) [44] Ferrag, M.A., Tihanyi, N., Debbah, M.: From LLM reasoning to autonomous ai agents: A comprehensive review. arXiv preprint arXiv:2504.19678 (2025) [45] Shi, J., Lee, D.K.C., Xu, W., Wang, Y.: Comparative analysis of open-source frameworks for agentic AI systems: Capabilities, design philosophies, and development experiences. World Scientific Annual Review of Fintech, 2450001 (2025) [46] Schardt, C., Adams, M.B., Owens, T., Keitz, S., Fontelo, P.: Utilization of the PICO framework to improve searching pubmed for clinical questions. BMC medical informatics and decision making 7(1), 16 (2007) [47] Derouiche, H., Brahmi, Z., Mazeni, H.: Agentic AI frameworks: Architectures, protocols, and design challenges. arXiv preprint arXiv:2508.10146 (2025) [48] Huang, J., Chang, K.C.-C.: Towards reasoning in large language models: A survey. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 1049–1065 (2023) [49] Zhang, Z., Zhang, A., Li, M., Smola, A.: Automatic chain of thought prompting in large language models. In: The Eleventh International Conference on Learning Representations (2022) [50] OpenAI: Introducing GPT-5.2. https://openai.com/index/introducing-gpt-5-2/. Accessed: 2025-12-xx (2025) [51] Liao, I., Gu, A.: Arc-agi without pretraining. arXiv preprint arXiv:2512.06104 (2025) [52] Chollet, F., Knoop, M., Kamradt, G., Landers, B., Pinkard, H.: ARC-AGI-2: A new challenge for frontier AI reasoning systems. arXiv preprint arXiv:2505.11831 (2025) [53] ARC Prize, Inc.: ARC-AGI-3 Leaderboard. https://arcprize.org/leaderboard. Accessed: 2026-04-12 (2026) [54] Wang, Z., Xia, M., He, L., Chen, H., Liu, Y., Zhu, R., Liang, K., Wu, X., Liu, 41

H., Malladi, S., et al.: Charxiv: Charting gaps in realistic chart understanding in multimodal LLMs. Advances in Neural Information Processing Systems 37, 113569–113697 (2024) [55] Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C.F., Denain, J.-S., Ho, A., Santos, E.d.O., et al.: Frontiermath: A benchmark for evaluating advanced mathematical reasoning in AI. arXiv preprint arXiv:2411.04872 (2024) [56] Zhou, P., Pujara, J., Ren, X., Chen, X., Cheng, H.-T., Le, Q.V., Chi, E., Zhou, D., Mishra, S., Zheng, H.S.: Self-discover: Large language models self-compose reasoning structures. Advances in Neural Information Processing Systems 37, 126032–126058 (2024) [57] Wohlin, C., Höst, M., Henningsson, K.: Empirical research methods in web and software engineering. Web engineering, 409–430 (2006) [58] Blair, E.: A reflexive exploration of two qualitative data coding techniques. Journal of Methods and Measurement in the Social Sciences 6(1), 14–29 (2015) [59] Pati, A.K.: Agentic AI: a comprehensive survey of technologies, applications, and societal implications. IEEE Access (2025) [60] Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., Farajtabar, M.: Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229 (2024) [61] Zhang, Z.: Comprehension without competence: Architectural limits of llms in symbolic computation and reasoning. arXiv preprint arXiv:2507.10624 (2025) [62] Hosseini, A., Sordoni, A., Toyama, D., Courville, A., Agarwal, R.: Not all LLM reasoners are created equal. arXiv preprint arXiv:2410.01748 (2024) [63] Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., Zhou, J.: Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825 (2023) [64] Tan, Z., Geng, H., Yu, X., Zhang, M., Wan, G., Zhou, Y., He, Q., Xue, X., Zhou, H., Fan, Y., et al.: Scaling behaviors of LLM reinforcement learning post-training: An empirical study in mathematical reasoning. arXiv preprint arXiv:2509.25300 (2025) [65] Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D., et al.: A careful examination of large language model performance on grade school arithmetic. Advances in Neural Information Processing Systems 37, 46819–46836 (2024)

[66] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering. Springer, Berlin, Heidelberg (2012)

Agentic Frameworks for Reasoning Tasks: An Empirical Study

Related documents