Agentic AI for Heterogeneous System Design Across Chip, Scale-Up, and Scale-Out Domains

arXiv:2610.23724v1 [cs.AR] 16 Oct 2026

ArchAgent: Agentic AI for Heterogeneous System Design Across Chip, Scale-Up, and Scale-Out Domains Murali Emani

Kashyap Manjusha

Argonne National Laboratory, IL, USA memani@anl.gov

University of Illinois Urbana-Champaign, IL, USA kr58@ieee.org

Abstract—The design space of modern AI/HPC systems has expanded rapidly across compute architectures, including GPUs, dataflow processors, and RISC-V accelerators; memory technologies, including HBM, GDDR, and SRAM; and interconnects such as NVLink, UALink, Infinity Fabric, InfiniBand, Slingshot, and Ultra Ethernet. No single architecture is optimal across LLM training, LLM serving, and AI-coupled HPC workloads, while the resulting combinatorial design space is too large for manual exploration. We present ArchAgent, a hierarchical multiagent framework in which an LLM-based planner decomposes system design into chip, scale-up, and scale-out subproblems, each handled by specialist agents. A deterministic analytical evaluator closes the loop, ensuring that all performance estimates are derived from analytical models rather than generated by the LLM. On a mixed AI+HPC workload under a 1 MW power budget, ArchAgent discovers heterogeneous designs that achieve a 1.93× weighted geometric mean speedup over a homogeneous GPU baseline, while requiring only 18 evaluations compared with 100 evaluations for hill climbing, which achieves 1.65× speedup. Index Terms—agentic AI, LLM agents, heterogeneous systems, design-space exploration, interconnects, co-design

I. I NTRODUCTION The hardware substrate for large-scale AI and HPC has fragmented along three axes. Compute spans SIMT GPUs (NVIDIA H100/H200 and Blackwell B200/GB300; AMD Instinct MI300X/MI325X/MI355X), matrix-engine ASICs with integrated networking (AWS Trainium2 [1]), systolic-array TPUs (Google TPU v5p, Trillium, and Ironwood [2]), SRAMresident dataflow processors (Groq’s LPU [3], Cerebras’ wafer-scale WSE-3 [4]), reconfigurable dataflow units (SambaNova SN40L [5]), near-memory compute architectures (dMatrix Corsair) and RISC-V native accelerators such as Tenstorrent’s Blackhole [6]. Memory spans stacked HBM (HBM2e through HBM3e, with HBM4 emerging), GDDR6/GDDR7, LPDDR5X coupled to host CPUs (e.g., Grace), on-chip SRAM ranging from hundreds of megabytes (LPU) to tens of gigabytes (WSE-3), tiered SRAM–HBM–DDR hierarchies (SN40L), and CXL-attached pooled memory [7]; these tiers differ by orders of magnitude in capacity, bandwidth, and cost per byte. Interconnects have fractured into proprietary and emerging scale-up domains (NVLink, AMD Infinity Fabric, Google’s ICI with optical circuit switching [2], AWS NeuronLink, and the open UALink [8] standard), Ethernetnative designs that blur the scale-up/scale-out boundary (Ten-

SC26 Workshops, November 15-20, 2026, Chicago, Illinois, USA 979-8-3195-1221-5/26/$31.00 ©2026 IEEE

storrent’s Ethernet mesh), and competing scale-out families (InfiniBand NDR/XDR, HPE Slingshot-11 [9], RoCEv2-based fabrics such as NVIDIA Spectrum-X, and Ultra Ethernet [10]). Together, these choices create a heterogeneous substrate in which compute architecture, memory hierarchy, and communication fabric are increasingly co-designed rather than independently selected. Workloads have diverged just as sharply. LLM training is dominated by GEMM and all-reduce operations, prefill is primarily compute-bound, decode is constrained by memory bandwidth, and AI-coupled HPC workloads alternate between FP64 solver phases and dense AI computation. As a result, no single chip–fabric combination performs well across all phases, motivating heterogeneous system compositions. However, the resulting design space is highly combinatorial, spanning chip types, partition sizes, per-partition interconnects, and workload assignments. This complexity makes the space difficult to navigate through human intuition alone, while conventional black-box auto-tuners often struggle because they lack the architectural priors needed to reason about these system-level trade-offs. LLM-based agents are well suited to this problem because they can leverage architectural priors when navigating the design space. For example, an agent can recognize that decode is typically bandwidth-bound, identify SRAM-resident dataflow architectures as a natural fit, and independently propose prefill/decode disaggregation. The risk of hallucinated performance claims can be addressed structurally: all performance estimates are produced by a deterministic analytical evaluator invoked as a tool, rather than generated by the model itself. In this design, the LLM is responsible for reasoning over architectural choices, while the evaluator provides the quantitative performance evidence used to assess them. In this work, our primary contributions are: (1) ArchAgent, a hierarchical multi-agent framework in which a planner decomposes system design into chip, scale-up, and scaleout subproblems, which are then handled by specialist subagents in a closed loop with a validating evaluator. (2) An open archetype library covering GPU, dataflow, and RISC-V compute architectures, together with ten fabric models and a calibrated analytical cost model based on a roofline chip model and α–β collective communication model, spanning LLM

Planner Agent (LLM)

Workload profile

Constraints

decompose objective · propose design JSON integrate specialist advice mutate on bottleneck feedback

LLM training / serving, AI–HPC kernels, weights

iso-power (1 MW), iso-cost budget

Chip Agent

Scale-Up Agent

Scale-Out Agent

GPU · dataflow · RISC-V · memory roofline reasoning

NVLink · UALink · IF · Eth. mesh domain sizing, α–β reasoning

IB · Slingshot · UEC topology collective mapping

Evaluator (deterministic tool)

schema validation → analytical cost model → scoring score · per-workload ratios · bottleneck attribution

Archetype library

Trace

chips + fabrics (specs only)

MLflow / JSON all evals

score + bottlenecks

design JSON

specialist critiques

Invariant: every performance number originates from the evaluator tool — never from an LLM. Fig. 1. ArchAgent overview. The planner proposes designs as JSON; chip, scale-up, and scale-out sub-agents critique their subsystems; a deterministic evaluator validates and scores every candidate, returning bottleneck attribution.

training, disaggregated serving, and four AI+HPC kernels. (3) A mixed-workload evaluation methodology under both isopower and iso-cost constraints, including a same-generation control that isolates the benefits of heterogeneity from those of process-generation improvements. (4) A systematic evaluation showing that agent-designed heterogeneous systems can outperform homogeneous procurement, complemented by searchefficiency and ablation studies that isolate the contributions of the agent hierarchy and evaluator feedback.

through effective utilization factors and focus on system-level performance across heterogeneous compute and interconnects. To our knowledge, ArchAgent is the first framework to pair hierarchical LLM agents with a deterministic evaluator to jointly explore chip, scale-up, and scale-out design, auditable configurations together with the reasoning behind key design choices. III. T HE A RCH AGENT F RAMEWORK A. Design Representation

II. R ELATED W ORK a) Architecture DSE: Prior work has extensively explored automated accelerator design-space exploration. ArchGym [11] uses RL, GAMMA [12] employs genetic search, and AutoDSE [13] applies bottleneck-guided search; related systems such as Timeloop/Accelergy [14] and MAESTRO [15] explore accelerator mappings and dataflows. These approaches generally treat the evaluator as a black box and optimize within a predefined parameter space, without explicitly reasoning about architectural relationships that transfer across workloads. ArchAgent, in contrast, uses an LLM to exploit architectural priors while coordinating system-level search. b) LLMs for hardware: Recent work applies LLMs to hardware development, including ChipNeMo [16] for chipdesign workflows and SpecLLM [17] and RTL-oriented systems [18], [19] for specification and RTL generation. These efforts primarily target individual chips or design artifacts; we instead consider full-system composition, jointly selecting compute architectures, scale-up fabrics, and scale-out networks. c) Performance modeling: Our evaluator follows the analytical modeling tradition of Calculon [20] and LLMCompass [21] for transformer performance and ASTRA-sim [22] for hierarchical collective communication. We abstract intrachip mapping, addressed by Timeloop/Accelergy [14], [23],

B. Design Representation We represent each candidate system as a JSON document containing a list of partitions. Each partition specifies a chip archetype, a scale-up fabric, a scale-out fabric, a fraction of the facility power budget, and a workload role (general, prefill, decode, solver, or ai). Workload roles allow the evaluator to model disaggregated execution without requiring the agent to explicitly manage placement. For example, a serving configuration can assign prefill to a GPU partition and decode to an SRAM-resident dataflow partition. Before any performance evaluation, a schema validator checks the design for basic feasibility, including fabric bandwidth that exceeds the chip’s available port capacity, power allocations that exceed the facility budget, and insufficient FP64 capability for HPC workloads. Validation errors are returned to the agent as structured feedback, preventing evaluation cycles from being spent on invalid or unsupported configurations. C. Agent Hierarchy The planner agent proposes and iteratively revises complete system designs. For each candidate, it must invoke the evaluate_design tool, which provides the planner with the hardware catalog, design schema, and optimization objective, but not with predicted performance improvements. Three specialist sub-agents provide additional guidance during

the search: a chip agent reasons about compute archetypes, roofline constraints, and memory hierarchies; a scale-up agent applies α–β reasoning to intra-domain communication and domain sizing; and a scale-out agent analyzes collective communication and system topology. Their recommendations are returned as concise parameter-level guidance and attached to the corresponding evaluation result. The evaluator is a deterministic analytical model. Given a valid configuration, it returns the mixed-workload score, per-workload performance relative to the reference system, and an attribution of the dominant bottleneck among compute throughput, memory bandwidth, fabric latency, and bisection bandwidth. A key structural invariant is that the LLM agents never generate or assert performance estimates themselves; all performance quantities originate from the deterministic evaluator. This separation makes the search process auditable: the complete sequence of candidate designs, evaluation results, and agent recommendations can be recorded and inspected. D. Analytical Cost Model We model conventional accelerator performance using a roofline formulation with dtype-specific peak FLOP rates, machine-utilization factors (MFU), and memory-bandwidth efficiency factors. For dataflow architectures whose sharded weights fit within on-chip SRAM, we instead model pipelined layer sharding, accounting for per-stage computation and point-to-point activation transfers rather than tensor-parallel all-reduce. This captures the execution model of SRAMresident architectures such as Groq and Cerebras. Communication costs are modeled using the standard α–β formulation [24]. The model supports ring and hierarchical all-reduce, with communication first within a domain and then across domains, as well as all-to-all and point-to-point transfers. For LLM training, we combine tensor, pipeline, and data parallelism (TP/PP/DP)[25], including pipeline-bubble overhead and a ZeRO-style[26] memory-feasibility check. For serving, we model disaggregated prefill and decode [27] and assign each phase to partitions according to their workload roles. The HPC suite consists of four workloads chosen to exercise distinct compute and communication regimes: a 27-point FP64 stencil dominated by halo exchange, an HPCG-like conjugategradient (CG) kernel with latency-sensitive scalar all-reduce, a 3D FFT dominated by all-to-all communication and global bisection bandwidth, and a coupled AI-surrogate–solver loop in which successive phases can be pipelined across heterogeneous partitions. Section IV-B describes calibration of the analytical model against published MLPerf results [28] and vendor-reported measurements. IV. E VALUATION A. Methodology All systems are provisioned under an iso-power constraint of 1,MW at the facility level, including the power consumed by fabric ports. Results under an iso-cost constraint, based on publicly available list prices, are reported

TABLE I B EST MIXED - WORKLOAD SCORE BY SEARCH METHOD (1 MW ISO - POWER ; REFERENCE : HOMOGENEOUS H100+NVL INK 4+IB-NDR = 1.000).

Method

Planner LLM

Evals

Runs

Score (mean±sd)

ArchAgent ArchAgent ArchAgent Hill climb Hill climb Random

gpt-oss-120b gpt-oss-120b gemma-4-31B-it — — —

100 50 50 100 50 100

1 2 3 1 1 1

1.93 1.71±0.16 0.87±0.15 1.65 1.61 1.51

as a secondary analysis. We optimize a weighted geometric mean of per-workload speedups relative to a homogeneous H100+NVLink4+InfiniBand-NDR reference system, whose score is normalized to 1.0. The workload weights are 0.25, 0.30, 0.10, 0.10, 0.10, 0.15 for training, serving, stencil, HPCG, FFT, and the coupled workload, respectively, representing a mixed AI+HPC facility profile. A configuration receives a score of zero if any workload cannot be executed. To isolate the benefits of architectural heterogeneity from those of newer process generations, the headline comparison restricts all candidates to a common hardware generation. This constraint is necessary because allowing unrestricted generational mixing can make the optimization trivial: in preliminary searches, the optimizer selected B200 accelerators throughout, with the performance advantage attributable primarily to the newer generation rather than to heterogeneous system design. Random search and hill climbing are evaluated under the same number of evaluator calls as ArchAgent. B. Model Calibration An automated verification suite comprising 32 checks validates the model at three levels: specification invariants against vendor datasheets, asymptotic identities for collective communication models, and end-to-end performance predictions against published measurements. As representative calibration points, the model predicts 910 tok/s/GPU for BF16 training of Llama-70B on a 1,024-H100 cluster, consistent with the approximately 750–1,100 tok/s/GPU reported by MLPerf-class runs [28]. For FP8 decode at TP4 with batch size 32, the model predicts 109 tok/s/user, within the approximately 60– 150 tok/s/user range reported for TRT-LLM. For HPCG, the model produces iteration times on the millisecond scale, consistent with the memory-bandwidth-limited behavior reported for comparable systems. C. Heterogeneous vs Homogeneous (iso-power) Table I summarizes design quality across search methods and planner models. The best design identified by ArchAgent (Fig.2) emerged from the planner’s response to evaluatorreported bottlenecks rather than from random exploration. The resulting configuration improves performance across all workloads, achieving 1.72× for training, 1.67× for serving, 4.76× for HPCG, and 2.61× for the coupled AI+HPC workload; FFT, at 1.37×, is the binding constraint.

exceed a score of 1.5. The faster convergence is consistent with the architectural structure encoded in the agent hierarchy: the planner’s early proposals already incorporate multi-partition configurations and workload-role assignments, whereas hill climbing must discover these structural choices through local mutations. E. Implications for Future Systems

Fig. 2. Per-workload speedup of the best agent-discovered design (score 1.93) over the homogeneous reference at iso-power.

Fig. 3. Best score vs. evaluations (100-eval budget): ArchAgent (gpt-oss120b) vs. hill climb and random search.

Planner capability also affects search outcomes. Under identical evaluation budgets and guardrails, gemma-4-31Bit converges to a score of 0.87, below the homogeneous reference, and produces 7 schema-invalid and 4 otherwise infeasible designs in 50 evaluations. In comparison, gpt-oss120b produces 5 schema-invalid and no infeasible designs under the same conditions. These results suggest that the framework can also serve as a controlled probe of how model capability affects architectural search, although the observed differences reflect the complete agent-and-model system rather than model capability alone. The hardware used to host the planner affects only search wall-clock time, as expected given the deterministic evaluator. Finally, the reported designs benefit in part from the generational advantage of B200 over the H100 reference. The same-generation control experiment, reported in the artifact, isolates the contribution of architectural heterogeneity from this generational effect. D. Search Efficiency Fig. 3 shows convergence under a matched 100-evaluation budget. ArchAgent surpasses hill climbing’s final score of 1.65 by evaluation 18 and reaches its best score of 1.93 by approximately evaluation 36, after which no further improvement is observed. Random search requires 48 evaluations to

The remaining performance gaps in the winning configurations consistently concentrate along two axes: all-to-all communication and FP64 memory bandwidth. In particular, FFT performance remains constrained by global bisection bandwidth, while the stencil workload is limited by FP64 memory throughput. These results suggest three directions for future mixed AI+HPC systems. (i) Scale-out fabrics would benefit from higher injection bandwidth, with ≥200GB/s per endpoint as a representative target, together with hardware support for in-network collective and all-to-all communication. (ii) Memory systems with HBM4-class bandwidth (≥10TB/s per socket), or accelerators that combine high memory bandwidth with substantial FP64 capability through dataflow or near-memory compute, could reduce the performance gap on bandwidth-bound HPC workloads. (iii) System procurement could be organized around workload roles rather than a single system. Across our search results, the optimizer frequently allocates 2–10% of the power budget to specialized decode or solver partitions, suggesting that a relatively small heterogeneous component can complement a predominantly AIoriented system and account for a substantial portion of the improvement over homogeneous configurations. V. D ISCUSSION AND L IMITATIONS Our conclusions are bounded by the fidelity of the analytical evaluator. The current model does not capture switch-level congestion, uses fixed factors for compute/communication overlap, and does not yet account for KV-cache transfer in disaggregated serving. These simplifications can affect the relative evaluation of communication-intensive configurations. The wafer-scale archetype is approximated as a set of composable slices, which may underestimate intra-wafer bandwidth and overestimate inter-slice communication performance. Similarly, the iso-cost analysis inherits uncertainty from publicly reported list prices. These limitations concern the evaluator rather than the framework: the agent loop is evaluator-agnostic, allowing the analytical model to be replaced or augmented by higherfidelity models. As a next step, we plan to validate selected top-ranked configurations with simulator-in-the-loop evaluation, using ASTRA-sim [22] for collective communication and Timeloop [14]-class tools for chip-level mapping, while keeping such expensive evaluations outside the primary search loop. On the agent side, LLM stochasticity motivates reporting results across multiple random seeds. Search outcomes also depend on planner capability. We view this dependence as an experimentally useful property of the framework: because

performance estimates are generated exclusively by the deterministic evaluator, differences in planning capability can be reflected directly in the quality of the resulting configurations under a fixed evaluation environment. This makes ArchAgent a potential benchmark for studying open-weight models on hardware-design search tasks. Ablations isolating the contributions of the specialist-agent hierarchy and bottleneckattribution feedback remain future work. VI. C ONCLUSION In this paper, we present ArchAgent, a framework that combines hierarchical LLM agents with a deterministic evaluator to explore heterogeneous system designs. Our results show that this approach can search complex architectural design spaces more efficiently than stochastic baselines while identifying higher-quality configurations. Under a mixed AI+HPC workload profile and an iso-power constraint, ArchAgent achieves a 1.93× improvement over the homogeneous GPU reference and reaches a design quality exceeding hill climbing’s final result within 18 evaluations. The configurations discovered by ArchAgent also exhibit a consistent architectural pattern: a large GPU-like partition complemented by smaller, role-specialized dataflow and FP64 partitions. This result suggests that future computing facilities may benefit from composing hardware around workload roles rather than relying exclusively on homogeneous accelerator fleets. At the component level, the remaining performance bottlenecks highlight two capabilities that are less prominent in current AI-oriented accelerators: global bisection bandwidth and FP64 memory bandwidth. These results point to both capabilities as promising directions for future accelerator and system design. VII. ACKNOWLEDGEMENT Argonne National Laboratory’s work was supported by the U.S. Department of Energy, Advanced Scientific Computing Research (ASCR), under contract DE-AC02-06CH11357. Claude was used to generate Figure 1 and to assist with refining the manuscript text. R EFERENCES [1] Amazon Web Services, “AWS Trainium2: Architecture and NeuronLink interconnect,” AWS Neuron Documentation, 2024. [2] N. Jouppi, G. Kurian, S. Li, P. Ma, R. Nagarajan, L. Nai et al., “TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings,” in Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA), 2023. [3] D. Abts, G. Kimmell, A. C. Ling, J. Kim, M. Boyd, A. Bitar, S. Parmar, I. Ahmed, R. DiCecco, D. Han, J. Thompson, M. Bye, J. Hwang, J. Fowers, P. Lillian, A. Murthy, E. Mehtabuddin, C. Tekur, T. Sohmers, K. Kang, S. Maresh, and J. Ross, “A software-defined tensor streaming multiprocessor for large-scale machine learning,” in Proceedings of the 49th Annual International Symposium on Computer Architecture (ISCA), 2022, pp. 567–580. [4] S. Lie, “The cerebras wafer-scale engine: Cluster-scale deep learning acceleration on a single chip,” in Hot Chips 35, 2023. [5] R. Prabhakar, R. Sivaramakrishnan, D. Gandhi et al., “SambaNova SN40L: Scaling the AI memory wall with dataflow and composition of experts,” in Proceedings of the 57th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2024.

[6] J. Vasiljevic and D. Capalija, “Blackhole & tt-metalium: The standalone ai computer and its programming model,” in Hot Chips 36, 2024. [7] CXL Consortium, “Compute Express Link (CXL) specification, revision 3.1,” 2023. [8] UALink Consortium, “Ualink 1.0 specification,” 2025. [9] D. De Sensi, S. Di Girolamo, K. H. McMahon, D. Roweth, and T. Hoefler, “An in-depth analysis of the slingshot interconnect,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2020. [10] Ultra Ethernet Consortium, “Ultra ethernet specification 1.0,” 2025. [11] S. Krishnan, A. Yazdanbaksh, S. Prakash, J. Jabbour, I. Uchendu, S. Ghosh, B. Boroujerdian, D. Richins, D. Tripathy, A. Faust, and V. J. Reddi, “ArchGym: An open-source gymnasium for machine learning assisted architecture design,” in Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA), 2023. [12] S.-C. Kao and T. Krishna, “GAMMA: Automating the HW mapping of DNN models on accelerators via genetic algorithm,” in IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2020. [13] A. Sohrabizadeh, C. H. Yu, M. Gao, and J. Cong, “AutoDSE: Enabling software programmers to design efficient FPGA accelerators,” ACM Transactions on Design Automation of Electronic Systems, vol. 27, no. 4, pp. 32:1–32:27, 2022. [14] A. Parashar, P. Raina, Y. S. Shao, Y.-H. Chen, V. A. Ying, A. Mukkara, R. Venkatesan, B. Khailany, S. W. Keckler, and J. Emer, “Timeloop: A systematic approach to DNN accelerator evaluation,” in 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2019, pp. 304–315. [15] H. Kwon, P. Chatarasi, M. Pellauer, A. Parashar, V. Sarkar, and T. Krishna, “Understanding reuse, performance, and hardware cost of DNN dataflow: A data-centric approach,” in Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2019, pp. 754–768. [16] M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu, B. Bhaskaran, B. Catanzaro, A. Chaudhuri, S. Clay, B. Dally, L. Dang, P. Deshpande, S. Dhodhi, S. Halepete, E. Hill, J. Hu, S. Jain, A. Jindal, B. Khailany, G. Kokai, K. Kunal, X. Li, C. Lind, H. Liu, S. Oberman, S. Omar, G. Pasandi, S. Pratty, J. Raiman, A. Sarkar, Z. Shao, H. Sun, P. P. Suthar, V. Tej, W. Turner, K. Xu, and H. Ren, “ChipNeMo: Domain-adapted LLMs for chip design,” arXiv preprint arXiv:2311.00176, 2023. [17] M. Li, W. Fang, Q. Zhang, and Z. Xie, “SpecLLM: Exploring generation and review of VLSI design specification with large language model,” arXiv preprint arXiv:2401.13266, 2024. [18] Y. Lu, S. Liu, Q. Zhang, and Z. Xie, “RTLLM: An open-source benchmark for design RTL generation with large language model,” in Proceedings of the 29th Asia and South Pacific Design Automation Conference (ASP-DAC), 2024. [19] S. Liu, W. Fang, Y. Lu, Q. Zhang, H. Zhang, and Z. Xie, “RTLCoder: Outperforming GPT-3.5 in design RTL generation with our open-source dataset and lightweight solution,” in IEEE International Workshop on LLM-Aided Design (LAD), 2024. [20] M. Isaev, N. McDonald, L. Dennison, and R. Vuduc, “Calculon: A methodology and tool for high-level codesign of systems and large language models,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2023. [21] H. Zhang, A. Ning, R. B. Prabhakar, and D. Wentzlaff, “LLMCompass: Enabling efficient hardware design for large language model inference,” in Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA), 2024. [22] W. Won, T. Heo, S. Rashidi, S. Sridharan, S. Srinivasan, and T. Krishna, “ASTRA-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale,” in 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2023, pp. 283–294. [23] Y. N. Wu, J. S. Emer, and V. Sze, “Accelergy: An architecture-level energy estimation methodology for accelerator designs,” in Proceedings of the IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2019. [24] R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of collective communication operations in mpich,” International Journal of High Performance Computing Applications, vol. 19, no. 1, pp. 49–66, 2005. [25] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro,

A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on gpu clusters using megatron-lm,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2021. [26] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “ZeRO: Memory optimizations toward training trillion parameter models,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2020. [27] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating prefill and decoding for goodputoptimized large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 193–210. [28] P. Mattson, C. Cheng, C. Coleman, G. Diamos, P. Micikevicius, D. A. Patterson, H. Tang, G.-Y. Wei, P. Bailis, V. Bittorf, D. Brooks, D. Chen, D. Dutta, U. Gupta, K. M. Hazelwood, A. Hock, X. Huang, A. Ike, B. Jia, D. Kang, D. Kanter, N. Kumar, J. Liao, G. Ma, D. Narayanan, T. Oguntebi, G. Pekhimenko, L. Pentecost, V. J. Reddi, T. Robie, T. St. John, T. Tabaru, C.-J. Wu, L. Xu, M. Yamazaki, C. Young, and M. Zaharia, “MLPerf training benchmark,” in Proceedings of Machine Learning and Systems (MLSys), 2020, originally released as arXiv:1910.01500 in 2019.

A PPENDIX A. Discovered System Configurations We report the best design at each level of the pipeline: the agent’s design artifact, (the exact JSON it emitted, reproducible from the released trace), the deployment specification implied by iso-power provisioning, and the realized workload mapping that is, the placement and parallelism the evaluator selects when executing each workload on the design. All numbers below are for the best gpt-oss-120b run (100-evaluation budget, score 1.93). B. Design Artifact (agent output) The agent’s final design artifact is shown below. It specifies the hardware partitions, their roles, power allocations, and interconnects. {"name":"Iter33", "partitions":[ {"chip":"B200","role":"general","power_frac":.98, "scaleup":"NVLink5","scaleout":"IB-XDR"}, {"chip":"WSE3s","role":"decode","power_frac":.02, "scaleup":"UALink","scaleout":"UEC"}]}

C. Deployment Specification (1,MW iso-power) Under the 1 MW iso-power constraint, the design yields 896 B200 accelerators organized into twelve NVL72 domains and four WSE-3 slices in a single UALink domain. The B200 and WSE-3 partitions use InfiniBand XDR and Ultra Ethernet 800, respectively, for scale-out. Table II summarizes the resulting deployment. The role labels are used as inputs to workload mapping but do not impose hard placement constraints. TABLE II D ESIGN PROPOSED

Role (agent label) Chip Scale-up Scale-out

Partition 1

Partition 2

general B200 × 896 NVLink5, 72-GPU domains (12 NVL72 racks) InfiniBand XDR 100 GB/s/endpoint

decode WSE-3 slice × 4 UALink, 4-slice domain (1 node) Ultra Ethernet 800 100 GB/s/endpoint

D. Realized Workload Mapping The evaluator maps each workload to the best role-eligible partition and searches parallelism per workload; Table III shows the resulting execution configuration i.e., what an operator would actually deploy. TABLE III E XECUTION CONFIGURATION SELECTED BY THE EVALUATOR FOR EACH WORKLOAD ON DESIGN I T E R 33 (L LAMA -70B FOR LLM WORKLOADS ). Workload

Partition

Configuration

Result

Training

B200

1.72× ref.

Serving: prefill

B200

Serving: decode

B200

best TP/PP/DP mapping, global batch 1024 TP1, batch 8, 896 replica groups TP1, batch 128, 896 replicas, FP8,

Stencil / HPCG / FFT Coupled AI+HPC: solver phase Coupled AI+HPC: AI phase

B200 B200

FP64, all 896 GPUs FP64 stencil phase

WSE-3

FNO surrogate, SRAM-resident

12.96M tok/s 4.71M tok/s (41 tok/s/user, mem-BW bound) 1.44 / 4.76 / 1.37× 2.61× (pipelined)

E. Role Labels vs. Realized Placement The role assigned by the agent does not necessarily determine the partition used during execution. In Iter33, the wafer-scale partition is labeled decode, but the evaluator assigns both serving stages to the B200 partition. With only four wafer slices available, the 70B FP8 model cannot be accommodated in on-slice SRAM: sharding requires 35,GB per slice, compared with 5.5,GB of available SRAM. Consequently, SRAM-resident decode is not feasible at this budget, and the evaluator selects the B200 partition, which provides the required capacity through HBM3e. The WSE-3 partition is instead used for the AI-surrogate phase of the coupled AI+HPC workload. For the FNO proxy, its dense FP16 throughput is higher than the effective throughput of the B200 partition, and pipelining the AI and solver phases produces a 2.61× coupled speedup. Thus, role labels serve as guidance to the workload mapper rather than as hard placement constraints. The realized mapping reported in Table III reflects the actual execution configuration, while both the agent-specified roles and the resulting placement can be reconstructed from the audit trail. At larger power budgets (≥ 8 wafer slices per replica), SRAMresident decode becomes feasible, causing the decode role and the realized placement to coincide.

Related documents

• Toward System-of-Systems Integration for Composable Cloud-HPC-Edge AI Platforms#1118139
• Toward System-of-Systems Integration for Composable Cloud-HPC-Edge AI Platforms#1108727
• Throughput-Optimized Networks at Scale#1131552
• SyclKittens: A Tile Programming Model for Programmers and Coding Agents on Intel GPUs#1199551
• SyclKittens: A Tile Programming Model for Programmers and Coding Agents on Intel GPUs#1194409