Conceptio › Archive › arXiv CS

arXiv CSopen access

Mitigating the Memory Bottleneck with Machine Learning-Driven and Data-Aware Microarchitectural Techniques

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

kernel operatingsystems virtualization

operating systems, kernel, virtualization

DISS. ETH NO. - 31461

MITIGATING THE MEMORY BOTTLENECK WITH MACHINE LEARNING-DRIVEN AND DATA-AWARE

arXiv:2603.07683v1 [cs.AR] 8 Mar 2026

MICROARCHITECTURAL TECHNIQUES A thesis submitted to attain the degree of DOCTOR OF SCIENCES (Dr. sc. ETH Zürich)

presented by RAHUL BERA born on 12.09.1992

accepted on the recommendation of Prof. Dr. Onur Mutlu, examiner Dr. Aamer Jaleel, co-examiner Prof. Dr. Boris Grot, co-examiner Chris Wilkerson, co-examiner Prof. Dr. Daniel Jiménez, co-examiner Dr. Pradip Bose, co-examiner

2025

In the memory of my beloved baba, Mr. Rebati Raman Bera (1960 - 2020)

Acknowledgments This dissertation is the outcome of more than half a decade’s worth of effort, which would not be possible without the help, advice, motivation, and encouragement from numerous people in my life. While it may not be possible to acknowledge all of them in this short section, I am truly thankful to everyone who shaped my PhD journey, knowingly or unknowingly. First and foremost, I am thankful to my advisor, Prof. Onur Mutlu, for believing in me and providing the opportunity, resources, and continuous enthusiasm during the entire course of the PhD. Without his invaluable guidance and deep expertise, this dissertation would not have been possible. The scientific rigor he instilled in me during this journey has not only shaped me as a researcher, but also as a problem solver in real life. I would extend my sincere gratitude to all other mentors in my life who had contributed profoundly to making me what I am today. My interest in computer architecture started from the lectures of Prof. Atal Chaudhuri at Jadavpur University. Then Prof. Mainak Chaudhuri at IIT Kanpur introduced me to the cutting-edge research in computer architecture during my master’s studies. During this period, I also had the opportunity to work on challenging problems in industry under the mentorship of Dr. Kanishka Lahiri at AMD. Lastly, I will be forever grateful to my Intel mentors—Anant V. Nori, Dr. Shankar Balachandran, and Sreenivas Subramoney—for their invaluable guidance and mentorship before, during, and after the PhD. In particular, I am especially thankful to Anant who not only helped me grow professionally as my mentor, but also in my personal life as an elder brother. I also thank my doctoral exam committee members—Dr. Aamer Jaleel, Prof. Boris Grot, Chris Wilkerson, Prof. Daniel Jimenez, and Dr. Pradip Bose—for taking time out of their busy schedules to provide thoughtful feedback and constructive criticism that immensely helped strengthen this dissertation. I thank Prof. Kaveh Razavi for chairing my doctoral exam. My sincere gratitude goes to all the funding partners: ETH Future Computing Lab (EFCL), Futurewei, Google, Huawei, Intel, Microsoft, Semiconductor Research Corporation (SRC), and VMware. Their support was instrumental in making this dissertation possible. The role of the SAFARI Research Group has been quintessential during my PhD journey. I thank all current and past SAFARI members for providing such a unique, stimulating, yet friendly environment for doing research. I am especially indebted to my PhD buddies— Konstantinos Kanellopoulos and Nika Mansouri Ghiasi—for always being there to share the many joys, sorrows, and frustrations along this journey. Their warm friendship, solidarity, and support helped me to function as a human during this long marathon, which otherwise would have surely driven me to despair. I feel deeply fortunate that the three of us embarked on this journey nearly at the same time and now stand at its conclusion nearly together. When I look back, I could not be more proud of what we have achieved together and the person we have grown into. I also extend my sincere gratitude to my mentees, especially Konstantinos Sgouras,

Liana Koleva, and Zhenrong Lang, for teaching me how to become a good mentor. I am immensely proud of the work we did together and I cannot wait to see their future endeavors. I also sincerely thank Andreas Kosmas Kakolyris, Ataberk Olgun, Christina Giannoula, Gagandeep Singh, Geraldo Francisco de Oliveira Junior, A. Giray Yağlıkçı, Harsh Songara, Harshita Gupta, Ismail E. Yuksel, Konstantina Koliogeorgi, Mayank Kabra, Mohammad Sadrosadati, Nisa Bostanci, and Rakesh Nadig, for their invaluable friendship and presence that made SAFARI a family away from home. I thank Tracy Ewen and Tulasi Blake for their administrative support along the PhD journey. Outside SAFARI, I thank all the friends I have met and made during my PhD who eventually became the family outside work: Ahalya, Akash, Akriti, Ankita, Arun, Ashabari, Blake, Dimos, Maria, Martina, Neethu, Saurav, Shimony, Siddharth, and Vaisakh. It is because of their friendship that Zürich felt like home. Finally, I owe everything to my family. I express my profound gratitude to my mother, Sukla Bera, whose lifelong sacrifices created the foundation upon which I built my path. She consistently placed my growth and aspirations above her own comfort and interests. Without her enduring love, unwavering support, and boundless affection, I would not have become the person I am today. I express my deepest gratitude to my wife, Moumita Dey, for her unwavering love, steadfast support, and remarkable patience. The doctoral journey did not always proceed along a smooth or predictable path. Her constant presence and encouragement provided the mental resilience required to confront uncertainty with confidence and perseverance. This dissertation stands as much a testament to her strength and sacrifice as it does to my own efforts. And lastly, I would like to thank my late father, Rebati Raman Bera, who is the reason I embarked on this journey in the first place. I still fondly recall the day I was preparing to depart for Zürich, uncertain whether it was right to move far away from my parents at a stage in life when they might need me by their side. My father assured me that I should stop worrying about them and chase my dream. I lost him within a year of my PhD and since then, everyday I contemplate the decision I took. Wherever he may be, I hope he would be proud to see me here today. This dissertation is dedicated to my father for his unwavering support, encouragement, and love.

Rahul Bera March 8, 2026

Abstract Modern applications operate on massive amounts of data, easily overwhelming the storage and retrieval capabilities of modern memory systems. As a result, memory is (and will likely continue to be) the key performance and energy efficiency bottleneck of computing systems. To alleviate this bottleneck, architects have proposed and employed numerous microarchitectural techniques in general-purpose processors that aim to hide or tolerate the long latency of memory accesses. These techniques have consistently pushed the boundaries of performance and energy efficiency of state-of-the-art general-purpose processors. Yet, as the growth in data footprints continues to far outpace the scaling of process technology nodes, we are in constant need of better microarchitectural techniques that can extract ever-more performance and energy efficiency scaling. This dissertation shows that even though modern microarchitectures observe a large amount of application data (e.g., an application’s control-flow information such as program counter value and branch outcome, or data-flow information such as memory address and data value) and system-generated data (e.g., memory bandwidth usage, cache pollution) during their course of operation, the decisions they make online are often agnostic to the data they are observing. Using four case studies of state-of-the-art microarchitectural techniques employed at various parts of modern processors, we quantitatively show that this data-agnosticism of the microarchitectural techniques cost them a severe opportunity loss of performance and energy efficiency improvements. To alleviate this, this dissertation advocates to fundamentally shift microarchitecture design from being data-agnostic to data-informed. We posit that by enabling microarchitectural techniques (1) to adapt their policies by continuously learning from the data (i.e., making them data-driven), and (2) to tailor their decisions exploiting characteristics/semantics of the application data (i.e., making them data-aware), we can significantly improve performance and energy efficiency of state-of-the-art processors that were otherwise untapped by the conventional data-agnostic microarchitectural techniques. To substantiate this hypothesis, we exploit various forms of lightweight and practical machine learning (ML) techniques, as well as previously-underexplored application data characteristics, and drive the architectural decision making in four key components of a state-ofthe-art processor. First, we exploit online reinforcement learning (RL) to enable a hardware data prefetcher to autonomously learn patterns in an application’s memory access stream and prefetch adaptively by taking various system-level feedback into account. Second, we exploit perceptron learning to accurately identify memory requests that would likely go offchip even after deploying state-of-the-art prefetching mechanisms and reduce their memory latency. Third, we study the interference between data prefetching and off-chip prediction and propose a lightweight RL-based mechanism to synergize the two speculative techniques

vii

by autonomously learning from their behavior online. Fourth, we exploit the consistent repeatability in the memory address and the data value loaded by certain memory instructions present in modern applications to propose an efficient instruction execution mechanism that safely eliminates execution of certain memory instructions. In all four cases, we show that our proposed ML-driven and data-aware microarchitectural techniques significantly improve performance and/or energy efficiency compared to the best prior conventional techniques. We further show that our proposed ML-driven techniques are capable of generalizing their performance benefits beyond the workloads considered at design-time by evaluating them on a large corpus of previously-unseen workload traces from emerging AI, graph mining, and real-world datacenter workloads collected as part of the 4th Data Prefetching Championship (DPC4). Overall, this dissertation builds a comprehensive understanding of how the data-agnostic nature of state-of-the-art microarchitectural techniques limits their effectiveness in mitigating the memory bottleneck and proposes novel data-driven and data-aware microarchitectural techniques. We hope and believe that the insights and techniques presented in this dissertation will encourage the next-generation of data-driven and data-aware microarchitectures to mitigate the ever-growing memory bottleneck problem. Such architectures will not only improve the performance and energy efficiency of computing systems, but will also greatly reduce the system architect’s burden in designing sophisticated policies.

viii

Zusammenfassung Moderne Anwendungen verarbeiten enorme Datenmengen, die die Speicher- und Abrufkapazitäten heutiger Speichersysteme leicht überfordern. Der Speicher stellt daher gegenwärtig – und voraussichtlich auch künftig – den zentralen Leistungs- und Energieeffizienzengpass moderner Rechnersysteme dar. Um diesen Engpass zu entschärfen, wurden zahlreiche mikroarchitektonische Techniken für allgemeine Prozessoren entwickelt, die darauf abzielen, die hohe Latenz von Speicherzugriffen zu verbergen oder zu tolerieren. Diese Techniken haben die Leistungsfähigkeit und Energieeffizienz moderner Prozessoren erheblich verbessert. Da jedoch das Wachstum der Datenmengen die Fortschritte der Halbleiterskalierung weiterhin deutlich übertrifft, besteht weiterhin Bedarf an neuen mikroarchitektonischen Ansätzen, die zusätzliche Leistungs- und Energieeffizienzgewinne ermöglichen. Diese Dissertation zeigt, dass moderne Mikroarchitekturen während ihres Betriebs zwar große Mengen an Anwendungsdaten – etwa Kontrollflussinformationen wie Programmzählerwerte und Verzweigungsergebnisse sowie Datenflussinformationen wie Speicheradressen und Datenwerte – und systemgenerierten Daten, beispielsweise zur Speicherbandbreitennutzung oder Cache-Verschmutzung, beobachten. Dennoch treffen viele mikroarchitektonische Mechanismen ihre Entscheidungen weitgehend unabhängig von diesen Daten. Anhand von vier Fallstudien zu modernen mikroarchitektonischen Techniken in verschiedenen Komponenten eines Prozessors zeigen wir quantitativ, dass diese Datenunabhängigkeit zu erheblichen ungenutzten Potenzialen bei Leistungs- und Energieeffizienzverbesserungen führt. Diese Dissertation plädiert daher für einen grundlegenden Paradigmenwechsel im Mikroarchitekturdesign: weg von datenunabhängigen hin zu dateninformierten Ansätzen. Leistungsfähigkeit und Energieeffizienz moderner Prozessoren lassen sich deutlich steigern, wenn mikroarchitektonische Techniken (1) ihre Strategien kontinuierlich durch Lernen aus beobachteten Daten anpassen (datengetrieben) und (2) ihre Entscheidungen unter Ausnutzung der Eigenschaften beziehungsweise Semantik der Anwendungsdaten treffen (datenbewusst). Dadurch können Leistungs- und Energieeffizienzpotenziale erschlossen werden, die von konventionellen datenunabhängigen Techniken bislang ungenutzt bleiben. Zur Überprüfung dieser Hypothese nutzt diese Dissertation leichtgewichtige und praxisnahe Verfahren des maschinellen Lernens (ML) sowie bislang wenig untersuchte Eigenschaften von Anwendungsdaten, um architektonische Entscheidungen in vier zentralen Komponenten moderner Prozessoren zu steuern. Erstens verwenden wir Online-ReinforcementLearning (RL), um einem Hardware-Prefetcher zu ermöglichen, Muster im Speicherzugriffsverhalten einer Anwendung autonom zu erlernen und sein Prefetching adaptiv unter Berücksichtigung verschiedener Systemrückmeldungen anzupassen. Zweitens nutzen wir Perzeptron-Lernen, um Speicheranforderungen zu identifizieren, die selbst bei Einsatz moderner Prefetching-Mechanismen mit hoher Wahrscheinlichkeit den Hauptspeicher erreichen

würden, und reduzieren dadurch ihre effektive Speicherlatenz. Drittens untersuchen wir die Interferenz zwischen Datenprefetching und Off-Chip-Vorhersage und schlagen einen leichtgewichtigen RL-basierten Mechanismus vor, der beide spekulativen Techniken durch OnlineLernen ihres Verhaltens koordiniert. Viertens nutzen wir die konsistente Wiederholbarkeit sowohl der Speicheradresse als auch des geladenen Datenwerts bestimmter Speicherinstruktionen moderner Anwendungen und entwickeln einen effizienten Ausführungsmechanismus, der die Ausführung solcher Instruktionen sicher vermeiden kann. In allen vier Fällen zeigen wir, dass die vorgeschlagenen ML-basierten und datenbewussten mikroarchitektonischen Techniken Leistung und/oder Energieeffizienz gegenüber den besten bisherigen konventionellen Ansätzen deutlich verbessern. Darüber hinaus zeigen wir, dass die vorgeschlagenen ML-basierten Techniken ihre Leistungsgewinne auch über die während der Entwurfsphase betrachteten Workloads hinaus verallgemeinern können. Hierzu evaluieren wir sie anhand eines umfangreichen Korpus zuvor ungesehener Workload-Traces aus aufkommenden KI-, Graph-Mining- und realen Rechenzentrums-Workloads, die im Rahmen der 4. Data Prefetching Championship (DPC4) gesammelt wurden. Insgesamt vermittelt diese Dissertation ein umfassendes Verständnis dafür, wie die datenunabhängige Natur moderner mikroarchitektonischer Techniken ihre Wirksamkeit bei der Abschwächung des Speicherengpasses begrenzt, und stellt neuartige datengetriebene sowie datenbewusste mikroarchitektonische Ansätze vor. Die in dieser Arbeit gewonnenen Erkenntnisse können die Entwicklung einer nächsten Generation datengetriebener und datenbewusster Mikroarchitekturen fördern, die dem stetig wachsenden Speicherengpass wirksam begegnen und gleichzeitig die Leistungsfähigkeit sowie Energieeffizienz moderner Rechnersysteme verbessern.

Contents Acknowledgments

Abstract

vii

Zusammenfassung

List of Figures

xvii

List of Tables

xxiii

1 Introduction 1.1 Key Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Dissertation Objective and Vision . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 Machine-Learning-Driven Microarchitecture . . . . . . . . . . . . . . 1.4.2 Data-Aware Microarchitecture . . . . . . . . . . . . . . . . . . . . . . 1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Dissertation Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 2 5 5 5 6 9 10 11

2 Related Work 2.1 Techniques to Hide Memory Latency . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Prefetching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Techniques to Tolerate Memory Latency . . . . . . . . . . . . . . . . . . . . . 2.2.1 Out-of-Order Execution . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Multithreading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Speculative Execution via Data Dependence Prediction . . . . . . . . . 2.3 Techniques to Reduce Memory Latency . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Reducing Memory Address Generation Latency . . . . . . . . . . . . . 2.3.2 Reducing Memory Access Latency by Exploiting Device-Level Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Techniques to Improve Processor Efficiency . . . . . . . . . . . . . . . . . . . 2.4.1 Memoization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Dead Instruction Elimination . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Dynamic Instruction Optimization . . . . . . . . . . . . . . . . . . . . 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13 13 13 13 15 15 16 16 17 17

18 18 18 19 20 20

xii

CONTENTS

3 A Primer on Reinforcement and Perceptron Learning 3.1 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Perceptron Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Application of Machine Learning in Processor Design . . . . . . . . . . . . . . 3.3.1 ML-Managed Microarchitectures . . . . . . . . . . . . . . . . . . . . . 3.3.2 Application of ML in High-Level Processor/System Design . . . . . . . 3.4 Summary and Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . .

21 21 23 23 23 24 24

4 Validating Generalization of ML-Driven Microarchitectural Techniques to Emerging Artificial Intelligence (AI) and Industrial Workloads 25 4.1 The Generalization Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.2 The 4th Data Prefetching Championship (DPC4) . . . . . . . . . . . . . . . . . 26 4.3 Exploiting DPC4 Infrastructure to Validate Generalization of ML-Driven Microarchitectural Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 5 Hardware Prefetching using Online Reinforcement Learning 5.1 Motivation and Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Key Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.2 Our Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Formulating Prefetching using Reinforcement Learning . . . . . . . . . . . . . 5.2.1 Why is RL a Good Fit for Modeling Prefetching? . . . . . . . . . . . . 5.3 Pythia: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 Reward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Pythia: Detailed Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 RL-Based Prefetching Algorithm . . . . . . . . . . . . . . . . . . . . . 5.4.2 Detailed Design of Pythia . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.3 Automated Design-Space Exploration . . . . . . . . . . . . . . . . . . 5.4.4 Storage Overhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.1 Workloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.2 Prefetchers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.1 Prefetch Coverage and Overprediction Analysis . . . . . . . . . . . . . 5.6.2 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.3 Performance Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . 5.6.4 Performance Comparison Against Hybrid Prefetching Schemes . . . . 5.6.5 Understanding Pythia using a Case Study . . . . . . . . . . . . . . . . 5.6.6 Performance Benefits via Customization . . . . . . . . . . . . . . . . . 5.6.7 Performance Evaluation using DPC4 Traces . . . . . . . . . . . . . . . 5.6.8 Overhead Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29 29 30 32 32 32 33 34 34 35 35 36 38 41 43 43 44 44 45 45 46 49 53 54 56 57 59

CONTENTS

xiii

5.7

59 60

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.1 Influence on the Research Community . . . . . . . . . . . . . . . . . .

6 Accelerating Long-Latency Loads via Perceptron-Based Off-Chip Load Prediction 61 6.1 Motivation and Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 6.1.1 Our Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 6.2 Hermes: Headroom and Challenges . . . . . . . . . . . . . . . . . . . . . . . . 64 6.2.1 Headroom Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 6.2.2 Key Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.3 Hermes: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 6.4 Hermes: Detailed Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 6.4.1 POPET Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 6.4.2 Hermes Datapath Design . . . . . . . . . . . . . . . . . . . . . . . . . 72 6.4.3 Storage Overhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.5 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.5.1 Workloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.5.2 Evaluated System Configurations . . . . . . . . . . . . . . . . . . . . . 74 6.6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 6.6.1 POPET Prediction Analysis . . . . . . . . . . . . . . . . . . . . . . . . 76 6.6.2 Single-Core Performance Analysis . . . . . . . . . . . . . . . . . . . . 77 6.6.3 Eight-Core Performance Analysis . . . . . . . . . . . . . . . . . . . . . 81 6.6.4 Performance Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . 81 6.6.5 Effect of Prefetchers on Off-Chip Prediction . . . . . . . . . . . . . . . 84 6.6.6 Limit Study by Varying Hermes Request Issue Latency . . . . . . . . . 85 6.6.7 Performance Evaluation using DPC4 Traces . . . . . . . . . . . . . . . 87 6.6.8 Power Overhead Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 6.7.1 Influence on the Research Community . . . . . . . . . . . . . . . . . . 89 7 Synergizing Prefetching and Off-Chip Prediction via Online Reinforcement Learning 91 7.1 Motivation and Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 7.1.1 Key Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 7.1.2 Our Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 7.2 Formulating Prefetcher-OCP Coordination using RL . . . . . . . . . . . . . . . 96 7.2.1 Why is RL a Good Fit for Prefetcher-OCP Coordination? . . . . . . . . 96 7.3 Athena: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 7.3.1 State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 7.3.2 Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 7.3.3 Reward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 7.4 Athena: Detailed Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 7.4.1 QVStore Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

CONTENTS

xiv

7.5

7.6

7.7

7.4.2 State Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.3 Automated Design-Space Exploration . . . . . . . . . . . . . . . . . . 7.4.4 Overhead Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.1 Workloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.2 Evaluated Prior Prefetcher Control Policies . . . . . . . . . . . . . . . 7.5.3 Evaluated Cache Designs . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.4 Evaluated Data Prefetchers . . . . . . . . . . . . . . . . . . . . . . . . 7.5.5 Evaluated Off-Chip Predictors . . . . . . . . . . . . . . . . . . . . . . . Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.1 Single-Core Evaluation Overview . . . . . . . . . . . . . . . . . . . . . 7.6.2 Performance Sensitivity Analysis in CD1 . . . . . . . . . . . . . . . . . 7.6.3 Performance Sensitivity Analysis in CD4 . . . . . . . . . . . . . . . . . 7.6.4 Multi-Core Evaluation Overview . . . . . . . . . . . . . . . . . . . . . 7.6.5 Understanding Athena . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.6 Athena for Prefetcher-Only Management . . . . . . . . . . . . . . . . 7.6.7 Performance Evaluation using DPC4 Traces . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7.1 Influence on the Research Community . . . . . . . . . . . . . . . . . .

102 103 104 104 105 106 106 107 107 108 108 112 114 116 117 120 120 122 122

8 Improving Performance and Power Efficiency by Safely Eliminating Load Instruction Execution 123 8.1 Brief Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 8.2 Motivation and Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 8.2.1 Our Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 8.3 Performance Headroom of Constable . . . . . . . . . . . . . . . . . . . . . . . 125 8.3.1 Global-Stable Loads in Real Workloads . . . . . . . . . . . . . . . . . . 125 8.3.2 Why Do Global-Stable Loads Exist? . . . . . . . . . . . . . . . . . . . . 127 8.3.3 Can Increasing Architectural Registers Eliminate Global-Stable Loads at Compile Time? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 8.3.4 Resource Dependence on Global-Stable Loads . . . . . . . . . . . . . . 130 8.3.5 Performance Headroom . . . . . . . . . . . . . . . . . . . . . . . . . . 131 8.4 Constable: Key Insight . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 8.5 Constable: Microarchitecture Design . . . . . . . . . . . . . . . . . . . . . . . 133 8.5.1 Design Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 8.5.2 Identifying Likely-Stable Loads . . . . . . . . . . . . . . . . . . . . . . 133 8.5.3 Eliminating Load Execution . . . . . . . . . . . . . . . . . . . . . . . . 134 8.5.4 Updating Constable Structures . . . . . . . . . . . . . . . . . . . . . . 135 8.5.5 Disambiguating Eliminated Loads from In-Flight Stores . . . . . . . . 136 8.5.6 Maintaining Coherence in Multi-Core Systems . . . . . . . . . . . . . 137 8.5.7 Other Design Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . 138 8.5.8 An Illustrative Example . . . . . . . . . . . . . . . . . . . . . . . . . . 139

CONTENTS

8.6

8.7

8.8

8.5.9 Storage Overhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.1 Performance Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.2 Power Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.3 Workloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.4 Evaluated Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.5 Functional Verification of Constable . . . . . . . . . . . . . . . . . . . Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7.1 Performance Improvement Analysis . . . . . . . . . . . . . . . . . . . 8.7.2 Performance Comparison with Prior Works . . . . . . . . . . . . . . . 8.7.3 Loads Eliminated by Constable . . . . . . . . . . . . . . . . . . . . . . 8.7.4 Impact on Pipeline Resource Utilization . . . . . . . . . . . . . . . . . 8.7.5 Power Improvement Analysis . . . . . . . . . . . . . . . . . . . . . . . 8.7.6 Performance Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . 8.7.7 Effect of In-Flight Stores on Elimination Coverage . . . . . . . . . . . 8.7.8 Effect of Clean Evictions on Elimination Coverage . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8.1 Influence on the Research Community . . . . . . . . . . . . . . . . . .

140 140 140 141 142 142 143 143 143 145 146 147 148 149 150 150 152 152

9 Conclusions and Future Directions 153 9.1 Putting It All Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 9.2 Future Research Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 9.2.1 Improving and Extending the Proposed Techniques . . . . . . . . . . . 155 9.2.2 New Avenues for ML-Driven and Data-Aware Microarchitectures . . . 156 9.3 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 A Comprehensive View of the Author’s Contributions

159

B Complete List of the Author’s Contributions 163 B.1 Major Contributions Led by the Author . . . . . . . . . . . . . . . . . . . . . . 163 B.2 Co-Supervised Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 B.3 Other Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 C Curriculum Vitae of the Author

167

Bibliography

173

List of Figures 1.1

1.2

3.1 3.2

Performance comparison of three contemporary prefetchers - stride [302], SPP [465], and Bingo [120] - across diverse applications. No prefetcher provides the highest performance across all applications. . . . . . . . . . . . . . . Fraction of dynamic loads that fetch the same data value from the same load address throughout the entire execution of an application (on the primary yaxis), and the performance improvement potential by ideally value-predicting and eliminating the execution of such loads (on the secondary y-axis). . . . . Interaction between an agent and the environment in a reinforcement learning system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overview of a single-layer perceptron model. Each blue circle denotes an input and the green circle denotes the output of the perceptron. . . . . . . . . . . . .

Comparison of (a) coverage, overprediction, and (b) performance of two recently-proposed prefetchers, SPP [465] and Bingo [120], and our new proposal, Pythia. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Formulating the prefetcher as an RL-agent. . . . . . . . . . . . . . . . . . . . . 5.3 Overview of Pythia. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 (a) The QVStore is comprised of multiple vaults. (b) Each vault is comprised of multiple planes. (c) Index generation from feature value. . . . . . . . . . . . . 5.5 Pipelined organization of QVStore search operation. The illustration depicts three program features, each having three planes. . . . . . . . . . . . . . . . . 5.6 Coverage and overprediction with respect to the baseline LLC misses in the single-core system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 Performance improvement in single-core workloads. St=Stride, S=SPP, B=Bingo, D=DSPatch, and M=MLOP. . . . . . . . . . . . . . . . . . . . . . . . 5.8 Performance line graph of 150 single-core traces. . . . . . . . . . . . . . . . . . 5.9 Performance in the four-core system. . . . . . . . . . . . . . . . . . . . . . . . 5.10 Performance line graph of 272 four-core trace mixes. . . . . . . . . . . . . . . 5.11 Performance on unseen traces. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.12 Performance of memory bandwidth-oblivious Pythia versus the basic Pythia. . 5.13 Geomean performance improvement of prefetchers in systems with varying (a) number of cores, (b) DRAM million transfers per second (MTPS), and (c) LLC size. Each DRAM bandwidth configuration roughly matches MTPS/core of various commercial processors [4, 5, 29]. The baseline bandwidth/LLC configuration is marked in red. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21 23

5.1

xvii

31 33 36 39 40 45 46 47 48 48 49 50

xviii

LIST OF FIGURES

5.14 Performance sensitivity of Pythia towards (a) the exploration rate (𝜖), and (b) the learning rate (𝛼) hyperparameter values. The values in basic Pythia configuration are marked in red. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 5.15 Performance, coverage, and overprediction of Pythia with different feature combinations. The x-axis shows experiments with different feature combinations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 5.16 Performance sensitivity of all prefetchers to number of warmup instructions. . 53 5.17 Performance of multi-level prefetching schemes (a) in the baseline 2400-MTPS configuration, and (b) with varying main memory bandwidth. . . . . . . . . . 53 5.18 Performance of Pythia vs. the context prefetcher [660] using hardware contexts. 54 5.19 Performance comparison against IBM POWER7 prefetcher [416]. . . . . . . . 55 5.20 Q-value curves of PC+Delta feature values (a) 0x436a81+0 and (b) 0x4377c5+0 in 459.GemsFDTD-1320B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.21 Performance and main memory bandwidth usage of prefetchers in Ligra-CC. 57 5.22 Performance of the basic and strict Pythia configurations in the Ligra workloads. 57 5.23 Performance of the basic and feature-optimized Pythia on the SPEC CPU2006 suite. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.24 Performance improvement in 483 single-core DPC4 traces. . . . . . . . . . . . 58 5.25 Performance improvement in 966 four-core DPC4 traces. . . . . . . . . . . . . 59 6.1

6.2

6.3 6.4

6.5 6.6 6.7 6.8 6.9

The distribution of ROB-blocking and non-blocking load requests (on the left y-axis), and LLC misses per kilo instructions (on the right y-axis) in the absence and presence of a state-of-the-art hardware data prefetcher [136]. . . . . . . . The average number of cycles a core stalls due to an off-chip load blocking any instruction from retiring from the ROB across all workload categories. The dark portion in each bar shows the cycles that can be completely eliminated by removing the on-chip cache access latency from an off-chip load’s critical path. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Comparison of the execution timeline of an off-chip load request in a conventional processor and in Hermes. . . . . . . . . . . . . . . . . . . . . . . . . . . (a) Speedup of Ideal Hermes by itself and when combined with Pythia in single-core workloads. (b) Speedup of Ideal Hermes when combined with four recently-proposed prefetchers: Bingo [120], SPP [148, 465], MLOP [761], and SMS [792]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Percentage of loads that miss the LLC and goes off-chip (on the left y-axis) and the LLC MPKI (on the right y-axis) in the baseline system with Pythia. . . . . Overview of Hermes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stages to make a prediction by POPET. . . . . . . . . . . . . . . . . . . . . . . Comparison of (a) accuracy and (b) coverage of POPET against those of HMP [905] and TTP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The accuracy and coverage of POPET using each program feature individually and in various combinations. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

63 64

65 66 67 69 76 77

LIST OF FIGURES

6.10 Line graph of POPET’s (a) accuracy and (b) coverage using each of the five program features individually across all 110 single-core workloads. No single feature can provide the best accuracy or coverage across all workloads. . . . . 6.11 Speedup in single-core workloads. . . . . . . . . . . . . . . . . . . . . . . . . . 6.12 Single-core performance of all 110 workloads. . . . . . . . . . . . . . . . . . . 6.13 Speedup of Hermes with three off-chip load predictors (HMP, TTP, and POPET) and the Ideal Hermes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.14 (a) Reduction in stall cycles caused by off-chip loads. (b) Overhead in the main memory requests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.15 Speedup in eight-core workloads. . . . . . . . . . . . . . . . . . . . . . . . . . 6.16 Performance sensitivity to (a) main memory bandwidth and (b) baseline prefetcher. The baseline system configuration is highlighted in green. Other highlighted configurations closely match with various commercial processors [4, 5, 29]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.17 Performance sensitivity to (a) Hermes request issue latency and (b) on-chip cache hierarchy access latency. The baseline system configuration is highlighted in green. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.18 Performance sensitivity to (a) reorder buffer size and (b) LLC size. The baseline system configuration is highlighted in green. . . . . . . . . . . . . . . . . . . . 6.19 Effect of the activation threshold on POPET’s accuracy and coverage (on the left y-axis) and Hermes’s speedup (on the right y-axis) for all single-core workloads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.20 (a) Variation of off-chip load prediction accuracy and coverage and (b) the increase in the main memory requests with different data prefetchers. . . . . . 6.21 Performance of Hermes and Pythia+Hermes while varying Hermes request issue latency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.22 Speedup in 483 single-core DPC4 workload traces. . . . . . . . . . . . . . . . . 6.23 (a) Speedup of Hermes with three different off-chip predictors, HMP, TTP, and POPET in single-core DPC4 workloads. Comparison of (b) accuracy and (c) coverage of POPET against those of HMP and TTP in single-core DPC4 workloads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.24 Performance sensitivity to baseline prefetcher in single-core DPC4 workloads. 6.25 Processor power consumption of Hermes, Pythia, and Hermes combined with Pythia. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1

7.2 7.3

Performance line graph of a state-of-the-art off-chip predictor (OCP), POPET [133], and a state-of-the-art prefetcher, Pythia [136], across 100 workloads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Geomean speedup of POPET, Pythia, Naive, and StaticBest combinations across all workloads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fraction of prefetch fills from off-chip main memory that are inaccurate. . . .

xix

78 78 79 79 80 81

83 83

84 85 86 87

88 89 89

92 93 95

LIST OF FIGURES

7.4

Geomean speedup of Naive, HPAC, MAB, and StaticBest combinations across all workloads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 High-level overview of Athena as an RL agent. . . . . . . . . . . . . . . . . . . 7.6 Organization of QVStore. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Speedup in cache design 1 (CD1). . . . . . . . . . . . . . . . . . . . . . . . . . 7.8 (a) Workload category-wise performance analysis in CD1. (b) Performance comparison with StaticBest in CD1. . . . . . . . . . . . . . . . . . . . . . . . . 7.9 Comparison of (a) the number of main memory requests and (b) average lastlevel cache (LLC) load miss latency. . . . . . . . . . . . . . . . . . . . . . . . . 7.10 Speedup in cache design 2 (CD2). . . . . . . . . . . . . . . . . . . . . . . . . . 7.11 Speedup in cache design 3 (CD3). . . . . . . . . . . . . . . . . . . . . . . . . . 7.12 Speedup in cache design 4 (CD4). . . . . . . . . . . . . . . . . . . . . . . . . . 7.13 Performance sensitivity to underlying prefetching mechanism at L2C in CD1. 7.14 Performance sensitivity to off-chip prediction mechanisms in CD1. . . . . . . 7.15 Performance sensitivity off-chip predicted request issue latency in CD1. . . . . 7.16 Performance sensitivity to prefetching mechanism at L1D in CD4. . . . . . . . 7.17 Performance sensitivity to main memory bandwidth in CD4. . . . . . . . . . . 7.18 Speedup in four-core workloads. . . . . . . . . . . . . . . . . . . . . . . . . . . 7.19 Speedup in eight-core workloads. . . . . . . . . . . . . . . . . . . . . . . . . . 7.20 Distribution of Athena’s action in coordinating Pythia and POPET and speedup of different Pythia-POPET combinations in compute_fp_78 workload from CVP suite, while varying memory bandwidth: 3.2 GB/s and 25.6 GB/s. . . 7.21 Contribution of individual state features and reward components to Athena’s geomean performance across all 100 workloads. . . . . . . . . . . . . . . . . . 7.22 Geomean performance of Athena coordinating two L2C prefetchers without OCP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.23 Speedup in (a) CD1 and (b) CD2 across 483 DPC4 traces. . . . . . . . . . . . . 7.24 Speedup in (a) CD3 and (b) CD4 across 483 DPC4 traces. . . . . . . . . . . . . 8.1 8.2

8.3 8.4 8.5

Execution timeline of a code example in a processor (a) without a load value predictor (LVP), (b) with LVP, and (c) with LVP and load elimination. . . . . . (a) Fraction of dynamic loads that are global-stable. Distribution of globalstable loads by their (b) addressing mode and (c) inter-occurrence distance. (d) Distribution of inter-occurrence distance of global-stable loads from each addressing mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Code example and disassembly from 541.leela_r of SPEC CPU 2017 suite. The highlighted load instructions are global-stable. . . . . . . . . . . . . . . . Code example and disassembly from 557.xz_r of SPEC CPU 2017 suite. The highlighted load instructions are global-stable. . . . . . . . . . . . . . . . . . . Fraction of all dynamic loads that are global-stable in workloads compiled without and with APX (on the left y-axis) and the reduction in dynamic loads by APX (on the right y-axis). . . . . . . . . . . . . . . . . . . . . . . . . . . . .

95 98 101 108 109 110 111 111 112 113 113 114 115 115 116 117

118 119 120 121 121

125

126 128 128

129

LIST OF FIGURES

Distribution of global-stable loads by their addressing modes in workloads without and with APX. Each number on the x-axis corresponds to the respective workload from SPEC CPU 2017 suite. . . . . . . . . . . . . . . . . . . . . . 8.7 (a) Fraction of total execution cycles where at least one load port is utilized (we call such cycles load-utilized). (b) Categorization of load-utilized cycles based on whether or not a global-stable load utilizes a load port. . . . . . . . . . . . 8.8 Speedup of Ideal Constable against Ideal Stable LVP and a processor with 2× load execution width of the baseline. . . . . . . . . . . . . . . . . . . . . . . . 8.9 Overview of Constable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.10 (a) Average number of SLD updates per cycle during rename stage. (b) Change in performance when Constable’s structures are updated only by correct path instructions vs. all instructions without updating Constable’s structures on branch misprediction recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.11 An illustrative example of Constable’s operation. . . . . . . . . . . . . . . . . . 8.12 Speedup over the baseline (noSMT). . . . . . . . . . . . . . . . . . . . . . . . . 8.13 Speedup of all workloads (noSMT). . . . . . . . . . . . . . . . . . . . . . . . . 8.14 Speedup of Constable by eliminating execution of only PC-relative, stackrelative, and register-relative loads. . . . . . . . . . . . . . . . . . . . . . . . . 8.15 Speedup over the baseline (SMT2). . . . . . . . . . . . . . . . . . . . . . . . . . 8.16 Speedup of Constable over ELAR and RFP. . . . . . . . . . . . . . . . . . . . . 8.17 Load coverage of Constable versus EVES. . . . . . . . . . . . . . . . . . . . . . 8.18 Breakdown of eliminated and non-eliminated loads as fractions of globalstable loads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.19 Reduction in (a) RS allocations and (b) L1-D accesses. . . . . . . . . . . . . . . 8.20 (a) Overall core power consumption normalized to baseline. Expanded view of (b) OOO power and (c) MEU power. . . . . . . . . . . . . . . . . . . . . . . . . 8.21 Performance sensitivity to (a) load execution width, and (b) pipeline depth. . . 8.22 (a) Fraction of loads eliminated by Constable that violate memory ordering. (b) Increase in instructions allocated to ROB in presence of Constable. . . . . . . . 8.23 (a) Speedup and (b) coverage of Constable with AMT invalidation on L1D eviction compared to a vanilla Constable. . . . . . . . . . . . . . . . . . . . . . . .

xxi

8.6

130

131 131 134

138 139 144 144 145 145 146 146 147 148 149 150 151 151

List of Tables 4.1

DPC4 workload traces used for evaluation. . . . . . . . . . . . . . . . . . . . .

5.1 5.2 5.3

Example program features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Basic Pythia configuration derived from our automated design-space exploration. 41 List of program control-flow and data-flow components used to derive the list of features for exploration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Storage overhead of Pythia. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Simulated system parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Workloads used for evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Configuration of evaluated prefetchers. . . . . . . . . . . . . . . . . . . . . . . 45 Area and power overhead of Pythia. . . . . . . . . . . . . . . . . . . . . . . . . 59

5.4 5.5 5.6 5.7 5.8

6.2 6.3 6.4 6.5 6.6

The initial set of program features used for automated feature selection. ⊕ represents a bitwise XOR operation. . . . . . . . . . . . . . . . . . . . . . . . . POPET configuration parameters. . . . . . . . . . . . . . . . . . . . . . . . . . Storage overhead of Hermes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simulated system parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . Workloads used for evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . Storage overhead of all evaluated mechanisms. . . . . . . . . . . . . . . . . . .

7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8

Candidate features considered for Athena’s state representation. . . . . . . . . 98 Constituent metrics of Athena’s reward. . . . . . . . . . . . . . . . . . . . . . 101 Final Athena configuration derived through automated design-space exploration. 103 Storage overhead of Athena. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Simulated system parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Workloads used for evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Evaluated cache designs (CD) and corresponding comparison points. . . . . . 106 Storage overhead of all evaluated mechanisms. . . . . . . . . . . . . . . . . . . 107

8.1 8.2 8.3

Storage overhead of Constable. . . . . . . . . . . . . . . . . . . . . . . . . . . . Simulation parameters. IDQ: Instruction Decode Queue, SB: Store Buffer. . . . Access energy, leakage power, and area estimates of Constable’s structures in 14nm technology. R: read port, W: write port. . . . . . . . . . . . . . . . . . . Workloads used for evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . Overhead of all evaluated mechanisms. . . . . . . . . . . . . . . . . . . . . . .

6.1

8.4 8.5

xxiii

70 70 73 74 75 75

140 141 142 142 143

Chapter 1

Introduction Modern applications generate and process massive amounts of data. Applications spanning key domains such as high-performance computing, large-scale graph analytics, machine learning, and genome analysis, exhibit massive data footprints that easily overwhelms the storage and retrieval capabilities of state-of-the-art memory systems [92, 157, 159, 176, 286, 440, 860]. Multiple prior works have shown that such applications, irrespective of whether they execute on cloud servers or mobile systems, spend a significant fraction of their total execution time simply waiting for the data to be retrieved from the memory system [72,103,107–111,128,213, 233, 286, 312, 313, 368, 392, 403, 440, 506, 537, 539, 541, 592, 593, 596, 618, 713, 724, 725, 730, 778, 779, 805,812,846]. As such, memory constitutes (and will likely continue to be) the key performance and energy efficiency bottleneck of modern computing systems [578, 580, 582, 591]. To mitigate this memory bottleneck, computer architects have proposed and employed numerous microarchitectural techniques that aim to hide or tolerate long memory access latency [96, 105, 106, 117, 119, 120, 125, 130, 131, 141, 148, 170, 174, 181, 184, 191, 192, 201, 202, 210, 212, 216–218, 220, 231, 232, 237, 259, 263, 265, 267, 268, 289, 297, 302, 316, 329, 357, 359, 371, 372, 378, 384, 388, 389, 408, 414, 419–421, 425, 428, 433, 434, 443, 452, 457, 465, 475, 479, 483, 485, 486, 515, 520, 553, 556, 569, 570, 575, 584, 585, 587, 592, 593, 598, 600, 601, 612, 628, 640, 647, 663–667, 669, 671, 671, 687, 692, 693, 712, 720, 722, 742, 754, 761, 763–765, 770, 791, 792, 799, 799, 813, 814, 831, 841–845,868–870,874,876,877,881,882,935].1 Some key examples of latency hiding techniques include efficiently caching of data in faster on-chip caches [154, 517, 872], and prefetching data to on-chip caches before the application demands it [97, 428, 780]. Some key examples of latency tolerance techniques include out-of-order (OoO) execution [97, 649, 650, 832], multithreading [365, 424, 643, 654, 781, 782, 839, 894], and speculative execution via data-dependence prediction [220, 515, 553, 569, 570, 687, 720, 722, 841, 844]. These microarchitectural techniques have collectively pushed the boundaries of performance and energy efficiency of state-of-theart general-purpose processors. Yet, as the growth in application data footprint continues to far outpace the scaling of logic and memory technology nodes, we are in a constant need for better microarchitectural designs to extract ever-more performance and energy efficiency scaling. 1 Along with hiding and tolerating memory latency, literature is also rich with techniques that aim to funda-

mentally reduce the memory latency. These techniques, while largely orthogonal to this dissertation, are summarized in §2.3.

CHAPTER 1: INTRODUCTION

1.1

Key Problem

Even though modern microarchitectures observe a massive amount of data2 during its course of operation, the decisions the microarchitecture makes online are, by and large, agnostic to the data it observes. The decisions are often dictated by simple static rules that we, as human architects, typically select at design time, which offers limited flexibility for dynamic adaptation at runtime. We observe that this data-agnosticism severely limits the effectiveness of many state-of-the-art microarchitectural techniques and leaves large potential for both performance and energy efficiency improvement on the table. We identify two concrete types of data-agnosticism typically found in microarchitectural techniques and demonstrate how these impact the overall effectiveness of such techniques. Inadequate Adaptation to the Observed Data Many microarchitectural techniques inadequately adapt their policies based on the data they observe during their online operation. These techniques make decisions based on rigid, and often myopic, human-crafted heuristics selected at design time, completely disregarding the massive amounts of data available during runtime. While effective for the set of applications considered during the design, these techniques often fail to adapt dynamically to diverse application behaviors and/or complex system configurations.3 To demonstrate this observation, we show an example using hardware data prefetching techniques. A hardware data prefetcher predicts the addresses of memory requests and fetches the corresponding data to the on-chip caches before the application demands it [117, 201, 302, 428]. Modern prefetchers typically exploit human-crafted program context information (we call this a program feature; e.g., the address of the instruction generating the memory request, the physical/virtual address of the memory request) to identify patterns in the addresses of memory requests. This program feature is selected at design time and remains fixed throughout the course of the prefetcher’s operation, regardless of the positive or negative impact of the prefetcher’s decisions on the overall system [117,141,148,201,290,301,302,384,428,465,475,479,556,584,628,671,761,765,792,799]. As a result, while a given prefetcher may successfully capture patterns in certain applications, it often fails to dynamically adapt to others, resulting in inconsistent gains (or even degradation) in performance. 2 The data observed by a microarchitecture can be broadly classified into two categories: application data and

system-generated data. Application data consists of information related to an application’s own execution, such as its control-flow information (e.g., program counter value, instruction type, branch outcome) and data-flow information (e.g., memory addresses touched, data value loaded by a memory instruction). System-generated data, on the other hand, consists of information related to the behavior of the system when executing given application(s), such as memory bandwidth usage, cache pollution, and energy consumption. 3 Although this limitation manifests in a wide range of microarchitectural techniques, we note that prior works (most notably in branch prediction literature [405–407,410–413,525,823,824,907,908,937] and, to a lesser extent, in other domains [383,410,576,594,776,825]) have proposed policies that adaptively learn online, thereby relying less on human-designed heuristics. In this dissertation, we either integrate such techniques into our evaluation baseline, or directly compare our proposed mechanisms against them to demonstrate the advancement of the state-of-the-art.

1.1. KEY PROBLEM

Figure 1.1 shows the performance of three contemporary prefetchers, stride [302], SPP [465], and Bingo [120], across a diverse set of applications.4 Each prefetcher exploits a different program feature to identify patterns in memory addresses. As we can see, no single prefetcher provides the highest performance across all applications. This is primarily due to the static nature of the program feature exploited by each prefetcher. For example, SPP provides the highest performance in application class 2, where the program feature it exploits (i.e., the sequence of last-four deltas in cacheline addresses) successfully predicts the future memory addresses. Yet, it does not perform well in other application classes (or even degrades performance in application class 1), where the same program context information cannot identify patterns in memory addresses. This demonstrates that a prefetcher’s reliance on a single, static program feature for finding address patterns inherently limits its adaptability to diverse workloads, leaving significant potential for performance improvement behind. Stride

Bingo

Geomean speedup over baseline 42 9 62 .m 3. cf xa -1 la 84 nc B bm k_ sLi 10 gr aB Pa Li g eR gr aan 45 C k om 9. G p em on sF en 60 D ts 7. TD ca -1 ct uB 41 SS 8B N _ s 41 -2 0. 42 bw 1B av 60 es 3. -1 bw 96 av 3B es _s -2 45 93 0. 1B so pl 48 ex -2 2. sp 47 hi B nx 313 95 4 B 03 46 .g 2. cc lib -1 qu 6B an tu m -7 14 B

2.2

SPP

2.0

Application class 1

Application class 2

Application class 3

1.8 1.6 1.4 1.2 1.0 0.8

Figure 1.1: Performance comparison of three contemporary prefetchers - stride [302], SPP [465], and Bingo [120] - across diverse applications. No prefetcher provides the highest performance across all applications.

Insufficient Exploitation of Data Characteristics Beyond their limited ability to learn, many microarchitectural techniques insufficiently exploit the inherent characteristics and semantics of the application data they process. By design, these techniques often treat data uniformly, even though different data can exhibit widely varying properties (e.g., criticality, value locality, compressibility, approximability) that may 4 §5.5 discusses the evaluation methodology in detail.

CHAPTER 1: INTRODUCTION

influence the overall performance and energy efficiency of the system.5 Neither the programmer conveys these data characteristic information to the underlying microarchitectural techniques, nor the techniques themselves try to infer and exploit such characteristics in the hardware. This unawareness of data characteristics leaves significant potential for performance and energy efficiency improvement on the table. For instance, we find that nearly one-third of all dynamic load instructions, on average across a diverse range of real-world applications, repeatedly fetch the same value from the same memory address (as plotted in the primary y-axis of Figure 1.2).6 Prior techniques such as load value prediction (LVP) [515,553,720,722] can exploit only the repeatable-value aspect of such loads to mitigate data dependence by speculating their values. However, as LVP ignores the repeatable-address aspect, it still requires executing such loads to verify the prediction, thereby consuming scarce pipeline resources. As a result, LVP mitigates data dependence but not resource dependence. By fully exploiting both value and address repeatability, these loads can be safely eliminated, thereby removing not only their data dependence but also their resource dependence. We show that ideally eliminating such load instructions provides a 9.1% performance improvement potential (as shown in the secondary y-axis in Figure 1.2), which is more than double of the performance potential provided by ideally value predicting all such loads. This highlights how overlooking data characteristics can leave a significant potential of performance and energy efficiency improvement on the table. Performance potential by ideally value predicting such loads

Performance potential by ideally eliminating such loads

50%

1.2

40%

1.16

30%

1.12

20%

1.08

10%

1.04

Geomean speedup over the baseline system

Fraction of all dynamic loads

Loads that fetch same data value from same memory address throughout the entire application

1 Client

Enterprise

FSPEC17

ISPEC17

Server

AVERAGE

Figure 1.2: Fraction of dynamic loads that fetch the same data value from the same load address throughout the entire execution of an application (on the primary y-axis), and the performance improvement potential by ideally value-predicting and eliminating the execution of such loads (on the secondary y-axis).

In summary, we observe that the data-agnostic design of many current microarchitectural 5 While this limitation applies to many microarchitectural policies and techniques (e.g., the policy to allo-

cate resources (like reorder buffer entry, reservation station, load/store queue entry) in an out-of-order processor, cache management policy, branch prediction), there exist proposals that incorporate certain forms of data-awareness, like, awareness to data criticality (e.g., criticality-driven instruction fetch [246], instruction execution [296, 297, 800], and prefetching [518, 633]), and data repeatability (e.g., load value prediction [515, 553, 720, 722]). In this dissertation, we build on top of such policies as baseline and demonstrate that by exploiting previously underexplored characteristics of data, one can unlock further performance and energy efficiency improvements. 6 §8.6 details the applications and the simulation methodology used in this study.

1.2. DISSERTATION OBJECTIVE AND VISION

techniques significantly limits their effectiveness in addressing the memory bottleneck. Their failure to learn from runtime data and their lack of awareness of data characteristics together prevent them from fully leveraging the abundant information inherently available during execution. As the data footprint of modern (and likely future) applications continues to grow, these limitations will only get more pronounced.

1.2

Dissertation Objective and Vision

The primary objective of this dissertation is to design fundamentally-better microarchitectural techniques that can inherently exploit the ever-growing data footprint to mitigate the memory bottleneck. We advocate for a fundamental shift from a data-agnostic approach toward a more data-driven approach that can continuously learn from runtime data to dynamically adapt their policies, and a data-aware approach that can tailor their actions by exploiting the characteristics and semantics of data.

1.3

Thesis Statement

To this end, this dissertation posits, and later quantitatively substantiates, the following thesis statement: By designing microarchitectural techniques that (1) adapt their policies through continuous learning from application and system-generated data, and (2) tailor their decisions by exploiting the characteristics and semantics of the data, we can significantly improve the performance and energy efficiency of state-of-the-art processors that was otherwise untapped by the conventional data-agnostic microarchitectural techniques.

1.4

Our Approach

To substantiate our thesis statement, we conduct the research in two broad stages. First, we analyze various state-of-the-art microarchitectural techniques to build a deeper understanding of how the data-agnostic nature of their design limits their effectiveness in mitigating the memory bottleneck. Then, we exploit our understanding of the limitations in current microarchitectures to design novel data-driven and data-aware techniques. For data-driven microarchitecture design, we exploit, design, and evaluate various lightweight machine learning (ML) techniques to guide architectural decision making at various points in the memory hierarchy.7 For data-aware microarchitecture design, we explore and evaluate previously-underexplored 7 Our approach of using ML for guiding or managing microarchitectural techniques at runtime fundamentally

differs from the orthogonal research direction that applies ML to explore microarchitecture, processor, and system design space (e.g., processor design and performance modeling [382, 602], network-on-chip design [294, 509, 904, 922], chip floorplanning optimization [207, 332, 543, 559–561]). Nonetheless, we provide a comprehensive literature review of applied ML in processor/system design in §3.3.2.

CHAPTER 1: INTRODUCTION

characteristics of application data, that are seemingly abundant in a wide variety of workloads, to propose new microarchitectural techniques for improving performance and energy efficiency. In the remainder of this section, we briefly introduce the four techniques developed in this dissertation.

1.4.1

Machine-Learning-Driven Microarchitecture

Efficient Hardware Data Prefetching via Online Reinforcement Learning Prefetching is a key speculation technique for designing high-performance processors that hides the long memory access latency by predicting the addresses of memory requests and fetching their data to on-chip caches before the processor demands it. As we discuss in §1.1, a prefetcher typically exploits human-crafted program context information (also called a feature; e.g., the address of the load instruction) for identifying patterns in the addresses of memory requests. Even though prior research has proposed numerous prefetching mechanisms, we observe that most of them exhibit three key limitations. First, they typically rely on a single, static program feature fixed at design time, which significantly limits their applicability across diverse workloads [117,141,148,201,290,301,302,384,428,465,475,479,556,628,671,761,765,792, 799]. Second, they either lack awareness to the inherent system-level feedback (e.g., memory bandwidth usage, cache pollution) [117,120,148,201,290,301,302,384,428,465,475,479,556,628, 671, 761, 765, 792], or incorporate system awareness as an afterthought (i.e., a separate control component) to the underlying system-unaware prefetch algorithm [141, 192, 265, 267, 268, 483, 485, 510, 511, 583, 799, 932]. Third, they lack the ability to be customized on-the-fly to make them adapt to diverse workloads and system configurations. To alleviate these limitations, we aim to design a single prefetching framework that (1) can holistically learn to prefetch using both multiple different types of program features and systemlevel feedback information, and (2) can be easily customized in silicon via simple configuration registers to exploit different types of program features and/or to change the objective of the prefetcher (e.g., increasing/decreasing coverage, accuracy, or timeliness) without any changes to the underlying hardware. To this end, we formulate prefetching from the grounds-up as a reinforcement learning (RL) problem where the prefetcher acts as the RL agent, the processor and memory system act as the environment, and the agent (i.e., the prefetcher) autonomously learns to prefetch by interacting with its environment. More specifically, for every demand request, the prefetcher extracts a set of program features and uses it as a state information to take a prefetch action based on its prior experience. For every prefetch action, the prefetcher receives a numerical reward which evaluates the efficacy of the prefetch action (e.g., accuracy and timeliness of the prefetch) under various system-level feedback information (e.g., memory bandwidth usage). The prefetcher uses this reward to reinforce the correlation between various program features and the prefetch action. By formulating prefetching in this way, we enable the prefetcher to autonomously learn from its own experience and continuously adapt to changing workloads and system configurations for generating effective prefetch requests, without relying on

1.4. OUR APPROACH

rigid, and often myopic, human-crafted heuristics. Our extensive evaluation using both hardware simulation and synthesis shows that our RL-based prefetcher (called Pythia [45, 136]) consistently outperforms multiple state-of-the-art prefetchers in single-core, multi-core, and bandwidth-constrained core configurations across a diverse set of workloads, while incurring a modest area and power overhead. We further show that Pythia is capable of generalizing its performance benefits beyond the workloads considered at the design time by evaluating it across a large corpus of previously-unseen workload traces collected as part of the 4th Data Prefetching Championship (DPC4) [135]. We open-source our artifact-evaluated implementation of Pythia, and all necessary evaluation infrastructure, to facilitate future research: https://github.com/CMU-SAFARI/Pythia. An earlier version of Pythia is published at MICRO 2021 [136] and an extended version on arXiv [137]. Pythia has influenced multiple subsequent works both as a state-of-the-art baseline [151, 152, 199, 260, 274, 319, 404, 490, 513, 554, 670, 777, 833, 928], as well as a reference framework for modeling architectural decision making using machine learning [91, 205, 330, 375, 526, 530, 594, 642, 776, 890, 892, 902, 910, 916, 929]. Accelerating Loads via Perceptron Learning-Based Off-Chip Load Prediction While Pythia advances the state-of-the-art in prefetching, it only successfully predicts and prefetches nearly half of the memory load requests that would have otherwise gone to the offchip main memory. For the remaining off-chip loads, we observe that, a large fraction of their latency is spent in accessing the on-chip cache hierarchy to solely determine that they need to go off-chip. As the size of on-chip caches continue to increase to cater to the ever-growing data footprint of workloads, the latency overhead stemming from on-chip cache access is going to exacerbate further in next-generation processors. To address this, we aim to accelerate off-chip load requests by removing the on-chip cache access latency from their critical path. To this end, we propose a new technique (called Hermes [22, 133]), whose key idea is to: (1) accurately predict which load requests might go offchip, and (2) speculatively fetch the data required by the predicted off-chip loads directly from the main memory, while also concurrently accessing the cache hierarchy for such loads. To enable Hermes, we develop a new lightweight, perceptron learning-based off-chip load prediction technique that learns to identify off-chip load requests using multiple program features (e.g., sequence of program counters, byte offset of a load request). For every load request generated by the processor, the predictor observes a set of program features to predict whether or not the load would go off-chip. If the load is predicted to go off-chip, Hermes issues a speculative load request directly to the main memory controller once the load’s physical address is generated. If the prediction is correct, the load eventually misses the cache hierarchy and waits for the ongoing speculative load request to finish, and thus Hermes completely hides the on-chip cache hierarchy access latency from the critical path of the correctly-predicted off-chip load. Our extensive evaluation using a wide range of workloads shows that Hermes provides consistent performance improvement on top of a state-of-the-art baseline system across a wide range of configurations with varying core count, main memory bandwidth, highperformance data prefetchers, and on-chip cache hierarchy access latencies, while incurring

CHAPTER 1: INTRODUCTION

only modest storage overhead. We further show that Hermes is capable of generalizing its performance benefits beyond the workloads considered at the design time by evaluating it across the large, previously-unseen DPC4 trace corpus. We open-source our artifact-evaluated implementation of Hermes, and all necessary evaluation infrastructure, to facilitate future research: https://github.com/CMU-SAFARI/Hermes. An earlier version of Hermes is published at MICRO 2022 [133] and an extended version on arXiv [134]. Hermes has already influenced multiple subsequent works as a state-of-the-art baseline [282, 399, 478, 633, 719, 886]. One follow-up work [204] has revealed the existence of a similar speculation mechanism already employed in Intel 3rd generation Xeon processors [31]. Synergizing Data Prefetching and Off-Chip Load Prediction via Lightweight Reinforcement Learning Prefetching and off-chip prediction, while differing in their speculation mechanism, share the same goal of hiding memory access latency. A prefetcher hides latency by predicting addresses of future memory requests and proactively fetching their data closer to the core, whereas an off-chip predictor (OCP) hides latency by predicting loads that would likely go off-chip and removing the on-chip cache access latency from their critical path. However, we demonstrate that: (1) a prefetcher and an OCP often provide complementary performance benefits, yet (2) naively combining these two mechanisms often fails to realize their full performance potential, and (3) existing prefetcher control policies (both heuristic- and learning-based) leave significant room for performance improvement behind. To address this, our goal is to design a holistic framework that can autonomously learn to coordinate an off-chip predictor with multiple prefetchers employed at various cache levels, delivering consistent performance benefits across a wide range of workloads and system configurations. To this end, we propose a new technique called Athena, which models the coordination between prefetchers and off-chip predictor (OCP) as a reinforcement learning (RL) problem. Athena acts as the RL agent that observes multiple system-level features (e.g., prefetcher/OCP accuracy, bandwidth usage) over an epoch of program execution, and uses them as state information to select a coordination action (i.e., enabling the prefetcher and/or OCP, and adjusting prefetcher aggressiveness). At the end of every epoch, Athena receives a numerical reward that measures the change in multiple system-level metrics (e.g., number of cycles taken to execute an epoch). Athena uses this reward to autonomously and continuously learn a policy to coordinate prefetchers with OCP. Athena makes a key observation that using performance improvement as the sole RL reward, as used in prior work, is unreliable, as it confounds the effects of the agent’s actions with inherent variations in workload behavior. To address this limitation, Athena introduces a composite reward framework that separates (1) system-level metrics directly influenced by Athena’s actions (e.g., last-level cache misses) from (2) metrics primarily driven by workload phase changes (e.g., number of mispredicted branches). This allows Athena to autonomously learn a coordination policy by isolating the true impact of its actions from inherent variations in workload behavior. Our extensive evaluation using a diverse set of 100 memory-intensive workloads shows that Athena consistently outperforms multiple prior state-of-the-art coordination policies across a wide range of

1.4. OUR APPROACH

system configurations with various combinations of underlying prefetchers at various cache levels, OCPs, and main memory bandwidths, while incurring only modest storage overhead and design complexity. We further demonstrate Athena’s capability of generalizing its performance benefits beyond the workloads considered at the design time by evaluating it across the large, previously-unseen DPC4 trace corpus. We open-source our artifact-evaluated implementation of Athena, and all necessary evaluation infrastructure, to facilitate future research: https://github.com/CMU-SAFARI/Athena. An earlier version of Athena is published at HPCA 2026 [138] and an extended version on arXiv [139].

1.4.2

Data-Aware Microarchitecture

Improving Performance and Power Efficiency by Eliminating Load Instruction Execution While prefetching and off-chip prediction are very effective in hiding memory access latency, modern processors also employ numerous mechanisms inside the processor core that tolerate latency by preventing stalls caused by long-latency instructions (e.g., a memory load instruction). Two such widely studied techniques are load value prediction (LVP) and memory renaming (MRN), both of which break the data dependency on a load instruction by speculating the value fetched by the load. However, even when their predictions are correct, these techniques still execute the predicted load instructions, like any other load instruction, to verify the result, thereby consuming scarce pipeline resources that could otherwise be used by other load instructions. This limitation leaves a significant performance and power efficiency improvement opportunity untapped. Our analysis reveals that a substantial fraction of dynamic load instructions in real workloads repeatedly fetch the same value from the same memory address throughout the execution. This recurring stability in load behavior represents a valuable characteristic that is often underexplored by conventional techniques. We aim to exploit this stability to not only break load data dependence but completely eliminate the entire execution of the load instruction, thereby mitigating both data and resource dependence. To this end, we propose Constable [143], a data-aware microarchitectural technique that dynamically detects likely-stable loads (i.e., loads that have historically fetched the same data from the same address) and safely eliminates their execution. For every likely-stable load, Constable (1) tracks modifications to its source architectural registers and memory location via lightweight hardware structures, and (2) eliminates the execution of subsequent instances of the load instruction until there is a write to its source register or a store or snoop request to its load address. In doing so, Constable treats likely-stable loads differently from other loads, tailoring the execution policy based on the inherent characteristics of the load instructions. Our extensive evaluation shows that Constable simultaneously improves performance and reduces core dynamic power by over a strong superscalar baseline processor design, with larger benefits in processors with simultaneous multithreading support. We open-source a binary instrumentation tool that we use to identify load instructions that repeatedly fetch

CHAPTER 1: INTRODUCTION

the same value from the same memory address in any off-the-self x86-64 binary at https: //github.com/CMU-SAFARI/Load-Inspector. An earlier version of Constable is published at ISCA 2024 [143] and an extended version on arXiv [142]. Constable has influenced both academic research and industrial product development. A later work has independently verified Constable’s key observation of loads with repeated address-value stability in off-the-shelf programs [629]. To the best of our knowledge, the key idea of Constable is in the process of getting transferred to a real-world commercial processor design, and has been the subject matter of a patent application filed by Intel Corporation [683].

1.5

Contributions

This dissertation makes the following contributions: • We build a comprehensive understanding of how the data-agnostic nature of conventional microarchitecture design restricts the effectiveness of state-of-the-art mechanisms deployed in high-performance general-purpose processors. Chapters 5 to 8 dissect four case studies of state-of-the-art microarchitectural techniques employed at various parts of modern processors and quantify the opportunity loss of performance and energy efficiency improvements due to data-agnostic designs. • We advocate and quantitatively substantiate a fundamental shift from data-agnostic techniques towards machine-learning (ML)-driven techniques that can continuously learn from runtime information to dynamically adapt their policies, and data-aware techniques that can tailor their actions by exploiting the characteristics and semantics of data. • We design and release a large-scale, community-wide workload traces and evaluation infrastructure through the 4th Data Prefetching Championship (DPC4), comprising 610 traces from emerging artificial intelligence (AI), graph mining, and real-world datacenter workloads, 483 of which are previously unseen by the architecture research community (see Chapter 4). We further develop a rigorous stress-testing methodology using these traces to systematically validate the performance generalization capability of ML-driven microarchitectural techniques without workload-specific tuning. • We propose Pythia (see Chapter 5), the first practical reinforcement-learning-based hardware data prefetching framework. We show that Pythia dynamically learns correlations between program features and system-level feedback to generate accurate, timely, and bandwidth-aware prefetches, achieving robust performance across diverse workloads and configurations. • We propose Hermes (see Chapter 6), that employs the first perceptron-learning-based off-chip load predictor. Hermes formulates the off-chip prediction as a binary classification problem, learning to accurately identify loads that would go off-chip early and

1.6. DISSERTATION OUTLINE

initiating speculative requests directly to the main memory, thereby effectively hiding the long main memory access latency. • We propose Athena (see Chapter 7), the first reinforcement-learning-based framework designed for prefetcher–OCP coordination. Athena autonomously synergizes prefetching and off-chip prediction by observing system-level metrics and learning to control both mechanisms in unison, consistently outperforming both mechanisms individually across a diverse set of workloads and system configuration. • We propose Constable (see Chapter 8), the first data-aware technique that exploits consistent repeatability in both the address and data value of a load instruction to safely eliminate its entire execution in today’s multi-core processors, providing simultaneous improvement in performance and power efficiency. • We open-source the implementations, workload infrastructures, and/or accompanying toolchains for all proposals to ensure reproducibility, enable rigorous comparison, and facilitate future research. By releasing complete implementations and evaluation artifacts, this dissertation aims to promote transparency and accelerate community-driven progress in ML-driven and data-aware microarchitecture design.

1.6

Dissertation Outline

This dissertation is organized into 8 chapters. Chapter 2 provides a comprehensive view of various microarchitectural techniques to improve processor performance and energy efficiency by mitigating the memory bottleneck. Chapter 3 provides a primer on two types of machine learning methods relevant to this dissertation: reinforcement learning and perceptron learning, and their prior applications in processor design. Chapter 4 introduces a new set of previously-unseen workload traces and a rigorous and systematic methodology using those traces to empirically validate the performance generalization capability of the MLdriven techniques proposed in this dissertation. Chapter 5 introduces and evaluates Pythia, a reinforcement-learning-based hardware data prefetcher. Chapter 6 introduces and evaluates Hermes, a perceptron-learning-based off-chip load predictor. Chapter 7 introduces and evaluates Athena, a reinforcement-learning-based prefetcher-OCP coordinator. Chapter 8 introduces and evaluates Constable, a purely-microarchitectural technique to safely eliminate load instruction execution. Chapter 9 summarizes this dissertation and discusses the possibilities of designing numerous other microarchitectural techniques using machine-learning-driven and data-aware principles.

Chapter 2

Related Work Long-latency memory accesses remain a fundamental performance and energy efficiency bottleneck in modern general-purpose processors. To address this challenge, researchers have proposed a wide range of microarchitectural techniques, which can be broadly categorized into four classes: (1) latency-hiding techniques, (2) latency-tolerance techniques, (3) latencyreduction techniques, and (4) techniques that improve processor energy and/or power efficiency. This section surveys these classes of techniques and positions them in relation to the approaches developed in this dissertation.

2.1

Techniques to Hide Memory Latency

2.1.1

Caching

Caching [154,517,872] hides memory access latency by exploiting temporal and spatial locality in memory accesses. Most modern computing systems employ caches, which are often organized in a hierarchical way with varying capacities and latencies. Researchers have proposed numerous cache management policies [84, 85, 116, 121, 127, 155, 173, 182, 194–198, 203, 208, 210, 254, 255, 261, 261–263, 279, 280, 316–318, 356, 371, 389, 390, 393, 394, 398, 408, 409, 414, 419–421, 448–453,457,482,484,520,555,573,603,627,634,641,656–659,669,676,680,681,692,693,718,741, 742, 745–752, 755, 756, 760, 821, 826, 829, 830, 842, 842, 845, 854–857, 864, 865, 874, 876, 898, 920] to improve cache performance. Prior works have also proposed various mechanisms that reduce memory access latency in deep cache hierarchies by predicting the cache level at which a given memory request is likely to hit [397, 524, 552, 655, 677, 728, 729]. We evaluate each proposed technique presented in this dissertation on a processor configuration that incorporates a deep cache hierarchy and state-of-the-art cache management policies [395, 876]. Moreover, Hermes introduces off-chip prediction that works orthogonally to cache management policies and bypasses intermediate cache levels during cache fills [9, 39, 210, 263, 389, 408, 414, 452, 457, 520, 669, 692, 693, 842, 845, 874].

2.1.2

Prefetching

Prefetching [97,117,428,780] is a well-studied speculation technique that predicts the addresses of long-latency memory requests and fetches the corresponding data to on-chip caches before the program demands it, thus hiding long memory access latency. Prior works on prefetching

CHAPTER 2: RELATED WORK

can be broadly categorized into three groups: software, hardware, and pre-computation-based prefetching. Software Prefetching In software prefetching, the programmer or compiler inserts explicit prefetch instructions into the program to fetch data before it is required [78–80, 111, 177, 209, 400, 471, 499, 514, 532, 534, 574, 575, 715, 880, 918]. This approach is effective for workloads with regular access patterns that can be statically predicted, but it is generally ineffective for workloads with irregular or hard-to-predict access behavior. Hardware Prefetching In hardware prefetching, dedicated microarchitectural structures detect patterns in the stream of observed memory addresses and issue prefetch requests transparently to the programmer and compiler. Based on the underlying pattern-matching algorithm, hardware prefetchers can be broadly classified into two categories. Temporal prefetchers [119, 130, 212, 216, 289, 372, 388, 425, 443, 791, 851, 868–870, 881, 882] memorize long sequences of cacheline addresses demanded by the processor. When a previously-seen address is encountered again, a temporal prefetcher issues prefetch requests to addresses that previously followed the currently-seen cacheline address. However, temporal prefetchers usually have high storage requirements (often multi-megabytes of metadata storage, which necessitates storing metadata in memory [388, 791, 868]). Spatial prefetchers [86–88, 117, 120, 141, 148, 191, 193, 201, 232, 268, 302, 379, 384, 415, 428, 465, 466, 475, 479, 556, 628, 635, 638, 671, 761, 765, 792, 799, 852, 918], on the other hand, predict addresses of future memory requests by learning program access patterns over different spatial memory regions. Spatial prefetchers provide high-accuracy prefetches, usually with lower storage overhead than temporal prefetchers. Our proposed prefetcher, Pythia, belongs to this category. We extensively evaluate Pythia with other spatial prefetchers [120,141,148,465,761] in this dissertation and demonstrate that Pythia delivers state-of-the-art performance benefits. Pre-Computation-Based Prefetching Pre-computation-based prefetching executes a program’s own code (either the complete code, or a slice of it) ahead of the actual execution for the sole purpose of generating future memory requests. Since prefetch requests are generated by pre-executing the code, these techniques can achieve high prefetch accuracy even when the program’s memory access stream lacks easilyidentifiable patterns. Based on the mechanism for constructing the code for pre-execution, these techniques can be classified into two broad categories. Thread-based pre-computation [189, 190, 227, 228, 258, 314, 357, 431, 433, 533, 533, 568, 644, 675, 682, 705, 707, 790, 802, 817, 861, 919, 925, 936] mechanisms explicitly construct a simplified version of the program’s own code (denoted as a helper thread), often derived from slices of the original instruction stream, to generate memory prefetches ahead of demand. Software-directed approaches [189, 190, 228, 314, 533, 533, 644, 704, 936] rely on the compiler or

2.2. TECHNIQUES TO TOLERATE MEMORY LATENCY

programmer to explicitly extract and spawn helper threads. In contrast, hardware-generated approaches [227, 357, 433, 568, 675, 704, 706, 802, 817, 925] transparently derive and execute precomputation slices in hardware, thus reducing programmer burden while still exposing substantial memory-level parallelism. Runahead execution [259, 592, 593], while fundamentally following pre-computationbased prefetching principle, does not require extracting and spawning explicit helper threads. Instead, it checkpoints the architectural state of the processor in an event of a full-window stall (i.e., when the reorder buffer neither can retire instructions due to a long-latency operation blocking the retirement, nor can it accept new instructions because it is full) and enters into runahead mode. In this mode, the processor speculatively continues to fetch, decode, and execute program’s own instructions without updating the architectural state, thereby exposing additional memory-level parallelism. While runahead execution provides performance benefits with relatively modest hardware overhead, its effectiveness depends on the accuracy of the speculative execution and the ability to generate useful prefetches. As such, subsequent works have refined runahead policies to maximize coverage while minimizing unnecessary speculation [357, 433, 584, 586, 587, 598–601, 697]. Prefetcher Coordination Along with the pattern-matching algorithm, prefetcher coordination policies also play a crucial role in designing a high-performance prefetcher. Several prior works incorporate prefetching metrics such as coverage, accuracy, and bandwidth consumption to selectively throttle or discard prefetch requests, aiming to reduce memory bandwidth usage and prefetch-induced cache pollution [237,265,267,268,274,378,483,485,640,671,742,799,877]. In this dissertation, we show that prefetcher coordination policies are often applied as an afterthought to an otherwise system-unaware prefetch algorithm. Moreover, many coordination policies rely on static, and often myopic, human-designed heuristics and thresholds. These two caveats together significantly limit the effectiveness a prefetcher. Pythia addresses these challenges by incorporating system-level feedback inherently to its design and autonomously learning from the workload behavior, without relying on static heuristics.

2.2

Techniques to Tolerate Memory Latency

Although caching and prefetching hide memory access latency, processors can still stall while waiting for memory (in general, any long-latency) operations to complete. To address this, prior works have introduced a variety of latency-tolerance techniques that identify and execute useful instructions during such stalls. This section summarizes such techniques.

2.2.1

Out-of-Order Execution

Out-of-order (OoO) execution [97, 649, 650, 832] is a fundamental latency-tolerance technique employed in modern processors. By dynamically reordering instructions at runtime, OoO exe-

CHAPTER 2: RELATED WORK

cution enables independent instructions to execute by bypassing stalled ones, thereby exploiting instruction-level parallelism that would otherwise remain hidden. This mechanism significantly mitigates the impact of long-latency operations, such as memory accesses, on overall performance. However, the scalability of OoO execution is constrained by the complexity and energy cost of maintaining large instruction windows and supporting structures. All proposed mechanisms in this dissertation are evaluated on top of an aggressive OoO baseline processor.

2.2.2

Multithreading

Multithreading [271,365,365,424,441,623,643,654,781,782,786,828,838–840,858,894] is a wellestablished technique to tolerate long memory access latencies by overlapping the execution of multiple threads, thereby improving overall processor utilization and performance. The literature distinguishes three major forms of multithreading: fine-grained, coarse-grained, and simultaneous. Fine-grained multithreading [476, 605, 781, 828] switches between threads at every cycle to issue instructions from a ready thread, thus hiding latency at the cost of reduced single-thread throughput. Coarse-grained multithreading [613] switches only on long-latency events (e.g., a cache miss), which reduces the overhead of frequent context switching but provides less opportunity to overlap latency. Simultaneous multithreading (SMT) [365, 731, 839, 840, 894, 895] issues instructions from multiple threads in the same cycle, enabling the processor to better utilize its wide issue width, but increasing contention for shared pipeline resources. While these forms of multithreading can improve performance by exploiting otherwise idle cycles, they also exacerbate competition for scarce hardware resources. Constable mitigates such resource pressure and demonstrates higher performance gains when deployed in SMT configurations.

2.2.3

Speculative Execution via Data Dependence Prediction

Load data dependence significantly limits instruction-level parallelism (ILP) [106,429,649,785, 801, 831]. A major class of latency-tolerance techniques extracts ILP by exploiting speculative execution to overcome the data dependence induced by load instructions. Load Value Prediction (LVP) [125, 170, 174, 329, 434, 515, 516, 553, 663–667, 712, 720–722, 754, 763, 764, 926, 927] breaks load data dependence by predicting the value of a load instruction and speculatively executing load-data-dependent instructions with the predicted value. The predicted value is later verified by executing the load instruction. A correct prediction increases ILP, but an incorrect prediction leads to re-execution of load-dependent instructions, incurring both performance and power overheads. Memory Renaming (MRN) [569, 570, 687, 841, 844] learns the dependency relationship between a store-load instruction pair, and speculatively executes the load-dependent instructions by forwarding the data directly from the associated store instruction. The forwarded data is later verified by executing the load. A correct data forwarding increases ILP, whereas an incorrect forwarding incurs both performance and power overhead due to re-execution of load-dependent instructions. Memory Dependence Prediction (MDP) [220, 467, 571, 572, 668, 758, 806, 905] extends speculation further by predicting whether a load is dependent on a preceding store with an unresolved address; if predicted independent,

2.3. TECHNIQUES TO REDUCE MEMORY LATENCY

the load executes early, exposing additional instruction-level parallelism. These mechanisms collectively reduce stalls by overlapping useful computation with pending memory operations, thus significantly improving tolerance to long-latency loads. However, all such speculative techniques still require the predicted load instruction to execute for correctness verification, which consumes critical hardware resources (e.g., reservation station, load port, and cache access bandwidth). As a result, while they effectively mitigate data dependence, they do not eliminate resource dependence, leaving untapped performance potential. This shortcoming directly motivates Constable, which aims to mitigate both load data and resource dependence to deliver higher power efficiency.

2.3

Techniques to Reduce Memory Latency

While prior techniques primarily aim to hide or tolerate memory latency, they do not fundamentally reduce the latency experienced by a memory instruction. A complementary line of research targets direct latency reduction by shortening the latency of either of the two component operations of a memory instruction, i.e., memory address generation and memory access. We classify such works into two broad categories: (1) techniques that reduce memory address generation latency, and (2) techniques that reduce memory access latency by exploiting device-level characteristics and optimizations.

2.3.1

Reducing Memory Address Generation Latency

Prior works propose both speculative and non-speculative techniques to accelerate the address generation of a load memory instruction. [105] uses a fast, speculative carry-free addition to speed up load address computation. [106] caches the values of recently-used registers to speculatively compute load address. Register file prefetching (RFP) [770] predicts the load address of an instruction to prefetch its data to the register file. Early load address resolution (ELAR) [131], on the other hand, tracks stack register values using a small computation unit in the decode stage to safely and non-speculatively compute the load address of most stack loads immediately after the decode stage. While Constable bears resemblance to these works, it significantly differs from them in a major way: These prior works necessitate the execution of the load instruction whose load address has been computed early. Works that speculatively compute load address [105, 106, 770] need to execute the load to verify the speculation. ELAR, which employs a safe technique to non-speculatively compute the address of stack loads, still needs to fetch the load data from the memory hierarchy. Constable safely eliminates both the address computation and the data fetch operations of a load execution altogether. We evaluate Constable against ELAR and RFP demonstrating their performance benefits.

CHAPTER 2: RELATED WORK

2.3.2

Reducing Memory Access Latency by Exploiting Device-Level Characteristics

In this section, we review a substantial body of prior work aimed at reducing memory access latency by exploiting device-level characteristics of memory technologies. Given that dynamic random-access memory (DRAM) serves as the de-facto main memory technology in modern computing systems, we first discuss techniques that reduce DRAM access latency by leveraging its microarchitectural organization and device-level properties. The techniques we propose in this dissertation are largely orthogonal to the latency reduction techniques mentioned in this section. Exploiting DRAM Microarchitecture. A significant corpus of research has reduced DRAM access latency by exploiting its internal microarchitectural structure [186, 188, 215, 345, 359, 363,470,489,529,610,647,732–735,737,793,917]. Subarray-Level Parallelism (SALP) [470] overlaps operations within different subarrays of a bank, alleviating bank-level serialization and thereby speeding up access latencies with minimal hardware additions. Tiered-Latency DRAM (TL-DRAM) [489] divides long bitlines into “near” and “far” segments using isolation transistors, allowing near-segment accesses at much lower latency without substantially increasing cost per bit. Low-Cost Inter-Linked Subarrays (LISA) [186] introduces inexpensive isolation transistors between adjacent subarrays to enable ultra-fast inter-subarray bulk data movement. Copy-Row DRAM (CROW) [360] duplicates selected rows using an in-DRAM bulk copy mechanism [735] and accesses them by simultaneously activating the regular and copied row. This dual-row activation drives the sense amplifiers faster, thereby reducing access latency. CLR-DRAM [535] enables dynamic per-row reconfiguration between high-capacity and lowlatency modes, significantly reducing DRAM timing parameters with modest area overhead. Exploiting DRAM Device-Level Characteristics. DRAM manufacturers specify conservative timing margins to guarantee correct operation under worst-case conditions such as high temperature, low voltage, and process variation. These margins are often overly pessimistic for common-case operation, and multiple works show that they can be safely reduced while still ensuring correctness. Prior research has exploited diverse DRAM device-level characteristics (e.g., operating temperature [183, 488], DRAM cell’s charge retention behavior [361, 863], and latency variation across cells and subarrays [185, 460, 487]) to achieve this goal. Other studies also leverage voltage–latency trade-offs [187], application error tolerance [477], and refresh/access parallelization [188, 893] to reduce DRAM latency.

2.4

Techniques to Improve Processor Efficiency

2.4.1

Memoization

Memoization [558] caches computed results from repeated code executions, enabling a program or a microarchitecture to skip redundant computations when encountering identical in-

2.4. TECHNIQUES TO IMPROVE PROCESSOR EFFICIENCY

put sets. Memoization has been applied in both software [229,230,558,690,818,819] and in hardware at various program granularities, including instruction-level [222, 355, 563, 691, 787, 788], basic-block-level [373], trace-level [340], and function-level [223]. Early works on instructionlevel memoization aim to accelerate long-latency operations (e.g., floating point multiplication and division) by storing their operands and results in value caches [611, 691]. Sodani and Sohi propose a PC-indexed reuse buffer to store the results of (multiple) dynamic instances of every static instruction [787]. Molina et al. improve upon the reuse buffer to capture reuse of results across dynamic instances of different static instructions [563]. Constable, in principle, resembles instruction-level memoization with three key differences that make Constable more performant, lightweight, and usable in today’s high-performance multi-core processors. First, prior works aim to memoize (multiple) results of every static instruction, irrespective of whether or not the results would be useful for instruction elimination [787, 789]. This requires a large memoization buffer, often as large as L1 data cache [221, 913, 914], to capture elimination opportunities across long inter-occurrence distances (see §8.3.1). Gonzalez et al. have shown that, while such a large memoization buffer may provide a significant performance benefit, the benefits reduce significantly when the latency to access the buffer is considered [339]. Constable, on the other hand, (a) only targets loads, and (b) employs a confidence-based mechanism to filter out likely-stable load instructions from all loads. This significantly reduces Constable’s storage overhead and design complexity (e.g., port requirements of SLD as discussed in §8.5.7) while providing high elimination coverage. Second, prior works may delay retrieving the memoized instruction output until its source register values are available [222, 355, 563, 691]. As a result, these works may not alleviate resource dependence on hardware structures like the reservation station. Constable, however, explicitly monitors changes in source architectural registers of likely-stable load instructions and eliminates them early in the pipeline, alleviating resource dependence from both load reservation station and load execution unit. Third, prior works may not be applied in today’s high-performance multi-core processors as they do not address challenges related to (a) keeping memoization buffer coherent across multiple cores, and (b) maintaining program correctness in presence of out-of-order load issuing. Constable addresses both these challenges (§8.5.5 and §8.5.6) and we extensively verify its correctness via functional simulation (§8.6.5).

2.4.2

Dead Instruction Elimination

Another line of research aims to improve processor energy efficiency by eliminating instructions that do not contribute to the final program outcome. Such dead instructions [171] may arise due to compiler limitations, aggressive speculation, or dynamic program behavior, and they unnecessarily consume execution resources and energy. Prior works propose both compile-time analyses to statically remove dead code [240, 472, 887] and runtime mechanisms to dynamically detect and eliminate instructions [171,172] whose results are never used. These approaches reduce pipeline pressure, free scarce microarchitectural resources, and improve overall energy efficiency without compromising correctness. In contrast, Constable targets

CHAPTER 2: RELATED WORK

redundant but live load instructions, whose results are used in the program but repeatedly identical, and eliminates their execution by exploiting address-value stability.

2.4.3

Dynamic Instruction Optimization

Prior works on trace-cache-based optimizations [300, 387, 702, 703, 784] and rePLay framework [277, 648] enable a wide range of runtime code optimizations (e.g., move elimination, zero-idiom elimination) that can eliminate instructions in microarchitecture. Continuous optimization (CO) [278] builds on these works and enables removing redundant instructions (including loads) using the register renaming logic. Constable differs from these works in two key ways. First, these schemes learn optimizations offline on a per-trace (or frame) basis. Constable learns optimization online and applies directly to the program’s dynamic instruction stream. Second, unlike Constable, CO does not eliminate a load instruction in a multi-core system in order to maintain coherence.

2.5

Summary

In summary, the literature encompasses a rich body of techniques that aim to mitigate the memory bottleneck through diverse approaches, including caching and prefetching to hide latency, out-of-order and speculative execution to tolerate latency, microarchitectural and device-level optimizations to reduce latency, and instruction elimination mechanisms to improve energy efficiency. These works have substantially advanced the state-of-the-art in addressing the long-standing memory bottleneck problem and its impact on processor performance and energy efficiency. Building on this foundation, this dissertation proposes new MLdriven and data-aware microarchitectural techniques that further extend these directions and push the state-of-the-art in alleviating the ever-growing memory bottleneck.

Chapter 3

A Primer on Reinforcement and Perceptron Learning Reinforcement learning [820] and perceptron learning [700] are two powerful machine learning (ML) methods that have found increasing adoption in various aspects of processor design. In this chapter, we first provide a brief background on these two learning methods, and then survey prior works that exploit them for processor design—ranging from managing microarchitectural mechanisms (e.g., caching, prefetching) to higher-level design tasks (e.g., floorplanning).

3.1

Reinforcement Learning

Reinforcement learning (RL) [383, 820], in its simplest form, is the algorithmic approach to learn how to take an action in a given situation to maximize a numerical reward signal. A typical RL system comprises of two main components: the agent and the environment, as shown in Figure 3.1. The agent is the entity that takes actions. The agent resides in the environment and interacts with it in discrete timesteps. At each timestep 𝑡, the agent observes the current state of the environment 𝑆𝑡 and takes action 𝐴𝑡 . Upon receiving the action, the environment transitions to a new state St+1 , and emits an immediate reward 𝑅𝑡+1 , which is delivered immediately or later to the agent. The reward scheme encapsulates the agent’s objective and drives the agent towards taking optimal actions.

Agent State (St )

Reward (Rt+1 ) Action (At )

Environment Figure 3.1: Interaction between an agent and the environment in a reinforcement learning system.

CHAPTER 3: A PRIMER ON REINFORCEMENT AND PERCEPTRON LEARNING

The policy of the agent dictates it to take a certain action in a given state. The agent’s goal is to find the optimal policy that maximizes the cumulative reward collected from the environment over time. The expected cumulative reward by taking an action 𝐴 in a given state 𝑆 is defined as the Q-value of the state-action pair (denoted as 𝑄 (𝑆, 𝐴)). At every timestep 𝑡, the agent iteratively optimizes its policy in two steps: (1) the agent updates the Q-value of a state-action pair using the reward collected in the current timestep, and (2) the agent optimizes its current policy using the newly updated Q-value. Updating Q-values Prior works have proposed numerous algorithms to update the Q-values with varying computational complexities. SARSA [711] is one such algorithm that strikes a good trade-off between the learning accuracy and the computational complexity, which makes it suitable for online architectural decision making. In SARSA, if at a given timestep 𝑡, the agent observes a state 𝑆𝑡 , takes an action 𝐴𝑡 , while the environment transitions to a new state 𝑆𝑡+1 and emits a reward 𝑅𝑡+1 and the agent takes action 𝐴𝑡+1 in the new state, the Q-value of the old state-action pair 𝑄 (𝑆𝑡 , 𝐴𝑡 ) is iteratively optimized using the SARSA [711,820] algorithm, as shown in Eqn. (3.1): 𝑄 (𝑆𝑡 , 𝐴𝑡 ) ← 𝑄 (𝑆𝑡 , 𝐴𝑡 ) (3.1) + 𝛼 [𝑅𝑡+1 + 𝛾𝑄 (𝑆𝑡+1, 𝐴𝑡+1 ) − 𝑄 (𝑆𝑡 , 𝐴𝑡 )] 𝛼 is the learning rate parameter that controls the convergence rate of Q-values. 𝛾 is the discount factor, which is used to assign more weight to the immediate reward received by the agent at any given timestep than to the delayed future rewards. A 𝛾 value closer to 1 gives a “far-sighted" planning capability to the agent, i.e., the agent can trade off a low immediate reward to gain higher rewards in the future. This is particularly useful in creating an autonomous agent that can anticipate the long-term effects of taking an action to optimize its policy that gets closer to optimal over time. Policy Optimization To find a policy that maximizes the cumulative reward collected over time, a purely-greedy agent always exploits the action 𝐴 in a given state 𝑆 that provides the highest Q-value 𝑄 (𝑆, 𝐴). However, greedy exploitation can leave the state-action space under-explored. Thus, in order to strike a balance between exploration and exploitation, an 𝜖-greedy agent stochastically takes a random action with a low probability of 𝜖 (called exploration rate); otherwise, it selects the action that provides the highest Q-value [820]. In short, the Q-value serves as the cornerstone of reinforcement learning. By iteratively learning Q-values of state-action pairs, an RL-agent continuously optimizes its policy to take actions that get closer to optimal over time.

3.2. PERCEPTRON LEARNING

3.2

Perceptron Learning

Perceptron learning, whose roots can be traced back to [548] and which was demonstrated by Rosenblatt [700], is a simplified learning model that mimics biological neurons. Figure 3.2 shows a single-layer perceptron network where each input is connected to the output via an artificial neuron. Each artificial neuron is represented by a numeric value, called weight.

…

xn Perceptron inputs

Perceptron output Artiﬁcial neuron

y Figure 3.2: Overview of a single-layer perceptron model. Each blue circle denotes an input and the green circle denotes the output of the perceptron.

The perceptron network as a whole iteratively learns a binary classification function 𝑓 (𝑥) (shown in Eq. 3.2), a function that maps the input 𝑋 (a vector of 𝑛 values) to a single binary output. 𝑓 (𝑥) =

1 0

Í if 𝑤 0 + 𝑛𝑖=1 𝑤𝑖 𝑥𝑖 > 0 otherwise

(3.2)

The perceptron learning algorithm starts by initializing the weights of all neurons and iteratively trains the weights using each input vector from the training dataset in two steps. First, for an input vector 𝑋 , the perceptron network computes a binary output using Eq. 3.2 and the current weight values of its neurons. Second, if the computed output differs from the desired output for that input vector provided by the dataset, the weight of each neuron is updated [700]. This iterative process is repeated until the error between the computed and desired output falls below a user-specified threshold.

3.3

Application of Machine Learning in Processor Design

The works that apply ML-based methods in processor design can be broadly classified into two categories: (1) works that use ML to manage microarchitectural decision making, and (2) works that use ML for high-level design-space exploration. We briefly summarize such works in this section.

3.3.1

ML-Managed Microarchitectures

Researchers have proposed ML-based algorithms for various microarchitectural decisionmaking tasks. Prominent examples include memory scheduling [383, 576], cache manage-

CHAPTER 3: A PRIMER ON REINFORCEMENT AND PERCEPTRON LEARNING

ment [122, 519, 743, 766, 825], branch direction prediction [309, 405, 410–412, 823, 824, 907, 908, 937], branch target prediction [315], branch confidence estimation [83], address translation [542], cache reuse prediction [414, 825], hardware prefetching [260, 358, 660, 661, 767, 767, 768, 909], and prefetch usefulness prediction [148, 399].

3.3.2

Application of ML in High-Level Processor/System Design

Researchers have also explored ML techniques to explore the large microarchitectural design space, e.g., system modeling [238, 243, 252, 508], GPU performance modeling [99, 100, 124, 402, 423, 624, 879], CPU performance modeling [276, 426, 427, 481, 602, 923, 924], data-center-scale system modeling [175, 235, 338, 385, 447, 540, 709, 891], NoC design [224, 249, 250, 269, 294, 509, 849, 903, 904, 922], chip floorplanning [207, 332, 543, 559–561], hardware resource assignment and task allocation [257, 275, 306, 333, 391, 442, 528, 885, 899], power management [70, 118, 226, 236, 366, 381, 430, 538, 631, 686, 873], scheduling [285, 853, 871], and various aspects of chip design [206, 349, 367, 512, 522, 531, 550, 606, 646, 688, 699, 708, 744, 859, 878, 884, 906, 930, 933].

3.4

Summary and Further Reading

In summary, this chapter outlines the foundations of reinforcement and perceptron learning and surveys a broad set of works that employ machine learning for processor design, ranging from microarchitectural mechanisms to higher-level design tasks. Together, these discussions underscore the increasing importance of data-driven methods in modern architecture, complementing the dissertation’s broader vision of moving beyond data-agnostic design. We refer interested readers other works that give a more comprehensive view of machine-learning-aided techniques in processor and system design [580,662,883]. The next chapters build on the background presented here to progressively propose, design, and evaluate machine-learning-based microarchitectural techniques.

Chapter 4

Validating Generalization of ML-Driven Microarchitectural Techniques to Emerging Artificial Intelligence (AI) and Industrial Workloads 4.1

The Generalization Challenge

The design of conventional microarchitectural techniques has predominantly relied on rigid, rule-based heuristics. These heuristics, often derived from a computer architect’s intuition or exhaustive search over a narrow set of workloads, aim to identify a general-case behavior that yields performance gains across common benchmarks. While effective for well-characterized workloads, such static designs lack the intrinsic flexibility required to adapt to rapidly-evolving modern workloads, as we show in Chapter 1. In this dissertation, we propose a fundamental shift in this design paradigm by replacing human-crafted heuristics with machine learning (ML)-driven microarchitectural policies. By exploiting ML methods, such as reinforcement learning (RL) in Pythia [136] and Athena [138] and perceptron learning in Hermes [133], we enable the microarchitecture to autonomously and continuously learn from application and system-generated data. However, such ML-driven policies necessitate a rigorous verification of their ability to generalize their performance benefits across a broad range of workloads beyond those used at the design time. We observe that the evaluation of many prior microarchitectural techniques often suffers from design-time bias, i.e., these techniques are often iteratively tuned and evaluated to maximize the performance on the same set of well-known standard benchmarks (e.g., SPEC CPU workloads) [84, 85, 104, 117, 151, 203, 237, 261, 264–268, 274, 319, 325, 357, 378, 383, 384, 388, 389, 397, 408, 414, 428, 452, 468, 475, 479, 483, 485, 524, 556, 566, 588, 589, 628, 640, 659, 671, 677, 728, 729, 741, 742, 742, 761, 799, 799, 808, 856, 877, 881, 882, 898, 920]. We argue that this conventional evaluation methodology is inadequate for validating an ML-driven microarchitectural technique for two reasons. First, while standard benchmark suites provide a strong baseline, prior works have shown that they often do not represent complex workload behaviors of contemporary production workloads, such as emerging artificial intelligence (AI) 25

CHAPTER 4: VALIDATING GENERALIZATION OF ML-DRIVEN TECHNIQUES

applications [253, 536, 678, 679, 698, 834] and workloads running in large-scale datacenter fleets [111,156,281,286–288,440,454–456,527,630,684,723,726,797,867]. Second, using the same workloads for both iterative tuning and final evaluation risks overfitting the ML-driven policy. Thus, to truly demonstrate the benefits of the ML-driven microarchitectural techniques presented in this dissertation, we must demonstrate that the proposed techniques do not merely memorize the characteristics of well-known standard benchmarks, but instead generalize and adapt to previously-unseen emerging workloads. The goal of this chapter is to establish a rigorous and systematic methodology to validate the generalization of the ML-driven microarchitectural techniques introduced in this dissertation. To this end, we leverage the unique infrastructure and workload traces we developed for the 4th Data Prefetching Championship (DPC4) [135] to serve as an independent, industrial-grade stress test for our mechanisms.

4.2

The 4th Data Prefetching Championship (DPC4)

We organized the 4th Data Prefetching Championship (DPC4) [135], in conjunction with the 32nd IEEE International Symposium on High-Performance Computer Architecture (HPCA) on February 1, 2026, in Sydney, Australia. The goal of the championship was to evaluate innovative data prefetching ideas under a common evaluation framework using a broad range of diverse emerging workloads. While prior data prefetching championships primarily relied on conventional benchmark suites, most notably SPEC CPU workloads, as the basis for evaluation [1, 2], we substantially expanded the workload corpus used in DPC4. We collected and open-sourced 610 single-core workload traces spanning multiple emerging application domains. Of these, 483 traces have never been publicly released or evaluated by the computer architecture research community prior to DPC4. This includes (1) 81 traces from emerging state-of-the-art AI applications (e.g., Llama2 [834], ViT [253], CLIP [678], Stable Diffusion [698], Whisper [679], and BioGPT [536]), (2) 43 traces from graph mining workloads [147], and (3) 359 traces captured from real-world workloads running on Google’s datacenter fleet [21]. These workloads exhibit diverse and complex memory access characteristics that differ significantly from those observed in traditional SPEC CPU benchmarks. By incorporating these traces into a unified evaluation framework, DPC4 provided a substantially more representative stress-testing environment for modern data prefetching mechanisms. All DPC4 competitors were evaluated using the entire trace corpus across three processor configurations to tangibly improve the state-of-the-art in data prefetching [152, 199, 490, 554, 670, 777, 833, 928]. All traces can be downloaded freely from the DPC4 repository [14].

4.3: EXPLOITING DPC4 INFRASTRUCTURE FOR VALIDATION

4.3

Exploiting DPC4 Infrastructure to Validate Generalization of ML-Driven Microarchitectural Techniques

We leverage the DPC4 infrastructure to establish a rigorous evaluation methodology to validate the generalization capability of the three ML-driven microarchitectural techniques proposed in this dissertation (i.e., Pythia [136] in Chapter 5, Hermes [133] in Chapter 6, and Athena [138] in Chapter 7). More specifically, we use the corpus of 483 previously-unseen single-core DPC4 workload traces that span across three application domains of AI, graph mining, and Google datacenter, to evaluate all three ML-driven techniques. To further increase our evaluation rigor, we also construct 966 random four-core trace mixes from the 483 single-core traces using both intra-domain (i.e., picking four traces at random from the same application domain) and interdomain (i.e., picking four traces at random from any application domain) trace mixing strategies. Table 4.1 summarizes the traces considered in this evaluation methodology.

Single-core

Four-core

Workload type AI/ML applications (AIML) Graph Mining Suite (GMS) Google datacenter workloads (Google) Total AI application mix (AIML-mix) Graph Mining application mix (GMS-mix) Google workload mixes (Google-mix) Random application mixes (Random-mix) Total

# traces 81 43 359 483 81 43 359 483 966

Table 4.1: DPC4 workload traces used for evaluation.

Rationale Behind the Methodology. As all three ML-driven techniques proposed in this dissertation - Pythia, Hermes, and Athena - have been designed, implemented, and finalized long before the collection of DPC4 workload traces, this trace corpus never contributed to the design-space exploration (e.g., feature selection, hyperparameter optimization, reward design, policy structure selection, or architectural parameter tuning) of Pythia, Hermes, or Athena. Moreover, we do not perform any additional hyperparameter search, feature engineering, or architectural adjustment for these traces. Instead, we evaluate each mechanism with its baseline configuration. Thus, this methodology enables us to empirically answer a central research question: do ML-driven microarchitectural techniques merely memorize the characteristics of familiar benchmark suites, or do they learn policies that generalize across fundamentally different and previously-unseen workloads? By demonstrating sustained performance improvements under this “blind” evaluation methodology (as we show in §5.6.7, §6.6.7, and §7.6.7), we provide strong empirical evidence that the proposed mechanisms generalize beyond their design-time workloads and adapt effectively to emerging AI and industrial applications.

4.4

CHAPTER 4: VALIDATING GENERALIZATION OF ML-DRIVEN TECHNIQUES

Summary

This chapter addresses the methodological challenge of validating whether ML-driven microarchitectural techniques generalize beyond their design-time workloads. We argue that conventional evaluation practices, which rely heavily on standard benchmark suites such as SPEC CPU, risk design-time bias and potential overfitting. To address this limitation, we introduce the 4th Data Prefetching Championship (DPC4) infrastructure, which expands the evaluation space to emerging AI applications, graph mining workloads, and real-world workloads running on Google’s datacenter fleet. The DPC4 trace corpus contains 610 traces, including 483 traces previously unseen by the research community and exhibiting memory behaviors distinct from traditional benchmarks. We leverage this corpus to construct a blind stress-testing methodology for Pythia, Hermes, and Athena. We evaluate all three mechanisms on the 483 unseen single-core traces and on 966 randomly generated four-core mixes derived from them. Importantly, we finalized the three mechanisms prior to the collection of these traces and performed no additional hyperparameter tuning, feature engineering, or architectural adjustments. The results presented in subsequent chapters demonstrate that the proposed ML-driven techniques sustain their performance benefits under these rigorous and previously-unseen workloads, thereby providing strong empirical evidence of their generalization capability.

Chapter 5

Hardware Prefetching using Online Reinforcement Learning Prefetching is a key memory latency-hiding technique employed in most high-performance processor. A prefetcher predicts the addresses of long-latency memory requests and fetches the corresponding data from main memory to on-chip caches before the program executing on the processor demands it. In this chapter, we study the behavior of a wide range of prefetchers proposed in the literature, build a comprehensive understanding of how the data-agnostic nature of their design restricts their effectiveness, and propose a novel data-driven hardware prefetcher exploiting reinforcement learning that advances the state-of-the-art in prefetching.

5.1

Motivation and Goal

As a program repeatedly accesses over its data structures, it creates patterns in its memory request addresses. A prefetcher tries to identify such memory access patterns from past memory requests to predict the addresses of future memory requests. To quickly identify a memory access pattern, a prefetcher typically uses some program context information to examine only a subset of memory requests. We call this program context a feature. The prefetcher associates a memory access pattern with a feature and generates prefetches following the same pattern when the feature reoccurs during program execution. Past research has proposed numerous prefetchers that consistently pushed the limits of prefetch coverage (i.e., the fraction of memory requests predicted by the prefetcher) and accuracy (i.e., the fraction of prefetch requests that are actually demanded by the program) by exploiting various program features, e.g., program counter (PC), cacheline address (Address), page offset of a cacheline (Offset), or a simple combination of such features using simple operations like concatenation (+) [117,120,141,148,201,290,301,302,384,428,465,475,479,556,628, 671, 761, 765, 792, 799]. For example, a PC-based stride prefetcher [301, 302, 428] uses PC as the feature to learn the constant stride between two consecutive memory accesses caused by the same PC. VLDP [765] and SPP [465] use a sequence of cacheline address deltas as the feature to predict the next cacheline address delta. Kumar and Wilkerson [479] use PC+Address of the first access in a memory region as the feature to predict the spatial memory access footprint in the entire memory region. SMS [792] empirically finds PC+Offset of the first access in a memory region to be a better feature to predict the memory access footprint. Bingo [120] combines the features from [479] and SMS and uses PC+Address and PC+Offset as its features. 29

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

Accurate and timely prefetch requests reduce the long memory access latency experienced by the processor, thereby improving overall system performance. However, speculative prefetch requests can cause undesirable effects on the system (e.g., increased memory bandwidth consumption, cache pollution, memory access interference, etc.), which can reduce or negate the performance improvement gained by hiding memory access latency [267, 799]. Thus, a good prefetcher aims to maximize its benefits while minimizing its undesirable effects on the system.

5.1.1

Key Observations

Even though there is a large number of prefetchers proposed in the literature, we observe three key shortcomings in almost every prior prefetcher design that significantly limits its performance benefits over a wide range of workloads and system configurations: (1) the use of mainly a single program feature for prefetch prediction, (2) lack of inherent system awareness, and (3) lack of ability to customize the prefetcher design to seamlessly adapt to a wide range of workload and system configurations. Prefetching using a Single Program Feature Almost every prior prefetcher relies on only one program feature to correlate with the program memory access pattern and generate prefetch requests [117, 141, 148, 201, 290, 301, 302, 384, 428, 465, 475, 479, 556, 628, 671, 761, 765, 792, 799]. As a result, a prefetcher typically provides good (or poor) performance benefits in mainly those workloads where the correlation between the feature used by the prefetcher and program’s memory access pattern is dominantly present (or absent). To demonstrate this, we show the coverage and overpredictions (i.e., prefetched memory requests that do not get demanded by the processor) of two recently proposed prefetchers, SPP [465] and Bingo [120], and our new proposal Pythia (see §5.4) for six example workloads (§5.5 discusses our experimental methodology) in Figure 5.1(a). Figure 5.1(b) shows the performance of SPP, Bingo and Pythia on the same workloads. As we see in Figure 5.1(a), Bingo provides higher prefetch coverage than SPP in sphinx3, PARSEC-Canneal, and PARSEC-Facesim, where the correlation exists between the first access in a memory region and the other accesses in the same region. As a result, Bingo performs better than SPP in these workloads (Figure 5.1(b)). In contrast, for workloads like GemsFDTD that have regular access patterns within a physical page, SPP’s sequence of deltas feature provides better coverage and performance than Bingo. Lack of Inherent System Awareness All prior prefetchers either completely neglect their undesirable effects on the system (e.g., memory bandwidth usage, cache pollution, memory access interference, system energy consumption, etc.) [117, 120, 148, 201, 290, 301, 302, 384, 428, 465, 475, 479, 556, 628, 671, 761, 765, 792] or incorporate system awareness as an afterthought (i.e., a separate control component) to the underlying system-unaware prefetch algorithm [141, 192, 265, 267, 268, 483, 485, 510, 511, 583,

5.1: PYTHIA: MOTIVATION AND GOAL

574%

250%

302%

368%

529%

(a)

Fraction of baseline LLC misses

Covered 200%

Uncovered Overpredicted

150% 100% 50% 0% SPP

Bingo Pythia

482.sphinx3-417B

SPP

Bingo Pythia

PARSEC-Canneal

SPP

Bingo Pythia

PARSEC-Facesim

SPP

Bingo Pythia

459.GemsFDTD-765B

SPP

Bingo Pythia Ligra-CC

SPP

Bingo Pythia

Ligra-PageRankDelta

60% IPC improvement over baseline (%)

SPP

Bingo

(b)

Pythia

40% 20% 0% -20% 482.sphinx3-417B

PARSEC-Canneal

PARSEC-Facesim

459.GemsFDTD-765B

Ligra-CC

Ligra-PageRankDelta

Figure 5.1: Comparison of (a) coverage, overprediction, and (b) performance of two recentlyproposed prefetchers, SPP [465] and Bingo [120], and our new proposal, Pythia.

799, 932]. Due to the lack of inherent system awareness, a prefetcher often loses its performance gain in resource-constrained scenarios. For example, as shown in Figure 5.1(a), Bingo achieves similar prefetch coverage in Ligra-CC as compared to PARSEC-Canneal, while generating significantly lower overpredictions in Ligra-CC than PARSEC-Canneal. However, Bingo loses performance in Ligra-CC by 1.9% compared to a no-prefetching baseline, whereas it improves performance by 6.4% in PARSEC-Canneal (Figure 5.1(b)). This contrasting outcome is due to Bingo’s lack of awareness of the memory bandwidth usage. Without prefetching, Ligra-CC consumes higher memory bandwidth than PARSEC-Canneal. As a result, each overprediction made by Bingo in Ligra-CC wastes more precious memory bandwidth and is more detrimental to performance than that in PARSEC-Canneal. Lack of Online Prefetcher Design Customization The high design complexity of architecting a multi-feature, system-aware prefetcher has traditionally compelled architects to statically select only one program feature at design time. With every new prefetcher, architects design new rigid hardware structures to exploit the selected program feature. To exploit a new program feature for higher performance benefits, one must design a new prefetcher from scratch and extensively evaluate and verify it both in pre-silicon and post-silicon realization. Due to the rigid design-time decisions, the hardware structures proposed by prior prefetchers cannot be customized online in silicon either to exploit any other program feature or to change the prefetcher’s objective (e.g., to increase/decrease coverage, accuracy, or timeliness) so that it can seamlessly adapt to varying workloads and system configurations.

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

5.1.2

Our Goal

Our goal is to design a single prefetching framework that (1) can holistically learn to prefetch using both multiple different types of program features and system-level feedback information that is inherent to the design, and (2) can be easily customized in silicon via simple configuration registers to exploit different types of program features and/or to change the objective of the prefetcher (e.g., increasing/decreasing coverage, accuracy, or timeliness) without any changes to the underlying hardware.

5.2

Formulating Prefetching using Reinforcement Learning

To this end, we formulate prefetching from the grounds-up as a reinforcement learning (RL) problem. More specifically, we propose a new hardware data prefetcher, named Pythia,1 based on RL, where the prefetcher itself acts as the RL agent, the processor and memory system act as the environment, and the agent (i.e., the prefetcher) autonomously learns to prefetch by interacting with its environment.

5.2.1

Why is RL a Good Fit for Modeling Prefetching?

RL has been recently successfully demonstrated to solve complex problems like mastering human-like control on Atari [562] and Go [772, 773]. We argue that RL is an inherent fit to model a hardware prefetcher for three key reasons. Adaptive Learning in a Complex State Space. As we show in §5.1, the benefits of a prefetcher not only depend on its coverage and accuracy but also on its undesirable effects on the system, like memory bandwidth usage. In other words, it is not sufficient for a prefetcher only to make highly accurate predictions. Instead, a prefetcher should be performance-driven. A prefetcher should have the capability to adaptively trade-off coverage for higher accuracy (and vice-versa) depending on its impact on the overall system to provide a robust performance improvement with varying workloads and system configurations. This adaptive and performance-driven nature of prefetching in a complex state space makes RL a good fit for modeling a prefetcher as an autonomous agent who learns to prefetch by interacting with the system. Online Learning. An RL agent does not require an expensive offline training phase. Instead, it can continuously learn online by iteratively optimizing its policy using the rewards received from the environment. A hardware prefetcher, similar to an RL agent, also needs to continuously learn from the changing workload behavior and system conditions to provide consistent performance benefits. The online learning requirement of prefetching makes RL an inherent fit to model a hardware prefetcher. 1 Pythia, according to Greek mythology, is the oracle of Delphi who is known for accurate prophecies [65].

5.3: PYTHIA: OVERVIEW

Ease of Implementation. Prior works have evaluated many sophisticated machine learning models like simple neural networks [661], LSTMs [358, 767], and Graph Neural Networks (GNNs) [768] as models for hardware prefetching. Even though these techniques show encouraging results in accurately predicting memory accesses, they fall short especially in two major aspects. First, these models’ sizes often exceed even the largest caches in traditional processors [358, 661, 767, 768], making them impractical (or at best very difficult) to implement. Second, due to the vast amount of computation they require for inference, these models’ inference latency is much higher than an acceptable latency of a prefetcher at any cache level. On the other hand, we can efficiently implement an RL-based model, as we demonstrate in this work (see §5.4), that can quickly make predictions and can be relatively easily adopted in a real processor.

5.3

Pythia: Overview

Pythia formulates prefetching as a reinforcement learning problem, as shown in Figure 5.2. Specifically, Pythia as the RL-agent that learns to make accurate, timely, and system-aware prefetch decisions by interacting with the environment, i.e., the processor and the memory subsystem. Each timestep corresponds to a new demand request seen by Pythia. With every new demand request, Pythia observes the state of the processor and the memory subsystem and takes a prefetch action. For every prefetch action (including not to prefetch), Pythia receives a numerical reward that evaluates the accuracy and timeliness of the prefetch action taking into account various system-level feedback information. Pythia’s goal is to find the optimal prefetching policy that would maximize the number of accurate and timely prefetch requests, taking system-level feedback information into account. While Pythia’s framework is general enough to incorporate any type of system-level feedback into its decision making, in this work we demonstrate Pythia using memory bandwidth usage as the system-level feedback information.

Prefetcher Features of memory request to address A (e.g., PC)

Reward

Prefetch from address A+oﬀset (O)

Processor & Memory Subsystem Figure 5.2: Formulating the prefetcher as an RL-agent.

We now formally define the three pillars of our RL-based prefetcher: the state space, the actions, and the reward scheme.

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

5.3.1

State

We define the state as a 𝑘-dimensional vector of program features. 𝑆 ≡ {𝜙𝑆1, 𝜙𝑆2, . . . , 𝜙𝑆𝑘 }

(5.1)

Each program feature is composed of at most two components: (1) program control-flow component, and (2) program data-flow component. The control-flow component is further made up of simple information like load-PC (i.e., the PC of a load instruction) or branch-PC (i.e., the PC of a branch instruction that immediately precedes a load instruction), and a history that denotes whether this information is extracted only from the current demand request or a series of past demand requests. Similarly, the data-flow component is made up of simple information like cacheline address, physical page number, page offset, cacheline delta, and its corresponding history. Table 5.1 shows some example program features. Although Pythia can theoretically learn to prefetch using any number of such program features, we fix the statevector dimension (i.e., 𝑘) at design time given a limited storage budget in hardware. However, the exact selection of 𝑘 program features out of all possible program features is configurable online using simple configuration registers. In §5.4.3, we provide an automated feature selection method to find a vector of program features to be used at design time. Control-flow Feature Last 3-PCs Last 4-deltas PC+Delta Last 4-PCs+Page no.

Data-flow

Info.

History

Info.

History

PC ✖ PC PC

last 3 ✖ current last 4

✖ Cacheline delta Cacheline delta Page no.

✖ last 4 current current

Table 5.1: Example program features.

5.3.2

Action

We define the action of the RL-agent as selecting a prefetch offset (i.e., a delta, "O" in Figure 5.2, between the predicted and the demanded cacheline address) from a set of candidate prefetch offsets. As every post-L1-cache prefetcher generates prefetch requests within a physical page [120, 141, 148, 290, 384, 465, 475, 479, 556, 628, 671, 761, 765, 792, 799], the list of prefetch offsets only contains values in the range of [−63, 63] for a system with a traditionally-sized 4KB page and 64B cacheline. Using prefetch offsets as actions (instead of full cacheline addresses) drastically reduces the action space size. We further reduce the action space size by fine tuning, as described in§5.4.3. A prefetch offset of zero means no prefetch is generated.

5.4: PYTHIA: DETAILED DESIGN

5.3.3

Reward

The reward structure defines the prefetcher’s objective. We define five different reward levels as follows. • Accurate and timely (R𝐴𝑇 ). This reward is assigned to an action whose corresponding prefetch address gets demanded after the prefetch fill. • Accurate but late (R𝐴𝐿 ). This reward is assigned to an action whose corresponding prefetch address gets demanded before the prefetch fill. • Loss of coverage (R𝐶𝐿 ). This reward is assigned to an action whose corresponding prefetch address is to a different physical page than the demand access that led to the prefetch. • Inaccurate (R𝐼 𝑁 ). This reward is assigned to an action whose corresponding prefetch address does not get demanded in a temporal window. The reward is classified into two sub-levels: inaccurate given low bandwidth usage (R𝐿𝐼𝑁 ) and inaccurate given high bandwidth usage (R𝐻𝐼𝑁 ). • No-prefetch (R𝑁 𝑃 ). This reward is assigned when Pythia decides not to prefetch. This reward level is also classified into two sub-levels: no-prefetch given low bandwidth usage (R𝐿𝑁 𝑃 ) and no-prefetch given high bandwidth usage (R𝐻𝑁 𝑃 ). By increasing (decreasing) a reward level value, we reinforce (deter) Pythia to collect such rewards from the environment in the future. R𝐴𝑇 and R𝐴𝐿 are used to guide Pythia to generate more accurate and timely prefetch requests. R𝐶𝐿 is used to guide Pythia to generate prefetches within the physical page of the triggering demand request. R𝐼 𝑁 and R𝑁 𝑃 are used to define Pythia’s prefetching strategy with respect to memory bandwidth usage feedback. In §5.4.3, we provide an automated method to configure the reward values. The reward values can be easily customized further for target workload suites to extract higher performance gains (see §5.6.6).

5.4

Pythia: Detailed Design

Figure 5.3 shows a high-level overview of Pythia. Pythia is mainly comprised of two hardware structures: Q-Value Store (QVStore) and Evaluation Queue (EQ). The purpose of QVStore is to record Q-values for all state-action pairs that are observed by Pythia. The purpose of EQ is to maintain a first-in-first-out list of Pythia’s recently-taken actions.2 Every EQ entry holds three pieces of information: (1) the taken action, (2) the prefetch address generated for the corresponding action, and (3) a filled bit. A set filled bit indicates that the prefetch request has been filled into the cache. 2 Pythia keeps track of recently-taken actions because it cannot always immediately assign a reward to an

action, as the usefulness of the generated prefetch request (i.e., if and when the prefetched address is demanded by the processor) is not immediately known while the action is being taken. During EQ residency, if the address of a demand request matches with the prefetch address stored in an EQ entry, the corresponding action is considered to have generated a useful prefetch request.

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

For every new demand request, Pythia first checks the EQ with the demanded memory address ( 1 ). If the address is present in the EQ (i.e., Pythia has issued a prefetch request for this address in the past), it signifies that the prefetch action corresponding to the EQ entry has generated a useful prefetch request. As such, Pythia assigns a reward (either R𝐴𝑇 or R𝐴𝐿 ) to the EQ entry, based on whether or not the EQ entry’s filled bit is set. Next, Pythia extracts the state-vector from the attributes of the demand request (e.g., PC, address, cacheline delta, etc.) ( 2 ) and looks up QVStore to find the action with the maximum Q-value for the given statevector ( 3 ). Pythia selects the action with the maximum Q-value to generate prefetch request and issues the request to the memory hierarchy ( 4 ). At the same time, Pythia inserts the selected prefetch action, its corresponding prefetched memory address, and the state-vector into EQ ( 5 ). Note that, a no-prefetch action or an action that prefetches an address beyond the current physical page is also inserted into EQ. The reward for such an action is instantaneously assigned to the EQ entry. When an EQ entry gets evicted, the state-action pair and the reward stored in the evicted EQ entry are used to update the Q-value in the QVStore ( 6 ). For every prefetch fill in cache, Pythia looks up EQ with the prefetch address and sets the filled bit in the matching EQ entry indicating that the prefetch request has been filled into the cache ( 7 ). Pythia uses this filled bit in 1 to classify actions that generated timely or late prefetches.3 ?#%+&,:(&!.,#-%&)#,:&@*A&1<6*B5(

!"#$%& '"()"*+

" ,+$+" -".+/0

3--4 5/& 167,-'(

=(%('*,( /'(9(,.: !"

!"#

8,9(2:* ;',2#256:

!$ !%

0>#.,&01&(%,'2&*%+& 5/+*,(&167,-'(

+/0#$%,*1&(2, -+01&(2,. $

!"#$%#&'()*+%,%, -!+. ! 7(,&9#BB(+&C#, !""#$%&'()*'+&,-& .-''("/-%+#%$ 01&(%,'2

8%"(',&/'(9(,.:&*.,#-%&;& 7,*,(<!.,#-%&/*#'&#%&01

32,4,&56*7'$$*

Figure 5.3: Overview of Pythia.

5.4.1

RL-Based Prefetching Algorithm

Algorithm 1 shows Pythia’s RL-based prefetching algorithm. Initially, all entries in QVStore 1 are reset to the highest possible Q-value ( 1−𝛾 ) and the EQ is cleared (lines 2-3). For every 3 In this work, we define prefetch timeliness as a binary value due to its measurement simplicity. One can

easily make the definition non-binary by storing three timestamps per EQ entry: (1) when the prefetch is issued (𝑡𝑖𝑠𝑠𝑢𝑒 ), (2) when the prefetch is filled (𝑡 𝑓 𝑖𝑙𝑙 ), and (3) when a demand is generated for the same prefetched address (𝑡𝑑𝑒𝑚𝑎𝑛𝑑 ).

5.4: PYTHIA: DETAILED DESIGN

Algorithm 1 Pythia’s reinforcement learning based prefetching algorithm 1: procedure Initialize 1 2: initialize QVStore: 𝑄 (𝑆, 𝐴) ← 1−𝛾 3: clear EQ 4: 5: procedure Train_and_Predict(Addr) ⊲ Called for every demand request 6: 𝑒𝑛𝑡𝑟𝑦 ← 𝑠𝑒𝑎𝑟𝑐ℎ_𝐸𝑄 (𝐴𝑑𝑑𝑟 ) ⊲ Search EQ with the demand address 7: if entry is valid then 8: if 𝑒𝑛𝑡𝑟𝑦.𝑓 𝑖𝑙𝑙𝑒𝑑 == 𝑡𝑟𝑢𝑒 then 9: 𝑒𝑛𝑡𝑟𝑦.𝑟𝑒𝑤𝑎𝑟𝑑 ← R𝐴𝑇 ⊲ Prefetch already filled means accurate and timely prefetch 10: else 11: 𝑒𝑛𝑡𝑟𝑦.𝑟𝑒𝑤𝑎𝑟𝑑 ← R𝐴𝐿 ⊲ Otherwise, the prefetch is accurate but late 12: 𝑆 ← 𝑔𝑒𝑡_𝑠𝑡𝑎𝑡𝑒 () ⊲ Extract the state-vector from current demand request 13: if 𝑟𝑎𝑛𝑑 () ≤ 𝜖 then 14: 𝑎𝑐𝑡𝑖𝑜𝑛 ← 𝑔𝑒𝑡_𝑟𝑎𝑛𝑑𝑜𝑚_𝑎𝑐𝑡𝑖𝑜𝑛() ⊲ Explore state-action space with a small probability 15: else 16: 𝑎𝑐𝑡𝑖𝑜𝑛 ← argmax𝑎 𝑄 (𝑆, 𝑎) ⊲ Otherwise, select the action with the highest Q-value 17: 𝑝𝑟𝑒 𝑓 𝑒𝑡𝑐ℎ(𝐴𝑑𝑑𝑟 + 𝑂 𝑓 𝑓 𝑠𝑒𝑡 [𝑎𝑐𝑡𝑖𝑜𝑛]) ⊲ Generate prefetch address from the selected offset 18: 𝑒𝑛𝑡𝑟𝑦 ← 𝑐𝑟𝑒𝑎𝑡𝑒_𝐸𝑄_𝑒𝑛𝑡𝑟𝑦 (𝑆, 𝑎𝑐𝑡𝑖𝑜𝑛, 𝐴𝑑𝑑𝑟 + 𝑂 𝑓 𝑓 𝑠𝑒𝑡 [𝑎𝑐𝑡𝑖𝑜𝑛]) ⊲ Create a new EQ entry 19: if no prefetch action then 𝐿 𝐻 or 𝑅 𝐿 ) for no-prefetch ⊲ Immediately assign reward (𝑅𝑁 20: 𝑒𝑛𝑡𝑟𝑦.𝑟𝑒𝑤𝑎𝑟𝑑 ← R𝐻 𝑁 𝑃 or R𝑁 𝑃 𝑃 𝑁𝑃 21: else if out-of-page prefetch then 22: 𝑒𝑛𝑡𝑟𝑦.𝑟𝑒𝑤𝑎𝑟𝑑 ← R𝐶𝐿 ⊲ Immediately assign reward 𝑅𝐶𝐿 for out-of-page prefetch 23: 𝑑𝑞_𝑒𝑛𝑡𝑟𝑦 ← 𝑖𝑛𝑠𝑒𝑟𝑡_𝐸𝑄 (𝑒𝑛𝑡𝑟𝑦) ⊲ Insert the new EQ entry and get the evicted EQ entry. 24: if ℎ𝑎𝑠_𝑟𝑒𝑤𝑎𝑟𝑑 (𝑑𝑞_𝑒𝑛𝑡𝑟𝑦) == 𝑓 𝑎𝑙𝑠𝑒 then 𝐿 𝐿 25: 𝑑𝑞_𝑒𝑛𝑡𝑟𝑦.𝑟𝑒𝑤𝑎𝑟𝑑 ← R𝐻 ⊲ Unused prefetch; assign R𝐻 𝐼 𝑁 or R𝐼 𝑁 𝐼 𝑁 or R𝐼 𝑁 based on bandwidth 26: 𝑅 ← 𝑑𝑞_𝑒𝑛𝑡𝑟𝑦.𝑟𝑒𝑤𝑎𝑟𝑑 ⊲ Get the reward stored in the evicted entry 27: 𝑆 1 ← 𝑑𝑞_𝑒𝑛𝑡𝑟𝑦.𝑠𝑡𝑎𝑡𝑒; 𝐴1 ← 𝑑𝑞_𝑒𝑛𝑡𝑟𝑦.𝑎𝑐𝑡𝑖𝑜𝑛 ⊲ Get state and action from evicted EQ entry 28: 𝑆 2 ← 𝐸𝑄.ℎ𝑒𝑎𝑑.𝑠𝑡𝑎𝑡𝑒; 𝐴2 ← 𝐸𝑄.ℎ𝑒𝑎𝑑.𝑎𝑐𝑡𝑖𝑜𝑛 ⊲ Get state and action from EQ head 29: 𝑄 (𝑆 1, 𝐴1 ) ← 𝑄 (𝑆 1, 𝐴1 ) + 𝛼 [𝑅 + 𝛾𝑄 (𝑆 2, 𝐴2 ) − 𝑄 (𝑆 1, 𝐴1 )] ⊲ Perform the SARSA update 30: 31: procedure Prefetch_Fill(Addr) 32: 𝑠𝑒𝑎𝑟𝑐ℎ_𝑎𝑛𝑑_𝑚𝑎𝑟𝑘_𝐸𝑄 (𝐴𝑑𝑑𝑟, FILLED) ⊲ Mark prefetch fill bit

demand request to a cacheline address 𝐴𝑑𝑑𝑟 , Pythia searches for 𝐴𝑑𝑑𝑟 in EQ (line 6). If a matching entry is found, Pythia assigns a reward (either R𝐴𝑇 or R𝐴𝐿 ) based on the filled bit in the EQ entry (lines 8-11). Pythia then extracts the state-vector to stochastically select a prefetching action that provides the highest Q-value (lines 13-16). Pythia uses the selected action to generate the prefetch request (line 17) and creates a new EQ entry with the current state-vector, the selected action, and its corresponding prefetched address (line 18). In case of a no-prefetch action, or an action that prefetches beyond the current physical page, Pythia immediately assigns the reward to the newly-created EQ entry (lines 19-22). The EQ entry is then inserted, which evicts an entry from EQ. If the evicted EQ entry does not already have a reward assigned (indicating that the corresponding prefetch address is not demanded by the processor so far), Pythia assigns the reward R𝐻𝐼𝑁 or R𝐿𝐼𝑁 based on the current memory bandwidth usage (lines 25). Finally, the Q-value of the evicted state-action pair is updated via the SARSA algorithm (see §3.1), using the reward stored in the evicted EQ entry and the Q-value of the state-action pair in the head of the EQ-entry (lines 26-29).

5.4.2

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

Detailed Design of Pythia

We describe the organization of QVStore (§5.4.2), how Pythia searches QVStore to get the action with the maximum Q-value for a given state-vector ( 3 ) (§5.4.2), how Pythia assigns rewards to each taken action and how it updates Q-values ( 6 ) (§5.4.2). Organization of QVStore The purpose of QVStore is to record Q-values for all state-action pairs that Pythia observes. Unlike prior real-world applications of RL [562, 772, 773], which use deep neural networks to approximately store Q-values of every state-action pair, we propose a new, table-based, hierarchical QVStore organization that is custom-designed to our RL-agent. Figure 5.4(a) shows the high-level organization of QVStore and how the Q-value is retrieved from QVStore for a given state 𝑆 (which is a k-dimensional vector of program features, {𝜙𝑆1, 𝜙𝑆2, . . . , 𝜙𝑆𝑘 }) and an action 𝐴. As the state space grows rapidly with the state-vector dimension (𝑘) and the bits used to represent each feature, we employ a hierarchical organization for QVStore. We organize QVStore in 𝑘 partitions, each of which we call a vault. Each vault corresponds to one constituent feature of the state-vector and records the Q-values for the feature-action pair, 𝑄 (𝜙𝑆𝑖 , 𝐴). During the Q-value retrieval for a given state-action pair 𝑄 (𝑆, 𝐴), Pythia queries each vault in parallel to retrieve the Q-values of constituent featureaction pairs 𝑄 (𝜙𝑆𝑖 , 𝐴). The final Q-value of the state-action pair 𝑄 (𝑆, 𝐴) is computed as the maximum of all constituent feature-action Q-values, as Equation (5.2) shows). The maximum operation ensures that the state-action Q-value is driven by the constituent feature of the state-vector that has the highest feature-action Q-value. The vault organization enables QVStore to efficiently scale up to higher state-vector dimensions: one can increase the state-vector dimension by simply adding a new vault to the QVStore. 𝑄 (𝑆, 𝐴) = max 𝑄 (𝜙𝑆𝑖 , 𝐴)

(5.2)

𝑖∈(1,𝑘)

Figure 5.4(a) shows the organization of QVStore as a collection of multiple vaults. The purpose of a vault is to record Q-values of all feature-action pairs that Pythia observes for a specific feature type. A vault can be conceptually visualized as a monolithic two-dimensional table (as shown in Figure 5.4(a)), indexed by the feature and action values, that stores Q-value for every feature-action pair. However, the key challenge in implementing vault as a monolithic table is that the size of the table increases exponentially with a linear increase in the number of bits used to represent the feature. This not only makes the monolithic table organization impractical for implementation but also increases the design complexity to satisfy its latency and power requirements. One way to address this challenge is to quantize the feature space into a small number of tiles. Even though feature space quantization can achieve a drastic reduction in the monolithic table size, it requires a compromise between the resolution of a feature value and the generalization of feature values. We draw inspiration from tile coding [89, 383, 820] to strike a balance between resolution and generalization. Tile coding uses multiple overlapping hash functions

5.4: PYTHIA: DETAILED DESIGN

Action (A)

(a)

(b)

Index Generation

φ1S

Vault1

φ2S

Vault2

… φkS

Vaultk

φkS

Index Generation

Program feature

Plane 1

Index Generation

Q(φ1S , A)

Q(φ2S , A)

Q(φkS , A)

Feature-action Q-value

MAX

Q(φkS , A)

(c)

Feature-action Q-value

Shift

Q(S, A) State-action Q-value

φkS

Feature Index

Figure 5.4: (a) The QVStore is comprised of multiple vaults. (b) Each vault is comprised of multiple planes. (c) Index generation from feature value.

to quantize a feature value into smaller tiles. The quantization achieves generalization of similar feature values, whereas multiple hash functions increase resolution to represent a feature value. We leverage the idea of tile coding to organize a vault as a collection of 𝑁 small twodimensional tables, each of which we call a plane. Each plane entry stores a partial Q-value of a feature-action pair.4 As Figure 5.4(c) shows, to retrieve a feature-action Q-value 𝑄 (𝜙𝑆𝑖 , 𝐴), the given feature is first shifted by a shifting constant (which is randomly selected at design time), followed by a hashing to get the feature index for the given plane. This feature index, along with the action index, is used to retrieve the partial Q-value from the plane. The final feature-action Q-value is computed as the sum of all the partial Q-values from all planes, as shown in Figure 5.4(b). The use of tile coding provides two key advantages to Pythia. First, the tile coding of a feature enables the sharing of partial Q-values between similar feature values, which shortens prefetcher training time. Second, multiple planes reduces the chance of sharing partial Q-values between widely different feature values. Pipelined Organization of QVStore Search To generate a prefetch request, Pythia has to (1) look up the QVStore with the state-vector extracted from the current demand request, and (2) search for the action that has the maximum state-action Q-value ( 3 in Figure 5.3). As a result, the search operation lies on Pythia’s critical path and directly impacts Pythia’s prediction latency. To improve the prediction latency, we pipeline the search operation. The Q-value search operation is implemented in the following way. For a given statevector, Pythia iteratively retrieves the Q-value of each action. Pythia also maintains a variable, 4 Our application of tile coding is similar to that used in the self-optimizing memory controller (RLMC) [383].

The key difference is that RLMC uses a hybrid combination of feature and action values to index singledimensional planes, whereas Pythia uses feature and action values separately to index two-dimensional planes.

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

φ2S φ3S

Stage 1

Stage 2

+ +

Stage 3

Stage 4

φ1 S ,A )

Q(φ2S , A)

Q(S, A)

) 3 ,A φS

Get maximum of all feature-action Q-values to compute state-action Q-value Sum up all partial feature-action Retrieve partial Q-values feature-action Q-values

Q-MAX

φ1S

Index Generation

Stage 0

MAX

Track maximum state-action Q-value

Figure 5.5: Pipelined organization of QVStore search operation. The illustration depicts three program features, each having three planes.

𝑄𝑚𝑎𝑥 , that tracks the maximum Q-value found so far. 𝑄𝑚𝑎𝑥 gets compared to every retrieved Q-value. The search operation concludes when Q-values for all possible actions have been retrieved. We pipeline the search operation into five stages as Figure 5.5 shows. Pythia first computes the index for each plane and each constituent feature of the given state-vector (Stage 0). In Stage 1, Pythia uses the feature indices and an action index to retrieve the partial Q-values from each plane. In Stage 2, Pythia sums up the partial Q-values to get the feature-action Qvalue for each constituent feature. In Stage 3, Pythia computes the maximum of all featureaction Q-values to get the state-action Q-value. In Stage 4, the maximum state-action Q-value found so far is compared against the retrieved state-action Q-value, and the maximum Q-value is updated. Stage 2 (i.e., the partial Q-value summation) is the longest stage of the pipeline and thus it dictates the pipeline’s throughput. We accurately measure the area and power overhead of the pipelined implementation of the search operation by modeling Pythia using Chisel [10] hardware design language and synthesize the model using Synopsys design compiler [57] and 14-nm library from GlobalFoundries [18] (see §5.6.8). Assigning Rewards and Updating Q-values To track usefulness of the prefetched requests, Pythia maintains a first-in-first-out list of recently taken actions, along with their corresponding prefetch addresses in EQ. Every prefetch action is inserted into EQ. A reward gets assigned to every EQ entry before or when it gets evicted from EQ. During eviction, the reward and the state-action pair associated with the evicted EQ entry are used to update the corresponding Q-value in QVStore ( 6 in Figure 5.3). We describe how Pythia appropriately assigns rewards to each EQ entry. We divide the reward assignment into three classes based on when the reward gets assigned to an entry: (1) immediate reward assignment during EQ insertion, (2) reward assignment during EQ residency,

5.4: PYTHIA: DETAILED DESIGN

and (3) reward assignment during EQ eviction. If Pythia selects the action not to prefetch or one that generates a prefetch request beyond the current physical page, Pythia immediately assigns a reward to the EQ entry. For out-of-page prefetch action, Pythia assigns R𝐶𝐿 . For the action not to prefetch, Pythia assigns R𝐻𝑁 𝑃 or R𝐿𝑁 𝑃 , based on whether the current system memory bandwidth usage is high or low. If the address of a demand request matches with the prefetch address stored in an EQ entry during its residency, Pythia assigns R𝐴𝑇 or R𝐴𝐿 based on the filled bit of the EQ entry. If the filled bit is set, it indicates that the demand request is generated after the prefetch fill. Hence the prefetch is accurate and timely, and Pythia assigns the reward R𝐴𝑇 . Otherwise, Pythia assigns the reward R𝐴𝐿 . If a reward does not get assigned to an EQ entry until it is going to be evicted, it signifies that the corresponding prefetch address is not yet demanded by the processor. Thus, Pythia assigns a reward R𝐻𝐼𝑁 or R𝐿𝐼𝑁 to the entry during eviction based on whether the current system memory bandwidth usage is high or low.

5.4.3

Automated Design-Space Exploration

We propose an automated, performance-driven approach to systematically explore Pythia’s vast design space and derive a basic configuration5 with appropriate program features, action set, reward and hyperparameters. Table 5.2 shows the basic configuration. Features Prefetch Action List Reward Level Values Hyperparameters

PC+Delta, Sequence of last-4 deltas {-6,-3,-1,0,1,3,4,5,10,11,12,16,22,23,30,32} R𝐴𝑇 =20, R𝐴𝐿 =12, R𝐶𝐿 =−12, R𝐻 𝐼 𝑁 =−14, 𝐿 =−4 R𝐿𝐼𝑁 =−8, R𝐻 =−2, R 𝑁𝑃 𝑁𝑃 𝛼 = 0.0065, 𝛾 = 0.556, 𝜖 = 0.002

Table 5.2: Basic Pythia configuration derived from our automated design-space exploration.

Feature Selection We derive a list of possible program features for feature-space exploration in four steps. First, we derive a list of 4 control-flow components, and 8 data-flow components, which are mentioned in Table 5.3. Second, we combine each control-flow component with each data-flow component with the concatenation operation, to obtain a total of 32 possible program features. Third, we use the linear regression technique [348, 565, 727] to create any-one, any-two, and any-three feature-combinations from the set of 32 initial features, each providing a different state-vector. Fourth, we run Pythia with every state-vectors across all single-core workloads (§5.5) and select the winning state-vector that provides the highest performance gain over no-prefetching baseline. As Table 5.2 shows, the two constituent features of the winning state-vector are PC+Delta and Sequence of last-4 deltas. 5 Using a compute-grid with ten 28-core machines, the automated exploration across 150 workload traces

(mentioned in detail in §5.5) takes 44 hours to complete.

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

Control-flow Components

Data-flow Components

1. PC of load request 2. PC-path (XOR-ed last-3 PCs) 3. PC XOR-ed branch-PC 4. None

1. Load cacheline address 2. Page number 3. Page offset 4. Load address delta 5. Sequence of last-4 offsets 6. Sequence of last-4 deltas 7. Offset XOR-ed with delta 8. None

Table 5.3: List of program control-flow and data-flow components used to derive the list of features for exploration.

Rationale behind the winning state-vector. The winning state-vector is intuitive as its constituent features PC+Delta and Sequence of last-4 deltas closely match with the program features exploited by two prior state-of-the-art prefetchers, Bingo [120] and SPP [465], respectively. However, concurrently running SPP and Bingo as a hybrid prefetcher does not provide the same performance benefit as Pythia, as we show in §5.6.2. This is because combining SPP with Bingo not only improves their prefetch coverage, but also combines their prefetch overpredictions, leading to performance degradation, especially in resource-constrained systems. In contrast, Pythia’s RL-based learning strategy that inherently uses the same two features successfully increases prefetch coverage, while maintaining high prefetch accuracy. As a result, Pythia not only outperforms SPP and Bingo individually, but also outperforms the combination of the two prefetchers. Action Selection In a system with conventionally-sized 4KB pages and 64B cachelines, Pythia’s list of actions (i.e., the list of possible prefetch offsets) contains all prefetch offsets in the range of [−63, 63]. However, such a long action list poses two drawbacks. First, a long action list requires more online exploration to find the best prefetch offset given a state-vector, thereby reducing Pythia’s performance benefits. Second, a longer action list increases Pythia’s storage requirements. To avoid these problems, we prune the action list. We drop each action individually from the full action list [−63, 63] and measure the performance improvement relative to the performance improvement with the full action list, across all single-core workload traces. We prune any action that does not have any significant impact on the performance. Table 5.2 shows the final pruned action list. Reward and Hyperparameter Tuning We separately tune seven reward level values (i.e., R𝐴𝑇 , R𝐴𝐿 , R𝐶𝐿 , R𝐻𝐼𝑁 , R𝐿𝐼𝑁 , R𝐻𝑁 𝑃 , and R𝐿𝑁 𝑃 ) and three hyperparameters (i.e., learning rate 𝛼, discount factor 𝛾, and exploration rate 𝜖) in three steps. First, we create a test trace suite by randomly selecting 10 workload traces from all of our 150 workload traces (§5.5). Second, we create a list of tuning configurations

5.5: PYTHIA: EVALUATION METHODOLOGY

using the uniform grid search technique [144, 498]. To do so, we first define a value range for each parameter to be tuned and divide the value range into uniform grids. For example, each of the three hyperparameters (𝛼, 𝛾, and 𝜖) can take a value in the range of [0, 1]. We divide each hyperparameter range into ten exponentially-sized grids (i.e., 1𝑒 0 , 1𝑒 −1 , 1𝑒 −2 , etc.) to obtain 10 × 10 × 10 = 1000 possible tuning configurations. For each tuning configuration, we run Pythia on the test trace suite and select the top-25 highest-performing configurations for the third step. Third, we run Pythia on all single-core workload traces using each of the 25 selected configurations. We select the winning configuration that provides the highest average performance gain. Table 5.2 provides reward level and hyperparameter values of the basic Pythia.

5.4.4

Storage Overhead

Table 5.4 shows the storage overhead of Pythia in its basic configuration. Pythia requires only 25.5KB of metadata storage. QVStore consumes 24KB to store all Q-values. The EQ consumes only 1.5KB. Structure

Description

QVStore

• # vaults = 2 • # planes/vault = 3 • # entries/plane = feature dimension (128) × action dimension (16) • Entry size = Q-value width (16b)

24 KB

• # entries = 256 • Entry size = state (21b) + action index (5b) + reward (5b) + filledbit (1b) + address (16b)

1.5 KB

Total

Size

25.5 KB Table 5.4: Storage overhead of Pythia.

5.5

Methodology

We use the trace-driven ChampSim simulator [328] to evaluate Pythia and compare it to five prior prefetching proposals. We simulate an Intel Skylake [3]-like multi-core processor that supports up to 12 cores. Table 5.5 provides the key system parameters. For single-core simulations (1𝐶), we warm up the core using 100 M instructions from each workload and simulate the next 500 M instructions. For multi-core multi-programmed simulations (𝑛𝐶), we use 50 M and 150 M instructions from each workload respectively to warmup and simulate. If a core finishes early, the workload is replayed until every core finishes simulating 150 M instructions. We also implement Pythia using the Chisel [10] hardware design language (HDL) and functionally verify the resultant register transfer logic (RTL) design to accurately measure Pythia’s chip area and power overhead. The source-code of Pythia is freely available at [45].

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

Core L1/L2 Caches LLC Main Memory

1-12 cores, 4-wide OoO, 256-entry ROB, 72/56-entry LQ/SQ, Perceptron-based branch predictor [411], 20-cycle misprediction penalty Private, 32KB/256KB, 64B line, 8 way, LRU, 16/32 MSHRs, 4-cycle/14-cycle round-trip latency 2MB/core, 64B line, 16 way, SHiP [876], 64 MSHRs per LLC Bank, 34-cycle round-trip latency 1C: Single channel, 1 rank/channel; 4C: Dual channel, 2 ranks/channel; 8C: Quad channel, 2 ranks/channel; 8 banks/rank, 2400 MTPS, 64b data-bus per channel, 2KB row buffer/bank, tRCD = tRP = 15ns, tCAS = 12.5ns Table 5.5: Simulated system parameters.

5.5.1

Workloads

We evaluate Pythia using a diverse set of memory-intensive workloads spanning SPEC CPU2006 [52], SPEC CPU2017 [53], PARSEC 2.1 [149], Ligra [771], and Cloudsuite [286] benchmark suites. For SPEC CPU2006 and SPEC CPU20017 workloads, we reuse the instruction traces provided by the 2nd and the 3rd data prefetching championships (DPC) [1, 2]. For PARSEC and Ligra workloads, we collect the instruction traces using the Intel Pin dynamic binary instrumentation tool [43]. We do not consider workload traces that have lower than 3 last-level cache misses per kilo instructions (MPKI) in the baseline system with no prefetching. In all, we present results for 150 workload traces spanning 50 workloads. Table 5.6 shows a categorized view of all the workloads evaluated in this work. For multi-core multi-programmed simulations, we create both homogeneous and heterogeneous trace mixes from our single-core trace list. For an 𝑛-core homogeneous trace mix, we run 𝑛 copies of a trace from our singlecore trace list, one in each core. For a heterogeneous trace mix, we randomly select 𝑛 traces from our single-core trace list and run one trace in every core. All the single-core traces and multi-programmed trace mixes used in our evaluation are freely available online [45]. Suite

# Workloads

# Traces

Example Workloads

SPEC06 SPEC17 PARSEC Ligra Cloudsuite

16 12 5 13 4

28 18 11 40 53

gcc, mcf, cactusADM, lbm, ... gcc, mcf, pop2, fotonik3d, ... canneal, facesim, raytrace, ... BFS, PageRank, Bellman-ford, ... cassandra, cloud9, nutch, ...

Table 5.6: Workloads used for evaluation.

5.5.2

Prefetchers

We compare Pythia to five state-of-the-art prior prefetchers: SPP [465], SPP+PPF [148], SPP+DSPatch [141], Bingo [120], and MLOP [761]. We model each competing prefetcher using the source-code provided by their respective authors and fine-tune them in our environment

5.6: PYTHIA: EVALUATION

to extract the highest performance gain across all single-core traces. Table 5.7 shows the parameters of all evaluated prefetchers. Each prefetcher is trained on L1-cache misses and fills prefetched lines into L2 and LLC. We also compare Pythia against multi-level prefetchers found in commercial processors (e.g., stride prefetcher at L1-cache and streamer at L2 [13]) and IPCP [628] in §5.6.4. For fair comparison, we add a simple PC-based stride prefetcher [301, 302, 428] at the L1 level, along with Pythia at the L2 level for such multi-level comparisons. SPP [465] Bingo [120] MLOP [761]

256-entry ST, 512-entry 4-way PT, 8-entry GHR 2KB region, 64/128/4K-entry FT/AT/PHT 128-entry AMT, 500-update, 16-degree

6.2 KB 46 KB 8 KB

DSPatch [141]

Same configuration as in [141]

3.6 KB

PPF [148] Pythia

Same configuration as in [148] 2 features, 2 vaults, 3 planes, 16 actions

39.3 KB 25.5 KB

Table 5.7: Configuration of evaluated prefetchers.

5.6

Evaluation

5.6.1

Prefetch Coverage and Overprediction Analysis

Figure 5.6 shows the coverage and overprediction of each prefetcher in the single-core system, as measured at the LLC-main memory boundary. The key takeaway is that Pythia improves prefetch coverage, while simultaneously reducing overprediction compared to state-of-theart prefetchers. On average, Pythia provides 6.9%, 8.8%, and 14% higher coverage than MLOP, Bingo, and SPP respectively, while generating 83.8%, 78.2%, and 3.6% fewer overpredictions. 309%

Fraction of LLC misses

250% 200%

Covered

315%

Uncovered

Overpredicted

150% 100% 50%

SPEC06

SPEC17

PARSEC

Ligra

Cloudsuite

Pythia

MLOP

Bingo

SPP

Pythia

MLOP

Bingo

SPP

Pythia

MLOP

Bingo

SPP

Pythia

MLOP

Bingo

SPP

Pythia

Bingo

MLOP

SPP

Pythia

Bingo

MLOP

SPP

AVG

Figure 5.6: Coverage and overprediction with respect to the baseline LLC misses in the singlecore system.

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

5.6.2

Performance Analysis

Performance in Single-Core System

SPP

Bingo

MLOP

Pythia

1.4 1.3 1.2 1.1

N M EA

G EO

lo ud su ite

Li gr a

SE C PA R

SP EC 17

1.24

(b)

1.21 1.18 1.15 1.12 1.09 1.06 St St + St S +S St + +S B St + +S B +B +D +D +M Py th ia

(a)

Geomean speedup over baseline

1.5

SP EC 06

Geomean speedup over baseline

Figure 5.7(a) shows the performance improvement of each individual prefetcher in each workload category in the single-core system. We make two major observations. First, Pythia improves performance by 22.4% on average over a no-prefetching baseline. Pythia outperforms MLOP, Bingo, and SPP by 3.4%, 3.8%, and 4.3% on average, respectively. Second, only Bingo outperforms Pythia only in the PARSEC suite, by 2.3%. However, Bingo’s performance comes at the cost of a high overprediction rate, which hurts performance in multi-core systems (see §5.6.2). To demonstrate the novelty of Pythia’s RL-based prefetching approach using multiple program features, Figure 5.7(b) compares Pythia’s performance improvement with the performance improvement of various combinations of prior prefetchers. Pythia not only outperforms all prefetchers (stride, SPP, Bingo, DSPatch, and MLOP) individually, but also outperforms their combination by 1.4% on average, with less than half of the combined storage size of the five prefetchers. We conclude that Pythia’s RL-based prefetching approach using multiple program features under one single framework provides higher performance benefit than combining multiple prefetchers, each exploiting only one program feature.

Figure 5.7: Performance improvement in single-core workloads. St=Stride, S=SPP, B=Bingo, D=DSPatch, and M=MLOP.

Figure 5.8 shows the performance line graph of all prefetchers for the 150 single-core workload traces. The workload traces are sorted in ascending order of performance improvement of Pythia over the baseline without prefetching. We make three key observations. First, Pythia outperforms the no-prefetching baseline in every single-core trace, except 623.xalancbmk-592B (where it underperforms the baseline by 2.1%). 603.bwaves-2931B enjoys the highest performance improvement of 2.2× over the baseline. Performance of the top 80% of traces improve by at least 4.2% over the baseline. Second, Pythia underperforms Bingo in workloads like libquantum due to the heavy streaming nature of memory accesses. As libquantum streams through all physical pages, Bingo simply prefetches all cachelines of a

5.6: PYTHIA: EVALUATION

2.4 2.2 2 1.8 1.6 1.4 1.2 1 0.8 0.6

SPP

Bingo

MLOP

Pythia

603.bwaves_s-2931B 462.libquantum streamcluster

fluidanimate-9500M 623.xalancbmk_s-592B

pagerank-51B

BFSCC-22B 429.mcf

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115 121 127 133 139 145

Speedup over no prefetching

page at once just by seeing the first access to the page. As a result Bingo achieves higher timeliness and higher performance than Pythia. Third, Pythia significantly outperforms every competing prefetcher in workloads with irregular access patterns (e.g., mcf, pagerank). We conclude that Pythia provides consistent performance gains over the no-prefetching baseline and multiple prior state-of-the-art prefetchers over a wide range of workloads.

Workload number

Figure 5.8: Performance line graph of 150 single-core traces.

Performance in Four-Core System Figure 5.9(a) shows the performance improvement of each individual prefetcher in each workload category in the four-core system. We make two major observations. First, Pythia provides significant performance improvement over all prefetchers in every workload category in the four-core system. On average, Pythia outperforms MLOP, Bingo, and SPP by 5.8%, 8.2%, and 6.5% respectively. Second, unlike in the single-core system, Pythia outperforms Bingo in PARSEC by 3.0% in the four-core system. This is due to Pythia’s ability to dynamically increase prefetch accuracy during high DRAM bandwidth usage. Figure 5.9(b) shows that Pythia outperforms the combination of stride, SPP, Bingo, DSPatch, and MLOP prefetchers by 4.9% on average. Unlike in the single-core system, combining more prefetchers on top of stride+SPP in four-core system lowers the overall performance gain. This is due to the additive increase in the overpredictions made by each individual prefetcher, which leads to performance degradation in the bandwidth-constrained four-core system. Pythia’s RL-based framework holistically learns to prefetch using multiple program features and generates fewer overpredictions, outperforming all combinations of all individual prefetchers. Figure 5.10 shows the performance line graph of all prefetchers for 272 four-core workload trace mixes (including both homogeneous and heterogeneous mixes). The workload mixes are sorted in ascending order of performance improvement of Pythia over the baseline without prefetching. We make two key observations. First, Pythia outperforms the baseline without prefetching in all but one four-core trace mix. Pythia provides the highest performance gain in

SPP

Bingo

MLOP

Pythia

1.3 1.2 1.1

N M EA

M ix

G EO

lo ud su ite

Li gr a

SP EC 17 PA R SE C

1.35 1.3

(b)

1.25 1.2 1.15 1.1 1.05 1 St St + St S St +S+ St +S B +S +B +B +D +D +M Py th ia

1.4

(a)

SP EC 06

Geomean speedup over baseline

1.5

Geomean speedup over baseline

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

Figure 5.9: Performance in the four-core system.

3.0 2.5 2.0 1.5

437.leslie3d-271B

SPP

Bingo

MLOP

Pythia

462.libquantum-1343B raytrace-23.75B

429.mcf-184B

1.0

Mix-59

0.5 0.0

pagerank

Mix-240

1 10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181 190 199 208 217 226 235 244 253 262

Speedup over no prefetching

437.leslie3d-271B (2.1×) and lowest performance gain in 429.mcf-184B (-3.5%) over the noprefetching baseline. Second, Pythia also outperforms (or matches performance) all competing prefetchers in majority of trace mixes. Pythia underperforms Bingo in the 462.libquantum homogeneous trace mix due to the very regular streaming access pattern. On the other hand, Pythia significantly outperforms Bingo in Ligra workloads (e.g., pagerank) due to its adaptive prefetching strategy to trade-off coverage for accuracy in high memory bandwidth usage. We conclude that Pythia provides a consistent performance gain over multiple prior state-of-theart prefetchers over a wide range of workloads even in bandwidth-constrained multi-core systems.

Workload number

Figure 5.10: Performance line graph of 272 four-core trace mixes.

Performance on Unseen Traces To demonstrate Pythia’s ability to provide performance gains across workload traces that are not used at all to tune Pythia, we evaluate Pythia using an additional 500 traces from the second value prediction championship [50] on both single-core and four-core systems. These traces are classified into floating-point, integer, crypto, and server categories and each of them has at

5.6: PYTHIA: EVALUATION

SPP

1.6

Bingo

MLOP

Pythia

Geomean speedup over baseline

least 3 LLC MPKI in the baseline without prefetching. No prefetcher, including Pythia, has been tuned on these traces. In the single-core system, Pythia outperforms MLOP, Bingo, and SPP on average by 8.3%, 3.5%, and 4.9%, respectively, across these traces. In the four-core system, Pythia outperforms MLOP, Bingo, and SPP on average by 9.7%, 5.4%, and 6.7%, respectively. We conclude that, Pythia, tuned on a set of workload traces, provides equally high (or even better) performance benefits on unseen traces for which it has not been tuned.

1.5 1.4

(a) single-core

1.3 1.2 1.1 1 Crypto

INT

Server

SPP

1.5 1.4

Bingo

MLOP

Pythia

(b) four-core

1.3 1.2 1.1 1

GEOMEAN

Crypto

INT

Server

GEOMEAN

Figure 5.11: Performance on unseen traces.

Benefit of Awareness to Memory Bandwidth Usage To demonstrate the benefit of Pythia’s awareness of system memory bandwidth usage, we compare the performance of the full-blown Pythia with a new version of Pythia that is oblivious to system memory bandwidth usage. We create this bandwidth-oblivious version of Pythia by setting the high and low bandwidth usage variants of the rewards R𝐼 𝑁 and R𝑁 𝑃 to the same value (i.e., essentially removing the bandwidth usage distinction from the reward values). More specifically, we set R𝐻𝐼𝑁 = R𝐿𝐼𝑁 = −8 and R𝐻𝑁 𝑃 = R𝐿𝑁 𝑃 = −4. Figure 5.12 shows the performance benefit of the memory bandwidth-oblivious Pythia normalized to the basic Pythia as we vary the DRAM bandwidth. The key takeaway is that the bandwidth-oblivious Pythia loses performance by up to 4.6% on average across all single-core traces when the available memory bandwidth is low (150-MTPS to 600-MTPS configuration). However, when the available memory bandwidth is high (1200-MTPS to 9600-MTPS), the memory bandwidth-oblivious Pythia provides similar performance improvement to the basic Pythia. We conclude that, memory bandwidth awareness gives Pythia the ability to provide robust performance benefits across a wide range of system configurations.

5.6.3

Performance Sensitivity Analysis

Varying Number of Cores Figure 5.13(a) shows the performance improvement of all prefetchers averaged across all traces in single-core to 12-core systems. To realistically model modern commercial multi-core processors, we simulate 1-2 core, 4-6 core, and 8-12 core systems with one, two, and four DDR4-2400 DRAM [33] channels, respectively. We make two key observations from Figure 5.13(a). First, Pythia consistently outperforms MLOP, Bingo, and SPP in all system configurations. Second,

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

Performance normalized to basic Pythia

0% -0.2%

-0.2%

-1.2%

-2% -3% -4%

-0.3%

-0.4%

-1%

Memory BW-oblivious Pythia

-2.5% -4.6%

-5% 150

300

600 1200 2400 DRAM MTPS (in log scale)

4800

9600

Figure 5.12: Performance of memory bandwidth-oblivious Pythia versus the basic Pythia.

Pythia’s performance improvement over prior prefetchers increases as core count increases. In the single-core system, Pythia outperforms MLOP, Bingo, SPP, and an aggressive SPP with perceptron filtering (PPF [148]) by 3.4%, 3.8%, 4.3%, and 1.02% respectively. In four (and twelve) core systems, Pythia outperforms MLOP, Bingo, SPP, and SPP+PPF by 5.8% (7.7%), 8.2% (9.6%), 6.5% (6.9%), and 3.1% (5.2%), respectively. MLOP

SPP+PPF

Pythia

MTPS DRAM MTPSDRAM (in log scale)

1.2 1.18 (c)

1.16 0.5 1 2 LLC Size (in MB, log scale)

12 80 0

0.25

64 00

Geomean speedup over no prefetching

1.22

32 00

(b)

16 00 32 00 6146 000 12 80 0

4 6 8 10 Number of cores

(a)

Similar to AMD Threadripper 3990X

4 channels

800 0

0.8

2 channels

Similar to AMD EPYC Rome 7702P

40 0

0.9

0.85

Similar to Intel Xeon 6258R

20 0

0.95

1 channel

1.1

Geomean speedup over no prefetching

SPP+DSPatch

1.24

1.25 1.2 1.15 1.1 1.05 1 0.95 0.9 0.85 0.8

40 0

1.05

20 0

1.15

1.15 1.1

10 0

1.2

Geomean speedup over no prefetching

1.2

1.3 1.25

Bingo

1.25

10 0

SPP 1.35

Figure 5.13: Geomean performance improvement of prefetchers in systems with varying (a) number of cores, (b) DRAM million transfers per second (MTPS), and (c) LLC size. Each DRAM bandwidth configuration roughly matches MTPS/core of various commercial processors [4, 5, 29]. The baseline bandwidth/LLC configuration is marked in red.

Varying DRAM Bandwidth To evaluate Pythia in band-width-constrained, highly-multi-threaded commercial server-class processors, where each core can have only a fraction of a channel’s bandwidth, we simulate the single-core single-channel configuration by scaling the DRAM bandwidth (Figure 5.13(b)). Each bandwidth configuration roughly corresponds to the available per-core DRAM bandwidth in various commercial processors (e.g., Intel Xeon Gold [29], AMD EPYC Rome [5], and AMD Threadripper [4]). The key takeaway is that Pythia consistently outperforms all com1 peting prefetchers in every DRAM bandwidth configuration from 16 × to 4× bandwidth of the baseline system. Due to their large overprediction rates, the performance gains of MLOP and Bingo reduce sharply as DRAM bandwidth decreases. By actively trading off prefetch cov-

5.6: PYTHIA: EVALUATION

erage for higher accuracy based on memory bandwidth usage, Pythia outperforms MLOP, Bingo, SPP, and SPP+PPF by 16.9%, 20.2%, 3.7%, and 9.5% respectively in the most bandwidthconstrained configuration with 150 million transfers per second (MTPS). In the 9600-MTPS configuration, every prefetcher enjoys ample DRAM bandwidth. Pythia still outperforms MLOP, Bingo, SPP, and SPP+PPF by 3%, 2.7%, 4.4%, and 0.8%, respectively. Varying LLC Size Figure 5.13(c) shows performance of all prefetchers averaged across all traces in the single-core system while varying the LLC size from 18 × to 2× of the baseline 2MB LLC. The key takeaway is that Pythia consistently outperforms all prefetchers in every LLC size configuration. For 256KB (and 4MB) LLC, Pythia outperforms MLOP, Bingo, SPP, and SPP+PPF by 3.6% (3.1%), 5.1% (3.4%), 2.7% (4.8%), and 1.2% (0.8%), respectively. Varying Hyperparameters Figure 5.14(a) shows Pythia’s performance sensitivity to the exploration rate (𝜖) averaged across all single-core traces. The key takeaway from Figure 5.14(a) is that Pythia’s performance improvement drops sharply if the underlying RL-agent heavily explores the state-action space as opposed to exploiting the learned policy. Changing the 𝜖-value from 0.002 to 1.0 reduces Pythia’s performance improvement by 16.0%. Figure 5.14(b) shows Pythia’s performance sensitivity to learning rate parameter (𝛼), averaged across all single-core traces. The key takeaway from Figure 5.14(b) is that Pythia’s performance improvement reduces for both increasing or decreasing the learning rate parameter. Increasing the learning rate reduces the hysteresis in Q-values (i.e., Q-values change significantly with the immediate reward received by Pythia), which reduces Pythia’s performance improvement. Similarly, decreasing the learning rate also reduces Pythia’s performance as it increases the hysteresis in Q-values. Pythia achieves optimal performance improvement for 𝛼 = 0.0065. 1.23 Speedup over no prefetching

Speedup over no prefetching

1.24 1.2 1.16 1.12

(a) Epsilon (𝜀) 1.08 1.04 1.E-6

1.E-5

1.E-4

1.E-3 values

1.E-2

1.E-1

1.E+0

1.22 1.21 1.2 1.19 1.18

(b) Alpha (𝛼)

1.17 1.16 1.E-6

1.E-5

1.E-4

1.E-3 values

1.E-2

1.E-1

1.E+0

Figure 5.14: Performance sensitivity of Pythia towards (a) the exploration rate (𝜖), and (b) the learning rate (𝛼) hyperparameter values. The values in basic Pythia configuration are marked in red.

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

Varying Program Features

72% 71%

1.225

70%

1.22

69% 1.215 68% 1.21

67% 66%

1.2

65%

Speedup

Overprediction

1.225

33% 32% 31%

1.22

30%

1.215

29% 28%

1.21

27% 1.205

26%

1.2

25% 1 13 25 37 49 61 73 85 97 109 121 133 145 157 169 181 193 205

1.205

1.23

Experiment number

Overprediction of Pythia (lower is better)

Coverage

Speedup over baseline (higher is better)

Speedup

Coverage of Pythia (higher is better)

1.23

1 13 25 37 49 61 73 85 97 109 121 133 145 157 169 181 193 205

Speedup over baseline (higher is better)

Figure 5.15 shows the performance, coverage, and overprediction of Pythia averaged across all single-core traces with different feature combinations during automated feature selection (§5.4.3). For brevity, we show results for all experiments with any-one and any-two combinations of 20 features taken from the full list of 32 features. Both graphs are sorted in ascending order of performance improvement of Pythia over the baseline without prefetching. We make three key observations. First, Pythia’s performance gain over the no-prefetching baseline improves from 20.7% to 22.4% by varying the feature combination. We select the feature combination that provides the highest performance gain as the basic Pythia configuration (Table 5.2). Second, Pythia’s coverage and overprediction also change significantly with varying feature combination. Pythia’s coverage improves from 66.2% to 71.5%, whereas overprediction improves from 32.2% to 26.7% by changing feature combination. Third, Pythia’s performance gain positively correlates with Pythia’s coverage in single-core configuration. We conclude that automatic design-space exploration can significantly optimize Pythia’s performance, coverage, and overpredictions.

Figure 5.15: Performance, coverage, and overprediction of Pythia with different feature combinations. The x-axis shows experiments with different feature combinations.

Varying Number of Warmup Instructions Figure 5.16 shows performance sensitivity of all prefetchers to the number of warmup instructions averaged across all single-core traces. Our baseline simulation configuration uses 100 million warmup instructions. The key takeaway from Figure 5.16 is that Pythia consistently outperforms prior prefetchers in a wide range of simulation configurations using different number of warmup instructions. In the baseline simulation configuration using 100M warmup instructions, Pythia outperforms MLOP, Bingo, and SPP by 3.4%, 3.8%, and 4.4% respectively. In a simulation configuration with no warmup instruction, Pythia continues to outperform MLOP, Bingo, and SPP by 2.8%, 3.7%, and 4.2% respectively. We conclude that, Pythia can quickly learn to prefetch from a program’s memory access pattern and provides higher performance than other heuristics-based prefetching techniques over a wide range of simulation configurations using different number of warmup instructions.

5.6: PYTHIA: EVALUATION

Geomean speedup over baseline

SPP

Bingo

MLOP

Pythia

1.23 1.21 1.19 1.17 1.15 0

10 15 Warmup instructions (in millions)

100

Figure 5.16: Performance sensitivity of all prefetchers to number of warmup instructions.

5.6.4

Performance Comparison Against Hybrid Prefetching Schemes

Comparison to Multi-Level Prefetching

lo ud su ite G EO M EA N

Li gr a

SE C PA R

SP EC 17

Stride-L1+Streamer-L2 IPCP

0.9

Stride-L1+Pythia-L2

0.85 12 80 0

1.1

1 0.95

64 00

1.2

1.1 1.05

32 00

1.3

1.15

16 00

Stride-L1+Pythia-L2

(b)

80 0

IPCP

1.4

1.2

40 0

1.5

1.25

20 0

Stride-L1+Streamer-L2

10 0

(a)

Geomean speedup over no prefetching

1.6

SP EC 06

Geomean speedup over no prefetching

Figure 5.17(a) shows the performance comparison of Pythia in single-core system with the baseline 2400-MTPS main memory bandwidth against two state-of-the-art multi-level prefetching schemes: (1) stride prefetcher [301, 302, 428] at L1 and streamer [201] at L2 cache found in commercial Intel processors [13], and (2) IPCP, the winner of the third data prefetching championship [2]. For fair comparison, we add a stride prefetcher in the L1 cache along with Pythia in the L2 cache for this experiment and measure performance over the no prefetching baseline. The key takeaway is that Stride+Pythia outperforms Stride+Streamer and IPCP by 2.4% and 1.3%, respectively. To demonstrate Pythia’s adaptiveness to memory bandwidth usage, we also show the average performance of Stride+Streamer, IPCP, and Stride+Pythia with varying main memory bandwidth in Figure 5.17(b). The key takeaway is that Stride+Pythia consistently outperforms Stride+Streamer and IPCP in every DRAM bandwidth configuration. Stride+Pythia outperforms Stride+Streamer and IPCP by 6.5% and 14.2% in the 150-MTPS configuration and by 2.3% and 1.0% in the 9600-MTPS configuration, respectively.

DRAM MTPS (in log scale)

Figure 5.17: Performance of multi-level prefetching schemes (a) in the baseline 2400-MTPS configuration, and (b) with varying main memory bandwidth.

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

Comparison to the Context Prefetcher

1.2 1.1

M ix G EO M EA N

1 te

G EO M EA N

C lo ud su i

Li gr a

PA R SE C

SP EC 17

Pythia

C lo ud su i

1.1

CP-HW

1.3

Li gr a

1.2

1.4

PA R SE C

Pythia

SP EC 17

CP-HW 1.3

(b) four-core

1.5

SP EC 06

1.4

Geomean speedup over baseline

(a) single-core

1.5

SP EC 06

Geomean speedup over baseline

Context Prefetcher (CP [660]) is the first work that explored the application of RL for prefetching. However, unlike Pythia, CP relies on both hardware and software contexts. A tailor-made compiler needs to encode the software contexts using special NOP instructions, which are decoded by the core front-end to pass the context to the CP. For a fair comparison, we implement the context prefetcher using only hardware contexts (CP-HW) and show the performance comparison of Pythia and CP-HW in Figure 5.18. The key takeaway is that Pythia outperforms the CP-HW prefetcher by 5.3% and 7.6% in single-core and four-core configurations, respectively. Pythia’s performance improvement over CP-HW mainly comes from two key aspects: (1) Pythia’s ability to take memory bandwidth usage into consideration while taking prefetch actions, and (2) the far-sighted predictions made by Pythia as opposed to myopic predictions by CP-HW.

Figure 5.18: Performance of Pythia vs. the context prefetcher [660] using hardware contexts.

Comparison to the IBM POWER7 Adaptive Prefetcher Figure 5.19 compares Pythia against the IBM POWER7 adaptive prefetcher [416]. The POWER7 prefetcher dynamically tunes its prefetch aggressiveness (e.g., selecting prefetch depth, enabling stride-based prefetching) by monitoring program performance. We make two observations from Figure 5.19. First, Pythia outperforms the POWER7 prefetcher by 4.5% in single-core system. This is mostly due to Pythia’s ability to capture different types of address patterns than just streaming/stride patterns. Second, Pythia outperforms POWER7 prefetcher by 6.5% in four-core and 6.1% in eight-core systems (not plotted), respectively. The increase in performance improvement from single to four (or eight) core configuration suggests that Pythia is more adaptive than the POWER7 prefetcher.

5.6.5

Understanding Pythia using a Case Study

We delve deeper into an example workload trace, 459.GemsFDTD-1320B, from SPEC CPU2006 suite to provide more insight into Pythia’s prefetching strategy and benefits. In this trace, the top two most selected prefetch offsets by Pythia are +23 and +11, which cumulatively account for nearly 72% of all offset selections. For each of these offsets, we examine the program feature value that selects that offset the most. For simplicity, we only focus on the PC+Delta feature

te G EO M EA N

C lo ud su i

Li gr a

PA R SE C

SP EC 17

1.2 1.1 1 M ix G EO M EA N

1.1

Pythia

1.2

POWER7 1.3

C lo ud su i

1.3

1.4

Li gr a

Pythia

PA R SE C

POWER7

SP EC 17

1.4

(b) four-core

1.5

SP EC 06

Geomean speedup over baseline

(a) single-core

1.5

SP EC 06

Geomean speedup over baseline

5.6: PYTHIA: EVALUATION

Figure 5.19: Performance comparison against IBM POWER7 prefetcher [416].

here. The PC+Delta feature values 0x436a81+0 and 0x4377c5+0 select the offsets +23 and +11 the most, respectively. Figure 5.20(a) and (b) show the Q-value curve of different actions for these feature. The x-axis shows the number of Q-value updates to the corresponding feature. Each color-coded line represents the Q-value of the respective action. As Figure 5.20(a) shows, the Q-value of action +23 for feature value 0x436a81+0 consistently stays higher than all other actions (only three other representative actions are shown in Figure 5.20(a)). This means Pythia actively favors to prefetch using +23 offset whenever the PC 0x436a81 generates the first load to a physical page (hence the delta 0). By dumping the program trace, we indeed find that whenever PC 0x436a81 generates the first load to a physical page, there is only one more address demanded in that page that is 23 cachelines ahead from the first loaded cacheline. In this case, the positive reward for generating a correct prefetch with offset +23 drives the Q-value of +23 much higher than those of other offsets and Pythia successfully uses the offset +23 for prefetch request generation given the feature value 0x436a81+0. We see similar a trend for the feature value 0x4377c5+0 with offset +11 (Figure 5.20(b)). 78&9 +1

78)9 +3

78""9 +22

78")9 +23

78(9 +1

!&$ 16

$(# 12

!&# 14

$(% 10

78(%9 +10

78((9 +11

$' 8

Q-value .!/0123

!&" 12

Q-value -./0123

78*9 +3

!&' 10 !% 8 !$ 6

$& 6 $" 4 $# 2 $% 0

!# 4

!# -2

!" 2

!" -4

'! 1k &( 2k "( 3k )( 4k #( 5k *( 6k $( 7k +( 8k %( 9k ,( &'( 10k 4563

# of Q-value updates

%$ 1k () 2k #) 3k *) 4k ") 5k +) 6k &) ,) ') -) 10k 7k 8k 9k (%) 4563

# of Q-value updates

Figure 5.20: Q-value curves of PC+Delta feature values (a) 0x436a81+0 and (b) 0x4377c5+0 in 459.GemsFDTD-1320B.

5.6.6

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

Performance Benefits via Customization

In this section, we show two examples of Pythia’s online customization ability to extract even higher performance gain than the baseline Pythia configuration in target workload suites. First, we customize Pythia’s reward level values for the Ligra graph processing workloads. Second, we customize the program features used by Pythia for the SPEC CPU2006 workloads. Customizing Reward Levels For workloads from the Ligra graph processing suite, we observe a general trend that a prefetcher with higher prefetch accuracy typically provides higher performance benefits. This is because any incorrect prefetch request wastes precious main memory bandwidth, which is already heavily used by the demand requests of the workload. Thus, to improve Pythia’s performance benefit in the Ligra suite, we create a new strict configuration of Pythia that favors not to prefetch over generating inaccurate prefetches. We create this strict configuration by simply reducing the reward level values for inaccurate prefetch (i.e., R𝐻𝐼𝑁 = −22 and R𝐿𝐼𝑁 = −20) and increasing the reward level values for no prefetch (i.e., R𝐻𝑁 𝑃 = R𝐿𝑁 𝑃 = 0). Figure 5.21 shows the percentage of the total runtime the workload spends in different bandwidth usage buckets in primary y-axis and the overall performance improvement in the secondary y-axis for each competing prefetcher in one example workload from the Ligra suite, Ligra-CC. We make two key observations. First, with MLOP and Bingo prefetchers enabled, Ligra-CC spends a much higher percentage of runtime consuming more than half of the peak DRAM bandwidth than in the no prefetching baseline. As a result, MLOP and Bingo underperforms the no prefetching baseline by 11.8% and 1.8%, respectively. In contrast, basic Pythia leads to only a modest memory bandwidth usage overhead, and outperforms the no prefetching baseline by 6.9%. Second, in the strict configuration, Pythia has even less memory bandwidth usage overhead, and provides 3.5% higher performance than the basic Pythia configuration (10.4% over the no prefetching baseline), without any hardware changes. Figure 5.22 shows the performance benefits of the basic and strict Pythia configurations for all workloads from Ligra. The key takeaway is that by simply changing the reward level values via configuration registers on the silicon, strict Pythia provides up to 7.8% (2.0% on average) higher performance than basic Pythia. We conclude that the objectives of Pythia can be easily customized via simple configuration registers for target workload suites to extract even higher performance benefits, without any changes to the underlying hardware. Customizing Feature Selection To maximize the performance benefits of Pythia on the SPEC CPU2006 workload suite, we run all one-combination and two-combination of program features from the initial set of 32 supported features. For each workload, we fine-tune Pythia using the feature combination that provides the highest performance benefit. We call this the feature-optimized configuration of Pythia for SPEC CPU2006 suite. Figure 5.23 shows the performance benefits of the basic and optimized configurations of Pythia for all SPEC CPU2006 workloads. The key takeaway is

5.6: PYTHIA: EVALUATION

25%-50% of peak

50%-75% of peak

>=75% of peak

Performance

100%

15%

80%

60%

40%

-3%

20%

-9%

IPC improvement over baseline

% of total runtime

<25% of peak

-15% Baseline

SPP

Bingo

MLOP

Basic Pythia

Strict Pythia

2.0 1.8

Basic Pythia

1.6

Strict Pythia

1.4 1.2

N M EA

G EO

BF S

M BF IS SBi tv ec to r BF SC C

ad ii R

an Fo rd Tr ia ng le

C C

lm Be l

Pa C ge F R an kD elt a

1.0 Pa ge R an k

IPC normalized to baseline

Figure 5.21: Performance and main memory bandwidth usage of prefetchers in Ligra-CC.

Figure 5.22: Performance of the basic and strict Pythia configurations in the Ligra workloads.

2.2 2.0

Basic Pythia

Feature-optimized Pythia

1.8 1.6 1.4 1.2

43 3. m

ilc 48 2. sp hi nx 3 48 46 1. w 2. rf lib qu an tu m 43 7. le sli e3 d 41 0. bw av G es EO M EA N

43 6. c

44 5. g

40 3. g

k tu sA D 47 M 1. om ne tp p 47 3. 48 as ta 3. r xa la nc bm k 42 9. m cf 47 0. lb m 45 0. so 45 pl 9. ex G em sF D TD

1.0 ob m

IPC normalized to baseline

that by simply fine-tuning the program feature selection, Pythia delivers up to 5.1% (1.5% on average) performance improvement on top of the basic Pythia configuration.

Figure 5.23: Performance of the basic and feature-optimized Pythia on the SPEC CPU2006 suite.

5.6.7

Performance Evaluation using DPC4 Traces

Figure 5.24 shows the geometric mean performance improvement of SPP, Bingo, MLOP, and Pythia over the no-prefetcher baseline across all 483 single-core DPC4 workloads, categorized into AIML, GMS, and Google datacenter workload categories (see §4.3). We make three key

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

observations. First, Pythia achieves the highest overall geometric mean performance improvement across all 483 DPC4 traces, outperforming all prior prefetching mechanisms without requiring any workload-specific tuning. Pythia improves performance by 14.8% over the noprefetching baseline, whereas SPP, Bingo, and MLOP improve performance by 9.1%, 11.9%, and 14.2%, respectively. Second, Pythia delivers significant performance improvements in Google datacenter workload category. Across these 359 workload traces, Pythia outperforms SPP, Bingo, and MLOP on average by 7.2%, 6.3%, and 3.9%. This result strongly indicates Pythia’s ability to adapt to complex and often irregular memory access behavior found in real production-scale workloads running on datacenters. Third, Pythia underperforms MLOP and Bingo on AIML workloads. This is because these workloads exhibit pronounced streaming memory access pattern, where aggressive multi-degree spatial prefetching employed by Bingo and MLOP achieve higher gains. As Pythia’s learning-driven policy does not explicitly bias toward such highly regular streaming patterns, Pythia significantly underperforms Bingo and MLOP by 11.2% and 14.2%, respectively. This observation highlights a potential direction for future work: incorporating adaptive multi-degree prefetch control within Pythia’s RL framework to better capture highly regular streaming access patterns while also preserving its general adaptability.

Geomean speedup over baseline

1.4 SPP

Bingo

MLOP

Pythia

1.3 1.2 1.1 1 AIML

GMS

Google

GEOMEAN

Figure 5.24: Performance improvement in 483 single-core DPC4 traces.

Figure 5.25 shows the category-wise geometric mean performance improvement of SPP, Bingo, MLOP, and Pythia across 966 four-core DPC4 workload mixes (see §4.3). We make three key observations. First, similar to the single-core evaluation results, Pythia achieves the highest overall geometric mean performance across all four-core mixes without requiring any hyperparameter tuning or architectural modification for these trace mixes. Pythia improves performance by 13.9% over the no-prefetching baseline, whereas SPP, Bingo, and MLOP improves performance by 7.2%, 12.1%, and 10.8%, respectively. Second, Pythia significantly outperforms SPP, Bingo, and MLOP by 7.2%, 5.9%, and 7.3% on average in Google datacenter workload mixes. Third, similar to the single-core evaluation, Pythia significantly underperforms Bingo and MLOP (by 20.7% and 16.2%, respectively) on AIML workloads mixes. Overall, these results provide strong empirical evidence that Pythia generalizes beyond its design-time workloads. Despite never observing these 483 single-core and 966 four-core workload traces during development and despite operating in a plug-and-play configuration without additional tuning, Pythia sustains state-of-the-art performance across both singlecore and multi-core configurations.

5.7: PYTHIA: SUMMARY

Geomean speedup over baseline

1.5

SPP

1.4

Bingo

MLOP

Pythia

1.3 1.2 1.1 1 AIML-mix

GMS-mix

Google-mix

Random-mix

GEOMEAN

Figure 5.25: Performance improvement in 966 four-core DPC4 traces.

5.6.8

Overhead Analysis

To accurately estimate Pythia’s chip area and power overheads, we implement the full-blown Pythia, including all fixed-point adders, multipliers, and the pipelined QVStore search operation (§5.4.2), using the Chisel [10] hardware design language (HDL). We extensively verify the functional correctness of the resultant register transfer logic (RTL) design and synthesize the RTL design using Synopsys Design Compiler [57] and 14-nm library from GlobalFoundries [18] to estimate Pythia’s area and power overhead. Pythia consumes 0.33 mm2 of area and 55.11 mW of power in each core. The QVStore component consumes 90.4% and 95.6% of the total area and power of Pythia, respectively. With respect to the overall die area and power consumption of a 4-core desktop-class Skylake processor with the lowest TDP budget [27], and a 28-core server-class Skylake processor with the highest TDP budget, Pythia (implemented in all cores) incurs area & power overheads of only 1.03% & 0.4%, and 1.33% & 0.75%, respectively. We conclude that Pythia’s performance benefits come at a very modest cost in area and power overheads across a variety of commercial processors. Pythia’s area: 0.33 mm2/core; Pythia’s power: 55.11 mW/core Overhead compared to real systems

Area

Power

4-core Skylake D-2123IT, 60W TDP [27] 18-core Skylake 6150, 165W TDP [28] 28-core Skylake 8180M, 205W TDP [30]

1.03% 1.24% 1.33%

0.37% 0.60% 0.75%

Table 5.8: Area and power overhead of Pythia.

5.7

Summary

We introduce Pythia, the first customizable prefetching framework that formulates prefetching as a reinforcement learning (RL) problem. Pythia autonomously learns to prefetch using multiple program features and system-level feedback information to predict memory accesses. Our extensive evaluations show that Pythia not only outperforms five state-of-the-art prefetchers but also provides robust performance benefits across a wide-range of workloads and system configurations. Pythia’s benefits come with very modest area and power overheads.

5.7.1

CHAPTER 5: HARDWARE PREFETCHING USING REINFORCEMENT LEARNING

Influence on the Research Community

Pythia has been presented at the 54th IEEE/ACM International Symposium on Microarchitecture (MICRO) on October, 2021 [136] and has been officially artifact evaluated with all three badges (i.e., available, functional, and reproducible). We have made Pythia freelydownloadable from our GitHub repository [45] with all evaluated workload traces, scripts, and implementation code required to reproduce and extend it. Since its release, Pythia has influenced numerous subsequent works: both as a state-of-theart baseline [151, 260, 274, 319, 404, 513], as well as a definitive reference for modeling architectural decision making using machine learning [91, 205, 330, 375, 526, 530, 594, 642, 776, 890, 892, 902, 910, 916, 929]. Most notably, a subsequent work, Micro-Armed Bandit (MAB) [319], extends Pythia by distilling its learning tables and demonstrates performance benefits similar to Pythia while reducing Pythia’s overhead by two orders of magnitude. Another follow-up work, Micro-MAMA [151], improves upon Pythia and MAB further to demonstrate higher performance gains in multi-core processor configurations. Pythia served as one of the baseline prefetchers in the 4th Data Prefetching Championship (DPC4) [135] and was independently verified by competing teams to provide the state-of-the-art performance gains among known prefetching techniques at that time. We believe and hope that Pythia will continue to encourage the the design of the next generation data-driven autonomous prefetchers that automatically learn far-sighted prefetching policies by interacting with the system. Such prefetchers can not only improve performance under a wide variety of workloads and system configurations, but also reduce the system architect’s burden in designing sophisticated prefetching mechanisms.

Chapter 6

Accelerating Long-Latency Loads via Perceptron-Based Off-Chip Load Prediction Load requests that miss the on-chip cache hierarchy and go to off-chip main memory often block instruction retirement from the reorder buffer (ROB) of modern out-of-order (OOO) processors, preventing the processor from allocating new instructions into the ROB [357, 585, 592, 593], limiting performance. To tolerate long memory latency, architects have primarily relied on two key latency-hiding techniques. First, they have significantly scaled up the size of on-chip caches in modern high-performance processor (e.g., each Intel Alder Lake core [44] employs 4.3MB on-chip cache (including L1, L2 and a per-core last-level cache (LLC) slice), which is 1.88× larger than the on-chip cache in the previous-generation Skylake core [3]). Second, architects have designed and deployed increasingly sophisticated hardware prefetchers (likes of Pythia, discussed in chapter 5; or complex prefetchers in real processors [19, 26]) that can more effectively predict the addresses of load requests in advance and fetch their corresponding data to on-chip caches before the program demands it, thereby completely or partially hiding the long off-chip load latency for a fraction of off-chip loads [19,136,141,148].

6.1

Motivation and Goal

Despite these advances, we observe two key trends in processor design that leave a significant performance improvement opportunity on the table: (1) a large fraction of load requests continues to go off-chip even in the presence of state-of-the-art prefetchers, and (2) an increasing fraction of the latency of an off-chip load request is spent accessing the increasingly larger on-chip caches. Large Fraction of Loads are Still Uncovered by State-of-the-Art Prefetchers Over the past decades, researchers have proposed many hardware prefetching techniques that have consistently pushed the limits of performance improvement (e.g., [117, 136, 141, 148, 201, 290, 302, 384, 428, 465, 475, 479, 518, 556, 628, 671, 761, 765, 792, 799]). We observe that stateof-the-art prefetchers provide a large performance gain by accurately predicting future load addresses. Yet, a large fraction of off-chip load requests cannot be predicted even by the most

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

Fraction of off-chip loads in the No-prefetching system

100%

25 Blocking

Non-blocking

MPKI 20

75%

15 50% 10 25%

SPEC06

SPEC17

PARSEC

Ligra

CVP

Pythia

No-prefetching

Pythia

No-prefetching

Pythia

No-prefetching

Pythia

No-prefetching

Pythia

No-prefetching

Pythia

0 No-prefetching

LLC misses per kilo instructions (MPKI)

advanced prefetchers. These uncovered requests limit the processor’s performance by blocking instruction retirement in the ROB. Figure 6.1 shows a stacked graph of total number off-chip load requests in a no-prefetching system and a system with the recently-proposed hardware data prefetcher Pythia [136], normalized to the no-prefetching system, across 110 workload traces categorized into five workload categories.1 Each bar further categorizes load requests into two classes: loads that block instruction retirement from the ROB (called blocking) and loads that do not (called non-blocking). §6.5 discusses our evaluation methodology.

AVG

Figure 6.1: The distribution of ROB-blocking and non-blocking load requests (on the left yaxis), and LLC misses per kilo instructions (on the right y-axis) in the absence and presence of a state-of-the-art hardware data prefetcher [136].

We make two key observations from Figure 6.1. First, on average, Pythia accurately prefetches nearly half of all off-chip load requests in the no-prefetching system, thereby improving the overall performance (not shown here; see §6.6.2). Second, the remaining half of the off-chip loads are not prefetched even by a sophisticated prefetcher like Pythia. 71.4% of these non-prefetched off-chip loads block instruction retirement from the ROB, significantly limiting performance. We conclude that, state-of-the-art prefetchers, while effective at improving performance, still leave a significant performance improvement opportunity on the table. An Increasing Fraction of Off-Chip Load Latency is Spent in Accessing the On-Chip Cache Hierarchy We observe that the on-chip cache hierarchy has not only grown tremendously in size but also in design complexity (e.g., sliced last-level cache organization [129, 356, 458]) in recent processors, in order to cater to workloads with large data footprints. A larger on-chip cache hierarchy, on the one hand, improves a core’s performance by preventing more load requests from going off-chip. On the other hand, all on-chip caches need to be accessed to determine if a load request should be sent off-chip. As a result, on-chip cache access latency significantly 1 We select Pythia as the baseline prefetcher as it provides the highest prefetch coverage and performance

benefit among the five contemporary prefetchers considered in this work (see §6.5.2 and §6.6.4). Nonetheless, our qualitative observation holds equally true for other prefetchers considered in this work (see §6.5.2).

6.1: HERMES: MOTIVATION AND GOAL

contributes to the total latency of an off-chip load. With increasing on-chip cache sizes, and the complexity of the cache hierarchy design and the on-chip network [145, 866], the on-chip cache access latency is increasing in processors [25, 35]. An analysis of the Intel Alder Lake core suggests that the load-to-use latency of an LLC access has increased to 14 ns (which is equivalent to 55 cycles for a core running at 4 GHz) [25, 56, 60]. To demonstrate the effect of long on-chip cache access latency on the total latency of an off-chip load, Figure 6.2 plots the average number of cycles a core stalls due to an off-chip load blocking any instruction from retiring from the ROB, averaged across each workload category in our baseline system with Pythia. Each bar further shows the average number of cycles an off-chip load spends for accessing the on-chip cache hierarchy. Our simulation configuration faithfully models an Intel Alder Lake performance-core with a large ROB, large on-chip caches and publicly-reported cache access latencies (see §6.5). As Figure 6.2 shows, an off-chip load stalls the core for an average of 147.1 cycles. 40.1% of these stall cycles (i.e., 58.9 cycles) can be completely eliminated by removing the on-chip cache access latency from the off-chip load’s critical path. We conclude that a large and complex on-chip cache hierarchy is directly responsible for a large fraction of the overall stall cycles caused by an off-chip load request. We envision that this problem will only get exacerbated with new processor designs as on-chip caches continue to grow in size and complexity [35]. # stall cycles due to an off-chip load blocking instruction retirement from ROB

180

147.1

160 140 120 100 80 60 40 20 0 SPEC06

SPEC17

PARSEC

Ligra

CVP

AVG

Figure 6.2: The average number of cycles a core stalls due to an off-chip load blocking any instruction from retiring from the ROB across all workload categories. The dark portion in each bar shows the cycles that can be completely eliminated by removing the on-chip cache access latency from an off-chip load’s critical path.

6.1.1

Our Goal

Our goal is to improve processor performance by removing the on-chip cache access latency from the critical path of off-chip load requests.

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

6.2

Hermes: Headroom and Challenges

To this end, we propose a new technique called Hermes, whose key idea is to predict which load requests might go off-chip and start fetching their corresponding data directly from the main memory, while also concurrently accessing the cache hierarchy for such a load.2 By doing so, Hermes hides the on-chip cache access latency under the shadow of the main memory access latency (as illustrated in Fig. 6.3), thereby significantly reducing the overall latency of an off-chip load request.

Hermes

Baseline

Latency tolerance limit of ROB L1

LLC

LLC Main Memory

Processor is stalled Main Memory

Saved stall cycles

2 Start fetching data from main memory a6ter physical address is available 1 Predict whether the load will go o=f-chip

Figure 6.3: Comparison of the execution timeline of an off-chip load request in a conventional processor and in Hermes.

6.2.1

Headroom Analysis

To understand the potential performance benefits of Hermes, we model an Ideal Hermes system in simulation where we reduce the main memory access latency of every off-chip load request by the post-L1 on-chip cache hierarchy access latency (which includes L2 and LLC access, and interconnect latency). In other words, in the Ideal Hermes system, we (1) magically and perfectly know if a load request would go off-chip after its physical address is available (i.e., after the translation lookaside buffer access, which happens in parallel with the L1 data cache access in modern processors [126, 178, 652, 875]), and (2) directly access the off-chip main memory for such a load, eliminating the non-L1-cache related on-chip cache hierarchy access latency from such a load’s total latency. Figure 6.4(a) shows the speedup of Ideal Hermes by itself and when combined with Pythia normalized to the no-prefetching system in single-core workloads. We make two key observations from Figure 6.4(a). First, Ideal Hermes combined with Pythia outperforms Pythia alone by 8.3% on average across all workloads. Second, Ideal Hermes by itself provides nearly 80% of the performance improvement that Pythia provides. Figure 6.4(b) shows the speedup of Ideal Hermes when combined with four other recently-proposed high-performance prefetchers: Bingo [120], SPP [465] (with perceptron filter [148]), MLOP [761], and SMS [792]. Ideal Hermes improves performance by 9.4%, 8.2%, 2 Hence named after Hermes, the Olympian deity [64] who can quickly move between the realms of the divine

(i.e., the processor) and the mortals (i.e., the main memory).

6.2: HERMES: HEADROOM AND CHALLENGES

10.9%, and 13.3% on top of four state-of-the-art prefetchers Bingo, SPP, MLOP, and SMS, respectively. Based on these results, we conclude that Hermes has high potential performance benefit not only when implemented alone but also when combined with a wide variety of high-performance prefetchers.

Geomean speedup over the No-prefetching system

1.35 1.3

(a)

Ideal Hermes

1.29

Pythia (baseline)

8.3%

Pythia + Ideal Hermes

1.25

1.20

1.2

1.16

1.15 1.1 1.05 1

Geomean speedup over the No-prefetching system

SPEC06 1.35 1.3 1.25 1.2

(b)

SPEC17 1.29

1.29

8.3% 1.20

PARSEC

Ligra

Prefetcher-only

9.4%

CVP

Prefetcher + Ideal Hermes 1.24

1.23

1.19

10.9%

8.2% 1.14

1.15

GEOMEAN

1.13

1.1

1.19

13.3% 1.06

1.05 1 Pythia

Bingo

SPP

MLOP

SMS

Figure 6.4: (a) Speedup of Ideal Hermes by itself and when combined with Pythia in singlecore workloads. (b) Speedup of Ideal Hermes when combined with four recently-proposed prefetchers: Bingo [120], SPP [148, 465], MLOP [761], and SMS [792].

6.2.2

Key Challenges

Even though Hermes has a significant potential to improve performance, Hermes’s performance gain heavily depends on the accuracy (i.e., the fraction of predicted off-chip loads that actually go off-chip) and the coverage (i.e., the fraction of off-chip loads that are successfully predicted) of the off-chip load prediction. A low-accuracy off-chip load predictor generates useless main memory requests, which incur both latency and bandwidth overheads, and causes interference to the useful requests in the main memory. A low-coverage predictor loses opportunity to improve performance. We identify two key challenges in designing an off-chip load predictor with high accuracy and high coverage. First, only a small fraction of the total loads generated by a workload goes off-chip in presence of a sophisticated data prefetcher. As shown in 6.5, on average 7.9 loads

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

per kilo instructions miss the LLC and go off-chip in our baseline system with Pythia. However, these loads constitute only 5.1% of the total loads generated by a workload. This small fraction of off-chip loads makes it difficult for an off-chip load predictor to accurately learn from the workload behavior to produce highly-accurate predictions. Off-chip rate

LLC MPKI

6% 4%

2% 0%

0 SPEC06

SPEC17

PARSEC

Ligra

CVP

LLC misses per kilo instructions

Fraction of loads that goes off-chip

10%

AVG

Figure 6.5: Percentage of loads that miss the LLC and goes off-chip (on the left y-axis) and the LLC MPKI (on the right y-axis) in the baseline system with Pythia.

Second, the off-chip predictability of a workload can change in the presence of modern sophisticated data prefetchers. This is because in the presence of a sophisticated prefetcher, the likelihood of a load request going off-chip not only depends on the program behavior but also on the prefetcher’s ability to successfully prefetch for the load. In this work, we overcome these two key challenges by designing a new off-chip load prediction technique, called POPET, based on perceptron learning [412, 548, 700]. By learning to identify off-chip loads using multiple program features (e.g., sequence of program counters, byte offset of a load request, page number of the load address), POPET provides both higher accuracy and coverage than a prior cache hit-miss prediction technique [905] and higher accuracy than another off-chip load prediction technique that we develop (see §6.5.2), in the presence of modern sophisticated prefetchers, without requiring large metadata storage overhead. With small changes to the existing on-chip datapath design, we demonstrate that Hermes with POPET significantly outperforms the baseline system with a state-of-the-art prefetcher across a wide range of workloads and system configurations.

6.3

Hermes: Overview

Figure 6.6 shows a high-level overview of Hermes. POPET is the key component of Hermes that is responsible for making highly-accurate off-chip load predictions. For every demand load request generated by the processor, POPET predicts whether or not the load request would go off-chip ( 1 ). If the load is predicted to go off-chip, Hermes issues a speculative memory request (called a Hermes request) directly to the main memory controller once the load’s physical address is generated to start fetching the corresponding data from the main memory ( 2 ). This Hermes request is serviced by the main memory controller concurrently with the regular load request (i.e., the load issued by the processor that generated the Hermes request) that accesses the on-chip cache hierarchy. If the prediction is correct, the regular load

6.4: HERMES: DETAILED DESIGN

request to the same address eventually misses the LLC and waits for the ongoing Hermes request to finish, thereby completely hiding the on-chip cache hierarchy access latency from the critical path of the correctly-predicted off-chip load ( 3 ). If a Hermes request returns from the main memory but there has been no regular load request to the same address, Hermes drops the request and does not fill the data into the cache hierarchy. By doing so, Hermes keeps the on-chip cache hierarchy fully coherent even in case of a misprediction. For every regular load request returning to the core, Hermes trains POPET based on whether or not this load has actually gone off-chip ( 4 ). 1

Core

POPET Predict whether the load will go o/f-chip

Train POPET 4

L1-D Existing datapath

New datapath

Issue Hermes request for predicted o/f-chip load

LLC Regular load request missing the LLC waits for the Hermes request to ﬁnish

Main Memory

Figure 6.6: Overview of Hermes.

6.4

Hermes: Detailed Design

We first describe the design of POPET in §6.4.1, followed by the changes introduced by Hermes to the on-chip cache access datapath in §6.4.2.

6.4.1

POPET Design

The purpose of POPET is to accurately predict whether or not a load request generated by the processor will go off-chip. We model POPET using the multi-feature perceptron learning mechanism [148, 315, 405, 411, 412, 414, 700, 825], more specifically as a hashed-perceptron [823] model. A hashed-perceptron model hashes multiple feature values to retrieve weights of each feature from small tables. If the sum of these weights exceeds a threshold, the model

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

makes a positive prediction. Hashed-perceptron, as compared to other perceptron models, is lightweight and easy to implement in hardware. Prior works successfully apply hashedperceptron for various microarchitectural predictions, e.g., branch outcome [315,405,412], LLC reuse [414,825], prefetch usefulness [148]. This is the first work that applies hashed-perceptron to off-chip load prediction. Why is Perceptron Learning a Good Fit for Modeling Off-Chip Prediction? We choose to model POPET based on perceptron learning for two key reasons. First, by learning using multiple program features, perceptron learning can provide highly accurate predictions that could not be otherwise provided by simple history-based learning prediction (e.g., HMP [905]). Second, perceptron learning can be implemented with low storage overhead, without requiring any impractical metadata support (e.g., extending TLB [728, 729] or in-memory metadata storage [397]). POPET Design Overview POPET is organized as a collection of one-dimensional tables (each called a weight table), where each table corresponds to a single program feature. Each table entry stores a weight value, implemented using a 5-bit saturating signed integer, that represents the correlation between the corresponding program feature value and the true outcome (i.e., whether a given load actually went off-chip). A weight value saturated near the maximum (i.e., +15) or the minimum (i.e., −16) value represents a strong positive or negative correlation between the program feature value and the true outcome, respectively. A weight value closer to zero signifies a weak correlation. The weights are adjusted during training (step 4 in Figure 6.6) to update POPET’s prediction with the true outcome. Each weight table is sized differently based on its corresponding program feature (see Table 6.3). Making a Prediction using POPET During load queue (LQ) allocation for a load generated by the core (step 1 in Figure 6.6), POPET makes a binary prediction on whether or not the load request would go off-chip. The prediction happens in three stages as shown in Figure 6.7. In the first stage, POPET extracts a set of program features from the current load request and a history of prior requests (§6.4.1 shows the list of program features used by POPET). In the second stage, each feature value is hashed and used as an index to retrieve a weight value from the weight table of the corresponding feature. In the third stage, all weight values from individual features are accumulated to generate the cumulative perceptron weight (𝑊𝜎 ). If 𝑊𝜎 exceeds a predefined threshold (called the activation threshold, 𝜏𝑎𝑐𝑡 ), POPET makes a positive prediction (i.e., it predicts that the current load request would go off-chip). Otherwise, POPET makes a negative prediction. The hashed feature values, the cumulative perceptron weight 𝑊𝜎 , and the predicted outcome are stored in the LQ entry to be reused to train POPET when the load request returns to the processor core (step 4 in Figure 6.6).

6.4: HERMES: DETAILED DESIGN

Stage 1

Stage 2

Feature1

(e.g., PC + o8fset)

hash

Feature2

.. .. . FeatureN

index

Stage 3

Weight Table1 weight1

index

hash

Weight Table2

weight2

.. .

𝚺

≥ τact

Sum weights

Activation

Predict to go o8f-chip

weightn

index

Weight TableN

hash

Figure 6.7: Stages to make a prediction by POPET.

Training POPET POPET training is invoked when a demand load request returns to the core and prepares to release its corresponding LQ entry (step 4 in Figure 6.6). Every demand load that misses the LLC and goes to the main memory controller is marked as a true off-chip load request. This true off-chip outcome, along with the predicted outcome stored in the LQ entry of the demand load, are used to appropriately train the feature weights of POPET. The training happens in two stages. In the first stage, the 𝑊𝜎 (computed during prediction) is retrieved from the LQ entry. If 𝑊𝜎 is neither positively nor negatively saturated (i.e., 𝑊𝜎 lies within a negative and a positive training threshold, 𝑇𝑁 and 𝑇𝑃 , respectively), the weight training is triggered. This saturation check prevents the individual feature weight values from getting over-saturated, thereby helping POPET to quickly adapt its learning to program phase changes. In the second stage, if the weight training is triggered, the weights for each individual program feature are retrieved from their corresponding weight table using the hashed feature indices stored in the LQ entry. If the true outcome is positive (meaning the load actually went off-chip), the weight value for each feature is incremented by one. If the true outcome is negative, the weight values are decremented by one. This simple weight update mechanism moves each individual feature weight towards the direction of the true outcome, thus gradually increasing the prediction accuracy. Automated Feature Selection The selection of the program features used to make the off-chip load prediction is critical to POPET’s performance. A carefully-crafted and selected set of features can significantly improve the accuracy and the coverage of POPET. In this section we propose an automated, offline, performance-driven methodology to find a set of program features for POPET.

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

We initially select a set of 16 individual program features using our domain expertise that can correlate well with a load going off-chip. Table 6.1 shows the initial feature set. Features without control-flow information 1. Load virtual address 2. Virtual page number 3. Cacheline offset in page 4. First access 5. Cacheline offset + first access 6. Byte offset in cacheline 7. Word offset in cacheline

Features with control-flow information 8. Load PC 9. PC ⊕ load virtual address 10. PC ⊕ virtual page number 11. PC ⊕ cacheline offset 12. PC + first access 13. PC ⊕ byte offset 14. PC ⊕ word offset 15. Last-4 load PCs 16. Last-4 PCs

Table 6.1: The initial set of program features used for automated feature selection. ⊕ represents a bitwise XOR operation.

The automated feature selection process happens offline during the design time of POPET. The process starts with the initial set of 16 individual program features and iteratively creates a list of feature sets, each containing 𝑛 features, at every iteration 𝑛 in the following way. In the first iteration, we design POPET with each of the 16 initial program features and test its prediction accuracy in 10 randomly-selected workload traces (called testing workloads). We select the top-10 features that produce the highest prediction accuracy for the second iteration. In the second iteration, we create 160 two-combination feature sets (meaning, each feature set contains two initial features from Table 6.1) by combining each of the 16 initial features with each of the 10 winning feature sets from the last iteration, and test the prediction accuracy on the testing workloads. We select the top-10 two-combination feature sets that produce the highest prediction accuracy for the third iteration. This iterative process repeats until the maximum prediction accuracy gets saturated (i.e., the difference in accuracy of two successive iterations is less than 3%).3 Table 6.2 shows the final list of program features selected by the automated feature selection process.

Selected features

Threshold values

• PC ⊕ cacheline offset • PC ⊕ byte offset • PC + first access • Cacheline offset + first access • Last-4 load PCs 𝜏𝑎𝑐𝑡 = −18, 𝑇𝑁 = −35, 𝑇𝑃 = 40

Table 6.2: POPET configuration parameters.

Rationale for Selected Features. Each selected feature correlates with the likelihood of ob3 For simplicity, our automated feature selection process optimizes for accuracy. A more comprehensive fea-

ture selection process can also include coverage or directly optimize for performance (i.e., execution time).

6.4: HERMES: DETAILED DESIGN

serving an off-chip load request with a different program context information. We explain the rationale for each selected feature below. (1) PC ⊕ cacheline offset. This feature is computed by XOR-ing the load PC value with the cacheline offset of the load address in the virtual page of the load request. The goal of this feature is to learn the likelihood of a load request going off-chip when a given load PC touches a certain cacheline offset in a virtual page. The use of cacheline offset information, instead of load virtual address or virtual page number, enables this feature to apply the learning across different virtual pages. (2) PC ⊕ cacheline byte offset. This feature is computed by XOR-ing the load PC with the byte offset of the load cacheline address. This feature is particularly useful in accurately predicting off-chip load requests when a program has a streaming access pattern over a linearly allocated data structure. For example, when a program streams through a large array of 4B integers, every 16𝑡ℎ load (as a 64B cacheline stores 16 integers) generated by a load PC that is iterating over the array will go off-chip, and the remaining loads will hit in on-chip caches. In this case, this feature learns to identify only those loads that have a byte offset of 0 to go off-chip. (3) PC + first access. This feature is computed by left-shifting the load PC and adding the first access hint at the most-significant bit position. The first access hint is a binary value that represents whether or not a cacheline has been recently touched by the program. The hint is computed using a small 64-entry buffer (called the page buffer) that tracks the demanded cachelines from last 64 virtual pages. Each page buffer entry holds two pieces of information: a virtual page tag, and a 64-bit bitmap, where each bit represents one cacheline in the virtual page. During every load request generation, POPET searches the page buffer with the virtual page number of the load address. If a matching entry is found, POPET uses the value of the bit corresponding to the cacheline offset in the matching page buffer entry’s bitmap as the first access hint. If the bit is set (or unset), it signifies that the corresponding cacheline has (not) been recently accessed by the program. If the bit is unset, POPET sets the bit in the page buffer entry’s bitmap. The first access hint provides a crude estimate of a cacheline’s reuse in a short temporal window. However, it alone cannot determine the cacheline’s residency in on-chip caches, as the memory footprint tracked by the page buffer is much smaller than the total cache size. (4) Cacheline offset + first access. This feature is similar to the PC + first access feature, except that it learns the likelihood of a load request going off-chip when a given cacheline offset is recently touched by the program. (5) Last-4 load PCs. This feature value is computed as a shifted-XOR of last four load PCs. It represents the execution path of a program and correlates it with the likelihood of observing an off-chip load request whenever the program follows the same execution path. Parameter Threshold Tuning POPET has three tunable parameters: negative and positive training thresholds (𝑇𝑁 and 𝑇𝑃 , respectively), and the activation threshold (𝜏𝑎𝑐𝑡 ). Properly tuning the values of all these three

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

parameters is also critical to POPET’s performance, since both POPET’s accuracy and coverage are sensitive to parameter values. We employ a three-step grid search technique to tune each of the three parameters separately. In the first stage, we uniformly sample values from a parameter’s range. For example, 𝜏𝑎𝑐𝑡 can take values in the range [−80, 75].4 We uniformly sample values from this range with a grid size of 5. In the second stage, we run Hermes with the randomly-selected 10 test workloads (as mentioned in §6.4.1) for each of the sampled values and pick the top-10 values that provide the highest performance gain. In the third stage, we run Hermes with all single-core workload traces using the selected 10 parameter values from the second stage. We finally select the value that provides the highest average performance gain. Table 6.2 shows the selected threshold values of each parameter.

6.4.2

Hermes Datapath Design

In this section, we describe the key changes introduced to the existing well-optimized on-chip cache access datapath to incorporate Hermes. First, we show how the core issues a Hermes request directly to the main memory controller if POPET predicts the load would go off-chip and how a regular load request that misses the LLC waits for an ongoing Hermes request (see §6.4.2). Second, we discuss how the data fetched from main memory is properly sent back to the core in presence of Hermes while maintaining cache coherence (see §6.4.2). Issuing a Hermes Request For every load request predicted to go off-chip, Hermes issues a Hermes request directly to the main memory controller (step 2 in Figure 6.6) once the load’s physical address is generated. The main memory controller enqueues the Hermes request in its read queue (RQ) and starts fetching the corresponding data from the main memory as dictated by its scheduling policy, while the regular load request is concurrently accessing the on-chip cache hierarchy. If the offchip prediction is correct, the regular load request eventually misses the LLC and checks the main memory controller’s RQ for any ongoing main memory access to the same load address (step 3 ). If the address is found, the regular load request waits for the ongoing Hermes request to finish before sending the Hermes-fetched data back to the core. Hermes’s performance gain depends on the latency to directly issue a Hermes request to the main memory controller (called Hermes request issue latency). Although a Hermes request experiences a significantly shorter latency to arrive at the main memory controller than its corresponding regular load request because a Hermes request bypasses the cache hierarchy and on-chip queueing delays, a Hermes request nonetheless pays for a latency to route through the on-chip network. We model two variants of Hermes using an optimistic and a pessimistic estimate of Hermes request issue latency to take into account a wide range of potential differences in on-chip interconnect designs (see §6.5.2). In §6.6.4, we also evaluate Hermes with 4 As POPET uses five program features (see §6.4.1), the sum of all five weights (each represented by a 5-

bit saturating signed integer as described in §6.4.1) can take a maximum and minimum value of 75 and −80, respectively.

6.5: HERMES: EVALUATION METHODOLOGY

a wide range of Hermes request issue latencies (from 0 cycle to 24 cycles) and show that Hermes consistently provides performance benefit even with the most pessimistic Hermes request issue latency. Returning Data to the Core For every Hermes request returning from main memory, Hermes checks the RQ of the main memory controller and returns the fetched data back to the LLC if there is a regular load request already waiting for the same load address. If there is no regular load request waiting for the completed Hermes request, Hermes drops the request and does not fill the data into the cache hierarchy, which keeps the on-chip cache hierarchy internally coherent.

6.4.3

Storage Overhead

Table 6.3 shows the total storage overhead of Hermes. Hermes requires only 4 KB of metadata storage per processor core. POPET consumes 3.2 KB, whereas the metadata stored in LQ for POPET training consumes 0.8 KB. Structure

Description

Size

• Perceptron weight tables

POPET

– PC ⊕ cacheline offset: 1024 × 5𝑏 – PC ⊕ byte offset: 1024 × 5𝑏 – PC + first access: 1024 × 5𝑏 – Cacheline offset + first access: 128 × 5𝑏 – Last-4 load PCs: 1024 × 5𝑏

3.2 KB

• Page buffer: 64 × 80𝑏 LQ Metadata

Hashed PC: 128×32𝑏; Last-4 PC: 128×10𝑏; First access: 128×1𝑏; perceptron weight: 128 × 5𝑏; prediction: 128 × 1𝑏

Total

0.8 KB 4.0 KB

Table 6.3: Storage overhead of Hermes.

6.5

Methodology

We use the ChampSim trace-driven simulator [328] to evaluate Hermes. We faithfully model the latest-generation Intel Alder Lake performance-core [19] with its large ROB, large caches with publicly-reported on-chip cache access latencies [35, 56, 60], and the state-of-the-art prefetcher Pythia [136] at the LLC. Table 6.4 shows the key microarchitectural parameters. For single-core simulations, we warm up the core using 100M instructions and simulate the next 500M instructions. For multi-programmed simulations, we use 50M and 100M instructions from each workload for warmup and simulation, respectively. If a core finishes early, the workload is replayed until every core has finished executing at least 100M instructions. The

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

source code of Hermes, along with all workload traces and scripts to reproduce our results are freely available at [22]. Core L1/L2 Caches LLC Main Memory Hermes

1 and 8 cores, 6-wide fetch/execute/commit, 512-entry ROB, 128/72-entry LQ/SQ, Perceptron branch predictor [411] with 17-cycle misprediction penalty Private, 48KB/1.25MB, 64B line, 12/20-way, 16/48 MSHRs, LRU, 5/15-cycle round-trip latency [60] 3MB/core, 64B line, 12 way, 64 MSHRs/slice, SHiP [876], 55-cycle round-trip latency [56, 60], Pythia prefetcher [136] 1C: 1 channel, 1 rank per channel; 8C: 4 channels, 2 ranks per channel; 8 banks per rank, DDR4-3200 MTPS, 64b data-bus per channel, 2KB row buffer per bank, tRCD=12.5ns, tRP=12.5ns, tCAS=12.5ns Hermes-O/P: 6/18-cycle Hermes request issue latency Table 6.4: Simulated system parameters.

6.5.1

Workloads

We evaluate Hermes using a wide range of memory-intensive workloads spanning SPEC CPU2006 [52], SPEC CPU2017 [53], PARSEC [149], Ligra graph processing workload suite [771], and commercial workloads from the 2nd data value prediction championship (CVP [50]). For SPEC CPU2006 and SPEC CPU2017 workloads, we reuse the instruction traces provided by the 2nd and the 3rd data prefetching championships (DPC [1, 2]). For PARSEC and Ligra workloads, we reuse the instruction traces open-sourced by Pythia [136]. The CVP workload traces are collected by the Qualcomm Datacenter Technologies and capture complex program behavior from various integer, floating-point, cryptographic, and server applications in the field. We only consider workload traces in our evaluation that have at least 3 LLC misses per kilo instructions (MPKI) in the no-prefetching system. In total, we evaluate Hermes using 110 single-core workload traces from 73 workloads, which are summarized in Table 6.5. For multiprogrammed simulations, we create both homogeneous and heterogeneous trace mixes. For an eight-core homogeneous multi-programmed simulation, we run eight copies of each trace from our single-core trace list, one trace in each core. For heterogeneous multi-programmed simulation, we randomly select any eight traces from our single-core trace list and run one trace in each core. In total, we evaluate Hermes using 110 homogeneous and 110 heterogeneous eight-core workloads.

6.5.2

Evaluated System Configurations

For a comprehensive analysis, we compare Hermes with various off-chip load prediction mechanisms, as well as in combination with various recently proposed prefetchers. Table 6.6 compares the storage overhead of all evaluated mechanisms. (1) Various off-chip prediction mechanisms. We compare POPET against two cache hit/miss prediction techniques: (1) HMP, proposed by Yoaz et al. [905], and (2) a simple cacheline tag-tracking based predictor, called TTP, which we design. HMP uses three predictors sim-

6.5: HERMES: EVALUATION METHODOLOGY

Suite

# Workloads

# Traces

Example Workloads

SPEC06 SPEC17 PARSEC Ligra CVP

14 11 4 11 33

22 23 12 20 33

gcc, mcf, cactusADM, lbm, ... gcc, mcf, pop2, fotonik3d, ... canneal, facesim, raytrace, ... BFS, PageRank, Radii, ... integer, floating-point, server, ...

Table 6.5: Workloads used for evaluation.

ilar to a hybrid branch predictor: local [900], gshare [549,900], and gskew [557], each of which individually predicts off-chip loads using a different prediction mechanism. For a given load, HMP consults each individual predictor and selects the majority prediction. We design TTP by taking inspiration from prior cacheline address tracking-based mechanisms [397, 524, 728]. TTP tracks partial tags of cacheline addresses that are likely to be present in the entire on-chip cache hierarchy in a separate metadata structure. For every cache fill (LLC eviction), the partial tag of the filled (evicted) cacheline address is inserted into (evicted from) TTP’s metadata. To predict whether or not a given load would go off-chip, TTP searches the metadata structure with the partial tag of the load address. If the tag is not present in the metadata structure, TTP predicts the load would go off-chip. We open-source TTP in our repository [22]. (2) Various data prefetchers. We evaluate Hermes combined with five recently-proposed high-performance prefetching techniques: Pythia [136], Bingo [120], SPP [465] (with perceptron filter [148]), MLOP [761], and SMS [792]. As mentioned in Table 6.4, Pythia is incorporated in our baseline system. HMP [905] with local, gshare, and gskew predictors TTP with a metadata budget similar to the L2 cache

11 KB 1536 KB

Pythia [136] with the same configuration in [136] Bingo [120] with the same configuration in [120] SPP [465] with perceptron-based prefetch filter [148] MLOP [761] with the same configuration in [761] SMS [792] with the same configuration in [792]

25.5 KB 46 KB 39.3 KB 8 KB 20 KB

Hermes with POPET (this work)

4 KB

Table 6.6: Storage overhead of all evaluated mechanisms.

We evaluate two variants of Hermes: Hermes-O and Hermes-P. These two variants differ only in Hermes request issue latency. Hermes-O (i.e., the optimistic Hermes) and Hermes-P (i.e., the pessimistic Hermes) use a request issue latency of 6 cycles and 18 cycles, respectively. Unless stated otherwise, Hermes represents the optimistic variant Hermes-O.

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

6.6

Evaluation

6.6.1

POPET Prediction Analysis

Accuracy and Coverage of POPET Figure 6.8 shows the comparison of POPET’s off-chip load prediction accuracy and coverage against those of HMP and TTP in the baseline system. The key takeaway is that POPET has significantly higher accuracy and coverage than HMP. POPET provides 77.1% accuracy with 74.3% coverage on average across all single-core workloads, whereas HMP provides 47% accuracy with 22.3% coverage. TTP, with a metadata budget of 1.5 MB, provides the highest coverage (94.8%) but with a significantly lower accuracy (16.6%). POPET’s superior accuracy and coverage directly translates to performance benefits both in single-core and eight-core system configuration (see §6.6.2 and §6.6.3). HMP 100%

TTP

HMP

POPET 100%

(a)

POPET

(b)

80% Coverage %

Accuracy %

80%

TTP

60% 40% 20%

0% SPEC06

SPEC17 PARSEC

Ligra

CVP

AVG

SPEC06

SPEC17 PARSEC

Ligra

CVP

AVG

Figure 6.8: Comparison of (a) accuracy and (b) coverage of POPET against those of HMP [905] and TTP.

Effect of Different POPET Features Figure 6.9 shows the accuracy and coverage of POPET using the five selected program features used individually and in various combinations. We make two key observations. First, each program feature individually produces predictions with a wide range of accuracy and coverage. The PC ⊕ cacheline offset feature produces the lowest-quality predictions with only 53.4% accuracy and 14.5% coverage, whereas the cacheline offset + first access feature produces the highest-quality predictions with 70.6% accuracy and 48.1% coverage. Second, by stacking multiple features together, the final POPET design achieves both higher accuracy and coverage than those provided by any single individual program feature. We conclude that POPET is capable of learning from multiple program features to achieve both higher off-chip load prediction accuracy and coverage than any individual program feature can provide. Usefulness of all features To understand the usefulness of multi-feature learning, we analyze per-trace accuracy and coverage of POPET using each individual program feature. Figure 6.10(a) shows the line graph of POPET’s prediction accuracy with each of the five program features individually for all

6.6: HERMES: EVALUATION

Accuracy and coverage %

80%

Accuracy

Coverage

60%

40%

20%

0% Last-4 load PC ⊕ byte PC + first Cacheline PC ⊕ PCs (2) cacheline offset (3) access (4) offset + first access (5) offset (1)

1+2

1+2+3

1+2+3+4

All (POPET)

Figure 6.9: The accuracy and coverage of POPET using each program feature individually and in various combinations.

single-core workload traces. The traces are sorted in ascending order of POPET accuracy using the feature cacheline offset + first access, since this feature individually has the highest average accuracy (as shown in Figure 6.9(a)). The key takeaway from Figure 6.10(a) is that there is no single program feature that individually provides the highest prediction accuracy across all workloads. Out of 110 workload traces, the features PC + first access, cacheline offset + first access, PC ⊕ byte offset, PC ⊕ cacheline offset, and last-4 load PCs provide the highest prediction accuracy in 47, 29, 20, 9, and 5 workload traces, respectively. We observe similar variability in POPET’s coverage, as shown in Figure 6.10(b), where no single program feature individually provides the highest coverage across all workloads. This large variability of accuracy/coverage with different features in different workloads warrants learning using all features in unison to provide higher accuracy and coverage than any individual program feature across a wide range of workloads.

6.6.2

Single-Core Performance Analysis

Performance Improvement Figure 6.11 shows performance of Hermes (O and P), Pythia, and Hermes combined with Pythia normalized to the no-prefetching system in single-core workloads. We make three key observations. First, Hermes provides nearly half of the performance benefit of Pythia with only 15 × the storage overhead. On average, Hermes-O improves performance by 11.5% over a no-prefetching system, whereas Pythia improves performance by 20.3%. Second, Hermes-O (Hermes-P) combined with Pythia outperforms Pythia by 5.4% (4.3%). Third, Hermes combined with Pythia consistently outperforms Pythia in every workload category. To better understand Hermes’s performance improvement, Figure 6.12 shows the performance line graph of Hermes, Pythia, and Hermes combined with Pythia for every single-core workload trace. The traces are sorted in ascending order of performance gains by Hermes combined with Pythia over the no-prefetching system. We make four key observations from Figure 6.12. First, Hermes combined with Pythia outperforms the no-prefetching system in all but three single-core workload traces. The compute_int_539 and 605.mcf_s-782B traces experi-

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

PC ⊕ ^ cacheline offset 100%

Last-4 load PCs

PC ⊕ ^ byte offset

PC + first access

Cacheline offset + first access

(a)

Accuracy %

80% 60% 40% 20%

PC + first access provides highest accuracy

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109

PC⊕ ^ byte offset provides highest accuracy

Workload number

PC ⊕ ^ cacheline offset 100%

Last-4 load PCs

PC ⊕ ^ byte offset

PC + first access

Cacheline offset + first access

(b)

Coverage %

80% 60% 40% 20%

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109

0% Workload number

1.35 Hermes-P

Hermes-O

Pythia (baseline)

SPEC17

PARSEC

Pythia + Hermes-P

Pythia + Hermes-O 1.25 1.26

1.3

1.20

1.25 1.2 1.09 1.12

Geomean speedup over the No-prefetching system

Figure 6.10: Line graph of POPET’s (a) accuracy and (b) coverage using each of the five program features individually across all 110 single-core workloads. No single feature can provide the best accuracy or coverage across all workloads.

1.15 1.1 1.05 1 SPEC06

Ligra

CVP

GEOMEAN

Figure 6.11: Speedup in single-core workloads.

ence the highest and the lowest speedup (2.3× and 0.8×, respectively). Second, unlike Pythia, Hermes always improves performance over the no-prefetching system in every workload trace. Third, Hermes outperforms Pythia by 7.9% on average in 51 traces (e.g., streamcluster-6B, Ligra_PageRank-79B). In the remaining 59 traces, Pythia outperforms Hermes by 26% on average. Fourth, Hermes combined with Pythia consistently outperforms both Hermes and Pythia alone in almost every workload trace. Based on our performance results, we conclude that, Hermes provides significant and con-

6.6: HERMES: EVALUATION

2.50

Hermes-O

2.25

Pythia (baseline)

compute_int_539

Pythia + Hermes-O

602.gcc_s-2226B

2.00

605.mcf_s-782B

1.75

623.xalancbmk_s-10B

1.50

streamcluster-6B

1.25

Ligra_Triangle-25B server_612 compute_int_264

1.00 Ligra_PageRank-79B Ligra_Components-22B

0.75 0.50

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109

Speedup over the No-prefetching system

sistent performance improvements over a wide range of workloads both by itself and when combined with the state-of-the-art prefetcher Pythia.

Workload number

Figure 6.12: Single-core performance of all 110 workloads.

Effect of the Off-chip Load Prediction Mechanism Figure 6.13 shows the performance of Hermes with POPET, Hermes-HMP, Hermes-TTP, and the Ideal Hermes (see §6.2.1) combined with Pythia normalized to the no-prefetching system in single-core workloads. We make two key observations. First, Hermes with POPET outperforms both Hermes-HMP and Hermes-TTP. On average, Hermes-HMP, Hermes-TTP, and Hermes with POPET combined with Pythia provide 0.8%, 1.7%, and 5.4% performance improvement over Pythia, respectively. Second, Hermes-POPET provides nearly 90% of the performance improvement provided by the Ideal Hermes that employs an ideal off-chip load predictor with 100% accuracy and coverage. We conclude that Hermes provides performance gains due to both the high off-chip load prediction accuracy and coverage of POPET. Thus, designing a good off-chip predictor is critical for Hermes to improve performance. Pythia + Hermes-HMP

Pythia + Hermes-TTP

Pythia + Hermes-POPET

Pythia + Ideal Hermes

1.4 1.203 1.211 1.220 1.257 1.286

Geomean speedup over the No-prefetching system

Pythia (baseline) 1.35 1.3 1.25 1.2 1.15 1.1 1.05 1

SPEC06

SPEC17

PARSEC

Ligra

CVP

GEOMEAN

Figure 6.13: Speedup of Hermes with three off-chip load predictors (HMP, TTP, and POPET) and the Ideal Hermes.

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

Effect on Stall Cycles Figure 6.14(a) plots the distribution of the percentage reduction in stall cycles due to off-chip load requests in a system with Hermes over the baseline system in single-core workloads as a box-and-whiskers plot.5 The key observation is that Hermes reduces the stall cycles caused by off-chip loads by 16.2% on average (up to 51.8%) across all workloads. PARSEC workloads experience the highest average stall cycle reduction of 23.8%. 90 out of 110 workloads experience at least 10% stall cycle reduction. We conclude that Hermes considerably reduces the stall cycles due to off-chip load requests, which leads to performance improvement.

20% 10% 0%

Hermes-O

CVP

-10%

70% 60%

Pythia (baseline)

Pythia + Hermes-O

(b) 38.5% 44.4%

30%

Ligra

50% 40% 30% 20%

5.5%

40%

(a)

PARSEC

over the No-prefetching system

50%

SPEC17

% increase in main memory requests

% reduction of stall cycles due to off-chip loads

SPEC06 60%

10% 0% SPEC06

SPEC17

PARSEC

Ligra

CVP

AVG

Figure 6.14: (a) Reduction in stall cycles caused by off-chip loads. (b) Overhead in the main memory requests.

Overhead in Main Memory Requests Figure 6.14(b) shows the percentage increase in main memory requests in Hermes, Pythia, and Hermes combined with Pythia over the no-prefetching system in all single-core workloads. We make two key observations. First, Hermes increases main memory requests by only 5.5% (on average) over the no-prefetching system, whereas Pythia by 38.5%. This means that, every 1% performance gain (see Figure 6.11) comes at a cost of only 0.5% increase in main memory requests in Hermes, whereas nearly 2% increase in main memory requests in Pythia. We attribute this result to the highly-accurate predictions made by POPET, as compared to less-accurate prefetch decisions made by Pythia. Second, Hermes combined with Pythia further increases main memory requests by only 5.9% over Pythia. This means that, every 1% performance benefit by Hermes on top of Pythia comes at a cost of only 1% overhead in main memory requests. We conclude that, Hermes, due to its underlying high-accuracy prediction mechanism, adds considerably lower overhead in main memory requests while providing significant performance improvement both by itself and when combined with Pythia. 5 Each box is lower-bounded by the first quartile (i.e., the middle value between the lowest value and the

median value of the data points) and upper-bounded by the third quartile (i.e., the middle value between the median and the highest value of the data points). The inter-quartile range (𝐼𝑄𝑅) is the distance between the first and the third quartile (i.e., the length of the box). Whiskers extend an additional 1.5 × 𝐼𝑄𝑅 on the either side of the box. Any outlier values that falls outside the range of whiskers are marked by dots. The cross marked value within each box represents the mean.

6.6: HERMES: EVALUATION

6.6.3

Eight-Core Performance Analysis

Pythia + Hermes-HMP

Pythia+Hermes-TTP

SPEC17

Ligra

Pythia + Hermes-POPET 1.174

Pythia (baseline) 1.25 1.2

1.123 1.129 1.102

Geomean speedup over the No-prefetching system

Figure 6.15 shows the performance of Pythia, Hermes-HMP, Hermes-TTP, and Hermes-POPET combined with Pythia normalized to the no-prefetching system in all eight-core workloads. The key takeaway is that due to the highly-accurate predictions by POPET, Hermes-POPET combined with Pythia consistently outperforms Pythia in every workload category. On average, Hermes-HMP, Hermes-TTP, and Hermes-POPET combined with Pythia provide 0.6%, −2.1%, and 5.1% higher performance on top of Pythia, respectively. Due to its inaccurate predictions, TTP generates many unnecessary main memory requests, which reduce the performance of Hermes-TTP combined with Pythia as compared to Pythia alone in the bandwidthconstrained four-core configuration. We conclude that Hermes provides significant and consistent performance improvement in the bandwidth-constrained eight-core system due to its highly-accurate off-chip load prediction.

1.15 1.1 1.05 1 SPEC06

PARSEC

CVP

MIX

GEOMEAN

Figure 6.15: Speedup in eight-core workloads.

6.6.4

Performance Sensitivity Analysis

Effect of Main Memory Bandwidth Figure 6.16(a) shows the speedup of Hermes, Pythia, and Hermes combined with Pythia over the no-prefetching system in single-core workloads by scaling the main memory bandwidth. We make two key observations. First, Hermes combined with Pythia consistently outperforms 1 Pythia in every main memory bandwidth configuration from 16 × to 4× of the baseline system. Hermes combined with Pythia outperforms Pythia alone by 6.2% and 5.5% in the main memory bandwidth configuration with 200 and 12800 million transfers per second (MTPS), respectively. Second, Hermes by itself outperforms Pythia in highly-bandwidth-constrained configurations. This is due to the highly-accurate off-chip load predictions made by POPET, which incurs less main memory bandwidth overhead than the aggressive, less-accurate prefetching decisions made by Pythia. Hermes outperforms Pythia by 2.8% and 8.9% in 400 and 200 MTPS configurations, respectively.

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

Geomean speedup over the No-prefetching system

Hermes 1.3

Pythia

Pythia+Hermes

(a)

1.25 1.2 1.15 1.1 1.05

~Intel Xeon 6258R

~AMD EPYC Rome 7720P

0.95

~AMD Threadripper 3990X

0.9 200

400

800

1600 3200 6400 12800

Prefetcher-only 1.3 Geomean speedup over the No-prefetching system

Prefetcher + Hermes-P

Prefetcher + Hermes-O

(b)

1.25 1.2 1.15 1.1 1.05 1 Pythia

Main Memory MTPS (in log scale)

Bingo

SPP

MLOP

SMS

Prefetcher types

Figure 6.16: Performance sensitivity to (a) main memory bandwidth and (b) baseline prefetcher. The baseline system configuration is highlighted in green. Other highlighted configurations closely match with various commercial processors [4, 5, 29].

Effect of the Baseline Prefetcher We evaluate Hermes combined with four recently-proposed data prefetchers: Bingo [120], SPP [465] (with perceptron filter [148]), MLOP [761], and SMS [792]. For each experiment, we replace the baseline LLC prefetcher Pythia with a new prefetcher and measure the performance improvement of the prefetcher by itself and Hermes combined with the prefetcher. Figure 6.16(b) shows the performance of the baseline prefetcher, and Hermes-P/O combined with the baseline prefetcher, normalized to the no-prefetching system in single-core workloads. The key takeaway is that Hermes combined with any baseline prefetcher consistently outperforms the baseline prefetcher by itself for all four evaluated prefetching techniques. Hermes+prefetcher outperforms the prefetcher alone by 6.2%, 5.1%, 7.6%, and 7.7%, for Bingo, SPP, MLOP, and SMS as the baseline prefetcher. Effect of the Hermes Request Issue Latency To analyze the performance benefit of Hermes over a wide range of processor designs with simple or complex on-chip datapath, we perform a performance sensitivity study by varying the Hermes request issue latency. Figure 6.17(a) shows the performance of Hermes combined with Pythia normalized to the no-prefetching system in single-core workloads as Hermes request issue latency varies from 0 cycles to 24 cycles. The dashed-line represents the performance of Pythia alone. We make two key observations. First, the speedup of Hermes combined with Pythia decreases as the Hermes request issue latency increases. Second, even with a pessimistic Hermes request issue latency of 24 cycles, Hermes combined with Pythia outperforms Pythia. Pythia+Hermes outperforms Pythia by 5.7% and 3.6% with 0-cycle and 24-cycle Hermes request issue latency, respectively. Effect of the On-chip Cache Hierarchy Access Latency We evaluate Hermes by varying the on-chip cache hierarchy access latency. For each experiment, we keep the L1 and L2 cache access latencies unchanged and vary the LLC access latency from 25-cycles to 50-cycles, to mimic the access latencies of a wide range of sliced LLC

(a)

1.25 1.24

Pythia+ Hermes-O

1.23

Pythia+ Hermes-P

1.22 1.21 1.2

Pythia alone

1.19 1.18 0

1.27

Pythia + Hermes-P

(b)

1.25 1.23

Pythia + Hermes-O

~Intel i9-12900K

1.26

Pythia

1.29

~AMD Ryzen 5950X

1.27

Geomean speedup over the No-prefetching system

6.6: HERMES: EVALUATION

1.21 1.19 1.17 1.15 40

12 15 18 21 24

Hermes request issue latency (in processor cycles)

45 50 55 60 65 On-chip cache hierarchy access latency (in processor cycles)

Figure 6.17: Performance sensitivity to (a) Hermes request issue latency and (b) on-chip cache hierarchy access latency. The baseline system configuration is highlighted in green.

designs with simple or complex on-chip networks. Figure 6.17(b) shows the performance of Pythia, and Hermes (O and P) combined with Pythia, normalized to the no-prefetching system in single-core workloads. We make two key observations. First, Hermes combined with Pythia consistently outperforms Pythia for every on-chip cache hierarchy latency. Hermes-O combined with Pythia outperforms Pythia alone by 3.6% and 6.2% in system with 40-cycle and 65-cycle on-chip cache hierarchy access latency, respectively. Second, the performance improvement by Hermes combined with Pythia increases as the on-chip cache hierarchy access latency increases. Thus, we posit that Hermes can provide even higher performance benefit in future processors with longer on-chip cache access latencies. Effect of Reorder Buffer Size

Geomean speedup over the No-prefetching system

Hermes

Pythia

Pythia+Hermes

1.35

(a)

1.3 1.25 1.2 1.15 1.1 1.05 1 256

512 768 ROB Size

1024

Geomean speedup over the No-prefetching system

Figure 6.18(a) shows the performance of Hermes, Pythia, and Hermes combined with Pythia normalized to the no-prefetching system in single-core workloads as the size of reorder buffer (ROB) varies from 256 entries to 1024 entries. The key takeaway is that Hermes combined with Pythia outperforms Pythia alone in every ROB size configuration. Pythia+Hermes outperforms Pythia by 6.7% and 5.3% in a system with 256-entry and 1024-entry ROB. Hermes

Pythia

Pythia+Hermes

1.3

(b)

1.25 1.2 1.15 1.1 1.05 1 3

6 12 LLC size per core (in MB)

Figure 6.18: Performance sensitivity to (a) reorder buffer size and (b) LLC size. The baseline system configuration is highlighted in green.

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

Effect of LLC Size Figure 6.18(b) shows the performance of Hermes, Pythia, and Hermes combined with Pythia normalized to the no-prefetching system in single-core workloads as the per-core last-level cache (LLC) size varies from 3 MB to 24 MB. The key takeaway is that Hermes combined with Pythia outperforms Pythia alone in every LLC size configuration. Even in a system with a 12 MB and 24 MB LLC per core, Pythia+Hermes provides 2.5% and 1.3% performance benefit over Pythia alone. Effect of Perceptron Activation Threshold

Accuracy/Coverage %

100%

Speedup

Accuracy

Coverage

1.26

80%

1.256

60%

1.252

40%

1.248

20%

1.244

1.24 -38

-34

-30

-26

-22 -18 -14 Activation threshold

-10

-6

-2

Geomean speedup over the No-prefetching system

We evaluate the impact of the perceptron activation threshold (𝜏𝑎𝑐𝑡 ) on Hermes’s performance by varying 𝜏𝑎𝑐𝑡 . Figure 6.19 shows POPET’s accuracy and coverage (as line graphs on the left y-axis) and the performance of Hermes combined with Pythia over the no-prefetching system (as a bar graph on the right y-axis) across all single-core workloads as 𝜏𝑎𝑐𝑡 varies from −38 to 2. The key takeaway from Figure 6.19 is that POPET’s accuracy (coverage) increases (decreases) as 𝜏𝑎𝑐𝑡 increases. However, Hermes’s performance gain peaks near 𝜏𝑎𝑐𝑡 = −26, which favors higher coverage by trading off accuracy. As POPET’s accuracy directly impacts Hermes’s main memory request overhead (and hence its performance in bandwidth-constrained configurations), we set 𝜏𝑎𝑐𝑡 = −18 in POPET. Doing so simultaneously optimizes both POPET’s accuracy and coverage.

Figure 6.19: Effect of the activation threshold on POPET’s accuracy and coverage (on the left y-axis) and Hermes’s speedup (on the right y-axis) for all single-core workloads.

6.6.5

Effect of Prefetchers on Off-Chip Prediction

Effect on Prediction Accuracy and Coverage Figure 6.20(a) shows the off-chip load prediction accuracy and coverage when Hermes is combined with different baseline data prefetchers. We make two key observations. First, POPET’s accuracy and coverage varies widely based on the baseline data prefetcher. When combined with Pythia, Bingo, SPP, MLOP, and SMS, POPET provides accuracy of 77.3%, 78.1%, 73.4%,

6.6: HERMES: EVALUATION

79.9%, and 76.0%, while providing coverage of 74.2%, 77.6%, 65.9%, 81.7%, and 84.7%, respectively. Second, in a system without any baseline data prefetcher, POPET provides significantly higher accuracy (88.9%) and coverage (93.6%) than any configuration with a baseline prefetcher. This shows that, the prefetch requests generated by a sophisticated data prefetcher interfere with the off-chip load prediction. This is why POPET’s accuracy and coverage increases in absence of a data prefetcher. 100% Accuracy

Coverage

90% 80% 70% 60% 50%

Increase in main memory requests over the No-prefetching system

60%

(a)

(b) 50%

Prefetcher

5.9%

Prefetcher+Hermes

5.8%

8.6%

15.6%

40% 7.6%

30% 20% 10%

Py th ia +H er m Bi es ng o+ H er m es SP P+ H er m M es LO P+ H er m es SM S+ H er m es H er m es al on e

0% Pythia

Bingo

SPP

MLOP

SMS

Figure 6.20: (a) Variation of off-chip load prediction accuracy and coverage and (b) the increase in the main memory requests with different data prefetchers.

Effect on Main Memory Requests Figure 6.20(b) shows the percentage increase in the main memory requests over the noprefetching system by different types of data prefetchers alone, and in combination with Hermes in all single-core workloads. Combining Hermes with the baseline prefetcher increases the main memory request overhead by 5.9%, 7.6%, 5.9%, 8.6%, and 15.6% for the baseline prefetchers Pythia, Bingo, SPP, MLOP, and SMS, respectively.

6.6.6

Limit Study by Varying Hermes Request Issue Latency

As we discuss earlier in §6.4.2, Hermes’s performance gain significantly depends on the Hermes request issue latency. We already conduct a bounded performance sensitivity study with Hermes request latency varying from 0 cycle to 24 cycles in §6.6.4. In this section, we conduct a performance limit study by pushing the range of the Hermes request issue latency even further to understand two key aspects: (1) under what range of Hermes request issue latency, Hermes’s performance benefits become negligible, and (2) how much performance benefit we can expect from Hermes in a real system under a realistic on-chip network latency. To accurately estimate the on-chip network latency in a real commercial processor, we first estimate the latency to access a banked-SRAM array of equal size to the L2/LLC of our baseline processor (which is modeled after the Intel Alder Lake processor; see §6.5) using PCACTI [42], and then subtract the SRAM array access latency from the publicly-reported cache access

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

latency (which inherently includes the on-chip network access latency). Our evaluation yields an optimistic estimate of 2.01ns (equivalent to 8 processor cycles for our baseline processor clocked at 4GHz) and 2.65ns (equivalent to 11 processor cycles) latency for the L2- and LLCsized SRAM arrays, respectively. This gives us a pessimistic estimate of 31 cycles6 for the total on-chip network latency for accessing L2 and LLC. Figure 6.21 shows the geomean performance of Hermes and Pythia+Hermes under varying Hermes request issue latency. We make two key observations. First, Hermes’s performance gain, both in standalone and in presence of Pythia, reduces for Hermes request issue latency of 51 cycles or more. This is expected, as with such latency, a speculative Hermes request would arrive at the memory controller nearly at the same time as its corresponding regular load request,7 thus loosing the latency hiding opportunity. With 51-cycle Hermes request issue latency, Hermes improves performance by only 2.1% on average over a no-prefetching system, and Hermes combined with Pythia improves performance by only 1.5% over Pythia-alone. Second, even with our realistic 31-cycle on-chip network latency, Hermes-alone provides a significant 6.3% performance gain over no-prefetching system, and Hermes with Pythia provides 3.3% performance over Pythia-alone. Thus we conclude that Hermes would provide tangible performance benefit even on a real-system with complex on-chip network topology and latency. Pythia+Hermes

1.3 1.25 1.2

Pythia+ Hermes-O

1.15

Pythia alone

Pythia+ Hermes-P

Hermes Request Issue Latency to expect in an Intel Alder Lake-like processor

1.1 Hermes-O

1.05

Hermes-P

1 0

Geomean speedup over the No-prefetching system

Hermes

Hermes request issue latency (in processor cycles)

Figure 6.21: Performance of Hermes and Pythia+Hermes while varying Hermes request issue latency.

6 Estimated latency spent on on-chip network for accessing L2 = reported L2 access latency (i.e., 10 cycles) -

estimated L2-sized SRAM access latency (i.e., 8 cycles) = 2 cycles; Estimated latency spent on on-chip network for accessing LLC = reported LLC access latency (i.e., 40 cycles) - estimated LLC-sized SRAM access latency (i.e., 11 cycles) = 29 cycles; 7 The minor performance gains we observe even with Hermes request issue latencies higher than 51 cycles is primarily stemming from hiding any latency caused by queuing delays during cache hierarchy traversal.

6.6: HERMES: EVALUATION

6.6.7

Performance Evaluation using DPC4 Traces

Figure 6.22 shows performance of Hermes (P and O), Pythia, and Hermes combined with Pythia normalized to the no-prefetching system in 483 single-core DPC4 workloads. The key observation is that, without any additional tuning for these traces, Hermes-O alone and combined with Pythia improve performance over a no-prefetching system by 2.0% and 8.8% on average, respectively, whereas Pythia alone provides performance gain of 7.7%.8 In every workload category, Hermes provides performance benefit when applied alone and when combined with Pythia. Hermes-O

Pythia (baseline)

Pythia + Hermes-P

Pythia + Hermes-O

1.08

1.088

1.077

1.1

1.085

1.12

1.04 1.02

1.020

1.06 1.016

Geomean speedup over the No-prefetching system

Hermes-P

1 AIML

GMS

Google

GEOMEAN

Figure 6.22: Speedup in 483 single-core DPC4 workload traces.

Effect of the Off-Chip Load Prediction Mechanism. Figure 6.23(a) shows the geomean performance of Pythia alone and Pythia combined with Hermes with three different offchip prediction mechanisms: POPET, HMP, and TTP, across all 483 DPC4 workloads. We make two key observations. First, Hermes with POPET consistently outperforms both Pythia alone and Pythia with Hermes-HMP in all workload categories. Second, when combined with Pythia, Hermes-POPET underperforms Hermes-TTP. To understand the reason behind POPET’s under-performance, Figure 6.23(b) and Figure 6.23(c) report the off-chip prediction accuracy and coverage of HMP, TTP, and POPET, respectively. Although POPET achieves a substantially higher prediction accuracy (79.5%) than TTP (15.5%), its coverage is significantly lower than that of TTP. POPET only covers 41.2% of the true off-chip load requests, whereas TTP achieves a coverage of 92.1%. This limited coverage constrains POPET’s overall effectiveness and ultimately results in lower performance compared to TTP when combined with Pythia. Effect of the Baseline Prefetcher. Figure 6.24 shows the geomean performance of the baseline prefetcher, and Hermes-P/O combined with the baseline prefetcher, normalized to the 8 Note that, Pythia’s performance benefit shown here is substantially lower than that shown in §5.6.7. This is because while §5.6.7 evaluates Pythia at the L2 cache (see §5.5), here we evaluate Pythia at the LLC (see §6.5). Relocating Pythia from the L2 cache to the LLC significantly reduces the geometric mean performance improvement for AIML and Google workloads. Specifically, the performance improvement decreases from 23.6% to 2.2% for AIML workloads and from 14% to 9.1% for Google workloads. In contrast, GMS workloads exhibit only a marginal change, with performance improvement decreasing from 5.9% to 5.1%.

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION Pythia + Hermes-TTP

Pythia + Hermes-POPET

(a)

1.077

1.1 1.08

1.088

1.12

Pythia + Hermes-HMP

1.082

Geomean speedup over the No-prefetching system

Pythia (Baseline) 1.14

1.103

1.06 1.04 1.02 1 AIML HMP

80%

TTP

Google

POPET

(b)

60% 40%

GEOMEAN HMP

100% Coverage %

Accuracy %

100%

GMS

TTP

POPET

(c)

80% 60% 40% 20%

20%

0% AIML

GMS

Google

AVG

AIML

GMS

Google

AVG

Figure 6.23: (a) Speedup of Hermes with three different off-chip predictors, HMP, TTP, and POPET in single-core DPC4 workloads. Comparison of (b) accuracy and (c) coverage of POPET against those of HMP and TTP in single-core DPC4 workloads.

no-prefetching system across all 483 single-core DPC4 workload traces. The key takeaway is that Hermes combined with any baseline prefetcher consistently outperforms the baseline prefetcher by itself for all four evaluated prefetching techniques. Hermes+prefetcher outperforms the prefetcher alone by 1.1%, 1.0%, 1.2%, 1.0%, 1.5%, for Pythia, Bingo, SPP, MLOP, and SMS as the baseline prefetcher, respectively. Overall, the performance results with DPC4 traces provide empirical evidence that Hermes generalizes beyond its design-time workloads. Despite never observing these traces during development and despite operating in a plug-and-play configuration without additional tuning, Hermes provides considerable, and often the state-of-the-art, performance gains in emerging workloads.

6.6.8

Power Overhead Analysis

To accurately estimate Hermes’s dynamic power consumption, we model our single-core configuration in McPAT [502] and compute processor power consumption using statistics from performance simulations. Figure 6.25 shows the runtime dynamic power consumed by Hermes, Pythia, and Hermes combined with Pythia, normalized to the no-prefetching system for all single-core workloads. We make two key observations. First, Hermes increases processor power consumption by only 3.6% on average over the no-prefetching system, whereas Pythia increases power consumption by 8.7%. Second, Hermes combined with Pythia incurs only 1.5%

6.7: HERMES: SUMMARY

Geomean speedup over the No-prefetching system

Prefetcher-only

Prefetcher + Hermes-P

Prefetcher + Hermes-O

1.1 1.08 1.06 1.04 1.02 1 Pythia

Bingo

SPP

MLOP

SMS

Prefetcher type

Figure 6.24: Performance sensitivity to baseline prefetcher in single-core DPC4 workloads.

Hermes

Pythia (baseline)

Bus

Hermes

L1-D

No-prefetching

L1-I

No-prefetching

1.2

Pythia (baseline)

1.4

Others

1 0.8 0.6 0.4 0.2

SPEC06

SPEC17

PARSEC

Ligra

CVP

Pythia + Hermes

Hermes

Pythia (baseline)

No-prefetching

Pythia + Hermes

Pythia (baseline)

Hermes

No-prefetching

Pythia + Hermes

Pythia (baseline)

Pythia + Hermes

Hermes

No-prefetching

Pythia + Hermes

Hermes

Pythia (baseline)

0 No-prefetching

Runtime dynamic power normalized to the No-prefetching system

additional power overhead on top of Pythia. We conclude that Hermes incurs only a modest power overhead and is more efficient than Pythia alone.

AVG

Figure 6.25: Processor power consumption of Hermes, Pythia, and Hermes combined with Pythia.

6.7

Summary

We introduce Hermes, a technique that accelerates long-latency off-chip load requests by eliminating the on-chip cache hierarchy access latency from their critical path. To enable Hermes, we propose a perceptron-learning based off-chip load predictor (POPET) that accurately predicts which load requests might go off-chip. Our extensive evaluations using a wide range of workloads and system configurations show that Hermes provides significant performance benefits over a baseline system with a state-of-the-art prefetcher.

6.7.1

Influence on the Research Community

Hermes has been presented at the 55th IEEE/ACM International Symposium on Microarchitecture (MICRO) on October, 2022 [133] and was recognized with the Best Paper Award at

CHAPTER 6: PERCEPTRON-BASED OFF-CHIP LOAD PREDICTION

MICRO 2022 [47]. Hermes has been officially artifact evaluated with all three badges (i.e., available, functional, and reproducible). We have made Hermes freely-downloadable from our GitHub repository [22] with all evaluated workload traces, scripts, and implementation code required to reproduce and extend it. Since its release, Hermes has already influenced multiple subsequent works as a state-ofthe-art baseline [282, 399, 478, 633, 719, 886]. Most notably, one follow-up work, CLIP [633], extends Hermes’ key observation of predicting only off-chip loads as performance-critical to predicting any load that might block the retirement from reorder-buffer as critical loads. CLIP exploits this prediction to make more informed prefetch decision. Another notable follow-up work, PrefetchX [204], has revealed the existence of an previously-undocumented speculation mechanism already employed in Intel 3rd generation Xeon processors [31] that is similar to Hermes in principle. We believe and hope that Hermes’ key observation and off-chip load prediction mechanism would continue inspiring future works to explore a multitude of other memory system optimizations.

Chapter 7

Synergizing Prefetching and Off-Chip Prediction via Online Reinforcement Learning In last two chapters, we introduce data-driven designs for prefetching and off-chip prediction - two speculative mechanisms that share the same goal of hiding memory access latency. While each improves performance individually, their simultaneous use often requires careful orchestration to avoid negating one another’s benefits. In this chapter, we demonstrate how data-driven design enables autonomous coordination of prefetching and off-chip prediction, unlocking performance gains beyond what either mechanism can achieve alone.

7.1

Motivation and Goal

Data prefetching and off-chip prediction are two key techniques used for hiding long memory access latency in high-performance processors. Data prefetching is a well-studied speculation technique that predicts addresses of memory requests and fetches their corresponding data into on-chip caches before the processor demands them. When accurate, prefetching improves performance by hiding the memory access latency. However, incorrect speculation can lead to significant memory bandwidth overhead and cache pollution [268,799]. Prior works have shown that prefetchers often lose their performance benefits (and even severely degrade performance) in processors with limited memory bandwidth or cache capacity [136, 151, 265, 267, 268, 483, 485, 633, 799]. Off-chip prediction is a more recently proposed speculation technique that predicts which memory requests would go off-chip and fetches their data directly from main memory [133, 397, 478]. Unlike a prefetcher that predicts full cacheline addresses of future memory requests, an off-chip predictor (OCP) makes a binary prediction on a memory request with known cacheline address: speculating whether or not it will access the off-chip main memory. This key difference allows OCP to often produce more accurate predictions than a prefetcher (see §7.1.1). However, OCP can only hide on-chip cache access latency from the critical path of an off-chip memory request, offering lower timeliness than a prefetcher.

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

7.1.1

Key Observations

We make three key observations, highlighting room for performance improvement in processors that employ both OCP and prefetchers: (1) prefetcher and OCP often provide complementary performance benefits, especially in a bandwidth-constrained processor configuration, yet (2) naively combining these two techniques often fails to realize their full performance potential, and (3) existing microarchitectural policies are either not capable of coordinating OCP with multiple prefetchers, or leave substantial room for performance gain. Off-chip Prediction and Prefetching Provide Complementary Performance Benefits Figure 7.1 shows the performance line graph of a state-of-the-art OCP, POPET [133], against a state-of-the-art data prefetcher, Pythia [136], deployed at the L2 cache (L2C) in a memory bandwidth-constrained single-core processor1 across 100 workloads. The graph is sorted in increasing order of Pythia’s speedup over the baseline system without a prefetcher or an OCP. POPET Prefetcher-adverse

Prefetcher-friendly

secret_compute_int_12

1.25

1.50 1.25

Pythia

1.50 483.xalancbmk-127B

1.00

PageRankDelta-24B

0.75

450.soplex-247B

0.50

secret_compute_int_452

0.25

605.mcf_s-1554B

Workload number

0.25 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97

0.00

1.75

POPET

Pythia

Speedup over no prefetching and no OCP

2.00

Workload number

Figure 7.1: Performance line graph of a state-of-the-art off-chip predictor (OCP), POPET [133], and a state-of-the-art prefetcher, Pythia [136], across 100 workloads.

We make three key observations. First, even though Pythia improves performance for the majority of workloads (highlighted in green), it also degrades performance for a significant number of workloads (40 out of 100; highlighted in red) even with its built-in bandwidth-aware throttling mechanism. For ease of discussion, we call the workloads with performance improvement (or degradation) prefetcher-friendly (prefetcher-adverse). Second, in many prefetcher-adverse workloads, POPET improves performance over the baseline. Pythia degrades performance by 11.6% on average across all prefetcher-adverse workloads, whereas POPET improves performance by 1.4%. This contrast arises because, in prefetcheradverse workloads, it is often easier to predict whether a memory request would go offchip than to predict the full cacheline address of a future memory request. For instance, in 1 We model the memory bandwidth-constrained processor with 3.2 GB/s of main memory bandwidth

(see §7.5). This configuration closely matches the per-core main memory bandwidth of many commercial datacenter-class processors, e.g., AMD EPYC 9754S [66], AmpereOne A192 [69], Amazon Graviton 3 [67], and the Arm Neoverse V2 platform [68,169]. Nonetheless, our technique’s benefits hold true for a wide range of memory bandwidth configurations, as shown in §7.6.3.

7.1: ATHENA: MOTIVATION AND GOAL

483.xalancbmk-127B, a workload known for its irregular memory access pattern, POPET predicts off-chip requests with 84.1% accuracy, while Pythia generates prefetch requests with only 28.7% accuracy. As a result, POPET improves performance by 10.3%, whereas Pythia degrades performance by 10.5%. Third, in prefetcher-friendly workloads, however, Pythia provides significantly higher performance benefits (16.0% on average) than POPET (5.9% on average). This is because, in these workloads, Pythia brings data to the cache well ahead of demand, hiding more memory access latency than only hiding the on-chip cache access latency by POPET.2 We conclude that OCP and prefetching often provide complementary performance benefits due to their fundamentally different forms of speculation. Naively Combining OCP with Prefetching Often Fails to Realize Their Full Performance Potential Although OCP and prefetching offer different tradeoffs for different workload categories, naively combining the two mechanisms often fails to realize their full performance potential together. Figure 7.2 compares the performance of POPET and Pythia individually against two combinations of them: (1) Naive<POPET, Pythia>, that simultaneously enables both POPET and Pythia without any coordination, and (2) StaticBest<POPET, Pythia>, that retrospectively (i.e., using end-to-end workload execution results offline) selects the best-performing option for each workload among four possibilities: POPET-only, Pythia-only, both enabled, and both disabled.3 The error bar indicates the range between the first and third quartiles.

Geomean speedup over no prefetching and no OCP

POPET

Pythia

Naive<POPET, Pythia>

StaticBest<POPET, Pythia>

1.4 1.2 1.0 0.8 0.6 Prefetcher-adverse

Prefetcher-friendly

Overall

Figure 7.2: Geomean speedup of POPET, Pythia, Naive, and StaticBest combinations across all workloads.

We make two key observations from Figure 7.2. First, even though Naive provides 4.7% performance improvement over the baseline across all workloads, Naive degrades performance by 11.2% in prefetcher-adverse workloads, effectively masking the performance improvement 2 Even though we demonstrate this observation using POPET and Pythia as the OCP and prefetcher, respec-

tively, we observe this dichotomy across various prefetcher and OCP implementations. In §7.6, we extend this observation to six prefetcher types [136, 148, 465, 604, 628, 761, 792] and three OCP types [133, 397, 905]. 3 While StaticBest estimates the performance headroom of an intelligent coordination policy, it is possible to further improve upon StaticBest by dynamically identifying the best combination for each workload phase, rather than the entire end-to-end workload. In §7.6.5 we show that, by adapting to each workload phase, our proposed technique can outperform StaticBest.

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

that POPET alone could have delivered otherwise (i.e., 1.2% on average). This shows that, even though off-chip prediction provides complementary performance benefits to prefetching, especially in prefetcher-adverse workloads, naively combining both techniques does not realize their full performance potential together. Second, StaticBest combination provides consistent performance benefits in both prefetcher-adverse and prefetcher-friendly workloads and significantly outperforms the Naive combination (by 6.5% on average) across all workloads. These observations demonstrate the need to design an intelligent coordination mechanism between OCPs and prefetchers. Existing Coordination Policies are Either Inflexible or Leave a Large Performance Potential Behind While researchers have proposed numerous techniques to control multiple prefetchers (e.g., [165, 166, 219, 267, 268, 274, 319, 377, 396, 416, 422, 435, 551, 632, 636, 637, 639, 640, 673, 710, 798, 799, 822, 896, 897]), TLP [399] is the only prior technique that aims to control a prefetcher in the presence of an OCP. TLP uses off-chip prediction as a hint to filter out prefetch requests to the L1 data cache (L1D), based on the empirical observation that prefetches filled from off-chip main memory into L1D are often inaccurate (i.e., the cacheline is not subsequently demanded during its cache residency) [399]. While TLP’s observation is often effective for an L1D prefetcher, we observe that it may not hold true for prefetchers employed at higher (i.e., further away from the core) cache levels. Figure 7.3 shows the fraction of prefetch fills from the off-chip main memory that are inaccurate as a box-and-whisker plot.4 We show the fraction for two state-of-the-art prefetchers, IPCP [628] and Pythia, individually employed at two different cache levels. IPCP fills prefetch requests to L1D, whereas Pythia fills to L2C. The key observation is that, while 50.6% prefetch fills to L1D caused by IPCP are inaccurate, only 28.1% of the prefetch fills to L2C caused by Pythia are inaccurate. In other words, an offchip prefetch fill to L2C is nearly half as likely to be inaccurate as an off-chip prefetch fill to L1D, while employing state-of-the-art prefetchers. This fundamental limitation in TLP’s key observation significantly limits its ability to coordinate OCPs with prefetchers that are placed beyond L1D (as we demonstrate in §7.6.1 and §7.6.1). Besides TLP, other prior techniques focus solely on prefetcher control, without considering OCP. We extend two best-performing prior techniques, heuristic-based HPAC [267] and learning-based MAB [319], to coordinate an OCP and a prefetcher. Figure 7.4 compares the performance of these techniques, coordinating POPET as the OCP and Pythia as the L2C prefetcher against Naive and StaticBest combinations across all workloads.5 We make two key observations. First, in prefetcher-adverse workloads, both HPAC and MAB considerably mitigate the performance degradation of the Naive combination. However, neither policy matches the performance of the baseline (without prefetching or OCP), let alone harnesses the potential performance gains of StaticBest. Second, in prefetcher-friendly workloads, these coordination 4 Each box is lower- (upper-) bounded by the first (third) quartile. The box size represents the inter-quartile

range (IQR). The whiskers extend to 1.5× IQR range on each side. The cross-marked value represents the mean. 5 Figure 7.4 excludes TLP as its key observation does not reliably extend to coordinating an L2C prefetcher (Pythia) with an OCP, as shown earlier.

7.1: ATHENA: MOTIVATION AND GOAL

50.6% 28.1%

Figure 7.3: Fraction of prefetch fills from off-chip main memory that are inaccurate.

techniques fall short of the Naive combination. The heuristic-based HPAC falls short due to its reliance on the statically tuned thresholds that are optimized for average-case behavior across workloads. These fixed thresholds cannot adapt to per-workload or phase-specific characteristics, causing HPAC to make conservative coordination decisions even when prefetching is beneficial. While MAB avoids such static thresholds, it still falls short as it makes decisions agnostic to any system-level features (e.g., prefetcher/OCP accuracy, prefetch-induced cache pollution). Naive<POPET, Pythia>

HPAC<POPET, Pythia>

MAB<POPET, Pythia>

StaticBest<POPET, Pythia>

Geomean speedup over no prefetching and no OCP

1.6 1.4 1.2 1.0 0.8 0.6 Prefetcher-adverse

Prefetcher-friendly

Overall

Figure 7.4: Geomean speedup of Naive, HPAC, MAB, and StaticBest combinations across all workloads.

We conclude that while there is a rich literature on prefetcher coordination techniques, the only OCP-aware prefetcher control mechanism (i.e., TLP) lacks flexibility (i.e., the ability to coordinate OCP with multiple prefetchers employed at various levels of the cache hierarchy), while other techniques (e.g., HPAC, MAB) leave significant performance potential behind.

7.1.2

Our Goal

Our goal in this work is to design a holistic framework that can autonomously coordinate offchip prediction with multiple prefetching techniques employed at various levels of the cache hierarchy by taking multiple system-level features into account, thereby delivering consistent

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

performance benefits, regardless of the underlying prefetcher-OCP combination, workload, and system configuration.

7.2

Formulating Prefetcher-OCP Coordination using Reinforcement Learning

To this end, we formulate the dynamic coordination between prefetching and off-chip prediction as a reinforcement learning (RL) problem. More specifically, we propose Athena,6 an RLbased agent that dynamically learns to synergize off-chip prediction with multiple prefetching techniques employed throughout the cache hierarchy of modern state-of-the-art processor by interacting with the processor and memory system.

7.2.1

Why is RL a Good Fit for Prefetcher-OCP Coordination?

RL is well-suited for coordinating data prefetchers and OCP due to the following three key advantages. Less Reliance on Static Heuristics and Thresholds. Heuristic-based prefetcher coordination policies typically rely on statically defined thresholds [264, 265, 267, 268, 799]. These thresholds are manually tuned and inherently inflexible, often resulting in suboptimal performance when workloads or system conditions change [383, 576]. Formulating prefetcher-OCP coordination as an RL problem allows a hardware architect to focus on what performance targets the coordinator should achieve and which system-level features might be useful, rather than spending time on manually devising fixed algorithms and/or thresholds that describe precisely how the coordinator should achieve that target. This not only significantly reduces the human effort needed for a coordinator design, but also yields higher-performing coordination (as shown in §7.6). Online Feedback-Driven Learning. RL provides two key benefits over supervised learning methods (e.g., SVM [234, 850], decision tree [168]) in formulating the prefetcher-OCP coordination problem. First, prefetcher-OCP coordination lacks a well-defined ground-truth label (e.g., which mechanism is beneficial to enable) for each system state. The optimal coordination decision depends on delayed, system-level performance outcomes that manifest only after executing an action under dynamic resource contention and changing workload behavior. As a result, generating labeled training data would require exhaustive offline exploration across workloads, phases, and system configurations, and such labels may not generalize as the underlying hardware, prefetchers, or OCPs change. Second, supervised learning models are inherently static once trained and cannot naturally adapt online to changing workload behavior without repeated retraining. In contrast, RL directly optimizes long-term performance using online reward feedback and continuously updates its policy, making it better suited to model prefetcher-OCP coordination. 6 Named after the Greek goddess of wisdom and strategic warfare [62].

7.3: ATHENA: OVERVIEW

Q-Value-Driven Prefetcher Aggressiveness Control. The Q-values learned by the RL agent provide a natural ranking over available actions, reflecting their expected utility. For actions involving prefetching (either standalone or combined with OCP), the corresponding Q-values implicitly encode the agent’s confidence in the prefetcher’s effectiveness relative to alternative actions. An RL agent can leverage these Q-values to also dynamically control prefetcher aggressiveness. Higher Q-values correspond to stronger confidence, prompting more aggressive prefetching, whereas lower Q-values imply uncertain benefits, resulting in more conservative prefetching. Importantly, this Q-value-driven prefetcher aggressiveness control incurs no additional hardware overhead, since the learned Q-values jointly govern both prefetcher/OCP selection and prefetcher aggressiveness.

7.3

Athena: Overview

Athena formulates the coordination of data prefetchers and the off-chip predictor as an RL problem, as illustrated in Figure 7.5. Here, Athena acts as an RL agent that continuously learns and adapts its prefetcher-OCP coordination policy by interacting with the processor and memory system. As Figure 7.5 shows, Athena comprises a key hardware structure, Q-Value Storage (QVStore), whose purpose is to store the Q-values of state-action pairs encountered during Athena’s online operation. Each timestep for Athena corresponds to a fixed-length epoch of workload execution (e.g., 𝑁 retired instructions). During an execution epoch, Athena observes and records various system-level features (e.g., prefetcher/OCP accuracy, memory bandwidth usage). At the end of every epoch, Athena uses the recorded feature values as state information to index into the QVStore (step 1 in Figure 7.5) to select a coordination action, i.e., whether to enable only prefetcher, only OCP, both mechanisms, or none of them (step 2 ). If Athena decides to enable the prefetcher, it further determines the prefetcher aggressiveness based on the magnitude of the selected action’s Q-value (see §7.3.2). At the end of every epoch, Athena receives a numerical reward that measures the change in multiple system-level metrics (e.g., the number of cycles taken to execute an epoch) to evaluate the impact of its actions on system performance (step 3 ). Athena uses this reward to autonomously and continuously learn a prefetcher-OCP coordination policy, that can adapt to diverse workloads and system configurations.

7.3.1

State

We define the state as a vector of system-level features, where each feature encapsulates a distinct behavior of the memory subsystem (e.g., the accuracy of the prefetcher, the main memory bandwidth usage) observed during the current execution epoch. Collectively, these features represent the runtime conditions relevant to making a prefetcher-OCP coordination decision. While Athena can, in principle, learn a coordination policy using any arbitrary set of features, increasing the state dimension rapidly increases the storage required to maintain the Q-values of all observed state-action pairs. To bound the storage overhead of Athena, we fix the state representation offline using a two-step process. First, guided by domain knowledge, we identify a set of seven candidate system-level features that are expected to influence

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

State

(e.g., bandwidth usage, prefetcher/OCP accuracy)

Processor & Memory Subsystem

Reward

Athena

Q-Value Q-Value Table Table QVStore

Action

1) enable/disable prefetcher and OCP 2) set prefetcher aggressiveness

Action

Figure 7.5: High-level overview of Athena as an RL agent.

the effectiveness of prefetcher-OCP coordination. Table 7.1 summarizes each feature, its computation method, and the rationale behind its inclusion. Second, we perform offline feature selection using automated design-space exploration (as described in §7.4.3) to determine the final subset of features that Athena uses to construct the state vector. Feature Prefetcher accuracy

Measurement # demand hits # prefetches issued

Rationale Effectiveness of prefetching

# off-chip demand hits # off-chip predictions

Effectiveness of OCP

Bandwidth usage

current DRAM bandwidth max DRAM bandwidth

Memory bus pressure

Cache pollution

# prefetch-evicted demand misses # total demand misses

Interference caused by prefetches

# prefetch requests to DRAM # total DRAM requests

Prefetcher’s share of memory traffic

OCP bandwidth

# OCP requests to DRAM # total DRAM requests

OCP’s share of memory traffic

Demand bandwidth

# demand requests to DRAM # total DRAM requests

Demand’s share of memory traffic

OCP accuracy

Prefetch bandwidth

Table 7.1: Candidate features considered for Athena’s state representation.

7.3: ATHENA: OVERVIEW

7.3.2

Action

Athena’s action space consists of four coordination decisions, i.e., whether to enable (1) only prefetcher, (2) only OCP, (3) both mechanisms, or (4) none of them. These actions allow Athena to explicitly coordinate the prefetcher and the OCP at a coarse granularity, i.e., only enabling or disabling any given mechanism as a whole. However, when Athena selects an action that enables prefetching, it further determines the prefetcher aggressiveness. This aggressiveness is derived directly from the learned Q-values as a function of the relative difference between the Q-value of the selected action and the average Q-value of the remaining actions. The underlying rationale for this Q-value-driven prefetcher aggressiveness control is that the magnitude of the selected action’s Q-value implicitly encodes Athena’s confidence in the prefetcher’s effectiveness. A larger separation between the Q-value of the selected action (that enables the prefetcher, either without or with the OCP) and those of the alternative actions indicates stronger historical evidence that enabling prefetching is beneficial in the current state, thus warranting more aggressive prefetching. On the other hand, a smaller separation reflects uncertainty in prefetcher effectiveness, prompting more conservative prefetching. Algorithm 2 formalizes this Q-value-based prefetcher aggressiveness control mechanism.7 Algorithm 2 Q-value-driven prefetcher aggressiveness control 1: procedure SelectPrefetchDegree 2: 𝑎 ∗ ← arg max𝑎∈A 𝑄 (𝑎) 3: 𝑎𝑣𝑔𝑄 ← average Q-value of all actions except 𝑎 ∗ 4: Δ𝑄 ← 𝑄 (𝑎 ∗ ) − 𝑎𝑣𝑔𝑄 ⊲ compute the Q-value confidence 5: 𝑟 ← min(1, Δ𝑄/𝜏) ⊲ normalize confidence w.r.t. hyperparam 𝜏 6: 𝑑 ← ⌊𝑟 · 𝑑 max ⌋ ⊲ adjust prefetch degree based on normalized confidence 7: return 𝑑

First, Athena selects the action 𝑎 ∗ with the highest Q-value. It then estimates the confidence of this decision by comparing 𝑄 (𝑎 ∗ ) against the average Q-value of all remaining actions. The resulting confidence ratio Δ𝑄 captures how strongly the selected action is preferred over the alternatives. Using this confidence signal, Athena determines the final prefetch degree as a fraction of 𝑑 max , where 𝑑 max denotes the number of prefetch requests that the underlying prefetcher can issue while operating at full aggressiveness. If this confidence ratio exceeds a hyperparameter 𝜏, Athena enables the prefetcher at full aggressiveness, reflecting high confidence in the benefits of prefetching in the current system state. Otherwise, Athena scales the prefetch degree proportionally to Δ/𝜏, issuing fewer prefetch requests.

7.3.3

Reward

The reward defines the optimization objective for Athena. Prior works often use the change in instructions committed per cycle (IPC) as the only system-level metric to train the RL 7 Here, we represent prefetcher aggressiveness using the prefetch degree, i.e., the number of prefetch requests

issued per demand trigger. However, the proposed Q-value-driven aggressiveness control mechanism can also be applied to alternative aggressiveness definitions.

100

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

agent [151, 274, 319, 396]. However, a change in IPC may originate from two different sources: (1) the coordination actions taken by the agent, and (2) the inherent variations in workload behavior, that are independent of the agent’s actions. As such, using IPC as the sole reward can be unreliable and may mislead the learned policy. To address this limitation, Athena introduces a composite reward framework that explicitly separates the effects of Athena’s action on the system from the inherent variations in workload behavior. More specifically, we define the reward for Athena at a given timestep 𝑡 using two components: (1) correlated reward (𝑅𝑡𝑐𝑜𝑟𝑟 ), which encapsulates the effect of Athena’s action on the system, and (2) uncorrelated reward (𝑅𝑡𝑢𝑛𝑐𝑜𝑟𝑟 ), which encapsulates the inherent change in program behavior. The overall reward (𝑅𝑡 ) is defined using these two component rewards as: 𝑅𝑡 = 𝑅𝑡𝑐𝑜𝑟𝑟 − 𝑅𝑡𝑢𝑛𝑐𝑜𝑟𝑟

(7.1)

By subtracting the uncorrelated reward component from the correlated reward, Athena aims to isolate the performance impact that is causally attributable to its coordination actions from variations induced by inherent workload behavior. This allows Athena to learn a higherperforming prefetcher-OCP coordination policy than it would have otherwise learned using a single, conflated reward signal (as we show in §7.6.5). Correlated Reward. We define the correlated reward at a given timestep 𝑡 as a linear combination of the changes in the constituent system-level metrics that are influenced by Athena’s actions in two consecutive timesteps. Formally, the correlated reward, 𝑅𝑡𝑐𝑜𝑟𝑟 , is defined as: 𝑅𝑡𝑐𝑜𝑟𝑟 =

∑︁

𝑐𝑜𝑟𝑟 𝜆𝑖 · Δ𝑀𝑖,𝑡

(7.2)

𝑖 𝑐𝑜𝑟𝑟 denotes the change in the 𝑖-th correlated system-level metric observed beHere, Δ𝑀𝑖,𝑡

tween timesteps (𝑡 − 1) and 𝑡, and 𝜆𝑖 is a hyperparameter that captures the relative weight of this metric to the overall reward.8 In principle, any system-level metric that is directly affected by Athena’s actions (e.g., execution cycles, number of last-level cache misses) can be incorporated into the correlated reward. In practice, we conduct an offline sensitivity analysis over a broad set of candidate metrics and select three metrics shown in Table 7.2 as the constituents of the correlated reward that provide a stable and informative learning signal across diverse workloads and system configurations. Uncorrelated Reward. We define the uncorrelated reward at a given timestep 𝑡 as a linear combination of the changes in its constituent metrics that are largely independent of Athena’s actions but are influenced by the inherent variations in workload behavior. Formally, the uncorrelated reward, 𝑅𝑡𝑢𝑛𝑐𝑜𝑟𝑟 , is defined as: 𝑅𝑡𝑢𝑛𝑐𝑜𝑟𝑟 =

∑︁

𝜆 𝑗 · Δ𝑀 𝑢𝑛𝑐𝑜𝑟𝑟 𝑗,𝑡

(7.3)

𝑗 8 The values of weight parameters are tuned offline using the automated design-space exploration (see §7.4.3).

7.4: ATHENA: DETAILED DESIGN

101

Here, Δ𝑀 𝑢𝑛𝑐𝑜𝑟𝑟 denotes the change in the 𝑗-th uncorrelated metric observed between 𝑗,𝑡 timesteps (𝑡 − 1) and 𝑡, and 𝜆 𝑗 is a hyperparameter that captures its relative weight to the overall reward. While any metric that is affected by changes in workload behavior can be incorporated into the uncorrelated reward, we select two metrics shown in Table 7.2 as the constituents of the uncorrelated reward, based on offline sensitivity analysis. Reward Component 𝑅𝑡corr 𝑅𝑡uncorr

Constituent Metric # Cycles # LLC misses LLC miss latency # Load instructions # Mispredicted branches

Weight 𝜆cycle 𝜆LLCm 𝜆LLCt 𝜆load 𝜆MBr

Table 7.2: Constituent metrics of Athena’s reward.

7.4

Athena: Detailed Design

7.4.1

QVStore Organization

The QVStore maintains the Q-values of all state-action pairs encountered by Athena during online execution. Unlike prior RL-based approaches that rely on deep neural networks to approximate Q-values [396], or operate without state information [151, 319], Athena adopts a lightweight and hardware-friendly tabular organization for storing Q-values, tailored for lowlatency access and online updates.

# hash1

. . . System-level featureF

State vector

System-level feature2

concatenate

(e.g., prefetcher accuracy)

(32-bit)

# hash2

index

q1(s, a)

index

Plane2 . . .

. . . # hashk

Plane1

index

q2(s, a)

qk(s, a)

𝚺

Q(s, a)

argmax

System-level feature1

sum partial Q-values

Planek

Figure 7.6: Organization of QVStore.

Figure 7.6 demonstrates the organization of QVStore and the procedure for retrieving the Q-value of a given state 𝑆 and action 𝐴. As the number of Q-values that need to be stored for every possible state-action pair grows rapidly with (1) the number of constituent features in a state vector, and (2) the number of bits used to represent each feature, naively implementing

102

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

the QVStore as a monolithic table quickly becomes impractical due to its storage overhead. Such a design also incurs prohibitive access latency and power overhead, making it unsuitable for designing a timing-critical hardware RL agent. To address these challenges, Athena organizes the QVStore as a partitioned structure comprising 𝑘 independent tables, each of which we call a plane. Each plane stores a partial Q-value of a given state-action pair. This partitioned organization enables Athena to decouple the storage cost from the full combinatorial state space while simultaneously supporting fast, parallel access. For a given state-action pair, Athena retrieves its corresponding Q-value from the partitioned QVStore in three stages, as shown in Figure 7.6. First, Athena constructs the state vector by concatenating all feature values. Second, Athena applies 𝑘 distinct hash functions to the state vector, each producing an index to a plane to retrieve the corresponding partial Qvalue in parallel. Third, Athena computes the final Q-value by summing the partial Q-values across all planes. At the end of each epoch, Athena updates the Q-value using the SARSA update rule (see §3.1), applying the update independently to each plane. The partitioned, multi-hash organization of QVStore provides two key benefits. First, hashing the same state into multiple planes strikes a balance between generalization and resolution: similar states are likely to collide in at least some planes, promoting value sharing, while dissimilar states are likely to be de-aliased through independent hashes. Second, partitioning keeps each plane compact, enabling low-latency, energy-efficient parallel reads and updates. Together, these properties substantially reduce storage overhead while supporting fast and scalable Q-value access and updates during Athena’s operation.

7.4.2

State Measurement

Prefetcher Accuracy Athena employs a Bloom filter [153] to track prefetcher accuracy. For every prefetch issued, the corresponding address is inserted into the Bloom filter. Upon each demand access, Athena queries the filter to determine whether the address was prefetched. Prefetch accuracy is computed as the ratio of demand accesses that hit in the filter over the number of issued prefetches. Athena resets the filter at the end of each epoch. OCP Accuracy Athena measures OCP accuracy using two simple counters. When OCP predicts that a demand load will go off-chip, it issues a speculative request to the memory controller to start fetching the data directly from main memory. Athena tracks such predictions using a dedicated counter. When a demand request misses all cache levels and arrives at the memory controller, it indicates that this request indeed went off-chip. Athena increments another counter to track correct predictions. OCP accuracy is computed as the ratio of correctly predicted off-chip accesses over the total number of off-chip predictions. Athena resets both counters at the end of each epoch.

7.4: ATHENA: DETAILED DESIGN

103

Prefetch-Induced Cache Pollution Athena uses a Bloom filter [153] to track prefetch-induced cache pollution at the last-level cache (LLC). When a cache block is evicted from the LLC for a prefetch fill, Athena inserts the evicted address into the filter. If the address of a subsequent LLC miss hits the filter, Athena increments a dedicated counter. Athena resets the filter and the counter at the end of each epoch. This method of measuring prefetch-induced cache pollution is similar to prior works [268,799].

7.4.3

Automated Design-Space Exploration

We employ automated design-space exploration (DSE) to select the optimal state features (see §7.3.1), reward weights (see §7.3.3), and hyperparameters (𝛼, 𝛾, 𝜖, 𝜏, and epoch length). To prevent overfitting, DSE is conducted on 20 selected workloads, which are not included in the final set of 100 workloads. We perform DSE using Cache Design 1 (CD1) in a single-core configuration, with POPET as the OCP and Pythia as the L2C prefetcher (see §7.5.3). The configuration that achieves the best performance across the 20 selected workloads is then applied unaltered to the full set of 100 evaluation workloads, to all other cache designs (see §7.6.1), OCPs (see §7.6.2), prefetchers (see §7.6.3), and multi-core experiments (see §7.6.4). Feature Selection We derive the program features through an iterative process. Starting from the initial set of seven candidate features (see §7.3.1), we begin with the feature that yields the highest standalone performance gain. In each iteration, we include the feature that provides the greatest additional performance improvement while retaining the previously selected features. We observe diminishing performance gains after the fourth iteration. Consequently, we fix the feature set to the following four: prefetcher accuracy, OCP accuracy, bandwidth usage, and prefetch-induced cache pollution, as summarized in §7.4.3.

Category Selected Features Reward Weights Hyperparameters

Final Values (1) prefetcher accuracy, (2) OCP accuracy, (3) bandwidth usage, (4) prefetch-induced cache pollution 𝜆cycle = 1.6, 𝜆LLCm = 0.0, 𝜆LLCt = 0.0, 𝜆load = 0.6, 𝜆MBr = 1.0 𝛼 = 0.6, 𝛾 = 0.6, 𝜖 = 0.0, 𝜏 = 0.12, Epoch length (𝑁 ) = 2𝐾 instructions

Table 7.3: Final Athena configuration derived through automated design-space exploration.

Reward and Hyperparameter Tuning We use grid search [144, 498] to tune reward weights (see §7.3.3) and hyperparameters (see §3.1). For each reward weight and hyperparameter, we define a search range and discretize it into equally spaced grid points. The learning rate 𝛼, the discount factor 𝛾, and the

104

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

exploration rate 𝜖 are searched over [0, 1] in steps of 0.1. Reward weights 𝜆𝑖 and 𝜆 𝑗 are searched over [0, 2] in steps of 0.2. The final reward weights and hyperparameters are shown in §7.4.3.

7.4.4

Overhead Analysis

Storage Overhead §7.4.4 summarizes the storage overhead of Athena. The QVStore is organized into eight planes, each containing 64 rows and 4 columns (one per action). Each entry stores an 8-bit Q-value. To size the Bloom filter used for prefetcher accuracy tracking, we experimentally observe an average of 49 prefetch requests per epoch (i.e., 2𝐾 retired instructions), with a standard deviation (SD) of 50. We therefore size the Bloom filter for prefetcher accuracy tracking at 4096 bits, which yields a false positive rate of 1% when accommodating three SDs above the average (i.e., 199 requests). Similarly, we observe an average of 62 LLC evictions per epoch, with an SD of 58. Thus, we size the Bloom filter for prefetch-induced cache pollution tracking at 4096 bits, providing a false positive rate of 1% when inserting three SDs more evictions than the average (i.e., 236 evictions).

Structure QVStore Accuracy Tracker Pollution Tracker Total

Description # planes = 8, # rows = 64 # columns = 4, entry size = 8 bits 4096-bit Bloom filter, 2 hashes 4096-bit Bloom filter, 2 hashes

Size 2 KB 0.5 KB 0.5 KB 3 KB

Table 7.4: Storage overhead of Athena.

Latency Overhead At the end of every epoch, Athena needs to compute the overall reward from its constituent partial rewards and update the QVStore, both of which incur considerable computation overhead. To account for such computations, we model Athena with a delayed QVStore update latency of 50 cycles (i.e., the QVStore is updated 50 cycles after the end of an epoch). We also sweep the update latency and observe that Athena’s performance benefit is not sensitive to the update latency. This is because Athena needs to query the updated QVStore only at the end of the current epoch (i.e., 2𝐾 retired instructions), which takes considerably longer than even the pessimistic estimate of the update latency.

7.5

Methodology

We evaluate Athena using the ChampSim trace-driven simulator [328]. We faithfully model an Intel Golden Cove-like microarchitecture [19], including its large reorder buffer (ROB),

7.5: ATHENA: EVALUATION METHODOLOGY

105

multi-level cache hierarchy, and publicly reported on-chip cache access latencies [35, 56, 60]. Table 7.5 summarizes the key microarchitectural parameters. The source code of Athena is freely available at [7]. Core L1I/D L2C LLC Main Memory

6-wide fetch/issue/commit, 512-entry ROB, 128-entry LQ, 72-entry SQ, perceptron branch predictor [411], 17-cycle misprediction penalty Private, 48KB, 64B line, 12-way, 16 MSHRs, LRU, 4/5-cycle round-trip latency Private, 1.25MB, 64B line, 20-way, 48 MSHRs, LRU, 15-cycle round-trip latency [60] Shared, 3MB/core, 64B line, 12-way, 64 MSHRs/slice, SHiP [876], 55-cycle round-trip latency [56, 60] 1 rank per channel, 8 banks per rank, 64-bit data bus, 2KB row buffer, 𝑡 RCD = 𝑡 RP = 𝑡 CAS = 12.5ns; DDR4 with 3.2 GB/s per core Table 7.5: Simulated system parameters.

7.5.1

Workloads

We evaluate Athena using a diverse set of workload traces spanning SPEC CPU 2006 [52], SPEC CPU 2017 [53], PARSEC [149], Ligra [771], and real-world commercial workloads from the first Value Prediction Championship (CVP [15]). We only consider workloads in our evaluation that have at least 3 LLC misses per kilo instructions (MPKI) in the no-prefetching and no-OCP system. In total, we use 100 memory-intensive single-core workload traces, as summarized in Table 7.6. SPEC CPU 2006 and SPEC CPU 2017 workloads are collectively referred to as SPEC. All the workload traces used in our evaluation are freely available online [7]. Suite

# Workloads

Example Workloads

SPEC CPU06 SPEC CPU17 Ligra PARSEC CVP

29 20 13 13 25

astar, leslie3d, libquantum, milc, omnetpp, sphinx3, soplex ... bwaves, cactuBSSN, fotonik3d, gcc, lbm, mcf, xalancbmk ... BC, BFS, BFSCC, PageRankDelta, Radii, ... canneal, facesim, fluidanimate, raytrace, streamcluster ... integer, floating point, ... Table 7.6: Workloads used for evaluation.

For multi-core evaluation, we construct 90 four-core and 90 eight-core workload mixes, each comprising three categories. (1) 30 prefetcher-adverse mixes: each workload is randomly selected from the prefetcher-adverse workloads. (2) 30 prefetcher-friendly mixes: each workload is randomly selected from the prefetcher-friendly workloads. (3) 30 random mixes: workloads are drawn uniformly at random from the entire set of 100 workloads. For all single-core simulations, we perform a warm-up of 100 million (M) instructions. SPEC workloads are simulated for 500M, PARSEC, Ligra, and CVP workloads for 150M instructions. For multi-core simulations, each core performs a warm-up of 10M instructions, followed by simulating 50M instructions. The workloads are replayed as needed to ensure all cores reach the required number of simulated instructions.

106

7.5.2

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

Evaluated Prior Prefetcher Control Policies

We compare Athena against three prior prefetcher control policies: TLP [399], HPAC [267], and MAB [319]. Two Level Perceptron (TLP) [399] explicitly incorporates OCP into its coordination framework and combines it with prefetch filtering [399] at the L1D. We adopt the same features, prediction thresholds (𝜏low , 𝜏high ), and filtering threshold (𝜏pref ) as specified in [399]. Hierarchical Prefetcher Aggressiveness Control (HPAC) [267] compares various systemlevel feature values against static thresholds to make prefetch control decisions. Although not designed with OCP in mind, we adapt HPAC to coordinate prefetchers and OCP. We use three system-level features for local aggressiveness control: prefetcher accuracy, OCP accuracy, and main-memory bandwidth usage. We use the bandwidth needed by each core as the feature for global aggressiveness control. For each feature, we tune its static threshold via extensive grid search using the same set of tuning workloads that we use to tune Athena (see §7.4.3). Micro-Armed Bandit (MAB) [319] uses a multi-armed bandit algorithm [696, 827] for its decision-making. Although originally not designed for OCP, we adapt MAB to coordinate OCP with prefetchers. MAB selects whether to enable the prefetcher and OCP based on previous reward feedback derived from the system’s IPC. Thus, our implementation of MAB uses four (eight) arms while coordinating one OCP in the presence of one (two) prefetcher(s). We find the best-performing hyperparameters via grid search [144, 498] using the same set of tuning workloads that we use to tune Athena (see §7.4.3).

7.5.3

Evaluated Cache Designs

To demonstrate Athena’s adaptability, we evaluate Athena across diverse cache designs (CDs), as summarized in Table 7.7. All these cache designs include an OCP alongside the three-level cache hierarchy and differ only in the number and placement of prefetchers, closely mimicking cache hierarchy designs found in commercial processors. For each cache design, we identify suitable prior approaches as comparison points. HPAC and MAB can be adapted to all four cache designs to coordinate OCP with prefetchers. In contrast, TLP, by design, is restricted to cache designs involving an L1D prefetcher (i.e., CD2 and CD4; see §7.1.1). Unless stated otherwise, CD1 serves as the default cache configuration. Cache Design CD1 CD2 CD3 CD4

Description

Comparison Points

OCP + 1 L2C prefetcher OCP + 1 L1D prefetcher OCP + 2 L2C prefetchers OCP + 1 L1D + 1 L2C prefetcher

HPAC, MAB HPAC, MAB, TLP HPAC, MAB HPAC, MAB, TLP

Table 7.7: Evaluated cache designs (CD) and corresponding comparison points.

7.5: ATHENA: EVALUATION METHODOLOGY

7.5.4

107

Evaluated Data Prefetchers

We assess Athena’s flexibility by integrating six prefetchers, i.e., IPCP [628], Berti [604], Pythia [136], SPP [465] with perceptron-based prefetch filter (PPF) [148], SMS [792], and MLOP [761], at various levels of the cache hierarchy. IPCP and Berti are evaluated at L1D and are trained using all memory requests looking up the L1D. Pythia, SPP+PPF, MLOP, and SMS operate at L2C and are trained using all memory requests looking up the L2C. All prefetchers prefetch in the physical address space. Unless stated otherwise, we use IPCP as the default L1D prefetcher, and Pythia as the default L2C prefetcher.

7.5.5

Evaluated Off-Chip Predictors

OCPs

POPET [133], with 5 features HMP [905], with 3 component predictors TTP [397], with metadata budget ~L2 cache size

4 KB 11 KB 1536 KB

TLP, as in [399] HPAC [267], adapted for OCP MAB [319], adapted for OCP Athena (this work)

6.98 KB 0.5 KB 0.1 KB 3 KB

L2C

L1D

0.7 KB 2.55 KB 25.5 KB 39.3 KB 8 KB 20 KB

Prefetchers

IPCP [628], as an L1D-only prefetcher Berti, as configured in [604] Pythia, as configured in [136] SPP+PPF, as configured in [148, 465] MLOP, as configured in [761] SMS, as configured in [792]

Policies

We also evaluate Athena across three OCPs: POPET [133], HMP [905], and TTP [133, 397]. POPET uses a hashed-perceptron network with five program features to make accurate offchip predictions. We evaluate the exact POPET configuration presented in [133]. HMP combines three prediction techniques analogous to hybrid branch prediction: local [900, 901], gshare [549], and gskew [557]. TTP, as introduced by [133, 397], predicts off-chip loads by tracking cacheline tags. We evaluate the exact TTP configuration open-sourced by [133]. Similar to prior work [133], all speculative load requests issued by the OCPs incur a 6-cycle latency before reaching the memory controller. Unless stated otherwise, we use POPET as the default OCP. Table 7.8 summarizes the storage overhead of all evaluated mechanisms.

Table 7.8: Storage overhead of all evaluated mechanisms.

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

108

7.6

Evaluation

7.6.1

Single-Core Evaluation Overview

CD1: OCP with One L2C Prefetcher Figure 7.7 shows the performance improvement of Naive, HPAC, MAB, and Athena when coordinating POPET as the OCP and Pythia as the L2C prefetcher. We make two key observations. First, in prefetcher-adverse workloads, Naive degrades performance by 11.1% compared to the baseline with no prefetching or OCP. This degradation arises because Pythia negatively impacts performance in these workloads, undermining POPET’s gains. In contrast, Athena dynamically identifies that POPET is beneficial and improves performance by 14.0% over Naive, even surpassing POPET’s standalone performance. Second, although Naive harms performance in prefetcher-adverse workloads, it significantly improves performance by 16.7% in prefetcher-friendly workloads. Athena dynamically determines that enabling both POPET and Pythia is advantageous in these workloads, thus closely matching Naive’s performance. Overall, Athena outperforms Naive, HPAC, and MAB by 5.7%, 7.9%, and 5.0%, respectively, across all 100 workloads, demonstrating consistent performance improvements.

Geomean speedup over no prefetching and no OCP

POPET HPAC<POPET, Pythia>

Pythia MAB<POPET, Pythia>

Naive<POPET, Pythia> Athena<POPET, Pythia>

1.6 1.4 1.2 1.0 0.8 SPEC

PARSEC

Ligra

CVP

Prefetcher Prefetcher adverse friendly

Overall

Figure 7.7: Speedup in cache design 1 (CD1).

Workload Category-Wise Performance Analysis. To further analyze Athena’s performance gains in CD1, Figure 7.8(a) shows the workload category-wise speedups as a boxand-whisker plot. We make three key observations. First, for prefetcher-adverse workloads, Athena substantially raises the lower quartile as well as the lower-end whisker relative to Naive, HPAC, and MAB. This implies that Athena improves performance broadly across all prefetcher-adverse workloads, rather than merely alleviating a small number of extreme slowdowns. Second, for prefetcher-friendly workloads, Athena increases both the upper quartile and the upper-end whisker compared to HPAC and MAB, demonstrating consistent performance improvements over these policies across all prefetcher-friendly workloads. Third, when considering all workloads together, Athena improves mean performance relative to HPAC and

7.6: ATHENA: EVALUATION

109

MAB while simultaneously elevating both the lower and upper quartiles. This result indicates that Athena delivers robust and consistent performance gains across a broad spectrum of workload behaviors. Pythia HPAC<POPET, Pythia> Athena<POPET, Pythia>

1.3 1.2 1.1 1 0.9 0.8

Athena<POPET, Pythia>

(b)

1.106 1.111

1.2

(a) Geomean speedup over no prefetching and no OCP

Speedup over no prefetching and no OCP

1.4

HPAC<POPET, Pythia>

MAB<POPET, Pythia> StaticBest<POPET, Pythia>

1.6 1.5

Naive<POPET, Pythia>

1.1

1.046 1.032 1.051

POPET Naive<POPET, Pythia> MAB<POPET, Pythia>

1.0

0.9

0.7 0.6 0.5

0.8 Prefetcher-adverse Prefetcher-friendly

Overall

Prefetcher adverse

Prefetcher friendly

Overall

Figure 7.8: (a) Workload category-wise performance analysis in CD1. (b) Performance comparison with StaticBest in CD1.

Performance Comparison with StaticBest. Figure 7.8(b) compares the performance of Athena against the StaticBest combination (see §7.1.1). The key takeaway is that, by dynamically coordinating POPET and Pythia, Athena provides similar performance gains as the StaticBest combination for both prefetcher-adverse and prefetcher-friendly workload categories. On average, Athena improves performance by 10.3% over the baseline with no prefetching or OCP, whereas StaticBest improves performance by 11.1%. Effect of Athena on Main Memory Requests. Figure 7.9(a) shows the number of main memory requests issued by POPET-alone, Pythia-alone, Naive, HPAC, MAB, and Athena across all 100 workloads in CD1. We make two key observations. First, in prefetcher-adverse workloads, Pythia-alone significantly increases the main memory requests due to its poor prefetch accuracy. Naively combining POPET with Pythia further increases the main memory requests (by 46.5% on top of the baseline system without any prefetcher or OCP). This overhead in main memory requests severely harms the overall performance (see §7.6.1). Athena, by dynamically coordinating POPET and Pythia, substantially reduces the main memory request overhead (only 5.9% over the baseline). Second, across all workloads, Naive, HPAC, and MAB increase the main memory requests by 21.9%, 15.2%, and 12.7%, respectively, whereas Athena increases them by only 5.8% over the baseline without any prefetcher or OCP. Effect of Athena on LLC Load Miss Latency. Figure 7.9(b) shows the average last-level cache (LLC) load miss latency, normalized to the baseline system without prefetching or OCP,

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

110

Main memory requests normalized to no prefetching and no OCP

POPET HPAC<POPET, Pythia> 1.5 1.4

Pythia MAB<POPET, Pythia>

(a)

1.3 1.2 1.1 1.0 0.9 0.8 SPEC

PARSEC

Ligra

POPET HPAC<POPET, Pythia> Avg. LLC load miss latency normalized to no prefetching and no OCP

Naive<POPET, Pythia> Athena<POPET, Pythia>

1.8 1.6

CVP

Pythia MAB<POPET, Pythia>

Prefetcher- Prefetcheradverse friendly

Overall

Naive<POPET, Pythia> Athena<POPET, Pythia>

(b)

1.4 1.2 1.0 0.8 0.6 SPEC

PARSEC

Ligra

CVP

Prefetcher- Prefetcheradverse friendly

Overall

Figure 7.9: Comparison of (a) the number of main memory requests and (b) average last-level cache (LLC) load miss latency.

across all 100 workloads. We make three key observations. First, naively enabling both Pythia and POPET increases average LLC miss latency, particularly for prefetcher-adverse workloads, indicating that uncoordinated speculation can exacerbate memory-system contention and interference. Second, HPAC and MAB partially reduce this latency overhead, but still fall short of the baseline. Third, Athena consistently reduces the LLC miss latency overhead across all workload categories by dynamically coordinating Pythia and POPET. Overall, Naive, HPAC, and MAB increase the average LLC load miss latency by 28.3%, 16.7%, and 7.3%, respectively, whereas Athena increases it by only 1.7% over the baseline without any prefetcher or OCP. CD2: OCP with One L1D Prefetcher Figure 7.10 shows the performance improvement of Naive, TLP, HPAC, MAB, and Athena when coordinating POPET as the OCP and IPCP as the L1D prefetcher. We make two observations. First, TLP outperforms Naive by 5.5% in prefetcher-adverse workloads by filtering out prefetches that are predicted to go off-chip. However, this filtering strategy hurts performance in prefetcher-friendly workloads where prefetch requests are indeed helpful, causing TLP to underperform Naive by 12.0%. Second, Athena, by dynamically learning using multiple system-level features, outperforms TLP in both prefetcher-adverse and prefetcher-friendly

7.6: ATHENA: EVALUATION

111

workloads by 6.5% and 10.4%, respectively, highlighting its robust and consistent performance improvements. Overall, Athena outperforms Naive, TLP, HPAC, and MAB on average by 4.5%, 8.7%, 8.4%, and 5.2%, respectively.

Geomean speedup over no prefetching and no OCP

POPET TLP<POPET, IPCP> Athena<POPET, IPCP>

IPCP HPAC<POPET, IPCP>

Naive<POPET, IPCP> MAB<POPET, IPCP>

1.8 1.6 1.4 1.2 1.0 0.8 SPEC

PARSEC

Ligra

CVP

Prefetcher adverse

Prefetcher friendly

Overall

Figure 7.10: Speedup in cache design 2 (CD2).

CD3: OCP with Two L2C Prefetchers

Geomean speedup over no prefetching and no OCP

Figure 7.11 shows the performance improvement of Naive, HPAC, MAB, and Athena when coordinating POPET as the OCP, along with SMS and Pythia as two L2C prefetchers. We observe that, in prefetcher-adverse workloads, HPAC and MAB only partially alleviate Naive’s performance degradation, failing to match the baseline with no prefetching or OCP. Athena, in contrast, achieves a 3.2% improvement over the baseline, surpassing POPET’s standalone performance. In prefetcher-friendly workloads, Athena matches Naive’s performance. Overall, Athena outperforms Naive, HPAC, and MAB on average by 10.1%, 10.4%, and 6.4%, respectively, underscoring Athena’s effectiveness irrespective of the cache design. POPET

SMS+Pythia

Naive<POPET, SMS+Pythia>

HPAC<POPET, SMS+Pythia>

MAB<POPET, SMS+Pythia>

Athena<POPET, SMS+Pythia>

1.6 1.4 1.2 1.0 0.8 0.6 SPEC

PARSEC

Ligra

CVP

Prefetcher adverse

Figure 7.11: Speedup in cache design 3 (CD3).

Prefetcher friendly

Overall

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

112

CD4: OCP with One L1D and One L2C Prefetcher

Geomean speedup over no prefetching and no OCP

Figure 7.12 shows the performance improvement of Naive, TLP, HPAC, MAB, and Athena when coordinating POPET as the OCP, IPCP as the L1D prefetcher, and Pythia as the L2C prefetcher. We make two key observations. First, in prefetcher-adverse workloads, enabling both prefetchers and the OCP without coordination results in a severe 26.8% performance degradation, the worst among all evaluated cache designs. TLP, due to its lack of control over the L2C prefetcher, fails to throttle harmful L2C prefetch requests, resulting in a performance degradation of 16.7%. Athena, benefiting from its flexibility, effectively coordinates prefetchers across two cache levels, significantly outperforming TLP by 19.9%. Second, in prefetcherfriendly workloads, TLP closely matches Naive’s performance since it inherently has no control over the L2C prefetcher. In contrast, Athena dynamically determines that prefetching is beneficial, thereby providing higher performance compared to TLP. Overall, Athena outperforms Naive, TLP, HPAC, and MAB on average by 14.9%, 9.9%, 10.3%, and 7.0%, respectively.

1.6

POPET TLP<POPET, IPCP>+Pythia Athena<POPET, IPCP, Pythia>

IPCP+Pythia HPAC<POPET, IPCP, Pythia>

Naive<POPET, IPCP, Pythia> MAB<POPET, IPCP, Pythia>

1.4 1.2 1.0 0.8 0.6 0.4 SPEC

PARSEC

Ligra

CVP

Prefetcher adverse

Prefetcher friendly

Overall

Figure 7.12: Speedup in cache design 4 (CD4).

Based on our extensive evaluation, we conclude that Athena consistently outperforms prior coordination mechanisms (e.g., TLP, HPAC, and MAB) across diverse cache designs that employ OCP with multiple prefetchers at different cache levels.

7.6.2

Performance Sensitivity Analysis in CD1

While §7.6.1 demonstrates the benefits of Athena across various cache designs, this section further demonstrates Athena’s adaptability by fixing the cache design to CD1 and varying the underlying L2C prefetcher and OCP type. Effect of L2C Prefetcher Type Figure 7.13 shows the performance improvement of Naive, HPAC, MAB, and Athena across all workloads while using POPET as the OCP, and varying the underlying L2C prefetcher. The key observation is that, by autonomously learning using system-level features and telemetry information, Athena consistently outperforms Naive, HPAC, and MAB for every prefetcher

7.6: ATHENA: EVALUATION

113

type, without requiring any changes to its configuration. On average, Athena outperforms the next-best-performing MAB by 5.0%, 5.4%, 3.6%, and 5.0%, when coordinating POPET with four types of L2C prefetchers, Pythia, SPP+PPF, MLOP, and SMS, respectively. We conclude that Athena is able to adapt and provide consistent performance benefits across diverse prefetcher types.

Geomean speedup over no prefetching and no OCP

Naive<POPET, Prefetcher>

HPAC<POPET, Prefetcher>

MAB<POPET, Prefetcher>

Athena<POPET, Prefetcher>

1.15 1.10 1.05 1.00 0.95 0.90 0.85 Pythia

SPP+PPF MLOP L2C prefetcher type

SMS

Figure 7.13: Performance sensitivity to underlying prefetching mechanism at L2C in CD1.

Effect of Off-Chip Predictor Type Figure 7.14 shows the performance improvement of Naive, HPAC, MAB, and Athena while using Pythia as the L2C prefetcher and varying the underlying OCP. The key takeaway is that Athena consistently outperforms Naive, HPAC, and MAB for every OCP type. On average, Athena outperforms the next-best-performing MAB by 5.0%, 4.7%, and 8.2%, when employing POPET, HMP, and TTP as the underlying OCP, respectively. We conclude that Athena is able to adapt and provide consistent performance benefits across diverse OCP types.

Geomean speedup over no prefetching and no OCP

OCP

Naive<OCP, Pythia>

HPAC<OCP, Pythia>

MAB<OCP, Pythia>

Athena<OCP, Pythia>

1.15 1.10 1.05 1.00 0.95 0.90 0.85 POPET

HMP OCP type

TTP

Figure 7.14: Performance sensitivity to off-chip prediction mechanisms in CD1.

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

114

Effect of OCP Request Issue Latency For each load request predicted to go off-chip, an OCP issues a speculative memory request (we call it an OCP request) directly to the main memory controller as soon as the physical address of the load becomes available. While an OCP request experiences substantially lower latency than a regular demand load, it still incurs a latency to traverse the on-chip network. To faithfully evaluate OCP under a wide range of on-chip network designs, we vary the latency to directly issue an OCP request to the main memory controller (we call this OCP request issue latency) from 6 to 30 cycles, in line with prior work [133]. Figure 7.15 shows the performance improvement of Naive, HPAC, MAB, and Athena across all workloads when coordinating Pythia as the L2C prefetcher, POPET as the OCP, and varying the OCP request issue latency. We make three key observations. First, POPET’s performance gains decrease by 2.5% as the OCP request issue latency increases from 6 to 30 cycles, consistent with prior work [133]. Second, although the overall benefit of OCP reduces with higher request latency, Athena’s performance decreases by only 0.8%, demonstrating robust adaptability to varying request delays. Third, Athena consistently outperforms Naive, HPAC, and MAB for all evaluated OCP request issue latencies. We conclude that Athena is able to adapt to diverse system configurations with variations in on-chip network design.

Geomean speedup over no prefetching and no OCP

POPET

Naive<POPET, Pythia>

HPAC<POPET, Pythia>

MAB<POPET, Pythia>

Athena<POPET, Pythia>

1.15 1.10 1.05 1.00 0.95 0.90 0.85 6 cycles

18 cycles OCP request issue latency

30 cycles

Figure 7.15: Performance sensitivity off-chip predicted request issue latency in CD1.

7.6.3

Performance Sensitivity Analysis in CD4

This section further demonstrates Athena’s adaptability in CD4, which employs one prefetcher each at L1D and L2C, by varying the L1D prefetcher type and main memory bandwidth. Effect of L1D Prefetcher Type Figure 7.16 shows the performance improvement of Naive, TLP, HPAC, MAB, and Athena across all workloads while varying the underlying L1D prefetcher, but keeping POPET as the OCP and Pythia as the L2C prefetcher. We make two key observations. First, Berti, due to its

7.6: ATHENA: EVALUATION

115

Geomean speedup over no prefetching and no OCP

higher prefetch accuracy, provides a higher performance gain than IPCP. Berti improves performance by 4.3% on average over the baseline without any prefetcher or OCP, whereas IPCP degrades performance by 3.1%. Second, Athena consistently improves performance, both over the baseline and prior coordination techniques, irrespective of the L1D prefetcher type. Athena outperforms the next-best-performing MAB by 7.0% and 5.0% on average, while coordinating IPCP and Berti at L1D, respectively. L1D Prefetcher + Pythia

Naive<POPET, L1D Prefetcher, Pythia>

TLP<POPET, L1D Prefetcher> + Pythia

HPAC<POPET, L1D Prefetcher, Pythia>

MAB<POPET, L1D Prefetcher, Pythia>

Athena<POPET, L1D Prefetcher, Pythia>

1.2 1.1 1.0 0.9 0.8 0.7 IPCP

L1D prefetcher type

Berti

Figure 7.16: Performance sensitivity to prefetching mechanism at L1D in CD4.

Effect of Main Memory Bandwidth

Geomean speedup over no prefetching and no OCP

Figure 7.17 shows the performance improvement of Naive, TLP, HPAC, MAB, and Athena over the baseline with no prefetcher or OCP across all workloads, while varying the main memory bandwidth (measured in gigabytes per second (GB/s)).

1.5

POPET Naive<POPET, IPCP, Pythia> HPAC<POPET, IPCP, Pythia> Athena<POPET, IPCP, Pythia>

IPCP + Pythia TLP<POPET, IPCP> + Pythia MAB<POPET, IPCP, Pythia>

1.3 1.0 0.8 0.5 1.6 GB/s

3.2 GB/s 6.4 GB/s Main memory bandwidth

12.8 GB/s

Figure 7.17: Performance sensitivity to main memory bandwidth in CD4.

We make three key observations. First, while Naive significantly improves performance over the baseline in a system with ample main memory bandwidth, Naive’s performance gain,

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

116

which is largely dominated by the prefetchers, significantly deteriorates when the system has limited main memory bandwidth, akin to datacenter-class processors [66–69, 169]. For example, Naive improves performance by 33.5% on average in the system with 12.8 GB/s main memory bandwidth, while it degrades performance by 18.9% in the system with 1.6 GB/s main memory bandwidth. Second, OCP alone also hurts performance in severely bandwidth-limited configurations, despite its highly accurate predictions. POPET degrades performance by 1.1% on average over the baseline in the system with 1.6 GB/s main memory bandwidth. This indicates that neither prefetching nor off-chip prediction is beneficial for performance in severely bandwidth-constrained configurations. Third, by autonomously learning using system-level features such as bandwidth usage, Athena consistently outperforms Naive, TLP, HPAC, and MAB across all bandwidth configurations. Athena’s benefit is more prominent in bandwidthconstrained configurations since no static combination (i.e., POPET-alone, Pythia-alone, naive combination of POPET and Pythia, or none) is consistently good across all phases of all workloads. However, in a system with ample bandwidth, Naive often yields good performance, and Athena correctly identifies this combination as the best-performing. Overall, Athena outperforms Naive (MAB) by 21.4% (8.8%) and 1.7% (11.6%) in 1.6 GB/s and 12.8 GB/s bandwidth configurations, respectively.

7.6.4

Multi-Core Evaluation Overview

Four-Core Performance Analysis Figure 7.18 shows the performance improvement of Naive, HPAC, MAB, and Athena when coordinating POPET as the OCP and Pythia as the L2C prefetcher in four-core workloads.

Geomean speedup over no prefetching and no OCP

POPET HPAC<POPET, Pythia>

Pythia MAB<POPET, Pythia>

Naive<POPET, Pythia> Athena<POPET, Pythia>

1.3 1.2 1.1 1.0 0.9 0.8 Prefetcher-adverse mix

Preftcher-friendly mix

Random mix

Overall

Figure 7.18: Speedup in four-core workloads.

We make three key observations. First, across all workload mixes, Athena outperforms Naive, HPAC, and MAB by 5.3%, 7.7%, and 3.0%, respectively, despite utilizing hyperparameters exclusively tuned for single-core workloads (i.e., hyperparameters derived from the automated DSE described in §7.4.3 are applied directly without alteration). Second, Athena consistently

7.6: ATHENA: EVALUATION

117

outperforms all prior coordination mechanisms in every workload mix category. Athena’s performance gains over Naive, HPAC, and MAB are largest in prefetcher-adverse mixes, where no static configuration (i.e., POPET-alone, Pythia-alone, naive combination of POPET and Pythia, or none) is consistently good across all workload mixes. In contrast, Athena’s relative benefit is smallest in prefetcher-friendly mixes, where the Naive combination is often beneficial for performance. More specifically, Athena outperforms Naive (MAB) by 10.1% (5.5%), 0.4% (1.2%), and 4.5% (1.9%) on average in prefetcher-adverse, prefetcher-friendly, and random workload mixes, respectively. These results highlight Athena’s ability to adaptively select effective coordination policies across diverse multi-core workload compositions. Eight-Core Performance Analysis

Geomean speedup over no prefetching and no OCP

Figure 7.19 shows the performance improvement of Naive, HPAC, MAB, and Athena when coordinating POPET as the OCP and Pythia as the L2C prefetcher in eight-core workloads. We highlight two key observations. First, similar to four-core mixes, Athena consistently outperforms Naive, HPAC, and MAB by 9.7%, 9.6%, and 4.3%, respectively, across all eight-core mixes, despite using hyperparameters that are exclusively tuned for single-core workloads. Second, Athena consistently outperforms all prior coordination mechanisms in every workload mix category. POPET

Pythia

Naive<POPET, Pythia>

HPAC<POPET, Pythia>

MAB<POPET, Pythia>

Athena<POPET, Pythia>

1.2 1.1 1.0 0.9 0.8 0.7 Prefetcher-adverse mix

Prefetcher-friendly mix

Random mix

Overall

Figure 7.19: Speedup in eight-core workloads.

The multi-core results demonstrate that Athena surpasses existing coordination policies in multi-core settings, without requiring workload-specific tuning. We believe Athena could improve even further by tuning it specifically for multi-core.

7.6.5

Understanding Athena

Understanding Athena’s Decision-Making using a Case Study To provide deeper insights into Athena’s decision-making process, we analyze its actions in coordinating POPET as the OCP and Pythia as the L2C prefetcher in a representative work-

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

118

35% Enable none

Enable POPET

Enable Pythia

Enable both

61%

Speedup over

47%

22%

Enable none

Enable POPET

Enable Pythia

Enable both

no prefetching and no OCP

Speedup over

(c)

14%

no prefetching and no OCP

With 3.2 GB/s main memory bandwidth

(a)

With 25.6 GB/s main memory bandwidth

load, compute_fp_78, from the CVP suite. As Figure 7.20(a) shows, in the system with 3.2 GB/s main memory bandwidth, Athena disables both POPET and Pythia, or enables only POPET in 47% and 35% of its actions, respectively. It enables only Pythia or both mechanisms in only 14% and 4% of its actions. To find the rationale behind this action distribution, we independently evaluate the performance of three static combinations: POPET-alone, Pythia-alone, and Naive, for this workload. Figure 7.20(b) shows both Pythia-alone and Naive substantially degrade performance in the 3.2 GB/s bandwidth configuration. While POPET-alone also incurs performance degradation, it does so less severely. By selectively enabling POPET in specific epochs and disabling both mechanisms in most others, Athena effectively outperforms all three static combinations. When we evaluate the same workload in the system with 25.6 GB/s main memory bandwidth, we observe that the action distribution significantly changes, and Athena strongly favors enabling both Pythia and POPET. As Figure 7.20(c) shows, Athena enables both Pythia and POPET in 61% of its actions. The performance graph in Figure 7.20(d) shows that, unlike in the 3.2 GB/s configuration, both Pythia-alone and Naive significantly improve performance. We conclude that Athena is not only dynamically learning to find the best coordination, but also adapting to system configuration changes. 1.4

(b)

1.2

1.074 0.970

1.0

0.792

0.8 0.6

1.4 1.2

POPET-alone

(d) 1.154

Pythia-alone

1.213

0.765

Naive <POPET, Athena <POPET, Pythia> Pythia>

1.265

1.260

1.0 0.8 0.6

POPET-alone

Pythia-alone

Naive <POPET, Athena <POPET, Pythia>

Pythia>

Figure 7.20: Distribution of Athena’s action in coordinating Pythia and POPET and speedup of different Pythia-POPET combinations in compute_fp_78 workload from CVP suite, while varying memory bandwidth: 3.2 GB/s and 25.6 GB/s.

7.6: ATHENA: EVALUATION

119

Understanding the Sources of Athena’s Performance Gains via Ablation Study To better understand the source of Athena’s performance gains, we conduct an ablation study to evaluate the contribution of each state feature and reward component to overall performance. We begin with a version of Athena that uses no state information and employs only IPC as the correlated reward (see 7.3.3). We call this configuration Stateless Athena. We then progressively introduce each state feature (i.e., prefetcher accuracy, OCP accuracy, bandwidth utilization, and prefetch-induced cache pollution), and finally include the uncorrelated reward (see 7.3.3). Figure 7.21 illustrates the effect of each state feature and reward component on Athena’s geomean performance. Each bar represents Athena’s performance up to and including the added state feature or reward component.

Geomean speedup over no prefetching and no OCP

Stateless Athena (SA) OCP Accuracy (OA) Prefetch-Induced Cache Pollution (CP)

Prefetcher Accuracy (PA) Bandwidth Usage (BW) Uncorrelated Reward

1.12 1.10 1.08 1.06 1.04

MAB <POPET, Pythia>

SA+PA

SA+PA+OA

SA+PA+OA +BW

SA+PA+OA Athena +BW+CP <POPET, Pythia>

Figure 7.21: Contribution of individual state features and reward components to Athena’s geomean performance across all 100 workloads.

We make three key observations from Figure 7.21. First, the stateless Athena, which operates without any state information, performs slightly worse than MAB, consistent with prior work [319]. This result stems from the difference in exploration strategies: MAB employs Discounted Upper Confidence Bound (DUCB), whereas Athena uses 𝜖-greedy. In the stateless configuration, 𝜖-greedy selects random actions uniformly, with a non-decaying exploration rate, leading to slightly lower efficiency. Second, incorporating prefetcher accuracy, OCP accuracy, bandwidth utilization, and prefetch-induced cache pollution progressively improves performance by 1.4%, 1.7%, 0.8%, and 0.1%, respectively, relative to the preceding configuration. However, as discussed in §7.4.3, adding additional features beyond prefetch-induced cache pollution yields diminishing returns. Hence, we limit Athena’s state to four features. Finally, adding the uncorrelated reward further improves performance by 1.0%, highlighting that the uncorrelated reward component significantly helps improve performance by isolating the true impact of Athena’s actions from inherent variations in the workload.

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

120

7.6.6

Athena for Prefetcher-Only Management

To evaluate the generality of Athena and its applicability beyond OCP-enabled systems, we conduct a generalizability study by comparing Athena against HPAC and MAB in a cache hierarchy without an OCP. Specifically, we evaluate Athena using a configuration that employs SMS and Pythia at the L2C (similar to CD3 in §7.6.1, but without OCP). Figure 7.22 shows the geomean performance of Naive, HPAC, MAB, and Athena, when coordinating SMS and Pythia as the L2C prefetchers. SMS + Pythia

HPAC<SMS, Pythia>

MAB<SMS, Pythia>

Athena<SMS, Pythia>

Geomean speedup over no prefetching

1.6 1.4 1.2 1 0.8 0.6 SPEC

PARSEC

Ligra

CVP

Prefetcher- Prefetcheradverse friendly

Overall

Figure 7.22: Geomean performance of Athena coordinating two L2C prefetchers without OCP.

We make two key observations. First, in prefetcher-adverse workloads, both HPAC and MAB fail to adequately throttle prefetching, leading to performance degradation below the baseline with no prefetching. In contrast, Athena effectively mitigates these slowdowns, maintaining performance close to the baseline. However, in the absence of an OCP, which could otherwise act as a complementary mechanism, Athena only prevents performance loss as opposed to improving performance, as observed in §7.6.1. Second, in prefetcher-friendly workloads, Athena consistently outperforms HPAC and MAB by 5.1% and 7.8%, respectively. Overall, Athena achieves 7.6% and 8.8% higher performance than HPAC and MAB, respectively. We conclude that Athena generalizes across system configurations with multiple prefetchers and maintains its adaptability even when the alternative mechanism, OCP, is absent.

7.6.7

Performance Evaluation using DPC4 Traces

Figure 7.23(a) shows the geomean performance improvement of Naive, HPAC, MAB, and Athena when coordinating POPET as the OCP and Pythia as the L2C prefetcher across all 483 DPC4 workloads. The key observation is that while Athena provides the best performance gain among three coordination policies considered in this work, Athena still underperforms the naive combination of POPET and Pythia. This underperformance is primarily due to the suboptimal coordination actions taken during the state-action space exploration by Athena (see §3.1). On average, Athena improves performance by 3.1% on average over the baseline without any prefetcher or OCP, whereas Naive, HPAC, and MAB improve performance by 4.4%, 1.8%, and 1.9%.

7.6: ATHENA: EVALUATION

121

Figure 7.23(b) shows the geomean performance improvement of Naive, TLP, HPAC, MAB, and Athena when coordinating POPET as the OCP and IPCP as the L1D prefetcher across all DPC4 workloads. Unlike CD1, in this case, Athena provides better performance gains than Naive, HPAC, and MAB, but falls short to TLP. On average, Athena improves performance by 0.4% on average over the baseline without any prefetcher or OCP, whereas Naive degrades performance by 0.2% and TLP, HPAC, and MAB improve performance by 1.2%, 0.1%, and 0.01%. POPET

Pythia

Naive<POPET, Pythia>

HPAC<POPET, Pythia>

MAB<POPET, Pythia>

Athena<POPET, Pythia> 1.05

(a)

Geomean speedup over no prefetching and no OCP

1.10

1.05

1.00

POPET TLP<POPET, IPCP> Athena<POPET, IPCP>

IPCP HPAC<POPET, IPCP>

Naive<POPET, IPCP> MAB<POPET, IPCP>

(b) 1.00

0.95

0.95 AIML

GMS

Google

AIML

GEOMEAN

GMS

Google

GEOMEAN

Figure 7.23: Speedup in (a) CD1 and (b) CD2 across 483 DPC4 traces.

Figure 7.24(a) shows the geomean performance improvement of Naive, HPAC, MAB, and Athena when coordinating POPET as the OCP, along with SMS and Pythia as two L2C prefetchers across all DPC4 workloads. Similar to CD1, here Athena provides the best performance gain among three coordination policies considered in this work, yet underperforms the naive combination. On average, Athena improves performance by 3.0% on average over the baseline without any prefetcher or OCP, whereas Naive, HPAC, and MAB improve performance by 4.0%, 1.8%, and 2.0%. Figure 7.24(b) shows the geomean performance improvement of Naive, HPAC, MAB, and Athena when coordinating POPET as the OCP, IPCP as the L1D prefetcher, and Pythia as the L2C prefetcher across all DPC4 workloads. Unlike all other cache designs, here in CD4 Athena provides the best performance gain among all coordination policies. On average, Athena improves performance by 1.9% on average over the baseline without any prefetcher or OCP, whereas Naive and HPAC degrade performance by 1.5% and 0.9%, and TLP and MAB improve performance by 1.7% and 0.2%, respectively. POPET

SMS+Pythia

Naive<POPET, SMS+Pythia>

HPAC<POPET, SMS+Pythia>

MAB<POPET, SMS+Pythia>

Athena<POPET, SMS+Pythia>

POPET Naive<POPET, IPCP, Pythia> HPAC<POPET, IPCP, Pythia> Athena<POPET, IPCP, Pythia> 1.10

(a)

Geomean speedup over no prefetching and no OCP

1.10

IPCP+Pythia TLP<POPET, IPCP>+Pythia MAB<POPET, IPCP, Pythia>

1.05

1.00

0.95

(b) 1.05

1.00

0.95

0.90 AIML

GMS

Google

GEOMEAN

AIML

GMS

Google

GEOMEAN

Figure 7.24: Speedup in (a) CD3 and (b) CD4 across 483 DPC4 traces.

Overall, these results suggest that while Athena is able to outperform prior coordination mechanisms in many cache designs without any additional fine-tuning for DPC4 traces, it can still be improved further for higher performance benefit.

122

7.7

CHAPTER 7: PREFETCHER-OCP COORDINATION VIA REINFORCEMENT LEARNING

Summary

We introduce Athena, a reinforcement learning (RL)-based policy that coordinates data prefetchers and off-chip predictor (OCP) by autonomously learning from system behavior. Athena measures multiple system-level metrics (e.g., prefetcher/OCP accuracy, memory bandwidth usage) and uses them as state information to take an action: enabling the prefetcher and/or OCP and adjusting the prefetcher aggressiveness. Athena introduces a holistic reward framework that disentangles events correlated to its own actions (e.g., improvement in IPC) from the events that are uncorrelated to its actions (e.g., change in mispredicted branch instructions). This allows Athena to autonomously learn a coordination policy by isolating the true impact of its actions from inherent variations in the workload. Our extensive evaluation shows that Athena consistently outperforms a naive prefetcher-OCP combination, heuristicbased HPAC, and learning-based TLP and MAB across a wide range of system configurations with various combinations of underlying prefetchers at various cache levels, OCPs, and main memory bandwidth, while incurring only modest storage overhead.

7.7.1

Influence on the Research Community

Athena has been presented at the 32nd International Symposium on High-Performance Computer Architecture (HPCA) on February, 2026 [138]. Athena has been officially artifact evaluated with all three badges (i.e., available, functional, and reproducible) and has been recognized with the Distinguished Artifact Award at HPCA 2026 [48]. We have made Athena freelydownloadable from our GitHub repository [22] with all evaluated workload traces, scripts, and implementation code required to reproduce and extend it. We hope that Athena and its novel reward policy would influence future works on datadriven coordination policy design. Such policies would not only improve system performance and efficiency under a wide range of configurations, but would also reduce an architect’s burden in designing sophisticated control policies.

Chapter 8

Improving Performance and Power Efficiency by Safely Eliminating Load Instruction Execution In the preceding chapters, we demonstrated how data-driven prefetching (Pythia), off-chip prediction (Hermes), and their synergistic orchestration (Athena) can effectively hide long memory access latency. In this chapter, we shift our focus from latency-hiding mechanisms to latency-tolerance mechanisms employed within the processor core. We quantitatively demonstrate how conventional latency-tolerance mechanism often fail to realize their potential due to their inability to fully exploit the underlying data characteristics. We introduce a new dataaware technique that exploits the repetitive characteristics of load instructions, thereby unlocking performance and power efficiency benefit beyond what state-of-the-art mechanisms provide.

8.1

Brief Background

Extracting high instruction-level parallelism (ILP) [298, 685] is essential in providing high single-thread and multi-thread performance in modern processors [364, 417, 814]. Unfortunately, ILP often gets limited by data dependence and resource dependence between instructions [106, 429, 649, 785, 831].1 Data dependence limits ILP due to data flow (communications) between instructions, whereas resource dependence (also called structural dependence) limits ILP due to contention for limited hardware resources in the system (e.g., execution unit, load port). Load instructions are a major source of ILP limitation in modern workloads due to both data and resource dependence [106]. Load instructions typically have longer latency than most non-memory instructions since they perform multiple component operations (i.e., address computation and data fetch) in a single instruction. This exacerbates stalls due to load data dependence, thus limiting ILP. Load instructions also use several hard-to-scale pipeline resources (e.g., reservation station (RS) entry, ports to access address generation unit (AGU) and L1 data cache), which often cause resource dependence in the pipeline, thus limiting ILP. Researchers have proposed numerous techniques to mitigate load data dependence by tol1 ILP also gets limited by frequent control dependence [649, 650, 831], which is outside the scope of this work.

123

124

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

erating load instruction latency. Load Value Prediction (LVP) and Memory Renaming (MRN) are two such key techniques that mitigate load data dependence by speculatively executing load data-dependent instructions using a predicted load value. Chapter 2.2.3 provides an extensive literature review of LVP and MRN.

8.2

Motivation and Goal

Even though LVP and MRN provide performance benefit by breaking load data dependence, the predicted load gets executed nonetheless to verify the speculated load value, which takes scarce and hard-to-scale hardware resources that otherwise could have been utilized for executing other load instructions. In other words, LVP and MRN provide performance benefits by mitigating load data dependence, but they do not mitigate load resource dependence. To illustrate how LVP provides performance benefit by mitigating data dependence,2 yet the benefit may get limited by resource dependence, Figure 8.1(a) and (b) show the execution timeline of a code example in a processor without and with LVP, respectively. For simplicity, we assume that the OOO processor has fetch, issue, and retire bandwidth of two instructions, and one load execution unit (comprised of an AGU and a load port). We also assume a perfect LVP. As Figure 8.1(a) shows, 𝐼 1 gets issued to the load execution unit in cycle-5, thus stalling 𝐼 2 . In cycle-6, an older load instruction 𝐼𝑥 (not shown in the figure) becomes ready to execute and gets issued to the load execution unit, thus stalling 𝐼 2 even further. Stalls like these, where a load instruction gets delayed due to limited hardware resources (i.e., resource dependence), frequently occur in a modern high-performance processor with deeper and wider pipeline, as we quantitatively show in §8.3.4. These stalls get exacerbated further in the presence of performance-enhancement techniques like simultaneous multithreading (SMT) [271], where a hardware resource may get shared across SMT threads. When LVP is employed, as shown in Figure 8.1(b), both loads 𝐼 1 and 𝐼 2 get value-predicted and the data-dependent instruction 𝐼 3 retires 4 cycles earlier than in the processor without LVP. However, since both 𝐼 1 and 𝐼 2 need to get executed to verify their respective predicted values, 𝐼 2 still experiences stalls in cycle-5 and 6 due to resource dependence. If we can safely eliminate the execution of 𝐼 1 while breaking its data dependence, as shown in Figure 8.1(c), we can enable 𝐼 2 to get issued to the load execution unit in cycle-5, which provides an additional 2 cycles savings on top of the processor with LVP.3 We conclude that LVP and MRN may improve performance by mitigating load data dependence, but they leave performance improvement opportunity by not mitigating load resource dependence. 2 Since LVP and MRN work on conceptually similar principles, we use LVP for this discussion without loss of generality. 3 Similarly, 𝐼 can potentially be eliminated as well, providing further savings in execution time (not shown in 2 Figure 8.1).

8.3: CONSTABLE: PERFORMANCE HEADROOM

Fetch

Decode

Rename

Allocate 1

(A) Without LVP

125

Issue 5

Execute 8

Memory 13

Writeback 18

Retire

Bubble as I1 takes the load execution unit I1: mov

rcx, [rsp+0x14]

I2: mov

rdx, [r11+rax*8]

I3: imul rcx,rdx

Another bubble as an older load Ix (not shown here) becomes ready to execute and takes the load execution unit

X X X X X X X X X X

(B) With LVP

Value predicted load I1: mov

rcx, [rsp+0x14]

I2: mov

rdx, [r11+rax*8]

I3: imul rcx,rdx

These bubbles remain as value-predicted load get executed nonetheless

X X

4 cycles saved

X X X X

Eliminated load I1: mov

rcx, [rsp+0x14]

I2: mov

rdx, [r11+rax*8]

I3: imul rcx,rdx

Value predicted load

Eliminating I1 allows I2 to get the load execution unit, saving the bubbles X X

2 more cycles saved

Figure 8.1: Execution timeline of a code example in a processor (a) without a load value predictor (LVP), (b) with LVP, and (c) with LVP and load elimination.

8.2.1

Our Goal

Our goal in this work is to improve ILP by mitigating both load data dependence and resource dependence. To this end, we propose a lightweight, purely-microarchitectural technique called Constable, which safely eliminates the entire execution of a load instruction (i.e., both load address computation and data fetch from memory hierarchy).

8.3

Performance Headroom of Constable

To understand the performance headroom of Constable, we first study the static load instructions that repeatedly fetch the same value from the same load address across the entire workload trace. We call such a load global-stable. Essentially, a global-stable load is a prime candidate for elimination since both load address computation and data fetch operations of its execution produce the same result across all dynamic instances of the instruction. We then quantify the resource dependence on global-stable loads (§8.3.4), and the performance benefit of ideally eliminating all global-stable load execution (§8.3.5).

8.3.1

Global-Stable Loads in Real Workloads

Intuitively, global-stable load instructions would be hard to find in real workloads since such instructions should already be optimized by the compiler. However, we observe that a significant fraction of load instructions in real workloads are global-stable even after aggressive compiler optimizations applied. Figure 8.2 shows the fraction of dynamic load instructions

40%

34.2%

30% 20% 10%

V G

Stack-relative

Reg-relative

80% 60% 40% 20% 0%

(b)

20%

V G

(c)

[100,250)

250+

80% 60% 40% 20% 0%

(d)

at iv e

40%

[50,100)

eg -r el

60%

[0-50)

80%

100%

St ac kre la tiv e

100%

at iv e

250+

PC -r el

[100,250)

Fraction of global-stable loads that use respective addressing mode

[50,100)

C lie nt En te rp ri FS se PE C 17 IS PE C 17 Se rv er

Fraction of global-stable loads

(a)

[0-50)

PC-relative

100%

C li En ent te rp ri FS se PE C 17 IS PE C 17 Se rv er A V G

50%

C lie nt En te rp ri FS se PE C 17 IS PE C 17 Se rv er

Fraction of all dynamic loads

Global-Stable loads

60%

Fraction of global-stable loads

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

126

Figure 8.2: (a) Fraction of dynamic loads that are global-stable. Distribution of global-stable loads by their (b) addressing mode and (c) inter-occurrence distance. (d) Distribution of interoccurrence distance of global-stable loads from each addressing mode.

that are global-stable on average across 90 workloads divided into five categories. §8.6 discusses our evaluation methodology. We make two key observations. First, 34.2% of all dynamic loads are global-stable. Second, the fraction of global-stable loads are much higher in Client, Enterprise, and Server workloads as compared to SPEC CPU 2017 workloads (i.e., ISPEC17 and FSPEC17 categories). We conclude that global-stable load instructions are relatively abundant in real workloads. Characterization of Global-Stable Loads To understand the source of the global-stable loads in workloads (e.g., accessing variables in global scope, memory accesses in a tight loop), we further characterize these loads by their addressing mode and the inter-occurrence distance (i.e., the number of instructions between two successive dynamic instances of the same global-stable load instruction). Figure 8.2(b) shows the breakdown of global-stable loads based on their addressing mode. The key takeaway is that global-stable loads use various different addressing modes. On average, 20%, 42.6%, and 37.4% of all global-stable loads use PC-relative (e.g., loads that access variables in the global scope), stack-relative (i.e., loads that access stack segment using RSP or RBP as their only source register), and register-relative (i.e., loads that use other general-purpose architectural

8.3: CONSTABLE: PERFORMANCE HEADROOM

127

registers as their source) addressing. Figure 8.2(c) shows the breakdown of global-stable loads based on their inter-occurrence distance. The key takeaway is that global-stable loads have a bimodal inter-occurrence distance distribution. 31.9% of global-stable loads reoccur within 50 instructions (e.g., loads in a tight loop) on average, whereas 31.8% loads reoccur more than 250 instructions away (e.g., accessing a global-scope variable across function calls). Figure 8.2(d) further shows the distribution of inter-occurrence distance of global-stable loads from each addressing mode. As we can see, global-stable loads that use PC-relative addressing have long inter-occurrence distance (52% of these loads have inter-occurrence distance of 250 or more instructions), whereas global-stable loads that use register-relative addressing have short interoccurrence distance (39.6% of these loads have inter-occurrence distance of less than 50 instructions). We conclude with three key takeaways. First, global-stable load instructions pose diverse characteristics, both in addressing mode and inter-occurrence distance. Second, the interoccurrence distance of global-stable loads changes significantly depending on their addressing mode. Third, an effective load elimination technique should capture elimination opportunities across both short and long inter-occurrence distances.

8.3.2

Why Do Global-Stable Loads Exist?

To understand why a compiler with aggressive optimization fails to avoid global-stable load instructions, we use a custom-made binary instrumentation tool4 to analyze the disassembly of workload binaries compiled with full optimization using a state-of-the-art off-the-shelf compiler. Figure 8.3(a) and (b) show a code example from 541.leela_r from SPEC CPU 2017 [53] benchmark suite and its disassembly, respectively. The workload is compiled using the latest GNU g++-13.2 compiler [16] at full optimization (i.e., using -O3 flag [17]) for x86-64 instruction set architecture to produce the most optimized binary. The highlighted load instruction in Figure 8.3 fetches the object pointer s_rng from memory. Since s_rng gets initialized only once at the beginning of the workload, the pointer variable effectively acts as a runtime constant and thus the highlighted load instruction is global-stable. The compiler could not eliminate this load instruction since it cannot reserve an architectural register across the global scope of the program to be reused for accessing the s_rng pointer. Figure 8.4(c) and (d) show two more examples of global-stable load instructions in a code example from 557.xz_r of SPEC CPU 2017 suite and its disassembly, compiled in the same way as the previous workload. Each highlighted load instruction accesses an argument variable to the function rc_shift_low that does not change during the function invocation. Since the function is repeatedly called using the same arguments from the same caller function throughout the workload trace, both the load instructions act as global-stable loads. However, as the function rc_shift_low gets inlined within the body of its caller function (not shown here), the compiler could not allocate architectural registers to store and reuse these variables due to register pressure [180]. 4 We call this tool Load Inspector, which is freely available at https://github.com/CMU-SAFARI/Load-Ins

pector.

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

128

Random* Random::s_rng = 0;

0000000000432620 <Random::get_Rng()>:

432620: endbr64

Random* Random::get_Rng(void)

432624: mov

rax,QWORD PTR [rip+0x1f4ac5]

; rax = Random::s_rng

{

43262b: test

rax,rax

; if(s_rng == 0)

432638 <Random::get_Rng()+0x18>

if (s_rng == 0)

43262e: je

{

432630: ret

s_rng = new Random;

432631: nop

DWORD PTR [rax+0x0]

Global-stable load accessing a global-scope variable, which is effectively a runtime constant

}

432638: sub

rsp,0x8

return s_rng;

43263c: mov

edi,0xc

432641: call

438460 <operator new(unsigned long)>

}

(a)

(b)

; new Random

Figure 8.3: Code example and disassembly from 541.leela_r of SPEC CPU 2017 suite. The highlighted load instructions are global-stable. 1

static inline bool

4134c8: mov r10d,edi

rc_shift_low(lzma_range_encoder *rc, uint8_t *out,

4134cb: mov rdi,QWORD PTR [r15]

3 4

size_t *out_pos, size_t out_size)

; rdi = *out_pos

4134ce: jmp 4134f0 <lzma_lzma_encode+0x3f0>

{

4134d0: movzx r8d,BYTE PTR [rbx+0x14]

...

4134d5: mov r14,QWORD PTR [rsp]

4134d9: add r8d,r10d

{

4134dc: mov BYTE PTR [r14+rdi*1],r8b if (*out_pos == out_size)

; r8d = rc->cache ; r14 = out

Global-stable loads accessing function arguments. Function is repeatedly called by the same caller with the same arguments

4134e0: inc rdi

return true;

out[*out_pos] = rc->cache + (uint8_t)(rc->low >> 32);

4134e6: dec QWORD PTR [rbx+0x8]

++*out_pos;

4134ea: mov BYTE PTR [rbx+0x14],0xff

rc->cache = 0xFF;

4134ee: je 413500 <lzma_lzma_encode+0x400>

; ++*out_pos

4134e3: mov QWORD PTR [r15],rdi ; --rc->cache_size ; Breaking the while loop

} while (--rc->cache_size != 0);

4134f0: cmp QWORD PTR [rsp+0x8],rdi

; if (*out_pos == out_size)

...

4134f5: jne 4134d0 <lzma_lzma_encode+0x3d0>

; do-while loop

4134f7: jmp 4133e9 <lzma_lzma_encode+0x2e9>

; return true

}

(a)

(b)

Figure 8.4: Code example and disassembly from 557.xz_r of SPEC CPU 2017 suite. The highlighted load instructions are global-stable.

We conclude that a state-of-the-art off-the-shelf compiler often fails to optimize globalstable loads due to various empirically-observed reasons, such as accessing runtime constants and local variables of inline functions, combined with a limited number of architectural registers.

8.3.3

Can Increasing Architectural Registers Eliminate Global-Stable Loads at Compile Time?

Increasing architectural registers enables a compiler to exploit additional registers to capture data reuse, which otherwise would have been reused via memory. Thus increasing architectural registers typically reduces the number of load and store instructions in a program. To understand the effect of increasing architectural registers on global-stable load instructions, we compile all C/C++-based workloads from SPEC CPU 2017 rate suite [53] without and with Intel APX extension [32], that doubles the number of architectural registers in x86-64 ISA from 16 to 32, using Clang 18.1.3 [37]. We run these workloads using the test input and profile their end-to-end execution using the Load Inspector tool (see §8.3.1) to observe (1) the reduction in dynamic loads caused by APX, and (2) the fraction of all dynamic loads that are global-stable in workloads without and with APX. Figure 8.5 shows the fraction of all dynamic loads that are global-stable in each workload without and with APX (as bar graph on the left y-axis) and the reduction in dynamic loads

8.3: CONSTABLE: PERFORMANCE HEADROOM

129

FSPEC17

ISPEC17

Overall

ISPEC17

FSPEC17

557.xz_r

541.leela_r

Reduction in dynamic loads w.r.t. baseline binary without APX

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

Reduction

531.deepsjeng_r

523.xalancbmk_r

With APX

520.omnetpp_r

500.perlbench_r

544.nab_r

538.imagick_r

526.blender_r

519.lbm_r

511.povray_r

Without APX

505.mcf_r

50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 508.namd_r

Fraction of all dynamic loads that are global-stable

with APX extension (as markers on the right y-axis) for all C/C++-based workloads from SPEC CPU 2017 suite.5

AVG

Figure 8.5: Fraction of all dynamic loads that are global-stable in workloads compiled without and with APX (on the left y-axis) and the reduction in dynamic loads by APX (on the right y-axis).

We make two key observations from Figure 8.5. First, the fraction of dynamic loads that are global-stable (i.e., the elimination opportunities for Constable) is much higher than the reduction in dynamic loads by doubling the number of architectural registers. APX reduces the number of dynamic loads by 11.7% on average. 500.perlbench_r is an outlier that observes a reduction of 98.3% of loads. Without it, APX reduces the dynamic loads by only 4.5% on average. On the other hand, 13.7% and 14.2% of all dynamic loads on average are global-stable in workloads without and with APX, respectively.6 The difference is more prominent for FSPEC17 workloads, where APX reduces dynamic loads by only 3.7%, whereas 20.8% and 21.2% of dynamic loads are global-stable in without and with APX, respectively. Second, the fraction of dynamic loads that are global-stable is nearly the same in workloads without and with APX. 500.perlbench_r and 557.xz_r are only two workloads that show more than 3% absolute change in the global-stable load fraction. This shows that the elimination opportunities for Constable is largely orthogonal to the benefits of increasing architectural registers. To further analyze the change in characteristics of global-stable loads in presence of APX, we break down the global-stable loads based on their addressing modes in workloads both without and with APX in Figure 8.6. We make two key observations from this figure. First, the fraction of stack-relative global-stable loads reduces in presence of APX. On average, 21.1% and 16% of all global-stable loads use stack-relative addressing in workloads without and with 5 502.gcc_r, 510.parest_r and 525.x264_r are omitted from this study due to failed compilation using

Clang. 6 Note that, the global-stable load fraction reported here is slightly lower than that reported in Figure 8.2(a). This is because, the earlier study in Figure 8.2(a) uses the representative sections of the workloads (see §8.6.3) to limit the simulation overhead, while this study instruments each workload end-to-end.

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

130

APX, respectively. This is expected, since increasing architectural registers predominantly reduces stack loads. Second, the fraction of PC-relative global-stable loads stays nearly the same in presence of APX (38.3% without APX as compared to 38.9% with APX). This shows that doubling architectural registers alone cannot eliminate all memory accesses to global-scope variables which are effectively runtime constant. PC-relative

Stack-relative

80% 60% 40%

508

541

ISPEC17

APX

ISPEC17

NOAPX

APX

FSPEC17

NOAPX

APX

557

NOAPX

APX

NOAPX

APX

531

NOAPX

APX

523

NOAPX

APX

520

FSPEC17

NOAPX

APX

505

NOAPX

APX

500

NOAPX

APX

544

NOAPX

APX

538

NOAPX

APX

526

NOAPX

APX

519

NOAPX

APX

511

NOAPX

APX

NOAPX

20%

NOAPX

Fraction of stable loads

100%

Overall

AVG

Figure 8.6: Distribution of global-stable loads by their addressing modes in workloads without and with APX. Each number on the x-axis corresponds to the respective workload from SPEC CPU 2017 suite.

Based on these results, we conclude that the two load elimination techniques - at compile time by increasing architectural registers and at runtime by Constable - are largely orthogonal to each other. Thus, Constable would likely be equally-performant and power-efficient in presence of increased architectural registers, as it is with the current set of architectural registers.

8.3.4

Resource Dependence on Global-Stable Loads

To quantify the loss of ILP due to resource dependence stemming from global-stable loads, we analyze the utilization of load ports. Other hardware resources that are used during a load execution (e.g., RS entry and AGU port) may also cause resource dependence, but we omit them here due to brevity. Figure 8.7(a) shows the fraction of total execution cycles where at least one load port is utilized (we call such cycles load-utilized), in our baseline processor7 augmented with a state-of-the-art load value predictor EVES [754]. As we can see, on average 32.7% of the total execution cycles are load-utilized. Figure 8.7(b) further categorizes the load-utilized cycles of each workload category based on whether or not a global-stable load utilizes a load port. As we can see, for 23.0% of all load-utilized cycles, a global-stable load takes a load port for its execution, while a non-global-stable load (i.e., a static load instruction that does not fetch the same value from the same load addresses across all dynamic instances) is waiting to be scheduled on the same port. Had the execution of global-stable loads be eliminated, the non-global-stable loads could have been scheduled faster, which in turn would provide 7 Our baseline processor has an issue width of six instructions per cycle with three AGU and three load ports,

which support a maximum throughput of three loads per cycle.

8.3: CONSTABLE: PERFORMANCE HEADROOM

131

performance benefit. Thus, we conclude that global-stable load instructions causes significant resource dependence, which can be mitigated by eliminating their execution altogether.

32.7%

30% 20% 10%

Load port is utilized by a: • Global-stable load while a non-global-stable load is waiting on the same port • Global-stable load while no non-global-stable load is waiting on the same port • Non-global-stable load

80%

60%

40%

20% 23.0%

V G

Fraction of load-utilized cycles

At least one load port is utilized 40%

C lie nt En te rp ris e FS PE C 17 IS PE C 17 Se rv er

Fraction of total execution cycles

100% 50%

0% Client

Enterprise FSPEC17

(a)

ISPEC17

Server

AVG

(b)

Figure 8.7: (a) Fraction of total execution cycles where at least one load port is utilized (we call such cycles load-utilized). (b) Categorization of load-utilized cycles based on whether or not a global-stable load utilizes a load port.

8.3.5

Performance Headroom

To measure the performance headroom of eliminating global-stable loads, we model an Ideal Constable configuration that identifies all global-stable load instructions offline and eliminates both component operations of their execution (i.e., load address computation and data fetch). We also compare Ideal Constable’s performance against three other configurations: (1) Ideal Stable LVP, where all global-stable load instructions identified offline are perfectly value predicted, and they are are also executed to verify the predictions, (2) Ideal Stable LVP with data fetch elimination, where all global-stable load instructions are perfectly value predicted, and the value-predicted loads are executed only until the end of address generation, and (3) 2× load execution width configuration, where the number of load execution units are doubled over the baseline. Figure 8.8 shows the speedup of each configuration over baseline. Ideal Stable LVP Ideal Stable LVP + data fetch elimination

1.15

2X load execution width

1.10

Ideal Constable

1.05

1.043 1.0669 1.088 1.091

Geomean speedup over the baseline

1.20

1.00 Client

Enterprise

FSPEC17

ISPEC17

Server

GEOMEAN

Figure 8.8: Speedup of Ideal Constable against Ideal Stable LVP and a processor with 2× load execution width of the baseline.

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

132

We make four key observations from Figure 8.8. First, Ideal Constable provides 9.1% performance improvement on average over the baseline. This shows that eliminating the execution of global-stable loads has high performance headroom. Second, Ideal Constable significantly outperforms Ideal Stable LVP (4.3% on average). This shows that mitigating both data and resource dependence (as done by Ideal Constable) has higher performance potential than only mitigating data dependence (as done by Ideal Stable LVP). Third, Ideal Stable LVP with data fetch elimination outperforms Ideal Stable LVP (6.7% on average), yet it falls short to the Ideal Constable. This shows that eliminating both the address computation and data fetch operations of a load execution has higher performance potential than just eliminating the data fetch. Fourth, Ideal Constable even slightly outperforms 2× load execution width configuration, which incurs significantly higher area and power overhead. We conclude that Constable has significant potential performance benefit by mitigating both load data and resource dependence.

8.4

Constable: Key Insight

Constable is based on the key insight that a dynamic instance 𝐼 2 of a static load instruction 𝐼 is bound to fetch the same value from the same memory location as the previous dynamic instance 𝐼 1 of the same static load instruction if the following two conditions are satisfied: • Condition 1: None of the source registers of 𝐼 has been written between the occurrences of 𝐼 1 and 𝐼 2 . • Condition 2: No store or snoop request has arrived to the memory address of 𝐼 1 between the occurrences of 𝐼 1 and 𝐼 2 . Satisfying Condition 1 ensures that 𝐼 2 would have the same load address as 𝐼 1 , and thus the address computation operation of 𝐼 2 can be safely eliminated. Satisfying Condition 2 ensures that 𝐼 2 would fetch the same value from the memory as 𝐼 1 , and thus the data fetch operation of 𝐼 2 can be safely eliminated. Constable exploits this observation to operate in two key steps. First, Constable dynamically identifies load instructions that have repeatedly fetched the same value from the same load address. We call such loads likely-stable.8 Second, when Constable gains enough confidence that a given load instruction is likely-stable, Constable tracks modifications to the source architectural registers of the load instruction and its memory location via two small hardware structures. Constable eliminates the execution of all future instances of the likely-stable load and breaks the load data dependence using the last-fetched value - until there is a write to the source registers or a store or snoop request to the load address. 8 Hence the name Constable, that polices the likely-stable loads to safely eliminate them [63].

8.5: CONSTABLE: MICROARCHITECTURE DESIGN

8.5

Constable: Microarchitecture Design

8.5.1

Design Overview

133

Figure 8.9 shows a high-level overview of Constable. Constable is comprised of three main hardware structures: Stable Load Detector (SLD). SLD is a program counter (PC)-indexed table that serves three key purposes. First, SLD identifies whether or not a given load instruction is likely-stable by analyzing its past dynamic instances. Second, SLD decides whether or not the execution of a load instruction can be eliminated. Third, SLD provides the last-computed load address and the last-fetched data of a given likely-stable load instruction. Register Monitor Table (RMT). RMT is an architectural-register-indexed table whose key purpose is to monitor modifications to architectural registers and avoid eliminating a load instruction when its source architectural register gets modified. Each RMT entry stores a list of load PCs that are currently getting eliminated that use the corresponding architectural register as their source. In the rename stage, every instruction looks up RMT using its destination architectural register and resets the elimination status of any load PC from the corresponding RMT entry in SLD to ensure that any future instances of that load instruction will not be eliminated. In essence, RMT enforces the Condition 1 for eliminating a load instruction (§8.4). Address Monitor Table (AMT). AMT is a physical-address-indexed table whose key purpose is to monitor modifications in the memory and avoid eliminating a load instruction when the memory location from which it fetches the data gets modified. Each AMT entry stores a list of load PCs that are currently getting eliminated that access the corresponding physical memory address. Every store or snoop request looks up AMT using its physical address and resets the elimination status of any load PC from the corresponding AMT entry in SLD to ensure that any subsequent instances of that load will not be eliminated further. In essence, AMT enforces the Condition 2 for eliminating a load instruction (§8.4).

8.5.2

Identifying Likely-Stable Loads

SLD employs a confidence-based learning mechanism to identify likely-stable load instructions based on the execution outcomes of their past dynamic instances. Each SLD entry stores four key pieces of information: (1) last-computed load address, (2) last-fetched value, (3) a 5bit stability confidence level and (4) a can_eliminate flag that represents whether or not an instance of this load instruction can be eliminated. When a non-eliminated load instruction completes execution in the writeback stage, Constable checks the SLD using the load PC to compare the last-computed load address and last-fetched value with the current load address and value. If both the address and value match, Constable increments the stability confidence level by one; otherwise, it halves the confidence. If the stability confidence level surpasses a

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

134

Stable Load Detector

Decode

Rename

Fetch

Address Monitor Table Allocate

Issue

Execute

Memory

Writeback

Retire

(a) Newly added structures in the pipeline Physical address of a store is generated

Snoop request arrives at the core

7 9 Register Monitor Table (RMT)

Insert load PC

4 into RMT

10 Address Monitor Table (AMT)

Insert load PC

5 into AMT

A likely-stable non-eliminated load completes execution

8 Reset can_eliminate >lag in SLD entry 1 New load

Look up with load PC

Stable Load Detector (SLD)

6 Set can_eliminate >lag

2 If can_eliminate >lag is set

Break load data dependence using last-fetched value and eliminate load execution

1. Mark the load “likely-stable” 3 If can_eliminate >lag is not set

if conﬁdence is above threshold 2. Execute the load

(b) Key operations in Constable

Figure 8.9: Overview of Constable.

threshold (set to 30 in our evaluation), Constable identifies subsequent load instances from the same PC as likely-stable.

8.5.3

Eliminating Load Execution

During the rename stage, a load instruction first checks the SLD using the load PC ( 1 in Figure 8.9). If the can_eliminate flag is set in the corresponding SLD entry, Constable breaks the load data dependence using the last-fetched value stored in the SLD entry and eliminates its execution ( 2 ). If the can_eliminate flag is not set, Constable checks the stability confidence level stored in the SLD entry. If the confidence level is above threshold, Constable marks the load instruction as likely-stable and executes it normally as the baseline ( 3 ). Only a load instruction marked as likely-stable can set the can_eliminate flag during the writeback stage of its execution (see §8.5.4). Microarchitecture for Breaking Load Data Dependence. Breaking load data dependence requires supplying the load value to all dependent in-flight instructions. Prior works on LVP achieve this by writing the value to the physical register file (PRF) or to a separate value table [763]. Since writing to PRF either requires adding expensive write ports to PRF [625, 665, 667] or a latency-sensitive arbitration of the existing write ports [663, 763], Constable

8.5: CONSTABLE: MICROARCHITECTURE DESIGN

135

implements load data dependence breaking using a small extra register file (only 32 entries), called xPRF, which is dedicated to hold the values of the in-flight eliminated load instructions.9 If SLD decides to eliminate the load execution, Constable stores the last-fetched value provided by SLD in an available xPRF register and converts the load instruction into a threeoperand register move instruction: the source is the xPRF register, the destination is the destination architectural register of the load, and the third operand is the last-computed load address provided by the SLD. If there is no available xPRF register, Constable does not eliminate the load and executes it normally as the baseline. We observe this happens rarely (only in 0.2% of the instances) in our evaluation with a 32-entry xPRF. In the rename stage, the converted register move instruction simply maps its destination register to the source xPRF register to complete its execution (similar to move elimination [278, 300]). Doing so enables the dependents of the converted register move instruction to get scheduled by reading the xPRF register value. In the allocation stage, the converted register move instruction allocates a reorder buffer (ROB) entry and a load buffer (LB) entry. The address field in the LB entry gets updated with the last-computed load address embedded within the move instruction as the third operand. This address field in the LB entry is later required to correctly disambiguate the eliminated load from the in-flight stores [305] as discussed in §8.5.5. Since the execution of the converted register move instruction has already been completed in the rename stage, the instruction bypasses the remaining pipeline stages and resources directly to retirement based on the in-order retirement logic.

8.5.4

Updating Constable Structures

Updates When a Likely-Stable Non-Eliminated Load Finishes Execution During the writeback stage of the pipeline, when a likely-stable yet not eliminated load finishes its execution, Constable updates its structures to eliminate subsequent instances of the same load instruction. This happens in three steps. First, Constable looks up RMT with its source architectural registers. For each source register, Constable inserts the load PC into the corresponding RMT entry ( 4 ). Second, Constable looks up AMT with the physical address of the load instruction. If the load address is found, Constable inserts the load PC into the corresponding AMT entry ( 5 ). If the address is not found, Constable inserts a new AMT entry for the load address and inserts the load PC into the new AMT entry. Third, Constable looks up SLD with the load PC and sets the can_eliminate flag of the corresponding entry ( 6 ). Setting the can_eliminate flag allows Constable to eliminate the execution of subsequent instances of the same load instruction. 9 We implement Constable using xPRF as having a small PRF to break data dependence has been shown to

be more area- and energy-efficient than adding new write ports to the existing PRF [763]. However, Constable can also be implemented by adding or arbitrating PRF write ports. For a fair evaluation, we also implement the LVP and MRN techniques considered in this work using xPRF. As such, xPRF is not considered as an additional structure for Constable.

136

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

Updates during Register Renaming In the rename stage, Constable checks the destination architectural register of every instruction and updates its structures to avoid eliminating subsequent instances of any load instruction that uses the destination register as its source. This happens in two steps. First, Constable looks up RMT with the architectural destination register of every instruction ( 7 ). If there is any load PC in the corresponding RMT entry, Constable looks up the SLD using each load PC and resets the can_eliminate flag in the corresponding entry in SLD ( 8 ). Updates on a Store Instruction When the address of a store instruction gets generated, Constable updates its structures to avoid eliminating subsequent instances of any load instruction that fetches data from the same memory address as the store. This happens in two steps. First, Constable looks up AMT using the physical store address ( 9 ). If the address is found in AMT, Constable looks up SLD using each load PC in the AMT entry and resets the can_eliminate flag from the corresponding entry in SLD ( 8 ). Second, after resetting can_eliminate flag for all load PCs in the AMT entry, Constable evicts the AMT entry. Updates on a Snoop Request To safely eliminate loads in multi-core systems, Constable monitors snoop requests coming to the core and updates its structures to avoid eliminating subsequent instances of any load instruction that fetches data from the same memory address as the snoop. Constable handles a snoop request in a similar way as a store request. When a snoop request arrives at the core, Constable looks up AMT using the snoop address ( 10 ). If the address is found, Constable looks up SLD using each load PC in the AMT entry and resets the can_eliminate flag from the corresponding entry in SLD ( 8 ). Finally, Constable evicts the AMT entry.

8.5.5

Disambiguating Eliminated Loads from In-Flight Stores

When a store instruction computes its address, Constable accesses AMT and resets the can_eliminate flag for all load instructions accessing the same memory location (see §8.5.4). This prevents Constable from eliminating any subsequent occurrences of those load instructions. However, in a processor that aggressively issues loads out-of-order [299, 305, 567], there may be eliminated loads in the pipeline that are younger than the store instruction and whose addresses match with the store address. We observe that this happens rarely (see §8.7.7 in the extended version [142]) since Constable considers a load instruction to be eligible for elimination only if it meets the stability confidence level threshold. In such infrequent cases, Constable exploits the existing memory disambiguation logic [202, 305] that matches the store address with the address of every load in the LB. If a violation is caught, Constable flushes the pipeline and re-executes all younger instructions, including the incorrectly-eliminated load (see the example in §8.5.8).

8.5: CONSTABLE: MICROARCHITECTURE DESIGN

8.5.6

137

Maintaining Coherence in Multi-Core Systems

Constable relies on monitoring snoop requests for tracking modifications in the memory by other processor cores to safely eliminate loads in a multi-core system. However, monitoring snoop requests poses the following two key challenges. Loss of Elimination Opportunity due to Clean Evictions In a multi-core system with a directory-based coherence protocol [179], when a cacheline gets evicted from a core-private cache, the core-valid bit (CV-bit) corresponding that core (i.e., the own core) gets reset in the directory entry of that cacheline [71,74,347,714]. Since resetting CVbit prevents the directory from sending any further snoop request to that cacheline to the core, on every core-private cache eviction, Constable needs to avoid eliminating any load instruction that accesses the evicted cacheline. This poses two key drawbacks. First, if the evicted cacheline is clean (e.g., eviction due to limited cache capacity or cache conflict), Constable loses elimination opportunity (we quantify the impact of such elimination opportunity loss in §8.7.8 in [142]). Second, for every core-private cache eviction, Constable needs to look up and invalidate the corresponding AMT entry, which increases design complexity. To address these drawbacks, we propose to pin the own core’s CV-bit of a cacheline that is accessed by an eliminated load instruction. When the memory request of a likely-stable yet not eliminated load returns from the cache hierarchy, Constable pins the own core’s CV-bit in the directory entry of that cacheline. Pinning CV-bit ensures that (1) the coherence protocol would send any snoop request to that cacheline to the own core, even if the cacheline gets clean-evicted from the core-private cache, and (2) Constable does not need to look up AMT on every core-private cache eviction. The CV-bit is reset as soon as a snoop request is delivered to the core, as per the normal directory-based coherence protocol. Tracking Snoop Requests at Cacheline Address Granularity Unlike a store instruction that contains a full memory address, a snoop request contains a cacheline address. Thus, to support AMT lookup using a snoop address, Constable indexes AMT using physical addresses at cacheline granularity. This may cause loss of elimination opportunities due to false address collisions (e.g., a store to a cacheline may reset the can_eliminate flag of a load instruction that accesses different bytes of the same cacheline accessed by the store). However, we find that the performance impact of such elimination opportunity loss is negligible. Constable with a cacheline-address-indexed AMT has only 0.4% lower average performance than a Constable with full-address-indexed AMT. This is primarily because the compiler tends to lay out memory addresses accessed by likely-stable load instructions together (e.g., a group of function arguments laid out in the same cacheline of stack memory segment), which reduces the overhead of false address collisions.

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

138

8.5.7

Other Design Decisions

Architecting SLD

Average number of SLD updates per cycle

3.5 3

(a)

2.5 2 1.5 1 0.5 0

t e 17 rver 17 en ris Cli terp SPEC SPEC Se I F En

Change in performance

Designing SLD with sufficient read/write ports is crucial for realizing Constable’s performance benefits. Constable reads SLD for every load instruction to identify likely-stable loads in the rename stage ( 1 in Figure 8.9). Thus SLD needs to support the read bandwidth of the expected number of load instructions in a group of instructions getting renamed together in every cycle (we call this a rename group).10 We observe that a rename group contains 1.93 loads on average across all workloads, and 98.3% of all rename groups have less than or equal to two loads. Thus, we model SLD with three read ports. If there are more than three loads in a rename group, we stall the rename stage until Constable finishes SLD lookup for every load in that group. Constable may need to update the can_eliminate flag in SLD on every RMT update, which happens for each instruction in a rename group ( 7 and 8 ). Since each RMT entry may contain a list of likely-stable load PCs, the expected number of SLD updates per cycle can vary in a large range. Figure 8.10(a) shows the average number of observed SLD updates per cycle for every workload as a box-and-whiskers plot.11 As we can see, we observe only 0.28 SLD updates per cycle on average across all workloads. 98.23% of all cycles on average across all workloads have two or fewer SLD updates. This is because, at any point in time, only a small fraction of all load PCs (14.7% on average) satisfies the stability confidence level threshold in order to be tracked by RMT entries. Thus, we model SLD with two write ports. If there are more than two SLD updates in a cycle, we stall the rename stage until Constable finishes SLD update for every load instruction in that rename group. 5% 4% 3% 2% 1% 0% -1% -2% -3%

(b)

t e 17 17 ver en ris Cli terp SPEC SPEC Ser I F En

Figure 8.10: (a) Average number of SLD updates per cycle during rename stage. (b) Change in performance when Constable’s structures are updated only by correct path instructions vs. all instructions without updating Constable’s structures on branch misprediction recovery.

10 We model a six-wide rename architecture (see §8.6.1). 11 Each box is lower- (upper-) bounded by the first (third) quartile. The box size represents the inter-quartile range (IQR). The whiskers extend to 1.5×IQR range on each side, and the cross-marked values in the box show the mean.

8.5: CONSTABLE: MICROARCHITECTURE DESIGN

139

Handling Wrong Path Execution In presence of branch prediction, Constable’s structures may get updated by wrong path instructions (especially, steps 7 and 8 in Figure 8.9). This may result in an unnecessary loss of elimination opportunity, unless the structures are restored on a branch misprediction recovery. To understand the need for restoring Constable’s structures, we measure the change in performance of Constable when its structures are updated only by the instructions on the correct path against when they are updated by all instructions without an update mechanism on branch misprediction recovery, and show it as a box-and-whiskers plot in Figure 8.10(b). The key observation is that 82 out of 90 workloads show less than 1% absolute change in performance, while the average performance change is only 0.2%. Thus, we model Constable without any update mechanism for its structures on a branch misprediction recovery. Handling Changes in Physical Address Mapping AMT monitors memory locations accessed by all eliminated load instructions in physical address space. This poses a challenge: when the physical memory mapping changes, the physical memory address tracked by an AMT entry may not be associated with the corresponding eliminated load anymore. In that case, to avoid incorrectly eliminating load execution, Constable resets the can_eliminate flag of all SLD entries and invalidates all RMT and AMT entries when the physical memory mapping changes (e.g., context switch).

8.5.8

An Illustrative Example

To put it together, Figure 8.11 illustrates an example of Constable’s operation. For this example, we consider that the loads 𝐿𝐷 1 , 𝐿𝐷 2 , and 𝐿𝐷 3 are three dynamic instances of the static load instruction 𝐿𝐷 with a PC value 𝑃𝐶𝑥 and the source registers of 𝐿𝐷 do not get modified between 𝐿𝐷 1 and 𝐿𝐷 3 . We also assume the stability confidence level threshold is set to 30. LD1 from PCX gets decoded

Likely-stable LD1 completes loading from address A

LD2 from PCX gets decoded

ST1 from PCY gets decoded

LD3 from PCX gets decoded

Look up SLD

1 Pins CV-bit in directory

Look up SLD

ST1 Glows through the pipeline to generate its store address

Look up SLD

ST1 address generated. Stores to address A. F

LD3 re-executes and completes execution G

time PC conf can_elim PCx 30 0

can_eliminate is not set but conf (30) matches the threshold (30) Mark LD1 as likely-stable and execute

Insert AMT entry Addr A

PC List PCx

PC conf can_elim PCx 31 1

can_eliminate is set

Eliminate LD2

3 Update SLD entry

PC conf can_elim PCx 30 0 PC conf can_elim PCx 31 1

Younger load completed execution. Re-execute LD3.

Look up AMT with store address A Addr A

PC List PCx

Reset can_eliminate Glag in SLD entry

Update conﬁdence in SLD entry

PC conf can_elim PCx 31 1

PC conf can_elim PCx 31 0

Invalidate AMT entry of address A

Memory disambiguation logic compares store address A with every load buLfer entry

PC conf can_elim PCx 31 0

PC conf can_elim PCx 15 0

Figure 8.11: An illustrative example of Constable’s operation.

When 𝐿𝐷 1 gets decoded ( A in Figure 8.11), Constable checks SLD and finds that the stability confidence level of 𝑃𝐶𝑥 matches the threshold, yet the can_eliminate flag is not set. In this case, Constable marks 𝐿𝐷 1 as likely-stable and executes it normally. When 𝐿𝐷 1 finishes its execution ( B ), Constable (1) pins the CV-bit corresponding to the own core in the coherence directory entry of address A, (2) updates AMT and RMT (not shown here), and (3) increments the stability confidence level in SLD. Since 𝐿𝐷 1 is marked as likely-stable, Constable sets the

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

140

can_eliminate flag in SLD entry. When 𝐿𝐷 2 gets decoded ( C ), Constable eliminates executing 𝐿𝐷 2 since the can_eliminate flag is set. Now a store 𝑆𝑇1 from a different 𝑃𝐶𝑦 gets decoded ( D ). This store instruction would ultimately modify the memory address touched by 𝐿𝐷. However, before 𝑆𝑇1 could generate its store address, 𝐿𝐷 3 , which is younger than 𝑆𝑇1 in program order, gets decoded ( E ) and Constable incorrectly eliminates its execution since the can_eliminate flag is set. When 𝑆𝑇1 finally generates its store address ( F ), Constable resets the can_eliminate flag to prevent eliminating subsequent instances of 𝐿𝐷 and evicts the AMT entry. However, the existing memory disambiguation logic probes the load buffer with the store address and finds out that a younger load 𝐿𝐷 3 has been incorrectly completed. As a result, the memory disambiguation logic aborts and re-executes 𝐿𝐷 3 (and all instructions younger than 𝐿𝐷 3 that are not shown here). When 𝐿𝐷 3 completes its execution ( G ), it halves the stability confidence level counter.

8.5.9

Storage Overhead

Table 8.1 shows the storage overhead of Constable. Constable requires only 12.4 KB storage per core of the processor (see §8.6.1). Structure

Description

SLD

• # entries: 512 (32 sets × 16 ways) • Entry size: tag (24𝑏) + addr (32𝑏) + val (64𝑏) + confidence level (5𝑏) + can_eliminate flag (1𝑏)

7.9 KB

RMT

• 16 load PCs for each stack registers (RSP and RBP) • 8 load PCs for each remaining 14 architectural registers in x86-64

0.4 KB

AMT

• # entries: 256 (32 sets × 8 ways) • Entry size: physical address tag (32𝑏) + # hashed load PCs (4 × 24𝑏)

4.0 KB

Total

Size

12.4 KB Table 8.1: Storage overhead of Constable.

8.6

Methodology

8.6.1

Performance Modeling

We evaluate Constable using an in-house, cycle-accurate, industry-grade simulator that simultaneously runs both functional and microarchitectural simulation on a workload. We faithfully model a 6-wide out-of-order processor core configured similar to the Intel Golden Cove [19, 20, 26, 44] as our baseline. Table 8.2 shows the key microarchitectural parameters. We include MRN and various dynamic optimizations in the rename stage of the baseline processor, as highlighted in bold. For a comprehensive analysis, we evaluate Constable and other competing mechanisms on the baseline system, both without SMT (called noSMT ) and with 2way SMT (called SMT2). For noSMT configuration, all hardware resources inside core are fully

8.6: CONSTABLE: EVALUATION METHODOLOGY

141

available to the single running software context. For SMT2 configuration, resources inside core (including Constable) are either statically-partitioned or dynamically-shared between both software contexts [225]. Unless stated otherwise, all reported results are from noSMT simulations. Basic Fetch & Decode

Rename Allocate Issue & Retire

Caches

Memory

x86-64 core clocked at 3.2 GHz with 2-way SMT support 8-wide fetch, TAGE/ITTAGE branch predictors [757], 20-cycle misprediction penalty, 32KB 8-way L1-I cache, 4K-entry 8-way micro-op cache, 6-wide decode, 144-entry IDQ, loop-stream detector [150] 6-wide, 288 integer, 220 512-bit and 320 256-bit physical registers, Memory Renaming [844], zero elimination [300], move elimination [278, 300], constant folding [278, 300], branch folding [251] 512-entry ROB, 240-entry LB, 112-entry SB, 248-entry RS 6-wide issue to 12 execution ports; 5, 3, 2, and 2 ports for ALU, load, store-address, and store-data execution. Port 0, 1, and 5 are used for vector instructions. Aggressive out-of-order load scheduling with memory dependence prediction [220, 569], 6-wide retire L1-D: 48KB, 12-way, 5-cycle latency, LRU, PC-based stride prefetcher [302]; L2: 2MB, 16-way, 12-cycle round-trip latency, LRU, stride + streamer [201] + SPP [465]; LLC: 3MB, 12-way, 50-cycle data round trip latency [44, 133], dead-block-aware replacement policy [452], streamer, MESIF [564] protocol 4 channels, 2 ranks/channel, 8 banks/rank, 2KB row-buffer/rank, 64b bus/channel, DDR4, tCAS=22ns, tRCD=22ns, tRP=22ns, tRAS=56ns

Table 8.2: Simulation parameters. IDQ: Instruction Decode Queue, SB: Store Buffer.

8.6.2

Power Modeling

We use an in-house, RTL-validated power model to measure the dynamic power consumption of the core. We report the overall core power consumption by breaking it down into four key units: (1) front end (FE), (2) out-of-order (OOO), (3) non-memory execution unit (EU), and (4) memory execution unit (MEU). We further break down the OOO power into three subunits: RS, Register Alias Table (RAT), and ROB. We also break down the MEU power into two sub-units: L1-D cache, and data translation look-aside buffer (DTLB). To faithfully model the power consumption of Constable, we estimate the read/write access energy and leakage power of Constable’s structures using CACTI 7.0 [123] 22nm library. We scale the estimates to 14nm technology using [803] to make the estimates compatible with our core power model. Table 8.3 shows the access energy, leakage power, and the area estimate of Constable’s structures. We report RMT and SLD power in the RAT component and AMT power in the L1-D component of the core power model.

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

142

Component

Port count

SLD RMT AMT

3R/2W 2R/6W 1R/1W

Read access energy (pJ) 10.76 0.15 1.58

Write access energy (pJ) 16.70 0.20 4.22

Leakage power (mW) 1.02 0.31 0.74

Area (mm2) 0.211 0.004 0.017

Table 8.3: Access energy, leakage power, and area estimates of Constable’s structures in 14nm technology. R: read port, W: write port.

8.6.3

Workloads

We evaluate Constable using 90 workload traces that span across a diverse set of 58 workloads. Our workload suite contains all benchmarks from the SPEC CPU 2017 suite [53], and many well-known Client, Enterprise, and Server workloads. Each trace contains a snapshot of the processor and the memory state (1) to drive both the functional and microarchitecture simulation models, and (2) to faithfully simulate wrong-path execution. Each trace is carefully selected to be representative of the overall workload. Table 8.4 summarizes the complete list of workloads. Suite

#Workloads

#Traces

Client

Enterprise

FSPEC17 ISPEC17

13 10

29 11

Server

Example Workloads DaCapo [12], SYSmark [58], TabletMark [59], JetStream2 [34] SPECjEnterprise [55], SPECjbb [54], LAMMPS [36] All from SPECrate FP 2017 [53] All from SPECrate Integer 2017 [53] Hadoop [6], Linpack [24], Snort [51], BigBench [8]

Table 8.4: Workloads used for evaluation.

8.6.4

Evaluated Mechanisms

For a comprehensive analysis, we evaluate Constable standalone and in combination with three prior works: (1) a state-of-the-art load value predictor EVES [754], (2) early load address resolution (ELAR) [131], and (3) register file prefetching (RFP) [770]. For EVES, we use the optimized implementation that won the first championship value prediction (CVP-1) in the 32 KB storage budget track [15]. For ELAR, we follow the same microarchitecture design as proposed in [131]. For RFP, we sweep and select the configuration parameter values that provide the highest performance benefit over the baseline. EVES, RFP, and Constable apply their optimizations to load instructions with data size up to 64 bits. Table 8.5 shows the overheads of all evaluated mechanisms.

8.7: CONSTABLE: EVALUATION

Mechanism EVES [754] with the same configuration as the winner of CVP-1 32 KB storage budget track [15] ELAR [131] with an additional adder in decode stage and a new copy of the ESP register RFP [770] with 2K-entry prefetch table, 64-entry page address table, and 128-entry RFP-Inflight table

143

Overhead 32 KB Adder + 64-b register

Constable (this work)

12.4 KB 15.4 KB

Table 8.5: Overhead of all evaluated mechanisms.

8.6.5

Functional Verification of Constable

Since Constable completely eliminates a load instruction in microarchitectural simulation, we cannot verify the functional correctness of Constable in the same way as LVP or MRN techniques. To functionally verify Constable, we enforce a golden check at the retirement stage of every load instruction. The golden check matches the load address and the load data from the functional simulation model with those from the microarchitectural simulation model. In case of a mismatch, the golden check aborts the simulation. We extensively verify the functional correctness of Constable using a broader set of 3400 traces and ensure that no single trace fails the simulation.

8.7

Evaluation

8.7.1

Performance Improvement Analysis

noSMT Configuration Figure 8.12 shows the geomean performance of EVES, Constable, and Constable and Ideal Constable (§8.3.5) combined with EVES normalized to the baseline for each workload category. We make three key observations. First, Constable alone improves performance by 5.1% on average over the baseline, which is similar to the performance improvement of EVES (4.7% on average) while incurring 21 × of EVES’ storage overhead. Second, when combined with EVES, Constable improves performance by 8.5% on average over the baseline, which is 3.7% higher than EVES alone. Third, when combined with EVES, Constable provides 82.9% of the performance improvement provided by Ideal Constable. Per-Workload Performance. To better understand Constable’s performance improvement, Figure 8.13 shows the performance line graph of EVES, Constable, and Constable combined with EVES for every workload. Workloads are sorted in ascending order of the performance gain of EVES over the baseline. We make three key observations. First, Constable outperforms EVES by 4.9% on average in 60 of the 90 workloads (highlighted in green). In the remaining 30 workloads (highlighted in red), EVES outperforms Constable by 9.2% on average. Second,

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

144

EVES

Constable

EVES+Constable

EVES+Ideal Constable

1.15

1.047 1.051 1.085 1.103

Geomean speedup over the baseline

1.20

1.10 1.05 1.00 Client

Enterprise

FSPEC17

ISPEC17

Server

GEOMEAN

Figure 8.12: Speedup over the baseline (noSMT).

EVES

Speedup over the baseline

1.4 1.3 1.2

Constable

1.62

Constable combined with EVES consistently outperforms both EVES and Constable alone in every workload. EVES+Constable jetstream2-richards_wasm 554.roms_r 527.cam4_r 549.fotonik3d_r

SYSMark-chrome

hadoop_kmeans

538.imagick_r

1.1 1

jetstream2-richards Constable provides higher performance than EVES

EVES provides higher performance than Constable

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88

0.9

Workload number (sorted)

Figure 8.13: Speedup of all workloads (noSMT).

Load Category-Wise Performance. To understand the performance benefits contributed by different load categories, Figure 8.14 compares the geomean performance of Constable when it eliminates only PC-relative, stack-relative, and register-relative loads with that of a fullblown Constable. The key takeaway is that each three individual types of loads contribute towards Constable’s overall performance benefit. Eliminating only PC-relative, stack-relative, and register-relative loads provide a performance improvement of 1.1%, 2.6%, and 1.8%, respectively, which nearly get added up to a 5.1% improvement by the full-blown Constable. Based on these results, we conclude that (1) Constable provides a significant performance benefit over a wide range of workloads both by itself and when combined with a state-of-theart load value predictor EVES, and (2) Constable’s performance benefit comes from eliminating all types of loads. SMT2 Configuration Figure 8.15 shows the geomean performance of EVES, Constable, and Constable combined with EVES normalized to the baseline. We make two key observations. First, unlike in noSMT

8.7: CONSTABLE: EVALUATION

PC-relative loads

Stack-relative loads

All loads 1.051

1.08 1.06

1.011 1.026 1.018

Geomean speedup over the baseline

1.10

145

1.04 1.02 1.00 Client

Enterprise

FSPEC17

ISPEC17

Server

GEOMEAN

Figure 8.14: Speedup of Constable by eliminating execution of only PC-relative, stackrelative, and register-relative loads.

configuration, Constable significantly outperforms EVES in the baseline with SMT2. Constable alone improves performance by 8.8% on average over the baseline, whereas EVES alone improves performance by 3.6%. This is because, unlike EVES, Constable’s load elimination fundamentally reduces utilization of load execution resources, which face increased contention in presence of SMT. Second, combining Constable with EVES continues to provide additional performance benefit than EVES alone. Constable with EVES improves performance by 11.3% on average over the baseline in SMT2. We conclude that Constable provides even more performance benefit in presence of SMT as compared to non-SMT system, due to high resource contention in SMT systems. EVES

EVES+Constable

FSPEC17

ISPEC17

1.088 1.113

Constable

1.15 1.10 1.036

Geomean speedup over the baseline

1.20

1.05 1.00 Client

Enterprise

Server

GEOMEAN

Figure 8.15: Speedup over the baseline (SMT2).

8.7.2

Performance Comparison with Prior Works

Figure 8.16 shows the geomean performance of Constable standalone and when combined with ELAR and RFP in the baseline. We make two key observations. First, Constable alone outperforms both ELAR and RFP. ELAR, RFP, and Constable improve performance on average by 0.74%, 4.4% and 5.1%, respectively over the baseline. ELAR provides relatively small performance benefit over baseline as our baseline already implements constant folding [278,300], which can track stack register modifications in the form of 𝑅𝑆𝑃 ← 𝑅𝑆𝑃 ± 𝑖𝑚𝑚𝑒𝑑𝑖𝑎𝑡𝑒 before the execution stage. Second, when combined with ELAR and RFP, Constable provides more

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

146

performance benefit than ELAR and RFP alone, respectively. This shows that Constable can be applied along with these proposals to provide even more performance benefit. ELAR

RFP

Constable

ELAR+Constable

RFP+Constable

1.15

1.007 1.0448 1.051 1.054 1.081

Geomean speedup over the baseline

1.20

1.10 1.05 1.00 0.95 Client

Enterprise

FSPEC17

ISPEC17

Server

GEOMEAN

Figure 8.16: Speedup of Constable over ELAR and RFP.

8.7.3

Loads Eliminated by Constable

Figure 8.17 shows the load coverage (i.e., the fraction of load instructions that are either eliminated or value-predicted by Constable or EVES, respectively) of EVES, Constable, and Constable and Ideal Constable combined with EVES in the baseline system. We make three key observations. First, Constable alone covers 23.5% of the loads, whereas EVES covers 27.3%. This is because Constable target loads which show both value and address locality (i.e., repeatedly fetching same value from same memory address), whereas EVES target loads that show only value locality. Despite its lower coverage, Constable matches performance of EVES as Constable mitigates both data dependence and resource dependence on covered loads. Second, Constable combined with EVES has higher load coverage (35.5% on average) than EVES alone. Third, when combined with EVES, Constable provides 85.4% of the coverage of Ideal Constable. We conclude that Constable covers a significant fraction of the load instructions both by itself and combined with EVES. EVES

Constable

EVES+Constable

EVES+Ideal Constable 27.3% 23.5% 35.5% 41.6%

Fraction of all loads

60% 40% 20% 0% Client

Enterprise

FSPEC17

ISPEC17

Server

Figure 8.17: Load coverage of Constable versus EVES.

AVG

8.7: CONSTABLE: EVALUATION

147

Coverage of Global-Stable Loads

150%

Global-stable and eliminated

Global-stable but not eliminated

Not global-stable but eliminated

125% 100% 75% 50% 25%

Client

Enterprise

FSPEC17

ISPEC17

Server

Reg-rel

All loads

Stack-rel

PC-rel

All loads

Reg-rel

PC-rel

Stack-rel

Reg-rel

All loads

Stack-rel

PC-rel

All loads

Reg-rel

PC-rel

Stack-rel

All loads

Reg-rel

Stack-rel

PC-rel

Reg-rel

All loads

Stack-rel

0% PC-rel

Fraction of stable load in each load category

To understand Constable’s coverage of global-stable loads (see §8.3), Figure 8.18 shows the breakdown of loads in each addressing-mode category into three classes: (1) loads that are global-stable and eliminated by Constable, (2) loads that are global-stable but not eliminated, and (3) loads that are not global-stable but eliminated. We make three key observations. First, PC-relative and register-relative global-stable loads see the highest and the lowest runtime elimination coverage of 70.2% and 33.2%, respectively. Second, Constable successfully eliminates 56.4% of all global-stable loads on average at runtime. For the remaining 43.6% globalstable loads, Constable misses their elimination opportunity due to three key reasons (not shown in the figure): (a) at least one source architectural register of a global-stable load instruction gets written between its two successive dynamic instances (for 23.3% of all global-stable loads), (b) a silent store [132, 495–497] occurs between two successive dynamic instances of a global-stable load (for 14.1% of all global-stable loads), and (c) coverage loss due to other reasons, e.g., stability confidence learning and limited hardware budget for likely-stable load tracking (for 6.2% of all global-stable loads). Third, on average, 13.5% more loads are eliminated by Constable at runtime which are not identified as global-stable. This is because these loads are not stable across the entire workload trace, but stable in a workload phase to meet the stability confidence threshold and and hence get eligible for elimination. Based on these results, we conclude that Constable eliminates a significant fraction of the global-stable loads at runtime. However many elimination opportunities are still left, which can be unlocked by future works to achieve even higher performance and power efficiency improvements.

AVG

Figure 8.18: Breakdown of eliminated and non-eliminated loads as fractions of global-stable loads.

8.7.4

Impact on Pipeline Resource Utilization

Reduction in RS Allocation Figure 8.19(a) plots the percentage reduction in RS allocations in a system with Constable over the baseline system as a box-and-whiskers plot. The key observation is that Constable reduces

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

148

40% 30% 20% 10% 0% -10% -20%

(a)

t e 17 rver 17 en ris Cli terp SPEC SPEC Se I F En

% reduction in L1-D accesses than the baseline

% reduction in RS allocations than the baseline

the RS allocation by 8.8% on average (up to 35.1%) across all workloads. Server and ISPEC17 workloads experience the highest and the lowest average RS allocation reductions of 12.8% and 1.3%, respectively. 37 of the 90 workloads experience a reduction in RS allocation by more than 10%. 100% 80% 60% 40% 20% 0% -20%

(b)

t e 17 rver 17 en ris Cli terp SPEC SPEC Se I F En

Figure 8.19: Reduction in (a) RS allocations and (b) L1-D accesses.

Reduction in L1-D Access Figure 8.19(b) plots the percentage reduction in L1-D accesses in a system with Constable over the baseline as a box-and-whiskers plot. The key observation is that Constable reduces L1-D allocation by 26.0% on average. Similar to the reduction in RS allocation, Server and ISPEC17 workloads experience the highest and the lowest average L1-D access reduction of 39.7% and 3.9%. Based on these results, we conclude that, by eliminating load execution, Constable significantly reduces RS allocations and L1-D accesses, both of which aid in improving performance (§8.7.1) and reducing dynamic power consumption (§8.7.5).

8.7.5

Power Improvement Analysis

Figure 8.20(a) shows the core power consumption (and its breakdown) in a system with EVES, Constable, and EVES+Constable normalized to the baseline. The key takeaway is that Constable reduces the core power consumption by 3.4% on average over the baseline, whereas EVES reduces power by only 0.2%. This is because, unlike EVES where the value-predicted load instructions get executed nonetheless, Constable eliminates executing likely-stable loads altogether. To understand the distribution of the power benefit across various core structures, we further expand the power consumption of OOO and MEU units in Figure 8.20(b) and (c) respectively. As Figure 8.20(b) shows, Constable reduces the power consumed by OOO unit by 4.5% on average over the baseline. The RS sub-unit of OOO unit experiences the highest power reduction of 5.1% (marked by braces). This is because Constable significantly reduces the number of RS allocations (§8.7.4). As Figure 8.20(c) shows, Constable also reduces the

8.7: CONSTABLE: EVALUATION

149

power consumed by MEU unit by 7.2% on average over the baseline. The MEU power reduction is dominated by L1-D cache, which experiences 9.1% reduction in power (marked by braces) on average. This is largely due to the reduction is L1-D accesses (§8.7.4). We conclude that Constable, unlike value prediction, reduces the core power by fundamentally eliminating load execution. RS

RAT

ROB

Others

Fraction of total OOO power

100%

60% 40% 20%

80% 60% 40% -5.1%

20%

(b)

EV ES C on EV st ES ab le + C on st ab le

EV ES

(a)

DTLB

Others

80% 60% 40% -9.1%

20% 0%

Ba se lin e

0% C on st ab le + C on st ab le

L1-D 100%

(c)

EV ES C on EV st ES ab le + C on st ab le

Others

Ba se lin e

MEU

Fraction of total MEU power

80%

Ba se lin e

Fraction of total core dynamic power

FE OOO 100%

Figure 8.20: (a) Overall core power consumption normalized to baseline. Expanded view of (b) OOO power and (c) MEU power.

8.7.6

Performance Sensitivity Analysis

Effect of Load Execution Width Scaling Figure 8.21(a) shows the geomean speedup of Constable and the baseline system over the baseline configuration when we increase the load execution width (i.e., increasing both number of AGU and load ports). We make two key observations. First, Constable consistently adds performance on top of the baseline system even if we naively scale the load execution width. With increasing AGU and load ports (while keeping the pipeline depth resources same), the resource dependence stemming from load reduces. Yet, Constable outperforms the baseline system by 3.5% with 2× load execution width than the baseline configuration. Second, adding Constable on the baseline system configuration (i.e., with 3 load execution width) essentially provides the similar performance benefit as the baseline system with one extra load execution width, while incurring lower area overhead and reducing power consumption. Effect of Pipeline Depth Scaling Figure 8.21(a) shows the geomean speedup of Constable and the baseline system over the baseline configuration when we scale pipeline depth resources (i.e., size of ROB, RS, LB and SB). The key takeaway is that Constable consistently adds performance on top of the baseline system even if we naively scale the pipeline depth. With 4× depth scaling, Constable improves performance of the baseline system by 3.4% on average.

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION Baseline system

1.14

Constable Geomean speedup over baseline processor configuration

Geomean speedup over baseline processor configuration

150

1.12 1.1 1.08

1 width savings

1.06 1.04

+5% perf

1.02 1

(a)

0.98 3

4 5 Load execution width

Baseline system

1.16

Constable

1.14 1.12 1.1 1.08 1.06 1.04 1.02 1

(b)

0.98 1

2 3 Pipeline depth scaling factor

Figure 8.21: Performance sensitivity to (a) load execution width, and (b) pipeline depth.

8.7.7

Effect of In-Flight Stores on Elimination Coverage

When the computed address of an in-flight store instruction matches with that of an eliminated load younger than the store, the existing memory disambiguation logic catches such memory ordering violation and re-executes all instructions younger than (and including) the incorrectly-eliminated load (see §8.5.5). Thus a frequent memory ordering violation by eliminated loads may incur a significant performance and power overhead on Constable. To understand such overhead, we show the fraction of loads eliminated by Constable that violate memory ordering as a box-and-whiskers plot in Figure 8.22(a). As we can see, an eliminated load rarely violates memory ordering. On average, only 0.09% of all eliminated loads violate memory ordering. Less than 0.5% of eliminated loads violate memory ordering in 86 out of 90 workloads. This is primarily due to Constable’s confidence-based mechanism that considers a load instruction eligible for elimination only if it meets a sufficiently-high stability confidence level threshold (set to 30 in our evaluation). Figure 8.22(b) further shows the increase in instructions allocated to the ROB in presence of Constable as compared to the baseline system to understand the effect of such rare memory ordering violations. As we can see, Constable increases the allocated instructions by only 0.3% on average across all workloads. 79 out of 90 workloads observe an increase of less than 1%. Thus, we conclude that Constable observes a very insignificant overhead due to rare memory ordering violations by incorrectly-eliminated loads.

8.7.8

Effect of Clean Evictions on Elimination Coverage

In order to correctly eliminate load instructions in a multi-core system, Constable proposes pinning the CV-bit of a cacheline that is accessed by an eliminated load instruction (see §8.5.6). However, the change in the coherence protocol may complicate hardware verification. Another alternative design could be to avoid elimination on every core-private cache eviction. However, this design choice may lose elimination opportunities if the evicted cacheline is

8.7: CONSTABLE: EVALUATION

(a)

Increase in allocated instructions in presence of Constable

Fraction of eliminated loads that violate memory ordering

3.0%

151

2.5% 2.0% 1.5% 1.0% 0.5% 0.0%

t e 7 7 ver ie n r p r is EC 1 EC 1 Cl Se r e P P t S S I F En

12% 10%

(b)

8% 6% 4% 2% 0% -2% -4%

t e 7 7 ver ie n r p r is EC 1 EC 1 Se r e P P t S S I F En

Figure 8.22: (a) Fraction of loads eliminated by Constable that violate memory ordering. (b) Increase in instructions allocated to ROB in presence of Constable.

clean. In this section, we quantify impact of such elimination opportunity loss on Constable’s performance and elimination coverage. To understand the effect, we model a Constable variant that looks up AMT for every L1 data (L1-D) cache eviction and invalidates the AMT entry. This prevents Constable from eliminating any further load instructions that access the evicted cacheline. We call this Constable variant Constable-AMT-I. Figure 8.23(a) compares the speedup of Constable-AMT-I with the vanilla Constable. Constable-AMT-I loses 0.9% performance improvement than the vanilla Constable on average across all workloads. 11 out of 90 workloads observe a performance loss of more than 5% in Constable-AMT-I (with the highest performance loss of 10.4% in 554.roms_r) as compared to vanilla Constable. The performance loss is primarily attributed to the loss in elimination coverage. As Figure 8.23(b) shows, Constable-AMT-I provides 3.4% less load elimination coverage than the vanilla Constable. 17 out of 90 workloads observe a coverage loss of more than 5% in Constable-AMT-I (with the highest coverage loss of 27% in 554.roms_r). Thus, we conclude that CV-bit pinning is a more-performant design choice than avoiding elimination on every core-private cache eviction. 40%

Constable-AMT-I

35%

1.08 1.06 1.04 1.02

(b)

Constable

Constable-AMT-I 23.5% 20.2%

Constable

Fraction of all loads

(a)

1.051 1.042

30% 25% 20% 15% 10% 5%

1.00

r G EO M AE VA GN

Se rv e

IS PE C 17

FS PE C 17

En te rp ris e

r G EO M EA N

Se rv e

IS PE C 17

FS PE C 17

En te rp ris e

C lie nt

0% C lie nt

Geomean speedup over the baseline

1.10

Figure 8.23: (a) Speedup and (b) coverage of Constable with AMT invalidation on L1D eviction compared to a vanilla Constable.

CHAPTER 8: SAFELY ELIMINATING LOAD INSTRUCTION EXECUTION

152

8.8

Summary

We introduce Constable, a purely-microarchitectural technique that safely eliminates the execution of load instructions while breaking the load data dependence. Our extensive evaluation using a wide range of workloads and system configurations shows that Constable provides significant performance benefit and reduced dynamic power consumption by eliminating load execution.

8.8.1

Influence on the Research Community

Constable has been presented at the the 51st International Symposium on Computer Architecture (ISCA) on June, 2024 and was recognized with the Best Paper Award at ISCA 2024 [46]. We have released an open-source binary instrumentation tool, called the Load Inspector [38], that can identify and provide insights on global-stable loads present in any off-the-shelf x86(64) binary. Constable has influenced both academic research and industrial product development. A follow-up work [629] has independently verified Constable’s key observation of global-stable loads in off-the-shelf programs using a modified version of our Load Inspector tool and has uncovered new insights on why modern compilers fail to eliminate such loads at compile time. To the best of our knowledge, the key idea of Constable is in the process of getting transferred to a real-world commercial processor design, and has been the subject matter of a patent application filed by Intel Corporation [683]. As hardware resource scaling is becoming increasingly challenging in today’s processor design, we believe and hope that Constable’s key observations and insights would inspire future works to explore a multitude of other optimizations that mitigate ILP loss due to resource dependence and load instruction execution.

Chapter 9

Conclusions and Future Directions 9.1

Putting It All Together

In pursuit of advancing the performance and energy efficiency of general-purpose processors, this dissertation argues for a fundamental shift in microarchitectural design: from the conventional data-agnostic approaches to data-driven and data-aware approaches, where microarchitecture would tailor their policies by continuously learning and exploiting the characteristics of application data and system-generated metadata. We build a detailed understanding of how the data-agnostic nature of state-of-the-art mechanisms for hiding or tolerating memory latency limits their effectiveness, and use this understanding to propose four new mechanisms, designed from the ground up with data-driven and data-aware principles. Across all four cases, we show that our proposed data-driven and data-aware microarchitectural techniques significantly improve performance and/or energy efficiency over the best conventional techniques. First, we begin with data prefetching, a fundamental mechanism to hide memory latency by predicting future memory addresses. In Chapter 5 we show that state-of-the-art prefetchers are limited by their reliance on a single fixed program feature and their lack of awareness to system-level feedback (e.g., memory bandwidth usage). To address these limitations, we formulate prefetching as a reinforcement learning problem, where the prefetcher acts as an autonomous agent that learns to prefetch using multiple program features and system-level feedback, without depending on rigid heuristics. We show that our proposed prefetcher, Pythia, consistently outperforms several state-of-the-art prefetchers across a wide range of workloads and system configurations, while incurring only modest area and power overhead. Our evaluation of Pythia across a large-scale previously-unseen trace corpus (see Chapter 4) collected for the 4th Data Prefetching Championship (DPC4) empirically validates that Pythia’s performance benefits generalize well beyond the workloads used at design time. Second, our experience with prefetching reveals that even an advanced prefetcher like Pythia successfully predicts only about half of the load requests that eventually access the main memory. For the remaining off-chip loads, we find that a large fraction of their latency is spent accessing the on-chip cache hierarchy to solely determine that they need to go offchip. To address this inefficiency, we propose off-chip prediction, a fundamentally different form of speculation than prefetching, that predicts which loads will miss the on-chip cache hierarchy and speculatively fetches their data directly from main memory, thereby removing cache access latency from their critical path. In Chapter 6, we design Hermes, which exploits perceptron learning to accurately identify off-chip requests from diverse program features.

153

154

CHAPTER 9: CONCLUSIONS AND FUTURE DIRECTIONS

We show that Hermes delivers significant additional performance improvements on top of state-of-the-art prefetchers across a wide range of workloads and system configurations. Our evaluation of Hermes on the previously-unseen DPC4 trace corpus empirically validates that Hermes’ performance benefits generalize well beyond the workloads used at design time. Third, we further study prefetching and off-chip prediction in unison and find that, while each technique improves performance individually, their simultaneous use often requires careful coordination to avoid negating one another’s benefits, particularly in bandwidthconstrained systems. To address this, we formulate their coordination as a reinforcement learning problem. In Chapter 7, we propose Athena, an reinforcement learning-based coordinator that autonomously learns effective policies by observing system-level metrics and the impact of its own decisions on the overall system’s performance, without relying on rigid, and often myopic, human-crafted heuristics and thresholds. Through extensive evaluation, we show that Athena consistently outperforms multiple state-of-the-art coordination policies across diverse system configurations, prefetcher–OCP combinations, and memory bandwidth settings, while incurring only modest storage overhead. Moreover, by evaluating Athena on the previously-unseen DPC4 trace corpus, we empirically validate that Athena’s performance benefits generalize well beyond the workloads used at design time. Fourth, we analyze the execution of load instructions in the processor core and observe that a significant fraction of dynamic load instructions repeatedly fetch the same value from the same memory address. Executing such redundant loads wastes scarce pipeline resources and limits instruction-level parallelism. To address this inefficiency, we propose Constable (see Chapter 8), a purely-microarchitectural technique that dynamically identifies loads with stable address–value behavior and safely eliminates their execution while preserving correctness. We show that Constable improves both performance and power efficiency on top of aggressively-optimized out-of-order processors, providing further gains when combined with state-of-the-art load value prediction and simultaneous multithreading. Together, these four works demonstrate how data-driven and data-aware principles can fundamentally reshape microarchitectural design. By leveraging lightweight machine learning methods and exploiting overlooked data characteristics, we show that, techniques that adapt their policies by continuously learning from the vast amount of application data and system-generated metadata, as well as exploit characteristics of application data, indeed deliver performance and energy efficiency improvements that are otherwise untapped by conventional data-agnostic microarchitectural techniques.

9.2

Future Research Directions

While this dissertation primarily focuses on improving performance and energy efficiency of traditional general-purpose processors, we believe that the principles and techniques presented here transcend to various other types of computing systems and paradigms. In this section, we discuss several promising directions for extending the principles more broadly across modern computing systems.

9.2. FUTURE RESEARCH DIRECTIONS

9.2.1

155

Improving and Extending the Proposed Techniques

Although the four techniques presented in this dissertation significantly advance their respective state-of-the-art, we believe they can be further improved and extended to various aspects of microarchitecture design. We review several such possibilities in this section. Enabling Adaptive Multi-Degree Prefetching While Pythia’s reinforcement learning–based formulation establishes a foundation for autonomous, data-driven prefetching, several components of its design remain statically determined at design time through extensive design-space exploration. Two key examples are the construction of the set of prefetch actions and the selection of prefetch degree. As discussed in §5.4.3 and shown in Table 5.2, we fix the set of candidate prefetch actions at design time. Although this static list yields state-of-the-art performance across a wide range of workloads, it risks missing useful prefetch opportunities when the most beneficial offset is absent from the predefined set. Similarly, Pythia currently employs a heuristic-based degree selection policy that adjusts the prefetch degree using static thresholds, which cannot adapt to workload or system dynamics at runtime. We believe that future research can explore mechanisms that will allow Pythia to autonomously and adaptively select both its candidate prefetch actions and prefetch degree during execution. Exploiting Off-Chip Prediction for Transparent Offloading to Processing-In-Memory Devices Processing-in-memory (PIM) paradigm places computing mechanisms in memory/storage [73, 90, 98, 146, 167, 211, 242, 270, 292, 293, 303, 304, 307, 352, 362, 380, 462, 463, 503–505, 590, 615, 616, 621, 626, 645, 689, 732–734, 736, 736, 738–740, 759, 762, 795, 796, 811, 835, 836, 889, 911] or near [75, 76, 81, 82, 93–95,101, 102, 112–115, 146,156–164, 176, 200, 214,239, 241, 244, 247,248, 256, 272,273,283,284,291,308,310,311,320–324,326,327,331,334,334,335,335–337,337,342,343,346, 350,353,357,369,370,374,376,386,401,444–446,459,461,464,464,473,474,477,480,492–494,501, 507, 521, 523, 546, 581, 596, 597, 607, 617–620, 622, 651, 653, 672, 701, 716, 717, 769, 774, 775, 804, 815, 837,862,888,912,915,931,934] where the data is stored to reduce/eliminate the data movement bottleneck between the computation units and the memory/storage system. Although PIM offers a fundamental solution to address memory bottleneck, its practical adoption remains hindered by the critical challenge of identifying program regions that are suitable for offloading, especially in a manner that is transparent to programs and programmers. We believe Hermes’s off-chip prediction, in conjunction with cache-reuse prediction [280, 408, 414, 452, 547, 825], presents an opportunity to address this challenge. More specifically, when a given load instruction is predicted to go off-chip and the data loaded by such instruction will also likely have no reuse, this combination of speculations can serve as a lightweight hint to dynamically offload that load instruction, and any instructions dependent on it, to a PIM-enabled device. This approach could enable seamless and adaptive exploitation of PIM benefits without requiring explicit code annotations, manual partitioning, or compiler intervention.

156

CHAPTER 9: CONCLUSIONS AND FUTURE DIRECTIONS

Extending Off-Chip Prediction in Modern Disaggregated Memory Systems With the advent of emerging interconnect technologies (e.g., CXL [11], NVLink [41]), modern systems employ increasing disaggregated and tiered memory architectures composed of multiple memory levels with various capacity and memory access latency. In such systems, memory requests typically incur additional latency due to the sequential traversal of each tier in the memory hierarchy [77, 341, 344, 354, 500, 544, 545, 674, 794, 816]. The fundamental idea behind Hermes, i.e., speculatively bypassing intermediate memory levels by predicting where the requested data will ultimately reside, can be naturally extended to this context. Applying such speculation in disaggregated memory systems could yield even greater benefits than in traditional cache hierarchies, by both reducing effective access latency and improving overall energy efficiency. Extending Stable-Load Elimination Coverage via Compiler-Microarchitecture CoOptimization Our detailed workload analysis and evaluation of Constable discussed in Chapter 8 provide two high-level insights. First, it shows why global-stable loads remain abundant in real-world workloads even after aggressive compiler optimizations, demonstrating the limitations of current compiler techniques in eliminating such instructions at compile time. Second, despite its sophisticated hardware mechanisms, Constable is unable to eliminate 43.6% (see §8.7.3) of the global-stable loads identified offline, highlighting scenarios where elimination opportunities are left untapped. These findings open two complementary avenues for future research. On the compiler side, more intelligent optimization strategies (such as improved register allocation and profile-guided analysis) can be developed to eliminate global-stable loads during compile time. On the microarchitectural side, Constable’s mechanisms can be extended to address remaining cases, e.g., by designing hardware support that effectively handles elimination losses caused by silent stores.

9.2.2

New Avenues for ML-Driven and Data-Aware Microarchitectures

Microarchitectures of modern computing systems (e.g., CPU, GPU, specialized accelerators) employ numerous techniques to enhance their performance and energy efficiency. We believe many of such techniques can be fundamentally reimagined using ML-driven and data-aware approaches presented in this dissertation. In this section, we highlight several such opportunities, noting that they represent only a subset of the broader possibilities that future research may uncover. Coordinating Caching, Prefetching, and Main Memory Scheduling using MultiAgent Collaborative Reinforcement Learning Caching, prefetching, and main memory scheduling policies [104, 184, 264, 266, 266, 325, 383, 418, 468, 469, 484, 566, 577, 579, 588, 589, 694, 695, 807–810, 847, 848, 921, 921] jointly deter-

9.2. FUTURE RESEARCH DIRECTIONS

157

mine the average memory access latency experienced by a processor. Although prior works have proposed reinforcement learning (RL)-based agents for each of these components (e.g., caching [530, 743], prefetching [136, 319, 660], and memory scheduling [383, 576]), such agents typically operate in isolation to optimize for their local objective with their local view of the system, without the awareness of the global system behavior. Such lack of coordination can lead to suboptimal or even detrimental performance. We believe that a multi-agent RLbased approach poses a viable solution to co-ordinate caching, prefetching, and main memory scheduling policies to improve overall system’s performance, energy efficiency, and many other metrics (e.g., fairness), beyond what any single technique can achieve independently. Reinforcement Learning for Dynamic Instruction Criticality Prediction Instruction criticality prediction has been studied as a means to identify dynamic instructions that disproportionately influence program performance, thereby enabling targeted latency reduction techniques [296]. Traditional approaches rely on handcrafted heuristics (for example, loads feeding to mispredicted branches or other cache-missing loads [800]), which are inherently ad-hoc, prone to oversight, and lack adaptability. Moreover, criticality prediction lacks a fixed ground truth; whether or not an instruction would be critical evolves based on speculative optimizations applied on that instruction, making critical instruction identification a moving target. We believe that re-framing criticality prediction as a reinforcement learning problem could overcome these limitations. An RL-based framework could continuously and autonomously learn which instructions are critical by systematically applying optimizations to dynamic instructions and observing their real-time impact on system performance. This approach may eliminate dependence on manual heuristics and could surface previously unknown patterns of criticality, providing novel insights to processor architects. Exploiting Accurate Speculation Metadata for Lazy Instruction Execution Modern processors rely heavily on two fundamental types of speculative techniques that break control dependencies (e.g., branch prediction [411, 491, 753, 783, 900]) and data dependencies (e.g., value prediction [125, 170, 174, 329, 434, 515, 553, 663–667, 712, 720, 722, 754, 763, 764], memory renaming [569, 570, 687, 841, 844]). Since these mechanisms have been designed to achieve very high accuracy (as their mispredictions cost significant performance penalty), we believe their repetitive correctness can be systematically exploited. More specifically, when the control- and data-dependence on a given instruction is predicted with high confidence, it can be executed lazily (i.e., with a time slack [295–297]) without delaying its dependents. Such awareness of prediction metadata can potentially free scarce and power-intensive pipeline resources (e.g., ports to execution units and/or L1 data cache) for other non-predictable instructions, thereby improving both performance and energy efficiency. Extending Data-Driven and Data-Aware Designs beyond CPUs Although we demonstrate Pythia, Hermes, Athena, and Constable in the context of generalpurpose processors, the key principles underlying these techniques are broadly applicable to

158

CHAPTER 9: CONCLUSIONS AND FUTURE DIRECTIONS

other computing systems, including GPUs, TPUs, and specialized accelerators. For example, as GPUs increasingly serve large-scale general-purpose workloads, sophisticated prefetching mechanisms such as Pythia could mitigate the latency of irregular memory accesses. Similarly, given the rapid growth of last-level cache capacity in commercial GPUs (e.g., from 4 MB in Nvidia P100 to 40 MB in A100 within four years [40]), Hermes can provide significant benefits by eliminating on-chip cache traversal latency for off-chip loads. Constable’s principle of exploiting load-value stability also holds promise in GPUs and accelerators, where redundant memory accesses are common, offering an opportunity to reduce pipeline pressure and improve energy efficiency. We believe that extending data-driven and data-aware designs beyond CPUs presents a promising research direction to address the diverse performance and energy efficiency bottlenecks of heterogeneous platforms.

9.3

Concluding Remarks

This dissertation makes two broad contributions. First, it develops a detailed understanding of how the data-agnostic nature of conventional microarchitectural techniques limits their ability to mitigate the memory bottleneck. Second, it proposes four novel techniques that embody data-driven and data-aware design principles. Pythia formulates prefetching as a reinforcement learning problem, enabling adaptive and system-aware prefetch decisions. Hermes leverages perceptron learning to accurately predict off-chip memory accesses, thereby removing costly on-chip cache access latency from their critical path. Athena employs reinforcement learning to synergize prefetching and off-chip prediction, delivering robust performance across diverse workloads and system configurations. Constable exploits load-value stability to safely eliminate redundant loads, improving both performance and energy efficiency. We believe that the insights and techniques presented in this dissertation will not only advance the state-of-the-art microarchitectural design, but also pave the way for a new generation of datadriven and data-aware systems that can more effectively address the ever-growing memory bottleneck.

Appendix A

Comprehensive View of the Author’s Contributions During the course of my graduate studies, I led/co-led five successful projects in the broader topic of processor microarchitecture and memory system design. Four of these projects collectively form the core contributions of this dissertation. First, I started exploring data prefetching and proposed DSPatch [141], a bandwidth-aware spatial prefetcher that advanced the stateof-the-art prefetcher performance. Second, I merged my experience on prefetching with machine learning methods and proposed Pythia [136], open-sourcing its artifact-evaluated infrastructure [45]. Pythia has served as both a state-of-the-art baseline for subsequent studies [260, 274, 319, 404, 513] and as a reference framework for modeling architectural decision making using machine learning [91,205,330,375,526,530,594,642,776,890,892,902,910,916,929]. Third, I continued exploring the intersection of microarchitecture and machine learning and proposed Hermes [133], along with its artifact-evaluated implementation and infrastructure [22, 23]. To our knowledge, Hermes is the first work to propose off-chip prediction and to demonstrate the applicability of perceptron-learning for off-chip prediction. Hermes influenced multiple follow-up works [282, 399, 478, 633, 719, 886] and was recognized with the Best Paper Award at MICRO 2022 [47]. Fourth, I shifted the focus from the memory system to the processor core to explore memory-related inefficiencies at the source. This exploration resulted in Constable [143], releasing the open-source Load Inspector [38] tool as a part of the project. Constable was recognized with the Best Paper Award at ISCA 2024 [46] for its contribution to the state-of-the-art. Most recently, with my mentee Zhenrong Lang, I co-developed Athena [138] and released its artifact-evaluated implementation and evaluation infrastructure [7]. Athena was recognized with the Distinguished Artifact Award at HPCA 2026 [48]. Lastly, in our recent IEEE Micro article [140], my co-author Rakesh Nadig and I presented a comprehensive overview of the progress we made over the past half decade in ML-driven microarchitectural design for memory and storage systems. Throughout my graduate studies, I also contributed to other research projects. These works can be classified into four groups based on the broader topic of the research. The rest of this section gives a comprehensive summary of such works. Machine-Learning-Driven Storage System Design. In collaboration with Gagandeep Singh and Rakesh Nadig, we explored the application of ML in managing hybrid storage systems (HSS), which employ storage devices with widely varying capacity and access latency. 159

160

APPENDIX A: COMPREHENSIVE VIEW OF THE AUTHOR’S CONTRIBUTIONS

Together, we first proposed Sibyl [776], which demonstrates the first RL-based framework to make adaptive data placement decisions by autonomously learning from multiple program/system features (e.g., size and type of the current storage request, remaining capacity of the fast storage device) and system-level feedback (e.g., request latency), without relying on human-designed heuristics or thresholds. We show that Sibyl not only significantly outperforms the best conventional heuristics-based and learning-based policies, but also provides extensibility to wide range of HSS. While Sibyl manages the data placement in HSS, it does not address the data migration between different storage devices of an HSS. To this end, we proposed Harmonia [594], which demonstrates the first multi-agent RL-based framework for HSS to manage both data placement and data migration in a synergistic way. Together, Sibyl and Harmonia further reinforce this dissertation’s central statement that data-driven decision making can provide substantial performance and efficiency gains beyond conventional dataagnostic techniques. Address Translation Subsystem Design. In collaboration with Konstantinos Kanellopoulos, we explored various avenues to accelerate the address translation subsystem of generalpurpose processors. First, we designed Utopia [436], an virtual-to-physical address mapping scheme that allows both conventional flexible address mapping and a hash-based restrictive address mapping to co-exist, thereby exploiting the benefits of both worlds in reducing address translation overhead without sacrificing the core virtual memory functionalities (e.g., data sharing). Second, we designed Victima [437], that exploits under-utilized L2 data cache as a victim cache to store evicted second-level translation look-aside buffer (TLB) translation entries. By doing so, Victima converts a significant number of L2 TLB misses from costly page table walks to faster L2 cache hits. Third, we designed Revelator [439], which employs a OS-hardware co-designed approach to enable accurate speculative virtual-to-physical address translation. Lastly, we open-sourced our flexible development infrastructure that we used for evaluating Utopia, Victima, Revelator, and numerous prior works on OS-architecture codesigned address translation schemes as a standalone simulator named Virtuoso [61,438]. This infrastructure was recognized with the Distinguished Artifact Award at MICRO 2023 [49]. Processing Data Where It Makes Sense. I also collaborated in various projects that enable seamless processing of data closer to where they reside. I significantly contributed to REDUCT [608,609], which enables efficient deep learning inference in multi-core CPUs by enabling near-cache computation. In collaboration with Alain Denzler, we designed Casper [245], where we extended the key learning from REDUCT to enable efficient stencil kernel computation in CPU via near-cache computation. In collaboration with Mayank Kabra, we proposed CIPHERMATCH [432], which accelerates homomorphic encryption (HE)-based string matching via novel data packing scheme and in-storage processing. Lastly, in collaboration with Rakesh Nadig, we proposed Conduit [595], a general-purpose near-data processing framework for SSDs that transparently offloads fine-grained computations to various computation resources available inside an SSD (e.g., SSD controller cores, SSD-internal DRAM chips, SSD’s flash chips).

COMPREHENSIVE VIEW OF THE AUTHOR’S CONTRIBUTIONS

161

Other Works. Together with Ataberk Olgun, we developed Sectored DRAM [614], which enables fine-grained DRAM data transfer and DRAM row activation in three steps. First, Sectored DRAM predicts which words of a requested cacheblock is likely going to be used during its cache residency. Second, it only activates a smaller set of cells in DRAM that contains the predicted words. Third, it only transfers the predicted words over the DRAM channel. By doing so, Sectored DRAM significantly reduces the overall energy consumption of the system, while simultaneously improving performance. In collaboration with Jawad Haj-Yahya, we proposed BurstLink [351], which improves the efficiency of video streaming in mobile devices by exploiting the full bandwidth of modern display interfaces and remote buffering a decoded frame.

Appendix B

Complete List of the Author’s Contributions This section lists the author’s contributions to the literature in reverse chronological order under three categories: (1) major contributions that the author led, (2) co-supervised contributions by the author, and (3) other contributions.

B.1

Major Contributions Led by the Author

1. Rahul Bera, Rakesh Nadig, Onur Mutlu, “Machine Learning-Driven Intelligent Memory System Design: From On-Chip Caches to Storage”, in IEEE Micro, 2026 2. Rahul Bera, Zhenrong Lang, Caroline Hengartner, Konstantinos Kanellopoulos, Rakesh Kumar, Mohammad Sadrosadati, Onur Mutlu, “Athena: Synergizing Data Prefetching and Off-Chip Prediction via Online Reinforcement Learning”, in HPCA, 2026. Artifact available: https://github.com/CMU-SAFARI/Athena. Distinguished Artifact Award. 3. Rahul Bera, Adithya Ranganathan, Joydeep Rakshit, Sujit Mahto, Anant V. Nori, Jayesh Gaur, Ataberk Olgun, Konstantinos Kanellopoulos, Mohammad Sadrosadati, Sreenivas Subramoney, Onur Mutlu, “Constable: Improving Performance and Power Efficiency by Safely Eliminating Load Instruction Execution”, in ISCA, 2024. Toolkit available: https: //github.com/CMU-SAFARI/Load-Inspector. Best Paper Award. Subject matter of US patent US20260003626A1, filed: 2024-06-27. 4. Rahul Bera, Konstantinos Kanellopoulos, Shankar Balachandran, David Novo, Ataberk Olgun, Mohammad Sadrosadati, Onur Mutlu, “Hermes: Accelerating Long-Latency Load Requests via Perceptron-Based Off-Chip Load Prediction”, in MICRO, 2022. Artifact available: https://github.com/CMU-SAFARI/Hermes. Best Paper Award. 5. Rahul Bera, Konstantinos Kanellopoulos, Anant V. Nori, Taha Shahroodi, Sreenivas Subramoney, Onur Mutlu, “Pythia: A Customizable Hardware Prefetching Framework using Online Reinforcement Learning”, in MICRO, 2021. Artifact available: https://github.c om/CMU-SAFARI/Pythia 6. Rahul Bera, Anant V. Nori, Onur Mutlu, Sreenivas Subramoney, “DSPatch: Dual Pattern

163

APPENDIX B: COMPLETE LIST OF THE AUTHOR’S CONTRIBUTIONS

164

Spatial Prefetcher”, in MICRO, 2019. Subject matter of US patent US11874773B2, filed: 2019-12-28; published: 2021-03-25.

B.2

Co-Supervised Contributions

1. Konstantinos Kanellopoulos, Rahul Bera, Kosta Stojiljkovic, Nisa Bostanci, Can Firtina, Rachata Ausavarungnirun, Rakesh Kumar, Nastaran Hajinazar, Mohammad Sadrosadati, Nandita Vijaykumar, Onur Mutlu, “Utopia: Efficient Address Translation using Hybrid Virtual-to-Physical Address Mapping”, in MICRO, 2023. Artifact available: https: //github.com/CMU-SAFARI/Utopia. 2. Anant V Nori, Rahul Bera, Shankar Balachandran, Joydeep Rakshit, Om J Omer, Avishaii Abuhatzera, Kuttanna Belliappa, Sreenivas Subramoney, “REDUCT: Keep it Close, Keep it Cool!: Effcient Scaling of DNN Inference on Multi-Core CPUs with Near-Cache Compute”, in ISCA, 2021. Subject matter of US patent US12405890B2, filed: 2021-12-23; published: 2023-06-29.

B.3

Other Contributions

1. Rakesh Nadig, Vamanan Arulchelvan, Rahul Bera, Taha Shahroodi, Gagandeep Singh, Andreas Kosmas Kakolyris, Ismail Emir Yuksel, Mohammad Sadrosadati, Jisung Park, Onur Mutlu, “Harmonia: Enhancing Data Placement and Migration in Hybrid Storage Systems via Multi-Agent Reinforcement Learning”, in ICS, 2026 2. Rakesh Nadig, Vamanan Arulchelvan, Mayank Kabra, Harshita Gupta, Rahul Bera, Nika Mansouri Ghiasi, Nanditha Rao, Qingcai Jiang, Andreas Kosmas Kakolyris, Yu Liang, Mohammad Sadrosadati, Onur Mutlu, “Conduit: Programmer-Transparent Near-Data Processing Using Multiple Compute-Capable Resources in SSDs”, in HPCA, 2026 3. Konstantinos Kanellopoulos, Konstantinos Sgouras, Harsh Songara, Andreas Kosmas Kakolyris, Vlad-Petru Nitu, Rahul Bera, Rakesh Kumar, Onur Mutlu, “Revelator: Rapid Data Fetching via OS-Driven Hash-based Speculative Address Translation”, arXiv (under peer-review), 2025 4. Konstantinos Kanellopoulos, Konstantinos Sgouras, F. Nisa Bostanci, Andreas Kosmas Kakolyris, Berkin Kerim Konar, Rahul Bera, Mohammad Sadrosadati, Rakesh Kumar, Nandita Vijaykumar, Onur Mutlu, “Virtuoso: Enabling Fast and Accurate Virtual Memory Research via an Imitation-based Operating System Simulation Methodology”, in ASPLOS, 2025. Artifact available: https://github.com/CMU-SAFARI/Virtuoso. 5. Mayank Kabra, Rakesh Nadig, Harshita Gupta, Rahul Bera, Manos Frouzakis, Vamanan Arulchelvan, Yu Liang, Haiyu Mao, Mohammad Sadrosadati, Onur Mutlu, “CIPHERMATCH: Accelerating Homomorphic Encryption-Based String Matching via MemoryEfficient Data Packing and In-Flash Processing”, in ASPLOS, 2025

B.3. OTHER CONTRIBUTIONS

165

6. Ataberk Olgun, F. Nisa Bostanci, Geraldo F. Oliveira, Yahya Can Tugrul, Rahul Bera, A. Giray Yaglikci, Hasan Hassan, Oguz Ergin, Onur Mutlu, “Sectored DRAM: An EnergyEfficient High-Throughput and Practical Fine-Grained DRAM Architecture”, in TACO, 2024. Artifact available: https://github.com/CMU-SAFARI/Sectored-DRAM. 7. Konstantinos Kanellopoulos, Hong Chul Nam, Nisa Bostanci, Rahul Bera, Mohammad Sadrosadati, Rakesh Kumar, Davide Basilio Bartolini, Onur Mutlu, “Victima: Drastically Increasing Address Translation Reach by Leveraging Underutilized Cache Resources”, in MICRO, 2023. Artifact available: https://github.com/CMU-SAFARI/Victima. Distinguished Artifact Award. 8. Alain Denzler, Geraldo F. Oliveira, Nastaran Hajinazar, Rahul Bera, Gagandeep Singh, Juan Gómez-Luna, Onur Mutlu, “Casper: Accelerating Stencil Computation using Nearcache Processing”, in IEEE Access, 2023 9. Gagandeep Singh, Rakesh Nadig, Jisung Park, Rahul Bera, Nastaran Hajinazar, David Novo, Juan Gómez-Luna, Onur Mutlu, “Sibyl: Adaptive and Extensible Data Placement in Hybrid Storage Systems Using Online Reinforcement Learning”, in ISCA, 2022 10. Jawad Haj-Yahya, Jisung Park, Rahul Bera, Juan Gómez Luna, Efraim Rotem, Taha Shahroodi, Jeremie Kim, Onur Mutlu, “BurstLink: Techniques for Energy-Efficient Conventional and Virtual Reality Video Display”, in MICRO, 2021.

Appendix C

Curriculum Vitae of the Author Education Sept 2019 Dec 2025

ETH Zürich, Doctor of Science Advisor: Prof. Dr. Onur Mutlu Thesis: “Mitigiating the Memory Bottleneck with Machine-Learning-Driven and Data-Aware Microarchitectural Techniques”

Jul 2014 Jan 2017

Indian Institute of Technology, Kanpur, Master of Technology Advisor: Prof. Dr. Mainak Chaudhuri Thesis: “Adaptive Prefetch Filter to Mitigate Prefetcher Induced Pollution”.

Jul 2010 Jun 2014

Jadavpur University, Bachelor of Engineering Thesis: “Design & Analysis of Logarithmic Multipliers and Dividers using VHDL”.

Professional Experience Mar 2023 Aug 2023

Processor Architecture Research Lab, Intel Labs Graduate Research Intern Mentors: Anant V. Nori and Sreenivas Subramoney. Worked on dynamic instruction elimination for high-performance and powerefficient processors.

Feb 2017 Aug 2019

Processor Architecture Research Lab, Intel Labs Architecture Researcher Mentors: Anant V. Nori and Sreenivas Subramoney. Worked on high-performance prefetcher design and near-cache computation for DNN inference.

May 2015 Dec 2015

Advanced Micro Devices Co-op Engineer Mentor: Dr. Kanishka Lahiri. Developed simulator to model AMD Data Fabric for performance projections of extremely-threaded SoCs. 167

APPENDIX C: CURRICULUM VITAE OF THE AUTHOR

168

Honors and Recognitions 2026

Distinguished artifact award at HPCA 2026 for “Athena: Synergizing Data Prefetching and Off-Chip Load Prediction via Online Reinforcement Learning”.

2025

MLCommons ML and Systems Rising Stars, awarded in recognition of research at the intersection of machine learning and systems.

2024

Best paper award at ISCA 2024 for “Constable: Improving Performance and Power Efficiency by Safely Eliminating Load Instruction Execution”.

2023

Distinguished artifact award at MICRO 2023 for “Victima: Drastically Increasing Address Translation Reach by Leveraging Underutilized Cache Resources”.

2022

Best paper award at MICRO 2022 for “Hermes: Accelerating Long-Latency Load Requests via Perceptron-Based Off-Chip Load Prediction”.

Key Publications Please visit https://dblp.org/pid/250/2580.html for a complete list of publications. HPCA 2026

Athena: Synergizing Data Prefetching and Off-Chip Load Prediction via Online Reinforcement Learning Rahul Bera, Zhenrong Lang, Caroline Hengartner, Konstantinos Kanellopoulos, Rakesh Kumar, Mohammad Sadrosadati, Onur Mutlu Distinguished artifact award at HPCA 2026.

HPCA 2026

Conduit: Programmer-Transparent Near-Data Processing Using Multiple Compute-Capable Resources in SSDs Rakesh Nadig, Vamanan Arulchelvan, Mayank Kabra, Harshita Gupta, Rahul Bera, Nika Mansouri Ghiasi, Nanditha Rao, Qingcai Jiang, Andreas Kosmas Kakolyris, Yu Liang, Mohammad Sadrosadati, Onur Mutlu

ASPLOS 2025

Virtuoso: Enabling Fast and Accurate Virtual Memory Research via an Imitation-based Operating System Simulation Methodology Konstantinos Kanellopoulos, Konstantinos Sgouras, F. Nisa Bostanci, Andreas Kosmas Kakolyris, Berkin Kerim Konar, Rahul Bera, Mohammad Sadrosadati, Rakesh Kumar, Nandita Vijaykumar, and Onur Mutlu

ASPLOS 2025

CIPHERMATCH: Accelerating Homomorphic Encryption-Based String Matching via Memory-Efficient Data Packing and In-Flash Processing Mayank Kabra, Rakesh Nadig, Harshita Gupta, Rahul Bera, Manos Frouzakis, Vamanan Arulchelvan, Yu Liang, Haiyu Mao, Mohammad Sadrosadati, and Onur Mutlu

CURRICULUM VITAE OF THE AUTHOR

169

ISCA 2024

Constable: Improving Performance and Power Efficiency by Safely Eliminating Load Instruction Execution Rahul Bera, Adithya Ranganathan, Joydeep Rakshit, Sujit Mahto, Anant V. Nori, Jayesh Gaur, Ataberk Olgun, Konstantinos Kanellopoulos, Mohammad Sadrosadati, Sreenivas Subramoney, and Onur Mutlu Best paper award at ISCA 2024.

MICRO 2023

Victima: Drastically Increasing Address Translation Reach by Leveraging Underutilized Cache Resources Konstantinos Kanellopoulos, Hong Chul Nam, F. Nisa Bostanci, Rahul Bera, Mohammad Sadrosadati, Rakesh Kumar, Davide Basilio Bartolini, and Onur Mutlu Distinguished artifact award at MICRO 2023.

MICRO 2023

Utopia: Efficient Address Translation using Hybrid Virtual-to-Physical Address Mapping Konstantinos Kanellopoulos, Rahul Bera, Kosta Stojiljkovic, Can Firtina, Rachata Ausavarungnirun, Nastaran Hajinazar, Jisung Park, Nandita Vijaykumar, and Onur Mutlu

MICRO 2022

Hermes: Accelerating Long-Latency Load Requests via PerceptronBased Off-Chip Load Prediction Rahul Bera, Konstantinos Kanellopoulos, Shankar Balachandran, David Novo, Ataberk Olgun, Mohammad Sadrosadati, and Onur Mutlu Best paper award at MICRO 2022.

ISCA 2022

Sibyl: Adaptive and Extensible Data Placement in Hybrid Storage Systems Using Online Reinforcement Learning Gagandeep Singh, Rakesh Nadig, Jisung Park, Rahul Bera, Nastaran Hajinazar, David Novo, Juan Gómez Luna, Sander Stuijk, Henk Corporaal, and Onur Mutlu

MICRO 2021

Pythia: A Customizable Hardware Prefetching Framework Using Online Reinforcement Learning Rahul Bera, Konstantinos Kanellopoulos, Anant V. Nori, Taha Shahroodi, Sreenivas Subramoney, and Onur Mutlu

MICRO 2021

BurstLink: Techniques for Energy-Efficient Conventional and Virtual Reality Video Display Jawad Haj-Yahya, Jisung Park, Rahul Bera, Juan Gómez Luna, Efraim Rotem, Taha Shahroodi, Jeremie Kim, and Onur Mutlu

ISCA 2021

REDUCT: Efficient Scaling of DNN Inference on Multi-core CPUs with Near-Cache Compute Anant V. Nori, Rahul Bera, Shankar Balachandran, Joydeep Rakshit, Om J. Omer, Avishaii Abuhatzera, Kuttanna Belliappa, and Sreenivas Subramoney

APPENDIX C: CURRICULUM VITAE OF THE AUTHOR

170

MICRO 2019

DSPatch: Dual Spatial Access Prefetcher Rahul Bera, Anant V. Nori, Onur Mutlu, and Sreenivas Subramoney

Key Patents 2024

Methods & Apparatus for Efficiently Processing Load Instruction Sequences Adithya Ranganathan, Rahul Bera, Joydeep Rakshit, Sujit Mahto, Anant V. Nori, Jayesh Gaur, and Sreenivas Subramoney US Patent US20260003626A1.

2023

Method & Apparatus for Leveraging Simultaneous Multithreading for Bulk Compute Operations Anant V. Nori, Rahul Bera, Shankar Balachandran, Joydeep Rakshit, Om Ji Omer, Sreenivas Subramoney, Avishaii Abuhatzera, and Belliappa Kuttanna US Patent US20230205692A1.

2021

Apparatuses, Methods, and Systems for Dual Spatial Pattern Prefetcher Rahul Bera, Anant V. Nori, and Sreenivas Subramoney US Patent US20210089456A1.

2020

Adaptive Spatial Access Prefetcher Apparatus and Method Rahul Bera, Anant V. Nori, Sreenivas Subramoney, and Hong Wang US Patent US10713053B2.

Invited Talks May 2025

Mitigating the Memory Bottleneck with Data-Driven and Data-Aware Microarchitectures Stanford University, Meta, and Tenstorrent.

Nov 2024

Data-Driven and Data-Aware Microarchitectures for Mitigating Memory Bottleneck in High-Performance Computing System Apple Lonestar Design Center, Austin, US.

Nov 2024

Data-Driven and Data-Aware Microarchitectures for Mitigating Memory Bottleneck in High-Performance Computing System AMD Research and Advanced Development, Austin, US.

Sept 2024

A Case for Data-Aware Microarchitecture for Alleviating Memory Bottleneck Huawei Research Center, Zürich, Switzerland.

CURRICULUM VITAE OF THE AUTHOR

171

Nov 2022

Taming the Memory Wall via Machine Learning Assisted Microarchitecture Design Huawei Research Center, Zürich, Switzerland.

Nov 2022

Hermes: Accelerating Long-Latency Load Requests via PerceptronBased Off-Chip Load Prediction Processor Architecture Research Lab, Intel Labs, India.

172

APPENDIX C: CURRICULUM VITAE OF THE AUTHOR

Bibliography [1] “2nd Data Prefetching Championship,” http://comparch-conf.gatech.edu/dpc2/. [2] “3rd Data Prefetching Championship,” https://dpc3.compas.cs.stonybrook.edu. [3] “6th Generation Intel® Processor Family,” https://www.intel.com/content/www/us/e n/processors/core/desktop-6th-gen-core-family-spec-update.html. [4] “AMD Ryzen Threadripper 3990X,” https://en.wikichip.org/wiki/amd/ryzen_threadrip per/3990x. [5] “AMD Zen2 EPYC 7702P,” https://en.wikichip.org/wiki/amd/epyc/7702p. [6] “Apache Hadoop,” https://hadoop.apache.org/. [7] “Athena GitHub Repository,” https://github.com/CMU-SAFARI/Athena. [8] “BigBench,” https://blog.cloudera.com/blog/2014/11/bigbench-toward-an-industry-s tandard-benchmark-for-big-data-analytics/. [9] “Caching of Temporal vs. Non-Temporal Data - Intel® 64 and IA-32 Architectures Developer’s Manual,” https://www.intel.com/content/www/us/en/architecture-and-tec hnology/64-ia-32-architectures-software-developer-vol-1-manual.html. [10] “Chisel/FIRRTL Hardware Compiler Framework,” https://www.chisel-lang.org. [11] “Compute Express Link,” https://computeexpresslink.org. [12] “DaCapo Benchmark Suite,” https://www.dacapobench.org. [13] “Disclosure of Hardware Prefetcher Control on Some Intel® Processors,” https://soft ware.intel.com/content/www/us/en/develop/articles/disclosure-of-hw-prefetcher-co ntrol-on-some-intel-processors.html. [14] “DPC4 - GitHub,” https://github.com/CMU-SAFARI/DPC4. [15] “First Championship Value Prediction (CVP-1) - Leaderboard,” https://microarch.org/ cvp1/cvp1online/contestants.html. [16] “GCC 13 Release Series,” https://gcc.gnu.org/gcc-13/. [17] “GCC Optimization Options,” https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Optimize -Options.html. [18] “GlobalFoundries 14nm FinFET Technology,” https://www.globalfoundries.com/sites/ default/files/product-briefs/pb-14lpp.pdf.

173

174

BIBLIOGRAPHY

[19] “Golden Cove Microarchitecture (P-Core) Examined,” https://www.anandtech.com/sh ow/16881/a-deep-dive-into-intels-alder-lake-microarchitectures/3. [20] “Golden Cove’s Vector Register File,” https://chipsandcheese.com/2023/01/15/golden -coves-vector-register-file-checking-with-official-spr-data/. [21] “Google Workload Traces Version 2,” https://console.cloud.google.com/storage/brows er/external-traces-v2. [22] “Hermes GitHub Repository,” https://github.com/CMU-SAFARI/Hermes. [23] “Hermes Zenodo Repository,” https://doi.org/10.5281/zenodo.6909799. [24] “HP-LINPACK,” https://www.netlib.org/benchmark/hpl/. [25] “Intel Core i5-12600K DDR4 Alder Lake CPU Review,” https://www.thefpsreview.com /2021/12/08/intel-core-i5-12600k-ddr4-alder-lake-cpu-review/6/. [26] “Intel Details Golden Cove: Next-Generation Big Core For Client and Server SoCs,” https://fuse.wikichip.org/news/6111/intel-details-golden-cove-next-generation-bigcore-for-client-and-server-socs/. [27] “Intel Xeon D-2123IT,” https://en.wikichip.org/wiki/intel/xeon_d/d-2123it. [28] “Intel Xeon Gold 6150,” https://en.wikichip.org/wiki/intel/xeon_gold/6150. [29] “Intel Xeon Gold 6258R,” https://en.wikichip.org/wiki/intel/xeon_gold/6258r. [30] “Intel Xeon Platinum 8180M,” https://en.wikichip.org/wiki/intel/xeon_platinum/818 0m. [31] “Intel® Xeon® Platinum 8376H Processor,” https://www.intel.com/content/www/us/e n/products/sku/204096/intel-xeon-platinum-8376h-processor-38-5m-cache-2-60-ghz /specifications.html. [32] “Introducing Intel® Advanced Performance Extensions,” https://www.intel.com/cont ent/www/us/en/developer/articles/technical/advanced-performance-extensions-apx .html. [33] “JEDEC-DDR4,” https://www.jedec.org/sites/default/files/docs/JESD79-4.pdf. [34] “JetStream 2.0,” https://browserbench.org/JetStream2.0/. [35] “L3 Cache Latency Comparison at Base Frequency,” https://www.cpuagent.com/cpu/i ntel-core-i9-10900k/benchmarks/l3-cache-latency-at-base-frequency/nvidia-geforce -rtx-2080-ti?res=1&quality=ultra. [36] “LAMMPS Benchmark,” https://www.lammps.org/bench.html. [37] “LLVM 18.1.4 Release,” https://github.com/llvm/llvm-project/releases/tag/llvmorg-18 .1.4.

BIBLIOGRAPHY

175

[38] “Load Inspector GitHub Repository,” https://github.com/CMU-SAFARI/Load-Inspect or. [39] “MOVNTI - x86 ISA,” https://www.felixcloutier.com/x86/movnti. [40] “NVIDIA A100 Tensor Core GPU Architecture,” https://images.nvidia.com/aem-dam/ en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf. [41] “NVIDIA NVLink,” https://www.nvidia.com/en-us/products/workstations/nvlink-brid ges/. [42] “PCACTI Tool - USC SPORTlab,” https://sportlab.usc.edu/downloads/download. [43] “Pin - A Dynamic Binary Instrumentation Tool,” https://software.intel.com/en-us/art icles/pin-a-dynamic-binary-instrumentation-tool. [44] “Popping the Hood on Golden Cove,” https://chipsandcheese.com/2021/12/02/poppin g-the-hood-on-golden-cove/. [45] “Pythia GitHub Repository,” https://github.com/CMU-SAFARI/Pythia. [46] “SAFARI News: Best Paper Award at ISCA 2024 for Constable,” https://safari.ethz.ch/b est-paper-award-at-isca-2024/. [47] “SAFARI News: Best Paper Award at MICRO 2022 for Hermes,” https://safari.ethz.ch/b est-paper-award-at-micro22/. [48] “SAFARI News: Distinguished Artifact Award at HPCA 2026 for Athena,” https://safa ri.ethz.ch/distinguished-artifact-award-at-hpca-2026-for-athena/. [49] “SAFARI News: Distinguished Artifact Award at MICRO 2023 for Victima,” https://sa fari.ethz.ch/micro-distinguished-artifact-award-for-victima/. [50] “Second Championship Value Prediction (CVP-2),” https://www.microarch.org/cvp1/c vp2/rules.html. [51] “Snort 3,” https://snort.org/snort3. [52] “SPEC CPU 2006,” https://www.spec.org/cpu2006/. [53] “SPEC CPU 2017,” https://www.spec.org/cpu2017/. [54] “SPECjbb© 2015,” https://www.spec.org/jbb2015/. [55] “SPECjEnterprise© 2010,” https://www.spec.org/jEnterprise2010/. [56] “Surprisingly High Latency Discovered During Alder Lake Test,” https://wccftech.com /surprisingly-high-latency-discovered-during-alder-lake-test-with-ddr5-6400-memo ry/. [57] “Synopsys DC Ultra,” https://www.synopsys.com/implementation-and-signoff/rtl-sy nthesis-test/dc-ultra.html.

176

BIBLIOGRAPHY

[58] “SYSmark 30,” https://bapco.com/products/sysmark-30/. [59] “TabletMark v3,” https://bapco.com/products/end-of-life-products/tabletmark/. [60] “The Intel 12th Gen Core i9-12900K Review: Hybrid Performance Brings Hybrid Complexity,” https://www.anandtech.com/show/17047/the-intel-12th-gen-core-i912900kreview-hybrid-performance-brings-hybrid-complexity/6. [61] “Virtuoso GitHub Repository,” https://github.com/CMU-SAFARI/Virtuoso. [62] “Wikipedia - Athena,” https://en.wikipedia.org/wiki/Athena. [63] “Wikipedia - Constable,” https://en.wikipedia.org/wiki/Constable. [64] “Wikipedia - Hermes,” https://en.wikipedia.org/wiki/Hermes. [65] “Wikipedia - Pythia,” https://en.wikipedia.org/wiki/Pythia. [66] “AMD Zen2 EPYC 7702P,” 2021, https://www.amd.com/en/products/processors/serve r/epyc/4th-generation-9004-and-8004-series/amd-epyc-9754s.html. [67] “Graviton3,” 2021, https://en.wikichip.org/wiki/annapurna_labs/graviton/graviton3. [68] “ARM Neoverse V2,” 2023, https://chipsandcheese.com/p/hot-chips-2023-arms-neove rse-v2. [69] “AmpereOne® 64-bit Multi-Core Processors,” 2024, https://amperecomputing.com/br iefs/ampereone-family-product-brief. [70] N. AbouGhazaleh, A. Ferreira, C. Rusu, R. Xu, F. Liberato, B. Childers, D. Mosse, and R. Melhem, “Integrated CPU and L2 Cache Voltage Scaling using Machine Learning,” in LCTES, 2007. [71] M. Acacio, J. Gonzalez, J. Garcia, and J. Duato, “A New Scalable Directory Architecture for Large-Scale Multiprocessors,” in HPCA, 2001. [72] A. Addisie, H. Kassa, O. Matthews, and V. Bertacco, “Heterogeneous Memory Subsystem for Natural Graph Analytics,” in IISWC, 2018. [73] S. Aga, S. Jeloka, A. Subramaniyan, S. Narayanasamy, D. Blaauw, and R. Das, “Compute Caches,” in HPCA, 2017. [74] A. Agarwal, R. Simoni, J. L. Hennessy, and M. Horowitz, “An Evaluation of Directory Schemes for Cache Coherence,” in ISCA, 1988. [75] J. Ahn, S. Hong, S. Yoo, O. Mutlu, and K. Choi, “A Scalable Processing-in-memory Accelerator for Parallel Graph Processing,” in ISCA, 2015. [76] J. Ahn, S. Yoo, O. Mutlu, and K. Choi, “PIM-Enabled Instructions: A Low-Overhead, Locality-Aware Processing-in-Memory Architecture,” in ISCA, 2015.

BIBLIOGRAPHY

177

[77] M. Ahn, A. Chang, D. Lee, J. Gim, J. Kim, J. Jung, O. Rebholz, V. Pham, K. T. Malladi, and Y. S. Ki, “Enabling CXL Memory Expansion for In-Memory Database Management Systems,” in DaMoN, 2022. [78] S. Ainsworth and T. M. Jones, “Graph Prefetching Using Data Structure Knowledge,” in ICS, 2016. [79] S. Ainsworth and T. M. Jones, “Software Prefetching for Indirect Memory Accesses,” in CGO, 2017. [80] S. Ainsworth and T. M. Jones, “Software Prefetching for Indirect Memory Accesses,” in CGO, 2017. [81] B. Akin, F. Franchetti, and J. C. Hoe, “Data Reorganization in Memory Using 3DStacked DRAM,” in ISCA, 2015. [82] B. Akın, J. C. Hoe, and F. Franchetti, “HAMLeT: Hardware Accelerated Memory Layout Transform within 3D-Stacked DRAM,” in HPEC, 2014. [83] H. Akkary, S. T. Srinivasan, R. Koltur, Y. Patil, and W. Refaai, “Perceptron-Based Branch Confidence Estimation,” in HPCA, 2004. [84] A. Alameldeen and D. Wood, “Adaptive Cache Compression for High-Performance Processors,” in ISCA, 2004. [85] A. Alameldeen and D. Wood, “Frequent Pattern Compression: A Significance-Based Compression Scheme for L2 Caches,” University of Wisconsin-Madison Department of Computer Sciences, Tech. Rep., 2004. [86] L. M. AlBarakat, P. V. Gratz, and D. A. Jiménez, “MTB-Fetch: Multithreading Aware Hardware Prefetching for Chip Multiprocessors,” in IEEE CAL, 2018. [87] L. M. AlBarakat, P. V. Gratz, and D. A. Jiménez, “SB-Fetch: Synchronization Aware Hardware Prefetching for Chip Multiprocessors,” in ICS, E. Ayguadé, W. W. Hwu, R. M. Badia, and H. P. Hofstee, Eds., 2020. [88] L. M. AlBarakat, P. V. Gratz, and D. A. Jiménez, “SLAP-CC: Set-Level Adaptive Prefetching for Compressed Caches,” in ICCD, 2022. [89] J. S. Albus, “A New Approach to Manipulator Control: The Cerebellar Model Articulation Controller (CMAC),” in Journal of Dynamic Systems, Measurement, and Control, 1975. [90] M. F. Ali, A. Jaiswal, and K. Roy, “In-Memory Low-Cost Bit-Serial Addition Using Commodity DRAM Technology,” in TCAS-I, 2019. [91] N. Alkassab, C.-T. Huang, and T. L. Botran, “DeePref: Deep Reinforcement Learning For Video Prefetching In Content Delivery Networks,” in ICCCN, 2024. [92] M. Alser, Z. Bingöl, D. S. Cali, J. Kim, S. Ghose, C. Alkan, and O. Mutlu, “Accelerating Genome Analysis: A Primer on an Ongoing Journey,” in IEEE Micro, 2020.

178

BIBLIOGRAPHY

[93] M. A. Z. Alves, M. Diener, P. C. Santos, and L. Carro, “Large Vector Extensions Inside the HMC,” in DATE, 2016. [94] M. A. Z. Alves, P. C. Santos, M. Diener, and L. Carro, “Opportunities and Challenges of Performing Vector Operations Inside the DRAM,” in MEMSYS, 2015. [95] M. A. Z. Alves, P. C. Santos, F. B. Moreira, M. Diener, and L. Carro, “Saving Memory Movements Through Vector Processing in the DRAM,” in CASES, 2015. [96] M. Alves, “Increasing Energy Efficiency of Processor Caches via Line Usage Predictors,” Ph.D. dissertation, Universidade Federal do Rio Grande do Sul, 2014. [97] D. W. Anderson, F. J. Sparacio, and R. M. Tomasulo, “The IBM System/360 Model 91: Machine Philosophy and Instruction-Handling,” in IBM Journal of Research and Development, 1967. [98] S. Angizi and D. Fan, “GraphiDe: A Graph Processing Accelerator Leveraging InDRAM-Computing,” in GLSVLSI, 2019. [99] N. Ardalani, C. Lestourgeon, K. Sankaralingam, and X. Zhu, “Cross-architecture Performance Prediction (XAPP) using CPU Code to Predict GPU Performance,” in MICRO, 2015. [100] N. Ardalani, U. Thakker, A. Albarghouthi, and K. Sankaralingam, “A Static AnalysisBased Cross-Architecture Performance Prediction using Machine Learning,” in arXiv preprint arXiv:1906.07840, 2019. [101] H. Asghari-Moghaddam, A. Farmahini-Farahani, K. Morrow et al., “Near-DRAM Acceleration with Single-ISA Heterogeneous Processing in Standard Memory Modules,” in IEEE Micro, 2016. [102] H. Asghari-Moghaddam, Y. H. Son, J. H. Ahn, and N. S. Kim, “Chameleon: Versatile and Practical Near-DRAM Acceleration Architecture for Large Memory Systems,” in MICRO, 2016. [103] T. Asheim, T. A. Khan, B. Kasicki, and R. Kumar, “Impact of Microarchitectural State Reuse on Serverless Functions,” in WoSC, 2022. [104] R. Ausavarungnirun, K. K.-W. Chang, L. Subramanian, G. H. Loh, and O. Mutlu, “Staged Memory Scheduling: Achieving High Performance and Scalability in Heterogeneous Systems,” in ISCA, 2012. [105] T. M. Austin, D. N. Pnevmatikatos, and G. S. Sohi, “Streamlining Data Cache Access with Fast Address Calculation,” in ISCA, 1995. [106] T. M. Austin and G. S. Sohi, “Zero-Cycle Loads: Microarchitecture Support for Reducing Load Latency,” in MICRO, 1995. [107] A. J. Awan, M. Brorsson, V. Vlassov, and E. Ayguade, “Performance Characterization of In-Memory Data Analytics on a Modern Cloud Server,” in BdCloud, 2015.

BIBLIOGRAPHY

179

[108] A. J. Awan, M. Brorsson, V. Vlassov, and E. Ayguade, “Micro-Architectural Characterization of Apache Spark on Batch and Stream Processing Workloads,” in BDCloud, 2016. [109] A. J. Awan, M. Ohara, E. Ayguade, K. Ishizaki, M. Brorsson, and V. Vlassov, “Identifying the Potential of near Data Processing for Apache Spark,” in MEMSYS, 2017. [110] G. Ayers, J. H. Ahn, C. Kozyrakis, and P. Ranganathan, “Memory Hierarchy for Web Search,” in HPCA, 2018. [111] G. Ayers, N. P. Nagendra, D. I. August, H. K. Cho, S. Kanev, C. Kozyrakis, T. Krishnamurthy, H. Litz, T. Moseley, and P. Ranganathan, “AsmDB: Understanding and Mitigating Front-End Stalls in Warehouse-Scale Computers,” in ISCA, 2019. [112] E. Azarkhish, C. Pfister, D. Rossi, I. Loi, and L. Benini, “Logic-Base Interconnect Design for Near Memory Computing in the Smart Memory Cube,” in IEEE VLSI, 2016. [113] E. Azarkhish, D. Rossi, I. Loi, and L. Benini, “A Case for Near Memory Computation Inside the Smart Memory Cube,” in EMS, 2016. [114] E. Azarkhish, D. Rossi, I. Loi, and L. Benini, “Neurostream: Scalable and Energy Efficient Deep Learning with Smart Memory Cubes,” in TPDS, 2018. [115] O. O. Babarinsa and S. Idreos, “JAFAR: Near-Data Processing for Databases,” in SIGMOD, 2015. [116] L. Backes and D. A. Jiménez, “The Impact of Cache Inclusion Policies on Cache Management Techniques,” in MEMSYS, 2019. [117] J.-L. Baer and T.-F. Chen, “An Effective On-chip Preloading Scheme to Reduce Data Access Penalty,” in SC, 1991. [118] Y. Bai, V. W. Lee, and E. Ipek, “Voltage Regulator Efficiency Aware Power Management,” in ASPLOS, 2017. [119] M. Bakhshalipour, P. Lotfi-Kamran, and H. Sarbazi-Azad, “Domino Temporal Data Prefetcher,” in HPCA, 2018. [120] M. Bakhshalipour, M. Shakerinava, P. Lotfi-Kamran, and H. Sarbazi-Azad, “Bingo Spatial Data Prefetcher,” in HPCA, 2019. [121] V. Balaji, N. C. Crago, A. Jaleel, and B. Lucia, “P-OPT: Practical Optimal Cache Replacement for Graph Analytics,” in HPCA, 2021. [122] A. Balasubramanian, A. Kumar, Y. Liu, H. Cao, S. Venkataraman, and A. Akella, “Accelerating Deep Learning Inference via Learned Caches,” 2021. [123] R. Balasubramonian, A. B. Kahng, N. Muralimanohar, A. Shafiee, and V. Srinivas, “CACTI 7: New Tools for Interconnect Exploration in Innovative Off-Chip Memories,” in TACO, 2017. [124] I. Baldini, S. J. Fink, and E. Altman, “Predicting GPU Performance from CPU Runs using Machine Learning,” in SBAC-PAD, 2014.

180

BIBLIOGRAPHY

[125] S. Bandishte, J. Gaur, Z. Sperber, L. Rappoport, A. Yoaz, and S. Subramoney, “Focused Value Prediction,” in ISCA, 2020. [126] A. Basu, M. D. Hill, and M. M. Swift, “Reducing Memory Reference Energy with Opportunistic Virtual Caching,” in ISCA, 2012. [127] A. Basu, N. Kırman, M. Kırman, M. Chaudhuri, and J. F. Martínez, “Scavenger: A New Last-Level Cache Architecture with Global Block Priority,” in MICRO, 2007. [128] N. N. Bavarsad, H. M. Makrani, H. Sayadi, L. Landis, S. Rafatirad, and H. Homayoun, “HosNa: A DPC++ Benchmark Suite for Heterogeneous Architectures,” in ICCD, 2021. [129] B. M. Beckmann and D. A. Wood, “Managing Wire Delay in Large Chip-Multiprocessor Caches,” in MICRO, 2004. [130] M. Bekerman, S. Jourdan, R. Ronen, G. Kirshenboim, L. Rappoport, A. Yoaz, and U. Weiser, “Correlated Load-Address Predictors,” in ISCA, 1999. [131] M. Bekerman, A. Yoaz, F. Gabbay, S. Jourdan, M. Kalaev, and R. Ronen, “Early Load Address Resolution via Register Tracking,” in ISCA, 2000. [132] G. B. Bell, K. M. Lepak, and M. H. Lipasti, “Characterization of Silent Stores,” in PACT, 2000. [133] R. Bera, K. Kanellopoulos, S. Balachandran, D. Novo, A. Olgun, M. Sadrosadati, and O. Mutlu, “Hermes: Accelerating Long-Latency Load Requests via Perceptron-Based Off-Chip Load Prediction,” in MICRO, 2022. [134] R. Bera, K. Kanellopoulos, S. Balachandran, D. Novo, A. Olgun, M. Sadrosadati, and O. Mutlu, “Hermes: Accelerating Long-Latency Load Requests via Perceptron-Based Off-Chip Load Prediction - Extended Version,” https://arxiv.org/abs/2209.00188, 2022. [135] R. Bera, K. Kanellopoulos, and O. Mutlu, “The 4th Data Prefetching Championship (DPC4),” https://sites.google.com/view/dpc4-2026/home. [136] R. Bera, K. Kanellopoulos, A. Nori, T. Shahroodi, S. Subramoney, and O. Mutlu, “Pythia: A Customizable Hardware Prefetching Framework Using Online Reinforcement Learning,” in MICRO, 2021. [137] R. Bera, K. Kanellopoulos, A. Nori, T. Shahroodi, S. Subramoney, and O. Mutlu, “Pythia: A Customizable Hardware Prefetching Framework Using Online Reinforcement Learning - Extended Version,” https://arxiv.org/abs/2109.12021, 2021. [138] R. Bera, Z. Lang, C. Hengartner, K. Kanellopoulos, R. Kumar, M. Sadrosadati, and O. Mutlu, “Athena: Synergizing Data Prefetching and Off-Chip Prediction via Online Reinforcement Learning,” in HPCA, 2026. [139] R. Bera, Z. Lang, C. Hengartner, K. Kanellopoulos, R. Kumar, M. Sadrosadati, and O. Mutlu, “Athena: Synergizing Data Prefetching and Off-Chip Prediction via Online Reinforcement Learning - Extended Version,” https://arxiv.org/abs/2601.17615, 2026.

BIBLIOGRAPHY

181

[140] R. Bera, R. Nadig, and O. Mutlu, “Machine Learning-Driven Intelligent Memory System Design: From On-Chip Caches to Storage,” in IEEE Micro, 2026. [141] R. Bera, A. V. Nori, O. Mutlu, and S. Subramoney, “DSPatch: Dual Spatial Pattern Prefetcher,” in MICRO, 2019. [142] R. Bera, A. Ranganathan, J. Rakshit, S. Mahto, A. V. Nori, J. Gaur, A. Olgun, K. Kanellopoulos, M. Sadrosadati, S. Subramoney, and O. Mutlu, “Constable: Improving Performance and Power Efficiency by Safely Eliminating Load Execution - Extended Version,” https://arxiv.org/abs/2406.18786, 2024. [143] R. Bera, A. Ranganathan, J. Rakshit, S. Mahto, A. V. Nori, J. Gaur, A. Olgun, K. Kanellopoulos, M. Sadrosadati, S. Subramoney, and O. Mutlu, “Constable: Improving Performance and Power Efficiency by Safely Eliminating Load Instruction Execution,” in ISCA, 2024. [144] J. Bergstra and Y. Bengio, “Random Search for Hyper-Parameter Optimization,” in JMLR, 2012. [145] M. Besta, S. M. Hassan, S. Yalamanchili, R. Ausavarungnirun, O. Mutlu, and T. Hoefler, “Slim NoC: A Low-diameter On-chip Network Topology for High Energy Efficiency and Scalability,” in ASPLOS, 2018. [146] M. Besta, R. Kanakagiri, G. Kwasniewski, R. Ausavarungnirun, J. Beránek, K. Kanellopoulos, K. Janda, Z. Vonarburg-Shmaria, L. Gianinazzi, I. Stefan et al., “SISA: SetCentric Instruction Set Architecture for Graph Mining on Processing-in-Memory Systems,” in MICRO, 2021. [147] M. Besta, Z. Vonarburg-Shmaria, Y. Schaffner, L. Schwarz, G. Kwasniewski, L. Gianinazzi, J. Beranek, K. Janda, T. Holenstein, S. Leisinger et al., “GraphMineSuite: Enabling High-Performance and Programmable Graph Mining Algorithms with Set Algebra,” in Proceedings of the VLDB Endowment, 2021. [148] E. Bhatia, G. Chacon, S. Pugsley, E. Teran, P. V. Gratz, and D. A. Jiménez, “PerceptronBased Prefetch Filtering,” in ISCA, 2019. [149] C. Bienia, S. Kumar, J. P. Singh, and K. Li, “The PARSEC Benchmark Suite: Characterization and Architectural Implications,” in PACT, 2008. [150] N. Black, “Daytripper: Dynamic Translation for Intel’s Loop Stream Decoder.” [151] C. Block, G. Gerogiannis, and J. Torrellas, “Micro-MAMA: Multi-Agent Reinforcement Learning for Multicore Prefetching,” in MICRO, 2025. [152] C. Block, P. Palacios, A. Farrell, G. Gerogiannis, and J. Torrellas, “Performance-Driven Composite Prefetching with Bandits,” in 4th Data Prefetching Championship (DPC4), 2026. [153] B. H. Bloom, “Space/time Trade-Offs in Hash Coding with Allowable Errors,” in Communications of the ACM, 1970.

182

BIBLIOGRAPHY

[154] L. Bloom, M. Cohen, and S. Porter, “Considerations in the Design of A Computer With High Logic-to-Memory Speed Ratio,” in Proc. Gigacycle Computing Systems, 1962. [155] F. Bodin and A. Seznec, “Skewed-Associativity Improves Performance and Enhances Predictability,” in IEEE Transactions on Computers, 1997. [156] A. Boroumand, “Practical Mechanisms for Reducing Processor-Memory Data Movement in Modern Workloads,” Ph.D. dissertation, Carnegie Mellon University, 2020. [157] A. Boroumand, S. Ghose, B. Akin, R. Narayanaswami, G. F. Oliveira, X. Ma, E. Shiu, and O. Mutlu, “Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottlenecks,” in PACT, 2021. [158] A. Boroumand, S. Ghose, B. Akin, R. Narayanaswami, G. F. Oliveira, X. Ma, E. Shiu, and O. Mutlu, “Mitigating Edge Machine Learning Inference Bottlenecks: An Empirical Study on Accelerating Google Edge Models,” arXiv:2103.00768 [cs.AR], 2021. [159] A. Boroumand, S. Ghose, Y. Kim, R. Ausavarungnirun, E. Shiu, R. Thakur, D. Kim, A. Kuusela, A. Knies, P. Ranganathan et al., “Google Workloads for Consumer Devices: Mitigating Data Movement Bottlenecks,” in ASPLOS, 2018. [160] A. Boroumand, S. Ghose, B. Lucia, K. Hsieh, K. Malladi, H. Zheng, and O. Mutlu, “LazyPIM: An Efficient Cache Coherence Mechanism for Processing-in-Memory,” in CAL, 2017. [161] A. Boroumand, S. Ghose, G. F. Oliveira, and O. Mutlu, “Polynesia: Enabling Effective Hybrid Transactional/Analytical Databases with Specialized Hardware/Software CoDesign,” arXiv:2103.00798 [cs.AR], 2021. [162] A. Boroumand, S. Ghose, G. F. Oliveira, and O. Mutlu, “Polynesia: Enabling HighPerformance and Energy-Efficient Hybrid Transactional/Analytical Databases with Hardware/Software Co-Design,” in ICDE, 2022. [163] A. Boroumand, S. Ghose, M. Patel, H. Hassan, B. Lucia, R. Ausavarungnirun, K. Hsieh, N. Hajinazar, K. T. Malladi, H. Zheng et al., “CoNDA: Efficient Cache Coherence Support for Near-Data Accelerators,” in ISCA, 2019. [164] A. Boroumand, S. Ghose, M. Patel, H. Hassan, B. Lucia, N. Hajinazar, K. Hsieh, K. T. Malladi, H. Zheng, and O. Mutlu, “LazyPIM: Efficient Support for Cache Coherence in Processing-in-Memory Architectures,” arXiv:1706.03162 [cs.AR], 2017. [165] P. Bose, A. Buyuktosunoglu, M. R. Dooley, M. S. Floyd, D. S. Ray, and B. J. Ronchetti, “Adaptive Data Prefetch,” US Patent, 2012, filed: 2009-01-15; Published: 2012-04-10. [Online]. Available: https://patents.google.com/patent/US8156287B2/en [166] P. Bose, A. Buyuktosunoglu, V. J. Jimenez Perez, and F. P. O’Connell, “Intelligent Bandwidth Shifting Mechanism,” US Patent, 2017, filed: 2015-01-13; Published: 2017-05-09. [Online]. Available: https://patents.google.com/patent/US9645935/en

BIBLIOGRAPHY

183

[167] F. N. Bostancı, A. Olgun, L. Orosa, A. G. Yağlikçi, J. S. Kim, H. Hassan, Oğuz, and O. Mutlu, “DR-STRaNGe: End-to-End System Design for DRAM-Based True Random Number Generators,” in HPCA, 2022. [168] L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification and Regression Trees. Chapman and Hall/CRC, 2017. [169] M. Bruce, “Arm Neoverse v2 Platform: Leadership Performance and Power Efficiency for Next-Generation Cloud Computing, ML and HPC Workloads,” in Hot Chips, 2023. [170] M. Burtscher and B. G. Zorn, “Exploring Last N Value Prediction,” in PACT, 1999. [171] J. A. Butts and G. Sohi, “Dynamic Dead-Instruction Detection and Elimination,” in ASPLOS, 2002. [172] J. A. Butts and G. S. Sohi, “Characterizing and predicting value degree of use,” in MICRO, 2002. [173] A. Buyuktosunoglu, D. Trilla, B. Abali, D. Berger, C. Walters, and J.-S. Lee, “EnterpriseClass Cache Compression Design,” in HPCA, 2024. [174] B. Calder, G. Reinman, and D. M. Tullsen, “Selective Value Prediction,” in ISCA, 1999. [175] R. N. Calheiros, E. Masoumi, R. Ranjan, and R. Buyya, “Workload Prediction using ARIMA Model and Its Impact on Cloud Applications’ QoS,” in IEEE transactions on cloud computing, 2014. [176] D. S. Cali, G. S. Kalsi, Z. Bingöl, C. Firtina, L. Subramanian, J. S. Kim, R. Ausavarungnirun, M. Alser, J. Gomez-Luna, A. Boroumand et al., “GenASM: A High-Performance, Low-Power Approximate String Matching Acceleration Framework for Genome Sequence Analysis,” in MICRO, 2020. [177] D. Callahan, K. Kennedy, and A. Porterfield, “Software Prefetching,” in ASPLOS, 1991. [178] M. Cekleov and M. Dubois, “Virtual-address Caches. Part 1: Problems and Solutions in Uniprocessors,” in IEEE Micro, 1997. [179] Censier and Feautrier, “A New Solution to Coherence Problems in Multicache Systems,” in IEEE TC, 1978. [180] G. J. Chaitin, “Register Allocation & Spilling via Graph Coloring,” in CC, 1982. [181] K. Chakraborty, P. M. Wells, and G. S. Sohi, “Computation Spreading: Employing Hardware Migration to Specialize CMP Cores On-the-Fly,” in ASPLOS, 2006. [182] J. Chandarlapati and M. Chaudhuri, “LEMap: Controlling Leakage in Large ChipMultiprocessor Caches via Profile-Guided Virtual Address Translation,” in ICCD, 2007. [183] K. Chandrasekar, S. Goossens, C. Weis, M. Koedam, B. Akesson, N. Wehn, and K. Goossens, “Exploiting Expendable Process-Margins in DRAMs for Run-Time Performance Optimization,” in DATE, 2014.

184

BIBLIOGRAPHY

[184] K. K. Chang, “Understanding and Improving the Latency of DRAM-Based Memory Systems,” Ph.D. dissertation, Carnegie Mellon University, 2017. [185] K. K. Chang, A. Kashyap, H. Hassan, S. Ghose, K. Hsieh, D. Lee, T. Li, G. Pekhimenko, S. Khan, and O. Mutlu, “Understanding Latency Variation in Modern DRAM Chips: Experimental Characterization, Analysis, and Optimization,” in SIGMETRICS, 2016. [186] K. K. Chang, P. J. Nair, D. Lee, S. Ghose, M. K. Qureshi, and O. Mutlu, “Low-Cost InterLinked Subarrays (LISA): Enabling Fast Inter-Subarray Data Movement in DRAM,” in HPCA, 2016. [187] K. K. Chang, A. G. Yağlıkçı, S. Ghose, A. Agrawal, N. Chatterjee, A. Kashyap, D. Lee, M. O’Connor, H. Hassan, and O. Mutlu, “Understanding Reduced-Voltage Operation in Modern DRAM Devices: Experimental Characterization, Analysis, and Mechanisms,” in POMACS, 2017. [188] K. K.-W. Chang, D. Lee, Z. Chishti, A. R. Alameldeen, C. Wilkerson, Y. Kim, and O. Mutlu, “Improving DRAM Performance by Parallelizing Refreshes with Accesses,” in HPCA, 2014. [189] R. S. Chappell, J. Stark, S. P. Kim, S. K. Reinhardt, and Y. N. Patt, “Simultaneous Subordinate Microthreading (SSMT),” in ISCA, 1999. [190] R. S. Chappell, F. Tseng, Y. N. Patt, and A. Yoaz, “Difficult-Path Branch Prediction Using Subordinate Microthreads,” in ISCA, 2002. [191] M. Charney, “Correlation-Based Hardware Prefetching,” Ph.D. dissertation, Cornell University, 1995. [192] M. J. Charney and T. R. Puzak, “Prefetching and Memory System Behavior of the SPEC95 Benchmark Suite,” in IBM Journal of Research and Development, 1997. [193] M. J. Charney and A. P. Reeves, “Generalized Correlation-Based Hardware Prefetching,” Cornell Univ., Tech. Rep., 1995. [194] M. Chaudhuri, “PageNUCA: Selected Policies for Page-Grain Locality Management in Large Shared Chip-Multiprocessor Caches,” in HPCA, 2009. [195] M. Chaudhuri, “Pseudo-LIFO: The Foundation of a New Family of Replacement Policies for Last-Level Caches,” in MICRO, 2009. [196] M. Chaudhuri, “Zero Inclusion Victim: Isolating Core Caches from Inclusive Last-Level Cache Evictions,” in ISCA, 2021. [197] M. Chaudhuri, J. Gaur, N. Bashyam, S. Subramoney, and J. Nuzman, “Introducing Hierarchy-Awareness in Replacement and Bypass Algorithms for Last-Level Caches,” in PACT, 2012. [198] M. Chaudhuri, J. Gaur, and S. Subramoney, “Bandwidth-Aware Last-Level Caching: Efficiently Coordinating Off-Chip Read and Write Bandwidth,” in ICCD, 2019.

BIBLIOGRAPHY

185

[199] J. Chen, T. Zhang, X. Liu, X. Zhang, P. Qu, and Y. Zhang, “Emender: Optimizing Prefetch Priority and Throttling in VBerti+Pythia,” in 4th Data Prefetching Championship (DPC4), 2026. [200] K. Chen, R. Nadig, M. Frouzakis, N. M. Ghiasi, Y. Liang, H. Mao, J. Park, M. Sadrosadati, and O. Mutlu, “REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing,” in ISCA, 2025. [201] T.-F. Chen and J.-L. Baer, “Effective Hardware-Based Data Prefetching for HighPerformance Processors,” in IEEE TC, 1995. [202] W. Y.-W. Chen, “Data Preload for Superscalar and VLIW Processors,” Ph.D. dissertation, University of Illinois at Urbana-Champaign, 1993. [203] X. Chen, L. Yang, R. P. Dick, L. Shang, and H. Lekatsas, “C-Pack: A High-Performance Microprocessor Cache Compression Algorithm,” in IEEE TVLSI, 2009. [204] Y. Chen, A. Hajiabadi, L. Pei, and T. E. Carlson, “PREFETCHX: Cross-Core CacheAgnostic Prefetcher-based Side-Channel Attacks,” in HPCA, 2024. [205] Z. Chen, C. Wu, Y. Gu, R. Jia, J. Li, and M. Guo, “Gaze into the Pattern: Characterizing Spatial Patterns with Internal Temporal Correlations for Hardware Prefetching,” in HPCA, 2025. [206] C.-K. Cheng, A. B. Kahng, I. Kang, and L. Wang, “Replace: Advancing Solution Quality and Routability Validation in Global Placement,” in IEEE TCAD, 2018. [207] C.-K. Cheng, A. B. Kahng, S. Kundu, Y. Wang, and Z. Wang, “Assessment of Reinforcement Learning for Macro Placement,” in ISPD, 2023. [208] H. Cheng, J. Zhao, J. Sampson, M. J. Irwin, A. Jaleel, Y. Lu, and Y. Xie, “LAP: LoopBlock Aware Inclusion Properties for Energy-Efficient Asymmetric Last Level Caches,” in ISCA, 2016. [209] C.-Y. Cher, A. Hosking, and T. N. Vijaykumar, “Software Prefetching for Mark-Sweep Garbage Collection: Hardware Analysis and Software Redesign,” in ASPLOS, 2004. [210] C.-H. Chi and H. Dietz, “Improving Cache Performance by Selective Cache Bypass,” in HICSS, 1989. [211] P. Chi, S. Li, C. Xu, T. Zhang, J. Zhao, Y. Liu, Y. Wang, and Y. Xie, “PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAMBased Main Memory,” in ISCA, 2016. [212] T. M. Chilimbi and M. Hirzel, “Dynamic Hot Data Stream Prefetching for GeneralPurpose Programs,” in PLDI, 2002. [213] A. Chilukuri and S. Akram, “Analyzing and Improving the Scalability of In-Memory Indices for Managed Search Engines,” in ISMM, 2023.

186

BIBLIOGRAPHY

[214] S. Cho, H. Choi, E. Park, H. Shin, and S. Yoo, “McDRAM v2: In-Dynamic Random Access Memory Systolic Array Accelerator to Address the Large Model Problem in Deep Neural Networks on the Edge,” in IEEE Access, 2020. [215] J. Choi, W. Shin, J. Jang, J. Suh, Y. Kwon, Y. Moon, and L.-S. Kim, “Multiple Clone Row DRAM: A Low Latency and Area Optimized DRAM,” in ISCA, 2015. [216] Y. Chou, “Low-cost Epoch-based Correlation Prefetching for Commercial Applications,” in MICRO, 2007. [217] Y. Chou, B. Fahs, and S. Abraham, “Microarchitecture Optimizations for Exploiting Memory-Level Parallelism,” in ISCA, 2004. [218] Y. Chou, L. Spracklen, and S. G. Abraham, “Store Memory-Level Parallelism Optimizations for Commercial Applications,” in MICRO, 2005. [219] Y. C. Chou, “Selectively Dropping Prefetch Requests Based on Prefetch Accuracy Information,” US Patent, 2014, filed: 2011-11-29; Published: 2014-11-18. [Online]. Available: https://patents.google.com/patent/US8892822B2/en [220] G. Z. Chrysos and J. S. Emer, “Memory Dependence Prediction Using Store Sets,” in ISCA, 1998. [221] D. Citron and D. Feitelson, “Revisiting Instruction Level Reuse,” in WDDD, 2002. [222] D. Citron, D. Feitelson, and L. Rudolph, “Accelerating Multi-media Processing by Implementing Memoing in Multiplication and Division Units,” in ASPLOS, 1998. [223] D. Citron and D. G. Feitelson, “Hardware Memoization of Mathematical and Trigonometric Functions,” in Tech. Rep. TR-2000-5, 2000. [224] M. Clark, A. Kodi, R. Bunescu, and A. Louri, “LEAD: Learning-enabled Energy-aware Dynamic Voltage/Frequency Scaling in NoCs,” in DAC, 2018. [225] M. Clark, “ A New, High Performance x86 Core Design from AMD,” in Hot Chips, 2016. [226] R. Cochran, C. Hankendi, A. K. Coskun, and S. Reda, “Pack & Cap: Adaptive DVFS and Thread Packing Under Power Caps,” in MICRO, 2011. [227] J. D. Collins, D. M. Tullsen, H. Wang, and J. P. Shen, “Dynamic Speculative Precomputation,” in MICRO, 2001. [228] J. D. Collins, H. Wang, D. M. Tullsen, C. Hughes, Y.-F. Lee, D. Lavery, and J. P. Shen, “Speculative Precomputation: Long-range Prefetching of Delinquent Loads,” in ISCA, 2001. [229] D. A. Connors and W.-m. W. Hwu, “Compiler-Directed Dynamic Computation Reuse: Rationale and Initial Results,” in MICRO, 1999. [230] D. A. Connors, H. C. Hunter, B.-C. Cheng, and W.-m. W. Hwu, “Hardware Support for Dynamic Activation of Compiler-Directed Computation Reuse,” in ASPLOS, 2000.

BIBLIOGRAPHY

187

[231] D. A. Connors, Eliminating Dynamic Computation Redundancy. University of Illinois at Urbana-Champaign, 2000. [232] R. Cooksey, S. Jourdan, and D. Grunwald, “A Stateless, Content-Directed Data Prefetching Mechanism,” in ASPLOS, 2002. [233] S. Corda, B. Veenboer, A. J. Awan, A. Kumar, R. Jordans, and H. Corporaal, “Near Memory Acceleration on High Resolution Radio Astronomy Imaging,” in MECO, 2020. [234] C. Cortes and V. Vapnik, “Support-Vector Networks,” in Machine Learning, 1995. [235] E. Cortez, A. Bonde, A. Muzio, M. Russinovich, M. Fontoura, and R. Bianchini, “Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms,” in SOSP, 2017. [236] S. M. P. D., H. Yu, H. Huang, and D. Xu, “A Q-learning Based Self-Adaptive I/O Communication for 2.5 D Integrated Many-Core Microprocessor and Memory,” in IEEE TC, 2015. [237] F. Dahlgren, M. Dubois, and P. Stenström, “Sequential Hardware Prefetching in SharedMemory Multiprocessors,” in IEEE TPDS, 1995. [238] D. Dai, F. S. Bao, J. Zhou, and Y. Chen, “Block2vec: A Deep Learning Strategy on Mining Block Correlations in Storage Systems,” in ICPPW, 2016. [239] G. Dai, T. Huang, Y. Chi, J. Zhao, G. Sun, Y. Liu, Y. Wang, Y. Xie, and H. Yang, “GraphH: A Processing-in-Memory Architecture for Large-Scale Graph Processing,” in TCAD, 2018. [240] F. Damiani and P. Giannini, “Automatic Useless-Code Elimination for HOT Functional Programs,” in Journal of Functional Programming, 2000. [Online]. Available: https://api.semanticscholar.org/CorpusID:5967783 [241] J. P. C. de Lima, P. C. Santos, M. A. Alves, A. Beck, and L. Carro, “Design Space Exploration for PIM Architectures in 3D-Stacked Memories,” in CF, 2018. [242] Q. Deng, L. Jiang, Y. Zhang, M. Zhang, and J. Yang, “DrAcc: A DRAM Based Accelerator for Accurate CNN Inference,” in DAC, 2018. [243] Z. Deng, L. Zhang, N. Mishra, H. Hoffmann, and F. T. Chong, “Memory Cocktail Therapy: A General Learning-Based Framework to Optimize Dynamic Tradeoffs in Nvms,” in MICRO, 2017. [244] A. Denzler, R. Bera, N. Hajinazar, G. Singh, G. F. Oliveira, J. Gómez-Luna, and O. Mutlu, “Casper: Accelerating Stencil Computation using Near-Cache Processing,” arXiv:2112.14216 [cs.AR], 2021. [245] A. Denzler, G. F. Oliveira, N. Hajinazar, R. Bera, G. Singh, J. Gómez-Luna, and O. Mutlu, “Casper: Accelerating Stencil Computations using Near-Cache Processing,” in IEEE Access, 2023.

188

BIBLIOGRAPHY

[246] A. Deshmukh and Y. N. Patt, “Criticality Driven Fetch,” in MICRO, 2021. [247] F. Devaux, “The True Processing in Memory Accelerator,” in Hot Chips, 2019. [248] S. Diab, A. Nassereldine, M. Alser, J. Gómez Luna, O. Mutlu, and I. El Hajj, “A Framework for High-Throughput Sequence Alignment using Real Processing-in-Memory Systems,” in Bioinformatics, 2023. [249] D. DiTomaso, T. Boraten, A. Kodi, and A. Louri, “Dynamic Error Mitigation in NoCs Using Intelligent Prediction Techniques,” in MICRO, 2016. [250] D. DiTomaso, A. Sikder, A. Kodi, and A. Louri, “Machine Learning Enabled Poweraware Network-on-chip Design,” in DATE, 2017. [251] D. R. Ditzel and H. R. McLellan, “Branch Folding in the CRISP Microprocessor: Reducing Branch Delay to Zero,” in ISCA, 1987. [252] X. Dong, N. P. Jouppi, and Y. Xie, “A Circuit-Architecture Co-Optimization Framework for Exploring Nonvolatile Memory Hierarchies,” in TACO, 2013. [253] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale,” in ICLR, 2021. [254] N. Drach, A. Gefflaut, P. Joubert, and A. Seznec, “About Cache Associativity in LowCost Shared Memory Multi-Microprocessors,” in Parallel Processing Letters, 1995. [255] N. Drach and A. Seznec, “Semi-Unified Caches,” in ICPP, 1993. [256] M. Drumond, A. Daglis, N. Mirzadeh, D. Ustiugov, J. Picorel, B. Falsafi, B. Grot, and D. Pnevmatikatos, “The Mondrian Data Engine,” in ISCA, 2017. [257] C. Dubach, T. M. Jones, E. V. Bonilla, and M. F. O’Boyle, “A Predictive Model for Dynamic Microarchitectural Adaptivity Control,” in MICRO, 2010. [258] M. Dubois and Y. Song, “Assisted Execution,” in University of Southern California CENG Technical Report, 1998. [259] J. Dundas and T. Mudge, “Improving Data Cache Performance by Pre-executing Instructions Under a Cache Miss,” in ICS, 1997. [260] Q. Duong, A. Jain, and C. Lin, “A New Formulation of Neural Data Prefetching,” in ISCA, 2024. [261] J. Dusser, T. Piquet, and A. Seznec, “Zero-Content Augmented Caches,” in ICS, 2009. [262] J. Dusser and A. Seznec, “Decoupled Zero-Compressed Memory,” in HiPEAC, 2011. [263] H. Dybdahl and P. Stenström, “Enhancing Last-Level Cache Performance by Block Bypassing and Early Miss Determination,” in ACSAC, 2006.

BIBLIOGRAPHY

189

[264] E. Ebrahimi, C. J. Lee, O. Mutlu, and Y. N. Patt, “Fairness via Source Throttling: A Configurable and High-performance Fairness Substrate for Multi-core Memory Systems,” in ASPLOS, 2010. [265] E. Ebrahimi, C. J. Lee, O. Mutlu, and Y. N. Patt, “Prefetch-aware Shared Resource Management for Multi-core Systems,” in ISCA, 2011. [266] E. Ebrahimi, R. Miftakhutdinov, C. Fallin, C. J. Lee, J. A. Joao, O. Mutlu, and Y. N. Patt, “Parallel Application Memory Scheduling,” in MICRO, 2011. [267] E. Ebrahimi, O. Mutlu, C. J. Lee, and Y. N. Patt, “Coordinated Control of Multiple Prefetchers in Multi-Core Systems,” in MICRO, 2009. [268] E. Ebrahimi, O. Mutlu, and Y. N. Patt, “Techniques for Bandwidth-Efficient Prefetching of Linked Data Structures in Hybrid Prefetching Systems,” in HPCA, 2009. [269] M. Ebrahimi, M. Daneshtalab, F. Farahnakian, J. Plosila, P. Liljeberg, M. Palesi, and H. Tenhunen, “HARAQ: Congestion-aware Learning Model for Highly Adaptive Routing Algorithm in On-Chip Networks,” in NOCS, 2012. [270] C. Eckert, X. Wang, J. Wang, A. Subramaniyan, R. Iyer, D. Sylvester, D. Blaauw, and R. Das, “Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networks,” in ISCA, 2018. [271] S. J. Eggers, J. S. Emer, H. M. Levy, J. L. Lo, R. L. Stamm, and D. M. Tullsen, “Simultaneous Multithreading: A Platform for Next-Generation Processors,” in IEEE Micro, 1997. [272] D. Elliott, W. Snelgrove, and M. Stumm, “Computational Ram: A Memory-SIMD Hybrid and its Application to DSP,” in IEEE CICC, 1992. [273] D. G. Elliott, M. Stumm, W. M. Snelgrove et al., “Computational RAM: Implementing Processors in Memory,” in Design and Test of Computers, 1999. [274] F. Eris, M. Louis, K. Eris, J. Abellán, and A. Joshi, “Puppeteer: A Random Forest Based Manager for Hardware Prefetchers Across the Memory Hierarchy,” in TACO, 2022. [275] S. Esteves, H. Galhardas, and L. Veiga, “Adaptive Execution of Continuous and DataIntensive Workflows with Machine Learning,” in Middleware, 2018. [276] S. Eyerman, K. Hoste, and L. Eeckhout, “Mechanistic-empirical Processor Performance Modeling for Constructing CPI Stacks on Real Hardware,” in ISPASS, 2011. [277] B. Fahs, S. Bose, M. Crum, B. Slechta, F. Spadini, T. Tung, S. J. Patel, and S. S. Lumetta, “Performance Characterization of a Hardware Mechanism for Dynamic Optimization,” in MICRO, 2001. [278] B. Fahs, T. Rafacz, S. J. Patel, and S. S. Lumetta, “Continuous Optimization,” in ISCA, 2005. [279] P. Faldu, J. Diamond, and B. Grot, “Domain-Specialized Cache Management for Graph Analytics,” in HPCA, 2020.

190

BIBLIOGRAPHY

[280] P. Faldu and B. Grot, “Leeway: Addressing Variability in Dead-Block Prediction for Last-Level Caches,” in PACT, 2017. [281] B. Falsafi, M. Ferdman, and B. Grot, “Server Architecture From Enterprise to PostMoore,” in IEEE Micro, 2024. [282] J. Fang, J. Li, and Z. Teng, “DHCM: A Dynamic Hierarchy Coordination Mechanism for Memory Optimization,” in J Supercomput., 2025. [283] A. Farmahini-Farahani, J. H. Ahn, K. Compton, and N. S. Kim, “DRAMA: An Architecture for Accelerated Processing Near Memory,” in IEEE CAL, 2014. [284] A. Farmahini-Farahani, J. H. Ahn, K. Morrow, and N. S. Kim, “NDA: Near-DRAM Acceleration Architecture Leveraging Commodity DRAM Devices and Standard Memory Modules,” in HPCA, 2015. [285] A. Fedorova, D. Vengerov, and D. Doucette, “Operating System Scheduling on Heterogeneous Core Systems,” in Proceedings of the Workshop on Operating System Support for Heterogeneous Multicore Architectures, 2007. [286] M. Ferdman, A. Adileh, O. Kocberber, S. Volos, M. Alisafaee, D. Jevdjic, C. Kaynak, A. D. Popescu, A. Ailamaki, and B. Falsafi, “Clearing the Clouds: A Study of Emerging Scale-out Workloads on Modern Hardware,” in ASPLOS, 2012. [287] M. Ferdman, A. Adileh, O. Kocberber, S. Volos, M. Alisafaee, D. Jevdjic, C. Kaynak, A. D. Popescu, A. Ailamaki, and B. Falsafi, “Quantifying the Mismatch between Emerging Scale-Out Applications and Modern Processors,” in TOCS, 2012. [288] M. Ferdman, A. Adileh, O. Kocberber, S. Volos, M. Alisafaee, D. Jevdjic, C. Kaynak, A. D. Popescu, A. Ailamaki, and B. Falsafi, “A Case for Specialized Processors for Scale-Out Workloads,” in IEEE Micro, 2014. [289] M. Ferdman and B. Falsafi, “Last-touch Correlated Data Streaming,” in ISPASS, 2007. [290] M. Ferdman, S. Somogyi, and B. Falsafi, “Spatial Memory Streaming with Rotated Patterns,” in In 1st JILP Data Prefetching Championship, 2009. [291] I. Fernandez, R. Quislant, E. Gutiérrez, O. Plata, C. Giannoula, M. Alser, J. Gómez-Luna, and O. Mutlu, “NATSA: A Near-Data Processing Accelerator for Time Series Analysis,” in ICCD, 2020. [292] J. D. Ferreira, G. Falcao, J. Gómez-Luna, M. Alser, L. Orosa, M. Sadrosadati, J. S. Kim, G. F. Oliveira, T. Shahroodi, A. Nori et al., “pLUTo: In-DRAM Lookup Tables to Enable Massively Parallel General-Purpose Computation,” arXiv:2104.07699 [cs.AR], 2021. [293] J. D. Ferreira, G. Falcao, J. Gómez-Luna, M. Alser, L. Orosa, M. Sadrosadati, J. S. Kim, G. F. Oliveira, T. Shahroodi, A. Nori et al., “pLUTo: Enabling Massively Parallel Computation in DRAM via Lookup Tables,” in MICRO, 2022. [294] Q. Fettes, M. Clark, R. Bunescu, A. Karanth, and A. Louri, “Dynamic Voltage and Frequency Scaling in NoCs with Supervised and Reinforcement Learning Techniques,” in IEEE TC, 2018.

BIBLIOGRAPHY

191

[295] B. Fields, R. Bodík, and M. D. Hill, “Slack: Maximizing Performance Under Technological Constraints,” in ISCA, 2002. [296] B. Fields, S. Rubin, and R. Bodik, “Focusing Processor Policies via Critical-Path Prediction,” in ISCA, 2001. [297] B. A. Fields, “Using Criticality to Attack Performance Bottlenecks,” Ph.D. dissertation, University of California, Berkeley, 2006. [298] J. A. Fisher and R. Rau, “Instruction-Level Parallel Processing,” in Science, 1991. [299] M. Franklin and G. S. Sohi, “ARB: A Hardware Mechanism for Dynamic Memory Disambiguation,” in IEEE ToC, 1996. [300] D. H. Friendly, S. J. Patel, and Y. N. Patt, “Putting the Fill Unit to Work: Dynamic Optimizations for Trace Cache Microprocessors,” in MICRO, 1998. [301] J. W. C. Fu and J. H. Patel, “Data Prefetching in Multiprocessor Vector Cache Memories,” in ISCA, 1991. [302] J. W. C. Fu, J. H. Patel, and B. L. Janssens, “Stride Directed Prefetching in Scalar Processors,” in MICRO, 1992. [303] D. Fujiki, S. Mahlke, and R. Das, “In-Memory Data Parallel Processor,” in ASPLOS, 2018. [304] D. Fujiki, S. Mahlke, and R. Das, “Duality Cache for Data Parallel Acceleration,” in ISCA, 2019. [305] D. M. Gallagher, W. Y. Chen, S. A. Mahlke, J. C. Gyllenhaal, and W.-m. W. Hwu, “Dynamic Memory Disambiguation Using the Memory Conflict Buffer,” in ASPLOS, 1994. [306] A. Ganapathi, K. Datta, A. Fox, and D. Patterson, “A Case for Machine Learning to Optimize Multicore Performance,” in USENIX HotPar, 2009. [307] F. Gao, G. Tziantzioulis, and D. Wentzlaff, “ComputeDRAM: In-Memory Compute Using Off-the-Shelf DRAMs,” in MICRO, 2019. [308] F. Gao, G. Tziantzioulis, and D. Wentzlaff, “FracDRAM: Fractional Values in Off-theShelf DRAM,” in MICRO, 2022. [309] H. Gao and H. Zhou, “Adaptive Information Processing: An Effective Way to Improve Perceptron Branch Predictors,” in JILP, 2005. [310] M. Gao and C. Kozyrakis, “HRL: Efficient and Flexible Reconfigurable Logic for NearData Processing,” in HPCA, 2016. [311] M. Gao, J. Pu, X. Yang, M. Horowitz, and C. Kozyrakis, “TETRIS: Scalable and Efficient Neural Network Acceleration with 3D Memory,” in ASPLOS, 2017. [312] W. Gao, J. Zhan, L. Wang, C. Luo, D. Zheng, F. Tang, B. Xie, C. Zheng, X. Wen, X. He, H. Ye, and R. Ren, “Data Motifs: a Lens Towards Fully Understanding Big Data and AI Workloads,” in PACT, 2018.

192

BIBLIOGRAPHY

[313] W. Gao, J. Zhan, L. Wang, C. Luo, D. Zheng, X. Wen, R. Ren, C. Zheng, X. He, H. Ye, H. Tang, Z. Cao, S. Zhang, and J. Dai, “BigDataBench: A Scalable and Unified Big Data and AI Benchmark Suite,” in arXiv: Distributed, Parallel, and Cluster Computing, 2018. [314] A. Garg and M. C. Huang, “A Performance-Correctness Explicitly-Decoupled Architecture,” in MICRO, 2008. [315] E. Garza, S. Mirbagher-Ajorpaz, T. A. Khan, and D. A. Jimenez, “Bit-level Perceptron Prediction for Indirect Branches,” in ISCA, 2019. [316] J. Gaur, M. Chaudhuri, and S. Subramoney, “Bypass and Insertion Algorithms for Exclusive Last-Level Caches,” in ISCA, 2011. [317] J. Gaur, M. Chaudhuri, and S. Subramoney, “Bypass and Insertion Algorithms for Exclusive Last-Level Caches,” in ISCA, 2011. [318] J. Gaur, R. Srinivasan, S. Subramoney, and M. Chaudhuri, “Efficient Management of Last-Level Caches in Graphics Processors for 3D Scene Rendering Workloads,” in MICRO, 2013. [319] G. Gerogiannis and J. Torrellas, “Micro-Armed Bandit: Lightweight & Reusable Reinforcement Learning for Microarchitecture Decision-Making,” in MICRO, 2023. [320] N. M. Ghiasi, J. Park, H. Mustafa, J. Kim, A. Olgun, A. Gollwitzer, D. S. Cali, C. Firtina, H. Mao, N. A. Alserr et al., “GenStore: A High-Performance and Energy-Efficient InStorage Computing System for Genome Sequence Analysis,” in ASPLOS, 2022. [321] N. M. Ghiasi, M. Sadrosadati, H. Mustafa, A. Gollwitzer, C. Firtina, J. Eudine, H. Ma, J. Lindegger, M. B. Cavlak, M. Alser et al., “MetaStore: High-Performance Metagenomic Analysis via In-Storage Computing,” in arXiv preprint arXiv:2311.12527, 2023. [322] N. M. Ghiasi, M. Sadrosadati, H. Mustafa, A. Gollwitzer, C. Firtina, J. Eudine, H. Mao, J. Lindegger, M. B. Cavlak, M. Alser et al., “MegIS: High-Performance, Energy-Efficient, and Low-Cost Metagenomic Analysis with In-Storage Processing,” in ISCA, 2024. [323] N. M. Ghiasi, N. Vijaykumar, G. F. Oliveira, L. Orosa, I. Fernandez, M. Sadrosadati, K. Kanellopoulos, N. Hajinazar, J. G. Luna, and O. Mutlu, “ALP: Alleviating CPUMemory Data Movement Overheads in Memory-Centric Systems,” in IEEE T-ETC, 2023. [324] S. Ghose, A. Boroumand, J. S. Kim, J. Gómez-Luna, and O. Mutlu, “Processing-inMemory: A Workload-Driven Perspective,” in IBM JRD, 2019. [325] S. Ghose, H.-G. Lee, and J. F. Martínez, “Improving Memory Scheduling via ProcessorSide Load Criticality Information,” in ISCA, 2013. [326] C. Giannoula, I. Fernandez, J. G. Luna, N. Koziris, G. Goumas, and O. Mutlu, “SparseP: Towards Efficient Sparse Matrix Vector Multiplication on Real Processing-in-Memory Architectures,” in SIGMETRICS, 2022. [327] C. Giannoula, N. Vijaykumar, N. Papadopoulou, V. Karakostas, I. Fernandez, J. GómezLuna, L. Orosa, N. Koziris, G. Goumas, and O. Mutlu, “SynCron: Efficient Synchronization Support for Near-Data-Processing Architectures,” in HPCA, 2021.

BIBLIOGRAPHY

193

[328] N. Gober, G. Chacon, L. Wang, P. V. Gratz, D. A. Jiménez, E. Teran, P. Seth, and J. Kim, “The Championship Simulator: Architectural Simulation for Education and Competition,” in arXiv, 2022. [329] B. Goeman, H. Vandierendonck, and K. de Bosschere, “Differential FCM: Increasing Value Prediction Accuracy by Improving Table Usage Efficiency,” in HPCA, 2001. [330] K. Gogineni, S. S. Dayapule, J. Gómez-Luna, K. Gogineni, P. Wei, T. Lan, M. Sadrosadati, O. Mutlu, and G. Venkataramani, “Swiftrl: Towards Efficient Reinforcement Learning on Real Processing-In-Memory Systems,” in ISPASS, 2024. [331] M. Gokhale, B. Holmes, and K. Iobst, “Processing in Memory: The Terasys Massively Parallel PIM Array,” in Computer, 1995. [332] A. Goldie, A. Mirhoseini, and J. Dean, “That Chip Has Sailed: A Critique of Unfounded Skepticism Around AI for Chip Design,” 2024. [Online]. Available: https://arxiv.org/abs/2411.10053 [333] F. J. Gomez, D. Burger, and R. Miikkulainen, “A Neuro-Evolution Method for Dynamic Resource Allocation on a Chip Multiprocessor,” in IJCNN, 2001. [334] J. Gómez-Luna, I. El Hajj, I. Fernandez, C. Giannoula, G. F. Oliveira, and O. Mutlu, “Benchmarking Memory-Centric Computing Systems: Analysis of Real Processing-inMemory Hardware,” in CUT, 2021. [335] J. Gómez-Luna, I. El Hajj, I. Fernandez, C. Giannoula, G. F. Oliveira, and O. Mutlu, “Benchmarking a New Paradigm: Experimental Analysis and Characterization of a Real Processing-in-Memory System,” in IEEE Access, 2022. [336] J. Gómez-Luna, Y. Guo, S. Brocard, J. Legriel, R. Cimadomo, G. F. Oliveira, G. Singh, and O. Mutlu, “Evaluating Machine Learning Workloads on Memory-Centric Computing Systems,” in ISPASS, 2023. [337] J. Gómez-Luna, I. E. Hajj, I. Fernández, C. Giannoula, G. F. Oliveira, and O. Mutlu, “Benchmarking a New Paradigm: An Experimental Analysis of a Real Processing-inMemory Architecture,” arXiv:2105.03814 [cs.AR], 2021. [338] Z. Gong, X. Gu, and J. Wilkes, “Press: Predictive Elastic Resource Scaling for Cloud Systems,” in CNSM, 2010. [339] A. González, J. Tubella, and C. Molina, “The Performance Potential of Data Value Reuse,” in University of Politecenica of Catalunya Technical Report: UPC-DAC-1998-23, 1998. [340] A. González, J. Tubella, and C. Molina, “Trace-Level Reuse,” in ICPP, 1999. [341] D. Gouk, M. Kwon, H. Bae, S. Lee, and M. Jung, “Memory Pooling With CXL,” in IEEE Micro, 2023. [342] P. Gu, S. Li, D. Stow, R. Barnes, L. Liu, Y. Xie, and E. Kursun, “Leveraging 3D Technologies for Hardware Security: Opportunities and Challenges,” in GLSVLSI, 2016.

194

BIBLIOGRAPHY

[343] P. Gu, X. Xie, Y. Ding, G. Chen, W. Zhang, D. Niu, and Y. Xie, “iPIM: Programmable InMemory Image Processing Accelerator using Near-Bank Architecture,” in ISCA, 2020. [344] Y. Gu, A. Khadem, S. Umesh, N. Liang, X. Servot, O. Mutlu, R. Iyer, and R. Das, “PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference,” in ASPLOS, 2025. [345] N. D. Gulur, R. Manikantan, M. Mehendale, and R. Govindarajan, “Multiple Sub-Row Buffers in DRAM: Unlocking Performance and Energy Improvement Opportunities,” in ICS, 2012. [346] Q. Guo, N. Alachiotis, B. Akin, F. Sadi, G. Xu, T. M. Low, L. Pileggi, J. C. Hoe, and F. Franchetti, “3D-Stacked Memory-Side Acceleration: Accelerator and System Design,” in WoNDP, 2014. [347] A. Gupta, W.-D. Weber, and T. Mowry, “Reducing Memory and Traffic Requirements for Scalable Directory-Based Cache Coherence Schemes,” in ICPP, 1992. [348] I. Guyon and A. Elisseeff, “An Introduction to Variable and Feature Selection,” in Journal of machine learning research, 2003. [349] W. Haaswijk, E. Collins, B. Seguin, M. Soeken, F. Kaplan, S. Süsstrunk, and G. De Micheli, “Deep Learning for Logic Optimization Algorithms,” in ISCAS, 2018. [350] R. Hadidi, L. Nai, H. Kim, and H. Kim, “CAIRO: A Compiler-Assisted Technique for Enabling Instruction-Level Offloading of Processing-in-Memory,” in TACO, 2017. [351] J. Haj-Yahya, J. Park, R. Bera, R. Bera, T. Shahroodi, J. S. Kim, E. Rotem, and O. Mutlu, “BurstLink: Techniques for Energy-Efficient Video Display for Conventional and Virtual Reality Systems,” in MICRO, 2021. [352] N. Hajinazar, G. F. Oliveira, S. Gregorio, J. D. Ferreira, N. M. Ghiasi, M. Patel, M. Alser, S. Ghose, J. Gómez-Luna, and O. Mutlu, “SIMDRAM: A Framework for Bit-Serial SIMD Processing Using DRAM,” in ASPLOS, 2021. [353] M. Hall, P. Kogge, J. Koller, P. Diniz, J. Chame, J. Draper, J. LaCoss, J. Granacki, J. Brockman, A. Srivastava et al., “Mapping Irregular Applications to DIVA, a PIM-Based DataIntensive Architecture,” in SC, 1999. [354] H. Ham, J.-I. Hong, G. Park, Y. Shin, O. Woo, W. Yang, J. Bae, E. Park, H. Sung, E. Lim, and G. T. Kim, “Low-Overhead General-Purpose Near-Data Processing in CXL Memory Expanders,” in MICRO, 2024. [355] S. P. Harbison, “An Architectural Alternative to Optimizing Compilers,” in ASPLOS, 1982. [356] N. Hardavellas, M. Ferdman, B. Falsafi, and A. Ailamaki, “Reactive NUCA: NearOptimal Block Placement and Replication in Distributed Caches,” in ISCA, 2009. [357] M. Hashemi, O. Mutlu, and Y. N. Patt, “Continuous Runahead: Transparent Hardware Acceleration for Memory Intensive Workloads,” in MICRO, 2016.

BIBLIOGRAPHY

195

[358] M. Hashemi, K. Swersky, J. A. Smith, G. Ayers, H. Litz, J. Chang, C. Kozyrakis, and P. Ranganathan, “Learning Memory Access Patterns,” in arXiv preprint arXiv:1803.02329, 2018. [359] H. Hassan, “Improving DRAM Performance, Reliability, and Security by Rigorously Understanding Intrinsic DRAM Operation,” Ph.D. dissertation, ETH Zürich, 2022. [360] H. Hassan, M. Patel, J. S. Kim, A. G. Yaglikci, N. Vijaykumar, N. M. Ghiasi, S. Ghose, and O. Mutlu, “CROW: A Low-cost Substrate for Improving DRAM Performance, Energy Efficiency, and Reliability,” in ISCA, 2019. [361] H. Hassan, G. Pekhimenko, N. Vijaykumar, V. Seshadri, D. Lee, O. Ergin, and O. Mutlu, “ChargeCache: Reducing DRAM Latency by Exploiting Row Access Locality,” in HPCA, 2016. [362] Z. He, L. Yang, S. Angizi, A. S. Rakin, and D. Fan, “Sparse BD-Net: A Multiplication-Less DNN with Sparse Binarized Depth-Wise Separable Convolution,” in JETC, 2020. [363] H. Hidaka, H. Terada, M. Tsuji, T. Yamada, and K. Mori, “The Cache DRAM Architecture: A DRAM with an On-Chip Cache Memory,” in MICRO, 1990. [364] M. D. Hill and M. R. Marty, “Amdahl’s Law in the Multicore Era,” in Computer, 2008. [365] H. Hirata, K. Kimura, S. Nagamine, Y. Mochizuki, A. Nishimura, Y. Nakase, and T. Nishizawa, “An Elementary Processor Architecture with Simultaneous Instruction Issuing from Multiple Threads,” in ISCA, 1992. [366] H. Hoffmann, “Jouleguard: Energy Guarantees for Approximate Applications,” in SOSP, 2015. [367] A. Hosny, S. Hashemi, M. Shalan, and S. Reda, “DRiLLS: Deep Reinforcement Learning for Logic Synthesis,” in ASP-DAC, 2020. [368] S. Hsia, U. Gupta, M. Wilkening, C.-J. Wu, G.-Y. Wei, and D. Brooks, “Cross-Stack Workload Characterization of Deep Recommendation Systems,” in IISWC, 2020. [369] K. Hsieh, E. Ebrahimi, G. Kim, N. Chatterjee, M. O’Connor, N. Vijaykumar, O. Mutlu, and S. W. Keckler, “Transparent Offloading and Mapping (TOM) Enabling ProgrammerTransparent Near-Data Processing in GPU Systems,” in ISCA, 2016. [370] K. Hsieh, S. Khan, N. Vijaykumar, K. K. Chang, A. Boroumand, S. Ghose, and O. Mutlu, “Accelerating Pointer Chasing in 3D-Stacked Memory: Challenges, Mechanisms, Evaluation,” in ICCD, 2016. [371] Z. Hu, S. Kaxiras, and M. Martonosi, “Timekeeping in the Memory System: Predicting and Optimizing Memory Behavior,” in ISCA, 2002. [372] Z. Hu, M. Martonosi, and S. Kaxiras, “TCP: Tag Correlating Prefetchers,” in HPCA, 2003. [373] J. Huang and D. J. Lilja, “Exploiting Basic Block Value Locality with Block Reuse,” in HPCA, 1999.

196

BIBLIOGRAPHY

[374] J. Huang, R. R. Puli, P. Majumder, S. Kim, R. Boyapati, K. H. Yum, and E. J. Kim, “ActiveRouting: Compute on the Way for Near-Data Processing,” in HPCA, 2019. [375] Y. Huang and Z. Wang, “RLOP: A Framework Design for Offset Prefetching Combined with Reinforcement Learning,” in CENet, 2023. [376] Y. Huang, L. Zheng, P. Yao, J. Zhao, X. Liao, H. Jin, and J. Xue, “A Heterogeneous PIM Hardware-Software Co-Design for Energy-Efficient Graph Processing,” in IPDPS, 2020. [377] T. J. Huberty, S. G. Meier, and K. Khubaib, “Prefetch Throttling in a Multi-Core System,” US Patent, 2018, filed: 2016-04-07; Published: 2018-02-27. [Online]. Available: https://patents.google.com/patent/US9904624B1/en [378] I. Hur and C. Lin, “Memory Prefetching Using Adaptive Stream Detection,” in MICRO, 2006. [379] S. Iacobovici, L. Spracklen, S. Kadambi, Y. Chou, and S. G. Abraham, “Effective StreamBased and Execution-Based Data Prefetching,” in ICS, 2004. [380] M. Imani, S. Gupta, Y. Kim, and T. Rosing, “FloatPIM: In-Memory Acceleration of Deep Neural Network Training with High Precision,” in ISCA, 2019. [381] C. Imes, S. Hofmeyr, and H. Hoffmann, “Energy-efficient Application Resource Scheduling using Machine Learning Classifiers,” in ICPP, 2018. [382] E. Ipek, S. A. McKee, K. Singh, R. Caruana, B. R. d. Supinski, and M. Schulz, “Efficient Architectural Design Space Exploration via Predictive Modeling,” in TACO, 2008. [383] E. Ipek, O. Mutlu, J. F. Martínez, and R. Caruana, “Self-Optimizing Memory Controllers: A Reinforcement Learning Approach,” in ISCA, 2008. [384] Y. Ishii, M. Inaba, and K. Hiraki, “Access Map Pattern Matching for Data Cache Prefetch,” in ISC, 2009. [385] S. Islam, J. Keung, K. Lee, and A. Liu, “Empirical Prediction Models for Adaptive Resource Provisioning in the Cloud,” in Future Generation Computer Systems, 2012. [386] M. Item, G. F. Oliveira, J. Gómez-Luna, M. Sadrosadati, Y. Guo, and O. Mutlu, “TransPimLib: Efficient Transcendental Functions for Processing-in-Memory Systems,” in ISPASS, 2023. [387] Q. Jacobson and J. E. Smith, “Instruction Pre-Processing in Trace Processors,” in HPCA, 1999. [388] A. Jain and C. Lin, “Linearizing Irregular Memory Accesses for Improved Correlated Prefetching,” in MICRO, 2013. [389] A. Jain and C. Lin, “Back to the Future: Leveraging Belady’s Algorithm for Improved Cache Replacement,” in ISCA, 2016. [390] A. Jain and C. Lin, “Rethinking Belady’s Algorithm to Accommodate Prefetching,” in ISCA, 2018.

BIBLIOGRAPHY

197

[391] R. Jain, P. R. Panda, and S. Subramoney, “Machine Learned Machines: Adaptive CoOptimization of Caches, Cores, and On-Chip Network,” in DATE, 2016. [392] R. Jain, S. Cheng, V. Kalagi, V. Sanghavi, S. Kaul, M. Arunachalam, K. Maeng, A. Jog, A. Sivasubramaniam, M. T. Kandemir et al., “Optimizing CPU Performance for Recommendation Systems At-Scale,” in ISCA, 2023. [393] A. Jaleel, M. Mattina, and B. L. Jacob, “Last level cache (LLC) performance of data mining workloads on a CMP - a case study of parallel bioinformatics workloads,” in HPCA, 2006. [394] A. Jaleel, J. Nuzman, A. Moga, S. C. S. Jr., and J. S. Emer, “High performing cache hierarchies for server workloads: Relaxing inclusion to capture the latency benefits of exclusive caches,” in HPCA, 2015. [395] A. Jaleel, K. B. Theobald, S. C. Steely Jr, and J. Emer, “High Performance Cache Replacement using Re-Reference Interval Prediction (RRIP),” in ISCA, 2010. [396] M. Jalili and M. Erez, “Managing Prefetchers with Deep Reinforcement Learning,” in IEEE CAL, 2022. [397] M. Jalili and M. Erez, “Reducing Load Latency with Cache Level Prediction,” in HPCA, 2022. [398] A. V. Jamet, L. Alvarez, D. A. Jiménez, and M. Casas, “Characterizing the Impact of Last-Level Cache Replacement Policies on Big-Data Workloads,” in IISWC, 2020. [399] A. V. Jamet, G. Vavouliotis, D. A. Jiménez, L. Alvarez, and M. Casas, “A Two Level Neural Approach Combining Off-Chip Prediction with Adaptive Prefetch Filtering,” in HPCA, 2024. [400] S. Jamilan, T. A. Khan, G. Ayers, B. Kasikci, and H. Litz, “APT-GET: Profile-Guided Timely Software Prefetching,” in EuroSys, 2022. [401] J. Jang, J. Heo, Y. Lee, J. Won, S. Kim, S. J. Jung, H. Jang, T. J. Ham, and J. W. Lee, “Charon: Specialized Near-Memory Processing Architecture for Clearing Dead Objects in Memory,” in MICRO, 2019. [402] W. Jia, K. A. Shaw, and M. Martonosi, “Stargazer: Automated Regression-Based GPU Design Space Exploration,” in ISPASS, 2012. [403] Z. Jia, J. Zhan, L. Wang, C. Luo, W. Gao, Y. Jin, R. Han, and L. Zhang, “Understanding Big Data Analytics Workloads on Modern Processors,” in TPDS, 2017. [404] S. Jiang, Q. Yang, and Y. Ci, “Merging Similar Patterns for Hardware Prefetching,” in MICRO, 2022. [405] D. A. Jiménez, “Fast Path-Based Neural Branch Prediction,” in MICRO, 2003. [406] D. A. Jiménez, “Improved Latency and Accuracy for Neural Branch Prediction,” in ACM TC, 2005.

198

BIBLIOGRAPHY

[407] D. A. Jiménez, “Generalizing Neural Branch Prediction,” in TACO, 2009. [408] D. A. Jiménez, “Dead Block Replacement and Bypass with a Sampling Predictor,” in JWAC, 2010. [409] D. A. Jiménez, “Insertion and Promotion for Tree-Based PseudoLRU Last-Level Caches,” in MICRO, M. K. Farrens and C. Kozyrakis, Eds., 2013. [410] D. A. Jiménez, “Multiperspective Perceptron Predictor,” in 5th Championship Branch Prediction (CBP-5), 2016. [411] D. A. Jiménez and C. Lin, “Dynamic Branch Prediction with Perceptrons,” in HPCA, 2001. [412] D. A. Jiménez and C. Lin, “Neural Methods for Dynamic Branch Prediction,” in TOCS, 2002. [413] D. A. Jiménez and G. H. Loh, “Controlling the Power and Area of Neural Branch Predictors for Practical Implementation in High-Performance Processors,” in SBAC-PAD, 2006. [414] D. A. Jiménez and E. Teran, “Multiperspective Reuse Prediction,” in MICRO, 2017. [415] D. A. Jiménez, E. Teran, and P. V. Gratz, “Last-Level Cache Insertion and Promotion Policy in the Presence of Aggressive Prefetching,” in IEEE CAL, 2023. [416] V. Jiménez, F. J. Cazorla, R. Gioiosa, A. Buyuktosunoglu, P. Bose, F. P. O’Connell, and B. G. Mealey, “Adaptive Prefetching on POWER7: Improving Performance and Power Consumption,” in TOPC, 2014. [417] J. A. Joao, M. A. Suleman, O. Mutlu, and Y. N. Patt, “Bottleneck Identification and Scheduling in Multithreaded Applications,” in ASPLOS, 2012. [418] J. A. Joao, M. A. Suleman, O. Mutlu, and Y. N. Patt, “Utility-Based Acceleration of Multithreaded Applications on Asymmetric CMPs,” in ISCA, 2013. [419] T. L. Johnson, D. A. Connors, and W.-m. W. Hwu, “Run-Time Adaptive Cache Management,” in HICSS, 1998. [420] T. L. Johnson, D. A. Connors, M. C. Merten, and W.-m. W. Hwu, “Run-Time Cache Bypassing,” in IEEE TC, 1999. [421] T. L. Johnson and W.-m. W. Hwu, “Run-Time Adaptive Cache Hierarchy Management via Reference Analysis,” in ISCA, 1997. [422] W. E. I. Jones, “Up/Down Prefetcher,” US Patent, 2018, filed: 2016-06-13; Published: 2018-09-11. [Online]. Available: https://patents.google.com/patent/US10073785B2/en [423] A. Jooya, N. Dimopoulos, and A. Baniasadi, “Multiobjective GPU Design Space Exploration Optimization,” in Microprocessors and Microsystems, 2019.

BIBLIOGRAPHY

199

[424] H. F. Jordan, “Performance Measurements on HEP-A Pipelined MIMD Computer,” in ISCA, 1983. [425] D. Joseph and D. Grunwald, “Prefetching using Markov Predictors,” in ISCA, 1997. [426] P. Joseph, K. Vaswani, and M. J. Thazhuthaveetil, “A Predictive Performance Model for Superscalar Processors,” in MICRO, 2006. [427] P. Joseph, K. Vaswani, and M. J. Thazhuthaveetil, “Construction and use of Linear Regression Models for Processor Performance Analysis,” in HPCA, 2006. [428] N. P. Jouppi, “Improving Direct-mapped Cache Performance by the Addition of a Small Fully-associative Cache and Prefetch Buffers,” in ISCA, 1990. [429] N. Jouppi and D. Wall, “Available Instruction-Level Parallelism for Superscalar and Superpipelined Machines,” in ASPLOS, 1989. [430] D.-C. Juan and D. Marculescu, “Power-aware Performance Increase via Core/uncore Reinforcement Control for Chip-Multiprocessors,” in ISLPED, 2012. [431] C. Jung, D. Lim, J. Lee, and Y. Solihin, “Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems,” in IPDPS, 2006. [432] M. Kabra, R. Nadig, H. Gupta, R. Bera, M. Frouzakis, V. Arulchelvan, Y. Liang, H. Mao, M. Sadrosadati, and O. Mutlu, “CIPHERMATCH: Accelerating Homomorphic Encryption-Based String Matching via Memory-Efficient Data Packing and In-Flash Processing,” in ASPLOS, 2025. [433] D. Kadjo, J. Kim, P. Sharma, R. Panda, P. Gratz, and D. Jimenez, “B-fetch: Branch Prediction Directed Prefetching for Chip-Multiprocessors,” in MICRO, 2014. [434] K. Kalaitzidis and A. Seznec, “Value Speculation Through Equality Prediction,” in ICCD, 2019. [435] M. A. Kandaswamy and S. C. Steely, “Adaptive Control of Multiple Prefetchers,” US Patent, 2008, filed: 2007-03-31; Published: 2008-10-02. [Online]. Available: https://patents.google.com/patent/US20080243268A1/en [436] K. Kanellopoulos, R. Bera, K. Stojiljkovic, F. N. Bostanci, C. Firtina, R. Ausavarungnirun, R. Kumar, N. Hajinazar, M. Sadrosadati, N. Vijaykumar et al., “Utopia: Fast and Efficient Address Translation via Hybrid Restrictive & Flexible Virtual-To-Physical Address Mappings,” in MICRO, 2023. [437] K. Kanellopoulos, H. C. Nam, N. Bostanci, R. Bera, M. Sadrosadati, R. Kumar, D. B. Bartolini, and O. Mutlu, “Victima: Drastically Increasing Address Translation Reach by Leveraging Underutilized Cache Resources,” in MICRO, 2023. [438] K. Kanellopoulos, K. Sgouras, F. N. Bostanci, A. K. Kakolyris, B. K. Konar, R. Bera, M. Sadrosadati, R. Kumar, N. Vijaykumar, and O. Mutlu, “Virtuoso: Enabling Fast and Accurate Virtual Memory Research via an Imitation-Based Operating System Simulation Methodology,” in ASPLOS, 2025.

200

BIBLIOGRAPHY

[439] K. Kanellopoulos, K. Sgouras, A. K. Kakolyris, V.-P. Nitu, B. K. Konar, R. Bera, and O. Mutlu, “Revelator: Rapid Data Fetching via OS-Driven Hash-based Speculative Address Translation,” in arXiv preprint arXiv:2508.02007, 2025. [440] S. Kanev, J. P. Darago, K. Hazelwood, P. Ranganathan, T. Moseley, G.-Y. Wei, and D. Brooks, “Profiling a Warehouse-scale Computer,” in ISCA, 2015. [441] D. Kang and J.-L. Gaudiot, “Speculation Control for Simultaneous Multithreading,” in IPDPS, 2004. [442] S.-C. Kao, G. Jeong, and T. Krishna, “ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning,” in MICRO, 2020. [443] M. Karlsson, F. Dahlgren, and P. Stenstrom, “A Prefetching Technique for Irregular Accesses to Linked Data Structures,” in HPCA, 2000. [444] W. H. Kautz, “Cellular Logic-in-Memory Arrays,” in IEEE TC, 1969. [445] L. Ke, X. Zhang, J. So, J.-G. Lee, S.-H. Kang, S. Lee, S. Han, Y. Cho, J. H. Kim, Y. Kwon et al., “Near-Memory Processing in Action: Accelerating Personalized Recommendation with AxDIMM,” in IEEE Micro, 2021. [446] C. D. Kersey, H. Kim, and S. Yalamanchili, “Lightweight SIMT Core Designs for Intelligent 3D Stacked DRAM,” in MEMSYS, 2017. [447] A. Khan, X. Yan, S. Tao, and N. Anerousis, “Workload Characterization and Prediction in the Cloud: A Multiple Time Series Approach,” in NOMS, 2012. [448] S. M. Khan, A. R. Alameldeen, C. Wilkerson, J. Kulkarni, and D. A. Jiménez, “Improving Multi-Core Performance using Mixed-Cell Cache Architecture,” in HPCA, 2013. [449] S. M. Khan, A. R. Alameldeen, C. Wilkerson, O. Mutlu, and D. A. Jiménez, “Improving Cache Performance using Read-Write Partitioning,” in HPCA, 2014. [450] S. M. Khan and D. A. Jiménez, “Insertion Policy Selection using Decision Tree Analysis,” in ICCD, 2010. [451] S. M. Khan, D. A. Jiménez, D. Burger, and B. Falsafi, “Using Dead Blocks as a Virtual Victim Cache,” in PACT, V. Salapura, M. Gschwind, and J. Knoop, Eds., 2010. [452] S. M. Khan, Y. Tian, and D. A. Jimenez, “Sampling Dead Block Prediction for Last-Level Caches,” in MICRO, 2010. [453] S. M. Khan, Z. Wang, and D. A. Jiménez, “Decoupled Dynamic Cache Segmentation,” in HPCA, 2012. [454] T. A. Khan, N. Brown, A. Sriraman, N. K. Soundararajan, R. Kumar, J. Devietti, S. Subramoney, G. A. Pokam, H. Litz, and B. Kasikci, “Twig: Profile-Guided BTB Prefetching for Data Center Applications,” in MICRO, 2021.

BIBLIOGRAPHY

201

[455] T. A. Khan, M. Ugur, K. Nathella, D. Sunwoo, H. Litz, D. A. Jiménez, and B. Kasikci, “Whisper: Profile-Guided Branch Misprediction Elimination for Data Center Applications,” in MICRO, 2022. [456] T. A. Khan, D. Zhang, A. Sriraman, J. Devietti, G. Pokam, H. Litz, and B. Kasikci, “Ripple: Profile-Guided Instruction Cache Replacement for Data Center Applications,” in ISCA, 2021. [457] M. Kharbutli and Y. Solihin, “Counter-Based Cache Replacement and Bypassing Algorithms,” in IEEE TC, 2008. [458] C. Kim, D. Burger, and S. W. Keckler, “An Adaptive, Non-Uniform Cache Structure for Wire-Delay Dominated On-Chip Caches,” in ASPLOS, 2002. [459] D. Kim, J. Kung, S. Chai, S. Yalamanchili, and S. Mukhopadhyay, “Neurocube: A Programmable Digital Neuromorphic Architecture with High-Density 3D Memory,” in ISCA, 2016. [460] J. Kim, M. Patel, H. Hassan, and O. Mutlu, “Solar-DRAM: Reducing DRAM Access Latency by Exploiting the Variation in Local Bitlines,” in ICCD, 2018. [461] J. S. Kim, D. S. Cali, H. Xin, D. Lee, S. Ghose, M. Alser, H. Hassan, O. Ergin, C. Alkan, and O. Mutlu, “GRIM-Filter: Fast Seed Location Filtering in DNA Read Mapping Using Processing-in-Memory Technologies,” in BMC Genomics, 2018. [462] J. S. Kim, M. Patel, H. Hassan, and O. Mutlu, “The DRAM Latency PUF: Quickly Evaluating Physical Unclonable Functions by Exploiting the Latency-Reliability Tradeoff in Modern Commodity DRAM Devices,” in HPCA, 2018. [463] J. S. Kim, M. Patel, H. Hassan, L. Orosa, and O. Mutlu, “D-RaNGe: Using Commodity DRAM Devices to Generate True Random Numbers With Low Latency and High Throughput,” in HPCA, 2019. [464] J. S. Kim, D. Senol, H. Xin, D. Lee, S. Ghose, M. Alser, H. Hassan, O. Ergin, C. Alkan, and O. Mutlu, “GRIM-Filter: Fast Seed Filtering in Read Mapping using Emerging Memory Technologies,” arXiv:1708.04329 [q-bio.GN], 2017. [465] J. Kim, S. H. Pugsley, P. V. Gratz, A. Reddy, C. Wilkerson, and Z. Chishti, “Path Confidence Based Lookahead Prefetching,” in MICRO, 2016. [466] J. Kim, E. Teran, P. V. Gratz, D. A. Jiménez, S. H. Pugsley, and C. Wilkerson, “Kill the Program Counter: Reconstructing Program Behavior in the Processor Cache Hierarchy,” in ASPLOS, 2017. [467] S. S. Kim and A. Ros, “Effective Context-Sensitive Memory Dependence Prediction,” in HPCA, 2024. [468] Y. Kim, D. Han, O. Mutlu, and M. Harchol-Balter, “ATLAS: A Scalable and HighPerformance Scheduling Algorithm for Multiple Memory Controllers,” in HPCA, 2010.

202

BIBLIOGRAPHY

[469] Y. Kim, M. Papamichael, O. Mutlu, and M. Harchol-Balter, “Thread Cluster Memory Scheduling: Exploiting Differences in Memory Access Behavior,” in MICRO, 2010. [470] Y. Kim, V. Seshadri, D. Lee, J. Liu, and O. Mutlu, “A Case for Exploiting Subarray-level Parallelism (SALP) in DRAM,” in ISCA, 2012. [471] A. C. Klaiber and H. M. Levy, “An Architecture for Software-Controlled Data Prefetching,” in ISCA, 1991. [472] J. Knoop, O. Rüthing, and B. Steffen, “Partial Dead Code Elimination,” in PLDI, 1994. [473] P. M. Kogge, “EXECUBE-A New Architecture for Scaleable MPPs,” in ICPP, 1994. [474] P. Kogge, T. Sunaga, H. Miyataka, K. Kitamura, and E. Retter, “Combined DRAM and Logic Chip for Massively Parallel Systems,” in ARVLSI, 1995. [475] S. Kondguli and M. Huang, “Division of Labor: A More Effective Approach to Prefetching,” in ISCA, 2018. [476] P. Kongetira, K. Aingaran, and K. Olukotun, “Niagara: A 32-Way Multithreaded SPARC Processor,” in IEEE Micro, 2005. [477] S. Koppula, L. Orosa, A. G. Yağlıkçı, R. Azizi, T. Shahroodi, K. Kanellopoulos, and O. Mutlu, “EDEN: Enabling Energy-Efficient, High-Performance Deep Neural Network Inference using Approximate DRAM,” in MICRO, 2019. [478] A. M. Krause, P. C. Santos, and P. O. A. Navaux, “Avoiding Unnecessary Caching with History-Based Preemptive Bypassing,” in SBAC-PAD, 2022. [479] S. Kumar and C. Wilkerson, “Exploiting Spatial Locality in Data Caches using Spatial Footprints,” in ISCA, 1998. [480] Y.-C. Kwon, S. H. Lee, J. Lee, S.-H. Kwon, J. M. Ryu, J.-P. Son, O. Seongil, H.-S. Yu, H. Lee, S. Y. Kim et al., “A 20nm 6GB Function-in-Memory DRAM, Based on HBM2 with a 1.2 TFLOPS Programmable Computing Unit using Bank-Level Parallelism, for Machine Learning Applications,” in ISSCC, 2021. [481] B. C. Lee and D. M. Brooks, “Accurate and Efficient Regression Modeling for Microarchitectural Performance and Power Prediction,” in ACM SIGOPS operating systems review, 2006. [482] C. J. Lee, E. Ebrahimi, V. Narasiman, O. Mutlu, and Y. N. Patt, “DRAM-Aware Last-Level Cache Replacement,” in HPS Technical Report, 2010. [483] C. J. Lee, O. Mutlu, V. Narasiman, and Y. N. Patt, “Prefetch-aware DRAM Controllers,” in MICRO, 2008. [484] C. J. Lee, V. Narasiman, E. Ebrahimi, O. Mutlu, and Y. N. Patt, “DRAM-Aware LastLevel Cache Writeback: Reducing Write-Caused Interference in Memory Systems,” in HPS Technical Report, 2010.

BIBLIOGRAPHY

203

[485] C. J. Lee, V. Narasiman, O. Mutlu, and Y. N. Patt, “Improving Memory Bank-level Parallelism in the Presence of Prefetching,” in MICRO, 2009. [486] D. Lee, “Reducing DRAM Latency at Low Cost by Exploiting Heterogeneity,” Ph.D. dissertation, Carnegie Mellon University, 2016. [487] D. Lee, S. Khan, L. Subramanian, S. Ghose, R. Ausavarungnirun, G. Pekhimenko, V. Seshadri, and O. Mutlu, “Design-induced Latency Variation in Modern DRAM Chips: Characterization, Analysis, and Latency Reduction Mechanisms,” in POMACS, 2017. [488] D. Lee, Y. Kim, G. Pekhimenko, S. Khan, V. Seshadri, K. Chang, and O. Mutlu, “Adaptivelatency DRAM: Optimizing DRAM Timing for the Common-Case,” in HPCA, 2015. [489] D. Lee, Y. Kim, V. Seshadri, J. Liu, L. Subramanian, and O. Mutlu, “Tiered-latency DRAM: A Low Latency and Low Cost DRAM Architecture,” in HPCA, 2013. [490] H. J. Lee and W. W. Ro, “Crossing the Boundary: Virtual-Address Based Inter-Page Prefetching for Lower Level Caches,” in 4th Data Prefetching Championship (DPC4), 2026. [491] J. K. Lee and A. J. Smith, “Branch Prediction Strategies and Branch Target Buffer Design,” in IEEE Computer, 1984. [492] J. H. Lee, J. Sim, and H. Kim, “BSSync: Processing Near Memory for Machine Learning Workloads with Bounded Staleness Consistency Models,” in PACT, 2015. [493] S. Lee, K. Kim, S. Oh, J. Park, G. Hong, D. Ka, K. Hwang, J. Park, K. Kang, J. Kim, J. Jeon, N. Kim, Y. Kwon, K. Vladimir, W. Shin, J. Won, M. Lee, H. Joo et al., “A 1ynm 1.25V 8Gb, 16Gb/s/pin GDDR6-based Accelerator-in-Memory Supporting 1TFLOPS MAC Operation and Various Activation Functions for Deep-Learning Applications,” in ISSCC, 2022. [494] S. Lee, S.-h. Kang, J. Lee, H. Kim, E. Lee, S. Seo, H. Yoon, S. Lee, K. Lim, H. Shin et al., “Hardware Architecture and Software Stack for PIM Based on Commercial DRAM Technology: Industrial Product,” in ISCA, 2021. [495] K. M. Lepak, G. B. Bell, and M. H. Lipasti, “Silent Stores and Store Value Locality,” in IEEE TC, 2001. [496] K. M. Lepak and M. H. Lipasti, “Silent Stores for Free,” in MICRO, 2000. [497] K. M. Lepak and M. H. Lipasti, “Temporally Silent Stores,” in ASPLOS, 2002. [498] P. Lerman, “Fitting Segmented Regression Models by Grid Search,” in Journal of the Royal Statistical Society: Series C (Applied Statistics), 1980. [499] H. Li, K. Liu, T. Liang, Z. Li, T. Lu, H. Yuan, Y. Xia, Y. Bao, M. Chen, and Y. Shan, “HoPP: Hardware-Software Co-Designed Page Prefetching for Disaggregated Memory,” in HPCA, 2023. [500] H. Li, D. S. Berger, S. Novakovic, L. R. Hsu, D. Ernst, P. Zardoshti, M. Shah, S. Rajadnya, S. Lee, I. Agarwal, M. D. Hill, M. Fontoura, and R. Bianchini, “Pond: CXL-Based Memory Pooling Systems for Cloud Platforms,” in ASPLOS, 2022.

204

BIBLIOGRAPHY

[501] J. Li, X. Wang, A. Tumeo, B. Williams, J. D. Leidel, and Y. Chen, “PIMS: A Lightweight Processing-in-Memory Accelerator for Stencil Computations,” in MEMSYS, 2019. [502] S. Li, J. H. Ahn, R. D. Strong, J. B. Brockman, D. M. Tullsen, and N. P. Jouppi, “McPAT: An Integrated Power, Area, and Timing Modeling Framework for Multicore and Manycore Architectures,” in MICRO, 2009. [503] S. Li, A. O. Glova, X. Hu, P. Gu, D. Niu, K. T. Malladi, H. Zheng, B. Brennan, and Y. Xie, “SCOPE: A Stochastic Computing Engine for DRAM-Based In-Situ Accelerator,” in MICRO, 2018. [504] S. Li, D. Niu, K. T. Malladi, H. Zheng, B. Brennan, and Y. Xie, “DRISA: A DRAM-Based Reconfigurable In-Situ Accelerator,” in MICRO, 2017. [505] S. Li, C. Xu, Q. Zou, J. Zhao, Y. Lu, and Y. Xie, “Pinatubo: A Processing-in-Memory Architecture for Bulk Bitwise Operations in Emerging Non-Volatile Memories,” in DAC, 2016. [506] M. Liang, Y. Gan, Y. Li, C. Torres, A. Dhanotia, M. Ketkar, and C. Delimitrou, “Ditto: End-to-End Application Cloning for Networked Cloud Services,” in ASPLOS, 2023. [507] H. Lim and G. Park, “Triple Engine Processor (TEP): A Heterogeneous Near-Memory Processor for Diverse Kernel Operations,” in TACO, 2017. [508] T.-R. Lin, Y. Li, M. Pedram, and L. Chen, “Design Space Exploration of Memory Controller Placement in Throughput Processors with Deep Learning,” in IEEE CAL, 2019. [509] T.-R. Lin, D. Penney, M. Pedram, and L. Chen, “A Deep Reinforcement Learning Framework for Architectural Exploration: A Routerless NoC Case Study,” in HPCA, 2020. [510] W.-F. Lin, S. Reinhardt, and D. Burger, “Reducing DRAM Latencies with an Integrated Memory Hierarchy Design,” in HPCA, 2001. [511] W.-F. Lin, S. Reinhardt, D. Burger, and T. Puzak, “Filtering Superfluous Prefetches using Density Vectors,” in ICCD, 2001. [512] Y. Lin, S. Dhar, W. Li, H. Ren, B. Khailany, and D. Z. Pan, “Dreamplace: Deep Learning Toolkit-Enabled Gpu Acceleration for Modern Vlsi Placement,” in DAC, 2019. [513] Y. Lin, W. Lin, J. Xu, Y. Chen, Z. Jin, J. Qin, J. He, S. Cai, Y. Zhang, Z. Wang et al., “Pars: a Pattern-Aware Spatial Data Prefetcher Supporting Multiple Region Sizes,” in IEEE TCAD, 2024. [514] M. H. Lipasti, W. J. Schmidt, S. R. Kunkel, and R. R. Roediger, “SPAID: Software Prefetching in Pointer- and Call-Intensive Environments,” in MICRO, 1995. [515] M. H. Lipasti and J. P. Shen, “Exceeding the Dataflow Limit via Value Prediction,” in MICRO, 1996. [516] M. H. Lipasti, C. B. Wilkerson, and J. P. Shen, “Value Locality and Load Value Prediction,” in ASPLOS, 1996.

BIBLIOGRAPHY

205

[517] J. S. Liptay, “Structural Aspects of the System/360 Model 85, II: The Cache,” in IBM Systems Journal, 1968. [518] H. Litz, G. Ayers, and P. Ranganathan, “CRISP: Critical Slice Prefetching,” in ASPLOS, 2022. [519] E. Z. Liu, M. Hashemi, K. Swersky, P. Ranganathan, and J. Ahn, “An Imitation Learning Approach for Cache Replacement,” 2020. [520] H. Liu, M. Ferdman, J. Huh, and D. Burger, “Cache Bursts: A New Approach for Eliminating Dead Blocks and Increasing Cache Efficiency,” in MICRO, 2008. [521] J. Liu, H. Zhao, M. A. Ogleari, D. Li, and J. Zhao, “Processing-in-Memory for EnergyEfficient Neural Network Training: A Heterogeneous Approach,” in MICRO, 2018. [522] M. Liu, K. Zhu, X. Tang, B. Xu, W. Shi, N. Sun, and D. Z. Pan, “Closing the Design Loop: Bayesian Optimization Assisted Hierarchical Analog Layout Synthesis,” in DAC, 2020. [523] E. Lockerman, A. Feldmann, M. Bakhshalipour, A. Stanescu, S. Gupta, D. Sanchez, and N. Beckmann, “Livia: Data-Centric Computing Throughout the Memory Hierarchy,” in ASPLOS, 2020. [524] G. H. Loh and M. D. Hill, “Efficiently Enabling Conventional Block Sizes for Very Large Die-Stacked DRAM Caches,” in MICRO, 2011. [525] G. H. Loh and D. A. Jiménez, “Modulo Path History for the Reduction of Pipeline Overheads in Path-based Neural Branch Predictors,” in Int. J. Parallel Program., 2008. [526] X. Long, X. Gong, B. Zhang, and H. Zhou, “Deep Learning Based Data Prefetching in CPU-GPU Unified Virtual Memory,” in Journal of Parallel and Distributed Computing, 2023. [527] P. Lotfi-Kamran, B. Grot, M. Ferdman, S. Volos, O. Kocberber, J. Picorel, A. Adileh, D. Jevdjic, S. Idgunji, E. Ozer et al., “Scale-Out Processors,” in ASPLOS, 2012. [528] S. Lu, R. Tessier, and W. Burleson, “Reinforcement Learning for Thermal-Aware ManyCore Task Allocation,” in GLSVLSI, 2015. [529] S.-L. Lu, K. Sankaralingam, M. Kandemir, and C. R. Das, “Improving DRAM Latency with Dynamic Asymmetric Subarrays,” in MICRO, 2015. [530] X. Lu, H. Najafi, J. Liu, and X.-H. Sun, “CHROME: Concurrency-Aware Holistic Cache Management Framework with Online Reinforcement Learning,” in HPCA, 2024. [531] Y.-C. Lu, J. Lee, A. Agnesina, K. Samadi, and S. K. Lim, “GAN-CTS: A Generative Adversarial Framework for Clock Tree Prediction and Optimization,” in ICCAD, 2019. [532] C.-K. Luk, “Optimizing the Cache Performance of Non-Numeric Applications,” Ph.D. dissertation, University of Toronto, 2000. [533] C.-K. Luk, “Tolerating Memory Latency Through Software-Controlled Pre-Execution in Simultaneous Multithreading Processors,” in ISCA, 2001.

206

BIBLIOGRAPHY

[534] C.-K. Luk and T. C. Mowry, “Compiler-Based Prefetching for Recursive Data Structures,” in ASPLOS, 1996. [535] H. Luo, T. Shahroodi, H. Hassan, M. Patel, A. G. Yağlıkçı, L. Orosa, J. Park, and O. Mutlu, “CLR-DRAM: A Low-Cost DRAM Architecture Enabling Dynamic Capacity-Latency Trade-Off,” in ISCA, 2020. [536] R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu, “BioGPT: Generative Pre-Trained Transformer for Biomedical Text Generation and Mining,” in Briefings in Bioinformatics, 2022. [537] Z. Luo, S. Son, S. Ratnasamy, and S. Shenker, “Harvesting Memory-bound CPU Stall Cycles in Software with MSH,” in OSDI, 2024. [538] K. Ma, X. Li, W. Chen, C. Zhang, and X. Wang, “Greengpu: A Holistic Approach to Energy Efficiency in Gpu-Cpu Heterogeneous Architectures,” in ICPP, 2012. [539] S. Mahar, H. Wang, W. W. Shu, and A. Dhanotia, “Workload Behavior Driven Memory Subsystem Design for Hyperscale,” in ArXiv, 2023. [540] F. Mahdisoltani, I. Stefanovici, and B. Schroeder, “Proactive Error Prediction to Improve Storage System Reliability,” in USENIX ATC, 2017. [541] H. M. Makrani, H. Sayadi, S. M. P. Dinakarra, S. Rafatirad, and H. Homayoun, “A Comprehensive Memory Analysis of Data Intensive Workloads on Server Class Architecture,” in MEMSYS, 2018. [542] A. Margaritov, D. Ustiugov, E. Bugnion, and B. Grot, “Virtual Address Translation via Learned Page Table Indexes,” in ML for Systems at NeurIPS, 2018. [543] I. L. Markov, “Reevaluating Google’s Reinforcement Learning for IC Macro Placement,” in Commun. ACM, 2024. [544] A. Maruf, A. Ghosh, J. Bhimani, D. Campello, A. Rudoff, and R. Rangaswami, “MULTICLOCK: Dynamic Tiering for Hybrid Memory Systems,” in HPCA, 2022. [545] H. A. Maruf, H. Wang, A. Dhanotia, J. Weiner, N. Agarwal, P. Bhattacharya, C. Petersen, M. Chowdhury, S. O. Kanaujia, and P. Chauhan, “TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory,” in ASPLOS, 2022. [546] K. K. Matam, G. Koo, H. Zha, H.-W. Tseng, and M. Annavaram, “GraphSSD: Graph Semantics Aware SSD,” in ISCA, 2019. [547] C. Mazumdar, P. Mitra, and A. Basu, “Dead Page and Dead Block Predictors: Cleaning TLBs and Caches Together,” in ISCA, 2021. [548] W. S. McCulloch and W. Pitts, “A Logical Calculus of the Ideas Immanent in Nervous Activity,” in BMB, 1943. [549] S. McFarling, “Combining Branch Predictors,” Digital Western Research Laboratory, Tech. Rep., 1993.

BIBLIOGRAPHY

207

[550] A. Mehrabi, A. Manocha, B. C. Lee, and D. J. Sorin, “Prospector: Synthesizing Efficient Accelerators via Statistical Learning.” in DATE, 2020. [551] S. G. Meier, T. J. Huberty, N. Gupta, F. Spadini, and G. Levinsky, “Prefetch Circuit with Global Quality Factor to Reduce Aggressiveness in Low Power Modes,” US Patent, 2019, filed: 2017-02-17; Published: 2019-06-25. [Online]. Available: https://patents.google.com/patent/US10331567B1/en [552] G. Memik, G. Reinman, and W. H. Mangione-Smith, “Just Say No: Benefits of Early Cache Miss Determination,” in HPCA, 2003. [553] A. Mendelson and F. Gabbay, “Speculative Execution Based on Value Prediction,” Tech. Rep., Technion, Tech. Rep., 1997. [554] M. Merrell, L. Wang, P. Gratz, and S. Kalafatis, “SPPAM: Signature Pattern Prediction and Access-Map Prefetcher,” in 4th Data Prefetching Championship (DPC4), 2026. [555] P. Michaud, “A Statistical Model of Skewed Associativity,” in ISPASS, 2003. [556] P. Michaud, “Best-offset Hardware Prefetching,” in HPCA, 2016. [557] P. Michaud, A. Seznec, and R. Uhlig, “Trading Conflict and Capacity Aliasing in Conditional Branch Predictors,” in ISCA, 1997. [558] D. Michie, ““Memo” Functions and Machine Learning,” in Nature, 1968. [559] A. Mirhoseini, A. Goldie, H. Pham, B. Steiner, Q. V. Le, and J. Dean, “A Hierarchical Model for Device Placement,” in ICLR, 2018. [560] A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y.-J. Lee, E. Johnson, O. Pathak, A. Nazi, J. Pak, A. Tong, K. Srinivasa, W. Hang, E. Tuncer, Q. V. Le, J. Laudon, R. Ho, R. Carpenter, and J. Dean, “A Graph Placement Methodology for Fast Chip Design,” in Nature, 2021. [561] A. Mirhoseini, H. Pham, Q. V. Le, B. Steiner, R. Larsen, Y. Zhou, N. Kumar, M. Norouzi, S. Bengio, and J. Dean, “Device Placement Optimization with Reinforcement Learning,” in ICML, 2017. [562] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level Control Through Deep Reinforcement Learning,” in Nature, 2015. [563] C. Molina, A. Gonzalez, and J. Tubella, “Dynamic Removal of Redundant Computations,” in ICS, 1999. [564] D. Molka, D. Hackenberg, R. Schöne, and W. E. Nagel, “Cache Coherence Protocol and Memory Performance of the Intel Haswell-EP Architecture,” in ICPP, 2015. [565] D. C. Montgomery, E. A. Peck, and G. G. Vining, Introduction to Linear Regression Analysis. John Wiley & Sons, 2012.

208

BIBLIOGRAPHY

[566] T. Moscibroda and O. Mutlu, “Memory Performance Attacks: Denial of Memory Service in Multi-Core Systems,” in USENIX Security, 2007. [567] A. Moshovos, S. E. Breach, T. N. Vijaykumar, and G. S. Sohi, “Dynamic Speculation and Synchronization of Data Dependences,” in ISCA, 1997. [568] A. Moshovos, D. N. Pnevmatikatos, and A. Baniasadi, “Slice-Processors: An Implementation of Operation-Based Prediction,” in ICS, 2001. [569] A. Moshovos and G. S. Sohi, “Streamlining Inter-operation Memory Communication via Data Dependence Prediction,” in MICRO, 1997. [570] A. Moshovos and G. S. Sohi, “Speculative Memory Cloaking and Bypassing,” in International Journal of Parallel Programming, 1999. [571] A. Moshovos and G. S. Sohi, “Memory Dependence Prediction in Multimedia Applications,” in JILP, 2000. [572] A. Moshovos and G. S. Sohi, “Memory Dependence Speculation Tradeoffs in Centralized, Continuous-Window Superscalar Processors,” in HPCA, 2000. [573] S. Mostofi, S. Gupta, A. Hassani, K. Tibrewala, E. Teran, P. V. Gratz, and D. A. Jiménez, “Light-weight Cache Replacement for Instruction Heavy Workloads,” in ISCA, 2025. [574] T. C. Mowry and C.-K. Luk, “Predicting Data Cache Misses in Non-Numeric Applications through Correlation Profiling,” in MICRO, 1997. [575] T. C. Mowry, “Tolerating Latency Through Software-Controlled Data Prefetching,” Ph.D. dissertation, Stanford University, 1995. [576] J. Mukundan and J. F. Martinez, “MORSE: Multi-objective Reconfigurable SelfOptimizing Memory Scheduler,” in ISCA, 2012. [577] S. P. Muralidhara, L. Subramanian, O. Mutlu, M. Kandemir, and T. Moscibroda, “Reducing Memory Interference in Multicore Systems via Application-Aware Memory Channel Partitioning,” in MICRO, 2011. [578] O. Mutlu, “Memory Scaling: A Systems Architecture Perspective,” in IMW, 2013. [579] O. Mutlu, “Main Memory Scaling: Challenges and Solution Directions,” in More Than Moore Technologies for Next Generation Computer Design. Springer-Verlag, 2015. [580] O. Mutlu, “Intelligent Architectures for Intelligent Computing Systems,” in DATE, 2021. [581] O. Mutlu, S. Ghose, J. Gómez-Luna, and R. Ausavarungnirun, “Processing Data Where It Makes Sense: Enabling In-Memory Computation,” in MicPro, 2019. [582] O. Mutlu, S. Ghose, J. Gómez-Luna, and R. Ausavarungnirun, “A Modern Primer on Processing in Memory,” in Emerging computing: from devices to systems: looking beyond Moore and Von Neumann. Springer, 2022.

BIBLIOGRAPHY

209

[583] O. Mutlu, H. Kim, D. N. Armstrong, and Y. N. Patt, “Using the First-Level Caches as Filters to Reduce the Pollution Caused by Speculative Memory References,” in IJPP, 2005. [584] O. Mutlu, H. Kim, and Y. N. Patt, “Address-Value Delta (AVD) Prediction: Increasing the Effectiveness of Runahead Execution by Exploiting Regular Memory Allocation Patterns,” in MICRO, 2005. [585] O. Mutlu, H. Kim, and Y. N. Patt, “Techniques for Efficient Processing in Runahead Execution Engines,” in ISCA, 2005. [586] O. Mutlu, H. Kim, and Y. N. Patt, “Address-Value Delta (AVD) Prediction: A Hardware Technique for Efficiently Parallelizing Dependent Cache Misses,” in IEEE TC, 2006. [587] O. Mutlu, H. Kim, and Y. N. Patt, “Efficient Runahead Execution: Power-efficient Memory Latency Tolerance,” in IEEE Micro, 2006. [588] O. Mutlu and T. Moscibroda, “Stall-Time Fair Memory Access Scheduling for Chip Multiprocessors,” in MICRO, 2007. [589] O. Mutlu and T. Moscibroda, “Parallelism-Aware Batch Scheduling: Enhancing Both Performance and Fairness of Shared DRAM Systems,” in ISCA, 2008. [590] O. Mutlu, A. Olgun, G. F. Oliveira, and I. E. Yuksel, “Memory-Centric Computing: Recent Advances in Processing-in-DRAM,” in IEDM, 2024. [591] O. Mutlu, A. Olgun, and İ. E. Yüksel, “Memory-Centric Computing: Solving Computing’s Memory Problem,” in IMW, 2025. [592] O. Mutlu, J. Stark, C. Wilkerson, and Y. N. Patt, “Runahead Execution: An Alternative to Very Large Instruction Windows for Out-Of-Order Processors,” in HPCA, 2003. [593] O. Mutlu, J. Stark, C. Wilkerson, and Y. N. Patt, “Runahead Execution: An Effective Alternative to Large Instruction Windows,” in IEEE Micro, 2003. [594] R. Nadig, V. Arulchelvan, R. Bera, T. Shahroodi, G. Singh, M. Sadrosadati, J. Park, and O. Mutlu, “Harmonia: A Multi-Agent Reinforcement Learning Approach to Data Placement and Migration in Hybrid Storage Systems,” in ICS, 2026. [595] R. Nadig, M. Kabra, H. Gupta, R. Bera, N. M. Ghiasi, N. Rao, Q. Jiang, A. K. Kakolyris, Y. Liang, M. Sadrosadati, and O. Mutlu, “Conduit: Programmer-Transparent Near-Data Processing Using Multiple Compute-Capable Resources in SSDs,” in HPCA, 2026. [596] L. Nai, R. Hadidi, J. Sim, H. Kim, P. Kumar, and H. Kim, “GraphPIM: Enabling Instruction-Level PIM Offloading in Graph Computing Frameworks,” in HPCA, 2017. [597] R. Nair, S. F. Antao, C. Bertolli, P. Bose et al., “Active Memory Cube: A Processing-inMemory Architecture for Exascale Systems,” in IBM JRD, 2015. [598] A. Naithani, S. Ainsworth, T. M. Jones, and L. Eeckhout, “Vector Runahead,” in ISCA, 2021.

210

BIBLIOGRAPHY

[599] A. Naithani and L. Eeckhout, “Reliability-Aware Runahead,” in HPCA, 2022. [600] A. Naithani, J. Feliu, A. Adileh, and L. Eeckhout, “Precise Runahead Execution,” in HPCA, 2020. [601] A. Naithani, J. Roelandts, S. Ainsworth, T. M. Jones, and L. Eeckhout, “Decoupled Vector Runahead,” in MICRO, 2023. [602] A. Nasr-Esfahany, M. Alizadeh, V. Lee, H. Alam, B. W. Coon, D. Culler, V. Dadu, M. Dixon, H. M. Levy, S. Pandey, P. Ranganathan, and A. Yazdanbakhsh, “Concorde: Fast and Accurate CPU Performance Modeling with Compositional Analytical-ML Fusion,” in ISCA, 2025. [603] R. Natarajan and M. Chaudhuri, “Characterizing Multi-Threaded Applications for Designing Sharing-Aware Last-Level Cache Replacement Policies,” in IISWC, 2013. [604] A. Navarro-Torres, B. Panda, J. Alastruey-Benedé, P. Ibáñez, V. Viñals-Yúfera, and A. Ros, “Berti: An Accurate Local-Delta Data Prefetcher,” in MICRO, 2022. [605] M. D. Nemirovsky, F. Brewer, and R. C. Wood, “DISC: Dynamic Instruction Stream Computer,” in MICRO, 1991. [606] W. L. Neto, M. Austin, S. Temple, L. Amaru, X. Tang, and P.-E. Gaillardon, “LSOracle: A Logic Synthesis Framework Driven by Artificial Intelligence,” in ICCAD, 2019. [607] D. Niu, S. Li, Y. Wang, W. Han, Z. Zhang, Y. Guan, T. Guan, F. Sun, F. Xue, L. Duan et al., “184QPS/W 64Mb/𝑚𝑚 2 3D Logic-to-DRAM Hybrid Bonding with Process-NearMemory Engine for Recommendation System,” in ISSCC, 2022. [608] A. V. Nori, R. Bera, S. Balachandran, J. Rakshit, O. J. Omer, A. Abuhatzera, B. Kuttanna, and S. Subramoney, “Proximu$: Efficiently Scaling DNN Inference in Multi-core CPUs Through Near-Cache Compute,” 2020. [609] A. V. Nori, R. Bera, S. Balachandran, J. Rakshit, O. J. Omer, A. Abuhatzera, B. Kuttanna, and S. Subramoney, “REDUCT: Keep It Close, Keep It Cool!: Effcient Scaling of DNN Inference on Multi-Core CPUs with Near-Cache Compute,” in ISCA, 2021. [610] S. O, Y. H. Son, Y. Ro, J. W. Lee, and J. H. Ahn, “Row-Buffer Decoupling: A Case for Low-Latency DRAM Microarchitecture,” in ISCA, 2014. [611] S. F. Oberman and M. J. Flynn, “Reducing Division Latency with Reciprocal Caches,” in Reliable Computing, 1996. [612] J. M. O’Connor, “Energy Efficient High Bandwidth DRAM for Throughput Processors,” Ph.D. dissertation, The University of Texas at Austin, 2021. [613] R. R. Oehler and M. W. Blasgen, “IBM RISC System/6000: Architecture and Performance,” in IEEE Micro, 1991. [614] A. Olgun, F. N. Bostanci, G. Francisco de Oliveira Junior, Y. C. Tugrul, R. Bera, A. G. Yaglikci, H. Hassan, O. Ergin, and O. Mutlu, “Sectored DRAM: A Practical EnergyEfficient and High-Performance Fine-Grained DRAM Architecture,” in TACO, 2024.

BIBLIOGRAPHY

211

[615] A. Olgun, J. G. Luna, K. Kanellopoulos, B. Salami, H. Hassan, O. Ergin, and O. Mutlu, “PiDRAM: A Holistic End-to-End FPGA-Based Framework for Processing-in-DRAM,” in TACO, 2022. [616] A. Olgun, M. Patel, A. G. Yağlıkçı, H. Luo, J. S. Kim, F. N. Bostancı, N. Vijaykumar, O. Ergin, and O. Mutlu, “QUAC-TRNG: High-Throughput True Random Number Generation Using Quadruple Row Activation in Commodity DRAMs,” in ISCA, 2021. [617] G. F. Oliveira, “New Tools, Programming Models, and System Support for Processingin-Memory Architectures,” Ph.D. dissertation, ETH Zürich, 2025. [618] G. F. Oliveira, J. Gómez-Luna, L. Orosa, S. Ghose, N. Vijaykumar, I. Fernandez, M. Sadrosadati, and O. Mutlu, “DAMOV: A New Methodology and Benchmark Suite for Evaluating Data Movement Bottlenecks,” in IEEE Access, 2021. [619] G. F. Oliveira, M. Kabra, Y. Guo, K. Chen, A. G. Yağlıkçı, M. Soysal, M. Sadrosadati, J. O. Bueno, S. Ghose, J. Gómez-Luna et al., “Proteus: Enabling High-Performance Processing-Using-DRAM with Dynamic Bit-Precision, Adaptive Data Representation, and Flexible Arithmetic,” in ICS, 2025. [620] G. F. Oliveira, A. Kohli, D. Novo, A. Olgun, A. G. Yaglikci, S. Ghose, J. Gómez-Luna, and O. Mutlu, “DaPPA: A Data-Parallel Programming Framework for Processing-inMemory Architectures,” 2025. [621] G. F. Oliveira, A. Olgun, A. G. Yağlıkçı, F. N. Bostancı, J. Gómez-Luna, S. Ghose, and O. Mutlu, “MIMDRAM: An End-to-End Processing-using-DRAM System for HighThroughput, Energy-Efficient and Programmer-Transparent Multiple-Instruction Multiple-Data Computing,” in HPCA, 2024. [622] G. F. Oliveira, P. C. Santos, M. A. Alves, and L. Carro, “NIM: An HMC-Based Machine for Neuron Computation,” in ARC, 2017. [623] K. Olukotun, B. A. Nayfeh, L. Hammond, K. G. Wilson, and K. Chang, “The Case for a Single-Chip Multiprocessor,” in ASPLOS, 1996. [624] K. O’Neal, P. Brisk, E. Shriver, and M. Kishinevsky, “HALWPE: Hardware-assisted Light Weight Performance Estimation for GPUs,” in DAC, 2017. [625] L. Orosa, R. Azevedo, and O. Mutlu, “AVPP: Address-first Value-next Predictor with Value Prefetching for Improving the Efficiency of Load Value Prediction,” in TACO, 2018. [626] L. Orosa, Y. Wang, M. Sadrosadati, J. Kim, M. Patel, I. Puddu, H. Luo, K. Razavi, J. Gómez-Luna, H. Hassan, N. M. Ghiasi, S. Ghose, and O. Mutlu, “CODIC: A LowCost Substrate for Enabling Custom In-DRAM Functionalities and Optimizations,” in ISCA, 2021. [627] M. Otoom, A. Jaleel, and P. Trancoso, “Using personality metrics to improve cache interference management in multicore processors,” in CF, 2017.

212

BIBLIOGRAPHY

[628] S. Pakalapati and B. Panda, “Bouquet of Instruction Pointers: Instruction Pointer Classifier-based Spatial Hardware Prefetching,” in ISCA, 2020. [629] S. Pal, J. Ryoo, and L. K. John, “The Curious Case of Global Stable Loads,” in IISWC, 2025. [630] T. Palit, Y. Shen, and M. Ferdman, “Demystifying Cloud Benchmarking,” in ISPASS, 2016. [631] G.-Y. Pan, J.-Y. Jou, and B.-C. Lai, “Scalable Power Management using Multilevel Reinforcement Learning for Multiprocessors,” in TODAES, 2014. [632] B. Panda, “SPAC: A Synergistic Prefetcher Aggressiveness Controller for Multi-Core Systems,” in IEEE TC, 2016. [633] B. Panda, “CLIP: Load Criticality Based Data Prefetching for Bandwidth-constrained Many-core Systems,” in MICRO, 2023. [634] B. Panda and S. Balachandran, “CSHARP: Coherence and Sharing Awareness Replacement Policies for Parallel Applications,” in SBAC-PAD, 2012. [635] B. Panda and S. Balachandran, “Hardware Prefetchers for Emerging Parallel Applications,” in PACT, 2012. [636] B. Panda and S. Balachandran, “TCPT: Thread Criticality-Driven Prefetcher Throttling,” in PACT, 2013. [637] B. Panda and S. Balachandran, “Introducing Thread Criticality Awareness in Prefetcher Aggressiveness Control,” in DATE, 2014. [638] B. Panda and S. Balachandran, “XStream: Cross-Core Spatial Streaming Based MLC Prefetchers for Parallel Applications in CMPs,” in PACT, 2014. [639] B. Panda and S. Balachandran, “CAFFEINE: A Utility-Driven Prefetcher Aggressiveness Engine for Multicores,” in ACM TACO, 2015. [640] B. Panda and S. Balachandran, “Expert Prefetch Prediction: An Expert Predicting the Usefulness of Hardware Prefetchers,” in IEEE CAL, 2016. [641] B. Panda and A. Seznec, “Synergistic Cache Layout for Reuse and Compression,” in PACT, 2018. [642] S. Pandey, L. Siddhu, and P. R. Panda, “NeuroCool: Dynamic Thermal Management of 3D DRAM for Deep Neural Networks Through Customized Prefetching,” in ACM TODAES, 2023. [643] G. M. Papadopoulos and K. R. Traub, “Multithreading: A Revisionist View of Dataflow Architectures,” in ISCA, 1991. [644] R. Parihar and M. C. Huang, “Accelerating Decoupled Look-Ahead via Weak Dependence Removal: A Metaheuristic Approach,” in HPCA, 2014.

BIBLIOGRAPHY

213

[645] J. Park, R. Azizi, G. F. Oliveira, M. Sadrosadati, R. Nadig, D. Novo, J. Gómez-Luna, M. Kim, and O. Mutlu, “Flash-Cosmos: In-Flash Bulk Bitwise Operations Using Inherent Computation Capability of NAND Flash Memory,” in MICRO, 2022. [646] G. Pasandi, S. Nazarian, and M. Pedram, “Approximate Logic Synthesis: A Reinforcement Learning-Based Technology Mapping Approach,” in ISQED, 2019. [647] M. Patel, “Enabling Effective Error Mitigation In Memory Chips That Use On-Die Error-Correcting Codes,” Ph.D. dissertation, ETH Zürich, 2022. [648] S. Patel and S. Lumetta, “rePLay: A Hardware Framework for Dynamic Optimization,” in IEEE TC, 2001. [649] Y. N. Patt, W. mei Hwu, and M. C. Shebanow, “HPS, A New Microarchitecture: Rationale and Introduction,” in MICRO: Annual Microprogramming Workshop, 1985. [650] Y. N. Patt, S. W. Melvin, W. Hwu, and M. C. Shebanow, “Critical Issues Regarding HPS, a High Performance Microarchitecture,” in MICRO: Annual Microprogramming Workshop, 1985. [651] D. A. Patterson, T. E. Anderson, N. Cardwell et al., “A Case for Intelligent RAM,” in IEEE Micro, 1997. [652] D. A. Patterson and J. L. Hennessy, Computer Organization and Design: The Hardware Software Interface. Morgan kaufmann, 2016. [653] A. Pattnaik, X. Tang, A. Jog, O. Kayiran, A. K. Mishra, M. T. Kandemir, O. Mutlu, and C. R. Das, “Scheduling Techniques for GPU Architectures with Processing-in-Memory Capabilities,” in PACT, 2016. [654] R. C. Pearce and J. C. Majithia, “Analysis of a Shared Resource MIMD Computer Organization,” in IEEE TC, 1978. [655] J.-K. Peir, S.-C. Lai, S.-L. Lu, J. Stark, and K. Lai, “Bloom Filtering Cache Misses for Accurate Data Speculation and Prefetching,” in ICS, 2002. [656] G. Pekhimenko, “Practical Data Compression for Modern Memory Hierarchies,” Ph.D. dissertation, Carnegie Mellon University, 2016. [657] G. Pekhimenko, E. Bolotin, N. Vijaykumar, O. Mutlu, T. C. Mowry, and S. W. Keckler, “A Case for Toggle-Aware Compression for GPU Systems,” in HPCA, 2016. [658] G. Pekhimenko, T. Huberty, R. Cai, O. Mutlu, P. B. Gibbons, M. A. Kozuch, and T. C. Mowry, “Exploiting Compressed Block Size as an Indicator of Future Reuse,” in HPCA, 2015. [659] G. Pekhimenko, V. Seshadri, O. Mutlu, P. B. Gibbons, M. A. Kozuch, and T. C. Mowry, “Base-Delta-Immediate Compression: Practical Data Compression for OnChip Caches,” in PACT, 2012. [660] L. Peled, S. Mannor, U. Weiser, and Y. Etsion, “Semantic Locality and Context-Based Prefetching using Reinforcement Learning,” in ISCA, 2015.

214

BIBLIOGRAPHY

[661] L. Peled, U. Weiser, and Y. Etsion, “A Neural Network Memory Prefetcher using Semantic Locality,” 2018. [662] D. D. Penney and L. Chen, “A Survey of Machine Learning Applied to Computer Architecture Design,” in arXiv preprint arXiv:1909.12373, 2019. [663] A. Perais, “Leveraging Targeted Value Prediction to Unlock New Hardware Strength Reduction Potential,” in MICRO, 2021. [664] A. Perais and A. Seznec, “Revisiting Value Prediction,” in INRIA Tech. Report, 2012. [665] A. Perais and A. Seznec, “EOLE: Paving the Way for an Effective Implementation of Value Prediction,” in ISCA, 2014. [666] A. Perais and A. Seznec, “Practical Data Value Speculation for Future High-End Processors,” in HPCA, 2014. [667] A. Perais and A. Seznec, “BeBoP: A Cost Effective Predictor Infrastructure for Superscalar Value Prediction,” in HPCA, 2015. [668] A. Perais and A. Seznec, “Cost Effective Speculation with the Omnipredictor,” in PACT, 2018. [669] T. Piquet, O. Rochecouste, and A. Seznec, “Exploiting Single-Usage for Effective Memory Management,” in ACSAC, 2007. [670] G. Posluns and M. Jeffrey, “Global Berti: Simultaneous Streaming and Spatial Prefetching,” in 4th Data Prefetching Championship (DPC4), 2026. [671] S. H. Pugsley, Z. Chishti, C. Wilkerson, P.-f. Chuang, R. L. Scott, A. Jaleel, S.-L. Lu, K. Chow, and R. Balasubramonian, “Sandbox Prefetching: Safe Run-Time Evaluation of Aggressive Prefetchers,” in HPCA, 2014. [672] S. H. Pugsley, J. Jestes, H. Zhang, R. Balasubramonian et al., “NDC: Analyzing the Impact of 3D-Stacked Memory+Logic Devices on MapReduce Workloads,” in ISPASS, 2014. [673] S. Punyamurtula and B. N. Swamy, “Hybrid Prefetch Method and Apparatus,” US Patent, 2013, filed: 2010-09-09; Published: 2013-11-12. [Online]. Available: https://patents.google.com/patent/US8583894B2/en [674] A. Puri, K. Bellamkonda, K. Narreddy, J. Jose, V. Tamarapalli, and V. Narayanan, “DRackSim: Simulating CXL-enabled Large-Scale Disaggregated Memory Systems,” in SIGSIM-PADS, 2023. [675] Z. Purser, K. Sundaramoorthy, and E. Rotenberg, “A Study of Slipstream Processors,” in MICRO, 2000. [676] M. K. Qureshi, A. Jaleel, Y. N. Patt, S. C. S. Jr., and J. S. Emer, “Set-Dueling-Controlled Adaptive Insertion for High-Performance Caching,” in IEEE Micro, 2008.

BIBLIOGRAPHY

215

[677] M. K. Qureshi and G. H. Loh, “Fundamental Latency Trade-off in Architecting DRAM Caches: Outperforming Impractical SRAM-Tags with a Simple and Practical Design,” in MICRO, 2012. [678] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” in ICML, 2021. [679] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” in ICML, 2023. [680] K. Raghavendra, B. Panda, and M. Mutyam, “MBZip: A Case for Compressing Multiple Data Blocks,” in PACT, 2015. [681] K. Raghavendra, B. Panda, and M. Mutyam, “PBC: Prefetched Blocks Compaction,” in IEEE TC, 2016. [682] T. Ramirez, A. Pajuelo, O. Santana, and M. Valero, “Runahead Threads to Improve SMT Performance,” in HPCA, 2008. [683] A. Ranganathan, R. Bera, J. Rakshit, S. Mahto, A. V. Nori, J. Gaur, and S. Subramoney, “Methods and Apparatus for Efficiently Processing Load Instruction Sequences,” 2026, uS Patent App. 18/757,432. [684] P. Ranganathan, D. Stodolsky, J. Calow, J. Dorfman, M. Guevara, C. W. Smullen IV, A. Kuusela, R. Balasubramanian, S. Bhatia, P. Chauhan et al., “Warehouse-Scale Video Acceleration: Co-Design and Deployment in the Wild,” in ASPLOS, 2021. [685] B. R. Rau and J. A. Fisher, “Instruction-Level Parallel Processing: History, Overview, and Perspective,” in J Supercomput., 1993. [686] G. S. Ravi and M. H. Lipasti, “Charstar: Clock Hierarchy Aware Resource Scaling in Tiled Architectures,” in ISCA, 2017. [687] G. Reinman, B. chCalder, D. Tullsen, G. Tyson, and T. Austin, “Classifying Load and Store Instructions for Memory Renaming,” in ICS, 1999. [688] H. Ren and M. Fojtik, “Nvcell: Standard Cell Layout in Advanced Technology Nodes with Reinforcement Learning,” in DAC, 2021. [689] S. H. S. Rezaei, M. Modarressi, R. Ausavarungnirun, M. Sadrosadati, O. Mutlu, and M. Daneshtalab, “NoM: Network-on-Memory for Inter-Bank Data Transfer in HighlyBanked Memories,” in CAL, 2020. [690] S. E. Richardson, “Caching Function Results: Faster Arithmetic by Avoiding Unnecessary Computation,” Sun Microsystems, Inc., Tech. Rep., 1992. [691] S. E. Richardson, “Exploiting Trivial and Redundant Computation,” in ARITH, 1993. [692] J. A. Rivers and E. S. Davidson, “Reducing Conflicts in Direct-Mapped Caches with a Temporality-Based Design,” in ICPP, 1996.

216

BIBLIOGRAPHY

[693] J. A. Rivers, E. S. Tam, G. S. Tyson, E. S. Davidson, and M. Farrens, “Utilizing Reuse Information in Data Cache Management,” in ICS, 1998. [694] S. Rixner, “Memory Controller Optimizations for Web Servers,” in MICRO, 2004. [695] S. Rixner, W. J. Dally, U. J. Kapasi, P. Mattson, and J. D. Owens, “Memory Access Scheduling,” in ISCA, 2000. [696] H. Robbins, “Some Aspects of the Sequential Design of Experiments,” in Bulletin of the American Mathematical Society, 1952. [697] J. Roelandts, A. Naithani, S. Ainsworth, T. M. Jones, and L. Eeckhout, “Scalar Vector Runahead,” in MICRO, 2024. [698] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” in CVPR, 2022. [699] J. P. Rosa, D. J. Guerra, N. C. Horta, R. M. Martins, N. C. Lourenço et al., Using Artificial Neural Networks for Analog Integrated Circuit Design Automation. Springer, 2020. [700] F. Rosenblatt, “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain,” in Psychological review, 1958. [701] P. Rosenfeld, “Performance Exploration of the Hybrid Memory Cube,” Ph.D. dissertation, University of Maryland, 2014. [702] E. Rotenberg, Q. Jacobson, Y. Sazeides, and J. Smith, “Trace Processors,” in MICRO, 1997. [703] E. Rotenberg and J. Smith, “Control Independence in Trace Processors,” in MICRO, 1999. [704] A. Roth, A. Moshovos, and G. S. Sohi, “Dependance Based Prefetching for Linked Data Structures,” in ASPLOS, 1998. [705] A. Roth, A. Moshovos, and G. S. Sohi, “Improving virtual function call target prediction via dependence-based pre-computation,” in ICS, 1999. [706] A. Roth and G. S. Sohi, “Effective Jump-Pointer Prefetching for Linked Data Structures,” in ISCA, 1999. [707] A. Roth and G. S. Sohi, “Speculative Data-Driven Multithreading,” in HPCA, 2001. [708] M. Rotman and L. Wolf, “Electric Analog Circuit Design with Hypernetworks and a Differential Simulator,” in ICASSP, 2020. [709] N. Roy, A. Dubey, and A. Gokhale, “Efficient Autoscaling in the Cloud using Predictive Models for Workload Forecasting,” in IEEE CLOUD, 2011. [710] S. Roy, V. Ahuja, and S. Banerjee, “Data Cache Prefetch Controller,” US Patent, 2016, filed: 2014-02-20; Published: 2016-03-22. [Online]. Available: https://patents.google.c om/patent/US9292447B2/en [711] G. A. Rummery and M. Niranjan, On-line Q-learning using Connectionist Systems. University of Cambridge, Department of Engineering Cambridge, UK, 1994.

BIBLIOGRAPHY

217

[712] C. Sakhuja, A. Subramanian, P. Joshi, A. Jain, and C. Lin, “Combining Branch History and Value History for Improved Value Prediction,” in Second Championship Value Prediction, 2019. [713] S. Samudrala, J. Wu, C. Chen, H. Shan, J. Ku, Y. Chen, and J. Rajendran, “Performance Analysis of Zero-Knowledge Proofs,” in IISWC, 2024. [714] D. Sanchez and C. Kozyrakis, “SCD: A Scalable Coherence Directory With Flexible Sharer Set Encoding,” in HPCA, 2012. [715] F. J. Sánchez and A. González, “Software Data Prefetching for Software Pipelined Loops,” in J. Parallel Distributed Comput., 1999. [716] P. C. Santos, G. F. Oliveira, J. P. Lima, M. A. Alves, L. Carro, and A. C. Beck, “Processing in 3D Memories to Speed Up Operations on Complex Data Structures,” in DATE, 2018. [717] P. C. Santos, G. F. Oliveira, D. G. Tomé, M. A. Z. Alves, E. C. Almeida, and L. Carro, “Operand Size Reconfiguration for Big Data Processing in Memory,” in DATE, 2017. [718] S. H. Sardashti, A. Seznec, and D. A. Wood, “Skewed Compressed Caches,” in MICRO, 2014. [719] K. Sato and N. Yamasaki, “A Learning-based Control Scheme for Prioritized SMT Processor,” in CANDARW, 2024. [720] Y. Sazeides and J. E. Smith, “The Predictability of Data Values,” in MICRO, 1997. [721] Y. Sazeides and J. E. Smith, “Limits of Data Value Predictability,” in IJPP, 1999. [722] Y. Sazeides, S. Vassiliadis, and J. E. Smith, “The Performance Potential of Data Dependence Speculation and Collapsing,” in MICRO, 1996. [723] D. Schall, M. Durackova, and B. Grot, “The Last-Level Branch Predictor Revisited,” in HPCA, 2026. [724] D. Schall, A. Margaritov, D. Ustiugov, A. Sandberg, and B. Grot, “Lukewarm Serverless Functions: Characterization and Optimization,” in ISCA, 2022. [725] D. Schall, A. Sandberg, and B. Grot, “Warming Up a Cold Front-End with Ignite,” in MICRO, 2023. [726] D. Schall, A. Sandberg, and B. Grot, “The Last-Level Branch Predictor,” in MICRO, 2024. [727] G. A. Seber and A. J. Lee, Linear Regression Analysis.

John Wiley & Sons, 2012.

[728] A. Sembrant, E. Hagersten, and D. Black-Schaffer, “The Direct-To-Data (D2D) Cache: Navigating the Cache Hierarchy with a Single Lookup,” in ISCA, 2014. [729] A. Sembrant, E. Hagersten, and D. Black-Schaffer, “A Split Cache Hierarchy for Enabling Data-Oriented Optimizations,” in HPCA, 2017. [730] R. Sen and Y. Tian, “Microarchitectural Analysis of Graph BI Queries on RDBMS,” in DaMoN, 2023.

218

BIBLIOGRAPHY

[731] M. J. Serrano, W. Yamamoto, R. C. Wood, and M. Nemirovsky, “A Model for Performance Estimation in a Multistreamed Superscalar Processor,” in TOOLS, G. Haring and G. Kotsis, Eds., 1994. [732] V. Seshadri, A. Boroumand, D. Lee, O. Mutlu, P. B. Gibbons, M. A. Kozuch, and T. C. Mowry, “Ambit: In-Memory Accelerator for Bulk Bitwise Operations Using Commodity DRAM Technology,” in MICRO, 2017. [733] V. Seshadri, A. Boroumand, O. Mutlu, P. B. Gibbons, M. A. Kozuch, and T. C. Mowry, “Buddy-RAM: Improving the Performance and Efficiency of Bulk Bitwise Operations Using DRAM,” in arXiv, 2016. [734] V. Seshadri, K. Hsieh, A. Boroumand, D. Lee, M. A. Kozuch, O. Mutlu, P. B. Gibbons, and T. C. Mowry, “Fast Bulk Bitwise AND and OR in DRAM,” in IEEE CAL, 2015. [735] V. Seshadri, Y. Kim, C. Fallin, D. Lee, R. Ausavarungnirun, G. Pekhimenko, Y. Luo, O. Mutlu, P. B. Gibbons, M. A. Kozuch et al., “RowClone: Fast and Energy-Efficient in-DRAM Bulk Data Copy and Initialization,” in MICRO, 2013. [736] V. Seshadri, Y. Kim, C. Fallin, D. Lee, R. Ausavarungnirun, G. Pekhimenko, Y. Luo, O. Mutlu, P. B. Gibbons, M. A. Kozuch et al., “RowClone: Accelerating Data Movement and Initialization Using DRAM,” arXiv:1805.03502 [cs.AR], 2018. [737] V. Seshadri, T. Mullins, A. Boroumand, O. Mutlu, P. B. Gibbons, M. A. Kozuch, and T. C. Mowry, “Gather-Scatter DRAM: In-DRAM Address Translation to Improve the Spatial Locality of Non-Unit Strided Accesses,” in MICRO, 2015. [738] V. Seshadri and O. Mutlu, “The Processing Using Memory Paradigm: In-DRAM Bulk Copy, Initialization, Bitwise AND and OR,” arXiv:1610.09603 [cs.AR], 2016. [739] V. Seshadri and O. Mutlu, “Simple Operations in Memory to Reduce Data Movement,” in Advances in Computers, Volume 106. Elsevier, 2017. [740] V. Seshadri and O. Mutlu, “In-DRAM Bulk Bitwise Execution Engine,” in CoRR, 2019. [Online]. Available: http://arxiv.org/abs/1905.09822 [741] V. Seshadri, O. Mutlu, M. A. Kozuch, and T. C. Mowry, “The Evicted-Address Filter: A Unified Mechanism to Address Both Cache Pollution and Thrashing,” in PACT, 2012. [742] V. Seshadri, S. Yedkar, H. Xin, O. Mutlu, P. B. Gibbons, M. A. Kozuch, and T. C. Mowry, “Mitigating Prefetcher-Caused Pollution using Informed Caching Policies for Prefetched Blocks,” in TACO, 2015. [743] S. Sethumurugan, J. Yin, and J. Sartori, “Designing a Cost-Effective Cache Replacement Policy using Machine Learning,” in HPCA, 2021. [744] K. Settaluri, A. Haj-Ali, Q. Huang, K. Hakhamaneshi, and B. Nikolic, “Autockt: Deep Reinforcement Learning of Analog Circuit Designs,” in arXiv preprint arXiv:2001.01808, 2020. [745] A. Seznec, “A Case for Two-Way Skewed-Associative Cache,” in ISCA, 1993.

BIBLIOGRAPHY

219

[746] A. Seznec, “About Set and Skewed Associativity on Second-Level Caches,” in ICCD, 1993. [747] A. Seznec, “Decoupled Sectored Caches: Reconciling Low Tag Volume and Low Miss Ratio,” in ISCA, 1994. [748] A. Seznec, “DASC Cache,” in HPCA, 1995. [749] A. Seznec, “Don’t Use the Page Number, But a Pointer to It,” in ISCA, 1996. [750] A. Seznec, “A New Case for Skewed-Associativity,” IRISA, Tech. Rep., 1997. [751] A. Seznec, “Decoupled Sectored Caches,” in IEEE Transactions on Computers, 1997. [752] A. Seznec, “Concurrent Support of Multiple Page Sizes on a Skewed Associative TLB,” in IEEE Transactions on Computers, 2003. [753] A. Seznec, “A New Case for the TAGE Branch Predictor,” in MICRO, 2011. [754] A. Seznec, “Exploring Value Prediction With the EVES Predictor,” in 1st Championship Value Prediction (CVP-1), 2018. [755] A. Seznec and F. Bodin, “Skewed-Associative Caches,” in PARLE, 1993. [756] A. Seznec and F. Lloansi, “About Effective Miss Penalty on Out-of-Order Microprocessors,” IRISA, Tech. Rep., 1995. [757] A. Seznec and P. Michaud, “A Case for (Partially) TAgged GEometric History Length Branch Prediction,” in JILP, 2006. [758] T. Sha, M. M. K. Martin, and A. Roth, “NoSQ: Store-Load Communication without a Store Queue,” in MICRO, 2006. [759] A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, “ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars,” in ISCA, 2016. [760] A. Shahab, M. Zhu, A. Margaritov, and B. Grot, “Farewell My Shared LLC! A Case for Private Die-Stacked DRAM Caches for Servers,” in MICRO, 2018. [761] M. Shakerinava, M. Bakhshalipour, P. Lotfi-Kamran, and H. Sarbazi-Azad, “Multilookahead Offset Prefetching,” in 3rd Data Prefetching Championship, 2019. [762] M. Sharad, D. Fan, and K. Roy, “Ultra Low Power Associative Computing with Spin Neurons and Resistive Crossbar Memory,” in DAC, 2013. [763] R. Sheikh, H. W. Cain, and R. Damodaran, “Load Value Prediction via Path-Based Address Prediction: Avoiding Mispredictions Due to Conflicting Stores,” in MICRO, 2017. [764] R. Sheikh and D. Hower, “Efficient Load Value Prediction Using Multiple Predictors and Filters,” in HPCA, 2019.

220

BIBLIOGRAPHY

[765] M. Shevgoor, S. Koladiya, R. Balasubramonian, C. Wilkerson, S. H. Pugsley, and Z. Chishti, “Efficiently Prefetching Complex Address Patterns,” in MICRO, 2015. [766] Z. Shi, X. Huang, A. Jain, and C. Lin, “Applying Deep Learning to the Cache Replacement Problem,” in MICRO, 2019. [767] Z. Shi, A. Jain, K. Swersky, M. Hashemi, P. Ranganathan, and C. Lin, “A Hierarchical Neural Model of Data Prefetching,” in ASPLOS, 2021. [768] Z. Shi, K. Swersky, D. Tarlow, P. Ranganathan, and M. Hashemi, “Learning Execution Through Neural Code Fusion,” 2019. [769] H. Shin, D. Kim, E. Park, S. Park, Y. Park, and S. Yoo, “McDRAM: Low Latency and Energy-Efficient Matrix Computations in DRAM,” in IEEE TCADICS, 2018. [770] S. Shukla, S. Bandishte, J. Gaur, and S. Subramoney, “Register File Prefetching,” in ISCA, 2022. [771] J. Shun and G. E. Blelloch, “Ligra: a Lightweight Graph Processing Framework for Shared Memory,” in PPoPP, 2013. [772] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al., “Mastering the Game of Go with Deep Neural Networks and Tree Search,” in Nature, 2016. [773] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel et al., “A General Reinforcement Learning Algorithm That Masters Chess, Shogi, and Go Through Self-Play,” in Science, 2018. [774] G. Singh, D. Diamantopoulos, C. Hagleitner, J. Gomez-Luna, S. Stuijk, O. Mutlu, and H. Corporaal, “NERO: A Near High-Bandwidth Memory Stencil Accelerator for Weather Prediction Modeling,” in FPL, 2020. [775] G. Singh, J. Gómez-Luna, G. Mariani, G. F. Oliveira, S. Corda, S. Stuijk, O. Mutlu, and H. Corporaal, “NAPEL: Near-Memory Computing Application Performance Prediction via Ensemble Learning,” in DAC, 2019. [776] G. Singh, R. Nadig, J. Park, R. Bera, N. Hajinazar, D. Novo, J. Gómez-Luna, S. Stuijk, H. Corporaal, and O. Mutlu, “Sibyl: Adaptive and Extensible Data Placement in Hybrid Storage Systems using Online Reinforcement Learning,” in ISCA, 2022. [777] S. Singh, A. N. Torres, and A. Ros, “Pushing the Limits of the Berti Prefetcher,” in 4th Data Prefetching Championship (DPC4), 2026. [778] U. Sirin and A. Ailamaki, “Micro-architectural Analysis of OLAP: Limitations and Opportunities,” in VLDB, 2020. [779] U. Sirin, A. Yasin, and A. Ailamaki, “A Methodology for OLTP Micro-Architectural Analysis,” in DAMON, 2017. [780] A. J. Smith, “Sequential Program Prefetching in Memory Hierarchies,” in Computer, 1978.

BIBLIOGRAPHY

221

[781] B. J. Smith, “A Pipelined, Shared Resource MIMD Computer,” in ICPP, 1978. [782] B. J. Smith, “Architecture and Applications of the HEP Multiprocessor Computer System,” in Real-Time signal processing IV, 1982. [783] J. E. Smith, “A Study of Branch Prediction Strategies,” in ISCA, 1981. [784] J. E. Smith and S. Vajapeyam, “Trace Processors: Moving to Fourth-Generation Microarchitectures,” in Computer, 2002. [785] M. D. Smith, M. Johnson, and M. A. Horowitz, “Limits on Multiple Instruction Issue,” in ASPLOS, 1989. [786] A. Snavely and D. M. Tullsen, “Symbiotic Job Scheduling for A Simultaneous Mutlithreading Processor,” in SIGP, 2000. [787] A. Sodani and G. S. Sohi, “Dynamic Instruction Reuse,” in ISCA, 1997. [788] A. Sodani and G. S. Sohi, “An Empirical Analysis of Instruction Repetition,” in ASPLOS, 1998. [789] A. Sodani and G. S. Sohi, “Understanding the Differences Between Value Prediction and Instruction Reuse,” in MICRO, 1998. [790] Y. Solihin, J. Lee, and J. Torrellas, “Using a User-Level Memory Thread for Correlation Prefetching,” in ISCA, 2002. [791] S. Somogyi, T. F. Wenisch, A. Ailamaki, and B. Falsafi, “Spatio-Temporal Memory Streaming,” in ISCA, 2009. [792] S. Somogyi, T. F. Wenisch, A. Ailamaki, B. Falsafi, and A. Moshovos, “Spatial Memory Streaming,” in ISCA, 2006. [793] Y. H. Son, S. O, Y. Ro, J. W. Lee, and J. H. Ahn, “Reducing Memory Access Latency with Asymmetric DRAM Bank Organizations,” in ISCA, 2013. [794] K. Song, J. Yang, Z. Wang, J. Zhao, S. Liu, and G. Pekhimenko, “FreqTier: Lightweight Adaptive Tiering for CXL Memory Systems,” in NVMW, 2025. [795] L. Song, X. Qian, H. Li, and Y. Chen, “PipeLayer: A Pipelined ReRAM-Based Accelerator for Deep Learning,” in HPCA, 2017. [796] L. Song, Y. Zhuo, X. Qian, H. Li, and Y. Chen, “GraphR: Accelerating Graph Processing Using ReRAM,” in HPCA, 2018. [797] S. Song, T. A. Khan, S. M. Shahri, A. Sriraman, N. K. Soundararajan, S. Subramoney, D. A. Jiménez, H. Litz, and B. Kasikci, “Thermometer: Profile-Guided BTB Replacement for Data Center Applications,” in ISCA, 2022. [798] A. Sridharan, B. Panda, and A. Seznec, “Band-Pass Prefetching: An Effective Prefetch Management Mechanism Using Prefetch-Fraction in Multicore Systems,” in ACM TACO, 2017.

222

BIBLIOGRAPHY

[799] S. Srinath, O. Mutlu, H. Kim, and Y. N. Patt, “Feedback Directed Prefetching: Improving the Performance and Bandwidth-Efficiency of Hardware Prefetchers,” in HPCA, 2007. [800] S. T. Srinivasan, R. D.-C. Ju, A. R. Lebeck, and C. Wilkerson, “Locality vs. Criticality,” in ISCA, 2001. [801] S. T. Srinivasan and A. R. Lebeck, “Load Latency Tolerance in Dynamically Scheduled Processors,” in MICRO, 1998. [802] S. T. Srinivasan, R. Rajwar, H. Akkary, A. Gandhi, and M. Upton, “Continual Flow Pipelines,” in ASPLOS, 2004. [803] A. Stillmaker and B. Baas, “Scaling Equations for the Accurate Prediction of CMOS Device Performance from 180nm to 7nm,” in Integration, 2017. [804] H. S. Stone, “A Logic-in-Memory Computer,” in IEEE TC, 1970. [805] W. Su, A. Dhanotia, C. Torres, J. Gandhi, N. Gholkar, S. Kanaujia, M. Naumov, K. Subramanian, V. Andrei, Y. Yuan, and C. Tang, “DCPerf: An Open-Source, Battle-Tested Performance Benchmark Suite for Datacenter Workloads,” in ISCA, 2025. [806] S. Subramaniam and G. H. Loh, “Store Vectors for Scalable Memory Dependence Prediction and Scheduling,” in HPCA, 2006. [807] L. Subramanian, D. Lee, V. Seshadri, H. Rastogi, and O. Mutlu, “The Blacklisting Memory Scheduler: Achieving High Performance and Fairness at Low Cost,” in ICCD, 2014. [808] L. Subramanian, D. Lee, V. Seshadri, H. Rastogi, and O. Mutlu, “BLISS: Balancing Performance, Fairness and Complexity in Memory Access Scheduling,” in TPDS, 2016. [809] L. Subramanian, V. Seshadri, A. Ghosh, S. Khan, and O. Mutlu, “The Application Slowdown Model: Quantifying and Controlling the Impact of Inter-Application Interference at Shared Caches and Main Memory,” in MICRO, 2015. [810] L. Subramanian, V. Seshadri, Y. Kim, B. Jaiyen, and O. Mutlu, “MISE: Providing Performance Predictability and Improving Fairness in Shared Main Memory Systems,” in HPCA, 2013. [811] A. Subramaniyan and R. Das, “Parallel Automata Processor,” in ISCA, 2017. [812] A. Subramaniyan, Y. Gu, T. Dunn, S. Paul, M. Vasimuddin, S. Misra, D. Blaauw, S. Narayanasamy, and R. Das, “GenomicsBench: A Benchmark Suite for Genomics,” in ISPASS, 2021. [813] M. A. Suleman, O. Mutlu, J. A. Joao, and Y. N. Patt, “Data Marshaling for Multi-Core Architectures,” in ISCA, 2010. [814] M. A. Suleman, O. Mutlu, M. K. Qureshi, and Y. N. Patt, “Accelerating Critical Section Execution with Asymmetric Multi-Core Architectures,” in ASPLOS, 2009.

BIBLIOGRAPHY

223

[815] W. Sun, Z. Li, S. Yin, S. Wei, and L. Liu, “ABC-DIMM: Alleviating the Bottleneck of Communication in DIMM-Based Near-Memory Processing with Inter-DIMM Broadcast,” in ISCA, 2021. [816] Y. Sun, Y. Yuan, Z. Yu, R. Kuper, C. Song, J. Huang, H. Ji, S. Agarwal, J. Lou, I. Jeong, R. Wang, J. H. Ahn, T. Xu, and N. S. Kim, “Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices,” in MICRO, 2023. [817] K. Sundaramoorthy, Z. Purser, and E. Rotenberg, “Slipstream Processors: Improving Both Performance and Fault Tolerance,” in ASPLOS, 2000. [818] A. Suresh, E. Rohou, and A. Seznec, “Compile-Time Function Memoization,” in ICCC, 2017. [819] A. Suresh, B. N. Swamy, E. Rohou, and A. Seznec, “Intercepting Functions for Memoization: A Case Study using Transcendental Functions,” in TACO, 2015. [820] R. S. Sutton and A. G. Barto, “Reinforcement Learning: An Introduction,” in The MIT Press, 2017. [821] Sweta, P. Priyadarshini, and B. Panda, “Drishti: Do Not Forget Slicing While Designing Last-Level Cache Replacement Policies for Many-Core Systems,” in MICRO, 2025. [822] P. P. Tang, H. G. Rotithor, R. L. Carlson, and N. Aboulenein, “Prefetch Optimization in Shared Resource Multi-Core Systems,” US Patent, 2014, filed: 2013-04-16; Published: 2014-12-30. [Online]. Available: https://patents.google.com/patent/US8924651B2/en [823] D. Tarjan and K. Skadron, “Merging Path and Gshare Indexing in Perceptron Branch Prediction,” in TACO, 2005. [824] S. J. Tarsa, C.-K. Lin, G. Keskin, G. Chinya, and H. Wang, “Improving Branch Prediction By Modeling Global History with Convolutional Neural Networks,” 2019. [825] E. Teran, Z. Wang, and D. A. Jiménez, “Perceptron Learning for Reuse Prediction,” in MICRO, 2016. [826] B. Testa, S. M. Ajorpaz, and D. A. Jiménez, “Dynamic set stealing to improve cache performance,” in SBAC-PAD, 2022. [827] W. R. Thompson, “On the Likelihood That One Unknown Probability Exceeds Another in View of the Evidence of Two Samples,” in Biometrika, 1933. [828] J. E. Thornton, “Parallel Operation in the Control Data 6600,” in AFIPS, 1964. [829] Y. Tian and D. A. Jiménez, “Sampling Temporal Touch Hint (STTH) Inclusive Cache Management Policy,” in PACT, L. Rauchwerger and V. Sarkar, Eds., 2011. [830] Y. Tian, S. M. Khan, and D. A. Jiménez, “Temporal-Based Multilevel Correlating Inclusive Cache Replacement,” in TACO, 2013. [831] G. Tjaden and M. Flynn, “Detection and Parallel Execution of Independent Instructions,” in IEEE TC, 1970.

224

BIBLIOGRAPHY

[832] R. M. Tomasulo, “An Efficient Algorithm for Exploiting Multiple Arithmetic Units,” in IBM Journal of Research and Development, 1967. [833] A. N. Torres, S. Singh, B. Panda, and A. Ros, “The Entangling Data Prefetcher,” in 4th Data Prefetching Championship (DPC4), 2026. [834] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models,” in arXiv preprint arXiv:2307.09288, 2023. [835] M. S. Truong, E. Chen, D. Su, L. Shen, A. Glass, L. R. Carley, J. A. Bain, and S. Ghose, “RACER: Bit-Pipelined Processing using Resistive Memory,” in MICRO, 2021. [836] M. S. Truong, L. Shen, A. Glass, A. Hoffmann, L. R. Carley, J. A. Bain, and S. Ghose, “Adapting the RACER Architecture to Integrate Improved In-ReRAM Logic Primitives,” in JETCAS, 2022. [837] P.-A. Tsai, C. Chen, and D. Sanchez, “Adaptive Scheduling for Systems with Asymmetric Memory Hierarchies,” in MICRO, 2018. [838] D. M. Tullsen and J. A. Brown, “Handling Long-Latency Loads in a Simultaneous Multithreading Processor,” in MICRO, 2001. [839] D. M. Tullsen, S. J. Eggers, J. S. Emer, H. M. Levy, J. L. Lo, and R. L. Stamm, “Exploiting Choice: Instruction Fetch and Issue on an Implementable Simultaneous Multithreading Processor,” in ISCA, 1996. [840] D. M. Tullsen, S. J. Eggers, and H. M. Levy, “Simultaneous Multithreading: Maximizing On-Chip Parallelism,” in ISCA, 1995. [841] G. S. Tyson and T. M. Austin, “Improving the Accuracy and Performance of Memory Communication Through Renaming,” in MICRO, 1997. [842] G. S. Tyson, M. Farrens, J. Matthews, and A. R. Pleszkun, “A Modified Approach to Data Cache Management,” in MICRO, 1995. [843] G. S. Tyson, “The Effects of Predicated Execution on Branch Prediction,” in MICRO, 1994. [844] G. S. Tyson and T. M. Austin, “Memory Renaming: Fast, Early and Accurate Processing of Memory Communication,” in IJPP, 1999. [845] G. S. Tyson, M. Farrens, J. Matthews, and A. R. Pleszkun, “Managing Data Caches using Selective Cache Line Replacement,” in IJPP, 1997. [846] J. Umeike, N. Patel, A. Manley, A. Mamandipoor, H. Yun, and M. Alian, “Profiling gem5 Simulator,” in ISPASS, 2023. [847] H. Usui, L. Subramanian, K. Chang, and O. Mutlu, “SQUASH: Simple QoS-Aware High-Performance Memory Scheduler for Heterogeneous Systems with Hardware Accelerators,” 2015. [Online]. Available: https://arxiv.org/abs/1505.07502

BIBLIOGRAPHY

225

[848] H. Usui, L. Subramanian, K. K.-W. Chang, and O. Mutlu, “DASH: Deadline-Aware HighPerformance Memory Scheduler for Heterogeneous Systems With Hardware Accelerators,” in TACO, 2016. [849] S. Van Winkle, A. K. Kodi, R. Bunescu, and A. Louri, “Extending the Power-efficiency and Performance of Photonic Interconnects for Heterogeneous Multicores with Machine Learning,” in HPCA, 2018. [850] V. N. Vapnik, “An Overview of Statistical Learning Theory,” in IEEE Transactions on Neural Networks, 1999. [851] D. A. Varkey, B. Panda, and M. Mutyam, “RCTP: Region Correlated Temporal Prefetcher,” in ICCD, 2017. [852] G. Vavouliotis, G. Chacon, L. Alvarez, P. V. Gratz, D. A. Jiménez, and M. Casas, “Page size aware cache prefetching,” in MICRO, 2022. [853] D. Vengerov, “A Reinforcement Learning Framework for Utility-Based Scheduling in Resource-Constrained Systems,” in Future Generation Computer Systems, 2009. [854] Y. Verma, D. Mishra, and M. Chaudhuri, “LeakyRand: An Efficient High-Fidelity Covert Channel in Fully Associative Last-Level Caches with Random Eviction,” in ACM Transactions on Embedded Computing Systems, 2025. [855] N. Vijaykumar, G. Pekhimenko, A. Jog, A. Bhowmick, R. Ausavarungnirun, C. Das, M. Kandemir, T. C. Mowry, and O. Mutlu, “A Case for Core-Assisted Bottleneck Acceleration in GPUs: Enabling Flexible Data Compression with Assist Warps,” in ISCA, 2015. [856] L. Villa, M. Zhang, and K. Asanović, “Dynamic Zero Compression for Cache Energy Reduction,” in MICRO, 2000. [857] S. Volos, D. Jevdjic, B. Falsafi, and B. Grot, “Fat Caches for Scale-Out Servers,” in IEEE Micro, 2017. [858] S. Wallace, B. Calder, and D. M. Tullsen, “Threaded Multiple Path Execution,” in ISCA, 1998. [859] H. Wang, K. Wang, J. Yang, L. Shen, N. Sun, H.-S. Lee, and S. Han, “GCN-RL Circuit Designer: Transferable Transistor Sizing with Graph Neural Networks and Reinforcement Learning,” in DAC, 2020. [860] L. Wang, J. Zhan, C. Luo, Y. Zhu, Q. Yang, Y. He, W. Gao, Z. Jia, Y. Shi, S. Zhang et al., “BigDataBench: A Big Data Benchmark Suite from Internet Services,” in HPCA, 2014. [861] P. H. Wang, J. D. Collins, H. Wang, D. Kim, B. Greene, K.-M. Chan, A. B. Yunus, T. Sych, S. F. Moore, and J. P. Shen, “Helper Threads via Virtual Multithreading on an Experimental Itanium®2 Processor-based Platform,” in ASPLOS, 2004. [862] Y. Wang, L. Orosa, X. Peng, Y. Guo, S. Ghose, M. Patel, J. S. Kim, J. G. Luna, M. Sadrosadati, N. M. Ghiasi et al., “FIGARO: Improving System Performance via Fine-Grained In-DRAM Data Relocation and Caching,” in MICRO, 2020.

226

BIBLIOGRAPHY

[863] Y. Wang, A. Tavakkol, L. Orosa, S. Ghose, N. M. Ghiasi, M. Patel, J. S. Kim, H. Hassan, M. Sadrosadati, and O. Mutlu, “Reducing DRAM Latency via Charge-Level-Aware Look-Ahead Partial Restoration,” in MICRO, 2018. [864] Z. Wang, S. M. Khan, and D. A. Jiménez, “Improving Writeback Efficiency with Decoupled Last-Write Prediction,” in ISCA, 2012. [865] Z. Wang, S. M. Khan, and D. A. Jiménez, “Rank Idle Time Prediction Driven Last-Level Cache Writeback,” in PLDI, 2012. [866] Z. Wang, J. Weng, J. Lowe-Power, J. Gaur, and T. Nowatzki, “Stream Floating: Enabling Proactive and Decentralized Cache Optimizations,” in HPCA, 2021. [867] T. F. Wenisch, M. Ferdman, A. Ailamaki, B. Falsafi, and A. Moshovos, “Temporal Streams in Commercial Server Applications,” in IISWC, 2008. [868] T. F. Wenisch, M. Ferdman, A. Ailamaki, B. Falsafi, and A. Moshovos, “Practical OffChip Meta-Data for Temporal Memory Streaming,” in HPCA, 2009. [869] T. F. Wenisch, M. Ferdman, A. Ailamaki, B. Falsafi, and A. Moshovos, “Making AddressCorrelated Prefetching Practical,” in IEEE Micro, 2010. [870] T. F. Wenisch, S. Somogyi, N. Hardavellas, J. Kim, A. Ailamaki, and B. Falsafi, “Temporal Streaming of Shared Memory,” in ISCA, 2005. [871] S. Whiteson and P. Stone, “Adaptive Job Routing and Scheduling,” in Engineering Applications of Artificial Intelligence, 2004. [872] M. V. Wilkes, “Slave Memories and Dynamic Storage Allocation,” in IEEE Transactions on Electronic Computers, 1965. [873] J.-Y. Won, X. Chen, P. Gratz, J. Hu, and V. Soteriou, “Up by Their Bootstraps: Online Learning in Artificial Neural Networks for CMP Uncore Power Management,” in HPCA, 2014. [874] W. A. Wong and J.-L. Baer, “Modified LRU Policies for Improving Second-level Cache Behavior,” in HPCA, 2000. [875] D. A. Wood, S. J. Eggers, G. Gibson, M. D. Hill, and J. M. Pendleton, “An In-Cache Address Translation Mechanism,” in ISCA, 1986. [876] C.-J. Wu, A. Jaleel, W. Hasenplaugh, M. Martonosi, S. C. Steely Jr, and J. Emer, “SHiP: Signature-based Hit Predictor for High Performance Caching,” in MICRO, 2011. [877] C.-J. Wu, A. Jaleel, M. Martonosi, S. C. Steely Jr, and J. Emer, “PACMan: Prefetch-Aware Cache Management for High Performance Caching,” in MICRO, 2011. [878] G. Wu, Y. Xu, D. Wu, M. Ragupathy, Y.-y. Mo, and C. Chu, “Flip-flop Clustering by Weighted K-means Algorithm,” in DAC, 2016. [879] G. Wu, J. L. Greathouse, A. Lyashevsky, N. Jayasena, and D. Chiou, “GPGPU Performance and Power Estimation using Machine Learning,” in HPCA, 2015.

BIBLIOGRAPHY

227

[880] H. Wu, Z. Ji, S. Zhu, and Z. Chen, “A Performance Study of Software Prefetching for Tracing Garbage Collectors,” in Advanced Parallel Programming Technologies, 2013. [881] H. Wu, K. Nathella, J. Pusdesris, D. Sunwoo, A. Jain, and C. Lin, “Temporal Prefetching Without the Off-Chip Metadata,” in MICRO, 2019. [882] H. Wu, K. Nathella, D. Sunwoo, A. Jaleel, and C. Lin, “Efficient Metadata Management for Irregular Data Prefetching,” in ISCA, 2019. [883] N. Wu and Y. Xie, “A Survey of Machine Learning for Computer Architecture and Systems,” in ACM CSUR, 2022. [884] N. Wu, Y. Xie, and C. Hao, “Ironman: GNN-assisted Design Space Exploration in HighLevel Synthesis via Reinforcement Learning,” in GLSVLSI, 2021. [885] W. Wu and B. C. Lee, “Inferred Models for Dynamic and Sparse Hardware-Software Spaces,” in MICRO, 2012. [886] Y. Wu, X. Lu, X. Chen, Y. Han, and X.-H. Sun, “Concurrency-Aware Cache Miss Cost Prediction with Perceptron Learning,” in GLSVLSI, 2025. [887] H. Xi, “Dead Code Elimination Through Dependent Types,” in PADL, 1999. [888] S. L. Xi, O. O. Babarinsa, M. Athanassoulis, and S. Idreos, “Beyond the Wall: Near-Data Processing for Databases,” in Int. Workshop on Data Management on New Hardware, 2015. [889] X. Xin, Y. Zhang, and J. Yang, “ELP2IM: Efficient and Low Power Bitwise Operation Processing in DRAM,” in HPCA, 2020. [890] S. Xing and Y. Wang, “Proactive Data Placement in Heterogeneous Storage Systems via Predictive Multi-Objective Reinforcement Learning,” in IEEE Access, 2025. [891] Y. Xu, K. Sui, R. Yao, H. Zhang, Q. Lin, Y. Dang, P. Li, K. Jiang, W. Zhang, J.-G. Lou et al., “Improving Service Availability of Cloud Systems by Predicting Disk Error,” in USENIX ATC, 2018. [892] F. Xue, J. Wu, C. Han, X. Li, T. Zhang, T. Liu, and F. Zhang, “Augur: Semantics-Aware Temporal Prefetching for Linked Data Structure,” in TACO, 2025. [893] A. G. Yağlikçi, A. Olgun, M. Patel, H. Luo, H. Hassan, L. Orosa, O. Ergin, and O. Mutlu, “HiRA: Hidden Row Activation for Reducing Refresh Latency of Off-The-Shelf DRAM Chips,” in MICRO, 2022. [894] W. Yamamoto and M. Nemirovsky, “Increasing Superscalar Performance Through Multistreaming,” in PACT, 1995. [895] W. Yamamoto, M. Serrano, A. Talcott, R. Wood, and M. Nemirovsky, “Performance Estimation of Multistreamed, Superscalar Processors,” in HICSS, 1994.

228

BIBLIOGRAPHY

[896] H. Yang, J. Fang, Y. Hou, X. Su, and N. N. Xiong, “Reinforcement Learning-Driven Adaptive Prefetch Aggressiveness Control for Enhanced Performance in Parallel System Architectures,” in IEEE TPDS, 2025. [897] H. Yang, J. Fang, X. Su, Z. Cai, and Y. Wang, “RL-CoPref: A Reinforcement LearningBased Coordinated Prefetching Controller for Multiple Prefetchers,” in J Supercomput., 2024. [898] J. Yang, Y. Zhang, and R. Gupta, “Frequent Value Compression in Data Caches,” in MICRO, 2000. [899] A. Yazdanbakhsh, C. Angermueller, B. Akin, Y. Zhou, A. Jones, M. Hashemi, K. Swersky, S. Chatterjee, R. Narayanaswami, and J. Laudon, “Apollo: Transferable Architecture Exploration,” in arXiv preprint arXiv:2102.01723, 2021. [900] T.-Y. Yeh and Y. N. Patt, “Two-level Adaptive Training Branch Prediction,” in MICRO, 1991. [901] T.-Y. Yeh and Y. N. Patt, “Alternative Implementations of Two-Level Adaptive Branch Prediction,” in ISCA, 1992. [902] X. Yi, H. Du, Y. Wang, J. Zhang, Q. Li, and C. J. Xue, “ArtMem: Adaptive Migration in Reinforcement Learning-Enabled Tiered Memory,” in ISCA, 2025. [903] J. Yin, Y. Eckert, S. Che, M. Oskin, and G. H. Loh, “Toward More Efficient NoC Arbitration: A Deep Reinforcement Learning Approach,” in AIDArc, 2018. [904] J. Yin, S. Sethumurugan, Y. Eckert, C. Patel, A. Smith, E. Morton, M. Oskin, N. E. Jerger, and G. H. Loh, “Experiences with ML-Driven Design: A NoC Case Study,” in HPCA, 2020. [905] A. Yoaz, M. Erez, R. Ronen, and S. Jourdan, “Speculation Techniques for Improving Load Related Instruction Scheduling,” in ISCA, 1999. [906] G. Zacharopoulos, A. Barbon, G. Ansaloni, and L. Pozzi, “Machine Learning Approach for Loop Unrolling Factor Prediction in High Level Synthesis,” in HPCS, 2018. [907] S. Zangeneh, S. Pruett, S. Lym, and Y. Patt, “BranchNet : Using Offline Deep Learning To Predict Hard-To-Predict Branches,” in MICRO, 2020. [908] S. Zangeneh, S. Pruett, and Y. Patt, “Branch Prediction with Multilayer Neural Networks: The Value of Specialization,” in Machine Learning for Computer Architecture and Systems, 2020. [909] Y. Zeng and X. Guo, “Long Short Term Memory Based Hardware Prefetcher: A Case Study,” in MEMSYS, 2017. [910] Z. Zeng, Z. Gen, W. Gu, J. Gao, and Z. Song, “Charten:Online Reinforcement Learning for Multi-Objective Optimization Data Placement in Hybrid Storage Systems,” in AINIT, 2025.

BIBLIOGRAPHY

229

[911] Y. Zha and J. Li, “Hyper-AP: Enhancing Associative Processing Through A Full-Stack Optimization,” in ISCA, 2020. [912] D. Zhang, N. Jayasena, A. Lyashevsky, J. L. Greathouse, L. Xu, and M. Ignatowski, “TOPPIM: Throughput-Oriented Programmable Processing in Memory,” in HPDC, 2014. [913] G. Zhang and D. Sanchez, “Leveraging Hardware Caches for Memoization,” in IEEE CAL, 2017. [914] G. Zhang and D. Sanchez, “Leveraging Caches to Accelerate Hash Tables and Memoization,” in MICRO, 2019. [915] M. Zhang, Y. Zhuo, C. Wang, M. Gao, Y. Wu, K. Chen, C. Kozyrakis, and X. Qian, “GraphP: Reducing Communication for PIM-Based Graph Processing with Efficient Data Partition,” in HPCA, 2018. [916] P. Zhang, R. Kannan, A. Srivastava, A. V. Nori, and V. K. Prasanna, “Resemble: Reinforced Ensemble Framework for Data Prefetching,” in SC, 2022. [917] T. Zhang, K. Chen, C. Xu, G. Sun, T. Wang, and Y. Xie, “Half-DRAM: A High-Bandwidth and Low-Power DRAM Architecture from the Rethinking of Fine-Grained Activation,” in ISCA, 2014. [918] T. Zhang, B. Grot, W. He, Y. Lv, P. Qu, F. Su, W. Wang, G. Zhang, X. Zhang, and Y. Zhang, “Hierarchical Prefetching: A Software-Hardware Instruction Prefetcher for Server Applications,” in ASPLOS, 2025. [919] W. Zhang, D. M. Tullsen, and B. Calder, “Accelerating and Adapting Precomputation Threads for Effcient Prefetching,” in HPCA, 2007. [920] Y. Zhang, J. Yang, and R. Gupta, “Frequent Value Locality and Value-Centric Data Cache Design,” in ASPLOS, 2000. [921] J. Zhao, O. Mutlu, and Y. Xie, “FIRM: Fair and High-Performance Memory Control for Persistent Memory Systems,” in MICRO, 2014. [922] H. Zheng and A. Louri, “An Energy-efficient Network-on-chip Design using Reinforcement Learning,” in DAC, 2019. [923] X. Zheng, L. K. John, and A. Gerstlauer, “Accurate Phase-Level Cross-Platform Power and Performance Estimation,” in DAC, 2016. [924] X. Zheng, P. Ravikumar, L. K. John, and A. Gerstlauer, “Learning-based Analytical Cross-Platform Performance Prediction,” in SAMOS, 2015. [925] H. Zhou, “Dual-core Execution: Building a Highly Scalable Single-Thread Instruction Window,” in PACT, 2005. [926] H. Zhou and T. M. Conte, “Enhancing Memory Level Parallelism via Recovery-Free Value Prediction,” in ICS, 2003.

230

BIBLIOGRAPHY

[927] H. Zhou, J. Flanagan, and T. M. Conte, “Detecting Global Stride Locality in Value Streams,” in ISCA, 2003. [928] J. Zhou, B. Chen, K. Li, and Y. Chen, “sBerti: Enhancing Berti with a Smart Stride Prefetcher for Better Coverage,” in 4th Data Prefetching Championship (DPC4), 2026. [929] Y. Zhou, F. Wang, Z. Shi, and D. Feng, “An Efficient Deep Reinforcement LearningBased Automatic Cache Replacement Policy in Cloud Block Storage Systems,” in IEEE TC, 2023. [930] K. Zhu, M. Liu, Y. Lin, B. Xu, S. Li, X. Tang, N. Sun, and D. Z. Pan, “GeniusRoute: A New Analog Routing Paradigm using Generative Neural Network Guidance,” in ICCAD, 2019. [931] Q. Zhu, T. Graf, H. E. Sumbul, L. Pileggi, and F. Franchetti, “Accelerating Sparse MatrixMatrix Multiplication with 3D-Stacked Logic-in-Memory Hardware,” in HPEC, 2013. [932] X. Zhuang and H.-H. Lee, “A Hardware-based Cache Pollution Filtering Mechanism for Aggressive Prefetches,” in ICPP, 2003. [933] C. Zhuo, B. Yu, and D. Gao, “Accelerating Chip Design with Machine Learning: From Pre-Silicon to Post-Silicon,” in SOCC, 2017. [934] Y. Zhuo, C. Wang, M. Zhang, R. Wang, D. Niu, Y. Wang, and X. Qian, “GraphQ: Scalable PIM-Based Graph Processing,” in MICRO, 2019. [935] C. Zilles and G. Sohi, “Understanding the Backward Slices of Performance Degrading Instructions,” in ISCA, 2000. [936] C. Zilles and G. Sohi, “Execution-based Prediction Using Speculative Slices,” in ISCA, 2001. [937] A. Zouzias, K. Kalaitzidis, and B. Grot, “Branch Prediction as a Reinforcement Learning Problem: Why, How and Case Studies,” in ArXiv, 2021.

Record · ID 2732 · SHA-256 cacd166b669c0ca1
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.