arXiv CSopen access

Ermes: a Stateful Serverless Platform for the Edge-to-Cloud Continuum

· arxiv_cs

arXiv CS · Papers · License: Open Access

cloud distributed-computing parallel-computing

distributed computing, parallel computing, cloud

Ermes: a Stateful Serverless Platform for the Edge-to-Cloud Continuum Matteo Cenzatoa , Dario d’Abatea,∗, Arianna Dragonia,∗, Giacomo Orsenigoa , Luca Tosettia , Matteo Briscinia , Alessandro Margaraa a Politecnico di Milano, Milan, Italy

Abstract

arXiv:2609.18924v1 [cs.DC] 16 Sep 2026

Function-as-a-Service (FaaS) is a widely adopted paradigm to simplify application deployment across the edge-to-cloud continuum. However, its stateless nature forces functions to retrieve their state from external, typically cloud-centric, data stores, reintroducing the very latency that edge computing aims to eliminate. This issue is further exacerbated by location-agnostic schedulers and rigid, one-size-fits-all consistency models that fail to capture the diverse requirements of edge applications. In this paper, we propose Ermes, a distributed platform that natively integrates state management into the FaaS paradigm, enabling the joint distribution of computational workloads and application state across the edgeto-cloud continuum. Ermes organizes application state into logical units, termed collections, and employs a distributed coordination algorithm that jointly maps collections and functions onto the available nodes seeking to minimize the latency perceived by the clients. In addition, it supports fine-grained replication and per-collection consistency levels, ranging from sequential to eventual consistency, leaving developers the choice of how to resolve the trade-off between consistency and performance. The experimental evaluation shows that Ermes quickly turns remote state accesses into local ones and sustains low latency as the workload grows, and as clients move across the edge. Keywords: Edge computing, edge-to-cloud continuum, stateful serverless computing, offloading, migration, distributed systems

1. Introduction Cloud computing has long served as the backbone of distributed applications, offering virtually unlimited computation and storage resources [1]. However, the physical distance between clients and remote data centers introduces unavoidable latency, which becomes prohibitive for a growing class of latency-sensitive applications [2]. Edge computing addresses this limitation by distributing resources across small-scale nodes located near the endusers [3]. The edge-to-cloud continuum integrates these decentralized resources with traditional cloud services [4], creating a unified infrastructure where applications can dynamically balance workloads between the cloud and the network periphery. In this landscape, serverless computing, and specifically Function-as-a-Service (FaaS), has emerged as a widely adopted paradigm to simplify application deployment [5]. Under the FaaS model, the execution logic is decoupled from infrastructure management. Indeed, the platform ∗ Corresponding author

Email addresses: [email protected] (Matteo Cenzato), [email protected] (Dario d’Abate), [email protected] (Arianna Dragoni), [email protected] (Giacomo Orsenigo), [email protected] (Luca Tosetti), [email protected] (Matteo Briscini), [email protected] (Alessandro Margara)

handles resource provisioning and scaling automatically, which is particularly beneficial in the edge-to-cloud continuum, where resources are inherently heterogeneous and geographically distributed. However, to facilitate the dynamic placement and migration of workloads, FaaS functions are typically designed to be stateless [6], which keeps the model lightweight and portable and enables rapid scaling even at the constrained network periphery. Within this model, the application state resides in external storage, typically centralized in the cloud. As a consequence, edge-deployed functions must fetch their state from remote data stores, reintroducing the very latency that edge computing was intended to eliminate. A first step toward mitigating this problem is to account for locality when placing computation. Several edgeoriented FaaS platforms already do so, routing invocations to nodes close to the requesting client [7, 8, 9, 10, 11]. These platforms, however, remain stateless: they optimize the placement of computation while leaving the application state in external storage. As a result, a function scheduled close to its client may still incur a remote roundtrip to fetch its state. Location-aware scheduling alone is therefore insufficient when the state itself is not brought close to the client. To address this, a more effective strategy involves co-locating application state and functions at the network periphery. However, optimizing state placement is non-trivial, as multiple functions often require ac-

cess to shared state portions. Replicating the state across edge nodes closer to the clients can reduce access latency, but it introduces a fundamental trade-off, as keeping replicas consistent requires coordination, and the cost of this coordination depends on the guarantees required. Strict consistency demands synchronization that may offset the latency benefits of edge placement, while weaker models reduce overhead at the expense of allowing temporary divergence across replicas. Most existing platforms do not move state toward the edge at all, and the few that do address this complexity only in part. Some bring a single copy of the state close to the client without replicating it, so that any other client accessing it still pays a remote round-trip. Others do replicate, but enforce rigid, one-size-fits-all consistency models [12] that lack the flexibility to support the diverse requirements of edge applications. As a result, the fundamental trade-off between data consistency and the low-latency performance required at the edge remains largely unresolved in current serverless architectures like Enoki [13], Apache Flink StateFun1 , and Faasm [14]. In this paper, we propose Ermes, a distributed architecture that natively integrates state management into the FaaS paradigm, enabling the joint distribution of both computational workloads and application state across the edge-to-cloud continuum. Ermes organizes application state into logical units termed collections, which are dynamically placed among available nodes to ensure that state is persisted and accessed in proximity to where computation occurs. To optimize placement, the framework employs a dual-engine approach. It distributes collections in proximity to the end-users based on both locality and resource availability, while simultaneously steering function execution toward the nodes hosting the relevant state, or as close as possible to them. To this end, Ermes automatically maps functions and state onto the available nodes through a distributed coordination algorithm that seeks to minimize the latency perceived by the clients. This coordination algorithm was proposed and validated in a companion work [15], which formulates the joint placement and scheduling problem and shows, in simulation, that its decentralized heuristic closely approximates an optimal allocation of the available resources To complement this decentralized storage model, Ermes supports fine-grained collections replication at the network periphery, supporting the diverse requirements of modern edge applications. Indeed, Ermes directly addresses the limitations of rigid, one-size-fits-all consistency models empowering developers to resolve the trade-off between consistency and performance by defining per-collection consistency levels. In doing so, it leaves developers the choice of whether to prioritize strict consistency or lower access latency, ranging from sequential consistency for sensitive data to eventual consistency for performance-critical tasks.

We integrate within Ermes a set of decentralized protocols that run among the nodes with no central coordinator, each deciding where to execute an invocation and where to keep its collections from the demand it observes. Whereas our companion work [15] studied the placement and scheduling heuristic in simulation, here we evaluate the fully implemented platform on real infrastructure, assessing its performance and scalability across the edgeto-cloud continuum. Furthermore, we compared our solution against a traditional centralized baseline to quantify the advantages of our decentralized approach. The results demonstrate that Ermes delivers significant performance benefits over conventional FaaS architectures, particularly in terms of request latency and system throughput, validating the framework as a robust solution that effectively bridges the gap between serverless simplicity and the performance requirements of modern edge applications. The remainder of this paper is organized as follows. We position Ermes within the landscape of serverless and edge computing in Sec. 2. We then present its programming model (Sec. 3) and an overview of the platform (Sec. 4), before detailing its two families of functionalities, function management (Sec. 5) and state management (Sec. 6). We describe our implementation in Sec. 7 and report the experimental evaluation in Sec. 8. Finally, Sec. 9 concludes the paper and outlines directions for future work. 2. Related Work This section positions Ermes within the landscape of serverless and edge computing, which we survey along three lines of research. The first develops lightweight execution runtimes, focused on running individual functions on a single node. The second builds full-fledged platforms that coordinate function execution across multiple nodes in the edge-to-cloud continuum. The third studies function scheduling and data placement as an optimization problem, in isolation from any specific system. We examine the three in turn, and close by positioning Ermes with respect to all of them. Runtimes. Lightweight execution runtimes minimize invocation overhead, making them well suited for resourceconstrained edge devices. Runtimes such as Sledge [16], TinyFaaS [17], Faasm [14], and Faasd are designed as lightweight execution engines for resource-constrained edge nodes, but none of them coordinates work across multiple nodes. Most adopt a stateless model, where functions have no persistent state across invocations. The exception is Faasm, which maintains persistent state and replicates it across function instances for fault tolerance. However, these runtimes are not full-fledged platforms: they do not include coordination protocols to manage function scheduling and state placement across geographically distributed nodes. Our own runtime, WASP [18], belongs to this same category but is designed to be modular: it decouples function

1 https://github.com/apache/flink-statefun

execution from state management, turning the execution engine, the datastore, and the caching policies into pluggable components that administrators tailor to each node without altering application code. Like the others, WASP runs on a single node and does not coordinate execution across the hierarchy, a role that Ermes fulfills by building on it (Sec. 5). Platforms. Commercial cloud FaaS platforms, managed by public vendors, address the scheduling problem by assuming homogeneous and virtually infinite resources, transparently instantiating new function instances without exposing placement decisions to the developer. Moreover, they follow a stateless execution model, delegating all data persistence to external storage services. The most widely adopted (AWS Lambda 2 , Google Cloud Functions 3 , and Azure Functions 4 ) follow a purely stateless model, where functions retain no persistent state across invocations, and any data persistence must be handled explicitly through external storage services. Some commercial platforms extend this model with native state management. Azure Durable Functions introduces Entity Functions that expose persistent state as if it was local memory, although the underlying storage remains centralized in a single region. Cloudflare Durable Objects 5 couples computation with storage following the actor model, binding each object to a single location close to its first client request to guarantee strong consistency, but without support for offloading or replication across nodes. AWS IoT Greengrass 6 extends the FaaS paradigm to IoT devices at the far edge, providing state abstraction through the Device Shadow service and automatically replicating local state to the cloud for durability. However, all these platforms are designed to run on the vendor’s own infrastructure, with no support for deployment on private, user-managed nodes. Moreover, most of them operate exclusively within cloud datacenters. The two exceptions that target edge scenarios are Cloudflare Durable Objects and Greengrass. However, they still tie execution to vendor-controlled deployment models. Indeed, Cloudflare Durable Objects execute on the vendor’s globally distributed edge nodes, not on user-owned infrastructure, while Greengrass runs on user-owned IoT devices but is orchestrated entirely from the cloud console. Open-source and academic platforms remove this constraint, as they can be deployed on arbitrary, usermanaged infrastructures. We summarize the most representative solutions in Table 1, comparing them along relevant dimensions. Specifically, we distinguish platforms that natively coordinate function execution across geographically distributed nodes (i.e., decentralized platforms) from those that confine operations to a single

cluster or region. We then distinguish between stateful platforms that manage persistent state across invocations and stateless ones that delegate persistence to external services. Within the former, we evaluate whether the platform provides high-level primitives to shield developers from underlying storage complexities (state abstraction). Finally, we examine how each platform makes function placement decisions. We distinguish platforms based on their decision-making logic. In centralized platforms, a global scheduler dictates task allocation, whereas distributed platforms rely on autonomous peer-to-peer negotiation. We also identify trivial placement, where execution is performed by the node receiving the client request without forwarding, and hybrid approaches, which partition the infrastructure into active schedulers and passive executors. We also evaluate whether the policy accounts for proximity to the requesting client (locality awareness) and available capacity on candidate nodes (resource awareness), and whether the platform supports data replication across nodes. Apache OpenWhisk [19], Lean OpenWhisk 7 , OpenFaaS [20], and FunLess [21] deploy a central control plane that manages all scheduling decisions based on available node capacity. Moreover, orchestration is confined to a single cluster with no support for geo-distributed coordination. Neptune [7] extends coordination beyond a single cluster, supporting placement across geographically distributed nodes. Indeed, its hierarchical scheduler accounts for both resource availability and client proximity, but placement decisions are still delegated to a central coordinator. Edgeless [22] takes a hybrid scheduling approach, where a designated subset of nodes collectively manages scheduling for the rest based on available resources, but without accounting for client proximity. DFaaS [9] and Serverledge [8] adopt a fully distributed scheduling policy, where nodes autonomously decide whether to handle a request locally or forward it to a neighbor. DFaaS bases this decision purely on load, while Serverledge also accounts for network proximity to the client. All the aforementioned platforms follow a stateless model, delegating data persistence to external services. A different class of platforms addresses this limitation by natively integrating a persistence layer into the serverless architecture. Enoki [13] adopts a local-first strategy. It provides state abstraction through keygroups and proactively replicates data to the executing node upon invocation, ensuring locality awareness. However, its scheduling is trivial, and it does not account for resource availability. Cloudburst and Apache Flink StateFun 8 provide both state abstraction and data replication, with a centralized scheduler that accounts for resource availability. They are designed for single-datacenter deployments and do not support geo-distributed orchestration. Enoki [13]supports

2 https://aws.amazon.com/lambda 3 https://cloud.google.com/functions 4 https://azure.microsoft.com/products/functions 5 https://developers.cloudflare.com/durable-objects/

7 https://github.com/kpavel/incubator-openwhisk/tree/lean

6 https://aws.amazon.com/greengrass

8 https://github.com/apache/flink-statefun

State model

State Abstraction

Scheduling Policy

Locality Awareness

Resource Awareness

Stateless Stateless Stateless Stateless Stateless Stateless Stateless Stateless Stateful

✓

Centralized Centralized Centralized Centralized Centralized Hybrid Distributed Distributed Centralized

× × × × ✓ × × ✓ ×

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Data Replication ✓

Stateful

✓

Centralized

✓

✓ ✓ ✓ ✓

Stateful Stateful Stateful Stateful

✓ × × ✓

Trivial Centralized Centralized Distributed

✓ × ✓ ✓

× ✓ ✓ ✓

✓ × ✓ ✓

Decentralized OpenWhisk Lean OpenWhisk OpenFaaS Funless Neptune Edgeless DFaaS Serverledge Cloudburst Apache Flink Statefun Enoki StructMesh FaDO Ermes

× × × × ✓ ✓ ✓ ✓ ×

Table 1: State model and scheduling policy overview

geo-distributed orchestration and provides state abstraction. However, its scheduling is trivial, as execution is bound to the receiving node with no possibility of offloading, and it does not account for resource availability. StructMesh [23] also adopts a decentralized architecture, but relies on a centralized scheduler that considers only resource availability. Moreover, unlike the previous platforms, it does not support data replication. FaDO [24] combines a decentralized architecture with data replication and a centralized scheduler that accounts for both resource availability and client proximity. However, like StructMesh, it does not provide state abstraction, exposing raw storage interfaces to the developer. To summarize, none of the existing platforms combines a decentralized stateful model with state abstraction, a distributed, locality-aware scheduling policy, and native data replication with configurable per-collection consistency.

Ermes infrastructure deploy

Administrator

coll

λ Developer

Client

function deploy r function invocation

coll

Figure 1: External actors interacting with Ermes.

by scheduling algorithms, a combination that no existing system offers.

Scheduling and data placement. A distinct line of research studies function scheduling and data placement as an optimization problem in isolation, independently of any specific system. Some works schedule functions without modeling their data dependencies, deciding where to execute each invocation based on load, network proximity, or energy [11, 10]. Others jointly optimize function scheduling and data placement, but either leave replication out of the model or treat consistency coarsely, without distinguishing among different consistency levels. Our companion work [15] addresses precisely this gap, formulating the joint placement and scheduling problem under heterogeneous consistency models and deriving a decentralized heuristic that closely approximates the optimum. Ermes draws on all these lines of work. It is a full-fledged platform that coordinates function execution across the hierarchy, executing functions on top of the WASP runtime and realizing the decentralized heuristic of our companion work [15] to jointly place computation and state. In doing so, it combines the lightweight, configurable execution of edge runtimes, the multi-node coordination of platforms, and the locality- and consistency-aware placement studied

3. Programming Model Ermes exposes its functionalities to three types of external actors, whose interactions with the platform are illustrated in Fig. 1. An administrator provisions and deploys the underlying infrastructure on which the platform runs. Developers implement the application logic as functions and deploy them to the platform. Clients trigger the execution of these functions through invocation requests, and the functions access the application state on their behalf. Ermes adopts the Function-as-a-Service paradigm, but, unlike traditional FaaS platforms, it is inherently stateful: functions can create, access, and modify persistent application state during their execution. The application state is organized into collections, the unit of state that functions create and operate on. Each collection is a set of records, key-value pairs that functions read and write individually. Clients are limited in accessing collections. Each collection carries an access policy, fixed when it is created, that determines who may operate on it: a private collection is accessible only to its owner, the client on whose behalf it was created; a public collection to every 4

client; and a shared collection to a restricted set of clients. To express sharing, developers organize clients into groups: the creator of a shared collection names the groups authorized to access it, and access is granted to the owner and to any client that belongs to at least one of them. Functions are the fundamental units of computation in Ermes, and the only means through which clients interact with the application state. When a client invokes a function, it provides a set of arguments, and the function executes on the client’s behalf, operating exclusively on the collections the client is authorized to access. A function manipulates the state through a small, uniform set of operations. It reads and writes the individual records of a collection through get and put operations, and it is the sole mechanism for creating new collections, fixing their consistency policy at creation time, and for deleting existing ones. Ermes maintains two kinds of metadata. A function is deployed with function metadata: it declares the collections it accesses by their properties rather than by their identity, with placeholders that the invocation’s arguments fill in, so a function never names its collections directly. Each collection, in turn, carries collection metadata: its properties, identity, owner, and consistency and access policies. At invocation time, the platform resolves the function metadata against the collection metadata, matching the properties a function declares against those each collection carries, so that the same function operates on different collections depending on the invoking client and the arguments it supplies. Any record a function writes, and any collection it creates, persists beyond the invocation, so that subsequent invocations, even of different functions and on behalf of different clients, observe it according to the consistency policy of the collection. As a running example, consider a patient monitoring application deployed in a hospital. A developer implements and deploys two functions: detectAnomaly, which analyzes a patient’s vitals to identify health anomalies, and analyzePopulation, which examines the medical history of a group of patients to detect common patterns. Each patient is a client whose monitoring device invokes detectAnomaly, while doctors invoke analyzePopulation over the data of the patients they follow. Ermes replicates collections, and thus some consistency guarantees should be preserved. To do so, Ermes provides two different consistency policies for a collection. A sequentially consistent collection ensures that all the read and write operations on the collection appear to the clients as if they were executed in a sequential order, and that the operations of each client appear in the order in which the client issued them. Developers thus obtain serializable accesses and read-your-writes, even when a client moves and its invocations are served by a different node. An eventually consistent collection relaxes ordering in favor of latency. A client is guaranteed that the responses it observes eventually converge to a single state, but has no guarantee during the transient, where it may read different states, and concurrent updates to the same record are

Developer

Administrator infrastructure deploy

function deploy

Ermes auth. function fetch

Group Token Provider user permissions

Cloud

function fetch

runtime c1

Fog Edge

Function Registry

runtime

function invocation

client

Figure 2: Functional model of the Ermes framework.

not all preserved. In the patient monitoring example, a developer stores the clinical data used to detect anomalies in a sequentially consistent collection, to preserve causality over life-critical records, and the append-only audit log of the accesses in an eventually consistent one, to favor throughput over ordering. A single invocation can access at most one sequentially consistent collection, and an arbitrary number of eventually consistent ones. This restriction is what allows Ermes to enforce a total order without running a consensus protocol: with a single collection involved, the updates of an invocation are ordered by construction, whereas spanning multiple sequentially consistent collections would require a distributed commit, whose stop-the-world barrier is prohibitive when inter-node latencies span the edge-tocloud continuum. This restriction is far less limiting than it may appear, as it mirrors a design choice common to distributed databases that support multiple consistency levels and deliberately confine strong guarantees to a single unit of data to avoid global coordination bottlenecks. Apache Cassandra, for instance, provides lightweight transactions that guarantee linearizability only within a single partition, with no cross-partition support9 . Ermes adopts the same principle: confining each invocation to a single sequentially consistent collection removes cross-collection coordination entirely, while still leaving eventually consistent collections unrestricted. 4. System Overview Ermes aims to minimize the latency observed by clients that invoke stateful functions from the edge of the network. 9 https://cassandra.apache.org

parent

Function Management

State Management

Access Scheduling

collection c1 c2 c3

child1

node n

replica direction parent, child1 local, child2 parent

Monitoring

leader direction parent — parent

Placement Execution

Replication

child2

Figure 3: Node n and the reachability information it keeps; a ⋆ marks a leader.

Figure 4: The functionalities of Ermes.

This latency is dominated by two costs: executing a function’s code and accessing the state it reads and writes. To minimize these two costs, Ermes co-locates each function with the state it accesses both close to the client. Where a function executes and where its state resides are thus decided jointly. In previous work, we formulated this joint problem of function scheduling and data placement exactly and showed that, although it admits an optimal solution, computing one is not feasible for the edge-to-cloud continuum [15]; we thus devised a heuristic that approximates the optimum well and adapts to dynamic scenarios. Ermes relies on this heuristic. Ermes is a distributed system: it runs on many nodes that cooperate, communicating and synchronizing to provide the service. The administrator deploys the same software, the Ermes runtime, on every node, and the runtimes together execute the application’s functions and manage the state those functions access. The nodes are organized into a hierarchy of tiers, illustrated in Fig. 2. Computational and storage capacity grows from the leaves toward the root, so the tiers closest to the clients are the most resource-constrained and those higher up the most capable. A node does not reach every other node directly: each tier connects to the one above it, and a node can always climb toward the root to find more compute and storage. We model the infrastructure as a tree, consistent with common edge-to-cloud hierarchies [4, 2]: although a physical deployment may wire nodes differently, this logical hierarchy captures that a node reaches richer resources by climbing toward the root. The nodes form a strict parent-child hierarchy rooted at a single cloud node. Throughout the paper we use a three-tier instance of this model, with an edge, a fog, and a cloud tier. The collections that make up the application state reside on these nodes, and clients issue their invocations to the edge tier closest to them. Two centralized components complete the deployment. The function registry stores all the functions available in the application, so that any node can retrieve a function it does not yet hold locally. The group token provider is the sole authority on the group memberships of clients: upon authentication, it issues a signed group token with a limited lifetime, which the client attaches to its invocation requests, and which any node can verify locally to enforce the access policies of the target collections without

contacting the provider on every request. No node holds a global view of the system. Instead, each node maintains reachability information (Fig. 3), used to provide its functionalities: for every collection it is aware of, the direction in which a replica lies, toward its parent or into a specific child’s subtree, and, for a sequentially consistent collection, the direction toward its leader. Note that direction and location differ: in Fig. 3, node n knows c1 lies toward its parent, though its replica actually sits two tiers up. A node assembles this view by interacting with its neighbors. We assume the topmost tier, the cloud, retains a replica of every collection: with effectively unbounded storage it can hold them all, so a node always reaches a collection by climbing toward the root. This makes the cloud the fallback from which every collection is reachable. Fig. 4 organizes the runtime’s functionalities into two groups that cooperate, each realized in a distributed fashion through protocols among the runtimes, with no central coordinator. Function management brings each invocation to a node and runs it there, through the scheduling and execution functionalities. State management keeps the collections spread across the hierarchy and serves the accesses functions make on them, through the access, monitoring, placement, and replication functionalities. When a client invokes a function, it contacts a node of the hierarchy, typically the closest one, which it selects itself. From that node, scheduling decides where the invocation is served, locally or forwarded to another node, and execution then runs the function there. Functions are stateful and operate on collections. A collection exists as a set of replicas whose roles follow its consistency policy: a sequentially consistent collection has a single leader replica that accepts writes and any number of read-only followers, while an eventually consistent one is multi-leader, every replica accepting both reads and writes. While a function runs, the serving node resolves the collections it needs and carries out its reads and writes (access). In the background, monitoring records how much demand each collection draws and from which direction, and placement consumes this to continuously reconsider where each collection and its replicas should reside, migrating them as the demand of the clients evolves. Finally, replication keeps the replicas of a collection synchronized, both while functions update them and while 6

Algorithm 1 Scheduling decision at a node n, computing the direction d it serves the invocation toward. Require: the required collections C 1: if C = ∅ then 2: d ← self 3: else if some c ∈ C is sequentially consistent then 4: if the function writes c then 5: d ← lead(c) 6: else 7: d ← arg mine ∈ repl(c) lat(e) 8: end if 9: else if self ∈ repl(c) for some c ∈ C then 10: d ← self 11: else 12: d ← arg maxe ∈ Sc∈C repl(c) |{ c ∈ C : e ∈ repl(c) }| 13: end if 14: if d = self then 15: return execute 16: else 17: return forward toward d 18: end if

they migrate. The two families are the subject of the two sections that follow: function management in Sec. 5 and state management in Sec. 6. 5. Function Management Function management covers the path of an invocation, from the moment a client issues it to the moment the platform returns its response. It brings each invocation to a node and runs the function there, through the scheduling and execution functionalities. Its half of the co-location goal (Sec. 4) is to serve each invocation on a node that already holds the data it needs, so that the function runs beside its state and the client observes low latency. Because the decision sits on the critical path of every invocation, a node never consults nodes beyond its neighbors, and either executes the invocation or forwards it one hop toward the data. And because collections migrate across the hierarchy while invocations are in flight (Sec. 6), it decides against a placement that is never fixed. Scheduling. The node a client contacts is not necessarily the one that should execute the invocation, as the data it needs may be hosted elsewhere. To route without coordination, each node relies on the reachability information it maintains locally (Sec. 4, Fig. 3): for every collection it is aware of, the direction in which a replica is hosted and, for a sequentially consistent collection, the direction toward its leader. This view is partial, as a node is aware only of the collections within its reach; a node that knows nothing of a required collection forwards the invocation to its parent, which is aware of more, up to the root, which is aware of every collection. When an invocation enters the system, its entry node resolves the set C of collections it requires (Sec. 3). If C is empty, nothing constrains the execution and the node serves the invocation itself. Otherwise, n looks each collection up in its reachability information: for a collection c it reads the neighbors repl(c) toward which a replica is hosted (self if n holds one) and, for a sequentially consistent c, the neighbor lead(c) toward its leader; it also knows the latency lat(e) to each neighbor e, its parent and children (lat(self) = 0), and treats a collection it is unaware of as reachable through its parent. From these it decides as Algorithm 1 summarizes, following the consistency policy of the collections involved. A sequentially consistent collection dominates the decision, since it alone constrains where execution may run: a write must reach the single leader, while a read may go to any replica, so n heads toward the nearest one. With only eventually consistent collections, n instead maximizes colocation, executing locally when it already holds a required collection and otherwise moving toward the neighbor that covers most of them. Any eventually consistent collection still missing at the chosen node is fetched from the nearest known replica while the function runs.

Because each node either executes or forwards one hop, an invocation climbs at most to the root, so routing always terminates. How often it must climb far depends on where replicas sit, which is exactly what collection placement (Sec. 6) works to improve, by keeping data close to the clients that use it. Execution. Once scheduling has selected the node that serves an invocation, that node executes the requested function locally. Two requirements shape how it does so. First, a node serves concurrently the invocations of different functions, on behalf of different clients: each invocation must therefore run in isolation from the others. Second, scheduling may direct an invocation to any node of the continuum, whose hardware ranges from constrained edge devices to cloud servers: the same function must thus be able to run, unmodified, wherever it lands. Ermes accordingly runs every invocation in an isolated execution environment, uniform across the infrastructure and private to the invocation it serves. It realizes this environment on WASP [18], a configurable substrate for stateful serverless execution in the edge-to-cloud continuum, whose implementation we detail in Sec. 7. 6. State Management Function management brings each invocation to a node and runs it there; it does so over a body of state that state management maintains. Ermes holds the application state as collections and continually replicates and relocates them across the hierarchy, so that each collection sits close to the clients that use it. State management is in this sense the counterpart of function management, and the two reinforce each other: placement pulls a collection 7

timer fires ∨ capacity report

toward the nodes that request it, shortening the path invocations travel, while the traffic that scheduling routes drives placement. State management comprises four functionalities (Fig. 4): (i) access serves the reads and writes that functions perform on collections, (ii) monitoring measures the demand each collection draws, (iii) placement decides where each collection and its replicas reside as that demand shifts, and (iv) replication keeps the replicas of a collection consistent as functions update them and as they move. Access runs on the critical path of every invocation; the other three run in the background.

Idle

capacity reports from all neighbors

Gossip

Decision

decisions taken

Figure 5: The placement state machine run by each node.

how that heuristic is mapped onto Ermes. A node acts on the accumulated demand periodically, through a cyclic state machine with three phases, idle, gossip, and decision; we call one full cycle an epoch (Fig. 5). Initially, the node remains idle for a fixed time, where it just serves invocations. This phase bounds how often the machine runs, so that epochs do not follow one another too closely. Two conditions drive the transition to the gossip phase: either a timeout fires after the fixed time, or a capacity report from a neighbor arrives. In this way, epochs stay loosely aligned across the tree with no global clock. Entering the gossip phase, the node sends its residual storage capacity to both its parent and children, and waits to receive theirs. Once it holds the capacity report of every neighbor, the node enters the decision phase. It evaluates each collection it holds independently, weighing, for every neighbor, the free storage it has just learned against the demand it has accumulated, and decides whether to keep the collection in place, migrate it to a neighbor, or replicate it. The node then returns to idle, and the migrations originated in the last phase proceed asynchronously. The two consistency policies supported by Ermes, call for different placement mechanisms, because they differ in whether the placement choices for a collection are coupled. Under eventual consistency placement is reactive: any replica accepts writes, so each one can be placed independently. Placement piggybacks on the natural flow of requests: when a function accesses a collection not stored locally, its node fetches it from the nearest known replica, and, as part of serving that request, the holder asynchronously provisions a new copy one hop closer to the consumer. Over time, collections under sustained demand migrate hop by hop toward the edge nodes that access them most. The decision to copy is subject to a storage threshold: a node sends a copy to a neighbor only if the neighbor has sufficient free capacity. Under sequential consistency, instead, placement is proactive: every write must reach the single leader, dictating where write traffic flows and where followers are worth placing. Ideally, the leader should be as close as possible to the clients issuing writes; but edge nodes have limited storage, and accumulating all replicas near the leaves is infeasible. Placement therefore balances bringing state close to clients against the storage constraints of resource-scarce nodes.

Access. Ermes enforces the access policy of each collection (Sec. 3) with a purely local check. The client proves its group memberships through the group token issued by the group token provider (Sec. 4), and the node serving an access verifies that token against the groups authorized on the collection, deciding locally and with no lookup elsewhere. Both the authorized groups of a collection and the memberships of a client may change over time, and each access is decided against their state at the moment it is served. The authorized groups are part of the collection metadata (Sec. 3): when an invocation enters the system, the platform resolves the collections the function declares against this metadata (Sec. 5). Once the collections are resolved, the function reads and writes their records through the get and put operations, unaware of where the collections physically reside. When a required replica is on the local node, the operation is served there; when it is not, the interface transparently forwards the operation to a node that holds one. How these operations leave the replicas of a collection mutually consistent is the subject of the replication functionality below. Monitoring. Placement needs to know how much demand each collection draws and from where;monitoring supplies it, continuously and off the critical path. Whenever a node finishes executing a function, it emits an access report for each collection the function accessed, recording whether the access was a read or a write and the latency the invocation accumulated on its way, one link at a time. Each report is routed toward a node that holds the collection, forwarded hop by hop by the same reachability information that guides scheduling (Sec. 5). As the reports travel through the hierarchy, each node keeps, for every collection it hosts, two running sums over the reports it receives: their number and the total latency they carry. Placement. A collection is the atomic unit of storage and replication: its records are never partitioned across nodes but reside together, so that a replica is self-contained and placement moves or replicates it as a whole. Each node periodically decides the placement of the collections it holds, weighing the demand monitoring has gathered against the free storage of its neighbors. These local decisions realize the decentralized heuristic that our previous work [15] derives for the placement problem (Sec. 4); here we describe 8

W Fext,j . Node j delegates the leader to k when the inward pull of k’s subtree, net of the sibling pulls and the tension, exceeds the cost of placing the replica one tier lower, namely its buoyancy at k and the extra link djk that each of the λW writes of the siblings would now traverse, by i a hysteresis margin τL that prevents oscillation between adjacent nodes (2).

Fbuoyancy (j) c

W Felastic (j, k)

j W Felastic (j, i)

subtree with high write demand

subtree with few remote writers

W (j, k) − Felastic

> Fbuoyancy (k) +

µ · size c stor free j

λW i · djk + τL

(2)

i∈ch(j) i̸=k

The delegation recurses with no additional bookkeeping. Once the leader resides on k, the writes of the rest of the tree keep flowing to it, now traversing the link between j and k: the tension k measures on its parent link therefore amounts to (3), the sibling pulls left at j, their extra hop, and the tension j was itself subject to. Node k can thus apply the same test to its own children, and the leader descends toward the region of highest write demand as long as the condition holds.

The heuristic casts the balance between locality and storage as a system of virtual forces on each replica (Fig. 6). Demand acts like a spring: it draws the replica toward the clients it serves, more strongly the more requests they issue and the farther away they are. The elastic force of a direction is exactly the total latency the node has accumulated for it. For a neighbor k, (1) factors this total into the number of requests λk that reach the collection from k’s subtree and the latency djk of the link between the node j and k. Recall that writes are directed only to the leader while reads can be served by any replica, so this pull splits in two: write demand anchors the leader, read demand decides where followers are placed. We accordingly split the count λk into the write and read counts λW k W and λR k aggregated from neighbor k, and write Felastic (j, k) R and Felastic (j, k) for the elastic force each induces. The opposing force is storage: a node j with little free storage behaves like a dense fluid that pushes its replicas up toward the parent, where free storage is more abundant. Its magnitude Fbuoyancy (j) grows with the size size c of the collection c and with the occupancy of the node, as the reciprocal of its free storage stor free j , which a node learns through gossip. The constant µ places this force on the same scale as the elastic one. Fbuoyancy (j) =

W W (j, i) − Fext,j Felastic

i∈ch(j) i̸=k

Figure 6: The leader replica of collection c on node j is pulled toward child k more than toward sibling i, which write less, and pushed upward by the buoyant force of a nearly-full node.

Felastic (j, k) = λk · djk ,

W Fext,k ←

W Felastic (j, i) +

i∈ch(j) i̸=k

W λW i · djk + Fext,j

(3)

i∈ch(j) i̸=k

Follower placement requires no such coordination. Because any replica can serve reads, a follower benefits only the subtree beneath it, so each node evaluates each child independently, without sibling or tension terms. Node j provisions a follower on child k when the read demand λR k aggregated from k’s subtree exceeds the storage pressure at k by a margin τR (4). R Felastic (j, k) > Fbuoyancy (k) + τR

(4)

Both mechanisms replicate aggressively toward the edge, but storage is finite, so placement is complemented by a hierarchical eviction that reclaims it. Each node keeps its replicas in least-recently-used (LRU) order and evicts them, on two occasions: reactively, when an incoming collection does not fit in the residual space, the node evicts replicas until it does; and proactively, at the end of each decision phase, when occupancy exceeds a configured safe level, the node migrates enough replicas to its parent to fall back below it. Evicted replicas are not discarded but pushed to the parent. A node that receives an evicted replica absorbs it if it has spare capacity, and otherwise pushes it further up, until a node with room takes it in, or it reaches the cloud, whose storage is unbounded. A node that receives a replica it already holds simply discards the incoming copy. Replication. Ermes keeps replicas synchronized, as functions update them and as placement provisions and moves them. Read operations are served synchronously during the execution. Writes, instead, never reach the store while the

(1)

At equilibrium, the leader is positioned to minimize write latency and the followers to minimize read latency, and the two positions are determined by separate conditions. The leader migrates one hop at a time, and each node re-evaluates the handoff to every child once per epoch. Consider the leader at node j in Fig. 6, where a write-heavy child k pulls it downward while the lighter siblings resist the move: relocating the leader to k reduces latency for the writes originating in k’s subtree, but increases it for all remaining writers, namely the siblings and the demand reaching j from above. Node j does not track this external demand explicitly: since every write reaches the leader, the demand originating outside j’s subtree arrives precisely through its parent link, and the force j measures on that link summarizes it in a single scalar, the tension 9

s tra

postpone u

nsf e

e nsf tra

store c ack

commit

store c

ack

re-apply u apply u

Figure 7: Up-and-down propagation of an update committed at the leader replica (L) on node n. Nodes a, r, and b hold no replica and forward without applying; the subtree of q holds no replica and is never reached.

(a) An update racing the transfer on the same link.

(b) An update and a replica crossing.

Figure 8: The two races between an update u for a collection c and a transfer of c. Arrows are messages exchanged between the two nodes; boxes are local actions on a node’s timeline.

function runs: they are accumulated in a per-collection set of dirty records, which the function hands back to the node together with its response when it returns. Only then the node commits atomically each set, as a single batch per collection, so that a failed execution leaves no partial writes behind. The two policies commit at different moments. The batch of the sequentially consistent collection is committed before the response returns to the client. The batch of the eventually consistent collections is instead committed asynchronously, after the response has returned, so the client never waits for them. Once committed on the executing node, an update disseminates through an up-and-down protocol rooted at that node (Fig. 7). Each node forwards the update in every direction its reachability information (Sec. 4) indicates a replica, excluding only the link the update arrived from, and applies it locally if it holds a replica itself. The updates directed to the same neighbor are delivered in order, one at a time. For sequentially consistent collections, at each commit, the leader increments a per-collection counter and tags the outgoing update with the resulting sequence number. Because a node acknowledges an update only after applying and forwarding it, and per-neighbor delivery is ordered, updates reach every follower in sequence-number order; a replica moreover applies an incoming update only if its sequence number exceeds the local one, discarding the duplicate and stale deliveries that can arise while replicas move, so every replica evolves monotonically toward the leader state. Reads can be served by any replica. Each response reports the sequence number the serving replica had reached, and each request carries the highest number the client has observed so far: a replica behind that number does not serve the invocation immediately, but re-examines its local state with an exponential backoff until it catches up, aborting the invocation after a bounded number of attempts. This preserves read-your-writes even when consecutive invocations of a moving client are served by different replicas. For eventually consistent collections, any replica accepts reads and writes, with no coordination on the critical path.

Concurrent updates to the same record are resolved by a per-record last-writer-wins (LWW) rule, where each update is timestamped at its origin, and ties are broken with the identifier of the originating node. The comparison and the write execute as a single atomic action on the local store, so that concurrent appliers cannot interleave. The rule totally orders the updates of each record, and replicas that have observed the same set of updates hold the same state. Placement moves replicas while updates are in flight, and a transfer and an update may travel the same link at the same time, in the same direction or in opposite ones; without precautions, both races could lose updates permanently. The protocol prevents this with two dual safeguards. A node that is transferring a replica postpones every update directed to the transfer destination: instead of shipping it, the node enqueues it, per destination and in arrival order (Fig. 8a). If the destination acknowledges the transfer, the node ships the postponed updates in their original order, so they reach the new replica right after the state they were racing; if the transfer fails, the node restores its local replica and applies the updates to it, again in order, and marks the destination as saturated until the next gossip round refreshes it. Dually, a node that propagates an update for a collection it does not hold keeps the update alive while the propagation is in flight (Fig. 8b): a completion counter, decremented as each outgoing message is delivered, re-applies the update to the local store after the last delivery if a replica of the collection has arrived in the meantime, as that replica may have left its source before the update reached it. A leader is not tied to a node forever: placement can migrate it, and while it moves the collection risks being left with no leader, or with two at once. Ermes transfers leadership through a strict handoff that rules this out. The source demotes itself and records the destination as the new leader, and only then starts the transfer; the destination assumes leadership upon receiving the collection, and if the transfer fails, the source resumes it. At most one 10

node is thus the leader at any time. A handoff never interrupts a running function: if placement decides one while invocations are accessing the collection, the move waits for them to finish. The price is a brief unavailability window, bounded by the invocations already admitted and by the transfer itself, during which invocations reaching the collection are rejected and succeed upon retry. Follower provisioning, in contrast, requires no handoff: a new follower is initialized from an existing replica and then receives subsequent updates through the regular propagation channel, without ever blocking ongoing writes.

cold

Function registry

remote fetch

warm

Tier 1 binaries, on disk

f@v1

f@v2

f@v1

f@v2

compile hot

Tier 2 modules, in memory instantiate

reset: drop instance

Worker pool busy: instance of f@v1 code (shared)

idle

linear memory (fresh)

7. Implementation We implemented Ermes in Go, whose lightweight concurrency suits the asynchronous, message-driven nature of the platform. We describe how the two families of functionalities are realized.

Figure 9: The cold, warm, and hot paths of an invocation on a node.

operation into its buffer and passes a pointer to a host function, which reads them, dispatches the operation to the state management functionalities, and writes the result back into the same buffer. The instance is discarded once the execution ends, and the worker returns clean to the pool.

7.1. Function Management Every node runs an instance of the Ermes runtime, which exposes a single HTTP endpoint: through it, the runtime receives both the invocations that clients issue directly to the node and those that a neighboring node forwards to it as the outcome of its scheduling decisions. Ermes builds execution on WASP (Sec. 5), which runs every function in a WebAssembly (WASM) [25] sandbox. WASM meets the two requirements of the execution functionality: its sandbox isolates each invocation from the others the node serves, and its portability lets the same function binary run on any node of the continuum. Each node keeps a pool of pre-warmed workers and serves every invocation it admits on one of them. The size of the pool bounds how many invocations the node executes concurrently. Before an instance can be created, though, the code of the function must be available locally, and Fig. 9 shows the path an invocation follows. A node retrieves the function binary from the function registry, which stores the available functions as versioned WASM modules and is backed by MinIO10 , an S3-compatible object store, only on the first invocation of a given version. To amortize this cost across invocations, the pool is backed by a two-tier cache: the first tier stores the raw binaries retrieved from the registry, the second the compiled modules in memory. A warm invocation thus only pays for compilation, and a hot one bypasses both retrieval and compilation, executing within a small constant factor of native code. A worker serves an invocation by instantiating a cached module: the compiled code is not copied but shared across concurrent instances, so that only the linear memory, which every instance gets fresh and for itself, grows with concurrency. That linear memory is also the channel through which the function reaches the application state: the guest serializes the parameters of an

7.2. State Management Among the data stores that WASP supports, Ermes uses Redis11 exclusively. As an in-memory store, it keeps every state access off the disk, in line with the platform’s goal of minimizing state-access latency (Sec. 4). The node persists collections and their metadata in Redis, organized into three keyspaces: meta: stores collection metadata as Redis hashes, data: stores the application records, and info: stores protocol state such as sequence numbers. Operations that must be atomic, such as the last-writerwins comparison-and-write and the commit of a batch, are implemented as Lua scripts executed directly in Redis, with a retry loop under randomized exponential backoff to absorb transient conflicts. On top of the store, each node keeps an in-memory LRU cache that bounds the number of collections it holds and drives eviction, and a partial view that materializes the reachability, traffic, and capacity information of its direct neighbors. The metadata-only index that backs collection resolution is built with RediSearch: queries are answered with FT.SEARCH over the meta: keyspace, and a user-defined metadata field introduced at runtime triggers an FT.ALTER that extends the index schema without interrupting service. This index is the concrete form of the reachability information that both resolution and scheduling consult; because it records only properties and the direction toward a replica, never the records, collections can be discovered without replicating their data and can move without changing how functions refer to them.

10 https://www.min.io

11 https://redis.io

RQ2. How does the client-perceived latency of Ermes evolve as the workload submitted to the system grows? RQ3. How effectively does Ermes adapt to client dynamics? RQ4. How does Ermes compare, in client-perceived latency, against the cloud-based deployments that represent current practice? These questions follow the logic of the problem Ermes addresses. Stateless FaaS platforms place computation close to the client but leave the state in remote storage, so every state access pays a remote round-trip. RQ1 tests, in isolation, whether relocating and replicating state actually shortens the path an access travels, and thus the latency a client observes, and what each consistency guarantee costs. RQ2 then asks whether this benefit survives at scale, since a realistic deployment serves many clients over a large body of state, with functions that differ in how much state they access. RQ3 asks whether the benefit holds as clients evolve rather than staying fixed: we assess how quickly the system converges for static clients and how well it follows mobile ones, since a platform that could not track moving clients would not fit the continuum. RQ4 finally places these results in perspective, quantifying the gain of Ermes over the cloud-based deployments that represent current practice, both at scale and under client dynamics. To answer RQ4 we compare Ermes against two cloudbased baselines that represent current practice. In the Cloud Only (CO)[19, 20] configuration, the classic serverless arrangement, both the functions and the application state reside in the cloud: an invocation issued at the edge is forwarded to the cloud, executes there with its state local, and returns, so the client pays a single round-trip to the cloud per invocation, regardless of how much state the function accesses. In the Cloud Data (CD)12 configuration instead, the application state is pinned to the cloud, while functions are scheduled at the edge when possible, close to the invoking client. Computation is thus brought close to the client, as in Ermes, but the state is not, since edge nodes hold no state and act only as executors, so every state access incurs a remote round-trip to the cloud. We deliberately do not compare against another platform: to the best of our knowledge, no complete stateful serverless platform for the edge-to-cloud continuum with dynamic, per-collection replication exists to serve as a direct competitor, so these two configurations are the most faithful references available. Before answering these questions, we fixed the execution substrate. Ermes builds on WASP (Sec. 5), which supports several WebAssembly engines and execution models; a detailed comparison among them, across execution time, memory footprint and stability, and scalability under concurrent load, is reported in the WASP paper [18]. Guided by that comparison, we adopt Wasmtime with just-in-time

Access is exposed to the execution functionality through a uniform interface that hides whether a collection resides locally or on a remote node. When a function issues a get or a put, the host function of Sec. 5 dispatches the operation to this interface, which serves it from the local store when a replica is present, and otherwise forwards it over HTTP to the node that holds one. Before committing a write to a sequentially consistent collection, the node validates the sequence number stored in the info: keyspace, enforcing the ordering the leader establishes. Placement is implemented as the finite-state machine of Sec. 6. Capacity reports are exchanged over a dedicated HTTP endpoint, while per-neighbor, per-collection demand is accumulated in penalized circular buffers held in the partial view: the buffers retain the reports of a bounded window of past epochs and weigh them by age, so that recent demand dominates the decision. In the decision phase, the node evaluates its collections concurrently, one goroutine each, and collects the outcomes through a channel; the resulting transfers, and the migrations that hierarchical eviction adds when the cache occupancy exceeds the safe level, are carried out over HTTP. Replication drives the up-and-down protocol over the same HTTP transport. A node maintains one outbound queue per neighbor, drained by a dedicated goroutine that sends a single message and waits for its acknowledgment before the next, which is what makes per-neighbor delivery ordered and keeps commits off the network. The two migration safeguards each add a per-destination structure. On the transferring node, a queue holds the updates postponed toward the destination, shipped in order once the transfer is acknowledged and replayed on the local replica if it is refused. On a node forwarding an update for a collection it does not hold, a completion counter, decremented as each outgoing copy is delivered, re-applies the update locally after the last delivery, in case a replica has meanwhile arrived.

8. Evaluation This section presents the experimental evaluation of Ermes. The central promise of the platform is to reduce the latency a client perceives when a function accesses application state. Ermes brings the state close to where the computation runs and keeps it there as demand shifts, rather than fetching that state from a remote data store. Whereas our prior work [15] studied the placement and scheduling algorithm in simulation, here we evaluate the fully implemented platform, and the latencies we report are those a client experiences end-to-end from the running system. We organize the evaluation around four research questions (RQ), each isolating a distinct aspect of this promise: RQ1. How does Ermes’ state management reduce the latency a client observes?

12 https://aws.amazon.com/lambda/edge/

Latency (ms)

compilation together with our hybrid caching mechanism: this combination achieves near-native warm-start performance with a stable memory footprint and full deployment portability, while avoiding the architecture-specific binaries that ahead-of-time engines require. Ermes holds state in memory, in Redis (Sec. 7.2), so the latencies we report reflect the cost of locating, replicating, and coordinating state across the hierarchy, not disk I/O. The remainder of this section is organized as follows. Sec. 8.1 details the experimental setup. Sec. 8.2 characterizes the latency impact of state management (RQ1). Sec. 8.3 studies scalability under a growing workload and its comparison against the baselines (RQ2 and RQ4). Sec. 8.4 evaluates adaptation to static and mobile clients, again against the baselines (RQ3 and RQ4). Sec. 8.5 summarizes the findings.

Writer (IT/Milan/Edge3) Reader 1 (IT/Rome/Edge1) Reader 2 (IT/Rome/Edge2) Reader 3 (IT/Milan/Edge1) Reader 4 (IT/Milan/Edge2) Remote access

50 40 30 20 10 0

Epoch number

Figure 10: Per-epoch latency of five clients, one issuing writes and four issuing reads, each from a distinct edge node.

Latency (ms)

Client 1 (IT/Rome/Edge2)

8.1. Experimental Setup We deployed Ermes on a virtualized infrastructure managed by Proxmox Virtual Environment on an Intel Xeon Gold 6418H server. We emulated a three-tier hierarchy consistent with the model of Sec. 4: one Cloud Node (IT, 8 vCPUs, 8 GB RAM), two Mid Nodes (IT/Rome and IT/Milan, 4 vCPUs, 4 GB RAM each), and five Edge Nodes (IT/Rome/Edge1-2 and IT/Milan/Edge1-3, 2 vCPUs, 2 GB RAM each), all running Ubuntu 24.04 LTS. The centralized components of the Ermes platform (Group Token Provider and Function Registry) are isolated on a dedicated virtual machine mirroring the Cloud Node specifications. Although the virtual machines are physically co-located, we used the Linux Traffic Control (tc) subsystem with the NetEm scheduler to inject realistic, heterogeneous RTT delays on each virtual network interface, emulating the wide-area network characteristics of a geodistributed topology. The injected round-trip delays range from 5 ms to 20 ms per link, with the shorter delays on the edge-to-fog links and the longer ones on the fog-to-cloud links. In all experiments, clients communicate exclusively with edge nodes and have negligible latency toward them, while incurring the injected delays toward the rest of the infrastructure. The workload consists of clients that invoke functions reading and writing the records of collections. The number of collections depends on the experiment and reaches up to a thousand in the scalability study (Sec. 8.3).

60 50 40 30 20 10 0

Sequential consistency

Client 2 (IT/Milan/Edge3)

Remote access

Eventual consistency

4 3 2 1 0

Epoch number

Figure 11: Per-epoch latency of two clients issuing reads and writes from opposite edges.

the framework detects a many-to-one read pattern and provisions follower replicas accordingly, without degrading write throughput. One client continuously issues write invocations from IT/Milan/Edge3, while four clients at the remaining edge nodes issue read invocations against the same collection. Since placement evaluates each collection independently, the behavior observed for a single collection generalizes to the multi-collection case: with N independent collections, the system runs N parallel placement decisions, each following the same logic. The only source of inter-collection interference is memory contention, which arises when the aggregate working set exceeds a node’s capacity and forces the migration of collections to the parent node. We show in Sec. 8.3 that placement handles this case effectively, preserving quality of service under memory pressure. Fig. 10 shows the latency evolution for each client over the seven epochs (0 to 6) the scenario spans, distinguishing accesses served locally from those that still incur at least one remote fetch. Initially, the collection resides on the writer’s node and the four readers access it remotely, incurring high read latency: reads start between ≈ 20 and ≈ 60 ms, depending on the reader’s distance from the collection. Placement detects the many-to-one read pattern and provisions four follower replicas in parallel, one per reader node, without interrupting ongoing writes. Crucially, the writer’s latency remains unaffected despite the additional write traffic required to synchronize the four new replicas: it starts at ≈ 2 ms and settles around ≈ 1 ms from epoch 2. The read latency of the four readers likewise drops to ≈ 1 ms once a local follower is available, confirming that the framework manages multiple concurrent replications without introducing bottlenecks. The epoch at

8.2. State Management and Perceived Latency (RQ1) To answer RQ1, we take the perspective of the individual client, reporting the latency each one perceives as its accesses move from remote to local. We evaluate two aspects of state management: how many epochs the placement mechanism takes to turn a client’s remote accesses into local ones, and the latency each client observes at steady state under each of the two consistency policies. We first isolate the reactivity of placement. Starting from a single-node placement, we measure how quickly 13

Latency (ms)

Ermes — Sequential

10 9 8 7 6 5 4

Number of collections 10 100

Cloud-Data — Sequential

1000

Cloud-Only — Sequential 34 32

35 34

24 22

Latency (ms)

200

400

600

Ermes — Eventual

800

1000

2.75 2.5 2.25 2

200

400

600

800

Cloud-Data — Eventual

1000

1.5 1.25 0

200

400

600

Users

800

1000

22 0

200

400

600

Users

800

200

34 32 30

1.75

1000

400

600

800

1000

600

800

1000

Cloud-Only — Eventual

400

Users

Figure 12: Steady-state latency versus concurrency and state cardinality.

cross-topology path. The hatched bar portions confirm that read traffic remains local throughout, served by follower replicas. Under eventual consistency, latency drops to ≈ 1 ms from the first epoch onward, as both reads and writes are served locally and state synchronization propagates asynchronously, without blocking user requests. Together, the two experiments answer RQ1: state management turns remote accesses into local ones within a few epochs, and the residual latency a client perceives at steady state is dictated by the consistency policy, ranging from the near-local cost of eventual consistency to the single-leader floor of sequential consistency.

which each reader transitions from remote to local access varies with its position in the topology: the two readers colocated with the writer’s Fog Node (IT/Milan/Edge1-2) converge at epoch 2, while the two on the distant Fog Node (IT/Rome/Edge1-2) converge only at epoch 4. This ordering reflects the hop-by-hop nature of placement, which migrates the collection first toward the readers that are closer in latency and hops, and reaches the more distant subtree later. The elevated first epoch after convergence for each client is the cost of the first execution of a function on a node, which downloads the function binary from the Function Registry. We then isolate the latency each consistency policy imposes under concurrent, bidirectional traffic. Keeping the same infrastructure, two clients issue alternating read and write invocations against the same collection from opposite ends of the topology: one from IT/Milan/Edge1, colocated with the initial collection placement, and one from the distant IT/Rome/Edge2. Whereas the previous experiment showed how state management removes remote reads, this one exposes the cost of remote writes, where the two policies diverge. Fig. 11 shows the latency evolution per epoch. Under sequential consistency, the client co-located with the collection initially executes locally, while the remote client incurs high write latency, as writes must reach the leader replica to enforce a total order. The system then converges to a stable equilibrium in which both clients settle around ≈ 18 ms. This is the structural cost of sequential consistency: a single leader serializes the updates, so write traffic cannot be local for all nodes at once. The value is deliberately a worst case: the two writers sit at opposite edges of the topology, so the leader is necessarily remote from one of them and each write traverses a long

8.3. Scalability under a Growing Workload (RQ2 and RQ4) This section answers RQ2 and, through the comparison, RQ4. To answer RQ2 we quantify how the latency characterized above holds as the workload grows; to answer RQ4 we measure Ermes against the CD and CO baselines defined above. All experiments are measured at steady state, after placement has settled. We evaluate scalability along two axes: the size of the workload, measured by the number of concurrent clients and the number of collections, and the data-intensity of individual functions, measured by the number of collection accesses per invocation. We first vary the size of the workload through a stress test over two independent load parameters: the concurrency level Nusers , from 1 to 1024 simultaneous clients, and the state cardinality Ncol , from 1 to 1000 collections, all of which are actively accessed by clients. We increase the number of client-collection associations accordingly. Each panel of Fig. 12 plots, for one configuration and consistency policy, the median request latency against Nusers , with one curve per value of Ncol ; the shaded band around 14

250

each curve is the interquartile range (IQR) of the measured latency. In the CD baseline, latency is dominated by the network RTT to remote storage and remains largely insensitive to both Nusers and Ncol , since edge nodes act only as stateless executors. Under both consistency policies, latency sits around ≈ 32 ms, with a wider IQR for sequential consistency. The gap in the IQR reflects how writes are committed: they are synchronous under sequential consistency, blocking until the remote commit completes, whereas under eventual consistency they return immediately after local buffering, so the RTT penalty is structurally lower. The CO baseline is numerically almost indistinguishable from CD: latency is insensitive to both Ncol and Nusers , and sits around ≈ 31 ms under both consistency policies. Unlike CD, the two policies also share the same narrow IQR (≈ 30–34 ms). With computation and state both in the cloud, every state operation executes locally there, so the consistency policy adds no network cost, and both the median and the IQR are set entirely by the single roundtrip that carries each invocation to the cloud. In Ermes, under sequential consistency, every write to a collection must be serialized by its single leader replica. As the number of concurrent clients grows, more geographically dispersed writers contend for that leader, which placement then keeps at a higher tier to balance them, so each write traverses additional links. Latency is therefore driven by concurrency and stays largely insensitive to Ncol , staying between ≈ 1 ms and ≈ 2 ms. Memory pressure at high concurrency surfaces only as a widening of the IQR, without shifting this median trend. Under eventual consistency, the multi-leader design lets every replica accept writes locally, so latency stays low across almost the entire range: it grows only mildly with concurrency, from ≈ 1.5 ms with a single client to ≈ 1.75 ms with 1024 clients. In both cases, having 1000 collections combined with more than ≈ 256 clients results in a widening of the IQR (up to ≈ 10 ms for sequential and ≈ 2.75 ms for eventual): the collections no longer fit in the memory of the edge nodes, so some are placed one tier higher and accessed remotely when possible. Across the whole range, both Ermes configurations remain far below the CD baseline: even at its worst operating point, Ermes latency (≈ 10 ms) is under a third of the ≈ 32 ms the CD baseline pays regardless of load. We then vary the data-intensity of individual functions. Fixing concurrency and state cardinality at representative values (Nusers = 64, Ncol = 100), we vary the number of state operations per invocation from 1 to 10, comparing Ermes against the CD and CO baselines. Writes are buffered and committed at the end of the execution, so only reads contribute to the measured latency; the xaxis therefore reports the number of reads per invocation, which are issued sequentially, with no batching. Fig. 13 isolates the benefit of data locality. In the CD baseline, latency grows linearly with the number of reads,

Latency (ms)

200 150

Ermes [sequential] Ermes [eventual] Cloud-Data [sequential] Cloud-Data [eventual] Cloud-Only [sequential] Cloud-Only [eventual]

100 50 0

Number of operations

Figure 13: Steady-state latency versus data-intensity.

from ≈ 32 ms at a single read to ≈ 301 ms at ten: computation runs at the edge but the state is remote, so each read is a separate round-trip up to the cloud, and Texec ≈ Nreads × RT Tcloud . Moving computation to the edge without its state is thus actively harmful for dataintensive functions. The CO baseline, in contrast, is nearly flat, at ≈ 31–42 ms: functions execute in the cloud, colocated with the state, so reads are local and the cost is dominated by the single round-trip that carries the invocation to the cloud and back. The bottleneck is link traversal, not data access, so the number of reads barely matters. In Ermes, the profile is flat like CO, but an order of magnitude lower, at ≈ 1.4–3 ms across both policies, because the collection has been replicated to the edge node serving the client, co-locating computation and state at the edge, so every read is a local lookup with no round-trip at all. The comparison highlights that removing the per-read network cost requires co-locating computation and state, but only co-locating them at the edge, as Ermes does, also eliminates the round-trip that keeps CO an order of magnitude above it. 8.4. Adaptation to Static and Mobile Clients (RQ3 and RQ4) RQ2 characterized steady-state behavior; RQ3 asks how quickly and how gracefully the system reaches and maintains it as clients evolve, and RQ4 compares this adaptation against the CD and CO baselines. Unlike RQ1, which follows individual clients, here we take the aggregate view, reporting the average latency across all clients and centering the analysis on the comparison with the cloud-based baselines. We consider the two extremes of client dynamics: static clients, for which placement must converge to a locality-optimal configuration, and mobile clients, for which the state must continuously migrate to follow the client across the edge tier. We consider a heavy workload with high concurrency and large state scenario. Both experiments fix the state at Ncol = 1000 collections, with Nusers = 1024 for the static scenario and Nusers = 64 for the mobile one. With the number of nodes fixed, the client count governs how 15

Latency (ms)

50 40

Latency (ms)

Cloud-Data [eventual] Cloud-Data [sequential] Cloud-Only [eventual] Cloud-Only [sequential] Ermes [eventual] Ermes [sequential]

30 20 10 1

Epoch

60 50 40 30 20 10 0

Cloud-Data [eventual] Cloud-Data [sequential] Cloud-Only [eventual] Cloud-Only [sequential] Ermes [eventual] Ermes [sequential]

Epoch

Figure 14: Latency across epochs for static clients.

Figure 15: Latency across epochs for mobile clients.

densely collections spread across the edge, so the two experiments cannot share it. The static experiment takes maximal concurrency. The mobility trace instead takes few clients: a denser population would keep collections replicated on every node, warming each destination in advance and hiding the migration cost, whereas a sparse one leaves destinations cold. We first consider static clients, tracing the latency evolution across logical epochs. Fig. 14 presents this evolution. All baseline configurations start high, between ≈ 55 and ≈ 60 ms, as the first epoch pays the initial collection resolution, and then stay flat: CO, under both policies, and CD under sequential consistency all settle at ≈ 30 ms, dominated by the roundtrip to the cloud (RT Tcloud ), since CO places both computation and state there and the co-location constraint of sequential consistency forces CD to do the same; CD under eventual consistency settles slightly lower, at ≈ 25 ms. None of the baselines improves further, since the state never leaves the cloud. Ermes starts comparably high, at ≈ 55 ms under sequential and ≈ 45 ms under eventual consistency, but, unlike the flat baselines, converges downward within the first epochs as the collections migrate to the edge nodes serving the clients: latency settles at ≈ 8 ms under sequential and ≈ 3 ms under eventual consistency. Moving the state to the edge thus proves faster than fetching it remotely even during the initial convergence phase. We then consider mobile clients, simulating a scenario in which all active clients transition across the edge nodes in a round-robin fashion, changing access point every five epochs. Fig. 15 compares the latency evolution. As in the static case, the baselines are unaffected by mobility: all start around ≈ 60 ms in the first epoch and then settle at ≈ 30 ms for CO (both policies) and CD under sequential consistency, and at ≈ 25 ms for CD under eventual consistency. Ermes, in contrast, exhibits a characteristic sawtooth: starting from ≈ 60 ms, it converges to a low steady state punctuated by a spike at each move, as placement reactively migrates the collection toward the client’s new

location. Under eventual consistency, stable phases reach ≈ 1 ms and the spikes are sharp but bounded (≈ 10 ms, at epochs 6, 11, and 16), with recovery in the following epoch. Under sequential consistency, the steady state is higher (≈ 10 ms) and the spikes reach ≈ 20 ms, an overhead intrinsic to the single-leader protocol that enforces total ordering; recovery is also slower. In both cases the latency between moves stays well below the ≈ 25–30 ms of the baselines, confirming that Ermes preserves its locality advantage under continuous mobility. 8.5. Discussion Taken together, the experiments answer the four research questions for the edge-to-cloud setting Ermes targets, in which clients invoke functions from the edge. On the latency impact of state management (RQ1), placement turns remote accesses into local ones within a few epochs, provisioning replicas without perturbing ongoing writes, and the residual latency a client perceives is then dictated by the consistency policy, near-local under eventual consistency and bounded by the single-leader floor under sequential consistency. On scalability (RQ2), Ermes sustains low and stable latency as concurrency, state cardinality, and data-intensity grow, degrading gracefully only under strict ordering or edge memory saturation. On adaptation to client dynamics (RQ3), placement converges from the first epochs for static clients and keeps pace with mobile ones, tracking each client as it roams. On the comparison with cloud-based deployments (RQ4), Ermes outperforms both the CD and CO baselines by a wide margin throughout; the comparison further shows that co-locating computation and state cuts the per-access network cost only when done at the edge, as in Ermes, and not in the cloud, as in CO. The advantage is largest for read-intensive, eventually consistent, and mobile workloads. 9. Conclusion This paper presented Ermes, a stateful serverless platform that natively integrates state management into the 16

Funding

FaaS paradigm and jointly distributes computation and application state across the edge-to-cloud continuum. We evaluated Ermes on an emulated geo-distributed deployment, showing that Ermes effectively brings state close to computation and keeps it there as demand shifts, sustaining low client-perceived latency across a range of workloads and consistency requirements and outperforming realistic deployment solutions. Several directions remain open. Placement currently weighs the storage capacity of the nodes, and we plan to extend it into a fully resource-aware policy that also accounts for the compute available at each node. On the consistency side, we intend to investigate protocols that adapt the guarantee of a collection to its observed access pattern. We also plan to strengthen the platform’s fault tolerance: the multi-leader design of eventually consistent collections, together with the cloud replica that every collection retains, already makes recovery straightforward in most scenarios, whereas sequential consistency additionally requires a leader re-election mechanism to survive the failure of a leader, an orthogonal concern we leave to future work. Finally, we plan to evaluate Ermes on larger-scale physical infrastructures.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-forprofit sectors. Declaration of generative AI and AI-assisted technologies in the manuscript preparation process During the preparation of this work, the authors used generative AI in order to refine language editing and support software development; all content was reviewed and verified by the authors, who assume full responsibility for the manuscript. References [1] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, M. Zaharia, A view of cloud computing, CACM 53 (4) (2010). [2] W. Shi, S. Dustdar, The promise of edge computing, Computer 49 (5) (2016).

CRediT authorship contribution statement

[3] W. Shi, J. Cao, Q. Zhang, Y. Li, L. Xu, Edge computing: Vision and challenges, IoT Jour. 3 (5) (2016).

Matteo Cenzato: Conceptualization, Methodology, Software, Validation, Visualization. Dario d’Abate: Conceptualization, Methodology, Validation, Visualization, Writing – original draft, Writing – review & editing, Supervision. Arianna Dragoni: Conceptualization, Methodology, Validation, Visualization, Writing – original draft, Writing – review & editing, Supervision. Giacomo Orsenigo: Conceptualization, Methodology, Software, Validation, Visualization. Luca Tosetti: Conceptualization, Methodology, Software, Validation, Visualization. Matteo Briscini: Conceptualization, Methodology, Software, Validation, Visualization. Alessandro Margara: Conceptualization, Methodology, Validation, Writing – review & editing, Supervision, Resources.

[4] F. Bonomi, R. Milito, J. Zhu, S. Addepalli, Fog computing and its role in the internet of things, in: MCC, ACM, 2012. [5] Y. Li, Y. Lin, Y. Wang, K. Ye, C. Xu, Serverless computing: State-of-the-art, challenges and opportunities, TSC 16 (2) (2023). [6] J. M. Hellerstein, J. Faleiro, J. E. Gonzalez, J. Schleier-Smith, V. Sreekanti, A. Tumanov, C. Wu, Serverless computing: One step forward, two steps back (2018). arXiv:1812.03651. URL https://arxiv.org/abs/1812.03651

Data availability

[7] L. Baresi, D. Y. X. Hu, G. Quattrocchi, L. Terracciano, Neptune: A comprehensive framework for managing serverless functions at the edge, TAAS 19 (1) (2024).

The source code of the Ermes platform, the experiment configurations and raw results, and the code that regenerate all the figures in this paper are openly available on Zenodo [26].

[8] G. R. Russo, T. Mannucci, V. Cardellini, F. L. Presti, Serverledge: Decentralized function-as-a-service for the edge-cloud continuum, in: PerCom, IEEE, 2023. [9] M. Ciavotta, D. Motterlini, M. Savi, A. Tundo, Dfaas: Decentralized function-as-a-service for federated edge computing, in: CloudNet, IEEE, 2021.

Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

[10] T. Rausch, A. Rashed, S. Dustdar, Optimized container scheduling for data-intensive serverless edge computing, FGCS 114 (2021). 17

[11] C. Cicconetti, M. Conti, A. Passarella, A decentralized framework for serverless edge computing in the internet of things, TNSM 18 (2) (2021).

[25] A. Haas, A. Rossberg, D. L. Schuff, B. L. Titzer, M. Holman, D. Gohman, L. Wagner, A. Zakai, J. Bastien, Bringing the web up to speed with webassembly, in: PLDI, ACM, 2017.

[12] S. Gilbert, N. Lynch, Perspectives on the cap theorem, Computer 45 (2) (2012).

[26] M. Cenzato, D. d’Abate, A. Dragoni, M. Briscini, G. Orsenigo, L. Tosetti, A. Margara, Artifact for “"Ermes: a Stateful Serverless Platform for the Edge-toCloud Continuum”, [software], Zenodo, version v1.0.0 (2026). doi:10.5281/zenodo.21737909. URL https://doi.org/10.5281/zenodo.21737909

[13] T. Pfandzelter, D. Bermbach, Enoki: Stateful distributed faas from edge to cloud, in: MiddleWEdge, ACM, 2023. [14] S. Shillaker, P. Pietzuch, Faasm: lightweight isolation for efficient stateful serverless computing, in: ATC, USENIX Association, 2020. [15] M. Cenzato, D. d’Abate, A. Dragoni, M. Briscini, A. Margara, Data replication meets function scheduling in the edge-cloud continuum (2026). arXiv: 2606.30563. URL https://arxiv.org/abs/2606.30563 [16] P. K. Gadepalli, S. McBride, G. Peach, L. Cherkasova, G. Parmer, Sledge: a serverless-first, light-weight wasm runtime for the edge, in: Middleware, ACM, 2020. [17] T. Pfandzelter, D. Bermbach, tinyfaas: A lightweight faas platform for edge environments, in: ICFC, IEEE, 2020. [18] M. Cenzato, D. d’Abate, A. Dragoni, G. Orsenigo, L. Tosetti, A. Margara, Wasp: A configurable framework for portable stateful serverless applications (2026). arXiv:2607.25493. URL https://arxiv.org/abs/2607.25493 [19] A. Alabbas, A. Kaushal, O. Almurshed, O. Rana, N. Auluck, C. Perera, Performance analysis of apache openwhisk across the edge-cloud continuum, in: CLOUD, IEEE, 2023. [20] D.-N. Le, S. Pal, P. K. Pattnaik, OpenFaaS, Wiley, 2022, Ch. 17. [21] G. De Palma, S. Giallorenzo, J. Mauro, M. Trentin, G. Zavattaro, Funless: Functions-as-a-service for private edge cloud systems, in: ICWS, IEEE, 2024. [22] C. Cicconetti, E. Carlini, R. Hetzel, R. Mortier, A. Paradell, M. Sauer, Edgeless: A software architecture for stateful faas at the edge, in: HPDC, ACM, 2024. [23] D. Carrizales-Espinoza, D. D. Sanchez-Gallegos, J. Gonzalez-Compean, J. Carretero, Structmesh: A storage framework for serverless computing continuum, FGCS 159 (2024). [24] C. P. Smith, A. Jindal, M. Chadha, M. Gerndt, S. Benedict, Fado: Faas functions and data orchestrator for multiple serverless edge-cloud clusters, in: ICFEC, IEEE, 2022. 18

Record · ID 965377 · SHA-256 3f9234b8043d4788
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.