CocoRNA: Collective RNA Design with Cooperative Multi-agent Reinforcement Learning

CocoRNA: Collective RNA Design with Cooperative Multi-agent Reinforcement Learning* Tianmeng Hu1,2 , Biao Luo3‡ , Ke Li1,2‡

arXiv:2610.23820v1 [cs.LG] 16 Oct 2026

1

Living Systems Institute, University of Exeter, United Kingdom 2 Department of Computer Science, University of Exeter, United Kingdom 3 School of Automation, Central South University, China Email: tianmeng0824@gmail.com, biao.luo@hotmail.com, k.li@exeter.ac.uk

Abstract Designing RNA sequences that reliably fold into specific secondary structures is essential for understanding their biological functions but remains a challenging computational problem. We propose CocoRNA, a cooperative multi-agent reinforcement learning framework for RNA inverse design. CocoRNA simplifies the design task by decomposing it into smaller subproblems, each solved collaboratively by multiple agents. This approach reduces the complexity of the problem and improves the exploration of design policies. During training, a centralized critic uses global structural information to guide the agents, enabling them to jointly optimize their design strategies. As a result, CocoRNA learns high-quality RNA design policies that generalize effectively to unseen structures without additional training. Experiments on the Rfam dataset demonstrate that CocoRNA substantially outperforms state-of-the-art methods in both success rate and design speed. Further experiments on other biological sequence design tasks highlight the effectiveness and broad potential of CocoRNA for complex design tasks.

1

Introduction

RNA is a polymeric molecule composed of nucleotides. Each nucleotide contains one of four bases: adenine (A), guanine (G), cytosine (C), or uracil (U). These bases carry genetic information. Nucleotides fold into complex secondary structures following Watson-Crick and Wobble base-pairing rules. Common structural elements include stems, loops, bulges, and junctions. For RNA, secondary structures are often more critical for function than they are in proteins [43]. These structures participate directly in catalysis, ligand recognition, and molecular scaffolding across diverse RNA classes [71, 11]. Examples include ribosomal RNAs [56], messenger RNAs [58], and small nuclear RNAs [19]. Secondary structures also regulate translation initiation and influence RNA polymerase II activity, thus controlling gene translation and splicing [24]. However, the precise roles of RNA secondary structures in many biological processes remain unclear [24]. To fully understand RNA functions, researchers need reliable methods to investigate and manipulate these structures [71]. Computational inverse design can address this need, guiding experiments and reducing costs associated with testing structural hypotheses [14, 52]. Computational RNA secondary structure design is essentially an inverse problem. The goal is to find RNA sequences that fold into a specific secondary structure with minimum free energy. This problem was first introduced in the early 1990s [28]. It is mathematically ill-posed in the Hadamard sense because multiple distinct sequences can fold into the same structure, as illustrated in Figure 1. Solving this inverse design problem is critical for advancing functional RNA research [68, 65]. Key applications include synthetic RNA biology [15] and RNA nanostructure design [26]. * ‡

Accepted as a conference paper at ICML 2026 Corresponding author

[ COLAlab Report | June 1, 2026

_ 1/29

A

B Text

Figure 1: A. Examples of RNA sequences and structures visualized using Forna [39]. (a) is an unfolded RNA sequence. Structures (b) and (c) show two different folded forms of this sequence. Structure (c) is specifically generated by RNAfold [45], which uses a minimum free energy algorithm. Structure (d) is folded from a different sequence, yet it shares the same secondary structure as (c). B. Structure density surface for RNA sequences of length 150. The plot illustrates the distribution of Hamming distances between RNA sequences and their corresponding secondary structure distances. A major challenge in computational RNA design is the exponential growth of the sequence space. For an RNA sequence of length ℓ, there are 4ℓ possible sequences. However, only a small fraction of these sequences will fold into the desired structure. For example, a 74-nucleotide tRNA with 40 paired and 34 unpaired positions results in a search space of approximately 1036 sequences. This enormous search space makes brute-force approaches computationally infeasible. Stochastic optimization is a widely used approach for computational RNA design. These methods typically rely on RNA folding prediction tools, such as ViennaRNA’s RNAfold [45], mfold/UNAFold [48], and RNAstructure [2]. These tools predict RNA structures by minimizing free energy according to established thermodynamic parameters [49, 50]. The design process begins with an initial seed sequence. Techniques such as adaptive random walks [28] or stochastic local search iteratively mutate and evaluate candidate sequences until a suitable solution emerges. Well-known algorithms in this domain are RNASSD [4] and INFO-RNA [12]. Other effective strategies draw from evolutionary algorithms [69, 18, 17], constraint programming [21, 22], ant colony optimization [40], and ensemble optimization [78]. Neural combinatorial optimization (NCO) has recently emerged as a powerful approach for combinatorial problems. Examples include the traveling salesman problem [10], maximum cut [8], and bin packing [37]. NCO combines the representational strength of deep neural networks with the search efficiency of reinforcement learning (RL). In computational RNA design, Eastman et al. [16] first introduced a pre-trained RL policy. Their policy learned RNA folding rules to guide the search process. Experiments on the Eterna100 benchmark [3] demonstrated encouraging performance compared to stochastic optimization methods. Later, Runge et al. [59] proposed LEARNA, an RL-based approach that directly generates RNA sequences, enhancing success rates and computational efficiency. Despite these advances, RL-based RNA inverse design still faces three major challenges: Œ Combinatorial explosion and sample inefficiency: The RNA design space grows exponentially with sequence length, requiring enormous sample sizes. As a result, RL models struggle to accurately estimate value functions from limited data. Œ Sparse rewards and local optima: Very few sequences exactly match the target structure, creating sparse rewards. Additionally, the RNA design landscape contains numerous local optima (see Figure 1). This combination makes policy improvement challenging. Œ Delayed rewards and misleading auxiliaries: Strict success feedback in RNA inverse design is only available when a correct sequence is found. Random exploration rarely finds these sequences, causing most attempts to yield no useful feedback. Long task horizons further complicate intermediate evaluations. Some methods [5, 59] use auxiliary rewards to guide exploration. However, these signals can overshadow the primary objective in long tasks, causing policies to overfit short-term gains and become trapped in local optima.

_ 2/29

These challenges significantly limit the effectiveness of current RL methods [16, 59]. As demonstrated by our experiments in Section 4, existing approaches typically require substantially longer computation time, yet achieve only modest success rates. Furthermore, once algorithms become trapped in local optima, additional search efforts yield diminishing returns. û Conceptually, navigating the RNA sequence space resembles exploring a complex maze. Multiple explorers, each starting from different points and sharing their discoveries, can map this maze more efficiently than a single explorer. Inspired by this analogy, we propose using multi-agent reinforcement learning (MARL) to tackle computational RNA design.  We formulate computational RNA design as a collective task, where multiple agents cooperate to generate RNA sequences. To our knowledge, this is the first study to frame biological sequence design explicitly as a multi-agent decision-making problem and the first to apply MARL in this domain.  We develop CocoRNA, a MARL framework designed for this collective RNA design task. CocoRNA instantiates cooperative MARL for RNA inverse design through a Dec-POMDP formulation over joint sequence–structure states, RNA-aware decomposition schemes, and a search-augmented exploration mechanism to enhance learning efficiency.  We rigorously evaluate CocoRNA on the Rfam dataset, along with extensive ablation studies. Under identical time constraints, CocoRNA achieves a 97.75% success rate, significantly surpassing the best baseline by 27.25 percentage points (38.7% relative improvement). Even when the strongest baseline receives 20 times more computational time, CocoRNA still maintains an 8.21 percentage point lead (9.2% relative advantage). Importantly, CocoRNA achieves an average solution time of only 3.24 seconds, which is over 70 times faster than the best-performing baseline.

2

Preliminaries

RNA secondary structure representation In computational biology, RNA secondary structures are commonly represented using dot-bracket notation. Unpaired nucleotides appear as dots (.), while base pairs are indicated by matching parentheses. An opening parenthesis (() marks the 5′ end of a base pair, and a closing parenthesis ()) marks the 3′ end. For example, ((. . .)) represents a hairpin loop, with paired bases at both ends and unpaired nucleotides in the middle. Problem formulation In RNA inverse design, the input is a target secondary structure rather than a source RNA sequence, and the goal is to generate RNA sequences whose predicted folded structures match the target. This setting is motivated by practical applications, where researchers often start from a desired structural motif and seek sequences that can realize it. Moreover, because multiple distinct RNA sequences can fold into the same secondary structure, inverse design also enables optimization of additional properties such as GC content and minimum free energy while preserving the target fold. Decentralized partially observable Markov decision process We model the fully cooperative MARL problem as a decentralized partially observable Markov decision process [53]. Formally, this is defined by the tuple ⟨N , S, Oi , Ai , P, R, γ⟩, where: • N = {1, . . . , N }: The set of N agents. • S: The global state space. • Oi : Local observation space of agent i. • Ai : The action space for agent i. • P : S × A × S → [0, 1]: State transition probability. _ 3/29

Dot-bracket notation structure

(

(

(

· · · ) ) )

C G U A A A A C G Randomly initialized sequence

··· ··· ··· Secondary Structure

Multi-Agent Actor-Critic Network

Output Sequence

Figure 2: Overview of the CocoRNA workflow. • R : S × A → [Rmin , Rmax ]: Shared reward function. • γ ∈ [0, 1]: The discount factor. At each time step t, the environment is in state st ∈ S, and each agent i ∈ N observes oit ∈ Oi and selects ait ∈ Ai according to π i (ait | oit ). The joint action is at = (a1t , . . . , aN t ). The environment transitions to st+1 with probability P (st+1 | st , at ), and all agents receive the shared reward R(st , at ). P∞ t The goal is to find π = (π 1 , . . . , π N ) maximizing G = E t=0 γ R(st , at ) . Centralized training with decentralized execution (CTDE) CTDE is a widely used paradigm in MARL [57, 77, 32]. During training, agents access global information to evaluate and improve their policies; during execution, each agent acts only from local observations. This mitigates non-stationarity and improves scalability. Actor-Critic methods with CTDE In the standard Actor-Critic framework [9], the Critic estimates the value function, while the Actor learns and updates its policy using the Critic’s value estimates. Under CTDE, this naturally extends to centralized Critics and decentralized Actors [46, 36, 77]: during training, the Critic leverages global information to guide each agent’s Actor; at execution time, agents act independently from local observations.

3

Proposed Method

The workflow of CocoRNA is presented in Figure 2 and its pseudocode is provided in Appendix A. In the following paragraphs, we will delineate its key algorithmic components step by step.

3.1

Problem decomposition

In a canonical MDP, a policy π : S → A maps states to actions. Ideally, the state space S should capture all information relevant for decision-making. RNA design decisions depend primarily on two factors: the target RNA secondary structure, and the current RNA sequence. These factors matter because nucleotide positions have interdependencies due to base-pairing constraints. Thus, the state space S must combine RNA sequence and structural information: n

S = (sseq , sstruct ) sseq ∈ {A, U, G, C}ℓ , o

sstruct ∈ {., (, )}ℓ .

(1)

where sseq is the nucleotide sequence, and sstruct is the target secondary structure in dot-bracket notation. For sequences of length ℓ, the total number of possible states is |S| = (4 × 3)ℓ , covering all _ 4/29

possible sequence-structure combinations. Such exponential growth results in a high-dimensional policy space, complicating policy learning and exploration. To disentangle this complexity, we propose using MARL within the CTDE framework. We decompose the RNA design problem into multiple subproblems, assigning them to different agents who solve these tasks cooperatively. Each agent then focuses only on specific positions or substructures. This targeted approach reduces the dimensionality of each agent’s state and policy spaces. In this work, we propose two decomposition schemes: • Position-based decomposition: The RNA sequence is split into n segments, each assigned to a distinct agent. Each agent decides nucleotides exclusively for its segment. • Structure-type-based decomposition: Each agent is assigned to a particular structural type (e.g., stems, loops). Agents design nucleotides solely at positions corresponding to their structural assignment. Each agent receives a lower-dimensional local observation containing only a fragment of the RNA sequence and structure around its current position. Formally, an agent’s observation space O is: 

O = (oseq , ostruct ) oseq ∈ ({A, U, G, C, ∅})m , 

(2)

ostruct ∈ ({., (, ), ∅})m , where m is the length of the observation window (an odd integer). The symbol ∅ serves as a placeholder for positions beyond sequence boundaries. For agent i, its local observation oit is: 

[i−κ, i+κ]

i+κ] oit = s[i−κ, , sstruct seq



,

(3)

i where κ = m−1 2 . ot consists of the local nucleotide and structural subsequences centered at position i. To handle variable-length inputs, sequences are padded with ∅ to a maximum length lmax . This local observation provides sufficient context for the agent’s decision. Although our primary implementation uses dot-bracket notation, the CocoRNA framework only requires a computable state representation and can therefore be instantiated with alternative encodings (e.g., a base-pairing matrix). In Section 4, we further implement a matrix-based variant that uses a three-channel input (Structure, Sequence, Position) processed by a CNN, enabling CocoRNA to naturally handle pseudoknots. By decomposing the complex RNA design task into smaller subproblems, we leverage the collective strengths of MARL. Each agent specializes either in a sequence segment or a specific structural type, enabling more focused and efficient policy learning.

3.2

Policy optimization

We adopt multi-agent proximal policy optimization (MAPPO) [77] as our base algorithm due to its robust performance in various MARL tasks. Our framework provides each agent with two distinct networks: an Actor for decision-making and a Critic for policy evaluation. Specifically, the Actor is implemented as an MLP, while the Critic uses a CNN-based architecture to process the global sequence–structure state. Importantly, each agent maintains its own network parameters, enabling independent learning and specialized adaptation to its specific design role. Actor network Each agent’s Actor network receives a local observation oit as input, which provides relevant context around the agent’s assigned position. The network then outputs a probability distribution over the discrete action space: A = {A, U, G, C},

(4)

where each action corresponds to selecting a nucleotide type. We denote agent i’s policy as πθi (ait | oit ), where θi denotes the parameters of its Actor.

_ 5/29

Base-pair compatibility is enforced by the environment as a hard constraint. When a position whose paired partner has already been assigned is to be filled, the environment applies a validity mask so that only nucleotides compatible with the existing assignment remain admissible according to Watson-Crick and wobble pairing rules (A-U, G-C, and G-U). For example, if one side of a pair has already been assigned G, the paired position can only take C or U. The Actor still outputs a probability distribution over all four nucleotides, but invalid actions are masked out before execution. During interaction with the environment, each agent selects an action ait from the valid action set and executes it by placing the corresponding nucleotide. Critic network The Critic network takes the global state st as input, including both the current RNA sequence and its target secondary structure. We process these two components separately using two convolutional modules. Their outputs are concatenated and fed into subsequent Critic layers to produce a scalar value estimate V (st ). This estimate reflects the expected cumulative  future rewards under the joint policy π: V π (st ) = Eπ

P∞

k=0 γ

kr t+k

st . Using the Critic’s value

estimates, we calculate the temporal difference (TD) error δt = rt + γV (st+1 ) − V (st ). Next, we apply P k generalized advantage estimation (GAE) [64] to compute advantage estimates At = ∞ k=0 (γλ) δt+k , i where λ ∈ [0, 1]. These advantage estimates guide updates to each agent’s Actor parameters θ . We follow the standard MAPPO update rule. The convergence of our algorithm is established by the following lemma, assuming standard conditions for policy gradient convergence are satisfied [41, 67]. The detailed proof is in Appendix B. Lemma 3.1. Consider CocoRNA’s multi-agent actor-critic algorithm, where a centralized Critic estimates the joint advantage function, and decentralized policies are updated according to the gradient: "

g = Eπ

# X

∇θi log π (a | o ) A(s, a) . i

i

i

(5)

i

Suppose the joint policy π is differentiable, uses an appropriate learning rate, and the Critic is compatible with π [41, 67]. Then, with probability 1, the algorithm converges to a local maximum of the expected joint return J π . That is: lim inf ∇θ J π (θk ) = 0 w.p. 1. (6) k→∞

The centralized Critic uses global information to estimate values, guiding local policy updates of each agent. This strategy ensures individual actions align effectively with the team objective. By combining centralized value estimation and decentralized decision-making, we promote cooperative behavior despite agents acting independently based on local observations.

3.3

Reward function

We formulate RNA inverse design as a MDP which treats RNA design as a navigation task through the large design space. Our reward function is inspired by standard navigation problems, explicitly encouraging both exploration and optimization. First, we use the ViennaRNA [45], a minimum free energy (MFE) folding algorithm, to predict the secondary structure xf of the current RNA sequence. We then compute the Hamming distance, P H(xf , xt ), between this predicted structure and the target structure xt as: H(xf , xt ) = ℓi=1 1(xfi ̸= xti ), where 1(·) is an indicator function returning 1 if xfi ̸= xti , and 0 otherwise. Reward function consists of two components: an intermediate reward and a final reward. At each step t, the multi-agent team selects actions, and the environment provides an intermediate reward reflecting the normalized reduction in Hamming distance from the previous time step. If the Hamming distance reaches 0, it means the predicted structure exactly matches the target structure. We then assign a substantial final reward C, and the episode terminates. Formally, the reward Rt is defined as: (Ht−1 − Ht )/ℓ, if Ht > 0, C, if Ht = 0,

(

Rt =

(7) _ 6/29

where C > 0 is a constant. The intermediate reward encourages stepwise improvements, rewarding incremental reductions in structural differences. The substantial final reward C strongly incentivizes agents to achieve a perfect match, thereby aiding them in overcoming local optima. We further design multi-objective variants of the reward function. Specifically, we add an auxiliary term raux that is only activated when the structural design is successful (i.e., Ht = 0); otherwise, raux = 0. For example, to steer the GC content toward a target ct , we use rGC = −α|cGC − ct |, where cGC ∈ [0, 1] is the normalized GC content and α > 0 is a scaling factor. To optimize minimum free energy, we use rMFE = −βE, where E is the MFE predicted by a folding engine and β > 0 is a scaling factor. We provide empirical results of these variants in Section 4.

3.4

Search-augmented exploration

The enormous RNA design space often causes cold-start issues for learning-based methods [72]. In the early training stages, random policies often generate low-quality experiences, which limit learning efficiency. To address this, we introduce a heuristic called search-augmented exploration (SAE). SAE employs a limited greedy search early in training to enhance the quality of experiences stored in the replay buffer. Specifically, after an agent selects an action a based on its current policy π and receives reward r, we perform a brief local search. This search checks whether an alternative action a′ can yield a higher reward r′ . If such an improved action is found, we replace the original experience tuple (o, a, r) with (o, a′ , r′ ). By improving data quality early on, SAE accelerates the agents’ learning progress. In practice, CocoRNA applies SAE during the first 30% of training, effectively mitigating coldstart challenges by exposing agents to more rewarding experiences earlier. Importantly, SAE does not introduce additional learning objectives or modify the original optimization goals. Therefore, SAE preserves MAPPO’s convergence guarantees, as detailed in Section 3.2 and Appendix B.

4

Empirical Study

4.1

Experiments on RNA inverse design

Dataset We evaluate CocoRNA on RNA secondary structures from the Rfam dataset [60]. We compute pairwise edit distances among 65, 000 structures and apply mini-batch K-means to partition them into 100 clusters. One cluster (650 structures) serves as the test set, while the remaining data are filtered to remove structures highly similar to the test set, yielding 60, 000 training structures. The minimum edit distance between training and test structures is 111, with an average of 188.42. To ensure robustness and prevent data leakage, we also construct six alternative splits by randomly selecting different clusters as test sets (details in Appendix C.1). Experimental setup We use position-based decomposition with n = 4 agents, where each RNA structure is divided into four subsequences of approximately equal length, each assigned to one agent. An episode has at most 400 steps and is considered successful if the agents collectively generate a sequence that folds into the target structure. We train the model for a total of 25 million steps. During evaluation, we generate sequences for each test structure with up to 15 attempts under a 30-second time limit. Network architectures and detailed hyperparameters are provided in Appendix F. Metrics & baselines We use success rate and diversity as primary metrics. Success is an exact structural match (Hamming distance = 0), and diversity is the average pairwise normalized Hamming distance among generated sequences. We compare CocoRNA with five state-of-the-art baselines: antaRNA [40], LEARNA and Meta-LEARNA-Adapt [59], SAMFEO [78], libLEARNA [61] and OmniGenome [75]. OmniGenome is a recent large-scale foundation model with 186M parameters, functioning as a generative model assisting genetic algorithms for RNA inverse design. antaRNA and SAMFEO are stochastic optimization methods, both requiring significant optimization time. LEARNA and its meta-learning variant, Meta-LEARNA-Adapt, perform online learning per sequence, demanding instance-specific policy training and additional computational resources. To ensure fair efficiency comparisons, we evaluate

_ 7/29

Table 1: Evaluation of CocoRNA and baseline methods on the RNA inverse design task. The time limit for CocoRNA is set to 30 seconds. The suffixes in baseline names indicate different time constraints; for example, “−600s" denotes a time limit of 600 seconds per structure. Method

Number of solved samples

Mean success rate

Average diversity

CocoRNA

635.40 ± 2.58

97.75%

0.405

LEARNA-30s LEARNA-300s LEARNA-600s

63.80 ± 3.03 368.20 ± 6.18 382.20 ± 4.82

9.82% 56.65% 58.80%

0.311 0.339 0.336

Meta-LEARNA-Adapt-30s Meta-LEARNA-Adapt-300s Meta-LEARNA-Adapt-600s

381.20 ± 5.63 564.20 ± 4.09 582.00 ± 4.98

58.65% 86.80% 89.54%

0.377 0.392 0.400

libLEARNA-30s libLEARNA-300s libLEARNA-600s

440.20 ± 7.16 554.60 ± 3.58 583.60 ± 2.97

67.72% 85.32% 89.79%

0.385 0.351 0.371

SAMFEO-30s SAMFEO-60s SAMFEO-600s

458.20 ± 6.10 488.60 ± 7.64 560.00 ± 2.76

70.50% 75.17% 86.15%

0.361 0.388 0.394

antaRNA-60s antaRNA-600s antaRNA-1200s

87.80 ± 2.70 110.00 ± 6.67 171.40 ± 5.03

13.51% 16.92% 26.37%

0.361 0.390 0.387

OmniGenome-60s OmniGenome-600s OmniGenome-1200s

261.4 ± 4.04 502.20 ± 4.21 532.40 ± 4.51

40.22% 77.26% 81.91%

0.389 0.380 0.388

these baselines under multiple time constraints. Note that we give CocoRNA the most strict time limit 30 seconds in all experiments. CocoRNA achieves the highest success rate Table 1 summarizes the results over five runs. Under a strict 30-second limit, CocoRNA achieves 97.75% success, improving by 27.25 absolute percentage points (38.7% relative) over the best baseline at the same budget (SAMFEO-30s). Even with 600s budgets (20× longer), CocoRNA still outperforms Meta-LEARNA-Adapt-600s by 8.21 points (9.2% relative), establishing CocoRNA as a highly reliable method. CocoRNA is highly efficient Beyond accuracy, CocoRNA is highly efficient: it finds solutions in 3.24 seconds on average, over 70× faster than Meta-LEARNA-Adapt (234.2 seconds at the 600-second limit). Figure 3 shows the time distribution and its dependence on sequence length, confirming that CocoRNA is consistently faster than all baselines. Notably, OmniGenome (186M parameters) does not outperform classical stochastic optimizers, and antaRNA performs poorly (success rate 26.37% at 1, 200s), often getting stuck in local optima. CocoRNA maintains good diversity In addition to high success and efficiency, CocoRNA produces diverse solutions. As shown in Table 1, CocoRNA achieves the highest average pairwise normalized Hamming distance (0.405), indicating broader exploration of the sequence space. Long RNA design and family-wise results CocoRNA is not inherently limited by sequence length. While our implementation uses a fixed input dimension (450 nt), longer sequences can be handled via a sliding-window representation. To demonstrate scalability, we tested the original

_ 8/29

Table 2: Multi-objective design with CocoRNA. Objectives Struct. Struct. + GC Struct. + MFE Struct. + GC + MFE CocoRNA

Time (seconds)

GC

MFE

97.75% 97.20% 97.07% 97.26%

0.62 0.51 – 0.53

−125.1 – −158.5 −151.2

Meta-LEARNA-Adapt-600s

LEARNA-600s

1000

Success rate

SAMFEO

OmniGenome

1000

1000

1000

1000

100

100

100

100

100

10

10

10

10

10

1

1

1

1

1

0.1

0.1

0.1

0.1

0.1

0.01

0.01

0.01 0

100

200

300

400

Sequence Length

500

0

100

200

300

400

Sequence Length

500

0.01

0.01 0

100

200

300

400

Sequence Length

500

0

100

200

300

400

Sequence Length

500

0

100

200

300

400

Sequence Length

Figure 3: CPU time distribution of CocoRNA and four baselines over successfully solved cases. CocoRNA model (without retraining) on 500 RNA sequences from Rfam ranging from 500 nt to 971 nt, using a sliding-window approach (a 600s time limit, repeated 5 times). Despite being evaluated on sequences significantly longer (up to ∼ 2×) than those seen during training, CocoRNA achieves a 92.4% success rate, highlighting strong zero-shot generalization. Matrix-based CocoRNA and pseudoknot design The dot-bracket representation used in our primary implementation cannot naturally express pseudoknots; however, the CocoRNA framework is representation-agnostic and can be instantiated with alternative encodings. As shown in Figure 4 (middle), our matrix-based variant achieves a 96.95% success rate, which is comparable to the dotbracket implementation. We further evaluate the matrix-based variant on a pseudoknotted dataset [40], with IPknot [63] as the oracle and fine-tune it using only 10% of the original training budget. On three held-out test sets, CocoRNA achieves mean success rate 92.67%, outperforming antaRNA by over 28 percentage points. Multi-objective design We further study multi-objective reward variants that control GC content (rGC , target ct = 0.5) and minimize free energy (rMFE ), where auxiliary rewards are applied only when the structural design is successful (see Section 3.3). Table 2 shows that CocoRNA can steer GC content and improve MFE without sacrificing structural accuracy. CocoRNA generalizes across different oracles We train CocoRNA with ViennaRNA and evaluate using RibonanzaNet [27], a deep learning–based structure predictor trained on large-scale experimental data. As shown in Figure 4, CocoRNA achieves 94.90% success; the slight drop vs. ViennaRNA is expected, but the strong performance indicates robust generalization beyond a single oracle. Family-wise performance We construct four family-specific test sets (500 structures each, deduplicated from Rfam seed). Figure 4 (right) shows results on five families: CocoRNA achieves > 99% success on three, while ribozymes are harder (94.0%). When extending the time limit to 600s, the ribozyme success rate recovers to 98.8%, indicating that certain ribozyme structures present a more rugged optimization landscape and require extended search time for CocoRNA to locate the global optimum. Other biological sequence design tasks We apply CocoRNA to solve other biological sequence design problems in the widely-used benchmark [70], including TF Bind 8, TF Bind 10 [7], ChEMBL [23], _ 9/29

Figure 4: (Left) Cross-oracle validation experiment using RibonanzaNet. (Middle) Matrix representation and pseudoknot design results. (Right) Success rate breakdown across RNA families.

Figure 5: (Left) Comparison of sequence recovery rates during training between the Single-agent variant and CocoRNA. (Right) Effect of removing the SAE heuristic from CocoRNA. and UTR [62]. Experimental results show strong generalization. Details are provided in Appendix D.

4.2

Ablation studies

We perform ablation studies to dissect the contributions of different design choices in CocoRNA. Multi-agent architecture yields clear gains To assess the benefit of collective design, we compare CocoRNA with a Single-agent variant that controls all positions. As shown in Figure 5 (left), the single-agent variant consistently underperforms CocoRNA during training, indicating that multi-agent decomposition improves exploration and accelerates convergence. SAE stabilizes and accelerates training We further evaluate a variant of CocoRNA without the SAE heuristic (CocoRNA-ablated). The learning curves in Figure 5 (right panel) show that this ablated model converges more slowly and exhibits higher variance. This indicates that SAE provides high-quality experiences early in training, significantly improving stability and efficiency. More detailed SAE-related analyses are deferred to Appendix C.3. Additional robustness analyses To comprehensively evaluate CocoRNA’s robustness, we also investigated alternative decomposition schemes, reward functions, numbers of agents, parameter-sharing strategies, dataset splits, and hyperparameter settings. Results consistently confirm the robustness and versatility of our framework. Full details and discussions are presented in Appendix C.

5

Related Works

The RNA inverse design problem was first introduced by Hofacker et al. [28] in the Vienna RNA package, which uses an adaptive random walk algorithm. There are various follow-up approaches. For example, RNA-SSD [4] enhanced sequence initialization and reduced complexity by dividing _ 10/29

sequences into smaller substructures. INFO-RNA [12] further combined dynamic programming for sequence initialization with stochastic local search. Later approaches explored larger search spaces and incorporated additional constraints using evolutionary algorithms [69, 18, 17], Monte Carlo tree search [76, 55], and simulated annealing [51]. Other notable methods utilized constraint programming [21, 22], ant colony optimization [40], or ensemble optimization [78]. However, these methods primarily rely on stochastic optimization guided by handcrafted heuristics. Such iterative search approaches are computationally expensive, with complexity rapidly increasing as sequence length grows. Moreover, partially due to the no free lunch theorem [74], these heuristics hardly generalize to new problems. In contrast, machine learning methods can automatically discover effective heuristics from data. Recent learning-based methods have gained prominence, particularly generative models [35], large language models [44, 66], and RL approaches [5, 31]. These methods are successfully applied in various biological design tasks, such as protein function optimization [47, 73], protein sequence design [13, 20], and drug repurposing [34]. For RNA secondary structure design, Eastman et al. [16] first utilized RL with pre-trained policies, effectively replacing handcrafted heuristics. Later, Runge et al. [59] combined RL with neural architecture search to simultaneously optimize network architectures and hyperparameters. They further refined generative policies through instance-specific fine-tuning. Recent generative methods like RNAinformer [54] show promising results but are computationally intensive, limiting practical applicability to sequences under 100 nucleotides and lacking open-source availability. Similarly, OmniGenome [75], a large-scale 186M-parameter generative model, tackles RNA structure design by learning direct structure-to-sequence mappings. We clarify key distinctions between CocoRNA and related methods. CocoRNA employs a MARL framework, decomposing the RNA design problem into manageable sub-tasks solved cooperatively. While RNA-SSD [4] also divides sequences into independently optimized subsequences, it does not effectively incorporate global structural information and thus often fails to achieve globally optimal solutions by combining local solutions. In contrast, CocoRNA uses a centralized Critic with complete global context, guiding agents to jointly reach optimal solutions. Another related method, GameOpt [6], frames combinatorial Bayesian optimization as a cooperative game among optimization variables. Although both CocoRNA and GameOpt adopt decomposition strategies, their approaches differ fundamentally: CocoRNA trains dynamic policies for cooperative adaptation, whereas GameOpt searches for a fixed solution over predefined optimization variables.

6

Conclusion

We propose CocoRNA, a collective design method that employs cooperative MARL to address the challenges of RNA secondary structure design. By formulating the RNA design task as a collective problem and decomposing it into multiple sub-tasks assigned to individual agents, CocoRNA reduces the complexity faced by each agent, enhances policy exploration, and improves learning efficiency. Experimental results fully demonstrate the effectiveness and generalizability of CocoRNA. Future work could explore more accurate reward models, adaptive decomposition strategies, and scalable credit-assignment mechanisms for larger multi-agent biological design problems.

Acknowledgments This work was supported by the UKRI Future Leaders Fellowship under Grant MR/S017062/1 and MR/X011135/1; in part by National Natural Science Foundation of China under Grant 62376056 and 62076056; in part by the Science and Technology Innovation Program of Hunan Province under Grant 2024RC1011 and the Major Basic Research Projects in Hunan Province under Grant 2026JC0001; in part by the Isambard-AI, Royal Society Faraday Discovery Fellowship (FDF/S2/251014), BBSRC Transformative Research Technologies (UKRI1875), Royal Society International Exchanges Award (IES/R3/243136), Kan Tong Po Fellowship (KTP/R1/231017); and the Amazon Research Award and Alan Turing Fellowship.

_ 11/29

Impact Statement The proposed method may potentially accelerate advances in synthetic biology, RNA-based drug development, and the study of gene regulatory mechanisms, thereby contributing to the development of novel therapeutic strategies. However, applying AI models to biological sequence design may introduce risks related to interpretability and reliability. Enhancing the interpretability of AI models could help mitigate such concerns.

References [1] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J. Ballard, Joshua Bambrick, Sebastian W. Bodenstein, David A. Evans, Chia-Chun Hung, Michael O’Neill, David Reiman, Kathryn Tunyasuvunakool, Zachary Wu, Akvilė Žemgulytė, Eirini Arvaniti, Charles Beattie, Ottavia Bertolli, Alex Bridgland, Alexey Cherepanov, Miles Congreve, Alexander I. Cowen-Rivers, Andrew Cowie, Michael Figurnov, Fabian B. Fuchs, Hannah Gladman, Rishub Jain, Yousuf A. Khan, Caroline M. R. Low, Kuba Perlin, Anna Potapenko, Pascal Savy, Sukhdeep Singh, Adrian Stecula, Ashok Thillaisundaram, Catherine Tong, Sergei Yakneen, Ellen D. Zhong, Michal Zielinski, Augustin Žídek, Victor Bapst, Pushmeet Kohli, Max Jaderberg, Demis Hassabis, and John M. Jumper. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 630(8016):493–500, 2024. [2] Sara E. Ali, Abhinav Mittal, and David H. Mathews. RNA secondary structure analysis using RNAstructure. Curr. Protoc., 3(7):e846, 2023. [3] Jeff Anderson-Lee, Eli Fisker, Vineet Kosaraju, Michelle Wu, Justin Kong, Jeehyung Lee, Minjae Lee, Mathew Zada, Adrien Treuille, and Rhiju Das. Principles for predicting RNA secondary structure design difficulty. J. Mol. Biol., 428(5, Part A):748–757, 2016. [4] Mirela Andronescu, Anthony P Fejes, Frank Hutter, Holger H Hoos, and Anne Condon. A new algorithm for RNA secondary structure design. Journal of molecular biology, 336(3):607–624, 2004. [5] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In ICLR’20: Proc. of the 8th International Conference on Learning Representations, 2020. [6] Melis Ilayda Bal, Pier Giuseppe Sessa, Mojmir Mutny, and Andreas Krause. Optimistic games for combinatorial bayesian optimization with applications to protein design. In NeurIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World, 2023. [7] Luis A Barrera, Anastasia Vedenko, Jesse V Kurland, Julia M Rogers, Stephen S Gisselbrecht, Elizabeth J Rossin, Jaie Woodard, Luca Mariani, Kian Hong Kock, Sachi Inukai, et al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 351 (6280):1450–1454, 2016. [8] Thomas Barrett, William Clements, Jakob Foerster, and Alex Lvovsky. Exploratory combinatorial optimization with reinforcement learning. In AAAI’20: Proc. of the 34th AAAI Conference on Artificial Intelligence, pages 3243–3250, 2020. [9] Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, SMC-13(5):834–846, 1983. [10] Irwan Bello, Hieu Pham, Quoc V Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. CoRR, abs/1611.09940, 2016.

_ 12/29

[11] Ritwika Bose, Irfana Saleem, and Anthony M Mustoe. Causes, functions, and therapeutic possibilities of RNA secondary structure ensembles and alternative states. Cell chemical biology, 31(1):17–35, 2024. [12] Anke Busch and Rolf Backofen. INFO-RNA: a fast approach to inverse RNA folding. Bioinformatics, 22(15):1823–1831, 2006. [13] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning–based protein sequence design using proteinmpnn. Science, 378(6615):49–56, 2022. [14] Ivan Dotu, Gloria Lozano, Peter Clote, and Encarnacion Martinez-Salas. Using RNA inverse folding to identify ires-like structural subdomains. RNA Biology, 10(12):1842–1852, 2013. [15] Peter B. Dykstra, Matias Kaplan, and Christina D. Smolke. Engineering synthetic RNA devices for cell control. Nat. Rev. Genet., 23(4):215–228, 2022. [16] Peter Eastman, Jade Shi, Bharath Ramsundar, and Vijay S Pande. Solving the RNA design problem with reinforcement learning. PLoS computational biology, 14(6):e1006176, 2018. [17] Ali Esmaili-Taheri and Mohammad Ganjtabesh. ERD: a fast and reliable tool for RNA design including constraints. BMC Bioinformatics, 16(1):20, 2015. [18] Ali Esmaili-Taheri, Mohammad Ganjtabesh, and Morteza Mohammad-Noori. Evolutionary solution for the RNA design problem. Bioinformatics, 30(9):1250–1258, 2014. [19] Sebastian M Fica, Nicole Tuttle, Thaddeus Novak, Nan-Sheng Li, Jun Lu, Prakash Koodathingal, Qing Dai, Jonathan P Staley, and Joseph A Piccirilli. RNA catalyses nuclear pre-mRNA splicing. Nature, 503(7475):229–234, 2013. [20] Zhangyang Gao, Cheng Tan, Yijie Zhang, Xingran Chen, Lirong Wu, and Stan Z Li. ProteinInvBench: Benchmarking protein inverse folding on diverse tasks, models, and metrics. In NeurIPS’23: Proc. of the 37th Annual Conference on Neural Information Processing Systems, 2023. [21] Juan Antonio Garcia-Martin, Peter Clote, and Ivan Dotu. RNAiFOLD: a constraint programming algorithm for RNA inverse folding and molecular design. J. Bioinform. Comput. Biol., 11(2): 1350001, 2013. [22] Juan Antonio Garcia-Martin, Peter Clote, and Ivan Dotu. RNAiFold 2.0: a web server and software to design custom and Rfam-based RNA molecules. Nucleic Acids Res., 43(W1):W513–W521, 2015. [23] Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 40(D1):D1100–D1107, 2012. [24] Ilias Georgakopoulos-Soares, Guillermo E Parada, and Martin Hemberg. Secondary structures in RNA synthesis, splicing and translation. Computational and structural biotechnology journal, 20: 2871–2884, 2022. [25] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In ICML’17: Proc. of the 34th International Conference on Machine Learning, pages 1263–1272, 2017. [26] Peixuan Guo. The emerging field of RNA nanotechnology. Nat. Nanotechnol., 5(12):833–842, 2010.

_ 13/29

[27] Shujun He, Rui Huang, Jill Townley, Rachael C Kretsch, Thomas G Karagianes, David BT Cox, Hamish Blair, Dmitry Penzar, Valeriy Vyaltsev, Elizaveta Aristova, et al. Ribonanza: Deep learning of RNA structure through dual crowdsourcing. bioRxiv, 2024. [28] Ivo L. Hofacker, Walter Fontana, Peter F. Stadler, L. Sebastian Bonhoeffer, Manfred Tacker, and Peter Schuster. Fast folding and comparison of RNA secondary structures. Monatsh. Chem., 125 (2):167–188, 1994. [29] Tianmeng Hu and Biao Luo. PA2D-MORL: Pareto ascent directional decomposition based multiobjective reinforcement learning. In AAAI’24: Proc. of the 38th AAAI Conference on Artificial Intelligence, volume 38, pages 12547–12555, 2024. [30] Tianmeng Hu, Biao Luo, Chunhua Yang, and Tingwen Huang. MO-MIX: Multi-objective multiagent cooperative decision-making with deep reinforcement learning. IEEE Trans. Pattern Anal. Mach. Intell., 45(10):12098–12112, 2023. [31] Tianmeng Hu, Yongzheng Cui, Biao Luo, and Ke Li. RIDER: 3D RNA inverse design with reinforcement learning-guided diffusion. In ICLR’26: Proc. of the 14th International Conference on Learning Representations, 2026. [32] Tianmeng Hu, Yongzheng Cui, Rui Tang, Biao Luo, and Ke Li. Beyond monotonicity: Revisiting factorization principles in multi-agent q-learning. In AAAI’26: Proc. of the 40th AAAI Conference on Artificial Intelligence, pages 21876–21884, 2026. [33] Tianmeng Hu, Biao Luo, and Ke Li. Human-in-the-loop policy optimization for preference-based multi-objective reinforcement learning. In ICML’26: Proc. of the 43rd International Conference on Machine Learning, 2026. [34] Kexin Huang, Payal Chandak, Qianwen Wang, Shreyas Havaldar, Akhil Vaid, Jure Leskovec, Girish N Nadkarni, Benjamin S Glicksberg, Nils Gehlenborg, and Marinka Zitnik. A foundation model for clinician-centered drug repurposing. Nature Medicine, 30(12):3601–3613, 2024. [35] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. In NeurIPS’19: Proc. of the 33rd Annual Conference on Neural Information Processing Systems, 2019. [36] Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In ICML’19: Proc. of the 36th International Conference on Machine Learning, pages 2961–2970, 2019. [37] Yuan Jiang, Zhiguang Cao, and Jie Zhang. Learning to solve 3-d bin packing problem via deep reinforcement learning and constraint programming. IEEE transactions on cybernetics, 53(5): 2864–2875, 2023. [38] Ioanna Kalvari, Eric P Nawrocki, Nancy Ontiveros-Palacios, Joanna Argasinska, Kevin Lamkiewicz, Manja Marz, Sam Griffiths-Jones, Claire Toffano-Nioche, Daniel Gautheret, Zasha Weinberg, et al. Rfam 14: expanded coverage of metagenomic, viral and microRNA families. Nucleic Acids Research, 49(D1):D192–D200, 2021. [39] Peter Kerpedjiev, Stefan Hammer, and Ivo L Hofacker. Forna (force-directed RNA): Simple and effective online RNA secondary structure diagrams. Bioinformatics, 31(20):3377–3379, 2015. [40] Robert Kleinkauf, Martin Mann, and Rolf Backofen. antaRNA: ant colony-based RNA sequence design. Bioinformatics, 31(19):3114–3121, 2015. [41] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In NIPS’99: Proc. of the 12th Annual Conference on Neural Information Processing Systems, pages 1008–1014, 1999.

_ 14/29

[42] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In NeurIPS’16: Proc. of the Annual Conference on Neural Information Processing Systems, pages 3675–3683, 2016. [43] Diana Kwon. RNA function follows form – why is it so hard to predict? Nature, 639(8056): 1106–1108, 2025. [44] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130, 2023. [45] Ronny Lorenz, Stephan H Bernhart, Christian Höner zu Siederdissen, Hakim Tafer, Christoph Flamm, Peter F Stadler, and Ivo L Hofacker. ViennaRNA package 2.0. Algorithms for Molecular Biology, 6(1):26, 2011. [46] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In NeurIPS’17: Proc. of the 31st Annual Conference on Neural Information Processing Systems, pages 6382–6393, 2017. [47] Isaac D Lutz, Shunzhi Wang, Christoffer Norn, Alexis Courbet, Andrew J Borst, Yan Ting Zhao, Annie Dosey, Longxing Cao, Jinwei Xu, Elizabeth M Leaf, et al. Top-down design of protein architectures with reinforcement learning. Science, 380(6642):266–273, 2023. [48] Nicholas R. Markham and Michael Zuker. UNAFold: software for nucleic acid folding and hybridization. Methods Mol. Biol., 453:3–31, 2008. [49] David H. Mathews, Michael Zuker, and Douglas H. Turner. Expanded sequence dependence of thermodynamic parameters improves prediction of RNA secondary structure. J. Mol. Biol., 288 (5):911–940, 1999. [50] David H. Mathews, Matthew D. Disney, Jessica L. Childs, Susan J. Schroeder, Michael Zuker, and Douglas H. Turner. Incorporating chemical modification constraints into a dynamic programming algorithm for prediction of RNA secondary structure. Proc. Natl. Acad. Sci. USA, 101(19): 7287–7292, 2004. [51] Marco C Matthies, Stefan Bienert, and Andrew E Torda. Dynamics in sequence space for RNA secondary structure design. Journal of chemical theory and computation, 8(10):3663–3670, 2012. [52] Edoardo Morandi, Martijn J van Hemert, and Danny Incarnato. SHAPE-guided RNA structure homology search and motif discovery. Nature Communications, 13(1):1722, 2022. [53] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate Q-value functions for decentralized POMDPs. Journal of Artificial Intelligence Research, 32:289–353, 2008. [54] Sharat Patil, Frederic Runge, Jörg KH Franke, and Frank Hutter. Towards generative RNA design with tertiary interactions. In GEM: ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design, 2024. [55] Fernando Portela. An unexpectedly effective monte carlo technique for the rna inverse folding problem. BioRxiv, page 345587, 2018. [56] V Ramakrishnan. The ribosome emerges from a black box. Cell, 159(5):979–984, 2014. [57] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML’18: Proc. of the 35th International Conference on Machine Learning, pages 4295–4304, 2018.

_ 15/29

[58] Silvi Rouskin, Meghan Zubradt, Stefan Washietl, Manolis Kellis, and Jonathan S Weissman. Genome-wide probing of RNA structure reveals active unfolding of mRNA structures in vivo. Nature, 505(7485):701–705, 2014. [59] Frederic Runge, Danny Stoll, Stefan Falkner, and Frank Hutter. Learning to design RNA. In ICLR’19: Proc. of the 7th International Conference on Learning Representations, 2019. [60] Frederic Runge, Karim Farid, Jorg KH Franke, and Frank Hutter. RnaBench: A comprehensive library for in silico RNA modelling. bioRxiv, pages 1–24, 2024. [61] Frederic Runge, Jörg Franke, Daniel Fertmann, Rolf Backofen, and Frank Hutter. Partial RNA design. Bioinformatics, 40(Supplement_1):i437–i445, 2024. [62] Paul J Sample, Ban Wang, David W Reid, Vlad Presnyak, Iain J McFadyen, David R Morris, and Georg Seelig. Human 5’ UTR design and variant effect prediction from a massively parallel translation assay. Nature biotechnology, 37(7):803–809, 2019. [63] Kengo Sato, Yuki Kato, Michiaki Hamada, Tatsuya Akutsu, and Kiyoshi Asai. Ipknot: fast and accurate prediction of rna secondary structures with pseudoknots using integer programming. Bioinformatics, 27(13):i85–i93, 2011. [64] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. CoRR, abs/1506.02438, 2015. [65] Alexander Serganov and Evgeny Nudler. A decade of riboswitches. Cell, 152(1–2):17–24, 2013. [66] Jithendaraa Subramanian, Shivakanth Sujit, Niloy Irtisam, Umong Sain, Derek Nowrouzezahrai, Samira Ebrahimi Kahou, and Riashat Islam. Reinforcement learning for sequence design leveraging protein language models. CoRR, abs/2407.03154, 2024. [67] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS’99: Proc. of the 12th Annual Conference on Neural Information Processing Systems, pages 1057–1063, 1999. [68] Ryan J. Taft, Ken C. Pang, Timothy R. Mercer, Marcel Dinger, and John S. Mattick. Non-coding RNAs: regulators of disease. J. Pathol., 220(2):126–139, 2010. [69] Akito Taneda. Multi-objective genetic algorithm for pseudoknotted RNA sequence design. Front. Genet., 3:36, 2012. [70] Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. In ICML’22: Proc. of the 39th International Conference on Machine Learning, pages 21658–21676, 2022. [71] Lee E Vandivier, Stephen J Anderson, Shawn W Foley, and Brian D Gregory. The conservation and function of RNA secondary structure in plants. Annual review of plant biology, 67(1):463–488, 2016. [72] Yanan Wang, Yong Ge, Zhepeng Li, Li Li, and Rui Chen. M3 Rec: A Context-Aware Offline Meta-Level Model-Based Reinforcement Learning Approach for Cold-Start Recommendation. ACM Transactions on Information Systems, 42(6):1–27, 2024. [73] Yi Wang, Hui Tang, Lichao Huang, Lulu Pan, Lixiang Yang, Huanming Yang, Feng Mu, and Meng Yang. Self-play reinforcement learning guides protein engineering. Nature Machine Intelligence, 5 (8):845–860, 2023. [74] David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1):67–82, 1997. _ 16/29

[75] Heng Yang, Renzhi Chen, and Ke Li. Bridging sequence-structure alignment in RNA foundation models. In AAAI’25: Proc. of the 39th AAAI Conference on Artificial Intelligence, pages 21929– 21937, 2025. [76] Xiufeng Yang, Kazuki Yoshizoe, Akito Taneda, and Koji Tsuda. RNA inverse folding using Monte Carlo tree search. BMC bioinformatics, 18(1):468, 2017. [77] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In NeurIPS’22: Proc. of the 36th Annual Conference on Neural Information Processing Systems, pages 24611–24624, 2022. [78] Tianshuo Zhou, Ning Dai, Sizhen Li, Max Ward, David H Mathews, and Liang Huang. RNA design via structure-aware multifrontier ensemble optimization. Bioinformatics, 39(Supplement_1): i563–i571, 2023.

_ 17/29

A

Pseudocode for CocoRNA

Algorithm 1 presents the pseudocode for the proposed CocoRNA method. CocoRNA leverages the MAPPO algorithm, an extension of the PPO method tailored for multi-agent environments. In our implementation of CocoRNA, each agent is equipped with an Actor network responsible for selecting actions based on local observations, while the centralized Critic network evaluates the global state to provide value estimates for policy updates. Algorithm 1 CocoRNA: Cooperative Multi-Agent RNA Design 1: Input: Target RNA secondary structure dataset X, maximum episodes E, maximum steps per episode T , horizon H, number of agents n 2: Initialize Actor networks πθi and Critic network Vϕ for each agent i 3: Initialize experience replay buffer D 4: for episode = 1 to E do 5: Reset environment, randomly select a target RNA secondary structure xt from X, and initialize RNA sequence sseq 6: Decompose the RNA design task among n agents according to the chosen decomposition scheme 7: for step t = 1 to T do 8: for each agent i = 1 to n in parallel do 9: Observe local observation oit 10: Obtain validity mask Mti from the environment based on assigned paired partners 11: Select masked action ait ∼ πθi (· | oit , Mti ) 12: end for 13: Execute joint action at = (a1t , a2t , . . . , ant ) in the environment’s fixed update order 14: Update RNA sequence sseq with masked valid actions at 15: Predict folded structure xf using MFE folding algorithm 16: Calculate Hamming distance Ht = H(xf , xt ) 17: Calculate reward rt based on Ht and Ht−1 18: Store transition (ot , at , rt , ot+1 ) in buffer D 19: if using SAE then 20: Perform local search to find better actions a′t 21: Update reward rt′ based on improved Hamming distance Ht′ 22: Replace transition with (ot , a′t , rt′ , ot+1 ) in buffer D 23: end if 24: if Ht = 0 then 25: Assign final reward C to all agents 26: Break 27: end if 28: if buffer size reaches horizon H then 29: Update Actor networks πθi and Critic network Vϕ using mini-batch updates from buffer D with MAPPO 30: Clear experience buffer D 31: end if 32: end for 33: end for

B

Theoretical Analysis

B.1

Convergence Analysis

In this section, we provide a convergence analysis of the proposed algorithm. We demonstrate that, under certain conditions, the multi-agent actor-critic method used in CocoRNA converges to a local maximum of the expected joint return. _ 18/29

In CocoRNA, we employ a centralized Critic network to evaluate the joint policy. Specifically, we use the joint advantage function A(s, a), defined as: A(s, a) = Q(s, a) − V (s),

(8)

where Q(s, a) denotes the joint action-value function and V (s) is the state-value function. The joint policy gradient is then given by: #

"

g = Eπ

X

∇θ log π i (ai | oi ) A(s, a)

(9)

i

"

= Eπ

# X

∇θ log π (a | o ) Q(s, a) − Eπ i

i

i

" X

i

#

∇θ log π (a | o ) V (s) , i

i

(10)

i

i

where π denotes the joint policy, and θ represents the parameters of the Actor networks. The second term of (10) can be expanded as: gV = −Eπ =−

" X

X

#

∇θ log π i (ai | oi ) V (s)

i π

d (s)

XX

s

π(a−i | o−i )

a−i

i

(11)

X

∇θ π i (ai | oi ) V (s),

(12)

ai

where dπ (s) denotes the discounted ergodic state distribution [67], a−i and o−i represent the joint actions and observations of all agents except agent i. Due to the normalization property of the policy: X

π i (ai | oi ) = 1,

(13)

ai

we have:

X

∇θ π i (ai | oi ) = ∇θ

ai

π i (ai | oi ) = ∇θ 1 = 0.

(14)

π(a−i | o−i ) V (s) ∇θ 1

(15)

X ai

Therefore, gV = −

X

dπ (s)

XX

s

i a−i

= 0.

(16)

Thus, the policy gradient in (10) reduces to only the first term: " X

g = Eπ

#

∇θ log π (a | o ) Q(s, a) . i

i

(17)

i

i

Since the joint policy can be expressed as the product of individual agent policies: π(a | s) =

π i (ai | oi ),

Y

(18)

i

and using the logarithmic identity log

i xi =

Q

P

i log xi , we can rewrite the policy gradient as:

#

"

g = Eπ

X

∇θ log π (a | o ) Q(s, a) i

i

i

(19)

i

#

"

= Eπ ∇θ log

Y

π (a | o ) Q(s, a) i

i

i

(20)

i

= Eπ [∇θ log π(a | s) Q(s, a)] .

(21)

This result shows that (21) is equivalent to the single-agent actor-critic policy gradient, where the multiple agents are considered as a single joint agent (or super-agent). Konda and Tsitsiklis [41] proved the convergence of policy gradient algorithms under the condition that the policy π is differentiable and the algorithm parameters satisfy certain conditions. Therefore, by following the gradient in (9), the multi-agent actor-critic algorithm converges to a local maximum of the expected joint return J π . _ 19/29

B.2

Limitation and Discussion

In theory, gradient-based deep learning and reinforcement learning methods generally guarantee convergence to local optima rather than global optima due to the non-convex nature of the optimization landscape. However, in practice, the ability of reinforcement learning algorithms to go beyond local optima and approach global optima depends significantly on how well they balance exploration and exploitation. Specifically, CocoRNA is built upon PPO, which includes an entropy bonus in the objective function, which explicitly encourages the policy to maintain high entropy. This prevents the policy from becoming too deterministic too quickly, avoiding premature convergence to suboptimal policies and promoting continued exploration. In the context of the RNA design task, the multi-agent framework of CocoRNA further aids in overcoming local optima. Unlike single-agent methods that mutate one nucleotide at one step, our method allows multiple agents to act simultaneously on different parts of the RNA sequence. This introduces greater diversity and increases the exploration of the state space, thereby increasing the opportunities to discover better policies.

C

Additional Results

In this section, we present a series of experiments under various settings to comprehensively demonstrate the effectiveness of CocoRNA.

C.1

Dataset Splitting

In the main experiments, we cluster the dataset based on structural similarity and randomly select one cluster out of 100 as the test set. To more rigorously rule out potential data leakage, we further construct six different train-test splits by randomly selecting six distinct clusters as test sets. Independent training and evaluation are then conducted on each split. Additionally, we evaluate two of the strongest baselines on these test sets. The results are reported in Table 3. The results show that CocoRNA achieves an average success rate of 97.44% across these new train-test splits, which is highly consistent with the results reported in the main paper. These findings further demonstrate the strong generalization ability of CocoRNA. Table 3: Performance of CocoRNA, Meta-LEARNA-Adapt-600s, and SAMFEO-600s across six different train-test splits. The mean and standard deviation are reported. Method

Number of solved samples

Mean success rate

CocoRNA

633.33 ± 5.47

97.44%

Meta-LEARNA-Adapt-600s

578.17 ± 9.60

88.95%

SAMFEO-600s

568.16 ± 6.09

87.41%

Table 4: Performance of CocoRNA with different problem decomposition schemes on Rfam dataset across six different train-test splits. The mean and standard deviation over are reported. Method

Number of solved samples

Mean success rate

CocoRNA-PBD

633.33 ± 5.47

97.44%

CocoRNA-SBD

628.50 ± 4.99

96.69%

CocoRNA-random

596.33 ± 6.70

91.74%

_ 20/29

Table 5: Performance of CocoRNA and ablated versions on Rfam dataset. The mean and standard deviation over five independent runs are reported.

C.2

Method

Number of solved samples

Mean success rate

CocoRNA

635.40 ± 2.58

97.75%

CocoRNA-ablated

599.60 ± 7.64

92.25%

Single-agent version

340.80 ± 5.63

52.43%

Using Different Decomposition Schemes

Table 4 provides the testing results for different problem decomposition schemes used in CocoRNA. We employ three decomposition methods: Position-based decomposition (CocoRNA-PBD), Structuretype-based decomposition (CocoRNA-SBD) and random decomposition(CocoRNA-random). For Position-based decomposition, the complete RNA structure is divided into approximately equal segments, with each segment assigned to a different agent. For Structure-type-based decomposition, agents are assigned specific structural types based on the dot-bracket notation; for example, if Agent 1 is responsible for non-paired structures, it sequentially designs nucleotides at all non-paired positions within the structure. In our experiments, CocoRNA-PBD and CocoRNA-SBD only show minor performance differences, demonstrating the flexibility of our method concerning decomposition choices. For CocoRNASBD, structure-type-based decomposition allows agents to focus more on specific structural elements, but the relative positional relationships between different agents may become completely disrupted as the design process progresses, potentially causing slight adverse effects on the learning process. Although CocoRNA-SBD performed slightly worse than CocoRNA-PBD in testing, both methods significantly outperformed the single-agent approach. CocoRNA-random is worse than the other two decomposition methods. We hypothesize that this is because randomly assigning design positions is entirely unstructured. This increases the difficulty for agents to cooperate and interferes with the learning process. However, the performance of random decomposition does not completely collapse, suggesting that the MARL mechanism itself is robust.

C.3

Ablation Study on SAE

Table 5 presents the testing results of two ablated versions of CocoRNA. Removing the SAE method results in a slight performance decrease in CocoRNA-ablated, demonstrating the effectiveness of the SAE approach. Additionally, as shown in the right plot of Figure 5, CocoRNA-ablated continues to improve towards the end of the training process, suggesting that further training may yield better performance. The single-agent version shows significantly poorer performance. In fact, the single-agent version is similar to the LEARNA method but lacks architectural and hyperparameter optimizations and does not undergo additional training during the design phase. Table 6: Performance of CocoRNA and CocoRNA-ablated under different reward settings on the Rfam dataset. Reward Setting / Method

Number of solved samples

Success rate

Delayed Reward / CocoRNA

634/650

97.54%

Delayed Reward / CocoRNA-ablated

621/650

95.54%

Terminal Reward / CocoRNA

604/650

92.92%

Terminal Reward / CocoRNA-ablated

570/650

87.69%

_ 21/29

1.000

0.975

0.975

0.950

0.950

0.925

0.925

0.900

0.900

SRR

SRR

1.000

0.875 0.850

0.850

Setting

Original Rewards Delayed Rewards Delayed Rewards (No SAER)

0.825 0.800 0.0

0.5

1.0

Step

1.5

0.875

2.0

Setting

0.825

Terminal Rewards Terminal Rewards (No SAER)

0.800

2.5 1e7

0.0

0.5

1.0

Step

1.5

2.0

2.5 1e7

Figure 6: Learning curves under different reward settings on the Rfam dataset. Each experiment is performed over 6 independent runs with different random seeds. The shaded areas represent the 95% confidence intervals. Left: Learning curves using delayed reward (22). Right: Learning curves using terminal reward (23).

C.4

Ablation Study on Reward Signals

In addition to the standard reward signal, we test two different delayed/sparse reward settings on the Rfam dataset. • Delayed Reward: The reward signal is calculated every 10 steps instead of every step. The original reward function (7) is modified as follows: Rt =

 Ht−10 −Ht  ,  l 

C,

  0,

if Ht > 0 and t mod 10 = 0, if Ht = 0 and t mod 10 = 0, otherwise.

(22)

• Terminal Reward Only: The reward is given only at the end of an episode or upon successful design. The reward function is defined as: Rt =

 Ht   1 − l ,

C,

  0,

if Ht ̸= 0 and t reaches maximum steps (episode ends), if Ht = 0, otherwise.

(23)

We also test the impact of removing the SAE method under these two reward settings. The results are shown in Figure 6 and Table 6. We denote the ablated version without SAE as CocoRNA-ablated. It can be observed that under the delayed reward setting, CocoRNA experiences almost no performance loss. When only terminal rewards are available, the algorithm’s performance decreases slightly due to the excessively sparse reward signal significantly increasing the difficulty of policy learning. Under both reward settings, CocoRNA-ablated performs worse than CocoRNA, demonstrating the effectiveness of the SAE method. Overall, this experiment shows that CocoRNA is robust to different reward signals and performs well even under harsh reward settings. Notably, the delayed reward setting reduces the number of calls to the RNA folding algorithm to one-tenth of the original, significantly speeding up training.

C.5

Effect of Varying the Number of Agents

In the main results presented earlier, we set the number of agents to n = 4. In this section, we adjust the parameter n to examine the impact of different numbers of agents on the algorithm’s performance. We experiment with four different settings: using 2, 4, 6, and 8 agents. Table 7 shows the performance of CocoRNA under these different agent sizes on the Rfam dataset. The corresponding learning curves are presented in Figure 7. _ 22/29

1.00 0.95

SRR

0.90 0.85

Setting

0.80

2 agents 4 agents 6 agents 8 agents

0.75 0.0

0.5

1.0

Step

1.5

2.0

2.5 1e7

Figure 7: Learning curves of CocoRNA under different agent size settings on the Rfam dataset. Each experiment is performed over 6 independent runs with different random seeds. The shaded areas represent the 95% confidence intervals. Table 7: Performance of CocoRNA under different agent size settings on the Rfam dataset. Number of Agents (n)

Number of solved samples

Success rate

n=2

601/650

92.46%

n=4

636/650

97.85%

n=6

629/650

96.77%

n=8

615/650

94.62%

From the results, we observe that using 4 or 6 agents achieves the best performance. When the number of agents is too small or too large, the performance tends to decrease. With too few agents, the benefits of multi-agent problem decomposition cannot be fully exploited, as each agent handles a larger portion of the task, leading to higher complexity per agent and potentially less efficient learning. On the other hand, with too many agents, the environment becomes more non-stationary from the perspective of each agent due to the increased interactions and dependencies among agents, making cooperation more challenging. Table 8: Performance of CocoRNA with independent and shared policies on the Rfam dataset. Method

C.6

Number of solved samples

Success rate

CocoRNA-PBD

636/650

97.85%

CocoRNA-PBD-shared

622/650

95.69%

CocoRNA-SBD

629/650

96.77%

CocoRNA-SBD-shared

621/650

95.54%

Effect of Sharing Policy Parameters

We investigate the performance of CocoRNA when agents share policy parameters under both decomposition methods. In these experiments, although different agents receive different observations, all agents share the same set of policy parameters. This approach is referred to as CocoRNAPBD-shared and CocoRNA-SBD-shared for position-based decomposition and structure-type-based decomposition, respectively. The learning curves and test results are shown in Figure 8 and Table 8, respectively. From the results, we observe that enforcing shared policy parameters among multiple agents leads to a slight _ 23/29

1.00 0.95

SRR

0.90 0.85 0.80

Setting

Individual policies Shared policy

0.75 0.0

0.5

1.0

Step

1.5

2.0

2.5 1e7

Figure 8: Learning curves of CocoRNA-PBD and CocoRNA-PBD-shared on the Rfam dataset. Each experiment is performed over 6 independent runs with different random seeds. The shaded areas represent the 95% confidence intervals. decrease in algorithm performance compared to using independent policies, although the gap is relatively small. These results potentially suggest that different agents learn distinct local policies that are specialized for their assigned sub-tasks. When agents share policy parameters, the policy learning process is constrained because all agents must use the same policy function, despite facing different sub-tasks or local environments. This can limit the agents’ ability to adapt their policies to their specific roles in the RNA design task. In contrast, when using independent policies, each agent can tailor its policy parameters to its particular sub-task without interference from the data of other agents, providing more flexibility in policy learning. 1.00 0.95

SRR

0.90 0.85 0.80 0.75

Setting

0.70 0.65

learning rate = 1e-3 learning rate = 1e-4 0.0

0.5

1.0

Step

1.5

2.0

2.5 1e7

Figure 9: Learning curves of CocoRNA using different learning rates. Each experiment is performed over 6 independent runs with different random seeds. The shaded areas represent the 95% confidence intervals.

C.7

Training Parameters

We observe that the algorithm is relatively sensitive to the learning rate compared to other hyperparameters. In our main experiments, we use a learning rate of 1 × 10−5 , which is lower than the settings commonly used in RL algorithms. Figure 9 shows the learning curves when using higher learning rates. It can be seen that excessively high learning rates lead to unstable training and poor convergence. In multi-agent environments, a high learning rate can exacerbate the non-stationarity of the environment. This issue arises because each agent’s policy update affects the environment dynamics perceived by other agents. With higher learning rates, these changes become more abrupt, making _ 24/29

Cumulative Solved Instances

600 500 400 300 200 100 0

0

5

Solved count Median time (1.47s) 10 15 20 Solve Time (seconds)

Figure 10: Number of solved samples as a function of the allotted runtime. Table 9: Evaluation of CocoRNA and the single-agent ablated version on biological sequence design tasks. Results represent the average of 8 independent runs, with standard deviations also reported.

CocoRNA Single-agent version

TF Bind 8 1.000 ± 0.000 0.901 ± 0.093

TF Bind 10 1.000 ± 0.000 0.725 ± 0.140

ChEMBL 0.811 ± 0.001 0.595 ± 0.166

UTR 0.810 ± 0.001 0.690 ± 0.010

it difficult for agents to adapt and learn stable policies. Consequently, the training process becomes highly unstable. For other important hyperparameters, such as the discount factor γ, the GAE parameter λ, and the PPO clip range ϵ, we use standard values commonly adopted in the literature. These settings are detailed in Appendix F.

C.8

Other Results

Figure 10 shows how the number of solved samples increases as a function of the allotted runtime. The median solving time of CocoRNA is only 1.47 s. Table 10 summarizes the number of redesign attempts (iterations) ni required to design different RNA structures on the Rfam dataset using CocoRNA, LEARNA-600s, and Meta-LEARNA-Adapt600s. For the majority of sequences (95.54%), our method succeeds within a single episode. In contrast, the two baseline methods require numerous iterative optimizations. Although Meta-LEARNA-Adapt performs meta-learning on the entire training set, only 3.38% of sequences can be solved in a single attempt.

D

Other biological sequence design tasks

Beyond RNA inverse design, we further apply CocoRNA to four biological sequence design tasks. These tasks are formulated as RL environments based on DesignBench [70]. Specifically, they include the DNA functional optimization tasks TF Bind 8, TF Bind 10 [7], and UTR [62], as well as the drug molecule design task ChEMBL [23]. Apart from minor adjustments to the state space, action space, and hyperparameters such as the learning rate, CocoRNA requires no additional modifications. Results Given that the algorithms in Section 4.1 are specifically tailored for RNA design, we can hardly adapt them to other tasks without nearly redesigning a new algorithm from scratch. Therefore, we compare CocoRNA with its single-agent ablated variant to further validate the effectiveness of the multi-agent framework. The results, presented in Table 9, clearly show that the MARL framework of CocoRNA consistently outperforms its single-agent counterpart across all tasks. Moreover, CocoRNA successfully reaches the global optimum in TF Bind 8 and TF Bind 10. Considering that CocoRNA was not originally designed for these tasks, these results further demonstrate its adaptability and effectiveness across diverse problem settings.

_ 25/29

1500 1250 1000 750 500 250 0

Method

ni = 1

1 < ni ≤ 15

ni > 15

CocoRNA LEARNA Meta-LEARNA

621 1 22

29 16 137

0 633 491

Rfam Training Set

Rfam Test Set 20 Frequency

Frequency

Table 10: Distribution of RNA sequences by redesign attempts ni intervals for different methods on the Rfam dataset.

15 10 5

100

200 300 Length

400

0

100

200 300 Length

400

Figure 11: Length distributions of RNA secondary structures in different datasets.

E

Datasets Details

We use the Rfam dataset [60] to train and evaluate our algorithm. The dataset contains only RNA secondary structure information without explicit sequence labels, as the task is to design sequences that fold into the provided structures. The dataset is constructed by applying RNA folding algorithms to RNA sequences from the Rfam database [38]. The structures in this dataset are computationally predicted using tools like the ViennaRNA package [45], which produces secondary structures based on minimum free energy (MFE) folding. Figure 11 shows the length distributions of RNA secondary structures in the Rfam training set and test set. Table 11 provides statistical information about the RNA structures in these datasets, including the sample size, average length, maximum length, and minimum length of the RNA secondary structures.

F

Algorithm Details and Hyperparameters

This section provides a detailed description of the CocoRNA algorithm, including network architectures, training details, and hyperparameter settings.

F.1

Network Architectures

Each agent in the CocoRNA framework is equipped with its own Actor and Critic networks. Although the Critic network leverages shared global information, the network parameters are not shared across agents. The Actor network takes local observations as input and outputs a discrete action (the nucleotide to be placed in the RNA sequence). The Critic network, on the other hand, processes the global state, which includes the full RNA sequence and secondary structure, and outputs a scalar value representing the estimated cumulative future reward for the entire system. • Actor Network: The Actor network consists of several fully connected layers followed by a softmax output layer. Each agent’s observation is passed through the network to compute a probability distribution over the action space {A, U, G, C}. _ 26/29

Table 11: Statistics of the datasets. Dataset

Sample Size

Average Length

Max Length

Min Length

Rfam (Training Set)

60000

243.73

450

50

Rfam (Test Set)

650

238.78

450

50

Table 12: Training hyperparameters. Hyperparameter

Value

Learning Rate

0.00001

Discount Factor (γ)

0.99

GAE Parameter (λ)

0.95

PPO Clip Range (ϵ)

0.2

Entropy Coefficient

0.02

Horizon Length

256

Batch Size

512

Number of Epochs

10

Gradient Clipping

0.5

• Critic Network: The Critic network takes the entire RNA sequence and secondary structure as input. This input is processed through two separate CNN modules: one for the sequence and one for the structure. The outputs of these modules are concatenated and passed through a fully connected network to produce a value estimate for the entire system.

F.2

Training and Optimization

The training procedure follows on-policy reinforcement learning paradigm: experiences are collected over a fixed horizon, stored in a replay buffer, and subsequently used to update the Actor and Critic networks. We adopt PPO to optimize the policy by maximizing the clipped surrogate objective, which stabilizes training, and employ Generalized Advantage Estimation (GAE) to compute advantages with reduced variance. Training on a single NVIDIA 4090 GPU requires approximately 8 to 10 hours. As shown in Appendix C.4, this can be reduced to 3 to 4 hours under a delayed reward formulation with negligible performance degradation. Inference is highly efficient, averaging only 3.24 seconds per successful design.

Table 13: Network hyperparameters. Hyperparameter

Value

Actor Network Layers

[128, 64]

Critic Network Layers

[256, 64]

CNN Filter Size

[8, 4, 3]

CNN Number of Filters

[32, 64, 64]

Activation Function

ReLU

Optimizer

Adam _ 27/29

Table 14: Additional hyperparameters. Parameter

Value

Maximum Steps per Episode (Training)

350

Maximum Steps per Episode (Testing)

400

Observation Radius

30

Local Observation Window Size

61

Global State Dimension

900

Total Training Steps

F.3

2.5 × 107

Hyperparameter Settings

Table 12 and Table 13 summarize the key hyperparameters used in training the CocoRNA model, including training parameters and network parameters.

F.4

Other Settings

Table 14 presents additional parameter settings used in the CocoRNA framework. During training, the maximum number of steps per episode is set to 350, while for testing, it is increased to 400. The training is conducted for a total of 25 million steps. Each agent has an observation radius of 30, resulting in a local observation window size of 61. Since the maximum RNA structure length in the dataset is 450, we standardize the global state dimension to 900 (450 for the sequence information and 450 for the structure information). We use the ViennaRNA package [45] version 2.6.4 to compute the folded RNA structures during the design process.

G

Discussion

G.1

Graph Neural Networks

Many biological molecules are well-suited to be modeled as graphs due to their inherent structural properties. Transforming RNA structures into graph representations and utilizing Graph Neural Networks (GNNs) [25] could potentially better capture the relationships between different nucleotides, enabling more effective policy learning. Graph-based representations naturally model the interactions and dependencies within RNA structures, which could enhance the agent’s decision-making process. However, employing more complex network architectures like GNNs also introduces higher computational and memory overheads, potentially reducing training efficiency. In our work, although we did not use GNNs, we addressed the issue of partial observations by employing CNNs within the centralized Critic to extract global structural information. This approach helps mitigate the limitations of local observations by providing a holistic view of the RNA structure, thereby supporting coordinated policy optimization without incurring excessive computational costs. Nevertheless, integrating GNNs with reinforcement learning-based methodologies represents a promising direction for future research.

G.2

Limitations and Future Work

While CocoRNA demonstrates promising results in RNA secondary structure design, there are several limitations. First, it is non-trivial to design an accurate and reliable reward model given the complexity of biological systems. We may resort to large-scale pretrained models [1]. However, these models are often too large and thus are computationally expensive. One potential solution is to go with a model-based RL, where an explicit model of the environment is learned and used to predict future states and rewards more efficiently. Another future direction is to extend the framework to multi-objective

_ 28/29

settings [29, 30, 33], jointly considering structural fidelity and additional biological objectives such as free energy [45] and GC content. Another potential limitation is the decomposition method considered in the current CocoRNA. Given the high-dimensional nucleotide design space, it will be more promising to study an adaptive decomposition mechanism. This might be achieved by designing hierarchical policies [42], where high-level agents make decisions about task decomposition and low-level agents focus on specific sub-tasks. In the current framework, all agents share a global reward defined by the structural quality of the full RNA sequence, and credit assignment is handled implicitly through centralized value estimation rather than explicit per-agent rewards. As the number of interacting agents increases, the environment becomes increasingly non-stationary from the perspective of each individual agent, making coordination and credit assignment more challenging. This effect is reflected in our ablation studies, where performance degrades moderately when the number of agents becomes too large. Developing more scalable coordination and credit-assignment mechanisms for large multi-agent biological design problems remains an important direction for future work.

_ 29/29

Related documents

• MORAD: Reinforcement Learning with Multi-Objective Rewards for RNA Inverse Design#1297657
• Building RNA coarse-grained force fields: Design principles and training strategies.#380836
• miRNAProtPred: computational prediction of human miRNA binding based on seed complementarity and thermodynamic stability.#351485
• Discrete Compositional Generation via General Soft Operators and Robust Reinforcement Learning#799865
• Counteracting host reactions to synthetic reprogramming in animal forms of artificial life#691057