1
Federated Foundation Models over Vehicular Networks
arXiv:2606.06786v1 [cs.LG] 5 Jun 2026
Kasra Borazjani, Student Member, IEEE, Fardis Nadimi, Student Member, IEEE, Payam Abdisarabshali, Student Member, IEEE, Owen Palinski, Student Member, IEEE, Allan Salihovic, Dinh Nguyen, Senior Member, IEEE, Minghui Liwang, Senior Member, IEEE, and Seyyedali Hosseinalipour, Senior Member, IEEE
centralized training/fine-tuning of M3T FMs, which is fundamentally misaligned with the data acquisition realities of vehicular networks. In particular, data in vehicular networks is often geo-distributed across vehicles, roadside units (RSUs), and edge/cloud infrastructure, and centralizing these data for training/fine-tuning M3T FMs may raise privacy concerns (by exposing sensitive sensing, location, or behavioral information to untrusted parties) and impose substantial communication overhead due to the transmission of raw multi-modal data. Such challenges motivate a shift toward the use of distributed learning principles, with the most prominent example being federated learning (FL) [3], for the training/fine-tuning of M3T FMs over vehicular networks. This shift has recently given rise to a new and rapidly emerging research direction in ML, called multi-modal multi-task federated foundation models (M3T FedFMs) [4], which enables collaborative training/finetuning of M3T FMs across decentralized data sources while preserving data locality. However, despite attracting attention in the ML community, the application of M3T FedFMs in I. I NTRODUCTION vehicular networks remains largely unexplored. The field of machine learning (ML) has recently witnessed a Motivated by this gap, in this work, we present a vision major transformation with the advent of large language models for integrating M3T FedFMs into vehicular networks, with (LLMs), which have demonstrated remarkable capabilities in our contributions summarized as follows: (i) We articulate understanding, generation, and reasoning of natural language the modular architecture of M3T FedFMs, positioning them across a wide range of tasks (e.g., text summarization, trans- as a promising yet underexplored paradigm for vehicular lation, and semantic analysis) [1]. While LLMs (e.g., GPT 2, intelligence. (ii) We conduct a case study to demonstrate and Gemini 1) may be viewed as the most prominent recent the potential of M3T FedFMs for vehicular networks, and breakthrough, ML is simultaneously undergoing a broader and publicly release the corresponding source code to establish a arguably more structural shift driven by the emergence of multi- benchmark for future evaluations in this underexplored area. modal multi-task foundation models (M3T FMs). In particular, (iii) We identify the unique characteristics of vehicular networks M3T FMs (e.g., GPT 5, and Gemini 3) extend the text-centric that impact the design and performance of M3T FedFMs, capabilities of LLMs to jointly process, fuse, and reason over and envision how future M3T FedFM frameworks should heterogeneous data modalities (e.g., vision, audio, and text), accommodate these characteristics through a set of forwardwhile supporting the concurrent learning of multiple tasks (e.g., looking research directions. image generation, classification, segmentation, and captioning) within a unified model architecture. II. R ELATED W ORK AND BACKGROUND Concurrent with these major innovations in ML, vehicular networks are undergoing a paradigm shift driven by the deploy- A. FL in Vehicular Networks ment of edge-native ML applications. In particular, vehicles are Conventional FL proceeds iteratively through three main increasingly envisioned as mobile edge nodes equipped with steps until model convergence: (i) FL devices (e.g., vehicles) multi-modal sensors, including light detection and ranging independently train local models using their private data; (ii) (LiDAR), millimeter-wave radar, wide-angle cameras, and they periodically transmit model updates (e.g., parameters global navigation satellite systems (GNSS), and expected to or gradients) to a coordinating server; and (iii) the server execute multiple downstream tasks, such as 3D object detection, aggregates the received updates (e.g., via weighted averaging) trajectory prediction, and collaborative planning/navigation [2]. to construct a global model, which is then broadcast back to the This evolution presents a unique opportunity to integrate M3T clients to synchronize their local models and initiate the next FMs within the vehicular networking ecosystem. Despite its model training round. Owing to its privacy-preserving nature strong promise, realizing such an integration in practice remains (as raw data never leaves clients during model training), FL non-trivial: a primary challenge stems from the predominantly has been widely adopted in vehicular networks. Representative Abstract—This paper presents a forward-looking vision for integrating the emerging multi-modal multi-task federated foundation models (M3T FedFMs) into vehicular networks, with the goal of unifying the expressive power of multi-modal multi-task foundation models (M3T FMs) with the privacy-preserving and distributed learning capabilities of federated learning (FL). Given the largely underexplored nature of this research direction, we first introduce the fundamental training/fine-tuning principles of M3T FedFMs. We then discuss a range of their representative use cases in vehicular networks, illustrating the significant potential of M3T FedFMs to enable next-generation vehicular intelligence. Afterwards, we identify key constraints inherent to vehicular environments that challenge the practical deployment of M3T FedFMs, and articulate a set of forward-looking research directions to address these challenges. Furthermore, through a case study conducted on a real-world vehicular dataset (i.e., Waymo Open Dataset), we demonstrate the promise of M3T FedFMs for vehicular networks and release our implementation to facilitate reproducibility and stimulate research in this emerging area (repository: https://github.com/KasraBorazjani/vehicular-fedfm).
2
applications include driver monitoring, personalized modeling of individual driving behaviors, advanced driver-assistance systems (ADAS) and steering control, cooperative perception, and traffic prediction/management [5].
B. M3T FMs in Vehicular Systems Exploring the applications of various M3T FMs in vehicular networks is a recent research topic. For example, EMMA (Endto-End Multimodal Model for Autonomous Driving) is an M3T FM that integrates camera inputs and navigation instructions to produce outputs such as planner trajectories, perception objects, and road graph elements [6]. Also, Drive Anywhere is an M3T FM designed for generalizable end-to-end autonomous driving [7]. Further, DeepInteraction++ is an M3T FM for autonomous driving [8] with exceptional performance in 3D object detection and end-to-end driving tasks. Collectively, these studies demonstrate the potential of integrating M3T FMs into vehicular networks; however, they rely on centralized training/fine-tuning of M3T FMs, where data is first collected at a centralized location (e.g., a cloud server) prior to model training. Such an assumption may not naturally extend to realistic vehicular networks, in which data are generated and stored in a geo-distributed manner across vehicles and RSUs.
Fig. 1. Schematic of the M3T FM architecture, consisting of modality encoders, a shared backbone (e.g., Mixture-of-Experts (MoE) or stacked transformers), and task heads. In addition, lightweight adaptation mechanisms, such as prompt tuning, low-rank adaptation (LoRA), and adapter layers can be employed for parameter-efficient model fine-tuning.
A. Architecture of M3T FMs
As illustrated in Fig. 1, the architecture of an M3T FM can be decomposed into the following three components: 1. Modality Encoders: Modality encoders serve as the front-end of an M3T FM, with each encoder responsible for processing a specific input modality (e.g., LiDAR, wideangle cameras, or GNSS signals). Specifically, each modality encoder transforms raw sensory inputs into a latent embedding representation, which captures the salient features of the corresponding modality and is subsequently passed to the C. M3T FedFMs: Blending FL and M3T FMs model’s backbone for further processing. M3T FedFMs are gaining high traction in both ML and 2. Backbone: The model backbone is responsible for wireless/communication communities as one of the major next cross-modal alignment and fusion, producing unified latent technologies [4]. Nevertheless, integration of M3T FedFMs representations that can be adapted to downstream tasks. This within vehicular networks is highly unexplored, with very backbone can take a variety of architectural forms, two common few works existing in this domain. The most related work is choices of which are outlined below: pFedLVM in [9], which addresses the challenges of deploying (i) Stacked Transformers (e.g., GPT, Gemini, CLIP [12]): Large Vision Models (LVM) in federated settings by keeping Stacked transformer backbone architectures employ multiple the LVM backbone centralized and sharing latent features with layers of self-attention and feed-forward networks to capture vehicles, enabling personalized learning. To our knowledge, long-range dependencies and complex interactions across pFedLVM is among the first to present a framework for the modalities. implementation of M3T FedFMs over vehicular networks. (ii) Mixture-of-Experts (e.g., DeepSeek [13]) MixtureSubsequently, our overarching goal in this work is to elucidate of-Experts (MoE) backbones decompose the model into a the unique challenges and design considerations associated collection of specialized expert subnetworks, with a gating with training/fine-tuning M3T FedFMs in vehicular settings, mechanism dynamically routing inputs to a subset of experts. thereby catalyzing further research in this emerging area and This design enables conditional computation, allowing different outlining a roadmap toward next-generation M3T FedFM- modalities, tasks, or input contexts to activate different experts. enabled intelligence in modern vehicular systems. 3. Task Heads: Task heads are typically implemented as lightweight neural modules that map the embeddings produced by the model’s backbone to concrete task-specific predictions, III. L EARNING A RCHITECTURE OF M3T F ED FM S such as control commands or action/decision probabilities. In a nutshell, M3T FedFMs enable the distributed training of M3T FMs across a set of decentralized devices (e.g., vehicles). B. Architecture of M3T FedFMs Owing to the recency and ongoing evolution of this research direction, there is currently no unique learning architecture for Given that M3T FMs are typically large-scale models with M3T FMs, and, by extension, for M3T FedFMs. Subsequently, billions/trillions of parameters, they commonly undergo an to ground our discussion, and informed by recent advances initial pre-training phase, which is often centralized and in this area [10], [11], we delineate a modular architecture exposes the model to massive datasets. Subsequently, these for M3T FMs by decomposing their design into three core models are trained/fine-tuned on narrower, task-specific datasets components, and then discuss how this modularity extends to to adapt them to the downstream tasks of interest. In this the M3T FedFM paradigm. context, M3T FedFMs enable the distributed training/fine-
3
Fig. 2. Schematic of the M3T FedM architecture over a set of vehicular edge nodes (the orange box on the left collects the existing data modalities in the diagram). Each vehicular edge node performs local fine-tuning of its model using parameter-efficient adaptation techniques (e.g., prompt tuning, LoRA, or adapter layers) to train/fine-tune its local M3T FM and transmits the resulting model updates to a server. The server then aggregates the received model updates to refine the global model parameters and disseminates the updated global model back to the nodes for subsequent training rounds.
tuning of M3T FMs across geo-distributed devices that collect and aggregated across clients. • Mixture-of-Experts (MoE) Training: MoE training updates data relevant to the target downstream tasks. a subset of expert subnetworks activated for specific inputs, Specifically, in M3T FedFMs (see Fig. 2), full-model modalities, or tasks; the parameters of these experts (and, training is typically avoided, as it is often infeasible across when applicable, the gating network) constitute the PEFT resource-constrained clients. Instead, clients employ parametermodules that are trained and aggregated across clients. efficient fine-tuning (PEFTs) techniques to adapt selected model components/modules locally, and subsequently transmit their locally learned model updates (e.g., gradients or module IV. U SE C ASES OF M3T F ED FM S IN V EHICULAR parameters) to a coordinating server. The server aggregates N ETWORKS these updates and disseminates the refined parameters back In this section, we outline representative use cases of M3T to the clients for subsequent training rounds. While this FedFMs across key stakeholders in vehicular ecosystems, aggregation process largely mirrors that of conventional FL, a including autonomous vehicles, human drivers and passengers, key distinction is that it operates at the module-level, rather than and vehicular network operators. For each use case, we (i) over the entire model (i.e., weighted aggregation is performed identify the typical data modalities involved, (ii) describe the over the parameters associated with PEFT-enabled modules). In expected downstream tasks, and (iii) present an example that particular, representative PEFT techniques and their subsequent highlights the potential of M3T FedFMs in vehicular settings. PEFT-enabled modules (see Fig. 1 for their placement within the local M3T FMs of nodes) are as follows: A. Use Case 1: M3T FedFM-Powered Autonomous Vehicles • Low-Rank Adaptation (LoRA): Low-Rank Adaptation (LoRA) injects trainable low-rank matrices into selected M3T FedFMs across autonomous vehicles (e.g., self-driving backbone layers; these matrices constitute the PEFT modules cars, buses, and delivery fleets) allow each vehicle to adapt that are trained and aggregated across clients. an M3T FM to its local sensing conditions and operational • Prompt Tuning: Prompt tuning prepends a small set of objectives/tasks without sharing raw sensory data. In particular, trainable prompt embeddings to the backbone input; these autonomous vehicles are typically equipped with multi-modal embeddings constitute the PEFT modules that are trained sensing capabilities, including LiDAR, cameras, millimeterand aggregated across clients. wave radar, GNSS signals, inertial sensors, and vehicle-state • Adapter Tuning: Adapter tuning inserts lightweight, train- telemetry. Leveraging these locally collected data modalities, able bottleneck layers between selected backbone layers; vehicles pursue a diverse set of downstream objectives/tasks, these layers constitute the PEFT modules that are trained such as localized perception (e.g., 3D object detection and
4
tracking), prediction (e.g., trajectory and intent inference), planning and control, localization and mapping, and cooperative perception with nearby vehicles or roadside infrastructure. Within this context, M3T FedFMs facilitate collaborative, module-level fine-tuning of M3T FMs across vehicles. For example, in an M3T FedFM paradigm, vehicles observing different portions or viewpoints of the same environment can collaboratively fine-tune their M3T FMs using their own sensor data to improve their cooperative perception performance, such as detecting occluded objects or enhancing scene completeness. B. Use Case 2: M3T FedFMs for Human Drivers/Passengers
these operators through the M3TFedFM paradigm enables the training of a global model that progressively improves its understanding of traffic dynamics and roadway conditions across the broader vehicular ecosystem. V. C HALLENGES AND F UTURE R ESEARCH D IRECTIONS We next focus on the challenges associated with implementing M3T FedFMs in vehicular networks. To provide a more structured presentation, we organize our discussions around three key challenges, each paired with a set of research directions.
M3T FedFMs enable the distributed fine-tuning of M3T A. Challenge 1: M3T FedFMs under Unpredictable Vehicle FMs across vehicles to support intelligent, interactive services Availability and Edge Handoffs for human drivers and passengers, allowing each vehicle to Vehicular networks exhibit highly heterogeneous and timeadapt an M3T FM to user-specific behaviors, preferences, varying client/vehicle availability, which poses a fundamental and contexts without sharing raw in-cabin or diagnostic data. challenge to the training and fine-tuning of M3T FedFMs. In In particular, modern vehicles collect a wide range of local practice, vehicles participate in model training under vastly data modalities, including cabin cameras, microphones, driverdifferent operational conditions: for example, an electric monitoring sensors, dashboard signals, vehicle diagnostics, and vehicle parked at a charging station or in a parking lot may contextual information from infotainment systems. Using these have ample energy, stable connectivity, and idle compute locally collected modalities, vehicles support downstream tasks resources, allowing it to engage in local model fine-tuning with such as driver monitoring, personalized assistance, voice- and minimal impact on its primary functions. In contrast, vehicles vision-based interaction, comfort and safety optimization, and that are actively driving must balance model training with in-vehicle diagnostics. For example, when a driver encounters latency-critical perception, planning, and control tasks, often a dashboard warning or suspected mechanical issue, the driver can first capture an image of the warning indicator or the resulting in intermittent or constrained participation in learning. relevant vehicle component. The locally fine-tuned M3T FM Moreover, even when vehicles are capable of performing local then analyzes this visual input, along with available contextual training while in motion, they frequently traverse the coverage signals (e.g., vehicle diagnostics or recent sensor readings), areas of multiple base stations, leading to frequent handoffs. As to provide an initial explanation and recommended actions. If a result, the set of vehicles participating in each training round needed, the driver can follow the model’s guidance to capture of M3T FedFM can vary over time and cannot be assumed additional images or provide further contextual input, allowing to be stable or known a priori. These dynamics introduce the model to iteratively refine the diagnosis and troubleshooting training fragmentation, where model updates are generated by steps, thereby augmenting individual mechanical expertise and a continuously changing subset of vehicles with heterogeneous reducing reliance on rather expensive vehicle service centers. availability, connectivity, and resource profiles. Consequently, M3T FedFMs in vehicular networks operate in an inherently on-the-fly learning regime, in which C. Use Case 3: M3T FedFMs for Vehicular Network Operators participant selection, training duration, and communication M3T FedFMs can unlock the distributed fine-tuning of opportunities must adapt dynamically to evolving mobility and M3T FMs across vehicular network operators, allowing trans- network conditions. Addressing this challenge calls for the portation authorities, city planners, and mobility-as-a-service development of availability- and mobility-aware M3T FedFM providers (e.g., ride-hailing services, on-demand shuttle oper- frameworks that explicitly account for unpredictable vehicle ators, and shared-mobility fleet managers) to collaboratively participation and edge handoffs. In this context, promising train their M3T FMs using their region-specific traffic patterns research directions include (i) dynamic vehicle scheduling and infrastructure conditions without centralizing raw data. policies for M3T FedFMs that prioritize participants based In particular, vehicular network operators collect and access on availability, resource profile, and predicted connectivity a diverse set of data modalities, including vehicle reports, duration, while accounting for the underlying PEFT method RSUs’ collected data, traffic cameras, environmental sensors, used; (ii) opportunistic model aggregation mechanisms for M3T and infrastructure telemetry. Further, the expected downstream FedFMs that tolerate partial and asynchronous PEFT module tasks include traffic flow prediction, road-condition monitoring, updates across vehicles. accident detection and reporting, fault and liability analysis (e.g., identifying contributing factors in collisions), intelligent traffic signal coordination (e.g., adaptive red-light scheduling), B. Challenge 2: M3T FedFMs under Vehicular Task and and proactive congestion mitigation. For example, operators in Hardware Diversity different regions can locally fine-tune their M3T FMs on their Vehicular networks are inherently characterized by task and traffic and infrastructure data to improve incident detection and hardware diversity across participating edge nodes, which traffic control policies, where the model aggregation across poses a major challenge to the training and fine-tuning of
5
M3T FedFMs. Specifically, different edge nodes within the modalities and tasks evolve, while mitigating catastrophic vehicular ecosystem, including autonomous vehicles, human- forgetting across previously learned modalities and tasks across driven vehicles, RSUs, and other sensing or data-collection the vehicular edge nodes; (ii) instantaneous model tracking infrastructure, are tasked with distinct learning objectives. For for M3T FedFMs under time-varying modality–task profiles, instance, vehicles may prioritize perception, prediction, and focusing on maintaining the best-performing PEFT modules for planning tasks, while RSUs may focus on traffic monitoring, the current modality-task profile across vehicular edge nodes, incident detection, or infrastructure-aware sensing. As a result, prioritizing rapid PEFT module adaptation to recent data over participating nodes often contribute data and model updates long-term knowledge retention. corresponding to heterogeneous and partially overlapping task sets. This challenge is further exacerbated by hardware and VI. C ASE S TUDY sensing heterogeneity across these edge nodes: vehicles and We next perform a case study on a representative challenge infrastructure units are equipped with different sensing suites, in M3T FedFMs: the ability to incorporate new tasks into ranging from cameras, LiDAR, and radar to microphones, an already-trained model without retraining from scratch or environmental sensors, and vehicle diagnostics, leading to disrupting existing capabilities. This challenge, known as task variation in the available data modalities across clients. onboarding or continual task arrival, is acute in vehicular Consequently, M3T FedFMs must be trained under un- networks, where a vehicle fleet may have been trained on a set balanced modality-task availability, where certain modalities of perception capabilities and must integrate new tasks over or tasks are presented at some nodes and absent at others. time as operational requirements evolve. Addressing this challenge calls for the development of task- and The details of our implementations, including all modality-aware M3T FedFM frameworks. Promising research source codes, are provided in our GitHub repository: directions include (i) selective and modular model training https://github.com/KasraBorazjani/vehicular-fedfm strategies for M3T FedFMs, in which only task-relevant or modality-specific submodules of the local M3T FedFMs are updated and aggregated based on local task-modalities profile A. Dataset and Tasks We implement an M3T FedFM over an urban vehicular of the participating clients; (ii) adaptive client scheduling mechanisms for M3T FedFMs under unbalanced modality–task scenario comprising 20 vehicles/clients based on the realavailability, which dynamically select participants in model world Waymo Open Dataset’s perception segment (available training and their training configurations according to their online at https://waymo.com/open/data/perception/). The dataset available modalities and tasks; such scheduling mechanisms contains data gathered from cameras deployed on cars across must be revisited and tailored to the underlying PEFT approach San Francisco, Mountain View, Los Angeles, Detroit, Seattle, and Phoenix. Each segment comprises a 20-second driving clip used for model fine-tuning. sampled at approximately 10 Hz, captured by five surroundview cameras providing full 360° coverage and one mid-range C. Challenge 3: M3T FedFMs under Time-Varying Vehicular spinning LiDAR sensor alongside five short-range LiDAR Data sensors. We dedicate one dataset segment to each of the 20 Vehicular networks operate in environments where both data clients, where each segment corresponds to a distinct 20-second modalities and learning tasks evolve/drift over time, posing driving clip from a single vehicle. This induces non-IID data a challenge to the training and deployment of M3T FedFMs. distributions across clients, as segments were collected in In practice, the sensing conditions experienced by vehicles different cities, under different weather/lighting conditions, and roadside infrastructure can change due to factors such and with varying object class frequencies. as traffic density, weather, time of day, road topology, and We train our M3T FedFM on four perception tasks spanning hardware usage patterns. As a result, the availability, quality, both camera and LiDAR modalities alongside an augmented and relevance of different data modalities (e.g., camera, LiDAR, text modality: (i) 3D Semantic Segmentation, where the input radar, or contextual signals) may drift over time. Similarly, modalities are image, LiDAR, and text, while the output is the the importance and definition of downstream tasks can shift, per-point semantic class label in the LiDAR point cloud across as vehicles transition between operational contexts, such as 23 object categories. The performance metric used for this urban driving, highway cruising, construction zones, or adverse task is mean Intersection over Union (mIoU). (ii) 3D Object weather conditions, each emphasizing different perception, Detection, where the input modalities are image, LiDAR, and prediction, or control objectives. These temporal variations text, while the output is 3D axis-aligned bounding boxes with induce modality and task drift across edge nodes in vehicular heading angles for all objects in the scene. The performance ecosystems, leading to non-stationary learning environments in metric used for this task is Mean Average Precision weighted which data distributions and task priorities change continuously. by Heading accuracy at difficulty level L2 (mAPH/L2). (iii) 2D Consequently, M3T FedFMs must operate under evolving Video Panoptic Segmentation, where the input modalities are modality–task profiles in vehicular networks. In this context, image and text, while the output is the per-pixel semantic class promising research directions include (i) continual learning and temporally consistent instance ID across all frames of the for M3T FedFMs under modality and task drift, focusing 20-second segment for 28 semantic classes. The performance on developing PEFT-driven continual learning mechanisms metric used for this task is Weighted Segmentation and Tracking that allow M3T FedFMs to incorporate new information as Quality (wSTQ). (iv) 2D Object Detection, where the input
6
Fig. 3. Architecture of the proposed M3T FedFM framework and the corresponding federated adaptation process. The model consists of three modality-specific K K transformer encoder branches devoted to processing vision/image, LiDAR, and text modalities, denoted by {Vk }K k=1 , {Lk }k=1 , and {Tk }k=1 , respectively (similar to CLIP, the total number of transformer blocks is K = 12). The image, LiDAR, and text inputs are first transformed into patch/token embeddings of size 3 × H × W , 3 × H × W , and 77 × 512 for the image, LiDAR, and text modalities, respectively, and subsequently propagated through their corresponding transformer backbones. Here, H and W correspond to the height and width of each image/LiDAR patch. To enable parameter-efficient fine-tuning, lightweight K K trainable adapters, denoted by {δv,k }K k=1 , {δℓ,k }k=1 , and {δt,k }k=1 , are inserted into the frozen backbone blocks via residual adapter injections. The resulting modality-specific semantic embeddings are denoted by π Vis ∈ R196×512 , π Lid ∈ R196×512 , and π Txt ∈ R512 , corresponding to the latent feature representations extracted from the vision, LiDAR, and text branches, respectively. The vision and LiDAR embeddings are fused through the multi-modal fusion module and processed by a shared task adapter that captures transferable cross-task knowledge across tasks. The resulting shared representation is subsequently delivered to multiple task-specific heads supporting the downstream tasks. Ultimately, at the end of each task head, a cosine similarity operation is performed on the token embeddings from the text modality via dot product.
modalities are image and text, while the output is 2D axis- task and usable for all arriving tasks. This also lightens the aligned bounding boxes for vehicles, pedestrians, cyclists, and burden on the model backbone’s parameters that are intended to signs in each of the five camera images. The performance extract multi-modal features from the input rather than adapting metric used for this task is Mean Average Precision at IoU to the tasks present in the system. Our ultimate method, called threshold 0.5 ([email protected]). FedAdapt, adopts an adapter-based PEFT technique (adapters across the image/vision, LiDAR, and text encoder branches are denoted by δv,· , δℓ,· , and δt,· in Fig. 3) and integrates B. Local M3T FM Architecture and Task Onboarding Setup We consider CLIP [12] as the underlying M3T FM model this shared task adapter along with Conflict-Averse Gradient deployed across the clients. Since this model is originally Descent (CAGrad [14]) to account for gradient normalization designed for handling text-image modalities and performing given the different scale of the gradients of tasks during the a single task, we make the following advancements to the backpropagation. For each reported result, one task is designated as the arriving model: (i) we add a new encoder branch for LiDAR, where the transformers are initialized with those of the image branch task while the remaining three tasks are treated as pre-trained (the encoder branches for vision/image, LiDAR, and text are tasks whose parameters are considered fixed and simultaneously denoted by V· , L· , and T· in Fig. 3), and (ii) we supplement the obtained via M3T FedFM prior to the new task arrival. When model with a series of task heads, each consisting of two de- the arriving task is onboarded, the following training protocol convolution layers to create the corresponding task’s output (see is applied: the task head corresponding to the arriving task is our aforementioned GitHub link for the exact implementation). initialized from scratch and trained; the shared task adapter To handle the task onboarding in the M3T FedFM setup, (where applicable) and all modality-specific adapters (visual, we design and add a shared task adapter to the model, which LiDAR, and text) are fine-tuned to accommodate the new task; captures cross-task representations (see Fig. 3). The rationale and all other parameters, including the frozen CLIP backbone behind this design relies on the capability of the shared task layers and the task heads of the three pre-trained tasks, remain adapter to contain information from the pre-trained tasks and unchanged throughout. act as an anchor/teacher for the arriving task. Thus, we will have 1) Baselines: We compare the performances of our method, task-related information that is re-adapted with each arriving FedAdapt, to the following baselines (across all methods,
7
Fig. 4. Task onboarding performance over the M3T FedFM aggregation rounds for the Waymo Open Dataset. In each sub-plot, the arriving task, mentioned in the title of the sub-plot, is onboarded (using the modalities mentioned in the top-left corner of the sub-plot) while the remaining three tasks are pre-trained and frozen. The consistent gap between FedAdapt and the baselines across all tasks demonstrates the benefit of our introduced shared task adapter regardless of the gradient conflict resolution strategy employed during the multi-task training.
FedAvg [3] is used for model aggregation): • No Task Adapter (NTA): Each client trains its local model with modality-specific adapters (visual, LiDAR, and text) but without a shared task adapter. • NTA+GR: This baseline combines NTA with GradNorm gradient balancing [15], but without using the shared task adapter. GradNorm dynamically reweights per-task loss gradients to enforce balanced task learning rates, using the ratio of each task’s current loss to its initial loss as a proxy for relative training speed. • NTA+CA: This baseline combines NTA with CAGrad [14], but without using the shared task adapter. • FedAdapt no CA: The baseline takes the CAGrad component out of our proposed model training pipeline. • FedAdapt no CA+GR: This baseline replaces CAGrad with GradNorm [15] in our original model training pipeline. C. Discussion of the Results We present the results in Fig. 4, which depicts the task onboarding performance over the model aggregation rounds for each of the four arriving tasks. The most consistent finding is that the presence of our proposed shared task adapter provides a clear and consistent performance benefit, with FedAdapt exhibiting the best performance across all baselines, confirming that our introduced shared task adapter serves as an effective knowledge transfer mechanism in the task onboarding setting in M3T FedFMs. More specifically, the effect of gradient conflict resolution is nuanced and task-dependent: for 3D Object Detection and 2D Object Detection, FedAdapt no CA+GR underperforms FedAdapt no CA, suggesting that operating directly on gradient directions (as in CAGrad) rather than loss magnitudes (as in GradNorm) provides a more robust conflict resolution mechanism across diverse task combinations when the M3T FedFM training pipeline comprises our introduced shared task adapter. These findings highlight the potential of M3T FedFMs for vehicular network operators, as they suggest that operators can leverage auxiliary, more general tasks collected across heterogeneous regions to enhance learning efficiency and
performance on critical, domain/vehicular-specific objectives, such as traffic understanding or incident reasoning, without centralizing raw data or sacrificing data privacy. VII. C ONCLUSION In this paper, we presented a forward-looking vision for the deployment of M3T FedFMs in vehicular networks. We first introduced a modular architectural perspective for M3T FedFMs along with their training/fine-tuning paradigms. We then explored representative use cases of M3T FedFMs in vehicular networks across three key stakeholder groups: autonomous vehicles, human drivers and passengers, and vehicular network operators. Furthermore, we outlined a set of open challenges and research directions, aiming to guide future work toward scalable, privacy-preserving, and intelligent vehicular systems powered by M3T FedFMs. Finally, we presented a case study to demonstrate the potential of M3T FedFMs in vehicular environments and publicly released our implementation to catalyze research in this area. R EFERENCES [1] Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al., “A survey on evaluation of large language models,” ACM transactions on intelligent systems and technology, vol. 15, no. 3, pp. 1–45, 2024. [2] K. Huang, B. Shi, X. Li, X. Li, S. Huang, and Y. Li, “Multi-modal sensor fusion for auto driving perception: A survey,” arXiv preprint arXiv:2202.02703, 2022. [3] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics, pp. 1273–1282, PMLR, 2017. [4] C. Ren, H. Yu, H. Peng, X. Tang, B. Zhao, L. Yi, A. Z. Tan, Y. Gao, A. Li, X. Li, et al., “Advances and open challenges in federated foundation models,” IEEE Communications Surveys & Tutorials, 2025. [5] Z. Du, C. Wu, T. Yoshinaga, K.-L. A. Yau, Y. Ji, and J. Li, “Federated learning for vehicular internet of things: Recent advances and open issues,” IEEE Open Journal of the Computer Society, vol. 1, pp. 45–61, 2020. [6] J.-J. Hwang, R. Xu, H. Lin, W.-C. Hung, J. Ji, K. Choi, D. Huang, T. He, P. Covington, B. Sapp, et al., “EMMA: End-to-end multimodal model for autonomous driving,” arXiv preprint arXiv:2410.23262, 2024.
8
[7] T.-H. Wang, A. Maalouf, W. Xiao, Y. Ban, A. Amini, G. Rosman, S. Karaman, and D. Rus, “Drive anywhere: Generalizable end-to-end autonomous driving with multi-modal foundation models,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6687– 6694, IEEE, 2024. [8] Z. Yang, N. Song, W. Li, X. Zhu, L. Zhang, and P. H. Torr, “Deepinteraction++: Multi-modality interaction for autonomous driving,” arXiv preprint arXiv:2408.05075, 2024. [9] W.-B. Kou, Q. Lin, M. Tang, S. Xu, R. Ye, Y. Leng, S. Wang, G. Li, Z. Chen, G. Zhu, et al., “pFedLVM: A large vision model (LVM)-driven and latent feature-based personalized federated learning framework in autonomous driving,” arXiv preprint arXiv:2405.04146, 2024. [10] J. Chen and A. Zhang, “On disentanglement of asymmetrical knowledge transfer for modality-task agnostic federated learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 11311–11319, 2024. [11] H. Chen, Y. Zhang, D. Krompass, J. Gu, and V. Tresp, “FedDAT: An approach for foundation model finetuning in multi-modal heterogeneous federated learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 11285–11293, 2024. [12] D. Rothman, Transformers for Natural Language Processing and Computer Vision: Explore Generative AI and Large Language Models with Hugging Face, ChatGPT, GPT-4V, and DALL-E 3. Packt Publishing Ltd, 2024. [13] A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo, et al., “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,” arXiv preprint arXiv:2405.04434, 2024. [14] B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu, “Conflict-averse gradient descent for multi-task learning,” Advances in neural information processing systems, vol. 34, pp. 18878–18890, 2021. [15] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, “GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” in International conference on machine learning, pp. 794–803, PMLR, 2018.
Kasra Borazjani is a Ph.D. student in the Department of Electrical Engineering at the University at Buffalo–SUNY, USA.
Fardis Nadimi is a Ph.D. student in the Department of Electrical Engineering at the University at Buffalo–SUNY, USA.
Payam Abdisarabshali is a Ph.D. student in the Department of Electrical Engineering at the University at Buffalo–SUNY, USA.
Owen Palinski is an M.Sc. student in the Department of Electrical Engineering at the University at Buffalo–SUNY, USA.
Allan Salihovic is a Ph.D. student in the Department of Electrical Engineering at the University at Buffalo–SUNY, USA.
Dinh Nguyen is an assistant professor of Electrical and Computer Engineering at the University of Alabama–Huntsville, USA.
Minghui Liwang is an associated professor of Electrical and Computer Engineering at Tongji University, China.
Seyyedali Hosseinalipour is an assistant professor of Electrical Engineering at the University at Buffalo–SUNY, USA.