LMPath: Language-Mediated Priors and Path Generation for Aerial Exploration
arXiv:2605.13782v1 [cs.RO] 13 May 2026
Jonathan A. Diller, Fernando Cladera, Camillo J. Taylor, Vijay Kumar
Abstract—Traditional autonomous UAV search missions rely on geometric coverage patterns that ignore the semantic context of the target, leading to significant time waste in large-scale environments. In this paper we present LMPath, a pipeline for generating language-mediated exploration priors for Unmanned Aerial Vehicle (UAV) search missions that leverages semantics. Given a basic geofence and an object of interest prompt, LMPath uses generative language models to determine what regions of the environment should contain that object and a foundation vision model ran over satellite imagery to segment sub-regions that form the exploration prior. This prior can then be used to generate UAV paths with various objectives, such as minimizing the expected time to locate the object of interest, maximizing the probability that the object is found given a limited travel distance, or narrowing down the search space to sub-regions that are most likely to contain the object. To demonstrate it’s capabilities, we used LMPath to generate various UAV paths and ran them using a real UAV over large-scale environments. We also ran simulations to demonstrate how paths generated using LMPath outperform traditional path planning approaches for search missions. The code of LMPath is available open source1 . Index Terms—aerial exploration, path finding, language semantics, foundation models
I. I NTRODUCTION Unmanned aerial vehicle (UAV) mission planning of overhead flight is usually performed manually. The traditional approach to path planning for UAV-search missions is for the human operator to manually select where the vehicle should go using GPS coordinates [1], [2]. These conventional methods severely limit the scalability and adaptability of autonomous UAV operations, as manual planning needs to be performed to change the UAV task. The advent of foundation models that can understand semantics in language and perform zero-shot vision tasks provides a means for rethinking how the user tasks UAVs. Large Language Models (LLMs) can understand the semantics of objects and the environments that they are customarily found in. For example, we expect to find cars in parking lots and along streets. Furthermore, foundation models for image segmentation, such as SAM 32 , have demonstrated the ability to zero-shot segmentation tasks over varying domains. These powerful models, combined with high-resolution satellite imagery, could be used to generate language-mediated This work was funded in part by the ARL DCIST CRA W911NF-17-2-0181. All authors are with the GRASP Laboratory, University of Pennsylvania. {diller, fclad, cjtaylor,
kumar}@seas.upenn.edu 1 https://github.com/KumarRobotics/LMPath 2 https://ai.meta.com/research/sam3
Fig. 1. Top: search prior generated by LMPath, targeting the label car, and UAV path to minimize expected search time. Bottom: orthomosaic generated after flying the mission, showing detected cars in red.
exploration priors that can be leveraged for generating UAV paths for search missions. Existing methods for using foundation models on satellite imagery for UAVs tend to focus on locating larger, static objects that are visible in satellite imagery, such as large buildings [3], [4]. These methods lack the ability to reason about the semantics between dynamic objects and their environment, such as searching for cars in parking lots. While local reactive navigation methods that utilize language embeddings to guide frontier exploration have shown promise in leveraging semantics [5], they lack a mechanism to translate high-level environmental common sense into global, optimized search priors for large-scale areas. In this paper we present LMPath, a pipeline for generating language-mediated exploration priors that leverages semanticspatial reasoning for optimized UAV path generation. Our contributions are as follows: 1) A semantic-spatial reasoning framework: We leverage an LLM to infer semantically relevant labels that provide a reasoning layer between a search target and its likely environmental context. 2) A multi-objective path-finding system: We integrate
Object Prompt LLM Agent
Geofence
Exploration Prior
Segmentation Labels Segmentation Model
Path Planner
Optimized Mission
Tile Composer
Satellite Image
Waypoint Generator
Coverage Waypoints
Fig. 2. The LMPath pipeline. LMPath takes a user-provided object prompt and geofence bounds for exploration and generates a heatmap that serves as an exploration prior. The exploration prior is then crossed with waypoints that provide sensor coverage of the area of interest and fed into a path planner.
these language-mediated priors into an Integer-Linear Program (ILP) to optimize paths for diverse mission objectives, such as minimizing search time. We demonstrate the system’s utility through real-world field tests and high-fidelity simulations where LMPath outperformed semantic-agnostic baselines by up to 88.0%. II. L ANGUAGE -M EDIATED P RIORS F RAMEWORK The LMPath pipeline (shown in figure 2) consists of an LLM agent for generating semantically relevant labels, a vision foundation model for segmenting regions of the environment that match the semantic labels, a waypoint generator based on sensor coverage, and an ILP for path finding. A. Generating An Exploration Prior The LMPath pipeline initializes by taking in a user-provided target label and geospatial metadata that defines the operational geofence boundaries, no-fly zones, and the UAV’s starting location, b ∈ R2 . This geospatial metadata is formatted as a QGroundControl (QGC) plan mission file. Using these boundaries, a Tile Composer queries a web-based map tile service to retrieve and stitch together high-resolution tiles, forming a complete global satellite image of the search area, denoted as I. To generate a heatmap prior for an arbitrary, user-defined target object O (e.g., car), the framework prompts an LLM to infer likely spatial contexts. The LLM outputs a set of semantically relevant labels L = {l1 , l2 , . . . , lk } (e.g., parking lot, road, driveway). This process effectively bridges the gap between the specific search target and the broader semantic features recognizable in overhead imagery. With the semantic labels L defined, the pipeline employs a foundation segmentation model, SAM 3, to generate a probabilistic heat map over I. Because satellite imagery is typically too large to process in a single forward pass, we implement a sliding window approach. Let W be the set of overlapping image windows extracted from I. For each window w ∈ W and each label l ∈ L, SAM 3 generates a binary segmentation mask Mw,l (x, y) ∈ {0, 1}. To mitigate boundary artifacts and ensure smooth transitions,
the overlapping masks are averaged. The aggregated mask for a given label l across the entire image is computed as P w∈W(x,y) Mw,l (x, y) Ml (x, y) = , |W(x, y)| where W(x, y) represents the subset of windows that contain the pixel coordinate (x, y). The final semantic prior, or heat map, H(x, y), is generated by summing the masks for all labels and normalizing the distribution: P Ml (x, y) H(x, y) = RR P l∈L . l∈L Ml (u, v) du dv I This results in a 2D probability density function where higher values indicate a greater semantic likelihood of the object O being present. B. Waypoint Generation and Path Finding To translate the continuous heat map H(x, y) into a discrete graph suitable for path planning, a Waypoint Generator creates a set of candidate waypoints N = {n1 , n2 , . . . , nm } over the search area. The Waypoint Generator sets the number of waypoints based on the UAV’s sensor coverage footprint and the total area of I. The pipeline then partitions the operational space using a Voronoi tessellation based on N . The Voronoi cell vi corresponding to waypoint ni is defined as vi = {p ∈ R2 | ∥p − ni ∥ ≤ ∥p − nj ∥ ∀j ̸= i}. The “heat” or probability mass associated with each waypoint, denoted as pi ∈ P , is calculated by integrating the semantic prior over its corresponding Voronoi cell and normalizing by the cell’s area A(vi ) to prevent bias toward disproportionately large cells: ZZ 1 pi = H(x, y) dx dy. A(vi ) vi The final component of the pipeline is a Path Planner for generating UAV paths based on various objectives over P .
(a) Geofence & Launch Position
(b) Satellite Image
(c) LMPath-Generated Heatmap
(d) LMPath-Generated Path
(e) Geofence & Launch Position
(f) Satellite Image
(g) LMPath-Generated Heatmap
(h) LMPath-Generated Path
Fig. 3. LMPath examples for generating heatmaps and flight paths to find the object “car” in minimum time (a-d), and a threshold-reduced flight path to find the object “building.” (e-h). The car example shows complete coverage but explores parking lots before the grass field, while the building example demonstrates the UAV exploring two distinctive sub-regions.
1) Minimizing Expected Search Time: In the context of searching for a specific object of interest, the user may want to minimize the expected time required to locate that object. Let xij ∈ {0, 1} be a binary decision variable that equals 1 if the UAV travels from waypoint ni to nj , and 0 otherwise. Let ti denote the time that the UAV visits waypoint i. This problem can be formulated as the following ILP: X min pi ti (1) i∈N
subject to X
xji + xb,i = 1,
∀i ∈ N
(2a)
xij + xi,b = 1,
∀i ∈ N
(2b)
j∈N
X j∈N
X
xb,i = 1
(2c)
i∈N
xii = 0, tj ≥ ti +
dij − M(1 − xij ), v
∀i ∈ N
(2d)
∀i, j ∈ N
(2e)
ti ≥
db,i − M(1 − xb,i ), v
∀i ∈ N
(2f)
Constraints 2a and 2b specify that waypoint i must be entered and exited, respectively, exactly once, either from another waypoint j or from the base station b. Constraint 2c ensures that the UAV goes to exactly one node after leaving the base station while constraint 2d prevents single waypoint cycles. If the UAV travels from i to j (i.e., xij = 1), then constraint 2e forces tj to be greater than or equal to ti plus the time to travel distance dij at speed v using an arbitrarily large value M. Constraint 2f enforces the same restriction on the first waypoint in the route. 2) Target-Focused Path in Large Environments: LMPath can generate missions for large-scale environments where covering the entire space is not expected to be as helpful as focusing on mission-relevant sub-regions. To generate an efficient, mission-focused paths in large environments the Path Planner filters out waypoints i ∈ N where pi < ρ, for some threshold value ρ. The remaining waypoints can then be treated as an input to the Traveling Salesman Problem (TSP), for which there are many well studied algorithms.
III. E XPERIMENTAL E VALUATION A. Experimental Setup We utilized SAM 3 as our foundation segmentation model and GPT-4o-mini as the LLM agent. The path planner was implemented in C++ using the Gurobi solver (version 13.01). For the geospatial raster tile service, Mapbox provided highquality satellite tiles at high zoom levels, though ESRI World Imagery and Google Maps were also evaluated. Empirically, a 75% overlap in the sliding window mask mosaic and a subregion size of 100 square meters yielded consistent heat map priors across various environments. B. Real-World Validation To evaluate the LMPath pipeline, we generated several path plans for distinct operational missions and ran them on a UAV. Figures 3a through 3d show an example of searching for a car by solving the ILP presented in Section II-B1. The UAV path initially goes over the parking lot before covering the remainder of the search space, which mostly consistent of a grass field. Figures 3e through 3h show an example of searching for buildings within a large-scale environment (300 m × 450 m) using the path planning approach described in Section II-B2. The UAV path covers two large, open spaces and ignores large sections of forest. These missions were successfully executed in real-world field tests using a Falcon 4 UAV with a nadir-viewing RGB camera [6]. C. Simulation Environments To quantitatively assess the framework’s ability to minimize search time, we conducted a series of Gazebo simulations where the UAV was tasked with locating a randomly selected target car. The simulations were evaluated across two distinct environments: • PolyCity: A synthetic, low-complexity urban layout. • Industrial Park: A high-fidelity 3D mesh reconstructed from actual UAV-captured imagery (Figure 4). Crucially, for the real-world mesh environment, LMPath was provided with standard web-sourced satellite imagery of that same geographic location. This established a highly realistic operational scenario where the generated semantic prior covers the correct spatial area but reflects the environment at a different point in time than the simulated world. We evaluated the efficiency of the LMPath framework by measuring the total flight time required to successfully locate a randomly selected target car. We repeated this experiment 50 times, using the same UAV path but selecting different cars using a uniform distribution. We compared our approach against a baseline TSP planner, which calculates an efficient route to uniformly cover the entire operational space without the guidance of a semantic prior. The path generated using LMPath found the target faster than the baseline 66.0% of the time in PolyCity and 88.0% of the time in the Industrial Park. The results demonstrate that LMPath can significantly reduce the expected search time by intelligently prioritizing high-probability regions over a semantic-agnostic exhaustive coverage approach.
(a) Exploration prior
(b) Drone simulator
Fig. 4. Exploration prior from real satellite image (a) and drone simulator for Industrial Park.
IV. C ONCLUSIONS In this paper, we introduced LMPath, a novel pipeline that leverages Large Language Models and foundation vision models to generate semantic exploration priors for UAV search missions. By linking user-defined prompts with satellite imagery, LMPath transforms continuous environments into discrete, probability-weighted graphs for advanced path planning. Both our real-world Falcon 4 field tests and high-fidelity simulations demonstrate that intelligently targeting high-probability sub-regions significantly outperforms traditional, non-semantic coverage methods. One of the limitations of LMPath is that it can only segment locations that are clearly visible from the given satellite image. This could become problematic in areas where large buildings or vegetation create occlusions. Future work will focus on merging the LMPath exploration prior with local, frontier exploration algorithms [5]. Additionally, the current heat map formulation assumes uniform weighting to labels and masks. Future iterations could incorporate label confidence and conditional dependencies (e.g., favor “driveway” when adjacent to “building”) to further refine the search prior. R EFERENCES [1] J. Diller and Q. Han, “Energy-aware drone path finding with a fixed-trajectory ground vehicle,” ACM J. Auton. Transport. Syst., vol. 2, no. 4, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3716894 [2] F. Cladera, Z. Ravichandran, J. Hughes, V. Murali, C. Nieto-Granda, M. Ani Hsieh, G. J. Pappas, C. J. Taylor, and V. Kumar, “Air-ground collaboration for language-specified missions in unknown environments,” IEEE Transactions on Field Robotics, vol. 2, pp. 626–642, 2025. [3] O. Sautenkov, A. Akhmetkazy, Y. Yaqoot, M. A. Mustafa, G. Tadevosyan, A. Lykov, V. Serpiva, and D. Tsetserukou, “Uav-vlpa*: Vision-language guided global-local uav mission planning from satellite imagery,” in 2025 IEEE International Conference on Robotics and Biomimetics (ROBIO), 2025, pp. 2354–2359. [4] J. Li, P. Chen, M. Li, and L. Ren, “Ib-amg: Aircraft mission generation with inference-based vision-language-action model,” in 2025 40th Youth Academic Annual Conference of Chinese Association of Automation (YAC), 2025, pp. 2323–2328. [5] Y. Tao, D. Ong, F. Cladera, J. Hughes, C. J. Taylor, P. Chaudhari, and V. Kumar, “Halo: High-altitude language-conditioned monocular aerial exploration and navigation,” 2025. [Online]. Available: https: //arxiv.org/abs/2511.17497 [6] F. Cladera, K. Chaney, M. A. Hsieh, C. J. Taylor, and V. Kumar, “Evmapper: High-altitude orthomapping with event cameras,” in 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 310–316.