Machine-learned monocular depth estimation and semantic segmentation for 6-DOF … — Wing Aviation Llc (US12307710B2)

ABSTRACT

Abstract

A method includes receiving a two-dimensional (2D) image captured by a camera on a unmanned aerial vehicle (UAV) and representative of an environment of the UAV. The method further includes applying a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, where the semantic image comprises one or more semantic labels. The method additionally includes retrieving reference depth data representative of the environment, wherein the reference depth data includes reference semantic labels. The method also includes aligning the depth image of the environment with the reference depth data representative of the environment to determine a location of the UAV in the environment, where the aligning associates the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data.

Description

BACKGROUND

An unmanned vehicle, which may also be referred to as an autonomous vehicle, is a vehicle capable of travel without a physically-present human operator. An unmanned vehicle may operate in a remote-control mode, in an autonomous mode, or in a partially autonomous mode.

When an unmanned vehicle operates in a remote-control mode, a pilot or driver that is at a remote location can control the unmanned vehicle via commands that are sent to the unmanned vehicle via a wireless link. When the unmanned vehicle operates in autonomous mode, the unmanned vehicle typically moves based on pre-programmed navigation waypoints, dynamic automation systems, or a combination of these. Further, some unmanned vehicles can operate in both a remote-control mode and an autonomous mode, and in some instances may do so simultaneously. For instance, a remote pilot or driver may wish to leave navigation to an autonomous system while manually performing another task, such as operating a mechanical system for picking up objects, as an example.

Various types of unmanned vehicles exist for various different environments. For instance, unmanned vehicles exist for operation in the air, on the ground, underwater, and in space. Examples include quad-copters and tail-sitter UAVs, among others. Unmanned vehicles also exist for hybrid operations in which multi-environment operation is possible. Examples of hybrid unmanned vehicles include an amphibious craft that is capable of operation on land as well as on water or a floatplane that is capable of landing on water as well as on land. Other examples are also possible.

SUMMARY

The present embodiments are directed to systems and methods that may help to localize an unmanned aerial vehicle within an environment. A machine learning model may be trained to process two-dimensional (2D) images of a terrain within the environment. The model may include separate depth and semantics branches to produce both depth images and semantic images. The depth and semantic information may be used in combination to localize the vehicle in the environment relative to reference depth and semantic data.

In one aspect, a method is provided including receiving a two-dimensional (2D) image captured by a camera on a unmanned aerial vehicle (UAV) and representative of an environment of the UAV. The method further includes applying a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, where the machine learning model has been trained with a semantics branch to produce the semantic image and a depth branch to produce the depth image, and where the semantic image comprises one or more semantic labels. The method also includes retrieving reference depth data representative of the environment, where the reference depth data includes reference semantic labels. The method additionally includes aligning the depth image of the environment with the reference depth data representative of the environment to determine a location of the UAV in the environment, where the aligning associates the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data.

In another aspect an unmanned aerial vehicle is described, including a camera and a control system configured to receive a two-dimensional (2D) image captured by the camera and representative of an environment of the UAV; apply a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, where the machine learning model has been trained with a semantics branch to produce the semantic image and a depth branch to produce the depth image, and where the semantic image comprises one or more semantic labels; retrieve reference depth data representative of the environment, where the reference depth data includes reference semantic labels; and align the depth image of the environment with the reference depth data representative of the environment to determine a location of the UAV in the environment, where the aligning associates the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data.

In another aspect, a non-transitory computer readable medium is described including program instructions executable by one or more processors to perform operations. The operations include receiving a two-dimensional (2D) image captured by a camera on a unmanned aerial vehicle (UAV) and representative of an environment of the UAV; applying a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, where the machine learning model has been trained with a semantics branch to produce the semantic image and a depth branch to produce the depth image, and where the semantic image comprises one or more semantic labels; retrieving reference depth data representative of the environment, where the reference depth data includes reference semantic labels; and aligning the depth image of the environment with the reference depth data representative of the environment to determine a location of the UAV in the environment, where the aligning associates the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data.

These as well as other aspects, advantages, and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description with reference where appropriate to the accompanying drawings. Further, it should be understood that the description provided in this summary section and elsewhere in this document is intended to illustrate the claimed subject matter by way of example and not by way of limitation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 A is a simplified illustration of an unmanned aerial vehicle, according to an example embodiment.

FIG. 1 B is a simplified illustration of an unmanned aerial vehicle, according to an example embodiment.

FIG. 1 C is a simplified illustration of an unmanned aerial vehicle, according to an example embodiment.

FIG. 1 D is a simplified illustration of an unmanned aerial vehicle, according to an example embodiment.

FIG. 1 E is a simplified illustration of an unmanned aerial vehicle, according to an example embodiment.

FIG. 2 is a simplified block diagram illustrating components of an unmanned aerial vehicle, according to an example embodiment.

FIG. 3 is a simplified block diagram illustrating a UAV system, according to an example embodiment.

FIG. 4 is a diagram illustrating training and inference phases of a machine learning model, in accordance with example embodiments.

FIG. 5 is a diagram illustrating application of a machine learning model, in accordance with example embodiments.

FIG. 6 A illustrates an image captured by a UAV, in accordance with example embodiments.

FIG. 6 B illustrates reference depth data representative of an environment of a UAV, in accordance with example embodiments.

FIG. 6 C illustrates semantically labeled depth data generated by a machine learning model, in accordance with example embodiments.

FIG. 6 D illustrates superimposed reference depth data and semantically labeled depth data generated by a machine learning model, in accordance with example embodiments.

FIG. 6 E illustrates reference depth data aligned with semantically labeled depth data generated by a machine learning model, in accordance with example embodiments.

FIG. 7 is a block diagram of a method, in accordance with example embodiments.

DETAILED DESCRIPTION

Exemplary methods and systems are described herein. It should be understood that the word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any implementation or feature described herein as “exemplary” or “illustrative” is not necessarily to be construed as preferred or advantageous over other implementations or features. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example implementations described herein are not meant to be limiting. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

I. Overview

Determining the absolute location (longitude, latitude, and altitude) of a delivery drone or other unmanned aerial vehicle (UAV) is often critical for mission success. A common way of determining absolute location is to rely on a Global Navigation Satellite System (GNSS), such as a Global Positioning System (GPS). However, in some situations, GPS can become unavailable or the receiver can produce an erroneous location measurement. Consequently, the ability to derive an absolute location of a UAV using a different modality may serve as a complement and/or backup to GPS.

Examples described herein enable a UAV to use a machine-learned model to produce a semantically labeled three-dimensional (3D) point cloud (e.g., a 3D model of a scene). The semantically labeled 3D point cloud can then be registered against a pre-existing point cloud. In some examples, the pre-existing point cloud may be derived from a vendor or built during a preceding visit by the UAV and/or by other UAVs. Registering the semantically labeled 3D point cloud against the pre-existing point cloud may be performed to derive the vehicle's absolute position and orientation (e.g., to recover the vehicle's position and orientation with six degrees of freedom). In some examples, the absolute position can be used alone in the absence of GPS to complete a mission assigned to the UAV. In further examples, the absolute position can be used to cross-check GPS measurements and/or other sensor measurements. In yet further examples, the absolute position can be blended with the GPS measurement to arrive at a more certain belief of the UAV's whearabouts.

II. Illustrative Unmanned Vehicles

Herein, the terms “unmanned aerial vehicle” and “UAV” refer to any autonomous or semi-autonomous vehicle that is capable of performing some functions without a physically present human pilot.

A UAV can take various forms. For example, a UAV may take the form of a fixed-wing aircraft, a glider aircraft, a tail-sitter aircraft, a jet aircraft, a ducted fan aircraft, a lighter-than-air dirigible such as a blimp or steerable balloon, a rotorcraft such as a helicopter or multicopter, and/or an ornithopter, among other possibilities. Further, the terms “drone,” “unmanned aerial vehicle system” (UAVS), or “unmanned aerial system” (UAS) may also be used to refer to a UAV.

FIG. 1 A is an isometric view of an example UAV 100 . UAV 100 includes wing 102 , booms 104 , and a fuselage 106 . Wings 102 may be stationary and may generate lift based on the wing shape and the UAV's forward airspeed. For instance, the two wings 102 may have an airfoil-shaped cross section to produce an aerodynamic force on UAV 100 . In some embodiments, wing 102 may carry horizontal propulsion units 108 , and booms 104 may carry vertical propulsion units 110 . In operation, power for the propulsion units may be provided from a battery compartment 112 of fuselage 106 . In some embodiments, fuselage 106 also includes an avionics compartment 114 , an additional battery compartment (not shown) and/or a delivery unit (not shown, e.g., a winch system) for handling the payload. In some embodiments, fuselage 106 is modular, and two or more compartments (e.g., battery compartment 112 , avionics compartment 114 , other payload and delivery compartments) are detachable from each other and securable to each other (e.g., mechanically, magnetically, or otherwise) to contiguously form at least a portion of fuselage 106 .

In some embodiments, booms 104 terminate in rudders 116 for improved yaw control of UAV 100 . Further, wings 102 may terminate in wing tips 117 for improved control of lift of the UAV.

In the illustrated configuration, <figure-callout id="100" label="UAV" filenames="US12307710-20250520-D00001.png" state

BACKGROUND

An unmanned vehicle, which may also be referred to as an autonomous vehicle, is a vehicle capable of travel without a physically-present human operator. An unmanned vehicle may operate in a remote-control mode, in an autonomous mode, or in a partially autonomous mode.

When an unmanned vehicle operates in a remote-control mode, a pilot or driver that is at a remote location can control the unmanned vehicle via commands that are sent to the unmanned vehicle via a wireless link. When the unmanned vehicle operates in autonomous mode, the unmanned vehicle typically moves based on pre-programmed navigation waypoints, dynamic automation systems, or a combination of these. Further, some unmanned vehicles can operate in both a remote-control mode and an autonomous mode, and in some instances may do so simultaneously. For instance, a remote pilot or driver may wish to leave navigation to an autonomous system while manually performing another task, such as operating a mechanical system for picking up objects, as an example.

Various types of unmanned vehicles exist for various different environments. For instance, unmanned vehicles exist for operation in the air, on the ground, underwater, and in space. Examples include quad-copters and tail-sitter UAVs, among others. Unmanned vehicles also exist for hybrid operations in which multi-environment operation is possible. Examples of hybrid unmanned vehicles include an amphibious craft that is capable of operation on land as well as on water or a floatplane that is capable of landing on water as well as on land. Other examples are also possible.

SUMMARY

The present embodiments are directed to systems and methods that may help to localize an unmanned aerial vehicle within an environment. A machine learning model may be trained to process two-dimensional (2D) images of a terrain within the environment. The model may include separate depth and semantics branches to produce both depth images and semantic images. The depth and semantic information may be used in combination to localize the vehicle in the environment relative to reference depth and semantic data.

In one aspect, a method is provided including receiving a two-dimensional (2D) image captured by a camera on a unmanned aerial vehicle (UAV) and representative of an environment of the UAV. The method further includes applying a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, where the machine learning model has been trained with a semantics branch to produce the semantic image and a depth branch to produce the depth image, and where the semantic image comprises one or more semantic labels. The method also includes retrieving reference depth data representative of the environment, where the reference depth data includes reference semantic labels. The method additionally includes aligning the depth image of the environment with the reference depth data representative of the environment to determine a location of the UAV in the environment, where the aligning associates the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data.

In another aspect an unmanned aerial vehicle is described, including a camera and a control system configured to receive a two-dimensional (2D) image captured by the camera and representative of an environment of the UAV; apply a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, where the machine learning model has been trained with a semantics branch to produce the semantic image and a depth branch to produce the depth image, and where the semantic image comprises one or more semantic labels; retrieve reference depth data representative of the environment, where the reference depth data includes reference semantic labels; and align the depth image of the environment with the reference depth data representative of the environment to determine a location of the UAV in the environment, where the aligning associates the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data.

In another aspect, a non-transitory computer readable medium is described including program instructions executable by one or more processors to perform operations. The operations include receiving a two-dimensional (2D) image captured by a camera on a unmanned aerial vehicle (UAV) and representative of an environment of the UAV; applying a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, where the machine learning model has been trained with a semantics branch to produce the semantic image and a depth branch to produce the depth image, and where the semantic image comprises one or more semantic labels; retrieving reference depth data representative of the environment, where the reference depth data includes reference semantic labels; and aligning the depth image of the environment with the reference depth data representative of the environment to determine a location of the UAV in the environment, where the aligning associates the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data.

These as well as other aspects, advantages, and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description with reference where appropriate to the accompanying drawings. Further, it should be understood that the description provided in this summary section and elsewhere in this document is intended to illustrate the claimed subject matter by way of example and not by way of limitation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 A is a simplified illustration of an unmanned aerial vehicle, according to an example embodiment.

FIG. 1 B is a simplified illustration of an unmanned aerial vehicle, according to an example embodiment.

FIG. 1 C is a simplified illustration of an unmanned aerial vehicle, according to an example embodiment.

FIG. 1 D is a simplified illustration of an unmanned aerial vehicle, according to an example embodiment.

FIG. 1 E is a simplified illustration of an unmanned aerial vehicle, according to an example embodiment.

FIG. 2 is a simplified block diagram illustrating components of an unmanned aerial vehicle, according to an example embodiment.

FIG. 3 is a simplified block diagram illustrating a UAV system, according to an example embodiment.

FIG. 4 is a diagram illustrating training and inference phases of a machine learning model, in accordance with example embodiments.

FIG. 5 is a diagram illustrating application of a machine learning model, in accordance with example embodiments.

FIG. 6 A illustrates an image captured by a UAV, in accordance with example embodiments.

FIG. 6 B illustrates reference depth data representative of an environment of a UAV, in accordance with example embodiments.

FIG. 6 C illustrates semantically labeled depth data generated by a machine learning model, in accordance with example embodiments.

FIG. 6 D illustrates superimposed reference depth data and semantically labeled depth data generated by a machine learning model, in accordance with example embodiments.

FIG. 6 E illustrates reference depth data aligned with semantically labeled depth data generated by a machine learning model, in accordance with example embodiments.

FIG. 7 is a block diagram of a method, in accordance with example embodiments.

DETAILED DESCRIPTION

Exemplary methods and systems are described herein. It should be understood that the word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any implementation or feature described herein as “exemplary” or “illustrative” is not necessarily to be construed as preferred or advantageous over other implementations or features. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example implementations described herein are not meant to be limiting. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

I. Overview

Determining the absolute location (longitude, latitude, and altitude) of a delivery drone or other unmanned aerial vehicle (UAV) is often critical for mission success. A common way of determining absolute location is to rely on a Global Navigation Satellite System (GNSS), such as a Global Positioning System (GPS). However, in some situations, GPS can become unavailable or the receiver can produce an erroneous location measurement. Consequently, the ability to derive an absolute location of a UAV using a different modality may serve as a complement and/or backup to GPS.

Examples described herein enable a UAV to use a machine-learned model to produce a semantically labeled three-dimensional (3D) point cloud (e.g., a 3D model of a scene). The semantically labeled 3D point cloud can then be registered against a pre-existing point cloud. In some examples, the pre-existing point cloud may be derived from a vendor or built during a preceding visit by the UAV and/or by other UAVs. Registering the semantically labeled 3D point cloud against the pre-existing point cloud may be performed to derive the vehicle&#39;s absolute position and orientation (e.g., to recover the vehicle&#39;s position and orientation with six degrees of freedom). In some examples, the absolute position can be used alone in the absence of GPS to complete a mission assigned to the UAV. In further examples, the absolute position can be used to cross-check GPS measurements and/or other sensor measurements. In yet further examples, the absolute position can be blended with the GPS measurement to arrive at a more certain belief of the UAV&#39;s whearabouts.

II. Illustrative Unmanned Vehicles

Herein, the terms “unmanned aerial vehicle” and “UAV” refer to any autonomous or semi-autonomous vehicle that is capable of performing some functions without a physically present human pilot.

A UAV can take various forms. For example, a UAV may take the form of a fixed-wing aircraft, a glider aircraft, a tail-sitter aircraft, a jet aircraft, a ducted fan aircraft, a lighter-than-air dirigible such as a blimp or steerable balloon, a rotorcraft such as a helicopter or multicopter, and/or an ornithopter, among other possibilities. Further, the terms “drone,” “unmanned aerial vehicle system” (UAVS), or “unmanned aerial system” (UAS) may also be used to refer to a UAV.

FIG. 1 A is an isometric view of an example UAV 100 . UAV 100 includes wing 102 , booms 104 , and a fuselage 106 . Wings 102 may be stationary and may generate lift based on the wing shape and the UAV&#39;s forward airspeed. For instance, the two wings 102 may have an airfoil-shaped cross section to produce an aerodynamic force on UAV 100 . In some embodiments, wing 102 may carry horizontal propulsion units 108 , and booms 104 may carry vertical propulsion units 110 . In operation, power for the propulsion units may be provided from a battery compartment 112 of fuselage 106 . In some embodiments, fuselage 106 also includes an avionics compartment 114 , an additional battery compartment (not shown) and/or a delivery unit (not shown, e.g., a winch system) for handling the payload. In some embodiments, fuselage 106 is modular, and two or more compartments (e.g., battery compartment 112 , avionics compartment 114 , other payload and delivery compartments) are detachable from each other and securable to each other (e.g., mechanically, magnetically, or otherwise) to contiguously form at least a portion of fuselage 106 .

In some embodiments, booms 104 terminate in rudders 116 for improved yaw control of UAV 100 . Further, wings 102 may terminate in wing tips 117 for improved control of lift of the UAV.

In the illustrated configuration, UAV 100 includes a structural frame. The structural frame may be referred to as a “structural H-frame” or an “H-frame” (not shown) of the UAV. The H-frame may include, within wings 102 , a wing spar (not shown) and, within booms 104 , boom carriers (not shown). In some embodiments the wing spar and the boom carriers may be made of carbon fiber, hard plastic, aluminum, light metal alloys, or other materials. The wing spar and the boom carriers may be connected with clamps. The wing spar may include pre-drilled holes for horizontal propulsion units 108 , and the boom carriers may include pre-drilled holes for vertical propulsion units 110 .

In some embodiments, fuselage 106 may be removably attached to the H-frame (e.g., attached to the wing spar by clamps, configured with grooves, protrusions or other features to mate with corresponding H-frame features, etc.). In other embodiments, fuselage 106 similarly may be removably attached to wings 102 . The removable attachment of fuselage 106 may improve quality and or modularity of UAV 100 . For example, electrical/mechanical components and/or subsystems of fuselage 106 may be tested separately from, and before being attached to, the H-frame. Similarly, printed circuit boards (PCBs) 118 may be tested separately from, and before being attached to, the boom carriers, therefore eliminating defective parts/subassemblies prior to completing the UAV. For example, components of fuselage 106 (e.g., avionics, battery unit, delivery units, an additional battery compartment, etc.) may be electrically tested before fuselage 106 is mounted to the H-frame. Furthermore, the motors and the electronics of PCBs 118 may also be electrically tested before the final assembly. Generally, the identification of the defective parts and subassemblies early in the assembly process lowers the overall cost and lead time of the UAV. Furthermore, different types/models of fuselage 106 may be attached to the H-frame, therefore improving the modularity of the design. Such modularity allows these various parts of UAV 100 to be upgraded without a substantial overhaul to the manufacturing process.

In some embodiments, a wing shell and boom shells may be attached to the H-frame by adhesive elements (e.g., adhesive tape, double-sided adhesive tape, glue, etc.). Therefore, multiple shells may be attached to the H-frame instead of having a monolithic body sprayed onto the H-frame. In some embodiments, the presence of the multiple shells reduces the stresses induced by the coefficient of thermal expansion of the structural frame of the UAV. As a result, the UAV may have better dimensional accuracy and/or improved reliability.

Moreover, in at least some embodiments, the same H-frame may be used with the wing shell and/or boom shells having different size and/or design, therefore improving the modularity and versatility of the UAV designs. The wing shell and/or the boom shells may be made of relatively light polymers (e.g., closed cell foam) covered by the harder, but relatively thin, plastic skins.

The power and/or control signals from fuselage 106 may be routed to PCBs 118 through cables running through fuselage 106 , wings 102 , and booms 104 . In the illustrated embodiment, UAV 100 has four PCBs, but other numbers of PCBs are also possible. For example, UAV 100 may include two PCBs, one per the boom. The PCBs carry electronic components 119 including, for example, power converters, controllers, memory, passive components, etc. In operation, propulsion units

108 and 110 of UAV 100 are electrically connected to the PCBs.

Many variations on the illustrated UAV are possible. For instance, fixed-wing UAVs may include more or fewer rotor units (vertical or horizontal), and/or may utilize a ducted fan or multiple ducted fans for propulsion. Further, UAVs with more wings (e.g., an “x-wing” configuration with four wings), are also possible. Although FIG. 1 illustrates two wings 102 , two booms 104 , two horizontal propulsion units 108 , and six vertical propulsion units 110 per boom 104 , it should be appreciated that other variants of UAV 100 may be implemented with more or less of these components. For example, UAV 100 may include four wings 102 , four booms 104 , and more or less propulsion units (horizontal or vertical).

Similarly, FIG. 1 B shows another example of a fixed- wing UAV 120 . The fixed- wing UAV 120 includes a fuselage 122 , two wings 124 with an airfoil-shaped cross section to provide lift for the UAV 120 , a vertical stabilizer 126 (or fin) to stabilize the plane&#39;s yaw (turn left or right), a horizontal stabilizer 128 (also referred to as an elevator or tailplane) to stabilize pitch (tilt up or down), landing gear 130 , and a propulsion unit 132 , which can include a motor, shaft, and propeller.

FIG. 1 C shows an example of a UAV 140 with a propeller in a pusher configuration. The term “pusher” refers to the fact that a propulsion unit 142 is mounted at the back of the UAV and “pushes” the vehicle forward, in contrast to the propulsion unit being mounted at the front of the UAV. Similar to the description provided for FIGS. 1 A and 1 B , FIG. 1 C depicts common structures used in a pusher plane, including a fuselage 144 , two wings 146 , vertical stabilizers 148 , and the propulsion unit 142 , which can include a motor, shaft, and propeller.

FIG. 1 D shows an example of a tail- sitter UAV 160 . In the illustrated example, the tail- sitter UAV 160 has fixed wings 162 to provide lift and allow the UAV 160 to glide horizontally (e.g., along the x-axis, in a position that is approximately perpendicular to the position shown in FIG. 1 D ). However, the fixed wings 162 also allow the tail- sitter UAV 160 to take off and land vertically on its own.

For example, at a launch site, the tail- sitter UAV 160 may be positioned vertically (as shown) with its fins 164 and/or wings 162 resting on the ground and stabilizing the UAV 160 in the vertical position. The tail- sitter UAV 160 may then take off by operating its propellers 166 to generate an upward thrust (e.g., a thrust that is generally along the y-axis). Once at a suitable altitude, the tail- sitter UAV 160 may use its flaps 168 to reorient itself in a horizontal position, such that its fuselage 170 is closer to being aligned with the x-axis than the y-axis. Positioned horizontally, the propellers 166 may provide forward thrust so that the tail- sitter UAV 160 can fly in a similar manner as a typical airplane.

Many variations on the illustrated fixed-wing UAVs are possible. For instance, fixed-wing UAVs may include more or fewer propellers, and/or may utilize a ducted fan or multiple ducted fans for propulsion. Further, UAVs with more wings (e.g., an “x-wing” configuration with four wings), with fewer wings, or even with no wings, are also possible.

As noted above, some embodiments may involve other types of UAVs, in addition to or in the alternative to fixed-wing UAVs. For instance, FIG. 1 E shows an example of a rotorcraft that is commonly referred to as a multicopter 180 . The multicopter 180 may also be referred to as a quadcopter, as it includes four rotors 182 . It should be understood that example embodiments may involve a rotorcraft with more or fewer rotors than the multicopter 180 . For example, a helicopter typically has two rotors. Other examples with three or more rotors are possible as well. Herein, the term “multicopter” refers to any rotorcraft having more than two rotors, and the term “helicopter” refers to rotorcraft having two rotors.

Referring to the multicopter 180 in greater detail, the four rotors 182 provide propulsion and maneuverability for the multicopter 180 . More specifically, each rotor 182 includes blades that are attached to a motor 184 . Configured as such, the rotors 182 may allow the multicopter 180 to take off and land vertically, to maneuver in any direction, and/or to hover. Further, the pitch of the blades may be adjusted as a group and/or differentially, and may allow the multicopter 180 to control its pitch, roll, yaw, and/or altitude.

It should be understood that references herein to an “unmanned” aerial vehicle or UAV can apply equally to autonomous and semi-autonomous aerial vehicles. In an autonomous implementation, all functionality of the aerial vehicle is automated; e.g., pre-programmed or controlled via real-time computer functionality that responds to input from various sensors and/or pre-determined information. In a semi-autonomous implementation, some functions of an aerial vehicle may be controlled by a human operator, while other functions are carried out autonomously. Further, in some embodiments, a UAV may be configured to allow a remote operator to take over functions that can otherwise be controlled autonomously by the UAV. Yet further, a given type of function may be controlled remotely at one level of abstraction and performed autonomously at another level of abstraction. For example, a remote operator could control high level navigation decisions for a UAV, such as by specifying that the UAV should travel from one location to another (e.g., from a warehouse in a suburban area to a delivery address in a nearby city), while the UAV&#39;s navigation system autonomously controls more fine-grained navigation decisions, such as the specific route to take between the two locations, specific flight controls to achieve the route and avoid obstacles while navigating the route, and so on.

More generally, it should be understood that the example UAVs described herein are not intended to be limiting. Example embodiments may relate to, be implemented within, or take the form of any type of unmanned aerial vehicle.

III. Illustrative UAV Components

FIG. 2 is a simplified block diagram illustrating components of a UAV 200 , according to an example embodiment. UAV 200 may take the form of, or be similar in form to, one of the

UAVs

100 , 120 , 140 , 160 , and 180 described in reference to FIGS. 1 A- 1 E . However, UAV 200 may also take other forms.

UAV 200 may include various types of sensors, and may include a computing system configured to provide the functionality described herein. In the illustrated embodiment, the sensors of UAV 200 include an inertial measurement unit (IMU) 202 , ultrasonic sensor(s) 204 , and a GPS 206 , among other possible sensors and sensing systems.

In the illustrated embodiment, UAV 200 also includes one or more processors 208 . A processor 208 may be a general-purpose processor or a special purpose processor (e.g., digital signal processors, application specific integrated circuits, etc.). The one or more processors 208 can be configured to execute computer- readable program instructions 212 that are stored in the data storage 210 and are executable to provide the functionality of a UAV described herein.

The data storage 210 may include or take the form of one or more computer-readable storage media that can be read or accessed by at least one processor 208 . The one or more computer-readable storage media can include volatile and/or non-volatile storage components, such as optical, magnetic, organic or other memory or disc storage, which can be integrated in whole or in part with at least one of the one or more processors 208 . In some embodiments, the data storage 210 can be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc storage unit), while in other embodiments, the data storage 210 can be implemented using two or more physical devices.

As noted, the data storage 210 can include computer- readable program instructions 212 and perhaps additional data, such as diagnostic data of the UAV 200 . As such, the data storage 210 may include program instructions 212 to perform or facilitate some or all of the UAV functionality described herein. For instance, in the illustrated embodiment, program instructions 212 include a navigation module 214 and a tether control module 216 .

A. Sensors

In an illustrative embodiment, IMU 202 may include both an accelerometer and a gyroscope, which may be used together to determine an orientation of the UAV 200 . In particular, the accelerometer can measure the orientation of the vehicle with respect to earth, while the gyroscope measures the rate of rotation around an axis. IMUs are commercially available in low-cost, low-power packages. For instance, an IMU 202 may take the form of or include a miniaturized MicroElectroMechanical System (MEMS) or a NanoElectroMechanical System (NEMS). Other types of IMUs may also be utilized.

An IMU 202 may include other sensors, in addition to accelerometers and gyroscopes, which may help to better determine position and/or help to increase autonomy of the UAV 200 . Two examples of such sensors are magnetometers and pressure sensors. In some embodiments, a UAV may include a low-power, digital 3-axis magnetometer, which can be used to realize an orientation independent electronic compass for accurate heading information. However, other types of magnetometers may be utilized as well. Other examples are also possible. Further, note that a UAV could include some or all of the above-described inertia sensors as separate components from an IMU.

UAV 200 may also include a pressure sensor or barometer, which can be used to determine the altitude of the UAV 200 . Alternatively, other sensors, such as sonic altimeters or radar altimeters, can be used to provide an indication of altitude, which may help to improve the accuracy of and/or prevent drift of an IMU.

In a further aspect, UAV 200 may include one or more sensors that allow the UAV to sense objects in the environment. For instance, in the illustrated embodiment, UAV 200 includes ultrasonic sensor(s) 204 . Ultrasonic sensor(s) 204 can determine the distance to an object by generating sound waves and determining the time interval between transmission of the wave and receiving the corresponding echo off an object. A typical application of an ultrasonic sensor for unmanned vehicles or IMUs is low-level altitude control and obstacle avoidance. An ultrasonic sensor can also be used for vehicles that need to hover at a certain height or need to be capable of detecting obstacles. Other systems can be used to determine, sense the presence of, and/or determine the distance to nearby objects, such as a light detection and ranging (LIDAR) system, laser detection and ranging (LADAR) system, and/or an infrared or forward-looking infrared (FLIR) system, among other possibilities.

In some embodiments, UAV 200 may also include one or more imaging system(s). For example, one or more still and/or video cameras may be utilized by UAV 200 to capture image data from the UAV&#39;s environment. As a specific example, charge-coupled device (CCD) cameras or complementary metal-oxide-semiconductor (CMOS) cameras can be used with unmanned vehicles. Such imaging sensor(s) have numerous possible applications, such as obstacle avoidance, localization techniques, ground tracking for more accurate navigation (e.g., by applying optical flow techniques to images), video feedback, and/or image recognition and processing, among other possibilities.

UAV 200 may also include a GPS receiver 206 . The GPS receiver 206 may be configured to provide data that is typical of well-known GPS systems, such as the GPS coordinates of the UAV 200 . Such GPS data may be utilized by the UAV 200 for various functions. As such, the UAV may use its GPS receiver 206 to help navigate to the caller&#39;s location, as indicated, at least in part, by the GPS coordinates provided by their mobile device. Other examples are also possible.

B. Navigation and Location Determination

The navigation module 214 may provide functionality that allows the UAV 200 to, e.g., move about its environment and reach a desired location. To do so, the navigation module 214 may control the altitude and/or direction of flight by controlling the mechanical features of the UAV that affect flight (e.g., its rudder(s), elevator(s), aileron(s), and/or the speed of its propeller(s)).

In order to navigate the UAV 200 to a target location, the navigation module 214 may implement various navigation techniques, such as map-based navigation and localization-based navigation, for instance. With map-based navigation, the UAV 200 may be provided with a map of its environment, which may then be used to navigate to a particular location on the map. With localization-based navigation, the UAV 200 may be capable of navigating in an unknown environment using localization. Localization-based navigation may involve the UAV 200 building its own map of its environment and calculating its position within the map and/or the position of objects in the environment. For example, as a UAV 200 moves throughout its environment, the UAV 200 may continuously use localization to update its map of the environment. This continuous mapping process may be referred to as simultaneous localization and mapping (SLAM). Other navigation techniques may also be utilized.

In some embodiments, the navigation module 214 may navigate using a technique that relies on waypoints. In particular, waypoints are sets of coordinates that identify points in physical space. For instance, an air-navigation waypoint may be defined by a certain latitude, longitude, and altitude. Accordingly, navigation module 214 may cause UAV 200 to move from waypoint to waypoint, in order to ultimately travel to a final destination (e.g., a final waypoint in a sequence of waypoints).

In a further aspect, the navigation module 214 and/or other components and systems of the UAV 200 may be configured for “localization” to more precisely navigate to the scene of a target location. More specifically, it may be desirable in certain situations for a UAV to be within a threshold distance of the target location where a payload 228 is being delivered by a UAV (e.g., within a few feet of the target destination). To this end, a UAV may use a two-tiered approach in which it uses a more-general location-determination technique to navigate to a general area that is associated with the target location, and then use a more-refined location-determination technique to identify and/or navigate to the target location within the general area.

For example, the UAV 200 may navigate to the general area of a target destination where a payload 228 is being delivered using waypoints and/or map-based navigation. The UAV may then switch to a mode in which it utilizes a localization process to locate and travel to a more specific location. For instance, if the UAV 200 is to deliver a payload to a user&#39;s home, the UAV 200 may need to be substantially close to the target location in order to avoid delivery of the payload to undesired areas (e.g., onto a roof, into a pool, onto a neighbor&#39;s property, etc.). However, a GPS signal may only get the UAV 200 so far (e.g., within a block of the user&#39;s home). A more precise location-determination technique may then be used to find the specific target location.

<div id="p-0063" nu

CLAIMS

Claims ( 19 )

What is claimed is:

1. A method comprising:

receiving a two-dimensional (2D) image captured by a camera on a unmanned aerial vehicle (UAV) and representative of an environment of the UAV;

applying a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, wherein the machine learning model has been trained with a semantics branch to produce the semantic image and a depth branch to produce the depth image, and wherein the semantic image comprises one or more semantic labels;

retrieving reference depth data representative of the environment, wherein the reference depth data includes reference semantic labels; and

determining a location of the UAV in the environment, wherein determining the location of the UAV in the environment comprises aligning the depth image of the environment with the reference depth data representative of the environment, wherein aligning the depth image of the environment with the reference depth data representative of the environment is based on associating the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data; and

controlling the UAV to navigate in the environment based on the determined location of the UAV in the environment.

2. The method of claim 1 , further comprising:

controlling the UAV to navigate in the environment using a Global Navigation Satellite System (GNSS) system;

detecting a disruption in service from the GNSS system, wherein the location of the UAV in the environment is determined responsive to detecting the disruption in service from the GNSS system; and

subsequent to detecting the disruption in service from the GNSS system, controlling the UAV to navigate in the environment based on the determined location of the UAV in the environment.

3. The method of claim 1 , further comprising:

controlling the UAV to navigate in the environment using a GNSS system; and

using the determined location of the UAV in the environment to cross-check location data from the GNSS system.

4. The method of claim 1 , further comprising:

determining a GNSS location of the UAV in the environment using a GNSS system;

determining a refined location of the UAV in the environment based on the GNSS location of the UAV in the environment and the determined location of the UAV in the environment; and

controlling the UAV to navigate in the environment based on the refined location of the UAV in the environment.

5. The method of claim 1 , wherein the aligning the depth image of the environment with the reference depth data representative of the environment to determine the location of the UAV in the environment comprises using an iterative closest point (ICP) algorithm.

6. The method of claim 5 , wherein the ICP algorithm aligns points from the reference depth data with points from the depth image such that the reference semantic labels from the reference depth data correspond to the one or more semantic labels from the semantic image.

7. The method of claim 1 , wherein the semantic image of the environment and the depth image of the environment produced by the machine learning model have the same dimensions.

8. The method of claim 1 , wherein the camera on the UAV faces downward, and wherein the 2D image captured by the camera is representative of a terrain in the environment below the UAV.

9. The method of claim 1 , wherein the semantics branch and the depth branch of the machine learning model operate on a commonly generated feature set.

10. The method of claim 1 , wherein the machine learning model has been trained based on ground truth depth data, wherein the ground truth depth data is based on performance of a structure from motion (SfM) algorithm on images captured by one or more UAVs.

11. The method of claim 1 , wherein the machine learning model has been trained based on ground truth semantic data, wherein the ground truth semantic data is based on operator labeling of images captured by one or more UAVs.

12. The method of claim 1 , wherein the machine learning model has been trained using a scale invariant loss for training of the depth branch.

13. The method of claim 12 , further comprising applying a scale factor to the depth image, wherein the scale factor comprises a ratio of an altitude of the UAV above ground level relative to a median of a monocular depth map, wherein the monocular depth map is based on the reference depth data.

14. The method of claim 12 , further comprising applying a scale factor to the depth image, wherein the scale factor comprises a ratio of an altitude of the UAV above ground level relative to an above ground level estimate from a monocular depth map, wherein the monocular depth map is based on the reference depth data.

15. The method of claim 1 , wherein the one or more semantic labels are selected from a predetermined set of labels, wherein the predetermined set of labels comprises at least the following labels: building, road, vegetation, vehicle, driveway, lawn, and sidewalk.

16. The method of claim 1 , further comprising retrieving the reference depth data in advance of a flight of the UAV, wherein the reference depth data is selected based on a planned flight path of the UAV.

17. The method of claim 1 , further comprising applying a Kalman filter to the determined location of the UAV in the environment to control navigation of the UAV in the environment.

18. An unmanned aerial vehicle (UAV), comprising:

a camera; and

a control system configured to:

receive a two-dimensional (2D) image captured by the camera and representative of an environment of the UAV;

apply a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, wherein the machine learning model has been trained with a semantics branch to produce the semantic image and a depth branch to produce the depth image, and wherein the semantic image comprises one or more semantic labels;

retrieve reference depth data representative of the environment, wherein the reference depth data includes reference semantic labels; and

determine a location of the UAV in the environment, wherein determining the location of the UAV in the environment comprises aligning the depth image of the environment with the reference depth data representative of the environment, wherein aligning the depth image of the environment with the reference depth data representative of the environment is based on associating the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data.

19. A non-transitory computer readable medium comprising program instructions executable by one or more processors to perform operations, the operations comprising:

receiving a two-dimensional (2D) image captured by a camera on a unmanned aerial vehicle (UAV) and representative of an environment of the UAV;

applying a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, wherein the machine learning model has been trained with a semantics branch to produce the semantic image and a depth branch to produce the depth image, and wherein the semantic image comprises one or more semantic labels;

retrieving reference depth data representative of the environment, wherein the reference depth data includes reference semantic labels; and

determining a location of the UAV in the environment, wherein determining the location of the UAV in the environment comprises aligning the depth image of the environment with the reference depth data representative of the environment, wherein aligning the depth image of the environment with the reference depth data representative of the environment is based on associating the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data; and

controlling the UAV to navigate in the environment based on the determined location of the UAV in the environment.

US17/813,188

2022-07-18

2022-07-18

Machine-learned monocular depth estimation and semantic segmentation for 6-DOF absolute localization of a delivery drone

Active

2043-04-29

US12307710B2

( en )

Priority Applications (5)

Application Number

Priority Date

Filing Date

Title

US17/813,188

US12307710B2

( en )

2022-07-18

2022-07-18

Machine-learned monocular depth estimation and semantic segmentation for 6-DOF absolute localization of a delivery drone

EP23843588.7A

EP4552080A4

( en )

2022-07-18

2023-07-17

MACHINE-LEVEL MONOCULAR DEEP ESTIMATION AND SEMANTIC SEGMENTATION FOR 6-DOF ABSOLUTE LOCALIZATION OF A DELIVERY DRONE

PCT/US2023/027913

WO2024019975A1

( en )

2022-07-18

2023-07-17

Machine-learned monocular depth estimation and semantic segmentation for 6-dof absolute localization of a delivery drone

AU2023308867A

AU2023308867A1

( en )

2022-07-18

2023-07-17

Machine-learned monocular depth estimation and semantic segmentation for 6-dof absolute localization of a delivery drone

US19/200,000

US20250265730A1

( en )

2022-07-18

2025-05-06

Machine-Learned Monocular Depth Estimation and Semantic Segmentation for 6-DOF Absolute Localization of a Delivery Drone

Applications Claiming Priority (1)

Application Number

Priority Date

Filing Date

Title

US17/813,188

US12307710B2

( en )

2022-07-18

2022-07-18

Machine-learned monocular depth estimation and semantic segmentation for 6-DOF absolute localization of a delivery drone

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US19/200,000

Continuation

US20250265730A1

( en )

2022-07-18

2025-05-06

Machine-Learned Monocular Depth Estimation and Semantic Segmentation for 6-DOF Absolute Localization of a Delivery Drone

Publications (2)

Publication Number

Publication Date

US20240020876A1

US20240020876A1 ( en )

2024-01-18

US12307710B2

true

US12307710B2 ( en )

2025-05-20

Family

ID=89510162

Family Applications (2)

Application Number

Title

Priority Date

Filing Date

US17/813,188

Active

2043-04-29

US12307710B2

( en )

2022-07-18

2022-07-18

Machine-learned monocular depth estimation and semantic segmentation for 6-DOF absolute localization of a delivery drone

US19/200,000

Pending

US20250265730A1

( en )

2022-07-18

2025-05-06

Machine-Learned Monocular Depth Estimation and Semantic Segmentation for 6-DOF Absolute Localization of a Delivery Drone

Family Applications After (1)

Application Number

Title

Priority Date

Filing Date

US19/200,000

Pending

US20250265730A1

( en )

2022-07-18

2025-05-06

Machine-Learned Monocular Depth Estimation and Semantic Segmentation for 6-DOF Absolute Localization of a Delivery Drone

Country Status (4)

Country

Link

US

( 2 )

US12307710B2

( en )

EP

( 1 )

EP4552080A4

( en )

AU

( 1 )

AU2023308867A1

( en )

WO

( 1 )

WO2024019975A1

( en )

Families Citing this family (12)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20250004468A1

( en )

*

2023-06-30

2025-01-02

Fca Us Llc

Method to control a vehicle as a function of data from a vehicle-based drone

US20250095156A1

( en )

*

2023-09-20

2025-03-20

Skydio, Inc.

Semantic Segmentation Rendering For Precise Localization Of Structure Components On Captured Images

US20250093882A1

( en )

*

2023-09-20

2025-03-20

Skydio, Inc.

Multi-Phase Semantic Three-Dimensional Scan For Structure Inspection

US20250094936A1

( en )

*

2023-09-20

2025-03-20

Skydio, Inc.

Unmanned Aerial Vehicle-Based Semantic Understanding For Structure Inspection

US20250218009A1

( en )

*

2023-12-27

2025-07-03

Faurecia Irystec Inc.

Automatic monocular depth perception calibration for camera

US12608931B2

( en )

2024-04-19

2026-04-21

Wing Aviation Llc

Generating aerial paths based on properties of aerial image data

USD1109672S1

( en )

*

2024-06-29

2026-01-20

Wing Aviation Llc

Uncrewed aerial vehicle

USD1105270S1

( en )

*

2024-08-07

2025-12-09

Guangdong Syma Model Aircraft Industrial Co., Ltd.

Toy aircraft

CN119025888A

( en )

*

2024-08-14

2024-11-26

浙江大学

A method for identifying unknown drone clusters based on signal semantic space construction

EP4737854A1

( en )

*

2024-10-31

2026-05-06

Helsing GmbH

Methods, systems, and computer program products for georeferencing

CN120334979B

( en )

*

2025-06-09

2025-08-22

太原理工大学

GNSS/vision loosely coupled positioning method integrating environmental semantics

CN120871631B

( en )

*

2025-09-23

2025-12-02

浙江孚临科技有限公司

A Multi-Agent Reinforcement Learning Method for Cooperative Decision-Making in UAV Swarms

Citations (24)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20120195471A1

( en )

*

2011-01-31

2012-08-02

Microsoft Corporation

Moving Object Segmentation Using Depth Images

US20130147911A1

( en )

*

2011-12-09

2013-06-13

Microsoft Corporation

Automatic 2d-to-stereoscopic video conversion

US9359074B2

( en )

2014-09-08

2016-06-07

Qualcomm Incorporated

Methods, systems and devices for delivery drone security

US9580173B1

( en )

2014-08-28

2017-02-28

X Development Llc

Translational correction of payload-release device based on tracked position

CN107444665A

( en )

2017-07-24

2017-12-08

长春草莓科技有限公司

A kind of unmanned plane Autonomous landing method

US20180101782A1

( en )

*

2016-10-06

2018-04-12

Gopro, Inc.

Systems and methods for determining predicted risk for a flight path of an unmanned aerial vehicle

US10049589B1

( en )

2016-09-08

2018-08-14

Amazon Technologies, Inc.

Obstacle awareness based guidance to clear landing space

US20180259960A1

( en )

2015-08-20

2018-09-13

Motionloft, Inc.

Object detection and analysis via unmanned aerial vehicle

US10353388B2

( en )

2016-10-17

2019-07-16

X Development Llc

Drop-off location planning for delivery vehicle

US20190286153A1

( en )

2018-03-15

2019-09-19

Nvidia Corporation

Determining drivable free-space for autonomous vehicles

US10642284B1

( en )

*

2018-06-08

2020-05-05

Amazon Technologies, Inc.

Location determination using ground structures

US10679509B1

( en )

2016-09-20

2020-06-09

Amazon Technologies, Inc.

Autonomous UAV obstacle avoidance using machine learning from piloted UAV flights

US20200241574A1

( en )

*

2019-01-30

2020-07-30

Adobe Inc.

Generalizable robot approach control techniques

US20210004976A1

( en )

*

2019-07-06

2021-01-07

Toyota Research Institute, Inc.

Systems and methods for semi-supervised training using reprojected distance loss

US20210041889A1

( en )

2019-08-08

2021-02-11

Pegatron Corporation

Semantic map orientation device and method, and robot

US20210150917A1

( en )

*

2019-11-20

2021-05-20

Wing Aviation Llc

Map including data for routing aerial vehicles during gnss failure

US20210174513A1

( en )

2019-12-09

2021-06-10

Naver Corporation

Method and apparatus for semantic segmentation and depth completion using a convolutional neural network

US20210224556A1

( en )

2018-02-27

2021-07-22

Nvidia Corporation

Real-time detection of lanes and boundaries by autonomous vehicles

US20210240195A1

( en )

*

2020-01-31

2021-08-05

Amazon Technologies, Inc.

Systems and methods for utilizing images to determine the position and orientation of a vehicle

US11195011B2

( en )

2017-09-21

2021-12-07

Amazon Technologies, Inc.

Object detection and avoidance for aerial vehicles

US20220026918A1

( en )

2020-07-23

2022-01-27

Toyota Research Institute, Inc.

System and method for monocular depth estimation from semantic information

US20220111869A1

( en )

2020-10-08

2022-04-14

Nec Laboratories America, Inc.

Indoor scene understanding from single-perspective images

US20220343521A1

( en )

*

2022-02-25

2022-10-27

Intel Corporation

Methods and apparatus for metric depth estimation using a monocular visual-inertial system

US20230406536A1

( en )

*

2020-11-13

2023-12-21

Mitsubishi Heavy Industries, Ltd.

Aircraft position control system, aircraft, and aircraft position control method

Family Cites Families (6)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US11295458B2

( en )

*

2016-12-01

2022-04-05

Skydio, Inc.

Object tracking by an unmanned aerial vehicle using visual sensors

US10387727B2

( en )

*

2017-09-13

2019-08-20

Wing Aviation Llc

Backup navigation system for unmanned aerial vehicles

US10685229B2

( en )

*

2017-12-21

2020-06-16

Wing Aviation Llc

Image based localization for unmanned aerial vehicles, and associated systems and methods

US11999372B2

( en )

*

2019-09-12

2024-06-04

Motional Ad Llc

Operation of an autonomous vehicle based on availability of navigational information

US11321863B2

( en )

*

2019-09-23

2022-05-03

Toyota Research Institute, Inc.

Systems and methods for depth estimation using semantic features

CA3158124A1

( en )

*

2019-10-15

2021-04-22

Alarm.Com Incorporated

Navigation using selected visual landmarks

2022

2022-07-18

US

US17/813,188

patent/US12307710B2/en

active

Active

2023

2023-07-17

EP

EP23843588.7A

patent/EP4552080A4/en

active

Pending

2023-07-17

WO

PCT/US2023/027913

patent/WO2024019975A1/en

not_active

Ceased

2023-07-17

AU

AU2023308867A

patent/AU2023308867A1/en

active

Pending

2025

2025-05-06

US

US19/200,000

patent/US20250265730A1/en

active

Pending

Patent Citations (24)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20120195471A1

( en )

*

2011-01-31

2012-08-02

Microsoft Corporation

Moving Object Segmentation Using Depth Images

US20130147911A1

( en )

*

2011-12-09

2013-06-13

Microsoft Corporation

Automatic 2d-to-stereoscopic video conversion

US9580173B1

( en )

2014-08-28

2017-02-28

X Development Llc

Translational correction of payload-release device based on tracked position

US9359074B2

( en )

2014-09-08

2016-06-07

Qualcomm Incorporated

Methods, systems and devices for delivery drone security

US20180259960A1

( en )

2015-08-20

2018-09-13

Motionloft, Inc.

Object detection and analysis via unmanned aerial vehicle

US10049589B1

( en )

2016-09-08

2018-08-14

Amazon Technologies, Inc.

Obstacle awareness based guidance to clear landing space

US10679509B1

( en )

2016-09-20

2020-06-09

Amazon Technologies, Inc.

Autonomous UAV obstacle avoidance using machine learning from piloted UAV flights

US20180101782A1

( en )

*

2016-10-06

2018-04-12

Gopro, Inc.

Systems and methods for determining predicted risk for a flight path of an unmanned aerial vehicle

US10353388B2

( en )

2016-10-17

2019-07-16

X Development Llc

Drop-off location planning for delivery vehicle

CN107444665A

( en )

2017-07-24

2017-12-08

长春草莓科技有限公司

A kind of unmanned plane Autonomous landing method

US11195011B2

( en )

2017-09-21

2021-12-07

Amazon Technologies, Inc.

Object detection and avoidance for aerial vehicles

US20210224556A1

( en )

2018-02-27

2021-07-22

Nvidia Corporation

Real-time detection of lanes and boundaries by autonomous vehicles

US20190286153A1

( en )

2018-03-15

2019-09-19

Nvidia Corporation

Determining drivable free-space for autonomous vehicles

US10642284B1

( en )

*

2018-06-08

2020-05-05

Amazon Technologies, Inc.

Location determination using ground structures

US20200241574A1

( en )

*

2019-01-30

2020-07-30

Adobe Inc.

Generalizable robot approach control techniques

US20210004976A1

( en )

*

2019-07-06

2021-01-07

Toyota Research Institute, Inc.

Systems and methods for semi-supervised training using reprojected distance loss

US20210041889A1

( en )

2019-08-08

2021-02-11

Pegatron Corporation

Semantic map orientation device and method, and robot

US20210150917A1

( en )

*

2019-11-20

2021-05-20

Wing Aviation Llc

Map including data for routing aerial vehicles during gnss failure

US20210174513A1

( en )

2019-12-09

2021-06-10

Naver Corporation

Method and apparatus for semantic segmentation and depth completion using a convolutional neural network

US20210240195A1

( en )

*

2020-01-31

2021-08-05

Amazon Technologies, Inc.

Systems and methods for utilizing images to determine the position and orientation of a vehicle

US20220026918A1

( en )

2020-07-23

2022-01-27

Toyota Research Institute, Inc.

System and method for monocular depth estimation from semantic information

US20220111869A1

( en )

2020-10-08

2022-04-14

Nec Laboratories America, Inc.

Indoor scene understanding from single-perspective images

US20230406536A1

( en )

*

2020-11-13

2023-12-21

Mitsubishi Heavy Industries, Ltd.

Aircraft position control system, aircraft, and aircraft position control method

US20220343521A1

( en )

*

2022-02-25

2022-10-27

Intel Corporation

Methods and apparatus for metric depth estimation using a monocular visual-inertial system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party

Title

Guizilini et al., " Semantically-Guided Representation Learning for Self-Supervised Monocular Depth, " arXiv:2002.12319v1, ICLR 2020, 14 pages.

Parkison et al., " Semantic Iterative Closest Point through Expectation-Maximization, " Proceedings of the British Machine Vision Conference, 2018, 18 pages.

Also Published As

Publication number

Publication date

US20250265730A1

( en )

2025-08-21

WO2024019975A1

( en )

2024-01-25

EP4552080A1

( en )

2025-05-14

AU2023308867A1

( en )

2025-03-06

US20240020876A1

( en )

2024-01-18

EP4552080A4

( en )

2026-03-18

Similar Documents

Publication

Publication Date

Title

US20250265730A1

( en )

2025-08-21

Machine-Learned Monocular Depth Estimation and Semantic Segmentation for 6-DOF Absolute Localization of a Delivery Drone

US10353388B2

( en )

2019-07-16

Drop-off location planning for delivery vehicle

US12602924B2

( en )

2026-04-14

Method for semantic localization of an unmanned aerial vehicle

EP3682192B1

( en )

2026-04-15

A backup navigation system for unmanned aerial vehicles

US12283099B2

( en )

2025-04-22

Semantic abort of unmanned aerial vehicle deliveries

US12221211B2

( en )

2025-02-11

Semantic adjustment of unmanned aerial vehicle delivery points

US12050475B2

( en )

2024-07-30

Unmanned aerial vehicle trajectories for nudging and un-nudging

US12608931B2

( en )

2026-04-21

Generating aerial paths based on properties of aerial image data

US20250376259A1

( en )

2025-12-11

Automatic Selection of Delivery Zones Using Survey Flight 3D Scene Reconstructions

AU2022203829B2

( en )

2024-04-18

Stereo abort of unmanned aerial vehicle deliveries

US12148206B2

( en )

2024-11-19

Method for controlling an unmanned aerial vehicle to avoid obstacles

WO2025221369A2

( en )

2025-10-23

Image-derived text delivery location descriptions

US12164308B2

( en )

2024-12-10

Tether-based wind estimation

Legal Events

Date

Code

Title

Description

2022-07-18

FEPP

Fee payment procedure

Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

2022-07-19

AS

Assignment

Owner name : WING AVIATION LLC, CALIFORNIA

Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SHOEB, ALI;REEL/FRAME:060552/0340

Effective date : 20220628

2022-11-23

STPP

Information on status: patent application and granting procedure in general

Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION

2024-07-29

STPP

Information on status: patent application and granting procedure in general

Free format text : NON FINAL ACTION MAILED

2024-12-08

STPP

Information on status: patent application and granting procedure in general

Free format text : RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

2025-01-22

STPP

Information on status: patent application and granting procedure in general

Free format text : NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

2025-05-07

STCF

Information on status: patent grant

Free format text : PATENTED CASE

Related documents

• Forest Inspection Dataset for Aerial Semantic Segmentation and Depth Estimation#1342282
• SemanticSeg4EO: An open-source unified framework and QGIS plugin for semantic segmentation in Earth Observation#229860
• ImageNet3D: Towards General-Purpose Object-Level 3D Understanding.#368057
• DeFM: Learning Foundation Representations from Depth for Robotics#1297026
• HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis#1081342