Conceptio › Archive › Google Patents

Google Patents open access

Mixed-reality visor for in-situ vehicular operations training — Wael Zohni (US12112440B2)

Wael Zohni · Google Patents

Google Patents · Patents

en google-patents intellectual-property patent

and a computer vision enabled three-dimensional point-cloud rendering subsystem configured to identify targeted regions of interest, the computer vision enabled three-dimensional point-cloud rendering subsystem comprising: at least one camera with a focal length positioned to provide binocular … Mixed-reality visor for in-situ vehicular operations training — Wael Zohni (US12112440B2)

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

ABSTRACT

Abstract

A Mixed-Reality visor (MR-visor) system and method utilizing environmental sensor feedback for replicating restricted external visibility during operation of manned vehicles, such as marine or aircraft. Adaptive hardware and software enable the user to reliably limit, modify and/or block views outside window(s) areas of the vehicle while maintaining visibility of the cabin interior and instrument control panel(s) without need for complex mechanical hardware alignment and setup. In the case of aircraft pilot training, the MR-visor can be worn by a pilot to replicate Instrument Meteorological Conditions (IMC) and other challenging scenarios.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is an U.S. National Phase Application under 35 U.S.C. 371 of PCT International Application No. PCT/US2021/064043, filed Dec. 17, 2021, which claims the benefit of the following: U.S. Provisional Application No. 63/128,163, filed on Dec. 20, 2020; U.S. Provisional Application No. 63/180,040, filed on Apr. 26, 2021; and U.S. Provisional Application No. 63/190,138, filed on May 18, 2021. The entire disclosures of each of the above applications are incorporated herein by reference.

FIELD

The present disclosure relates to mixed reality devices and, more particularly, relates to a mixed-reality visor device for selective control of a user's field of view.

BACKGROUND AND SUMMARY

This section provides background information related to the present disclosure, which is not necessarily prior art. This section provides a general summary of the disclosure and is not a comprehensive disclosure of its full scope or all of its features.

According to the principles of the present teachings, systems and methods are provided for modifying a view perceived by a user who is substantially contained within an enclosure. The present systems and methods provide benefits and applications in a wide variety of industries, activities, environments, and situations. In the interest of providing a robust disclosure illustrative of the unique contributions to the art, however, the present disclosure will be provided in connection with aircraft flight training applications. This description should not be regarded as limiting the potential uses, benefits, and/or claims, unless specifically stated.

In some embodiments according to the principles of the present teachings, a system is provided including the following: a view-blocking wearable user visor-headset having a display surface and see-through camera; a distinguishing system configured to detect regions corresponding to an exterior of the enclosure from regions contained within an interior of the enclosure and output a region signal; and a vision system configured to overlay imagery graphics upon the display surface of the view-blocking wearable user visor-headset based on the region signal. Details relating thereto will be provided herein.

Generally, according to the principles of the present teachings, a mixed reality device is provided that is to be worn by a user in the area of flight training, particularly preparation and certification for flight in reduced visibility conditions resulting from clouds, fog, haze, smoke, or other adverse weather conditions or lack of sunlight (night operations).

By way of non-limiting example, pilots completing basic flight training are initially qualified to fly only under conditions permitting substantial visibility outside the cockpit so that aircraft orientation relative to the ground or distant horizon is easily viewed. Having a visible ground reference enables the pilot to both control the aircraft and visually see obstructions and other air traffic by pilot. This initial condition or restriction of qualification is termed Visual Flight Rules (VFR) by the U.S. Federal Aviation Administration (FAA). In order to fly when visibility is restricted, such as by clouds or fog, a pilot must demonstrate proficiency at maintaining flight control with reference only the instrument panel; this is termed as flight under Instrument Flight Rules (IFR) and requires additional training and certification.

The FAA defines weather-related flight conditions for VFR and IFR in terms of specific values for cloud ceiling and visibility. U.S. Federal Regulations for VFR require a ceiling greater than 3,000 feet above-ground-level (AGL) and horizontal visibility of greater than 3 miles in most airspace (i.e., visual meteorological conditions (VMC)). VFR establishes that VMC is sufficient for pilots to visually maintain separation from clouds and other aircraft. When weather conditions or other factors limit or reduce visibility and/or cloud ceilings below VMC, then these conditions are generally referred to as instrument meteorological conditions (IMC) and require a pilot to fly under Instrument Flight Rules (IFR). By way of example, IMC may exist when cloud ceilings drop to less than 1,000 feet above ground level (AGL) and/or horizontal visibility reduces to less than 3 miles.

Due to these reduced weather conditions or other factors that can result in pilot disorientation, a pilot trainee or pilot must complete specialized training in order to fly under IFR conditions because there may be little to no outward visibility from the cockpit to the exterior environment. Such training includes receiving specialized instruction from a certified flight instructor to simulate conditions where visibility outside the aircraft is limited. This is typically accomplished by the pilot trainee or pilot wearing simple view-limiting devices (VLDs), such as goggles, hoods, or visors (see FIG. 1 A ) that obstruct the pilot trainee or pilot's view outside the cabin or aircraft. Therefore, the pilot trainee or pilot must utilize only flight instruments (such as an artificial horizon, airspeed indicator, and other gauges) to maintain controlled flight and navigation. An accompanying flight instructor supervises the pilot trainee or pilot operating while âunder the hoodâ in order to simulate IMC, although the flight may actually be taking place under actual VMC. This artificial view restriction facilitates IFR training irrespective of actual weather conditions thus facilitating more rapid training.

There are a number of relevant points regarding IFR vs VFR flight. For example, IFR flying challenges pilots with multi-tasking as they visually scan an array of instruments monitoring everything from equipment status to aircraft orientation to area navigation. Task-saturation occurs when the pilot becomes overwhelmed with information and can no longer keep up with flying the aircraft. Saturation may result from unexpected events such as equipment failures or inadvertent flight into compromised weather conditions. Such disorientation or confusion has led to loss of control accidents. It is therefore important that both new and veteran IFR pilots establish and maintain a high level of proficiency in IFR flying.

Visual Meteorological Conditions (VMC) generally require 3 statute miles visibility with aircraft remaining clear of clouds at a minimum of 500 feet below, 1000 feet above, and 2000 feet horizontally. These minimums may increase or decrease slightly based on the type of controlled airspace (near vs away from an airport for example). VMC is a regulatory prerequisite of VFR flying.

Separate from the aforementioned discussion, Mixed-Reality (MR)ânot to be confused with Virtual-Reality (VR) or Augmented-Reality (AR)âis an interactive experience where computer-generated perceptual information is super-imposed on a predominantly real-world environment. MR can be defined as a system that fulfills three basic features: a combination of real and virtual worlds, real-time interaction, and accurate three-dimensional (3D) registration of virtual and real objects. The overlaid sensory information can be constructive (i.e., additive to the natural environment), or destructive (i.e., masking of the natural environment). This experience is commonly implemented in the form of specialized goggle or visor hardware that the user wears to seamlessly interweave the real physical world with elements of computer-generated imagery. In this way, mixed reality only modifies a user's perception of a chiefly real-world environment, whereas virtual reality completely replaces the real-world environment with a simulated one.

The primary value of mixed reality is the way components of the digital world blend into a person's perception of the real world, not as a simple display of data, but through the integration of immersive sensations, which are perceived as natural parts of an environment. Commercial mixed reality experiences have been largely limited to entertainment and gaming businesses with some industrial applications in medicine and other areas.

Augmented Reality (AR) is associated with visors designed to project generated digital imagery upon a clear, see-through lens that permits users to directly view the remaining natural environment. Because a clear lens is essentially used as a computer screen in this case, the associated digital imaging overlay is characteristically translucent such as with a Heads-Up-Display (HUD) and therefore cannot be used to, as effectively, fully block a user's view of surroundings. For example, AR applications typically generate text data overlays to a work environment such as during medical procedures where a surgeon prefers not to look away from the patient for any duration.

Widespread commercial use of MR technology for IFR flight training has not been pursued due in part to complexities involved with processing a dynamic environment such as an aircraft cockpit during flight operations. The present teachings describe materials and methods that enable implementation of streamlined MR hardware and software that offers improved cost-effectiveness, safety and quality of training.

Conventional IFR training employs long-standing View Limiting Devices (VLDs) to block views outside the aircraft's windows. Aircraft cockpit windows are typically placed above and to the sides of an instrument gauge panel. Industry standard VLD goggles are correspondingly shaped like blinders with opaque surfaces that inhibit views beyond the instrument panel. These IFR goggles, visor, or âhoodsâ are usually constructed from inexpensive plastic and are head-mounted using elastic or fabric straps. Some common types available to date are illustrated in FIGS. 1 A- 2 B and terminology is used interchangeably herein.

An accompanying flight instructor or safety pilot supervises the student wearing the visor or goggles to ensure it is worn properly while also monitoring aircraft motion and orientation with reference to external views. Such partial view blocking visors or goggles are also used during practical flight tests where a candidate is required to demonstrate proficiency in IFR flight to an FAA examiner.

Being essentially blinders, conventional VLDs pose shortcomings in effectively replicating IFR conditions. Often the fit and positioning of the formed view-blocking areas do not conform well to the span of the instrument panel and user's height, requiring the pilot to maintain an unnatural and uncomfortable head-down position to prevent view of the aircraft exterior. Such head repositioning has a direct effect on how aircraft motion is sensed and interpreted by the user thus presents potentially dissimilar effects to those that would be experienced under real IFR conditions. Furthermore, aircraft movements due to turbulence or maneuvering may cause inadvertent head movements that momentarily expose an exterior view to the user. Such glances, however brief, can provide enough information to reorient the pilot user hence diminishing value of the training session. VLDs also do not offer the capability to impose more complex IFR scenarios such as sudden transitions from clear to obscure weather conditions. One of the major risk factors with flight safety is inadvertent flight into IMC such as clouds during night flying. In such cases there is a surprise factor that makes maintaining proper aircraft control a challenge. VLDs are worn and removed deliberately therefore do not offer possibility for replicating sudden and unintended flight into IFR conditions. Nor do they offer methods for gradual changes in exterior visibility.

The present teachings provide numerous advantages. For example, the present teachings provide improved safety, efficiency, and effectiveness of training for vehicular operations during adverse conditions such as poor visibility due to fog or rain. In the case of aircraft, particularly small general aviation aircraft, serious accidents resulting from pilots inadvertently flying from clear weather (VMC) into inclement weather (IFR or IMC) unfortunately continue to occur on a regular basis despite increased training and awareness. Such accidents frequently result in a loss of control of the aircraft or controlled flight into high-elevation terrain such as mountains or high-rise objects. Oftentimes, even experienced IFR-rated pilots encounter mishaps in IMC due to lapses in judgement and eroded skills. The rate of these loss of control in IMC accidents continue to be of concern to the FAA and general aviation community.

A recognized contributor to these weather-related accidents is a lack of adequate primary or recurrent IFR flight training. Much of this training takes place in ground-based flight simulators or employ the use of VLD hoods or goggles to simulate instrument conditions during actual flight. These simple tools offer limited realism in terms of replicating instrument meteorological conditions as well as limited control over simulated training conditions. For example, although ground-based flight simulators used in primary flight training can block cockpit exterior views as desired, they typically do not incorporate motion; a major factor contributing to loss of spatial orientation leading to loss of aircraft control. Real life instrument flight conditions remove visual reference to the earth's horizon, which normally provides a means for the pilot to maintain orientation and aircraft control. Losing this visual reference may lead to misinterpretation of aircraft movements leading to pilot disorientation and subsequent loss of aircraft control.

In the case of actual flight with conventional view-limiting devices such as head-mounted visors or goggles, variations in the wearer's height, external lighting, and movements of the aircraft due to turbulence or maneuvering may unintentionally offer momentary glimpses of the aircraft exterior sufficient to reorient the pilot trainee. These unintended breaks in blocked visibility detract from the difficulty of practice conditions so can lead to significant deficiencies in skill over time. Furthermore, trainees need to apply conventional IFR hoods or visors manually for IFR-training phases of flight, which removes the element of surprise that often accompanies actual encounters with IMC such as inadvertent flight into clouds. Pilots accidentally flying into IMC can experience significant anxiety and disorientation due to the sudden loss of outside visual reference combined with abrupt aircraft movements associated with turbulence and other convective activity associated with inclement weather.

An object of the present teachings is enablement of simplified integration of visualization control offered by computer-based simulation with real life training conditions via a novel mixed-reality (MR) system and method. In some embodiments, the system is provided having an MR-visor headset worn by a pilot-user during actual IFR in-flight training. In some embodiments, the system utilizes a built-in viewer, internal- and external-radiation energy sources and sensors such that the user's view outside cockpit windows can be easily set and controlled during any phase of flight. In this way, the trainee can be subject to obstructed or altered views outside the cockpit regardless of head position and at the discretion of a flight instructor or examiner. An MR-visor for IFR offers a level of realism and control well beyond simple conventional VLD headwear used to date. Enhanced realism during IFR training can better prepare new instrument pilots, help maintain proficiency with experienced IFR-rated pilots, and provide flight examiners more rigorous methods for assessing a candidate's capabilities.

Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

DRAWINGS

The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

FIG. 1 A is an image of a virtual reality headset representing existing art in the field of invention.

FIG. 1 B is a drawing of augmented-reality glasses representing existing art in the field of invention.

FIG. 2 A is an image of conventional hood-style vision blockers commonly used for Instrument-Flight-Rules (IFR) flight training and exa

CROSS-REFERENCE TO RELATED APPLICATIONS

FIELD

The present disclosure relates to mixed reality devices and, more particularly, relates to a mixed-reality visor device for selective control of a user's field of view.

BACKGROUND AND SUMMARY

DRAWINGS

The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

FIG. 1 A is an image of a virtual reality headset representing existing art in the field of invention.

FIG. 1 B is a drawing of augmented-reality glasses representing existing art in the field of invention.

FIG. 2 A is an image of conventional hood-style vision blockers commonly used for Instrument-Flight-Rules (IFR) flight training and examination.

FIG. 2 B is an image of conventional visor-style vision blockers commonly used for IFR flight training and examination.

FIG. 3 is a perspective view from inside an aircraft cockpit displaying the interior instrument panel and an unobstructed forward view of the exterior environment.

FIG. 4 A is a comparative perspective view from inside an aircraft cockpit displaying the interior instrument panel and an unobstructed forward window view of the exterior environment.

FIG. 4 B is a comparative perspective view from inside an aircraft cockpit displaying the interior instrument panel and a fully obstructed forward window view of the exterior environment.

FIG. 5 is a diagram describing basic elements of the mixed-reality visor (MR-visor) headset provided by the present disclosure.

FIG. 6 illustrates core components of the MR-visor headset system according to some aspects of the present disclosure.

FIG. 7 A illustrates a variation of the MR-visor headset of the present disclosure employing multiple fixed-position lenses corresponding to different focal lengths.

FIG. 7 B illustrates a variation of the MR-visor headset of the present disclosure utilizing multiple moveable optical lenses that can be mechanically cycled via a rotating barrel assembly to provide different focal length views to user.

FIGS. 8 A and 8 B illustrates use of a forward-looking time-of-flight (ToF) depth camera where continuous measurement of forward objects is used to adjust lens focal length.

FIG. 9 illustrates a flow diagram describing methodology for implementing adaptive focus capability to MR-visor headset see-through cameras.

FIG. 10 illustrates a cross-section view of a small fixed-wing aircraft forward section showing relative locations of user and environmental lighting effects.

FIG. 11 illustrates a methodology flow diagram for tracking and pattern recognition of viewed objects.

FIG. 12 A illustrates an aircraft pilot's view of the interior cabin and forward exterior during Visual-Flight-Rules (VFR) conditions.

FIG. 12 B illustrates the same perspective view as FIG. 12 A , wherein views of regions contained by the aircraft window are modified by a computer-generated mixed-reality (MR) overlay.

FIG. 13 illustrates a window overlay graphics region being adjusted by user hand-gestures sensed and tracked by the MR-visor headset of the present disclosure.

FIG. 14 illustrates data input and output to a computer vision hardware and software controller generating overlay graphics for the MR-visor headset of the present disclosure.

FIG. 15 illustrates a methodology flow diagram for initialization and calibration set up for mixed-reality image overlay.

FIG. 16 illustrates the freedom of movement provided by MR-visor headset of the present disclosure.

FIG. 17 illustrates pattern-recognized overlay regions corresponding to an aircraft's windows upon which mixed-reality graphics can be superimposed to the user's perspective.

FIG. 18 illustrates an exemplary overlayed graphics of external objects that can be generated by the mixed-reality controller of the present disclosure within the exterior view regions of an aircraft cockpit.

FIG. 19 illustrates an example overlayed graphics of features that can be generated by the mixed-reality controller of the present disclosure within the interior view regions of an aircraft cockpit.

FIG. 20 illustrates electromagnetic energy emitters affixed to an aircraft exterior providing supplemental location data inputs to the MR-visor headset controller.

FIG. 21 illustrates electromagnetic energy emitters affixed to the backside of an aircraft interior providing supplemental location data to the MR-visor headset controller.

FIG. 22 illustrates electromagnetic energy emitters affixed to the top of an aircraft interior providing supplemental location data to the MR-visor headset controller.

FIG. 23 illustrates electromagnetic energy emitters affixed to the side console of an aircraft interior providing supplemental location data to the MR-visor headset controller.

FIG. 24 illustrates a location registration device affixed to the topside of an aircraft instrument panel.

FIG. 25 provides a diagram of an MR-visor headset in accordance with the present disclosure employing time-of-flight (ToF) infra-red (IR) light emitter distance measuring equipment.

FIG. 26 illustrates a console-mounted stereo camera that maintains positional data on an aircraft flight instrument panel.

FIG. 27 illustrates a three-dimensional (3D) mixed-reality graphic image generated from the stereo camera ( FIG. 26 ) replicating at least portions of the aircraft instrument panel and windshield regions.

FIG. 28 illustrates an embodiment of the MR-visor headset including inward-facing cameras for capturing user eye pupil movements and gaze-tracking.

FIG. 29 illustrates a methodology flow diagram for generating and collecting instant gaze tracking of multiple Areas-of-Interest (AOI) by the user.

FIG. 30 illustrates multiple inward-facing RGB video cameras that capture the MR-visor headset user's facial expressions.

FIG. 31 illustrates sensors embedded in the MR-visor that track and record physiological parameters of the user.

FIG. 32 illustrates an MR-visor with embedded audio speakers located near the user's ears.

FIG. 33 illustrates an MR-visor with onboard Global-Positioning-System (GPS) tracking, data-input and -output device.

FIG. 34 illustrates a variant of the MR-visor employing a forward-facing transparent Liquid-Crystal-Display (LCD) or equivalent see-through display screen that permits direct view of the user's surroundings.

FIG. 35 illustrates a transparent display variant similar to FIG. 34 wherein additional emitters located within the visor direct software-controlled lighting effects towards the user's eyes.

FIG. 36 illustrates a system-level overview of a visor pose localization process in accordance with the present disclosure.

Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference to the accompanying drawings.

Prior Art: VR-, AR-, and MR-Reality Technology

FIG. 1 A is an image of a virtual-reality (VR) visor-headset representing relevant existing art in the field of invention. VR headsets like commercial gaming platforms Oculus Riftâ¢ and Sony PlayStationâ¢ VR employ head mounted displays (HMDs) to generate what appears to be life size, fully immersive, three-dimensional (3D) virtual environments. VR headsets are worn like goggles and incorporate image feeds into individual LCD displays and lenses (one per eye) thereby providing a stereoscopic 3D image of an artificial computer-generated environment. The VR headset allows the user to move his/her head to view these virtual surroundings within a predefined field of view typically from 110 degrees to full 360 degrees rotation. The computer vision control generating the associated moving or still images require a minimum frame rate 60 frames per second (fps) but are more typically capable or 90 or 120 fps in order to provide a more realistic perspective to the user. The VR headset hardware and software implement head tracking to maintain the proper location of image features relative to the user's perception. Head tracking utilizes a 6DoF (six degrees of freedom) system that plots the location of the user's head in a virtual coordinate plan so that head motions forward and backwards, side to side and shoulder to shoulder, otherwise known as pitch, yaw and roll can be registered in real time. Some of the internal components used in a head-tracking system include built-in gyroscope, accelerometer and a magnetometer but may also incorporate an external camera monitoring locator signals from the headset. Head tracking computing needs to be low latency in order to keep up with user head movements. Most conventional VR headsets can achieve latencies of 50 milliseconds or less. In addition to head tracking, VR headsets may integrate motion and eye tracking of the user through extended internal and external hardware in order to further enhance realism and the user experience.

FIG. 1 B is a diagram of augmented reality (AR) glasses, representing other existing art in the field of invention. Whereas VR headsets provide a fully fabricated 3D virtual environment, AR instead provides an overlay of virtual elements onto actual surroundings. A commercial example of AR-glasses was provided by Google's Glassâ¢ product released in 2013. This head-mounted hardware was worn similarly to conventional reading glasses. Computer-generated images are projected onto the clear lens positioned in front of the user's eyes so that data can be superimposed onto actual surroundings. Industrial applications for AR to date have concentrated on specialized industrial users such as surgical medicine, child learning, and security personnel. AR hardware is comprised of many of the same elements as VR headsets with the main difference being AR glasses provide direct view of actual surroundings via a see-through transparent screen instead of being blocked out as with VR's fully virtual environment.

Another variation of related art, termed Mixed-Reality (MR), lies between VR and AR. Transparency of the see-through lens screens of AR visors limit the opaqueness of computer-generated imagery on the resulting view of the environment. With MR, computer-generated imagery is combined with that of actual surroundings but without use of the clear-, see-through lens LCD screen used for AR. Instead, MR employs a fully enclosed visor similar to VR, that utilizes a built-in dual-lens camera to access 3D views of actual surroundings. This type of hardware facilitates opaque computer-generated graphics (as with VR) that can now augment visuals of actual surroundings via processing of camera-based imagery. Hence, MR can offer a more immersive version of augmented reality that is not limited by the transparent nature of a clear lens display. For example, an MR display can impose opaque three-dimensional (3D) objects such as extra virtual solid walls to a room whereas an AR-display would have difficulty preventing visual transparency through such a virtual wall from the user.

Conventional View-Limiting-Devices (VLDs)

FIG. 2 A is an image of conventional hood-style vision blockers commonly used for IFR flight training and examination. This hood is mounted to the user's head via lateral and topside straps or fasteners. The user (pilot trainee) is thereby prevented from seeing the aircraft exterior through the cockpit windows so that flight can be conducted only by reference to the forward instrument panel. This blocked view is subject to head movements of the user, where momentary shifts in position due to turbulence or other motion may expose portions of the external window view to the user. FIG. 2 B is a more compact design vision blocker comprised of goggles that block most of the user's forward and lateral views so that ideally only the flight instrument panel is visible. IFR hoods or goggles are typically constructed from low-cost molded plastic and are applied and removed deliberately by the user before and after use. Conventional IFR hoods require the pilot trainee to apply the head-mount and then maintain an unnatural downward head position to prevent views of the aircraft exterior. Since human physiology relating to balance and orientation involves the inner ear and head position, it would be preferable for trainees to maintain natural movements throughout training scenarios.

FIGS. 3 and 4 A show a perspective view from inside an aircraft cockpit displaying the interior instrument panel 3 and an unobstructed forward view 1 of the exterior environment. This diagram represents Visual Flight Rules (VFR) conditions where views outside the cockpit window 5 are clear 1 of cloud obstruction and allow sighting of the actual ground and/or horizon 2 . Flight under VFR is conducted with visual reference to the ground horizon 2 in order to maintain aircraft orientation and direction. Flight instruments on the forward panel 3 such as the artificial horizon 4 provide supplemental flight information for pitch and roll during VFR operations. FIG. 4 B represents the same perspective as FIGS. 3 and 4 A except under Instrument Flight Rules (IFR) conditions 17 where the exterior view outside the cockpit window is limited or completely blocked due to weather and/or lack of lighting. Under such conditions a pilot cannot rely on outside visual references such as terrain or horizon so must maintain control of the aircraft using only the instrument panel 3 which houses various gauges that describe aircraft orientation and motion. Pilots require specialized training to obtain an IFR rating that permits them to fly during such conditions of limited or no visibility 17 . Without proper training, a pilot is liable to lose orientation and control of the aircraft due to lack of skill and physiological vulnerabilities that amplify disorientation under such conditions.

MR-IFR Visor of the Present Disclosure

FIG. 5 provides a visual representation 6 and description of the MR- visor invention 7 referencing the user's head 8 and torso 10 areas to illustrate fastening of the invention 7 by means of straps or bands 9 . The MR- visor 7 may include all typical components of existing commercial VR-, AR- or MR-headsets including but not limited to a combiner, camera, sensors, registration features, signal emitters, and computer hardware and software driving image generation and overlay.

In accordance with some aspects of the present teachings, the basic MR-IFR visor utilizes standard components of a head-worn virtual reality (VR) display (i.e. VR headset) which utilizes video see-through display technology for immersing the user in a digitally-enhanced visual environment. Such standalone VR headsets typically include the following:

a) A dual Liquid-Crystal Display (LCD) panel or an Organic Light Emitting Diodes (OLED) technology-based display panel supporting a refresh rate of at least 60-120 fps for each eye at high resolution (e.g., 1920 pixels); b) Lenses with Interpupillary Distance (IPD) adjustable mechanically for each user; c) A Central Processor Unit (CPU) that generates virtual imagery graphics with additional processing units dedicated to computing motion-tracking outcomes; d) An internal Random-Access-Memory (RAM) unit; e) An embedded long-term digital storage unit; f) A battery unit acting as the power source; and g) A headband that straps the headset onto the user's head.

Additional sensors may be used for tracking extended head movements as well as specific objects in the surroundings. FIG. 6 illustrates the core components of the system according to some embodiments. In the case of a pilot user, the eye- forward display 20 enables MR content to be superimposed both inside the aircraft cockpit environment as well as the external environment viewable outside the aircraft's window panels. The hardware thus becomes a mixed-reality (MR) headset that covers the full field-of-view of the user, preventing any peering from the side, below, or above the display as frequently occurs with conventional head-worn view limiting devices.

The MR display is therefore similar to VR headsets in form, but now capable of adding precision-located holographic content to the actual surroundings by use of camera-assisted tracking and see-through technology. For example, this basic embodiment may include four (4) head-tracking cameras, two (2) directed forward 21 (above right and left eyes) and two (2) directed diagonally to the left side 22 and the right side 23 . By using sensor fusion-based positional tracking methods, these cameras continuously track the position of the user's head in relation to the physical environment without need for any additional external measurement devices. Each of the head-tracking cameras contains an Inertial Measurement Unit (IMU) which in turn includes an accelerometer and a gyroscope that allow high-frequency measurement of headset orientation. Together the cameras and their IMUs enable precise and reliable positional tracking based on sensor fusion. Inside-out optical positional tracking utilizes Simultaneous Localization and Mapping (SLAM) algorithms applied to the image stream of the head-tracking cameras. This âinside-outâ approach is contrary to the most common âoutside-inâ positional tracking approach employed in consumer-grade VR headsets. Inertial tracking methods based on the data stream produced by the IMUs supplement the optical positional tracking methods, which is particularly useful in the event of abrupt head movement.

Two (2) forward-looking high- definition RGB cameras 24 are used for creating a video see-through MR imagery. The cameras provide live view of actual surroundings while also permitting video recording and MR tracking of marker-less landmarks. Whereas conventional MR displays are typically designed to function only within close-range views such as a small room, the MR-IFR cameras and their variations are to provide for both near- and far-vision; thereby facilitating alternating views between the aircraft interior and far-off objects seen through cabin windows. The MR-IFR concept introduces the use of additional sets of physical lenses or high-speed auto-focusing lenses to provide rapid and reliable transition from near- to far-sight views. In one variation as shown in FIG. 7 A , multiple lenses 66 corresponding to different focal lengths are placed near each other to minimize variation in 3D perspective viewpoint between cameras. Another variation as shown in FIG. 7 B shows is an example of how multiple lenses can be mechanically cycled to rapidly adjust focus between nearby and distant targets (such as between an instrument gauge and a distant mountain range). In this case the lenses can be embedded into a rotating âbarrelâ 67 that rapidly switches cameras without any change to the user's 3D perspective. Two (2) subsystems are utilized to determine which lenses (near- or far-view) to use at any point in time:

a) Use of the forward-looking time-of-flight (ToF) depth camera 26 of the basic embodiment, where continuous measurement of the distance of the objects directly in front of the user at any given moment is used to indicate the appropriate focal length lens selection (illustrated in FIGS. 8 A and 8 B ); and b) Gaze-tracking capability (as described in accordance with some embodiments) where the gaze direction is measured directly from monitoring and detecting changes in the user's pupils.

FIG. 9 provides a flow diagram that describes associated methodology for achieving adaptive focus capability with the MR-IFR visor for both forward-looking ToF and gaze tracking cases. Sight target information from either source is used to generate a corresponding distance calculation that forms the basis of determining lens or focus selection and subsequent image processing. This flow is accomplished via control software that seamlessly integrates actual interior and exterior views with desired overlays of MR content.

A light sensor array 25 facing forward and to the sides of the headset allows measurement of luminous intensity of the natural light surrounding the user. Specifically, this sensor array provides detection and tracking of:

a) the main light source, whether it be natural sunlight, internal lighting or other lighting, its relative direction and the brightness as perceived by the user's eyes; and b) isolation and designation of external light energy entering via the aircraft's windshield, cabin windows, or other exposures to the aircraft exterior from the cockpit view

FIG. 10 provides a cross-section view of a small fixed-wing aircraft forward section showing relative locations of user and environmental lighting effects. The pilot/user 6 is shown seated in the cockpit area of a small fixed-wing single- engine aircraft 13 such as a Cessna 172 wearing said MR- visor 7 invention. Light energy 12 such as from the sun 11 radiates through the windows 18 of the cabin to light the interior of the cabin as well as illuminating exterior objects via reflected light from the sun or other energy sources. The MR- visor 7 incorporates built-in light energy sensors distributed in the forward and lateral sections of the headset such that a sufficient degree of image resolution data describing interior and exterior lighting is generated and provided to the computing vision system of the MR- visor 7 . This image data is utilized to distinguish visual reference regions for interior elements such as the instrument panel 3 from those visible through cabin window areas 18 such the nose-mounted propeller 19 ahead of the aircraft cabin as well as all exterior views to the environment 5 . Such pattern recognition can provide easier and more reliable definition of MR-overlay areas compared with conventional set up and programming techniques that rely heavily on mechanical reference and adjustment hardware.

Natural lighting intensity and distribution both for the interior and exterior of the aircraft can vary significantly over the course of a flight as weather conditions and relative position of the sun change over time and location. The present disclosure introduces MR hardware and an associated methodology that is akin to radio signal modulation in order to achieve accurate, consistent, and stable fixation of visible and obstructed regions desired by the IFR pilot trainee. For example, the primary measures in radio receivers are gain, selectivity, sensitivity, and stability. In similar fashion, the invention can provide user parameters and software settings that utilize similar parameters to easily set and maintain the desired boundaries between viewable and unviewable areas provided by the MR-visor headset. Gain describes the amount of amplification a signal may require in order to be properly registered by a receiver or sensor. Adjusting gain may assist in defining an aircraft cabin's window areas by strengthening the signal from low-light external environmental conditions during such times as sunrise or when the sky is overcast. Selectivity is the ability to filter out certain frequencies of energy so that the receiver or sensor can tune in to a particular bandwidth of electromagnetic energy. Adjusting selectivity can assist in distinguishing outside natural light from interior lighting sources by tuning in to specific wavelengths that are not shared with interior artificial aircraft lighting. In this way, light sensors on the MR-visor can more easily distinguish interior and exterior views of the cabin. Relatedly, sensitivity is the ability for the receiving hardware or detectors to distinguish true signals from naturally occurring background noise. Users of the MR-visor can set the sensitivity level of detectors to assist in defining visibility boundaries as well. For example, nighttime or other low-light conditions may require users to increase the sensitivity of visor-mounted sensors in order to provide sufficient signal contrast for detecting the interior areas of the cabin. Finally, stability describes how well the desired signal is maintained over the duration of use. For embodiments of the present disclosure, stability translates to how well the MR-visor maintains the original visibility boundaries set by the user as external conditions such as lighting, head position, aircraft position, and acceleration forces change over time. Such hardware is to utilize manual user input settings, software-based control, and optional software settings to easily and efficiently set and automatically maintain signal-to-noise ratios required for fixing the desired visibility boundaries. The MR-visor hardware includes detectors or sensors that feed signal read data to a computing unit that may reside on the headset or a nearby console. Software may also be designed to fix window overlay areas based only on initial user settings.

With reference to FIGS. 6 , 11 , 12 A, 12 B, and 13 , by combining the measurements from the light sensor array 25 , computer vision-based edge detection algorithms, natural feature detection algorithms, and SLAM algorithms are used for defining and continuous tracking of the cockpit window area, particularly the edges of the windows, regardless of the user's head position or aircraft orientation. Rapid and consistent tracking of exterior view areas is achieved with the software flow shown in FIG. 11 which is comprised of keypoint detection, landmark detection, edge detection, and anchor localization functions. This methodology differs from conventional SLAM-based approaches by implementing advanced machine learning algorithms that utilizes light sensor data to distinguish the cabin window areas from the cockpit interior. These stabilized and tracked exterior views become the canvas for mixed reality elements such as fog or clouds at various levels of transparency up through and including full opaqueness.

FIGS. 12 A and 12 B illustrate a VFR cockpit view and corresponding mixed-reality IFR replication by the invention, respectively. FIG. 12 A represents a pilot's view of the interior cabin and forward mounted instrument panel 3 along with an unobstructed view outside a forward cockpit window 18 during VMC. FIG. 12 B illustrates the same perspective view by a user of the MR-visor where the regions contained by the aircraft window 18 regions are now obstructed or modified by a computer-generated overlay 17 by the MR-headset. This MR overlay 17 visual pattern(s) can be designed and varied as desired by the user. For example, the flight instructor may adjust the transparency of this overlay 17 area in order to simulate gradual transition from VFR to IFR conditions. In other cases, the instructor may abruptly convert from VFR to IFR to simulate inadvertent flight into clouds. Furthermore, the overlay area can be programmed to replicate certain external lighting effects, such as partial view of ground lighting in order to produce optical illusions that exacerbate pilot spatial disorientation. Such exercises can help reinforce reliance on aircraft instruments in the presence of conflicting physiological sensory inputs, which is a central challenge to IFR flight. The present disclosure permits the user to always wear the MR-visor so that a host of IFR encounter situations can be replicated during flight.

FIG. 14 describes data input and output of the computer vision hardware and software controlling mixed-reality elements of the invented MR-visor. As shown, inputs from the external environment can be obtained by means including but not limited to light energy sensors or detectors, supplemental energy emitters, gyroscopic sensors, and/or other energy and motion sensors. These peripherals may occur on the surface of, within or outside the MR-visor headset itself. The external sense data is fed to the computing vision hardware which utilizes optimized operating software to achieve the desired window overlay 17 area. Computing hardware typically comprising a central processing unit(s) and memory devices may also reside onboard and/or external to the MR-visor headset. Some embodiments may incorporate an accompanying handheld console in order to minimize size and weight of the head-mounted unit. Such console may include a separate display that allows observers such as flight instructors to v

CLAIMS

Claims ( 58 )

What is claimed is:

1. A system for modifying a view perceived by a user who is substantially contained within an enclosure, the system comprising:

a view-blocking wearable user visor-headset having a display surface and see-through camera;

a distinguishing system configured to detect regions corresponding to an exterior region of the enclosure from an interior region contained within the enclosure and output a region signal, the distinguishing system is configured to generate a three-dimensional point cloud model of a substantial portion of the interior region of the enclosure;

computer memory storage and data transfer capability of the three-dimensional point cloud model;

a vision system configured to overlay imagery graphics upon the display surface of the view-blocking wearable user visor-headset based on the region signal;

first position and motion detection sensors affixable to a portion of the enclosure;

second position and motion detection sensors integrated with the view-blocking wearable user visor-headset; and

hardware configured for data communication between the first position and motion detection sensors, the second position and motion detection sensors, and at least one processing unit that is configured to generate and utilize differential data between the first and the second position and motion detection sensors to reduce vision mapping compute requirements.

2. The system of claim 1 , wherein the first and the second position and motion sensors include at least one of an inertial measuring unit, a gyroscopic sensor, and an accelerometer.

3. The system of claim 1 , further comprising at least one electromagnetic energy emitter and at least one electromagnetic energy receiver operably coupled to a processing system.

4. A system for modifying a user's view of a region of interest from within an enclosure in which the user is present, the system comprising:

a user wearable device having a transparent display screen substantially covering a user's field of view;

a first processing unit configured to detect regions of interest and output a region signal;

second processing units configured to detect and transmit object three-dimensional orientation provided by first position and motion sensors affixed in place and second position and motion sensors integrated with the user wearable device;

a vision system configured to receive the region signal and display an image upon at least one selected region of interest;

at least one electromagnetic energy emitter and at least one electromagnetic energy receiver operably coupled to the vision system, the at least one electromagnetic energy emitter and the at least one electromagnetic energy receiver utilize infra-red electromagnetic energy; and

a computer vision enabled three-dimensional point-cloud rendering subsystem configured to identify targeted regions of interest, the computer vision enabled three-dimensional point-cloud rendering subsystem comprising:

at least one camera with a focal length positioned to provide binocular three-dimensional views of a surrounding environment; and

a computer-based system integrating at least one of a three-dimensional point cloud model of internal features of the enclosure in which the user is present, an inertial measurement unit (IMU), and light-detection-and-ranging (LIDAR) depth sensing filter for determining limits of the enclosure and providing environmental references for point cloud model overlay, sizing, and position.

5. The system of claim 4 , wherein the transparent display screen is a Micro-Electro-Mechanical-Systems (MEMS) display utilizing holographic or diffractive extraction of light from a waveguide where at least two (2) micro-displays beam an image through a lens toward a surface of the transparent display screen.

6. The system of claim 5 , wherein headset onboard inward-facing visible light emitters direct light energy towards eyes of the user.

7. The system of claim 5 , wherein at least one electromagnetic energy emitter is located within an interior of the enclosure in which the user is present.

8. The system of claim 4 , wherein the enclosure is an aircraft cockpit with interior regions including an instrument panel, and windowpane regions of interest providing view of exterior regions to the aircraft.

9. The system of claim 4 , wherein the at least one camera is an optical see-through camera that alters between near- and far-vision by utilizing at least one of mechanical cycling of multiple sets of physical lenses and high-speed auto-focusing lenses.

10. The system of claim 4 , further comprising a programmable Global Positioning System (GPS) tracking system that provides at least one of location, orientation, speed, and acceleration data to an image-generating Central-Processing-Unit (CPU).

11. The system of claim 4 , further comprising inward-facing sensors or cameras utilized for at least one of gaze-tracking, eye-monitoring, and face-tracking of the user.

12. The system of claim 4 , further comprising supplemental physiological measurement devices having at least one of a heart rate sensor, electroencephalogram (EEG) sensors, and Galvanic Skin Response (GSR) sensors.

13. The system of claim 4 , further comprising an embedded surround sound personal audio system component to the user wearable device.

14. The system of claim 5 , wherein at least one electromagnetic energy emitter is located within an exterior of the enclosure in which the user is present.

15. The system of claim 3 , wherein the at least one electromagnetic energy emitter and the at least one electromagnetic energy receiver utilize infra-red electromagnetic energy.

16. A system for modifying a view perceived by a user who is substantially contained within an enclosure, the system comprising:

a view-blocking wearable user visor-headset having a display surface and see-through camera;

a distinguishing system configured to detect regions corresponding to an exterior region of the enclosure from an interior region contained within the enclosure and output a region signal;

a vision system configured to overlay imagery graphics upon the display surface of the view-blocking wearable user visor-headset based on the region signal; and

at least one camera with a focal length positioned to provide binocular three-dimensional views of a surrounding environment; and

17. The system of claim 16 , wherein the see-through camera is configured to alter between near- and far-vision by utilizing at least one of mechanical cycling of multiple sets of physical lenses and high-speed auto-focusing lenses.

18. The system of claim 1 , wherein the display surface is a Micro-Electro-Mechanical-Systems (MEMS) display utilizing holographic or diffractive extraction of light from a waveguide where at least two (2) micro-displays beam an image through a lens toward a surface of the display surface.

19. The system of claim 18 , wherein inward-facing visible light emitters onboard the view-blocking wearable user visor-headset direct light energy towards eyes of the user.

20. The system of claim 3 , wherein at least one electromagnetic energy emitter is located within an interior of the enclosure in which the user is present.

21. The system of claim 3 , wherein at least one electromagnetic energy emitter is located within an exterior of the enclosure in which the user is present.

22. The system of claim 1 , wherein the enclosure is an aircraft cockpit with interior regions including an instrument panel, and windowpane regions of interest providing view of exterior regions to the aircraft.

23. The system of claim 1 , further comprising a programmable Global Positioning System (GPS) tracking system that provides at least one of location, orientation, speed, and acceleration data to an image-generating Central-Processing-Unit (CPU).

24. The system of claim 1 , further comprising inward-facing sensors or cameras utilized for at least one of gaze-tracking, eye-monitoring, and face-tracking of the user.

25. The system of claim 1 , further comprising supplemental physiological measurement devices having at least one of a heart rate sensor, electroencephalogram (EEG) sensors, and Galvanic Skin Response (GSR) sensors.

26. The system of claim 1 , further comprising an embedded surround sound personal audio system component to the user wearable device.

27. The system of claim 16 , further comprising at least one electromagnetic energy emitter and at least one electromagnetic energy receiver operably coupled to a processing system.

28. The system of claim 27 , wherein the at least one electromagnetic energy emitter and the at least one electromagnetic energy receiver utilize infra-red electromagnetic energy.

29. The system of claim 27 , wherein the at least one electromagnetic energy emitter is located within the interior of the enclosure in which the user is present.

30. The system of claim 27 , wherein at least one electromagnetic energy emitter is located within an exterior of the enclosure in which the user is present.

31. The system of claim 1 , wherein the see-through camera is configured to alter between near- and far-vision by utilizing at least one of mechanical cycling of multiple sets of physical lenses and high-speed auto-focusing lenses.

32. The system of claim 16 , wherein the display surface is a Micro-Electro-Mechanical-Systems (MEMS) display utilizing holographic or diffractive extraction of light from a waveguide where at least two micro-displays beam an image through a lens toward a surface of the display surface.

33. The system of claim 32 , wherein the view-blocking wearable user visor-headset comprises inward-facing visible light emitters onboard directing light energy towards eyes of the user.

34. The system of claim 16 , wherein the enclosure is an aircraft cockpit having interior regions including an instrument panel and windowpane regions of interest providing view of exterior regions to the aircraft.

35. The system of claim 16 , further comprising a programmable Global Positioning System (GPS) tracking system configured to provide at least one of location, orientation, speed, and acceleration data to an image-generating Central-Processing-Unit (CPU).

36. The system of claim 16 , wherein the view-blocking wearable user visor-headset comprises inward-facing sensors or cameras configured for at least one of gaze-tracking, eye-monitoring, and face-tracking of the user.

37. The system of claim 16 , further comprising a supplemental physiological measurement device having at least one of a heart rate sensor, electroencephalogram (EEG) sensors, and Galvanic Skin Response (GSR) sensors.

38. The system of claim 16 , further comprising an embedded surround sound personal audio system component to the view-blocking wearable user visor-headset.

39. The system of claim 1 wherein the see-through camera comprises a liquid lens configured for dynamic autofocus and zoom, a system configured to adjust a lens curvature and focal length of the liquid lens via at least one of an electric field and electro-wetting mechanism.

40. The system of claim 16 wherein the see-through camera comprises a liquid lens configured for dynamic autofocus and zoom, a system configured to adjust a lens curvature and focal length of the liquid lens via at least one of an electric field and electro-wetting mechanism.

41. The system of claim 1 wherein the distinguishing system is configured to employ machine learning algorithms to detect regions corresponding to the exterior region of the enclosure from the interior region over time.

42. The system of claim 16 wherein the distinguishing system is configured to employ machine learning algorithms to detect regions corresponding to the exterior region of the enclosure from the interior region over time.

43. The system of claim 1 wherein the hardware configured for data communication is configured for use with a plurality communication protocols comprising Bluetooth, Wi-Fi, and NFC.

44. The system of claim 1 further comprising alternative peripherals configured to interface with the vision system.

45. The system of claim 4 further comprising alternative peripherals configured to interface with the vision system.

46. The system of claim 16 further comprising alternative peripherals configured to interface with the vision systemâ².

47. The system of claim 1 wherein the vision system is configured to automatically adapt the imagery graphics overlay based on changes in the surrounding environment.

48. The system of claim 4 wherein the vision system is configured to automatically adapt the imagery graphics overlay based on changes in the surrounding environment.

49. The system of claim 16 wherein the vision system is configured to automatically adapt the imagery graphics overlay based on changes in the surrounding environment.

50. The system of claim 1 wherein the three-dimensional point cloud model is configured to be exported in a standardized format.

51. The system of claim 4 wherein the three-dimensional point cloud model is configured to be exported in a standardized format.

52. The system of claim 16 wherein the three-dimensional point cloud model is configured to be exported in a standardized format.

53. The system of claim 1 further comprising a gesture recognition module configured to allow the user to manually define and modify regions corresponding to the exterior region or the interior region through hand or object gestures, the distinguishing system configured to detect the hand or object gesture and update the region signal.

54. The system of claim 4 further comprising a gesture recognition module configured to allow the user to manually define and modify regions corresponding to the exterior region or the interior region through hand or object gestures, the distinguishing system configured to detect the hand or object gesture and update the region signal.

55. The system of claim 16 further comprising a gesture recognition module configured to allow the user to manually define and modify regions corresponding to the exterior region or the interior region through hand or object gestures, the distinguishing system configured to detect the hand or object gesture and update the region signal.

56. The system of claim 1 wherein the distinguishing system is configured to employ artificial intelligence and machine learning algorithms.

57. The system of claim 4 wherein the distinguishing system is configured to employ artificial intelligence and machine learning algorithms.

58. The system of claim 16 wherein the distinguishing system is configured to employ artificial intelligence and machine learning algorithms.

US17/919,304

2020-12-20

2021-12-17

Mixed-reality visor for in-situ vehicular operations training

Active

US12112440B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US17/919,304

US12112440B2

( en )

2020-12-20

2021-12-17

Mixed-reality visor for in-situ vehicular operations training

Applications Claiming Priority (5)

Application Number

Priority Date

Filing Date

Title

US202063128163P

2020-12-20

US202163180040P

2021-04-26

US202163190138P

2021-05-18

US17/919,304

US12112440B2

( en )

2020-12-20

2021-12-17

Mixed-reality visor for in-situ vehicular operations training

PCT/US2021/064043

WO2022133219A1

( en )

2020-12-20

2021-12-17

Mixed-reality visor for in-situ vehicular operations training

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

PCT/US2021/064043

A-371-Of-International

WO2022133219A1

( en )

2020-12-20

2021-12-17

Mixed-reality visor for in-situ vehicular operations training

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US18/791,998

Continuation

US20240394990A1

( en )

2020-12-20

2024-08-01

Mixed-Reality Visor For In-Situ Vehicular Operations Training

Publications (2)

Publication Number

Publication Date

US20230334788A1

US20230334788A1 ( en )

2023-10-19

US12112440B2

true

US12112440B2 ( en )

2024-10-08

Family

ID=82058328

Family Applications (2)

Application Number

Title

Priority Date

Filing Date

US17/919,304

Active

US12112440B2

( en )

2020-12-20

2021-12-17

Mixed-reality visor for in-situ vehicular operations training

US18/791,998

Pending

US20240394990A1

( en )

2020-12-20

2024-08-01

Mixed-Reality Visor For In-Situ Vehicular Operations Training

Family Applications After (1)

Application Number

Title

Priority Date

Filing Date

US18/791,998

Pending

US20240394990A1

( en )

2020-12-20

2024-08-01

Mixed-Reality Visor For In-Situ Vehicular Operations Training

Country Status (2)

Country

Link

( 2 )

US12112440B2

( en )

( 1 )

WO2022133219A1

( en )

Cited By (1)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20250218127A1

( en )

2023-12-27

2025-07-03

Frank F. Dasmacci

Physical Navigation through Virtual Course within Virtual Environment

Families Citing this family (17)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US11993394B2

( en )

2021-11-10

2024-05-28

Rockwell Collins, Inc.

Flight safety demonstration and infotainment through mixed reality

US11960783B2

( en )

2022-04-28

2024-04-16

Dell Products Lp

Method and apparatus for extended reality head mounted display device

TWI873926B

( en )

2022-06-30

2025-02-21

å®éåéé»åè¡ä»½æéå¬å¸

Image sensing device and head-mounted display

US12706001B2

( en )

2022-06-30

2026-08-11

Red Six Aerospace Inc

Bi-directional communications for vehicle and virtual game situations

TWI848311B

( en )

2022-06-30

2024-07-11

å®éåéé»åè¡ä»½æéå¬å¸

Image sensing device and head-mounted display

US12028505B2

( en )

2022-06-30

2024-07-02

Htc Corporation

Image sensing device and head-mounted display

US20240071249A1

( en )

2022-08-31

2024-02-29

Shahriar SHAMS

System, Apparatus and Method for Advance View Limiting Device

DE102022124717A1

( en )

2022-09-26

2024-03-28

Bayerische Motoren Werke Aktiengesellschaft

Driving a motor vehicle in a virtual environment

US20240145066A1

( en )

2022-10-31

2024-05-02

United States Of America As Represented By The Administrator Of Nasa

Pervasive and Ubiquitous Decentralized Biocybernetic Adaptation and Biofeedback Training System

CN117542253B

( en )

2023-10-31

2026-04-24

ä¸å½äººæ°è§£æ¾åç©ºåèªç©ºå¤§å¦

Pilot cockpit training system

WO2025106992A1

( en )

2023-11-17

2025-05-22

Red Six Aerospace Inc.

Operator training using artificial intelligence agent

WO2025111371A1

( en )

2023-11-20

2025-05-30

Vrgineers, Inc.

Modular reconfigurable simulator

US12592143B2

( en )

2023-11-21

2026-03-31

Schon Mobility Inc.

System and device for threat monitoring

US12552546B2

( en )

2024-05-30

2026-02-17

Honeywell International s.r.o

Gaze-assisted verification of pilot activities

US12493350B1

( en )

2024-06-11

2025-12-09

Honda Motor Co., Ltd.

Vehicle simulation system and method

US12560720B1

( en )

2024-08-21

2026-02-24

Nvidia Corporation

Illuminated multi-view sensing using reflectance for in-cabin applications

CN119761075B

( en )

2025-03-06

2025-07-22

å¹¿ä¸æ ¼ç»¿æèè½ç§ææéå¬å¸

Outdoor electric awning equipment testing method and device based on shading holographic simulation

Citations (19)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20100001928A1

( en )

2008-06-30

2010-01-07

Honeywell International Inc.

Head-mountable cockpit display system

US20120206452A1

( en )

2010-10-15

2012-08-16

Geisner Kevin A

Realistic occlusion for a head mounted augmented reality display

US8411214B2

( en )

2008-06-24

2013-04-02

United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration

Variably transmittive, electronically-controlled eyewear

US9097891B2

( en )

2010-02-28

2015-08-04

Microsoft Technology Licensing, Llc

See-through near-eye display glasses including an auto-brightness control for the display brightness based on the brightness in the environment

US20160026242A1

( en )

2014-07-25

2016-01-28

Aaron Burns

Gaze-based object placement within a virtual reality environment

US9454022B1

( en )

2014-12-10

2016-09-27

Nickolas Joseph Sinopoli

Controlled selective visual occlusion system and method

US9759917B2

( en )

2010-02-28

2017-09-12

Microsoft Technology Licensing, Llc

AR glasses with event and sensor triggered AR eyepiece interface to external devices

US20170357332A1

( en )

2016-06-09

2017-12-14

Alexandru Octavian Balan

Six dof mixed reality input by fusing inertial handheld controller with hand tracking

US9995936B1

( en )

2016-04-29

2018-06-12

Lockheed Martin Corporation

Augmented reality systems having a virtual image overlaying an infrared portion of a live scene

US20190012928A1

( en )

2016-12-02

2019-01-10

AT Systems, LLC

Novel system and method for improving safety when operating aircraft in reduced- or modified-visibility conditions

US10180572B2

( en )

2010-02-28

2019-01-15

Microsoft Technology Licensing, Llc

AR glasses with event and user action control of external applications

US10249208B2

( en )

2016-12-02

2019-04-02

AT Systems, LLC

System and method for training aircraft pilots for sudden-onset reduced-visibility conditions

US10345593B2

( en )

2015-03-16

2019-07-09

Magic Leap, Inc.

Methods and systems for providing augmented reality content for treating color blindness

US20190324274A1

( en )

2018-04-24

2019-10-24

Apple Inc.

Head-Mounted Device with an Adjustable Opacity System

US10529248B2

( en )

2014-06-19

2020-01-07

Embraer S.A.

Aircraft pilot training system, method and apparatus for theory, practice and evaluation

US10539787B2

( en )

2010-02-28

2020-01-21

Microsoft Technology Licensing, Llc

Head-worn adaptive display

US10575920B2

( en )

2015-03-09

2020-03-03

National Cancer Center

Augmented reality image projection system

US20210049925A1

( en )

2018-04-27

2021-02-18

Red 6 Inc.

Augmented reality for vehicle operations

US20210295732A1

( en )

2018-08-02

2021-09-23

Elbit Systems Ltd.

In-flight training simulation displaying a virtual environment

2021

2021-12-17

PCT/US2021/064043

patent/WO2022133219A1/en

not_active

Ceased

2021-12-17

US17/919,304

patent/US12112440B2/en

active

Active

2024

2024-08-01

US18/791,998

patent/US20240394990A1/en

active

Pending

Patent Citations (21)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US8411214B2

( en )

2008-06-24

2013-04-02

United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration

Variably transmittive, electronically-controlled eyewear

US20100001928A1

( en )

2008-06-30

2010-01-07

Honeywell International Inc.

Head-mountable cockpit display system

US10180572B2

( en )

2010-02-28

2019-01-15

Microsoft Technology Licensing, Llc

AR glasses with event and user action control of external applications

US9097891B2

( en )

2010-02-28

2015-08-04

Microsoft Technology Licensing, Llc

See-through near-eye display glasses including an auto-brightness control for the display brightness based on the brightness in the environment

US20200192089A1

( en )

2010-02-28

2020-06-18

Microsoft Technology Licensing, Llc

Head-worn adaptive display

US10539787B2

( en )

2010-02-28

2020-01-21

Microsoft Technology Licensing, Llc

Head-worn adaptive display

US9759917B2

( en )

2010-02-28

2017-09-12

Microsoft Technology Licensing, Llc

AR glasses with event and sensor triggered AR eyepiece interface to external devices

US20120206452A1

( en )

2010-10-15

2012-08-16

Geisner Kevin A

Realistic occlusion for a head mounted augmented reality display

US10529248B2

( en )

2014-06-19

2020-01-07

Embraer S.A.

Aircraft pilot training system, method and apparatus for theory, practice and evaluation

US20160026242A1

( en )

2014-07-25

2016-01-28

Aaron Burns

Gaze-based object placement within a virtual reality environment

US9454022B1

( en )

2014-12-10

2016-09-27

Nickolas Joseph Sinopoli

Controlled selective visual occlusion system and method

US10575920B2

( en )

2015-03-09

2020-03-03

National Cancer Center

Augmented reality image projection system

US10345593B2

( en )

2015-03-16

2019-07-09

Magic Leap, Inc.

Methods and systems for providing augmented reality content for treating color blindness

US10359631B2

( en )

2015-03-16

2019-07-23

Magic Leap, Inc.

Augmented reality display systems and methods for re-rendering the world

US9995936B1

( en )

2016-04-29

2018-06-12

Lockheed Martin Corporation

Augmented reality systems having a virtual image overlaying an infrared portion of a live scene

US20170357332A1

( en )

2016-06-09

2017-12-14

Alexandru Octavian Balan

Six dof mixed reality input by fusing inertial handheld controller with hand tracking

US20190012928A1

( en )

2016-12-02

2019-01-10

AT Systems, LLC

Novel system and method for improving safety when operating aircraft in reduced- or modified-visibility conditions

US10249208B2

( en )

2016-12-02

2019-04-02

AT Systems, LLC

System and method for training aircraft pilots for sudden-onset reduced-visibility conditions

US20190324274A1

( en )

2018-04-24

2019-10-24

Apple Inc.

Head-Mounted Device with an Adjustable Opacity System

US20210049925A1

( en )

2018-04-27

2021-02-18

Red 6 Inc.

Augmented reality for vehicle operations

US20210295732A1

( en )

2018-08-02

2021-09-23

Elbit Systems Ltd.

In-flight training simulation displaying a virtual environment

Non-Patent Citations (12)

* Cited by examiner, â Cited by third party

Title

C. Campos, R. Elvira, J. Gomez Rodrigues, J. Montiel, J. Tordos, " ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map Slam ", IEEE Transactions and Robotics, Apr. 23, 2021.

C. Debeunne and D. Vivet. " A Review of Visual-LiDAR Fusion based Simultaneous Localization and Mapping, " Sensors 2020, 20, 2068; doi: 10.3390/s20072068, Accepted: Apr. 5, 2020; Published: Apr. 7, 2020.

David Tulis, " IMC on a VFR Day: ICARUS Face Shield Simulates Realistic Conditions: View Limiting Device Controlled by App " AOPA, Sep. 20, 2021.

G. Klein and D. Murray, " Parallel Tracking and Mapping for Small AR Workspaces, " 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, Nov. 13-16, 2007, pp. 225-234, doi: 10.1109/ISMAR.2007.4538852.

G. Klein and D. Murray, " Parallel Tracking and Mapping on a camera phone, " 2009 8th IEEE International Symposium on Mixed and Augmented Reality, Washington DC, Oct. 19-22, 2009, pp. 83-86, doi: 10.1109/ISMAR.2009.5336495.

He, K., Gkioxari, G., Dollar, P. and Girshick, R., " Mask R-CNN, " 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Oct. 22-29, 2017, pp. 2980-2988, doi: 10.1109/ICCV.2017.322.

International Preliminary Report on Patentability, corresponding to PCT Application No. PCT/US2021/064043, dated Jun. 29, 2023.

International Search Report and Written Opinion regarding International Application No. PCT/US2021/64043, dated Mar. 8, 2022.

J. Carreria and C. Sminchisescu, " CPMC: Automatic Object Segmentation Using Constrained Parametric Min- Cuts, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, No. 7, pp. 1312-1328, Jul. 1, 2012, doi: 10.1109/TPAMI.2011.231.

P. Furgale, J. Rehder, and R. Siegwart, " Unified Temporal and Spatial Calibration for Multi-Sensor Systems. " In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan, Nov. 1, 2013.

R. Mur-Artal and J. D. Tardos, " Visual-Inertial Monocular Slam With Map Reuse, " in IEEE Robotics and Automation Letters, vol. 2, No. 2, pp. 796-803, Oct. 19, 2016, doi: 10.1109/LRA.2017.2653359.

TDK InvenSense, " SmartMotion ", Online, https://invensense.tdk.com/smartmotion/.

Cited By (2)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20250218127A1

( en )

2023-12-27

2025-07-03

Frank F. Dasmacci

Physical Navigation through Virtual Course within Virtual Environment

US12688656B2

( en )

2023-12-27

2026-07-21

Frank F. Dasmacci

Physical navigation through virtual course within virtual environment

Also Published As

Publication number

Publication date

WO2022133219A1

( en )

2022-06-23

US20240394990A1

( en )

2024-11-28

US20230334788A1

( en )

2023-10-19

Mixed-reality visor for in-situ vehicular operations training — Wael Zohni (US12112440B2)

Analyze & explore related research in Conceptio

Related documents