ABSTRACT
Abstract
An apparatus for mounting on a head including a frame, A face-wearable near-ocular optics and a micro-display for displaying data in front of the eyes is provided. A computing device is coupled to the micro-display. At least one sensor is coupled to the computing device for receiving biometric human information.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
This application is a continuation of U.S. patent application Ser. No. 17/396,976, filed Aug. 9, 2021, which application is a continuation of U.S. patent application Ser. No. 16/667,489, filed Oct. 29, 2019, now issued as U.S. Pat. No. 11,099,643 which is a continuation of U.S. patent application Ser. No. 14/853,851, filed Sep. 14, 2015, now issued as U.S. Pat. No. 10,509,466, which is a continuation of U.S. patent application Ser. No. 14/328,663, filed Jul. 10, 2014, which is a continuation of U.S. patent application Ser. No. 14/086,909, filed Nov. 21, 2013, which is a continuation of U.S. patent application Ser. No. 13/815,492. filed Mar. 5, 2013, which is a continuation-in-part of U.S. patent application Ser. No. 13/470,242, filed May 11, 2012, which claims the benefit of priority to U.S. Provisional Application Ser. No. 61/485,120, filed May 11, 2011, the contents of each application is incorporated herein by reference in their entireties.
FIELD OF THE INVENTION
The present invention relates to headwear and more particularly to headwear having cameras therein.
BACKGROUND OF THE INVENTION
Headwear such as glasses having cameras therein has been provided. Head-mounted displays have also been provided.
There is a need for headwear that can, for example, provide the user with an improved interactive experience with images viewable by the user through the headwear.
BRIEF DESCRIPTION OF THE DRAWINGS
The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.
FIG. 1 is a front perspective view of one embodiment of the headwear of the present invention, consisting of a pair of glasses.
FIG. 2 is a rear perspective view of the glasses of FIG. 1 .
FIG. 3 is a schematic illustration of one embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 4 is a schematic illustration of one embodiment of an eye tracking and three-dimensional distance focusing system for one embodiment of the headwear of the present invention.
FIG. 5 is a schematic illustration one embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 6 is a schematic illustration of another embodiment of a display apparatus for use with the headwear of the present invention, a portion of which has been enlarged in size.
FIG. 7 is a schematic illustration of the enlarged portion of the display apparatus of FIG. 6 , a portion of which has enlarged in three views to illustration alternate positions of the lenses shown therein.
FIG. 8 is a schematic illustration of another embodiment of a display apparatus and system incorporating the same for use with the headwear of the present invention.
FIG. 9 is a schematic illustration of a further embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 10 is a schematic illustration of another embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 11 is a schematic illustration of another embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 12 is a schematic illustration of a further embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 13 is a schematic illustration of another embodiment of the headwear of the present invention.
FIG. 14 is a schematic illustration of another embodiment of the headwear of the present invention.
FIG. 15 is a schematic illustration of a further embodiment of the headwear of the present invention.
FIG. 16 is a schematic illustration of another embodiment of the headwear of the present invention.
FIG. 17 is a schematic illustration of an embodiment of a head mounted sensor apparatus for use with the headwear of the present invention.
FIG. 18 is a schematic illustration of another embodiment of the headwear of the present invention.
FIG. 19 is a portion of one embodiment of a user interface for use with the headwear of the present invention.
FIG. 20 is another portion of the user interface of FIG. 19 .
FIG. 21 is a schematic illustration of a multi-user system utilizing a plurality of the headwear of the present invention.
FIG. 22 is an embodiment of a system utilizing the headwear of the present invention.
FIG. 23 is an embodiment of a transaction processing system utilizing the headwear of the present invention.
FIG. 24 is an embodiment of a processing system incorporating the headwear of the present invention.
FIG. 25 is an embodiment of a system utilizing the headwear of the present invention.
FIG. 26 is one embodiment of the computer architecture of the headwear of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
The headware of the present invention can include any suitable head-mounted device or apparatus or face-wearable device or apparatus that can specifically include any suitable eyewear such as glasses or goggles. The headwear can include any suitable display such as a head-mounted display. In one embodiment, the headware can be a pair of glasses 31 , such as illustrated in FIGS. 1 and 2 . The glasses 31 can include a frame 32 made from any suitable material such as plastic or metal, including any suitable shape memory alloy. The frame 32 can have a front piece 33 that can include a first or left lens, display or optical element holder 36 and a second or right lens, display or optical element holder 37 connected by a bridge 38 . The front piece 33 additionally includes a left end portion 41 and a right end portion 42 . A first or left optical element 43 and a second or right optical element 44 can be provided within respective left and right optical element holders 36 , 37 . Each of the optical elements 43 , 44 can be a lens, a display, a display assembly or a combination of the foregoing. Any of the display assemblies disclosed herein can be provided in glasses 31 . When the optical elements 43 , 44 include a display, they can each be referred to as near ocular digital displays and can show immersive volumetric three-dimensional graphics, stereo three-dimensional graphics or two-dimensional graphics and can include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, any other light-emitting diode (LED) display or a laser display. Each of the optical elements 43 , 44 includes an inner surface 45 that faces the user and is thus view are mounted on the head of the user. When the optical elements 43 , 44 include a display, the inner surface 45 is a display surface.
Frame 32 additionally includes a left arm or temple piece 46 and a second arm or temple piece 47 coupled to the respective left and right end portions 41 , 42 of the front piece 33 by any suitable means such as a hinge (not shown), so as to be pivotably coupled to the front piece 33 , or rigidly or fixably secured to the front piece so as to be integral with the front piece 33 . Each of the temple pieces or temples 46 , 47 can include a first portion 51 that is pivotably coupled to the respective end portion 41 , 42 of the front piece and any suitable second portion 52 , such as curved or arcuate piece, for coupling to the ear of the user. In one embodiment the front piece 33 can be formed from a single piece of material, so as to have a unitary or integral construction. In one embodiment, such as illustrated in FIGS. 1 and 2 , the entire frame can be formed from a single piece of material so as to have a unitary or integral construction.
Glasses 31 can include a computing device, such as computer 61 , which can be of any suitable type so as to be carried by the frame 32 and in one embodiment of a suitable size and shape so as to be at least partially disposed in one of the temples 46 , 47 and in one embodiment, as illustrated in FIGS. 1 and 2 , the computer is sized and shaped similar to the size and shape of one of the temples 46 , 47 and is thus disposed almost entirely if not entirely within the structure and confines of such temple 46 , 47 . In one embodiment, the computer 61 can be disposed in both of the temples. The computer 61 can include a central processing unit such as one or more micro processors (not shown), a suitable storage medium (not shown) such as a flash drive or memory that is electrically coupled to the central processing unit, and a suitable input device, a suitable output device or a combination of input and output devices that is electrically coupled to one or both of the central processing unit and the storage medium. The computer 61 additionally includes a battery 62 or other suitable portable power supply. In one embodiment, the battery 62 is disposed in one of the temples 46 , 47 , and in the glasses 31 shown in FIGS. 1 and 2 the battery 62 is shown as being disposed in left temple 46 and electrically coupled to the remainder of the computer 61 disposed in the right temple 47 . The one or more input and output devices can include a connector or port (not shown) accessible from the outside of frame 32 , a wireless receiver, transmitter or transceiver (not shown) or a combination of such devices.
Face wearable computing device or apparatus 31 , which can be in the form-factor of glasses, can include one or more input sensors or peripheral devices for any suitable purpose including the collection of environmental and biometric contextual data and information used as input to the computer 61 . Front piece 33 is provided with an outward-facing, forward-facing or front or outer surface 66 that faces forward or away from the user when the glasses 31 are mounted on the face of the user, and an opposite inward-facing, rearward-facing or rear or inner surface 67 that faces the face of the user when the glasses 31 are mounted on the face of the user. Such sensors can include inwardly-facing video sensors or digital imaging modules such as cameras 68 that can be mounted on or provided within the inner surface 67 of the front piece 33 or elsewhere on the frame 32 so as to be facing the user, and outwardly-facing video sensors or digital imaging modules such as cameras 69 that can be mounted on or provided with the outer surface 66 of the front piece 33 or elsewhere on the frame 32 so as to be facing away from the user. Such sensors, peripheral devices or peripherals can additionally include inward-facing digital sensors in the form of electro occulography sensors, or EOG sensors 71 , and inwardly-facing sensors in the form of electroencephalogram sensors or EEG sensors 72 . The EOG sensors 71 and EEG sensors 72 can each be mounted on or provided within the inner surface 67 of front frame piece 33 or elsewhere on the frame 32 so as to be facing the user. The outwardly-facing sensors can additionally include any suitable geometry sensor 73 . Each of the peripherals or sensors are electrically coupled to the computer 61 by any suitable means such as a conductive lead, trace or cable 74 , only a few of which are illustrated in the figures for simplicity. Additional peripheral devices or sensors for obtaining biometric inputs from the user can be provided and carried by or mounted on frame 32 . Left and right optical elements 43 , 44 can be additionally electrically coupled to the computer by any suitable means such as respective leads 74 when the optical elements include a display or other features that are controllable by the computer 61 .
In one embodiment, illustrated in FIGS. 1 and 2 , first and second inwardly-facing cameras 68 can be provided, one near the bottom center of each frame holder 36 and first and second outwardly-facing cameras 69 are provided, one near the top center of each frame holder 36 , 37 . The inwardly-facing cameras 68 can be used for any suitable purposes including the extraction of biometric data using image analysis. Such biometric data can include image-based eye tracking, iris, facial or other recognition for example for identification purposes, facial expressions and the mood of the user. The outwardly-facing cameras 69 can be used for any suitable purpose, for example to capture respective images similar to those capturable by the left and right eyes of the user. First and second EEG sensors 72 can be provided on a portion of the frame that contacts the skin of the user, for example on the inner surface 67 or the bridge 38 or front piece 33 . A plurality of EOG sensors 71 can be provided on the inner surface 67 around each of the left and right optical elements 43 , 44 so as to be registrable with the left and right eyes of the user. In one embodiment, the EOG sensors 71 contact the skin of the user. The outwardly-facing geometry sensor 73 can be used for any suitable purpose, including the scanning and capturing of three dimensional geometry.
Computer 61 can additionally include an operating system that can include software used to access and control the peripheral devices connected to the computer, including but not limited to peripherals 68 , 69 , 71 , 72 and 73 . The computer can process the data from the multiple input sources or peripherals and can then optionally output data and information to the human sensory system through the use of the near ocular digital displays 43 , 44 for consumption into the user's or wearer's eyes. For example, outwardly-facing digital sensors, peripherals or cameras 69 and geometry sensor 73 can be used to collect contextual data about the surroundings of the wearer, and sent to the computer 61 for processing as input data used by the computing systems operating within the computer. The inwardly-facing sensors, including sensors 78 , 71 and 72 , can be used to capture data from the wearer of the apparatus or glasses 31 such that this data can be sent to the computer 61 and the computing system of the computer can compute additional meaningful data from the input data sent from such sensors, which can further be utilized by the computing system to control various aspects of the computing system of the computer 61 , including
CROSS REFERENCE TO RELATED APPLICATIONS
This application is a continuation of U.S. patent application Ser. No. 17/396,976, filed Aug. 9, 2021, which application is a continuation of U.S. patent application Ser. No. 16/667,489, filed Oct. 29, 2019, now issued as U.S. Pat. No. 11,099,643 which is a continuation of U.S. patent application Ser. No. 14/853,851, filed Sep. 14, 2015, now issued as U.S. Pat. No. 10,509,466, which is a continuation of U.S. patent application Ser. No. 14/328,663, filed Jul. 10, 2014, which is a continuation of U.S. patent application Ser. No. 14/086,909, filed Nov. 21, 2013, which is a continuation of U.S. patent application Ser. No. 13/815,492. filed Mar. 5, 2013, which is a continuation-in-part of U.S. patent application Ser. No. 13/470,242, filed May 11, 2012, which claims the benefit of priority to U.S. Provisional Application Ser. No. 61/485,120, filed May 11, 2011, the contents of each application is incorporated herein by reference in their entireties.
FIELD OF THE INVENTION
The present invention relates to headwear and more particularly to headwear having cameras therein.
BACKGROUND OF THE INVENTION
Headwear such as glasses having cameras therein has been provided. Head-mounted displays have also been provided.
There is a need for headwear that can, for example, provide the user with an improved interactive experience with images viewable by the user through the headwear.
BRIEF DESCRIPTION OF THE DRAWINGS
The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.
FIG. 1 is a front perspective view of one embodiment of the headwear of the present invention, consisting of a pair of glasses.
FIG. 2 is a rear perspective view of the glasses of FIG. 1 .
FIG. 3 is a schematic illustration of one embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 4 is a schematic illustration of one embodiment of an eye tracking and three-dimensional distance focusing system for one embodiment of the headwear of the present invention.
FIG. 5 is a schematic illustration one embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 6 is a schematic illustration of another embodiment of a display apparatus for use with the headwear of the present invention, a portion of which has been enlarged in size.
FIG. 7 is a schematic illustration of the enlarged portion of the display apparatus of FIG. 6 , a portion of which has enlarged in three views to illustration alternate positions of the lenses shown therein.
FIG. 8 is a schematic illustration of another embodiment of a display apparatus and system incorporating the same for use with the headwear of the present invention.
FIG. 9 is a schematic illustration of a further embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 10 is a schematic illustration of another embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 11 is a schematic illustration of another embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 12 is a schematic illustration of a further embodiment of a display apparatus for use with the headwear of the present invention.
FIG. 13 is a schematic illustration of another embodiment of the headwear of the present invention.
FIG. 14 is a schematic illustration of another embodiment of the headwear of the present invention.
FIG. 15 is a schematic illustration of a further embodiment of the headwear of the present invention.
FIG. 16 is a schematic illustration of another embodiment of the headwear of the present invention.
FIG. 17 is a schematic illustration of an embodiment of a head mounted sensor apparatus for use with the headwear of the present invention.
FIG. 18 is a schematic illustration of another embodiment of the headwear of the present invention.
FIG. 19 is a portion of one embodiment of a user interface for use with the headwear of the present invention.
FIG. 20 is another portion of the user interface of FIG. 19 .
FIG. 21 is a schematic illustration of a multi-user system utilizing a plurality of the headwear of the present invention.
FIG. 22 is an embodiment of a system utilizing the headwear of the present invention.
FIG. 23 is an embodiment of a transaction processing system utilizing the headwear of the present invention.
FIG. 24 is an embodiment of a processing system incorporating the headwear of the present invention.
FIG. 25 is an embodiment of a system utilizing the headwear of the present invention.
FIG. 26 is one embodiment of the computer architecture of the headwear of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
The headware of the present invention can include any suitable head-mounted device or apparatus or face-wearable device or apparatus that can specifically include any suitable eyewear such as glasses or goggles. The headwear can include any suitable display such as a head-mounted display. In one embodiment, the headware can be a pair of glasses 31 , such as illustrated in FIGS. 1 and 2 . The glasses 31 can include a frame 32 made from any suitable material such as plastic or metal, including any suitable shape memory alloy. The frame 32 can have a front piece 33 that can include a first or left lens, display or optical element holder 36 and a second or right lens, display or optical element holder 37 connected by a bridge 38 . The front piece 33 additionally includes a left end portion 41 and a right end portion 42 . A first or left optical element 43 and a second or right optical element 44 can be provided within respective left and right optical element holders 36 , 37 . Each of the optical elements 43 , 44 can be a lens, a display, a display assembly or a combination of the foregoing. Any of the display assemblies disclosed herein can be provided in glasses 31 . When the optical elements 43 , 44 include a display, they can each be referred to as near ocular digital displays and can show immersive volumetric three-dimensional graphics, stereo three-dimensional graphics or two-dimensional graphics and can include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, any other light-emitting diode (LED) display or a laser display. Each of the optical elements 43 , 44 includes an inner surface 45 that faces the user and is thus view are mounted on the head of the user. When the optical elements 43 , 44 include a display, the inner surface 45 is a display surface.
Frame 32 additionally includes a left arm or temple piece 46 and a second arm or temple piece 47 coupled to the respective left and right end portions 41 , 42 of the front piece 33 by any suitable means such as a hinge (not shown), so as to be pivotably coupled to the front piece 33 , or rigidly or fixably secured to the front piece so as to be integral with the front piece 33 . Each of the temple pieces or temples 46 , 47 can include a first portion 51 that is pivotably coupled to the respective end portion 41 , 42 of the front piece and any suitable second portion 52 , such as curved or arcuate piece, for coupling to the ear of the user. In one embodiment the front piece 33 can be formed from a single piece of material, so as to have a unitary or integral construction. In one embodiment, such as illustrated in FIGS. 1 and 2 , the entire frame can be formed from a single piece of material so as to have a unitary or integral construction.
Glasses 31 can include a computing device, such as computer 61 , which can be of any suitable type so as to be carried by the frame 32 and in one embodiment of a suitable size and shape so as to be at least partially disposed in one of the temples 46 , 47 and in one embodiment, as illustrated in FIGS. 1 and 2 , the computer is sized and shaped similar to the size and shape of one of the temples 46 , 47 and is thus disposed almost entirely if not entirely within the structure and confines of such temple 46 , 47 . In one embodiment, the computer 61 can be disposed in both of the temples. The computer 61 can include a central processing unit such as one or more micro processors (not shown), a suitable storage medium (not shown) such as a flash drive or memory that is electrically coupled to the central processing unit, and a suitable input device, a suitable output device or a combination of input and output devices that is electrically coupled to one or both of the central processing unit and the storage medium. The computer 61 additionally includes a battery 62 or other suitable portable power supply. In one embodiment, the battery 62 is disposed in one of the temples 46 , 47 , and in the glasses 31 shown in FIGS. 1 and 2 the battery 62 is shown as being disposed in left temple 46 and electrically coupled to the remainder of the computer 61 disposed in the right temple 47 . The one or more input and output devices can include a connector or port (not shown) accessible from the outside of frame 32 , a wireless receiver, transmitter or transceiver (not shown) or a combination of such devices.
Face wearable computing device or apparatus 31 , which can be in the form-factor of glasses, can include one or more input sensors or peripheral devices for any suitable purpose including the collection of environmental and biometric contextual data and information used as input to the computer 61 . Front piece 33 is provided with an outward-facing, forward-facing or front or outer surface 66 that faces forward or away from the user when the glasses 31 are mounted on the face of the user, and an opposite inward-facing, rearward-facing or rear or inner surface 67 that faces the face of the user when the glasses 31 are mounted on the face of the user. Such sensors can include inwardly-facing video sensors or digital imaging modules such as cameras 68 that can be mounted on or provided within the inner surface 67 of the front piece 33 or elsewhere on the frame 32 so as to be facing the user, and outwardly-facing video sensors or digital imaging modules such as cameras 69 that can be mounted on or provided with the outer surface 66 of the front piece 33 or elsewhere on the frame 32 so as to be facing away from the user. Such sensors, peripheral devices or peripherals can additionally include inward-facing digital sensors in the form of electro occulography sensors, or EOG sensors 71 , and inwardly-facing sensors in the form of electroencephalogram sensors or EEG sensors 72 . The EOG sensors 71 and EEG sensors 72 can each be mounted on or provided within the inner surface 67 of front frame piece 33 or elsewhere on the frame 32 so as to be facing the user. The outwardly-facing sensors can additionally include any suitable geometry sensor 73 . Each of the peripherals or sensors are electrically coupled to the computer 61 by any suitable means such as a conductive lead, trace or cable 74 , only a few of which are illustrated in the figures for simplicity. Additional peripheral devices or sensors for obtaining biometric inputs from the user can be provided and carried by or mounted on frame 32 . Left and right optical elements 43 , 44 can be additionally electrically coupled to the computer by any suitable means such as respective leads 74 when the optical elements include a display or other features that are controllable by the computer 61 .
In one embodiment, illustrated in FIGS. 1 and 2 , first and second inwardly-facing cameras 68 can be provided, one near the bottom center of each frame holder 36 and first and second outwardly-facing cameras 69 are provided, one near the top center of each frame holder 36 , 37 . The inwardly-facing cameras 68 can be used for any suitable purposes including the extraction of biometric data using image analysis. Such biometric data can include image-based eye tracking, iris, facial or other recognition for example for identification purposes, facial expressions and the mood of the user. The outwardly-facing cameras 69 can be used for any suitable purpose, for example to capture respective images similar to those capturable by the left and right eyes of the user. First and second EEG sensors 72 can be provided on a portion of the frame that contacts the skin of the user, for example on the inner surface 67 or the bridge 38 or front piece 33 . A plurality of EOG sensors 71 can be provided on the inner surface 67 around each of the left and right optical elements 43 , 44 so as to be registrable with the left and right eyes of the user. In one embodiment, the EOG sensors 71 contact the skin of the user. The outwardly-facing geometry sensor 73 can be used for any suitable purpose, including the scanning and capturing of three dimensional geometry.
Computer 61 can additionally include an operating system that can include software used to access and control the peripheral devices connected to the computer, including but not limited to peripherals 68 , 69 , 71 , 72 and 73 . The computer can process the data from the multiple input sources or peripherals and can then optionally output data and information to the human sensory system through the use of the near ocular digital displays 43 , 44 for consumption into the user's or wearer's eyes. For example, outwardly-facing digital sensors, peripherals or cameras 69 and geometry sensor 73 can be used to collect contextual data about the surroundings of the wearer, and sent to the computer 61 for processing as input data used by the computing systems operating within the computer. The inwardly-facing sensors, including sensors 78 , 71 and 72 , can be used to capture data from the wearer of the apparatus or glasses 31 such that this data can be sent to the computer 61 and the computing system of the computer can compute additional meaningful data from the input data sent from such sensors, which can further be utilized by the computing system to control various aspects of the computing system of the computer 61 , including any software system of the computer 61 , or aspects of the computing system or attached peripherals of the computer, such as visual outputs to optical displays 43 , 44 or auditory outputs to speakers (not shown). For example, EEG sensors 72 can be used to measure the user's brain activity and state, for example voltage fluctuations within the neurons of the brain of the user, that can be sent to the computer 61 and used to control various functions within the software or operating system of the computer. An example is using EEG sensors 72 to mentally concentrate on a button on optical displays 43 , 44 in order to click on it.
Such output data can additionally include audio signals that can delivered to speakers (not shown) mounted on or carried by the frame 32 or coupleable to the frame for mounting elsewear for consumption into the wearer's ears. Other output methods such as haptics can be provided. Other outputs can include, for example, haptic capacitive touch surfaces on frames 32 of the glasses 31 , haptic/tactile virtual objects for example via hand-worn accessories such as gloves or rings, world-tracked three-dimensional spatially rendered audio, electrochromic sunglasses, LEDs or other visual displays for notifications, and simulating acceleration and/or gravity via electrical stimulation of the user's inner ear. Additionally, the output from computer 61 can be sent to other local applications, other networked application that have access to some or all of the data acquired by the biometric sensors of glasses 31 , or both.
As can be seen, the computer 61 can access and control connected sensors and peripherals, including without limitation peripherals 68 , 69 , 71 , 72 and 73 , which are electrically connected to the computer and send data to the computer over digital leads or cables 74 . The computer and peripherals, and a suitable power supply such as battery 62 , can be packaged and encased into a frame 32 that is designed as eyewear and can have the form factor of glasses. The glasses 31 can use optical displays 43 , 44 together with biometric sensors to create a natural user experience where biometric thoughts and feelings and moods and concentration can control the user interface, provided for example by displays 43 , 44 .
In an additional possible use, optical displays 43 , 44 can include electrochromic sunglasses. In this regard, software or electrical command from computer 61 causes the lenses 43 , 44 to change their color, darkness or both. The computer 61 can additionally change other optical properties of the displays 43 , 44 , such as the focus distance of the scene as seen through the lenses.
The headwear of the present invention can be configured to re-display the world that the user is seeing not just as a stereo three-dimensional scene, but in one embodiment as a more realistic volumetric three-dimensional scene. In such a volumetric three-dimensional scene, the light displayed to the user is in focus at the proper focus distance. In one embodiment, the focus distance of the light passing through the optical display assembly or optical element assembly of the invention is controlled by software or otherwise by a local computing systems such as computer 61 or any other networked computer system. In one embodiment illustrated in FIG. 3 , headwear 78 includes a support structure 79 , for example, frame 32 of glasses 31 . An optical display or element assembly 80 of the headwear 78 , such as left and right optical elements 43 , 44 of glasses 31 , can include a digital display matrix 81 and a plurality of optical layers or lenses disposed between the matrix 81 and the eye of the user that can be controlled by such software and/or computing system to adjust the focus distance of the light traveling through the assembly 30 . Such optical layers can include any of the optimal layers or lens arrays disclosed herein. The display matrix can be of any suitable type, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, any other light-emitting diode (LED) display or a laser display. In one embodiment, the plurality of optical layers or lenses includes a double concave lens 82 overlying the display matrix 81 and a double convex lens 83 overlying the double concave lens 82 so that the double concave lens 82 is disposed between the double convex lens 83 and the display matrix 81 . In one embodiment, each of the lens 82 , 83 has a cross-sectional area approximating the area of the display matrix 81 . The display matrix 81 and lenses 32 , 83 are each carried by the frame or support structure 79 of the headwear 78 , such as frame 32 or glasses 81 . Each of the lenses can be movable is directions towards and away from the display matrix 81 , and suitable linear movement devices such as linear actuators (not shown) can be carried by the support structure 79 for providing such respective lineal movement. In one embodiment (not shown), only one of lenses 82 , 83 is provided and such lens is movable towards and away from the display matrix 81 by its linear actuator. In the foregoing manner, one or both of lenses 82 , 83 serve as software or computing system focusable lens. It is appreciated that other software or computer focusable lens can be provided and be within the scope of the invention. For example, a liquid crystal with a tuneable refractive index, or one or more deformable liquid or fluid-filled lenses, can be utilized instead of lenses 82 , 83 and serve as the software or computer focusable lens of the invention.
The display matrix 81 and linear actuators are electrically coupled to a suitable miniaturized computer system or computer 86 , such as computer 61 of glasses, also carried by the support structure 79 . Each such optical element assembly 80 , which can serve for example as one or both of optical elements 43 and 44 of glasses 31 , can as described above include display matrix 81 , double concave lens 82 and double convex lens 83 as well as an outwardly-facing digital sensor or image display module 87 , such as camera 69 of glasses 31 , and one or more inwardly-facing digital sensors in the form of camera 88 , which can be similar to inwardly-facing cameras 68 of glasses 31 . The optical element assemblies 80 of the headwear 78 can each be carried by the support structure 79 and electrically coupled to computer 86 .
In operation, the one or more cameras 88 and/or other inwardly-facing sensors capture the scene in real-time and feed the data into the miniaturized computer 86 . The computer then runs an operating system, alone or in combination with local application and/or networked applications that have access to some or all of the sensor data from cameras 88 , and produces output that is sent to the display matrix 81 . If no application is running, the default behavior is to pass the data from the sensors 88 through to the computer 86 , as well as apply necessary camera transformations and other software procedures, to make the user see the unmodified view as if the user was not wearing the headwear 78 .
Once one or more of the local application, the networked application and the computer 61 produce output data, such data is sent to the display matrix 81 , which converts the data into visible photons on the matrix 81 . Next, those visible photons pass through one or more of the optical lenses 81 , 82 which enhance the realism of the displayed representation of reality by for example adjusting the optical focal distance between the eye and the image viewed on the matrix 81 . As an example, the combination of convex lensing element 82 , concave lensing element 83 , and a software or computer algorithm provided in one or more of computer 86 , the local applications or the networked applications, which algorithm is informed by both knowledge of the scene and by knowledge of the view point at which the user is looking, can be used to adjust the optical focus distance between the display matrix 81 and the eye of the viewer of the viewed scene, as shown schematically in FIG. 3 . An optical path 89 of a photon, from one of the pixels on display matrix 81 , is shown in FIG. 3 , and includes a first path segment 89 a from the display matrix 81 to one surface of convex lens 82 , a second path segment 89 b from the opposite surface of convex lens 82 to one surface of concave lens 83 and a third path segment 89 c from the opposite surface of concave lens 83 to the eye of the user. Adjusting the scene's focus distance can achieve several goals, including focusing individual objects to be at their correct optical distances, so as to achieve a level of realism called volumetric three dimensional graphics which is a better than standard stereo three dimensional graphics. Such adjustment of the scene's focus distance can also render objects at slightly different distances by using a software implementation of the user's optometry prescription. For example, if the user's prescription is minus 0.5 diopters in the left eye and minus 0.25 diopters in the right eye, then the objects shown to the left eye can just be drawn 0.5 diopters closer to the eye of the viewer than reality, and the objects shown to the right eye drawn 0.25 diopters closer to the eye of the viewer than reality, which achieve the optometry prescription without requiring any prescription lenses but instead an adjustment of the input data provided to the software algorithm.
It is appreciated that in one embodiment none of the lenses in optical display assembly 80 is movable, whether the assembly 80 include both lenses 82 , 83 or only one of such lenses 83 , 83 .
One embodiment of determining the three dimensional point that the user is focusing on and/or the distance from the user's eyes to the three dimensional point that the user is focusing on, sometimes referred to herein as Z distance, for the purpose of driving a focusable display, and/or as inputs into a computer system, is illustrated in FIG. 4 . An eye-tracking sensor, such as one or more cameras 68 on right optical element holder 37 of glasses 31 or the camera 88 of headwear 78 pertaining to the right eye of the user, determines the gaze vector 96 of the right eye of the user, that is the angle and direction at which the right eye of the user is pointed or gazing at a point in time, for example when viewing an object or virtual position in space. Another or the same eye-tracking sensor, such as one or more of cameras 68 on left optical element holder 36 of glasses 31 or the camera 88 of head wear 78 pertaining to the left eye of the user, simultaneously determines the gaze vector 97 of the left eye of the user, that is the angle and direction at which the left eye of the user is pointing or gazing at the point in time, for example when viewing the object or virtual position in space being viewed by the right eye at the time of measurement. The gaze vectors 96 , 97 converge so as to intersect at such object or virtual position in space. Any suitable sensing system that can include the left eye-tracking sensor and/or the right eve-tracking sensor and/or additional eye tracking sensors tracks properties of the user's right and left eyes, for example the two pupils, irises, and/or eye muscles of the eyes, to determine the user's inter-ocular distance 98 , mood, intent, or other biometric properties that can be computed and derived within a computer system, such as computer 61 of glasses 31 , computer 86 of head wear 78 , or other local or networked computers communicating with such computers 61 or 86 . The inter-ocular distance is the distance between the eyes of the user, as shown in FIG. 4 .
A computing system, such as computer 61 of glasses 31 , computer 86 of headwear 78 , or other local or networked computers communicating with such computers 61 or 86 , uses inputs 96 - 98 , and/or other inputs potentially including digital sensor data obtained from the headwear or elsewhere, waveforms, images and/or geometry of the scene that the user is looking at, to produce outputs including a three-dimensional point or gaze point 101 in space that the user is looking at and/or a Z distance 102 , expressed in linear distance or dioptric/opticn1 distance or both, from the user's eyes to the gaze point 101 . The right gaze vector 96 intersects the left gaze vector 97 at such gaze point 101 . The three-dimensional point 101 can be used as an input to a computer software system, for example computer 61 of glasses 31 , computer 86 of headwear 78 , or other local or networked computers communicating with such computers 61 or 86 , to map a virtual position in three dimensional space where the user is looking in order for the user to control aspects of such software system. The Z distance 102 , which can be expressed as dioptric distance where the dioptric distance equals one divided by the linear distance, is useful as an input to a computing and display system of the present invention that renders visible photons that are in focus at the proper optical focus distance that the user is looking at. More detail regarding the foregoing is disclosed in a paper entitled Volumetric World Display dated May 15, 2012 authored by Jon Rodriquez, the entire content of which is incorporated herein by this reference.
FIG. 5 illustrates a user viewing an immersive, computer-generated scene that looks just as realistic as the real-world by use of one embodiment of the headwear of the present invention. Although not illustrated in FIG. 5 for simplicity, the headwear 111 of FIG. 5 can be in the form of glasses, goggles, eyewear or any of the other headwear discussed above, and can include any suitable support structure (not shown), such as a frame, for supporting the components of the headwear on the head of the user. Headwear 111 can include any suitable left and right optical elements or assemblies 113 , including any of the optical elements discussed above, mounted on or secured to the support structure. In one embodiment, each of the left and right optical elements or assemblies 113 are in the form of any suitable light field array such as a suitable plenoptic lens system. In one embodiment, each assembly 113 includes a high density digital display or matrix 114 , for example any of the displays discussed above with respect to display matrix 81 , having a display surface 116 . A lens array 117 , which can be of micro scale or nano scale, overlies the display matrix 114 and is included in the assembly 113 .
In one embodiment, the lens array 117 includes a plurality of micro-scale lenses 118 arranged in an array that can be substantially centered on an eye of the user. Although the lens array 117 can be flat or have any other suitable configuration or shape, in one embodiment the lens array has suitable transverse dimensions so as to encompass the entire peripheral vision of the user. In one embodiment, the lenses 118 are arranged in an arcuate array 117 that can be substantially centered on an eye of the user so, for example, to warp around the eye of the user, and in one embodiment such arc has substantially the same curvature of the eye and has sufficient arcuate dimensions to encompass the entire peripheral vision of the user. In one embodiment, the curvature of the arcuate array is a radial curve. Each of the micro lenses can be made from any suitable material such as plastic, glass or another suitable transparent material, and can have an outer surface 122 that faces the eye of the user during use that can be either flat, concave or convex and is shown in FIG. 5 as being concave.
The lens array 117 can be carried by or mounted on any suitable member or element, and in one embodiment is mounted on a suitable support element 119 made from any suitable transparent materials such as glass or plastic. The support element includes an arcuate surface 120 on which the lenses 118 are mounted or secured and which defines the arcuate shape and dimensions of the arcuate array 117 . Although in the illustrated embodiment the lenses 118 are fixed or non-movable, it is appreciated that lenses 118 can be movably carried by the support element 119 and be within the scope of the invention. In this regard, for example, the support element 119 can include can include nano materials and/or nano particles that are movable or actuatable for example by magnetic or electric fields to cause the lenses 118 to move on the arcuate surface 120 of the support element 119 .
Each assembly 113 can further include a suitable focusing element 121 for eliminating artifacts such as chromatic aberrations from the light passing through the assembly. In one embodiment, the focusing element 121 is disposed between the display matrix 114 and the support element 119 of the lens stray 117 .
A plurality of optical paths 131 - 135 are shown in FIG. 5 and illustrate the travel of respective photons emitted from the display matrix 114 and include a first path segment 131 a - 135 a from the display surface 116 of the matrix 114 to the rear of the support element 119 , a second path segment 131 b - 135 b from the rear of the support element 119 to the rear surface of a micro lens 118 , and a third path segment 131 c - 135 c from the outer surface 122 of the micro lens 118 to the eye of the user. The exploded portion of FIG. 5 shows a subarray 141 of a plurality of pixels 142 of display matrix 114 , and a plurality of optical paths 143 , 144 of photons emitted from a couple of such pixels 142 and having first path lengths 143 a , 143 b from the display surface 116 of the subarray 141 to the rear surface of the micro lens 118 and respective second path lengths 143 b , 144 b from the outer concave surface 122 of the micro lens 118 . As can be seen from FIG. 5 , the concave outer surface 122 of each micro lenses 118 causes the optical paths of photons emitted from the pixels 142 of the subarray 141 operated on by such lens 118 to diverge from each other as they are emitted from the outer surface 122 of the lens 118 . A convex surface on the micro lenses 118 would cause the optical paths of such a subarray 141 to converge from the outer surface 122 of the lens 118 .
In one operation of such embodiment, data or signals from a miniaturized computer running an operating system, for example computer 61 or computer 86 , and/or local application(s) and/or networked application(s), is sent to display matrix 81 , which produces photons based on the input data or signal. The optical assembly 113 , which in one embodiment includes display matrix 81 , focusing element 121 and the plurality of micro leases 118 , not only can display software controlled color and intensity like a normal pixel, it can also send different color and intensity of light in different directions, for example in the manner discussed above. The array 117 of these superpixels 118 can thus render an optical four dimensional light field, which is sufficient to reproduce the complete visual appearance of the real world, including optical focus and multi-focus, that is having different objects be at different distances in a single frame. When the micro lenses 118 are movable, the steering of light in different directions can also be assisted by other micro- and macro-scale optics layers (not shown), including optics that mechanically move or deform in response to electromagnetic fields. Such optic layers can also used to direct the light, including optical paths 131 - 135 and 143 - 144 , so that it impinges upon the user's eye from all directions, for example in a wide-angle configuration, immersing the user in the virtual world.
FIG. 6 illustrates one embodiment of a user in an immersive, computer-generated scene that looks just as realistic as the real-world. Headwear 151 therein is similar to headwear 111 and would be used with a miniaturized computer running an operating system, for example computer 61 or computer 86 , and/or local application(s) and/or networked application(s). Although not illustrated in FIG. 6 for simplicity, the headwear 151 can be in the form of glasses, goggles, eyewear or any of the other headwear discussed above, and can include any suitable support structure (not shown), such as a frame, for supporting the components of the headwear on the head of the user. Headwear 151 can include any suitable left and right optical elements or assemblies 153 , including any of the optical elements discussed above, mounted on or secured to the support structure. In one embodiment, each of the left and right optical elements or assemblies 153 are in the form of any suitable light field array such as a suitable plenoptic lens system. In one embodiment, each assembly 153 includes a high density digital display or matrix 154 , for example any of the displays discussed above with respect to display matrix 81 , having a display surface 156 . A lens array 157 , which can be of micro scale or nano scale, overlies the display matrix 156 and is included in the assembly 153 .
In one embodiment, the lens array 157 includes a plurality of micro-scale lenses 158 arranged in an array that can be substantially centered on an eye of the user. Although the lens array 157 can be arcuate or have any other suitable configuration or shape, in one embodiment the lens array has suitable transverse dimensions so as to encompass the entire peripheral vision of the user. In one embodiment, the lenses 158 are arranged in a flat array 157 that can be substantially centered on an eye of the user and has sufficient dimensions to encompass the entire peripheral vision of the user. Headwear 151 is configured so that the assembly 153 is located very close to the eye of the user, and thus has a very near ocular distance, when in use. Each of the micro lenses can be made from any suitable material such as plastic, glass or another suitable transparent material, and can have an outer surface 162 that faces the eye of the user during, use that can be either flat, concave or convex and is shown in FIG. 6 as being concave.
The lens array 157 can be carried by or mounted on any suitable member or element, and in one embodiment is mounted on a suitable support element 166 made from any suitable transparent materials such as glass or plastic. The support element 166 includes a flat surface on which the lenses 158 overlie. Although in the illustrated embodiment the lenses 158 are movable, it is appreciated that lenses 158 can be fixably carried by the support element 166 and be within the scope of the invention.
In one embodiment, the lenses 158 are movable relative to the support element 166 . Although the assembly 153 can include any suitable means tor moving the lenses 158 individually or in unison relative to the support element 166 , in one embodiment each lens 158 is mounted on a first electromagnet 167 which overlies and is movable relative to a second electromagnet 168 . The electromagnets 167 and 168 can each be of nano or micro scale size and can extend parallel to but spaced apart from each other. The first electromagnet can move in directions, including first and second orthogonal directions in its plane and thus parallel to the second electromagnet 168 , and in one embodiment can additional move towards and away from the second electromagnet 168 regard. Any suitable means can be provided for causing such movements, and in one embodiment a magnetic particle fluid, for example nano materials and/or nano particles (not shown), is disposed between the first and second electromagnets 167 , 168 and a magnetic field or flux generator or other suitable means can be provided for causing the first electromagnet 167 to move in the desired direction(s) relative to the second electromagnet 168 . In one embodiment, electrostatics are utilized to cause such movement.
Each assembly 153 can further include an optional optical layer 171 , which in one embodiment can be highly refractive, disposed between display matrix 154 and the support element 166 .
Headwear 151 includes at least one eye-tracking sensor, and ideally at least one per eye, tracking one or both eyes of the user using the headwear. In one embodiment, the eye-tracking sensor(s) can each be in the form of an inwardly-facing digital sensor or camera 172 , which can be similar to inwardly-facing cameras 68 of glasses 31 .
The exploded portion of FIG. 6 includes a plurality of RGB matrices 173 provided, for example, on the display surface 156 of the display matrix 154 . Each of the three pixels of each matrix is capable of emitting a separate photon. In FIG. 7 , the first and second electromagnets 167 , 168 are shown in three relative positions with respect to each other. Additionally in FIG. 7 , a plurality of optical paths 176 - 178 are shown and illustrate the travel of respective photons emitted from the respective RGB matrix 173 and include a first path segment 176 a - 178 a from the matrix 173 to the rear of the micro lens 158 , and a second path segment 176 b - 178 b from the outer surface 162 of the micro lens 158 to the eye of the user. As can be seen from FIG. 7 , the concave outer surface 162 of each micro lenses 158 causes the second path segments 176 b - 178 b of photons emitted from the matrix 173 to continue in a direction parallel to the first path segments 176 a - 178 a when the first electromagnet 167 is aligned, registered or concentrically disposed with respect to the second electromagnet 168 . When viewed in a plane as in FIGS. 6 and 7 , the second path segments 176 b - 178 b of photons emitted from the matrix 173 are redirected to an inclined first direction relative to or with respect to the first path segments 176 a - 178 a when the first electromagnet 167 is offset in a first direction relative to the second electromagnet 168 and are redirected to an inclined second direction relative to or with respect to the first path segments 176 a - 178 a when the first electromagnet 167 is offset in a second direction, that is opposite to the first direction, relative to the second electromagnet 168 . A convex surface on the micro lenses 158 would cause the optical paths to similarly redirect when the first electromagnet 167 is moved to an offset position relative to the second electromagnet 168 .
In operation, the scene can rendered through a combination of several means. Light color and intensity data travels to the display matrix 154 , which emits visible photons. Simultaneously, electromagnetic commands controlled by the computer system are sent to the lens array 157 of mechanically actuated micro lenses 158 whose movement and/or deformation is actuated by the interaction of such electromagnetic commands with the first and second electromagnets 167 and 168 . The effect of this movement and/or deformation of the micro lenses 158 and/or their housings and/or a liquid is to steer the light from the display matrix 154 , 173 in software-controlled directions 176 b - 178 b , resulting in the visible photons emitted by the matrices having not just software-controlled color, intensity, and (x, y) coordinates, but also software controlled left-right and up-down direction of emission. In one embodiment, such commands can be encoded as electromagnetic or acoustic waveforms. When such an array 157 is tiled across a vast array, such as display matrix 154 , and/or built into or with an optical layer containing or adjacent to such array, the resulting total system has the power to render a so-called âfour dimensional light fieldâ, meaning that the system takes in data specified with (x,y,pan,tilt) spatial coordinates, or coordinates in one of several other equivalent parameterizations of four dimensional light fields. The resulting âfour dimensional light field displayâ or âfour dimensional plenoptic displayâ is able to render multiple objects that are in focus at a variety of different focal distancesâin other words, a scene with all the optical realism of physical reality.
The inclusion of the eye trackers 172 is useful for stabilizing the displayed scene relative to the rotation of the user's eyeball, became the optical center of the eyeball is not in the same place as the rotational center of the eyeball, such stabilization is beneficial. Such eye tracking is also useful as an input to a software or other operating system of the computer (not shown) of headwear 151 and/or local application(s) and/or networked application(s) communicating with such computer, for example in an interface where the user looks at an object to seamlessly and effortlessly interact with the object.
In addition, the input from eye tracker or inwarding facing sensor or camera 172 can be used for a display technique called âsaccade amplificationâ, which increases the user's field of view by âmoving the worldâ when the user moves their eyes. For example, if the user looks left by ten degrees, the world can move right by ten degrees, resulting in a total movement of 20 degrees for a 2Ã amplification of how many degrees the user can turn their view. This could also give the user the ability to extend their peripheral far beyond the normal human field of view, for example, to include views behind the person's head. This saccade amplification technique, combined with viewing data from cameras, for example any suitable outward-facing camera included in the headwear 151 , that see to the left and right of the user's head as well as forward, results in expanded peripheral vision and awareness for the user, a great boon for drivers and pilots who need to be aware of their surroundings in all directions.
A system and methods for interactive local and remote display and control of a computer controlled via signals and with output to a light based output system can be provided. In one embodiment, illustrated in FIG. 8 , headwear 191 can be in the form of glasses, goggles, eyewear or any of the other headwear discussed above, and can include any suitable support structure (not shown), such as a frame, for supporting the components of the headwear on the head of the user. Headwear 191 can include any suitable left and right optical elements or assemblies 192 , which can be similar to any of the optical elements or assemblies discussed herein including optical elements 43 , 44 of glasses 31 , display assembly 80 of headwear 78 , optical assembly 113 of headwear 111 or optical assembly 153 of headwear 151 . In one embodiment, the optical assembly 192 includes any suitable display matrix 193 , which can be similar to display matrix 81 discussed above, and any suitable optical layer 194 , which can be similar to any of the optical, layers herein including optical layers 82 , 83 , lens array 117 or lens array 157 .
Headwear 191 can include any suitable computing system, including any of the computers disclosed herein such as computers 61 and 86 . In one embodiment, a micro computer system or computer 201 powered by a suitable rechargeable battery 202 , which can be similar to battery 62 , is provided. Computer 201 can receive a data stream from one or more image sensors 203 , which can be of any suitable type such as camera 69 , geometry sensor 73 or as combination thereof, positioned such that the image sensor 202 senses the same scene as a human eye. One or more additional image sensors 206 , which can be of any suitable type such as similar to camera 68 , EOG sensor 71 or a combination of the foregoing, is positioned such that a human eye and surrounding region is visible in the field of view of the sensor. Sensor 206 delivers a data stream to the micro computer system 201 as well. One or more additional sensors 207 , which can be of any suitable type such as EOG sensors 71 , EEG sensors 72 or any other sensor for obtaining biometric information from the user, can also be connected to the micro computer 201 . Additionally, the micro computer system or computer 201 is connected to a means of data transmission 211 to one or more networked computers, which in one embodiment can include first networked computer 212 , second networked computer 213 and third networked computer 214 , such a way that data cart be simultaneously transmitted to and from the micro computer 201 by one or more of such networked computers. The data transmission means can be of any suitable form, including wired, a local area network, a wide area network, a dynamic area network, cellular transmission, peer to peer or a combination of any of the foregoing. Each of computers 212 - 214 can be of any suitable type, and each include at least a central processing unit 216 and one or more storage mediums 217 .
The micro computer system 201 connects to a digital display assembly 192 . The digital display system or assembly 192 is positioned such that light emitted from display matrix 193 passes through one or more layers 194 of material that may modify the path of the light such that the light is delivered to a human eye.
In one method of operation, the micro computer system 201 receives inputs from one or more of sensors 203 , 206 , 207 and executes procedures in response to the inputs. In one example, a procedure processes the input from the digital sensor system 206 to determine properties of the human eye that it senses. The procedure then modifies the signals delivered to the display matrix 193 in response to the measured properties. In another example, the input stream from the outward facing sensor system 203 is processed and output to the display matrix 193 . In another example, a procedure receives data from the outward facing sensor system 203 and delivers it along the means of networked communication 211 to a networked computer in the first networked computer 212 , which in one embodiment can be a grid computer, a computing cluster or a remote cloud computer. The first networked computer 212 stores the sensed data in its digital storage medium 217 and then a collection of software or other instructions running on one or more second networked computers 213 executes procedures on the data to extract information. The procedures on the second networked computers 213 can include reality as a platform software. The extracted information is then delivered back along the network 211 to the micro computer system 201 and a procedure running on the micro computer system 201 executes a procedure to output the information to the display matrix 193 .
The foregoing procedures may be modified in response to other procedures or signals received from inputs to the micro c
CLAIMS
Claims ( 20 )
What is claimed is:
1. An apparatus comprising:
a support structure for mounting on the head of a user;
a display matrix;
a sensor device for receiving biometric data;
a microcomputer;
a transceiver coupled with the microcomputer, wherein the microcomputer executes instructions that comprise:
generating an image stream that comprises a set of images;
accessing biometric data generated by the sensor device;
generating a request that includes the biometric data to a networked computer;
receiving, from the networked computer, a presentation of the image stream that includes an object rendered at a position within the image stream based on at least the biometric data; and
causing the display matrix to display the presentation of the image stream that includes the object rendered at the position within the image stream.
2. The apparatus of claim 1 , wherein the sensor device comprises an electrooculography (EOG) sensor, and the biometric data includes eye tracking data.
3. The apparatus of claim 1 , further comprising:
determining a gaze vector based on the biometric data; and
wherein the position of the object in the image stream is based on the gaze vector.
4. The apparatus of claim 1 , wherein the support structure comprises a glasses frame, and the display matrix and sensor device are integrated into the glasses frame.
5. The apparatus of claim 1 , further comprising:
merging the image stream and the biometric data to generate a data structure; and
wherein the request includes the data structure.
6. The apparatus of claim 1 , further comprising:
detecting biometric feedback based on the sensor device; and
presenting the object within the presentation of the image stream based on the biometric feedback.
7. The apparatus of claim 1 , further comprising:
extracting metadata from the set of images of the image stream; and
generating the request based on the biometric data and the metadata.
8. A method comprising:
generating an image stream that comprises a set of images;
accessing biometric data generated by a sensor device;
generating a request that includes the biometric data to a networked computer;
receiving, from the networked computer, a presentation of the image stream that includes an object rendered at a position within the image stream based on at least the biometric data; and
causing a display matrix to display the presentation of the image stream that includes the object rendered at the position within the image stream.
9. The method of claim 8 , wherein the sensor device comprises an electrooculography (EOG) sensor, and the biometric data includes eye tracking data.
10. The method of claim 8 , further comprising:
determining a gaze vector based on the biometric data; and
wherein the position of the object in the image stream is based on the gaze vector.
11. The method of claim 8 , further comprising an apparatus that comprises a glasses frame, and the display matrix and the sensor device are integrated into the glasses frame.
12. The method of claim 8 , further comprising:
merging the image stream and the biometric data to generate a data structure; and
wherein the request includes the data structure.
13. The method of claim 8 , further comprising:
detecting biometric feedback based on the sensor device; and
presenting the object within the presentation of the image stream based on the biometric feedback.
14. The method of claim 8 , further comprising:
extracting metadata from the set of images of the image stream; and
generating the request based on the biometric data and the metadata.
15. A non-transitory computer readable storage medium storing therein instructions that, when executed by a processor, cause an apparatus for mounting on the head of a user to perform operations comprising:
generating an image stream that comprises a set of images;
accessing biometric data generated by a sensor device;
generating a request that includes the biometric data to a networked computer;
receiving, from the networked computer, a presentation of the image stream that includes an object rendered at a position within the image stream based on at least the biometric data; and
causing a display matrix to display the presentation of the image stream that includes the object rendered at the position within the image stream.
16. The non-transitory computer readable storage medium of claim 15 , wherein the sensor device comprises an electrooculography (EOG) sensor, and the biometric data includes eye tracking data.
17. The non-transitory computer readable storage medium of claim 15 , further comprising:
determining a gaze vector based on the biometric data; and
wherein the position of the object in the image stream is based on the gaze vector.
18. The non-transitory computer readable storage medium of claim 15 , wherein the apparatus comprises a glasses frame, and the display matrix and the sensor device are integrated into the glasses frame.
19. The non-transitory computer readable storage medium of claim 15 , further comprising:
merging the image stream and the biometric data to generate a data structure; and
wherein the request includes the data structure.
20. The non-transitory computer readable storage medium of claim 15 , further comprising:
detecting biometric feedback based on the sensor device; and
presenting the object within the presentation of the image stream based on the biometric feedback.
US18/458,895
2011-05-11
2023-08-30
Headware with computer and optical element for use therewith and systems utilizing same
Active
2032-11-02
US12401772B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US18/458,895
US12401772B2
( en )
2011-05-11
2023-08-30
Headware with computer and optical element for use therewith and systems utilizing same
US19/278,155
US20250350707A1
( en )
2011-05-11
2025-07-23
Headware with computer and optical element for use therewith and systems utilizing same
Applications Claiming Priority (9)
Application Number
Priority Date
Filing Date
Title
US201161485120P
2011-05-11
2011-05-11
US201213470242A
2012-05-11
2012-05-11
US201313815492A
2013-03-05
2013-03-05
US201314086909A
2013-11-21
2013-11-21
US201414328663A
2014-07-10
2014-07-10
US14/853,851
US10509466B1
( en )
2011-05-11
2015-09-14
Headwear with computer and optical element for use therewith and systems utilizing same
US16/667,489
US11099643B1
( en )
2011-05-11
2019-10-29
Headware with computer and optical element for use therewith and systems utilizing same
US17/396,976
US11778149B2
( en )
2011-05-11
2021-08-09
Headware with computer and optical element for use therewith and systems utilizing same
US18/458,895
US12401772B2
( en )
2011-05-11
2023-08-30
Headware with computer and optical element for use therewith and systems utilizing same
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US17/396,976
Continuation
US11778149B2
( en )
2011-05-11
2021-08-09
Headware with computer and optical element for use therewith and systems utilizing same
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US19/278,155
Continuation
US20250350707A1
( en )
2011-05-11
2025-07-23
Headware with computer and optical element for use therewith and systems utilizing same
Publications (2)
Publication Number
Publication Date
US20230412780A1
US20230412780A1 ( en )
2023-12-21
US12401772B2
true
US12401772B2 ( en )
2025-08-26
Family
ID=68841431
Family Applications (5)
Application Number
Title
Priority Date
Filing Date
US14/853,851
Active
2032-10-31
US10509466B1
( en )
2011-05-11
2015-09-14
Headwear with computer and optical element for use therewith and systems utilizing same
US16/667,489
Active
US11099643B1
( en )
2011-05-11
2019-10-29
Headware with computer and optical element for use therewith and systems utilizing same
US17/396,976
Active
US11778149B2
( en )
2011-05-11
2021-08-09
Headware with computer and optical element for use therewith and systems utilizing same
US18/458,895
Active
2032-11-02
US12401772B2
( en )
2011-05-11
2023-08-30
Headware with computer and optical element for use therewith and systems utilizing same
US19/278,155
Pending
US20250350707A1
( en )
2011-05-11
2025-07-23
Headware with computer and optical element for use therewith and systems utilizing same
Family Applications Before (3)
Application Number
Title
Priority Date
Filing Date
US14/853,851
Active
2032-10-31
US10509466B1
( en )
2011-05-11
2015-09-14
Headwear with computer and optical element for use therewith and systems utilizing same
US16/667,489
Active
US11099643B1
( en )
2011-05-11
2019-10-29
Headware with computer and optical element for use therewith and systems utilizing same
US17/396,976
Active
US11778149B2
( en )
2011-05-11
2021-08-09
Headware with computer and optical element for use therewith and systems utilizing same
Family Applications After (1)
Application Number
Title
Priority Date
Filing Date
US19/278,155
Pending
US20250350707A1
( en )
2011-05-11
2025-07-23
Headware with computer and optical element for use therewith and systems utilizing same
Country Status (1)
Country
Link
US
( 5 )
US10509466B1
( en )
Families Citing this family (232)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US10509466B1
( en )
2011-05-11
2019-12-17
Snap Inc.
Headwear with computer and optical element for use therewith and systems utilizing same
US11502551B2
( en )
2012-07-06
2022-11-15
Energous Corporation
Wirelessly charging multiple wireless-power receivers using different subsets of an antenna array to focus energy at different locations
US10312715B2
( en )
*
2015-09-16
2019-06-04
Energous Corporation
Systems and methods for wireless power charging
US10439972B1
( en )
2013-05-30
2019-10-08
Snap Inc.
Apparatus and method for maintaining a message thread with opt-in permanence for entries
US10438631B2
( en )
2014-02-05
2019-10-08
Snap Inc.
Method for real-time video processing involving retouching of an object in the video
US9276886B1
( en )
2014-05-09
2016-03-01
Snapchat, Inc.
Apparatus and method for dynamically configuring application component tiles
US9537811B2
( en )
2014-10-02
2017-01-03
Snap Inc.
Ephemeral gallery of ephemeral messages
US9396354B1
( en )
2014-05-28
2016-07-19
Snapchat, Inc.
Apparatus and method for automated privacy protection in distributed images
US9225897B1
( en )
2014-07-07
2015-12-29
Snapchat, Inc.
Apparatus and method for supplying content aware photo filters
US10775996B2
( en )
2014-11-26
2020-09-15
Snap Inc.
Hybridization of voice notes and calling
US10311916B2
( en )
2014-12-19
2019-06-04
Snap Inc.
Gallery of videos set to an audio time line
US9385983B1
( en )
2014-12-19
2016-07-05
Snapchat, Inc.
Gallery of messages from individuals with a shared interest
EP3272078B1
( en )
2015-03-18
2022-01-19
Snap Inc.
Geo-fence authorization provisioning
US9668217B1
( en )
2015-05-14
2017-05-30
Snap Inc.
Systems and methods for wearable initiated handshaking
US10503264B1
( en )
2015-06-16
2019-12-10
Snap Inc.
Radial gesture navigation
US11710321B2
( en )
2015-09-16
2023-07-25
Energous Corporation
Systems and methods of object detection in wireless power charging systems
US10354425B2
( en )
2015-12-18
2019-07-16
Snap Inc.
Method and system for providing context relevant media augmentation
JP2017123050A
( en )
*
2016-01-07
2017-07-13
ã½ãã¼æ ªå¼ä¼ç¤¾
Information processor, information processing method, program, and server
US10055895B2
( en )
2016-01-29
2018-08-21
Snap Inc.
Local augmented reality persistent sticker objects
US10530731B1
( en )
2016-03-28
2020-01-07
Snap Inc.
Systems and methods for chat with audio and video elements
US10596371B1
( en )
*
2016-04-19
2020-03-24
Orbital Research Inc.
Galvanic vestibular stimulation (GVS) systems, devices and methods
US10474353B2
( en )
2016-05-31
2019-11-12
Snap Inc.
Application control using a gesture based trigger
US10102423B2
( en )
2016-06-30
2018-10-16
Snap Inc.
Object modeling and replacement in a video stream
US11404056B1
( en )
2016-06-30
2022-08-02
Snap Inc.
Remoteless control of drone behavior
CN118115357A
( en )
*
2016-08-22
2024-05-31
å¥è·å ¬å¸
Virtual, augmented and mixed reality systems and methods
US10379611B2
( en )
*
2016-09-16
2019-08-13
Intel Corporation
Virtual reality/augmented reality apparatus and method
US10609036B1
( en )
2016-10-10
2020-03-31
Snap Inc.
Social media post subscribe requests for buffer user accounts
WO2018089533A1
( en )
*
2016-11-08
2018-05-17
PogoTec, Inc.
A smart case for electronic wearable device
US11635872B2
( en )
2016-11-22
2023-04-25
Snap Inc.
Smart carousel of image modifiers
US20180210628A1
( en )
2017-01-23
2018-07-26
Snap Inc.
Three-dimensional interaction system
KR102322683B1
( en )
*
2017-03-20
2021-11-08
ì¼ì±ì ì 주ìíì¬
Electronic device and method for identifying falsification of biometric information
US12074452B2
( en )
2017-05-16
2024-08-27
Wireless Electrical Grid Lan, Wigl Inc.
Networked wireless charging system
US12074460B2
( en )
2017-05-16
2024-08-27
Wireless Electrical Grid Lan, Wigl Inc.
Rechargeable wireless power bank and method of using
US11462949B2
( en )
2017-05-16
2022-10-04
Wireless electrical Grid LAN, WiGL Inc
Wireless charging method and system
CN110770717B
( en )
2017-05-17
2024-04-16
è°·ææéè´£ä»»å ¬å¸
Automatic image sharing with designated users via communication network
US10219095B2
( en )
*
2017-05-24
2019-02-26
Glen A. Norris
User experience localizing binaural sound during a telephone call
IT201700057066A1
( en )
2017-05-25
2018-11-25
St Microelectronics Srl
PROCESSING SYSTEM IMPLEMENTING AN ALGORITHM FOR THE MERGER OF DATA FROM INERTIAL SENSORS, AND METHOD
US10579869B1
( en )
2017-07-18
2020-03-03
Snap Inc.
Virtual object machine learning
US11204949B1
( en )
2017-07-31
2021-12-21
Snap Inc.
Systems, devices, and methods for content selection
US11323398B1
( en )
2017-07-31
2022-05-03
Snap Inc.
Systems, devices, and methods for progressive attachments
US10591730B2
( en )
2017-08-25
2020-03-17
II Jonathan M. Rodriguez
Wristwatch based interface for augmented reality eyewear
US10809796B2
( en )
2017-09-29
2020-10-20
Apple Inc.
Monitoring a user of a head-wearable electronic device
US11531357B1
( en )
2017-10-05
2022-12-20
Snap Inc.
Spatial vector-based drone control
WO2019094618A1
( en )
2017-11-08
2019-05-16
Signall Technologies Zrt
Computer vision based sign language interpreter
US10217488B1
( en )
2017-12-15
2019-02-26
Snap Inc.
Spherical video editing
US10523606B2
( en )
2018-01-02
2019-12-31
Snap Inc.
Generating interactive messages with asynchronous media content
US10567321B2
( en )
2018-01-02
2020-02-18
Snap Inc.
Generating interactive messages with asynchronous media content
US10825015B2
( en )
*
2018-03-07
2020-11-03
Capital One Services, Llc
Secure payment using a network of wearable devices
US11227626B1
( en )
2018-05-21
2022-01-18
Snap Inc.
Audio response messages
US20190364256A1
( en )
*
2018-05-25
2019-11-28
North Inc.
Method and System for Dynamically Generating Scene-Based Display Content on a Wearable Heads-Up Display
US11063889B2
( en )
2018-06-08
2021-07-13
Snap Inc.
Generating interactive messages with entity assets
US10764556B2
( en )
2018-09-26
2020-09-01
Snap Inc.
Depth sculpturing of three-dimensional depth images utilizing two-dimensional input selection
US11334815B1
( en )
2018-09-28
2022-05-17
Snap Inc.
Cloud based machine learning
US10796482B2
( en )
2018-12-05
2020-10-06
Snap Inc.
3D hand shape and pose estimation
US11326763B1
( en )
2019-02-06
2022-05-10
Apple Inc.
Light-emitting diodes with optical filters
CN113661660B
( en )
2019-02-06
2023-01-24
è¾è¯ºæ ¼æå ¬å¸
Method for estimating optimal phase, wireless power transmitting device and storage medium
US11861941B1
( en )
*
2019-02-06
2024-01-02
Apple Inc.
Eye camera systems with polarized light
US11012390B1
( en )
2019-03-28
2021-05-18
Snap Inc.
Media content response in a messaging system
US12071228B1
( en )
*
2019-03-28
2024-08-27
Snap Inc.
Drone with propeller guard configured as an airfoil
US11019011B1
( en )
2019-03-29
2021-05-25
Snap Inc.
Messaging system with discard user interface
US11036368B1
( en )
2019-03-29
2021-06-15
Snap Inc.
Messaging system with message transmission user interface
US11106342B1
( en )
2019-06-03
2021-08-31
Snap Inc.
User interfaces to facilitate multiple modes of electronic communication
US11151794B1
( en )
2019-06-28
2021-10-19
Snap Inc.
Messaging system with augmented reality messages
JP7260425B2
( en )
*
2019-07-08
2023-04-18
ãã¡ããã¯æ ªå¼ä¼ç¤¾
Display device
US11238611B2
( en )
*
2019-07-09
2022-02-01
Electric Avenue Software, Inc.
System and method for eyewear sizing
US11307747B2
( en )
2019-07-11
2022-04-19
Snap Inc.
Edge gesture interface with smart interactions
WO2021035646A1
( en )
*
2019-08-29
2021-03-04
æ·±å³å¸å¤§çåæ°ç§ææéå ¬å¸
Wearable device and control method therefor, gesture recognition method, and control system
KR20260046176A
( en )
2019-09-09
2026-04-06
ì¤ë ì¸ì½í¬ë ì´í°ë
Hand pose estimation from stereo cameras
US11407098B2
( en )
2019-11-26
2022-08-09
Stmicroelectronics S.R.L.
Smart push button device utilizing MEMS sensors
US12450763B2
( en )
*
2019-12-13
2025-10-21
Nvidia Corporation
Depth estimation using a neural network
CN111103689A
( en )
*
2019-12-29
2020-05-05
æå°ç§ææéå ¬å¸
Wearable equipment
US11062498B1
( en )
2019-12-30
2021-07-13
Snap Inc.
Animated pull-to-refresh
US20210216099A1
( en )
*
2020-01-09
2021-07-15
Universal City Studios Llc
Interface device for wearable visualization device, systems and methods
US11488358B2
( en )
2020-02-05
2022-11-01
Snap Inc.
Augmented reality session creation using skeleton tracking
US11265274B1
( en )
2020-02-28
2022-03-01
Snap Inc.
Access and routing of interactive messages
US11675494B2
( en )
2020-03-26
2023-06-13
Snap Inc.
Combining first user interface content into second user interface
US11409368B2
( en )
2020-03-26
2022-08-09
Snap Inc.
Navigating through augmented reality content
US11960651B2
( en )
2020-03-30
2024-04-16
Snap Inc.
Gesture-based shared AR session creation
US11455078B1
( en )
2020-03-31
2022-09-27
Snap Inc.
Spatial navigation and creation interface
CN113467078B
( en )
*
2020-03-31
2022-12-30
京ä¸æ¹ç§æéå¢è¡ä»½æéå ¬å¸
Near-eye display device and method for manufacturing same
US11547302B2
( en )
*
2020-04-21
2023-01-10
Designs For Vision, Inc.
User wearable fluorescence enabled visualization system
CN113608576B
( en )
2020-05-05
2024-06-25
ææ³å导ä½è¡ä»½æéå ¬å¸
Electronic device control method, electronic device and software product thereof
IT202000009937A1
( en )
2020-05-05
2021-11-05
St Microelectronics Srl
METHOD OF CHECKING AN ELECTRONIC DEVICE BY CALCULATION OF AN OPENING ANGLE, RELATED ELECTRONIC DEVICE AND SOFTWARE PRODUCT
US11469626B2
( en )
2020-06-28
2022-10-11
Nucurrent, Inc.
Wireless power receiver for receiving high power high frequency transfer
US11005308B1
( en )
2020-06-28
2021-05-11
Nucurrent, Inc.
Wireless power transmitter for high fidelity communications and high power transfer
US11476724B2
( en )
*
2020-06-28
2022-10-18
Nucurrent, Inc.
Higher power high frequency wireless power transfer system
US11476725B2
( en )
2020-06-28
2022-10-18
Nucurrent, Inc.
Wireless power transmitter for high fidelity communications and high power transfer
WO2022006249A1
( en )
2020-06-30
2022-01-06
Snap Inc.
Skeletal tracking for real-time virtual effects
US11404918B2
( en )
2020-07-21
2022-08-02
Nucurrent, Inc.
Wireless charging in eyewear with enhanced positional freedom
EP4197180A1
( en )
2020-08-13
2023-06-21
Snap Inc.
User interface for pose driven virtual effects
US11671559B2
( en )
2020-09-30
2023-06-06
Snap Inc.
Real time video editing
CN112346568B
( en )
*
2020-11-05
2021-08-03
广å·å¸åæ¹äººåèµæºè¯ä»·ä¸å¿æéå ¬å¸
Method and device for dynamic presentation of VR test questions based on counter and brain waves
US12229342B2
( en )
2020-12-22
2025-02-18
Snap Inc.
Gesture control on an eyewear device
US11782577B2
( en )
2020-12-22
2023-10-10
Snap Inc.
Media content player on an eyewear device
US12105283B2
( en )
2020-12-22
2024-10-01
Snap Inc.
Conversation interface on an eyewear device
KR102861516B1
( en )
2020-12-22
2025-09-18
ì¤ë ì¸ì½í¬ë ì´í°ë
Media content player on eyewear devices
US11797162B2
( en )
2020-12-22
2023-10-24
Snap Inc.
3D painting on an eyewear device
EP4272406B1
( en )
2020-12-29
2024-10-02
Snap Inc.
Body ui for augmented reality components
KR102932214B1
( en )
*
2020-12-30
2026-03-03
ì¤ë ì¸ì½í¬ë ì´í°ë
Periodic parameter estimation visual inertial tracking system
US11994751B1
( en )
2020-12-30
2024-05-28
Snap Inc.
Dual system on a chip eyewear
EP4272060A1
( en )
2020-12-31
2023-11-08
Snap Inc.
Real-time video communication interface with haptic feedback
US11989348B2
( en )
2020-12-31
2024-05-21
Snap Inc.
Media content items with haptic feedback augmentations
CN116710885A
( en )
2020-12-31
2023-09-05
æ¯çº³æ®å ¬å¸
Communication interface with haptic feedback response
WO2022147449A1
( en )
2020-12-31
2022-07-07
Snap Inc.
Electronic communication interface with haptic feedback response
US12008152B1
( en )
2020-12-31
2024-06-11
Snap Inc.
Distance determination for mixed reality interaction
US11711112B2
( en )
2021-01-28
2023-07-25
Nucurrent, Inc.
Wireless power transmission systems and methods with selective signal damping active mode
US11489555B2
( en )
2021-01-28
2022-11-01
Nucurrent, Inc.
Wireless power transmitter for high fidelity communications with amplitude shift keying
US11722179B2
( en )
2021-01-28
2023-08-08
Nucurrent, Inc.
Wireless power transmission systems and methods for selectively signal damping for enhanced communications fidelity
US11476897B2
( en )
2021-01-28
2022-10-18
Nucurrent, Inc.
Wireless power transmitter for high fidelity communications at high power transfer
US11695449B2
( en )
2021-01-28
2023-07-04
Nucurrent, Inc.
Wireless power transmission systems and methods with signal damping operating modes
US11483032B2
( en )
2021-01-28
2022-10-25
Nucurrent, Inc.
Wireless power transmission systems and methods with selective signal damping at periodic active mode windows
CN112818835B
( en )
*
2021-01-29
2024-11-22
å京大å¦
A method to quickly identify and analyze 2D materials using machine learning methods
TWI858237B
( en )
*
2021-03-05
2024-10-11
大ç«å é»è¡ä»½æéå ¬å¸
Head-mounted device
US11734959B2
( en )
2021-03-16
2023-08-22
Snap Inc.
Activating hands-free mode on mirroring device
US11908243B2
( en )
<td itemprop="prio