ABSTRACT
Abstract
The present disclosure relates to a method of providing at least one image of at least one real object captured by at least one scene camera of a plurality of scene cameras mounted on a vehicle. The method includes: providing camera poses of respective scene cameras of the plurality of scene cameras relative to a reference coordinate system associated with the vehicle, providing user attention data related to a user captured by an information capturing device, providing at least one attention direction relative to the reference coordinate system from the user attention data, determining at least one of the scene cameras among the plurality of scene cameras according to the at least one attention direction and the respective camera pose of the at least one of the scene cameras, and providing at least one image of at least one real object captured by the at least one of the scene cameras.
Description
BACKGROUND
1. Technical Field
The present disclosure is related to a method and system for providing at least one image of at least one real object captured by at least one scene camera of a plurality of scene cameras mounted on a vehicle.
2. Background Information
In a potential situation, a person may be interested in an object of a real environment (often called âobject of interestâ), e.g. in a surrounding environment. The person (herein also referred to as user) may be interested in further identifying the object of interest and/or in determining similar objects. In a particular example, the user may want to find out if the same object (also called item) and/or similar objects are available for purchasing. The user may then wish to make an order for purchasing one or more objects (e.g. the same or a similar object) or may then wish to find a store for hands-on checking and purchasing the one or more objects.
For this purpose, for example, the user may use a camera to capture an image of the object of interest and identify the same or similar items based on image analysis.
U.S. Pat. No. 8,228,364 B2 discloses a method and system for an omnidirectional camera which can be used to record an event happening around a police vehicle. The system and method include an omnidirectional camera and a digital processor that processes the images taken by the camera. The direction of the signal generated by the microphone determines the region of interest.
U.S. Pat. No. 6,580,373 B1 discloses a vehicle-mounted image record system for encouraging safe driving of a vehicle by recording images of the surface of the road and part of the vehicle. The system includes one or more cameras mounted on the vehicle and a recording device that records the images captured by the cameras. In the event of an accident the recorded images can be used as proof of safe driving.
U.S. Pat. No. 7,119,832 B1 discloses an in-car video system where a wireless microphone is configured with bi-directional communications capability. When an RF activation signal is received, the wireless microphone is automatically switched on to capture an audio soundtrack that accompanies the images captured by the car-mounted video camera. A wireless microphone controller mounted in the car transmits the RF activation signal to the wireless microphone. When the video recording device starts recording, the wireless microphone controller transmits the RF activation signal.
Further, there is known a mobile phone application (so-called âAppâ) called âASAP54â that allows the user, upon seeing a particular outfit or style the user is interested in on a real person or magazine, to take a photograph of the outfit. By using image recognition the application can find that piece of clothing the user is looking at and a number of similar fashions. For employing this, the user is required to focus the camera of his/her mobile phone on the object of interest.
SUMMARY
It would be desirable to provide a method and system which facilitate for a user to retrieve information on a region of interest in the surrounding real environment of the user which can be electronically stored and processed for later use.
According to a first aspect, there is disclosed a method of providing at least one image of at least one real object captured by at least one scene camera of a plurality of scene cameras mounted to a vehicle, the method comprising providing camera poses of respective scene cameras of the plurality of scene cameras relative to a reference coordinate system associated with the vehicle, providing user attention data related to at least one user captured by an information capturing device, providing at least one attention direction relative to the reference coordinate system from the user attention data, determining at least one of the scene cameras among the plurality of scene cameras according to the at least one attention direction and the respective camera pose of the at least one of the scene cameras, and providing at least one image of at least one real object captured by the at least one of the scene cameras.
According to another aspect, there is disclosed a system for providing at least one image of at least one real object, comprising at least one processing device coupled to a plurality of scene cameras mounted to a vehicle, and to an information capturing device configured to capture user attention data related to at least one user. The at least one processing device is configured to provide camera poses of respective scene cameras of the plurality of scene cameras relative to a reference coordinate system associated with the vehicle, to provide at least one attention direction relative to the reference coordinate system from the user attention data, to determine at least one of the scene cameras among the plurality of scene cameras according to the at least one attention direction and the respective camera pose of the at least one of the scene cameras, and to provide at least one image of at least one real object captured by the at least one of the scene cameras.
For example, the at least one processing device according to the present disclosure, which may comprise one or more processing devices such as one or more microprocessors, is comprised, at least in part, in a mobile device (such as a mobile phone, wearable computer, tablet computer, mobile computer, often called laptop, or a head mounted display, such as used for optical see-through augmented reality applications), in the vehicle, and/or in a server computer adapted to communicate with the mobile device and/or the vehicle. The at least one processing device may be comprised in only one of these devices, e.g. in the mobile device or in the server computer, or may be a distributed system in which one or more processing tasks are distributed and processed by one or more processing devices of a processing system which are distributed and are communicating with each other, e.g. by point to point communication or via a network.
Any steps, embodiments, aspects and examples described herein with respect to the method can equally or analogously be implemented by the at least one processing device being configured (by software and/or hardware) to perform the respective steps, embodiments, aspects or examples. Any used processing device, such as one or more microprocessors, may be configured as such by software and/or hardware and communicate via a communication network, e.g. via a server computer or a point to point communication, with one or more cameras, displays, sensors and/or any other components disclosed herein.
According to another aspect, the disclosure is also related to a computer program product comprising software code sections which are adapted to perform a method according to the invention, particularly as set out in any one of the claims. Particularly, the software code sections are contained on a computer readable medium which is non-transitory. The software code sections may be loaded into the memory of one or more processing devices (such as microprocessors) as described herein. Any used processing devices, such as one or more microprocessors, may communicate via a communication network, e.g. via a server computer or a point to point communication, with other devices as described herein.
BRIEF DESCRIPTION OF THE DRAWINGS
Aspects and embodiments of the invention will now be described with respect to the drawings, in which:
FIG. 1 shows a flow diagram of a method according to an embodiment of the invention.
FIG. 2 shows an exemplary scenario according to an embodiment of the invention for providing at least one image of at least one real object captured by at least one camera mounted on a vehicle.
FIG. 3 shows an embodiment of a system setup which may be used in connection with aspects of the invention.
FIG. 4 shows another exemplary scenario according to an embodiment of the invention.
FIGS. 5 A and 5 B show further exemplary scenarios according to embodiments of the invention.
FIG. 6 shows another exemplary scenario according to an embodiment of the invention.
FIG. 7 shows a flow diagram of a method according to a further embodiment of the invention.
FIG. 8 shows a scenario of a user holding a mobile device that has a front facing camera, a back facing camera, and a display screen for recognizing, tracking and/or reconstructing an object of interest.
FIG. 9 shows another scenario of a user holding a mobile device that has a front facing camera, a back facing camera, and a display screen for recognizing, tracking and/or reconstructing an object of interest.
FIG. 10 shows a flow diagram of a method according to an embodiment of the invention.
FIG. 11 shows a flow diagram of a method according to another embodiment of the invention.
DESCRIPTION
Nowadays, people often spend a lot of time in a vehicle, for example when they are on the way to shopping, commuting or sightseeing. Potentially, there may be many different objects (e.g. pedestrians, clothes worn by pedestrians, advertisement posters, real stores, etc.) in the surroundings of the vehicle when it travels along its way. It would be difficult and impractical for people sifting in a vehicle to use, e.g., a mobile device equipped with a camera (e.g. a standard camera or a mobile phone with a camera) for capturing an image of an object of interest of an environment surrounding the vehicle. This is particularly the case for a driver who is driving the vehicle. The driver would not be able and allowed to hold the mobile device to capture an image during driving.
In a potential exemplary scenario, in which a person is driving a car from one location to another, he or she may find an object of interest, for instance a skirt worn by a pedestrian walking on a sidewalk next to the vehicle. The inventors have found that it is beneficial to employ cameras mounted on the vehicle to capture an image containing the object of interest, instead of the driver holding a camera for capturing an image.
Increasingly, multiple cameras are mounted on vehicles, such as cars. The inventors further considered that images captured by each of the car mounted cameras and the processing thereof would increase complexity of object detection and/or recognition. Thus, the inventors found that it would be beneficial to determine a subset (i.e. one or more) of the vehicle mounted cameras and further process or analyze only images captured by the determined subset of the vehicle mounted cameras.
Aspects of the present disclosure are related to a method and system for determining at least one of a plurality of vehicle mounted cameras for capturing at least one image of a part of a real environment based on the direction and/or position of the user's attention while being in the vehicle. At least one image of an environment captured by the determined at least one camera can be used to determine one or more items (i.e. objects of interest) of the real environment.
The provided user attention data, e.g. captured by an information capturing device, according to the present invention are related to the user. Particularly, the user attention data are indicative of a user's attention towards a real object (particularly the object of interest) or a part of the real environment containing the real object.
According to an embodiment of the invention, as set out in more detail below, it is possible to look at an object of interest in the surroundings of the vehicle and to activate one or more vehicle cameras, e.g. by using voice, or gesture, or gaze direction, to capture an image of the object of interest (e.g., a piece of clothing), and then to determine whether the object of interest or similar items are available for purchasing.
FIG. 1 shows a flow diagram of a method according to an embodiment of the invention providing at least one image of at least one real object captured by at least one camera mounted on a vehicle and further shows optional steps related to exemplary applications based on objects of interest determined according to the at least one image. FIG. 2 shows an exemplary scenario according to an embodiment of the invention for providing at least one image of at least one real object captured by at least one camera mounted on a vehicle. FIG. 3 shows an embodiment of a system setup which may be used in connection with aspects of the invention.
In the scenario of FIG. 2 , a vehicle, in this example a car 201 , is driving through a real environment 220 . The real environment 220 includes real objects fixed in the environment, e.g. a gas station 221 , a tree 222 , a building 223 , a park sign 224 , and a road 227 . The real environment 220 may also include movable real objects, like a person 226 and a person 225 . The car 201 may also be considered as a part of the real environment 220 . The car 201 is equipped with multiple scene cameras 211 - 214 mounted on the car. The driver 202 is looking or facing or pointing toward the <figure-callout id="203" label="
BACKGROUND
1. Technical Field
The present disclosure is related to a method and system for providing at least one image of at least one real object captured by at least one scene camera of a plurality of scene cameras mounted on a vehicle.
2. Background Information
In a potential situation, a person may be interested in an object of a real environment (often called âobject of interestâ), e.g. in a surrounding environment. The person (herein also referred to as user) may be interested in further identifying the object of interest and/or in determining similar objects. In a particular example, the user may want to find out if the same object (also called item) and/or similar objects are available for purchasing. The user may then wish to make an order for purchasing one or more objects (e.g. the same or a similar object) or may then wish to find a store for hands-on checking and purchasing the one or more objects.
For this purpose, for example, the user may use a camera to capture an image of the object of interest and identify the same or similar items based on image analysis.
U.S. Pat. No. 8,228,364 B2 discloses a method and system for an omnidirectional camera which can be used to record an event happening around a police vehicle. The system and method include an omnidirectional camera and a digital processor that processes the images taken by the camera. The direction of the signal generated by the microphone determines the region of interest.
U.S. Pat. No. 6,580,373 B1 discloses a vehicle-mounted image record system for encouraging safe driving of a vehicle by recording images of the surface of the road and part of the vehicle. The system includes one or more cameras mounted on the vehicle and a recording device that records the images captured by the cameras. In the event of an accident the recorded images can be used as proof of safe driving.
U.S. Pat. No. 7,119,832 B1 discloses an in-car video system where a wireless microphone is configured with bi-directional communications capability. When an RF activation signal is received, the wireless microphone is automatically switched on to capture an audio soundtrack that accompanies the images captured by the car-mounted video camera. A wireless microphone controller mounted in the car transmits the RF activation signal to the wireless microphone. When the video recording device starts recording, the wireless microphone controller transmits the RF activation signal.
Further, there is known a mobile phone application (so-called âAppâ) called âASAP54â that allows the user, upon seeing a particular outfit or style the user is interested in on a real person or magazine, to take a photograph of the outfit. By using image recognition the application can find that piece of clothing the user is looking at and a number of similar fashions. For employing this, the user is required to focus the camera of his/her mobile phone on the object of interest.
SUMMARY
It would be desirable to provide a method and system which facilitate for a user to retrieve information on a region of interest in the surrounding real environment of the user which can be electronically stored and processed for later use.
According to a first aspect, there is disclosed a method of providing at least one image of at least one real object captured by at least one scene camera of a plurality of scene cameras mounted to a vehicle, the method comprising providing camera poses of respective scene cameras of the plurality of scene cameras relative to a reference coordinate system associated with the vehicle, providing user attention data related to at least one user captured by an information capturing device, providing at least one attention direction relative to the reference coordinate system from the user attention data, determining at least one of the scene cameras among the plurality of scene cameras according to the at least one attention direction and the respective camera pose of the at least one of the scene cameras, and providing at least one image of at least one real object captured by the at least one of the scene cameras.
According to another aspect, there is disclosed a system for providing at least one image of at least one real object, comprising at least one processing device coupled to a plurality of scene cameras mounted to a vehicle, and to an information capturing device configured to capture user attention data related to at least one user. The at least one processing device is configured to provide camera poses of respective scene cameras of the plurality of scene cameras relative to a reference coordinate system associated with the vehicle, to provide at least one attention direction relative to the reference coordinate system from the user attention data, to determine at least one of the scene cameras among the plurality of scene cameras according to the at least one attention direction and the respective camera pose of the at least one of the scene cameras, and to provide at least one image of at least one real object captured by the at least one of the scene cameras.
For example, the at least one processing device according to the present disclosure, which may comprise one or more processing devices such as one or more microprocessors, is comprised, at least in part, in a mobile device (such as a mobile phone, wearable computer, tablet computer, mobile computer, often called laptop, or a head mounted display, such as used for optical see-through augmented reality applications), in the vehicle, and/or in a server computer adapted to communicate with the mobile device and/or the vehicle. The at least one processing device may be comprised in only one of these devices, e.g. in the mobile device or in the server computer, or may be a distributed system in which one or more processing tasks are distributed and processed by one or more processing devices of a processing system which are distributed and are communicating with each other, e.g. by point to point communication or via a network.
Any steps, embodiments, aspects and examples described herein with respect to the method can equally or analogously be implemented by the at least one processing device being configured (by software and/or hardware) to perform the respective steps, embodiments, aspects or examples. Any used processing device, such as one or more microprocessors, may be configured as such by software and/or hardware and communicate via a communication network, e.g. via a server computer or a point to point communication, with one or more cameras, displays, sensors and/or any other components disclosed herein.
According to another aspect, the disclosure is also related to a computer program product comprising software code sections which are adapted to perform a method according to the invention, particularly as set out in any one of the claims. Particularly, the software code sections are contained on a computer readable medium which is non-transitory. The software code sections may be loaded into the memory of one or more processing devices (such as microprocessors) as described herein. Any used processing devices, such as one or more microprocessors, may communicate via a communication network, e.g. via a server computer or a point to point communication, with other devices as described herein.
BRIEF DESCRIPTION OF THE DRAWINGS
Aspects and embodiments of the invention will now be described with respect to the drawings, in which:
FIG. 1 shows a flow diagram of a method according to an embodiment of the invention.
FIG. 2 shows an exemplary scenario according to an embodiment of the invention for providing at least one image of at least one real object captured by at least one camera mounted on a vehicle.
FIG. 3 shows an embodiment of a system setup which may be used in connection with aspects of the invention.
FIG. 4 shows another exemplary scenario according to an embodiment of the invention.
FIGS. 5 A and 5 B show further exemplary scenarios according to embodiments of the invention.
FIG. 6 shows another exemplary scenario according to an embodiment of the invention.
FIG. 7 shows a flow diagram of a method according to a further embodiment of the invention.
FIG. 8 shows a scenario of a user holding a mobile device that has a front facing camera, a back facing camera, and a display screen for recognizing, tracking and/or reconstructing an object of interest.
FIG. 9 shows another scenario of a user holding a mobile device that has a front facing camera, a back facing camera, and a display screen for recognizing, tracking and/or reconstructing an object of interest.
FIG. 10 shows a flow diagram of a method according to an embodiment of the invention.
FIG. 11 shows a flow diagram of a method according to another embodiment of the invention.
DESCRIPTION
Nowadays, people often spend a lot of time in a vehicle, for example when they are on the way to shopping, commuting or sightseeing. Potentially, there may be many different objects (e.g. pedestrians, clothes worn by pedestrians, advertisement posters, real stores, etc.) in the surroundings of the vehicle when it travels along its way. It would be difficult and impractical for people sifting in a vehicle to use, e.g., a mobile device equipped with a camera (e.g. a standard camera or a mobile phone with a camera) for capturing an image of an object of interest of an environment surrounding the vehicle. This is particularly the case for a driver who is driving the vehicle. The driver would not be able and allowed to hold the mobile device to capture an image during driving.
In a potential exemplary scenario, in which a person is driving a car from one location to another, he or she may find an object of interest, for instance a skirt worn by a pedestrian walking on a sidewalk next to the vehicle. The inventors have found that it is beneficial to employ cameras mounted on the vehicle to capture an image containing the object of interest, instead of the driver holding a camera for capturing an image.
Increasingly, multiple cameras are mounted on vehicles, such as cars. The inventors further considered that images captured by each of the car mounted cameras and the processing thereof would increase complexity of object detection and/or recognition. Thus, the inventors found that it would be beneficial to determine a subset (i.e. one or more) of the vehicle mounted cameras and further process or analyze only images captured by the determined subset of the vehicle mounted cameras.
Aspects of the present disclosure are related to a method and system for determining at least one of a plurality of vehicle mounted cameras for capturing at least one image of a part of a real environment based on the direction and/or position of the user's attention while being in the vehicle. At least one image of an environment captured by the determined at least one camera can be used to determine one or more items (i.e. objects of interest) of the real environment.
The provided user attention data, e.g. captured by an information capturing device, according to the present invention are related to the user. Particularly, the user attention data are indicative of a user's attention towards a real object (particularly the object of interest) or a part of the real environment containing the real object.
According to an embodiment of the invention, as set out in more detail below, it is possible to look at an object of interest in the surroundings of the vehicle and to activate one or more vehicle cameras, e.g. by using voice, or gesture, or gaze direction, to capture an image of the object of interest (e.g., a piece of clothing), and then to determine whether the object of interest or similar items are available for purchasing.
FIG. 1 shows a flow diagram of a method according to an embodiment of the invention providing at least one image of at least one real object captured by at least one camera mounted on a vehicle and further shows optional steps related to exemplary applications based on objects of interest determined according to the at least one image. FIG. 2 shows an exemplary scenario according to an embodiment of the invention for providing at least one image of at least one real object captured by at least one camera mounted on a vehicle. FIG. 3 shows an embodiment of a system setup which may be used in connection with aspects of the invention.
In the scenario of FIG. 2 , a vehicle, in this example a car 201 , is driving through a real environment 220 . The real environment 220 includes real objects fixed in the environment, e.g. a gas station 221 , a tree 222 , a building 223 , a park sign 224 , and a road 227 . The real environment 220 may also include movable real objects, like a person 226 and a person 225 . The car 201 may also be considered as a part of the real environment 220 . The car 201 is equipped with multiple scene cameras 211 - 214 mounted on the car. The driver 202 is looking or facing or pointing toward the direction 203 .
FIG. 3 shows an embodiment of an interior setup for the car 201 . There are provided two information capturing devices mounted in the car 201 . In principle, one may suffice. In this example, the information capturing devices are comprising cameras, herein called user cameras
205 and 206 . The car 201 may further be equipped with a communicating device 207 , such as a wireless communication device (e.g. WLAN device or SIM card device), and a processing device 208 , such as a microprocessor. All the steps or a part of the steps disclosed in this disclosure may be performed by the processing device 208 alone or in combination with any other processing device. All the steps or a part of the steps may also be performed by a remote processing device that is separate to the car 201 , such as a server computer or a mobile device. In the present case, the car 201 may communicate with the remote processing device through the communicating device 207 via cable or wirelessly. The remote processing device may be a server computer 301 (e.g. a workstation) or a mobile device, e.g. a mobile phone 302 .
According to the flow diagram of FIG. 1 , step 101 captures user attention data related to a user (in FIG. 2 , the driver 202 ) by an information capturing device (such as one of the cameras
205 , 206 according to FIG. 3 ). The user attention data may be any data that represents at least one aspect of a user attention. Particularly, the user attention data represents or encodes information related to at least one direction, at least one position, and/or at least one indicated space or area of the user attention.
The user attention may be indicated by a gaze (or a stare). A direction from the user's eyes to where the eyes are looking may represent the gaze direction, which may be considered as an attention direction. Further, a field of view of one eye or two eyes of the user represents a space of the user attention of the gaze. A position where the user is looking at represents a position of the user attention.
In another example, the user attention may also be indicated by the user's face (e.g. a pose of the face or head). The pose of the user's face may represent where the user is focusing. At least one attention direction may be derived from the pose of the face. In one implementation, the at least one attention direction may be the same as the normal direction of the frontal face.
In a further example, the user attention may be indicated by finger pointing or any gesture indicative of directions, positions, and/or areas. An attention direction may be modeled by a direction axis. For example, the direction axis may be represented by a 2-vector. Further, the attention direction may be modeled by a field of view. For example, the user's position may determine the view point, and the field of view of the user's eye may define an attention direction. In another example, standard deviations of the direction axis (e.g. estimated errors of the frontal face direction from the face pose estimation) may determine an angle (e.g. vertical or horizontal range) for the field of view, which may be considered as an attention direction.
For example, the user attention data comprises at least one of, but is not limited to, one or more images captured by one or more cameras, a bioelectric signal (e.g. electrooculogram), and a mechanical signal (e.g. hand pressure).
In one embodiment, the information capturing device may comprise a camera device called user camera. The user camera may capture at least one user image of at least part of the user. For example, the camera 205 (i.e. a user camera) mounted on the car 201 may capture an image (i.e. a user image) of the front face of the user 202 , as shown in FIG. 3 .
In another embodiment, the information capturing device may be an eye tracking device. The eye tracking device (also called eye tracker) may measure the orientation of one or two eyes of the user and, thus, can provide gaze directions of the user. There are different types of eye tracking methods, like eye-attached tracking, optical tracking, and electric potential measurement. The eye-attached tracking may be implemented as special contact lens with an embedded sensor (like mirror or magnetic field sensors). The optical tracking can employ cameras to capture images of the eyes and determine the eye orientation from the images, for example as disclosed in Kaminski, Jeremy Yrmeyahu, DotanKnaan, and AdiShavit. âSingle image face orientation and gaze detection.â Machine Vision and Applications 21.1 (2009): 85-98 (hereinafter âKaminski et al.â). The electric potential measurement devices can measure electric potentials with electrodes placed around the eyes. One technique called electrooculography (EOG) system can measure electric potentials (the measured signal called electrooculogram). Bulling et al. present a wearable EOG goggle; e.g. see Bulling, Andreas, Daniel Roggen, and Gerhard Tröster. âWearable EOG goggles; Seamless sensing and context-awareness in everyday environments.â Journal of Ambient Intelligence and Smart Environments 1.2 (2009): 157-171.
In another embodiment, the information capturing device may be a mechanical sensor, like a pressure or force sensor. For example, it measures force or pressure applied by the user. The mechanical sensor may be a mechanical joystick.
Step 102 provides at least one attention direction of the user relative to a reference coordinate system associated with a vehicle, wherein the at least one attention direction is derived from the user attention data. For example, the attention direction 203 of the user 202 (i.e. driver) could be determined or defined in the reference coordinate system 209 associated with car 201 . The attention direction 203 may indicate a gaze direction or face direction of the user 202 , which may be estimated from an image (i.e. the user attention data) of the face of the user 202 captured by the camera 205 (i.e. the information capturing device) mounted in the car 201 .
In some implementations, attention direction information may be contained in the captured user attention data. For example, an eye tracker may provide a gaze direction in the output signals. In other implementations, attention direction information may be derived (e.g. estimated) from the captured user attention data. For example, when a camera is used to capture one or more images of the user's face or eyes, the face pose or the gaze direction may have to be estimated from the camera images based on a computer vision method like that disclosed in Kaminski et al. or in Fanelli, Gabriele, Juergen Gall, and Luc Van Gool. âReal time head pose estimation with random regression forests.â Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011 (hereinafter âFanelli et al.). According to the invention, the step of providing at least one attention direction relative to the reference coordinate system from the user attention data shall encompass all of these implementations and embodiments.
The attention direction may be determined relative to the information capturing device. In order to have the attention direction in a reference coordinate system associated with the vehicle, a device spatial relationship, e.g. 6 DOF (degrees of freedom) rigid transformation, between the vehicle and the information capturing device may be required. Then, the attention direction relative to the reference coordinate system may be determined from the attention direction relative to the information capturing device and the device spatial relationship.
The device spatial relationship may be determined from a calibration procedure. The calibration procedure is, for example, a mechanical calibration. For instance, the information capturing device (e.g. the camera 205 ) may be mounted at a known pose in the reference coordinate system of the vehicle (e.g. the car 201 ) using mechanical arms. The camera 205 may also be mounted at an arbitrary pose (i.e. unknown at the moment of the mounting). In this case, the camera 205 could capture an image of a part of the car 201 . The image of the part of the car can be used to estimate the device spatial relationship based on a computer vision method (e.g. feature based pose estimation). It is also possible to use another tracking system to determine the device spatial relationship between the vehicle and the information capturing device. The tracking system may be a mechanical arm, an optical camera system, or a magnetic tracking system, or any motion or position sensor (e.g. gravity sensor, accelerometer, GPS).
In one embodiment, the information capturing device is or comprises one or more cameras. As shown in FIG. 3 , the camera 205 (and 206 ) mounted inside the car 201 is part of the information capturing device. For example, the camera 205 has a known device spatial relationship with the reference coordinate system 209 associated with the car 201 . It is possible to capture a user image of at least part of the face of the user 202 .
The pose of the user's face can be estimated from the user image based on various computer vision methods (like proposed in Fanelli et al.). From the face pose, a direction of the frontal face can be determined as an attention direction. The attention direction 203 shown in FIG. 2 may represent the face direction.
According to an embodiment, multiple attention directions are determined. When the user image contains multiple faces (of multiple users sitting in the car), multiple face poses may be determined. In another example, the face of the user may move, and then multiple face directions may be determined for the same face. In this case, it is possible to estimate one main direction from the multiple attention directions. Different mathematical methods can be employed to determine a main direction from multiple directions. For example, each direction could have an angle relative to a common coordinate system. The main direction may be determined by an angle, which may be computed as an average, maximum, minimum, median or mean of the angles associated with the multiple directions. In FIG. 2 , the attention direction 203 may be one direction estimated based on one user or a main direction based on multiple attention (e.g. face and/or gaze) directions estimated from one or more user images of one or more users.
It is also possible to estimate a gaze direction from the user image of least part of the face of the user 202 (e.g. as proposed in Kaminski et al.). The attention direction 203 shown in FIG. 2 can represent the gaze direction. The gaze direction of the user 202 may also be estimated from an eye tracker.
Further, when one or more user images capture at least part of a hand or an arm of the user, a hand pose can be estimated from the one or more user images according to any appropriate vision based method (e.g. like that proposed in de La Gorce, Martin, David J. Fleet, and Nikos Paragios. âModel-Based 3D Hand Pose Estimation from Monocular Videoâ or Erol, Ali, et al. âVision-based hand pose estimation: A review.â Computer Vision and Image Understanding 108.1 (2007): 52-73 (hereinafter âErol et al.â)). A hand gesture (or generally a gesture) may also be estimated, like according to a method as proposed in Erol et al. A direction (e.g. a pointing direction) as an attention direction may be derived from the hand pose. A hand pointing direction, or generally a gesture, may also be computed from the one or more user images. The attention direction 203 shown in FIG. 2 may represent such hand pointing direction.
According to an embodiment, face poses, gaze directions, and/or hand poses are estimated for one or more users from the same user attention data (e.g. from the same user image captured by a camera). Multiple directions (e.g. at least two of face directions, gaze directions, and hand directions) may be determined. The attention direction 203 may be determined as one of the multiple directions or as a main direction estimated based on the multiple directions.
In another embodiment, the camera 205 has unknown device spatial relationships with respect to the reference coordinate system 209 when a user image is captured by the camera 205 . It is possible to determine the attention direction 203 relative to the reference coordinate system from the user image captured by the camera 205 without knowing the device spatial relationship. In an example, the user image captured by the camera 205 contains at least part of the car and at least part of the user. The at least part of the car has a known pose relative to the reference coordinate system of the car. The at least part of the user may contain the user face. In this case, the pose of the user's face or gaze direction can be determined in the reference coordinate system of the car based on a computer vision method using pixel information of the captured user image.
Step 103 provides a plurality of scene cameras mounted to the vehicle at respective camera poses relative to the reference coordinate system. In the embodiment shown in FIG. 2 , four scene cameras 211 - 214 are mounted to the car 201 , and the respective camera poses of the scene cameras 211 - 214 are known in the reference coordinate system 209 .
Step 104 determines at least one of the scene cameras among the plurality of scene cameras according to the at least one attention direction and at least one respective camera pose. For example, it is possible to determine at least one of the scene cameras 211 - 214 as desired scene camera(s) according to the attention direction 203 and the respective camera poses of at least part of the scene cameras 211 - 214 . One thought behind this is that attention directions of the user and/or positions of the user could indicate where or in which region or along which direction an object of interest locates in the real environment.
In an embodiment, multiple scene cameras mounted to the vehicle capture different regions of the real environment. For example, the scene cameras 211 - 214 capture different regions of the real environment 220 (e.g. four different sides) around the car 201 . For example, they are arranged facing perpendicularly to one another, thus being directed towards four perpendicular sides. At least one attention direction of the user may be used to determine at least one scene camera among the scene cameras. The determined scene camera(s) could capture at least one scene image that may contain at least part of the object of interest indicated by the at least one attention direction.
The at least one scene image may be processed by various computer vision methods in order to recognize an object of interest, determine objects similar to the object of interest, reconstruct the 3D geometry of the object of interest, determine the position of the object of interest relative to the car or to the real environment, and/or determine the position of the vehicle in the real environment. Further, an attention direction relative to the car or relative to the scene camera may be considered in the computer vision methods. For example, the attention direction may be used to determine a region of interest in the at least one scene image. In another example, the attention direction may be used to determine image features based on distances between the corresponding features and the attention direction in 3D space or image distances between the image features and the image projection of the attention direction in the captured image.
A scene camera determined from attention directions:
It is possible to determine a scene camera (called desired scene camera) among the plurality of scene cameras according to an attention direction of the user. Further, multiple desired scene cameras among the plurality of scene cameras could also be determined similarly according to methods mentioned below. In one implementation, spatial relationships between the attention direction and each respective camera direction of at least part of the plurality of scene cameras is evaluated to determine a desired scene camera. The camera directions can be derived from related camera poses. A camera direction may indicate the direction of a respective camera optical axis.
In an example, a spatial relationship between the attention direction and a respective camera direction is an angle between the two direction axes. A threshold value may be given, and then a scene camera may be determined as a desired scene camera if the related angle is below the threshold. It is also possible to select a scene camera as a desired scene camera if the angle related to the selected scene camera is the smallest among angles related to the at least part of the scene cameras.
In another example, a spatial relationship between the attention direction and a respective camera direction is defined as an intersection. A camera direction may be defined as it originates from the position of the camera. It is also possible to select a scene camera as a desired scene camera if the direction axis of the selected scene camera intersects the attention direction. Further, when the attention direction intersects the direction axes of multiple scene cameras, the multiple scene cameras can be determined as desired scene cameras.
In a further embodiment, a desired scene camera is determined according to spatial relationships between the attention direction and each respective camera position of at least part of the plurality of scene cameras. A camera position may be derived from a camera pose of a related scene camera. A spatial relationship between the attention direction and a respective camera position may be defined as a distance from the camera position to the attention direction axis or a distance from the camera position to the user.
In a further embodiment, a desired scene camera is determined according to spatial relationships between the attention direction and each respective camera pose (including both direction and position) of at least part of the plurality of scene cameras.
In a further embodiment, a capturing coverage (e.g. field of view of a camera or manually defined) of a scene camera is provided. As shown in FIG. 2 , the scene camera 211 - 214 cover front left, back and right of the car 201 , respectively. When the attention direction is determined to point right of the car 201 , then the scene camera 214 is determined as a desired camera.
In another embodiment, a desired scene camera is determined according to spatial relationships between the attention direction and each respective camera field of view of at least part of the plurality of scene cameras. For example, a spatial relationship between the attention direction and a respective camera field of view may be defined as intersects, (partially) covered by, or (partially) covers. A scene camera may be determined as a desired scene camera if the related camera field of view covers the attention direction or has the largest cover (i.e. an uncovered part of the attention direction is the smallest) or covers a certain part (e.g. a certain part beginning from the user position) of the attention direction among the at least part of the scene cameras. In another implementation, when the attention direction is covered by multiple scene cameras, the multiple scene cameras may be determined as desired scene cameras. When depth information is available for the scene cameras, the camera field of view may be limited to a certain depth based on the depth information, as objects behind the certain depth with respect to the camera would be occluded and not be captured in images.
One or more embodiments of determining at least one scene camera as at least one desired scene cameras disclosed herein could be combined for the determination.
In another embodiment, a scene camera may be an omni-camera (or a wide-angle camera) mounted to the car. The at least one attention direction may be used to determine a region of interest in at least one scene image captured by the omni-camera.
In a further embodiment, it is possible to determine a scene camera (called desired scene camera) according to multiple attention directions. The multiple attention directions may come from one user or different users. The multiple attention directions may be obtained from the same or several different user attention data.
In one implementation, a desired attention direction (may be or may be not one of the multiple attention directions) may be estimated from the multiple attention directions. Each respective attention direction of the multiple attention directions has an angle relative to a common axis in a common coordinate system (e.g. the reference coordinate system of the car). The desired attention direction may be estimated to have a maximum, minimum, average, mean, or median angle based on the angles of the multiple attention directions.
The multiple attention directions may be clustered or grouped (e.g. according to their angles or direction axes). A desired attention direction may be estimated from a group with the majority of the multiple attention directions.
In a further implementation, for each respective attention direction of the multiple attention directions, it is possible to determine an angle, a distance, a covered region relative to a scene camera according to the embodiments mentioned above. For the scene camera, statistics related to angles, distances, covered regions of the multiple attention directions may be calculated. At least one scene camera may be selected (i.e. determined) from the at least part of the scene cameras according to the calculated statistics. For example, a sum of the angles of the multiple attention directions related to a scene camera may be calculated. One or more scene cameras having minimal values may be determined as desired cameras.
Step 105 provides at least one scene image of at least one real object captured by the determined at least one scene camera. In an embodiment, the plurality of scene cameras capture a plurality of scene images. Then, at least one scene image captured by the determined at least one scene camera (i.e. desired scene camera) is provided. For example, each of the scene cameras 211 - 214 captures a respective scene image. The scene camera 214 may be determined as the desired scene camera according to the attention direction 203 . The scene image captured by the scene camera 214 may be provided for further processing.
In an embodiment, the step of capturing the plurality of scene images and the step of capturing the user attention data may be synchronized.
In another embodiment, the step of capturing the plurality of scene images and the step of capturing the user attention data are not synchronized. For example, the user attention data may be captured and at least one attention direction is estimated. Then, at least one scene camera is determined according to the at least one attention direction. Afterwards, the determined at least one scene camera captures at least one scene image.
There are many computer vision applications designed for use in or with vehicles which could take advantages of the at least one scene image captured by the determined at least one scene camera mounted to the vehicle.
In an example, it is possible to determine at least one object of interest according to the at least one scene image as shown in the optional step 106 . Different potential embodiments related to the determination of the at least one object of interest are described together with FIG. 7 (e.g. for step 702 ).
It is also optional to perform step 107 to generate purchasing information related to the at least one object of interest. One specific potential embodiment is described together with FIG. 7 .
It is further optional to perform step 108 to integrate the at least one object of interest to a digital map.
When a user is driving a car, he or she normally relies solely on his/her ability to remember objects of interest surrounding the car in the environment, e.g. locations of shops of interest, or gas stations with exceptional prices, or the first free parking lot seen for a while. A problem is that a human's memory cannot always be trusted, and locations and details get lost.
According to embodiments, it is possible to add information related to the determined at least one object of interest to a digital map in order to customize the digital map. The information related to the determined at least one object of interest could be location, name, type of the object of interest. The location may be derived from the current location of the car (e.g. from GPS). The location may be further improved (e.g. improving its accuracy) by considering the attention direction and depth information along the attention direction. The depth information may be provided from a depth sensor or from two attention directions or from two optical cameras. For example, the two attention directions may be captured when the vehicle at two positions, which is described in detail below.
A name or a type of the object of interest may be determined from image classification methods based on the captured at least one scene image. For this, known reference image features or objects may be used for the classification.
According to an embodiment, the at least one scene image may be captured after the determination of the at least one scene camera, after the determination of the at least one attention direction, or after the capture of the user attention data. Often, computation and/or processing time is required for the determination of the at least one scene camera, the determination of the at least one attention direction, and/or the capture of the user attention data. A lag between the capture of the user attention data and the capture of the at least one scene image may exist. In reality, the vehicle may move. Therefore, based on embodiments disclosed above, the at least one scene image (i.e. the desired scene camera) may not capture an object of interest indicated by the at least one attention direction provided from the user attention data, and/or the at least one attention direction may not correctly indicate a region of interest where the object of interest is contained in the at least one scene image.
According to the example of FIG. 4 , the vehicle is at a first vehicle position where the user attention data is captured. As shown in FIG. 4 , the car 201 is at a first position 441 where the user looks at the person 226 and the user attention data is captured. The user attention data is indicative of the user attention direction 203 that may indicate an object of interest (e.g. the person 226 ). A first coordinate system may be derived from the reference coordinate system of the vehicle at the first vehicle position 441 . In FIG. 4 , the reference coordinate system 209 is the first coordinate system.
At a later time, the vehicle is at a current vehicle position (i.e. a second vehicle position) where the at least one scene image is captured. A second coordinate system may be derived from the reference coordinate system of the vehicle at the current vehicle position. As shown in FIG. 4 , the <figure-callout id="201" la
CLAIMS
Claims ( 20 )
The invention claimed is:
1. A non-transitory computer readable medium storing instructions executable by one or more processors to:
determine a camera pose of a first camera of a plurality of scene cameras mounted on a system;
determine an attention direction of a user of the system based on sensor data captured by a user-facing camera of the system;
determine that a field of view of the first camera overlaps the attention direction based on the camera pose and the attention direction; and
in response to the field of view overlapping the attention direction:
obtain an image captured by the first camera; and
identify an object of interest in the obtained image by projecting the attention direction into the obtained image to identify a region of interest within the image.
2. The non-transitory computer readable medium of claim 1 , wherein the sensor data includes an image of a hand of the user, and wherein the attention direction includes a direction in which the hand of the user is pointing.
3. The non-transitory computer readable medium of claim 1 , wherein sensor data includes an electrooculogram.
4. The non-transitory computer readable medium of claim 1 , wherein the instructions are further executable by the one or more processors to initiate display of enhanced information related to the object of interest.
5. The non-transitory computer readable medium of claim 4 , wherein the enhanced information related to the object of interest includes an indication of a material included in the object of interest, a size of the object of interest, a link to a webpage including additional information related to the object of interest, or a combination thereof.
6. The non-transitory computer readable medium of claim 1 , further comprising instructions executable by the one or more processors to obtain the image by selecting the obtained image from a plurality of images captured by the plurality of scene cameras.
7. The non-transitory computer readable medium of claim 6 , wherein capture of the image of the first camera of the plurality of scene cameras and capture of the sensor data by the user-facing camera are asynchronous.
8. A method comprising:
determining a camera pose of a first camera of a plurality of scene cameras mounted on a system, the camera pose;
determining an attention direction of a user of the system based on sensor data captured by a user-facing camera of the system;
determining that a field of view of the first camera overlaps the attention direction based on the camera pose and the attention direction; and
in response to the field of view overlapping the attention direction:
obtaining an image captured by the first camera; and
identifying an object of interest in the obtained image by projecting the attention direction into the obtained image to identify a region of interest within the image.
9. The method of claim 8 , wherein the sensor data includes an image of a hand of the user, and wherein the attention direction includes a direction in which the hand of the user is pointing.
10. The method of claim 8 , wherein sensor data includes an electrooculogram.
11. The method of claim 8 , further comprising:
displaying enhanced information related to the object of interest.
12. The method of claim 11 , wherein the enhanced information related to the object of interest includes an indication of a material included in the object of interest, a size of the object of interest, a link to a webpage including additional information related to the object of interest, or a combination thereof.
13. The method of claim 8 , further comprising:
obtaining the image by selecting the obtained image from a plurality of images captured by the plurality of scene cameras.
14. The method of claim 13 , wherein capture of the image of the first camera of the plurality of scene cameras and capture of the sensor data by the user-facing camera are asynchronous.
15. A system comprising:
a plurality of scene cameras;
a user-facing camera;
one or more processors;
one or more computer readable media storing instructions executable by the one or more processors to:
determine a camera pose of a first camera of the plurality of scene cameras mounted on the system, the camera pose associated with the system;
determine an attention direction of a user of the system based on sensor data captured by the user-facing camera of the system;
determine that a field of view of the first camera overlaps the attention direction based on the camera pose and the attention direction; and
in response to the field of view overlapping the attention direction:
obtain an image captured by the first camera; and
identify an object of interest in the obtained image by projecting the attention direction into the obtained image to identify a region of interest within the image.
16. The system of claim 15 , wherein the sensor data includes an image of a hand of the user, and wherein the attention direction includes a direction in which the hand of the user is pointing.
17. The system of claim 15 , further comprising instructions executable by the one or more processors to initiate display of enhanced information related to the object of interest.
18. The system of claim 17 , wherein the enhanced information related to the object of interest includes an indication of a material included in the object of interest, a size of the object of interest, a link to a webpage including additional information related to the object of interest, or a combination thereof.
19. The system of claim 15 , wherein the instructions are further executable by the one or more processors to obtain the image by selecting the obtained image from a plurality of images captured by the plurality of scene cameras.
20. The system of claim 19 , wherein capture of the image of the first camera of the plurality of scene cameras and capture of the sensor data by the user-facing camera are asynchronous.
US18/333,943
2014-08-27
2023-06-13
Method and system for providing at least one image captured by a scene camera of a vehicle
Active
US12244970B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US18/333,943
US12244970B2
( en )
2014-08-27
2023-06-13
Method and system for providing at least one image captured by a scene camera of a vehicle
US19/030,582
US20250168300A1
( en )
2014-08-27
2025-01-17
Method and System for Providing At Least One Image Captured By a Scene Camera of a Vehicle
Applications Claiming Priority (5)
Application Number
Priority Date
Filing Date
Title
PCT/EP2014/068165
WO2016029939A1
( en )
2014-08-27
2014-08-27
Method and system for determining at least one image feature in at least one image
US14/581,609
US10375357B2
( en )
2014-08-27
2014-12-23
Method and system for providing at least one image captured by a scene camera of a vehicle
US16/448,693
US10757373B2
( en )
2014-08-27
2019-06-21
Method and system for providing at least one image captured by a scene camera of a vehicle
US16/938,660
US20200358984A1
( en )
2014-08-27
2020-07-24
Method and System for Providing At Least One Image Captured By a Scene Camera of a Vehicle
US18/333,943
US12244970B2
( en )
2014-08-27
2023-06-13
Method and system for providing at least one image captured by a scene camera of a vehicle
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US16/938,660
Continuation
US20200358984A1
( en )
2014-08-27
2020-07-24
Method and System for Providing At Least One Image Captured By a Scene Camera of a Vehicle
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US19/030,582
Continuation
US20250168300A1
( en )
2014-08-27
2025-01-17
Method and System for Providing At Least One Image Captured By a Scene Camera of a Vehicle
Publications (2)
Publication Number
Publication Date
US20230328207A1
US20230328207A1 ( en )
2023-10-12
US12244970B2
true
US12244970B2 ( en )
2025-03-04
Family
ID=51454678
Family Applications (5)
Application Number
Title
Priority Date
Filing Date
US14/581,609
Active
2034-12-17
US10375357B2
( en )
2014-08-27
2014-12-23
Method and system for providing at least one image captured by a scene camera of a vehicle
US16/448,693
Active
US10757373B2
( en )
2014-08-27
2019-06-21
Method and system for providing at least one image captured by a scene camera of a vehicle
US16/938,660
Abandoned
US20200358984A1
( en )
2014-08-27
2020-07-24
Method and System for Providing At Least One Image Captured By a Scene Camera of a Vehicle
US18/333,943
Active
US12244970B2
( en )
2014-08-27
2023-06-13
Method and system for providing at least one image captured by a scene camera of a vehicle
US19/030,582
Pending
US20250168300A1
( en )
2014-08-27
2025-01-17
Method and System for Providing At Least One Image Captured By a Scene Camera of a Vehicle
Family Applications Before (3)
Application Number
Title
Priority Date
Filing Date
US14/581,609
Active
2034-12-17
US10375357B2
( en )
2014-08-27
2014-12-23
Method and system for providing at least one image captured by a scene camera of a vehicle
US16/448,693
Active
US10757373B2
( en )
2014-08-27
2019-06-21
Method and system for providing at least one image captured by a scene camera of a vehicle
US16/938,660
Abandoned
US20200358984A1
( en )
2014-08-27
2020-07-24
Method and System for Providing At Least One Image Captured By a Scene Camera of a Vehicle
Family Applications After (1)
Application Number
Title
Priority Date
Filing Date
US19/030,582
Pending
US20250168300A1
( en )
2014-08-27
2025-01-17
Method and System for Providing At Least One Image Captured By a Scene Camera of a Vehicle
Country Status (3)
Country
Link
US
( 5 )
US10375357B2
( en )
CN
( 2 )
CN105391970B
( en )
WO
( 1 )
WO2016029939A1
( en )
Families Citing this family (99)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US9877145B2
( en )
*
2014-04-24
2018-01-23
Sony Corporation
Wireless communication apparatus and method for a user wirelessly receiving information regarding belongings of a nearby person
WO2016029939A1
( en )
*
2014-08-27
2016-03-03
Metaio Gmbh
Method and system for determining at least one image feature in at least one image
US10769453B2
( en )
*
2017-05-16
2020-09-08
Samsung Electronics Co., Ltd.
Electronic device and method of controlling operation of vehicle
US9569693B2
( en )
*
2014-12-31
2017-02-14
Here Global B.V.
Method and apparatus for object identification and location correlation based on received images
GB201501510D0
( en )
*
2015-01-29
2015-03-18
Apical Ltd
System
ES2891150T3
( en )
*
2015-05-06
2022-01-26
Reactive Reality Ag
Method and system for producing output images
KR102146398B1
( en )
*
2015-07-14
2020-08-20
ì¼ì±ì ì주ìíì¬
Three dimensional content producing apparatus and three dimensional content producing method thereof
CN107292642B
( en )
*
2016-03-31
2020-12-08
èå®äºè®¡ç®æéå ¬å¸
An image-based product recommendation method and system
EP3242228A1
( en )
*
2016-05-02
2017-11-08
Artag SARL
Managing the display of assets in augmented reality mode
US10366290B2
( en )
*
2016-05-11
2019-07-30
Baidu Usa Llc
System and method for providing augmented virtual reality content in autonomous vehicles
US9996936B2
( en )
*
2016-05-20
2018-06-12
Qualcomm Incorporated
Predictor-corrector based pose detection
JP2019518287A
( en )
2016-06-13
2019-06-27
ã¸ã¼ã ã¤ã³ã³ã¼ãã¬ã¼ãããï¼¸ï½ ï½ï½ Ｉï½ï½ï¼
Method and system for car parking space management using virtual cycle
US10726576B2
( en )
*
2016-06-29
2020-07-28
Seeing Machines Limited
System and method for identifying a camera pose of a forward facing camera in a vehicle
US20180147986A1
( en )
*
2016-06-30
2018-05-31
Faraday&Future Inc.
Method and system for vehicle-based image-capturing
US10762358B2
( en )
*
2016-07-20
2020-09-01
Ford Global Technologies, Llc
Rear camera lane detection
EP3496969A4
( en )
*
2016-08-10
2020-09-16
Xevo Inc.
PROCEDURE AND SYSTEM FOR PROVIDING INFORMATION ON COLLECTED AND STORED METADATA WITH A DERIVED ATTENTION MODEL
CN108024049B
( en )
*
2016-10-31
2020-07-03
æ å·åé³éç¨çµåæéå ¬å¸
Orientation control method and device of vehicle-mounted shooting device
JP6342106B1
( en )
*
2016-11-30
2018-06-13
æ¥½å¤©æ ªå¼ä¼ç¤¾
Image processing apparatus, image processing method, server, and computer program
CN108227914B
( en )
*
2016-12-12
2021-03-05
è´¢å¢æ³äººå·¥ä¸ææ¯ç 究é¢
Transparent display device, control method using the same, and controller thereof
US10950132B2
( en )
2016-12-22
2021-03-16
Xevo Inc.
Method and system for providing artificial intelligence analytic (AIA) services using operator fingerprints and cloud data
KR102407815B1
( en )
*
2016-12-22
2022-06-13
ì¼ì±ì ì주ìíì¬
Apparatus and method for processing image
FR3063557B1
( en )
*
2017-03-03
2022-01-14
Valeo Comfort & Driving Assistance
DEVICE FOR DETERMINING THE STATE OF ATTENTION OF A VEHICLE DRIVER, ON-BOARD SYSTEM COMPRISING SUCH A DEVICE, AND ASSOCIATED METHOD
US20180262739A1
( en )
*
2017-03-10
2018-09-13
Denso International America, Inc.
Object detection system
US10803323B2
( en )
*
2017-05-16
2020-10-13
Samsung Electronics Co., Ltd.
Electronic device and method of detecting driving event of vehicle
WO2018217110A1
( en )
2017-05-26
2018-11-29
Motorola Solutions, Inc.
Method to identify watchers of objects
US10810773B2
( en )
*
2017-06-14
2020-10-20
Dell Products, L.P.
Headset display control based upon a user's pupil state
EP3441788A1
( en )
*
2017-08-08
2019-02-13
Koninklijke Philips N.V.
Apparatus and method for generating a representation of a scene
CN109426330A
( en )
*
2017-08-22
2019-03-05
åæ欧è²æ¾ç¤ºç§ææéå ¬å¸
wearable device and operation method thereof
DE102017215161A1
( en )
*
2017-08-30
2019-02-28
Volkswagen Aktiengesellschaft
Method and device for selecting an environment object in the environment of a vehicle
CN111133752B
( en )
*
2017-09-22
2021-12-21
æ ªå¼ä¼ç¤¾çµé
Expression recording system
CN107705259A
( en )
*
2017-09-24
2018-02-16
åè¥éºå¾ä¿¡æ¯ç§ææéå ¬å¸
A kind of data enhancement methods and device under mobile terminal preview, screening-mode
TWI787205B
( en )
*
2017-09-28
2022-12-21
æ¥åé»éè¡ä»½æéå ¬å¸
Expression recording system, stroller, and expression recording program
CN109584295B
( en )
2017-09-29
2022-08-26
é¿éå·´å·´éå¢æ§è¡æéå ¬å¸
Method, device and system for automatically labeling target object in image
CN107895386A
( en )
*
2017-11-14
2018-04-10
ä¸å½èªç©ºå·¥ä¸éå¢å ¬å¸è¥¿å®é£æºè®¾è®¡ç 究æ
A kind of multi-platform joint objective autonomous classification method
CN108447063B
( en )
*
2017-12-15
2020-06-19
æµæ±ä¸å»è¯å¤§å¦
Multimodal MRI segmentation method for brain glioblastoma
US11994405B2
( en )
2018-03-07
2024-05-28
Google Llc
Methods and systems for determining geographic orientation based on imagery
US10558895B2
( en )
*
2018-03-30
2020-02-11
Tobii Ab
Deep learning for three dimensional (3D) gaze prediction
CN108510525B
( en )
*
2018-03-30
2019-03-12
ç¾åº¦å¨çº¿ç½ç»ææ¯ï¼å京ï¼æéå ¬å¸
Template method for tracing, device, augmented reality system and storage medium
WO2019207499A2
( en )
*
2018-04-25
2019-10-31
Tanku LTD.
Method and system for performing fueling operations
US10739455B2
( en )
*
2018-05-24
2020-08-11
GM Global Technology Operations LLC
Method and apparatus for acquiring depth information using cameras from different vehicles
US20190385372A1
( en )
*
2018-06-15
2019-12-19
Microsoft Technology Licensing, Llc
Positioning a virtual reality passthrough region at a known distance
TWI719343B
( en )
*
2018-08-28
2021-02-21
財åæ³äººå·¥æ¥æè¡ç 究é¢
Method and display system for information display
US11630485B2
( en )
2018-09-10
2023-04-18
Apple Inc.
Housing structures and input-output devices for electronic devices
US10614318B1
( en )
*
2019-10-25
2020-04-07
7-Eleven, Inc.
Sensor mapping to a global coordinate system using a marker grid
US11030756B2
( en )
2018-10-26
2021-06-08
7-Eleven, Inc.
System and method for position tracking using edge computing
WO2020123967A2
( en )
2018-12-14
2020-06-18
Apple Inc.
Gaze-driven recording of video
SE542553C2
( en )
*
2018-12-17
2020-06-02
Tobii Ab
Gaze tracking via tracing of light paths
US10882398B2
( en )
2019-02-13
2021-01-05
Xevo Inc.
System and method for correlating user attention direction and outside view
US10984534B2
( en )
*
2019-03-28
2021-04-20
GM Global Technology Operations LLC
Identification of attention region for enhancement of sensor-based detection in a vehicle
KR102799709B1
( en )
*
2019-04-05
2025-04-23
íëìë차주ìíì¬
Apparatus and method for guiding path
WO2020212776A1
( en )
*
2019-04-18
2020-10-22
Alma Mater Studiorum - Universita' Di Bologna
Creating training data variability in machine learning for object labelling from images
CN110281847A
( en )
*
2019-06-28
2019-09-27
大éæèµ(ä¸å½)æéå ¬å¸
In-vehicle interactive system
EP3980921A1
( en )
*
2019-07-08
2022-04-13
Huawei Technologies Co., Ltd.
System and method to identify points of interest from within autonomous vehicles
US11200749B2
( en )
2019-07-19
2021-12-14
The Boeing Company
Systems and methods of augmented reality visualization based on sensor data
US10896547B1
( en )
2019-07-19
2021-01-19
The Boeing Company
Systems and methods of augmented reality visualization based on sensor data
JP7566524B2
( en )
*
2019-07-31
2024-10-15
ãã¤ãã³æ ªå¼ä¼ç¤¾
Camera System
US11893759B2
( en )
2019-10-24
2024-02-06
7-Eleven, Inc.
Homography error correction using a disparity mapping
US11113541B2
( en )
2019-10-25
2021-09-07
7-Eleven, Inc.
Detection of object removal and replacement from a shelf
US11023740B2
( en )
2019-10-25
2021-06-01
7-Eleven, Inc.
System and method for providing machine-generated tickets to facilitate tracking
US11893757B2
( en )
2019-10-25
2024-02-06
7-Eleven, Inc.
Self-serve beverage detection and assignment
US12062191B2
( en )
2019-10-25
2024-08-13
7-Eleven, Inc.
Food detection using a sensor array
US11403852B2
( en )
2019-10-25
2022-08-02
7-Eleven, Inc.
Object detection based on wrist-area region-of-interest
KR102736783B1
( en )
2019-10-25
2024-12-04
ì¸ë¸-ì¼ë ë¸,ì¸ì½í¬ë ì´í°ë
Action detection during image tracking
US11674792B2
( en )
2019-10-25
2023-06-13
7-Eleven, Inc.
Sensor array with adjustable camera positions
US11551454B2
( en )
2019-10-25
2023-01-10
7-Eleven, Inc.
Homography error correction using marker locations
US11557124B2
( en )
2019-10-25
2023-01-17
7-Eleven, Inc.
Homography error correction
US11023741B1
( en )
2019-10-25
2021-06-01
7-Eleven, Inc.
Draw wire encoder based homography
US11587243B2
( en )
2019-10-25
2023-02-21
7-Eleven, Inc.
System and method for position tracking using edge computing
US11501454B2
( en )
2019-10-25
2022-11-15
7-Eleven, Inc.
Mapping wireless weight sensor array for item detection and identification
US11450011B2
( en )
2019-10-25
2022-09-20
7-Eleven, Inc.
Adaptive item counting algorithm for weight sensor using sensitivity analysis of the weight sensor
US11003918B1
( en )
2019-10-25
2021-05-11
7-Eleven, Inc.
Event trigger based on region-of-interest near hand-shelf interaction
US11887372B2
( en )
2019-10-25
2024-01-30
7-Eleven, Inc.
Image-based self-serve beverage detection and assignment
US11887337B2
( en )
2019-10-25
2024-01-30
7-Eleven, Inc.
Reconfigurable sensor array
JP7401273B2
( en )
*
2019-11-26
2023-12-19
æ¥ç«Astemoæ ªå¼ä¼ç¤¾
Mobile body control device and method
EP4058874B1
( en )
*
2019-12-05
2026-03-18
Guangdong Oppo Mobile Telecommunications Corp., Ltd.
Method and system for associating device coordinate systems in a multi-person ar system
US11074040B2
( en )
*
2019-12-11
2021-07-27
Chian Chiu Li
Presenting location related information and implementing a task based on gaze, gesture, and voice detection
CN114761997B
( en )
*
2019-12-12
2025-12-19
Oppo广ä¸ç§»å¨éä¿¡æéå ¬å¸
Target detection methods, terminal equipment and media
CN111857341B
( en )
*
2020-06-10
2023-06-13
æµæ±å汤ç§æå¼åæéå ¬å¸
Display control method and device
CN112185120A
( en )
*
2020-10-09
2021-01-05
éåºå¤§å¦
Commuting private car identification method based on electronic license plate data
CN112261295B
( en )
*
2020-10-22
2022-05-20
Oppo广ä¸ç§»å¨éä¿¡æéå ¬å¸
Image processing method, device and storage medium
CN112423035A
( en )
*
2020-11-05
2021-02-26
ä¸æµ·èéç½ç»ç§ææéå ¬å¸
Method for automatically extracting visual attention points of user when watching panoramic video in VR head display
IL280256A
( en )
*
2021-01-18
2022-08-01
Emza Visual Sense Ltd
Device and method for determining connection to an item
US11482007B2
( en )
*
2021-02-10
2022-10-25
Ford Global Technologies, Llc
Event-based vehicle pose estimation using monochromatic imaging
CN113112565B
( en )
*
2021-04-08
2022-01-25
å®å¾½å·¥ç¨å¤§å¦
A map construction method, storage medium and device based on robot vision closed-loop detection algorithm
US12019706B2
( en )
*
2021-05-23
2024-06-25
Jingdong Technology Holding Co., Ltd.
Data augmentation for object detection via differential neural rendering
CN113611143B
( en )
*
2021-07-29
2022-10-18
åè´çµåç§æï¼å¦é¨ï¼æéå ¬å¸
Parking memory system and map building system thereof
DE102021123275B3
( en )
2021-09-08
2023-01-26
Cariad Se
Method and processor circuit for providing color image data, by means of which at least one object in a predetermined environment is represented in color, and motor vehicle with the processor circuit
US12261810B2
( en )
*
2021-11-15
2025-03-25
Micron Technology, Inc.
Sharing externally captured content in communications
CN114708309B
( en )
*
2022-02-22
2025-06-13
广ä¸å·¥ä¸å¤§å¦
Visual indoor positioning method and system based on building plan prior information
JP2023157200A
( en )
*
2022-04-14
2023-10-26
ãã¤ãã³æ ªå¼ä¼ç¤¾
Electronic equipment, control methods and programs
CN117095322A
( en )
*
2022-05-09
2023-11-21
å京åè·³ç½ç»ææ¯æéå ¬å¸
An object tracking and identification method, device, electronic equipment and storage medium
CN114663458B
( en )
*
2022-05-24
2022-10-11
éé¨å¡(èå·)ç§ææéå ¬å¸
Dynamic calibration method, dynamic calibration device, driving state detection method, driving state detection medium and driving state detection equipment
CN117311486A
( en )
2022-06-22
2023-12-29
京ä¸æ¹ç§æéå¢è¡ä»½æéå ¬å¸
An interactive method, device and light field display system for light field display
US12526379B2
( en )
2022-09-23
2026-01-13
Dell Products L.P.
Automatic image orientation via zone detection
JP2024068553A
( en )
*
2022-11-08
2024-05-20
ãã¤ãã³æ ªå¼ä¼ç¤¾
Data creation device, control method and program
WO2024119000A1
( en )
*
2022-12-01
2024-06-06
Zimeno Inc.
Operations playback
US12198377B1
( en )
*
2023-07-11
2025-01-14
Tomahawk Robotics, Inc.
Point of interest tracking and estimation
CN119450234A
( en )
*
2023-08-01
2025-02-14
æ¬ç°æç å·¥ä¸æ ªå¼ä¼ç¤¾
Perspective image display system and method
CN117409191B
( en )
*
2023-12-12
2024-03-08
ä¼å¤ç§æè¡ä»½æéå ¬å¸
Fire inspection early warning method based on unmanned aerial vehicle and improved YOLOv8 target detection algorithm
Citations (24)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
EP1301021A2
( en )
2001-10-04
2003-04-09
Eastman Kodak Company
Method and system for displaying an image
US20050202844A1
( en )
2004-03-15
2005-09-15
General Electric Company
Method and system for portability of images using a high-quality display
US20070008091A1
( en )
*
2005-06-09
2007-01-11
Hitachi, Ltd.
Method and system of monitoring around a vehicle
JP2007133639A
( en )
2005-11-10
2007-05-31
Toyota Motor Corp
Vehicle display device
US20080181591A1
( en )
2007-01-26
2008-07-31
Sanyo Electric Co., Ltd.
Camera posture estimation device, vehicle, and camera posture estimation method
US20090128618A1
( en )
2007-11-16
2009-05-21
Samsung Electronics Co., Ltd.
System and method for object selection in a handheld image capture device
US20090299684A1
( en )
2008-05-30
2009-12-03
Denso Corporation
Method for calibrating cameras installed on vehicle
US20090304232A1
( en )
2006-07-14
2009-12-10
Panasonic Corporation
Visual axis direction detection device and visual line direction detection method
US20100049393A1
( en )
2008-08-21
2010-02-25
<span itemprop="assi