ABSTRACT
Abstract
Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. According to particular embodiments, the spatial relationship between multiple images and video is analyzed together with location information data, for purposes of creating a representation referred to herein as a surround view for presentation on a device. A visual guide can provided for capturing the multiple images used in the surround view. The visual guide can be a synthetic object that is rendered in real-time into the images output to a display of an image capture device. The visual guide can help user keep the image capture device moving along a desired trajectory.
Description
CROSS-REFERENCE TO RELATED APPLICATION
This patent document is a continuation of U.S. patent application Ser. No. 15/979,214, filed May 14, 2018, by Holzer et al, and entitled, âLIVE AUGMENTED REALITY GUIDES,â which is a continuation of U.S. Pat. No. 9,996,945, filed Dec. 12, 2016, by Holzer et al, and entitled, âLIVE AUGMENTED REALITY GUIDES.â The entirety of U.S. patent application Ser. No. 15/979,214 and U.S. Pat. No. 9,996,945 are hereby incorporated herein by reference for all purposes.
TECHNICAL FIELD
The present disclosure relates to augmenting multi-view image data with synthetic objects. In one example, the present disclosure relates to using inertial measurement unit (IMU) and image data to generate views of synthetic objects to be placed in a multi-view image.
Augmented reality typically includes a view of a real-world environment, such as through video and/or image data of scenery, a sports game, an object, individual, etc. This view of the real-world environment is augmented by computer generated input such as images, text, video, graphics, or the like. Accordingly, augmented reality can take the form of a live-action video or photo series with added elements that are computer-generated. Augmented reality is distinct from virtual reality, in which a simulated environment is depicted through video and/or image data.
In some implementations, augmented reality applications may add three-dimensional (3D) information to video and image data. This is generally done by creating a 3D reconstruction of the scene. However, this process is computationally expensive and usually restricted to static scenes. Accordingly, improved methods of implementing augmented reality are desirable.
OVERVIEW
Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. According to particular embodiments, the spatial relationship between multiple images and video is analyzed together with location information data, for purposes of creating a representation referred to herein as a surround view for presentation on a device. An object included in the surround view may be manipulated along axes by manipulating the device along corresponding axes. In particular embodiments, an augmented reality (AR) system is used for the purposes of capturing images used in a surround view. For example, live image data from camera of a mobile device can be augmented with virtual guides that help a user position the mobile device during image capture.
BRIEF DESCRIPTION OF THE DRAWINGS
The disclosure may best be understood by reference to the following description taken in conjunction with the accompanying drawings, which illustrate particular embodiments of the present invention.
FIG. 1 illustrates an example of a surround view acquisition system.
FIG. 2 illustrates an example of a process flow for generating a surround view.
FIG. 3 illustrates one example of multiple camera views that can be fused into a three-dimensional (3D) model to create an immersive experience.
FIG. 4 illustrates one example of separation of content and context in a surround view.
FIGS. 5A-5B illustrate examples of concave view and convex views, respectively, where both views use a back-camera capture style.
FIGS. 6A-6D illustrate examples of various capture modes for surround views.
FIGS. 7A and 7B illustrate an example of a process flow for capturing images in a surround view using augmented reality.
FIGS. 8A and 8B illustrate examples of generating an Augmented Reality (AR) image capture track for capturing images used in a surround view.
FIG. 9 illustrates an example of generating an Augmented Reality (AR) image capture track for capturing images used in a surround view on a mobile device.
FIGS. 10A and 10B illustrate an example of generating an Augmented Reality (AR) image capture track including status indicators for capturing images used in a surround view.
FIGS. 11A and 11B illustrate an example of generating an Augmented Reality (AR) image capture track including camera tilt effects on a mobile device.
FIGS. 12A-12D illustrate an example of generating an augmented reality image using tracking points on a mobile device.
FIGS. 13A-13C illustrate an example of generating an augmented reality image using tracking points on a mobile device.
FIG. 14 illustrates an example of a process flow for generating augmented reality images using tracking points.
FIG. 15 illustrates a particular example of a computer system that can be used with various embodiments of the present invention.
DETAILED DESCRIPTION
Reference will now be made in detail to some specific examples of the invention including the best modes contemplated by the inventors for carrying out the invention. Examples of these specific embodiments are illustrated in the accompanying drawings. While the present disclosure is described in conjunction with these specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. Particular embodiments of the present invention may be implemented without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.
Various aspects of the present invention relate generally to systems and methods for analyzing the spatial relationship between multiple images and video together with location information data, for the purpose of creating a single representation, a surround view, which eliminates redundancy in the data, and presents a user with an interactive and immersive active viewing experience. According to various embodiments, active is described in the context of providing a user with the ability to control the viewpoint of the visual information displayed on a screen.
In particular example embodiments, augmented reality (AR) is used to aid a user in capturing the multiple images used in a surround view. For example, a virtual guide can be inserted into live image data from a mobile. The virtual guide can help the user guide the mobile device along a desirable path useful for creating the surround view. The virtual guide in the AR images can respond to movements of the mobile device. The movement of mobile device can be determined from a number of different sources, including but not limited to an Inertial Measurement Unit and image data.
According to various embodiments of the present invention, a surround view is a multi-view interactive digital media representation. With reference to FIG. 1 , shown is one example of a surround view acquisition system 100 . In the present example embodiment, the surround view acquisition system 100 is depicted in a flow sequence that can be used to generate a surround view. According to various embodiments, the data used to generate a surround view can come from a variety of sources.
In particular, data such as, but not limited to two-dimensional (2D) images 104 can be used to generate a surround view. These 2D images can include color image data streams such as multiple image sequences, video data, etc., or multiple images in any of various formats for images, depending on the application. As will be described in more detail below with respect to FIGS. 7A-11B , during an image capture process, an AR system can be used. The AR system can receive and augment live image data with virtual data. In particular, the virtual data can include guides for helping a user direct the motion of an image capture device.
Another source of data that can be used to generate a surround view includes environment information 106 . This environment information 106 can be obtained from sources such as accelerometers, gyroscopes, magnetometers, GPS, WiFi, IMU-like systems (Inertial Measurement Unit systems), and the like. Yet another source of data that can be used to generate a surround view can include depth images 108 . These depth images can include depth, 3D, or disparity image data streams, and the like, and can be captured by devices such as, but not limited to, stereo cameras, time-of-flight cameras, three-dimensional cameras, and the like.
In the present example embodiment, the data can then be fused together at sensor fusion block 110 . In some embodiments, a surround view can be generated a combination of data that includes both 2D images 104 and environment information 106 , without any depth images 108 provided. In other embodiments, depth images 108 and environment information 106 can be used together at sensor fusion block 110 . Various combinations of image data can be used with environment information at 106 , depending on the application and available data.
In the present example embodiment, the data that has been fused together at sensor fusion block 110 is then used for content modeling 112 and context modeling 114 . As described in more detail with regard to FIG. 4 , the subject matter featured in the images can be separated into content and context. The content can be delineated as the object of interest and the context can be delineated as the scenery surrounding the object of interest. According to various embodiments, the content can be a three-dimensional model, depicting an object of interest, although the content can be a two-dimensional image in some embodiments, as described in more detail below with regard to FIG. 4 . Furthermore, in some embodiments, the context can be a two-dimensional model depicting the scenery surrounding the object of interest. Although in many examples the context can provide two-dimensional views of the scenery surrounding the object of interest, the context can also include three-dimensional aspects in some embodiments. For instance, the context can be depicted as a âflatâ image along a cylindrical âcanvas,â such that the âflatâ image appears on the surface of a cylinder. In addition, some examples may include three-dimensional context models, such as when some objects are identified in the surrounding scenery as three-dimensional objects. According to various embodiments, the models provided by content modeling 112 and context modeling 114 can be generated by combining the image and location information data, as described in more detail with regard to FIG. 3 .
According to various embodiments, context and content of a surround view are determined based on a specified object of interest. In some examples, an object of interest is automatically chosen based on processing of the image and location information data. For instance, if a dominant object is detected in a series of images, this object can be selected as the content. In other examples, a user specified target 102 can be chosen, as shown in FIG. 1 . It should be noted, however, that a surround view can be generated without a user specified target in some applications.
In the present example embodiment, one or more enhancement algorithms can be applied at enhancement algorithm(s) block 116 . In particular example embodiments, various algorithms can be employed during capture of surround view data, regardless of the type of capture mode employed. These algorithms can be used to enhance the user experience. For instance, automatic frame selection, stabilization, view interpolation, filters, and/or compression can be used during capture of surround view data. In some examples, these enhancement algorithms can be applied to image data after acquisition of the data. In other examples, these enhancement algorithms can be applied to image data during capture of surround view data.
According to particular example embodiments, automatic frame selection can be used to create a more enjoyable surround view. Specifically, frames are automatically selected so that the transition between them will be smoother or more even. This automatic frame selection can incorporate blur- and overexposure-detection in some applications, as well as more uniformly sampling poses such that they are more evenly distributed.
In some example embodiments, stabilization can be used for a surround view in a manner similar to that used for video. In particular, keyframes in a surround view can be stabilized for to produce improvements such as smoother transitions, improved/enhanced focus on the content, etc. However, unlike video, there are many additional sources of stabilization for a surround view, such as by using IMU information, depth information, computer vision techniques, direct selection of an area to be stabilized, face detection, and the like.
For instance, IMU information can be very helpful for stabilization. In particular, IMU information provides an estimate, although sometimes a rough or noisy estimate, of the camera tremor that may occur during image capture. This estimate can be used to remove, cancel, and/or reduce the effects of such camera tremor.
In some examples, depth information, if available, can be used to provide stabilization for a surround view. Because points of interest in a surround view are three-dimensiona
CROSS-REFERENCE TO RELATED APPLICATION
This patent document is a continuation of U.S. patent application Ser. No. 15/979,214, filed May 14, 2018, by Holzer et al, and entitled, âLIVE AUGMENTED REALITY GUIDES,â which is a continuation of U.S. Pat. No. 9,996,945, filed Dec. 12, 2016, by Holzer et al, and entitled, âLIVE AUGMENTED REALITY GUIDES.â The entirety of U.S. patent application Ser. No. 15/979,214 and U.S. Pat. No. 9,996,945 are hereby incorporated herein by reference for all purposes.
TECHNICAL FIELD
The present disclosure relates to augmenting multi-view image data with synthetic objects. In one example, the present disclosure relates to using inertial measurement unit (IMU) and image data to generate views of synthetic objects to be placed in a multi-view image.
Augmented reality typically includes a view of a real-world environment, such as through video and/or image data of scenery, a sports game, an object, individual, etc. This view of the real-world environment is augmented by computer generated input such as images, text, video, graphics, or the like. Accordingly, augmented reality can take the form of a live-action video or photo series with added elements that are computer-generated. Augmented reality is distinct from virtual reality, in which a simulated environment is depicted through video and/or image data.
In some implementations, augmented reality applications may add three-dimensional (3D) information to video and image data. This is generally done by creating a 3D reconstruction of the scene. However, this process is computationally expensive and usually restricted to static scenes. Accordingly, improved methods of implementing augmented reality are desirable.
OVERVIEW
Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. According to particular embodiments, the spatial relationship between multiple images and video is analyzed together with location information data, for purposes of creating a representation referred to herein as a surround view for presentation on a device. An object included in the surround view may be manipulated along axes by manipulating the device along corresponding axes. In particular embodiments, an augmented reality (AR) system is used for the purposes of capturing images used in a surround view. For example, live image data from camera of a mobile device can be augmented with virtual guides that help a user position the mobile device during image capture.
BRIEF DESCRIPTION OF THE DRAWINGS
The disclosure may best be understood by reference to the following description taken in conjunction with the accompanying drawings, which illustrate particular embodiments of the present invention.
FIG. 1 illustrates an example of a surround view acquisition system.
FIG. 2 illustrates an example of a process flow for generating a surround view.
FIG. 3 illustrates one example of multiple camera views that can be fused into a three-dimensional (3D) model to create an immersive experience.
FIG. 4 illustrates one example of separation of content and context in a surround view.
FIGS. 5A-5B illustrate examples of concave view and convex views, respectively, where both views use a back-camera capture style.
FIGS. 6A-6D illustrate examples of various capture modes for surround views.
FIGS. 7A and 7B illustrate an example of a process flow for capturing images in a surround view using augmented reality.
FIGS. 8A and 8B illustrate examples of generating an Augmented Reality (AR) image capture track for capturing images used in a surround view.
FIG. 9 illustrates an example of generating an Augmented Reality (AR) image capture track for capturing images used in a surround view on a mobile device.
FIGS. 10A and 10B illustrate an example of generating an Augmented Reality (AR) image capture track including status indicators for capturing images used in a surround view.
FIGS. 11A and 11B illustrate an example of generating an Augmented Reality (AR) image capture track including camera tilt effects on a mobile device.
FIGS. 12A-12D illustrate an example of generating an augmented reality image using tracking points on a mobile device.
FIGS. 13A-13C illustrate an example of generating an augmented reality image using tracking points on a mobile device.
FIG. 14 illustrates an example of a process flow for generating augmented reality images using tracking points.
FIG. 15 illustrates a particular example of a computer system that can be used with various embodiments of the present invention.
DETAILED DESCRIPTION
Reference will now be made in detail to some specific examples of the invention including the best modes contemplated by the inventors for carrying out the invention. Examples of these specific embodiments are illustrated in the accompanying drawings. While the present disclosure is described in conjunction with these specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. Particular embodiments of the present invention may be implemented without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.
Various aspects of the present invention relate generally to systems and methods for analyzing the spatial relationship between multiple images and video together with location information data, for the purpose of creating a single representation, a surround view, which eliminates redundancy in the data, and presents a user with an interactive and immersive active viewing experience. According to various embodiments, active is described in the context of providing a user with the ability to control the viewpoint of the visual information displayed on a screen.
In particular example embodiments, augmented reality (AR) is used to aid a user in capturing the multiple images used in a surround view. For example, a virtual guide can be inserted into live image data from a mobile. The virtual guide can help the user guide the mobile device along a desirable path useful for creating the surround view. The virtual guide in the AR images can respond to movements of the mobile device. The movement of mobile device can be determined from a number of different sources, including but not limited to an Inertial Measurement Unit and image data.
According to various embodiments of the present invention, a surround view is a multi-view interactive digital media representation. With reference to FIG. 1 , shown is one example of a surround view acquisition system 100 . In the present example embodiment, the surround view acquisition system 100 is depicted in a flow sequence that can be used to generate a surround view. According to various embodiments, the data used to generate a surround view can come from a variety of sources.
In particular, data such as, but not limited to two-dimensional (2D) images 104 can be used to generate a surround view. These 2D images can include color image data streams such as multiple image sequences, video data, etc., or multiple images in any of various formats for images, depending on the application. As will be described in more detail below with respect to FIGS. 7A-11B , during an image capture process, an AR system can be used. The AR system can receive and augment live image data with virtual data. In particular, the virtual data can include guides for helping a user direct the motion of an image capture device.
Another source of data that can be used to generate a surround view includes environment information 106 . This environment information 106 can be obtained from sources such as accelerometers, gyroscopes, magnetometers, GPS, WiFi, IMU-like systems (Inertial Measurement Unit systems), and the like. Yet another source of data that can be used to generate a surround view can include depth images 108 . These depth images can include depth, 3D, or disparity image data streams, and the like, and can be captured by devices such as, but not limited to, stereo cameras, time-of-flight cameras, three-dimensional cameras, and the like.
In the present example embodiment, the data can then be fused together at sensor fusion block 110 . In some embodiments, a surround view can be generated a combination of data that includes both 2D images 104 and environment information 106 , without any depth images 108 provided. In other embodiments, depth images 108 and environment information 106 can be used together at sensor fusion block 110 . Various combinations of image data can be used with environment information at 106 , depending on the application and available data.
In the present example embodiment, the data that has been fused together at sensor fusion block 110 is then used for content modeling 112 and context modeling 114 . As described in more detail with regard to FIG. 4 , the subject matter featured in the images can be separated into content and context. The content can be delineated as the object of interest and the context can be delineated as the scenery surrounding the object of interest. According to various embodiments, the content can be a three-dimensional model, depicting an object of interest, although the content can be a two-dimensional image in some embodiments, as described in more detail below with regard to FIG. 4 . Furthermore, in some embodiments, the context can be a two-dimensional model depicting the scenery surrounding the object of interest. Although in many examples the context can provide two-dimensional views of the scenery surrounding the object of interest, the context can also include three-dimensional aspects in some embodiments. For instance, the context can be depicted as a âflatâ image along a cylindrical âcanvas,â such that the âflatâ image appears on the surface of a cylinder. In addition, some examples may include three-dimensional context models, such as when some objects are identified in the surrounding scenery as three-dimensional objects. According to various embodiments, the models provided by content modeling 112 and context modeling 114 can be generated by combining the image and location information data, as described in more detail with regard to FIG. 3 .
According to various embodiments, context and content of a surround view are determined based on a specified object of interest. In some examples, an object of interest is automatically chosen based on processing of the image and location information data. For instance, if a dominant object is detected in a series of images, this object can be selected as the content. In other examples, a user specified target 102 can be chosen, as shown in FIG. 1 . It should be noted, however, that a surround view can be generated without a user specified target in some applications.
In the present example embodiment, one or more enhancement algorithms can be applied at enhancement algorithm(s) block 116 . In particular example embodiments, various algorithms can be employed during capture of surround view data, regardless of the type of capture mode employed. These algorithms can be used to enhance the user experience. For instance, automatic frame selection, stabilization, view interpolation, filters, and/or compression can be used during capture of surround view data. In some examples, these enhancement algorithms can be applied to image data after acquisition of the data. In other examples, these enhancement algorithms can be applied to image data during capture of surround view data.
According to particular example embodiments, automatic frame selection can be used to create a more enjoyable surround view. Specifically, frames are automatically selected so that the transition between them will be smoother or more even. This automatic frame selection can incorporate blur- and overexposure-detection in some applications, as well as more uniformly sampling poses such that they are more evenly distributed.
In some example embodiments, stabilization can be used for a surround view in a manner similar to that used for video. In particular, keyframes in a surround view can be stabilized for to produce improvements such as smoother transitions, improved/enhanced focus on the content, etc. However, unlike video, there are many additional sources of stabilization for a surround view, such as by using IMU information, depth information, computer vision techniques, direct selection of an area to be stabilized, face detection, and the like.
For instance, IMU information can be very helpful for stabilization. In particular, IMU information provides an estimate, although sometimes a rough or noisy estimate, of the camera tremor that may occur during image capture. This estimate can be used to remove, cancel, and/or reduce the effects of such camera tremor.
In some examples, depth information, if available, can be used to provide stabilization for a surround view. Because points of interest in a surround view are three-dimensional, rather than two-dimensional, these points of interest are more constrained and tracking/matching of these points is simplified as the search space reduces. Furthermore, descriptors for points of interest can use both color and depth information and therefore, become more discriminative. In addition, automatic or semi-automatic content selection can be easier to provide with depth information. For instance, when a user selects a particular pixel of an image, this selection can be expanded to fill the entire surface that touches it. Furthermore, content can also be selected automatically by using a foreground/background differentiation based on depth. In various examples, the content can stay relatively stable/visible even when the context changes.
According to various examples, computer vision techniques can also be used to provide stabilization for surround views. For instance, keypoints can be detected and tracked. However, in certain scenes, such as a dynamic scene or static scene with parallax, no simple warp exists that can stabilize everything. Consequently, there is a trade-off in which certain aspects of the scene receive more attention to stabilization and other aspects of the scene receive less attention. Because a surround view is often focused on a particular object of interest, a surround view can be content-weighted so that the object of interest is maximally stabilized in some examples.
Another way to improve stabilization in a surround view includes direct selection of a region of a screen. For instance, if a user taps to focus on a region of a screen, then records a convex surround view, the area that was tapped can be maximally stabilized. This allows stabilization algorithms to be focused on a particular area or object of interest.
In some examples, face detection can be used to provide stabilization. For instance, when recording with a front-facing camera, it is often likely that the user is the object of interest in the scene. Thus, face detection can be used to weight stabilization about that region. When face detection is precise enough, facial features themselves (such as eyes, nose, and mouth) can be used as areas to stabilize, rather than using generic keypoints. In another example, a user can select an area of image to use as a source for keypoints.
According to various examples, view interpolation can be used to improve the viewing experience. In particular, to avoid sudden âjumpsâ between stabilized frames, synthetic, intermediate views can be rendered on the fly. This can be informed by content-weighted keypoint tracks and IMU information as described above, as well as by denser pixel-to-pixel matches. If depth information is available, fewer artifacts resulting from mismatched pixels may occur, thereby simplifying the process. As described above, view interpolation can be applied during capture of a surround view in some embodiments. In other embodiments, view interpolation can be applied during surround view generation.
In some examples, filters can also be used during capture or generation of a surround view to enhance the viewing experience. Just as many popular photo sharing services provide aesthetic filters that can be applied to static, two-dimensional images, aesthetic filters can similarly be applied to surround images. However, because a surround view representation is more expressive than a two-dimensional image, and three-dimensional information is available in a surround view, these filters can be extended to include effects that are ill-defined in two dimensional photos. For instance, in a surround view, motion blur can be added to the background (i.e. context) while the content remains crisp. In another example, a drop-shadow can be added to the object of interest in a surround view.
In various examples, compression can also be used as an enhancement algorithm 116 . In particular, compression can be used to enhance user-experience by reducing data upload and download costs. Because surround views use spatial information, far less data can be sent for a surround view than a typical video, while maintaining desired qualities of the surround view. Specifically, the IMU, keypoint tracks, and user input, combined with the view interpolation described above, can all reduce the amount of data that must be transferred to and from a device during upload or download of a surround view. For instance, if an object of interest can be properly identified, a variable compression style can be chosen for the content and context. This variable compression style can include lower quality resolution for background information (i.e. context) and higher quality resolution for foreground information (i.e. content) in some examples. In such examples, the amount of data transmitted can be reduced by sacrificing some of the context quality, while maintaining a desired level of quality for the content.
In the present embodiment, a surround view 118 is generated after any enhancement algorithms are applied. The surround view can provide a multi-view interactive digital media representation. In various examples, the surround view can include three-dimensional model of the content and a two-dimensional model of the context. However, in some examples, the context can represent a âflatâ view of the scenery or background as projected along a surface, such as a cylindrical or other-shaped surface, such that the context is not purely two-dimensional. In yet other examples, the context can include three-dimensional aspects.
According to various embodiments, surround views provide numerous advantages over traditional two-dimensional images or videos. Some of these advantages include: the ability to cope with moving scenery, a moving acquisition device, or both; the ability to model parts of the scene in three-dimensions; the ability to remove unnecessary, redundant information and reduce the memory footprint of the output dataset; the ability to distinguish between content and context; the ability to use the distinction between content and context for improvements in the user-experience; the ability to use the distinction between content and context for improvements in memory footprint (an example would be high quality compression of content and low quality compression of context); the ability to associate special feature descriptors with surround views that allow the surround views to be indexed with a high degree of efficiency and accuracy; and the ability of the user to interact and change the viewpoint of the surround view. In particular example embodiments, the characteristics described above can be incorporated natively in the surround view representation, and provide the capability for use in various applications. For instance, surround views can be used to enhance various fields such as e-commerce, visual search, 3D printing, file sharing, user interaction, and entertainment.
According to various example embodiments, once a surround view 118 is generated, user feedback for acquisition 120 of additional image data can be provided. In particular, if a surround view is determined to need additional views to provide a more accurate model of the content or context, a user may be prompted to provide additional views. Once these additional views are received by the surround view acquisition system 100 , these additional views can be processed by the system 100 and incorporated into the surround view.
With reference to FIG. 2 , shown is an example of a process flow diagram for generating a surround view 200 . In the present example, a plurality of images is obtained at 202 . According to various embodiments, the plurality of images can include two-dimensional (2D) images or data streams. These 2D images can include location information that can be used to generate a surround view. In some embodiments, the plurality of images can include depth images 108 , as also described above with regard to FIG. 1 . The depth images can also include location information in various examples.
As is described in more detail with respect to FIGS. 7A-11B , when the plurality of images is captured, images output to the user can be augmented with the virtual data. For example, the plurality of images can be captured using a camera system on a mobile device. The live image data, which is output to a display on the mobile device, can include virtual data, such as guides and status indicators, rendered into the live image data. The guides can help a user guide a motion of the mobile device. The status indicators can indicate what portion of images needed for generating a surround view have been captured. The virtual data may not be included in the image data captured for the purposes of generating the surround view.
According to various embodiments, the plurality of images obtained at 202 can include a variety of sources and characteristics. For instance, the plurality of images can be obtained from a plurality of users. These images can be a collection of images gathered from the internet from different users of the same event, such as 2D images or video obtained at a concert, etc. In some examples, the plurality of images can include images with different temporal information. In particular, the images can be taken at different times of the same object of interest. For instance, multiple images of a particular statue can be obtained at different times of day, different seasons, etc. In other examples, the plurality of images can represent moving objects. For instance, the images may include an object of interest moving through scenery, such as a vehicle traveling along a road or a plane traveling through the sky. In other instances, the images may include an object of interest that is also moving, such as a person dancing, running, twirling, etc.
In the present example embodiment, the plurality of images is fused into content and context models at 204 . According to various embodiments, the subject matter featured in the images can be separated into content and context. The content can be delineated as the object of interest and the context can be delineated as the scenery surrounding the object of interest. According to various embodiments, the content can be a three-dimensional model, depicting an object of interest, and the content can be a two-dimensional image in some embodiments.
According to the present example embodiment, one or more enhancement algorithms can be applied to the content and context models at 206 . These algorithms can be used to enhance the user experience. For instance, enhancement algorithms such as automatic frame selection, stabilization, view interpolation, filters, and/or compression can be used. In some examples, these enhancement algorithms can be applied to image data during capture of the images. In other examples, these enhancement algorithms can be applied to image data after acquisition of the data.
In the present embodiment, a surround view is generated from the content and context models at 208 . The surround view can provide a multi-view interactive digital media representation. In various examples, the surround view can include a three-dimensional model of the content and a two-dimensional model of the context. According to various embodiments, depending on the mode of capture and the viewpoints of the images, the surround view model can include certain characteristics. For instance, some examples of different styles of surround views include a locally concave surround view, a locally convex surround view, and a locally flat surround view. However, it should be noted that surround views can include combinations of views and characteristics, depending on the application.
With reference to FIG. 3 , shown is one example of multiple camera views that can be fused together into a three-dimensional (3D) model to create an immersive experience. According to various embodiments, multiple images can be captured from various viewpoints and fused together to provide a surround view. In the present example embodiment, three
cameras
312 , 314 , and 316 are positioned at
locations
322 , 324 , and 326 , respectively, in proximity to an object of interest 308 . Scenery can surround the object of interest 308 such as object 310 .
Views
302 , 304 , and 306 from their
respective cameras
312 , 314 , and 316 include overlapping subject matter. Specifically, each
view
302 , 304 , and 306 includes the object of interest 308 and varying degrees of visibility of the scenery surrounding the object 310 . For instance, view 302 includes a view of the object of interest 308 in front of the cylinder that is part of the scenery surrounding the object 310 . View 306 shows the object of interest 308 to one side of the cylinder, and view 304 shows the object of interest without any view of the cylinder.
In the present example embodiment, the
various views
302 , 304 , and 316 along with their associated
locations
322 , 324 , and 326 , respectively, provide a rich source of information about object of interest 308 and the surrounding context that can be used to produce a surround view. For instance, when analyzed together, the
various views
302 , 304 , and 326 provide information about different sides of the object of interest and the relationship between the object of interest and the scenery. According to various embodiments, this information can be used to parse out the object of interest 308 into content and the scenery as the context. Furthermore, as also described above with regard to FIGS. 1 and 2 , various algorithms can be applied to images produced by these viewpoints to create an immersive, interactive experience when viewing a surround view.
FIG. 4 illustrates one example of separation of content and context in a surround view. According to various embodiments of the present invention, a surround view is a multi-view interactive digital media representation of a scene 400 . With reference to FIG. 4 , shown is a user 402 located in a scene 400 . The user 402 is capturing images of an object of interest, such as a statue. The images captured by the user constitute digital visual data that can be used to generate a surround view.
According to various embodiments of the present disclosure, the digital visual data included in a surround view can be, semantically and/or practically, separated into content 404 and context 406 . According to particular embodiments, content 404 can include the object(s), person(s), or scene(s) of interest while the context 406 represents the remaining elements of the scene surrounding the content 404 . In some examples, a surround view may represent the content 404 as three-dimensional data, and the context 406 as a two-dimensional panoramic background. In other examples, a surround view may represent both the content 404 and context 406 as two-dimensional panoramic scenes. In yet other examples, content 404 and context 406 may include three-dimensional components or aspects. In particular embodiments, the way that the surround view depicts content 404 and context 406 depends on the capture mode used to acquire the images.
In some examples, such as but not limited to: recordings of objects, persons, or parts of objects or persons, where only the object, person, or parts of them are visible, recordings of large flat areas, and recordings of scenes where the data captured appears to be at infinity (i.e., there are no subjects close to the camera), the content 404 and the context 406 may be the same. In these examples, the surround view produced may have some characteristics that are similar to other types of digital media such as panoramas. However, according to various embodiments, surround views include additional features that distinguish them from these existing types of digital media. For instance, a surround view can represent moving data. Additionally, a surround view is not limited to a specific cylindrical, spherical or translational movement. Various motions can be used to capture image data with a camera or other capture device. Furthermore, unlike a stitched panorama, a surround view can display different sides of the same object.
FIGS. 5A-5B illustrate examples of concave and convex views, respectively, where both views use a back-camera capture style. In particular, if a camera phone is used, these views use the camera on the back of the phone, facing away from the user. In particular embodiments, concave and convex views can affect how the content and context are designated in a surround view.
With reference to FIG. 5A , shown is one example of a concave view 500 in which a user is standing along a vertical axis 508 . In this example, the user is holding a camera, such that camera location 502 does not leave axis 508 during image capture. However, as the user pivots about axis 508 , the camera captures a panoramic view of the scene around the user, forming a concave view. In this embodiment, the object of interest 504 and the distant scenery 506 are all viewed similarly because of the way in which the images are captured. In this example, all objects in the concave view appear at infinity, so the content is equal to the context according to this view.
With reference to FIG. 5B , shown is one example of a convex view 520 in which a user changes position when capturing images of an object of interest 524 . In this example, the user moves around the object of interest 524 , taking pictures from different sides of the object of interest from camera locations
528 , 530 , and 532 . Each of the images obtained includes a view of the object of interest, and a background of the distant scenery 526 . In the present example, the object of interest 524 represents the content, and the distant scenery 526 represents the context in this convex view.
FIGS. 6A-6D illustrate examples of various capture modes for surround views. Although various motions can be used to capture a surround view and are not constrained to any particular type of motion, three general types of motion can be used to capture particular features or views described in conjunction surround views. These three types of motion, respectively, can yield a locally concave surround view, a locally convex surround view, and a locally flat surround view. In some examples, a surround view can include various types of motions within the same surround view.
With reference to FIG. 6A , shown is an example of a back-facing, concave surround view being captured. According to various embodiments, a locally concave surround view is one in which the viewing angles of the camera or other capture device diverge. In one dimension this can be likened to the motion required to capture a spherical 360 panorama (pure rotation), although the motion can be generalized to any curved sweeping motion in which the view faces outward. In the present example, the experience is that of a stationary viewer looking out at a (possibly dynamic) context.
In the present example embodiment, a user 602 is using a back-facing camera 606 to capture images towards world 600 , and away from user 602 . As described in various examples, a back-facing camera refers to a device with a camera that faces away from the user, such as the camera on the back of a smart phone. The camera is moved in a concave motion 608 , such that
views
604 a , 604 b , and 604 c capture various parts of capture area 609 .
With reference to FIG. 6B , shown is an example of a back-facing, convex surround view being captured. According to various embodiments, a locally convex surround view is one in which viewing angles converge toward a single object of interest. In some examples, a locally convex surround view can provide the experience of orbiting about a point, such that a viewer can see multiple sides of the same object. This object, which may be an âobject of interest,â can be segmented from the surround view to become the content, and any surrounding data can be segmented to become the context. Previous technologies fail to recognize this type of viewing angle in the media-sharing landscape.
In the present example embodiment, a user 602 is using a back-facing camera 614 to capture images towards world 600 , and away from user 602 . The camera is moved in a convex motion 610 , such that
views
612 a , 612 b , and 612 c capture various parts of capture area 611 . As described above, world 600 can include an object of interest in some examples, and the convex motion 610 can orbit around this object.
Views
612 a , 612 b , and 612 c can include views of different sides of this object in these examples.
With reference to FIG. 6C , shown is an example of a front-facing, concave surround view being captured. As described in various examples, a front-facing camera refers to a device with a camera that faces towards the user, such as the camera on the front of a smart phone. For instance, front-facing cameras are commonly used to take âselfiesâ (i.e., self-portraits of the user).
In the present example embodiment, camera 620 is facing user 602 . The camera follows a concave motion 606 such that the
views
618 a , 618 b , and 618 c diverge from each other in an angular sense. The capture area 617 follows a concave shape that includes the user at a perimeter.
With reference to FIG. 6D , shown is an example of a front-facing, convex surround view being captured. In the present example embodiment, camera 626 is facing user 602 . The camera follows a convex motion 622 such that the
views
624 a , 624 b , and 624 c converge towards the user 602 . As described above, various modes can be used to capture images for a surround view. These modes, including locally concave, locally convex, and locally linear motions, can be used during capture of separate images or during continuous recording of a scene. Such recording can capture a series of images during a single session.
Next, details of an augmented reality system, which is usable in the image capture process for a surround view, is described with respect to FIG. 7A to FIG. 11B . In one embodiment, the augmented reality system can be implemented on a mobile device, such as a cell phone. In particular, the live camera data, which is output to a display on the mobile device, can be augmented with virtual objects. The virtual objects can be rendered into the live camera data. In one embodiment, the virtual objects can provide a user feedback when images are being captured for a surround view.
FIGS. 7A and 7B illustrate an example of a process flow for capturing images in a surround view using augmented reality. In 702 , live image data can be received from a camera system. For example, live image data can be received from one or more cameras on a hand-held mobile device, such as a smartphone. The image data can include pixel data captured from a camera sensor. The pixel data varies from frame to frame. In one embodiment, the pixel data can be 2-D. In other embodiments, depth data can be included with the pixel data.
In 704 , sensor data can be received. For example, the mobile device can include an IMU with accelerometers and gyroscopes. The sensor data can be used to determine an orientation of the mobile device, such as a tilt orientation of the device relative to the gravity vector. Thus, the orientation of the live 2-D image data relative to the gravity vector can also be determined. In addition, when the user applied accelerations can be separated from the acceleration due to gravity, it may be possible to determine ch
CLAIMS
Claims ( 20 )
What is claimed is:
1. A method comprising:
receiving live images from a camera of a mobile device comprising 2-D pixel data wherein the live images are output to a display of the mobile device and show content that is currently being captured by the camera;
capturing a plurality of images from among the live images to generate a multi-view interactive digital media representation of a vehicle appearing in the live images wherein a position and/or orientation of the camera varies during the capturing of the plurality of images;
during output of the live images to the display, determining a location on the vehicle in the 2-D pixel data from a first image from among the live images;
based upon the location, determining a first position of a tracking point in the 2-D pixel data from the first image among the live images;
receiving sensor data indicating the position and/or orientation of the camera;
as additional live images after the first image are captured by the camera and output to the display, determining a current position of the tracking point in each of the additional live images on an image by image basis;
based upon the current position of the tracking point and the sensor data, rendering a virtual object, separate from and addition to the vehicle, into each of the additional live images output to the display to generate a plurality of synthetic images wherein in each of the synthetic images the virtual object is positioned relative to the current position of the tracking point determined for each of the additional live images, wherein the virtual object provides information related to one or more of: i) a progress of the camera along a path, ii) a current position, iii) a current orientation of the camera relative to the path or iv) combinations thereof during the capture of the plurality of images used to generate the multi-view interactive digital media representation of the vehicle; and
outputting the plurality of synthetic images to the display, each of the synthetic images showing the vehicle as currently being captured by the camera and the virtual object.
2. The method of claim 1 , further comprising receiving via an input interface a selection of the location.
3. The method of claim 2 , wherein the input interface is a touchscreen of the mobile device.
4. The method of claim 1 , wherein the virtual object is specified in a 3-D coordinate system.
5. The method of claim 3 , further comprising rendering the virtual object from a 3-D coordinate system into the 2-D pixel data.
6. The method of claim 1 , wherein the virtual object is rendered over a damaged portion of the vehicle.
7. The method of claim 6 , wherein the virtual object comprises a tag indicating the damaged portion of the vehicle.
8. The method of claim 6 , wherein the virtual object virtual object comprises a view of the damaged portion of the vehicle in an undamaged state.
9. The method of claim 6 , wherein the virtual object comprises an indication of an assessment of severity of damage to the damaged portion of the vehicle.
10. The method of claim 6 , wherein the virtual object comprises an estimate of a cost to repair the damaged portion of the vehicle.
11. The method of claim 1 , wherein the virtual object includes a curve.
12. The method of claim 1 , further comprising rendering a second virtual object, separate from and addition to the vehicle and the virtual object, into each of the additional live images output to the display.
13. The method of claim 1 , further comprising recognizing the vehicle in the first image and rendering a target into the first image to mark the vehicle.
14. The method of claim 1 , further comprising recognizing a plurality of vehicles in the first image including the vehicle and rendering targets over each of the plurality of vehicles to mark the vehicles.
15. The method of claim 14 , further comprising receiving an input indicating a selection of the one of the plurality of vehicles.
16. The method of claim 1 , wherein the camera is coupled to a hand-held device which is guided by a user during the capturing of the plurality of images.
17. The method of claim 1 , wherein the sensor data is received from an inertial measurement unit.
18. The method of claim 1 wherein the mobile device is a head mounted.
19. The method of claim 1 , further comprising, based upon the sensor data, determining the tracking point has moved from the first position in the 2-D pixel data and determining a second position of the tracking point in the 2-D pixel data.
20. The method of claim 1 , further comprising, replacing the tracking point with a new tracking point at a second position different from the first position.
US16/564,598
2016-12-12
2019-09-09
Live augmented reality guides
Active
US11024093B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US16/564,598
US11024093B2
( en )
2016-12-12
2019-09-09
Live augmented reality guides
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
US15/376,537
US9996945B1
( en )
2016-12-12
2016-12-12
Live augmented reality guides
US15/979,214
US10535197B2
( en )
2016-12-12
2018-05-14
Live augmented reality guides
US16/564,598
US11024093B2
( en )
2016-12-12
2019-09-09
Live augmented reality guides
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US15/979,214
Continuation
US10535197B2
( en )
2016-12-12
2018-05-14
Live augmented reality guides
Publications (2)
Publication Number
Publication Date
US20190392650A1
US20190392650A1 ( en )
2019-12-26
US11024093B2
true
US11024093B2 ( en )
2021-06-01
Family
ID=62485417
Family Applications (3)
Application Number
Title
Priority Date
Filing Date
US15/376,537
Active
US9996945B1
( en )
2016-12-12
2016-12-12
Live augmented reality guides
US15/979,214
Active
2037-02-07
US10535197B2
( en )
2016-12-12
2018-05-14
Live augmented reality guides
US16/564,598
Active
US11024093B2
( en )
2016-12-12
2019-09-09
Live augmented reality guides
Family Applications Before (2)
Application Number
Title
Priority Date
Filing Date
US15/376,537
Active
US9996945B1
( en )
2016-12-12
2016-12-12
Live augmented reality guides
US15/979,214
Active
2037-02-07
US10535197B2
( en )
2016-12-12
2018-05-14
Live augmented reality guides
Country Status (1)
Country
Link
US
( 3 )
US9996945B1
( en )
Families Citing this family (132)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
KR20170019366A
( en )
2014-05-16
2017-02-21
ëë²ì í¸ í í¬ëë¡ì§ì¤, ì¸í¬.
Modular formed nodes for vehicle chassis and their methods of use
JP6820843B2
( en )
2014-07-02
2021-01-27
ãã¤ãã¼ã¸ã§ã³ã ãã¯ããã¸ã¼ãºï¼ ã¤ã³ã³ã¼ãã¬ã¤ãããＤï½ï½ï½ ï½ï½ï½ ï½ï½ ï¼´ï½ ï½ï½ï½ï½ï½ï½ï½ï½ï½ ï½ï¼ Ｉï½ï½ï¼
Systems and methods for manufacturing fittings
US10509469B2
( en )
2016-04-21
2019-12-17
Finch Technologies Ltd.
Devices for controlling computers based on motions and positions of hands
US10173255B2
( en )
2016-06-09
2019-01-08
Divergent Technologies, Inc.
Systems and methods for arc and node design and manufacture
US10210662B2
( en )
2016-12-09
2019-02-19
Fyusion, Inc.
Live augmented reality using tracking
US9996945B1
( en )
2016-12-12
2018-06-12
Fyusion, Inc.
Live augmented reality guides
US10665024B2
( en )
2016-12-12
2020-05-26
Fyusion, Inc.
Providing recording guidance in generating a multi-view interactive digital media representation
US10999602B2
( en )
2016-12-23
2021-05-04
Apple Inc.
Sphere projected motion estimation/compensation and mode decision
US10692262B2
( en )
*
2017-01-12
2020-06-23
Electronics And Telecommunications Research Institute
Apparatus and method for processing information of multiple cameras
CN106875444B
( en )
*
2017-01-19
2019-11-19
æµæ±å¤§åææ¯è¡ä»½æéå ¬å¸
A kind of object localization method and device
US10759090B2
( en )
2017-02-10
2020-09-01
Divergent Technologies, Inc.
Methods for producing panels using 3D-printed tooling shells
US11155005B2
( en )
2017-02-10
2021-10-26
Divergent Technologies, Inc.
3D-printed tooling and methods for producing same
US11259046B2
( en )
2017-02-15
2022-02-22
Apple Inc.
Processing of equirectangular object data to compensate for distortion by spherical projections
US10924747B2
( en )
2017-02-27
2021-02-16
Apple Inc.
Video coding techniques for multi-view video
US12251884B2
( en )
2017-04-28
2025-03-18
Divergent Technologies, Inc.
Support structures in additive manufacturing
US10705113B2
( en )
2017-04-28
2020-07-07
Finch Technologies Ltd.
Calibration of inertial measurement units attached to arms of a user to generate inputs for computer systems
US10898968B2
( en )
2017-04-28
2021-01-26
Divergent Technologies, Inc.
Scatter reduction in additive manufacturing
US10379613B2
( en )
2017-05-16
2019-08-13
Finch Technologies Ltd.
Tracking arm movements to generate inputs for computer systems
US10540006B2
( en )
2017-05-16
2020-01-21
Finch Technologies Ltd.
Tracking torso orientation to generate inputs for computer systems
US10703419B2
( en )
2017-05-19
2020-07-07
Divergent Technologies, Inc.
Apparatus and methods for joining panels
US11358337B2
( en )
2017-05-24
2022-06-14
Divergent Technologies, Inc.
Robotic assembly of transport structures using on-site additive manufacturing
US11093752B2
( en )
*
2017-06-02
2021-08-17
Apple Inc.
Object tracking in multi-view video
US11123973B2
( en )
2017-06-07
2021-09-21
Divergent Technologies, Inc.
Interconnected deflectable panel and node
US10919230B2
( en )
2017-06-09
2021-02-16
Divergent Technologies, Inc.
Node with co-printed interconnect and methods for producing same
US10781846B2
( en )
2017-06-19
2020-09-22
Divergent Technologies, Inc.
3-D-printed components including fasteners and methods for producing same
US10994876B2
( en )
2017-06-30
2021-05-04
Divergent Technologies, Inc.
Automated wrapping of components in transport structures
US10754242B2
( en )
2017-06-30
2020-08-25
Apple Inc.
Adaptive resolution and projection format in multi-direction video
US11022375B2
( en )
2017-07-06
2021-06-01
Divergent Technologies, Inc.
Apparatus and methods for additively manufacturing microtube heat exchangers
US10895315B2
( en )
2017-07-07
2021-01-19
Divergent Technologies, Inc.
Systems and methods for implementing node to node connections in mechanized assemblies
US10940609B2
( en )
2017-07-25
2021-03-09
Divergent Technologies, Inc.
Methods and apparatus for additively manufactured endoskeleton-based transport structures
US10751800B2
( en )
2017-07-25
2020-08-25
Divergent Technologies, Inc.
Methods and apparatus for additively manufactured exoskeleton-based transport structures
US10605285B2
( en )
2017-08-08
2020-03-31
Divergent Technologies, Inc.
Systems and methods for joining node and tube structures
US10357959B2
( en )
2017-08-15
2019-07-23
Divergent Technologies, Inc.
Methods and apparatus for additively manufactured identification features
US11306751B2
( en )
2017-08-31
2022-04-19
Divergent Technologies, Inc.
Apparatus and methods for connecting tubes in transport structures
US10960611B2
( en )
2017-09-06
2021-03-30
Divergent Technologies, Inc.
Methods and apparatuses for universal interface between parts in transport structures
US11292058B2
( en )
2017-09-12
2022-04-05
Divergent Technologies, Inc.
Apparatus and methods for optimization of powder removal features in additively manufactured components
US10668816B2
( en )
2017-10-11
2020-06-02
Divergent Technologies, Inc.
Solar extended range electric vehicle with panel deployment and emitter tracking
US10814564B2
( en )
2017-10-11
2020-10-27
Divergent Technologies, Inc.
Composite material inlay in additively manufactured structures
EP3474230B1
( en )
*
2017-10-18
2020-07-22
Tata Consultancy Services Limited
Systems and methods for edge points based monocular visual slam
US11786971B2
( en )
2017-11-10
2023-10-17
Divergent Technologies, Inc.
Structures and methods for high volume production of complex structures using interface nodes
US10926599B2
( en )
2017-12-01
2021-02-23
Divergent Technologies, Inc.
Suspension systems using hydraulic dampers
US11110514B2
( en )
2017-12-14
2021-09-07
Divergent Technologies, Inc.
Apparatus and methods for connecting nodes to tubes in transport structures
US10521011B2
( en )
*
2017-12-19
2019-12-31
Finch Technologies Ltd.
Calibration of inertial measurement units attached to arms of a user and to a head mounted device
US11085473B2
( en )
2017-12-22
2021-08-10
Divergent Technologies, Inc.
Methods and apparatus for forming node to panel joints
US11534828B2
( en )
2017-12-27
2022-12-27
Divergent Technologies, Inc.
Assembling structures comprising 3D printed components and standardized components utilizing adhesive circuits
US10509464B2
( en )
2018-01-08
2019-12-17
Finch Technologies Ltd.
Tracking torso leaning to generate inputs for computer systems
US11016116B2
( en )
2018-01-11
2021-05-25
Finch Technologies Ltd.
Correction of accumulated errors in inertial measurement units attached to a user
US11420262B2
( en )
2018-01-31
2022-08-23
Divergent Technologies, Inc.
Systems and methods for co-casting of additively manufactured interface nodes
US10751934B2
( en )
2018-02-01
2020-08-25
Divergent Technologies, Inc.
Apparatus and methods for additive manufacturing with variable extruder profiles
US11224943B2
( en )
2018-03-07
2022-01-18
Divergent Technologies, Inc.
Variable beam geometry laser-based powder bed fusion
US11267236B2
( en )
2018-03-16
2022-03-08
Divergent Technologies, Inc.
Single shear joint for node-to-node connections
US11872689B2
( en )
2018-03-19
2024-01-16
Divergent Technologies, Inc.
End effector features for additively manufactured components
US11254381B2
( en )
2018-03-19
2022-02-22
Divergent Technologies, Inc.
Manufacturing cell based vehicle manufacturing system and method
US11408216B2
( en )
2018-03-20
2022-08-09
Divergent Technologies, Inc.
Systems and methods for co-printed or concurrently assembled hinge structures
US10496366B2
( en )
*
2018-04-18
2019-12-03
Comcast Cable Communications, Llc
Method to determine intended direction of a vocal command and target for vocal interaction
US11613078B2
( en )
2018-04-20
2023-03-28
Divergent Technologies, Inc.
Apparatus and methods for additively manufacturing adhesive inlet and outlet ports
US11214317B2
( en )
2018-04-24
2022-01-04
Divergent Technologies, Inc.
Systems and methods for joining nodes and other structures
US11020800B2
( en )
2018-05-01
2021-06-01
Divergent Technologies, Inc.
Apparatus and methods for sealing powder holes in additively manufactured parts
US10682821B2
( en )
2018-05-01
2020-06-16
Divergent Technologies, Inc.
Flexible tooling system and method for manufacturing of composite structures
US11474593B2
( en )
2018-05-07
2022-10-18
Finch Technologies Ltd.
Tracking user movements to control a skeleton model in a computer system
US10416755B1
( en )
2018-06-01
2019-09-17
Finch Technologies Ltd.
Motion predictions of overlapping kinematic chains of a skeleton model used to control a computer system
US11389816B2
( en )
2018-05-09
2022-07-19
Divergent Technologies, Inc.
Multi-circuit single port design in additively manufactured node
US10691104B2
( en )
2018-05-16
2020-06-23
Divergent Technologies, Inc.
Additively manufacturing structures for increased spray forming resolution or increased fatigue life
US11590727B2
( en )
2018-05-21
2023-02-28
Divergent Technologies, Inc.
Custom additively manufactured core structures
US11441586B2
( en )
2018-05-25
2022-09-13
Divergent Technologies, Inc.
Apparatus for injecting fluids in node based connections
US11035511B2
( en )
2018-06-05
2021-06-15
Divergent Technologies, Inc.
Quick-change end effector
US11292056B2
( en )
2018-07-06
2022-04-05
Divergent Technologies, Inc.
Cold-spray nozzle
US10948297B2
( en )
*
2018-07-09
2021-03-16
Samsung Electronics Co., Ltd.
Simultaneous location and mapping (SLAM) using dual event cameras
US11009941B2
( en )
2018-07-25
2021-05-18
Finch Technologies Ltd.
Calibration of measurement units in alignment with a skeleton model to control a computer system
US11269311B2
( en )
2018-07-26
2022-03-08
Divergent Technologies, Inc.
Spray forming structural joints
US10836120B2
( en )
2018-08-27
2020-11-17
Divergent Technologies, Inc .
Hybrid composite structures with integrated 3-D printed elements
US11433557B2
( en )
2018-08-28
2022-09-06
Divergent Technologies, Inc.
Buffer block apparatuses and supporting apparatuses
US11826953B2
( en )
2018-09-12
2023-11-28
Divergent Technologies, Inc.
Surrogate supports in additive manufacturing
US11072371B2
( en )
2018-10-05
2021-07-27
Divergent Technologies, Inc.
Apparatus and methods for additively manufactured structures with augmented energy absorption properties
US11260582B2
( en )
2018-10-16
2022-03-01
Divergent Technologies, Inc.
Methods and apparatus for manufacturing optimized panels and other composite structures
CN109192111A
( en )
*
2018-10-19
2019-01-11
æè¾
A kind of guide system that the scenic spot based on AR technology is visited
US12115583B2
( en )
2018-11-08
2024-10-15
Divergent Technologies, Inc.
Systems and methods for adhesive-based part retention features in additively manufactured structures
US12194536B2
( en )
2018-11-13
2025-01-14
Divergent Technologies, Inc.
3-D printer with manifolds for gas exchange
US11504912B2
( en )
2018-11-20
2022-11-22
Divergent Technologies, Inc.
Selective end effector modular attachment device
USD911222S1
( en )
2018-11-21
2021-02-23
Divergent Technologies, Inc.
Vehicle and/or replica
US11529741B2
( en )
2018-12-17
2022-12-20
Divergent Technologies, Inc.
System and method for positioning one or more robotic apparatuses
US10663110B1
( en )
2018-12-17
2020-05-26
Divergent Technologies, Inc.
Metrology apparatus to facilitate capture of metrology data
US11449021B2
( en )
2018-12-17
2022-09-20
Divergent Technologies, Inc.
Systems and methods for high accuracy fixtureless assembly
US11885000B2
( en )
2018-12-21
2024-01-30
Divergent Technologies, Inc.
In situ thermal treatment for PBF systems
US20190141310A1
( en )
*
2018-12-28
2019-05-09
Intel Corporation
Real-time, three-dimensional vehicle display
US12378643B2
( en )
2019-01-18
2025-08-05
Divergent Technologies, Inc.
Aluminum alloys
US10931972B2
( en )
*
2019-01-24
2021-02-23
Dell Products, L.P.
Forward channel contextual error concealment and sync for virtual, augmented, or mixed reality (XR) content in connectivity-constrained environments
US10937247B1
( en )
*
2019-03-11
2021-03-02
Amazon Technologies, Inc.
Three-dimensional room model generation using ring paths and photogrammetry
US11024079B1
( en )
2019-03-11
2021-06-01
Amazon Technologies, Inc.
Three-dimensional room model generation using panorama paths and photogrammetry
US11203240B2
( en )
2019-04-19
2021-12-21
Divergent Technologies, Inc.
Wishbone style control arm assemblies and methods for producing same
US12314031B1
( en )
2019-06-27
2025-05-27
Divergent Technologies, Inc.
Incorporating complex geometric features in additively manufactured parts
US10809797B1
( en )
2019-08-07
2020-10-20
Finch Technologies Ltd.
Calibration of multiple sensor modules related to an orientation of a user of the sensor modules
US12280554B2
( en )
2019-11-21
2025-04-22
Divergent Technologies, Inc.
Fixtureless robotic assembly
US11912339B2
( en )
2020-01-10
2024-02-27
Divergent Technologies, Inc.
3-D printed chassis structure with self-supporting ribs
US11590703B2
( en )
2020-01-24
2023-02-28
Divergent Technologies, Inc.
Infrared radiation sensing and beam control in electron beam additive manufacturing
US11884025B2
( en )
2020-02-14
2024-01-30
Divergent Technologies, Inc.
Three-dimensional printer and methods for assembling parts via integration of additive and conventional manufacturing operations
US12194674B2
( en )
2020-02-14
2025-01-14
Divergent Technologies, Inc.
Multi-material powder bed fusion 3-D printer
US11479015B2
( en )
2020-02-14
2022-10-25
Divergent Technologies, Inc.
Custom formed panels for transport structures and methods for assembling same
US12203397B2
( en )
2020-02-18
2025-01-21
Divergent Technologies, Inc.
Impact energy absorber with integrated engine exhaust noise muffler
US11535322B2
( en )
2020-02-25
2022-12-27
Divergent Technologies, Inc.
Omni-positional adhesion device
US11421577B2
( en )
2020-02-25
2022-08-23
Divergent Technologies, Inc.
Exhaust headers with integrated heat shielding and thermal syphoning
US12337541B2
( en )
2020-02-27
2025-06-24
Divergent Technologies, Inc.
Powder bed fusion additive manufacturing system with desiccant positioned within hopper and ultrasonic transducer
US11413686B2
( en )
2020-03-06
2022-08-16
Divergent Technologies, Inc.
Methods and apparatuses for sealing mechanisms for realizing adhesive connections with additively manufactured components
KR20230035571A
( en )
2020-06-10
2023-03-14
ëë²ì í¸ í í¬ëë¡ì§ì¤, ì¸í¬.
Adaptive production system
CN113784148B
( en )
*
2020-06-10
2026-01-13
é¿éå·´å·´éå¢æ§è¡æéå ¬å¸
Data processing method, system, related device and storage medium
US11850804B2
( en )
2020-07-28
2023-12-26
Divergent Technologies, Inc.
Radiation-enabled retention features for fixtureless assembly of node-based structures
US11806941B2
( en )
2020-08-21
2023-11-07
Divergent Technologies, Inc.
Mechanical part retention features for additively manufactured structures
CN116457139A
( en )
2020-09-08
2023-07-18
æ´å¼æ ¹ç¹ææ¯æéå ¬å¸
Assembly sequence generation
CN116669885A
( en )
2020-09-22
2023-08-29
æ´å¼æ ¹ç¹ææ¯æéå ¬å¸
Method and apparatus for ball milling to produce powders for additive manufacturing
EP4182893A4
( en )
2020-10-08
2023-12-06
Samsung Electronics Co., Ltd.
METHOD AND ELECTRONIC DEVICE FOR GENERATING AUGMENTED REALITY CONTENT BASED ON INTENTION AND INTERACTION OF MULTIPLE OBJECTS
US12220819B2
( en )
2020-10-21
2025-02-11
Divergent Technologies, Inc.
3-D printed metrology feature geometry and detection
US12311612B2
( en )
2020-12-18
2025-05-27
Divergent Technologies, Inc.
Direct inject joint architecture enabled by quick cure adhesive
US12083596B2
( en )
2020-12-21
2024-09-10
Divergent Technologies, Inc.
Thermal elements for disassembly of node-based adhesively bonded structures
US12226824B2
( en )
2020-12-22
2025-02-18
Divergent Technologies, Inc.
Three dimensional printer with configurable build plate for rapid powder removal
US11872626B2
( en )
2020-12-24
2024-01-16
Divergent Technologies, Inc.
Systems and methods for floating pin joint design
US11947335B2
( en )
2020-12-30
2024-04-02
Divergent Technologies, Inc.
Multi-component structure optimization for combining 3-D printed and commercially available parts
US11928966B2
( en )
2021-01-13
2024-03-12
Divergent Technologies, Inc.
Virtual railroad
US12249812B2
( en )
2021-01-19
2025-03-11
Divergent Technologies, Inc.
Bus bars for printed structural electric battery modules
US12459377B2
( en )
2021-01-19
2025-11-04
Divergent Technologies, Inc.
Energy unit cells for primary vehicle structure
US20220288850A1
( en )
2021-03-09
2022-09-15
Divergent Technologies, Inc.
Rotational additive manufacturing systems and methods
US12090551B2
( en )
2021-04-23
2024-09-17
Divergent Technologies, Inc.
Removal of supports, and other materials from surface, and within hollow 3D printed parts
US12138772B2
( en )
2021-04-30
2024-11-12
Divergent Technologies, Inc.
Mobile parts table
CN117769486A
( en )
2021-05-24
2024-03-26
æ´å¼æ ¹ç¹ææ¯æéå ¬å¸
Robotic gripper equipment
US11681539B2
( en )
2021-05-28
2023-06-20
International Business Machines Corporation
Transforming asset operation video to augmented reality guidance model
WO2023278878A1
( en )
2021-07-01
2023-01-05
Divergent Technologies, Inc.
Al-mg-si based near-eutectic alloy composition for high strength and stiffness applications
US12583033B2
( en )
2021-08-13
2026-03-24
Divergent Technologies, Inc.
<td itemprop="