ABSTRACT
Abstract
Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. According to particular embodiments, the spatial relationship between multiple images and video is analyzed together with location information data, for purposes of creating a representation referred to herein as a surround view for presentation on a device. A real object can be tracked in the live image data for the purposes of creating a surround view using a number of tracking points. As a camera is moved around the real object, virtual objects can be rendered into live image data to create synthetic images where a position of the tracking points can be used to position the virtual object in the synthetic image. The synthetic images can be output in real-time. Further, virtual objects in the synthetic images can be incorporated into surround views.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This patent document is a continuation of and claims priority to U.S. patent application Ser. No. 15/374,910 by Holzer et al., filed on Dec. 9, 2016, entitled, âLive Augmented Reality Using Tracking.â U.S. patent application Ser. No. 15/374,910 is hereby incorporated by reference in its entirety and for all purposes.
TECHNICAL FIELD
The present disclosure relates to augmenting multi-view image data with synthetic objects. In one example, the present disclosure relates to using inertial measurement unit (IMU) and image data to generate views of synthetic objects to be placed in a multi-view image or rendered into live image data.
Augmented reality typically includes a view of a real-world environment, such as through video and/or image data of scenery, a sports game, an object, individual, etc. This view of the real-world environment is augmented by computer generated input such as images, text, video, graphics, or the like. Accordingly, augmented reality can take the form of a live-action video or photo series with added elements that are computer-generated. Augmented reality is distinct from virtual reality, in which a simulated environment is depicted through video and/or image data.
In some implementations, augmented reality applications may add three-dimensional (3D) information to video and image data. This is generally done by creating a 3D reconstruction of the scene. However, this process is computationally expensive and usually restricted to static scenes. Accordingly, improved methods of implementing augmented reality are desirable.
Overview
Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. According to particular embodiments, the spatial relationship between multiple images and video is analyzed together with location information data, for purposes of creating a representation referred to herein as a surround view for presentation on a device. An object included in the surround view may be manipulated along axes by manipulating the device along corresponding axes. In particular embodiments, an augmented reality (AR) system is used for the purposes of capturing images used in a surround view. For example, live image data from camera of a mobile device can be augmented with virtual guides that help a user position the mobile device during image capture.
BRIEF DESCRIPTION OF THE DRAWINGS
The disclosure may best be understood by reference to the following description taken in conjunction with the accompanying drawings, which illustrate particular embodiments of the present invention.
FIG. 1 illustrates an example of a surround view acquisition system.
FIG. 2 illustrates an example of a process flow for generating a surround view.
FIG. 3 illustrates one example of multiple camera views that can be fused into a three-dimensional (3D) model to create an immersive experience.
FIG. 4 illustrates one example of separation of content and context in a surround view.
FIGS. 5A-5B illustrate examples of concave view and convex views, respectively, where both views use a back-camera capture style.
FIGS. 6A-6D illustrate examples of various capture modes for surround views.
FIGS. 7A and 7B illustrate an example of a process flow for capturing images in a surround view using augmented reality.
FIGS. 8A and 8B illustrate examples of generating an augmented reality image capture track for capturing images used in a surround view.
FIG. 9 illustrates an example of generating an augmented reality image capture track for capturing images used in a surround view on a mobile device.
FIGS. 10A and 10B illustrate an example of generating an augmented reality image capture track including status indicators for capturing images used in a surround view.
FIGS. 11A and 11B illustrate an example of generating an augmented reality image capture track including camera tilt effects on a mobile device.
FIGS. 12A-12D illustrate an example of generating an augmented reality image using tracking points on a mobile device.
FIGS. 13A-13C illustrate an example of generating an augmented reality image using tracking points on a mobile device.
FIG. 14 illustrates an example of a process flow for generating augmented reality images using tracking points.
FIG. 15 illustrates a particular example of a computer system that can be used with various embodiments of the present invention.
DETAILED DESCRIPTION
Reference will now be made in detail to some specific examples of the invention including the best modes contemplated by the inventors for carrying out the invention. Examples of these specific embodiments are illustrated in the accompanying drawings. While the present disclosure is described in conjunction with these specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. Particular embodiments of the present invention may be implemented without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.
Various aspects of the present invention relate generally to systems and methods for analyzing the spatial relationship between multiple images and video together with location information data, for the purpose of creating a single representation, a surround view, which eliminates redundancy in the data, and presents a user with an interactive and immersive active viewing experience. According to various embodiments, active is described in the context of providing a user with the ability to control the viewpoint of the visual information displayed on a screen.
In particular example embodiments, augmented reality (AR) is used to aid a user in capturing the multiple images used in a surround view. For example, a virtual guide can be inserted into live image data from a mobile. The virtual guide can help the user guide the mobile device along a desirable path useful for creating the surround view. The virtual guide in the AR images can respond to movements of the mobile device. The movement of mobile device can be determined from a number of different sources, including but not limited to an Inertial Measurement Unit and image data.
According to various embodiments of the present invention, a surround view is a multi-view interactive digital media representation. With reference to FIG. 1 , shown is one example of a surround view acquisition system 100 . In the present example embodiment, the surround view acquisition system 100 is depicted in a flow sequence that can be used to generate a surround view. According to various embodiments, the data used to generate a surround view can come from a variety of sources.
In particular, data such as, but not limited to two-dimensional (2D) images 104 can be used to generate a surround view. These 2D images can include color image data streams such as multiple image sequences, video data, etc., or multiple images in any of various formats for images, depending on the application. As will be described in more detail below with respect to FIGS. 7A-11B , during an image capture process, an AR system can be used. The AR system can receive and augment live image data with virtual data. In particular, the virtual data can include guides for helping a user direct the motion of an image capture device.
Another source of data that can be used to generate a surround view includes environment information 106 . This environment information 106 can be obtained from sources such as accelerometers, gyroscopes, magnetometers, GPS, WiFi, IMU-like systems (Inertial Measurement Unit systems), and the like. Yet another source of data that can be used to generate a surround view can include depth images 108 . These depth images can include depth, 3D, or disparity image data streams, and the like, and can be captured by devices such as, but not limited to, stereo cameras, time-of-flight cameras, three-dimensional cameras, and the like.
In the present example embodiment, the data can then be fused together at sensor fusion block 110 . In some embodiments, a surround view can be generated a combination of data that includes both 2D images 104 and environment information 106 , without any depth images 108 provided. In other embodiments, depth images 108 and environment information 106 can be used together at sensor fusion block 110 . Various combinations of image data can be used with environment information at 106 , depending on the application and available data.
In the present example embodiment, the data that has been fused together at sensor fusion block 110 is then used for content modeling 112 and context modeling 114 . As described in more detail with regard to FIG. 4 , the subject matter featured in the images can be separated into content and context. The content can be delineated as the object of interest and the context can be delineated as the scenery surrounding the object of interest. According to various embodiments, the content can be a three-dimensional model, depicting an object of interest, although the content can be a two-dimensional image in some embodiments, as described in more detail below with regard to FIG. 4 . Furthermore, in some embodiments, the context can be a two-dimensional model depicting the scenery surrounding the object of interest. Although in many examples the context can provide two-dimensional views of the scenery surrounding the object of interest, the context can also include three-dimensional aspects in some embodiments. For instance, the context can be depicted as a âflatâ image along a cylindrical âcanvas,â such that the âflatâ image appears on the surface of a cylinder. In addition, some examples may include three-dimensional context models, such as when some objects are identified in the surrounding scenery as three-dimensional objects. According to various embodiments, the models provided by content modeling 112 and context modeling 114 can be generated by combining the image and location information data, as described in more detail with regard to FIG. 3 .
According to various embodiments, context and content of a surround view are determined based on a specified object of interest. In some examples, an object of interest is automatically chosen based on processing of the image and location information data. For instance, if a dominant object is detected in a series of images, this object can be selected as the content. In other examples, a user specified target 102 can be chosen, as shown in FIG. 1 . It should be noted, however, that a surround view can be generated without a user specified target in some applications.
In the present example embodiment, one or more enhancement algorithms can be applied at enhancement algorithm(s) block 116 . In particular example embodiments, various algorithms can be employed during capture of surround view data, regardless of the type of capture mode employed. These algorithms can be used to enhance the user experience. For instance, automatic frame selection, stabilization, view interpolation, filters, and/or compression can be used during capture of surround view data. In some examples, these enhancement algorithms can be applied to image data after acquisition of the data. In other examples, these enhancement algorithms can be applied to image data during capture of surround view data.
According to particular example embodiments, automatic frame selection can be used to create a more enjoyable surround view. Specifically, frames are automatically selected so that the transition between them will be smoother or more even. This automatic frame selection can incorporate blur- and overexposure-detection in some applications, as well as more uniformly sampling poses such that they are more evenly distributed.
In some example embodiments, stabilization can be used for a surround view in a manner similar to that used for video. In particular, keyframes in a surround view can be stabilized for to produce improvements such as smoother transitions, improved/enhanced focus on the content, etc. However, unlike video, there are many additional sources of stabilization for a surround view, such as by using IMU information, depth information, computer vision techniques, direct selection of an area to be stabilized, face detection, and the like.
For instance, IMU information can be very helpful for stabilization. In
CROSS-REFERENCE TO RELATED APPLICATIONS
This patent document is a continuation of and claims priority to U.S. patent application Ser. No. 15/374,910 by Holzer et al., filed on Dec. 9, 2016, entitled, âLive Augmented Reality Using Tracking.â U.S. patent application Ser. No. 15/374,910 is hereby incorporated by reference in its entirety and for all purposes.
TECHNICAL FIELD
The present disclosure relates to augmenting multi-view image data with synthetic objects. In one example, the present disclosure relates to using inertial measurement unit (IMU) and image data to generate views of synthetic objects to be placed in a multi-view image or rendered into live image data.
Augmented reality typically includes a view of a real-world environment, such as through video and/or image data of scenery, a sports game, an object, individual, etc. This view of the real-world environment is augmented by computer generated input such as images, text, video, graphics, or the like. Accordingly, augmented reality can take the form of a live-action video or photo series with added elements that are computer-generated. Augmented reality is distinct from virtual reality, in which a simulated environment is depicted through video and/or image data.
In some implementations, augmented reality applications may add three-dimensional (3D) information to video and image data. This is generally done by creating a 3D reconstruction of the scene. However, this process is computationally expensive and usually restricted to static scenes. Accordingly, improved methods of implementing augmented reality are desirable.
Overview
Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. According to particular embodiments, the spatial relationship between multiple images and video is analyzed together with location information data, for purposes of creating a representation referred to herein as a surround view for presentation on a device. An object included in the surround view may be manipulated along axes by manipulating the device along corresponding axes. In particular embodiments, an augmented reality (AR) system is used for the purposes of capturing images used in a surround view. For example, live image data from camera of a mobile device can be augmented with virtual guides that help a user position the mobile device during image capture.
BRIEF DESCRIPTION OF THE DRAWINGS
The disclosure may best be understood by reference to the following description taken in conjunction with the accompanying drawings, which illustrate particular embodiments of the present invention.
FIG. 1 illustrates an example of a surround view acquisition system.
FIG. 2 illustrates an example of a process flow for generating a surround view.
FIG. 3 illustrates one example of multiple camera views that can be fused into a three-dimensional (3D) model to create an immersive experience.
FIG. 4 illustrates one example of separation of content and context in a surround view.
FIGS. 5A-5B illustrate examples of concave view and convex views, respectively, where both views use a back-camera capture style.
FIGS. 6A-6D illustrate examples of various capture modes for surround views.
FIGS. 7A and 7B illustrate an example of a process flow for capturing images in a surround view using augmented reality.
FIGS. 8A and 8B illustrate examples of generating an augmented reality image capture track for capturing images used in a surround view.
FIG. 9 illustrates an example of generating an augmented reality image capture track for capturing images used in a surround view on a mobile device.
FIGS. 10A and 10B illustrate an example of generating an augmented reality image capture track including status indicators for capturing images used in a surround view.
FIGS. 11A and 11B illustrate an example of generating an augmented reality image capture track including camera tilt effects on a mobile device.
FIGS. 12A-12D illustrate an example of generating an augmented reality image using tracking points on a mobile device.
FIGS. 13A-13C illustrate an example of generating an augmented reality image using tracking points on a mobile device.
FIG. 14 illustrates an example of a process flow for generating augmented reality images using tracking points.
FIG. 15 illustrates a particular example of a computer system that can be used with various embodiments of the present invention.
DETAILED DESCRIPTION
Reference will now be made in detail to some specific examples of the invention including the best modes contemplated by the inventors for carrying out the invention. Examples of these specific embodiments are illustrated in the accompanying drawings. While the present disclosure is described in conjunction with these specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. Particular embodiments of the present invention may be implemented without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.
Various aspects of the present invention relate generally to systems and methods for analyzing the spatial relationship between multiple images and video together with location information data, for the purpose of creating a single representation, a surround view, which eliminates redundancy in the data, and presents a user with an interactive and immersive active viewing experience. According to various embodiments, active is described in the context of providing a user with the ability to control the viewpoint of the visual information displayed on a screen.
In particular example embodiments, augmented reality (AR) is used to aid a user in capturing the multiple images used in a surround view. For example, a virtual guide can be inserted into live image data from a mobile. The virtual guide can help the user guide the mobile device along a desirable path useful for creating the surround view. The virtual guide in the AR images can respond to movements of the mobile device. The movement of mobile device can be determined from a number of different sources, including but not limited to an Inertial Measurement Unit and image data.
According to various embodiments of the present invention, a surround view is a multi-view interactive digital media representation. With reference to FIG. 1 , shown is one example of a surround view acquisition system 100 . In the present example embodiment, the surround view acquisition system 100 is depicted in a flow sequence that can be used to generate a surround view. According to various embodiments, the data used to generate a surround view can come from a variety of sources.
In particular, data such as, but not limited to two-dimensional (2D) images 104 can be used to generate a surround view. These 2D images can include color image data streams such as multiple image sequences, video data, etc., or multiple images in any of various formats for images, depending on the application. As will be described in more detail below with respect to FIGS. 7A-11B , during an image capture process, an AR system can be used. The AR system can receive and augment live image data with virtual data. In particular, the virtual data can include guides for helping a user direct the motion of an image capture device.
Another source of data that can be used to generate a surround view includes environment information 106 . This environment information 106 can be obtained from sources such as accelerometers, gyroscopes, magnetometers, GPS, WiFi, IMU-like systems (Inertial Measurement Unit systems), and the like. Yet another source of data that can be used to generate a surround view can include depth images 108 . These depth images can include depth, 3D, or disparity image data streams, and the like, and can be captured by devices such as, but not limited to, stereo cameras, time-of-flight cameras, three-dimensional cameras, and the like.
In the present example embodiment, the data can then be fused together at sensor fusion block 110 . In some embodiments, a surround view can be generated a combination of data that includes both 2D images 104 and environment information 106 , without any depth images 108 provided. In other embodiments, depth images 108 and environment information 106 can be used together at sensor fusion block 110 . Various combinations of image data can be used with environment information at 106 , depending on the application and available data.
In the present example embodiment, the data that has been fused together at sensor fusion block 110 is then used for content modeling 112 and context modeling 114 . As described in more detail with regard to FIG. 4 , the subject matter featured in the images can be separated into content and context. The content can be delineated as the object of interest and the context can be delineated as the scenery surrounding the object of interest. According to various embodiments, the content can be a three-dimensional model, depicting an object of interest, although the content can be a two-dimensional image in some embodiments, as described in more detail below with regard to FIG. 4 . Furthermore, in some embodiments, the context can be a two-dimensional model depicting the scenery surrounding the object of interest. Although in many examples the context can provide two-dimensional views of the scenery surrounding the object of interest, the context can also include three-dimensional aspects in some embodiments. For instance, the context can be depicted as a âflatâ image along a cylindrical âcanvas,â such that the âflatâ image appears on the surface of a cylinder. In addition, some examples may include three-dimensional context models, such as when some objects are identified in the surrounding scenery as three-dimensional objects. According to various embodiments, the models provided by content modeling 112 and context modeling 114 can be generated by combining the image and location information data, as described in more detail with regard to FIG. 3 .
According to various embodiments, context and content of a surround view are determined based on a specified object of interest. In some examples, an object of interest is automatically chosen based on processing of the image and location information data. For instance, if a dominant object is detected in a series of images, this object can be selected as the content. In other examples, a user specified target 102 can be chosen, as shown in FIG. 1 . It should be noted, however, that a surround view can be generated without a user specified target in some applications.
In the present example embodiment, one or more enhancement algorithms can be applied at enhancement algorithm(s) block 116 . In particular example embodiments, various algorithms can be employed during capture of surround view data, regardless of the type of capture mode employed. These algorithms can be used to enhance the user experience. For instance, automatic frame selection, stabilization, view interpolation, filters, and/or compression can be used during capture of surround view data. In some examples, these enhancement algorithms can be applied to image data after acquisition of the data. In other examples, these enhancement algorithms can be applied to image data during capture of surround view data.
According to particular example embodiments, automatic frame selection can be used to create a more enjoyable surround view. Specifically, frames are automatically selected so that the transition between them will be smoother or more even. This automatic frame selection can incorporate blur- and overexposure-detection in some applications, as well as more uniformly sampling poses such that they are more evenly distributed.
In some example embodiments, stabilization can be used for a surround view in a manner similar to that used for video. In particular, keyframes in a surround view can be stabilized for to produce improvements such as smoother transitions, improved/enhanced focus on the content, etc. However, unlike video, there are many additional sources of stabilization for a surround view, such as by using IMU information, depth information, computer vision techniques, direct selection of an area to be stabilized, face detection, and the like.
For instance, IMU information can be very helpful for stabilization. In particular, IMU information provides an estimate, although sometimes a rough or noisy estimate, of the camera tremor that may occur during image capture. This estimate can be used to remove, cancel, and/or reduce the effects of such camera tremor.
In some examples, depth information, if available, can be used to provide stabilization for a surround view. Because points of interest in a surround view are three-dimensional, rather than two-dimensional, these points of interest are more constrained and tracking/matching of these points is simplified as the search space reduces. Furthermore, descriptors for points of interest can use both color and depth information and therefore, become more discriminative. In addition, automatic or semi-automatic content selection can be easier to provide with depth information. For instance, when a user selects a particular pixel of an image, this selection can be expanded to fill the entire surface that touches it. Furthermore, content can also be selected automatically by using a foreground/background differentiation based on depth. In various examples, the content can stay relatively stable/visible even when the context changes.
According to various examples, computer vision techniques can also be used to provide stabilization for surround views. For instance, keypoints can be detected and tracked. However, in certain scenes, such as a dynamic scene or static scene with parallax, no simple warp exists that can stabilize everything. Consequently, there is a trade-off in which certain aspects of the scene receive more attention to stabilization and other aspects of the scene receive less attention. Because a surround view is often focused on a particular object of interest, a surround view can be content-weighted so that the object of interest is maximally stabilized in some examples.
Another way to improve stabilization in a surround view includes direct selection of a region of a screen. For instance, if a user taps to focus on a region of a screen, then records a convex surround view, the area that was tapped can be maximally stabilized. This allows stabilization algorithms to be focused on a particular area or object of interest.
In some examples, face detection can be used to provide stabilization. For instance, when recording with a front-facing camera, it is often likely that the user is the object of interest in the scene. Thus, face detection can be used to weight stabilization about that region. When face detection is precise enough, facial features themselves (such as eyes, nose, and mouth) can be used as areas to stabilize, rather than using generic keypoints. In another example, a user can select an area of image to use as a source for keypoints.
According to various examples, view interpolation can be used to improve the viewing experience. In particular, to avoid sudden âjumpsâ between stabilized frames, synthetic, intermediate views can be rendered on the fly. This can be informed by content-weighted keypoint tracks and IMU information as described above, as well as by denser pixel-to-pixel matches. If depth information is available, fewer artifacts resulting from mismatched pixels may occur, thereby simplifying the process. As described above, view interpolation can be applied during capture of a surround view in some embodiments. In other embodiments, view interpolation can be applied during surround view generation.
In some examples, filters can also be used during capture or generation of a surround view to enhance the viewing experience. Just as many popular photo sharing services provide aesthetic filters that can be applied to static, two-dimensional images, aesthetic filters can similarly be applied to surround images. However, because a surround view representation is more expressive than a two-dimensional image, and three-dimensional information is available in a surround view, these filters can be extended to include effects that are ill-defined in two dimensional photos. For instance, in a surround view, motion blur can be added to the background (i.e. context) while the content remains crisp. In another example, a drop-shadow can be added to the object of interest in a surround view.
In various examples, compression can also be used as an enhancement algorithm 116 . In particular, compression can be used to enhance user-experience by reducing data upload and download costs. Because surround views use spatial information, far less data can be sent for a surround view than a typical video, while maintaining desired qualities of the surround view. Specifically, the IMU, keypoint tracks, and user input, combined with the view interpolation described above, can all reduce the amount of data that must be transferred to and from a device during upload or download of a surround view. For instance, if an object of interest can be properly identified, a variable compression style can be chosen for the content and context. This variable compression style can include lower quality resolution for background information (i.e. context) and higher quality resolution for foreground information (i.e. content) in some examples. In such examples, the amount of data transmitted can be reduced by sacrificing some of the context quality, while maintaining a desired level of quality for the content.
In the present embodiment, a surround view 118 is generated after any enhancement algorithms are applied. The surround view can provide a multi-view interactive digital media representation. In various examples, the surround view can include three-dimensional model of the content and a two-dimensional model of the context. However, in some examples, the context can represent a âflatâ view of the scenery or background as projected along a surface, such as a cylindrical or other-shaped surface, such that the context is not purely two-dimensional. In yet other examples, the context can include three-dimensional aspects.
According to various embodiments, surround views provide numerous advantages over traditional two-dimensional images or videos. Some of these advantages include: the ability to cope with moving scenery, a moving acquisition device, or both; the ability to model parts of the scene in three-dimensions; the ability to remove unnecessary, redundant information and reduce the memory footprint of the output dataset; the ability to distinguish between content and context; the ability to use the distinction between content and context for improvements in the user-experience; the ability to use the distinction between content and context for improvements in memory footprint (an example would be high quality compression of content and low quality compression of context); the ability to associate special feature descriptors with surround views that allow the surround views to be indexed with a high degree of efficiency and accuracy; and the ability of the user to interact and change the viewpoint of the surround view. In particular example embodiments, the characteristics described above can be incorporated natively in the surround view representation, and provide the capability for use in various applications. For instance, surround views can be used to enhance various fields such as e-commerce, visual search, 3D printing, file sharing, user interaction, and entertainment.
According to various example embodiments, once a surround view 118 is generated, user feedback for acquisition 120 of additional image data can be provided. In particular, if a surround view is determined to need additional views to provide a more accurate model of the content or context, a user may be prompted to provide additional views. Once these additional views are received by the surround view acquisition system 100 , these additional views can be processed by the system 100 and incorporated into the surround view.
With reference to FIG. 2 , shown is an example of a process flow diagram for generating a surround view 200 . In the present example, a plurality of images is obtained at 202 . According to various embodiments, the plurality of images can include two-dimensional (2D) images or data streams. These 2D images can include location information that can be used to generate a surround view. In some embodiments, the plurality of images can include depth images 108 , as also described above with regard to FIG. 1 . The depth images can also include location information in various examples.
As is described in more detail with respect to FIGS. 7A-11B , when the plurality of images is captured, images output to the user can be augmented with the virtual data. For example, the plurality of images can be captured using a camera system on a mobile device. The live image data, which is output to a display on the mobile device, can include virtual data, such as guides and status indicators, rendered into the live image data. The guides can help a user guide a motion of the mobile device. The status indicators can indicate what portion of images needed for generating a surround view have been captured. The virtual data may not be included in the image data captured for the purposes of generating the surround view.
According to various embodiments, the plurality of images obtained at 202 can include a variety of sources and characteristics. For instance, the plurality of images can be obtained from a plurality of users. These images can be a collection of images gathered from the internet from different users of the same event, such as 2D images or video obtained at a concert, etc. In some examples, the plurality of images can include images with different temporal information. In particular, the images can be taken at different times of the same object of interest. For instance, multiple images of a particular statue can be obtained at different times of day, different seasons, etc. In other examples, the plurality of images can represent moving objects. For instance, the images may include an object of interest moving through scenery, such as a vehicle traveling along a road or a plane traveling through the sky. In other instances, the images may include an object of interest that is also moving, such as a person dancing, running, twirling, etc.
In the present example embodiment, the plurality of images is fused into content and context models at 204 . According to various embodiments, the subject matter featured in the images can be separated into content and context. The content can be delineated as the object of interest and the context can be delineated as the scenery surrounding the object of interest. According to various embodiments, the content can be a three-dimensional model, depicting an object of interest, and the content can be a two-dimensional image in some embodiments.
According to the present example embodiment, one or more enhancement algorithms can be applied to the content and context models at 206 . These algorithms can be used to enhance the user experience. For instance, enhancement algorithms such as automatic frame selection, stabilization, view interpolation, filters, and/or compression can be used. In some examples, these enhancement algorithms can be applied to image data during capture of the images. In other examples, these enhancement algorithms can be applied to image data after acquisition of the data.
In the present embodiment, a surround view is generated from the content and context models at 208 . The surround view can provide a multi-view interactive digital media representation. In various examples, the surround view can include a three-dimensional model of the content and a two-dimensional model of the context. According to various embodiments, depending on the mode of capture and the viewpoints of the images, the surround view model can include certain characteristics. For instance, some examples of different styles of surround views include a locally concave surround view, a locally convex surround view, and a locally flat surround view. However, it should be noted that surround views can include combinations of views and characteristics, depending on the application.
With reference to FIG. 3 , shown is one example of multiple camera views that can be fused together into a three-dimensional (3D) model to create an immersive experience. According to various embodiments, multiple images can be captured from various viewpoints and fused together to provide a surround view. In the present example embodiment, three
cameras
312 , 314 , and 316 are positioned at
locations
322 , 324 , and 326 , respectively, in proximity to an object of interest 308 . Scenery can surround the object of interest 308 such as object 310 .
Views
302 , 304 , and 306 from their
respective cameras
312 , 314 , and 316 include overlapping subject matter. Specifically, each
view
302 , 304 , and 306 includes the object of interest 308 and varying degrees of visibility of the scenery surrounding the object 310 . For instance, view 302 includes a view of the object of interest 308 in front of the cylinder that is part of the scenery surrounding the object 310 . View 306 shows the object of interest 308 to one side of the cylinder, and view 304 shows the object of interest without any view of the cylinder.
In the present example embodiment, the
various views
302 , 304 , and 316 along with their associated
locations
322 , 324 , and 326 , respectively, provide a rich source of information about object of interest 308 and the surrounding context that can be used to produce a surround view. For instance, when analyzed together, the
various views
302 , 304 , and 326 provide information about different sides of the object of interest and the relationship between the object of interest and the scenery. According to various embodiments, this information can be used to parse out the object of interest 308 into content and the scenery as the context. Furthermore, as also described above with regard to FIGS. 1 and 2 , various algorithms can be applied to images produced by these viewpoints to create an immersive, interactive experience when viewing a surround view.
FIG. 4 illustrates one example of separation of content and context in a surround view. According to various embodiments of the present invention, a surround view is a multi-view interactive digital media representation of a scene 400 . With reference to FIG. 4 , shown is a user 402 located in a scene 400 . The user 402 is capturing images of an object of interest, such as a statue. The images captured by the user constitute digital visual data that can be used to generate a surround view.
According to various embodiments of the present disclosure, the digital visual data included in a surround view can be, semantically and/or practically, separated into content 404 and context 406 . According to particular embodiments, content 404 can include the object(s), person(s), or scene(s) of interest while the context 406 represents the remaining elements of the scene surrounding the content 404 . In some examples, a surround view may represent the content 404 as three-dimensional data, and the context 406 as a two-dimensional panoramic background. In other examples, a surround view may represent both the content 404 and context 406 as two-dimensional panoramic scenes. In yet other examples, content 404 and context 406 may include three-dimensional components or aspects. In particular embodiments, the way that the surround view depicts content 404 and context 406 depends on the capture mode used to acquire the images.
In some examples, such as but not limited to: recordings of objects, persons, or parts of objects or persons, where only the object, person, or parts of them are visible, recordings of large flat areas, and recordings of scenes where the data captured appears to be at infinity (i.e., there are no subjects close to the camera), the content 404 and the context 406 may be the same. In these examples, the surround view produced may have some characteristics that are similar to other types of digital media such as panoramas. However, according to various embodiments, surround views include additional features that distinguish them from these existing types of digital media. For instance, a surround view can represent moving data. Additionally, a surround view is not limited to a specific cylindrical, spherical or translational movement. Various motions can be used to capture image data with a camera or other capture device. Furthermore, unlike a stitched panorama, a surround view can display different sides of the same object.
FIGS. 5A-5B illustrate examples of concave and convex views, respectively, where both views use a back-camera capture style. In particular, if a camera phone is used, these views use the camera on the back of the phone, facing away from the user. In particular embodiments, concave and convex views can affect how the content and context are designated in a surround view.
With reference to FIG. 5A , shown is one example of a concave view 500 in which a user is standing along a vertical axis 508 . In this example, the user is holding a camera, such that camera location 502 does not leave axis 508 during image capture. However, as the user pivots about axis 508 , the camera captures a panoramic view of the scene around the user, forming a concave view. In this embodiment, the object of interest 504 and the distant scenery 506 are all viewed similarly because of the way in which the images are captured. In this example, all objects in the concave view appear at infinity, so the content is equal to the context according to this view.
With reference to FIG. 5B , shown is one example of a convex view 520 in which a user changes position when capturing images of an object of interest 524 . In this example, the user moves around the object of interest 524 , taking pictures from different sides of the object of interest from camera locations
528 , 530 , and 532 . Each of the images obtained includes a view of the object of interest, and a background of the distant scenery 526 . In the present example, the object of interest 524 represents the content, and the distant scenery 526 represents the context in this convex view.
FIGS. 6A-6D illustrate examples of various capture modes for surround views. Although various motions can be used to capture a surround view and are not constrained to any particular type of motion, three general types of motion can be used to capture particular features or views described in conjunction surround views. These three types of motion, respectively, can yield a locally concave surround view, a locally convex surround view, and a locally flat surround view. In some examples, a surround view can include various types of motions within the same surround view.
With reference to FIG. 6A , shown is an example of a back-facing, concave surround view being captured. According to various embodiments, a locally concave surround view is one in which the viewing angles of the camera or other capture device diverge. In one dimension this can be likened to the motion required to capture a spherical 360 panorama (pure rotation), although the motion can be generalized to any curved sweeping motion in which the view faces outward. In the present example, the experience is that of a stationary viewer looking out at a (possibly dynamic) context.
In the present example embodiment, a user 602 is using a back-facing camera 606 to capture images towards world 600 , and away from user 602 . As described in various examples, a back-facing camera refers to a device with a camera that faces away from the user, such as the camera on the back of a smart phone. The camera is moved in a concave motion 608 , such that
views
604 a , 604 b , and 604 c capture various parts of capture area 609 .
With reference to FIG. 6B , shown is an example of a back-facing, convex surround view being captured. According to various embodiments, a locally convex surround view is one in which viewing angles converge toward a single object of interest. In some examples, a locally convex surround view can provide the experience of orbiting about a point, such that a viewer can see multiple sides of the same object. This object, which may be an âobject of interest,â can be segmented from the surround view to become the content, and any surrounding data can be segmented to become the context. Previous technologies fail to recognize this type of viewing angle in the media-sharing landscape.
In the present example embodiment, a user 602 is using a back-facing camera 614 to capture images towards world 600 , and away from user 602 . The camera is moved in a convex motion 610 , such that views
612 a , 612 b , and 612 c capture various parts of capture area 611 . As described above, world 600 can include an object of interest in some examples, and the convex motion 610 can orbit around this object. Views
612 a , 612 b , and 612 c can include views of different sides of this object in these examples.
With reference to FIG. 6C , shown is an example of a front-facing, concave surround view being captured. As described in various examples, a front-facing camera refers to a device with a camera that faces towards the user, such as the camera on the front of a smart phone. For instance, front-facing cameras are commonly used to take âselfiesâ (i.e., self-portraits of the user).
In the present example embodiment, camera 620 is facing user 602 . The camera follows a concave motion 606 such that the
views
618 a , 618 b , and 618 c diverge from each other in an angular sense. The capture area 617 follows a concave shape that includes the user at a perimeter.
With reference to FIG. 6D , shown is an example of a front-facing, convex surround view being captured. In the present example embodiment, camera 626 is facing user 602 . The camera follows a convex motion 622 such that the
views
624 a , 624 b , and 624 c converge towards the user 602 . As described above, various modes can be used to capture images for a surround view. These modes, including locally concave, locally convex, and locally linear motions, can be used during capture of separate images or during continuous recording of a scene. Such recording can capture a series of images during a single session.
Next, details of an augmented reality system, which is usable in the image capture process for a surround view, is described with respect to FIG. 7A to FIG. 11B . In one embodiment, the augmented reality system can be implemented on a mobile device, such as a cell phone. In particular, the live camera data, which is output to a display on the mobile device, can be augmented with virtual objects. The virtual objects can be rendered into the live camera data. In one embodiment, the virtual objects can provide a user feedback when images are being captured for a surround view.
FIGS. 7A and 7B illustrate an example of a process flow for capturing images in a surround view using augmented reality. In 702 , live image data can be received from a camera system. For example, live image data can be received from one or more cameras on a hand-held mobile device, such as a smartphone. The image data can include pixel data captured from a camera sensor. The pixel data varies from frame to frame. In one embodiment, the pixel data can be 2-D. In other embodiments, depth data can be included with the pixel data.
In 704 , sensor data can be received. For example, the mobile device can include an IMU with accelerometers and gyroscopes. The sensor data can be used to determine an orientation of the mobile device, such as a tilt orientation of the device relative to t
CLAIMS
Claims ( 20 )
What is claimed is:
1. A method comprising:
receiving a request to capture a plurality of images used to generate a multi-view interactive digital media representation of a vehicle appearing in the plurality of images;
receiving first live images, comprising the vehicle, captured from a camera on a mobile device, the first live images being output to a display of the mobile device, the first live images comprising first 2-D pixel data captured by the camera;
receiving first sensor data indicating a first orientation of the camera associated with a first image of the first live images;
generating a first synthetic image comprising: 1) a location selector rendered into the first 2-D pixel data associated with the first image, the location selector being a movable first virtual object in the first synthetic image such that selection of the location selector causes a pixel location in the first image to be selected and 2) the vehicle;
receiving, via a touch screen of the display and via the location selector, a selection of the pixel location in the first 2-D pixel data from the first image;
determining a first pixel location in the first 2-D pixel data from the first image of a first tracking point, the first tracking point being within the first 2-D pixel data associated with the vehicle and the first tracking point being proximate to the pixel location selected via the location selector;
generating a second synthetic image comprising 1) a second virtual object rendered into the first 2-D pixel data from the first image, the second virtual object being positioned in the first 2-D pixel data from the first live image relative to first pixel location of the first tracking point;
outputting the second synthetic image to the display;
after the first image is captured, receiving second live images captured by the camera, the second live images comprising second 2-D pixel data, the second live images comprising the vehicle from a plurality of different views;
receiving second sensor data, associated with the second live images, indicating second orientations of the camera;
receiving second live image data comprising second 2-D pixel data from the camera;
based upon the first sensor data, the second sensor data, the first 2-D pixel data and the second 2-D pixel data, determining, as the view of the vehicle in the second live images changes, second pixel locations of the first tracking point in the second 2-D pixel data of the second live images on an image by image basis, the second pixel locations being determined using spatial intensity information or optical flows derived from the second 2-D pixel data;
generating a third synthetic images comprising the second virtual object rendered into the second 2-D pixel data at third pixel locations positioned relative to the second pixel locations of the first tracking point;
outputting the third synthetic images to the display, each of the third synthetic images showing one of the different views of the vehicle as captured by the camera and the second virtual object.
2. The method of claim 1 , wherein the pixel location is over the vehicle captured in the first 2-D pixel data.
3. The method of claim 1 , wherein the first virtual object is a target.
4. The method of claim 3 , further comprising receiving an input indicating a selection of the vehicle captured in the first 2-D pixel data over which the target is rendered.
5. The method of claim 3 , further comprising receiving input used to position the target over the pixel location.
6. The method of claim 5 , wherein the input is received from the touch screen over the display of mobile device.
7. The method of claim 1 , wherein one or more components of the vehicle that are moving are captured in the first live images and the second live images.
8. The method of claim 1 , wherein the camera is coupled to the mobile device comprising an Inertial Measurement unit (IMU) wherein the first sensor data and the second sensor data is received from the IMU.
9. The method of claim 8 , wherein the IMU includes at least one gyroscope and at least one accelerometer.
10. The method of claim 1 , wherein the second virtual object is a 2-D or a 3-D model specified in a 3-D coordinate system.
11. The method of claim 10 , further comprising projecting the second virtual object from the 3-D coordinate system into the first pixel data.
12. The method of claim 1 , wherein the second pixel location in the second 2-D pixel data is determined using an optical flow.
13. The method of claim 1 , wherein the multi-view interactive digital media representation of the vehicle includes a three dimensional representation of one or more components of the vehicle.
14. The method of claim 13 , wherein a size of the vehicle in the multi-view interactive digital media representation of the vehicle is configurable to be changed using a digital zoom of the camera.
15. The method of claim 13 , wherein the multi-view interactive digital media representation of the vehicle includes a three dimensional representation of a damaged component of the vehicle.
16. The method of claim 1 , further comprising
determining a fourth pixel location in the first 2-D pixel data of a second tracking point and
based upon the first sensor data, the second sensor data, the first 2-D pixel data and the second 2-D pixel data, determining a fifth pixel location of the second tracking point in the second 2-D pixel data using one of the spatial intensity information or the optical flows derived from the second 2-D pixel data.
17. The method of claim 16 , further comprising
determining a first distance in pixel space between the first pixel location of the first tracking point and the fourth location of the second tracking point;
determining a second distance between the second pixel location of the first tracking point and the fifth pixel location of the second tracking point in the pixel space and
based upon the first distance and the second distance, scaling the second virtual object during the rendering into the second pixel data used to generate the third synthetic images.
18. The method of claim 1 , further comprising receiving a selection of the second virtual object from among a plurality of virtual objects.
19. The method of claim 1 , further comprising generating the multi-view interactive digital media representation of the vehicle using the second live image data.
20. The method of claim 1 , further comprising generating the multi-view interactive digital media representation of the vehicle using the third synthetic images wherein the rendering of the second virtual object is included in the multi-view interactive digital media representation of the vehicle.
US16/186,994
2016-12-09
2018-11-12
Live augmented reality using tracking
Active
2036-12-12
US10713851B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US16/186,994
US10713851B2
( en )
2016-12-09
2018-11-12
Live augmented reality using tracking
Applications Claiming Priority (2)
Application Number
Priority Date
Filing Date
Title
US15/374,910
US10210662B2
( en )
2016-12-09
2016-12-09
Live augmented reality using tracking
US16/186,994
US10713851B2
( en )
2016-12-09
2018-11-12
Live augmented reality using tracking
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US15/374,910
Continuation
US10210662B2
( en )
2016-12-09
2016-12-09
Live augmented reality using tracking
Publications (2)
Publication Number
Publication Date
US20190096137A1
US20190096137A1 ( en )
2019-03-28
US10713851B2
true
US10713851B2 ( en )
2020-07-14
Family
ID=62490150
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US15/374,910
Active
2037-01-04
US10210662B2
( en )
2016-12-09
2016-12-09
Live augmented reality using tracking
US16/186,994
Active
2036-12-12
US10713851B2
( en )
2016-12-09
2018-11-12
Live augmented reality using tracking
Family Applications Before (1)
Application Number
Title
Priority Date
Filing Date
US15/374,910
Active
2037-01-04
US10210662B2
( en )
2016-12-09
2016-12-09
Live augmented reality using tracking
Country Status (1)
Country
Link
US
( 2 )
US10210662B2
( en )
Cited By (1)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
EP4250241A1
( en )
2022-03-21
2023-09-27
TeamViewer Germany GmbH
Method for generating an augmented image
Families Citing this family (129)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
KR20170019366A
( en )
2014-05-16
2017-02-21
ëë²ì í¸ í í¬ëë¡ì§ì¤, ì¸í¬.
Modular formed nodes for vehicle chassis and their methods of use
JP6820843B2
( en )
2014-07-02
2021-01-27
ãã¤ãã¼ã¸ã§ã³ã ãã¯ããã¸ã¼ãºï¼ ã¤ã³ã³ã¼ãã¬ã¤ãããＤï½ï½ï½ ï½ï½ï½ ï½ï½ ï¼´ï½ ï½ï½ï½ï½ï½ï½ï½ï½ï½ ï½ï¼ Ｉï½ï½ï¼
Systems and methods for manufacturing fittings
US10509469B2
( en )
2016-04-21
2019-12-17
Finch Technologies Ltd.
Devices for controlling computers based on motions and positions of hands
US10173255B2
( en )
2016-06-09
2019-01-08
Divergent Technologies, Inc.
Systems and methods for arc and node design and manufacture
CN109964245A
( en )
*
2016-12-06
2019-07-02
æ·±å³å¸å¤§çåæ°ç§ææéå ¬å¸
System and method for correcting wide-angle images
US10210662B2
( en )
2016-12-09
2019-02-19
Fyusion, Inc.
Live augmented reality using tracking
US9996945B1
( en )
2016-12-12
2018-06-12
Fyusion, Inc.
Live augmented reality guides
US10759090B2
( en )
2017-02-10
2020-09-01
Divergent Technologies, Inc.
Methods for producing panels using 3D-printed tooling shells
US11155005B2
( en )
2017-02-10
2021-10-26
Divergent Technologies, Inc.
3D-printed tooling and methods for producing same
US12251884B2
( en )
2017-04-28
2025-03-18
Divergent Technologies, Inc.
Support structures in additive manufacturing
US10705113B2
( en )
2017-04-28
2020-07-07
Finch Technologies Ltd.
Calibration of inertial measurement units attached to arms of a user to generate inputs for computer systems
US10898968B2
( en )
2017-04-28
2021-01-26
Divergent Technologies, Inc.
Scatter reduction in additive manufacturing
US10540006B2
( en )
2017-05-16
2020-01-21
Finch Technologies Ltd.
Tracking torso orientation to generate inputs for computer systems
US10379613B2
( en )
2017-05-16
2019-08-13
Finch Technologies Ltd.
Tracking arm movements to generate inputs for computer systems
US10703419B2
( en )
2017-05-19
2020-07-07
Divergent Technologies, Inc.
Apparatus and methods for joining panels
US11358337B2
( en )
2017-05-24
2022-06-14
Divergent Technologies, Inc.
Robotic assembly of transport structures using on-site additive manufacturing
US10009640B1
( en )
2017-05-31
2018-06-26
Verizon Patent And Licensing Inc.
Methods and systems for using 2D captured imagery of a scene to provide virtual reality content
US11123973B2
( en )
2017-06-07
2021-09-21
Divergent Technologies, Inc.
Interconnected deflectable panel and node
US10919230B2
( en )
2017-06-09
2021-02-16
Divergent Technologies, Inc.
Node with co-printed interconnect and methods for producing same
US10781846B2
( en )
2017-06-19
2020-09-22
Divergent Technologies, Inc.
3-D-printed components including fasteners and methods for producing same
US10994876B2
( en )
2017-06-30
2021-05-04
Divergent Technologies, Inc.
Automated wrapping of components in transport structures
US11022375B2
( en )
2017-07-06
2021-06-01
Divergent Technologies, Inc.
Apparatus and methods for additively manufacturing microtube heat exchangers
US10895315B2
( en )
2017-07-07
2021-01-19
Divergent Technologies, Inc.
Systems and methods for implementing node to node connections in mechanized assemblies
US10940609B2
( en )
2017-07-25
2021-03-09
Divergent Technologies, Inc.
Methods and apparatus for additively manufactured endoskeleton-based transport structures
US10751800B2
( en )
2017-07-25
2020-08-25
Divergent Technologies, Inc.
Methods and apparatus for additively manufactured exoskeleton-based transport structures
US10605285B2
( en )
2017-08-08
2020-03-31
Divergent Technologies, Inc.
Systems and methods for joining node and tube structures
US10357959B2
( en )
2017-08-15
2019-07-23
Divergent Technologies, Inc.
Methods and apparatus for additively manufactured identification features
US11306751B2
( en )
2017-08-31
2022-04-19
Divergent Technologies, Inc.
Apparatus and methods for connecting tubes in transport structures
US10960611B2
( en )
2017-09-06
2021-03-30
Divergent Technologies, Inc.
Methods and apparatuses for universal interface between parts in transport structures
US11292058B2
( en )
2017-09-12
2022-04-05
Divergent Technologies, Inc.
Apparatus and methods for optimization of powder removal features in additively manufactured components
US10668816B2
( en )
2017-10-11
2020-06-02
Divergent Technologies, Inc.
Solar extended range electric vehicle with panel deployment and emitter tracking
US10814564B2
( en )
2017-10-11
2020-10-27
Divergent Technologies, Inc.
Composite material inlay in additively manufactured structures
EP3474230B1
( en )
*
2017-10-18
2020-07-22
Tata Consultancy Services Limited
Systems and methods for edge points based monocular visual slam
US11786971B2
( en )
2017-11-10
2023-10-17
Divergent Technologies, Inc.
Structures and methods for high volume production of complex structures using interface nodes
US10926599B2
( en )
2017-12-01
2021-02-23
Divergent Technologies, Inc.
Suspension systems using hydraulic dampers
US11110514B2
( en )
2017-12-14
2021-09-07
Divergent Technologies, Inc.
Apparatus and methods for connecting nodes to tubes in transport structures
CN111684795B
( en )
*
2017-12-15
2022-08-12
Pcmsæ§è¡å ¬å¸
Method for using view paths in 360° video navigation
US10521011B2
( en )
*
2017-12-19
2019-12-31
Finch Technologies Ltd.
Calibration of inertial measurement units attached to arms of a user and to a head mounted device
US11085473B2
( en )
2017-12-22
2021-08-10
Divergent Technologies, Inc.
Methods and apparatus for forming node to panel joints
US11534828B2
( en )
2017-12-27
2022-12-27
Divergent Technologies, Inc.
Assembling structures comprising 3D printed components and standardized components utilizing adhesive circuits
US10509464B2
( en )
2018-01-08
2019-12-17
Finch Technologies Ltd.
Tracking torso leaning to generate inputs for computer systems
US11016116B2
( en )
2018-01-11
2021-05-25
Finch Technologies Ltd.
Correction of accumulated errors in inertial measurement units attached to a user
US11420262B2
( en )
2018-01-31
2022-08-23
Divergent Technologies, Inc.
Systems and methods for co-casting of additively manufactured interface nodes
US10751934B2
( en )
2018-02-01
2020-08-25
Divergent Technologies, Inc.
Apparatus and methods for additive manufacturing with variable extruder profiles
US11224943B2
( en )
2018-03-07
2022-01-18
Divergent Technologies, Inc.
Variable beam geometry laser-based powder bed fusion
US11267236B2
( en )
2018-03-16
2022-03-08
Divergent Technologies, Inc.
Single shear joint for node-to-node connections
US11872689B2
( en )
2018-03-19
2024-01-16
Divergent Technologies, Inc.
End effector features for additively manufactured components
US11254381B2
( en )
2018-03-19
2022-02-22
Divergent Technologies, Inc.
Manufacturing cell based vehicle manufacturing system and method
US11408216B2
( en )
2018-03-20
2022-08-09
Divergent Technologies, Inc.
Systems and methods for co-printed or concurrently assembled hinge structures
US11613078B2
( en )
2018-04-20
2023-03-28
Divergent Technologies, Inc.
Apparatus and methods for additively manufacturing adhesive inlet and outlet ports
US11214317B2
( en )
2018-04-24
2022-01-04
Divergent Technologies, Inc.
Systems and methods for joining nodes and other structures
US11020800B2
( en )
2018-05-01
2021-06-01
Divergent Technologies, Inc.
Apparatus and methods for sealing powder holes in additively manufactured parts
US10682821B2
( en )
2018-05-01
2020-06-16
Divergent Technologies, Inc.
Flexible tooling system and method for manufacturing of composite structures
US11474593B2
( en )
2018-05-07
2022-10-18
Finch Technologies Ltd.
Tracking user movements to control a skeleton model in a computer system
US10416755B1
( en )
2018-06-01
2019-09-17
Finch Technologies Ltd.
Motion predictions of overlapping kinematic chains of a skeleton model used to control a computer system
US11389816B2
( en )
2018-05-09
2022-07-19
Divergent Technologies, Inc.
Multi-circuit single port design in additively manufactured node
US10691104B2
( en )
2018-05-16
2020-06-23
Divergent Technologies, Inc.
Additively manufacturing structures for increased spray forming resolution or increased fatigue life
US11590727B2
( en )
2018-05-21
2023-02-28
Divergent Technologies, Inc.
Custom additively manufactured core structures
US11441586B2
( en )
2018-05-25
2022-09-13
Divergent Technologies, Inc.
Apparatus for injecting fluids in node based connections
US11035511B2
( en )
2018-06-05
2021-06-15
Divergent Technologies, Inc.
Quick-change end effector
US11292056B2
( en )
2018-07-06
2022-04-05
Divergent Technologies, Inc.
Cold-spray nozzle
US11009941B2
( en )
2018-07-25
2021-05-18
Finch Technologies Ltd.
Calibration of measurement units in alignment with a skeleton model to control a computer system
US11269311B2
( en )
2018-07-26
2022-03-08
Divergent Technologies, Inc.
Spray forming structural joints
US10836120B2
( en )
2018-08-27
2020-11-17
Divergent Technologies, Inc .
Hybrid composite structures with integrated 3-D printed elements
US11433557B2
( en )
2018-08-28
2022-09-06
Divergent Technologies, Inc.
Buffer block apparatuses and supporting apparatuses
US11826953B2
( en )
2018-09-12
2023-11-28
Divergent Technologies, Inc.
Surrogate supports in additive manufacturing
US11072371B2
( en )
2018-10-05
2021-07-27
Divergent Technologies, Inc.
Apparatus and methods for additively manufactured structures with augmented energy absorption properties
US11260582B2
( en )
2018-10-16
2022-03-01
Divergent Technologies, Inc.
Methods and apparatus for manufacturing optimized panels and other composite structures
US12115583B2
( en )
2018-11-08
2024-10-15
Divergent Technologies, Inc.
Systems and methods for adhesive-based part retention features in additively manufactured structures
US12194536B2
( en )
2018-11-13
2025-01-14
Divergent Technologies, Inc.
3-D printer with manifolds for gas exchange
US11504912B2
( en )
2018-11-20
2022-11-22
Divergent Technologies, Inc.
Selective end effector modular attachment device
USD911222S1
( en )
2018-11-21
2021-02-23
Divergent Technologies, Inc.
Vehicle and/or replica
US10663110B1
( en )
2018-12-17
2020-05-26
Divergent Technologies, Inc.
Metrology apparatus to facilitate capture of metrology data
US11529741B2
( en )
2018-12-17
2022-12-20
Divergent Technologies, Inc.
System and method for positioning one or more robotic apparatuses
US11449021B2
( en )
2018-12-17
2022-09-20
Divergent Technologies, Inc.
Systems and methods for high accuracy fixtureless assembly
US11885000B2
( en )
2018-12-21
2024-01-30
Divergent Technologies, Inc.
In situ thermal treatment for PBF systems
US12378643B2
( en )
2019-01-18
2025-08-05
Divergent Technologies, Inc.
Aluminum alloys
IT201900005806A1
( en )
*
2019-04-15
2020-10-15
Pikkart Srl
METHOD TO REALIZE AUGMENTED REALITY
US11203240B2
( en )
2019-04-19
2021-12-21
Divergent Technologies, Inc.
Wishbone style control arm assemblies and methods for producing same
US12314031B1
( en )
2019-06-27
2025-05-27
Divergent Technologies, Inc.
Incorporating complex geometric features in additively manufactured parts
US10809797B1
( en )
2019-08-07
2020-10-20
Finch Technologies Ltd.
Calibration of multiple sensor modules related to an orientation of a user of the sensor modules
CN110691175B
( en )
*
2019-08-19
2021-08-24
æ·±å³å¸å±å¾æ°ç ç§ææéå ¬å¸
Video processing method and device for simulating motion tracking of camera in studio
JP7379028B2
( en )
*
2019-09-06
2023-11-14
ãã¤ãã³æ ªå¼ä¼ç¤¾
Electronic devices, control methods for electronic devices, programs and storage media
CN110689570B
( en )
*
2019-09-29
2020-11-27
å京达佳äºèä¿¡æ¯ææ¯æéå ¬å¸
Live virtual image broadcasting method and device, electronic equipment and storage medium
US12280554B2
( en )
2019-11-21
2025-04-22
Divergent Technologies, Inc.
Fixtureless robotic assembly
US11912339B2
( en )
2020-01-10
2024-02-27
Divergent Technologies, Inc.
3-D printed chassis structure with self-supporting ribs
US11590703B2
( en )
2020-01-24
2023-02-28
Divergent Technologies, Inc.
Infrared radiation sensing and beam control in electron beam additive manufacturing
US12194674B2
( en )
2020-02-14
2025-01-14
Divergent Technologies, Inc.
Multi-material powder bed fusion 3-D printer
US11884025B2
( en )
2020-02-14
2024-01-30
Divergent Technologies, Inc.
Three-dimensional printer and methods for assembling parts via integration of additive and conventional manufacturing operations
US11479015B2
( en )
2020-02-14
2022-10-25
Divergent Technologies, Inc.
Custom formed panels for transport structures and methods for assembling same
US12203397B2
( en )
2020-02-18
2025-01-21
Divergent Technologies, Inc.
Impact energy absorber with integrated engine exhaust noise muffler
US11535322B2
( en )
2020-02-25
2022-12-27
Divergent Technologies, Inc.
Omni-positional adhesion device
US11421577B2
( en )
2020-02-25
2022-08-23
Divergent Technologies, Inc.
Exhaust headers with integrated heat shielding and thermal syphoning
CN111380511A
( en )
*
2020-02-26
2020-07-07
æ¡æçµåç§æ大å¦
An inertial trajectory tracking system
US12337541B2
( en )
2020-02-27
2025-06-24
Divergent Technologies, Inc.
Powder bed fusion additive manufacturing system with desiccant positioned within hopper and ultrasonic transducer
US11413686B2
( en )
2020-03-06
2022-08-16
Divergent Technologies, Inc.
Methods and apparatuses for sealing mechanisms for realizing adhesive connections with additively manufactured components
US11043038B1
( en )
2020-03-16
2021-06-22
Hong Kong Applied Science and Technology Research Institute Company Limited
Apparatus and method of three-dimensional interaction for augmented reality remote assistance
KR20230035571A
( en )
2020-06-10
2023-03-14
ëë²ì í¸ í í¬ëë¡ì§ì¤, ì¸í¬.
Adaptive production system
US11850804B2
( en )
2020-07-28
2023-12-26
Divergent Technologies, Inc.
Radiation-enabled retention features for fixtureless assembly of node-based structures
US11806941B2
( en )
2020-08-21
2023-11-07
Divergent Technologies, Inc.
Mechanical part retention features for additively manufactured structures
CN116457139A
( en )
2020-09-08
2023-07-18
æ´å¼æ ¹ç¹ææ¯æéå ¬å¸
Assembly sequence generation
CN116669885A
( en )
2020-09-22
2023-08-29
æ´å¼æ ¹ç¹ææ¯æéå ¬å¸
Method and apparatus for ball milling to produce powders for additive manufacturing
US12220819B2
( en )
2020-10-21
2025-02-11
Divergent Technologies, Inc.
3-D printed metrology feature geometry and detection
JP6959422B1
( en )
*
2020-10-22
2021-11-02
ã¤ãã¼æ ªå¼ä¼ç¤¾
Transmission program, terminal device and transmission method
CN112102395B
( en )
*
2020-11-09
2022-05-20
广ä¸ç§å¯è¾¾æºè½æºå¨äººæéå ¬å¸
Autonomous inspection method based on machine vision
US12311612B2
( en )
2020-12-18
2025-05-27
Divergent Technologies, Inc.
Direct inject joint architecture enabled by quick cure adhesive
US12083596B2
( en )
2020-12-21
2024-09-10
Divergent Technologies, Inc.
Thermal elements for disassembly of node-based adhesively bonded structures
US12226824B2
( en )
2020-12-22
2025-02-18
Divergent Technologies, Inc.
Three dimensional printer with configurable build plate for rapid powder removal
US11872626B2
( en )
2020-12-24
2024-01-16
Divergent Technologies, Inc.
Systems and methods for floating pin joint design
US11947335B2
( en )
2020-12-30
2024-04-02
Divergent Technologies, Inc.
Multi-component structure optimization for combining 3-D printed and commercially available parts
US11928966B2
( en )
2021-01-13
2024-03-12
Divergent Technologies, Inc.
Virtual railroad
US12459377B2
( en )
2021-01-19
2025-11-04
Divergent Technologies, Inc.
Energy unit cells for primary vehicle structure
US12249812B2
( en )
2021-01-19
2025-03-11
Divergent Technologies, Inc.
Bus bars for printed structural electric battery modules
US20220288850A1
( en )
2021-03-09
2022-09-15
Divergent Technologies, Inc.
Rotational additive manufacturing systems and methods
US12090551B2
( en )
2021-04-23
2024-09-17
Divergent Technologies, Inc.
Removal of supports, and other materials from surface, and within hollow 3D printed parts
US12138772B2
( en )
2021-04-30
2024-11-12
Divergent Technologies, Inc.
Mobile parts table
CN113615169B
( en )
*
2021-05-17
2022-06-10
é¦æ¸¯åºç¨ç§æç 究é¢æéå ¬å¸
Apparatus and method for augmenting a real user manual
US11169832B1
( en )
*
2021-05-17
2021-11-09
Hong Kong Applied Science and Technology Research Institute Company Limited
Apparatus and method for augmented reality user manual
CN117769486A
( en )
2021-05-24
2024-03-26
æ´å¼æ ¹ç¹ææ¯æéå ¬å¸
Robotic gripper equipment
WO2023278878A1
( en )
2021-07-01
2023-01-05
Divergent Technologies, Inc.
Al-mg-si based near-eutectic alloy composition for high strength and stiffness applications
US12583033B2
( en )
2021-08-13
2026-03-24
Divergent Technologies, Inc.
Integrating additively-manufactured components
US11865617B2
( en )
2021-08-25
2024-01-09
Divergent Technologies, Inc.
Methods and apparatuses for wide-spectrum consumption of output of atomization processes across multi-process and multi-scale additive manufacturing modalities
US12351238B2
( en )
2021-11-02
2025-07-08
Divergent Technologies, Inc.
Mo