ABSTRACT
Abstract
Systems, devices, media, and methods are presented for playing audio sounds, such as music, on a portable electronic device using a digital color image of a note matrix on a map. A computer vision engine, in an example implementation, includes a mapping module, a color detection module, and a music playback module. The camera captures a color image of the map, including a marker and a note matrix. Based on the color image, the computer vision engine detects a token color value associated with each field. Each token color value is associated with a sound sample from a specific musical instrument. A global state map is stored in memory, including the token color value and location of each field in the note matrix. The music playback module, for each column, in order, plays the notes associated with one or more the rows, using the corresponding sound sample, according to the global state map.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a Continuation of U.S. application Ser. No. 17/397,145 filed on Aug. 9, 2021, which is a Continuation of U.S. application Ser. No. 16/724,209 filed on Dec. 21, 2019, now U.S. Pat. No. 11,087,728, both of which are incorporated fully herein by reference.
TECHNICAL FIELD
Examples set forth in the present disclosure relate to portable electronic devices, including wearable devices such as eyewear. More particularly, but not by way of limitation, the present disclosure describes computer vision and mapping systems and methods for audio applications, such as playing musical sounds.
BACKGROUND
Many types of computers and electronic devices available today, including mobile devices (e.g., smartphones, tablets, and laptops), and wearable devices (e.g., smartglasses, digital eyewear, headwear, headgear, and head-mounted displays), include digital cameras, display screens, and interfaces through which a user can interact with displayed content.
BRIEF DESCRIPTION OF THE DRAWINGS
Features of the various implementations disclosed will be readily understood from the following detailed description, in which reference is made to the appending drawing figures. A reference numeral is used with each element in the description and throughout the several views of the drawing. When a plurality of similar elements is present, a single reference numeral may be assigned to like elements, with an added lower-case letter referring to a specific element.
The various elements shown in the figures are not drawn to scale unless otherwise indicated. The dimensions of the various elements may be enlarged or reduced in the interest of clarity. The several figures depict one or more implementations and are presented by way of example only and should not be construed as limiting. Included in the drawing are the following figures:
FIG. 1 A is a side view (right) of an example hardware configuration of an eyewear device with a primary input surface, which may be utilized in a selective control and transition system;
FIG. 1 B is a top, partly sectional view of a right chunk of the eyewear device of FIG. 1 A depicting a right visible-light camera, and a circuit board;
FIG. 1 C is a side view (left) of an example hardware configuration of the eyewear device of FIG. 1 A , which shows a left visible-light camera;
FIG. 1 D is a top, partly sectional view of a left chunk of the eyewear device of FIG. 1 C depicting the left visible-light camera, and a circuit board;
FIGS. 2 A and 2 B are rear views of example hardware configurations of an eyewear device utilized in the selective control and transition system;
FIG. 3 is a diagrammatic depiction of a three-dimensional scene, a left raw image captured by a left visible-light camera, and a right raw image captured by a right visible-light camera;
FIG. 4 is a functional block diagram of an example system including a mobile device, a wearable device (e.g., an eyewear device), a computer vision engine, and a server system connected via various networks;
FIG. 5 is a diagrammatic representation of an example hardware configuration for a mobile device including the computer vision engine of FIG. 4 ;
FIG. 6 is an illustration of an example map, including a marker, a note matrix, a percussion matrix, and a tempo scale, suitable for use with the computer vision engine of FIG. 4 ; and
FIG. 7 is an illustration of a set of example tokens ready to be cut and placed on the example map of FIG. 6 .
DETAILED DESCRIPTION
Various implementations and details are described with reference to an example: a computer vision engine for playing a musical composition on a portable electronic device using a digital color image of a note matrix on a map. A mobile device, such as a smartphone, includes a processor, a memory, a camera, a loudspeaker, and a display screen. The computer vision engine, in an example implementation, includes a mapping module, a color detection module, and a music playback module. The camera captures a color image of the map, including a marker and a note matrix (in rows and columns, similar to standard musical notation, with a note in each field). From the color image, the computer vision engine detects a token color value associated with each field. Each token color value is associated with a sound sample from a specific musical instrument. A global state map is stored in memory, including the token color value and location of each field in the note matrix. The music playback module, for each column, in order, plays the notes associated with one or more the rows, using the corresponding sound sample, according to the global state map.
The following detailed description includes systems, methods, techniques, instruction sequences, and computing machine program products illustrative of examples set forth in the disclosure. Numerous details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its relevant teachings. Those skilled in the relevant art, however, may understand how to apply the relevant teachings without such details. Aspects of the disclosed subject matter are not limited to the specific devices, systems, and method described because the relevant teachings can be applied or practice in a variety of ways. The terminology and nomenclature used herein is for the purpose of describing particular aspects only and is not intended to be limiting. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.
The term âcoupledâ or âconnectedâ as used herein refers to any logical, optical, physical, or electrical connection, including a link or the like by which the electrical or magnetic signals produced or supplied by one system element are imparted to another coupled or connected system element. Unless described otherwise, coupled or connected elements or devices are not necessarily directly connected to one another and may be separated by intermediate components, elements, or communication media, one or more of which may modify, manipulate, or carry the electrical signals. The term âonâ means directly supported by an element or indirectly supported by the element through another element integrated into or supported by the element.
The orientations of the eyewear device, the handheld device, associated components and any other complete devices incorporating a camera and/or an inertial measurement unit such as shown in any of the drawings, are given by way of example only, for illustration and discussion purposes. In operation, the eyewear device may be oriented in any other direction suitable to the particular application of the eyewear device; for example, up, down, sideways, or any other orientation. Also, to the extent used herein, any directional term, such as front, rear, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, side, horizontal, vertical, and diagonal are used by way of example only, and are not limiting as to the direction or orientation of any camera and/or inertial measurement unit as constructed as otherwise described herein.
Additional objects, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings or may be learned by production or operation of the examples. The objects and advantages of the present subject matter may be realized and attained by means of the methodologies, instrumentalities and combinations particularly pointed out in the appended claims.
Reference now is made in detail to the examples illustrated in the accompanying drawings and discussed below.
FIG. 1 A is a side view (right) of an example hardware configuration of an eyewear device 100 which includes a touch-sensitive input device or touchpad 181 . As shown, the touchpad 181 may have a boundary that is subtle and not easily seen; alternatively, the boundary may be plainly visible and/or include a raised or otherwise tactile edge that provides feedback to the user about the location and boundary of the touchpad 181 . In other implementations, the eyewear 100 may include a touchpad on the left side.
The surface of the touchpad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touches) for use with a GUI displayed by the eyewear, on a display, to allow the user to navigate through and select menu options in an intuitive manner, which enhances and simplifies the user experience.
Detection of finger inputs on the touchpad 181 can enable several functions. For example, touching anywhere on the touchpad 181 may cause the GUI to display and/or highlight an item on the screen of the image display, which may be projected onto at least one of the optical assemblies
180 A, 180 B. Double tapping on the touchpad 181 may select an item or icon. Sliding or swiping a finger in a particular direction (e.g., from front to back, back to front, up to down, or down to) may cause the items or icons to slide or scroll in a particular direction; for example, to move to a next item, icon, video, image, page, or slide. Sliding the finger in another direction may slide or scroll in the opposite direction; for example, to move to a previous item, icon, video, image, page, or slide. The touchpad 181 can be virtually anywhere on the eyewear device 100 .
In one example, when the identified finger gesture is a single tap on the touchpad 181 , this initiates selection or pressing of a graphical user interface element in the image presented on the image display of the optical assembly
180 A, 180 B. An adjustment to the image presented on the image display of the optical assembly
180 A, 180 B based on the identified finger gesture can be a primary action which selects or submits the graphical user interface element on the image display of the optical assembly
180 A, 180 B for further display or execution.
As shown, the eyewear 100 includes a right visible- light camera 114 B. As further described herein, two cameras
114 A, 114 B capture image information for a scene from two separate viewpoints. The two captured images may be used to project a three-dimensional display onto a screen for viewing with 3D glasses.
The eyewear device 100 includes a right optical assembly 180 B with an image display to present images, such as depth images. As shown in FIGS. 1 A and 1 B , the eyewear device 100 includes the right visible- light camera 114 B. The eyewear device 100 can include multiple visible- light cameras
114 A, 114 B that form a pas
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a Continuation of U.S. application Ser. No. 17/397,145 filed on Aug. 9, 2021, which is a Continuation of U.S. application Ser. No. 16/724,209 filed on Dec. 21, 2019, now U.S. Pat. No. 11,087,728, both of which are incorporated fully herein by reference.
TECHNICAL FIELD
Examples set forth in the present disclosure relate to portable electronic devices, including wearable devices such as eyewear. More particularly, but not by way of limitation, the present disclosure describes computer vision and mapping systems and methods for audio applications, such as playing musical sounds.
BACKGROUND
Many types of computers and electronic devices available today, including mobile devices (e.g., smartphones, tablets, and laptops), and wearable devices (e.g., smartglasses, digital eyewear, headwear, headgear, and head-mounted displays), include digital cameras, display screens, and interfaces through which a user can interact with displayed content.
BRIEF DESCRIPTION OF THE DRAWINGS
Features of the various implementations disclosed will be readily understood from the following detailed description, in which reference is made to the appending drawing figures. A reference numeral is used with each element in the description and throughout the several views of the drawing. When a plurality of similar elements is present, a single reference numeral may be assigned to like elements, with an added lower-case letter referring to a specific element.
The various elements shown in the figures are not drawn to scale unless otherwise indicated. The dimensions of the various elements may be enlarged or reduced in the interest of clarity. The several figures depict one or more implementations and are presented by way of example only and should not be construed as limiting. Included in the drawing are the following figures:
FIG. 1 A is a side view (right) of an example hardware configuration of an eyewear device with a primary input surface, which may be utilized in a selective control and transition system;
FIG. 1 B is a top, partly sectional view of a right chunk of the eyewear device of FIG. 1 A depicting a right visible-light camera, and a circuit board;
FIG. 1 C is a side view (left) of an example hardware configuration of the eyewear device of FIG. 1 A , which shows a left visible-light camera;
FIG. 1 D is a top, partly sectional view of a left chunk of the eyewear device of FIG. 1 C depicting the left visible-light camera, and a circuit board;
FIGS. 2 A and 2 B are rear views of example hardware configurations of an eyewear device utilized in the selective control and transition system;
FIG. 3 is a diagrammatic depiction of a three-dimensional scene, a left raw image captured by a left visible-light camera, and a right raw image captured by a right visible-light camera;
FIG. 4 is a functional block diagram of an example system including a mobile device, a wearable device (e.g., an eyewear device), a computer vision engine, and a server system connected via various networks;
FIG. 5 is a diagrammatic representation of an example hardware configuration for a mobile device including the computer vision engine of FIG. 4 ;
FIG. 6 is an illustration of an example map, including a marker, a note matrix, a percussion matrix, and a tempo scale, suitable for use with the computer vision engine of FIG. 4 ; and
FIG. 7 is an illustration of a set of example tokens ready to be cut and placed on the example map of FIG. 6 .
DETAILED DESCRIPTION
Various implementations and details are described with reference to an example: a computer vision engine for playing a musical composition on a portable electronic device using a digital color image of a note matrix on a map. A mobile device, such as a smartphone, includes a processor, a memory, a camera, a loudspeaker, and a display screen. The computer vision engine, in an example implementation, includes a mapping module, a color detection module, and a music playback module. The camera captures a color image of the map, including a marker and a note matrix (in rows and columns, similar to standard musical notation, with a note in each field). From the color image, the computer vision engine detects a token color value associated with each field. Each token color value is associated with a sound sample from a specific musical instrument. A global state map is stored in memory, including the token color value and location of each field in the note matrix. The music playback module, for each column, in order, plays the notes associated with one or more the rows, using the corresponding sound sample, according to the global state map.
The following detailed description includes systems, methods, techniques, instruction sequences, and computing machine program products illustrative of examples set forth in the disclosure. Numerous details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its relevant teachings. Those skilled in the relevant art, however, may understand how to apply the relevant teachings without such details. Aspects of the disclosed subject matter are not limited to the specific devices, systems, and method described because the relevant teachings can be applied or practice in a variety of ways. The terminology and nomenclature used herein is for the purpose of describing particular aspects only and is not intended to be limiting. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.
The term âcoupledâ or âconnectedâ as used herein refers to any logical, optical, physical, or electrical connection, including a link or the like by which the electrical or magnetic signals produced or supplied by one system element are imparted to another coupled or connected system element. Unless described otherwise, coupled or connected elements or devices are not necessarily directly connected to one another and may be separated by intermediate components, elements, or communication media, one or more of which may modify, manipulate, or carry the electrical signals. The term âonâ means directly supported by an element or indirectly supported by the element through another element integrated into or supported by the element.
The orientations of the eyewear device, the handheld device, associated components and any other complete devices incorporating a camera and/or an inertial measurement unit such as shown in any of the drawings, are given by way of example only, for illustration and discussion purposes. In operation, the eyewear device may be oriented in any other direction suitable to the particular application of the eyewear device; for example, up, down, sideways, or any other orientation. Also, to the extent used herein, any directional term, such as front, rear, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, side, horizontal, vertical, and diagonal are used by way of example only, and are not limiting as to the direction or orientation of any camera and/or inertial measurement unit as constructed as otherwise described herein.
Additional objects, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings or may be learned by production or operation of the examples. The objects and advantages of the present subject matter may be realized and attained by means of the methodologies, instrumentalities and combinations particularly pointed out in the appended claims.
Reference now is made in detail to the examples illustrated in the accompanying drawings and discussed below.
FIG. 1 A is a side view (right) of an example hardware configuration of an eyewear device 100 which includes a touch-sensitive input device or touchpad 181 . As shown, the touchpad 181 may have a boundary that is subtle and not easily seen; alternatively, the boundary may be plainly visible and/or include a raised or otherwise tactile edge that provides feedback to the user about the location and boundary of the touchpad 181 . In other implementations, the eyewear 100 may include a touchpad on the left side.
The surface of the touchpad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touches) for use with a GUI displayed by the eyewear, on a display, to allow the user to navigate through and select menu options in an intuitive manner, which enhances and simplifies the user experience.
Detection of finger inputs on the touchpad 181 can enable several functions. For example, touching anywhere on the touchpad 181 may cause the GUI to display and/or highlight an item on the screen of the image display, which may be projected onto at least one of the optical assemblies
180 A, 180 B. Double tapping on the touchpad 181 may select an item or icon. Sliding or swiping a finger in a particular direction (e.g., from front to back, back to front, up to down, or down to) may cause the items or icons to slide or scroll in a particular direction; for example, to move to a next item, icon, video, image, page, or slide. Sliding the finger in another direction may slide or scroll in the opposite direction; for example, to move to a previous item, icon, video, image, page, or slide. The touchpad 181 can be virtually anywhere on the eyewear device 100 .
In one example, when the identified finger gesture is a single tap on the touchpad 181 , this initiates selection or pressing of a graphical user interface element in the image presented on the image display of the optical assembly
180 A, 180 B. An adjustment to the image presented on the image display of the optical assembly
180 A, 180 B based on the identified finger gesture can be a primary action which selects or submits the graphical user interface element on the image display of the optical assembly
180 A, 180 B for further display or execution.
As shown, the eyewear 100 includes a right visible- light camera 114 B. As further described herein, two cameras
114 A, 114 B capture image information for a scene from two separate viewpoints. The two captured images may be used to project a three-dimensional display onto a screen for viewing with 3D glasses.
The eyewear device 100 includes a right optical assembly 180 B with an image display to present images, such as depth images. As shown in FIGS. 1 A and 1 B , the eyewear device 100 includes the right visible- light camera 114 B. The eyewear device 100 can include multiple visible- light cameras
114 A, 114 B that form a passive type of three-dimensional camera, such as stereo camera, of which the right visible- light camera 114 B is located on a right chunk 110 B. As shown in FIGS. 1 C-D , the eyewear device 100 also includes a left visible- light camera 114 A.
Left and right visible- light cameras
114 A, 114 B are sensitive to the visible-light range wavelength. Each of the visible- light cameras
114 A, 114 B have a different frontward facing field of view which are overlapping to enable generation of three-dimensional depth images, for example, right visible- light camera 114 B depicts a right field of view 111 B. Generally, a âfield of viewâ is the part of the scene that is visible through the camera at a particular position and orientation in space. The fields of view
111 A and 111 B have an overlapping field of view 813 . Objects or object features outside the field of view
111 A, 111 B when the visible-light camera captures the image are not recorded in a raw image (e.g., photograph or picture). The field of view describes an angle range or extent, which the image sensor of the visible- light camera
114 A, 114 B picks up electromagnetic radiation of a given scene in a captured image of the given scene. Field of view can be expressed as the angular size of the view cone, i.e., an angle of view. The angle of view can be measured horizontally, vertically, or diagonally.
In an example, visible- light cameras
114 A, 114 B have a field of view with an angle of view between 15° to 30°, for example 24°, and have a resolution of 480Ã480 pixels. The âangle of coverageâ describes the angle range that a lens of visible- light cameras
114 A, 114 B or infrared camera 220 (see FIG. 2 A ) can effectively image. Typically, the camera lens produces an image circle that is large enough to cover the film or sensor of the camera completely, possibly including some vignetting toward the edge. If the angle of coverage of the camera lens does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the edge, and the effective angle of view will be limited to the angle of coverage.
Examples of such visible- light cameras
114 A, 114 B include a high-resolution complementary metal-oxide-semiconductor (CMOS) image sensor and a digital VGA camera (video graphics array) capable of resolutions of 640p (e.g., 640Ã480 pixels for a total of 0.3 megapixels), 720p, or 1080p. Other examples of visible- light cameras
114 A, 114 B that can capture high-definition (HD) still images and store them at a resolution of 1642 by 1642 pixels (or greater); and/or record high-definition video at a high frame rate (e.g., thirty to sixty frames per second or more) and store the recording at a resolution of 1216 by 1216 pixels (or greater).
The eyewear device 100 may capture image sensor data from the visible- light cameras
114 A, 114 B along with geolocation data, digitized by an image processor, for storage in a memory. The left and right raw images captured by respective visible- light cameras
114 A, 114 B are in the two-dimensional space domain and comprise a matrix of pixels on a two-dimensional coordinate system that includes an X-axis for horizontal position and a Y-axis for vertical position. Each pixel includes a color attribute value (e.g., a red pixel light value, a green pixel light value, and/or a blue pixel light value); and a position attribute (e.g., an X-axis coordinate and a Y-axis coordinate).
In order to capture stereo images for later display as a three-dimensional projection, the image processor 912 (shown in FIG. 4 ) may be coupled to the visible- light cameras
114 A, 114 B to receive and store the visual image information. A timestamp for each image may be added by the image processor 912 or another processor which controls operation of the visible- light cameras
114 A, 114 B, which act as a stereo camera to simulate human binocular vision. The timestamp on each pair of images allows the images to be displayed together as part of a three-dimensional projection. Three-dimensional projections create an immersive, life-like experience that is desirable in a variety of contexts, including virtual reality (VR) and video gaming.
FIG. 3 is a diagrammatic depiction of a three- dimensional scene 715 , a left raw image 858 A captured by a left visible- light camera 114 A, and a right raw image 858 B captured by a right visible- light camera 114 B. The left field of view 111 A may overlap, as shown, with the right field of view 111 B. The overlapping field of view 813 represents that portion of the image captured by both cameras
114 A, 114 B. The term âoverlappingâ when referring to field of view means the matrix of pixels in the generated raw images overlap by thirty percent (30%) or more. âSubstantially overlappingâ means the matrix of pixels in the generated raw imagesâor in the infrared image of sceneâoverlap by fifty percent (50%) or more. As described herein, the two raw images 858 A, 858 B may be processed to include a timestamp, which allows the images to be displayed together as part of a three-dimensional projection.
For the capture of stereo images, as illustrated in FIG. 3 , a pair of raw red, green, and blue (RGB) images are captured of a real scene 715 at a given moment in timeâa left raw image 858 A captured by the left camera 114 A and right raw image 858 B captured by the right camera 114 B. When the pair of raw images 858 A, 858 B are processed (e.g., by the image processor 912 ), depth images are generated. The generated depth images may be viewed on an optical assembly
180 A, 180 B of an eyewear device, on another display (e.g., the image display 880 on a mobile device 890 ), or on a screen.
The generated depth images are in the three-dimensional space domain and can comprise a matrix of vertices on a three-dimensional location coordinate system that includes an X axis for horizontal position (e.g., length), a Y axis for vertical position (e.g., height), and a Z axis for depth (e.g., distance). Each vertex may include a color attribute (e.g., a red pixel light value, a green pixel light value, and/or a blue pixel light value); a position attribute (e.g., an X location coordinate, a Y location coordinate, and a Z location coordinate); a texture attribute and/or a reflectance attribute. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of color or intensities in a region of vertices of the depth image.
In one example, the system described herein includes the eyewear device 100 , which includes a frame 105 and a left temple 125 A extending from a left lateral side 170 A of the frame 105 and a right temple 125 B extending from a right lateral side 170 B of the frame 105 . The eyewear device 100 may further include at least two visible- light cameras
114 A, 114 B which may have overlapping fields of view. In one example, the eyewear device 100 includes a left visible- light camera 114 A with a left field of view 111 A, as illustrated in FIG. 3 . The left camera 114 A is connected to the frame 105 or the left temple 125 A to capture a left raw image 858 A from the left side of scene 715 . The eyewear device 100 further includes a right visible- light camera 114 B with a right field of view 111 B. The right camera 114 B is connected to the frame 105 or the right temple 125 B to capture a right raw image 858 B from the right side of scene 715 .
FIG. 1 B is a top cross-sectional view of a right chunk 110 B of the eyewear device 100 of FIG. 1 A depicting the right visible- light camera 114 B of the camera system, and a circuit board. FIG. 1 C is a side view (left) of an example hardware configuration of an eyewear device 100 of FIG. 1 A , which shows a left visible- light camera 114 A of the camera system. FIG. 1 D is a top cross-sectional view of a left chunk 110 A of the eyewear device of FIG. 1 C depicting the left visible- light camera 114 A of the three-dimensional camera, and a circuit board. Construction and placement of the left visible- light camera 114 A is substantially similar to the right visible- light camera 114 B, except the connections and coupling are on the left lateral side 170 A. As shown in the example of FIG. 1 B , the eyewear device 100 includes the right visible- light camera 114 B and a circuit board 140 B, which may be a flexible printed circuit board (PCB). The right hinge 226 B connects the right chunk 110 B to a right temple 125 B of the eyewear device 100 . In some examples, components of the right visible- light camera 114 B, the flexible PCB 140 B, or other electrical connectors or contacts may be located on the right temple 125 B or the right hinge 226 B.
The right chunk 110 B includes chunk body 211 and a chunk cap, with the chunk cap omitted in the cross-section of FIG. 1 B . Disposed inside the right chunk 110 B are various interconnected circuit boards, such as PCBs or flexible PCBs, that include controller circuits for right visible- light camera 114 B, microphone(s), low-power wireless circuitry (e.g., for wireless short-range network communication via Bluetoothâ¢), high-speed wireless circuitry (e.g., for wireless local area network communication via WiFi).
The right visible- light camera 114 B is coupled to or disposed on the flexible PCB 140 B and covered by a visible-light camera cover lens, which is aimed through opening(s) formed in the frame 105 . For example, the right rim 107 B of the frame 105 , shown in FIG. 2 A , is connected to the right chunk 110 B and includes the opening(s) for the visible-light camera cover lens. The frame 105 includes a front side configured to face outward and away from the eye of the user. The opening for the visible-light camera cover lens is formed on and through the front or outward-facing side of the frame 105 . In the example, the right visible- light camera 114 B has an outward-facing field of view 111 B (shown in FIG. 3 ) with a line of sight or perspective that is correlated with the right eye of the user of the eyewear device 100 . The visible-light camera cover lens can also be adhered to a front side or outward-facing surface of the right chunk 110 B in which an opening is formed with an outward-facing angle of coverage, but in a different outwardly direction. The coupling can also be indirect via intervening components.
As shown in FIG. 1 B , flexible PCB 140 B is disposed inside the right chunk 110 B and is coupled to one or more other components housed in the right chunk 110 B. Although shown as being formed on the circuit boards of the right chunk 110 B, the right visible- light camera 114 B can be formed on the circuit boards of the left chunk 110 A, the temples
125 A, 125 B, or the frame 105 .
FIGS. 2 A and 2 B are perspective views, from the rear, of example hardware configurations of the eyewear device 100 , including two different types of image displays. The eyewear device 100 is sized and shaped in a form configured for wearing by a user; the form of eyeglasses is shown in the example. The eyewear device 100 can take other forms and may incorporate other types of frameworks; for example, a headgear, a headset, or a helmet.
In the eyeglasses example, eyewear device 100 includes a frame 105 including a left rim 107 A connected to a right rim 107 B via a bridge 106 adapted to be supported by a nose of the user. The left and right rims
107 A, 107 B include respective apertures
175 A, 175 B, which hold a respective optical element
180 A, 180 B, such as a lens and a display device. As used herein, the term âlensâ is meant to include transparent or translucent pieces of glass or plastic having curved and/or flat surfaces that cause light to converge/diverge or that cause little or no convergence or divergence.
Although shown as having two optical elements
180 A, 180 B, the eyewear device 100 can include other arrangements, such as a single optical element (or it may not include any optical element
180 A, 180 B), depending on the application or the intended user of the eyewear device 100 . As further shown, eyewear device 100 includes a left chunk 110 A adjacent the left lateral side 170 A of the frame 105 and a right chunk 110 B adjacent the right lateral side 170 B of the frame 105 . The chunks
110 A, 110 B may be integrated into the frame 105 on the respective sides
170 A, 170 B (as illustrated) or implemented as separate components attached to the frame 105 on the respective sides
170 A, 170 B. Alternatively, the chunks
110 A, 110 B may be integrated into temples (not shown) attached to the frame 105 .
In one example, the image display of optical assembly
180 A, 180 B includes an integrated image display. As shown in FIG. 2 A , each optical assembly
180 A, 180 B includes a suitable display matrix 177 , such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or any other such display. Each optical assembly
180 A, 180 B also includes an optical layer or layers 176 , which can include lenses, optical coatings, prisms, mirrors, waveguides, optical strips, and other optical components in any combination. The optical layers 176 A, 176 B, . . . 176 N (shown as 176 A-N in FIG. 2 A and herein) can include a prism having a suitable size and configuration and including a first surface for receiving light from a display matrix and a second surface for emitting light to the eye of the user. The prism of the optical layers 176 A-N extends over all or at least a portion of the <figure-callout id="175A" label="respective apertures" filenames="US12080261-20240903-D00005.pn
CLAIMS
Claims ( 20 )
What is claimed is:
1. A method of playing a musical composition, comprising:
initiating a computer vision and mapping application on a wearable device comprising a processor, a memory, a camera, a loudspeaker, and a display;
storing in the memory a plurality of preset token color values, each associated with a set of sound samples;
capturing with the camera a color image of a note matrix comprising one or more fields defined by rows and columns, wherein each row is associated with a musical note and each column is associated with a beat duration;
detecting with the application one or more tokens in the captured color image, wherein the one or more tokens comprises at least a first token characterized by a first token color value wherein the first token is detected at a first token position relative to the fields of the note matrix, and a subsequent token characterized by a subsequent token color value wherein the subsequent token is detected at a subsequent token position relative to the fields of the note matrix;
retrieving from the memory a first set of sound samples based on the first token color value and a subsequent set of sound samples based on the subsequent token color value;
selecting a first playable sound from the first set of sound samples based on the first token position and a subsequent playable sound from the subsequent set of sound samples based on the subsequent token position; and
playing through the loudspeaker a musical composition comprising the first playable sound and the subsequent playable sound in a sequence according to each column, in order of the note matrix.
2. The method of claim 1 , further comprising:
detecting at least a portion of a hand in the color image; and
in response to detecting the portion, pausing the detecting of the one or more tokens.
3. The method of claim 1 , wherein capturing the color image further comprises capturing a sequence of subsequent color images, the method further comprising:
detecting at least a portion of a hand in a first image in the sequence of subsequent color images; and
in response to detecting the portion in the first image, pausing the detecting of the one or more tokens while and for as long as the portion is detected in any of the sequence of subsequent color images following the first image.
4. The method of claim 3 , further comprising:
initiating on the wearable device a hand detection algorithm, wherein the algorithm performs the detecting and the pausing.
5. The method of claim 1 , wherein the detecting of the one or more tokens further comprises:
calculating a difference between the first token color value associated with the first token and the first preset token color value; and
wherein the retrieving of the first set of sound samples is based on the difference.
6. The method of claim 5 , further comprising:
determining that at least one of the detected one or more token color values includes the first preset token color value if the calculated difference is less than a threshold value.
7. The method of claim 1 , further comprising:
detecting a background color value associated with an unmarked area associated with the note matrix;
calculating a ratio between the background color value and a preset background color value;
multiplying each of the detected one or more token color values by the ratio to obtain a modified set of detected token color values; and
calculating a variance between the modified set of detected token color values and the first preset token color value,
wherein the retrieving of the first set of sound samples is based on the calculated variance.
8. The method of claim 1 , wherein the display is characterized by a two-dimensional display coordinate system, the method further comprising:
establishing a map coordinate system relative to the note matrix in the captured color image, the established map coordinate system comprising a set of field coordinates associated with each of the plurality of fields of the note matrix; and
storing in the memory a global state map comprising the detected one or more token color values and their detected positions relative to the established map coordinate system,
wherein playing the musical composition comprises playing the one or more sound samples according to the global state map.
9. The method of claim 1 , wherein each set of sound samples stored in memory is associated with one or more musical instruments, such that a first preset token color value is associated with a first set of sound samples and a first musical instrument,
wherein playing the musical composition further comprises:
determining, for each of the fields of the note matrix, whether the first token is positioned near a select column and near a select row, wherein the select row is associated with a select note of a musical composition;
retrieving from the memory a select sound sample from the retrieved first set of sound samples; and
playing the retrieved select sound sample for the select note of the musical composition for a select beat duration associated with the select column.
10. The method of claim 9 , wherein the note matrix further comprises a pointer located along a tempo scale extending lengthwise from a minimum speed to a maximum speed, and wherein playing the musical composition further comprises:
identifying pointer coordinates associated with the pointer relative to the tempo scale; and
adjusting the beat duration associated with each column according to the pointer coordinates.
11. A system comprising:
a wearable device comprising a processor, a memory, a camera, a loudspeaker, and a display;
programming in the memory, wherein execution of the programming by the processor configures the wearable device to perform functions, including functions to:
store in the memory a plurality of preset token color values, each associated with a set of sound samples;
capture with the camera a color image of a note matrix comprising one or more fields defined by rows and columns, wherein each row is associated with a musical note and each column is associated with a beat duration;
detect one or more tokens in the captured color image, wherein the one or more tokens comprises at least a first token characterized by a first token color value wherein the first token is detected at a first token position relative to the fields of the note matrix, and a subsequent token characterized by a subsequent token color value wherein the subsequent token is detected at a subsequent token position relative to the fields of the note matrix;
retrieve from the memory a first set of sound samples based on the first token color value and a subsequent set of sound samples based on the subsequent token color value;
select a first playable sound from the first set of sound samples based on the first token position and a subsequent playable sound from the subsequent set of sound samples based on the subsequent token position; and
play through the loudspeaker a musical composition comprising the first playable sound and the subsequent playable sound in a sequence according to each column, in order, of the note matrix.
12. The system of claim 11 , wherein the execution of the programming further configures the wearable device to:
capture a sequence of subsequent color images;
detect at least a portion of a hand in a first image in the sequence of subsequent color images; and
pause the detecting of the one or more tokens while and for as long as the portion is detected in any of the sequence of subsequent color images following the first image.
13. The system of claim 11 , wherein the execution of the programming further configures the wearable device to:
calculate a difference between the first token color value associated with the first token and the first preset token color value; and
retrieve of the first set of sound samples based on the difference.
14. The system of claim 11 , wherein the execution of the programming further configures the wearable device to:
detect a background color value associated with an unmarked area associated with the note matrix;
calculate a ratio between the background color value and a preset background color value;
multiply each of the detected one or more token color values by the ratio to obtain a modified set of detected token color values;
calculate a variance between the modified set of detected token color values and the first preset token color value; and
retrieve of the first set of sound samples based on the calculated variance.
15. The system of claim 11 , wherein the execution of the programming further configures the wearable device to:
establish a two-dimensional display coordinate system associated with the display;
establish a map coordinate system relative to the note matrix in the captured color image, the established map coordinate system comprising a set of field coordinates associated with each of the plurality of fields of the note matrix;
store in the memory a global state map comprising the detected one or more token color values and their detected positions relative to the established map coordinate system; and
play the musical composition according to the global state map.
16. The system of claim 11 , wherein the execution of the programming further configures the wearable device to:
detect in the captured color image a pointer relative to a tempo scale extending lengthwise from a minimum speed to a maximum speed;
identify pointer coordinates associated with the pointer relative to the tempo scale; and
play the musical composition according to the pointer coordinates.
17. A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of:
initiating a computer vision and mapping application on a wearable device comprising a processor, a memory, a camera, a loudspeaker, and a display;
storing in the memory a plurality of preset token color values, each associated with a set of sound samples;
capturing with the camera a color image of a note matrix comprising one or more fields defined by rows and columns, wherein each row is associated with a musical note and each column is associated with a beat duration;
detecting with the application one or more tokens in the captured color image, wherein the one or more tokens comprises at least a first token characterized by a first token color value wherein the first token is detected at a first token position relative to the fields of the note matrix and a subsequent token characterized by a subsequent token color value wherein the subsequent token is detected at a subsequent token position relative to the fields of the note matrix;
retrieving from the memory a first set of sound samples based on the first token color value and a subsequent set of sound samples based on the subsequent token color value;
selecting a first playable sound from the first set of sound samples based on the first token position and a subsequent playable sound from the subsequent set of sound samples based on the subsequent token position; and
playing through the loudspeaker a musical composition comprising the first playable sound and the subsequent playable sound in a sequence according to each column, in order, of the note matrix.
18. The non-transitory computer-readable medium of claim 17 , wherein the execution of the stored program code is operative to further cause the electronic processor to perform the steps of:
capturing a sequence of subsequent color images;
detecting at least a portion of a hand in a first image in the sequence of subsequent color images; and
in response to detecting the portion in the first image, pausing the detecting of the one or more tokens while and for as long as the portion is detected in any of the sequence of subsequent color images following the first image.
19. The non-transitory computer-readable medium of claim 17 , wherein the execution of the stored program code is operative to further cause the electronic processor to perform the steps of:
detecting a background color value associated with an unmarked area associated with the note matrix;
calculating a ratio between the background color value and a preset background color value;
multiplying each of the detected one or more token color values by the ratio to obtain a modified set of detected token color values;
calculating a variance between the modified set of detected token color values and the first preset token color value; and
retrieving the first set of sound samples based on the calculated variance.
20. The non-transitory computer-readable medium of claim 17 , wherein the display is characterized by a two-dimensional display coordinate system, and wherein the execution of the stored program code is operative to further cause the electronic processor to perform the steps of:
establishing a map coordinate system relative to the note matrix in the captured color image, the established map coordinate system comprising a set of field coordinates associated with each of the plurality of fields of the note matrix; and
storing in the memory a global state map comprising the detected one or more token color values and their detected positions relative to the established map coordinate system,
wherein playing the musical composition comprises playing the one or more sound samples according to the global state map.
US18/142,181
2019-12-21
2023-05-02
Computer vision and mapping for audio
Active
US12080261B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US18/142,181
US12080261B2
( en )
2019-12-21
2023-05-02
Computer vision and mapping for audio
Applications Claiming Priority (3)
Application Number
Priority Date
Filing Date
Title
US16/724,209
US11087728B1
( en )
2019-12-21
2019-12-21
Computer vision and mapping for audio applications
US17/397,145
US11670267B2
( en )
2019-12-21
2021-08-09
Computer vision and mapping for audio applications
US18/142,181
US12080261B2
( en )
2019-12-21
2023-05-02
Computer vision and mapping for audio
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US17/397,145
Continuation
US11670267B2
( en )
2019-12-21
2021-08-09
Computer vision and mapping for audio applications
Publications (2)
Publication Number
Publication Date
US20230267900A1
US20230267900A1 ( en )
2023-08-24
US12080261B2
true
US12080261B2 ( en )
2024-09-03
Family
ID=77179106
Family Applications (3)
Application Number
Title
Priority Date
Filing Date
US16/724,209
Active
US11087728B1
( en )
2019-12-21
2019-12-21
Computer vision and mapping for audio applications
US17/397,145
Active
2039-12-23
US11670267B2
( en )
2019-12-21
2021-08-09
Computer vision and mapping for audio applications
US18/142,181
Active
US12080261B2
( en )
2019-12-21
2023-05-02
Computer vision and mapping for audio
Family Applications Before (2)
Application Number
Title
Priority Date
Filing Date
US16/724,209
Active
US11087728B1
( en )
2019-12-21
2019-12-21
Computer vision and mapping for audio applications
US17/397,145
Active
2039-12-23
US11670267B2
( en )
2019-12-21
2021-08-09
Computer vision and mapping for audio applications
Country Status (1)
Country
Link
US
( 3 )
US11087728B1
( en )
Families Citing this family (159)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US10509466B1
( en )
2011-05-11
2019-12-17
Snap Inc.
Headwear with computer and optical element for use therewith and systems utilizing same
US10439972B1
( en )
2013-05-30
2019-10-08
Snap Inc.
Apparatus and method for maintaining a message thread with opt-in permanence for entries
US10438631B2
( en )
2014-02-05
2019-10-08
Snap Inc.
Method for real-time video processing involving retouching of an object in the video
US9276886B1
( en )
2014-05-09
2016-03-01
Snapchat, Inc.
Apparatus and method for dynamically configuring application component tiles
US9537811B2
( en )
2014-10-02
2017-01-03
Snap Inc.
Ephemeral gallery of ephemeral messages
US9396354B1
( en )
2014-05-28
2016-07-19
Snapchat, Inc.
Apparatus and method for automated privacy protection in distributed images
US9225897B1
( en )
2014-07-07
2015-12-29
Snapchat, Inc.
Apparatus and method for supplying content aware photo filters
US10775996B2
( en )
2014-11-26
2020-09-15
Snap Inc.
Hybridization of voice notes and calling
US10311916B2
( en )
2014-12-19
2019-06-04
Snap Inc.
Gallery of videos set to an audio time line
US9385983B1
( en )
2014-12-19
2016-07-05
Snapchat, Inc.
Gallery of messages from individuals with a shared interest
EP3272078B1
( en )
2015-03-18
2022-01-19
Snap Inc.
Geo-fence authorization provisioning
US9668217B1
( en )
2015-05-14
2017-05-30
Snap Inc.
Systems and methods for wearable initiated handshaking
US10503264B1
( en )
2015-06-16
2019-12-10
Snap Inc.
Radial gesture navigation
US10354425B2
( en )
2015-12-18
2019-07-16
Snap Inc.
Method and system for providing context relevant media augmentation
US10055895B2
( en )
2016-01-29
2018-08-21
Snap Inc.
Local augmented reality persistent sticker objects
US10530731B1
( en )
2016-03-28
2020-01-07
Snap Inc.
Systems and methods for chat with audio and video elements
US10474353B2
( en )
2016-05-31
2019-11-12
Snap Inc.
Application control using a gesture based trigger
US11404056B1
( en )
2016-06-30
2022-08-02
Snap Inc.
Remoteless control of drone behavior
US10102423B2
( en )
2016-06-30
2018-10-16
Snap Inc.
Object modeling and replacement in a video stream
US10609036B1
( en )
2016-10-10
2020-03-31
Snap Inc.
Social media post subscribe requests for buffer user accounts
US11635872B2
( en )
2016-11-22
2023-04-25
Snap Inc.
Smart carousel of image modifiers
US20180210628A1
( en )
2017-01-23
2018-07-26
Snap Inc.
Three-dimensional interaction system
US10579869B1
( en )
2017-07-18
2020-03-03
Snap Inc.
Virtual object machine learning
US11204949B1
( en )
2017-07-31
2021-12-21
Snap Inc.
Systems, devices, and methods for content selection
US11323398B1
( en )
2017-07-31
2022-05-03
Snap Inc.
Systems, devices, and methods for progressive attachments
US10591730B2
( en )
2017-08-25
2020-03-17
II Jonathan M. Rodriguez
Wristwatch based interface for augmented reality eyewear
US11531357B1
( en )
2017-10-05
2022-12-20
Snap Inc.
Spatial vector-based drone control
WO2019094618A1
( en )
2017-11-08
2019-05-16
Signall Technologies Zrt
Computer vision based sign language interpreter
US10217488B1
( en )
2017-12-15
2019-02-26
Snap Inc.
Spherical video editing
US10567321B2
( en )
2018-01-02
2020-02-18
Snap Inc.
Generating interactive messages with asynchronous media content
US10523606B2
( en )
2018-01-02
2019-12-31
Snap Inc.
Generating interactive messages with asynchronous media content
US11227626B1
( en )
2018-05-21
2022-01-18
Snap Inc.
Audio response messages
US11063889B2
( en )
2018-06-08
2021-07-13
Snap Inc.
Generating interactive messages with entity assets
US11334815B1
( en )
2018-09-28
2022-05-17
Snap Inc.
Cloud based machine learning
US10796482B2
( en )
2018-12-05
2020-10-06
Snap Inc.
3D hand shape and pose estimation
US12071228B1
( en )
*
2019-03-28
2024-08-27
Snap Inc.
Drone with propeller guard configured as an airfoil
US11012390B1
( en )
2019-03-28
2021-05-18
Snap Inc.
Media content response in a messaging system
US11036368B1
( en )
2019-03-29
2021-06-15
Snap Inc.
Messaging system with message transmission user interface
US11019011B1
( en )
2019-03-29
2021-05-25
Snap Inc.
Messaging system with discard user interface
US11106342B1
( en )
2019-06-03
2021-08-31
Snap Inc.
User interfaces to facilitate multiple modes of electronic communication
US11151794B1
( en )
2019-06-28
2021-10-19
Snap Inc.
Messaging system with augmented reality messages
US11307747B2
( en )
2019-07-11
2022-04-19
Snap Inc.
Edge gesture interface with smart interactions
KR20260046176A
( en )
2019-09-09
2026-04-06
ì¤ë ì¸ì½í¬ë ì´í°ë
Hand pose estimation from stereo cameras
US11087728B1
( en )
*
2019-12-21
2021-08-10
Snap Inc.
Computer vision and mapping for audio applications
US11062498B1
( en )
2019-12-30
2021-07-13
Snap Inc.
Animated pull-to-refresh
US11488358B2
( en )
2020-02-05
2022-11-01
Snap Inc.
Augmented reality session creation using skeleton tracking
US11265274B1
( en )
2020-02-28
2022-03-01
Snap Inc.
Access and routing of interactive messages
US11409368B2
( en )
2020-03-26
2022-08-09
Snap Inc.
Navigating through augmented reality content
US11675494B2
( en )
2020-03-26
2023-06-13
Snap Inc.
Combining first user interface content into second user interface
US11960651B2
( en )
2020-03-30
2024-04-16
Snap Inc.
Gesture-based shared AR session creation
US11455078B1
( en )
2020-03-31
2022-09-27
Snap Inc.
Spatial navigation and creation interface
WO2022006249A1
( en )
2020-06-30
2022-01-06
Snap Inc.
Skeletal tracking for real-time virtual effects
EP4197180A1
( en )
2020-08-13
2023-06-21
Snap Inc.
User interface for pose driven virtual effects
US11671559B2
( en )
2020-09-30
2023-06-06
Snap Inc.
Real time video editing
US11782577B2
( en )
2020-12-22
2023-10-10
Snap Inc.
Media content player on an eyewear device
US12105283B2
( en )
2020-12-22
2024-10-01
Snap Inc.
Conversation interface on an eyewear device
US11797162B2
( en )
2020-12-22
2023-10-24
Snap Inc.
3D painting on an eyewear device
KR102861516B1
( en )
2020-12-22
2025-09-18
ì¤ë ì¸ì½í¬ë ì´í°ë
Media content player on eyewear devices
US12229342B2
( en )
2020-12-22
2025-02-18
Snap Inc.
Gesture control on an eyewear device
EP4272406B1
( en )
2020-12-29
2024-10-02
Snap Inc.
Body ui for augmented reality components
EP4272060A1
( en )
2020-12-31
2023-11-08
Snap Inc.
Real-time video communication interface with haptic feedback
US11989348B2
( en )
2020-12-31
2024-05-21
Snap Inc.
Media content items with haptic feedback augmentations
CN116710885A
( en )
2020-12-31
2023-09-05
æ¯çº³æ®å ¬å¸
Communication interface with haptic feedback response
US12008152B1
( en )
2020-12-31
2024-06-11
Snap Inc.
Distance determination for mixed reality interaction
WO2022147449A1
( en )
2020-12-31
2022-07-07
Snap Inc.
Electronic communication interface with haptic feedback response
US11734959B2
( en )
2021-03-16
2023-08-22
Snap Inc.
Activating hands-free mode on mirroring device
USD998637S1
( en )
2021-03-16
2023-09-12
Snap Inc.
Display screen or portion thereof with a graphical user interface
US11908243B2
( en )
2021-03-16
2024-02-20
Snap Inc.
Menu hierarchy navigation on electronic mirroring devices
US11809633B2
( en )
2021-03-16
2023-11-07
Snap Inc.
Mirroring device with pointing based navigation
US11978283B2
( en )
2021-03-16
2024-05-07
Snap Inc.
Mirroring device with a hands-free mode
US11798201B2
( en )
2021-03-16
2023-10-24
Snap Inc.
Mirroring device with whole-body outfits
US12573100B2
( en )
2021-03-31
2026-03-10
Snap Inc.
User-defined contextual spaces
US12050729B2
( en )
2021-03-31
2024-07-30
Snap Inc.
Real-time communication interface with haptic and audio feedback response
US12164689B2
( en )
2021-03-31
2024-12-10
Snap Inc.
Virtual reality communication interface with haptic feedback response
KR102840378B1
( en )
2021-03-31
2025-07-31
ì¤ë ì¸ì½í¬ë ì´í°ë
Virtual reality interface with haptic feedback response
US12314472B2
( en )
2021-03-31
2025-05-27
Snap Inc.
Real-time communication interface with haptic and audio feedback response
US11928306B2
( en )
2021-05-19
2024-03-12
Snap Inc.
Touchpad navigation for augmented reality display device
US11880542B2
( en )
2021-05-19
2024-01-23
Snap Inc.
Touchpad input for augmented reality display device
WO2022246399A1
( en )
2021-05-19
2022-11-24
Snap Inc.
Touchpad input for augmented reality display device
CN118891859A
( en )
2021-08-16
2024-11-01
æ¯çº³æ®å ¬å¸
Displaying profiles from messaging system contact feeds
US11670059B2
( en )
2021-09-01
2023-06-06
Snap Inc.
Controlling interactive fashion based on body gestures
US11900506B2
( en )
2021-09-09
2024-02-13
Snap Inc.
Controlling interactive fashion based on facial expressions
US11734866B2
( en )
2021-09-13
2023-08-22
Snap Inc.
Controlling interactive fashion based on voice
US11983826B2
( en )
2021-09-30
2024-05-14
Snap Inc.
3D upper garment tracking
US11636662B2
( en )
2021-09-30
2023-04-25
Snap Inc.
Body normal network light and rendering control
US11651572B2
( en )
2021-10-11
2023-05-16
Snap Inc.
Light and rendering of garments
US11748958B2
( en )
2021-12-07
2023-09-05
Snap Inc.
Augmented reality unboxing experience
US11960784B2
( en )
2021-12-07
2024-04-16
Snap Inc.
Shared augmented reality unboxing experience
US11861801B2
( en )
2021-12-30
2024-01-02
Snap Inc.
Enhanced reading with AR glasses
US12159412B2
( en )
2022-02-14
2024-12-03
Snap Inc.
Interactively defining an object segmentation
US11579747B1
( en )
2022-03-14
2023-02-14
Snap Inc.
3D user interface depth forgiveness
US12265663B2
( en )
2022-04-04
2025-04-01
Snap Inc.
Gesture-based application invocation
US11949527B2
( en )
2022-04-25
2024-04-02
Snap Inc.
Shared augmented reality experience in video chat
US12360663B2
( en )
2022-04-26
2025-07-15
Snap Inc.
Gesture-based keyboard text entry
US12277632B2
( en )
2022-04-26
2025-04-15
Snap Inc.
Augmented reality experiences with dual cameras
US12190739B2
( en )
2022-04-27
2025-01-07
Snap Inc.
Navigation correction for excessive wind
US12360531B2
( en )
2022-04-27
2025-07-15
Snap Inc.
Stabilization and navigation of an autonomous drone
US12287647B2
( en )
2022-04-27
2025-04-29
Snap Inc.
Autonomous drone navigation based on vision
US12372975B2
( en )
2022-04-27
2025-07-29
Snap Inc.
Unlocking an autonomous drone for takeoff
US12181945B2
( en )
2022-04-27
2024-12-31
Snap Inc.
Boot time for autonomous drones
US12282607B2
( en )
2022-04-27
2025-04-22
Snap Inc.
Fingerspelling text entry
US12344409B2
( en )
2022-04-27
2025-07-01
Snap Inc.
Fully autonomous drone flights
US12136160B2
( en )
2022-04-27
2024-11-05
Snap Inc.
Augmented reality experience power usage prediction
US12300275B2
( en )
2022-04-27
2025-05-13
Snap Inc.
Editing video captured by electronic devices
US12367606B2
( en )
2022-04-27
2025-07-22
Snap Inc.
Automatically cropping of landscape videos
US12517521B2
( en )
2022-04-27
2026-01-06
Snap Inc.
Landing an autonomous drone with gestures
US11960653B2
( en )
2022-05-10
2024-04-16
Snap Inc.
Controlling augmented reality effects through multi-modal human interaction
US12327302B2
( en )
2022-05-18
2025-06-10
Snap Inc.
Hand-tracked text selection and modification
US12373096B2
( en )
2022-05-31
2025-07-29
Snap Inc.
AR-based virtual keyboard
WO2023235485A1
( en )
2022-06-01
2023-12-07
Snap Inc.
User interface including multiple interaction zones
US12266057B2
( en )
2022-06-02
2025-04-01
Snap Inc.
Input modalities for AR wearable devices
US12001878B2
( en )
2022-06-03
2024-06-04
Snap Inc.
Auto-recovery for AR wearable devices
US12002168B2
( en )
2022-06-20
2024-06-04
Snap Inc.
Low latency hand-tracking in augmented reality systems
US12288298B2
( en )
2022-06-21
2025-04-29
Snap Inc.
Generating user interfaces displaying augmented reality graphics
US12382188B2
( en )
2022-06-22
2025-08-05
Snap Inc.
Hand-tracking pipeline dimming
US12332438B2
( en )
2022-06-23
2025-06-17
Snap Inc.
Color calibration tool for see-through augmented reality environment
US12204693B2
( en )