Conceptio › Archive › Google Patents

Google Patentsopen access

Apparatus for use in a virtual reality system — Microsoft Technology Licensing, Llc (US11379054B2)

Microsoft Technology Licensing, Llc · Google Patents

Google Patents · Patents · License: Open Access

patent, google patents, intellectual property, US11379054B2, Microsoft Technology Licensing, Llc, Edward Cutrell, en, 2022

ABSTRACT

Abstract

A virtual reality system is described herein. The virtual reality system includes a cane controller and a computing system. The cane controller comprises a rod, a sensor, and a brake mechanism, wherein the sensor is configured to generate a signal that is indicative of position, direction of movement, and velocity of the rod, and wherein the brake mechanism is configured to apply a force to the rod. The computing system receives the signal, computes a position, direction of movement, and velocity of a virtual rod in a virtual space, and outputs a control signal to the brake mechanism based upon such computation. The brake mechanism applies the force to the rod in a direction and with a magnitude indicated in the control signal, thereby preventing the user from causing the virtual rod to penetrate a virtual barrier in the virtual space.

Description

RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No. 15/821,713, filed on Nov. 22, 2017, and entitled âAPPARATUS FOR USE IN A VIRTUAL REALITY SYSTEMâ. The entirety of this application is incorporated herein by reference.

BACKGROUND

Virtual reality (VR) technologies have been conventionally applied to fields such as entertainment (e.g., gaming), education, and social activities. Further, potential of VR has been recognized with respect to training and rehabilitation for people with disabilities, such as those with dyslexia, people who have experienced a stroke, people with attention-deficit hyperactivity disorder (ADHD), etc.

While VR applications have great potential in a variety of use cases, conventional VR applications rely upon realistic visual feedback to provide an immersive experience to sighted people, for whom the visual sense is dominant. These existing VR applications, however, are generally not accessible to people with visual impairments, thereby preventing such people from benefiting from VR technologies.

Prior work in VR that has been developed for use by people with visual impairments has focused on creating auditory VRâfor instance, an audio-based environment simulator game has been developed that facilitate allowing blind users to interact with a virtual scene through auditory feedback. In such game, presence of objects is identified through sound rendering. For example, a conventional VR system employs an acoustic VR space, where a portable system generates spatial audio to create the illusion that objects in the virtual scene are covered with speakers emitting sounds depending on their physical characteristics (e.g., color, texture, etc.). In another exemplary conventional VR system, real-world navigation strategies of people with visual impairments are incorporated into a virtual space, where echolocation (use of sound reflecting off objects to localize them) is simulated in the virtual space to assist users in better understanding the virtual space.

Other conventional VR systems have introduced haptic sensations to provide benefits of VR to people with visual impairments. Haptic sensation by way of white canes is the main channel for people with visual impairments to understand a physical space, with audio serving as a channel for complementary information. When navigating the real-world, the white cane is commonly used for low resolution scanning of the environment and to detect obstacles and identify distances. Similarly, legs and feet are used to feel the ground surface, and the palm or fingers are used to precisely recognize the shape and texture of objects. In an exemplary conventional VR system, a joystick is employed by a user, where movement of the joystick corresponds to movement in a virtual space, and further where the joystick is vibrated to present cues to a user as to boundaries of the virtual space and virtual objects in the virtual space. This system, however, is unable to allow for natural movement of a user as the user navigates through the virtual space.

In yet another exemplary conventional VR system, a haptic glove is worn by a user and generates force feedback to fingers of the user as the user moves through a virtual space, thereby providing the illusion that the user is navigating the virtual space with a cane. The user can hold a real cane, the grasp posture of the user can be detected, and force feedback can be provided through the haptic glove (to simulate vibrations that would be felt based upon the tip of the cane impacting a virtual object in the virtual space). This real cane, however, penetrates virtual objects in the virtual space, thereby failing to provide a user with realistic proprioceptive feedback as the user employs the cane to navigate the virtual space.

SUMMARY

The following is a brief summary of subject matter that is described in greater detail herein. This summary is not intended to be limiting as to the scope of the claims.

Described herein are various technologies pertaining to a virtual reality (VR) system, wherein the VR system is well-suited for use by people with visual impairments. The VR system includes a computing system and an apparatus that is manipulated by a user of the VR system when navigating a virtual space. The computing system executes a VR application, wherein a VR environment (a virtual space) can be defined by way of the VR application. The virtual space can include barriers (such as walls), objects (such as tables, chairs, trash cans, etc.), steps, and so forth. Boundaries in the virtual space can also have materials assigned thereto, material thicknesses assigned thereto, colors assigned thereto, textures assigned thereto, etc. For instance, a floor in the virtual space can be carpeted, and accordingly is assigned a texture that corresponds to carpet; similarly, a floor in the virtual space can be tiled, and accordingly is assigned a texture that corresponds to ceramic. Still further, materials and textures in the virtual space are assigned audio characteristics. As will be described in greater detail herein, the computing system is configured to track movement of the user through the virtual space and interaction of the user with the virtual space based upon manipulation of the above-referenced apparatus by the user in the real-world.

The apparatus (referred to herein as a cane controller) includes a rod that comprises a proximal end and a distal end. In an example, the rod can have a length of between twelve inches and thirty-six inches. The proximal end of the rod is configured to be grasped by a hand of the user, and a sensor is coupled to the rod, wherein the sensor outputs a sensor signal that is indicative of position and orientation, direction of movement, and velocity of the distal end of the rod in the real-world (as the user manipulates the cane controller). The apparatus also includes a brake mechanism that is operably coupled to the rod, wherein the brake mechanism is configured to apply a resistive force to the rod, wherein magnitude and direction of the resistive force is based upon a control signal received from the computing system. The computing system generates such control signal based upon the signal output by the sensor that is coupled to the rod.

In an exemplary application, the computing system can construct a virtual cane of arbitrary length for use in the virtual environment, wherein position, velocity, and direction of movement of the virtual cane in the virtual space corresponds to position, velocity, and direction of movement of the rod in the real-world. Specifically, as location of the sensor on the rod is known, the computing system can receive the signal output by the sensor and compute a position, direction of movement, and velocity of any portion of the virtual cane (including the distal end of the virtual cane) based upon the sensor signal output by the sensor. When the VR application determines that the virtual cane impacts a surface (e.g., a wall, a virtual object, etc.) in the virtual space (based upon the signal output by the sensor and boundaries of the virtual space), the VR application can cause a control signal to be directed to the brake, wherein the brake applies resistive force such that the user (in the real-world) is unable to further move the rod. In other words, the virtual cane is prevented from penetrating boundaries in the virtual space. Thus, a user with a visual impairment navigating the virtual space will perceive physical boundaries as the user moves the rod about in the real-world (e.g., even though the virtual rod never impacts a real physical boundary).

The VR system described herein also can provide other channels of feedback to the user, including auditory feedback (e.g., three-dimensional spatial audio can be provided to the user by way of headphones), and a haptic mechanism can be coupled to the rod such that vibratory feedback is given to the user as the user navigates the virtual environment. For instance, when the virtual environment includes a carpeted floor, and the VR application determines that the distal end of the virtual cane is sweeping over and in contact with the carpet in the virtual space (e.g., where the determination is made based upon real-world manipulation of the rod by the user), the vibratory mechanism can provide vibratory feedback along the rod, where the vibratory feedback is configured to provide the user the sensation that a real cane is being moved over a real carpeted floor. The brake mechanism can also provide a force feedback as the distal end encounters friction with the carpet.

While the technologies have been described as well-suited for use by people with visual impairments, it is to be understood that the technologies described herein are not limited to such applications. For example, the technologies described herein may be well-suited for use with respect to physical rehabilitation, entertainment (gaming), etc. For instance, a virtual space may include a river, wherein the user is tasked with navigating the river in a kayak. The user may have one or more apparatuses attached thereto, wherein positions (including orientations), directions of movement, and velocities of oars in the virtual space are controlled by positions (including orientations), directions of movement, and velocities or rods manipulated by the user in the real-world. As an oar (controlled based upon movement of a rod) hits the water in the virtual space, the brake mechanism can be configured to provide viscous resistive force, thereby giving the user the sensation of additional resistance as the user is attempting to row through the virtual river.

In another exemplary application, the technologies described herein are well-suited for medical training. For instance, a surgeon can practice a surgery in a virtual environment, wherein position and movement of a distal end of the rod can be mapped to a virtual scalpel being held by the surgeon in a virtual surgery room. As the surgeon moves the rod in the real-world, such that the scalpel impacts skin of a virtual patient, the brake mechanism can be configured to emit resistive force to the rod, thereby giving the surgeon the perception of a change in resistance as the surgery is performed. In one or more of these applications, visual feedback can also be provided to enhance the experience in the virtual environment.

The above summary presents a simplified summary in order to provide a basic understanding of some aspects of the systems and/or methods discussed herein. This summary is not an extensive overview of the systems and/or methods discussed herein. It is not intended to identify key/critical elements or to delineate the scope of such systems and/or methods. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic that depicts a virtual reality (VR) system that is well-suited for use by people with visual impairments.

FIG. 2 illustrates an exemplary apparatus that is to be employed by a user of a VR system.

FIG. 3 is a functional block diagram of an exemplary VR application.

FIG. 4 is a schematic that depicts a user in a virtual space.

FIG. 5 is a flow diagram illustrating an exemplary methodology for configuring an apparatus that is well-suited for use in a virtual space.

FIG. 6 is a flow diagram illustrating an exemplary methodology for controlling an apparatus that is to be used in a virtual space.

FIG. 7 is a flow diagram illustrating an exemplary methodology for controlling an apparatus that is used in virtual space.

FIG. 8 is an exemplary computing system.

DETAILED DESCRIPTION

Various technologies pertaining to a virtual reality (VR) system that is well-suited for use by people with visual impairments are now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It may be evident, however, that such aspect(s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Further, it is to be understood that functionality that is described as being carried out by certain system components may be performed by multiple components. Similarly, for instance, a component may be configured to perform functionality that is described as being carried out by multiple components.

Moreover, the term âorâ is intended to mean an inclusive âorâ rather than an exclusive âor.â That is, unless specified otherwise, or clear from the context, the phrase âX employs A or Bâ is intended to mean any of the natural inclusive permutations. That is, the phrase âX employs A or Bâ is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. In addition, the articles âaâ and âanâ as used in this application and the appended claims should generally be construed to mean âone or moreâ unless specified otherwise or clear from the context to be directed to a singular form.

Further, as used herein, the terms âcomponentâ and âsystemâ are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor. The computer-executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices. Further, as used herein, the term âexemplaryâ is intended to mean serving as an illustration or example of something, and is not intended to indicate a preference.

Described herein are various technologies pertaining to an apparatus, referred to herein as a cane controller, that is well-suited for allowing people with visual impairments to use real-world cane skills to explore virtual spaces. The cane controller described herein was inspired by real-world use of white canes by people with visual impairments. The cane controller provides three types of feedback: 1) braking; 2) vibrotactile; and 3) auditory. With respect to braking, when a virtual cane (controlled based upon movement of the cane controller) hits a virtual object in a virtual space, the brake mechanism generates physical resistance that prevents the cane controller, in the real-world, from further movement that would cause the virtual cane to penetrate boundaries of the object. Hence, with this proprioceptive feedback, the virtual cane does not penetrate virtual objects in the virtual space, thereby providing the user with the perception of boundaries of virtual objects in the virtual space.

With respect to vibrotactile feedback, when any part of the virtual cane contacts a virtual object (taps an object or sweeps over a surface of an object), the cane controller generates vibrotactile feedback to simulate the corresponding tap or texture vibration that the user would feel in the real-world if the user were interacting with real-world objects through use of a virtual cane.

With respect to auditory feedback, the VR system described herein can generate three-dimensional audio feedback and provide such audio to earphones being worn by a user to simulate the sound a real cane makes when impacting objects in the real-world (i.e., when the virtual cane impacts a metal trash can in the virtual space, three-dimensional audio feedback is provided that simulates the sound made when a real cane impacts a metal trash can in the real world, where the three-dimensional audio feedback can assist a blind user with localizing the sound in the virtual space). Sound rendering depends on the surface type and collision speed.

Using these different types of feedback, the VR system described

RELATED APPLICATION

BACKGROUND

SUMMARY

The following is a brief summary of subject matter that is described in greater detail herein. This summary is not intended to be limiting as to the scope of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic that depicts a virtual reality (VR) system that is well-suited for use by people with visual impairments.

FIG. 2 illustrates an exemplary apparatus that is to be employed by a user of a VR system.

FIG. 3 is a functional block diagram of an exemplary VR application.

FIG. 4 is a schematic that depicts a user in a virtual space.

FIG. 5 is a flow diagram illustrating an exemplary methodology for configuring an apparatus that is well-suited for use in a virtual space.

FIG. 6 is a flow diagram illustrating an exemplary methodology for controlling an apparatus that is to be used in a virtual space.

FIG. 7 is a flow diagram illustrating an exemplary methodology for controlling an apparatus that is used in virtual space.

FIG. 8 is an exemplary computing system.

DETAILED DESCRIPTION

Using these different types of feedback, the VR system described herein supports at least three different cane strategies often employed by people with visual impairments: 1) shorelining; 2) two-point touch; and 3) constant contact.

Shorelining is a technique that uses touch and drag of a white cane to travel along an edge. A person can repeatedly tap the edge as the person travels along, moving the cane from the edge to the opposite side and then dragging the cane along the ground back toward the edge. In the VR system described herein, when a virtual cane hits a vertical surface, e.g., a virtual wall, in the horizontal left-right direction, the user can feel the physical resistance that enables them to perceive the boundary of the wall. The user can also feel the contact vibration and hear the 3-D sound, which simulates the collision between a real-world cane and wall. With this interaction, a person with a visual impairment can walk along a virtual wall with the cane controller described herein, by using shorelining techniques.

Two-point touch refers to swinging a cane from side to side and tapping the edges of a walking path on either side in an arc slightly wider than shoulders. Two-point touch is often used to protect both sides of a body of a walker. The cane controller described herein supports two-point touch, as when a virtual cane (controlled based upon the cane controller) taps on a surface, the cane controller generates corresponding vibration to simulate the tactile sensation. The user can also hear 3-D tapping sound from the VR system. With this interaction, the user can conduct two-point touch on different virtual surfaces, such as carpet, concrete, tactile domes, etc. by way of the cane controller.

Constant contact is a cane strategy that involves sweeping the cane from side to side and keeping the cane tip in contact with the surface at all times. This technique provides the most reliable detection of subtle surface changes at every point of the arc. Constant contact is supported by the VR system described herein, as the VR system generates resistive force, tactile and auditory feedback that simulates textures of different surfaces. Moreover, since tactile domes found at sidewalk crossings have bumps that would generate resistance when the cane sweeps across such bumps, the brake mechanism can be used to generate a short resistance at each bump, thereby providing the user with the illusion that a cane of the user hits the bumps. Further, the frequency and amplitude of the braking resistance can be modulated, as well as the vibrotactile feedback and the 3-D audio based on the virtual texture characteristics and sweeping speed of the cane controller.

With reference now to FIG. 1 , a schematic that illustrates an exemplary virtual reality (VR) system 100 that is well-suited for use by people with visual impairments is illustrated. The VR system 100 and includes a computing system 102 and an apparatus 104 (referred to herein as a cane controller), wherein the cane controller 104 includes componentry that is in communication with the computing system 102 . The VR system 100 optionally includes a VR headset 106 . A user 108 can wear the VR headset 106 (which can include goggles that comprise one or more screens for depicting imagery and speakers positioned over or near the ears of the user 108 ). In an exemplary embodiment, the VR headset 106 can include the computing system 102 . In other examples, the computing system 102 may be or include mobile telephone, a desktop computing device, a server computing device that is in network communication with the componentry of the cane controller 104 , etc.

The cane controller 104 includes a brake mechanism 110 that is anchored to the body of the user 108 . For example, the brake mechanism 110 may be coupled to a harness that is worn by the user 108 , such that the brake mechanism 110 is positionally fixed at or near the abdomen of the user 108 . The cane controller 104 additionally includes a rod 112 that is operably coupled to the brake mechanism 110 . The brake mechanism 110 is configured to provide a resistive force to the rod 112 , wherein the resistive force is in a direction that opposes the direction of force being applied to the rod 112 by the user 108 . The brake mechanism 110 can be configured to apply such resistive force in one or more directions, where (as will be described below) magnitude(s) and direction(s) of resistive force(s) applied by the brake mechanism 110 can be based upon a magnitude and direction of force being applied to the rod 112 by the user 108 . For instance, when the user 108 applies a leftward force to the rode 112 (with respect to the body of the user 108 ), the brake mechanism 110 can be configured to provide a resistive force in the rightward direction (or vice versa). Similarly, when the user 108 applies a downward force to the rod 112 , the brake mechanism 110 can be configured to provide an upward force (and vice versa). Finally, when the user 108 applies a forward force (in the direction that the user 108 is looking), the brake mechanism 110 can be configured to provide force in the reverse direction (and vice versa). As will be described herein, the brake mechanism 110 is controlled by the computing system 102 .

The cane controller 104 also includes a sensor 114 that is configured to output a sensor signal that is indicative of a position and orientation, direction of movement, and velocity of its 6DOF position. Therefore, the sensor 114 can be or include a 6DOF tracked sensor, an optical sensor, global positioning system (GPS) sensor, an accelerometer, a gyroscope, or other suitable inertial sensor. The cane controller 104 also includes a haptic device 116 that is configured to generate mechanical vibrations with controllable frequency and magnitude. In an exemplary embodiment, the rod 112 can be shorter than a conventional real-world cane, such as between twelve and thirty-six inches long. This shorter length reduces the possibility that the user 108 , when manipulating the cane controller 104 in the real-world, will cause the cane controller 104 to come into physical contact with a real-world surface. In other embodiments, however, the rod 112 may have a length that is proximate to length of a real-world cane (e.g., about sixty inches).

The computing system 102 comprises a processor 118 and memory 120 , wherein the memory has a VR application 122 loaded therein. The VR application 122 , when executed by the processor 118 , can construct a virtual space that is able to be navigated by the user 108 , wherein the virtual space includes virtual boundaries and virtual objects that are analogous to real-world boundaries and objects. Accordingly, for instance, a virtual object and/or a virtual boundary can be assigned a material type, material width, texture, sound characteristics, etc. In a more specific example, a virtual object that represents a trash can be assigned a material type of âmetalâ, a material thickness of â0.1 inchesâ, a texture of âridgedâ, and a sound characteristic (which may alter along a surface of the virtual object). In the example shown in FIG. 1 , the VR application 122 generates a virtual space that includes two three-dimensional virtual objects

124 and 126 . The VR application 122 is further configured to receive data from the VR headset 106 (e.g., generated by a sensor in the VR headset 106 ) that indicates a position and orientation of the head of the user 108 at any given time. The VR application 122 is further configured to receive signals output by the sensor 114 of the cane controller 104 .

The VR application 122 employs signals received from the VR headset 106 to generate three-dimensional audio signals and cause such 3-D audio signals to be transmitted to the VR headset 106 , which emits audio to the user by way of the speakers of the VR headset 106 . The VR application 122 further receives signals generated by the sensor 114 and controls the brake mechanism 110 and the haptic device 116 based upon the signals output by the sensor 114 . Moreover, the VR application 122 can employ the signal output by the sensor 114 to determine a position of the cane controller 104 , and thus the user 108 , in the virtual space constructed by the VR application 122 .

Still further, when the cane controller 104 is designed to be shorter than a typical real-world cane, the VR application 122 can construct a virtual cane 128 that extends from the rod 104 by a distance that can be selected by the user 108 and/or based upon the height of the user 108 , such that the virtual cane 128 has a length that is consistent with the length of a typical real-world cane (e.g., about sixty inches). In other words, as the length of the rod 112 is known, and as the position of the sensor 114 on the rod 112 is known, and further as the signal output by the sensor 112 is indicative of orientation of the rod 112 , the VR application 122 can construct the virtual cane 128 such that the virtual cane 128 extends from the rod 112 at an appropriate distance. Further, the VR application 122 can compute a position, orientation, direction of movement, and velocity of a distal end of the virtual cane 128 based upon the signal output by the sensor 114 .

Exemplary operation of the VR system 100 is now set forth. The user 108 places a VR headset 106 on her head, grasps the rod 112 of the cane controller 104 , and causes the computing device 102 to execute the VR application 122 , such that the VR space that includes the virtual objects

124 and 126 is constructed. The user 108 moves the rod 112 in the real-world as if using a cane, and the VR application 122 continuously constructs the virtual cane 128 based upon signals output by the sensor 114 , and additionally continuously computes position, orientation, direction of movement, and velocity of the distal end of the virtual cane 128 in the VR space based upon signals output by the sensor 112 . For example, the user 108 can swing the rod 112 in a downward arc, and the VR application 122 can determine when the distal end of the virtual cane 128 impacts the ground in the virtual space (and speed of the virtual cane 128 at the time of impact) based upon signals output by the sensor 114 .

When the VR application 122 determines that the distal end of the virtual cane 128 impacts the ground in the virtual space, the VR application 122 can cause a control signal to be transmitted to the VR headset 106 , the brake mechanism 110 , and/or the haptic device 116 . The control signal, in an example, may cause the brake mechanism 110 to generate a resistive force in the upward direction, thereby preventing the user 108 from swinging the rod 104 further downward along the arc (and accordingly preventing the tip of the virtual cane 128 from penetrating the ground in the virtual space). The control signal, in another example, may cause the haptic device 116 to emit mechanical vibrations (such that the rod 112 is vibrated), such that the user 108 has the sensation of a real-world cane hitting a floor. The control signal, in yet another example, may cause speakers of the VR headset 106 to emit three-dimensional audio that simulates a tip of a real-world cane impacting a floor (where the floor has the texture defined in the virtual space), even though the distal end of the rod never impacts a real-world surface. In other words, the user 108 is provided with tactile and audio feedback as if the user 108 were using a real cane in the real-world and tapping a floor with the cane.

Continuing with this example, the user 108 may then begin sweeping the cane controller 104 in a horizontal direction, from left to right and right to left with respect to the body of the user 108 , with the distal end of the rod 112 pointing towards the floor. In the virtual space constructed by the VR application 122 , the distal end of the virtual cane 128 is swept across the floor while maintaining contact with the floor. Based upon signals generated by the sensor 114 , the VR application 122 can compute a location of the distal end of the virtual cane 128 in the virtual space, and can control the brake mechanism 110 , the haptic device 116 , and/or the VR headset 106 based upon such signals. The VR application 122 can control the brake mechanism 110 such that the user 108 is prevented from pushing the distal end of the virtual cane 128 through the floor. The VR application 122 can control the haptic device 116 such that vibratory feedback is provided to the user 108 , where such feedback is based upon the texture of the floor in the virtual space across which the distal end of the virtual cane 128 is being dragged. Based upon this vibratory feedback, the user 108 can understand whether the floor in the virtual space is carpeted, wooden, concrete, etc. The VR application 122 can control the VR headset 106 such that audio that simulates a cane being swept over a floor is provided to the user 108 .

While sweeping the rod 112 , the VR application 122 can determine that, for example, the distal end of the virtual cane 128 has impacted the surface of the virtual object 124 in the virtual space. Specifically, the VR application 122 receives signals output by the sensor 114 and determines that the distal end of the virtual cane 128 has impacted a surface of the virtual object 124 based upon the signals. The <figure-callout id="122" label="VR application" filenames=

CLAIMS

Claims ( 20 )

What is claimed is:

1. A system that is configured for use in a virtual reality (VR) environment, the system comprising:

an apparatus;

a mechanism coupled to the apparatus that is configured to apply forces to the apparatus in a first plane, a second plane, and a third plane, wherein the first plane is orthogonal to the second plane, the third plane is orthogonal to both the first plane and the second plane, and further wherein the computing system, based upon the sensor signal, causes the mechanism to apply resistive forces that oppose the direction of movement of the apparatus in at least one of the first plane, the second plane, or the third plane.

2. The system of claim 1 , further comprising:

a haptic device that is coupled to the apparatus, wherein the haptic device configured to provide vibratory feedback to a hand of a user grasping the apparatus.

3. The system of claim 1 , wherein the computing system is a VR headset worn by a user that is grasping the apparatus.

4. The system of claim 3 , wherein a virtual version of the apparatus is displayed to the user on the VR headset, wherein the computing system causes the mechanism to apply the resistive forces when the virtual version of the apparatus touches a virtual object displayed to the user on the VR headset.

5. The system of claim 4 , wherein the virtual version of the apparatus is a first sword, wherein the virtual object is a second sword.

6. The system of claim 1 , wherein the mechanism is coupled to a harness worn by a user of the apparatus.

7. The system of claim 1 , wherein the resistive forces prevent the apparatus from rotating about an axis that is positioned at a center of the brake mechanism.

8. The system of claim 1 , wherein the apparatus has a proximal end and a distal end, wherein the proximal end is grasped by a user, wherein the sensor is coupled to the distal end, wherein the signal is indicative of a position of the distal end of the apparatus in the three dimensional space and a direction of movement of the distal end of the apparatus in three dimensional space.

9. The system of claim 1 , wherein the apparatus is oriented along the first plane, wherein the resistive forces are applied in the first plane, the second plane, and the third plane.

10. The system of claim 1 , wherein the computing system computes magnitudes of the resistive forces based upon the signal, wherein the computing system causes the mechanism to apply the resistive forces subsequent to computing the magnitudes of the resistive forces.

11. An system that is configured for use in a virtual reality (VR) environment, the apparatus comprising:

an apparatus;

a mechanism that is configured to apply directional forces to the apparatus in three planes that are orthogonal to one another; and

a sensor that is coupled to the apparatus, wherein the sensor is configured to output, to a computing system, a signal that is indicative of a position of the apparatus in three dimensional space and a direction of movement of the apparatus in the three dimensional space, and further wherein the computing system is configured to control, based upon the signal, magnitudes of the directional forces generated by the mechanism in the three planes.

12. The system of claim 11 , wherein the computing system comprises a VR headset worn by a user who is grasping the apparatus, wherein a virtual version of the apparatus is displayed to the user on the VR headset, wherein the mechanism applies the directional forces when the virtual version of the apparatus touches a virtual object displayed to the user on the VR headset, and further wherein the magnitudes of the resistive forces in the three planes are further based upon an impact of the virtual version of the apparatus with the virtual object in virtual space.

13. The system of claim 12 , wherein the computing system computes a position of the virtual version of the apparatus in the virtual space based upon the signal.

14. The system of claim 11 , wherein the computing system comprises a VR headset, wherein the VR headset emits audio based upon the signal that is output by the sensor.

15. The system of claim 11 , wherein a length of the apparatus is between twenty-four and forty-eight inches.

16. The system of claim 11 , wherein the mechanism rotates about an axis that is tangential to an abdomen of a user who is grasping the apparatus and parallel to a ground on which the user stands.

17. The system of claim 11 , wherein the directional forces are resistive forces that oppose the direction of movement of the apparatus in at least one of the three planes.

18. A system that is configured for use in a virtual reality (VR) system, the system comprising:

an apparatus;

braking means for applying resistive forces to the rod in three planes that are orthogonal to one another, wherein the resistive forces oppose a direction of movement of the apparatus; and

sensor means for outputting, to a computing system, a signal that is indicative of a position of the rod and the direction of movement of the rod, wherein a computing system is configured to receive the signal and transmit a control signal to the braking means based upon the signal, wherein the control signal indicates magnitudes of the resistive forces that are to be applied to the rod by the braking means, and further wherein the braking means applies the resistive forces with the magnitudes indicated in the control signal.

19. The system of claim 18 , further comprising:

vibration means coupled to the apparatus, the vibration means configured to generate vibrations along the apparatus based upon a second control signal received from the computing system.

20. The system of claim 18 being a video game controller.

US16/747,459

2017-11-22

2020-01-20

Apparatus for use in a virtual reality system

Active

2038-04-27

US11379054B2

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US16/747,459

US11379054B2

( en )

2017-11-22

2020-01-20

Apparatus for use in a virtual reality system

Applications Claiming Priority (2)

Application Number

Priority Date

Filing Date

Title

US15/821,713

US10551940B2

( en )

2017-11-22

Apparatus for use in a virtual reality system

US16/747,459

US11379054B2

( en )

2017-11-22

2020-01-20

Apparatus for use in a virtual reality system

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US15/821,713

Continuation

US10551940B2

( en )

2017-11-22

Apparatus for use in a virtual reality system

Publications (2)

Publication Number

Publication Date

US20200150784A1

US20200150784A1 ( en )

2020-05-14

US11379054B2

true

US11379054B2 ( en )

2022-07-05

Family

ID=64572587

Family Applications (2)

Application Number

Title

Priority Date

Filing Date

US15/821,713

Active

2038-03-05

US10551940B2

( en )

2017-11-22

Apparatus for use in a virtual reality system

US16/747,459

Active

2038-04-27

US11379054B2

( en )

2017-11-22

2020-01-20

Apparatus for use in a virtual reality system

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US15/821,713

Active

2038-03-05

US10551940B2

( en )

2017-11-22

Apparatus for use in a virtual reality system

Country Status (4)

Country

Link

( 2 )

US10551940B2

( en )

( 1 )

EP3694617B1

( en )

( 1 )

KR102655149B1

( en )

( 1 )

WO2019103901A1

( en )

Families Citing this family (17)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

WO2017043101A1

( en )

2015-09-08

2017-03-16

ã½ãã¼æ ªå¼ä¼ç¤¾

Information processing device, information processing method, and program

US11497673B2

( en )

2016-11-03

2022-11-15

Wewalk Limited

Motion-liberating smart walking stick

US10551940B2

( en )

2017-11-22

2020-02-04

Microsoft Technology Licensing, Llc

Apparatus for use in a virtual reality system

FR3079424B1

( en )

2018-03-27

2021-04-23

Protubevr

CONTROLLER SUPPORT FOR VIRTUAL REALITY VIDEO GAMES

US10739857B2

( en )

2018-04-21

2020-08-11

Dale Kazdan

Wearable tracking computer

US11000442B2

( en )

2019-01-22

2021-05-11

Victor Ramos

Mobility tool

US10667073B1

( en )

2019-06-10

2020-05-26

Bose Corporation

Audio navigation to a point of interest

CN113545968A

( en )

2020-04-24

2021-10-26

äººç±»å·¥ç¨æ ªå¼ä¼ç¤¾

Energy harvesting device and smart walking stick using electromagnetic induction

US20230259197A1

( en )

2020-07-02

2023-08-17

VirtuReal Pty Ltd

A Virtual Reality System

US20220295209A1

( en )

2021-03-12

2022-09-15

Jennifer Hendrix

Smart cane assembly

US20240157245A1

( en )

2021-03-22

2024-05-16

Sony Group Corporation

Information processing apparatus, information processing method, and program

KR102526843B1

( en )

2021-04-22

2023-04-27

íêµê¸°ì êµì¡ëíêµ ì°ííë ¥ë¨

AR medium based on augmented reality

CN114721518B

( en )

2022-03-31

2024-11-15

æå°æºè½ç§ææéå¬å¸

Control method and control device of electronic interactive device, and electronic interactive device

CN114712847B

( en )

2022-03-31

2024-11-22

æå°æºè½ç§ææéå¬å¸

An electronic interactive device

CN115019591B

( en )

2022-08-05

2022-11-04

ä¸æµ·åæ¨¡ç§ææéå¬å¸

Operation simulation method, device and storage medium

US20240325902A1

( en )

2023-03-30

2024-10-03

Electronic Arts Inc.

Dynamic indications of navigation assist in virtual interactive environments

KR102788781B1

( en )

2023-07-21

2025-04-02

ê¹ì í

Self-driving walking aid device using proprioceptive sense and low-speed autonomous driving device

Citations (7)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US5767839A

( en )

1995-01-18

1998-06-16

Immersion Human Interface Corporation

Method and apparatus for providing passive force feedback to human-computer interface systems

US20050143173A1

( en )

2000-02-22

2005-06-30

Barney Jonathan A.

Magical wand and interactive play experience

US7113166B1

( en )

1995-06-09

2006-09-26

Immersion Corporation

Force feedback devices using fluid braking

US20140168176A1

( en )

2012-12-17

2014-06-19

Microsoft Corporation

Multi-purpose stylus for a computing device

US20180277292A1

( en )

2017-03-27

2018-09-27

Ecole Polytechnique Federale De Lausanne (Epfl)

Electromagnetic actuator

US20180293980A1

( en )

2017-04-05

2018-10-11

Kumar Narasimhan Dwarakanath

Visually impaired augmented reality

US20190155404A1

( en )

2017-11-22

2019-05-23

Microsoft Technology Licensing, Llc

Apparatus for use in a virtual reality system

2017

2017-11-22

US15/821,713

patent/US10551940B2/en

active

Active

2018

2018-11-15

EP18812557.9A

patent/EP3694617B1/en

active

Active

2018-11-15

KR1020207014513A

patent/KR102655149B1/en

active

Active

2018-11-15

PCT/US2018/061160

patent/WO2019103901A1/en

not_active

Ceased

2020

2020-01-20

US16/747,459

patent/US11379054B2/en

active

Active

Patent Citations (7)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US5767839A

( en )

1995-01-18

1998-06-16

Immersion Human Interface Corporation

Method and apparatus for providing passive force feedback to human-computer interface systems

US7113166B1

( en )

1995-06-09

2006-09-26

Immersion Corporation

Force feedback devices using fluid braking

US20050143173A1

( en )

2000-02-22

2005-06-30

Barney Jonathan A.

Magical wand and interactive play experience

US20140168176A1

( en )

2012-12-17

2014-06-19

Microsoft Corporation

Multi-purpose stylus for a computing device

US20180277292A1

( en )

2017-03-27

2018-09-27

Ecole Polytechnique Federale De Lausanne (Epfl)

Electromagnetic actuator

US20180293980A1

( en )

2017-04-05

2018-10-11

Kumar Narasimhan Dwarakanath

Visually impaired augmented reality

US20190155404A1

( en )

2017-11-22

2019-05-23

Microsoft Technology Licensing, Llc

Apparatus for use in a virtual reality system

Non-Patent Citations (61)

* Cited by examiner, â Cited by third party

Title

" Geomagic Touch (formerly Geomagic Phantom Omni) Overview ", Retrieved From <<http://www.geomagic.com/en/products/phantom-omni/overview>>, Retrieved on: Oct. 30, 2017, 2 Pages.

" How Virtual Reality Facilitates Social Connection ", Retrieved From <<https://www.facebook.com/iq/articles/how-virtual-reality-facilitates-social-connection>>, Jan. 9, 2017, 16 Pages.

" Igroup presence questionnaire (IPQ) Database ", Retrieved From <<http://www.igroup.org/pqlipq/data.php>>, Retrieved on: Oct. 30, 2017, 7 Pages.

" International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/06160 ", dated Feb. 20, 2019, 15 Pages.

" Long Cane Techniques, Study Guide: APH Step-by-Step ", Retrieved From <<http://tech.aph.org/sbs/04_sbs_Ic_study.html>>, Retrieved on: Oct. 30, 2017, 92 Pages.

" Non-Final Office Action for U.S. Appl. No. 15/821,713 ", dated May 6, 2019, 10 Pages.

" Notice of Allowance and Fees Due for U.S. Appl. No. 15/821,713 ", dated Sep. 20, 2019, 7 Pages.

" NovintâProducts ", Retrieved From <<http://www.novint.com/index.php/products>>, Retrieved on: Oct. 30, 2017, 3 Pages.

" Office Action Issued in European Patent Application No. 18812557.9 ", dated Apr. 11, 2022, 5 Pages.

" Phantom Force Feedback ", Retrieved From <<http://www.cgl.ucsf.edu/chimera/1.2470/docs/ContributedSoftware/phantom/phantom.html>>, Aug. 2005, 6 Pages.

" Response to the Non-Final Office Action for U.S. Appl. No. 15/821,713 ", filed Aug. 7, 2019, 10 Pages.

Ando, et al., " A Contactless Haptic Cane for Blind People ", In Proceedings of 12th joint symposium on Man Science and Measurement, Sep. 3, 2008, pp. 147-152.

Ando, et al., " A Haptic Solution to Assist Visually Impaired in Mobility Tasks ", In Journal of IEEE Transactions on Human-Machine Systems, vol. 45, Issue 5, Oct. 2015, pp. 641-646.

Bates, Joseph, " Virtual Reality, Art, and Entertainment ", In Journal of Presence: Teleoperators and Virtual Environments, vol. 1, Issue 1, Jan. 2, 1992, pp. 1-9.

Borenstein, et al., " The GuideCaneâa computerized travel aid for the active guidance of blind pedestrians ", In Proceedings of IEEE International Conference on Robotics and Automation, Apr. 21, 1997, pp. 1-6.

Choi, et al., " Wolverine: A Wearable Haptic Interface for Grasping in Virtual Reality ", In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct. 9, 2016, 8 Pages.

Colwell, et al., " Haptic Virtual Reality for Blind Computer Users ", In Proceedings of the third international ACM conference on Assistive technologies, Apr. 15, 1998, pp. 92-99.

Colwell, et al., " Use of a Haptic Device By Blind and Sighted People: Perception of Virtual Textures and Objects ", In Publication of IOS Press, Jul. 1998, 8 Pages.

GonzÃ¡lez-Mora, et al., " Development of a new space perception system for blind people, based on the creation of a virtual acoustic space ", In Proceedings of International Work-Conference on Artificial Neural Networks, Jun. 2, 1999, 10 Pages.

GonzÃ¡lez-Mora, et al., " Seeing the world by hearing: Virtual Acoustic Space (VAS) a new space perception system for blind people ", In Journal of Information and Communication Technologies, Apr. 24, 2006, 6 Pages.

Jack, et al., " Virtual Reality-Enhanced Stroke Rehabilitation ", In Journal of IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 9, No. 3, Sep. 2001, pp. 308-318.

Jansson, et al., " Haptic virtual environments for blind people: exploratory experiments with two devices ", In International Journal of Virtual Reality, vol. 3, No. 4, Retrieved on: Nov. 2, 2011, 2017, pp. 8-17.

Kalyvioti, et al., " Virtual Environments and Dyslexia: A Literature Review ", In Journal of Procedia Computer Science, vol. 27, Dec. 2014, pp. 138-147.

Kawai, et al., " Evaluation of Interactive Tactile Display System ", Retrieved From <<https://pdfs.semanticscholar.org/c8da/db1e1caa0b4e317db3bb8f22c6a45a52f8c6.pdf>> Retrieved on: Nov. 2, 2011, 8 Pages.

Kim, et al., " Drop-off Detection with the Long Cane: Effects of Different Cane Techniques on Performance ", In Journal of visual impairment & blindness, vol. 103, Issue 9, Sep. 2009, pp. 1-14.

KÃ¶nig, et al., " Haptic Exploration of Virtual Buildings Using Non-Realistic Haptic Rendering ", In Proceedings of the 7th International Conference on Computers Helping People with Special Needs, Jul. 2000, 8 Pages.

Lahav, et al., " Exploration of Unknown Spaces by People Who Are Blind Using a Multi-sensory Virtual Environment ", In Journal of Special Education Technology, vol. 19, Issue 3, Jun. 2004, pp. 15-23.

Lahav, et al., " Multisensory virtual environment for supporting blind persons' acquisition of spatial cognitive mapping, orientation, and mobility skills ", In Proceedings of the Third International Conference on Disability, Virtual Reality and Associated Technologies, Sep. 23, 2000, pp. 53-58.

LÃ©cuyer, et al., " HOMERE: a Multimodal System for Visually Impaired People to Explore Virtual Environments ", In Proceedings of IEEE Virtual Reality, Mar. 22, 2003, 8 Pages.

Maidenbaum, et al., " Increasing Accessibility to the Blind of Virtual Environments, Using a Virtual Mobility Aid Based On the "EyeCane": Feasibility Study ", In Journal of PLoS One, vol. 8, Issue 8, Aug. 19, 2013, pp. 1-7.

Martin, et al., " The smart cane: An electrical engineering design project ", In Proceedings of ASEE North Central Section Conference, Apr. 2009, pp. 1-9.

Menikdiwela, et al., " Haptic based walking stick for visually impaired people ", In Proceedings of International conference on Circuits, Controls and Communications, Dec. 27, 2013, 7 Pages.

Merabei, et al., " Audio-Based Navigation Using Virtual Environments: Combining Technology and Neuroscience ", In Journal of Research and Practice in Visual Impairment and Blindness, vol. 2, Issue 3, Sep. 2009, pp. 128-137.

Merchant, et al., " Effectiveness of virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A meta-analysis ", In Journal of Computer Education, vol. 70, Jan. 2014, pp. 29-40.

Merians, et al., " Virtual Reality-Augmented Rehabilitation for Patients Following Stroke ", In Journal of Physical Therapy, vol. 82, Issue 9, Sep. 2002, pp. 898-915.

Parsons, et al., " A Controlled Clinical Comparison of Attention Performance in Children with ADHD in a Virtual Reality Classroom Compared to Standard Neuropsychological Methods ", In Journal of Child Neuropsychology, vol. 13, Issue 4, Jul. 2007, pp. 363-381.

Pascale, et al., " Bringing Haptics to Second Life for Visually Impaired People ", In Proceedings of International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, Jun. 10, 2008, pp. 1-10.

Picinali, et al., " Exploration of architectural spaces by blind people using auditory virtual reality for the construction of spatial knowledge ", In International Journal of Human Computer Studies, vol. 72, Issue 4, Apr. 2014, pp. 393-407.

Reardon, Sara, " Playing by Ear ", In Journal of Science, vol. 333, Issue 6051, Sep. 30, 2011, pp. 1816-1818.

Romano, et al., " Creating Realistic Virtual Textures from Contact Acceleration Data ", In Journal of IEEE Transactions on Haptics, vol. 5 Issue 2, Apr. 2012, pp. 109-119.

Sanchez, et al., " Assisting the mobilization through subway networks by users with visual disabilities ", In Proceedings of the Sixth International Conference on Disability, Virtual Reality and Associated Technologies, Sep. 18, 2006, pp. 183-190.

Sanchez, et al., " Virtual Environment Interaction Through 3D Audio by Blind Children ", In Journal of Cyberpsychology & behavior: the impact of the Internet, multimedia and virtual reality on behavior and society, vol. 2, No. 2, Feb. 1999, pp. 1-12.

Schloerb, et al., " BlindAid: Virtual environment system for self-reliant trip planning and orientation and mobility training ", In Proceedings of IEEE Haptics Symposium, Mar. 25, 2010, pp. 363-370.

Schubert, et al., " The Experience of Presence: Factor Analytic Insights ", In Journal of Presence: Teleoperators and virtual environments, vol. 10, Issue 3, Jun. 2001, pp. 266-281.

Seki, et al., " A training system of orientation and mobility for blind people using acoustic virtual reality ", In Proceedings of IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 19, Issue 1, Feb. 2011, pp. 95-104.

Semwal. S. K., " MoVE: Mobiltiy Training in Haptic Virtual Environment ", Retrieved From <<https://pdfs.semanticscholar.org/243e/b3d64990f34d0b126b36132acc17e4f50737.pdf>>, Retrieved on: Oct. 30, 2017, pp. 1-18.

Singh, et al., " "Smart" Cane for the Visually Impaired: Design and Controlled Field Testing of an Affordable Obstacle Detection System ", In Proceedings of 12th International Conference on Mobility and Transport for Elderly and Disabled Persons, vol. 53, Issue 9, Jun. 2010, 13 Pages.

Sveistrup, Heidi, " Motor rehabilitation using virtual reality ", In Journal of NeuroEngineering and Rehabilitation, vol. 1, Issue 1, Dec. 10, 2004, 8 Pages.

Tatsumi, et al., " Cane Walk in the Virtual Reality Space Using Virtual Haptic Sensing ", In Proceedings of International Conference on Systems, Man, and Cybernetics, Oct. 9, 2015, pp. 2360-2365.

Thaler, et al., " Mouth-clicks used by blind expert human echolocatorsâsignal description and model based signal synthesis ", In Journal of PLoS Computational Biology, vol. 13, No. 8, Aug. 31, 2017, pp. 1-17.

Tiponut, et al., " Acoustic Virtual Reality Performing Man-machine Interfacing of the Blind ", In Proceedings of 12th WSEAS International Conference on Systems, Jul. 22, 2008, pp. 345-349.

Torres-Gil, et al., " Applications of Virtual Reality for Visually Impaired People ", In Proceedings of WSEAS Transactions on Computers, vol. 9, Issue 2, Feb. 2010, pp. 184-193.

Tzovaras, et al., " Design and implementation of virtual environments training of the visually impaire ", In Proceedings of the fifth international ACM conference on Assistive technologies, Jul. 8, 2002, pp. 41-48.

Tzovaras, et al., " Interactive mixed reality white cane simulation for the training of the blind and the visually impaired ", In Journal of Personal and Ubiquitous Computing, vol. 13, Issue 1, Jan. 2009, pp. 51-58.

Ulrich, et al., " The GuideCaneâApplying Mobile Robot Technologies to Assist the Visually Impaired ", In Journal of IEEE Transactions on Systems, Man, and Cybernetics,âPart A: Systems and Humans, vol. 31, No. 2, Mar. 2001, pp. 1-12.

Virvou, et al., " On the usability and likeability of virtual reality games for education: The case of VR-ENGAGE ", In Journal of Computers and Education, vol. 50, Issue 1, Jan. 31, 2008, pp. 1-22.

Wahab, et al., " Smart Cane: Assistive Cane for Visually-impaired People ", In International Journal of Computer Science Issues, vol. 8, Issue 4, Jul. 2011, pp. 21-27.

White, et al., " Toward accessible 3D virtual environments for the blind and visually impaired ", In Proceedings of the 3rd international conference on Digital Interactive Media in Entertainment and Arts, Sep. 10, 2008, 8 Pages.

Williams, et al., " "Pray Before You Step Out": Describing Personal and Situational Blind Navigation Behaviors ", In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility, Oct. 21, 2013, 8 Pages.

Williams, et al., " Just let the cane hit it ": how the blind and sighted see navigation differently, In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility, Oct. 20, 2014, pp. 217-224.

Zyda, Michael, " From Visual Simulation to Virtual Reality to Games ", In Journal of Computer, vol. 38, Issue 9, Sep. 2005, pp. 25-32.

Also Published As

Publication number

Publication date

US20200150784A1

( en )

2020-05-14

US10551940B2

( en )

2020-02-04

KR20200089675A

( en )

2020-07-27

EP3694617A1

( en )

2020-08-19

WO2019103901A1

( en )

2019-05-31

EP3694617B1

( en )

2023-09-06

KR102655149B1

( en )

2024-04-05

US20190155404A1

( en )

2019-05-23

Apparatus for use in a virtual reality system — Microsoft Technology Licensing, Llc (US11379054B2)

Related documents