ABSTRACT
Abstract
A surgery visualization theatre comprising: an augmented/extended reality (AXR) headset; a digital viewport mounted on a cobotic arm; a monitor mounted on a monitor cobotic arm; a camera subsystem mounted on a camera cobotic arm; and a frame with cobotic arms featuring intelligence and command and control for the system and visualization methodologies, where the digital viewport cobotic arm, the monitor cobotic arm, and the camera cobotic arm are mounted on the frame and the AXR headset is connected thereto.
Description
CROSS REFERENCE
This application is based on and claims priority to U.S. Provisional Patent Application No. 63/019,796 filed May 4, 2020. It is also a continuation-in-part of U.S. patent application Ser. No. 15/073,144 filed Mar. 17, 2016, which issued on May 1, 2018 as U.S. Pat. No. 9,955,862, U.S. patent application Ser. No. 15/940,561 filed Mar. 29, 2018, which issued on Oct. 30, 2018 as U.S. Pat. No. 10,111,583, U.S. patent application Ser. No. 16/173,719 filed Oct. 29, 2018, which issued as U.S. Pat. No. 10,874,297 on Dec. 29, 2020, U.S. patent application Ser. No. 17/137,069 filed Dec. 29, 2020, U.S. patent application Ser. No. 17/137,093 filed Dec. 29, 2020, and U.S. patent application Ser. No. 17/151,174 filed Jan. 17, 2021, all of which claim the benefit of U.S. Provisional Patent Application No. 62/134,422 filed Mar. 17, 2015; of U.S. patent application Ser. No. 15/962,661 filed Apr. 25, 2018, which claims the benefit of U.S. Provisional Patent Application No. 62/489,801 filed Apr. 25, 2017; of U.S. patent application Ser. No. 16/511,202 filed Jul. 15, 2019, which claims the benefit of U.S. Provisional Patent Application No. 62,697,854 filed Jul. 13, 2018; of U.S. patent application Ser. No. 16/511,451 filed Jul. 15, 2019; of U.S. patent application Ser. No. 17/034,944 filed Sep. 28, 2020, which claims the benefit of U.S. Provisional Patent Application No. 62/907,300 filed Sep. 27, 2019; and of U.S. patent application Ser. No. 17/182,022 filed Feb. 22, 2021, which claims the benefit of U.S. Provisional Patent Application No. 62/979,999 filed Feb. 21, 2020, U.S. Provisional Patent Application No. 62/986,461 filed Mar. 6, 2020, U.S. Provisional Patent Application No. 63/005,202 filed Apr. 3, 2020, and U.S. Provisional Patent Application No. 63/019,796 filed May 4, 2020. All are incorporated herein by reference.
COPYRIGHT NOTICE
A portion of this disclosure contains material that is subject to copyright or trademark protection. The copyright and trademark owner has no objection to the facsimile reproduction by anyone of this patent document as it appears in the U.S. Patent and Trademark Office, patent file or records, but reserves all copyrights whatsoever in the subject matter presented herein. The trademark names of the systems herein are those selected by the inventors but are not exclusive of names which could be used.
BACKGROUND OF THE INVENTION
Field of the Invention
This invention relates generally to a surgery visualization theatre, with optional methods to view a surgery video feed, and more particularly, but not by way of limitation, to a surgery visualization theatre featuring an augmented reality headset, a virtual reality viewport, a 3D autostereoscopic no glasses monitor, and an all- digital 12K 3D microscope wherein the virtual reality viewport, autostereoscopic monitor, and 3D microscope are configured on cobotic arms. While the preferred embodiment for the surgery visualization theatre would provide all three digital viewing options, the system could comprise less than all three viewing options, or each of the viewing options may be used independently of the others. Further, while this system was first originated for medical and surgery applications, it could also be used in other sectors, for other non-surgical applications such as industrial applications where a real-time video feed is necessary, and several viewing options for the viewer or operator are advantageous.
Description of the Related Art
There is a need for surgeons to be able to access real-time and pre-recorded computer-generated and/or camera images in the operating room. In addition, the operating room (OR) healthcare professionals need multiple viewing ports providing 3D surgery site information to such personnel for them to accomplish their OR tasks. Since the 1800s, surgeons have relied on the standard optical microscope (SOM), which uses multiple lenses in a dual tube configuration to create optical zoom and magnification designed to be viewed by a human operator. In addition, OR personnel also need to be able to recall electronic medical record (EMR) information during surgery, such as 3D MRI's, CT scans, and other imaging, which currently is done through 2D viewing on a light board in the operating room or through multiple monitors throughout the OR, or in adjacent clinic/diagnostic rooms. In the recent past, some large medical device manufactures have introduced SOM's that project video through a wired connection onto 3D monitors which require the OR personnel to wear polarized 3D glasses, and which require the surgeon to strain, crane, or lean for hours to observe. As the 3D glasses are polarized up to 50%, this is somewhat akin to having a surgeon wearing sunglasses to surgery and does not provide the surgeon with good ergonomics or state-of-the-art digital technologies.
Recent trends in the field have shown some SOM manufacturers adding digital imaging to the legacy equipment, but this means that an already large SOM becomes larger and more cumbersome and takes ever larger highly-valued surgery space away from the OR personnel.
Further, a typical surgical SOM is traditionally limited by the resolution of the observer's eye with an upper bound of the retinal resolution. In contrast, a digital microscope can use algorithms to detect features that are imperceptible to the eye. Moreover, the typical SOM does not have the same zoom range, typically only 6Ã, as a digital microscope because a digital microscope has both optical and digital zoom, which can achieve magnifications of up to 500Ã.
Beyond this, a SOM cannot provide the best three-dimensional renderings. Although when using a stereo SOM, an operator can see 3D depth owing to the fact that the operator is receiving two slightly different views of the object under study through the ocular eyepieces, this distance is limited and based on maximum interpupillary distance (IPD) of a human's eye, while with a 3DAMD microscope one can adjust the system irrespective of IPD and apply a greater physical distance between the sensors and/or apply disparity 3D imaging algorithms, which makes for better quality 3D. In addition, a SOM requires the user to have a limited, single field-of-view at high zoom, meaning that the context tends to be lost, while with a digital microscope, you can have a picture-in-picture or picture-in-picture which references where you are in the larger structure to be viewed at higher zoom.
Even though some SOM systems provide the advantage of offering an ocular only view or a digital view, or both, the result comes with limitations in the form of posture constraints, which typically forces the surgeon to unergonomic positions for hours of surgery use, often causing neck, back, hip, and hamstring strains. Many surgeons find they have to retire early due to such stress and strain or must undergo surgery themselves to alleviate work-related musculoskeletal disorders (WMSDs), which are prevalent among surgeons and often result in practice modification. Some medical investigations show that WMSD's exist in as much as 66% to 94% of surgeons practicing open surgeries, with surgeons showing 73% to 100% WMSD's for conventional laparoscopic surgeries, and a rate of 54% to 87% for vaginal surgeries, and 23% to 80% of WMSD's reported from surgeons involved in robotic-assisted surgery.
Risk factors for WMSD's include the use of loupes, headlamps, OM microscopes, and robotic-type assisted surgery systems that have control booths where the operating surgeon is totally divorced from the patient and surgery site. In the instance of the robotic systems, the surgeon is typically secluded in an enclosed console, which may or may not be in the same room with the patient. So, the surgeon typically must depend on the surgery techs, who are physically present with the patient, to tell the surgeon if the robotic arms are in conflict, or if there is an unexpected patient event, such as a bleed. In the case of an unexpected patient event, typically the surgeon is dependent on a technician to advise him of that occurrence, and then the surgeon has to have the techs raise the robotic arms, while the surgeon is scrubbing up to sterilize to render aid. With the AXR technology of the present invention, such remoteness of the surgeon is overcome.
Moreover, the huge space which is needed for the existing robotic and legacy surgery equipment may compromise the surgeon's ability to view digitally or have the space needed for positioning during surgery, with one report concluding that: âFuture research must aim to develop objective surgical ergonomics instruments and guidelines and to correlate ergonomics assessments with pain and tissue-level damage in surgeons with WMSDs. Ergonomics training should be developed to protect surgeons from preventable, potentially career-altering injuries.â (Catanzarite T, Tan-Kim J, Whitcomb EL, Menefee S. Ergonomics in Surgery: A Review. Female Pelvic Med Reconstr Surg. 2018 January/February; 24(1):1-12. doi: 10.1097/SPV.0000000000000456. PMID: 28914699.)
Thus, the legacy systems and their add-ons have not provided the OR personnel with the best state-of-the-art in digital 3D technologies and better ergonomics that are provided by employing the multi-option fully digital 3D cobotic surgical system as set out herein.
Others have attempted to employ virtual reality (VR) surgical or diagnostic headsets. However, a VR headset totally immerses the user into the images presented, essentially totally blocking and replacing the user's field of vision of the real-world, often called the real-reality (RR), with virtual images and the impossibility to see the virtual image and the RR around the OR personnel. Such systems are defined by their replacement of the reality around the user with a total virtual substitute. This immersion locks the surgeon into a virtual space that is not easy to extract from in case of an urgency or emergency. The invention viewing option disclosed following transverses these limitations.
Such existing virtual reality surgery systems are generally uncomfortable and must be worn tight on the head, blocking out reality. VR systems seal out real-word light, sound, and the air around the surgeon's eyes and cheeks, making the device hot and uncomfortable. The heat generated by the surgeon wearing the VR headset and from the headset itself often causes condensation on the interior lenses, which makes the images appear foggy and requires the surgeon to take of the VR headset for cleaning during the surgery. Clearing the lenses typically only helps temporarily. Some such systems use a trackpad that is turned 90 degrees from the user interface, so that swiping forward actually moves right and swiping backward moves left. This can be frustrating for the user, particularly if the user is left-handed. Moreover, typing within a VR headset menu is a painstaking and time-consuming chore, making entering HIPPA compliant passwords for sensitive data difficult. Furthermore, such virtual reality systems are typically heavy, with most of the weight forward on the head, making it uncomfortable for the user.
To address these concerns, augmented/extended reality (AXR) surgical systems have been introduced for surgical use. Whereas virtual reality immerses the user into the images presented and closes RR, AXR permits the surgeon, nurse, assistant, or tech user to see RR and what is actually happening in the user's world and then adds computer-generated, computer-manipulated, or secondary camera images to RR. Thus, while virtual reality completely covers and replaces the user's field-of-vision with virtual images, augmented/extended reality provides the user with vision of the real-world plus an overlay of computer-generated and/or manipulated photographic imagery or video (âvirtualâ) images, which positions the user in the RR with virtual images added.
In an operating environment, an augmented/extended reality system permits the surgeon to both view and have magnified the virtual (streaming surgery) image or video of the operation site, while still having a visual sense of the operating or diagnostic room and being with all the other things happening in that space. The problem with current AXR surgical systems that typically rely on waveguide technologies or laser beam scanning plus waveguide technology is that they all offer a small field-of-view (FoV) and limited resolution and are typically waveguide or laser beam scanning technology combined with a waveguide. In addition, they typically are on a heavy wearable that is often tethered to the system by a large cord, limiting the surgeon's movements and putting strain on the surgeon's neck and back. Furthermore, current AXR surgical systems must block out a great deal of ambient light to make the AXR images visible and are difficult to see in daylight or highly-lighted conditions, making the systems function more like a virtual reality system than an AXR system, which erases the benefit of AXR.
Based on the foregoing, it is desirable to provide a true AXR surgery system that provides a plurality of potential video or image feeds or overlays of computer-generated images while maintaining a sufficient real-world view. Specifically, it is desirable to not use waveguide technologies but, rather use an AXR headset which is a wearable pupil-forming display apparatus, comprised of two axially symmetric pupil expanding ocular engine with a folded prism so that the micro-displays, which are the warmest electronic in the headset, are the furthers away from the wearer's head and body.
It is further desirable for the AXR headset be lightweight, comfortable, untethered, and is feature- and user-friendly
It is further desirable for the AXR headset to offer a wide FOV and high resolution, which is not currently possible with AR waveguide technology or laser beam scanning technologies combined with waveguides. Thus, in the preferred embodiment, the AXR headset may have it optical engine based on near-eye pupil forming catadioptric optical engine, which provides a wide field-of-view and pixel accurate photo-realistic imaging. By using this system with OLED micro-displays, it is easy to adapt this type of optical engine when the next level of imagers, the micro-LED engines to come out in the future. The near-eye pupil forming catadioptric optical engine may be designed with a bird-bath design so that new display technologies are adapted by OLED replacement LED plug and play technologies.
While the focus of this invention is on its application to the medical and surgical fields, it is further desirable for the same techniques to be utilized in other sectors while wearing a lightweight, comfortable, untethered, feature- and user-friendly AXR headset would be of benefit.
Further, the surgery visualization system presented herein may provide other viewing options besides the AXR wireless headset, such as (i) an autostereoscopic monitor featuring lenticular lenses or parallax barrier which does not need polarized 3D glasses for the user to view the surgery image in 3D, which may be mounted on a cobotic arm; (ii) a 3D digital viewport (3DDV) device, which may be mounted on a cobotic arm which may traverse to advance to the OR personnel and then, through sensor technologies, moves with user as his or her posture changes or slouches during surgery. Some of the sensor technologies which may be employed are SLAM technologies, facial-recognition technologies, head-tracking, eye-tracking technologies, and time of flight technologies. The one or more 3D viewing options may be provided in the first instance by a new type of 3D all-digital stereo microscope (3DADM) which may feature two full-frame 35 mm or higher 4K, 6K, or higher resolution sensors with large pixel size of 5 microns or more, providing as much as 69 billion possible colors, which is more than the human eye can distinguish, which also may be mounted on a cobotic arm. The surgery visualization theatre may also comprise at least one computer and graphics unit, together with multiple methods of transmission including wire connected transmission; or any of the existing wireless transmission technologies. The surgery visualization theatre may also comprise a model controller to control the components and the digital microscope and viewing options to keep them in sync with each other.
SUMMARY OF THE INVENTION
In general, in a first aspect, the invention relates to an all- digital multi-option 3D viewing theatre (ADMO3DV). The surgery visualization theatre may offer multiple options for viewing a surgery feed in 3D comprising: an augmented/exte
CROSS REFERENCE
This application is based on and claims priority to U.S. Provisional Patent Application No. 63/019,796 filed May 4, 2020. It is also a continuation-in-part of U.S. patent application Ser. No. 15/073,144 filed Mar. 17, 2016, which issued on May 1, 2018 as U.S. Pat. No. 9,955,862, U.S. patent application Ser. No. 15/940,561 filed Mar. 29, 2018, which issued on Oct. 30, 2018 as U.S. Pat. No. 10,111,583, U.S. patent application Ser. No. 16/173,719 filed Oct. 29, 2018, which issued as U.S. Pat. No. 10,874,297 on Dec. 29, 2020, U.S. patent application Ser. No. 17/137,069 filed Dec. 29, 2020, U.S. patent application Ser. No. 17/137,093 filed Dec. 29, 2020, and U.S. patent application Ser. No. 17/151,174 filed Jan. 17, 2021, all of which claim the benefit of U.S. Provisional Patent Application No. 62/134,422 filed Mar. 17, 2015; of U.S. patent application Ser. No. 15/962,661 filed Apr. 25, 2018, which claims the benefit of U.S. Provisional Patent Application No. 62/489,801 filed Apr. 25, 2017; of U.S. patent application Ser. No. 16/511,202 filed Jul. 15, 2019, which claims the benefit of U.S. Provisional Patent Application No. 62,697,854 filed Jul. 13, 2018; of U.S. patent application Ser. No. 16/511,451 filed Jul. 15, 2019; of U.S. patent application Ser. No. 17/034,944 filed Sep. 28, 2020, which claims the benefit of U.S. Provisional Patent Application No. 62/907,300 filed Sep. 27, 2019; and of U.S. patent application Ser. No. 17/182,022 filed Feb. 22, 2021, which claims the benefit of U.S. Provisional Patent Application No. 62/979,999 filed Feb. 21, 2020, U.S. Provisional Patent Application No. 62/986,461 filed Mar. 6, 2020, U.S. Provisional Patent Application No. 63/005,202 filed Apr. 3, 2020, and U.S. Provisional Patent Application No. 63/019,796 filed May 4, 2020. All are incorporated herein by reference.
COPYRIGHT NOTICE
A portion of this disclosure contains material that is subject to copyright or trademark protection. The copyright and trademark owner has no objection to the facsimile reproduction by anyone of this patent document as it appears in the U.S. Patent and Trademark Office, patent file or records, but reserves all copyrights whatsoever in the subject matter presented herein. The trademark names of the systems herein are those selected by the inventors but are not exclusive of names which could be used.
BACKGROUND OF THE INVENTION
Field of the Invention
This invention relates generally to a surgery visualization theatre, with optional methods to view a surgery video feed, and more particularly, but not by way of limitation, to a surgery visualization theatre featuring an augmented reality headset, a virtual reality viewport, a 3D autostereoscopic no glasses monitor, and an all- digital 12K 3D microscope wherein the virtual reality viewport, autostereoscopic monitor, and 3D microscope are configured on cobotic arms. While the preferred embodiment for the surgery visualization theatre would provide all three digital viewing options, the system could comprise less than all three viewing options, or each of the viewing options may be used independently of the others. Further, while this system was first originated for medical and surgery applications, it could also be used in other sectors, for other non-surgical applications such as industrial applications where a real-time video feed is necessary, and several viewing options for the viewer or operator are advantageous.
Description of the Related Art
There is a need for surgeons to be able to access real-time and pre-recorded computer-generated and/or camera images in the operating room. In addition, the operating room (OR) healthcare professionals need multiple viewing ports providing 3D surgery site information to such personnel for them to accomplish their OR tasks. Since the 1800s, surgeons have relied on the standard optical microscope (SOM), which uses multiple lenses in a dual tube configuration to create optical zoom and magnification designed to be viewed by a human operator. In addition, OR personnel also need to be able to recall electronic medical record (EMR) information during surgery, such as 3D MRI's, CT scans, and other imaging, which currently is done through 2D viewing on a light board in the operating room or through multiple monitors throughout the OR, or in adjacent clinic/diagnostic rooms. In the recent past, some large medical device manufactures have introduced SOM's that project video through a wired connection onto 3D monitors which require the OR personnel to wear polarized 3D glasses, and which require the surgeon to strain, crane, or lean for hours to observe. As the 3D glasses are polarized up to 50%, this is somewhat akin to having a surgeon wearing sunglasses to surgery and does not provide the surgeon with good ergonomics or state-of-the-art digital technologies.
Recent trends in the field have shown some SOM manufacturers adding digital imaging to the legacy equipment, but this means that an already large SOM becomes larger and more cumbersome and takes ever larger highly-valued surgery space away from the OR personnel.
Further, a typical surgical SOM is traditionally limited by the resolution of the observer's eye with an upper bound of the retinal resolution. In contrast, a digital microscope can use algorithms to detect features that are imperceptible to the eye. Moreover, the typical SOM does not have the same zoom range, typically only 6Ã, as a digital microscope because a digital microscope has both optical and digital zoom, which can achieve magnifications of up to 500Ã.
Beyond this, a SOM cannot provide the best three-dimensional renderings. Although when using a stereo SOM, an operator can see 3D depth owing to the fact that the operator is receiving two slightly different views of the object under study through the ocular eyepieces, this distance is limited and based on maximum interpupillary distance (IPD) of a human's eye, while with a 3DAMD microscope one can adjust the system irrespective of IPD and apply a greater physical distance between the sensors and/or apply disparity 3D imaging algorithms, which makes for better quality 3D. In addition, a SOM requires the user to have a limited, single field-of-view at high zoom, meaning that the context tends to be lost, while with a digital microscope, you can have a picture-in-picture or picture-in-picture which references where you are in the larger structure to be viewed at higher zoom.
Even though some SOM systems provide the advantage of offering an ocular only view or a digital view, or both, the result comes with limitations in the form of posture constraints, which typically forces the surgeon to unergonomic positions for hours of surgery use, often causing neck, back, hip, and hamstring strains. Many surgeons find they have to retire early due to such stress and strain or must undergo surgery themselves to alleviate work-related musculoskeletal disorders (WMSDs), which are prevalent among surgeons and often result in practice modification. Some medical investigations show that WMSD's exist in as much as 66% to 94% of surgeons practicing open surgeries, with surgeons showing 73% to 100% WMSD's for conventional laparoscopic surgeries, and a rate of 54% to 87% for vaginal surgeries, and 23% to 80% of WMSD's reported from surgeons involved in robotic-assisted surgery.
Risk factors for WMSD's include the use of loupes, headlamps, OM microscopes, and robotic-type assisted surgery systems that have control booths where the operating surgeon is totally divorced from the patient and surgery site. In the instance of the robotic systems, the surgeon is typically secluded in an enclosed console, which may or may not be in the same room with the patient. So, the surgeon typically must depend on the surgery techs, who are physically present with the patient, to tell the surgeon if the robotic arms are in conflict, or if there is an unexpected patient event, such as a bleed. In the case of an unexpected patient event, typically the surgeon is dependent on a technician to advise him of that occurrence, and then the surgeon has to have the techs raise the robotic arms, while the surgeon is scrubbing up to sterilize to render aid. With the AXR technology of the present invention, such remoteness of the surgeon is overcome.
Moreover, the huge space which is needed for the existing robotic and legacy surgery equipment may compromise the surgeon's ability to view digitally or have the space needed for positioning during surgery, with one report concluding that: âFuture research must aim to develop objective surgical ergonomics instruments and guidelines and to correlate ergonomics assessments with pain and tissue-level damage in surgeons with WMSDs. Ergonomics training should be developed to protect surgeons from preventable, potentially career-altering injuries.â (Catanzarite T, Tan-Kim J, Whitcomb EL, Menefee S. Ergonomics in Surgery: A Review. Female Pelvic Med Reconstr Surg. 2018 January/February; 24(1):1-12. doi: 10.1097/SPV.0000000000000456. PMID: 28914699.)
Thus, the legacy systems and their add-ons have not provided the OR personnel with the best state-of-the-art in digital 3D technologies and better ergonomics that are provided by employing the multi-option fully digital 3D cobotic surgical system as set out herein.
Others have attempted to employ virtual reality (VR) surgical or diagnostic headsets. However, a VR headset totally immerses the user into the images presented, essentially totally blocking and replacing the user's field of vision of the real-world, often called the real-reality (RR), with virtual images and the impossibility to see the virtual image and the RR around the OR personnel. Such systems are defined by their replacement of the reality around the user with a total virtual substitute. This immersion locks the surgeon into a virtual space that is not easy to extract from in case of an urgency or emergency. The invention viewing option disclosed following transverses these limitations.
Such existing virtual reality surgery systems are generally uncomfortable and must be worn tight on the head, blocking out reality. VR systems seal out real-word light, sound, and the air around the surgeon's eyes and cheeks, making the device hot and uncomfortable. The heat generated by the surgeon wearing the VR headset and from the headset itself often causes condensation on the interior lenses, which makes the images appear foggy and requires the surgeon to take of the VR headset for cleaning during the surgery. Clearing the lenses typically only helps temporarily. Some such systems use a trackpad that is turned 90 degrees from the user interface, so that swiping forward actually moves right and swiping backward moves left. This can be frustrating for the user, particularly if the user is left-handed. Moreover, typing within a VR headset menu is a painstaking and time-consuming chore, making entering HIPPA compliant passwords for sensitive data difficult. Furthermore, such virtual reality systems are typically heavy, with most of the weight forward on the head, making it uncomfortable for the user.
To address these concerns, augmented/extended reality (AXR) surgical systems have been introduced for surgical use. Whereas virtual reality immerses the user into the images presented and closes RR, AXR permits the surgeon, nurse, assistant, or tech user to see RR and what is actually happening in the user's world and then adds computer-generated, computer-manipulated, or secondary camera images to RR. Thus, while virtual reality completely covers and replaces the user's field-of-vision with virtual images, augmented/extended reality provides the user with vision of the real-world plus an overlay of computer-generated and/or manipulated photographic imagery or video (âvirtualâ) images, which positions the user in the RR with virtual images added.
In an operating environment, an augmented/extended reality system permits the surgeon to both view and have magnified the virtual (streaming surgery) image or video of the operation site, while still having a visual sense of the operating or diagnostic room and being with all the other things happening in that space. The problem with current AXR surgical systems that typically rely on waveguide technologies or laser beam scanning plus waveguide technology is that they all offer a small field-of-view (FoV) and limited resolution and are typically waveguide or laser beam scanning technology combined with a waveguide. In addition, they typically are on a heavy wearable that is often tethered to the system by a large cord, limiting the surgeon's movements and putting strain on the surgeon's neck and back. Furthermore, current AXR surgical systems must block out a great deal of ambient light to make the AXR images visible and are difficult to see in daylight or highly-lighted conditions, making the systems function more like a virtual reality system than an AXR system, which erases the benefit of AXR.
Based on the foregoing, it is desirable to provide a true AXR surgery system that provides a plurality of potential video or image feeds or overlays of computer-generated images while maintaining a sufficient real-world view. Specifically, it is desirable to not use waveguide technologies but, rather use an AXR headset which is a wearable pupil-forming display apparatus, comprised of two axially symmetric pupil expanding ocular engine with a folded prism so that the micro-displays, which are the warmest electronic in the headset, are the furthers away from the wearer's head and body.
It is further desirable for the AXR headset be lightweight, comfortable, untethered, and is feature- and user-friendly
It is further desirable for the AXR headset to offer a wide FOV and high resolution, which is not currently possible with AR waveguide technology or laser beam scanning technologies combined with waveguides. Thus, in the preferred embodiment, the AXR headset may have it optical engine based on near-eye pupil forming catadioptric optical engine, which provides a wide field-of-view and pixel accurate photo-realistic imaging. By using this system with OLED micro-displays, it is easy to adapt this type of optical engine when the next level of imagers, the micro-LED engines to come out in the future. The near-eye pupil forming catadioptric optical engine may be designed with a bird-bath design so that new display technologies are adapted by OLED replacement LED plug and play technologies.
While the focus of this invention is on its application to the medical and surgical fields, it is further desirable for the same techniques to be utilized in other sectors while wearing a lightweight, comfortable, untethered, feature- and user-friendly AXR headset would be of benefit.
Further, the surgery visualization system presented herein may provide other viewing options besides the AXR wireless headset, such as (i) an autostereoscopic monitor featuring lenticular lenses or parallax barrier which does not need polarized 3D glasses for the user to view the surgery image in 3D, which may be mounted on a cobotic arm; (ii) a 3D digital viewport (3DDV) device, which may be mounted on a cobotic arm which may traverse to advance to the OR personnel and then, through sensor technologies, moves with user as his or her posture changes or slouches during surgery. Some of the sensor technologies which may be employed are SLAM technologies, facial-recognition technologies, head-tracking, eye-tracking technologies, and time of flight technologies. The one or more 3D viewing options may be provided in the first instance by a new type of 3D all-digital stereo microscope (3DADM) which may feature two full-frame 35 mm or higher 4K, 6K, or higher resolution sensors with large pixel size of 5 microns or more, providing as much as 69 billion possible colors, which is more than the human eye can distinguish, which also may be mounted on a cobotic arm. The surgery visualization theatre may also comprise at least one computer and graphics unit, together with multiple methods of transmission including wire connected transmission; or any of the existing wireless transmission technologies. The surgery visualization theatre may also comprise a model controller to control the components and the digital microscope and viewing options to keep them in sync with each other.
SUMMARY OF THE INVENTION
In general, in a first aspect, the invention relates to an all- digital multi-option 3D viewing theatre (ADMO3DV). The surgery visualization theatre may offer multiple options for viewing a surgery feed in 3D comprising: an augmented/extended reality (AXR) headset which may be preferably connected wirelessly to the theatre, but may alternatively be wired; a 3D digital viewport, which may be mounted on a cobotic arm; an autostereoscopic monitor, which may be mounted on a monitor cobotic arm; a camera microscope subsystem, which may be mounted on a camera cobotic arm; and a frame, where the microscope cobotic arm, the monitor cobotic arm, and the camera cobotic arm are mounted on the frame. The frame may cantilever over the gurney and use any of hydraulic, pneumatic, electronic actuator, springs, or band to control the movement of the cobotic arms, which may be 6 axes more or less. The cobotic arms may be cartesian, SCARA, cylindrical, delta, polar, or vertically articulated or other robotic mechanism.
The AXR headset may have two micro-displays and each of the two micro-displays may have a resolution of at least 4K. The micro-displays may be capable of active pixel phase shift. The 3D digital viewport may have two micro-displays and each of the two micro-displays may have a resolution of at least 4K. The 3D digital viewport's micro-displays may be capable of active pixel phase shift. The 3D digital viewport mounted on the cobotic arm may be a six-axis more or less cobotic arms. The monitor may be a 3D autostereoscopic glasses-free monitor, meaning that no 3D glasses need be worn by the viewer(s), capable of providing, in the best embodiment, at least 4K resolution to each eye of a user.
The 3DADM subsystem may comprise two 4K 6K, 8K, or higher resolution cameras and a light source.
The monitor cobotic arm and/or the camera cobotic arm may be any type of robotic arm described herein. The frame may be capable of swiveling and reversing for right or left-handed use.
The surgery visualization theatre may further comprise and include a 3DDV with embedded SLAM technology and a plurality of sensors such that the system is capable of moving the 3DDV oculars on the cobotic arm to position the digital viewport in front of a user's face upon a cue by the user and then through the use of SLAM, time-of-flight (ToF), depth estimation from stereo cameras, a combination of ultrasonic sensors with single camera face tracking, to keep the digital viewport always moving in alignment with the surgeon as he moves, slouches, or adjust during a surgery. The digital viewport cobotic arm, the monitor cobotic arm, and/or the camera cobotic arm may use a low-friction, gravity compensated controller The digital viewport may comprise one or more eye cups or oculars for the user to place his eyes against, essentially like a surgeon would do to view images with a SOM, except that the surgeon would be, in this instance, looking at micro-displays projecting the surgery image or video feed to his eyes. In addition, the digital viewport may have an automatic or manual adjustment for interpupillary distance (IPD) so that when viewed with two eyes in both oculars the image presented is a 3D image.
The AXR headset may be capable of protecting a user's eyes while in the vicinity of medical lasers with either a film or liquid crystal layer on the exterior of the AXR headset, which may be comprised of a collector lens and an additional external lens which contains the film or liquid crystals. The AXR headset may further comprise a clip-on face mask, which may be an N1 filtration capability mask and/or use photocatalytic (UV energized titanium dioxide coated surface) active pathogen oxidation techniques. The clip-on face mask may be capable of using a combination of photocatalytic pathogen oxidization and post filtration to capture any spuriously created ozone before inhalation.
The 3DADM may contain a lens magnification and focusing system, and dual sensors for 3D viewing, software, algorithms, and processing capability, including a model controller, computer vision technologies, and computer graphics technologies. Hardware may include a series of lens beginning with an objective lens, which both camera channels share, from there the dual optical channels may magnify a certain specified amount in tandem, including optical zoom. The sensors may be the end of the optical engine and receive the information for processing and sending for viewing. In one embodiment, the sensors may be rotated on the aperture azimuth for viewing at multiple azimuth degrees. The 3DADM may contain internal lighting or can be used with external lighting like which is used in vitreoretinal surgeries.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a perspective view of the AXR surgical system in use;
FIG. 2 is a perspective view of the AXR surgical system headset;
FIG. 3 is a front view of the AXR surgical system headset;
FIG. 4 is an exploded view of the lenses;
FIG. 5 is a diagrammatic illustration of an eye box;
FIG. 6 is a diagrammatic view of the micro-displays;
FIG. 7 is a break-out of the near-eye pupil-forming catadioptric optical engine in the AXR headset;
FIG. 8 is a close-up view of the dynamic opacity and the rotating articulation of the dual headset cameras up to 90 degrees;
FIG. 9 is a view of the overall ADMO3DV surgery suite with its 3D microscope and three (3) 3D viewing options;
FIG. 10 is a view of the 3DDV mounted on a cobotic arm;
FIG. 11 is a view of the autostereoscopic 3D âglasses freeâ monitor on the system;
FIG. 12 is a diagrammatic view of the 3DADM dual camera sensor microscope with internal and external lighting options;
FIG. 13 is a back view of a person wearing the AXR headset, illustrating different views presented by the virtual overlay;
FIG. 14 is a perspective view of the charging cabinet housing the control system and computer and depicting the AXR headsets being charged and uploaded with surgery imaging data and EMR information;
FIG. 15 is a perspective view of the smart pedestal of the surgery system showing its automatic change of position upon command;
FIG. 16 is a perspective view of the AXR surgical system headset with a visor-type design;
FIG. 17 is a diagrammatic view of the picture-in-picture portion of the virtual overlay software and hardware, showing a surgeon at full magnification a smaller picture of the entire FOV for better orientation;
FIG. 18 is a perspective view of the surgery visualization theatre;
FIG. 19 is a diagrammatic illustration showing how lenticular lens placement may create a glasses-free 3D image on a television/display;
FIG. 20 is a diagrammatic illustration of the picture-on-picture superimposed imaging technology;
FIG. 21 is a diagrammatic illustration of the remote surgery assist feature with both surgeons seeing the same virtual information and area of interest;
FIG. 22 is a series of perspective views of the cabinet and drawer system;
FIG. 23 is a back perspective view of the AXR headset with a larger battery box;
FIG. 24 is a front perspective view of the AXR headset with the larger battery box; and
FIG. 25 is a view of the 3DADM lens system with dual optical systems.
Other advantages and features will be apparent from the following description and from the claims.
DETAILED DESCRIPTION OF THE INVENTION
The devices and methods discussed herein are merely illustrative of specific manners in which to make and use this invention and are not to be interpreted as limiting in scope.
While the devices and methods have been described with a certain degree of particularity, it is to be noted that many modifications may be made in the details of the construction and the arrangement of the devices and components without departing from the spirit and scope of this disclosure. It is understood that the devices and methods are not limited to the embodiments set forth herein for purposes of exemplification.
âAugmented and Extended Realityâ (AXR) is defined herein in its common scientific use, which may include an interactive experience typically in a see-through headset with lenses of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual images and information, sometimes across multiple sensory modalities, including visual, auditory, haptic technologies, somatosensory, and/or olfactory.
âExtended Realityâ is defined in its common scientific use, which is typically an umbrella term encapsulating augmented reality (AR) and/or virtual reality (VR) and/or mixed reality (MR) and/or real reality (RR) and everything in between. It may also include combined environments and human-machine interactions generated by computer technology such as 6 DoF and SLAM, and artificial intelligence (AI), including machine learning (ML), where the âXâ represents a variable for any current or future spatial computing technologies, including digital content of any sort; for instance, in the medical field, a 3D MRI or CT scan images or data visualizations, like patient vitals, superimposed or overlaid on an AXR headset in one of the several methods outlined herein.
âArtificial Intelligenceâ (AI), sometimes called âMachine Learningâ (ML), is used herein in its common scientific meaning, including referring to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions and decisions. The term may also be applied to an augmented reality headset that exhibits traits associated with a human mind, such as learning and/or problem-solving. AI may enable AR to interact with the physical environment in a multidimensional way. For instance, AI may permit object recognition and tracking, gestural input, eye-tracking, and voice command recognition to combine to let the user manipulate 2D and 3D objects in virtual space with the user's hands, eyes, and/or words.
âCoboticâ is used herein in its common scientific meaning, including a robotic function which is pre-programed and automatic, and human control, which augments the pre-programmed function. For instance, a cobotic arm could be set from âreposeâ position to fully extend, however the extension is augmented and guided by one or more of: facial recognition, eye-tracking, head-tracking, hand gesturing technology, verbal command, manual command, time-of-flight, dept perception, SLAM and Object Recognition, and 6 DoF (the cobotic arm knows where it exists in the world). It can operate on a relative coordinate system that tells it where to go; where to be stored on âreposeâ; and directs its travel to the user's preferred use position, based on user preference settings. However, a user can manually take control, which shuts off the end-point of the automatic control, and it responds to the manual placement by the user. Then, as the user begins to re-position, slumps, or adjusts, the cobotic arm carries the 3DDV instrument with the re-adjustment of the user position so that good ergonomics are always maintained. As used herein, cobotic is a term describing an articulating or robotic action arm of the appropriate length and joints of appropriate load bearing with the added dimension of the ability for human touch, voice, or other control to interrupt and re-control the pre-programmed robotic functions. As used herein, the cobotic arms may be a type of articulating arm robot, or a Cartesian coordinate robot arm, also called linear robot, whose three principal axes of control are linear, i.e. they move in a straight line rather than rotate, or a selective compliance assembly robot arm or selective compliance articulated robot arm (SCARA) robotic arm with parallel-axis joint layout, the arm is slightly compliant in the X-Y direction but rigid in the âZâ direction, hence the term: selective compliant, or any other type of mechanical programmable arm, with similar functions to a human arm; the arm may be the sum total of the mechanism or may be part of a more complex robot or the surgery visualization theatre herein.
The term âimage(s)â or âvirtual image(s) or âimagingâ or âvirtual objectsâ or âAXR imagingâ is defined for the purpose of this patent as visualization of either 2D images or video or 3D images or video. The definition also includes the concept that one or more 2D images can be viewed in stereoscopy to create one or more virtual 3D perspectives. Further included in the âimage(s)â definition, herein, is the idea that AXR 3D models may be viewed as a single or series of 2D images, as in a still picture or video, or a single or series of stereoscopic 3D images, as in a 3D images or video. The 3D effect may be created in the AXR headset by using an off-set paired perspective of a 3D model. In addition, 3D models in AXR can be viewed from different perspectives by the user or multiple users can view the same image from multiple perspectives.
The term âwirelessâ as used herein means the electromagnetic transfer of information between two or more points which are not connected by an electrical conductor, or a communication by technologies, such as light, magnetic, or electric fields, or the use of sound. The term âwiredâ communication as used herein includes all methods of wireline communication including, but not limited to, directly connected devices, telephone networks, ethernet connections, cable networks, internet access, fiber-optic communications, and waveguide (electromagnetism) connections.
The following are sensing and control technologies which may be utilized by the ADMO3DV system:
âSix Degrees of Freedomâ (6 DoF) is defined herein in its common meaning, including the way virtual objects can be moved in virtual space in AR. There are six total degrees of freedom in placing virtual images in AR. Three (3) correspond to rotational movement around the x, y, and z axes, commonly termed pitch, yaw, and roll. The other three (3) correspond to translational movement along those axes, which can be thought of as moving forward or backward, moving left or right, and moving up or down.
âInertial Measurement Unitsâ is used herein in its common scientific meaning, including referencing devices for measuring rotational movements, such as an accelerometer, a gyroscope, and a magnetometer, all located within the AXR headset. These IMUs may measure the headset's velocity, orientation, and gravitational forces to infer rotational orientation and movement.
âHaptic technologiesâ is used herein in its common scientific meaning and is sometimes called kinaesthetic communication or 3D touch. It may also refer to any technology which may create an experience of touch by applying forces, vibrations, or motions to the user or to an object. Haptics may enable users to feel the sense of touch via vibrations of forced motion. Haptic technologies can be used to create virtual objects in a computer simulation or virtual space, or to control those virtual objects, and may be used to enhance remote control of machines and devices (telerobotics). Haptic devices may incorporate tactile sensors that measure forces exerted by the user on the interface. This technology may employ touch sensors for control.
âObject Recognitionâ (OR) or âObject Identificationâ (OI) is used herein in its common scientific meaning, including a computer vision technique for identifying objects in images or videos. Object recognition may be a key output of deep learning and AI algorithms. When humans look at a photograph or watch a video, we can readily spot people, objects, scenes, and visual details. OR/OI does this from visual analysis based on a neural network algorithms reconciliation with pre-existing information.
âSimultaneous Localization and Mappingâ (SLAM) is used herein in its common scientific meaning, including a technology that understands the physical world through a 3D grid of feature points. SLAM maps what the camera and sensors see in three dimensions with correct spatial information and distancing. This may make it possible for AXR applications to recognize RR 3D objects and scenes, as well as to instantly track motion in the RR, and to overlay digital interactive augmentations. SLAM incorporates the application of sensors sensing dept, time-of-flight, and creating a 3D grid. SLAM also incorporates infrared sensing and measurements.
The following terms relate to the virtual/augmented images on the AXR headset:
The term âluxâ is the SI derived unit of illuminance and luminous emittance, measuring luminous flux per unit area. It is equal to one lumen per square meter.
The term âlumenâ is the SI derived unit of luminous flux, a measure of the total quantity of visible light emitted by a source per unit of time.
The term âluminanceâ is a photometric measure of the luminous intensity per unit area of light traveling in a given direction. It describes the amount of light that passes through, is emitted from, or is reflected from a particular area, and falls within a given solid angle.
The term âcandelaâ is the SI unit of luminous intensity. The candela per square meter is the derived SI unit of luminance. It is from the candela that we get the modern measurement of NIT, which is commonly referenced in wearable and cellular applications. The term âNITâ is a non-SI name also used for the candela per square meter. As a measure of light emitted per unit area, this unit is frequently used to specify the brightness of a cellular or wearable display device. The sRGB spec for monitors targets 80 cd/m2. Typically, calibrated monitors should have a brightness of 120 cd/m2. As system described herein uses a NIT reference for its light/brightness measurements.
The all- Digital Multi-Option 3D Viewing Theatre.
The ADMO3DV system may have one or more connected viewports in addition to multiple connected AXR headsets, which may derive a surgery feed from the 3DADM. The connected viewports may include one or more of a 3DAM and a 3DDV. The ADMO3DV may be a non-invasive robotic arm system with a large electronic actuator lift in the form of a cylinder mounted on a smart pedestal on which is hung a translating top to hold a balanced cobotic arm system. The system may have a cabinet and drawer system, as shown in FIG. 22 , and may contain the main computer, control, and wired and wireless connection/transmission system. The system may have its own keyboard and monitor for inputting settings, connecting with other OR equipment, and inputting or outputting EMR an imaging. The non-transitory model view controller (MVC) may synchronize the subsystems and may control all input and output according to the software programs. It may also house and control all the subsystems, including the AXR headset and cobotic arms with their viewing options as well as the 3DADM microscope. An operator may input the appropriate settings and model the control system may utilize keyboard, Bluetooth, voice control, eye-tracking, or gesture recognition or other technologies identified herein, and may also utilize SLAM and 6 DoF technologies to operate properly wirelessly or use any other sensing and control technique stated above. Alternatively, one or more of these technologies may be used together in order to access and manipulate a control of the viewing system or microscope system attached to the cobotic arms. This method may allow the user to control the AXR system, autostereoscopic 3D monitor, 3DDV, 3D microscope, or other external equipment or systems via wired or wireless connection without requiring input through foot pedals, buttons, hand dials, or other hardwired methods of control. Combining two or more methods of control, i.e., voice with eye-tracking, may provide redundancy and ensure proper operation of controls.
The ASMO3DV system may provide an enhancement to existing surgery systems in several ways. First, as time is money, a surgery team does not have to re-position the typical 3D television monitor as the cobotic arms move the monitor to the exact position needed by the OR healthcare provider, based on the sensing and control technologies outlines herein.
The AXR 3D Headset Technology.
In general, in a first aspect, the ADMO3DV system invention relates to an augmented and extended reality (AXR) surgical system option which can be either wired or typically the preferred wireless headset. The system may comprise a wearable device 1 , such as a head mounted display (HMD) or glasses, that provides the user with virtual reality (VR), augmented reality (AR), and/or mixed-extended reality (XR) for surgery visualization, as shown in FIG. 1 . Alternately, it may comprise a system as shown in FIG. 23 with a larger battery box 66 for longer surgeries. This may allow the user to access 2D or 3D imaging, magnification, virtual visualization, six-degrees of freedom (6 DoF) image and simultaneous localization and mapping (SLAM) management, and/or other images while still viewing real reality (RR) and thus maintaining a presence in the operating room. The AXR headset control of the virtual world may include sensors including haptic sensors which may be worn on the hands and connected to the headset for coordinated control.
The AXR headset system may comprise one or more micro-displays 2 , a head-tracking subsystem 3 , an eye-tracking subsystem 4 , and one or more cameras 5 , all of which may be included on the wearable device 1 . The system may further comprise one or more lenses 10 , where the micro-displays 2 are capable of projecting images on the lenses 10 , where the images may be reflected back to the user's eyes. For example, as shown in FIGS. 2 and 3 , the wearable device 1 may be a head mounted display with a pair of lenses 10 , one in front of each of the user's eyes. One or more micro-displays 2 may be located above the user's eyes and may be pointed toward the lenses 10 . The two or more AXR cameras 5 may be 4K or higher each and may provide image input, while the head-tracking subsystem 3 and the eye-tracking subsystem 4 may provide positional input, allowing the system to project the desired images to the desired location for the user to view the images. In addition, the cameras may articulate to rotate downward to a position which may be 90 degrees from the plane of the front of the AXR headset. In this fashion, instead of a surgeon wearing loops, which causes the surgeon to put his chin on his chest throughout the surgery, the AXR cameras tilt and to the surgery site, leaving the surgeon with a more ergonomic and comfortable posture. Additional image input may be provided from other sources such as SLAM or other sensing cameras 5 A. The AXR may be connected to the system and the 3DADM and my receive and transmit to the system for surgery visualization and commands back to the system.
All components may be controlled by a CPU and enabled by a GPU and one or more digital signal processors, cables, and battery source, which may be located on the wearable device 1 or remotely. Other components may include additional central processing units, one or more graphics processing units, one or more digital signal processors, firmware, hardware, software, and/or memory components, as well as other desired components, including a non-transitory model view controller. The high-level components may control the features and functions of the AXR headset 1 , including, but not limited to, its cameras 5 , micro-displays 2 , lenses 10 , sensors, communications, and subsystems.
Among virtual image display solutions for AXR viewing are catadioptric optics which are preferred in that they employ a partially transmissive curved mirror for directing image-bearing light to the viewer's eye and a partially reflective beam splitter for combining light generated at a 2D display with the real-world visible scene, which forms a superior 3D image and holographic images when viewed binocularly.
The headset may be wireless or wired. If wireless, the wireless module antenna may be connected to the main circuit board inside the headset and may radiate RF to the outside world through the WiFi, cellular, or 5G antennae 42 .
The AXR headset may contain a small worm gear or similar device connected to the two lens frames 43 , which may move closer and farther, approximately 5 mm, in order to adjust for interpupillary distance (IPD) for each person. This may be accomplished by the worm gear being connected to a spindle gear threaded on both ends, which may connect to the lens frames, which may be on a track that permits them this measure of movement. A remote Bluetooth connection may be housed in the charging station drawers, where it can automatically adjust based on the information preprogrammed into the ADMO3DV controller according to each user's IPD or can be accomplished manually through a small Bluetooth handheld device housed in each drawer and independently connected and secured to each device.
One such AXR headset, as shown in FIG. 7 and which may produce the best results for surgery, is an axially symmetric near-eye pupil-forming wearable AXR display apparatus comprising:
(i) relay of the image generator 44 to form a curved intermediate image 45 as a conjugate image. As a type of âaerialâ image, intermediate image 45 may be formed in air, serving as the optical âobjectâ for forming the virtual image. Intermediate image 45 may be formed along the curved focal surface of curved mirror 46 , with the approximate aerial position shown by a dashed line in FIG. 7 . (ii) an optical relay 47 , with particular structure as described in more detail subsequently, may conjugate the image formed from image generator 44 to the curved intermediate image 45 along the focal surface. (iii) a curved mirror 46 may be partially transmissive, such as between about 30% to 70% transmissive, for example, allowing visibility of the real-world object scene to the viewer. A nominal transmission range of 50 to 70% may be useful in many applications and the see-through may be increased with the use of brighter imaging source such as an LCD or other micro display. (iv) a beam splitter 49 may be used to reflect light from the relay 47 to the curved mirror 46 and may be an unpolarized or polarized beam splitter. It may transmit light from both the real reality external to the viewer and the virtual reality reflected off the surface of curved lens 46 . (v) use of a cylindrically curved quarter-wave plate (QWP) between mirror 48 and beam splitter 49 . Curvature of this element may help to reduce variations of the retardation imparted to the image-bearing light by the QWP over the field of view. (vi) large exit pupil 50 . System optics can form a 10 mm exit pupil at the viewer's eye-box for eye 51 . Forming a suitably sized pupil for the viewer may help to provide an eye box of reasonable dimensions to allow eye movement, without noticeable vignetting. Also, an enlarged eye box may permit the headset to move or slip without noticeable degradation of the viewed image(s). The apparatus may not need to provide pupil expansion, such as is used in existing wearable display apparatus, but may use pupil-forming optics for improved efficiency and brightness, as well as for improved image resolution.
Significantly, the eyes of the viewer may clearly see and be seen by others, with minimal impediment from the beam splitter and curved mirror optics that provide the electronically generated virtual image.
With the optical arrangement shown, the aperture stop AS may lie within prism 52 of the image relay, along or near the fold surface that is provided. This arrangement may be advantageous for component packaging and spacing, allowing the prism to be reduced in size over other configurations using a folding prism.
The given design may allow an FOV along the horizontal (x) axis, the axis parallel to a line between left and right pupils of the viewer's eyes, of greater than 50 degrees. The FOV aspect ratio (horizontal:vertical) may equal or exceeds 1.5. Digital correction may not be needed for distortion or lateral color.
According to an embodiment, curved reflector 46 may have a conic surface shape. The conic shape is advantaged, in the embodiment shown herein, helping to control chief ray angles, thus correcting for dist
CLAIMS
Claims ( 24 )
What is claimed is:
1. A surgery visualization theatre comprising:
a support pedestal;
a frame coupled to the support pedestal;
a cobotic arm system coupled to the frame and including a microscope cobotic arm, a monitor cobotic arm, and a camera cobotic arm, the monitor cobotic arm positioned between the microscope cobotic arm and the camera cobotic arm;
an augmented/extended reality (AXR) headset;
a digital microscope viewer mounted on the microscope cobotic arm;
a monitor mounted on the monitor cobotic arm;
a camera subsystem mounted on the camera cobotic arm; and
a processor programmed to receive images from the camera subsystem and display the received images on the AXR headset, the monitor, and the digital microscope viewer.
2. The surgery visualization theatre of claim 1 where the AXR headset is wireless.
3. The surgery visualization theatre of claim 1 where the camera subsystem comprises two 4K camera and a light source.
4. The surgery visualization theatre of claim 1 where the AXR headset has two micro-displays and where each of the two micro-displays has a resolution of at least 4K.
5. The surgery visualization theatre of claim 4 where the micro-displays are capable of active pixel phase shift.
6. The surgery visualization theatre of claim 1 where the digital microscope viewer has two micro-displays and where each of the two micro-displays has a resolution of at least 4K.
7. The surgery visualization theatre of claim 6 where the micro-displays are capable of active pixel phase shift.
8. The surgery visualization theatre of claim 1 where the microscope cobotic arm is a six-axis cobotic arm.
9. The surgery visualization theatre of claim 1 where the monitor is a 3D autostereoscopic no-glasses monitor capable of providing 4K resolution to each eye of a user.
10. The surgery visualization theatre of claim 9 where the monitor comprises:
an LCD panel; and
an optical filter, where the optical filter comprises a thin substrate on which one or more optically active layers are applied.
11. The surgery visualization theatre of claim 10 where the thin substrate is a film or a glass layer.
12. The surgery visualization theatre of claim 10 where the one or more optically active layers comprise lenticular lenses, parallax barriers, or both lenticular lenses and parallax barriers.
13. The surgery visualization theatre of claim 10 where the LCD panel has a resolution of 8K, and the optically active layers comprise lenticular lenses capable of producing two 4K displays such that the monitor is capable of displaying a 3D image to a user without requiring 3D glasses.
14. The surgery visualization theatre of claim 1 further comprising:
an eye-tracking subsystem associated with the monitor, such that the eye-tracking subsystem comprises at least one eye-tracking camera mounted on the monitor cobotic arm and where the eye-tracking subsystem is capable of tracking a user's eye position via the eye-tracking camera;
a wireless system associated with the monitor; or
both the eye-tracking subsystem associated with the monitor and the wireless system associated with the monitor.
15. The surgery visualization theatre of claim 1 where the monitor cobotic arm is a six-axis cobotic arm.
16. The surgery visualization theatre of claim 1 where the camera cobotic arm is a six-axis cobotic arm.
17. The surgery visualization theatre of claim 1 where the frame is capable of swiveling and reversing for right or left-handed use.
18. The surgery visualization theatre of claim 1 further comprising SLAM technology and a plurality of sensors coupled to the processor such that the processor is capable of moving the microscope cobotic arm to position the microscope viewer in front of a user's face upon a cue by the user.
19. The surgery visualization theatre of claim 1 where the microscope cobotic arm, the monitor cobotic arm, and/or the camera cobotic arm use a low-friction, zero-gravity controller.
20. The surgery visualization theatre of claim 1 where the digital microscope viewer comprises two eye cups.
21. The surgery visualization theatre of claim 1 where the AXR headset is capable of protecting a user's eyes while in the vicinity of medical lasers.
22. The surgery visualization theatre of claim 1 where the AXR headset further comprises a clip-on face mask.
23. The surgery visualization theatre of claim 22 where the clip-on face mask is an N1 filtration capability mask and/or uses photocatalytic (UV energized titanium dioxide coated surface) active pathogen oxidation techniques.
24. The surgery visualization theatre of claim 22 where the clip-on face mask is capable of using a combination of photocatalytic pathogen oxidization and post filtration to capture any spuriously created ozone before inhalation.
US17/307,771
2015-03-17
2021-05-04
Surgery visualization theatre
Active
2037-03-03
US12062430B2
( en )
Priority Applications (5)
Application Number
Priority Date
Filing Date
Title
PCT/US2021/030718
WO2021226134A1
( en )
2020-05-04
2021-05-04
Surgery visualization theatre
US17/307,771
US12062430B2
( en )
2015-03-17
2021-05-04
Surgery visualization theatre
US18/450,997
US12142367B2
( en )
2020-02-21
2023-09-06
Surgery visualization theatre
US18/606,941
US12211611B2
( en )
2020-02-21
2024-03-15
Surgery visualization theatre
US18/905,906
US12462924B2
( en )
2020-02-21
2024-10-03
AXR visualization system
Applications Claiming Priority (20)
Application Number
Priority Date
Filing Date
Title
US201562134422P
2015-03-17
2015-03-17
US15/073,144
US9955862B2
( en )
2015-03-17
2016-03-17
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US201762489801P
2017-04-25
2017-04-25
US15/940,561
US10111583B1
( en )
2015-03-17
2018-03-29
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US15/962,661
US11956414B2
( en )
2015-03-17
2018-04-25
Wearable image manipulation and control system with correction for vision defects and augmentation of vision and sensing
US201862697854P
2018-07-13
2018-07-13
US16/173,719
US10874297B1
( en )
2015-03-17
2018-10-29
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US16/511,202
US11461936B2
( en )
2015-03-17
2019-07-15
Wearable image manipulation and control system with micro-displays and augmentation of vision and sensing in augmented reality glasses
US16/511,451
US11016302B2
( en )
2015-03-17
2019-07-15
Wearable image manipulation and control system with high resolution micro-displays and dynamic opacity augmentation in augmented reality glasses
US201962907300P
2019-09-27
2019-09-27
US202062979999P
2020-02-21
2020-02-21
US202062986461P
2020-03-06
2020-03-06
US202063005202P
2020-04-03
2020-04-03
US202063019796P
2020-05-04
2020-05-04
US17/034,944
US11819273B2
( en )
2015-03-17
2020-09-28
Augmented and extended reality glasses for use in surgery visualization and telesurgery
US17/137,069
US12137974B1
( en )
2015-03-17
2020-12-29
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US17/137,093
US12237074B2
( en )
2015-03-17
2020-12-29
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US17/151,174
US11031120B1
( en )
2015-03-17
2021-01-17
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US17/182,022
US11628038B2
( en )
2020-02-21
2021-02-22
Multi-option all-digital 3D surgery visualization system and control
US17/307,771
US12062430B2
( en )
2015-03-17
2021-05-04
Surgery visualization theatre
Related Parent Applications (11)
Application Number
Title
Priority Date
Filing Date
US15/073,144
Continuation-In-Part
US9955862B2
( en )
2015-03-17
2016-03-17
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US15/940,561
Continuation-In-Part
US10111583B1
( en )
2015-03-17
2018-03-29
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US15/962,661
Continuation-In-Part
US11956414B2
( en )
2015-03-17
2018-04-25
Wearable image manipulation and control system with correction for vision defects and augmentation of vision and sensing
US16/173,719
Continuation-In-Part
US10874297B1
( en )
2015-03-17
2018-10-29
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US16/511,202
Continuation-In-Part
US11461936B2
( en )
2015-03-17
2019-07-15
Wearable image manipulation and control system with micro-displays and augmentation of vision and sensing in augmented reality glasses
US16/511,451
Continuation-In-Part
US11016302B2
( en )
2015-03-17
2019-07-15
Wearable image manipulation and control system with high resolution micro-displays and dynamic opacity augmentation in augmented reality glasses
US17/034,944
Continuation-In-Part
US11819273B2
( en )
2015-03-17
2020-09-28
Augmented and extended reality glasses for use in surgery visualization and telesurgery
US17/137,093
Continuation-In-Part
US12237074B2
( en )
2015-03-17
2020-12-29
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US17/137,069
Continuation-In-Part
US12137974B1
( en )
2015-03-17
2020-12-29
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US17/151,174
Continuation-In-Part
US11031120B1
( en )
2015-03-17
2021-01-17
System, method, and non-transitory computer-readable storage media related to correction of vision defects using a visual display
US17/182,022
Continuation-In-Part
US11628038B2
( en )
2015-03-17
2021-02-22
Multi-option all-digital 3D surgery visualization system and control
Related Child Applications (2)
Application Number
Title
Priority Date
Filing Date
US18/450,997
Continuation
US12142367B2
( en )
2020-02-21
2023-09-06
Surgery visualization theatre
US18/606,941
Continuation
US12211611B2
( en )
2020-02-21
2024-03-15
Surgery visualization theatre
Publications (2)
Publication Number
Publication Date
US20210335483A1
US20210335483A1 ( en )
2021-10-28
US12062430B2
true
US12062430B2 ( en )
2024-08-13
Family
ID=78222711
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US17/307,771
Active
2037-03-03
US12062430B2
( en )
2015-03-17
2021-05-04
Surgery visualization theatre
Country Status (1)
Country
Link
US
( 1 )
US12062430B2
( en )
Cited By (4)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20220011856A1
( en )
*
2021-09-24
2022-01-13
Intel Corporation
Strapless autonomous extended reality viewer
US12132984B2
( en )
2018-03-06
2024-10-29
Eyedaptic, Inc.
Adaptive system for autonomous machine learning and control in wearable augmented reality and virtual reality visual aids
US12282169B2
( en )
2018-05-29
2025-04-22
Eyedaptic, Inc.
Hybrid see through augmented reality systems and methods for low vision users
US12416062B2
( en )
2018-09-24
2025-09-16
Eyedaptic, Inc.
Enhanced autonomous hands-free control in electronic visual aids
Families Citing this family (19)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US12521194B2
( en )
*
2019-05-29
2026-01-13
Intuitive Surgical Operations, Inc.
Operating mode control systems and methods for a computer-assisted surgical system
US11357594B2
( en )
*
2020-08-07
2022-06-14
Johnson & Johnson Surgical Vision, Inc.
Jig assembled on stereoscopic surgical microscope for applying augmented reality techniques to surgical procedures
DE102021101694A1
( en )
*
2021-01-26
2022-07-28
Carl Zeiss Meditec Ag
Tool-Type-Agnostic Assistive Functionality in Surgical Procedures
US20220384016A1
( en )
2021-05-28
2022-12-01
Cilag Gmbh International
Monitoring a health care professional movement relative to a virtual boundary in an operating room
CN113887311B
( en )
*
2021-09-03
2023-05-02
ä¸å±±å¤§å¦ä¸å±±ç¼ç§ä¸å¿
Method, device and storage medium for protecting privacy of ophthalmic patient
CN114137727A
( en )
*
2021-11-08
2022-03-04
éå²æå°å£°å¦ç§ææéå ¬å¸
Pneumatic pupil distance adjusting device and intelligent head-mounted equipment
US20230168786A1
( en )
*
2021-11-30
2023-06-01
Verizon Patent And Licensing Inc.
Methods and Systems for Location-Based Accessing of Predesignated Data Payloads Using Extended Reality
DE102021213557B4
( en )
*
2021-11-30
2023-09-14
Siemens Healthcare Gmbh
Method and device for providing a perfusion image data set of a patient
CN114300114A
( en )
*
2021-12-24
2022-04-08
æ¦æ±èå½±æºèå»çç§ææéå ¬å¸
Method, device, equipment and storage medium for adjusting operation remote control device
GB2631647A
( en )
*
2022-03-21
2025-01-08
Headvantage Corp
Body worn camera, sensor and content delivery system
CA3259872A1
( en )
*
2022-08-08
2024-02-15
Alcon Inc.
Support assembly for ophthalmic visualization system
US20240064320A1
( en )
*
2022-08-22
2024-02-22
Karl Storz Se & Co. Kg
Systems and methods for scaling video streams
US12141340B2
( en )
*
2023-02-16
2024-11-12
Microsoft Technology Licensing, Llc
Muscle group movement tracking using RF sensors
EP4485043A1
( en )
*
2023-06-29
2025-01-01
Siemens Healthineers AG
Display system, medical system with a display system and method of displaying
WO2025042799A2
( en )
*
2023-08-18
2025-02-27
Vuzix Corporation
Offset-axis optical power image light guide system
WO2025059522A1
( en )
*
2023-09-14
2025-03-20
Onpoint Medical, Inc.
Augmented reality guidance methods and systems for use with microscopes or endoscopes in surgical procedures
EP4531058A1
( en )
*
2023-09-28
2025-04-02
Koninklijke Philips N.V.
Visualization for medical interventions
GB2643045A
( en )
*
2024-07-30
2026-02-04
Infinite Medical Ventures Ltd
Stereoscopic display system
US20260094709A1
( en )
*
2024-09-27
2026-04-02
Stephen McNutt
Integrated mixed reality visualization for diagnostic imaging and data mapping
Citations (36)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20020063807A1
( en )
*
1999-04-19
2002-05-30
Neal Margulis
Method for Performing Image Transforms in a Digital Display System
US20030223038A1
( en )
2002-02-19
2003-12-04
Yair Alster
Methods, devices and systems for assessing eye disease
US20050036109A1
( en )
2003-08-15
2005-02-17
Blum Ronald D.
Enhanced electro-active lens system
US20090218400A1
( en )
2005-09-02
2009-09-03
Boss Andre
Method for optimising the firing trigger of a weapon or artillery
US20100011959A1
( en )
*
2006-07-19
2010-01-21
Koninklijke Philips Electronics N.V.
Electrostatic particle filter
US20100149073A1
( en )
2008-11-02
2010-06-17
David Chaum
Near to Eye Display System and Appliance
US20130172902A1
( en )
*
2011-12-29
2013-07-04
Mako Surgical Corp.
Neural Monitor-Based Dynamic Haptics
KR20130082637A
( en )
2011-12-12
2013-07-22
íêµì ìíµì ì°êµ¬ì
Apparatus of medical service using transparent display
US20140094655A1
( en )
*
2012-09-28
2014-04-03
Endopodium, Inc.
Endoscope with multifunctional extendible arms and endoscopic instrument with integrated image capture for use therewith
US20140160264A1
( en )
*
2012-12-10
2014-06-12
The Johns Hopkins University
Augmented field of view imaging system
JP5571901B2
( en )
2008-10-20
2014-08-13
æ ªå¼ä¼ç¤¾ã¿ã¤ãã¼
Shooting game equipment
US20140275760A1
( en )
*
2013-03-13
2014-09-18
Samsung Electronics Co., Ltd.
Augmented reality image display system and surgical robot system comprising the same
WO2014140849A2
( en )
2013-03-15
2014-09-18
Amo Groningen B.V.
Wavefront generation for ophthalmic applications
US20140283429A1
( en )
2013-03-21
2014-09-25
Kms Consulting, Llc
Precision aiming system for a weapon
US20160091968A1
( en )
*
2014-09-26
2016-03-31
International Business Machines Corporation
Interacting with a display positioning system
WO2016133644A1
( en )
2015-02-20
2016-08-25
Covidien Lp
Operating room and surgical site awareness
US20160247418A1
( en )
*
2015-02-23
2016-08-25
The Regents Of The University Of Colorado, A Body Corporate
Microsurgical Training System and Related Methods
US20160252325A1
( en )
2013-10-08
2016-09-01
Horus Vision Llc
Compositions, methods and systems for external and internal environmental sensing
US20160262608A1
( en )
2014-07-08
2016-09-15
Krueger Wesley W O
Systems and methods using virtual reality or augmented reality environments for the measurement and/or improvement of human vestibulo-ocular performance
US20160270656A1
( en )
2015-03-16
2016-09-22
Magic Leap, Inc.
Methods and systems for diagnosing and treating health ailments
US20170011706A1
( en )
*
2015-07-07
2017-01-12
Samsung Electronics Co., Ltd.
Head-mounted display apparatus
US20170007351A1
( en )
*
2015-07-12
2017-01-12
Steven Sounyoung Yu
Head-Worn Image Display Apparatus for Microsurgery and Other Uses
US20170094167A1
( en )
*
2015-09-24
2017-03-30
Airbus Operations Gmbh
Virtual windows for airborne vehicles
WO2017064301A1
( en )
2015-10-16
2017-04-20
Medical Microinstruments S.R.L.
Robotic surgical assembly
US9651786B1
( en )
2015-05-05
2017-05-16
SA Photonics, Inc.
Systems and methods for augmented reality devices with light security
US20170293380A1
( en )
2016-04-12
2017-10-12
Benoit CHAUVEAU
Interface device with a touch screen and a control button
WO2018067611A1
( en )
2016-10-03
2018-04-12
Verb Surgical Inc.
Immersive three-dimensional display for robotic surgery
US20180116742A1
( en )
2016-01-14
2018-05-03
Trevor James DELL
System and method for configuring positions in a surgical positioning system
US20180250085A1
( en )
*
2015-10-16
2018-09-06
Medical Microinstruments S.p.A.
Surgical tool
US10323904B1
( en )
2016-11-02
2019-06-18
Guneye LLC
Infrared firearm sight camera attachment, system and method
US20190339528A1
( en )
2015-03-17
2019-11-07
Raytrx, Llc
Wearable image manipulation and control system with high resolution micro-displays and dynamic opacity augmentation in augmented reality glasses
US20190353457A1
( en )
2013-05-09
2019-11-21
Shooting Simulator, Llc
System and method for marksmanship training
US20190359351A1
( en )
2015-01-18
2019-11-28
Foundation Productions, Llc
Apparatus, Systems and Methods for Unmanned Aerial Vehicles
US20200038120A1
( en )
2017-02-17
2020-02-06
Nz Technologies Inc.
Methods and systems for touchless control of surgical environment
US20210149173A1
( en )
*
2017-05-22
2021-05-20
Carl Zeiss Microscopy Gmbh
Method and adapter for adapting a microscope objective to a digital microscope
EP4052647A1
( en )
*
2021-03-02
2022-09-07
Oncomfort SA
Hygiene protective cover for bodily worn devices
2021
2021-05-04
US
US17/307,771
patent/US12062430B2/en
active
Active
Patent Citations (37)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20020063807A1
( en )
*
1999-04-19
2002-05-30
Neal Margulis
Method for Performing Image Transforms in a Digital Display System
US20030223038A1
( en )
2002-02-19
2003-12-04
Yair Alster
Methods, devices and systems for assessing eye disease
US20050036109A1
( en )
2003-08-15
2005-02-17
Blum Ronald D.
Enhanced electro-active lens system
US8579194B2
( en )
2005-09-02
2013-11-12
Rheinmetall Air Defence Ag
Method for optimising the firing trigger of a weapon or artillery
US20090218400A1
( en )
2005-09-02
2009-09-03
Boss Andre
Method for optimising the firing trigger of a weapon or artillery
US20100011959A1
( en )
*
2006-07-19
2010-01-21
Koninklijke Philips Electronics N.V.
Electrostatic particle filter
JP5571901B2
( en )
2008-10-20
2014-08-13
æ ªå¼ä¼ç¤¾ã¿ã¤ãã¼
Shooting game equipment
US20100149073A1
( en )
2008-11-02
2010-06-17
David Chaum
Near to Eye Display System and Appliance
KR20130082637A
( en )
2011-12-12
2013-07-22
íêµì ìíµì ì°êµ¬ì
Apparatus of medical service using transparent display
US20130172902A1
( en )
*
2011-12-29
2013-07-04
Mako Surgical Corp.
Neural Monitor-Based Dynamic Haptics
US20140094655A1
( en )
*
2012-09-28
2014-04-03
Endopodium, Inc.
Endoscope with multifunctional extendible arms and endoscopic instrument with integrated image capture for use therewith
US20140160264A1
( en )
*
2012-12-10
2014-06-12
The Johns Hopkins University
Augmented field of view imaging system
US20140275760A1
( en )
*
2013-03-13
2014-09-18
Samsung Electronics Co., Ltd.
Augmented reality image display system and surgical robot system comprising the same
WO2014140849A2
( en )
2013-03-15
2014-09-18
Amo Groningen B.V.
Wavefront generation for ophthalmic applications
US20140283429A1
( en )
2013-03-21
2014-09-25
Kms Consulting, Llc
Precision aiming system for a weapon
US20190353457A1
( en )
2013-05-09
2019-11-21
Shooting Simulator, Llc
System and method for marksmanship training
US20160252325A1
( en )
2013-10-08
2016-09-01
Horus Vision Llc
Compositions, methods and systems for external and internal environmental sensing
US20160262608A1
( en )
2014-07-08
2016-09-15
Krueger Wesley W O
Systems and methods using virtual reality or augmented reality environments for the measurement and/or improvement of human vestibulo-ocular performance
US20160091968A1
( en )
*
2014-09-26
2016-03-31
International Business Machines Corporation
Interacting with a display positioning system
US20190359351A1
( en )
2015-01-18
2019-11-28
Foundation Productions, Llc
Apparatus, Systems and Methods for Unmanned Aerial Vehicles
WO2016133644A1
( en )
2015-02-20
2016-08-25
Covidien Lp
Operating room and surgical site awareness
US20160247418A1
( en )
*
2015-02-23
2016-08-25
The Regents Of The University Of Colorado, A Body Corporate
Microsurgical Training System and Related Methods
US20160270656A1
( en )
2015-03-16
2016-09-22
Magic Leap, Inc.
Methods and systems for diagnosing and treating health ailments
US20190339528A1
( en )
2015-03-17
2019-11-07
Raytrx, Llc
Wearable image manipulation and control system with high resolution micro-displays and dynamic opacity augmentation in augmented reality glasses
US9651786B1
( en )
2015-05-05
2017-05-16
SA Photonics, Inc.
Systems and methods for augmented reality devices with light security
US20170011706A1
( en )
*
2015-07-07
2017-01-12
Samsung Electronics Co., Ltd.
Head-mounted display apparatus
US20170007351A1
( en )
*
2015-07-12
2017-01-12
Steven Sounyoung Yu
Head-Worn Image Display Apparatus for Microsurgery and Other Uses
US20170094167A1
( en )
*
2015-09-24
2017-03-30
Airbus Operations Gmbh
Virtual windows for airborne vehicles
US20180250085A1
( en )
*
2015-10-16
2018-09-06
Medical Microinstruments S.p.A.
Surgical tool
WO2017064301A1
( en )
2015-10-16
2017-04-20
Medical Microinstruments S.R.L.
Robotic surgical assembly
US20180116742A1
( en )
2016-01-14
2018-05-03
Trevor James DELL
System and method for configuring positions in a surgical positioning system
US20170293380A1
( en )
2016-04-12
2017-10-12
Benoit CHAUVEAU
Interface device with a touch screen and a control button
WO2018067611A1
( en )
2016-10-03
2018-04-12
Verb Surgical Inc.
Immersive three-dimensional display for robotic surgery
US10323904B1
( en )
2016-11-02
2019-06-18
Guneye LLC
Infrared firearm sight camera attachment, system and method
US202000381