Conceptio › Archive › Google Patents

Google Patentsopen access

Software compensated robotics — Sanctuary Cognitive Systems Corporation (US11691274B2)

Sanctuary Cognitive Systems Corporation · Google Patents

Google Patents · Patents · License: Open Access

patent, google patents, intellectual property, US11691274B2, Sanctuary Cognitive Systems Corporation, Adrian Kaehler, en, 2023

ABSTRACT

Abstract

A software compensated robotic system makes use of recurrent neural networks and image processing to control operation and/or movement of an end effector. Images are used to compensate for variations in the response of the robotic system to command signals. This compensation allows for the use of components having lower reproducibility, precision and/or accuracy that would otherwise be practical.

Description

PRIORITY APPLICATIONS

The present application is a continuation of U.S. application Ser. No. 16/237,721 filed on Jan. 1, 2019, titled âSoftware Compensated Roboticsâ, the disclosure of which is hereby incorporated by reference in its entirety.

BACKGROUND

Field of the Invention

The invention is in the field of robotics, and in some embodiments the field of vision-controlled robotics.

Related Art

Control of a robot typically involves sending an electronic signal and activating an actuator based on the electronic signal. The actuator can include a DC motor, hydraulic device, synthetic muscle, pneumonic device, piezoelectric, a linear or rotational actuator, or other movement generation device. The generated movement may be scaled up or down using a gear box or lever, and then used to move a part of the robot. The amount of movement is optionally detected using an encoder. The encoder and other components are optionally embodied in a servo motor or other actuator. A robot having multiple degrees of freedom, e.g., 6-degrees, typically require at least one movement generation device for each degree of freedom.

Reaching a desired âposeâ for a robotic requires specification of both a location (x, y, z) and a set of angular values (Î±, Î², Î³). Reaching a desired pose depends on knowing an existing pose of the robot and applying motion to six movement generation devices to move from the current pose to a desired pose. Such movement is typically achieved by using a target pose and a model of the robot to calculate a movement needed in each degree of freedom. The precision and accuracy of reaching the desired pose is dependent on inverse kinematics, which requires knowledge of the initial pose, accuracy of the model of the robot and precision of the movement. Achieving high precision and accuracy can require expensive components, particularly when heavy loads are involved. Requirements for precision and accuracy also preclude, in many applications, use of some types of movement generation devices which may change over time, such as tendon mechanisms. Finally, in many applications, use of some types of materials are precluded for use in robotics for similar reasons.

SUMMARY

Vision based robot control includes a real-time feedback loop which compensates for variations in actuator response and/or models of the robot using data collected from cameras and other input devices. Images of actual robot movement in response to control signals are used to determine future control signals need to achieve desired robot movements. A computer vision software pipeline, which may be implemented as a multi-stage neural network, is configured to process received images and to generate control signals for reaching a desired movement goal of the robot. When implemented using a neural network, such a network may include at least one recurrent neural network block having a stored state that allows for dynamic temporal behavior. Specifically, such a neural network is configured such that images are the primary input used to control movement of the robot toward a specified goal, though other inputs, such as from servo encoders, potentiometers, contact sensors, and/or force sensors may also be included. Together, these inputs are used to detect responses of the robot to a prior set of control signals. The stored state of the recurrent neural network enables the incorporation of past responses in the prediction of future responses.

Various embodiments of the invention include a robotic system comprising: a movement generation device; a tendon coupled to the movement generation device and to a robotic manipulator, the tendon being configured to move the robotic manipulator in response to the movement generation device; an end effector attached to the robotic manipulator, a pose of the end effector being dependent on movement of the robotic manipulator; a camera configured to generate an image of the end effector; a multi-stage neural network including: a first perception block configured to receive the image and generate an image processing output representative of a state of an object within the image, a policy block configured to generate command signals for movement of the end effector, the generated command signals being based on at least i) a goal for the end effector, ii) the image processing output and optionally iii) a time dependent internal state of the policy block, and a compensation block configured to provide an output for control of the movement generation device based both the command signals and the image processing output; and control logic configured to provide the goal for the end effector to the policy block, or to select the policy block based on the goal for the end effector.

Various embodiments of the invention include a method of controlling a robot, the method comprising: capturing an image using a camera, the image optionally including an end effector connected to a robotic manipulator; processing the captured image to produce a representation of objects within the image, as well as a state of the robot itself; applying a policy to the representation of objects to produce command signals, the production of command signals being based on at least a goal and the representation of objects; compensating for a change in response to of the robotic manipulator to command signals, to produce compensated control signals, the compensation being based on prior command signals and the representation of objects; and activating the robot using the compensated control signals.

Various embodiments of the invention include a method of calibrating a robot, the method comprising: generating first control signals; providing the first control signals to a robot, the first control signals optionally being configured to generate an expected movement of an end effector attached to the robot; capturing an image showing a response of the robot to the control signals; generating second control signals; changing a state of the recurrent neural network responsive to the image and the expected movement; and generating second control signals; compensating the second control signals to produce compensated control signals using the recurrent neural network, the compensation being responsive to the changed state of the recurrent neural network, the compensation being configured to reduce a difference between the expected movement and a movement of the end effector indicated by the image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a robotic system, according to various embodiments of the invention.

FIG. 2 illustrates a robot, according to various embodiments of the invention.

FIG. 3 illustrates a neural network at different times, according to various embodiments of the invention.

FIG. 4 illustrates a neural network including one or more multiplex layer, according to various embodiments of the invention.

FIG. 5 illustrates methods of controlling a robot, according to various embodiments of the invention.

DETAILED DESCRIPTION

As used herein, a âmovement generation deviceâ is a device that causes movement or force. For example, a movement generation device can include a DC motor, an AC motor, a pneumonic device, a piezoelectric, an electro-magnetic driver, a stepper motor, a servo, and/or the like.

As used herein, an âactuatorâ includes a movement generation device, circuitry configured to control the movement generation device and an optional encoder configured to measure movement or force of the movement generation device.

As used herein, an âend effectorâ is a device configured to interact with or operate on an object. Examples, of end effectors include, a cutting tool, a gripping tool, a pushing tool, a pulling tool, a lifting tool, a welding tool, a gripping tool, an attachment tool, a heating tool, a soldering tool, a pressing tool, and/or the like. Tools need not make direct contact with an object. For example, a camera, laser, paint gun or a heat lamp may be used as an end effector. In some embodiments, an end effector includes a robotic hand, which has two or more fingers configured to manipulate objects, such as tools and/or work pieces.

As used herein, âlogicâ is used to refer to hardware, firmware, and/or firmware stored on a computer non-transitory readable medium. Logic includes computing instructions and electronic circuits configured to execute these instructions.

FIG. 1 illustrates a Robotic System 100 , according to various embodiments of the invention. Robotic System 100 can include a wide variety of alternative devices. For example, Robotic System 100 can include manipulators configured to move large objects or extremely small devices configured to perform delicate operations such as vascular surgery. Robotic System 100 can include self-guided vehicles such as drones. Robotic System 100 may include a human exoskeleton, or a prosthesis.

Robotic System 100 includes at least one Movement Generation Device 110 optionally configured to generate movement of at least one Tendon 120 . Movement Generation Device 110 can include any of the movement generation devices discussed herein. Movement Generation Device 110 is optionally coupled with a control circuit and/or encoder configured to control or measure movement respectively. Movement Generation Device 110 is optionally coupled with a device configured to measure appearance, temperature, pressure, strain, current, or some other indicator of a state of Movement Generation Device 110 .

Optionally Tendon 120 is a movable linkage such as a hydraulic coupling, a pneumonic coupling, a cable, a chain, a gear, a cam, a rope, a screw drive, a belt, a pully, and/or the like. Tendon 120 is coupled to Movement Generation Device 110 and at least one robotic Manipulator 130 . Each Tendon 120 is configured to convey movement from an instance of Movement Generation Device 110 to one or more respective robotic Manipulators 130 . For example, movement generated by an electric motor may be conveyed to a robotic manipulator via a pully and cable. In various embodiments, Tendon 120 may experience changes in length due to load, temperature, age, and/or other factors. Various embodiments include Tendons 120 configured in opposition. For example, a first Tendon 120 may be configured to rotate a joint in a first direction while a second Tendon 120 may be configured to rotate a joint in a second direction. Tendons 120 optionally comprise polymer fibers such as NylonÂ® and/or Spectra LineÂ®.

Manipulator 130 is typically a load bearing element, such as a wheel, robotic arm or truss. At least one of the one or more Manipulators 130 is configured to be attached to an End Effector 140 . A pose of End Effector 140 is dependent on movement of Manipulator 130 . Tendon 120 is configured to move Manipulator 130 or End Effector 140 in response to Movement Generation Device 110 . Manipulator 130 is optionally a member of a plurality of robotic manipulators configured to manipulate End Effector 140 in the six-dimensional space of pose. Minimally this implies six degrees of freedom, however a robotic system may have more degrees of freedom than this minimal number.

PRIORITY APPLICATIONS

BACKGROUND

Field of the Invention

The invention is in the field of robotics, and in some embodiments the field of vision-controlled robotics.

Related Art

SUMMARY

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a robotic system, according to various embodiments of the invention.

FIG. 2 illustrates a robot, according to various embodiments of the invention.

FIG. 3 illustrates a neural network at different times, according to various embodiments of the invention.

FIG. 4 illustrates a neural network including one or more multiplex layer, according to various embodiments of the invention.

FIG. 5 illustrates methods of controlling a robot, according to various embodiments of the invention.

DETAILED DESCRIPTION

Robotic System 100 optionally further includes one or more Camera 150 configured to generate an image of End Effector 140 , Manipulator 130 , and/or other objects within a three-dimensional environment. A pose of Camera 150 is optionally dependent on movement of an instance of Manipulator 130 . As such, Camera 150 may be positioned in a way similar to other examples of End Effector 140 . Some embodiments of Robotic System 100 include a first Manipulator 130 configured to move Camera 150 and a second Manipulator 130 configured to move End Effector 140 within a field of view of Camera 150 . Camera 150 is optionally a stereoscopic camera. In some embodiments, Camera 150 is replaced or augmented by an alternative detector such as a laser range finder, time-of-flight sensor, radar, sonic device, or other sensor capable of measuring depth either separately or in addition to color or grayscale imagery. Output from any such sensors or detectors may be processed along with images generated by Camera 150 .

Robotic System 100 further includes a Neural Network 160 . Neural Network 160 is a multi-stage neural network including at least a perception block, a policy block and a compensation block, (see FIG. 3 ). These various blocks are optionally combined in one or more neural network stages configured to perform the functionality of the discrete blocks discussed herein for clarity. For example, the all three blocks can be combined in a single neural network stage, or any two of these blocks can be combined in a particular stage. Particular neural network nodes may provide functionality of more than one of the blocks. As such, the boundaries between blocks may not be distinct and one or more neural network system including the functionality described as being provided by each block may be considered to include each of the perception, policy and compensation blocks. As used herein, neural network âblocksâ may or may not be distinct from each other.

The perception block is configured to receive an image generated by Camera 150 and to generate an image processing output representative of a state of an object with the image. The policy block is configured to generate command signals for movement of End Effector 140 (or Camera 150 ). The generated command signals are based on i) a goal for End Effector 140 , e.g., a desired pose or movement, ii) the image processing output, and optionally iii) time-dependent internal states of the policy block and/or compensation block. The compensation block is configured to provide an output for control of one or more of Movement Generation Device 110 based both the command signals and the image processing output. This output is typically an adapted version of the command signals generated by the policy block. Any of the perception block, policy block and compensation block can include recurrent neural network layers.

Robotic System 100 further includes Control Logic 170 . Control Logic 170 is configured to provide a goal for movement of End Effector 140 , to the policy block. Alternatively, Control Logic 170 may be configured to select a particular policy block configured to execute a specific goal. In specific examples, Control Logic 170 is configured to receive a set of instructions to move an object from a first location to a second location. This task is divided into multiple steps each represented by a goal. The specific goals may be to 1) move a gripping tool adjacent to the object, 2) grasp the object using the gripping tool, 3) lift the object using the gripping tool to a first intermediate position, 4) move the object to a second intermediate position, and 5) place the object on a designated surface. Control Logic 170 is optionally configured to divide a task in to specific goals. Each of these goals is optionally performed by a different policy block.

In some embodiments, a particular policy block is configured to perform multiple goals and/or specific classes of goals. In these embodiments, a specific goal is provided to the policy block at execution time. Control Logic 170 is optionally configured to select a policy block based on a specific goal class. For example, a specific policy block may be configured to execute âlinear movement goals.â This policy block may receive a destination and a velocity; or a vector, velocity and distance, and use this information to perform a specific movement goal. Other specific policy blocks may be configured to execute âgripping goals,â âattachment goals,â ârotation goals,â âposition relative to goals,â âinsert goals,â âcut goals,â and/or the like.

In some embodiments, Control Logic 170 is configured to include default goals, such as avoiding a collision between Manipulator 130 and a person nearby, or avoiding contact between two different instances of Manipulator 130 . Control Logic 170 may further be configured to select between different available end effectors for a task, for example between a gripping tool and a cutting tool. These different end effectors may be attached to different instances of Manipulator 130 or be alternatively attached to the same instance of Manipulator 130 . Control Logic 170 may be configured to provide goals related to movement of Camera 150 and/or goals related to identifying a particular object. For example, Control Logic 170 may provide goals to identify male and female parts of a connector and positioning Camera 150 such that insertion of the male part into the female part can best be observed. Other goals provided by Control Logic 170 can include object recognition goals, movement goals, gripping goals, cutting goals, attachment goals, insertion goals, heating goals, positioning goals, activation goals (e.g., press the ON button), rotation goals, lifting goals, releasing goals, placement goals, and/or goals relating to any other interactions between End Effector 140 and on object.

Goals generated by Control Logic 170 , and thereby selection of policy blocks, optionally depend on outputs of a perception block. For example, the outputs of a perception block may be used to identify a location, orientation and/or identity of an object. In one example, an orientation of an object may result in a goal of rotating the object to a different orientation. In another example, identification of a human hand by a perception block may result in a goal to avoid the hand or interact with the hand. In a specific example, the goal may be to avoid contact between a cutting tool and a moving hand or to accept an object from the hand.

In some embodiments, a goal generated by Control Logic 170 is configured for calibration of the compensating block. For example, Control Logic 170 may generate a series of movement goals for the purpose of observing a resulting movement of End Effector 140 . In this case, Camera 150 and the perception block are used to determine actual movements in response to control signals generated by the compensating block. Such movements and measured results cause a change in state of the compensating block and/or policy block, making the compensating block and/or policy block better able to generate command signals that will result in a desired movement.

In some embodiments, Control Logic 170 is configured to divide a task into goals of different magnitude. For example, a task of moving a gripping tool in position to grip an object may include a goal of moving a first distance, a goal of moving a second distance and a goal of moving a third distance. The first distance being larger than the second distance and the second distance being larger than the third distance. The goal of moving the second distance may be generated before after execution of the goal of moving the first distance. More specifically, a task of moving approximately 11 cm may be divided into a goal of making a 10 cm movement, a goal of making a 1 cm movement and one or more goals of making sub-1 mm movement. A result of executing 1 st goal is considered in defining the requirements of the 2 nd goal and a result of executing the 2 nd goal is considered in the number and requirements of subsequent goals. Such a task may be used, for example, to precisely place a pin in a hole.

A task performed using Control Logic 170 can include operation or activation of a machine. For example, a task may include electropolishing a part. Control Logic 170 can divide this task into goals such as picking up the part, attaching an electrode to the part, closing a protective cover, placing the part in an electropolishing bath, activating (turning on) an electropolishing circuit, opening the cover, removing the part from the bath, disconnecting the electrode, and/or placing the part on a transport device to be taken to a location of the next task to be performed on that part. Activating the electropolishing circuit can include pressing a button using an instance of End Effector 140 , activating a circuit using a command issued by Policy Block 320 , and/or the like. Machine activation as part of a task performed using Control Logic 170 can include activating a washing device, a heating device, a cutting device, a spraying device, drilling device, a mixing device, a pressing device, a deposition device, a programming device, and/or any other device used in logical, mechanical or chemical processing of an object.

In some embodiments the start or completion of a goal are determined by visual input from Camera 150 . For example, one or more images from Camera 150 may indicate that a gripping tool is in position to grip a target object, and subsequently that the gripping tool is in contact with the object. These images may be used to represent the completion of a positioning goal, the start of a gripping goal and the completion of a gripping goal. The one or more images are used to determine relative relationships between the objects, not necessarily absolute positions of the objects. This allows goals to be defined in terms of relative relationships between objects. For example, a goal may include moving a gripping tool to a pose (+/âsome margin of distance error) relative to a target object. This goal can then be achieved even if the location and/or orientation of the target object changes as the goal is being executed.

FIG. 2 illustrates a Robot 200 , according to various embodiments of the invention. The Robot 200 is meant as an illustrative example. Various embodiments of the invention include a wide variety of robotic architectures, designs and structures in addition to or instead of those illustrated in FIG. 2 , which is for illustrative purposes. Robot 200 can include a system of arbitrary complexity and may include multiple End Effectors 140 of any type known in the field of robotics. Robot 200 can include both robot arms, e.g., one or more Manipulators 130 and robot hands, e.g. End Effectors 140 , having one or more âfingers.â By using image input, the systems and methods described herein can be used to control both the robot arms and robot hands. Generated images detect the result of movement of both the âarmsâ and âhandsâ of Robot 200 and, as a result, a neural network trained using such images inherently provides an optimal balance between control of the movement of the arms and hands. For example, the generated movement of the arms and hands can have an optimal relative magnitude optimized to achieve a goal. In a specific case, picking up an object using a robot hand, e.g. End Effector 140 , can include movement of both an arm, e.g., one or more Manipulators 130 , and fingers of the hand. The neural network system described herein, trained based on images generated using Camera 150 , can result in an optimal movement. The optimization being with regard to minimal error toward achieving a desired goal, minimal total movement, minimal energy usage, most probable goal achievement, minimal adverse effects (e.g., damage to a target object or person), and or the like. Robot 200 can include large scale robots configured to manipulate heavy loads, small scale robotics configured to perform surgery or modify integrated circuits, mobile robots, and/or the like.

Robot 200 includes a Base 210 configured to support other elements of Robot 200 . Base 210 can be fixed, movable, or mobile. For example, in some embodiments Base 210 includes propulsion, a conveyor, wheels or tracks, and movement of an End Effector 140 optionally includes movement of Base 210 . Alternatively, Base 210 may be configured to be bolted or otherwise fixed to a floor and to support heavy loads manipulated by one or more End Effectors 140 . Alternatively, Base 210 may include a body of a walking robot in which End Effectors 140 include tracks, pads or feet. Alternatively, Base 210 may include a body of a floating or submersible embodiment of Robot 200 . Base 210 may be configured to support multiple robotic arms and End Effectors 140 .

Robot 200 further includes at least one Movement Generation Device 110 . Movement Generation Device 110 is configured to generate movement, e.g., rotational and/or linear movement. In some embodiments, Movement Generation Device 110 is attached to a Tendon 120 , Tendon 120 is attached to a Manipulator 130 , and Manipulator 130 is attached to End Effector 140 , such that the pose of End Effector 140 is responsive to movement generated by Movement Generation Device 110 . End Effector 140 , Manipulator 130 and/or Movement Generation Device 110 are optionally separated by at least one Robotic Joint 225 . In various embodiments, an instance of Movement Generation Device 110 is connected to a particular End Effector 140 by a Tendon 120 that traverses one, two, three or more Robotic Joints 225 .

Robotic Joint 225 can include, for example, linear joints, orthogonal joints, rotational joints, twisting joints, or revolving joints. Instances of Robotic Joint 225 can be configured to couple Bass 210 , Manipulators 130 , and/or End Effectors 140 . In various embodiments, End Effector 140 and/or Manipulator(s) 130 are separate by one or more Robotic Joints 225 . Tendon(s) 120 are optionally configured to traverse these Robotic Joints 225 . For example, as illustrated in FIG. 2 , Tendon 120 can extend from Movement Generation Device 110 , past one or more Robotic Joints 225 , past one or more Manipulators 130 , to terminate in a connection to one of Manipulators 120 or End Effector 140 .

FIG. 3 illustrates instances of Neural Network 160 at different times, according to various embodiments of the invention. Neural Network 160 includes at least a Perception Block 310 , a Policy Block 320 and a Compensation Block 330 . Neural Network 160 is configured to receive images, and based on those images generate command signals configured to control Movement Generation Device 110 . The command signals are generated to complete a goal, such as movement or operation of End Effector 140 .

Perception Block 310 includes a neural network configured to receive an image, and/or series of images, and generate an image processing output representative of the state of an object within the image. The image processing output can include object features, e.g., corners, edges, etc., identified within an image; and/or relationships therebetween. In a specific example, the image processing output can include joint angles and positional coordinates of the fingers of a robotic hand, and distances between these fingers and an object. The image processing output can include classifications and/or identifications of objects with an image. The image processing output can include data characterizing differences between two images, for example, a number of pixels an object has moved between images, or numbers of pixels particular object features have moved between images. In various embodiments Perception Block 310 is configured to generate an image processing output based on a stereo image. In various embodiments Perception Block 310 is configured to determine spatial relationships between objects. For example, Perception Block 310 may be configured to generate an image processing output representative of a distance between a target object and End Effector 140 . The image processing output optionally includes a representation of a pose of an object within the image and/or a pose of End Effector 140 .

Perception Block 310 optionally includes a recurrent neural network in which the processing of an image results in a change in state in of the neural network. The change in state is typically represented by a change in operation of specific nodes within the neural network. This change in operation is, optionally, a result of a previous (e.g., recurrent) output of that specific node or other nodes within the network. Specifically, a previous output may be included as a (recurrent) input to the operation of the node. Specific nodes, sets of nodes, levels of nodes, and/or entire blocks of nodes may be recurrent to any previous output, and thus their operational state may change over time. A recurrent instance of Perception Block 310 may be used to detect changes between images. For example, movement of objects as seen in different images or a change in viewpoint from which the image is obtained.

In some embodiments, Neural Network 160 includes a plurality of Perception Blocks 310 . Each of these Perception Blocks 310 are optionally associated with a different camera, the different cameras having overlapping fields of view such that they can be used to view an object from different viewpoints. As discussed elsewhere herein, e.g., with reference to FIG. 4 , a multiplex layer is optionally used to selectively communicate image processing outputs from each of the Perception Blocks 310 to one or more Policy Block 320 . The different Perception Blocks 310 are optionally configured to process images in different ways. For example, one Perception Block 310 may be configured to read barcodes, another Perception Block 310 may be configured to recognize particular objects, e.g., faces or end effectors, and/or another perception block may be configured to measure distances based on a stereo image pair. One of Perception Block 310 may be configured to detect geometric objects such as a bolt or an integrated circuit while another Perception Block 310 is configured to identify people, e.g., a hand in a work area. Perception Blocks 310 may process images in parallel or serially. For example, in parallel processing, a first Perception Block 310 may process an image at the same time that a second Perception Block 310 is processing the same image or a different image.

In various embodiments, image processing outputs of Perception Block 310 includes a representation of a distance between End Effector 140 and an object as seen within a processed image, and/or a distance between two objects with the image. The outputs can include a representation of an object within a three-dimensional environment. In various embodiments, image processing outputs include a representation of a change in state of an object within a processed image, as compared to a prior image. For example, the outputs can include information regarding translation or rotation of an object, a change in color of an object, filling of a seam, hole, or gap (as in a welding operation), addition of a material (as in a soldiering operation), alignment of objects or surfaces (as in positioning of an object at a desired place or a screw over an opening), insertion of one object into another, and/or the like.

In some embodiments, image processing outputs of Perception Block 310 includes estimates of positions of objects that are occluded by other objects within an image. For example, if a first object is moved in front of a second object, a position of the second object may be estimated from data received in prior images. The âmemoryâ of the position of the second object can be retained in a state of the Perception Block 310 , where Perception Block 310 includes one or more recurrent layers. Such memory may be otherwise stored in an external memory that is accessed by the neural network, such as with a Differentiable Neurocomputer.

Policy Block 320 is configured to generate command signals for movement of End Effector 140 . The generated command signals are based on at least: 1) a goal for movement of End Effector 140 , 2) the image processing output received from Perception Block(s) 310 , optionally 3) a time dependent internal state of Policy Block 320 , and optionally 4) feedback received from Compensation Block 330 . Neural Network 160 optionally includes multiple Policy Block 320 . Optionally, different instances of Policy Block 320 are configured to perform different tasks and/or goals. For example, one instance may be configured for accomplishing a welding goal while other instances are configured for accomplishing moving or gripping goals. An instance of Policy Block 320

CLAIMS

Claims ( 19 )

I claim:

1. A robotic system comprising:

a movement generation device;

a robotic arm having a movement in a three-dimensional space responsive to the movement generation device;

an end effector coupled to the robotic arm and having a spatial pose adjustable, at least in part, by the movement of the robotic arm in the three-dimensional space;

one or more sensing devices configured to capture one or more images of a portion of an environment including the end effector; and

a neural network configured to output controls for the movement generation device in response to receiving images captured by the one or more sensing devices, wherein the neural network comprises a first processing block configured to estimate a position of an occluded object in an image of the portion of the environment and generate an image processing output that is representative of a state of a first object within the image and that includes the estimate of the position of the occluded object, wherein the first object is different from the occluded object.

2. The robotic system of claim 1 , wherein the neural network retains a memory of the position of the occluded object within a state of the first processing block.

3. The robotic system of claim 2 , wherein the first processing block comprises recurrent neural network layers having states dependent on previously processed images.

4. The robotic system of claim 1 , wherein the neural network retains a memory of the position of the occluded object within an external memory communicatively coupled to the neural network.

5. The robotic system of claim 1 , wherein the image processing output includes at least one of a representation of a distance between the end effector and the first object and a representation of a distance between two objects within the image.

6. The robotic system of claim 1 , wherein the neural network comprises a second processing block configured to generate command signals for movement of the end effector based on at least one of a goal for the end effector, the image processing output, or a time dependent internal state of the second processing block.

7. The robotic system of claim 6 , wherein the second processing block comprises recurrent neural network layers in which a state of the second processing block is changed through processing of the image processing output such that processing of the image processing output by the second processing block is dependent on prior states of the second processing block.

8. The robotic system of claim 7 , wherein the third processing block comprises recurrent neural network layers in which a state of the third processing block is changed through processing of the image processing output and the command signal such that processing of the image processing output and the command signal is dependent on prior states of the third processing block.

9. The robotic system of claim 6 , wherein the neural network comprises a third processing block configured to generate the control for the movement generation device based on the image processing output and the command signals.

10. The robotic system of claim 1 , wherein the neural network comprises a multiple of the first processing block, and wherein each first processing block is configured to receive images from one of the one or more sensing devices.

11. A method of operating a robotic system, the method comprising:

receiving a first image of a portion of an environment including an end effector, wherein the end effector is coupled to a robotic arm that is responsive to a movement generation device;

generating a first image processing output representative of a state of a first object within the first image, wherein generating the image processing output comprises estimating a position of an occluded object in the first image, wherein the first image processing output includes the estimate of the position of the occluded object, and wherein the first object is different from the occluded object;

generating command signals for movement of the end effector; and

providing an output for control of the movement generation device based on the command signals and the image processing output, wherein providing the output causes the robotic arm to adjust at least one component of a spatial pose of the end effector.

12. The method of claim 11 , wherein generating command signals for movement of the end effector comprises identifying one or more movement goals for the end effector based on the image processing output.

13. The method of claim 11 , further comprising:

receiving a task for the end effector; and

identifying a first movement goal from the task;

wherein generating the command signals for the movement of the end effector comprises selecting a first processing block of a neural network configured to perform the first movement goal, and wherein selecting the first processing block causes the first processing block to generate the command signals.

14. The method of claim 11 , wherein the image processing output is generated by a processing block of a neural network, and wherein generating the image processing output further comprises capturing a memory of the position of the occluded object in a state of the processing block.

15. The method of claim 11 , wherein receiving the first image of the portion of the environment comprises receiving a plurality of images of the portion of the environment; and

wherein the image processing output is representative of states of objects within the plurality of images.

16. The method of claim 11 , wherein the output for control of the movement generation device is provided by a neural network having recurrent nodes, and further comprising adjusting the output for hysteresis in positioning of the end effector based on the first image and the recurrent nodes of the neural network.

17. One or more non-transitory computer-readable storage media storing computer-executable instructions for causing a computing system to perform operations comprising:

receiving an image captured by one or more sensing devices configured to capture images of a portion of an environment including an end effector, wherein the end effector is coupled to a robotic arm;

generating, by a processing block of a neural network, an image processing output representative of a state of a first object within the image, wherein generating the image processing output comprises estimating a position of one or more occluded objects in the image, wherein the image processing output includes the estimate of the position of the one or more occluded objects, wherein the first object is different from the one or more occluded objects;

generating command signals for movement of the end effector based on a movement goal for the end effector; and

providing an output for control of a movement generation device coupled to the robotic arm based on the set of command signals and the set of image processing outputs, wherein the output causes the robotic arm to adjust at least one component of a spatial pose of the end effector.

18. The one or more non-transitory computer-readable storage media of claim 17 , wherein the neural network includes recurrent layers, and wherein generating the image processing output further comprises capturing a memory of the position of the one or more occluded objects in a state of the neural network.

19. The one or more non-transitory computer-readable storage media of claim 17 , wherein the output for control of the movement of the generation device is provided by a neural network having recurrent nodes, and wherein the operations further comprise adjusting the output for hysteresis in positioning of the end effector based on the image and the recurrent nodes of the neural network.

US17/728,910

2019-01-01

2022-04-25

Software compensated robotics

Active

US11691274B2

( en )

Priority Applications (2)

Application Number

Priority Date

Filing Date

Title

US17/728,910

US11691274B2

( en )

2019-01-01

2022-04-25

Software compensated robotics

US18/209,853

US20230339104A1

( en )

2019-01-01

2023-06-14

Software Compensated Robotics

Applications Claiming Priority (2)

Application Number

Priority Date

Filing Date

Title

US16/237,721

US11312012B2

( en )

2019-01-01

Software compensated robotics

US17/728,910

US11691274B2

( en )

2019-01-01

2022-04-25

Software compensated robotics

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/237,721

Continuation

US11312012B2

( en )

2019-01-01

Software compensated robotics

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US18/209,853

Continuation

US20230339104A1

( en )

2019-01-01

2023-06-14

Software Compensated Robotics

Publications (2)

Publication Number

Publication Date

US20230018498A1

US20230018498A1 ( en )

2023-01-19

US11691274B2

true

US11691274B2 ( en )

2023-07-04

Family

ID=71123788

Family Applications (3)

Application Number

Title

Priority Date

Filing Date

US16/237,721

Active

2041-02-15

US11312012B2

( en )

2019-01-01

Software compensated robotics

US17/728,910

Active

US11691274B2

( en )

2019-01-01

2022-04-25

Software compensated robotics

US18/209,853

Pending

US20230339104A1

( en )

2019-01-01

2023-06-14

Software Compensated Robotics

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US16/237,721

Active

2041-02-15

US11312012B2

( en )

2019-01-01

Software compensated robotics

Family Applications After (1)

Application Number

Title

Priority Date

Filing Date

US18/209,853

Pending

US20230339104A1

( en )

2019-01-01

2023-06-14

Software Compensated Robotics

Country Status (1)

Country

Link

( 3 )

US11312012B2

( en )

Cited By (3)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20240037848A1

( en )

2022-07-27

2024-02-01

Sanctuary Cognitive Systems Corporation

Systems, methods, and computer program products for implementing object permanence in a simulated environment

US12521297B1

( en )

2020-10-09

2026-01-13

Apptronik, Inc.

Enhanced transparency exoskeleton

US12553500B2

( en )

2023-03-10

2026-02-17

Apptronik, Inc.

Linear actuator

Families Citing this family (26)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

JP7044047B2

( en )

2018-12-14

2022-03-30

ãã¨ã¿èªåè»æ ªå¼ä¼ç¤¾

robot

CN113924195B

( en )

2019-05-29

2025-03-14

ä¼å²æºå¨äººå¬å¸

Control of a multi-purpose robot arm

WO2021046530A1

( en )

2019-09-07

2021-03-11

Embodied Intelligence, Inc.

Three-dimensional computer vision system for robotic devices

CN114641378B

( en )

2019-09-07

2024-09-27

å·ç°æºè½æéå¬å¸

Systems and methods for robotic picking

US11911901B2

( en )

2019-09-07

2024-02-27

Embodied Intelligence, Inc.

Training artificial networks for robotic picking

JP7458741B2

( en )

2019-10-21

2024-04-01

ãã¤ãã³æ ªå¼ä¼ç¤¾

Robot control device and its control method and program

MX2022005751A

( en )

2019-11-12

2022-08-22

Bright Machines Inc

A software defined manufacturing/assembly system.

EP3904013B1

( en )

2020-04-27

2022-07-20

C.R.F. SocietÃ Consortile per Azioni

System for assisting an operator in a work station

US11998289B2

( en )

2020-05-12

2024-06-04

Aescape, Inc.

Method and system for autonomous therapy

US12122050B2

( en )

2020-09-09

2024-10-22

Aescape, Inc.

Method and system for generating a therapeutic massage plan

US11999061B2

( en )

2020-05-12

2024-06-04

Aescape, Inc.

Method and system for autonomous object manipulation

US11858144B2

( en )

2020-05-12

2024-01-02

Aescape, Inc.

Method and system for autonomous body interaction

DE102020113278B4

( en )

2020-05-15

2024-07-25

Gerhard Schubert Gesellschaft mit beschrÃ¤nkter Haftung

Method for controlling the operation of an industrial robot, industrial robot and arrangement for moving objects

CN112008696A

( en )

2020-09-07

2020-12-01

è²å°¼åæ¯(åäº¬)æºè½å¶é ææ¯å·¥ç¨æéå¬å¸

Industrial robot system based on vision

JP7531196B2

( en )

2020-09-09

2024-08-09

ãªã ãã³æ ªå¼ä¼ç¤¾

Robot system and picking method

CN116097315A

( en )

2020-10-20

2023-05-09

ä¸æçµåæ ªå¼ä¼ç¤¾

Electronic apparatus and control method thereof

US12204314B2

( en )

2020-11-10

2025-01-21

Bright Machines, Inc.

Method and apparatus for improved auto-calibration of a robotic cell

CN113255208B

( en )

2021-04-21

2023-05-12

æå·æ°åæºå¨äººææ¯è¡ä»½æéå¬å¸

Neural network model predictive control method for series elastic actuator of robot

US11845190B1

( en )

2021-06-02

2023-12-19

Google Llc

Injecting noise into robot simulation

WO2023288233A1

( en )

2021-07-16

2023-01-19

Bright Machines, Inc.

Method and apparatus for vision-based tool localization

US12427672B2

( en )

2021-09-02

2025-09-30

Khalifa University of Science and Technology and The Aerospace Holding Company LLC

Robotic manipulator with visual guidance and tactile sensing

US12280495B2

( en )

2021-09-07

2025-04-22

Khalifa University of Science and Technology

Robotic manipulator with camera and tactile sensing

CN113976390B

( en )

2021-09-18

2023-04-07

ç©åè½æºç§ææéå¬å¸

Flexible intelligent dispensing system and method based on machine learning

US20240058953A1

( en )

2022-08-19

2024-02-22

AMP Robotics Corporation

Object picking optimization

TWI894670B

( en )

2022-11-03

2025-08-21

ç¾åéå·§å¬å¸

Variable payload robot system, computer program product and method to programmatically control a robot

CN117021118B

( en )

2023-10-08

2023-12-15

ä¸åå¤§å¦

Dynamic compensation method for digital twin track error of parallel robot

Citations (4)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20170252922A1

( en )

2016-03-03

2017-09-07

Google Inc.

Deep machine learning methods and apparatus for robotic grasping

US20200114506A1

( en )

2017-12-05

2020-04-16

Google Llc

Viewpoint invariant visual servoing of robot end effector using recurrent neural network

US20200122321A1

( en )

2018-10-23

2020-04-23

X Development Llc

Machine learning methods and apparatus for automated robotic placement of secured object in appropriate location

US10930037B2

( en )

2016-02-25

2021-02-23

Fanuc Corporation

Image processing device for displaying object detected from input picture image

Family Cites Families (15)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US8060250B2

( en )

2008-12-15

2011-11-15

GM Global Technology Operations LLC

Joint-space impedance control for tendon-driven manipulators

US8204623B1

( en )

2009-02-13

2012-06-19

Hrl Laboratories, Llc

Planning approach for obstacle avoidance in complex environment using articulated redundant robot arm

US8364314B2

( en )

2009-04-30

2013-01-29

GM Global Technology Operations LLC

Method and apparatus for automatic control of a humanoid robot

US8260460B2

( en )

2009-09-22

2012-09-04

GM Global Technology Operations LLC

Interactive robot control system and method of use

US8467903B2

( en )

2009-09-22

2013-06-18

GM Global Technology Operations LLC

Tendon driven finger actuation system

US8511964B2

( en )

2009-09-22

2013-08-20

GM Global Technology Operations LLC

Humanoid robot

US8600552B2

( en )

2009-10-30

2013-12-03

Honda Motor Co., Ltd.

Information processing method, apparatus, and computer readable medium

US8805581B2

( en )

2012-02-21

2014-08-12

GM Global Technology Operations LLC

Procedural memory learning and robot control

US8996167B2

( en )

2012-06-21

2015-03-31

Rethink Robotics, Inc.

User interfaces for robot training

US20170106542A1

( en )

2015-10-16

2017-04-20

Amit Wolf

Robot and method of controlling thereof

JP2019518297A

( en )

2016-05-19

2019-06-27

Robot Assisted Object Learning Vision System

US11181886B2

( en )

2017-04-24

2021-11-23

Autodesk, Inc.

Closed-loop robotic deposition of material

EP3606707A2

( en )

2017-06-28

2020-02-12

Google LLC

Machine learning methods and apparatus for semantic robotic grasping

JP7122821B2

( en )

2017-12-15

2022-08-22

å·å´éå·¥æ¥æ ªå¼ä¼ç¤¾

Robot system and robot control method

US11014231B2

( en )

2018-05-03

2021-05-25

Fujitsu Limited

Robotic device task learning

2019

2019-01-01

US16/237,721

patent/US11312012B2/en

active

Active

2022

2022-04-25

US17/728,910

patent/US11691274B2/en

active

Active

2023

2023-06-14

US18/209,853

patent/US20230339104A1/en

active

Pending

Patent Citations (5)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US10930037B2

( en )

2016-02-25

2021-02-23

Fanuc Corporation

Image processing device for displaying object detected from input picture image

US20170252922A1

( en )

2016-03-03

2017-09-07

Google Inc.

Deep machine learning methods and apparatus for robotic grasping

US20190283245A1

( en )

2016-03-03

2019-09-19

Google Llc

Deep machine learning methods and apparatus for robotic grasping

US20200114506A1

( en )

2017-12-05

2020-04-16

Google Llc

Viewpoint invariant visual servoing of robot end effector using recurrent neural network

US20200122321A1

( en )

2018-10-23

2020-04-23

X Development Llc

Machine learning methods and apparatus for automated robotic placement of secured object in appropriate location

Cited By (4)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US12521297B1

( en )

2020-10-09

2026-01-13

Apptronik, Inc.

Enhanced transparency exoskeleton

US20240037848A1

( en )

2022-07-27

2024-02-01

Sanctuary Cognitive Systems Corporation

Systems, methods, and computer program products for implementing object permanence in a simulated environment

US12269169B2

( en )

2022-07-27

2025-04-08

Sanctuary Cognitive Systems Corporation

Systems, methods, and computer program products for implementing object permanence in a simulated environment

US12553500B2

( en )

2023-03-10

2026-02-17

Apptronik, Inc.

Linear actuator

Also Published As

Publication number

Publication date

US20230339104A1

( en )

2023-10-26

US20200206913A1

( en )

2020-07-02

US11312012B2

( en )

2022-04-26

US20230018498A1

( en )

2023-01-19

Software compensated robotics — Sanctuary Cognitive Systems Corporation (US11691274B2)

Related documents