ABSTRACT
Abstract
A robotic system includes a controller configured to obtain image data from one or more optical sensors and to determine one or more of a location and/or pose of a vehicle component based on the image data. The controller also is configured to determine a model of an external environment of the robotic system based on the image data and to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component. The controller also is configured to assign the tasks to the components of the robotic system and to communicate control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a continuation of U.S. patent application Ser. No. 16/240,237 (hereinafter the '237 Application), which was filed on Jan. 4, 2019, which is a continuation-in-part of U.S. patent application Ser. No. 15/292,605 (U.S. Pat. Pub. No. 2017-0341236), which was filed on Oct. 13, 2016, which claims priority to U.S. Provisional Application No. 62/342,510, filed May 27, 2016. The entire disclosure of each is incorporated herein by reference.
The '237 Application also is a continuation-in-part of U.S. patent application Ser. No. 15/885,289 (now U.S. Pat. No. 10,252,424), filed Jan. 31, 2018, which is a continuation of U.S. patent application Ser. No. 14/702,014 (now U.S. Pat. No. 9,889,566), filed May 1, 2015. The entire disclosure of each is incorporated herein by reference.
The '237 application also is a continuation-in-part of and claims priority to U.S. patent application Ser. No. 15/058,560 (now U.S. Pat. No. 10,272,573) filed Mar. 2, 2016, that claims priority to U.S. Provisional Application Nos. 62/269,523; 62/269,425; 62/269,377; and 62/269,481, all of which were filed Dec. 18, 2015. The entire disclosure of each is incorporated herein by reference.
FIELD
The subject matter described herein relates to systems and methods for autonomously maintaining vehicles.
BACKGROUND
The challenges in the modern vehicle yards are vast and diverse. Classification yards, or hump yards, play an important role as consolidation nodes in vehicle freight networks. At classification yards, inbound vehicle systems (e.g., trains) are disassembled and the cargo-carrying vehicles (e.g., railcars) are sorted by next common destination (or block). The efficiency of the yards in part drives the efficiency of the entire transportation network.
The hump yard is generally divided into three main areas: the receiving yard, where inbound vehicle systems arrive and are prepared for sorting; the class yard, where cargo-carrying vehicles in the vehicle systems are sorted into blocks; and the departure yard, where blocks of vehicles are assembled into outbound vehicle systems, inspected, and then depart.
Current solutions for field service operations are labor-intensive, dangerous, and limited by the operational capabilities of humans being able to make critical decisions in the presence of incomplete or incorrect information. Furthermore, efficient system level-operations require integrated system wide solutions, more than just point solutions to key challenges. The nature of these missions dictates that the tasks and environments cannot always be fully anticipated or specified at the design time, yet an autonomous solution may need the essential capabilities and tools to carry out the mission even if it encounters situations that were not expected.
Solutions for typical vehicle yard problems, such as brake bleeding, brake line lacing, coupling cars, etc., can require combining mobility, perception, and manipulation toward a tightly integrated autonomous solution. When placing robots in an outdoor environment, technical challenges largely increase, but field robotic application benefits both technically and economically. One key challenge in yard operation is that of bleeding brakes on inbound cars in the receiving yard. Railcars have pneumatic breaking systems that work on the concept of a pressure differential. The size of the brake lever is significantly small compared to the size of the environment and the cargo-carrying vehicles. Additionally, there are lots of variations on the shape, location, and the material of the brake levers. Coupled with that is the inherent uncertainty in the environment; every day, vehicles are placed at different locations, and the spaces between cars are very narrow and unstructured. As a result, an autonomous solution for maintenance (e.g., brake maintenance) of the vehicles presents a variety of difficult challenges.
BRIEF DESCRIPTION
In one embodiment, a robotic system includes a controller configured to obtain image data from one or more optical sensors and to determine one or more of a location and/or pose of a vehicle component based on the image data. The controller also is configured to determine a model of an external environment of the robotic system based on the image data and to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component. The controller also is configured to assign the tasks to the components of the robotic system and to communicate control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component. A propulsion system moves the robotic system based on the control signals, and a manipulator arm configured to actuate the rail vehicle component based on the control signals. The model of the external environment of the robotic system provides locations of objects external to the robotic system relative to the robotic system, grades off a surface on which the robotic system is traveling, and obstructions in the moving path of the robotic system, and the model of the external environment of the robotic system is determined only for a designated volume around the manipulator arm.
In one embodiment, a method includes obtaining image data from one or more optical sensors, determining one or more of a location or pose of a vehicle component based on the image data, determining a model of an external environment of the robotic system based on the image data, determining tasks to be performed by components of the robotic system to perform maintenance on the vehicle component, assigning the tasks to the components of the robotic system, and communicating control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component.
In one embodiment, a robotic system includes one or more optical sensors configured to generate image data representative of an external environment and a controller configured to obtain the image data and to determine one or more of a location or pose of a vehicle component based on the image data. The controller also can be configured to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component and to assign the tasks to the components of the robotic system based on the image data and based on a model of the external environment. The controller can be configured to communicate control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component. A propulsion system moves the robotic system based on the control signals, and a manipulator arm configured to actuate the rail vehicle component based on the control signals. The controller is configured to determine waypoints for the propulsion system of the robotic system to move the robotic system based on one or more of the tasks assigned to the propulsion system by the controller and on a mapping of a location of the robotic system in the model of the external environment determined by the controller, and 3D image data is examined using real-time simultaneous localization and mapping to determine the model of the external environment of the robotic system.
BRIEF DESCRIPTION OF THE DRAWINGS
The present inventive subject matter will be better understood from reading the following description of non-limiting embodiments, with reference to the attached drawings, wherein below:
FIG. 1 illustrates one embodiment of a robotic system;
FIG. 2 illustrates a control architecture used by the robotic system shown in FIG. 1 to move toward, grasp, and actuate a brake lever or rod according to one embodiment;
FIG. 3 illustrates 2D image data of a manipulator arm shown in FIG. 1 near a vehicle;
FIG. 4 illustrates one example of a model of an external environment around the manipulator arm;
FIG. 5 illustrates a flowchart of one embodiment of a method for autonomous control of a robotic system for vehicle maintenance;
FIG. 6 is a schematic block diagram of a robotic system in accordance with various embodiments;
FIG. 7 is a graph depicting control operations in accordance with various embodiments;
FIG. 8 is a schematic block diagram of a robotic control system in accordance with various embodiments;
FIG. 9 is a flowchart of a method for controlling a robot in accordance with various embodiments;
FIG. 10 schematically illustrates a brake system and an automation system according to one embodiment;
FIG. 11 A illustrates a peripheral view of an end-effector with an opening according to one embodiment;
FIG. 11 B illustrates a peripheral view of an end-effector without an opening according to one embodiment;
FIG. 12 illustrates a schematic block diagram of the automation system shown in FIG. 10 ; and
FIG. 13 illustrates a system block diagram of the automation system shown in FIG. 10 .
DETAILED DESCRIPTION
One or more embodiments of the inventive subject matter described herein provide robotic systems and methods that provide a large form factor mobile robot with an industrial manipulator arm to effectively detect, identify, and subsequently manipulate components to perform maintenance on vehicles, which can include inspection and/or repair of the vehicles. While the description herein focuses on manipulating brake levers of vehicles (e.g., rail vehicles) in order to bleed air brakes of the vehicles, not all maintenance operations performed by the robotic systems or using the methods described herein are limited to brake bleeding. One or more embodiments of the robotic systems and methods described herein can be used to perform other maintenance operations on vehicles, such as obtaining information from vehicles (e.g., AEI tag reading), inspecting vehicles (e.g., inspecting couplers between vehicles), air hose lacing, etc.
The robotic system autonomously navigates within a route corridor along the length of a vehicle system, moving from vehicle to vehicle within the vehicle system. An initial âcoarseâ estimate of a location of a brake rod or lever on a selected or designated vehicle in the vehicle system is provided to or obtained by the robotic system. This coarse estimate can be derived or extracted from a database or other memory structure that represents the vehicles present in the corridor (e.g., the vehicles on the same segment of a route within the yard). The robotic system moves through or along the vehicles and locates the brake lever rods on the side of one or more, or each, vehicle. The robotic system positions itself next to a brake rod to then actuate a brake release mechanism (e.g., to initiate brake bleeding) by manipulating the brake lever rod.
During autonomous navigation, the robotic system maintains a distance of separation (e.g., about four inches or ten centimeters) from the plane of the vehicle while moving forward toward the vehicle. In order to ensure real-time brake rod detection and subsequent estimation of the brake rod location, a two-stage detection strategy is utilized. Once the robotic system has moved to a location near to the brake rod, an extremely fast two-dimensional (2-D) vision-based search is performed by the robotic system to determine and/or confirm a coarse location of the brake rod. The second stage of the detection strategy involves building a dense model for template-based shape matching (e.g., of the brake rod) to identify the exact location and pose of the break rod. The robotic system can move to approach the brake rod as necessary to have the brake rod within reach of the robotic arm of the robotic system. Once the rod is within reach of the robotic arm, the robotic system uses the arm to manipulate and actuate the rod.
FIG. 1 illustrates one embodiment of a robotic system 100 . The robotic system 100 may be used to autonomously move toward, grasp, and actuate (e.g., move) a brake lever or rod on a vehicle in order to change a state of a brake system of the vehicle. For example, the robotic system 100 may autonomously move toward, grasp, and move a brake rod of an air brake system on a rail car in order to bleed air out of the brake system. The robotic system 100 includes a robotic vehicle 102 having a propulsion system 104 that operates to move the robotic system 100 . The propulsion system 104 may include one or more motors, power sources (e.g., batteries, alternators, generators, etc.), or the like, for moving the robotic system 100 . A controller 106 of the robotic system 100 includes hardware circuitry that includes and/or is connected with one or more processors (e.g., microprocessors, field programmable gate arrays, and/or integrated circuits) that direct operations of the robotic system 100 .
The robotic system 100 also includes
<figure-callout id="111" label="several sensors" filenames="US11865732-20240109-D00000.png,US
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a continuation of U.S. patent application Ser. No. 16/240,237 (hereinafter the '237 Application), which was filed on Jan. 4, 2019, which is a continuation-in-part of U.S. patent application Ser. No. 15/292,605 (U.S. Pat. Pub. No. 2017-0341236), which was filed on Oct. 13, 2016, which claims priority to U.S. Provisional Application No. 62/342,510, filed May 27, 2016. The entire disclosure of each is incorporated herein by reference.
The '237 Application also is a continuation-in-part of U.S. patent application Ser. No. 15/885,289 (now U.S. Pat. No. 10,252,424), filed Jan. 31, 2018, which is a continuation of U.S. patent application Ser. No. 14/702,014 (now U.S. Pat. No. 9,889,566), filed May 1, 2015. The entire disclosure of each is incorporated herein by reference.
The '237 application also is a continuation-in-part of and claims priority to U.S. patent application Ser. No. 15/058,560 (now U.S. Pat. No. 10,272,573) filed Mar. 2, 2016, that claims priority to U.S. Provisional Application Nos. 62/269,523; 62/269,425; 62/269,377; and 62/269,481, all of which were filed Dec. 18, 2015. The entire disclosure of each is incorporated herein by reference.
FIELD
The subject matter described herein relates to systems and methods for autonomously maintaining vehicles.
BACKGROUND
The challenges in the modern vehicle yards are vast and diverse. Classification yards, or hump yards, play an important role as consolidation nodes in vehicle freight networks. At classification yards, inbound vehicle systems (e.g., trains) are disassembled and the cargo-carrying vehicles (e.g., railcars) are sorted by next common destination (or block). The efficiency of the yards in part drives the efficiency of the entire transportation network.
The hump yard is generally divided into three main areas: the receiving yard, where inbound vehicle systems arrive and are prepared for sorting; the class yard, where cargo-carrying vehicles in the vehicle systems are sorted into blocks; and the departure yard, where blocks of vehicles are assembled into outbound vehicle systems, inspected, and then depart.
Current solutions for field service operations are labor-intensive, dangerous, and limited by the operational capabilities of humans being able to make critical decisions in the presence of incomplete or incorrect information. Furthermore, efficient system level-operations require integrated system wide solutions, more than just point solutions to key challenges. The nature of these missions dictates that the tasks and environments cannot always be fully anticipated or specified at the design time, yet an autonomous solution may need the essential capabilities and tools to carry out the mission even if it encounters situations that were not expected.
Solutions for typical vehicle yard problems, such as brake bleeding, brake line lacing, coupling cars, etc., can require combining mobility, perception, and manipulation toward a tightly integrated autonomous solution. When placing robots in an outdoor environment, technical challenges largely increase, but field robotic application benefits both technically and economically. One key challenge in yard operation is that of bleeding brakes on inbound cars in the receiving yard. Railcars have pneumatic breaking systems that work on the concept of a pressure differential. The size of the brake lever is significantly small compared to the size of the environment and the cargo-carrying vehicles. Additionally, there are lots of variations on the shape, location, and the material of the brake levers. Coupled with that is the inherent uncertainty in the environment; every day, vehicles are placed at different locations, and the spaces between cars are very narrow and unstructured. As a result, an autonomous solution for maintenance (e.g., brake maintenance) of the vehicles presents a variety of difficult challenges.
BRIEF DESCRIPTION
In one embodiment, a robotic system includes a controller configured to obtain image data from one or more optical sensors and to determine one or more of a location and/or pose of a vehicle component based on the image data. The controller also is configured to determine a model of an external environment of the robotic system based on the image data and to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component. The controller also is configured to assign the tasks to the components of the robotic system and to communicate control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component. A propulsion system moves the robotic system based on the control signals, and a manipulator arm configured to actuate the rail vehicle component based on the control signals. The model of the external environment of the robotic system provides locations of objects external to the robotic system relative to the robotic system, grades off a surface on which the robotic system is traveling, and obstructions in the moving path of the robotic system, and the model of the external environment of the robotic system is determined only for a designated volume around the manipulator arm.
In one embodiment, a method includes obtaining image data from one or more optical sensors, determining one or more of a location or pose of a vehicle component based on the image data, determining a model of an external environment of the robotic system based on the image data, determining tasks to be performed by components of the robotic system to perform maintenance on the vehicle component, assigning the tasks to the components of the robotic system, and communicating control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component.
In one embodiment, a robotic system includes one or more optical sensors configured to generate image data representative of an external environment and a controller configured to obtain the image data and to determine one or more of a location or pose of a vehicle component based on the image data. The controller also can be configured to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component and to assign the tasks to the components of the robotic system based on the image data and based on a model of the external environment. The controller can be configured to communicate control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component. A propulsion system moves the robotic system based on the control signals, and a manipulator arm configured to actuate the rail vehicle component based on the control signals. The controller is configured to determine waypoints for the propulsion system of the robotic system to move the robotic system based on one or more of the tasks assigned to the propulsion system by the controller and on a mapping of a location of the robotic system in the model of the external environment determined by the controller, and 3D image data is examined using real-time simultaneous localization and mapping to determine the model of the external environment of the robotic system.
BRIEF DESCRIPTION OF THE DRAWINGS
The present inventive subject matter will be better understood from reading the following description of non-limiting embodiments, with reference to the attached drawings, wherein below:
FIG. 1 illustrates one embodiment of a robotic system;
FIG. 2 illustrates a control architecture used by the robotic system shown in FIG. 1 to move toward, grasp, and actuate a brake lever or rod according to one embodiment;
FIG. 3 illustrates 2D image data of a manipulator arm shown in FIG. 1 near a vehicle;
FIG. 4 illustrates one example of a model of an external environment around the manipulator arm;
FIG. 5 illustrates a flowchart of one embodiment of a method for autonomous control of a robotic system for vehicle maintenance;
FIG. 6 is a schematic block diagram of a robotic system in accordance with various embodiments;
FIG. 7 is a graph depicting control operations in accordance with various embodiments;
FIG. 8 is a schematic block diagram of a robotic control system in accordance with various embodiments;
FIG. 9 is a flowchart of a method for controlling a robot in accordance with various embodiments;
FIG. 10 schematically illustrates a brake system and an automation system according to one embodiment;
FIG. 11 A illustrates a peripheral view of an end-effector with an opening according to one embodiment;
FIG. 11 B illustrates a peripheral view of an end-effector without an opening according to one embodiment;
FIG. 12 illustrates a schematic block diagram of the automation system shown in FIG. 10 ; and
FIG. 13 illustrates a system block diagram of the automation system shown in FIG. 10 .
DETAILED DESCRIPTION
One or more embodiments of the inventive subject matter described herein provide robotic systems and methods that provide a large form factor mobile robot with an industrial manipulator arm to effectively detect, identify, and subsequently manipulate components to perform maintenance on vehicles, which can include inspection and/or repair of the vehicles. While the description herein focuses on manipulating brake levers of vehicles (e.g., rail vehicles) in order to bleed air brakes of the vehicles, not all maintenance operations performed by the robotic systems or using the methods described herein are limited to brake bleeding. One or more embodiments of the robotic systems and methods described herein can be used to perform other maintenance operations on vehicles, such as obtaining information from vehicles (e.g., AEI tag reading), inspecting vehicles (e.g., inspecting couplers between vehicles), air hose lacing, etc.
The robotic system autonomously navigates within a route corridor along the length of a vehicle system, moving from vehicle to vehicle within the vehicle system. An initial âcoarseâ estimate of a location of a brake rod or lever on a selected or designated vehicle in the vehicle system is provided to or obtained by the robotic system. This coarse estimate can be derived or extracted from a database or other memory structure that represents the vehicles present in the corridor (e.g., the vehicles on the same segment of a route within the yard). The robotic system moves through or along the vehicles and locates the brake lever rods on the side of one or more, or each, vehicle. The robotic system positions itself next to a brake rod to then actuate a brake release mechanism (e.g., to initiate brake bleeding) by manipulating the brake lever rod.
During autonomous navigation, the robotic system maintains a distance of separation (e.g., about four inches or ten centimeters) from the plane of the vehicle while moving forward toward the vehicle. In order to ensure real-time brake rod detection and subsequent estimation of the brake rod location, a two-stage detection strategy is utilized. Once the robotic system has moved to a location near to the brake rod, an extremely fast two-dimensional (2-D) vision-based search is performed by the robotic system to determine and/or confirm a coarse location of the brake rod. The second stage of the detection strategy involves building a dense model for template-based shape matching (e.g., of the brake rod) to identify the exact location and pose of the break rod. The robotic system can move to approach the brake rod as necessary to have the brake rod within reach of the robotic arm of the robotic system. Once the rod is within reach of the robotic arm, the robotic system uses the arm to manipulate and actuate the rod.
FIG. 1 illustrates one embodiment of a robotic system 100 . The robotic system 100 may be used to autonomously move toward, grasp, and actuate (e.g., move) a brake lever or rod on a vehicle in order to change a state of a brake system of the vehicle. For example, the robotic system 100 may autonomously move toward, grasp, and move a brake rod of an air brake system on a rail car in order to bleed air out of the brake system. The robotic system 100 includes a robotic vehicle 102 having a propulsion system 104 that operates to move the robotic system 100 . The propulsion system 104 may include one or more motors, power sources (e.g., batteries, alternators, generators, etc.), or the like, for moving the robotic system 100 . A controller 106 of the robotic system 100 includes hardware circuitry that includes and/or is connected with one or more processors (e.g., microprocessors, field programmable gate arrays, and/or integrated circuits) that direct operations of the robotic system 100 .
The robotic system 100 also includes
several sensors
108 , 109 , 110 , 111 , 112 that measure or detect various conditions used by the robotic system 100 to move toward, grasp, and actuate brake levers. The sensors 108 - 111 are optical sensors, such as cameras, infrared projectors and/or detectors. While four optical sensors
108 , 110 are shown, alternatively, the robotic system 100 may have a single optical sensor, less than four optical sensors, or more than four optical sensors. In one embodiment, the sensors
109 , 111 are RGB cameras and the sensors
110 , 112 are structured-light three-dimensional (3-D) cameras, but alternatively may be another type of camera.
The sensor 112 is a touch sensor that detects when a manipulator arm 114 of the robotic system 100 contacts or otherwise engages a surface or object. The touch sensor 112 may be one or more of a variety of touch-sensitive devices, such as a switch (e.g., that is closed upon touch or contact), a capacitive element (e.g., that is charged or discharged upon touch or contact), or the like.
The manipulator arm 114 is an elongated body of the robotic system 100 that can move in a variety of directions, grasp, and pull and/or push a brake rod. The controller 106 may be operably connected with the propulsion system 104 and the manipulator arm 114 to control movement of the robotic system 100 and/or the arm 114 , such as by one or more wired and/or wireless connections. The controller 106 may be operably connected with the sensors 108 - 112 to receive data obtained, detected, or measured by the sensors 108 - 112 .
FIG. 2 illustrates a control architecture 200 used by the robotic system 100 to move toward, grasp, and actuate a brake lever or rod according to one embodiment. The architecture 200 may represent the operations performed by various components of the robotic system 100 . The architecture 200 is composed of three layers: a physical layer 202 , a processing layer 204 , and a planning layer 206 . The physical layer 202 includes the robotic vehicle 102 (including the propulsion system 104 , shown as âGrizzly Robotâ in FIG. 2 ), the sensors 108 - 112 (e.g., the âRGB Cameraâ as the sensors
109 , 111 and the âKinect Sensorâ as the sensors
108 , 110 in FIG. 2 ), and the manipulator arm 114 (e.g., the âSIA20F Robotâ in FIG. 2 ).
The processing layer 204 is embodied in the controller 106 , and dictates operation of the robotic system 100 . The processing layer 204 performs or determines how the robotic system 100 will move or operate to perform various tasks in a safe and/or efficient manner. The operations determined by the processing layer 204 can be referred to as modules. These modules can represent the algorithms or software used by the processing layer 204 to determine how to perform the operations of the robotic system 100 , or optionally represent the hardware circuitry of the controller 106 that determines how to perform the operations of the robotic system 100 . The modules are shown in FIG. 1 inside the controller 106 .
The modules of the processing layer 204 include a deliberation module 208 , a perception module 210 , a navigation module 212 , and a manipulation module 214 . The deliberation module 208 is responsible for planning and coordinating all behaviors or movements of the robotic system 100 . The deliberation module 208 can determine how the various physical components of the robotic system 100 move in order to avoid collision with each other, with vehicles, with human operators, etc., while still moving to perform various tasks. The deliberation module 208 receives processed information from one or more of the sensors 108 - 112 and determines when the robotic vehicle 102 and/or manipulator arm 114 are to move based on the information received or otherwise provided by the sensors 108 - 112 .
The perception module 210 receives data provided by the sensors 108 - 112 and processes this data to determine the relative positions and/or orientations of components of the vehicles. For example, the perception module 210 may receive image data provided by the sensors 108 - 111 and determine the location of a brake lever relative to the robotic system 100 , as well as the orientation (e.g., pose) of the brake lever. At least some of the operations performed by the perception module 210 are shown in FIG. 2 . For example, the perception module 210 can perform 2D processing of image data provided by the sensors
109 , 111 . This 2D processing can involve receiving image data from the sensors 109 , 111 (âDetectionâ in FIG. 2 ) and examining the image data to identify components or objects external to the robotic system 100 (e.g., components of vehicles, âSegmentationâ in FIG. 2 ). The perception module 210 can perform 3D processing of image data provided by the sensors
108 , 110 . This 3D processing can involve identifying different portions or segments of the objects identified via the 2D processing (â3D Segmentationâ in FIG. 2 ). From the 2D and 3D image processing, the perception module 210 may determine the orientation of one or more components of the vehicle, such as a pose of a brake lever (âPose estimationâ in FIG. 2 ).
The navigation module 212 determines the control signals generated by the controller 106 and communicated to the propulsion system 104 to direct how the propulsion system 104 moves the robotic system 100 . The navigation module 212 may use a real-time appearance-based mapping (RTAB-Map) algorithm (or a variant thereof) to plan how to move the robotic system 100 . Alternatively, another algorithm may be used.
The navigation module 212 may use modeling of the environment around the robotic system 100 to determine information used for planning motion of the robotic system 100 . Because the actual environment may not be previously known and/or may dynamically change (e.g., due to moving human operators, moving vehicles, errors or discrepancies between designated and actual locations of objects, etc.), a model of the environment may be determined by the controller 106 and used by the navigation module 212 to determine where and how to move the robotic system 100 while avoiding collisions. The manipulation module 214 determines how to control the manipulator arm 114 to engage (e.g., touch, grasp, etc.) one or more components of a vehicle, such as a brake lever.
In the planning layer 206 , the information obtained by the sensors 108 - 112 and state information of the robotic system 100 are collected from the lower layers
202 , 204 . According to the requirements of a task to be completed or performed by the robotic system 100 , the controller 106 (e.g., within the planning layer 206 ) will make different decisions based on the current task-relevant situation being performed or the next task to be performed by the robotic system 100 . A state machine can tie the
layers
202 , 204 , 206 together and transfer signals between the navigation module 212 and the perception module 210 , and then to the manipulation module 214 . If there is an emergency stop signal generated or there is error information reported by one or more of the modules, the controller 106 may responsively trigger safety primitives such as stopping movement of the robotic system 100 to prevent damage to the robotic system 100 and/or surrounding environment.
As shown in FIG. 2 , the processing layer 204 of the controller 106 may receive image data 216 from the sensors
108 , 110 . This image data 216 can represent or include 3D image data representative of the environment that is external to the robotic system 100 . This image data 216 is used by the processing layer 204 of the controller 106 to generate a model or other representation of the environment external to the robotic system 100 (âEnvironmental Modelingâ in FIG. 2 ) in one embodiment. The environmental modeling can represent locations of objects relative to the robotic system 100 , grades of the surface on which the robotic vehicle 102 is traveling, obstructions in the moving path of the robotic system 100 , etc. The 3D image data 216 optionally can be examined using real-time simultaneous localization and mapping (SLAM) to model the environment around the robotic system 100 .
The processing layer 204 can receive the image data 216 from the sensors
108 , 110 and/or image data 218 from the sensors
109 , 111 . The image data 218 can represent or include 2D image data representative of the environment that is external to the robotic system 100 . The 2D image data 218 can be used by the processing layer 204 of the controller 106 to identify objects that may be components of a vehicle, such as a brake lever (â2D Processingâ in FIG. 2 ). This identification may be performed by detecting potential objects (âDetectionâ in FIG. 2 ) based on the shapes and/or sizes of the objects in the 2D image data and segmenting the objects into smaller components (âSegmentationâ in FIG. 2 ). The 3D image data 216 can be used by the processing layer 204 to further examine these objects and determine whether the objects identified in the 2D image data 218 are or are not designated objects, or objects of interest, such as a component to be grasped, touched, moved, or otherwise actuated by the robotic system 100 to achieve or perform a designated task (e.g., moving a brake lever to bleed air brakes of a vehicle). In one embodiment, the processing layer 204 of the controller 106 uses the 3D segmented image data and the 2D segmented image data to determine an orientation (e.g., pose) of an object of interest (âPose estimationâ in FIG. 2 ). For example, based on the segments of a brake lever in the 3D and 2D image data, the processing layer 204 can determine a pose of the brake lever.
The planning layer 206 of the controller 106 can receive at least some of this information to determine how to operate the robotic system 100 . For example, the planning layer 206 can receive a model 220 of the environment surrounding the robotic system 100 from the processing layer 204 , an estimated or determined pose 222 of an object-of-interest (e.g., a brake lever) from the processing layer 204 , and/or a location 224 of the robotic system 100 within the environment that is modeled from the processing layer 204 .
In order to move in the environment, the robotic system 100 generates the model 220 of the external environment in order to understand the environment. In one embodiment, the robotic system 100 may be limited to moving only along the length of a vehicle system formed from multiple vehicles (e.g., a train typically about 100 rail cars long), and does not need to move longer distance. As a result, more global planning of movements of the robotic system 100 may not be needed or generated. For local movement planning and movement, the planning layer 206 can use a structured light-based SLAM algorithm, such as real-time appearance-based mapping (RTAB-Map), that is based on an incremental appearance-based loop closure detector. Using RTAB-Map, the planning layer 206 of the controller 106 can determine the location of the robotic system 100 relative to other objects in the environment, which can then be used to close a motion control loop and prevent collisions between the robotic system 100 and other objects. The point cloud data provided as the 3D image data can be used recognize the surfaces or planes of the vehicles. This information is used to keep the robotic system 100 away from the vehicles and maintain a pre-defined distance of separation from the vehicles.
In one embodiment, the model 220 is a grid-based representation of the environment around the robotic system 100 . The 3D image data collected using the sensors
108 , 110 can include point cloud data provided by one or more structured light sensors. The point cloud data points are processed and grouped into a grid.
FIG. 3 illustrates 2D image data 218 of the manipulator arm 114 near a vehicle. FIG. 4 illustrates one example of the model 220 of the environment around the manipulator arm 114 . The model 220 may be created by using grid cubes 402 with designated sizes (e.g., ten centimeters by ten centimeters by ten centimeters to represent different portions of the objects detected using the 2D and/or 3D image data
218 , 216 . In order to reduce the time needed to generate the model 220 , only a designated volume around the arm 114 may be modeled (e.g., the area within a sphere having a radius of 2.5 meters or another distance).
Returning to the description of the control architecture</f
CLAIMS
Claims ( 20 )
What is claimed is:
1. A robotic system comprising:
a controller configured to obtain image data from one or more sensors, the controller also configured to determine a location and a pose of a vehicle component based on the image data, the controller configured to determine a model of an external environment of the robotic system based on the image data, the controller configured to determine a mapping of a location of the robotic system in the model of the external environment, the model of the external environment providing locations of objects external to the robotic system relative to the location of the robotic system and grades of a surface on which the robotic system is configured to travel,
the controller configured to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component,
the controller configured to determine a sequence of movements of the components of the robotic system based at least in part on the tasks to be performed and the locations of the objects external to the robotic system relative to the location of the robotic system and the grades of the surface on which the robotic system is configured to travel,
the controller configured to communicate control signals to the components of the robotic system to move the components based on the sequence of movements of the components to perform the tasks.
2. The robotic system of claim 1 , wherein the model of the external environment of the robotic system provides obstructions in a moving path of the robotic system relative to the location of the robotic system in the model of the external environment.
3. The robotic system of claim 1 , wherein at least one of the tasks includes at least one of the components of the robotic system actuating the vehicle component.
4. The robotic system of claim 1 , wherein the controller is configured to autonomously move one or more of the components of the robotic system based on the model of the external environment and the sequence of movements of the one or more of the components.
5. The robotic system of claim 1 , wherein the model of the external environment provides a location of objects relative to one or more of the vehicle component or the components of the robotic system.
6. The robotic system of claim 1 , wherein the model of the external environment is a grid-based representation of the external environment based on the image data.
7. The robotic system of claim 1 , wherein the tasks are determined based on one or more of the model of the external environment or the location and pose of the vehicle component.
8. The robotic system of claim 1 , further comprising a propulsion system configured to move the robotic system based on the control signals.
9. The robotic system of claim 8 , wherein the propulsion system is configured to move the robotic system from a first location to a second location, wherein the controller is configured to determine a new model of the external environment based on movement of the robotic system from the first location to the second location, the new model of the external environment providing new locations of the objects external to the robotic system based on the movement of the robotic system to the second location.
10. The robotic system of claim 1 , wherein the model of the external environment is determined for a designated volume around one or more of the robotic system or the vehicle component.
11. The robotic system of claim 1 , wherein the model of the external environment is determined based on one or more of two-dimensional (2D) or three-dimensional (3D) image data from the one or more sensors.
12. A method comprising:
obtaining image data from one or more optical sensors;
determining a location and a pose of a vehicle component based on the image data;
determining a model of an external environment of a robotic system based on the image data;
determining a mapping of a location of the robotic system in the model of the external environment, the model of the external environment providing locations of objects external to the robotic system and grades of a surface on which the robotic system is configured to travel;
determining tasks to be performed by components of the robotic system to perform maintenance on the vehicle component;
determining a sequence of movements of the components of the robotic system based at least in part on the tasks to be performed and the locations of the objects external to the robotic system relative to the location of the robotic system and the grades of the surface on which the robotic system is configured to move; and
communicating control signals to the components of the robotic system to move the components based on the sequence of movements of the components to perform the tasks.
13. The method of claim 12 , wherein at least one of the tasks includes at least one of the components of the robotic system actuating the vehicle component.
14. The method of claim 12 , wherein the model of the external environment of the robotic system provides obstructions in a moving path of the robotic system relative to the location of the robotic system in the model of the external environment.
15. The method of claim 12 , further comprising autonomously moving one or more of the components of the robotic system based on the model of the external environment and the sequence of movements of the one or more of the components.
16. The method of claim 12 , wherein the model of the external environment provides a location of objects relative to one or more of the vehicle component or the components of the robotic system.
17. The method of claim 12 , wherein the model of the external environment is a grid-based representation of the external environment based on the image data.
18. The method of claim 12 , wherein the tasks are determined based on one or more of the model of the external environment or the location and pose of the vehicle component.
19. The method of claim 12 , wherein the model of the external environment is determined for a designated volume around one or more of the robotic system or the vehicle component.
20. A robotic system comprising:
a controller configured to obtain image data from one or more sensors, the controller also configured to determine a location and a pose of a vehicle component based on the image data and to determine a model of an external environment of the robotic system based on the image data, the controller configured to determine a mapping of a location of the robotic system in the model of the external environment, the model of the external environment providing locations of objects external to the robotic system relative to the location of the robotic system, grades of a surface on which the robotic system is configured to travel, and obstructions in a moving path of the robotic system relative to the location of the robotic system, the controller configured to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component; and
a propulsion system configured to move the robotic system based on the control signals, wherein the control signals of the propulsion system are based on one or more of the model of the external environment or the location and pose of the vehicle component,
the controller configured to determine a sequence of movements of the components of the robotic system based on one or more of the tasks to be performed by the components, the model of the external environment, or the location and pose of the vehicle component,
wherein the controller is configured to communicate control signals to the components of the robotic system to move the components according to the sequence of movements of the components to perform the tasks.
US17/246,009
2015-05-01
2021-04-30
Integrated robotic system and method for autonomous vehicle maintenance
Active
2035-09-04
US11865732B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US17/246,009
US11865732B2
( en )
2015-05-01
2021-04-30
Integrated robotic system and method for autonomous vehicle maintenance
US18/524,579
US20240091953A1
( en )
2015-05-01
2023-11-30
Integrated robotic system and method for autonomous vehicle maintenance
Applications Claiming Priority (11)
Application Number
Priority Date
Filing Date
Title
US14/702,014
US9889566B2
( en )
2015-05-01
2015-05-01
Systems and methods for control of robotic manipulation
US201562269523P
2015-12-18
2015-12-18
US201562269481P
2015-12-18
2015-12-18
US201562269377P
2015-12-18
2015-12-18
US201562269425P
2015-12-18
2015-12-18
US15/058,560
US10272573B2
( en )
2015-12-18
2016-03-02
Control system and method for applying force to grasp a brake lever
US201662342510P
2016-05-27
2016-05-27
US15/292,605
US20170341236A1
( en )
2016-05-27
2016-10-13
Integrated robotic system and method for autonomous vehicle maintenance
US15/885,289
US10252424B2
( en )
2015-05-01
2018-01-31
Systems and methods for control of robotic manipulation
US16/240,237
US11020859B2
( en )
2015-05-01
2019-01-04
Integrated robotic system and method for autonomous vehicle maintenance
US17/246,009
US11865732B2
( en )
2015-05-01
2021-04-30
Integrated robotic system and method for autonomous vehicle maintenance
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US16/240,237
Continuation
US11020859B2
( en )
2015-05-01
2019-01-04
Integrated robotic system and method for autonomous vehicle maintenance
Related Child Applications (1)
Application Number
Title
Priority Date
Filing Date
US18/524,579
Continuation-In-Part
US20240091953A1
( en )
2015-05-01
2023-11-30
Integrated robotic system and method for autonomous vehicle maintenance
Publications (2)
Publication Number
Publication Date
US20210252712A1
US20210252712A1 ( en )
2021-08-19
US11865732B2
true
US11865732B2 ( en )
2024-01-09
Family
ID=66326646
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US16/240,237
Active
2035-10-15
US11020859B2
( en )
2015-05-01
2019-01-04
Integrated robotic system and method for autonomous vehicle maintenance
US17/246,009
Active
2035-09-04
US11865732B2
( en )
2015-05-01
2021-04-30
Integrated robotic system and method for autonomous vehicle maintenance
Family Applications Before (1)
Application Number
Title
Priority Date
Filing Date
US16/240,237
Active
2035-10-15
US11020859B2
( en )
2015-05-01
2019-01-04
Integrated robotic system and method for autonomous vehicle maintenance
Country Status (1)
Country
Link
US
( 2 )
US11020859B2
( en )
Families Citing this family (27)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
GB2554363B
( en )
2016-09-21
2021-12-08
Cmr Surgical Ltd
User interface device
US11945115B2
( en )
*
2018-06-14
2024-04-02
Yamaha Hatsudoki Kabushiki Kaisha
Machine learning device and robot system provided with same
CN210418988U
( en )
*
2018-11-07
2020-04-28
ä¸æµ·å¾æ£®æªæ¥äººå·¥æºè½ç§ææéå ¬å¸
Mobile hoisting equipment control system, server and mobile hoisting equipment
JP7135750B2
( en )
*
2018-11-12
2022-09-13
å¯å£«éæ ªå¼ä¼ç¤¾
Learning program, learning method, learning device, detection program, detection method, and detection device
CN110370284A
( en )
*
2019-08-07
2019-10-25
å京å天ä¸çºªæ§è¡è¡ä»½æéå ¬å¸
A kind of automatic control system of explosion-proof autonomous crusing robot
CN114080304B
( en )
*
2019-08-22
2024-05-14
欧å§é¾æ ªå¼ä¼ç¤¾
Control device, control method, and control program
US11633853B2
( en )
*
2020-05-06
2023-04-25
Eagle Technology, Llc
Dynamic manipulator strength augmentation
WO2022054292A1
( en )
*
2020-09-14
2022-03-17
ä¸è±é»æ©æ ªå¼ä¼ç¤¾
Robot control device
CN112207839A
( en )
*
2020-09-15
2021-01-12
西å®äº¤é大å¦
Mobile household service robot and method
US12330720B1
( en )
2020-10-28
2025-06-17
Robotic Research Opco, Llc
Rear trailer hostler
US12393915B2
( en )
2020-12-18
2025-08-19
Strong Force Vcn Portfolio 2019, Llc
Variable-focus dynamic vision for robotic system
WO2022133016A1
( en )
2020-12-18
2022-06-23
Boston Dynamics, Inc.
Limiting arm forces and torques
US11999059B2
( en )
2020-12-18
2024-06-04
Boston Dynamics, Inc.
Limiting arm forces and torques
US20230109096A1
( en )
2020-12-18
2023-04-06
Strong Force Vcn Portfolio 2019, Llc
Maintenance Prediction and Health Monitoring for Robotic Fleet Management
US11931898B2
( en )
2020-12-22
2024-03-19
Boston Dynamics, Inc.
Arm and body coordination
CN113103225B
( en )
*
2021-03-03
2022-06-10
éåºå¤§å¦
Mobile robot multi-stage stable and autonomous docking method based on image measurement
US12384410B2
( en )
2021-03-05
2025-08-12
The Research Foundation For The State University Of New York
Task-motion planning for safe and efficient urban driving
GB2598037B
( en )
*
2021-04-29
2022-09-14
X Tend Robotics Inc
Robotic device for distributing designated items
EP4337467A4
( en )
2021-05-11
2025-05-07
Strong Force VCN Portfolio 2019, LLC
Systems, methods, kits, and apparatuses for edge-distributed storage and querying in value chain networks
US11766775B2
( en )
*
2021-05-20
2023-09-26
Ubkang (Qingdao) Technology Co., Ltd.
Method, device and computer-readable storage medium for designing serial manipulator
US11872512B2
( en )
*
2022-02-24
2024-01-16
Xtend Ai Inc.
Robot air filter
WO2023212044A1
( en )
2022-04-26
2023-11-02
Robotic Research Opco, Llc
Autonomous gladhands coupling system
CN114770567A
( en )
*
2022-04-28
2022-07-22
å½ç½å±±ä¸ççµåå ¬å¸éå²ä¾çµå ¬å¸
Remote control method and system for distribution live working robot
WO2024076856A1
( en )
2022-10-02
2024-04-11
Xtend Ai Inc.
Robotic device for distributing designated items
US12617097B2
( en )
*
2023-10-30
2026-05-05
International Business Machines Corporation
Dynamic alteration of operational parameters of a machine
WO2026006829A1
( en )
*
2024-06-28
2026-01-02
Gecko Robotics, Inc.
System, method, and apparatus for field support of inspection operations
CN119658694B
( en )
*
2025-01-03
2025-10-10
大è¿ç工大å¦
A multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning
Citations (6)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20080086241A1
( en )
*
2006-10-06
2008-04-10
Irobot Corporation
Autonomous Behaviors for a Remove Vehicle
US20100076631A1
( en )
*
2008-09-19
2010-03-25
Mian Zahid F
Robotic vehicle for performing rail-related actions
US20130311153A1
( en )
*
2012-05-15
2013-11-21
Caterpillar Inc.
Virtual environment and method for sorting among potential route plans for operating autonomous machine at work site
US20140324291A1
( en )
*
2003-03-20
2014-10-30
Agjunction Llc
Gnss and optical guidance and machine control
US9283674B2
( en )
*
2014-01-07
2016-03-15
Irobot Corporation
Remotely operating a mobile robot
US9452528B1
( en )
*
2012-03-05
2016-09-27
Vecna Technologies, Inc.
Controller device and method
Family Cites Families (1)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US11185985B2
( en )
*
2015-01-05
2021-11-30
Bell Helicopter Textron Inc.
Inspecting components using mobile robotic inspection systems
2019
2019-01-04
US
US16/240,237
patent/US11020859B2/en
active
Active
2021
2021-04-30
US
US17/246,009
patent/US11865732B2/en
active
Active
Patent Citations (7)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20140324291A1
( en )
*
2003-03-20
2014-10-30
Agjunction Llc
Gnss and optical guidance and machine control
US20080086241A1
( en )
*
2006-10-06
2008-04-10
Irobot Corporation
Autonomous Behaviors for a Remove Vehicle
US20100076631A1
( en )
*
2008-09-19
2010-03-25
Mian Zahid F
Robotic vehicle for performing rail-related actions
US9452528B1
( en )
*
2012-03-05
2016-09-27
Vecna Technologies, Inc.
Controller device and method
US20130311153A1
( en )
*
2012-05-15
2013-11-21
Caterpillar Inc.
Virtual environment and method for sorting among potential route plans for operating autonomous machine at work site
US9283674B2
( en )
*
2014-01-07
2016-03-15
Irobot Corporation
Remotely operating a mobile robot
US20160243698A1
( en )
*
2014-01-07
2016-08-25
Irobot Corporation
Remotely operating a mobile robot
Also Published As
Publication number
Publication date
US20210252712A1
( en )
2021-08-19
US20190134821A1
( en )
2019-05-09
US11020859B2
( en )
2021-06-01
Similar Documents
Publication
Publication Date
Title
US11020859B2
( en )
2021-06-01
Integrated robotic system and method for autonomous vehicle maintenance
US10252424B2
( en )
2019-04-09
Systems and methods for control of robotic manipulation
US10675765B2
( en )
2020-06-09
Control system and method for applying force to grasp a target object
US20170341236A1
( en )
2017-11-30
Integrated robotic system and method for autonomous vehicle maintenance
US11745355B2
( en )
2023-09-05
Control device, control method, and non-transitory computer-readable storage medium
US10471595B2
( en )
2019-11-12
Systems and methods for control of robotic manipulation
US10759051B2
( en )
2020-09-01
Architecture and methods for robotic mobile manipulation system
US11927969B2
( en )
2024-03-12
Control system and method for robotic motion planning and control
CA2883622C
( en )
2017-07-11
Localization within an environment using sensor fusion
CN115319764A
( en )
2022-11-11
Robot based on multi-mode fusion in complex limited environment and operation method
US20160023352A1
( en )
2016-01-28
SURROGATE: A Body-Dexterous Mobile Manipulation Robot with a Tracked Base
Rambow et al.
2012
Autonomous manipulation of deformable objects based on teleoperated demonstrations
US20170165839A1
( en )
2017-06-15
Control system and method for brake bleeding
Cacace et al.
2015
Aerial service vehicles for industrial inspection: task decomposition and plan execution
Al-Hussaini et al.
2020
Human-supervised semi-autonomous mobile manipulators for safely and efficiently executing machine tending tasks
CN119458364A
( en )
2025-02-18
A humanoid robot grasping method based on three-dimensional vision
JP5803769B2
( en )
2015-11-04
Mobile robot
US10933526B2
( en )
2021-03-02
Method and robotic system for manipulating instruments
US20240091953A1
( en )
2024-03-21
Integrated robotic system and method for autonomous vehicle maintenance
Rousseau et al.
2023
Constant distance and orientation following of an unknown surface with a cable-driven parallel robot
George et al.
2024
System for autonomous management of retail shelves using an omnidirectional dual-arm robot with a novel soft gripper
Chen et al.
2013
Semiautonomous industrial mobile manipulation for industrial applications
Koch et al.
2024
Active manipulator-aided docking of underactuated autonomous underwater vehicles
Changchun et al.
2015
Research of visual servo control system for space intelligent robot
Rafique
2022
Adaptive nonlinear control for unmanned aerial vehicles: Visual servoing and aerial manipulation
Legal Events
Date
Code
Title
Description
2021-04-30
AS
Assignment
Owner name : GE GLOBAL SOURCING LLC, CONNECTICUT
Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PATRICK, ROMANO;SEN, SHIRAJ;JAIN, ARPIT;AND OTHERS;SIGNING DATES FROM 20190103 TO 20190227;REEL/FRAME:056102/0018
Owner name : TRANSPORTATION IP HOLDINGS, LLC, CONNECTICUT
Free format text : CHANGE OF NAME;ASSIGNOR:GE GLOBAL SOURCING LLC;REEL/FRAME:056109/0317
Effective date : 20191112
2021-04-30
FEPP
Fee payment procedure
Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY
2021-07-13
STPP
Information on status: patent application and granting procedure in general
Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION
2022-11-01
STPP
Information on status: patent application and granting procedure in general
Free format text : NON FINAL ACTION MAILED
2023-06-27
STPP
Information on status: patent application and granting procedure in general
Free format text : RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER
2023-07-03
STPP
Information on status: patent application and granting procedure in general
Free format text : ADVISORY ACTION COUNTED, NOT YET MAILED
2023-07-25
STPP
Information on status: patent application and granting procedure in general
Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION
2023-08-29
STPP
Information on status: patent application and granting procedure in general
Free format text : NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS
2023-11-28
STPP
Information on status: patent application and granting procedure in general
Free format text : PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED
2023-12-20
STCF
Information on status: patent grant
Free format text : PATENTED CASE