ABSTRACT
Abstract
The technology relates to determining a source of a horn honk or other noise in an environment around one or more self-driving vehicles. Aspects of the technology leverage real-time information from a group of self-driving vehicles regarding received acoustical information. The location and pose of each self-driving vehicle in the group, along with the precise arrangement of acoustical sensors on each vehicle, can be used to triangulate or otherwise identify the actual location in the environment for the origin of the horn honk or other sound. Other sensor information, map data, and additional data can be used narrow down or refine the location of a likely noise source. Once the location and source of the noise is known, each self-driving vehicle can use that information to modify current driving operations and/or use it as part of a reinforcement learning approach for future driving situations.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
The present application is a continuation of U.S. patent application Ser. No. 16/934,132, filed Jul. 21, 2020, the entire disclosure of which is incorporated herein by reference.
BACKGROUND
Self-driving vehicles that operate in an autonomous driving mode may transport passengers, cargo or other items from one location to another. During operation, horn honks in the vehicle's external environment may provide insight into behavior of another vehicle that may or may not be operating in an autonomous driving mode, behavior of the self-driving vehicle itself, or interaction between the different vehicles. Unfortunately, it can be difficult to determine which other vehicle is honking, especially in a congested area, when the line-of-sight might be occluded, or in an otherwise noisy environment. The inability of the self-driving vehicle to properly determine which vehicle is honking or the reason for the honking may limit the ability of the self-driving vehicle to take corrective action or otherwise change its driving behavior.
BRIEF SUMMARY
The technology relates to approaches for determining what vehicle or other object in the environment is honking or issuing different acoustical information that may assist a self-driving vehicle in managing how it operates in an autonomous driving mode. In particular, aspects of the technology leverage real-time information from a group of self-driving vehicles regarding received acoustical information. The location and pose (e.g., position and orientation along the roadway or pitch, yaw and roll of the vehicle chassis) of each self-driving vehicle in the group, along with the precise arrangement of acoustical sensors on each vehicle, can be used to triangulate or otherwise identify the actual location in the environment for the origin of the horn honk or other sound. Information from other sensors of the group of vehicles, map data, and other information available to the group of vehicles can be used to narrow down or refine the location of a likely horn honk. Obtained information may be processed locally, for instance in real time by each vehicle, or one or more vehicles of the group (or a centralized remote computer processing system) may coordinate how certain information is processed by some or all of the vehicles.
According to one aspect, a method of operating a vehicle in an autonomous driving mode is provided. The method comprises obtaining, by one or more acoustical sensors of a perception system of the vehicle, audio sensor data, the one or more acoustical sensors being configured to detect sounds in an external environment around the vehicle; receiving, by one or more processors of the vehicle, audio sensor data from one or more other vehicles operating in an autonomous driving mode, the received audio sensor data including direction-of-arrival and timestamp information regarding a detected sound from the external environment; evaluating, by the one or more processors, the obtained audio sensor data and the received audio sensor data based on a location of the vehicle and locations of the one or more other vehicles associated with the timestamp information to identify an estimated location at which a specific sound emanated and a particular object that likely issued the specific sound; and based on a type of the specific sound, the estimated location, and a type of the particular object, the one or more processors controlling operation of the vehicle in the autonomous driving mode.
In one example, the method further comprising using non-acoustical sensor data to identify at least one of the estimated location or the type of the particular object. In this case, the non-acoustical sensor data may be received from the one or more other vehicles. The method may include identifying the type of the specific sound. In this case, the specific sound type may be a horn honk or a siren noise.
Identifying the estimated location may include evaluating the received audio sensor data according to positions of each acoustical sensor of an array of sensors from each of the one or more other vehicles. The evaluating may be further based on a pose of the vehicle and poses of each of the one or more other vehicles.
In another example, the method includes identifying a first likelihood region as the estimated location of the specific sound based on the audio sensor data obtained from the perception system of the vehicle; identifying one or more additional likelihood regions as the estimated location of the specific sound based on the audio sensor data received from each of the one or more other vehicles; and comparing the first likelihood region and the one or more additional likelihood regions to identify a highly localized region as the estimated location of the particular object.
The method may also include determining whether the specific sound relates to operation of the vehicle in the autonomous driving mode. Here, the method may further comprise changing a driving operation of the vehicle in response to determining that the specific sound relates to the operation of the vehicle in the autonomous driving mode. In this case, the method may include using the determination that the specific sound relates to the operation of the vehicle in the autonomous driving mode in a reinforcement learning process.
In yet another example, the method also includes transmitting the obtained audio sensor data to (i) at least one of the one or more other vehicles, or (ii) a back-end server system remote from the vehicle and the one or more other vehicles. Here, the process may include transmitting a set of non-acoustical sensor data to (i) the at least one of the one or more other vehicles, or (ii) to the back-end server system. The method may further comprise selecting the one or more other vehicles based on at least one of a proximity to the vehicle, an estimated proximity to the particular object, a driving condition, or an environmental condition. In this case, the method may also include pre-processing the obtained audio sensor data to perform one or more of noise cancellation, filtering, signal averaging or signal boosting. The pre-processing may include pre-processing multiple samples of the obtained audio sensor data to account for one or more of (i) changes over time in positioning of the one or more acoustical sensors of the perception system of the vehicle, (ii) signal attenuation, or (iii) a Doppler shift caused by relative movement of the particular object and the vehicle.
In another example, receiving the audio sensor data from one or more other vehicles may include one of (i) directly receiving the audio sensor data from the one or more other vehicles, or (ii) indirectly receiving the audio sensor data from a back-end server system. And in a further example, the method also includes evaluating a change in the specific sound over time; and controlling operation of the vehicle in the autonomous driving mode includes adjusting operation of the vehicle in the autonomous driving mode based on the change in the specific sound over time.
According to another aspect of the technology, a method is provided for assisting one or more vehicles operating in an autonomous driving mode. The method comprises receiving, by one or more processors of a server system, audio sensor data from a set of vehicles each operating in an autonomous driving mode, the received audio sensor data having been detected by one or more acoustical sensors of a perception system of each vehicle of the set of vehicles, the received audio sensor data including direction-of-arrival and timestamp information regarding a detected sound from an external environment of each corresponding vehicle; evaluating, by the one or more processors, the received audio sensor data based on locations of the vehicles associated with the timestamp information to identify an estimated location in the external environment at which a specific sound emanated and a particular object that likely issued the specific sound; and transmitting, to the set of vehicles, the estimated location of the specific sound and at least one of a type of the specific sound and a type of the particular object.
In one example, this method also includes identifying a plurality of likelihood regions as the estimated location of the specific sound based on the audio sensor data received from each of the vehicles in the set; and comparing the plurality of likelihood regions to identify a highly localized region as the estimated location of the particular object; wherein transmitting the estimated location comprises transmitting the highly localized region to the set of vehicles. In another example, this method includes determining whether the specific sound relates to operation of a given one of the set of vehicles.
BRIEF DESCRIPTION OF THE DRAWINGS
FIGS. 1 A-B illustrate example self-driving vehicles in accordance with aspects of the technology.
FIGS. 1 C-D illustrate an example cargo-type vehicle configured for use with aspects of the technology.
FIG. 2 illustrates components of a self-driving vehicle in accordance with aspects of the technology.
FIGS. 3 A-B are block diagrams of systems of an example cargo-type vehicle in accordance with aspects of the technology.
FIGS. 4 A-G illustrate views of an elevated roof sensor assembly in accordance with aspects of the technology.
FIGS. 5 A-B illustrate exemplary microphone sensor arrangements in accordance with aspects of the technology.
FIGS. 6 A-B illustrate an in-vehicle scenario in accordance with aspects of the technology.
FIGS. 7 A-C illustrate a multi-vehicle noise evaluation scenario in accordance with aspects of the technology.
FIGS. 8 A-B illustrate an example system in accordance with aspects of the technology.
FIG. 9 illustrates an example method in accordance with aspects of the technology.
FIG. 10 illustrates another example method in accordance with aspects of the technology.
DETAILED DESCRIPTION
Having multiple self-driving vehicles in the same area that are able to effectively communicate certain data in real time allows for accurate localization of the three-dimensional (3D) position of a honking horn or other sound. In contrast, when only one acoustical sensor or a single self-driving vehicle receives a sound, it may only be possible to obtain the direction-of-arrival for the sound, or, at best, a very coarse range resolution using a microphone array. This may be insufficient in many instances to accurately identify what vehicle or other object issued the sound, or may otherwise limit the ability of the system to determine the reason for a horn honk and either take corrective action or respond in a beneficial manner.
A self-driving vehicle that is configured to operate in an autonomous driving mode may be part of a fleet of vehicles, or otherwise able to communicate with other nearby self-driving vehicles. Thus, a self-driving vehicle may be in a situation where there are multiple nearby self-driving vehicles in the same area, e.g., within a few blocks of one another, along the same stretch of freeway, within line of sight, less than 100-250 meters away, etc. The likelihood of multiple self-driving vehicles being in the same area at the same time will increase as self-driving vehicles become more prevalent, for instance as part of a taxi-type service or a package delivery service.
By way of example and as explained further below, different self-driving vehicles in a group of nearby vehicles are able to share sensor data from a microphone or other acoustical array with the other nearby vehicles upon a detected honk or other noise. This sensor data could be raw or processed. In the former case, the audio data may not be particularly sizeable (e.g., on the order of tens to hundreds of kilobytes of data) and could be re-transmitted over low-data rate wireless links without significant latency. This could allow for highly granular (including sub-wavelength) range resolution. In the latter case, the processed data may include, for instance, just the direction-of-arrival and timestamp, or other information sufficient for triangulation. This would result in transmitting even fewer bytes of data, which would enable low latency communication at a small onboard processing cost at each vehicle.
Example Vehicle Systems
FIG. 1 A illustrates a perspective view of an example passenger vehicle 100 , such as a minivan or sport utility vehicle (SUV). FIG. 1 B illustrates a perspective view of another example passenger vehicle 150 , such as a sedan. The passenger vehicles may include various sensors for obtaining information about the vehicle's external environment. For instance, a roof-top housing unit (roof pod assembly) 102 may include a lidar sensor as well as various cameras (e.g., optical or infrared), radar units, acoustical sensors (e.g., microphone or sonar-type sensors), inertial (e.g., accelerometer, gyroscope, etc.) or other sensors (e.g., positioning sensors such as GPS sensors). Housing 104 , located at the front end of vehicle 100 , and housings
106 a , 106 b on the driver's and passenger's sides of the vehicle may each incorporate lidar, radar, camera and/or other sensors. For example, housing 106 a may be located in front of the driver's side door along a quarter panel of the vehicle. As shown, the passenger vehicle 100 also includes housings
108 a , 108 b for radar units, lidar and/or cameras also located towards the rear roof portion of the vehicle. Additional lidar, radar units and/or cameras (not shown) may be located at other places along the vehicle 100 . For instance, arrow 110 indicates that a sensor unit (not shown) may be positioned along the rear of the vehicle 100 , suc
CROSS REFERENCE TO RELATED APPLICATIONS
The present application is a continuation of U.S. patent application Ser. No. 16/934,132, filed Jul. 21, 2020, the entire disclosure of which is incorporated herein by reference.
BACKGROUND
Self-driving vehicles that operate in an autonomous driving mode may transport passengers, cargo or other items from one location to another. During operation, horn honks in the vehicle's external environment may provide insight into behavior of another vehicle that may or may not be operating in an autonomous driving mode, behavior of the self-driving vehicle itself, or interaction between the different vehicles. Unfortunately, it can be difficult to determine which other vehicle is honking, especially in a congested area, when the line-of-sight might be occluded, or in an otherwise noisy environment. The inability of the self-driving vehicle to properly determine which vehicle is honking or the reason for the honking may limit the ability of the self-driving vehicle to take corrective action or otherwise change its driving behavior.
BRIEF SUMMARY
The technology relates to approaches for determining what vehicle or other object in the environment is honking or issuing different acoustical information that may assist a self-driving vehicle in managing how it operates in an autonomous driving mode. In particular, aspects of the technology leverage real-time information from a group of self-driving vehicles regarding received acoustical information. The location and pose (e.g., position and orientation along the roadway or pitch, yaw and roll of the vehicle chassis) of each self-driving vehicle in the group, along with the precise arrangement of acoustical sensors on each vehicle, can be used to triangulate or otherwise identify the actual location in the environment for the origin of the horn honk or other sound. Information from other sensors of the group of vehicles, map data, and other information available to the group of vehicles can be used to narrow down or refine the location of a likely horn honk. Obtained information may be processed locally, for instance in real time by each vehicle, or one or more vehicles of the group (or a centralized remote computer processing system) may coordinate how certain information is processed by some or all of the vehicles.
According to one aspect, a method of operating a vehicle in an autonomous driving mode is provided. The method comprises obtaining, by one or more acoustical sensors of a perception system of the vehicle, audio sensor data, the one or more acoustical sensors being configured to detect sounds in an external environment around the vehicle; receiving, by one or more processors of the vehicle, audio sensor data from one or more other vehicles operating in an autonomous driving mode, the received audio sensor data including direction-of-arrival and timestamp information regarding a detected sound from the external environment; evaluating, by the one or more processors, the obtained audio sensor data and the received audio sensor data based on a location of the vehicle and locations of the one or more other vehicles associated with the timestamp information to identify an estimated location at which a specific sound emanated and a particular object that likely issued the specific sound; and based on a type of the specific sound, the estimated location, and a type of the particular object, the one or more processors controlling operation of the vehicle in the autonomous driving mode.
In one example, the method further comprising using non-acoustical sensor data to identify at least one of the estimated location or the type of the particular object. In this case, the non-acoustical sensor data may be received from the one or more other vehicles. The method may include identifying the type of the specific sound. In this case, the specific sound type may be a horn honk or a siren noise.
Identifying the estimated location may include evaluating the received audio sensor data according to positions of each acoustical sensor of an array of sensors from each of the one or more other vehicles. The evaluating may be further based on a pose of the vehicle and poses of each of the one or more other vehicles.
In another example, the method includes identifying a first likelihood region as the estimated location of the specific sound based on the audio sensor data obtained from the perception system of the vehicle; identifying one or more additional likelihood regions as the estimated location of the specific sound based on the audio sensor data received from each of the one or more other vehicles; and comparing the first likelihood region and the one or more additional likelihood regions to identify a highly localized region as the estimated location of the particular object.
The method may also include determining whether the specific sound relates to operation of the vehicle in the autonomous driving mode. Here, the method may further comprise changing a driving operation of the vehicle in response to determining that the specific sound relates to the operation of the vehicle in the autonomous driving mode. In this case, the method may include using the determination that the specific sound relates to the operation of the vehicle in the autonomous driving mode in a reinforcement learning process.
In yet another example, the method also includes transmitting the obtained audio sensor data to (i) at least one of the one or more other vehicles, or (ii) a back-end server system remote from the vehicle and the one or more other vehicles. Here, the process may include transmitting a set of non-acoustical sensor data to (i) the at least one of the one or more other vehicles, or (ii) to the back-end server system. The method may further comprise selecting the one or more other vehicles based on at least one of a proximity to the vehicle, an estimated proximity to the particular object, a driving condition, or an environmental condition. In this case, the method may also include pre-processing the obtained audio sensor data to perform one or more of noise cancellation, filtering, signal averaging or signal boosting. The pre-processing may include pre-processing multiple samples of the obtained audio sensor data to account for one or more of (i) changes over time in positioning of the one or more acoustical sensors of the perception system of the vehicle, (ii) signal attenuation, or (iii) a Doppler shift caused by relative movement of the particular object and the vehicle.
In another example, receiving the audio sensor data from one or more other vehicles may include one of (i) directly receiving the audio sensor data from the one or more other vehicles, or (ii) indirectly receiving the audio sensor data from a back-end server system. And in a further example, the method also includes evaluating a change in the specific sound over time; and controlling operation of the vehicle in the autonomous driving mode includes adjusting operation of the vehicle in the autonomous driving mode based on the change in the specific sound over time.
According to another aspect of the technology, a method is provided for assisting one or more vehicles operating in an autonomous driving mode. The method comprises receiving, by one or more processors of a server system, audio sensor data from a set of vehicles each operating in an autonomous driving mode, the received audio sensor data having been detected by one or more acoustical sensors of a perception system of each vehicle of the set of vehicles, the received audio sensor data including direction-of-arrival and timestamp information regarding a detected sound from an external environment of each corresponding vehicle; evaluating, by the one or more processors, the received audio sensor data based on locations of the vehicles associated with the timestamp information to identify an estimated location in the external environment at which a specific sound emanated and a particular object that likely issued the specific sound; and transmitting, to the set of vehicles, the estimated location of the specific sound and at least one of a type of the specific sound and a type of the particular object.
In one example, this method also includes identifying a plurality of likelihood regions as the estimated location of the specific sound based on the audio sensor data received from each of the vehicles in the set; and comparing the plurality of likelihood regions to identify a highly localized region as the estimated location of the particular object; wherein transmitting the estimated location comprises transmitting the highly localized region to the set of vehicles. In another example, this method includes determining whether the specific sound relates to operation of a given one of the set of vehicles.
BRIEF DESCRIPTION OF THE DRAWINGS
FIGS. 1 A-B illustrate example self-driving vehicles in accordance with aspects of the technology.
FIGS. 1 C-D illustrate an example cargo-type vehicle configured for use with aspects of the technology.
FIG. 2 illustrates components of a self-driving vehicle in accordance with aspects of the technology.
FIGS. 3 A-B are block diagrams of systems of an example cargo-type vehicle in accordance with aspects of the technology.
FIGS. 4 A-G illustrate views of an elevated roof sensor assembly in accordance with aspects of the technology.
FIGS. 5 A-B illustrate exemplary microphone sensor arrangements in accordance with aspects of the technology.
FIGS. 6 A-B illustrate an in-vehicle scenario in accordance with aspects of the technology.
FIGS. 7 A-C illustrate a multi-vehicle noise evaluation scenario in accordance with aspects of the technology.
FIGS. 8 A-B illustrate an example system in accordance with aspects of the technology.
FIG. 9 illustrates an example method in accordance with aspects of the technology.
FIG. 10 illustrates another example method in accordance with aspects of the technology.
DETAILED DESCRIPTION
Having multiple self-driving vehicles in the same area that are able to effectively communicate certain data in real time allows for accurate localization of the three-dimensional (3D) position of a honking horn or other sound. In contrast, when only one acoustical sensor or a single self-driving vehicle receives a sound, it may only be possible to obtain the direction-of-arrival for the sound, or, at best, a very coarse range resolution using a microphone array. This may be insufficient in many instances to accurately identify what vehicle or other object issued the sound, or may otherwise limit the ability of the system to determine the reason for a horn honk and either take corrective action or respond in a beneficial manner.
A self-driving vehicle that is configured to operate in an autonomous driving mode may be part of a fleet of vehicles, or otherwise able to communicate with other nearby self-driving vehicles. Thus, a self-driving vehicle may be in a situation where there are multiple nearby self-driving vehicles in the same area, e.g., within a few blocks of one another, along the same stretch of freeway, within line of sight, less than 100-250 meters away, etc. The likelihood of multiple self-driving vehicles being in the same area at the same time will increase as self-driving vehicles become more prevalent, for instance as part of a taxi-type service or a package delivery service.
By way of example and as explained further below, different self-driving vehicles in a group of nearby vehicles are able to share sensor data from a microphone or other acoustical array with the other nearby vehicles upon a detected honk or other noise. This sensor data could be raw or processed. In the former case, the audio data may not be particularly sizeable (e.g., on the order of tens to hundreds of kilobytes of data) and could be re-transmitted over low-data rate wireless links without significant latency. This could allow for highly granular (including sub-wavelength) range resolution. In the latter case, the processed data may include, for instance, just the direction-of-arrival and timestamp, or other information sufficient for triangulation. This would result in transmitting even fewer bytes of data, which would enable low latency communication at a small onboard processing cost at each vehicle.
Example Vehicle Systems
FIG. 1 A illustrates a perspective view of an example passenger vehicle 100 , such as a minivan or sport utility vehicle (SUV). FIG. 1 B illustrates a perspective view of another example passenger vehicle 150 , such as a sedan. The passenger vehicles may include various sensors for obtaining information about the vehicle's external environment. For instance, a roof-top housing unit (roof pod assembly) 102 may include a lidar sensor as well as various cameras (e.g., optical or infrared), radar units, acoustical sensors (e.g., microphone or sonar-type sensors), inertial (e.g., accelerometer, gyroscope, etc.) or other sensors (e.g., positioning sensors such as GPS sensors). Housing 104 , located at the front end of vehicle 100 , and housings
106 a , 106 b on the driver's and passenger's sides of the vehicle may each incorporate lidar, radar, camera and/or other sensors. For example, housing 106 a may be located in front of the driver's side door along a quarter panel of the vehicle. As shown, the passenger vehicle 100 also includes housings
108 a , 108 b for radar units, lidar and/or cameras also located towards the rear roof portion of the vehicle. Additional lidar, radar units and/or cameras (not shown) may be located at other places along the vehicle 100 . For instance, arrow 110 indicates that a sensor unit (not shown) may be positioned along the rear of the vehicle 100 , such as on or adjacent to the bumper. Depending on the vehicle type and sensor housing configuration(s), acoustical sensors may be disposed in any or all of these housings around the vehicle.
Arrow 114 indicates that the roof pod 102 as shown includes a base section coupled to the roof of the vehicle. And arrow 116 indicated that the roof pod 102 also includes an upper section raised above the base section. Each of the base section and upper section may house different sensor units configured to obtain information about objects and conditions in the environment around the vehicle. The roof pod 102 and other sensor housings may also be disposed along vehicle 150 of FIG. 1 B . By way of example, each sensor unit may include one or more sensors of the types described above, such as lidar, radar, camera (e.g., optical or infrared), acoustical (e.g., a passive microphone or active sound emitting sonar-type sensor), inertial (e.g., accelerometer, gyroscope, etc.) or other sensors (e.g., positioning sensors such as GPS sensors).
FIGS. 1 C-D illustrate an example cargo vehicle 150 , such as a tractor-trailer truck. The truck may include, e.g., a single, double or triple trailer, or may be another medium or heavy duty truck such as in commercial weight classes 4 through 8. As shown, the truck includes a tractor unit 152 and a single cargo unit or trailer 154 . The trailer 154 may be fully enclosed, open such as a flat bed, or partially open depending on the type of cargo to be transported. In this example, the tractor unit 152 includes the engine and steering systems (not shown) and a cab 156 for a driver and any passengers.
The trailer 154 includes a hitching point, known as a kingpin, 158 . The kingpin 158 is typically formed as a solid steel shaft, which is configured to pivotally attach to the tractor unit 152 . In particular, the kingpin 158 attaches to a trailer coupling 160 , known as a fifth-wheel, that is mounted rearward of the cab. For a double or triple tractor-trailer, the second and/or third trailers may have simple hitch connections to the leading trailer. Or, alternatively, each trailer may have its own kingpin. In this case, at least the first and second trailers could include a fifth-wheel type structure arranged to couple to the next trailer.
As shown, the tractor may have one or more sensor units
162 , 164 disposed therealong. For instance, one or more sensor units 162 may be disposed on a roof or top portion of the cab 156 , and one or more side sensor units 164 may be disposed on left and/or right sides of the cab 156 . Sensor units may also be located along other regions of the cab 156 , such as along the front bumper or hood area, in the rear of the cab, adjacent to the fifth-wheel, underneath the chassis, etc. The trailer 154 may also have one or more sensor units 166 disposed therealong, for instance along a side panel, front, rear, roof and/or undercarriage of the trailer 154 .
As with the sensor units of the passenger vehicle of FIGS. 1 A-B , each sensor unit of the cargo vehicle may include one or more sensors, such as lidar, radar, camera (e.g., optical or infrared), acoustical (e.g., microphone or sonar-type sensor), inertial (e.g., accelerometer, gyroscope, etc.) or other sensors (e.g., positioning sensors such as GPS sensors).
While certain aspects of the disclosure may be particularly useful in connection with specific types of vehicles, the vehicle may be different types of vehicle including, but not limited to, cars, motorcycles, cargo vehicles, buses, recreational vehicles, emergency vehicles, construction equipment, etc.
There are different degrees of autonomy that may occur for a vehicle operating in a partially or fully autonomous driving mode. The U.S. National Highway Traffic Safety Administration and the Society of Automotive Engineers have identified different levels to indicate how much, or how little, the vehicle controls the driving. For instance, Level 0 has no automation and the driver makes all driving-related decisions. The lowest semi-autonomous mode, Level 1, includes some drive assistance such as cruise control. At this level, the vehicle may operate in a strictly driver-information system without needing any automated control over the vehicle. Here, the vehicle's onboard sensors, relative positional knowledge between them, and a way for them to exchange data, can be employed to implement aspects of the technology as discussed herein. Level 2 has partial automation of certain driving operations, while Level 3 involves conditional automation that can enable a person in the driver's seat to take control as warranted. In contrast, Level 4 is a high automation level where the vehicle is able to drive without assistance in select conditions. And Level 5 is a fully autonomous mode in which the vehicle is able to drive without assistance in all situations. The architectures, components, systems and methods described herein can function in any of the semi or fully-autonomous modes, e.g., Levels 1-5, which are referred to herein as autonomous driving modes. Thus, reference to an autonomous driving mode includes both partial and full autonomy.
FIG. 2 illustrates a block diagram 200 with various components and systems of an exemplary vehicle, such as passenger vehicle
100 or 150 , to operate in an autonomous driving mode. As shown, the block diagram 200 includes one or more computing devices 202 , such as computing devices containing one or more processors 204 , memory 206 and other components typically present in general purpose computing devices. The memory 206 stores information accessible by the one or more processors 204 , including instructions 208 and data 210 that may be executed or otherwise used by the processor(s) 204 . The computing system may control overall operation of the vehicle when operating in an autonomous driving mode.
The memory 206 stores information accessible by the processors 204 , including instructions 208 and data 210 that may be executed or otherwise used by the processors 204 . For instance, the memory may include acoustic models or the like to perform, e.g., noise cancellation, triangulation, trilateration, vehicle or other object recognition, honk or other sound recognition, etc. The memory 206 may be of any type capable of storing information accessible by the processor, including a computing device-readable medium. The memory is a non-transitory medium such as a hard-drive, memory card, optical disk, solid-state, etc. Systems may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.
The instructions 208 may be any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by the processor(s). For example, the instructions may be stored as computing device code on the computing device-readable medium. In that regard, the terms âinstructionsâ, âmodulesâ and âprogramsâ may be used interchangeably herein. The instructions may be stored in object code format for direct processing by the processor, or in any other computing device language including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. The data 210 may be retrieved, stored or modified by one or more processors 204 in accordance with the instructions 208 . In one example, some or all of the memory 206 may be an event data recorder or other secure data storage system configured to store vehicle diagnostics and/or detected sensor data, which may be on board the vehicle or remote, depending on the implementation.
The processors 204 may be any conventional processors, such as commercially available CPUs. Alternatively, each processor may be a dedicated device such as an ASIC or other hardware-based processor. Although FIG. 2 functionally illustrates the processors, memory, and other elements of computing devices 202 as being within the same block, such devices may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, the memory 206 may be a hard drive or other storage media located in a housing different from that of the processor(s) 204 . Accordingly, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.
In one example, the computing devices 202 may form an autonomous driving computing system incorporated into vehicle 100 . The autonomous driving computing system may be capable of communicating with various components of the vehicle. For example, the computing devices 202 may be in communication with various systems of the vehicle, including a driving system including a deceleration system 212 (for controlling braking of the vehicle), acceleration system 214 (for controlling acceleration of the vehicle), steering system 216 (for controlling the orientation of the wheels and direction of the vehicle), signaling system 218 (for controlling turn signals), navigation system 220 (for navigating the vehicle to a location or around objects) and a positioning system 222 (for determining the position of the vehicle, e.g., including the vehicle's pose, e.g., position and orientation along the roadway or pitch, yaw and roll of the vehicle chassis relative to a coordinate system). The autonomous driving computing system may employ a planner module 223 , in accordance with the navigation system 220 , the positioning system 222 and/or other components of the system, e.g., for determining a route from a starting point to a destination or for making modifications to various driving aspects in view of current or expected traction conditions.
The computing devices 202 are also operatively coupled to a perception system 224 (for detecting objects in the vehicle's environment), a power system 226 (for example, a battery and/or gas or diesel powered engine) and a transmission system 230 in order to control the movement, speed, etc., of the vehicle in accordance with the instructions 208 of memory 206 in an autonomous driving mode which does not require or need continuous or periodic input from a passenger of the vehicle. Some or all of the wheels/ tires 228 are coupled to the transmission system 230 , and the computing devices 202 may be able to receive information about tire pressure, balance and other factors that may impact driving in an autonomous mode.
The computing devices 202 may control the direction and speed of the vehicle, e.g., via the planner module 223 , by controlling various components. By way of example, computing devices 202 may navigate the vehicle to a destination location completely autonomously using data from the map information and navigation system 220 . Computing devices 202 may use the positioning system 222 to determine the vehicle's location and the perception system 224 to detect and respond to objects when needed to reach the location safely. In order to do so, computing devices 202 may cause the vehicle to accelerate (e.g., by increasing fuel or other energy provided to the engine by acceleration system 214 ), decelerate (e.g., by decreasing the fuel supplied to the engine, changing gears, and/or by applying brakes by deceleration system 212 ), change direction (e.g., by turning the front or other wheels of vehicle 100 by steering system 216 ), and signal such changes (e.g., by lighting turn signals of signaling system 218 ). Thus, the acceleration system 214 and deceleration system 212 may be a part of a drivetrain or other type of transmission system 230 that includes various components between an engine of the vehicle and the wheels of the vehicle. Again, by controlling these systems, computing devices 202 may also control the transmission system 230 of the vehicle in order to maneuver the vehicle autonomously.
Navigation system 220 may be used by computing devices 202 in order to determine and follow a route to a location. In this regard, the navigation system 220 and/or memory 206 may store map information, e.g., highly detailed maps that computing devices 202 can use to navigate or control the vehicle. As an example, these maps may identify the shape and elevation of roadways, lane markers, intersections, crosswalks, speed limits, traffic signal lights, buildings, signs, real time traffic information, vegetation, or other such objects and information. The lane markers may include features such as solid or broken double or single lane lines, solid or broken lane lines, reflectors, etc. A given lane may be associated with left and/or right lane lines or other lane markers that define the boundary of the lane. Thus, most lanes may be bounded by a left edge of one lane line and a right edge of another lane line.
The perception system 224 includes sensors 232 for detecting objects external to the vehicle. The detected objects may be other vehicles, obstacles in the roadway, traffic signals, signs, trees, etc. The sensors may 232 may also detect certain aspects of weather conditions, such as snow, rain or water spray, or puddles, ice or other materials on the roadway.
By way of example only, the perception system 224 may include one or more microphones or other acoustical arrays, for instance arranged along the roof pod 102 and/or other sensor assembly housings. The microphones may be capable of detecting sounds across a wide frequency band (e.g., 50 Hz-25 KHz) such as to detect various types of noises, or may be designed to pick up sounds in specific narrow bands (e.g., 300-500 Hz) designed for use with horn honks or other vehicle noises. If such noises are subject to regulation (e.g., SAE J1849 for emergency vehicle sirens), the microphones may be configured to pick up such sounds. In one scenario, the microphones are able to detect sounds across a wide frequency band, and post-processing using one or more acoustic modules may be performed to identify sounds in particular limited frequency bands.
Other exterior sensors may include light detection and ranging (lidar) sensors, radar units, cameras (e.g., optical imaging devices, with or without a neutral-density filter (ND) filter), positioning sensors (e.g., gyroscopes, accelerometers and/or other inertial components), infrared sensors, and/or any other detection devices that record data which may be processed by computing devices 202 .
Such sensors of the perception system 224 may detect objects outside of the vehicle and their characteristics such as location, orientation relative to the roadway, size, shape, type (for instance, vehicle, pedestrian, bicyclist, etc.), heading, speed of movement relative to the vehicle, etc., as well as environmental conditions around the vehicle. The perception system 224 may also include other sensors within the vehicle to detect objects and conditions within the vehicle, such as in the passenger compartment. For instance, such sensors may detect, e.g., one or more persons, pets, packages, etc., as well as conditions within and/or outside the vehicle such as temperature, humidity, etc. Still further sensors 232 of the perception system 224 may measure the rate of rotation of the wheels 228 , an amount or a type of braking by the deceleration system 212 , and other factors associated with the equipment of the vehicle itself.
The raw data obtained by the sensors can be processed by the perception system 224 and/or sent for further processing to the computing devices 202 periodically or continuously as the data is generated by the perception system 224 . Computing devices 202 may use the positioning system 222 to determine the vehicle's location and perception system 224 to detect and respond to objects when needed to reach the location safely, e.g., via adjustments made by planner module 223 , including adjustments in operation to deal with occlusions and other issues. In addition, the computing devices 202 may perform validation or calibration of individual sensors, all sensors in a particular sensor assembly, or between sensors in different sensor assemblies or other physical housings. In some instances, validation or calibration may occur between acoustical sensors across multiple vehicles. For instance, information obtained by acoustical sensors of one self-driving vehicle could be used to validate (or calibrate) the acoustical sensors of another self-driving vehicle. By way of example, it would be possible to honk the horn of one of the vehicles (in various locations as it drives around). Detection of the honks by other vehicles may be used in the validation of the sensors of the other vehicles. Alternatively or additionally, one could place a speaker on the vehicle and play various siren sounds in a similar manner. As long as the system knows the ground-truth position of the emitter on one vehicle, that can be used to validate or calibrate the other vehicle(s) acoustical sensors. Driving around (and playing various types of sounds) helps ensure a diverse set of data for calibration and validation.
As illustrated in FIGS. 1 A-B , certain sensors of the perception system 224 may be incorporated into one or more sensor assemblies or housings. In one example, these may be integrated into front, rear or side perimeter sensor assemblies around the vehicle. In another example, other sensors may be part of the roof-top housing (roof pod) 102 . The computing devices 202 may communicate with the sensor assemblies located on or otherwise distributed along the vehicle. Each assembly may have one or more types of sensors such as those described above.
Returning to FIG. 2 , computing devices 202 may include all of the components normally used in connection with a computing device such as the processor and memory described above as well as a user interface subsystem 234 . The user interface subsystem 234 may include one or more user inputs 236 (e.g., a mouse, keyboard, touch screen and/or microphone) and one or more display devices 238 (e.g., a monitor having a screen or any other electrical device that is operable to display information). In this regard, an internal electronic display may be located within a cabin of the vehicle (not shown) and may be used by computing devices 202 to provide information to passengers within the vehicle. Other output devices, such as speaker(s) 240 may also be located within the passenger vehicle.
The vehicle may also include a communication system 242 . For instance, the communication system 242 may also include one or more wireless configurations to facilitate communication with other computing devices, such as passenger computing devices within the vehicle, computing devices external to the vehicle such as in other nearby vehicles on the roadway, and/or a remote server system. The network connections may include short range communication protocols such as Bluetoothâ¢, Bluetooth⢠low energy (LE), cellular connections, as well as various configurations and protocols including the Internet, World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi and HTTP, and various combinations of the foregoing.
FIG. 3 A illustrates a block diagram 300 with various components and systems of a vehicle, e.g., vehicle 150 of FIGS. 1 C-D . By way of example, the vehicle may be a truck, farm equipment or construction equipment, configured to operate in one or more autonomous modes of operation. As shown in the block diagram 300 , the vehicle includes a control system of one or more computing devices, such as computing devices 302 containing one or more processors 304 , memory 306 and other components similar or equivalent to
components
202 , 204 and 206 discussed above with regard to FIG. 2 . For instance, the memory may include acoustic models to perform, e.g., noise cancellation, triangulation, trilateration, vehicle or other object recognition, honk or other sound recognition, etc.
The control system may constitute an electronic control unit (ECU) of a tractor unit of a cargo vehicle. As with instructions 208 , the instructions 308 may be any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by the processor. Similarly, the data 310 may be retrieved, stored or modified by one or more processors 304 in accordance with the instructions 308 .
In one example, the computing devices 302 may form an autonomous driving computing system incorporated into vehicle 150 . Similar to the arrangement discussed above regarding FIG. 2 , the autonomous driving computing system of block diagram 300 may be capable of communicating with various components of the vehicle in order to perform route planning and driving operations. For example, the computing devices 302 may be in communication with various systems of the vehicle, such as a driving system including a deceleration system 312 , acceleration system 314 , steering system 316 , signaling system 318 , navigation system 320 and a positioning system 322 , each of which may function as discussed above regarding FIG. 2 .
<div id="p-0055" num="0
CLAIMS
Claims ( 21 )
The invention claimed is:
1. A method comprising:
obtaining, by a perception system of a vehicle, audio sensor data of a sound emanated in an external environment around the vehicle;
responsive to a change in the sound, receiving, by one or more processors of the vehicle, other audio sensor data of the sound from one or more other vehicles, the other audio sensor data including a direction-of-arrival and a timestamp;
determining, by the one or more processors based on (a) the audio sensor data, (b) the other audio sensor data, (c) a location of the vehicle at the timestamp, and (d) respective locations of the one or more other vehicles at the timestamp, a particular object from which the sound emanated; and
modifying, by the one or more processors based on the particular object, a driving operation of the vehicle operating in an autonomous driving mode.
2. The method of claim 1 , wherein the particular object is an emergency vehicle.
3. The method of claim 2 , wherein the emergency vehicle is one of: an ambulance, a fire truck or police car.
4. The method of claim 2 , wherein modifying the driving operation includes at least one of: slowing down for the emergency vehicle, changing lanes for the emergency vehicle, or pulling over for the emergency vehicle.
5. The method of claim 1 , further comprising:
obtaining, by the perception system, imagery of the external environment around the vehicle; and
determining, by the one or more processors based on the imagery and the audio sensor data, whether the particular object is an emergency vehicle.
6. The method of claim 5 , wherein determining whether the particular object is the emergency vehicle includes determining, by the one or more processors based on the imagery, whether lights associated with the particular object are lights associated with the emergency vehicle.
7. The method of claim 1 , wherein the particular object is a person.
8. The method of claim 7 , wherein the person is associated with an emergency vehicle.
9. The method of claim 8 , wherein the sound includes audio information from the person.
10. The method of claim 8 , wherein the emergency vehicle is of: an ambulance, a fire truck or police car.
11. The method of claim 1 , wherein the sound includes at least two of: a siren associated with an emergency vehicle, a horn honk, or audio information from a person.
12. The method of claim 11 , further comprising:
obtaining, by the perception system, imagery of the external environment around the vehicle; and
determining, by the one or more processors based on the imagery, whether lights associated with the particular object are lights associated with the emergency vehicle.
13. A method comprising:
receiving, by a microphone array of a vehicle operating in an autonomous driving mode, audio sensor data of a sound emanated in an external environment around the vehicle, the audio sensor data including a direction-of-arrival and a timestamp;
determining, by one or more processors of the vehicle based on the audio sensor data, a particular object from which the sound emanated;
determining, by the one or more processors based on (a) the audio sensor data, (b) a location of the vehicle at the timestamp, and (c) respective locations of one or more vehicles at the timestamp, whether the sound is directed at the vehicle operating in the autonomous driving mode or another road agent; and
modifying, by the one or more processors based on the particular object and the determining whether the sound is directed at the vehicle or the other road agent, a driving operation of the vehicle.
14. The method of claim 13 , wherein the vehicle and the other road agent are associated with a fleet of vehicles.
15. The method of claim 13 , further comprising receiving, by the one or more processors from a back-end system via a network, other audio sensor data of the sound from the other road agent, the other audio sensor data including another direction-of-arrival; and
wherein determining the particular object from which the sound emanated is further based on the other audio sensor data and a location of the other road agent.
16. The method of claim 13 , further comprising determining whether the sound is indicative of a problematic or dangerous condition on a roadway.
17. The method of claim 16 , further comprising determining whether the particular object is an emergency vehicle.
18. The method of claim 16 , further comprising determining whether the particular object is a person associated with an emergency vehicle.
19. The method of claim 13 , wherein:
the sound is a horn honk, and
the method further comprises:
determining, by the one or more processors based on at least one of: a timing, a frequency, a harmony, a pitch change, or an amplitude of the horn honk, whether the horn honk is a first type of horn honk or a second type of horn honk;
responsive to determining that the horn honk is the first type of horn honk, selecting, by the one or more processors, a first behavior model for predicting a behavior of another vehicle; and
responsive to determining that the sound is the second type of horn honk, selecting, by the one or more processors, a second behavior model for predicting the behavior of the other vehicle, the second behavior model being different from the first behavior model.
20. The method of claim 19 , wherein modifying the driving operation is further based on whether the first behavior model or the second behavior model is selected.
21. The method of claim 20 , wherein modifying the driving operation includes, responsive to selecting the second behavior model, at least one of: slowing down for the other vehicle, changing lanes for the other vehicle, or pulling over for the other vehicle.
US18/608,300
2020-07-21
2024-03-18
Identifying the position of a horn honk or other acoustical information using multiple autonomous vehicles
Active
US12280803B2
( en )
Priority Applications (1)
Application Number
Priority Date
Filing Date
Title
US18/608,300
US12280803B2
( en )
2020-07-21
2024-03-18
Identifying the position of a horn honk or other acoustical information using multiple autonomous vehicles
Applications Claiming Priority (2)
Application Number
Priority Date
Filing Date
Title
US16/934,132
US11958505B2
( en )
2020-07-21
2020-07-21
Identifying the position of a horn honk or other acoustical information using multiple autonomous vehicles
US18/608,300
US12280803B2
( en )
2020-07-21
2024-03-18
Identifying the position of a horn honk or other acoustical information using multiple autonomous vehicles
Related Parent Applications (1)
Application Number
Title
Priority Date
Filing Date
US16/934,132
Continuation
US11958505B2
( en )
2020-07-21
2020-07-21
Identifying the position of a horn honk or other acoustical information using multiple autonomous vehicles
Publications (2)
Publication Number
Publication Date
US20240217554A1
US20240217554A1 ( en )
2024-07-04
US12280803B2
true
US12280803B2 ( en )
2025-04-22
Family
ID=79687379
Family Applications (2)
Application Number
Title
Priority Date
Filing Date
US16/934,132
Active
2042-06-04
US11958505B2
( en )
2020-07-21
2020-07-21
Identifying the position of a horn honk or other acoustical information using multiple autonomous vehicles
US18/608,300
Active
US12280803B2
( en )
2020-07-21
2024-03-18
Identifying the position of a horn honk or other acoustical information using multiple autonomous vehicles
Family Applications Before (1)
Application Number
Title
Priority Date
Filing Date
US16/934,132
Active
2042-06-04
US11958505B2
( en )
2020-07-21
2020-07-21
Identifying the position of a horn honk or other acoustical information using multiple autonomous vehicles
Country Status (1)
Country
Link
US
( 2 )
US11958505B2
( en )
Families Citing this family (11)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20200333429A1
( en )
*
2017-12-29
2020-10-22
Ubicquia Iq Llc
Sonic pole position triangulation in a lighting system
US10297266B1
( en )
*
2018-06-15
2019-05-21
Cisco Technology, Inc.
Adaptive noise cancellation for multiple audio endpoints in a shared space
US20220050479A1
( en )
*
2020-08-12
2022-02-17
Gneiss Enterprises LLC
Uav-based protection from active shooters
US11899468B2
( en )
*
2020-12-22
2024-02-13
Waymo Llc
Sensor for flashing light detection
US11735205B2
( en )
*
2021-01-12
2023-08-22
Baidu Usa Llc
Audio logging for model training and onboard validation utilizing autonomous driving vehicle
US11430466B2
( en )
*
2021-01-13
2022-08-30
Baidu Usa Llc
Sound source detection and localization for autonomous driving vehicle
JP2023017571A
( en )
*
2021-07-26
2023-02-07
ã¦ã¼ãã³ã»ãã©ãããã»ãã¼ã«ãã£ã³ã°ã¹æ ªå¼ä¼ç¤¾
Remote support apparatus, remote support system, and remote support method
US12036996B2
( en )
*
2021-12-21
2024-07-16
Gm Cruise Holdings Llc
Automated method to detect road user frustration due to autonomous vehicle driving behavior
US20230356551A1
( en )
*
2022-05-05
2023-11-09
Tusimple, Inc.
Control subsystem and method for detecting and directing a response to a tire failure of an autonomous vehicle
US12491915B2
( en )
*
2023-12-11
2025-12-09
Waymo Llc
Autonomous vehicle driving path selection in the presence of emergency vehicle sounds
US12497000B2
( en )
*
2023-12-28
2025-12-16
Aspinity, Inc.
Vehicle in-cabin audio detection of vandalism
Citations (29)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20010007969A1
( en )
1999-12-14
2001-07-12
Matsushita Electric Industrial Co., Ltd.
Method and apparatus for concurrently estimating respective directions of a plurality of sound sources and for monitoring individual sound levels of respective moving sound sources
US20020003470A1
( en )
1998-12-07
2002-01-10
Mitchell Auerbach
Automatic location of gunshots detected by mobile devices
US6362749B1
( en )
2001-06-18
2002-03-26
William E. Brill
Emergency vehicle detection system
JP2007050724A
( en )
2005-08-16
2007-03-01
Yamaha Corp
Vehicle exterior warning sound notifying device
US20080150755A1
( en )
2006-11-15
2008-06-26
Alice Jane Van Zandt
Emergency Vehicle Indicator
US20090179774A1
( en )
2008-01-15
2009-07-16
Qnx Software Systems (Wavemakers), Inc.
Dynamic siren detection and notification system
US7616128B2
( en )
2004-07-23
2009-11-10
Panasonic Corporation
Audio identifying device, audio identifying method, and program
US7750814B2
( en )
2003-01-24
2010-07-06
Shotspotter, Inc.
Highly portable system for acoustic event detection
US8676427B1
( en )
2012-10-11
2014-03-18
Google Inc.
Controlling autonomous vehicle using audio data
US20140136096A1
( en )
2011-04-15
2014-05-15
Meijo University
Approaching vehicle detecting system and approaching vehicle detecting method
US20140241126A1
( en )
2011-09-20
2014-08-28
Meijo University
Sound source detection system
US8838321B1
( en )
2012-11-15
2014-09-16
Google Inc.
Modifying a vehicle state based on the presence of a special-purpose vehicle
US20150177363A1
( en )
2013-12-19
2015-06-25
Ford Global Technologies, Llc
Mobile gunshot detection
US20160009222A1
( en )
2014-07-09
2016-01-14
Eugene Taylor
Emergency alert audio interception
US9278689B1
( en )
2014-11-13
2016-03-08
Toyota Motor Engineering & Manufacturing North America, Inc.
Autonomous vehicle detection of and response to emergency vehicles
US20160070788A1
( en )
2014-09-04
2016-03-10
Aisin Seiki Kabushiki Kaisha
Siren signal source detection, recognition and localization
US9305223B1
( en )
2013-06-26
2016-04-05
Google Inc.
Vision-based indicator signal detection using spatiotemporal filtering
US9397630B2
( en )
2012-04-09
2016-07-19
Dts, Inc.
Directional based audio response to an external environment emergency signal
US9704397B1
( en )
2016-04-05
2017-07-11
Global Ip Holdings, Llc
Apparatus for use in a warning system to notify a land vehicle or a motorist of the vehicle of an approaching or nearby emergency vehicle or train
US20170249839A1
( en )
2016-02-29
2017-08-31
Faraday&Future Inc.
Emergency signal detection and response
WO2017151937A1
( en )
2016-03-04
2017-09-08
Emergency Vehicle Alert Systems Llc
Emergency vehicle alert and response system
US9840254B2
( en )
2015-12-31
2017-12-12
Sony Corporation
Method and system for adaptive detection and application of horn for an autonomous vehicle
US20180132052A1
( en )
2016-11-08
2018-05-10
Harman Becker Automotive Systems Gmbh
Vehicle sound processing system
US20180211528A1
( en )
2017-01-26
2018-07-26
Magna Electronics Inc.
Vehicle acoustic-based emergency vehicle detection
US20190027032A1
( en )
2017-07-24
2019-01-24
Harman International Industries, Incorporated
Emergency vehicle alert system
US10319228B2
( en )
2017-06-27
2019-06-11
Waymo Llc
Detecting and responding to sirens
US20200207371A1
( en )
2019-01-02
2020-07-02
Qualcomm Incorporated
Methods And Systems For Establishing Cooperative Driving Engagements With Vehicles Having Varying Levels Of Autonomy
US20200398832A1
( en )
2019-06-24
2020-12-24
GM Global Technology Operations LLC
System, vehicle and method for adapting a driving condition of a vehicle upon detecting an event in an environment of the vehicle
US20210109187A1
( en )
2019-10-10
2021-04-15
Uatc, Llc
Microphone Array for Sound Source Detection and Location
2020
2020-07-21
US
US16/934,132
patent/US11958505B2/en
active
Active
2024
2024-03-18
US
US18/608,300
patent/US12280803B2/en
active
Active
Patent Citations (30)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
US20020003470A1
( en )
1998-12-07
2002-01-10
Mitchell Auerbach
Automatic location of gunshots detected by mobile devices
US20010007969A1
( en )
1999-12-14
2001-07-12
Matsushita Electric Industrial Co., Ltd.
Method and apparatus for concurrently estimating respective directions of a plurality of sound sources and for monitoring individual sound levels of respective moving sound sources
US6362749B1
( en )
2001-06-18
2002-03-26
William E. Brill
Emergency vehicle detection system
US7750814B2
( en )
2003-01-24
2010-07-06
Shotspotter, Inc.
Highly portable system for acoustic event detection
US7616128B2
( en )
2004-07-23
2009-11-10
Panasonic Corporation
Audio identifying device, audio identifying method, and program
JP2007050724A
( en )
2005-08-16
2007-03-01
Yamaha Corp
Vehicle exterior warning sound notifying device
US20080150755A1
( en )
2006-11-15
2008-06-26
Alice Jane Van Zandt
Emergency Vehicle Indicator
US20090179774A1
( en )
2008-01-15
2009-07-16
Qnx Software Systems (Wavemakers), Inc.
Dynamic siren detection and notification system
US20140136096A1
( en )
2011-04-15
2014-05-15
Meijo University
Approaching vehicle detecting system and approaching vehicle detecting method
US20140241126A1
( en )
2011-09-20
2014-08-28
Meijo University
Sound source detection system
US9397630B2
( en )
2012-04-09
2016-07-19
Dts, Inc.
Directional based audio response to an external environment emergency signal
US8676427B1
( en )
2012-10-11
2014-03-18
Google Inc.
Controlling autonomous vehicle using audio data
US8838321B1
( en )
2012-11-15
2014-09-16
Google Inc.
Modifying a vehicle state based on the presence of a special-purpose vehicle
US9305223B1
( en )
2013-06-26
2016-04-05
Google Inc.
Vision-based indicator signal detection using spatiotemporal filtering
US20150177363A1
( en )
2013-12-19
2015-06-25
Ford Global Technologies, Llc
Mobile gunshot detection
US10061013B2
( en )
2013-12-19
2018-08-28
Ford Global Technologies, Llc
Mobile gunshot detection
US20160009222A1
( en )
2014-07-09
2016-01-14
Eugene Taylor
Emergency alert audio interception
US20160070788A1
( en )
2014-09-04
2016-03-10
Aisin Seiki Kabushiki Kaisha
Siren signal source detection, recognition and localization
US9278689B1
( en )
2014-11-13
2016-03-08
Toyota Motor Engineering & Manufacturing North America, Inc.
Autonomous vehicle detection of and response to emergency vehicles
US9840254B2
( en )
2015-12-31
2017-12-12
Sony Corporation
Method and system for adaptive detection and application of horn for an autonomous vehicle
US20170249839A1
( en )
2016-02-29
2017-08-31
Faraday&Future Inc.
Emergency signal detection and response
WO2017151937A1
( en )
2016-03-04
2017-09-08
Emergency Vehicle Alert Systems Llc
Emergency vehicle alert and response system
US9704397B1
( en )
2016-04-05
2017-07-11
Global Ip Holdings, Llc
Apparatus for use in a warning system to notify a land vehicle or a motorist of the vehicle of an approaching or nearby emergency vehicle or train
US20180132052A1
( en )
2016-11-08
2018-05-10
Harman Becker Automotive Systems Gmbh
Vehicle sound processing system
US20180211528A1
( en )
2017-01-26
2018-07-26
Magna Electronics Inc.
Vehicle acoustic-based emergency vehicle detection
US10319228B2
( en )
2017-06-27
2019-06-11
Waymo Llc
Detecting and responding to sirens
US20190027032A1
( en )
2017-07-24
2019-01-24
Harman International Industries, Incorporated
Emergency vehicle alert system
US20200207371A1
( en )
2019-01-02
2020-07-02
Qualcomm Incorporated
Methods And Systems For Establishing Cooperative Driving Engagements With Vehicles Having Varying Levels Of Autonomy
US20200398832A1
( en )
2019-06-24
2020-12-24
GM Global Technology Operations LLC
System, vehicle and method for adapting a driving condition of a vehicle upon detecting an event in an environment of the vehicle
US20210109187A1
( en )
2019-10-10
2021-04-15
Uatc, Llc
Microphone Array for Sound Source Detection and Location
Non-Patent Citations (4)
* Cited by examiner, â Cited by third party
Title
Bazilinskyy, P , et al., Take-over requests in highly automated driving: A crowdsourcing survey on auditory, vibrotactile, and visual displays, Transportation Research Part F, 2018, pp. 82-98.
Cartwri, Mark , et al., Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations, New York University, 2018, pp. 1-5.
Cartwright, Mark, et al., Seeing Sound: Investigating the Effects of Visualizations and Complexity on Crowdsourced Audio Annotations, PACM on Human-Computer Interaction, vol. 1, No. CSCW, Article 29. Publication date: Nov. 2017.
Patil, Nikhil , Content Centric Crowdsourcing and Dissemination of Realtime Public Transportation Information, Department of Computer Science and Engineering, Indian Institute of Technology, 2014, pp. 1-53.
Also Published As
Publication number
Publication date
US11958505B2
( en )
2024-04-16
US20220024484A1
( en )
2022-01-27
US20240217554A1
( en )
2024-07-04
Similar Documents
Publication
Publication Date
Title
US20240217554A1
( en )
2024-07-04
Identifying The Position Of A Horn Honk Or Other Acoustical Information Using Multiple Autonomous Vehicles
US12511581B2
( en )
2025-12-30
Road condition deep learning model
US12066836B2
( en )
2024-08-20
Detecting general road weather conditions
US12578724B2
( en )
2026-03-17
Detection of anomalous trailer behavior
KR102757272B1
( en )
2025-01-23
Detect and address abnormal driver behavior using driver assistance
US12365345B2
( en )
2025-07-22
Using audio to detect road conditions
US12344278B2
( en )
2025-07-01
Continuing lane driving prediction
US20200189567A1
( en )
2020-06-18
Determining Wheel Slippage on Self Driving Vehicle
JP2020525885A
( en )
2020-08-27
Siren detection and response to sirens
US20250222955A1
( en )
2025-07-10
Using the Presence of Road Surface and Surrounding Area Illumination to Detect Occluded Objects
US12384426B1
( en )
2025-08-12
Positional gaps for driver controllability
US20260141307A1
( en )
2026-05-21
Road condition deep learning model
Legal Events
Date
Code
Title
Description
2024-03-18
FEPP
Fee payment procedure
Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY
2024-03-19
AS
Assignment
Owner name : WAYMO LLC, CALIFORNIA
Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ARMSTRONG-CREWS, NICHOLAS;REEL/FRAME:066825/0640
Effective date : 20200720
2024-03-28
STPP
Information on status: patent application and granting procedure in general
Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION
2024-12-02
STPP
Information on status: patent application and granting procedure in general
Free format text : NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS
2025-02-22
STPP
Information on status: patent application and granting procedure in general
Free format text : NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS
2025-02-25
STPP
Information on status: patent application and granting procedure in general
Free format text : PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED
2025-04-02
STCF
Information on status: patent grant
Free format text : PATENTED CASE