ABSTRACT
Abstract
It is advantageous for a vehicle to detect road wetness or related environmental conditions. This is particularly true for self-driving vehicles, which can then adjust the manner of automated operation of the vehicle to increase safety by reducing speed, braking earlier, adjusting internal estimates of road traction parameters, or adjusting autonomous operation in some other manner. It is difficult to directly measure road wetness (e.g., using spectroscopy or other methods directed at the road surface), however, it is possible to indirectly estimate road wetness based on road noise audio signals detected via one or more microphones disposed on the vehicle. The location of the microphones, the type of post-processing applied to the audio signals, or other factors can be adapted to increase the useful road wetness-related content of such audio signals while reducing the presence of engine noise, road noise, or other confounding signals.
Description
BACKGROUND
Autonomous vehicles, such as vehicles that do not require a human driver, can be used to aid in the transport of passengers or cargo from one location to another. Such vehicles may operate in a fully autonomous mode or a partially autonomous mode where a person may provide some driving input. In order to operate in an autonomous mode, the vehicle may employ various on-board sensors to detect features of the external environment, and use received sensor information to perform various driving operations. Road conditions including water on the roadway may adversely impact operation of the vehicle, including how information from the sensor system is evaluated, when a wiper system is engaged, real-time and planned driving behavior, among other issues.
SUMMARY
Audio signals recorded from a variety of locations on or around a vehicle contain information that can be used, alone or in combination with other signals (e.g., from additional microphones, from other types of sensors, or from other information sources on or off the vehicle) to predict the condition of the environment of the vehicle. For example, the amount of water on a road surface, and thus a corresponding degree of traction possible against that road surface, could be predicted based on such audio signals. However, emplacing and configuring microphones on a vehicle to facilitate detection of audio signals that are relevant to predicting such environmental conditions can be difficult due to fouling of the microphones, detection of unwanted signals (e.g., wind noise, engine noise, or other noise that is unrelated to the environmental condition(s) of interest), or other confounding processes. For many vehicles, such as trucks or other vehicles configured to tow large trailers, microphones can be beneficially located between and behind one or more pairs of rear wheels of the vehicle. This position allows the microphone to detect road noise that is relevant to predicting road wetness or other environmental condition(s) of interest (e.g., the noise of the tires interacting with the road, the noise of mudflaps, noise reflected off of the road surface from other sources) while reducing the amount of engine noise, wind noise, or other unwanted noise signals that could confound the prediction of the environmental condition(s) of interest. The signal(s) from such microphone(s) could then be filtered, transformed, or otherwise processed prior to be applied to a model (e.g., a deep learning (DL) model) to predict road wetness or to generate some other prediction related to the condition of the environment of the vehicle. The output of such an audio-based prediction can be applied in a variety of ways to enhance autonomous vehicle operation, for instance by altering current driving actions, modifying planned routes or trajectories, activating on-board cleaning systems, etc.
According to one aspect, a system configured to operate a vehicle in an autonomous driving mode is provided. The system includes: (i) memory storing a road condition deep learning model, the model relating to a discrete classification or continuous regression/estimation of road wetness; and (ii) one or more processors operatively coupled to the memory. The one or more processors are configured to: (a) receive sensor data from one or more microphones of the vehicle while operating in the autonomous driving mode, the one or more microphones being configured to detect one or more road noise signals, wherein the vehicle includes a pair of front wheels and a first pair of rear wheels, and wherein a first microphone of the one or more microphones is disposed, relative to the vehicle, to the center of and behind the first pair of rear wheels; (b) use the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more road noise signals; and (c) use the generated information to control operation of the vehicle in the autonomous driving mode.
The one or more processors can additionally be configured to apply a highpass filter to the one or more road noise signals prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more road noise signals, wherein the highpass filter passes road noise detected by the one or more microphones at frequencies greater than 1 kHz.
The one or more processors can additionally be configured to apply a bandpass filter to the one or more road noise signals prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more road noise signals, wherein the bandpass filter passes road noise detected by the one or more microphones at frequencies between 1 kHz and 6 kHz.
The vehicle can further include a second pair of rear wheels that are disposed, relative to the vehicle, behind the first pair of rear wheels. In such examples, the first microphone can be disposed, relative to the vehicle, in front of the second pair of rear wheels, and a second microphone of the one or more microphones can be disposed, relative to the vehicle, to the center of and behind the second pair of rear wheels.
The one or more processors can additionally be configured to generate one or more features from the one or more road noise signals, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more road noise signals comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more features, and wherein the one or more features comprise at least one of: (i) a mean of one of the one or more road noise signals within a time window, (ii) a zero crossing rate of one of the one or more road noise signals, (iii) a moment of a time domain waveform of one of the one or more road noise signals, (iv) an energy in a frequency band of one of the one or more road noise signals, (v) a ratio between an energy in two different frequency bands of one of the one or more road noise signals, (vi) a moment of a spectrum of one of the one or more road noise signals, or (vii) a shape of a spectrum of one of the one or more road noise signals.
The one or more processors can additionally be configured to generate a spectrum image from the one or more road noise signals, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more road noise signals comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the spectrum image.
The model can be formed by evaluating a first set of training inputs of sensor data of an environment along a portion of a roadway from one or more on-board sensors and a second set of training inputs of off-board information associated with the portion of the roadway with respect to ground truth data for the portion of the roadway, the ground truth data including one or more measurements of water thickness across one or more areas of the portion of the roadway, and the one or more on-board sensors including the one or more microphones. In such examples, the second set of training inputs of off-board information can include one or more of weather station information, public weather forecasts, road graph data, crowdsourced information, or observations from one or more other vehicles
In an example, controlling operation of the vehicle in the autonomous mode using the generating information can include at least one of alteration of a current driving action, modification of a planned route or trajectory, or activation of an on-board cleaning system.
According to another aspect, a vehicle configured to operate in an autonomous driving mode is provided. The vehicle includes (i) a pair of front wheels; (ii) a first pair of rear wheels; (iii) a first microphone, wherein the first microphone is configured to detect a first road noise signal; (iv) memory storing a road condition deep learning model, the model relating to a discrete classification or continuous regression/estimation of road wetness; and (v) one or more processors operatively coupled to the memory. The one or more processors are configured to: (a) receive sensor data that includes the first road noise signal while operating in the autonomous driving mode; (b) use the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data; and (c) use the generated information to control operation of the vehicle in the autonomous driving mode.
In some examples, the first microphone is disposed, relative to the vehicle, to the center of and behind the first pair of rear wheels. In another example, the vehicle can additionally include a second pair of rear wheels that are disposed, relative to the vehicle, behind the first pair of rear wheels, wherein the first microphone is disposed, relative to the vehicle, in front of the second pair of rear wheels, to the center of and behind the first pair of rear wheels. The microphones can have other locations relative to the vehicle in other examples.
In yet another example, the vehicle can additionally include a pair of mudflaps, wherein each mudflap of the pair of mudflaps is located proximate to and behind a respective rear wheel of the first pair of rear wheels, and wherein the first microphone is located between the pair of mudflaps.
In an additional example, the one or more processors can be additionally configured to apply a high pass filter to the first road noise signal prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the first road noise signal, wherein the high pass filter passes frequencies greater than 1 kHz within the first road noise signal.
In yet another example, the one or more processors can be additionally configured to apply a band pass filter to the first road noise signal prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the first road noise signal, wherein the band pass filter passes frequencies between 1 kHz and 6 kHz within the first road noise signal.
In some examples, the vehicle can additionally include a third microphone that is configured to detect a third road noise signal and the one or more processors can additionally be configured to combine the first road noise signal and the third road noise signal to generate a combined road noise signal, using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the combined road noise signal, and wherein the combined road noise signal contains less of a confounding signal than the first road noise signal.
In yet another example, the one or more processors can be additionally configured to generate one or more features from the first road noise signal, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more features, and wherein the one or more features comprise at least one of: (i) a mean of the first road noise signal within a time window, (ii) a zero crossing rate of the first road noise signal, (iii) a moment of a time domain waveform of the first road noise signal, (iv) an energy in a frequency band of the first road noise signal, (v) a ratio between an energy in two different frequency bands of the first road noise signal, (vi) a moment of a spectrum of the first road noise signal, or (vii) a shape of a spectrum of the first road noise signal.
In an additional example, the one or more processors can be additionally configured to generate a spectrum image from the first road noise signal, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the spectrum image.
According to another aspect, a method for generating a road condition deep learning model is provided. The method includes: (i) receiving as a first set of training inputs, by one or more processors, sensor data of an environment along a portion of a roadway from one or more microphones of a vehicle, the one or more microphones being configured to detect one or more road noise signals, wherein the vehicle includes a pair of front wheels and a first pair of rear wheels, and wherein a first microphone of the one or more microphones is disposed, relative to the vehicle, to the center of and behind the first pair of rear wheels; (ii) receiving as a second set of training inputs, by the one or more processors, off-board information associated with the portion of the roadway; (iii) evaluating, by the one or more processors, the received first set of training inputs and the received second set of training inputs with respect to ground truth data for the portion of the roadway, the ground truth data including one or more measurements of water thickness across one or more areas of the portion of the roadway to give classification or continuous estimation of wetness along the one or more areas of the portion of the roadway, wherein the evaluating generates road wetness information based on the received first and second sets of training inputs and the ground truth data; (iv) generating the road condition deep learning model from the road wetness information; and (v) storing the generated road condition deep learning model in memory.
BRIEF DESCRIPTION OF THE DRAWINGS
FIGS. 1 A and 1 B illustrate an example passenger-type vehicle configured for use with aspects of the disclosure.
FIGS. 1 C and 1 D illustrate an example cargo-type vehicle configured for use with aspects of the disclosure.
FIG. 2 is a block diagram of systems of an example passenger-type vehicle in accordance with aspects of the disclosure.
FIGS. 3 A and 3 B are block diagrams of systems of an example cargo-type vehicle in accordance with aspects of the disclosure.
FIG. 4 illustrates an example of detecting ground truth in accordance with aspects of the disclosure.
FIG. 5 illustrates example sensor fields of view for a passenger-type vehicle in accordance with aspects of the disclosure.
FIGS. 6 A and 6 B illustrate example sensor fields of view for a cargo-type vehicle in accordance with aspects of the disclosure.
FIG. 7 illustrates an example system in accordance with aspects of the disclosure.
FIG. 8 illustrates examples of on-board and off-board training inputs in accordance with aspects of the disclosure.
FIG. 9 illustrates an example of roadway wetness in accordance with aspects of the disclosure.
FIG. 10 illustrates an example road condition deep learning model in accordance with aspects of the disclosure.<
BACKGROUND
Autonomous vehicles, such as vehicles that do not require a human driver, can be used to aid in the transport of passengers or cargo from one location to another. Such vehicles may operate in a fully autonomous mode or a partially autonomous mode where a person may provide some driving input. In order to operate in an autonomous mode, the vehicle may employ various on-board sensors to detect features of the external environment, and use received sensor information to perform various driving operations. Road conditions including water on the roadway may adversely impact operation of the vehicle, including how information from the sensor system is evaluated, when a wiper system is engaged, real-time and planned driving behavior, among other issues.
SUMMARY
Audio signals recorded from a variety of locations on or around a vehicle contain information that can be used, alone or in combination with other signals (e.g., from additional microphones, from other types of sensors, or from other information sources on or off the vehicle) to predict the condition of the environment of the vehicle. For example, the amount of water on a road surface, and thus a corresponding degree of traction possible against that road surface, could be predicted based on such audio signals. However, emplacing and configuring microphones on a vehicle to facilitate detection of audio signals that are relevant to predicting such environmental conditions can be difficult due to fouling of the microphones, detection of unwanted signals (e.g., wind noise, engine noise, or other noise that is unrelated to the environmental condition(s) of interest), or other confounding processes. For many vehicles, such as trucks or other vehicles configured to tow large trailers, microphones can be beneficially located between and behind one or more pairs of rear wheels of the vehicle. This position allows the microphone to detect road noise that is relevant to predicting road wetness or other environmental condition(s) of interest (e.g., the noise of the tires interacting with the road, the noise of mudflaps, noise reflected off of the road surface from other sources) while reducing the amount of engine noise, wind noise, or other unwanted noise signals that could confound the prediction of the environmental condition(s) of interest. The signal(s) from such microphone(s) could then be filtered, transformed, or otherwise processed prior to be applied to a model (e.g., a deep learning (DL) model) to predict road wetness or to generate some other prediction related to the condition of the environment of the vehicle. The output of such an audio-based prediction can be applied in a variety of ways to enhance autonomous vehicle operation, for instance by altering current driving actions, modifying planned routes or trajectories, activating on-board cleaning systems, etc.
According to one aspect, a system configured to operate a vehicle in an autonomous driving mode is provided. The system includes: (i) memory storing a road condition deep learning model, the model relating to a discrete classification or continuous regression/estimation of road wetness; and (ii) one or more processors operatively coupled to the memory. The one or more processors are configured to: (a) receive sensor data from one or more microphones of the vehicle while operating in the autonomous driving mode, the one or more microphones being configured to detect one or more road noise signals, wherein the vehicle includes a pair of front wheels and a first pair of rear wheels, and wherein a first microphone of the one or more microphones is disposed, relative to the vehicle, to the center of and behind the first pair of rear wheels; (b) use the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more road noise signals; and (c) use the generated information to control operation of the vehicle in the autonomous driving mode.
The one or more processors can additionally be configured to apply a highpass filter to the one or more road noise signals prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more road noise signals, wherein the highpass filter passes road noise detected by the one or more microphones at frequencies greater than 1 kHz.
The one or more processors can additionally be configured to apply a bandpass filter to the one or more road noise signals prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more road noise signals, wherein the bandpass filter passes road noise detected by the one or more microphones at frequencies between 1 kHz and 6 kHz.
The vehicle can further include a second pair of rear wheels that are disposed, relative to the vehicle, behind the first pair of rear wheels. In such examples, the first microphone can be disposed, relative to the vehicle, in front of the second pair of rear wheels, and a second microphone of the one or more microphones can be disposed, relative to the vehicle, to the center of and behind the second pair of rear wheels.
The one or more processors can additionally be configured to generate one or more features from the one or more road noise signals, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more road noise signals comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more features, and wherein the one or more features comprise at least one of: (i) a mean of one of the one or more road noise signals within a time window, (ii) a zero crossing rate of one of the one or more road noise signals, (iii) a moment of a time domain waveform of one of the one or more road noise signals, (iv) an energy in a frequency band of one of the one or more road noise signals, (v) a ratio between an energy in two different frequency bands of one of the one or more road noise signals, (vi) a moment of a spectrum of one of the one or more road noise signals, or (vii) a shape of a spectrum of one of the one or more road noise signals.
The one or more processors can additionally be configured to generate a spectrum image from the one or more road noise signals, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more road noise signals comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the spectrum image.
The model can be formed by evaluating a first set of training inputs of sensor data of an environment along a portion of a roadway from one or more on-board sensors and a second set of training inputs of off-board information associated with the portion of the roadway with respect to ground truth data for the portion of the roadway, the ground truth data including one or more measurements of water thickness across one or more areas of the portion of the roadway, and the one or more on-board sensors including the one or more microphones. In such examples, the second set of training inputs of off-board information can include one or more of weather station information, public weather forecasts, road graph data, crowdsourced information, or observations from one or more other vehicles
In an example, controlling operation of the vehicle in the autonomous mode using the generating information can include at least one of alteration of a current driving action, modification of a planned route or trajectory, or activation of an on-board cleaning system.
According to another aspect, a vehicle configured to operate in an autonomous driving mode is provided. The vehicle includes (i) a pair of front wheels; (ii) a first pair of rear wheels; (iii) a first microphone, wherein the first microphone is configured to detect a first road noise signal; (iv) memory storing a road condition deep learning model, the model relating to a discrete classification or continuous regression/estimation of road wetness; and (v) one or more processors operatively coupled to the memory. The one or more processors are configured to: (a) receive sensor data that includes the first road noise signal while operating in the autonomous driving mode; (b) use the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data; and (c) use the generated information to control operation of the vehicle in the autonomous driving mode.
In some examples, the first microphone is disposed, relative to the vehicle, to the center of and behind the first pair of rear wheels. In another example, the vehicle can additionally include a second pair of rear wheels that are disposed, relative to the vehicle, behind the first pair of rear wheels, wherein the first microphone is disposed, relative to the vehicle, in front of the second pair of rear wheels, to the center of and behind the first pair of rear wheels. The microphones can have other locations relative to the vehicle in other examples.
In yet another example, the vehicle can additionally include a pair of mudflaps, wherein each mudflap of the pair of mudflaps is located proximate to and behind a respective rear wheel of the first pair of rear wheels, and wherein the first microphone is located between the pair of mudflaps.
In an additional example, the one or more processors can be additionally configured to apply a high pass filter to the first road noise signal prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the first road noise signal, wherein the high pass filter passes frequencies greater than 1 kHz within the first road noise signal.
In yet another example, the one or more processors can be additionally configured to apply a band pass filter to the first road noise signal prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the first road noise signal, wherein the band pass filter passes frequencies between 1 kHz and 6 kHz within the first road noise signal.
In some examples, the vehicle can additionally include a third microphone that is configured to detect a third road noise signal and the one or more processors can additionally be configured to combine the first road noise signal and the third road noise signal to generate a combined road noise signal, using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the combined road noise signal, and wherein the combined road noise signal contains less of a confounding signal than the first road noise signal.
In yet another example, the one or more processors can be additionally configured to generate one or more features from the first road noise signal, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more features, and wherein the one or more features comprise at least one of: (i) a mean of the first road noise signal within a time window, (ii) a zero crossing rate of the first road noise signal, (iii) a moment of a time domain waveform of the first road noise signal, (iv) an energy in a frequency band of the first road noise signal, (v) a ratio between an energy in two different frequency bands of the first road noise signal, (vi) a moment of a spectrum of the first road noise signal, or (vii) a shape of a spectrum of the first road noise signal.
In an additional example, the one or more processors can be additionally configured to generate a spectrum image from the first road noise signal, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the spectrum image.
According to another aspect, a method for generating a road condition deep learning model is provided. The method includes: (i) receiving as a first set of training inputs, by one or more processors, sensor data of an environment along a portion of a roadway from one or more microphones of a vehicle, the one or more microphones being configured to detect one or more road noise signals, wherein the vehicle includes a pair of front wheels and a first pair of rear wheels, and wherein a first microphone of the one or more microphones is disposed, relative to the vehicle, to the center of and behind the first pair of rear wheels; (ii) receiving as a second set of training inputs, by the one or more processors, off-board information associated with the portion of the roadway; (iii) evaluating, by the one or more processors, the received first set of training inputs and the received second set of training inputs with respect to ground truth data for the portion of the roadway, the ground truth data including one or more measurements of water thickness across one or more areas of the portion of the roadway to give classification or continuous estimation of wetness along the one or more areas of the portion of the roadway, wherein the evaluating generates road wetness information based on the received first and second sets of training inputs and the ground truth data; (iv) generating the road condition deep learning model from the road wetness information; and (v) storing the generated road condition deep learning model in memory.
BRIEF DESCRIPTION OF THE DRAWINGS
FIGS. 1 A and 1 B illustrate an example passenger-type vehicle configured for use with aspects of the disclosure.
FIGS. 1 C and 1 D illustrate an example cargo-type vehicle configured for use with aspects of the disclosure.
FIG. 2 is a block diagram of systems of an example passenger-type vehicle in accordance with aspects of the disclosure.
FIGS. 3 A and 3 B are block diagrams of systems of an example cargo-type vehicle in accordance with aspects of the disclosure.
FIG. 4 illustrates an example of detecting ground truth in accordance with aspects of the disclosure.
FIG. 5 illustrates example sensor fields of view for a passenger-type vehicle in accordance with aspects of the disclosure.
FIGS. 6 A and 6 B illustrate example sensor fields of view for a cargo-type vehicle in accordance with aspects of the disclosure.
FIG. 7 illustrates an example system in accordance with aspects of the disclosure.
FIG. 8 illustrates examples of on-board and off-board training inputs in accordance with aspects of the disclosure.
FIG. 9 illustrates an example of roadway wetness in accordance with aspects of the disclosure.
FIG. 10 illustrates an example road condition deep learning model in accordance with aspects of the disclosure.
FIGS. 11 A and 11 B illustrate driving modification scenarios in accordance with aspects of the disclosure.
FIGS. 12 A and 12 B illustrate an example system in accordance with aspects of the disclosure.
FIG. 13 illustrates an example process in accordance with aspects of the disclosure.
FIG. 14 illustrates an example process in accordance with aspects of the disclosure.
FIGS. 15 A, 15 B, and 15 C illustrate an example vehicle with example locations, relative to other elements of the vehicle, of microphones that can be employed to acoustically detect environmental conditions of the vehicle in accordance with aspects of the disclosure.
FIG. 16 illustrates a simulated location of noise that can interfere with acoustical detection of environmental conditions of a vehicle in accordance with aspects of the disclosure.
DETAILED DESCRIPTION
As noted above, aspects of the technology use audio signals detected by one or more microphones positioned at advantageous locations on a vehicle in order to predict road wetness or other information about the condition of the environment of the vehicle in order to inform the autonomous operation of the vehicle. These audio signals, alone or in combination with other signals generated by sensors of the vehicle (e.g., LIDAR, camera, traction control sensor outputs) or received from information sources separate from the vehicle (e.g., servers providing live weather information), are then applied to a DL model or other algorithm to predict road wetness or some other variable of interest related to the environmental condition of the vehicle.
Ground truth information about road wetness, input from one or more of the following sources such as other on-board sensor signals and/or signals from other on-board modules, and off-board signals can be used to develop the DL model for road wetness classification, as well as to perform a road wetness regression analysis. Certain data (e.g., on-board microphones or other sensors and off-board signals) is used in the DL model, while other data (e.g., ground-truth info) may be used only for training. Thus, a deployed system does not require that the ground-truth sensors be installed on the vehicle. For instance, road noise signal(s) and/or other training inputs are evaluated with respect to ground truth information for a given roadway segment. The output of the DL model can be used in a variety of ways to enhance autonomous vehicle operation, for instance by altering current driving actions, modifying planned routes or trajectories, activating on-board cleaning systems, etc.
Note that, where an element (e.g., a microphone) is described as being âbehindâ a pair of wheels, this is intended to mean that the element is located, with respect to the usual direction of motion of the vehicle (the âforwardâ direction of motion), behind the center axis of the pair of wheels. Where an element (e.g., a microphone) is described as being to the âcenterâ of a pair of wheels, this is intended to mean that the element is located closer to the midline of the vehicle than the innermost surface of either wheel of the pair of wheels.
Example Vehicle Systems
FIG. 1 A illustrates a perspective view of an example passenger vehicle 100 , such as a minivan, sport utility vehicle (SUV) or other vehicle. FIG. 1 B illustrates a top-down view of the passenger vehicle 100 . The passenger vehicle 100 may include various sensors for obtaining information about the vehicle's external environment. For instance, a roof-top housing 102 may include a lidar sensor as well as one or more cameras, radar units, infrared and/or acoustical sensors. Housing 104 , located at the front end of vehicle 100 , and housings 106 a, 106 b on the driver's and passenger's sides of the vehicle may each incorporate lidar, radar, camera and/or other sensors. For example, housing 106 a may be located in front of the driver's side door along a quarter panel of the vehicle. As shown, the passenger vehicle 100 also includes housings 108 a, 108 b for radar units, lidar and/or cameras also located towards the rear roof portion of the vehicle. Additional lidar, radar units and/or cameras (not shown) may be located at other places along the vehicle 100 . For instance, arrow 110 indicates that a sensor unit ( 112 in FIG. 1 B ) may be positioned along the rear of the vehicle 100 , such as on or adjacent to the bumper. And arrow 114 indicates a series of sensor units 116 arranged along a forward-facing direction of the vehicle. In some examples, the passenger vehicle 100 also may include various sensors for obtaining information about the vehicle's interior spaces (not shown).
FIGS. 1 C and 1 D illustrate an example cargo vehicle 150 , such as a tractor-trailer truck. The truck may include, e.g., a single, double or triple trailer, or may be another medium or heavy duty truck such as in commercial weight classes 4 through 8. As shown, the truck includes a tractor unit 152 and a single cargo unit or trailer 154 . The trailer 154 may be fully enclosed, open such as a flat bed, or partially open depending on the type of cargo to be transported. In this example, the tractor unit 152 includes the engine and steering systems (not shown) and a cab 156 for a driver and any passengers. In a fully autonomous arrangement, the cab 156 may not be equipped with seats or manual driving components, since no human driver may be necessary.
The trailer 154 includes a hitching point, known as a kingpin, 158 . The kingpin 158 is typically formed as a solid steel shaft, which is configured to pivotally attach to the tractor unit 152 . In particular, the kingpin 158 attaches to a trailer coupling 160 , known as a fifth-wheel, that is mounted rearward of the cab. For a double or triple tractor-trailer, the second and/or third trailers may have simple hitch connections to the leading trailer. Or, alternatively, each trailer may have its own kingpin. In this case, at least the first and second trailers could include a fifth-wheel type structure arranged to couple to the next trailer.
As shown, the tractor may have one or more sensor units 162 , 164 disposed therealong. For instance, one or more sensor units 162 may be disposed on a roof or top portion of the cab 156 , and one or more side sensor units 164 may be disposed on left and/or right sides of the cab 156 . Sensor units may also be located along other regions of the cab 156 , such as along the front bumper or hood area, in the rear of the cab, adjacent to the fifth-wheel, underneath the chassis, etc. The trailer 154 may also have one or more sensor units 166 disposed therealong, for instance along a side panel, front, rear, roof and/or undercarriage of the trailer 154 .
By way of example, each sensor unit may include one or more sensors, such as lidar, radar, camera (e.g., optical or infrared), acoustical (e.g., microphone or sonar-type sensor), inertial (e.g., accelerometer, gyroscope, etc.) or other sensors (e.g., positioning sensors such as GPS sensors). While certain aspects of the disclosure may be particularly useful in connection with specific types of vehicles, the vehicle may be any type of vehicle including, but not limited to, cars, trucks, motorcycles, buses, recreational vehicles, etc.
There are different degrees of autonomy that may occur for a vehicle operating in a partially or fully autonomous driving mode. The U.S. National Highway Traffic Safety Administration and the Society of Automotive Engineers have identified different levels to indicate how much, or how little, the vehicle controls the driving. For instance, Level 0 has no automation and the driver makes all driving-related decisions. The lowest semi-autonomous mode, Level 1, includes some drive assistance such as cruise control. Level 2 has partial automation of certain driving operations, while Level 3 involves conditional automation that can enable a person in the driver's seat to take control as warranted. In contrast, Level 4 is a high automation level where the vehicle is able to drive without assistance in select conditions. And Level 5 is a fully autonomous mode in which the vehicle is able to drive without assistance in all situations. The architectures, components, systems and methods described herein can function in any of the semi or fully-autonomous modes, e.g., Levels 1-5, which are referred to herein as autonomous driving modes. Thus, reference to an autonomous driving mode includes both partial and full autonomy.
FIG. 2 illustrates a block diagram 200 with various components and systems of an exemplary vehicle, such as passenger vehicle 100 , to operate in an autonomous driving mode. As shown, the block diagram 200 includes one or more computing devices 202 , such as computing devices containing one or more processors 204 , memory 206 and other components typically present in general purpose computing devices. The memory 206 stores information accessible by the one or more processors 204 , including instructions 208 and data 210 that may be executed or otherwise used by the processor(s) 204 . The computing system may control overall operation of the vehicle when operating in an autonomous driving mode.
The memory 206 stores information accessible by the processors 204 , including instructions 208 and data 210 that may be executed or otherwise used by the processors 204 . The memory 206 may be of any type capable of storing information accessible by the processor, including a computing device-readable medium. The memory is a non-transitory medium such as a hard-drive, memory card, optical disk, solid-state, etc. Systems may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.
The instructions 208 may be any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by the processor. For example, the instructions may be stored as computing device code on the computing device-readable medium. In that regard, the terms âinstructionsâ, âmodulesâ and âprogramsâ may be used interchangeably herein. The instructions may be stored in object code format for direct processing by the processor, or in any other computing device language including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. The data 210 may be retrieved, stored or modified by one or more processors 204 in accordance with the instructions 208 . In one example, some or all of the memory 206 may be an event data recorder or other secure data storage system configured to store vehicle diagnostics and/or detected sensor data, which may be on board the vehicle or remote, depending on the implementation.
The processors 204 may be any conventional processors, such as commercially available CPUs. Alternatively, each processor may be a dedicated device such as an ASIC or other hardware-based processor. Although FIG. 2 functionally illustrates the processors, memory, and other elements of computing devices 202 as being within the same block, such devices may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, the memory 206 may be a hard drive or other storage media located in a housing different from that of the processor(s) 204 . Accordingly, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.
In one example, the computing devices 202 may form an autonomous driving computing system incorporated into vehicle 100 . The autonomous driving computing system may be capable of communicating with various components of the vehicle. For example, the computing devices 202 may be in communication with various systems of the vehicle, including a driving system including a deceleration system 212 (for controlling braking of the vehicle), acceleration system 214 (for controlling acceleration of the vehicle), steering system 216 (for controlling the orientation of the wheels and direction of the vehicle), signaling system 218 (for controlling turn signals), navigation system 220 (for navigating the vehicle to a location or around objects) and a positioning system 222 (for determining the position of the vehicle, e.g., including the vehicle's pose). The autonomous driving computing system may employ a planner module 223 , in accordance with the navigation system 220 , the positioning system 222 and/or other components of the system, e.g., for determining a route from a starting point to a destination or for making modifications to various driving aspects in view of current or expected traction conditions.
The computing devices 202 are also operatively coupled to a perception system 224 (for detecting objects and conditions in the vehicle's environment), a power system 226 (for example, a battery and/or gas or diesel powered engine) and a transmission system 230 in order to control the movement, speed, etc., of the vehicle in accordance with the instructions 208 of memory 206 in an autonomous driving mode which does not require or need continuous or periodic input from a passenger of the vehicle. Some or all of the wheels/tires 228 are coupled to the transmission system 230 , and the computing devices 202 may be able to receive information about tire pressure, balance and other factors that may impact driving in an autonomous mode.
The computing devices 202 may control the direction and speed of the vehicle, e.g., via the planner module 223 , by controlling various components. By way of example, computing devices 202 may navigate the vehicle to a destination location completely autonomously using data from the map information and navigation system 220 . Computing devices 202 may use the positioning system 222 to determine the vehicle's location and the perception system 224 to detect and respond to objects when needed to reach the location safely. In order to do so, computing devices 202 may cause the vehicle to accelerate (e.g., by increasing fuel or other energy provided to the engine by acceleration system 214 ), decelerate (e.g., by decreasing the fuel supplied to the engine, changing gears, and/or by applying brakes by deceleration system 212 ), change direction (e.g., by turning the front or other wheels of vehicle 100 by steering system 216 to the left or to the right), and signal such changes (e.g., by lighting turn signals of signaling system 218 ). Thus, the acceleration system 214 and deceleration system 212 may be a part of a drivetrain or other type of transmission system 230 that includes various components between an engine of the vehicle and the wheels of the vehicle. Again, by controlling these systems, computing devices 202 may also control the transmission system 230 of the vehicle in order to maneuver the vehicle autonomously.
Navigation system 220 may be used by computing devices 202 in order to determine and follow a route to a location. In this regard, the navigation system 220 and/or memory 206 may store map information, e.g., highly detailed maps that computing devices 202 can use to navigate or control the vehicle. As an example, these maps may identify the shape and elevation of roadways (e.g., including dips, angles, etc.), lane markers, intersections, crosswalks, speed limits, traffic signal lights, buildings, signs, real time traffic information, vegetation, or other such objects and information. The lane markers may include features such as solid or broken double or single lane lines, solid or broken lane lines, reflectors, etc. A given lane may be associated with left and/or right lane lines or other lane markers that define the boundary of the lane. Thus, most lanes may be bounded by a left edge of one lane line and a right edge of another lane line.
The perception system 224 includes sensors 232 for detecting objects and environmental factors external to the vehicle. The detected objects may be other vehicles, obstacles in the roadway, traffic signals, signs, trees, etc. The sensors 232 may also detect certain aspects of weather conditions, such as snow, rain or water spray, or puddles, ice or other materials on the roadway. Such sensors 232 may include one or more microphones positioned advantageously on the vehicle to facilitate detection of audio signals that can be used to predict road wetness or other environmental conditions. For example, the sensors 232 may include one or more microphones located between and behind one or more pairs of rear wheels of the vehicle. A selected vehicle may include enhanced sensors to provide water measurements for a roadway segment. By way of example only, a road weather information sensor from Lufft may be employed. Sensor data from such a selected vehicle could be used to train a DL model to predict road wetness or other environmental conditions based on road noise signals generated from one or more microphones and/or based on other sensor signals or other information that is likely to be available to the vehicle.
By way of example only, the perception system 224 may include one or more light detection and ranging (lidar) sensors and/or LED emitters, radar units, cameras (e.g., optical imaging devices, with or without a neutral-density filter (ND) filter), positioning sensors (e.g., gyroscopes, accelerometers and/or other inertial components), infrared sensors, acoustical sensors (e.g., microphones or sonar transducers), and/or any other detection devices that record data which may be processed by computing devices 202 . Such sensors of the perception system 224 may detect objects outside of the vehicle and their characteristics such as location, orientation, size, shape, type (for instance, vehicle, pedestrian, bicyclist, etc.), heading, speed of movement relative to the vehicle, etc. Ambient conditions (e.g., temperature and humidity) and roadway conditions such as surface temperature, dew point and/or relative humidity, water film thickness, precipitation type, etc. may also be detected by one or more types of these sensors.
The perception system 224 may also include other sensors within the vehicle to detect objects and conditions within the vehicle, such as in the passenger compartment. For instance, such sensors may detect, e.g., one or more persons, pets, packages, etc., as well as conditions within and/or outside the vehicle such as temperature, humidity, etc. Still further sensors 232 of the perception system 224 may measure the rate of rotation of the wheels 228 , an amount or a type of braking by the deceleration system 312 , and other factors associated with the equipment of the vehicle itself.
The raw data from the sensors, including the microphone(s) and/or other roadway condition sensors, and the aforementioned characteristics can be processed by the perception system 224 and/or sent for further processing to the computing devices 202 periodically or continuously as the data is generated by the perception system 224 . Computing devices 202 may use the positioning system 222 to determine the vehicle's location and perception system 224 to detect and respond to objects and roadway conditions when needed to reach the location safely, e.g., via adjustments made by planner module 223 . In addition, the computing devices 202 may perform calibration of individual sensors, all sensors in a particular sensor assembly, or between sensors in different sensor assemblies or other physical housings.
As illustrated in FIGS. 1 A and 1 B , certain sensors of the perception system 224 may be incorporated into one or more exterior sensor assemblies or housings. In one example, these may be integrated into the side-view mirrors on the vehicle. In another example, other sensors may be part of the roof-top housing 102 , or other sensor housings or units 104 , 106 a,b, 108 a,b, 112 and/or 116 . The computing devices 202 may communicate with the sensor assemblies located on or otherwise distributed along the vehicle. Each assembly may have one or more types of sensors such as those described above.
Returning to FIG. 2 , computing devices 202 may include all of the components normally used in connection with a computing device such as the processor and memory described above as well as a user interface subsystem 234 . The user interface subsystem 234 may include one or more user inputs 236 (e.g., a mouse, keyboard, touch screen and/or microphone) and one or more display devices 238 (e.g., a monitor having a screen or any other electrical device that is operable to display information). In this regard, an internal electronic display may be located within a cabin of the vehicle (not shown) and may be used by computing devices 202 to provide information to passengers within the vehicle. Other output devices, such as speaker(s) 240 may also be located within the passenger vehicle.
The passenger vehicle also includes a communication system 242 . For instance, the communication system 242 may also include one or more wireless configurations to facilitate communication with other computing devices, such as passenger computing devices within the vehicle, computing devices external to the vehicle such as in another nearby vehicle on the roadway, and/or a remote server system. The network connections may include short range communication protocols such as Bluetoothâ¢, Bluetooth⢠low energy (LE), cellular connections, as well as various configurations and protocols including the Internet, World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi and HTTP, and various combinations of the foregoing.
FIG. 3 A illustrates a block diagram 300 with various components and systems of a vehicle, e.g., vehicle 150 of FIG. 1 C . By way of example, the vehicle may be a truck, bus, farm equipment, construction equipment, emergency vehicle or the like, configured to operate in one or more autonomous modes of operation. As shown in the block diagram 300 , the vehicle includes a control system of one or more computing devices, such as computing devices 302 containing one or more processors 304 , memory 306 and other components similar or equivalent to components 202 , 204 and 206 discussed above with regard to FIG. 2 . The control system may constitute an electronic control unit (ECU) of a tractor unit of a cargo vehicle. As with instructions 208 , the instructions 308 may be any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by the processor. Similarly, the data 310 may be retrieved, stored or modified by one or more processors 304 in accordance with the instructions 308 .
In one example, the computing devices 302 may form an autonomous driving computing system incorporated into vehicle 150 . Similar to the arrangement discussed above regarding FIG. 2 , the autonomous driving computing system of block diagram 300 may be capable of communicating with various components of the vehicle in order to perform route planning and driving operations. For example, the computing devices 302 may be in communication with various systems of the vehicle, such as a driving system including a deceleration system 312 , acceleration system 314 , steering system 316 , signaling system 318 , navigation system 320 and a positioning system 322 , each of which may function as discussed above regarding FIG. 2 .
The computing devices 302 are also operatively coupled to a perception system 324 , a power system 326 and a transmission system 330 . Some or all of the wheels/tires 228 are coupled to the transmission system 230 , and the computing devices 202 may be able to receive information about tire pressure, balance, rotation rate and other factors that may impact driving in an autonomous mode. As with computing devices 202 , the computing devices 302 may control the direction and speed of the vehicle by controlling various components. By way of example, computing devices 302 may navigate the vehicle to a destination location completely autonomously using data from the map information and navigation system 320 . Computing devices 302 may employ a planner module 323 , in conjunction with the positioning system 322 , the perception system 324 and other subsystems to detect and respond to objects when needed to reach the location safely, similar to the manner described above for FIG. 2 .
Similar to perception system 224 , the perception system 324 also includes one or more microphones or other sensors or other components such as those described above for detecting objects and environmental condition (including roadway conditions) external to the vehicle, objects or conditions internal to the vehicle, and/or operation of certain vehicle equipment such as the wheels and deceleration system 312 . For instance, as indicated in FIG. 3 A the perception system 324 includes one or more sensor assemblies 332 . Each sensor assembly 332 includes one or more sensors. In one example, the sensor assemblies 332 may be arranged as sensor towers integrated into the side-view mirrors on the truck, farm equipment, construction equipment or the like. Sensor assemblies 332 may also be positioned at different locations on the tractor unit 152 or on the trailer 154 , as noted above with regard to FIGS. 1 C-D . The computing devices 302 may communicate with the sensor assemblies located on both the tractor unit 152 and the trailer 154 . Each assembly may have one or more types of sensors such as those described above.
Also shown in FIG. 3 A is a coupling system 334 for connectivity between the tractor unit and the trailer. The coupling system 334 may include one or more power and/or pneumatic connections (not shown), and a fifth-wheel 336 at the tractor unit for connection to the kingpin at the trailer. A communication system 338 , equivalent to communication system 242 , is also shown as part of vehicle system 300 .
FIG. 3 B illustrates an example block diagram 340 of systems of the trailer, such as trailer 154 of FIGS. 1 C-D . As shown, the system includes an ECU 342 of one or more computing devices, such as computing devices containing one or more processors 344 , memory 346 and other components typically present in general purpose computing devices. The memory 346 stores information accessible by the one or more processors 344 , including instructions 348 and data 350 that may be executed or otherwise used by the processor(s) 344 . The descriptions of the processors, memory, instructions and data from FIGS. 2 and 3 A apply to these elements of FIG. 3 B .
The ECU 342 is configured to receive information and control signals from the trailer unit. The on-board processors 344 of the ECU 342 may communicate with various systems of the trailer, including a deceleration system 352 , signaling system 254 , and a positioning system 356 . The ECU 342 may also be operatively coupled to a perception system 358 with one or more sensors for detecting objects and/or conditions in the trailer's environment and a power system 260 (for example, a battery power supply) to provide power to local components. Some or all of the wheels/tires 362 of the trailer may be coupled to the deceleration system 352 , and the processors 344 may be able to receive information about tire pressure, balance, wheel speed and other factors that may impact driving in an autonomous mode, and to relay that information to the processing system of the tractor unit. The deceleration system 352 , signaling system 354 , positioning system 356 , perception system 358 , power system 360 and wheels/tires 362 may operate in a manner such as described above with regard to FIGS. 2 and 3 A .
The trailer also includes a set of landing gear 366 , as well as a coupling system 368 . The landing gear provides a support structure for the trailer when decoupled from the tractor unit. The coupling system 368 , which may be a part of coupling system 334 , provides connectivity between the trailer and the tractor unit. Thus, the coupling system 368 may include a connection section 370 (e.g., for power and/or pneumatic links). The coupling system also includes a kingpin 372 configured for connectivity with the fifth-wheel of the tractor unit.
Example Implementations
While models for road surface and other conditions may be trained on human-labeled data, such an approach is subjective and can be error-ridden. Thus, selected sensor data is employed as a ground truth to the model. Various model architectures can be employed, for instance using a Neural Architecture Search (NAS) type model. Different model architectures can be used depending on the type(s) of data, such as one or more road noise audio signals or other relevant data (e.g., on-board lidar data and road graph information). Thus, any DL model that can be used to classify/regress road wetness using on-board microphone or other sensor signals and other available prior information (such as road graph data, etc.), may be employed.
Various sensors may be located at different places around the vehicle (see FIGS. 1 A-D ) to gather data from different parts of the external environment. Certain sensors may have different fields of view depending on their placement around the vehicle and the type of information they are designed to gather. For instance, different sensors may be used for near (short range) detection of objects or conditions adjacent to the vehicle (e.g., less than 2-10 meters), while others may be used for far (long range) detection of objects a hundred meters (or more or less) in front of the vehicle. Mid-range sensors may also be employed. Multiple sensor units such as lidars and radars may be positioned toward the front or rear of the vehicle for long-range object detection. And cameras and other image sensors may be arranged to provide good visibility around the vehicle. As described in greater detail below, microphones may be positioned between and/or behind one or more pairs of rear wheels of the vehicle(e.g., to generate audio signals that contain information relevant to road wetness or other environmental conditions of interest while reducing the amount of wind noise, engine, noise or other unwanted content in the audio signals). Depending on the configuration, certain types of sensors may include multiple individual sensors with overlapping fields of view. Alternatively, other sensors may provide redundant 360° fields of view.
FIG. 4 illustrates a scenario 400 in which a vehicle uses one or more sensors to detect the presence of water along the roadway in order to obtain ground truth data. For instance, the ground truth input may include measurements of the water thickness, e.g., water film thickness and/or ice coverage on road surfaces. This can be done at a very granular level, e.g., measuring the thickness on the order of microns. In this scenario, the vehicle may be configured to operate in an autonomous driving mode (or a manual mode), that includes various sensors at different locations along the exterior of the vehicle. This can include front and/or rear sensor units 402 , and a roof-based sensor unit 404 , each which may include lidar, radar, optical cameras, acoustic sensors and/or other sensors. These or other sensor units may be used to collect signals of the environment around the autonomous vehicle.
By way of example, the ground truth can be collected using sensors (e.g., front and/or rear sensors 402 ) designed for water thickness, e.g., water film thickness measurement and/or ice coverage. This could include, e.g., a road weather information sensor from Lufft. For instance, the front sensor may obtain data from scans shown via dashed lines 406 F , while the rear sensor may obtain data from scans shown via dashed lines 406 R . The roof-based sensor assembly may obtain information about objects or conditions around the vehicle as shown by dash-dot lines 408 . Notice that the sensors used to collect ground truth data may only be placed in selected vehicles for the training of deep learning models during the development phase. After deployment of such models on-board of the autonomous vehicles, these sensors that measure the road wetness do not need to be installed on vehicles.
The placement of the ground truth collecting sensor(s) around the vehicle may vary depending on the type of vehicle (e.g., sedan, truck, motorcycle, etc.) and other factors, so long as the sensor has a direct line of sight to the relevant portion of the roadway. Spray from tires or other vehicles could potentially have some effect, so to mitigate this the ground truth sensor should be covered by a protective housing. Also, water droplets passing across the sensor's sensing track can impact the optical sensing and affect the measurement. However, by avoiding mounting the sensor right above the tire tracks, the likelihood of water spray flying across the sensing track is small.
Besides sensors used for ground truth, FIG. 5 provides one example 500 of sensor fields of view relating to the sensors illustrated in FIG. 1 B . Here, should the roof-top housing 102 include a lidar sensor as well as various cameras, radar units, infrared and/or acoustical sensors, each of those sensors may have a different field of view. Thus, as shown, the lidar sensor may provide a 360° FOV 502 , while cameras arranged within the housing 102 may have individual FOVs 504 . A sensor within housing 104 at the front end of the vehicle has a forward facing FOV 506 , while a sensor within housing 112 at the rear end has a rearward facing FOV 508 . The housings 106 a, 106 b on the driver's and passenger's sides of the vehicle may each incorporate lidar, radar, camera and/or other sensors. For instance, lidars within housings 106 a and 106 b may have a respective FOV 510 a or 510 b, while radar units or other sensors within housings 106 a and 106 b may have a respective FOV 511 a or 511 b. Similarly, sensors within housings 108 a, 108 b located towards the rear roof portion of the vehicle each have a respective FOV. For instance, lidars within housings 108 a and 108 b may have a respective FOV 512 a or 512 b, while radar units or other sensors within housings 108 a and 108 b may have a respective FOV 513 a or 513 b. And the series of sensor units 116 arranged along a forward-facing direction of the vehicle may have respective FOVs 514 ,
CLAIMS
Claims ( 20 )
What is claimed is:
1. A system configured to operate a vehicle in an autonomous driving mode, the vehicle comprising a pair of front wheels, a first pair of rear wheels, and a second pair of rear wheels that are disposed, relative to the vehicle, behind the first pair of rear wheels, the system comprising:
memory storing a road condition deep learning model, the model relating to a discrete classification or continuous regression/estimation of road wetness; and
one or more processors operatively coupled to the memory, the one or more processors being configured to:
receive sensor data from a plurality of microphones of the vehicle while operating in the autonomous driving mode, the plurality of microphones being configured to detect road noise signals, the plurality of microphones including:
a first microphone disposed, relative to the vehicle, to the center of and behind the first pair of rear wheels and in front of the second pair of rear wheels; and
a second microphone disposed, relative to the vehicle, to the center of and behind the second pair of rear wheels;
use the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the road noise signals; and
use the generated information to control operation of the vehicle in the autonomous driving mode.
2. The system of claim 1 , wherein the one or more processors are additionally configured to:
apply a high pass filter to the road noise signals prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the road noise signals, wherein the high pass filter passes road noise detected by the first and second microphones at frequencies greater than 1 kHz.
3. The system of claim 1 , wherein the one or more processors are additionally configured to:
apply a band pass filter to the road noise signals prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the road noise signals, wherein the band pass filter passes road noise detected by the first and second microphones at frequencies between 1 kHz and 6 kHz.
4. The system of claim 1 , wherein the one or more processors are additionally configured to:
generate one or more features from the road noise signals, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the road noise signals comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more features, and wherein the one or more features comprise at least one of: (i) a mean of one of the road noise signals within a time window, (ii) a zero crossing rate of one of the road noise signals, (iii) a moment of a time domain waveform of one of the road noise signals, (iv) an energy in a frequency band of one of the road noise signals, (v) a ratio between an energy in two different frequency bands of one of the road noise signals, (vi) a moment of a spectrum of one of the road noise signals, or (vii) a shape of a spectrum of one of the road noise signals.
5. The system of claim 1 , wherein the one or more processors are additionally configured to:
generate a spectrum image from the road noise signals, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the noise signals comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the spectrum image.
6. The system of claim 1 , wherein the model is formed by evaluating a first set of training inputs of sensor data of an environment along a portion of a roadway from on-board sensors and a second set of training inputs of off-board information associated with the portion of the roadway with respect to ground truth data for the portion of the roadway, the ground truth data including one or more measurements of water thickness across one or more areas of the portion of the roadway, and the on-board sensors including the first and second microphones.
7. The system of claim 6 , wherein the second set of training inputs of off-board information includes one or more of weather station information, public weather forecasts, road graph data, crowdsourced information, or observations from one or more other vehicles; and
wherein controlling operation of the vehicle in the autonomous mode using the generating information includes at least one of alteration of a current driving action, modification of a planned route or trajectory, or activation of an on-board cleaning system.
8. A vehicle configured to operate in an autonomous driving mode, the vehicle comprising:
a pair of front wheels;
a first pair of rear wheels;
a second pair of rear wheels that are disposed, relative to the vehicle, behind the first pair of rear wheels;
a first microphone disposed, relative to the vehicle, to the center of and behind the pair of rear wheels and in front of the second pair of rear wheels, wherein the first microphone is configured to detect a first road noise signal;
a second microphone disposed, relative to the vehicle, to the center of and behind the second pair of rear wheels, wherein the second microphone is configured to detect a second road noise signal;
memory storing a road condition deep learning model, the model relating to a discrete classification or continuous regression/estimation of road wetness; and
one or more processors operatively coupled to the memory, the one or more processors being configured to:
receive sensor data that includes the first road noise signal and the second road noise signal while operating in the autonomous driving mode;
use the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data; and
use the generated information to control operation of the vehicle in the autonomous driving mode.
9. The vehicle of claim 8 , further comprising:
a third microphone, wherein the third microphone is configured to detect a third road noise signal, and wherein the one or more processors are additionally configured to:
combine the first road noise signal, the second road noise signal, and the third road noise signal to generate a combined road noise signal, using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the combined road noise signal.
10. The vehicle of claim 8 , further comprising:
a pair of mudflaps, wherein each mudflap of the pair of mudflaps is located proximate to and behind a respective rear wheel of the second pair of rear wheels, and wherein the second microphone is located to the center of and proximate to the mudflaps.
11. The vehicle of claim 8 , wherein the one or more processors are additionally configured to:
apply a high pass filter to the first road noise signal prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the first road noise signal, wherein the high pass filter passes frequencies greater than 1 kHz within the first road noise signal.
12. The vehicle of claim 8 , wherein the one or more processors are additionally configured to:
apply a band pass filter to the first road noise signal prior to using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the first road noise signal, wherein the band pass filter passes frequencies between 1 kHz and 6 kHz within the first road noise signal.
13. The vehicle of claim 8 , wherein the one or more processors are additionally configured to:
generate one or more features from the first road noise signal, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the one or more features, and wherein the one or more features comprise at least one of: (i) a mean of the first road noise signal within a time window, (ii) a zero crossing rate of the first road noise signal, (iii) a moment of a time domain waveform of the first road noise signal, (iv) an energy in a frequency band of the first road noise signal, (v) a ratio between an energy in two different frequency bands of the first road noise signal, (vi) a moment of a spectrum of the first road noise signal, or (vii) a shape of a spectrum of the first road noise signal.
14. The vehicle of claim 8 , wherein the one or more processors are additionally configured to:
generate a spectrum image from the first road noise signal, wherein using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the sensor data comprises using the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the spectrum image.
15. The system of claim 1 , wherein the vehicle further comprises:
a pair of mudflaps, wherein each mudflap of the pair of mudflaps is located proximate to and behind a respective rear wheel of the second pair of rear wheels, and wherein the second microphone is located to the center of and proximate to the mudflaps.
16. The system of claim 15 , wherein the second microphone is positioned to detect road noise related to the presence of road wetness on the mudflaps.
17. The system of claim 15 , wherein the plurality of microphones further includes a third microphone.
18. The system of claim 17 , wherein the one or more processors are additionally configured to:
combine a first road noise signal from the first microphone, a second road noise signal from the second microphone, and a third road noise signal from the third microphone to generate a combined road noise signal.
19. The system of claim 18 , wherein the one or more processors are additionally configured to:
use the stored model to generate information associated with the discrete classification or continuous regression/estimation of road wetness based on the combined road noise signal.
20. The vehicle of claim 10 , wherein the second microphone is positioned to detect road noise related to the presence of road wetness on the mudflaps.
US17/449,540
2021-09-30
2021-09-30
Using audio to detect road conditions
Active
2043-12-20
US12365345B2
( en )
Priority Applications (2)
Application Number
Priority Date
Filing Date
Title
US17/449,540
US12365345B2
( en )
2021-09-30
2021-09-30
Using audio to detect road conditions
EP22198254.9A
EP4159572B1
( en )
2021-09-30
2022-09-28
Using audio to detect road conditions
Applications Claiming Priority (1)
Application Number
Priority Date
Filing Date
Title
US17/449,540
US12365345B2
( en )
2021-09-30
2021-09-30
Using audio to detect road conditions
Publications (2)
Publication Number
Publication Date
US20230100827A1
US20230100827A1 ( en )
2023-03-30
US12365345B2
true
US12365345B2 ( en )
2025-07-22
Family
ID=84329749
Family Applications (1)
Application Number
Title
Priority Date
Filing Date
US17/449,540
Active
2043-12-20
US12365345B2
( en )
2021-09-30
2021-09-30
Using audio to detect road conditions
Country Status (2)
Country
Link
US
( 1 )
US12365345B2
( en )
EP
( 1 )
EP4159572B1
( en )
Families Citing this family (10)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
AU2018425455B2
( en )
*
2018-06-01
2025-05-01
Paccar Inc
Autonomous detection of and backing to trailer kingpin
JP7645389B2
( en )
*
2021-10-07
2025-03-13
æ¥ç«Astemoæ ªå¼ä¼ç¤¾
vehicle
US20230356551A1
( en )
*
2022-05-05
2023-11-09
Tusimple, Inc.
Control subsystem and method for detecting and directing a response to a tire failure of an autonomous vehicle
EP4283256A1
( en )
*
2022-05-23
2023-11-29
TuSimple, Inc.
Systems and methods for detecting road surface condition
US12583475B1
( en )
*
2022-06-06
2026-03-24
Zoox, Inc.
Sensor platform
US12091032B2
( en )
*
2022-08-11
2024-09-17
Toyota Motor Engineering & Manufacturing North America, Inc.
Engine sound enhancement during towing
US12325477B2
( en )
*
2023-03-09
2025-06-10
GM Global Technology Operations LLC
Vehicle control adaptation to sustained wind levels and gusts
GB2630811A
( en )
*
2023-06-09
2024-12-11
Jaguar Land Rover Ltd
Environmental condition monitoring for a vehicle
PL445479A1
( en )
*
2023-07-03
2025-01-07
Politechnika GdaÅska
System for acoustic determination of the road surface condition
CN121019582A
( en )
*
2024-05-27
2025-11-28
æ¯äºè¿ªè¡ä»½æéå ¬å¸
Road surface type identification methods, devices, vehicles, and cloud servers
Citations (12)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
JP5758759B2
( en )
*
2011-09-20
2015-08-05
ãã¨ã¿èªåè»æ ªå¼ä¼ç¤¾
Vehicle road surface determination device and driving support device
US20160221581A1
( en )
*
2015-01-29
2016-08-04
GM Global Technology Operations LLC
System and method for classifying a road surface
US20160349219A1
( en )
*
2013-12-18
2016-12-01
Antoine Paturle
Method for acoustic detection of the condition of the road and the tire
WO2018172464A1
( en )
2017-03-24
2018-09-27
Chazal Guillaume
Method and system for real-time estimation of road conditions and vehicle behavior
US20190101509A1
( en )
*
2017-10-04
2019-04-04
HELLA GmbH & Co. KGaA
Method for detecting moisture on a road surface
US20200018730A1
( en )
*
2017-03-24
2020-01-16
Compagnie Generale Des Etablissements Michelin
Sound measurement system for a motor vehicle
US20200189567A1
( en )
*
2018-12-12
2020-06-18
Waymo Llc
Determining Wheel Slippage on Self Driving Vehicle
EP3712020A1
( en )
2019-03-18
2020-09-23
Ask Industries Societa' per Azioni
System for monitoring an acoustic scene outside a vehicle
DE102019204609A1
( en )
2019-04-01
2020-10-01
Robert Bosch Gmbh
Method for determining a roadway condition while a vehicle is being driven
US20210101616A1
( en )
*
2019-10-08
2021-04-08
Mobileye Vision Technologies Ltd.
Systems and methods for vehicle navigation
US20210132628A1
( en )
*
2018-12-12
2021-05-06
Waymo Llc
Detecting General Road Weather Conditions
US11521127B2
( en )
2020-06-05
2022-12-06
Waymo Llc
Road condition deep learning model
2021
2021-09-30
US
US17/449,540
patent/US12365345B2/en
active
Active
2022
2022-09-28
EP
EP22198254.9A
patent/EP4159572B1/en
active
Active
Patent Citations (14)
* Cited by examiner, â Cited by third party
Publication number
Priority date
Publication date
Assignee
Title
JP5758759B2
( en )
*
2011-09-20
2015-08-05
ãã¨ã¿èªåè»æ ªå¼ä¼ç¤¾
Vehicle road surface determination device and driving support device
US20160349219A1
( en )
*
2013-12-18
2016-12-01
Antoine Paturle
Method for acoustic detection of the condition of the road and the tire
US20160221581A1
( en )
*
2015-01-29
2016-08-04
GM Global Technology Operations LLC
System and method for classifying a road surface
US20200324779A1
( en )
*
2017-03-24
2020-10-15
Compagnie Generale Des Etablissements Michelin
Method and system for real-time estimation of road conditions and vehicle behavior
WO2018172464A1
( en )
2017-03-24
2018-09-27
Chazal Guillaume
Method and system for real-time estimation of road conditions and vehicle behavior
US20200018730A1
( en )
*
2017-03-24
2020-01-16
Compagnie Generale Des Etablissements Michelin
Sound measurement system for a motor vehicle
US20190101509A1
( en )
*
2017-10-04
2019-04-04
HELLA GmbH & Co. KGaA
Method for detecting moisture on a road surface
US20200189567A1
( en )
*
2018-12-12
2020-06-18
Waymo Llc
Determining Wheel Slippage on Self Driving Vehicle
US20210132628A1
( en )
*
2018-12-12
2021-05-06
Waymo Llc
Detecting General Road Weather Conditions
US20200298756A1
( en )
*
2019-03-18
2020-09-24
Ask Industries Societa' Per Azioni
System for monitoring an acoustic scene outside a vehicle
EP3712020A1
( en )
2019-03-18
2020-09-23
Ask Industries Societa' per Azioni
System for monitoring an acoustic scene outside a vehicle
DE102019204609A1
( en )
2019-04-01
2020-10-01
Robert Bosch Gmbh
Method for determining a roadway condition while a vehicle is being driven
US20210101616A1
( en )
*
2019-10-08
2021-04-08
Mobileye Vision Technologies Ltd.
Systems and methods for vehicle navigation
US11521127B2
( en )
2020-06-05
2022-12-06
Waymo Llc
Road condition deep learning model
Non-Patent Citations (1)
* Cited by examiner, â Cited by third party
Title
JP-5758759-B2âEnglish Translation (Year: 2015).
*
Also Published As
Publication number
Publication date
EP4159572B1
( en )
2025-12-10
EP4159572A1
( en )
2023-04-05
US20230100827A1
( en )
2023-03-30
Similar Documents
Publication
Publication Date
Title
US12511581B2
( en )
2025-12-30
Road condition deep learning model
EP4159572B1
( en )
2025-12-10
Using audio to detect road conditions
US12066836B2
( en )
2024-08-20
Detecting general road weather conditions
US12578724B2
( en )
2026-03-17
Detection of anomalous trailer behavior
CN113167906B
( en )
2024-06-28
Pseudo Object Detection for Autonomous Vehicles
JP7697931B2
( en )
2025-06-24
DETECTING AND ACTING ON ABERRANT DRIVER BEHAVIOR USING DRIVER ASSISTANCE - Patent application
US12280803B2
( en )
2025-04-22
Identifying the position of a horn honk or other acoustical information using multiple autonomous vehicles
KR102733944B1
( en )
2024-11-27
Sensor field of view for autonomous vehicles
CN113176096A
( en )
2021-07-27
Detection of vehicle operating conditions
US20220291690A1
( en )
2022-09-15
Continuing Lane Driving Prediction
CN114435392B
( en )
2025-04-29
Use the presence of road surface and surrounding area lighting to detect occluded objects
US12498470B1
( en )
2025-12-16
Surface fouling detection
US12384426B1
( en )
2025-08-12
Positional gaps for driver controllability
US12532110B2
( en )
2026-01-20
Vehicle sensor modules with external audio receivers
US20260141307A1
( en )
2026-05-21
Road condition deep learning model
US20260004405A1
( en )
2026-01-01
Methods and Systems for Mitigating the Effects of Weather-related Attenuation on Radar Imagery
Legal Events
Date
Code
Title
Description
2021-09-30
FEPP
Fee payment procedure
Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY
2021-11-05
STPP
Information on status: patent application and granting procedure in general
Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION
2022-09-07
AS
Assignment
Owner name : WAYMO LLC, CALIFORNIA
Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHOU, XIN;ZHONG, XUAN;MCCOOL, COURTNEY;SIGNING DATES FROM 20220503 TO 20220906;REEL/FRAME:061016/0612
2024-05-28
STPP
Information on status: patent application and granting procedure in general
Free format text : NON FINAL ACTION MAILED
2024-07-06
STPP
Information on status: patent application and granting procedure in general
Free format text : RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER
2024-09-10
STPP
Information on status: patent application and granting procedure in general
Free format text : NON FINAL ACTION MAILED
2024-12-11
STPP
Information on status: patent application and granting procedure in general
Free format text : RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER
2024-12-17
STPP
Information on status: patent application and granting procedure in general
Free format text : FINAL REJECTION MAILED
2025-02-19
STPP
Information on status: patent application and granting procedure in general
Free format text : ADVISORY ACTION MAILED
2025-03-06
STPP
Information on status: patent application and granting procedure in general
Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION
2025-07-09
STCF
Information on status: patent grant
Free format text : PATENTED CASE