Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training

In a neural network with ReLU activations, the number of piecewise linear regions in the output can grow exponentially with depth. However, this is highly unlikely to happen when the initial parameters are sampled randomly, which therefore often leads to the use of networks that are unnecessarily la