Is net neutrality not important in a competitive market of internet providers?

In my understanding, allowing ISP to throttle sites that use a lot of traffic is just price discrimination. The ISP will charge sites like Netflix, which will then pass the cost to Netflix consumers. So basically people who use a lot of data will get charged more--seems perfectly fine to me. I understand the argument that in practice, the ISP won't build new infrastructure but simply put up roadblock to slow down the current speed. However, this seems like a problem only in the oligopolistic market of US internet providers. In a competitive market, putting up roadblock will cause an exodus to competitors. So, am I correct in thinking that preserving Net Neutrality is not important in a competitive telecom market? Instead of fighting to preserve Net Neutrality, may be the correct fight is to encourage more ISP to enter the market.