There is literally zero downside to add-ons. They only do exactly what the designers want them to do. Certain add-ons, such as DBM, are a little controversial. It only exists because Blizzard wants it to exist. There are dozens of ways they could get rid of the real-time processing needed to provide things like boss timers (which is all DBM really is).
Nobody forces you to use an add-on, so I've never understood how add-ons can hurt your game. What do you care if other people want to change the layout of their bars or modify the way debuffs show up.
In particular, games without Auction APIs generally have weak economies. The current TOR economy is pretty unstable. For a well-functioning capitalist market, you really need people who act as market makers by watching prices, holding inventory etc. This is impossible without good tools. Without add-ons, auction houses crumble.