100% it's the muffler. They can make all the difference, and with mufflers you get what you pay for. One time I tried a 3" Bosal twin tip exhaust on my wrx (no cats). It looked awesome, but was unbearably loud at low rpm and on the highway.
I sold it after a week and bought a Cobb 3" exhaust (no cats) and the difference was night and day. No droning, no unbearable booming at low speeds... Just a nice and clean sounding, free flowing exhaust