its seems like its toe or camber issue, small road inperfection throws it off the balance and forces to one side. I would do quick alignment check to rull out the whole suspension so you can focus more on mechanical parts. Also check all the rubber boots on all CV joints to make sure none of em are damages and stuff got lodged making it work harder, cause that would cause torque pull. Quick differential fluid check could tell you few things as well. You know, dirty fluid or with metal flakes - seriously someting wrong, etc..