Replace the grease on the swaybar bushings with silicone grease. Most swaybars come with white lithium grease, which burns off pretty quick and leaves sticky residue that "grabs" the bar and pops when you go over stuff or turn hard enough. If it was an issue with something in the steering system it would most likely be repeatable and frequent.
I assume it also does this going in and out of angled driveways? You can test my theory by spraying some silicone lube into the swaybar bushings before pulling them apart.