Mine doesn't ever happen while driving. Only after it rains and I've let my car sit around a week, not driven.
This sounds to me like the e-brake pads sticking to the drums. I had similar issues with my old truck after using the e-brake after raining and letting it sit overnight. The brake shoes seize to the drums and make an awful noise when you start to move. My solution was to park on level ground in gear without the e-brake on.
Did you notice any resistance when you first starting moving your car? If your situation is the same as mine was, I wouldn't be concerned.