I can speak for my experience with my 5mt and the cvt loaners I've had.
The manual is WAY more fun on dirt/clay/sandy roads. Gravel is where the abs, vdc, etc really ruin the fun. But I need RA flaps before I take any gravel or ROUGH roads too hard.
Sure is fun on graded dirt roads out in the ocala national forest. Did some "rally cross" of my own last weekend. Just spent two hours washing, although love bugs and blind mosquitos are mostly to blame.
When someone figures out a good way of eliminating all the "anti-fun" elements of the car, I'm sure there will be more go-fast goodies to help increase the fun-factor as well.