Anyone can "sue" anyone. Whether litigation takes place is a whole other story. Right now the guy has a lawyer and is probably trying to scare you into some form of settlement. The litigation process is very long and expensive. It is not in this guys best financial interest to take you to court.
Get a lawyer, and see what he/she says.
The fact that he hit the side of your car is proof enough that you did not hit him.