I hope no one finds this offensive, but what would you do?

This is obviously bothering me because it is now 2 weeks since the incident - the first night I couldn't sleep because I kept thinking "what if" and now I'm wondering if it should be reported and to whom......Here's the story. I belong to a cycling group, we are all older and we do different routes each ride. Our leader took us to a park area that he knew and to get to it we had to travel a long private driveway(about 1/4 mile)which was in the process of being graded with gravel. I found it very hard going on my road bike, so on the return trip I decided to walk the driveway ahead of the group. Partly along the way a fellow was on his tractor(I guess he was doing the grading)and when he saw me walking my bike, stopped and got off and came towards me. I naturally said Hi. He kept coming towards me and then put both his hands at his crotch, pulling at himself. I thought to myself "how rude", but kept walking and talking(I'm sure he answered me). Next thing he puts both hands down the inside the front of his shorts and I'm getting a little scared, wondering what the heck I should do. Thankfully at that point, I could hear(as could he) the voices of the rest of the group coming and almost ran the rest of the way down the driveway.
Because of confusion about which route we were taking home, I never did tell the group, although I did ask who the guy was and was told he was co-owner of the property.
I keep telling myself maybe he was mentally challenged and didn't know he was scaring me, or maybe just one of those really uncouth people who's offensive behaviour is the norm. Or he is a pervert and I honestly can't decide - I'm just grateful the rest of the group showed up when they did.

You know, I just feel better getting this off my chest! Maybe writing it down was needed.