I frequently go for walks in the afternoons on a trail that's close to my place. I drive and park in a nearby public parking lot and then walk from there. Today I finished my walk and was about to get into my car in the lot when I noticed a red car slowly approaching me. The lot was practically empty on the side I had parked so I was surprised when the car pulled up next to me. I didn't think much of it, got into my car and was about to drive away when I noticed the driver of the red car saying something to me so I rolled down my window. It was a young man, probably 20s. I was wearing my non-prescription sunglasses so I wasn't able to clearly make out his face. This is how the conversation went:

Man: Hey, is there a lake nearby this area?

Me: Yes, there is a lake and a river just 5 minutes from here. *proceeded to give him directions*

Man: Oh how long is the trail?

Me: About 20km one way. It's a loop and has many entrances.

Man: Oh do you go the whole way?

Me: No I do 5-10km depending on how much time I have.

Man: Do you go everyday at this time?

Me: Depends on my schedule, I try whenever I can. (At this point I was getting concerned as to where this was going and was on my guard)

Man: Oh wow, maybe we can go together sometime? Do you want to do that?

Me: Umm...........(I was feeling pretty uncomfortable and felt like he was hustling me. My mind was racing thinking how to get out of there safe and fast)

Man: How about we go for a coffee?

Me: Thanks but I'm going to have to say no.

Man: Why not? C'mon. Let's go.

Me: No, I can't, I have a husband. (I made this up so that he would leave me alone)

Man: So what?

Me: No.

I made a dismissive gesture and rolled up my window, preparing to drive. I waited for him to leave first as I didn't want him to follow me. He sat there for a minute (I couldn't see his expression) and then drove away. I waited for 5 minutes or so after I was sure that he was nowhere in sight, and then left. On the way home I was constantly checking my mirrors to make sure I wasn't being followed. After I thought about it a little, I subconsciously recalled that his car was actually parked in a different spot when I was walking towards my car so he was probably scouring me while I was walking. If I am not mistaken, I think I saw a red car parked in the lot yesterday too but I cannot be sure if it was him.

Now all this happened in broad daylight. That parking lot is on the ground level in the open and is not secluded at all. Pedestrians are often walking around it and there are vehicles passing by, so it is pretty safe. Also I had my big sunglasses so he never saw my full face (which makes me wonder, a man must be pretty desperate if he is asking out a woman without even seeing her full face). I don't think I will recognize him except that he had an accent since I didn't have my prescription glasses on, but I do remember his car.

A part of me is a little worried that he might return. I love going to that trail and would hate to have to go elsewhere because of this. I've been going there for many years and never had any uncomfortable incident ever. I am thinking of switching up my timings to noon instead of 4pm if my work permits (I work from home). I can also start parking elsewhere, so there are options. I guess my question is did I handle it right?

TLDR: Got hustled by a man as I was getting into my car. He asked me for a coffee and I declined. He left. I am concerned he might return. How to better navigate such situations in future?