The man touched the inner thigh of a woman in her 20s as she was walking along Park Street.
Thames Valley Police believe the incident happened between 2pm and 2.30pm.
The victim was walking along Park Street with a relative on the opposite side to Aqua Vale Swimming and Fitness Centre when the man approached them asking for help.
Once he was close enough he leaned over walking rail and touched the woman’s inner thigh and leg.
She said ‘no’ and shook the perpetrator off, after that he carried on walking away.
Police intelligence states the is an Asian man, in his 50s, roughly five foot 10 inches tall, with dark grey short hair.
He was wearing a short sleeved shirt which witnesses describe as being white, cream colour.
Investigating officer, PC Holly Andrews, based at Aylesbury police station, said: “We are appealing for anyone who may have seen what happened to please come forward.
“We’d also ask any motorists with dash-cams who may have been in the local area around that time of this incident to check any footage in case it may have captured something that could assist our investigation.
“Anyone with information can call 101 or make a report through the website, quoting reference 43220167944.
“Alternatively, if you wish to remain anonymous, you can contact the independent charity Crimestoppers on 0800 555 111 or via its website.”