Moving the library from near the bus station to the bottom of the high street may make it less accessible for the elderly and disabled, Aylesbury Town councillors said.
It was also suggested that the current location makes it a hub for young people.
Council leader Mike Smith said: “The library is the meeting point for young people. They use it in different ways but they do use it. You need to pay attention to the needs and wants of existing customers because you could find a chunk of your customer base not being there any more.”
Councillor Penni Thorne said moving it would make it hard for people with mobility difficulties at the top of town to use it, but Councillor Barbara Russel said the current location is not ideal for all.
She said: “If the library was at the bottom of the high street some of the older people in my ward would find it much more easy to get to.”
The plans are still at an early discussion stage.