If you want to access an NTP server by a virtual address, it will need to be supplied the -L flag when it is started.
There are debates about whether one needs to or should use virtual addresses for NTP servers. These debates can be found in this email thread.
A compelling argument for running at least one NTP server with a virtual address in some circumstances is because some network hardware only allows a single server to be listed in its configuration.
Other less compelling arguments have been discussed in the aforementioned email thread.