Do you actually restart the server or just the daemon? Restarting the server is rarely needed except for things like kernel patches, etc. At most you’d just restart the daemon or the PHP service.
I’m guessing your server time is not being updated properly. Can your run
date from SSH next time you restart to see what it says? Is this a virtual machine from a hosting provider? If so, what virtualization method is used (OpenVZ, KVM, etc.). Things like OpenVZ have their clocks set from the host. Have you done this more than twice to know that it’s always three days?