A howl on the wind…

One of the ways to seamlessly transition services from one machine to another is by setting the Time To Live of the domains to something very low a few days before, then switch their IP to the new, and setting TTL back to normal when you can be sure it has propagated.

While that works, I couldn’t do that; I did not have access to change the TTL of all the domains attached to the machine, and didn’t want to carefully coordinate with all users…it’s bound to mess up somewhere.

So, I turned to my old friends rsync and OpenSSH.

The Transition Process

• Set up and configure the basics of the new machine (see my previous post).
• Create all desired users on the new machine.
• Create the applicable /home skeletons (~/mail/ ~/public_html/ and such)
• Configure services: Apache vhosts, Dovecot maildir, etc.
• On the old machine, disable SSH logins for the users to be transitioned. If you allow FTP (why would you? SSH provides.), disable that too.
• rsync -avzrltSpP /home and /var/spool/mail from old machine to new machine
• Make sure everything works; can do this by temporarily changing your own hosts file to point to the new IP for the domains.
• rsync -avzrltSpP /home and /var/spool/mail from old machine to new machine again, to make sure there are only minor changes for next step.
• Stop services on the old machine
• rsync -avzrltSpP /home and /var/spool/mail from old machine to new machine again, final time. Data transferred here is the data that will be live in a sec.
• Start services on the new machine
• SSH forward applicable ports from the old machine to the new machine; in my case ports 25/SMTP, 80/HTTP, 110/POP3, 143/IMAP, 587/Submission. Remember to allow remote hosts to use the forwarded ports (cmdline option -g). Has to be done as root as the ports are below 1024.
• Change domains to point to the new IP, and notify owners of the domains you don’t control to do the same.
• Once all domains are over and the IP is propagated, kill the SSH tunnels.

If executed properly in the off-hours, this should cause downtime of a few minutes. Majority of the downtime is for rsync.

Should be noted there is no real need to transition everything in one go. Mail can be done seperately, if handled properly; in my case I had users store mail in ~/mail/ and sites in ~/public_html/ so it was simply easier to rsync the whole /home over and do all services at once.

Problems and Gotchas

• Due to the nature of SSH tunnels all requests that pass through them will appear on the target machine as coming from localhost. This may cause problems with some services and scripts.

  In my case, Sendmail was the problem: localhost is a trusted sender, so suddenly spam was being sent since it was being blindly tunneled from the old machine. This is easily avoidable by using Sendmail’s built-in relaying; I simply hadn’t thought about it.

• This method will not work for anything SSL (HTTPS, IMAPS, POP3S, etc), and that is a security feature.
• I did not have the need to transition databases; I already had MySQL and PostgreSQL running on a different machine. Those can and should be transitioned seperately before anything else, though, as they have quite different methods of doing so, and will require more downtime than other services.