A howl on the wind…

While the Apache HTTP Server is a wonderful piece of software, it sadly does not have any built-in way of running vhosts as the user they represent, which has been the headache of many an admin. Either have to fiddle with groups, running multiple daemons, or something worse. In most cases, it led to users being able to access eachother’s files. There was an experimental multi-processing module called perchild designed to combat this, but it ultimately never made it into the 2.2 branch. Speak no evil of the dead, and so forth.

If one is able to apply and maybe edit a patch, one can achieve near-perfect user seperation in a single Apache process, though.

The two projects, Telana peruser by Sean Gabriel Heacock and the ITK mpm by Steinar H. Gunderson, provide such patches. Each have their own gotchas and configuration, but once you get either of them working you’ll never want to go back to running Apache as a single global user. Why neither of these have been included upstream or in major distributions, one can but wonder…

One of the ways to seamlessly transition services from one machine to another is by setting the Time To Live of the domains to something very low a few days before, then switch their IP to the new, and setting TTL back to normal when you can be sure it has propagated.

While that works, I couldn’t do that; I did not have access to change the TTL of all the domains attached to the machine, and didn’t want to carefully coordinate with all users…it’s bound to mess up somewhere.

So, I turned to my old friends rsync and OpenSSH.

The Transition Process

• Set up and configure the basics of the new machine (see my previous post).
• Create all desired users on the new machine.
• Create the applicable /home skeletons (~/mail/ ~/public_html/ and such)
• Configure services: Apache vhosts, Dovecot maildir, etc.
• On the old machine, disable SSH logins for the users to be transitioned. If you allow FTP (why would you? SSH provides.), disable that too.
• rsync -avzrltSpP /home and /var/spool/mail from old machine to new machine
• Make sure everything works; can do this by temporarily changing your own hosts file to point to the new IP for the domains.
• rsync -avzrltSpP /home and /var/spool/mail from old machine to new machine again, to make sure there are only minor changes for next step.
• Stop services on the old machine
• rsync -avzrltSpP /home and /var/spool/mail from old machine to new machine again, final time. Data transferred here is the data that will be live in a sec.
• Start services on the new machine
• SSH forward applicable ports from the old machine to the new machine; in my case ports 25/SMTP, 80/HTTP, 110/POP3, 143/IMAP, 587/Submission. Remember to allow remote hosts to use the forwarded ports (cmdline option -g). Has to be done as root as the ports are below 1024.
• Change domains to point to the new IP, and notify owners of the domains you don’t control to do the same.
• Once all domains are over and the IP is propagated, kill the SSH tunnels.

If executed properly in the off-hours, this should cause downtime of a few minutes. Majority of the downtime is for rsync.

Should be noted there is no real need to transition everything in one go. Mail can be done seperately, if handled properly; in my case I had users store mail in ~/mail/ and sites in ~/public_html/ so it was simply easier to rsync the whole /home over and do all services at once.

Problems and Gotchas

• Due to the nature of SSH tunnels all requests that pass through them will appear on the target machine as coming from localhost. This may cause problems with some services and scripts.

  In my case, Sendmail was the problem: localhost is a trusted sender, so suddenly spam was being sent since it was being blindly tunneled from the old machine. This is easily avoidable by using Sendmail’s built-in relaying; I simply hadn’t thought about it.

• This method will not work for anything SSL (HTTPS, IMAPS, POP3S, etc), and that is a security feature.
• I did not have the need to transition databases; I already had MySQL and PostgreSQL running on a different machine. Those can and should be transitioned seperately before anything else, though, as they have quite different methods of doing so, and will require more downtime than other services.

Edited 2008-10-31: Refinements, further optimizations, Fedora 9.

This week I got a fresh machine from ServerBeach to play with, and thought it would be interesting to jot down what I do with a server before I consider it usable. The preinstalled OS is Fedora Core 6.

The order here is not chronological; more a general overview of steps.

Updating existing packages

• Easy step: yum upgrade

Replacement of some default packages

• Uninstalled the httpd package and all dependencies in favour of compiling the Apache HTTP Daemon myself. I never understood why Red Hat decided on the scattered structure with their package, so I install Apache from source to make sure it is all self-contained in /usr/local/apache2. Also for this machine I added the ITK MPM to run each vhost as a seperate user.
• Replaced the existing version of MySQL with the vendor RPMs.
• Ditto for PostgreSQL.
• Installed PHP from source.
• Installed Subversion from source.
• Replaced the existing version of Webmin with the vendor RPM.

Installation of new packages

• Midnight Commander from FC6 repository. Cannot live without this.
• Enabled Bind and set it up to serve as resolver for the machine by forwarding to existing resolvers. This greatly helps with lookup speeds when doing lots of lookups for the same hostname, such as Sendmail and Apache will be doing.
• Enabled Dovecot for IMAP.

File System

• Disable updating of last access time (option noatime in /etc/fstab).
• “tune2fs -o journal_data_writeback” to speed up even more, at the cost of crash recovery.
• “tune2fs -m1” to lower reserved space from 5% to 1%. On non-system partitions I set it to 0%.

PHP

• Installed the Alternative PHP Cache.
• Set up session and upload folders elsewhere than /tmp.
• Set up sessions to use multi-level folders, normally 3-levels deep. This prevents the single folder from becoming unusably huge.

Sendmail

• Configure to only allow user mail sending via authenticated submission (port 587).
• Add DNS based blacklists:
  SpamCop
  Distributed Sender Blackhole List
  Spamhaus SBL + XBL

Other stuff

• Edited /etc/sysctl.conf to include:
  kernel.shmmax = 536870912
  net.ipv4.tcp_fin_timeout = 10
• Configured logrotate to keep 30 days worth of logs instead of 4, and to compress rotated logs.
• Enabled logrotate for the root mailbox (I always forget to delete mails in it).
• Enabled logrotate for Apache access and error log.
• Added nightly incremental backup of /home, /etc, and /var/spool/mail to a remote server.
• Added nightly time synchronization to pool.ntp.org.
• Added nightly cleanup of old files in /tmp and other temporary folders.
• Enabled the firewall.

That should about cover it…

In order of importance:

• Run the game with your computer cabinet open to mitigate possible overheating.
• Clean inside your cabinet. Dust may be preventing your fans from turning.
• Test your memory with Memtest86 from http://www.memtest86.com/
• Test your CPU with Prime95 from http://www.mersenne.org/freesoft.htm
• Update your drivers.
• Reset your BIOS settings to factory default.
• Reset your BIOS settings to safe mode, if possible.
• Open both cabinet and use a window/fan/whatever to really make sure overheating is not the issue.
• Swap around internal power cables. Could be the one that goes to your GPU is overloaded and doesn’t feed enough power to maintain it.
• Could be your PSU is generally overloaded. Unplug any device that you are not using. If you have 2 HDDs, remove the excess one. If you have any unused PCI cards, take them out.
• At this point, pray. It could be anything and you need to do an individual component check, but it is most commonly the PSU that’s the culprit.
• …if all that fails, report back when you find what exactly was wrong.

I have Microsoft Windows Millenium Edition and I have 2 blocks of 512 MB RAM. Until yesterday I had only had 1 512 MB block installed, because of a limit in non-NT versions of Microsoft Windows (described by MS themselves). I found that a bit of a drag, so I set out to circumvent it in one way or the other. The theory is simple: Install a 512 MB RAMDrive before Windows boots, and use that as Virtual Memory (swap).

Problems

1. Windows ME skips both config.sys and autoexec.bat in the boot sequence.
2. The standard ramdrive.sys supports max 64 MB per drive.

Solutions

1. Install the Real DOS-Mode Patch for Windows Millennium (local mirror)
2. Use a alternate RAMDrive (local mirror)

But, that still didn’t quite solve my problem. Sure, I used all 1024 MB RAM, but Windows complained about “Not Enough Memory” when starting a DOS prompt. So, in addition, I used both of the tweaks from the MS article and added some lines to my system.ini:

1. Under the section [vcache] I added MaxFileCache=131072. This will make Windows use max 128 MB RAM for file cache, which in all cases is a good thing. If you want more, just set it to the full 524288 (512 MB).
2. Under the section [386Enh] I added MaxPhysPage=80000. What this should do, is to make Windows use at max 512 MB RAM. For some reason it is in MB, so I actually told it to use 524288 MB RAM, but it works nonetheless. If it doesn’t for you, use MaxPhysPage=200 or MaxPhysPage=512 instead.

The line I used for my RAMDrive in autoexec.bat is:
XMSDSK 524288 R: /T /C1 /Y

Notes

• I use a 512 MB RAMDrive because I have 1024 MB RAM total (1024 total minus 512 limit is 512 left for drive), but for people with 768 or 640 MB RAM total, or similar, then a smaller RAMDrive is probably better. Also, a RAMDrive smaller than 256 MB should not be used for Virtual Memory.
• Doing it in this way, by essentially patching the boot sequence, you probably break your Microsoft Windows Millenium Edition license.
• For users of Microsoft Windows 98 or 98 Second Edition, the Real DOS-Mode Patch for Windows Millennium would not do anything, and should not be installed. Asides from that, the XMSDSK should work with 98 and 98SE, as should the system.ini changes.