Still Here

Some of you may have noticed alarming messages on my site saying that my account has been suspended! Or maybe you saw “Bad Gateway” errors and broken links. It’s nothing to worry about.

I run the web server that hosts this site, so the only one who can suspend me is me. Actually, it’s a virtual private server (VPS) that InMotion Hosting has running in the cloud somewhere, and if they got mad at me, they could certainly kick me out, but it wouldn’t result in the web server showing an “Account Suspended” banner because there wouldn’t be a webserver.

I’ve just been changing servers. InMotion provisioned the server when I come on board, but they haven’t been upgrading it much since then. To be sure, I get a variety of security and maintenance updates to the software on the server, but they are still somewhat limited in what they can upgrade while I’m using it. So every few years I spin up a new hosting plan and move all the websites^[1]I host a few sites for friends. over to their bright new home.

Despite the complexity of modern hosting, it usually goes remarkably well: Using the cPanel management platform, I download a backup of each site from the old server, upload that backup onto the new server, and then switch the CloudFlare front-end to point to the new server’s IP address. It only takes a few minutes per site. Then I suspend the site account on the old server, so I can be sure that I really am serving the site from the new server.

This time that step was even easier, because the new versions of WHM have a slick transfer tool. I just give the new server my login credentials for the old server, and it connects over and lists all the sites. I pick the ones I want to move, and it downloads and installs them on the new server. All I have to do is switch IP addresses at CloudFlare.

There were a couple of glitches, however, which is why you might have noticed some problems. The rest of this post is about those glitches.

The first glitch was that something that didn’t work right with the origin server certificates. You may have noticed that my site shows up in your browser as having a secure connection — the URL starts with https and the certificate is valid. Normally, that would happen because I went through the traditional process of generating a digital server certificate and having it signed by a trusted authority. Here, however, it’s happening because your browser isn’t talking directly to my server, it’s talking to the proxy servers at CloudFlare and they add the appropriate SSL/TLS certificates on my behalf as a free service. It’s one click and you see a secure connection between your browser and CloudFlare.

The connection between my server and Cloudflare is a little more complicated. I could use a traditional SSL/TLS server ticket, signed by the appropriate authority, and CloudFlare would accept the signature to authenticate my website. But it turns out I can avoid doing that work by generating and installing a certificate that has been automatically signed by CloudFlare. No ordinary browser client will accept such a certificate, of course, but Cloudflare will. And that authenticates and secures my connection through CloudFlare to your browser.

The problem was that some of the certificates did not get installed in the new server, and it took me a while to figure out what had happened. During that time, I was switching back and forth between servers, so you may have caught a glimpse of the “Account Suspended” banner when I switched servers before I remembered to unsuspend my account. Eventually I got it sorted out.

The second glitch was far more confusing: Some of the posted images on my browser weren’t loading in the web pages. I verified that the server was sending 404 Not Found errors for those image requests. But why?

I quickly realized that the problem was with thumbnail images. If a page requested a full image, the server found it, but if the page tried to load a thumbnail, it wasn’t found. I checked the directory where all the media for posts are stored, and the thumbnail files were missing. There are ways to force WordPress to regenerate the thumbnails, but…this was ringing a bell…

I had done something unorthodox many years ago, and now I was paying the price.

It all started because I don’t like the way WordPress handles thumbnail images. If you upload a file named fluffy-cats.jpg, WordPress will immediately generate a bunch of pre-configured thumbnails, and give them names that encode the sizes: fluffy-cats-100x100.jpg, fluffy-cats-270x270.jpg, fluffy-cats-500x300.jpg, and so on. The problem I had was that it would put these thumbnails side-by-side in the same folder as the original image, creating a lot of clutter. For better or worse^[2]At the time, media management in WordPress was very primitive. It’s gotten better over the years, and I probably wouldn’t worry about the thumbnail clutter problem if I faced it today., I decided to change how this worked.

I wrote a WordPress plug-in to provide dynamic thumbnails on demand. WordPress operates as a front controller, meaning that it analyzes the URL of each web request and responds to it by generating the page content from its database. For the sake of efficiency, however, the Apache web server first checks the filesystem for a matching file. If it finds one, it serves it directly, without ever involving the WordPress code. On the other hand, if it doesn’t find the file, it still has to fire up WordPress to generate the response.

What my plug-in did was hook into WordPress to respond when the Apache web server couldn’t find an image file. So if a request came in for FluffyCat.jpg, the Apache web server would find that and never forward the request to WordPress. But if the request was for a file using the standard sizing notation such as FluffyCat-100x100.jpg, Apache would discover that there was no such file and forward the request to WordPress, where my plug-in would locate the source FluffyCat.jpg file — all 5 megabytes of it — resize it to 100 by 100, and send it back to the requesting browser client.^[3]Yes, this is less CPU efficient than pre-generated static thumbnails, but the generated files are still going to be cached by CloudFlare, so I’m not too worried about the additional load on the processor.

For some reason, that wasn’t working. WordPress was never seeing the request for the resized file. I poked around in the logs, and it didn’t even look like Apache was receiving the request. What the heck?

As it turns out, there’s one more layer to all this. The VPS is running a lightweight web server called nginx as a reverse-proxy cache, meaning that it keeps copies of the responses to some requests and next time it sees one of the same requests, it fills it directly from its copy without involving the Apache server or WordPress.^[4]This was another reason I wasn’t too worried about generating the thumbnail responses on the fly.

On the new server, however, there’s an added twist: For certain types of static content — files that rarely change, such as images and videos — the nginx server serves them up directly. And if it can’t find the matching file it doesn’t forward the request to Apache. It just sends a failure code. So Apache — and therefore WordPress – never receives the request, so the image is never generated, and nginx sends the dreaded 404 Not Found response.

To fix that, I had to learn yet another bit of technical arcana: nginx web server configuration files. By itself, that’s not too bad. I’m familiar with the basic principles of a web server, and nginx is well-documented. But of course there was a complication… InMotion Hosting doesn’t edit these configuration files by hand. They have scripts to do that. So I had to learn the hosting service scripting system first. Fortunately, they actually wrote documentation explaining how to use their nginx scripts. I just had to change a couple of things and re-run the scripts and it worked.

That was enough to get the sites up and running. Since then, I’ve got full site backups running, so I think I’m ready to tell InMotion to deprovision the old server. It should be smooth running from here…until the next time.