Resilient containers, auto-mounting drives and WoL

Lately, I have been enjoying the consequences of some renovation work in the building where I live. One day, this meant that we were going to be without electricity for some hours.

This made me think of my home server. For the most part, I have everything automated in a way that when the server would shutdown for whatever reason, when it came back online every service would be running flawlessly and I could just relax.

Sadly, since I set up some services recently, that is not the case anymore. This was the perfect excuse to automate that too, and be able to go on holidays without worrying for it to become unreachable.

The problems

Docker networking

When restarting the docker containers, they are getting different internal IPs which is messing with the way they resolve each other.

To solve this, we will make use of Docker networking. In particular, we can use a bridge network that both containers will be added to, so they can resolve each other via hostname.

To do this, we are gonna use two commands:

We first create a bridge network with:

docker network create myNetwork

Then, we only have to connect both containers to that new network we just created with:

docker network connect myNetwork myContainerOne

and 

docker network connect myNetwork myContainerTwo

And there we have it! Our first problem is solved and now both containers can resolve each other via hostname.

Auto-mounting drives

My home server happens to be a NUC. This devices are super compact and silent, and they consume very low power both when resting and when working at full load.

However, since they're so compact, they don't have space for many SATA drives inside of the case. My NUC in particular only has 1 SATA drive bay, which I'm using for the server's OS.

I could get a huge SSD and store stuff inside of it, but I prefer to keep my concerns separated and use some external storage for the services I'm running inside the NUC. This way if one of the drives dies, I would only lose data for one of the services (which is hopefully backed up somewhere else), but the server would still function.

For this reason, I got a USB3 to SATA3 dock that I have connected on one of the USB3 ports of the NUC. In that dock, I have two 2.5" SATA drives that I use for the storage of my personal cloud and other services I have running.

What happened when the NUC was restarted was that the drives would not auto-mount to Linux, so the Docker containers that required them were not working either.

Tools for automounting

To automount drives in Linux (Ubuntu server in my case), we are gonna need the following tools:

In my case, I already had the drives mounted manually before doing all of this, so I had already decided where the mounted external drives would live and mounted them successfully.

First thing we need to do is run fdisk -l to know what name the drive has. This will be something like /dev/sdb, and in my case, with only one partition called /dev/sdb1. After getting the name, I run df to see where that device is mounted in the filesystem.

You might not need to do this, but I have two drives that are identical in size, so it's not easy to tell which one is used for what purpose.

Next, I'd use blkid /dev/sdb1 to find out the unique identifier (UUID) for this partition. The UUID will look similar to this: dced1bd6-d26d-44as-1234-3b84sg745de1d.

With all of this information, we are ready to add our drive and partition to the /etc/fstab file.

I added a line at the bottom that looks like this:

UUID=dced1bd6-d26d-44as-1234-3b84sg745de1d /home/myUser/myMountPoint ext4 defaults 0 0

At this point, I would recommend you to look deeper into /etc/fstab syntax since I'm not an expert. I chose ext4 since my drive is formatted like that, but it can be different in your case.

But what if the drive is not available at boot?

Sadly, after doing the previous commands and restarting my NUC, I realised that my drives somehow had not been detected by Ubuntu, and the server was failing to boot since it required those to be present when trying to mount. As that was not the case, the server booted in emergency mode without ssh access or anything else that physical access via keyboard connected to it.

If that happened while I didn't have physical access to it, the server was useless.

I looked for some parameter that would let the OS boot without those drives, and I found it. This is not your ideal situation since the services that rely on those drives would not be able to function correctly. But you can always try to troubleshoot that via SSH which is better than having to plug a keyboard and screen to your server.

The change that solved the issue was to add a nofail flag to the /etc/fstab line like this:

UUID=dced1bd6-d26d-44as-1234-3b84sg745de1d /home/webmaster/plex ext4 defaults,nofail 0 0

Wake on LAN

With the drives automounted and the containers properly connected to each other, there was only a small problem left to squash.

The server needs to boot after a power outage, otherwise none of what we did would matter!

There are two steps to this:

For the BIOS part, you will need to search in your BIOS parameters for it, or if you have a specific server model like in my case, you can search online for the way to activate Wake on LAN.

Once that is out of the way, we need a machine in the same network that can send a packet to our server to tell it to wake up.

Luckily for me, I have a Raspberry Pi in my LAN. The Pi boots directly when the power is back since it has no switch to turn on. And even better, it is running an OpenVPN server that allows me to join my LAN remotely from the outside.

With all of that in place, I only needed to install a package called etherwake via:

sudo apt-get install etherwake

When the package is installed, waking up our server from the other machine is as simple as running:

wakeonlan {serverMACAdress}

The end of the adventure

Well, that was a bit longer than I expected but I wanted to make sure most of it was written down. For my own sake too, hehe.

I think I am satisfied with the setup I have for now, but it's tough to say because I always end up trying to install something else and failing for some time at it :)

If you want to, you can reach me at @J2TheKay on Twitter :D