My experience of managing a cloud-hosted server with NixOS

As you may already heard from any staunch advocates which I’ve even said something about those lines myself, there are a couple of good reasons to use NixOS. But I’m fairly sure you’ve heard a fair share of the same bullet points at this point so let me give you my perspective from using it for some time instead.

What I like about it is the perspective of building NixOS systems just like how one would program applications. You create a text file containing a Nix program, you pass the program to the compiler (or in this case, the Nix build daemon), then you "compiled" an operating system.

Furthermore, the overall Nix ecosystem has some things helping to create a NixOS system such as nixpkgs which does not only have one of the largest package repositories (as of 2023-02-14) but also a wide set of options including…​

All of those can be made to create NixOS systems for a variety of purposes.

Prior to this post, I use NixOS as my desktop OS of choice for at least a year. Let me show you some examples of how I use it.

The point is…​ NixOS is very flexible despite that it already gives you a complete set of knobs to tinker with especially with mechanisms in place such as Nix modules ^[3], nixpkgs overrides, overlays, and flakes.

Unlike configuring from a traditional Linux system (e.g., Debian, Arch, Fedora), NixOS is nicer in terms of UX for configuring the system since you only have to monitor one location to consolidate the system configuration.

It’s not as large as other ecosystems such as from Docker with its vast selection of images and widespread tools such as Kubernetes. The amount of things within the containers ecosystem is just too huge which rallies even more support behind it.

This doesn’t mean the Nix ecosystem is something to be scoffed at. There’s more to what you can do with NixOS especially there’s an ecosystem beyond nixpkgs.

Here’s some of the tools I use related to managing the server.

I’m managing one instance in Hetzner Cloud. While it doesn’t have a NixOS image, it is easy to initiate a NixOS instance with the nixos-infect script which will…​ infect a Linux system into a NixOS one (without the spreading of NixOS disease though).

Thankfully, this can be automated since Hetzner also has an ecosystem of libraries and tools for more flexibility including a command-line interface for managing Hetzner Cloud deployments.

hcloud network create --name smn-lan --ip-range 172.16.0.0/12
hcloud server create --location hel1 --type cx21 --image ubuntu-22.04 \
    --network smn-lan \
    --user-data-from-file ./hosts/plover/files/hcloud/hcloud-user-data.yml \
    --ssh-key foodogsquared@foodogsquared.one \
    --name nixos-plover

Looking into NixOS as a server is pretty similar to how one would configure any Linux-based systems as a server.

One would have to statically configure the hardware of the server, mostly relating to networking setup. For this, I used systemd-networkd service that integrates well in NixOS.

The following systemd-networkd configuration is based from the generated systemd network unit from Ubuntu 22.04 image from Hetzner Cloud. It assumes that you have an global IPv4 and IPv6 address and a private IPv4 subnet from networks feature.

{ config, options, lib, pkgs, ... }:

let
  # IPv4 is already configured through DHCP so no need to configure it.
  wan = {
    IPv6.address = "2001:db8:4723:d1ad::";
    IPv6.gateway = "fe80::1";
  };

  lan = {
    IPv4.address = "10.32.54.1";
    IPv4.gateway = "10.0.0.1";
    IPv6.address = "fd45:5643:1ade::";
    IPv6.gateway = "fe80::1";
  };
in
{
  systemd.network.networks = {
    "10-wan" = with wan; {
      matchConfig.Name = [ "ens3" "enp0s3" ];

      # Setting up IPv6.
      address = [ "${IPv6.address}/64" ];
      gateway = [ IPv6.gateway ];

      networkConfig = {
        # IPv6 has to be manually configured.
        DHCP = "ipv4";

        LinkLocalAddressing = "ipv6";
        IPForward = true;

        # This is based from
        # https://docs.hetzner.com/dns-console/dns/general/recursive-name-servers/.
        DNS = [
          "2a01:4ff:ff00::add:2"
          "2a01:4ff:ff00::add:1"
        ];
      };
    };

    "20-lan" = with lan; {
      matchConfig.Name = [ "ens10" "enp0s10" ];

      address = [
        "${IPv4.address}/16"
        "${IPv6.address}/64"
      ];

      gateway = [
        IPv4.gateway
        IPv6.gateway
      ];
    };
  };
}

{ config, options, lib, pkgs, ... }:

let
  # IPv4 is already configured through DHCP so no need to configure it.
  wan = {
    IPv6.address = "2001:db8:4723:d1ad::";
    IPv6.gateway = "fe80::1";
  };

  lan = {
    IPv4.address = "10.32.54.1";
    IPv4.gateway = "10.0.0.1";
    IPv6.address = "fd45:5643:1ade::";
    IPv6.gateway = "fe80::1";
  };
in
{
  networking = {
    enableIPv6 = true;
    defaultGatewayIPv6 = wan.IPv6.gateway;

    interfaces = {
      enp0s3 = {
        useDHCP = true;
        ipv6.addresses = [
          {
            address = wan.IPv6.address;
            prefixLength = 64;
          }
        ];
      };

      enp0s10 = {
        ipv4.addresses = [
          {
            address = lan.IPv4.address;
            prefixLength = 16;
          }
        ];

        ipv6.addresses = [
          {
            address = lan.IPv6.address;
            prefixLength = 64;
          }
        ];
      };
    };
  };
}

In a typical server setup, one would harden the server for security. Otherwise, unwanted traffic including potential attackers might take advantage of it. Fortunately, nixpkgs already has a profile that already does most of that for you and all you have to do is to import it in your NixOS configuration like in the following code listing.

{ config, modulesPath, ... }:

{
  imports = [ "${modulesPath}/profiles/hardened.nix" ];
}

With the most important things done, you can now proceed with the service configuration. Here’s a non-exhaustive list of things that I’ve done for the server.

The resulting NixOS server configuration is all available in my NixOS configuration repo. At the end, I was able to deploy instances of some of the applications I want to host and manage myself (among other things).

Alongside some important services to keep them in check such as the firewall for basic measures, fail2ban to slow down intrusions from these publicly available services, and using a hardened kernel.

I really like how it turned out. Though I’m worried how I’m putting it all in one basket but I think it’s mostly fine for a smaller instance as long as it is hardened and take precautions properly.

{ config, lib, pkgs, ... }:

let
  inherit (config.networking) domain;
  domainZoneFile = ./config/coredns/${domain}.zone;
  domainZoneFile' = "/etc/coredns/zones/${domain}.zone";

  dnsDomainName = "ns1.${domain}";
  corednsServiceName = "coredns";
in
{
  # Generating a certificate for the DNS-over-TLS feature.
  security.acme.certs."${dnsDomainName}".postRun = ''
    systemctl restart ${corednsServiceName}.service
  '';

  services.coredns.config = ''
    tls://. {
      tls {$CREDENTIALS_DIRECTORY}/cert.pem {$CREDENTIALS_DIRECTORY}/key.pem {$CREDENTIALS_DIRECTORY}/fullchain.pem
      forward . /etc/resolv.conf
    }
  '';

  systemd.services.${corednsServiceName} = {
    requires = [ "acme-finished-${dnsDomainName}.target" ];
    preStart =
      let
        secretsPath = path: config.sops.secrets."plover/${path}".path;
        replaceSecretBin = "${lib.getBin pkgs.replace-secret}/bin/replace-secret";
      in
      lib.mkBefore ''
        install -Dm0644 ${domainZoneFile} ${domainZoneFile'}

        ${replaceSecretBin} '#mailboxSecurityKey#' '${secretsPath "dns/${domain}/mailbox-security-key"}' '${domainZoneFile'}'
        ${replaceSecretBin} '#mailboxSecurityKeyRecord#' '${secretsPath "dns/${domain}/mailbox-security-key-record"}' '${domainZoneFile'}'
      '';
    serviceConfig.LoadCredential =
      let
        certDirectory = config.security.acme.certs."${dnsDomainName}".directory;
      in
      [
        "cert.pem:${certDirectory}/cert.pem"
        "key.pem:${certDirectory}/key.pem"
        "fullchain.pem:${certDirectory}/fullchain.pem"
      ];
  };
}

{ config, lib, pkgs, ... }:

let
  giteaDatabaseUser = "gitea";
in
{
  systemd.services.gitea = {
    # Gitea service module will have to set up certain things first which is
    # why we have to go first.
    preStart =
      let
        psqlBin = "${lib.getBin config.services.postgresql.package}/bin/psql";
        giteaBin = "${lib.getBin config.services.gitea.package}/bin/gitea";
        giteaAdminUsername = lib.escapeShellArg "foodogsquared";
      in
      lib.mkMerge [
        (lib.mkBefore ''
          # Setting up the appropriate schema for PostgreSQL secure schema usage.
          ${psqlBin} -tAc "SELECT 1 FROM information_schema.schemata WHERE schema_name='${giteaDatabaseUser}';" \
            grep -q 1 || ${psqlBin} -tAc "CREATE SCHEMA IF NOT EXISTS AUTHORIZATION ${giteaDatabaseUser};"
        '')

        (lib.mkAfter ''
          # Setting up the administrator account automated.
          ${giteaBin} admin user list --admin | grep -q ${giteaAdminUsername} \
            || ${giteaBin} admin user create \
              --username ${giteaAdminUsername} --email foodogsquared@${config.networking.domain} \
              --random-password --random-password-length 76 --admin
        '')
      ];
  };
}

As for the workflow, I can push my changes easily with deploy-rs (or even just nixos-rebuild as I’ve learned it from this post). The aspect that I have trouble so far is debugging the system properly which is something I don’t know how to do…​ properly.

Before I completely decided with NixOS, I shortly ran a server with Debian which I think its a good choice for a first timer especially with its welcoming user documentations which the Nix ecosystem lacks. A lot of the documentation usually found from Nix assumes you’re familiar with the overall of a Linux system which is not a bad thing.

Once I got the ropes of managing a Debian system (for at least a week), I jumped into managing one with NixOS partly because of impatience coupled with the problems of traditional deploying and managing a system like that. While I find a lot of resources using Debian and find a solution for deploying traditional systems like Ansible or Terraform ^[6], I eventually find it easier to map concepts from there to its NixOS counterparts. Once the NixOS system has been setup, there’s not much of a difference except it is deployed by my computer somewhere.

The most worrying part of the setup is secrets management. It is done with sops and sops-nix then manually putting the private key in its supposed place with good ol' scp. I don’t think it’s efficient but it is what it is. I’m fairly sure there’s a better way to provision the server alongside the secrets but solving that problem seems like diminishing returns.

The maintenance process also considers budget especially in the long run so let’s be a good I.T. department for ourselves and enumerate them in a table.

A total of €14.2 or ₱712 in my local currency which is nice in my personal budget of under ₱1000 (€17.3).

While creating a NixOS system has been nice, there are some things you have to keep in mind.

Overall, the process of managing servers with NixOS is a great experience so far especially with how much controls it gives you from it. It sparks joy the same way how I build a NixOS system for my desktop driver. However, the investment required just for this solution is quite high which is a shame considering how simple it really is once you’ve gained some familiarity with Nix compared to other solutions such as containers. This high investment exists due to how different ^[8] it is compared to the rest of the ecosystem.

While it is not the de-facto experience for managing a Linux server especially for a first-timer, it can be a gateway for managing servers in a fun way sort-of-deal. There are still some things I wonder how some things work such as scaling which is Kubernetes is good at. It isn’t completely needed for me right now but it is something to ponder when considering expansion for my server fleet.

All of these are new things to me but I’m expanding what I can manage. Other plans include managing a Borg server from a BuyVM instance especially with its block storage slabs which seems to be quite flexible compared to Hetzner Storage Boxes.

Among other things, I might add a more convenient way of deploying servers from cloud providers with Terraform. Xe Iaso already has a post detailing this kind of setup but I’ll have to adjust it to deploy to Hetzner Cloud if possible. While the additional setup is more of a nuisance since it is already deployed (for the most), it is still something that can be worth in the future.