Intro

IPv6 only

The whole setup will be IPv6 only. I will not configure any IPv4 because I don’t need legacy IP support. If you have legacy workloads that require IPv4 connectivity, you will have to think about adding a SIIT + NAT64 setup with something like JOOL or building everything dual stack. Both ways complicate the setup in different ways. You decide how you want to solve that issue. Maybe I’ll add a SIIT-DC setup later just for fun. It’s probably the easiest to get rid of your IPv4-only software anyway. If you want to create an IPv4 only setup you should be able to follow this guide and simply replace every mentioned IPv6 thing with your IPv4 equivalent. But honestly, why would you do that? It’s 2025, IPv6 is now over 30 years old. There should be no reason to not support it as first class citizen. Do this first and think about legacy support later when you actually need it.

There will be an exception for the Kubernetes Ingress Controller. It needs to be reachable by public IPv4 addresses so that legacy users can still reach my public services. They are punished enough by their ISPs, let’s not add more to that.

Public addresses

I will use public IPv6 addresses for many components. That means services are available to the public internet. Firewall rules will need to make sure that only our components talk to non-public endpoints. This will be done with nftables firewall rules that block outside traffic to our Pod and Service networks.

I’m still undecided if I should use ULA addresses for the Pod and Service networks or Global Unicast. There are many ways to set this up and none of them are wrong per se. Let’s see how thing evolve when we get to the CNI plugin setup.

Combined control plane and worker nodes

In a production setup you would usually separate kubernetes worker nodes and control plane nodes. This is better for security and it also gives you the ability to easier scale them independent from each other. Because I only have 3 dedicated nodes at the moment and I want to simplify the setup I will combine these roles in this guide. This doesn’t stop you from adding more nodes with dedicated roles in the future. In fact once this setup works I will migrate it from 3 VMs to 4 hardware nodes with one being only a worker node.

A control plane node would only host etcd, apiserver, scheduler, controller manager, addon manager and proxy services. A worker node would only host a container runtime, kubelet and proxy services. Depending on your network plugin it would probably also be installed on both node roles. In NixOS this is handled by the services.kubernetes.roles option. It overwrites the enabled option of kubernetes components. For that reason we will not use that and take fine grained control over our services.

Basic setup

This guide assumes you’re running NixOS or you have Nix installed on your machine. If you don’t want to do that, the Makefile will not be able to deploy the nodes for you. You will have to do that by connecting to the node and run the nixos-rebuild commands locally.

Git

We will put everything we do in a single git repository. So let’s set this up now.

Let’s start with a standard .gitignore file. The most important thing is the direnv part that keeps us from accidentally including some nix outputs. I will use PyCharm and Vim. Both of them will generate some files that you don’t want, so I’ll exclude these too. And finally we will exclude all OS level stuff. Adapt the file to your own needs. The gitignore.io API has templates for a lot of tools.

curl -sL "gitignore.io/api/linux,windows,macos,direnv,pycharm+all,vim" > ./.gitignore

You can put a .envrc file with the content use flake in the Git repos root dir, to load the flakes devshell automatically. You need to install direnv to make that work. On NixOS, you can use a faster implementation of direnv like nix-direnv.

If you don’t want to do that you can activate the devshell of the flake by running nix develop manually.

We will have a pki directory which consists of our certificate autorities and certificates. In there we need another gitignore file, so add a ./pki/.gitignore like this:

# don't include unencrypted certificates
*/*.pem

This will prevent you from accidentally making your precious certificates public.

We will also have a services directory which holds our Nix configuration of the services we will setup. This makes it easy to include them only on the nodes where we need them.

The next direcotry is secrets. It will hold our age encrypted secrets. So no worries this should be plenty secure to put them into git. At least for now. &xF609;

Lastly we will have a deployments directory with our Kubernetes manifests. I will probably come up with a better way of handling those at some point but for now they live there.

Nix flake

Let’s put a simple flake in there too.

{
  inputs = {
    nixpkgs.url = github:NixOS/nixpkgs/nixos-24.11;
    agenix = {
      url = github:ryantm/agenix;
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
  outputs = { self, nixpkgs, agenix }:
  let
    system = "x86_64-linux";
    pkgs = import nixpkgs { inherit system; };
  in
  {
    formatter.${system} = pkgs.nixpkgs-fmt;
    devShells.${system}.default = pkgs.mkShell {
      packages = with pkgs; [
        agenix.packages.${system}.default
        age # secrets debugging
        gnumake # automation
      ];
    };
  }
}

Different architectures

If the architectures of your dev machine and the architecture of your nodes don’t match, you can utilize flake-utils like this:

{
  inputs = {
    nixpkgs.url = github:NixOS/nixpkgs/nixos-24.11;
    agenix = {
      url = github:ryantm/agenix;
      inputs.nixpkgs.follows = "nixpkgs";
    };
    flake-utils.url = github:numtide/flake-utils;
  };
  outputs = { self, nixpkgs, agenix, flake-utils }: flake-utils.lib.eachDefaultSystem (system:
  let
    pkgs = import nixpkgs { inherit system; };
  in
  {
    formatter = pkgs.nixpkgs-fmt;
    devShells.default = pkgs.mkShell {
      packages = with pkgs; [
        agenix.packages.default
        age # secrets debugging
        gnumake # automation
      ];
    };
  })
}

This will generate the flakes dev shell in your local architecture and you can use the nodes system further down for the architecture of your nodes like this:

nixosConfigurations = let
    system = "x86_64-linux";
    pkgs = import nixpkgs { inherit system; };
in {
    "node-01".nixpkgs.lib.nixosSystem = { inherit system; };
    "node-02".nixpkgs.lib.nixosSystem = { inherit system; };
    "node-03".nixpkgs.lib.nixosSystem = { inherit system; };
};

You also need to redefine pkgs to use the correct architecture in the modules we will include. Keep this in mind for the next chapter when we setup the nodes.

Makefile

Let’s also add a helping Makefile that doesn’t do much for now:

.DEFAULT_GOAL := help

.PHONY: all
all: update-flake ## Update flake inputs

.PHONY: update-flake
update-flake: ## Update nix flake
    nix flake update

.PHONY: help
help: ## Display this help
    @grep -h -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

By default, it prints a nice help message.

Outro

This is the final layout so far:

.
├── Makefile
├── README.md
├── deployments/
├── flake.lock
├── flake.nix
├── pki
│   └── Makefile
├── secrets/
│   └── secrets.nix
└── services/

We will add more stuff later on.

Good bye for now!