The value of declarative and reproducible environments for developers

My employer MaibornWolff is a consulting and custom software development company. This means we work with a diverse range of customers, and our developers frequently switch between vastly different projects. For me, this variability is even more pronounced. As an infrastructure platform expert, I am often called upon to help establish a solid base platform setup at the start of a project or to assist when a project encounters infrastructure issues. This requires me to frequently jump between projects and sometimes manage two projects simultaneously.

Each project comes with its own unique setup and tools that I need to deal with. Sometimes the tool stack is completely different, while other times, only the versions differ. For instance, different Kubernetes cluster versions necessitate compatible versions of kubectl. Or a different version of terraform might be required. These variations present their own challenges as versions should not be mixed (e.g., using an older Terraform version with a state file produced by a newer version).

However, I am never alone in a project; we always work in a team, bringing together various specializations. At MaibornWolff, we have a choice of three operating systems: macOS, Windows 11 or (since end of last year) Ubuntu Linux. Therefore, the tool stack used in a project needs to be usable on all three systems and should behave consistently across them.

When we consider all these factors, we see a multitude of dimensions: projects, tools, their versions, and operating systems. Managing these moving parts is crucial to ensure we have a consistent developer experience for developers across all projects.

Onboarding a colleague into a project can be quite challenging. We need to gather all the tools used and their exact versions. Then the colleague must install them on their laptop. If they have a different version of a tool for another project, they must find a way for those to coexist.

Making things more finicky are differences in base tools like sed or grep between macOS and Linux. These tools can have different options and arguments available, making it difficult to write scripts that work the same on all environments a challenge.

All of this means that onboarding new colleagues to a project can take a significant amount of time, and jumping between projects brings its own hassles.

If we had a way to define developer environments in a declarative and reproducible manner, it would alleviate or outright remove most of these pain points. Moreover, if these environments could be isolated from each other and behave consistently regardless of operating system, it would be ideal.

Virtual environments

The first solution that comes to mind is using some sort of virtual environments. Many programming languages already provide such tooling. For example Python venv or npm local install. However, this approach does not address how to install the correct version of the language or tooling itself.

For Python, tools like pyenv or uv solve this issue by allowing us to have multiple concurrent versions installed, which can be selected on a per-project basis. But this solution only works for a single language.

What about all the other tools we need for our daily work, especially as platform engineers, such as kubectl, terraform, and many others. These tools do not have built-in support for virtual environments.

What do we need

We need a generic tool that can provide virtual environments and handle multiple languages, tools, and their versions.

In addition to managing versions, we also want a declarative approach. We don't want to write scripts that run curl to download tool versions from GitHub releases or use apt to install packages. Instead, we want to define a list of tools and their versions and have an automated environment manager handle the rest.

Moreover, we also want reproducible environments. This means that if two developers start an environment based on the same definition on different systems on different days, they should get the exact same tools and versions. To make it more complex, this should work regardless of whether I run Linux or macOS, Intel/AMD chips or Apple Silicon ARM64. We can exclude Windows from this requirement, as the Windows Subsystem for Linux (WSL) is a well-established solution for getting a Linux environment on Windows.

When I think declarative, I think Nix

Many people, when reading about Linux and declarative and reproducible packages/environments, will have come across Nix and NixOS, which promises to provide just that. Nix is built on its own functional language that allows users to define, using a concept called derivation, how to build packages and environments. This language is very powerful, but can be challenging to understand and learn.

Nix is available for both Linux and macOS, So using Nix would cover the declarative and reproducible requirements. Selecting different versions of tools can be done with Nix (and is a core concept because Nix can manage different versions of dependencies/libraries for different tools) but this functionality is not easily exposed for end users.

Devbox to the rescue

Devbox is a tool that promises to provide exactly what we need: isolated, declarative and reproducible development environments. It uses Nix in the background but presents a simple interface and configuration to hide the complexity and make it easy to use. At its core, it has a configuration file called devbox.json that describes the packages and their exact versions to install for an environment. If I have such a file in my repository, I can just navigate into its directory, run devbox shell and Devbox will initialize my shell session to have all the tools described in its configuration in exactly the defined versions. Even if I have some of the tools already installed on my local system, the versions from Devbox/Nix take precedence.

A quick example to show how this works:

$ kubectl version # Run kubectl that is installed globally on my system
Client Version: v1.31.3 
$ which kubectl # Check which path is used
/usr/bin/kubectl
# Init and enter a devbox
$ devbox init
$ devbox add kubectl@1.30.0
$ devbox shell
Info: Ensuring packages are installed.
✓ Computed the Devbox environment.
Starting a devbox shell...
(devbox) $ kubectl version # Run kubectl in the devbox shell
Client Version: v1.30.0
(devbox) $ which kubectl # Check which path is used
/home/swoehrl/myproject/.devbox/nix/profile/default/bin/kubectl

We can see that I have a globally installed version of kubectl, but as soon as I enter a devbox shell, I access the version defined there.

Since Devbox only modifies my current shell session, everything reverts to my system defaults as soon as I close that session. I can also open several sessions with Devbox shells from different projects at the same time, and each of these sessions will have the tools with the versions defined for that project.

In a devbox.json I can also define so-called init_hooks to, for example, set environment variables for my shell session. I could, for instance, point the AWS CLI to a project-specific configuration file that already includes all the AWS accounts and login mechanisms used in that project. Thus I have no need to copy that to my global config.

Devbox is available for Linux and macOS, so if a colleague uses macOS and I use Linux, and we use the same devbox.json, we can be certain we are using the same versions of our tools.

In this way, Devbox provides declarative, reproducible and isolated development environments. At the same time, it is not completely isolated, meaning I still have access to my system. So if I have my editor configured the way I like it or have a fancy shell prompt, I still have access to them.

Task runner

Having the same tools and versions across all developers is unfortunately only half the battle. Especially in platform engineering, there are often a myriad of partially-automated tasks we want or need to execute. From performing an SSO login for a system to starting local Kubernetes clusters with K3d. We could, of course, just write them all down in a README file and let developers copy-paste all day. But there are better alternatives: Task runners.

The probably oldest and most well known task runner is make. Anyone who has ever compiled a C or C++ project (or even many modern Go projects) will have seen and used a Makefile. They are powerful, but the more complex ones can be hard to read and understand. Modern variants like just make it a bit easier but can still be hard to use.

I have found Task to be a modern and easy-to-use alternative and use it in most of my projects now. It is configured using a Taskfile.yml file. Because it uses YAML, configuration is very explicit and not hidden behind implicit syntax or special characters. It also has features concerning dependencies between tasks, arguments, environment variables and many others. It also allows to document each task directly alongside the command to run (and generates automatic help texts from it).

But one feature, that is not prominently mentioned in the docs, is very important for our requirement of having the same behavior between different systems: Task uses its own shell to run commands, not the system shell. That means it doesn't matter if the developer has a bash, zsh, fish or a limited ash on their system. If a task runs on my system (and uses only external commands that are part of my Devbox shell), I can be certain the task will also run the same on the systems of my colleagues.

CI and Local

By using a combination of Devbox and Taskfile, we can achieve declarative, reproducible, and isolated developer environments regardless of the OS or architecture. This combination can even extend the developer environment into our CI/CD pipelines.

Devbox has an integration with GitHub Actions, providing an easy way to get the same environment in the CI runner a we have on our local machines. This removes much of the hassle of writing and debugging CI pipeline steps. If it works locally in the Devbox, it will very likely also work in CI. While there will still be differences due to how CI runners are set up (and often limited in what they can do) or how they get credentials for external APIs, all the problems with tools and versions can be eliminated.

For CI systems that use containers as their abstraction (like Gitlab), Devbox offers a way to build a container image containing the same Devbox shell as defined locally. Note that this does not use the Nix way of building container images (from derivations) but a fairly standard Dockerfile.

Onboarding made easy

Having the combination of Devbox and Taskfile in a project makes onboarding a new colleague (which in the consulting business happens all the time) a breeze. A new developer could spend hours and days finding and installing all the tools and versions we use on their particular system. Or they could simply have Nix and Devbox installed. All tools and versions are defined in the manifest contained in the project's git repository, and Devbox takes care of installing them. Any repetitive tasks we do in the project are likewise defined in the Taskfile. There is no need to go through bash histories or poorly-maintained READMEs, it's all there in the code and can be used instantly.

This reduces the technical onboarding time to mere minutes and eliminates any differences between Linux and macOS systems (Windows users should use WSL to get a Linux environment).

Additionally, if a developer is working on several projects at the same time, they no longer have version conflicts or need to worry about calling specific versions. They just start a Devbox shell for the project they need, and get the correct environment set up in seconds.

Being more isolationist

I've used the word "isolated" when describing developer environments with Devbox. The truth is, it is a minimal form of isolation as it just overwrites paths. I can still use any tools I have installed on my host, and any tasks or commands I run also have access to my system. So it is not isolation from a security perspective, but from the point of view of different projects having their own set of versioned tools without interfering with each other.

However, depending on the project or use case, a deeper or stricter isolation might be beneficial or even required. The go-to solution for isolation on the same system is containers. There is an open specification on how to use containers for development environments, called devcontainer. The concept originated in VSCode, but the widely used JetBrains suite also has support for them. You can define a container image as a base and then use a JSON file to define additional layers and extensions. This allows the IDE to start a container and connect your session to it so that you have your normal IDE window but its terminal and any tasks you start (like a compiler or tests) are run inside the container.

Devcontainer manifests can be stored in your project's git repository, and new developers can quickly spin up a complete development environment and get started by just checking out the repo. They get the same declarative and reproducible properties we want, but with more isolation. This reduces setup time but can also cause frustration because the isolation means the tools a developer might have installed on their personal system are not available in the container (in contrast to Devbox). So no fancy personal shell prompt.

Dev environments as a service

Tools like GitHub Codespaces or Gitpod, and the self-hosted alternative DevPod take the devcontainer concept a step further by hosting the containers on remote systems in the cloud. They can be accessed either by a local IDE or even using an IDE running in a browser. This enables a very lightweight onboarding process where new hires or project members can use virtually any device they want. This allows for bring-your-own-device policies or the use of inexpensive provided devices that can be quickly replaced in case of a fault. It could be as lightweight as a Chromebook, as long as it has internet and a browser. A developer just requests a new dev environment from one of these services, and a few minutes later, they can start coding. If needed, they can get a beefy container with ample resources for whenever they need to do some heavy compiling or testing that their local device cannot handle.

Personally, I have started to use Codespaces to get quick, powerful, and disposable test environments. To show potential customers how a platform for their use case could look, I developed a scaled-down and simplified version of our MaibornWolff Smart Factory reference architecture to run as a demonstration. It uses several Kubernetes clusters via K3d and sets up up a complete platform from simulated machine to data visualization. Initially, I had scripts to provision and configure this setup on my local laptop. This had two main limitations: it would consume all my CPU and memory, and it was hard to reproduce on other people's machines. So I built a proper devcontainer for it (actually using Devbox for tool versioning) and added it to GitHub. Now I can just start a Codespace from the repo page on GitHub and have a running demo in a few minutes. Not only does this make my life easier, but it also enables colleagues who are not platform engineers and do not continuously run Kubernetes on their machines to perform the demo. This has the added benefit of removing myself as a bottleneck for such sales calls.

Conclusion

In my job, having declarative, reproducible and isolated developer environments is essential. The combination of Devbox and Taskfile provides a lightweight and easy-to-use solution to jump between projects and maintain consistent environments with my colleagues, even if they are running different operating systems. It makes my life much easier and eliminates a whole class of "works on my machine" problems and the tedious debugging that comes with them.

I highly recommend that anyone working on projects with multiple colleagues or frequently switching between projects adopts a method to get declarative and reproducible developer environments. Devbox is just one solution, and there are many others depending on your requirements and preferences. In the end, it doesn't matter which solution you choose, but I strongly believe a modern development process needs one.