Luminescent Dreams

nix-shell, the command that creates a subshell after evaluating any nix expression, has a lot of uses. I found it very useful in my devops work when I had multiple environments to administer, but had to use different tools for each. The shell provides excellent help in isolating my required tools to the environments involved.

A trick, though, lay in learning how to acquire those tools when the tools were not available in the nixos channel. I figured it out, and so here is the example for one of the environments I was administering. Note that I include both Linux and Darwin builds, because I wanted to offer the nix environment to my replacement at the company.

• Packer – 0.10.1
• Terraform – 0.7.4
• Ansible 2
• Python 2.7

We were deploying in Amazon AWS. I used Packer to build the custom images that we were deploying. Autoscaling works a lot better if it has a complete image that only has to be started (the Crops, i.e., the systems that can be replaced almost instantly and thus do get replaced regularly). I love Terrafrom because I was able to describe everything I was doing in AWS using the tool. Ansible is present for those systems that get reconfigured regularly (primarily the Cattle machines, things that can be rebuilt from just the devops scripts, but that I do not want to terminate). Python 2.7 is present to support Ansible, though it is sometimes convenient to have at hand.

Neither Packer nor Terraform were available in my Nix channel, so I had to build derivations for those. The process is non-obvious until it is done. Here are my scripts for them. At the time I wrote these scripts, I was running NixOS 16.03, however I still use the same scripts after having upgraded to NixOS 16.09.

nix-deps/packer.nix

{ pkgs ? import <nixpkgs> {},
  stdenv ? pkgs.stdenv }:

let
  # suggestion from @clever of #nixos
  package =
         if stdenv.system == "x86_64-linux" then "packer_0.10.1_linux_amd64.zip"
    else if stdenv.system == "x86_64-darwin" then "packer_0.10.1_darwin_amd64.zip"
    else abort "unsupported platform";
  checksum =
         if stdenv.system == "x86_64-linux" then "7d51fc5db19d02bbf32278a8116830fae33a3f9bd4440a58d23ad7c863e92e28"
    else if stdenv.system == "x86_64-darwin" then "fac621bf1fb43f0cbbe52481c8dfda2948895ad52e022e46f00bc75c07a4f181"
    else abort "unsupported platform";
in
stdenv.mkDerivation rec {
  name = "packer-${version}";
  version = "0.10.1";

  buildCommand = ''
  mkdir -p $out/bin
  unzip $src
  mv packer $out/bin/packer
  echo Installed packer to $out/bin/packer
  '';

  src = pkgs.fetchurl {
    url = "https://releases.hashicorp.com/packer/0.10.1/${package}";
    sha256 = checksum;
    name = package;
  };

  buildInputs = [ pkgs.unzip ];
}

nix-deps/terraform.nix

{ pkgs ? import <nixpkgs> {},
  stdenv ? pkgs.stdenv }:

let
  # suggestion from @clever of #nixos
  package =
         if stdenv.system == "x86_64-linux" then "terraform_0.7.4_linux_amd64.zip"
    else if stdenv.system == "x86_64-darwin" then "terraform_0.7.4_darwin_amd64.zip"
    else abort "unsupported platform";
  checksum =
         if stdenv.system == "x86_64-linux" then "8950ab77430d0ec04dc315f0d2d0433421221357b112d44aa33ed53cbf5838f6"
    else if stdenv.system == "x86_64-darwin" then "21c8ecc161628ecab88f45eba6b5ca1fbf3eb897e8bc951b0fbac4c0ad77fb04"
    else abort "unsupported platform";
in
stdenv.mkDerivation rec {
  name = "terraform-${version}";
  version = "0.7.4";

  buildCommand = ''
  mkdir -p $out/bin
  unzip $src
  mv terraform $out/bin/terraform
  echo Installed terraform to $out/bin/terraform
  '';

  src = pkgs.fetchurl {
    url = "https://releases.hashicorp.com/terraform/0.7.4/${package}";
    sha256 = checksum;
    name = package;
  };

  buildInputs = [ pkgs.unzip ];
}

The structure of each script is relatively straightforward.

• declare that pkgs and stdenv are both required, as well as how to get them if they are absent

• based on the OS, declare what package I want to download and the relevant checksum

• declare the name and version of the derivation

• create the custom build command

  In many cases, the default build commands works perfectly, but that only works for projects that have to be built with autoconfig or with Stack (and possibly some other languages). Both Terraform and Packer are binaries, and so it is necessary for me to specify the build for the derivation.

  In this case, the build is simply to unzip the downloaded package (specified in $src) and copy the executable into the destination (which has a root at $out). It is vital that the executable end up in the bin/ directory. I am not sure of the mandated directory structure of a derivation, but I know that derivations that did not include the bin/ directory would fail. I assume that they failed because there was no executable to add to the path.

• specify precisely how to get the source package. In this case, through the fetchurl tool.

• specify additional build inputs. These have to be somewhere in the nix namespace. pkgs.unzip refers to nixpkgs.unzip in the standard channel.

Both of the files above must go in a subdirectory. I named the subdirectory nix-deps/. Some subtle interaction will cause an infinite recursion if the two files are included in the root directory of your project.

With those present, it is time to build the nix-shell command:

shell.nix

let
  pkgs = import <nixpkgs> {};
  stdenv = pkgs.stdenv;
  terraform = import nix-deps/terraform.nix {};
  packer = import nix-deps/packer.nix {};

in stdenv.mkDerivation {
  name = "v2-devops";

  buildInputs = [ pkgs.ansible2
                  terraform
                  packer
                  pkgs.python
                  pkgs.python27Packages.alembic
                  pkgs.python27Packages.boto
                  pkgs.python27Packages.psycopg2
                  pkgs.awscli
                ];
}

The only difficult part here was for me to figure out how to import my Terraform and Packer derivations. I handle that with the import nix-deps/<package>.nix {} lines. The result of each import statement is a derivation, and so it is valid to include in buildInputs.

buildInputs again just lists the packages that must be included in this derivation. So, I included all of the packages that I use directly.

Thus, from the root directory of my devops folder, I can simply run nix-shell and have exactly the version of Terraform, Packer, Ansible, and Python that I want. This also means that I can have completely different versions for a different devops repository (I was actually administering three different clouds, all with different standards). And, possibly best of all, if I could convince my co-workers to use Nix (the tool, not the operating system), they would have had a trivial way to set up their development environments, also.