Setting up a Custom Nix Channel

Savanni D'Gerinel 13 Sep, 2019

Happy Friday the 13th, everyone!

This week has gone extremely well, with me finishing a project for work that will actcually help save lives. Lots of lives. And I for the life of me cannot actually think of any way in which the application can be used to create harm.

At the same time, I’m tuning my tools, and this week, that means my shell.

I’ve been working on switching my shell environment away from being a nix-shell all the time and back to being something more like the config.nix based environment that the Nix maintainers envisioned.

In the midst of this, I decided that it would also be good for me to grab all of the derivations that I’ve been duplicating from one project repository to another, and drop them into a custom nix channel.

I describe all of my development environments in a shell.nix file that I keep with the application that I’m developing. Part of my discpline also involves standardizing on a particular version of my tools, such as Rust-1.33 instead of whatever version of Rust is currently available in the public NixPkgs repository. Until this week, that meant keeping a Rust derivation file in the project repository. That inevitably leadf to me duplicating gthat derivation file (and Node, and other things), across a variety of repositories.

This was probably optional with my work repositories, but the repositories I build for Luminescent Dreams ship with the shell.nix and default.nix, and so have to also ship with the other derivations.

My own custom channel became the obvious solution, but I did not think of it until late last week.

So, while the format is actually very straightforward, I did not find documentation on how to set up the channel repository and for a while did not think of the simplest, most obvious way of finding out the format. So, here I document it all.

You will start out by building a very simple derivation set in a default.nix file:

{ system ? builtins.currentSystem }:
let
  pkgs = import <nixpkgs> { inherit system; };
  self = in self
in rec {
    rust_1_33_0 = pkgs.callPackage ./pkgs/dev/rust-1.33 {
      inherit pkgs;
    };

    nodejs_9_10_0 = pkgs.callPackage ./pkgs/dev/node9.nix {
      inherit pkgs;
    };
    ...
  }

Every element in this set; in this case rust_1_33_0, nodejs_9_10_0, and so forth; will be available in the channel.

Next, make sure you actually define he derivations described here. Errors won’t be detected until somebody who subscribes to the channel actually tries to install something. Obvious errors, like failing to pass a parameter to a derivation, will completely break the channel and are thus easy to find.

Finally, move one directory up from your default.nix, and tar the entire directory:

> ls -l

total <whatever>
drwxr-xr-x 4 savanni staff  128 Sep 12 11:02 nixpkgs

I actually do not know whether the name of the directory matters, but I mimicked the naming convention that I found in the main nixpkgs channels. Tar up the nixpkgs directory, with bz2 compression, to the file nixexprs.tar.bz2:

> tar -cjf nixexprs.tar.bz2 nixpkgs

Finally, upload nixpkgs.tar.bz2 to a public location. This file alone is all it takes to make a channel.

To use the channel, add it to your nix channel list, but be sure to include the exact url. For instance, to get my Luminescent Dreams channel, this is your command:

nix-channel --add http://luminescent-dreams-apps.s3-website-us-west-2.amazonaws.com/nixexprs.tar.bz2 luminescent-dreams

My Channel

I am now running my own custom channel for Luminescent Dreams, in which I distribute both the exact versions of the tools that I use and the software that I develop. As of today, that includes:

  • rust-1.33.0
  • nodejs-9.10.0
  • nodejs-10.15.3
  • ansible-2.7.4
  • certbot-0.19.0
  • packer-1.1.3
  • terraform-0.12.2
  • vault-1.0.1
  • fitnesstrax-0.0.1
  • fitnesstrax-0.0.2

Soon I’ll add orizentic, palimpsest, and digikam-export to the channel.

Typescript Setup

Savanni D'Gerinel 11 Jun, 2019

I recently started a typescript project from scratch. With no pre-existing setup, and very little prior typescript experience, I had to set up the entire development environment to support it. In doing so, I found that there was really no coherent set of instructions for bootstrapping a Typescript project from base principles. Once I did all of the research, I decided it would be important for me to document not only my final environment, but how the various parts work together in a way that allows a bit of modular picking-and-choosing.

This is a straightforward React + Redux project with Jest tests, and I wanted a fairly complete development environment:

  • Build to a single .js file, with companion .html and .css files
  • Monitor and automatically rebuild all files in the project
  • Jest tests
  • Continuous linting with Ale for vim
  • Automatic formatting with Prettier

The final project is an environment dashboard for the Boston Cloud City Team. It is my first Typescript project, it allows me to test out some CSS layout techniques, and it gives us continuous ambient information around Alewife Station in Boston.

Base Setup

Because it is popular, I decided to go with webpack for the bundling tool. I’ll start out with just the webpack setup.

Dev Dependencies:

  • webpack: ^4.30.0
  • webpack-cli: ^3.3.1

package.json

{
  "name": "cc-boston-dashboard",
  "version": "1.0.0",
  "description": "A dashboard display for the Boston Cloud City team",
  "scripts": {
    "start": "webpack-dev-server",
    "build": "webpack",
  },
  "author": "Savanni D'Gerinel <savanni@luminescent-dreams.com>",
  "license": "BSD-3-Clause",
  "devDependencies": {
    "webpack": "^4.30.0",
    "webpack-cli": "^3.3.1",
    "webpack-dev-server": "^3.3.1"
  },
  "dependencies": {
  }
}

webpack.config.js

const path = require("path")

module.exports = {
  mode: "development",
  entry: "./src/main.tsx",
  devtool: "inline-source-map",
  devServer: {
    contentBase: path.join(__dirname, "dist"),
    compress: true,
    port: 9000,
  },
  module: { },
  resolve: {
    extensions: [".tsx", ".ts", ".js"],
  },
  output: {
    filename: "bundle.js",
    path: path.resolve(__dirname, "dist"),
  },
}

This is the most basic npm and webpack setup that I can think of. By itself, it does basically nothing since there are no modules defined for the build process, but it does form the template for doing development.

I include webpack-dev-server, but that is an optional component. It is nice to have because your browser will automatically reload the application after every file change. If you decide to leave that out, remove the devServer stanza from webpack.config.js.

Typescript

In adding Typescript, this is where we add in the programming language and can start writing and building code.

Dev Dependencies:

  • ts-loader: ^5.4.4
  • typescript: ^3.4.5

First, let’s set up the typescript rules.

tsconfig.json

{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "lib": ["es2015", "dom"],
    "jsx": "react",
    "strict": true,
    "esModuleInterop": true,
    "sourceMap": true
  }
}

Most of the options I don’t understand, but I know that they are necessary. For now, use these as a starting point and modify them to see what effects you get. There is extensive documentation of all of the compiler options available.

However, there are three that I understand well enough to describe:

  • jsx – this option must be set to “react” or to “react-native” for React applications. The documentation describes all of the settings.
  • strict – when set to true, Typescript will perform the strictest type checking available. I highly advise this for every project. Typescipt and Javascript code can coexist in the same repository, so if you are migrating a Javascript project into Typescript, migrate one file at a time but be prepared to migrate the entirity of that file at once.
  • sourceMap – this option includes source maps in the final build product.

Now we need to tell Webpack how to understand typescript files. This takes the form of a rule in the module section of webpack.config.js.

webpack.config.js

module.exports = {
  ...
  module: {
    rules: [
      {
        test: path => path.endsWith(".ts") || path.endsWith(".tsx"),
        loader: "ts-loader",
        options: {
          onlyCompileBundledFiles: true,
        },
        exclude: /node_modules/,
      },
    ]
  }
  ...
}

This rule will be applied to every file with the extensions .ts and .tsx in the source directory. Each of those files, when processed, will be passed into ts-loader, which is a compatibility layer between typescript and webpack.

options: { onlyCompileBundledFiles: true } is an important option to include to ensure that ts-loader only works on those files that will be part of the resulting package. When excluded, ts-loader will attempt to process *.test.ts and *.test.tsx files. This does not go well as test files in Javascript tend treat many of the testing idioms as part of the environment, not as libraries to be included. So, when ts-loader attempts to process a file intended for Jest, it will find a lot of errors of this form:

ERROR in /Users/savanni/src/dashboard/client/src/clients/mbta.test.ts
[tsl] ERROR in /Users/savanni/src/dashboard/client/src/clients/mbta.test.ts(6,1)
      TS2582: Cannot find name 'describe'. Do you need to install type definitions for a test runner? Try `npm i @types/jest` or `npm i @types/mocha`.

Jest, Enzyme, and Fetch Mocks

At this stage, I assume that we have some code to test. I have gotten most familiar with Jest over the last few months, and so it was natural for me to pick that up again.

Dev Dependencies:

  • @types/jest: ^24.0.11
  • jest: ^24.7.1
  • ts-jest: ^24.0.2

At this point, I add a new section, jest, to the package.json file to handle configuration.

package.json

  "jest": {
    "transform": {
      ".(ts|tsx)": "ts-jest"
    },
    "testPathIgnorePatterns": [
      "/node_modules/",
      "/lib/"
    ],
    "testRegex": "(/src/.*\\.(test|spec))\\.(ts|tsx)$",
    "moduleFileExtensions": [
      "ts",
      "tsx",
      "js",
      "json"
    ],
  },

Jest uses its own loader, ts-jest. It serves much the same function as ts-loader, but it provides definitions such as describe, it, test, and expect as part of the global environment in test files.

Once Jest is working, I want to add Enzyme into the mix. Enzyme is my favorite testing tool, as it has made it easy for me to test my React components. React components can be rendered based solely on their parameter inputs, or with the extra aid of mocked Redux and Apollo providers. This helps me ensure that the component responds correctly to input parameters, possibly through tricky logic.

Dev Dependencies:

  • @types/enzyme-adapter-react-16: ^1.0.5
  • @types/enzyme: ^3.9.1
  • enzyme-adapter-react-16: ^1.12.1
  • enzyme: ^3.9.0

Enzyme needs to be configured before each test. I can do with a simple beforeEach block in the test file.

package.json

    "setupFilesAfterEnv": [
      "./setupBeforeTest.ts"
    ]

component.test.ts

import { configure } from "enzyme"
import Adapter from "enzyme-adapter-react-16"

beforeEach(() => configure({ adapter: new Adapter() }))

However, I need to write this for every file which does component testing. Jest provides setupFilesAfterEnv to specify chunks of code that should be run before each test in the suite, allowing us to centralize the Enzyme configuration code for the entire project.

setupBeforeTest.ts

import { configure } from "enzyme"
import Adapter from "enzyme-adapter-react-16"

configure({ adapter: new Adapter() })

There is a lot to be said about how to make mocking an API effective. Done poorly, you get a client library that cannot actually talk to the server. Done well, you have a reliable set of tests that can isolate a problem to a mismatch in the wire protocol.

In most applications, I need to write, and test, a client library. Even if this is a library only for communication with my own service, I generally find it much easier to test without actually invoking the server, repeatedly using example responses from the server as controlled inputs for my tests. However, mocking the fetch command requires an extra library, jest-fetch-mock.

Dev Dependencies:

  • jest-fetch-mock: ^2.1.2

Once installed, the fetch mock needs to be enabled for each file. Like with Enzyme, we could provide the initialization code in every file, but Jest provides an option, setupFiles, which executes code at the beginning of each file. Again, we can centralize this configuration.

package.json

    "setupFiles": [
      "./setupOnFileLoad.ts"
    ],

setupOnFileLoad.ts

import { GlobalWithFetchMock } from "jest-fetch-mock"

const customGlobal: GlobalWithFetchMock = global as GlobalWithFetchMock
customGlobal.fetch = require("jest-fetch-mock")
customGlobal.fetchMock = customGlobal.fetch

With this in place, fetch calls, when called from a jest process, will be replaced with the mock.

Compiling in HTML and CSS

This part is not specific to typescript, but is useful for managing static assets. These additional loaders allow me to include .css and .html files alongside the typescript source files.

Dev Dependencies:

  • css-loader: ^2.1.1
  • file-loader: ^3.0.1
  • style-loader: ^0.23.1

webpack.config.js

  module: {
    rules: [
      { test: /\.css$/, loaders: ["style-loader", "css-loader"] },
      {
        test: /\.html$/,
        loader: "file-loader",
        options: {
          name: "[name].[ext]",
        },
      },
    ],
  },

With these rules in place, webpack will process require instructions such as the following.

const index = require("./index.html")
const styles = require("./styles.css")

HTML files will be copied into the distribution directory, where CSS files will be included in the target javascript bundle. This also means that the HTML file does not need to link to the CSS files.

Continuous Linting

Linting is very nearly free. tsserver, which gets installed in the typescript package, can be run in the background to continuously typecheck and lint the code. Ale for vim will automatically start tsserver if it can detect its presence. You just need a couple of small configuration options in your .vimrc.

.vimrc

let g:ale_fixers = {
\   '*': ['remove_trailing_lines', 'trim_whitespace'],
\   'typescript': ['prettier', 'eslint'],
\}
let g:ale_fix_on_save = 1
let g:ale_set_loclist = 0

All At Once

At this point, it’s time to put everything together. Whether you jumped directly here or read through the explanations, here is where the whole thing goes together.

There are six files to drop into your environment. Fortunately, you probably only need to modify package.json and webpack.config.js, while the others are just boilerplate.

package.json

{
  "name": "cc-boston-dashboard",
  "version": "1.0.0",
  "description": "A dashboard display for the Boston Cloud City team",
  "scripts": {
    "start": "webpack-dev-server",
    "build": "webpack",
  },
  "author": "Savanni D'Gerinel <savanni@luminescent-dreams.com>",
  "license": "BSD-3-Clause",
  "jest": {
    "transform": {
      ".(ts|tsx)": "ts-jest"
    },
    "testPathIgnorePatterns": [
      "/node_modules/",
      "/lib/"
    ],
    "testRegex": "(/src/.*\\.(test|spec))\\.(ts|tsx)$",
    "moduleFileExtensions": [
      "ts",
      "tsx",
      "js",
      "json"
    ],
    "setupFilesAfterEnv": [
      "./setupBeforeTest.ts"
    ]
    "setupFiles": [
      "./setupOnFileLoad.ts"
    ],
  },
  "devDependencies": {
    "@types/enzyme": "^3.9.1",
    "@types/enzyme-adapter-react-16": "^1.0.5",
    "@types/jest": "^24.0.11",
    "css-loader": "^2.1.1",
    "enzyme": "^3.9.0",
    "enzyme-adapter-react-16": "^1.12.1",
    "file-loader": "^3.0.1",
    "jest": "^24.7.1",
    "jest-fetch-mock": "^2.1.2",
    "style-loader": "^0.23.1",
    "ts-jest": "^24.0.2",
    "ts-loader": "^5.4.4",
    "typescript": "^3.4.5",
    "webpack": "^4.30.0",
    "webpack-cli": "^3.3.1",
    "webpack-dev-server": "^3.3.1"
  },
  "dependencies": {
  }
}

webpack.config.js

const path = require("path")

module.exports = {
  mode: "development",
  entry: "./src/main.tsx",
  devtool: "inline-source-map",
  devServer: {
    contentBase: path.join(__dirname, "dist"),
    compress: true,
    port: 9000,
  },
  module: {
    rules: [
      {
        test: path => path.endsWith(".ts") || path.endsWith(".tsx"),
        loader: "ts-loader",
        options: {
          onlyCompileBundledFiles: true,
        },
        exclude: /node_modules/,
      },
      { test: /\.css$/, loaders: ["style-loader", "css-loader"] },
      {
        test: /\.html$/,
        loader: "file-loader",
        options: {
          name: "[name].[ext]",
        },
      },
    ]
  },
  resolve: {
    extensions: [".tsx", ".ts", ".js"],
  },
  output: {
    filename: "bundle.js",
    path: path.resolve(__dirname, "dist"),
  },
}

tsconfig.json

{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "lib": ["es2015", "dom"],
    "jsx": "react",
    "strict": true,
    "esModuleInterop": true,
    "sourceMap": true
  }
}

component.test.ts

import { configure } from "enzyme"
import Adapter from "enzyme-adapter-react-16"

beforeEach(() => configure({ adapter: new Adapter() }))

setupBeforeTest.ts

import { configure } from "enzyme"
import Adapter from "enzyme-adapter-react-16"

configure({ adapter: new Adapter() })

setupOnFileLoad.ts

import { GlobalWithFetchMock } from "jest-fetch-mock"

const customGlobal: GlobalWithFetchMock = global as GlobalWithFetchMock
customGlobal.fetch = require("jest-fetch-mock")
customGlobal.fetchMock = customGlobal.fetch

At the end of this journey, you should have a working typescript development environment. You can simply drop the above files into the root of your development environment and start coding!

Good luck and go out to build some applications!

Winters End, 2019

Savanni D'Gerinel 24 Mar, 2019

As the last festiges of winter come to an end, here is a batch of photos that I took a month ago. This was in Middlesex Fells in Boston, a short distance from where I live, right after a snowstorm that drop a foot or so of snow for seventeen hours.

I had been starting to lose my mind from the cold and being trapped inside. Finally, at my partner’s advice, I got completely bundled up to take camera and tripod out for pictures. The results here are pretty fantastic, and my mental health improved immediately.

Sunset

Savanni D'Gerinel 10 Jan, 2019
Brilliant Sunset, 2019-01

We had a spectacular sunset in the Boston area last night.

Dogwood Challenge, Week 1, Self-Portait

Savanni D'Gerinel 6 Jan, 2019

Take a picture that tells us who you are, without actually showing your face.

_DSC2092.jpg

For my self-portrait, I could think of nothing more appropriate than my work desk. Not precisely as I have it day-to-day, but not greatly rearranged, either. The distinction between who I am and what I do is pretty minimal, for better or for worse.

  • trans
  • code on the screen
  • wacom tablet nearby
  • camera parts
  • home-made crochet hand warmers in bi pride colors
  • I Voted!
  • mug of tea in the background

Dreamer, Shaper, Seeker, Maker