Luminescent Dreams

Nix Development Environments

nix-shell, the command that creates a subshell after evaluating any nix expression, has a lot of uses. I found it very useful in my devops work when I had multiple environments to administer, but had to use different tools for each. The shell provides excellent help in isolating my required tools to the environments involved.

A trick, though, lay in learning how to acquire those tools when the tools were not available in the nixos channel. I figured it out, and so here is the example for one of the environments I was administering. Note that I include both Linux and Darwin builds, because I wanted to offer the nix environment to my replacement at the company.

  • Packer – 0.10.1
  • Terraform – 0.7.4
  • Ansible 2
  • Python 2.7

We were deploying in Amazon AWS. I used Packer to build the custom images that we were deploying. Autoscaling works a lot better if it has a complete image that only has to be started (the Crops, i.e., the systems that can be replaced almost instantly and thus do get replaced regularly). I love Terrafrom because I was able to describe everything I was doing in AWS using the tool. Ansible is present for those systems that get reconfigured regularly (primarily the Cattle machines, things that can be rebuilt from just the devops scripts, but that I do not want to terminate). Python 2.7 is present to support Ansible, though it is sometimes convenient to have at hand.

Neither Packer nor Terraform were available in my Nix channel, so I had to build derivations for those. The process is non-obvious until it is done. Here are my scripts for them. At the time I wrote these scripts, I was running NixOS 16.03, however I still use the same scripts after having upgraded to NixOS 16.09.

nix-deps/packer.nix

{ pkgs ? import <nixpkgs> {},
  stdenv ? pkgs.stdenv }:

let
  # suggestion from @clever of #nixos
  package =
         if stdenv.system == "x86_64-linux" then "packer_0.10.1_linux_amd64.zip"
    else if stdenv.system == "x86_64-darwin" then "packer_0.10.1_darwin_amd64.zip"
    else abort "unsupported platform";
  checksum =
         if stdenv.system == "x86_64-linux" then "7d51fc5db19d02bbf32278a8116830fae33a3f9bd4440a58d23ad7c863e92e28"
    else if stdenv.system == "x86_64-darwin" then "fac621bf1fb43f0cbbe52481c8dfda2948895ad52e022e46f00bc75c07a4f181"
    else abort "unsupported platform";
in
stdenv.mkDerivation rec {
  name = "packer-${version}";
  version = "0.10.1";

  buildCommand = ''
  mkdir -p $out/bin
  unzip $src
  mv packer $out/bin/packer
  echo Installed packer to $out/bin/packer
  '';

  src = pkgs.fetchurl {
    url = "https://releases.hashicorp.com/packer/0.10.1/${package}";
    sha256 = checksum;
    name = package;
  };

  buildInputs = [ pkgs.unzip ];
}

nix-deps/terraform.nix

{ pkgs ? import <nixpkgs> {},
  stdenv ? pkgs.stdenv }:

let
  # suggestion from @clever of #nixos
  package =
         if stdenv.system == "x86_64-linux" then "terraform_0.7.4_linux_amd64.zip"
    else if stdenv.system == "x86_64-darwin" then "terraform_0.7.4_darwin_amd64.zip"
    else abort "unsupported platform";
  checksum =
         if stdenv.system == "x86_64-linux" then "8950ab77430d0ec04dc315f0d2d0433421221357b112d44aa33ed53cbf5838f6"
    else if stdenv.system == "x86_64-darwin" then "21c8ecc161628ecab88f45eba6b5ca1fbf3eb897e8bc951b0fbac4c0ad77fb04"
    else abort "unsupported platform";
in
stdenv.mkDerivation rec {
  name = "terraform-${version}";
  version = "0.7.4";

  buildCommand = ''
  mkdir -p $out/bin
  unzip $src
  mv terraform $out/bin/terraform
  echo Installed terraform to $out/bin/terraform
  '';

  src = pkgs.fetchurl {
    url = "https://releases.hashicorp.com/terraform/0.7.4/${package}";
    sha256 = checksum;
    name = package;
  };

  buildInputs = [ pkgs.unzip ];
}

The structure of each script is relatively straightforward.

  • declare that pkgs and stdenv are both required, as well as how to get them if they are absent

  • based on the OS, declare what package I want to download and the relevant checksum

  • declare the name and version of the derivation

  • create the custom build command

    In many cases, the default build commands works perfectly, but that only works for projects that have to be built with autoconfig or with Stack (and possibly some other languages). Both Terraform and Packer are binaries, and so it is necessary for me to specify the build for the derivation.

    In this case, the build is simply to unzip the downloaded package (specified in $src) and copy the executable into the destination (which has a root at $out). It is vital that the executable end up in the bin/ directory. I am not sure of the mandated directory structure of a derivation, but I know that derivations that did not include the bin/ directory would fail. I assume that they failed because there was no executable to add to the path.

  • specify precisely how to get the source package. In this case, through the fetchurl tool.

  • specify additional build inputs. These have to be somewhere in the nix namespace. pkgs.unzip refers to nixpkgs.unzip in the standard channel.

Both of the files above must go in a subdirectory. I named the subdirectory nix-deps/. Some subtle interaction will cause an infinite recursion if the two files are included in the root directory of your project.

With those present, it is time to build the nix-shell command:

shell.nix

let
  pkgs = import <nixpkgs> {};
  stdenv = pkgs.stdenv;
  terraform = import nix-deps/terraform.nix {};
  packer = import nix-deps/packer.nix {};

in stdenv.mkDerivation {
  name = "v2-devops";

  buildInputs = [ pkgs.ansible2
                  terraform
                  packer
                  pkgs.python
                  pkgs.python27Packages.alembic
                  pkgs.python27Packages.boto
                  pkgs.python27Packages.psycopg2
                  pkgs.awscli
                ];
}

The only difficult part here was for me to figure out how to import my Terraform and Packer derivations. I handle that with the import nix-deps/<package>.nix {} lines. The result of each import statement is a derivation, and so it is valid to include in buildInputs.

buildInputs again just lists the packages that must be included in this derivation. So, I included all of the packages that I use directly.

Thus, from the root directory of my devops folder, I can simply run nix-shell and have exactly the version of Terraform, Packer, Ansible, and Python that I want. This also means that I can have completely different versions for a different devops repository (I was actually administering three different clouds, all with different standards). And, possibly best of all, if I could convince my co-workers to use Nix (the tool, not the operating system), they would have had a trivial way to set up their development environments, also.

Processing SqlValues in Haskell

Let us assume that, whether you are just learning Haskell for the first time or have long since mastered Haskell the language, you suddenly need to hook up to a database for the first time.

Don’t laugh. Many applications work extremely well without a database.

My applications, however, almost always have a relational database somewhere in their guts. I work with a lot of relational data. I have to learn Database.HDBC early, and many of you will, too.

You should know that I have actually practiced Haskell several times over the last six years. I probably have about one year of active development scattered across that time. This time I was finally able to comprehend some concepts that really kicked my ass in previous attempts. In that time, libraries have changed and Database.HDBC has emerged as the primary interface to SQL.

Putting data into the database is really easy. Even getting data back out is easy, up to a point. You can find basic instructions on connecting for the first time, putting data into the database, and running queries, over in Chapter 21 of Real World Haskell. My article does not cover that. Instead, it covers what to do with the data that HDBC hands back to you.

I have not written much about it, but I am building up a general Fitness Application that covers workouts of the form “x repetions, y sets”. This application is currently distinct from my Cycling application, but I may merge them some time in the future.

The application

I am slowly working my way through the 100 Pushup Challenge. I have not purchased the Android app or anything and have been keeping logs of my progress in a normal text-based journal. But, I want to do some data analysis so that I can see trend lines and whether and how quickly I am making progress.

Several different workout types all follow this data structure. Whether I am doing the 100 Pushup Challenge or the 200 Situp Challenge or any of the other challenges, all of the workouts share identical data structures. I call this a SetRepWorkout (a workout consisting of multiple sets of repetitions with rest time in between) and modelled it as such:

data WorkoutType = Pushups | Situps deriving (Show, Read, Eq)
data SetRep = CSet { reps :: Integer, rest :: Integer, next :: SetRep }
            | FSet { reps Integer }
data SetRepWorkout = SetRepWorkout { uuid :: Data.UUID.UUID,
                                     date :: Data.Time.Calendar.Day,
                                     workout_type :: WorkoutType,
                                     description :: String,
                                     sets :: SetRep }

In theory, I could have represented things like so:

data SetRep = CSet { reps :: Integer, rest :: Maybe Integer }
            | FSet { reps :: Integer }
data SetRepWorkout = SetRepWorkout { uuid :: Data.UUID.UUID,
                                     date :: Data.Time.Calendar.Day,
                                     workout_type :: WorkoutType,
                                     description :: String,
                                     sets :: [SetRep] }

In fact, you will see later that I use this representation when storing the data. Arguably, this could have worked out better, but I used the original recursive definition. Even more, I probably could have left out the rest time as not informative.

Now, given the representation, we have to work on the ever-annoying functions to move the data into and out of the database.

When I started, I investigated the Persistent database architecture. I will likely investigate it again later, but I discovered that Persistent does not handle recursive data structures. When I told Persistent to save a SetRep structure, it saved the first CSet just fine, but everything in the next field ended up serialized into the next column in the database. I find this representation unacceptable as it forces all data queries to load the data into the objects, making ad-hoc queries impossible.

Changing the representation to non-recusive would have fixed the problem, but I would not have created nearly so much fodder to explain. And, ultimately, when doing complex data modelling we will all find a case in which the ORM breaks down and we must use SQL instead.

For the final application, though, I do not rule out the possibility of changing the representation. I may even drop the rest time by dropping the SetRep structure completely and making sets a list of Integers. If I make this change, Persistent becomes more viable, but still has some problems. I will discuss those down in my Conclusions.

The really easy way

Most documentation online covers running queries and retrieving data, but it rarely focuses on retrieval beyond the basic fetchRow or fetchAllRows functions. Once you have called these functions (or fetchAllRowsAL or fetchAllRowsMap), you have SqlValues, not values relevant to your domain model. Consider this:

conn <- connectSqlite3 "example.sqlite"
stmt <- prepare conn "SELECT * FROM Workout WHERE uuid=?"
execute stmt [toSql "9d4f98cd-63eb-4fad-8d9d-070f191b72db"]
rows <- fetchAllRowsAL' stmt

This looks distressingly procedural, but I could have rearranged it into a »= pipe to make it feel less so. When we examine rows, in this case we see:

[[("uuid",SqlByteString "9d4f98cd-63eb-4fad-8d9d-070f191b72db"),
  ("day",SqlByteString "2012-04-18"),
  ("workout_type",SqlByteString "Pushups"),
  ("description",SqlByteString "Week 3, Day 3, Column 2")]]

Now, remember that I called fetchAllRowsAL'. This will fetch all of the rows (hence the outermost list that has only a single element), and return each row as an association list of (column name, value). Further, this particular function fetches all data strictly, so I don’t have to worry about laziness issues when I finalize the stmt.

So, extract just the interesting workout with

let row = head rows

Then we need to convert from [(String, SqlValue)] to Workout:

let uuid = fromJust $ fromString $ fromSql $ fromJust $ lookup "uuid" row
    day = parseTime defaultTimeLocale "%Y-%m-%d" $ fromSql $ fromJust $ lookup "day" row
    workout_type = workouttypeFromSql $ fromJust $ lookup "workout_type" row
    description = fromSql $ fromJust $ lookup "description" row
SetRepWorkout uuid day workout_type description (FSet 0)

This works, but it catches no errors. fromJust will throw an exception if it gets Nothing as an input. fromString :: String -> Maybe UUID return Nothing if it receives a string in the wrong format. So, two classes of errors:

  • column not present
  • invalid data

I feel comfortable with allowing a column not present error to crash the application. The application expects a particular structure for the database and does not get it, so I consider it a programming error. But data in the database can get corrupted, and I do not want the program to crash because of an invalid data format. I would flag the error and do something with it in a higher level of the application.

A more robust (and almost as easy) way

It turns out that a more robust implementation is almost as easy to implement, but figuring that out took me seven hours of experimenting.

Notice that HDBC contains safeFromSql :: Convertible SqlValue a => SqlValue -> ConvertResult a. This leads naturally to the definition of Convertible and to ConvertResult.

The code does not include an instance for Convertible SqlValue UUID, but such an instance would help. If you had such an instance, you could say this:

let uuid = safefromSql $ fromJust $ lookup "uuid" row

(Note: I left a fromJust. lookup will return Nothing if the “uuid” column does not exist, and I consider this a programming error. So, I actually want fromJust there to blow up the program in that event.)

In fact, with proper instances, you could replace much of the above code like this:

let uuid = safeFromSql $ fromJust $ lookup "uuid" row
    day = safeFromSql $ fromJust $ lookup "day" row
    workout_type = safeFromSql $ fromJust $ lookup "workout_type" row
    description = safeFromSql $ fromJust $ lookup "description" row

Now, each of the fields will contain Left (ConvertError ...) instead of exploding when the code encounters invalid data for that field. You cannot feed that directly into a SetRepWorkoutand unravelling would be tedious, butConvertResult bis just a type alias forEither ConvertError b`, and so it functions in the Either Monad. So the entire function above can look like this:

rowToWorkout :: [(String, SqlValue)] -> ConvertResult SetRepWorkout
rowToWorkout row = do
    uuid <- safeFromSql $ fromJust $ lookup "uuid" row
    day <- safeFromSql $ fromJust $ lookup "day" row
    workout_type <- safeFromSql $ fromJust $ lookup "workout_type" row
    description <- safeFromSql $ fromJust $ lookup "description" row
    return $ SetRepWorkout uuid day workout_type desciption (FSet 0)

To make this work, you must ensure that each of the above conversions has a Convertible a b instance.

As it turns out, HDBC already provides Convertible SqlValue Day and Convertible SqlValue String. I simply had to provide Convertible SqlValue UUID and Convertible SqlValue WorkoutType. Here I provide those implementations, complete with the compiler flags necessary to even make them possible. I put the compile flags at the top of the file, but you could put them on the command line or in your Cabal file.

{-# LANGUAGE TypeSynonymInstances, FlexibleInstances, MultiParamTypeClasses #-}

import Data.ByteString.UTF8 a BUTF8 (toString)
import Data.UUID
import Data.Convertible.Base


instance Convertible SqlValue UUID where
    safeConvert (Sqlstring a) = case fromString a of
        Just b -> Right b
        Nothing -> Left $ ConvertError (show a) "SqlValue" "UUID" "Could not parse UUID"
    safeConvert (SqlByteString a) = safeConvert $ SqlString $ BUTF8.toString a
    safeConvert a = Left $ ConvertError (show a) "SqlValue" "UUID" "No conversion available"

instance Convertible SqlValue WorkoutType where
    safeConvert (SqlString a) = case a of
        "Pushups" -> Right Pushups
        "Situps" -> Right Situps
        _ -> Left $ ConvertError (show a) "SqlValue" "WorkoutType" "Unrecognized value"
    safeConvert (SqlByteString a) = safeConvert $ SqlString $ BUTF8.toString a
    safeConvert a = Left $ ConvertError (show a) "SqlValue" "WorkoutType" "No conversion available"

Given these implementations, and the rowToWorkout function, the selectWorkoutByID function is not bad, though it could probably use some more refinement.

selectWorkoutByID :: IConnection a => UUID -> a -> IO (Either String SetRepWorkout)
selectWorkoutByID w_id conn = do
    stmt <- prepare conn "SELECT * FROM Workout WHERE uuid=?"
    execute stmt [toSql $ show w_id]
    workout <- fetchAllRowsAL' stmt >>= mapM (return . rowToWorkout)
    return $ case length workout of
        0 -> Left "No workout found"
        1 -> case (head workout) of
            Right w -> Right w
            Left err -> Left $ show err
        _ -> Left "Multiple hits for a single workout Id"

Conclusions

You will need this information. For one reason or another, you will one day encounter a case in which an ORM cannot function, and then you will need to go to SQL. The needs of the database absolutely should not dictate your domain model. Persistent annoys me because, in an effort to stick with “Don’t Repeat Yourself”, it actually forces database knowledge into the domain model, which I feel is a violation of the Onion Architecture. Ultimately, though, this is a simple application and such a violation may not be so bad.

This case, however, I used for practice and for teaching. The program is so simple that I really can explore the implications of the Onion Architecture and force in a level of architectural purity. I would like you to closely examine the code and consider for yourself whether such a setup makes sense in your application. Perhaps, when I learn TemplateHaskell, I will even develop a set of templates to describe this code and allow the compiler to figure it out for me.

More concerning to me, though, is that I am not sure how Persistent can be made to follow even First Normal Form when dealing with a composite data structure without modelling the data in some very odd ways. Now, I would accept this if I were going to a key-value store or a document database, but I’m going to a relational database and I want the option of running queries based on proper relations. In almost every relational database I build, I demand at least Second Normal Form, frequently also Third Normal Form, and possibly also some insights from even higher normal forms.

Model <-> Database code is annoying, tedious, and error-prone. I recently read an article by Patrick Loi, Dependency Injection Ceremony, and really hooked on the image of the Devil on one shoulder, the Angel on the other shoulder, and the Angel saying “You should be listening to that other guy”. Read the article, as it is hilarous and informative. It also is relevant here.

Martin Fowler has some objections to ORM Hate, and he is right in that rolling your own ORM almost never makes sense. On the other hand, the ORM always (no, really, ALWAYS) violates a good Onion Architecture because it forces you to define database representation in the middle of your domain model. Ew. Gross.

Your solution is going to vary. For now, due to the extreme pain of a project at work in which I definitely did not properly isolate the domain model from the database, I will practice quite a lot with the purest architecture I can get. After I have experienced the benefits and drawbacks of this purity, I will revisit the compromises I want to make on my projects.

I have the full set of code for this fitness project (which will slowly grow into an application) on Gitlab. If you have comments or feedback, please email them directly to me at savanni@alyra.org.

http://i.creativecommons.org/l/by-nc-sa/3.0/88x31.png

Processing SqlValues in Haskell by Savanni D'Gerinel is licensed under a Creative Commons Attribution-NonCommercial-SharAlike 3.0 Unported License. You can link to it, copy it, redistribute it, and modify it, but don't sell it or the modifications and don't take my name from it.

Programming a Neopixel Pride Flag

Some weeks ago, I had an instance in which I was planning to do in person volunteer recruitment for Freedom for All Massachusetts. Now, this is very different from canvassing, as I found out about when I was at Pride. There is no “hide your politics for a bit because we need accurate information”, this is “assume everyone you talk to is on our side because we’re trying to recruit them to help us”.

_DSC1459.jpg

So, what better way for me to show my pride, and be a techie, than to have a batch of LEDs rotating between the pride flags that I care most about?

Enter neopixels and an ATTiny86, in the form of an Adafruit Trinket. This is my first significant hardware project that involves both wiring and programming.

I started out with a neopixel RGBW jewel, but I was never able to get the signalling to work. Lacking any better ideas, I dropped back to a bunch of 8mm through-hole RGB neopixels and programmed for that. More on the problem later, but I think the programming library I was using does not actually signal RGBW neopixels correctly.

While it took a great many hours, mostly in setting up my environment and then getting the hang of very low level C programming, I did eventually succeed. But on a breadboard with long jumper wires. Still, proof of concept, and I’ll get a neopixel RGB jewel later to finish out the project.

All of the code for this project is in the pride-trinket github project.

AVR Builds

_DSC1460.jpg

First, I need a build environment. I am not using Adafruit’s development environment because I expect to pretty quickly go beyond the idea of a single tiny microcontroller, a microcontroller that Adafruit packaged and shipped. I’m already planning to go beyond Adafruit’s offerings, though their products are very nice.

For every AVR project, you need three things:

  • AVR binutils
  • Libc built for AVR
  • A GCC that targets AVR

I discovered bugs in the libc and GCC builds in my Nix repository, so I patched against those bugs and then repackaged all three packages for safety.

avrbinutils has warnings that cause the build to error, so I replaced the build line.

  src = fetchurl {
    url = "mirror://gnu/binutils/binutils-${version}.tar.bz2";
    sha256 = "028cklfqaab24glva1ks2aqa1zxa6w6xmc8q34zs1sb7h22dxspg";
  };
  configureFlags = "--target=avr --enable-languages=c,c++ --disable-werror";
  meta = with stdenv.lib; {

GCC needed a change to be able to find avrbinutils.

I glue it all together with a rather pedantic shell.nix.

    avrbinutils = import nixpkgs/avr/binutils {
        stdenv = pkgs.stdenv;
        fetchurl = pkgs.fetchurl;
    };
    avrgcc = import nixpkgs/avr/gcc {
        stdenv = pkgs.stdenv;
        fetchurl = pkgs.fetchurl;
        gmp = pkgs.gmp;
        mpfr = pkgs.mpfr;
        libmpc = pkgs.libmpc;
        zlib = pkgs.zlib;
        avrbinutils = avrbinutils;
        texinfo = pkgs.texinfo;
    };
    libc = import nixpkgs/avr/libc {
        stdenv = pkgs.stdenv;
        fetchurl = pkgs.fetchurl;
        avrgcc = avrgcc;
        avrbinutils = avrbinutils;
        automake = pkgs.automake;
        autoconf = pkgs.autoconf;
    };

    ...

    buildInputs = [
        avrbinutils
        avrgcc
        libc
        pkgs.avrdude
        ws2812.ws2812_avr
    ];

To get this working, I spent many hours trying to figure out why the libc libraries weren’t showing up in my path. I no longer remember searches that I used to find what I needed, but I eventually added a setup-hook.sh to my libc build. That script adds include paths and library paths to NIX_CFLAGS_COMPILE and NIX_LDFLAGS for each library directory under <libc>/avr/lib/.

I do not have a GCC wrapper working in my environment, so I had to modify my project Makefile to explicitely include these environment variables.

ALL_CFLAGS = -mmcu=$(MCU) -I. $(CFLAGS) $(NIX_CFLAGS_COMPILE) $(NIX_LDFLAGS)

I copied my base Makefile from the neopixel_i2c project, which I have used in previous experiments that involved an embedded Linux system.

One final element, the crtattiny85.o file, is missing. That file is found in in avrlibc hierarchy at <libc>/avr/lib/avr25/crtattiny85.o. I have no idea how the compiler knows that it wants that file in paricular, so I have not determined what environment variables I could use to tell the compiler where to find it. Instead, and somewhat out of laziness, I just copied the file into my working directory.

This gives me a working compiler!

If you want to use the environment, try pulling the shell.nix file and the avr directory. If you are not doing a neopixel project, you can leave out the ws2812 derivation, as that is a packaging of the library that handles neopixel signalling.

WS2812

_DSC1461.jpg

WS2812 refers to the onboard controller that neopixels use. Neopixels may look like LEDs, but they are actually clusters of LEDs with their own processor and power distribution. The control protocol works by sending precisely timed signals down the wire, one trio of signals per pixel, followed by a long time at logical 0. Each pixel captures the first three signals it receives and sends all remaining signals down the wire until the long logical 0 indicates that the entire message is done.

The neopixel ws2812 library includes clever assembly code that implements the timing based on an 800kHz clock.

I first worked with the library as I was learning to work with the neopixel_i2c library. However, I did not want to use git submodules. Instead, I decided to package this as a Nix derivation, including any patches that I wanted to include.

In this case, I had to build the derivation from scratch as I had no template to start with. Fortunately, I have the hang of building libraries, but I still provide a setup-hook file to set a convenient WS2812_PATH environment variable and a patch to the Makefile to depend on NIX_CFLAGS_COMPILE and NIX_LDFLAGS.

My derivation file provides both light-ws2812-source and the ws2812_avr library as separate derivations. The library actually provides code for multiple environments, but ws2812_avr is the only one I currently build.

While this is a “library”, it is unusual in the same way as libc in that my only target is a .o file, not a .so or .a. Like with the crtattiny85.o file, I manually copy the light_ws2812.o file into my project.

Some Hardware

_DSC1455.jpg

This is where the fun happens. For this project, I have:

I have standardized my wire colors on this project as RED for 5V power, BLUE for ground, and YELLOW for data. Since Neopixels are 5V driven, I made sure to use a 5V Trinket. Watch out for that, as I don’t know what would happen if you tried to use a 3.3V trinket with the pixels.

The Trinket accepts main power on the USB wire. The underside of the Trinket has pads to which one could attach incoming power, also. Importantly, the 5V pin and the GND pin on he sides of Trinket are for the Trinket to provide power to any attached hardware, not for the Trinket to receive power. Fortunately, I did not destroy anything before I understood this.

_DSC1458.jpg

I use Pin #3 on the Trinket as the data line. This pin number is hard-coded in the ws2812 library. You can change it by patching ws2812_pin in ws2812_config.h. If you wanted to do this, I recommend a patch that you provide in the patches phase of your derivation.

On this project I have only the single yellow wire because I carefully ensured that the data pins chain to one another. These particular pixels made that convenient by putting the data in and data out pins on the edge of the pixels with the power pins in the center.

The wiring is important. When you power the Trinket on for the first time, keep a finger near it and pull power as fast as you can if you feel heat. I made a mistake in my wire colors at first and accidentally shorted the Trinket’s power to ground. Fortunately, I noticed and pulled power before anything exploded.

When you apply power, all of the pixels will go to a random state while the Trinket boots up. While I don’t know the details of how to fix that, I suspect a pull-down resistor on the data line would help.

After a few seconds, though, the Trinket will start running the program. In my case, the program just runs forever, transitioning from one pride flag to the next in a loop. I get visible flicker in the pixels, which I suspect I could alleviate with capacitors on the inputs. Some folks have recommended 1uF capacitors, but said that 10uF, while overkill, would do the job.

Programming

Brush up your C, as this is low-level programming. Nevertheless, it is an odd combination of fun and frustrating. Remember, you have no debugging tools on the Trinket.

ws2812 provides a cRGB convenience data structure and a function, ws2812_setleds function which accepts a struct cRGB *ledarray. I had not done any C programming in quite a long time, so I completely missed that struct cRGB *ledarray is a standard C way of describing struct cRGB ledarray[x], where x is unknown. This means a single contiguous block of data, one cRGB after another.

Until I figured that out, I wrote all kinds of wrong code that mostly amounted to creating arrays of pointers to cRGB structures. Now, I create my pride flags as normal cRGB arrays, and happily pass them around as struct cRGB *.

const struct cRGB genderqueer_flag[LED_COUNT * 3] = {
    { .r = 128, .g = 0,     .b = 128 },
    { .r = 128, .g = 0,     .b = 128 },
    { .r = 128, .g = 128,   .b = 128 },
    { .r = 0,   .g = 128,   .b = 0 },
    { .r = 0,   .g = 128,   .b = 0 }
};

I wrote so many bugs that trial and error was insufficient, so I wrote in some conditional compilation logic that lets me run my application directly on my host machine. For the neopixel project, I needed in particular to provide my own _delay_ms function, declarations for uint8_t and uint16_t, and wrapper for ws2812_setleds. At the top of my program I have a conditional compilation block for initial declarations:

#if __HOST__
/* headers for the host machine */

typedef unsigned char uint8_t;
struct cRGB { uint8_t g; uint8_t r; uint8_t b; };

void _delay_ms(double ms) {
    int ms_int = round(ms);
    struct timeval delay = { .tv_sec = ms_int / 1000, .tv_usec = (ms_int % 1000) * 1000000 };
    select(0, NULL, NULL, NULL, &delay);
}

#define uint8_t unsigned char
#define uint16_t unsigned short
#else
/* headers for the AVR */
#endif

In the body of my code, I provide write_leds as a wrapper around ws2812_setleds, though I realize now that I could have simply redeclared ws2812_setleds in the __HOST__ block at the start of the application.

void write_leds(const struct cRGB *leds, uint16_t count) {
#if __HOST__
    for (int i = 0; i < count; i++) {
        printf("\t{ %d, %d, %d }\n", leds[i].r, leds[i].g, leds[i].b);
    }
#else
    ws2812_setleds((struct cRGB *)leds, count);
#endif
}

The main body of my code is a simple state machine that I modeled as a machine that is either RESTING on a particular pattern or TRANSITIONING to another pattern. When transitioning I do a simple interpolation between my starting pattern and my ending pattern, so I am able to base the “current” pattern solely on the start, end, current frame number, and total number of frames.

struct state_s {
    uint8_t current;
    uint8_t frame_count;

    const struct cRGB *start;
    const struct cRGB *end;
};
uint8_t interpolate_color (uint8_t start, uint8_t end, uint8_t frame_count) {
    uint8_t res = start + ((end - start) / TRANSITION_FRAMES * (frame_count + 1));
    return res;
}

Finally, my state machine block is rather straightforward:

        switch (state.current) {
            case RESTING:
                _delay_ms(REST_TIME_MS);
                state.current = TRANSITIONING;
                break;
            case TRANSITIONING:
                if (state.frame_count < TRANSITION_FRAMES) {
                    /* code to calculate write the current pattern */
                    state.frame_count = state.frame_count + 1;
                    _delay_ms(TRANSITION_TIME_MS / TRANSITION_FRAMES);
                } else {
                    state.current = RESTING;
                }
                break;
        }

Glue it all together and I get a program that displays a pattern for five seconds, then fades to the next one over the course of five seconds.

Program Loading

The final step to actually try out all of the work is to flash the program onto the chip. My Makefile provides a main.hex file, which is the AVR image to be sent to the controller. I use avrdude to do the programming. I include in my repository a Trinket-friendly, macOS version of avrdude.conf.

Every time I try programming, I have to do it twice in a row. The first time always fails with a checksum error, and the second time succeeds. I have no idea why this happens, but I made a script that does it for me:

avrdude -c usbtiny -p attiny85 -U flash:w:main.hex
sleep 5
avrdude -c usbtiny -p attiny85 -D -U flash:w:main.hex

Note that the two avrdude commands differ. Just issuing the command with -D gives me one checksum error. Issuing the command with the -D gives a different checksum error. Issuing the two commands in this order… just works? Let me know if you know why and especially if you can tell me the correct way of running the command.

Making sure to use a data cable, plug the Trinket into a USB port. The Trinket seems to automatically go into programming mode if it detects a data cable, so just run the script above. When the programming is done, the Trinket should immediately start running the application!

More things!

I still have more that I want to do. I want to get an RGB pixel jewel and wrap it into a wearable form. I want to add a bisexual pride flag. And I want to show it off!

And then I want to start on my Halloween costume. I have a Lapis Lazuli costume that I have wanted to make for years, and now I can add lights into the design. When I do this costume, instead of using a Trinket I want to drop down to a bare AtTiny85 so I can make the electronics as compact as possible.

But more than this, I want to do more work based on a Beagle Bone or other embedded Linux platform so I can use the i2c bus to link together sensors and lights with some logic. I have already done experiments with this and I am looking forward to figuring out how to mount sensors and lights on a piece of clothing.

More references

Progress on Illuminance

I started the Illuminance project back in February of this year. Immediately after my last contract ended, I started working on the code in order to make this application real.

Three months in, I both exault in everything I have learned, and despair at how much remains to be done. This looked like such a simple problem, yet in its solution I have pursued so much new information that I constantly swim underneath a bare understanding of all that I have learned.

Where am I now?

User Interface

I have a very basic user interface that allows me to load images, see their previews, see error messages that occur during load, and run a “naive” HDR calculation. Error messages get displayed in the location where an image preview would normally be. While this is not necessarily elegant, it allow me to display an error that is specific to one image, since a lot of things happen in parallel and in other threads.

Generally, no expensive operation occurs in the main gui thread, so the gui does not ever actually hang. I have not been able to test this exhaustively and I am sure that I will find places where I am wrong, but this does certainly seem to be the case.

Images load in millisecond time, even very large images. They also load with reasonable space usage. I had some great success with creating my own image codec library and representation, but then I discovered that a combination of JuicyPixels and Repa solve the problem better by virtue of being more complete.

On the HDR Calculation

I have spent a vast amount of time learning math in the hopes that I could figure out how to set up the HDR calculation. Ultimately, I read some Matlab code that somebody else wrote to figure out how to set up all of the terms for the problem.

I put all of that together, set up the matrices to solve, called into the LaPack sgelsd function to actually run the calculation… and got the wrong answer. I am stumped, however there is a lot that I can play with still. I have before solved problems where the wrong answer cannot be reverse engineered and where fixing the problem can only be done by carefully examining the inputs. I imagine that is what will be required here.

I do have what I call a naive calculation in the application. This calculation scales every pixel by the associated exposure value, and then averages the result. The end result is weird in the visual artifacts and doesn’t actually give me a high dynamic range anything. While I could improve this, all I wanted to do was to get something on the screen. I have since focused all of my efforts on getting the real calculation to work.

What needs to be done

Oh so much.

  • Get the HDR calculation actually working
  • Be able to save and load HDR files
  • Tonemapping to LDR
  • Making a nice GUI

sigh. In other words, what is done so far only barely scratches the surface of a complete application.

Diversion

In any event, I am diverting to a different task. Problems on Linux have left all of the current photo managers unusable, and have left me unable to even submit a bug report on the one that I’ve been using for several years, so I am scrambling to hack one out on my own. I have no idea whether this is going to work, because while I can handle most everything, there are GUI elements that I don’t know how to handle yet. We shall see. This does, however, put Illuminance on hold, but all of my photography is really on hold due to these bugs.

  • Shotwell – Ug. Doesn’t handle XMP files. Won’t update tags on a RAW file of any kind. Instead, squirrels tags away in some custom database in a hidden folder so that no other program can see them. This problem burned me very badly once, so I refuse to use the program again unles full XMP support gets added.
  • Digikam – An 8 million pound gorilla (requiring a full installation of KDE) that I’ve been using with good success for a couple of years. Writes all tags for RAW files into an XMP sidecar, so I haven’t lost a tag in years. Unfortunately, it is also crashing randomly with either the Abort signal (and no reportable stack trace) or a Segmentation Fault (which might be reportable as soon as I get symbols installed). I think the problem lies somewhere in the KDE libs, not in Digikam itself. Either way, the application is unusable and I have no way forward to fixing it.

So, maybe I can produce something a little lightweight. I’d been thinking of trying, anyway, but I had not intended to do so until Illuminance was done. Hopefully I will get results faster since this is just a GUI and data management problem.

Python Type Annotations

I have to admit, 40% of the reason I use Haskell lies in static type checking. Another 40% lies in strong, ironclad immutability. Only about 20% of my love of Haskell has anything to do with language expressiveness. Make no mistake, the language has expressiveness out the wazoo, but I truly love static typing and immutability.

One day recently turned into a disaster. One problem lead to another in a task that should have been trivial and instead involved four hours of beating my head against the desk. Part of the problem was that my system under test had only acceptance tests, executed only with all of standard out and standard error shunted away to /dev/null. Either way, after I got my changes made, I decided to step out of the office to think.

Python has expressiveness. It has neither static type checking or immutable values, and the language developers get really holy about this. I have no interest in arguing with them, as I believe they have decided to abandon all safety and flood the world with code that might blow up in production because maybe in some corner case that a STATIC TYPE CHECKER could have detected at compile time, they inadverdently passed an Integer to a function that expected a String. So, I will not change Python the language, but I do want to make things nicer. Even though my solution will not get checked at compile time, it can certainly make debugging easier when a function crashes immediately due to a blown data type assertion, rather than propogating that incorrect data type potentially quite some distance.

I have put the code in a Repository. You can get this particular file at type_checking.py

hg clone https://gitlab.com/savannidgerinel/python-tools

The gruesome way

You could do it like this. I have done this.

def authenticate (username, password):
    assert isinstance(username, types.StringType)
    assert isinstance(password, types.StringType)

    ... do a bunch of authentication stuff and talking to databases and many things that belong in the IO monad ...

    assert isinstance(ret, UserClass)
    return ret

Ew. It will work in a pinch… but ew. This gets especially bad if I have several places from which I can return from the function. Yes, it improves the readability of the input parameters, but it does little for the return parameter beyond making postconditions.

Slightly better

Assertions are things that “should never happen in code”. So, technically, an AssertionError is actually not a good thing to throw in the case of a type error. Python actually provides TypeError to indicate that a data type error has occurred. That is convenient. So, instead of calling assert, let’s create a function that will do the job and raise a better exception. And, let’s build in making the parameter optional.

def typecheck (param, type_, optional=False):
    if not optional and param is None:
        raise TypeError('None is not permitted for this value')
    if param is not None and not isinstance(param, type_):
        raise TypeError('Expected type %r, got %r' % (type_, type(param)))

With this, your above code would look like this:

def authenticate (username, password):
    typecheck(username, types.StringType)
    typecheck(password, types.StringType)

    ... do a bunch of authentication stuff and talking to databases and many things that belong in the IO monad ...

    typecheck(ret, UserClass)
    return ret

This doesn’t improve the code much, but it does make for more descriptive error messages. I’m rather liking this improvement. But I can do better.

Decorative rescue

I once read this round of pejoratives about static type users, and I wondered for a while what that meant. I looked things up, found a few references to using decorators to “decorate” type checks on functions, but I did not like the solutions. Maybe they were good solutions, but I wanted to solve it myself. Also, the typecheck module for Python appears to be almost seven years dead.

So I introduce a some code that I wrote in an hour yesterday.

First, I played a bit with the syntax, and then I put the syntax into a unit test. You do test your code, don’t you?

class TestTypeChecker(unittest.TestCase):
    @unittest.skip('disabled')
    def testNoParamsReturnString(self):
        @accepts()
        @returns(types.StringType)
        def f():
            return 'abcd'

        f()
        self.assertRaises(AssertionError, lambda: f('a'))

    @unittest.skip('disabled')
    def testParams(self):
        @accepts(types.StringType, types.IntType)
        @returns(types.NoneType)
        def f(var1, var2):
            return None

        f('abcd', 15)
        self.assertRaises(AssertionError, lambda: f('abcd', 'efgh'))
        self.assertRaises(AssertionError, lambda: f(15, 'efgh'))
        self.assertRaises(AssertionError, lambda: f())

In here you can see the syntax. Before each declaration of f(), I put an @accepts block and a returns block. The desired data types get passed into @accepts and @returns as though these two calls are function calls. As it happens, they are.

Additionally, I wanted to flag a parameter as optional. Not optional in that it can be omitted, but optional in that I could pass None instead of the declared type.

    def testMaybeParams(self):
        @accepts(types.StringType, Maybe(types.IntType))
        @returns(types.NoneType)
        def f(var1, var2):
            return None

        self.assertRaises(AssertionError, lambda: f('abcd', 'efgh'))
        f('abcd', None)
        f('abcd', 15)

def testOptions(self):
    @accepts(types.StringType, Options(types.NoneType, types.StringType, types.IntType))
    @returns(Options(types.NoneType, types.IntType))
    def f(var1, var2):
        if var1 == 'None':
            return None
        else:
            return 15

    f('abcd', 15)
    f('abcd', '15')
    f('abcd', None)
    self.assertRaises(TypeError, lambda: f('abcd', 5.5))
    self.assertRaises(TypeError, lambda: f(None, 'abcd'))

Note Maybe. Declarations will come soon, but I created Maybe as a class that accepts a data type as a single parameter. If either of the decorators see that the parameter type is Maybe, then it will allow None or the type passed in to Maybe in the corresponding parameter. And then, some time later, I created Options as a way to specify that a parameter can be any number of data types, including None.

So, finally, it is time to present the code itself. First, my two support classes. They are delightfully short.

class Maybe(object):
    def __init__(self, var_type):
        self.var_type = var_type

    def __repr__(self):
        return 'Maybe(%s)' % self.var_type

    def check(self, param):
        if param is None:
            return True
        if isinstance(param, self.var_type):
            return True
        return False

class Options(object):
    def __init__(self, *args):
        self.var_options = args

    def __repr__(self):
        return 'Options(%s)' % ','.join(map(repr, self.var_options))

    def check(self, param):
        for type_ in self.var_options:
            if isinstance(param, type_):
                return True
        return False

Both of these exist to give expressiveness to the type system, as above. In both cases, it became simplest to create a check operation that would actually run the check against a parameter and return whether the parameter passes.

The actual guts of the type checking happens in a series of standalone functions.

def format_param_mismatch(idx, arg_type, expected_type):
    return 'Incorrect type in parameter %d: got %s, expected %s' % (idx, arg_type, expected_type)

First, I have a function, format_param_mismatch to provide a good error message in the case of a parameter type mismatch. Note that the requirements are the index of the parameter, the argument type, and the expected argument type. I included the index because I found it necessary to say “Hey, a parameter doesn’t match and it is this parameter!”

def check_param(param, expected):
    if getattr(expected, 'check', None):
        return expected.check(param)
    return isinstance(param, expected)

This function is pretty simple. It only returns True or False. If the “expected” type has a check method, i.e., it is Maybe, Options, or some other supporting class that I have not created yet, get the result by calling the check method. Otherwise, just run isinstance.

def accepts(*var_types):
    def checked_function(f):
        def checker(*args, **kwargs):
            # if len(var_types) != len(args):
            mismatches = [
                (idx, type(arg), var_type)
                for (idx, var_type, arg) in zip(itertools.count(), var_types, args)
                if not check_param(arg, var_type)]
            if len(mismatches) != 0:
                raise TypeError('\n'.join(map(lambda x: format_param_mismatch(*x), mismatches)))
            return f(*args, **kwargs)
        return checker
    return checked_function

Decorators are complicated to code.

First, the decorator itself takes parameters. That is *var_types, and that is what allows the syntax above. That returns checked_function as a parameter.

Second, checked_function then gets applied to your original function, and the magic plumbing of the decorator replaces your binding with this new function that wraps your original function.

Third, the decorator needs to return a function, and that function will take your original function as a parameter. So, at compile time the original function and the types will all get linked together and your function binding will be replaced with the function that runs this check.

def returns(return_type):
    def checked_function(f):
        def checker(*args, **kwargs):
            val = f(*args, **kwargs)
            # assert isinstance(val, return_type), 'Incorrect return type: returned %s, expected %s' % (type(val), return_type)
            if not check_param(val, return_type):
                raise TypeError('Incorrect return type: return %s, expected %r' % (type(val), return_type))
            return val
        return checker
    return checked_function

returns works in exactly the same way as accepts, but applies the data type to the return value. With this, no matter how many return statements you have in your code, the actual returned value gets checked. Admittedly, it is getting checked after whatever side effects your function had, so if you return invalid data from your database update, your database has already been updated and potentially corrupted.

Limits

You have some limits still.

First, a part that I think is critical, is that you still will not know that you have a data type error until you actually exercise a code path that exhibits the data type error. On the other hand, at least you find out very quickly when you do so that your error is not a different kind of logic error.

Also, not quite obviously, I do not have a way for you to check arbitrary argument lists. Any part of *args that does not get captured by a named parameter will not be checked, and none of **kwargs will be checked. The decorator syntax is simply too limited to be able to describe such a check without the entire declaration becoming very cumbersome.

Generally, I would suggest avoiding arbitrary keyword arguments. It is not always a problem, but it does tend to lead to necessary but undocumented parameters. If you must use them, use them for cases where the arbitrary keyword arguments are just used to name optional arbitrary data fields, but that all actually necessary parameters are given an explicit name in your function declaration.

Overall, however, using these decorators liberally will help significantly with the task of tracking down problems that are ultimately data type errors. Additionally, the presence of the decorator helps document the API for the next person to come along, making explicit things that otherwise a programmer would have to dig into the code to find out.

If you are like me, then data type errors are the most common error you make, these decorators are going to be a big help.

http://i.creativecommons.org/l/by-nc-sa/3.0/88x31.png

Python Type Annotations by Savanni D'Gerinel is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. You can link to it, copy it, redistribute it, and modify it, but don't sell it or the modifications and don't take my name from it.