To Dream of Magick

Dreamer Shaper Seeker Maker

Programming a Neopixel Pride Flag

Posted on Mon Jul 9 01:00:00 UTC 2018

Some weeks ago, I had an instance in which I was planning to do in person volunteer recruitment for Freedom for All Massachusetts. Now, this is very different from canvassing, as I found out about when I was at Pride. There is no "hide your politics for a bit because we need accurate information", this is "assume everyone you talk to is on our side because we're trying to recruit them to help us".

_DSC1459.jpg

So, what better way for me to show my pride, and be a techie, than to have a batch of LEDs rotating between the pride flags that I care most about?

Enter neopixels and an ATTiny86, in the form of an Adafruit Trinket. This is my first significant hardware project that involves both wiring and programming.

I started out with a neopixel RGBW jewel, but I was never able to get the signalling to work. Lacking any better ideas, I dropped back to a bunch of 8mm through-hole RGB neopixels and programmed for that. More on the problem later, but I think the programming library I was using does not actually signal RGBW neopixels correctly.

While it took a great many hours, mostly in setting up my environment and then getting the hang of very low level C programming, I did eventually succeed. But on a breadboard with long jumper wires. Still, proof of concept, and I'll get a neopixel RGB jewel later to finish out the project.

All of the code for this project is in the pride-trinket github project.

AVR Builds

_DSC1460.jpg

First, I need a build environment. I am not using Adafruit's development environment because I expect to pretty quickly go beyond the idea of a single tiny microcontroller, a microcontroller that Adafruit packaged and shipped. I'm already planning to go beyond Adafruit's offerings, though their products are very nice.

For every AVR project, you need three things:

  • AVR binutils
  • Libc built for AVR
  • A GCC that targets AVR

I discovered bugs in the libc and GCC builds in my Nix repository, so I patched against those bugs and then repackaged all three packages for safety.

avrbinutils has warnings that cause the build to error, so I replaced the build line.

  src = fetchurl {
    url = "mirror://gnu/binutils/binutils-${version}.tar.bz2";
    sha256 = "028cklfqaab24glva1ks2aqa1zxa6w6xmc8q34zs1sb7h22dxspg";
  };
  configureFlags = "--target=avr --enable-languages=c,c++ --disable-werror";
  meta = with stdenv.lib; {

GCC needed a change to be able to find avrbinutils.

I glue it all together with a rather pedantic shell.nix.

    avrbinutils = import nixpkgs/avr/binutils {
        stdenv = pkgs.stdenv;
        fetchurl = pkgs.fetchurl;
    };
    avrgcc = import nixpkgs/avr/gcc {
        stdenv = pkgs.stdenv;
        fetchurl = pkgs.fetchurl;
        gmp = pkgs.gmp;
        mpfr = pkgs.mpfr;
        libmpc = pkgs.libmpc;
        zlib = pkgs.zlib;
        avrbinutils = avrbinutils;
        texinfo = pkgs.texinfo;
    };
    libc = import nixpkgs/avr/libc {
        stdenv = pkgs.stdenv;
        fetchurl = pkgs.fetchurl;
        avrgcc = avrgcc;
        avrbinutils = avrbinutils;
        automake = pkgs.automake;
        autoconf = pkgs.autoconf;
    };

    ...

    buildInputs = [
        avrbinutils
        avrgcc
        libc
        pkgs.avrdude
        ws2812.ws2812_avr
    ];

To get this working, I spent many hours trying to figure out why the libc libraries weren't showing up in my path. I no longer remember searches that I used to find what I needed, but I eventually added a setup-hook.sh to my libc build. That script adds include paths and library paths to NIX_CFLAGS_COMPILE and NIX_LDFLAGS for each library directory under <libc>/avr/lib/.

I do not have a GCC wrapper working in my environment, so I had to modify my project Makefile to explicitely include these environment variables.

ALL_CFLAGS = -mmcu=$(MCU) -I. $(CFLAGS) $(NIX_CFLAGS_COMPILE) $(NIX_LDFLAGS)

I copied my base Makefile from the neopixel_i2c project, which I have used in previous experiments that involved an embedded Linux system.

One final element, the crtattiny85.o file, is missing. That file is found in in avrlibc hierarchy at <libc>/avr/lib/avr25/crtattiny85.o. I have no idea how the compiler knows that it wants that file in paricular, so I have not determined what environment variables I could use to tell the compiler where to find it. Instead, and somewhat out of laziness, I just copied the file into my working directory.

This gives me a working compiler!

If you want to use the environment, try pulling the shell.nix file and the avr directory. If you are not doing a neopixel project, you can leave out the ws2812 derivation, as that is a packaging of the library that handles neopixel signalling.

WS2812

_DSC1461.jpg

WS2812 refers to the onboard controller that neopixels use. Neopixels may look like LEDs, but they are actually clusters of LEDs with their own processor and power distribution. The control protocol works by sending precisely timed signals down the wire, one trio of signals per pixel, followed by a long time at logical 0. Each pixel captures the first three signals it receives and sends all remaining signals down the wire until the long logical 0 indicates that the entire message is done.

The neopixel ws2812 library includes clever assembly code that implements the timing based on an 800kHz clock.

I first worked with the library as I was learning to work with the neopixel_i2c library. However, I did not want to use git submodules. Instead, I decided to package this as a Nix derivation, including any patches that I wanted to include.

In this case, I had to build the derivation from scratch as I had no template to start with. Fortunately, I have the hang of building libraries, but I still provide a setup-hook file to set a convenient WS2812_PATH environment variable and a patch to the Makefile to depend on NIX_CFLAGS_COMPILE and NIX_LDFLAGS.

My derivation file provides both light-ws2812-source and the ws2812_avr library as separate derivations. The library actually provides code for multiple environments, but ws2812_avr is the only one I currently build.

While this is a "library", it is unusual in the same way as libc in that my only target is a .o file, not a .so or .a. Like with the crtattiny85.o file, I manually copy the light_ws2812.o file into my project.

Some Hardware

_DSC1455.jpg

This is where the fun happens. For this project, I have:

I have standardized my wire colors on this project as RED for 5V power, BLUE for ground, and YELLOW for data. Since Neopixels are 5V driven, I made sure to use a 5V Trinket. Watch out for that, as I don't know what would happen if you tried to use a 3.3V trinket with the pixels.

The Trinket accepts main power on the USB wire. The underside of the Trinket has pads to which one could attach incoming power, also. Importantly, the 5V pin and the GND pin on he sides of Trinket are for the Trinket to provide power to any attached hardware, not for the Trinket to receive power. Fortunately, I did not destroy anything before I understood this.

_DSC1458.jpg

I use Pin #3 on the Trinket as the data line. This pin number is hard-coded in the ws2812 library. You can change it by patching ws2812_pin in ws2812_config.h. If you wanted to do this, I recommend a patch that you provide in the patches phase of your derivation.

On this project I have only the single yellow wire because I carefully ensured that the data pins chain to one another. These particular pixels made that convenient by putting the data in and data out pins on the edge of the pixels with the power pins in the center.

The wiring is important. When you power the Trinket on for the first time, keep a finger near it and pull power as fast as you can if you feel heat. I made a mistake in my wire colors at first and accidentally shorted the Trinket's power to ground. Fortunately, I noticed and pulled power before anything exploded.

When you apply power, all of the pixels will go to a random state while the Trinket boots up. While I don't know the details of how to fix that, I suspect a pull-down resistor on the data line would help.

After a few seconds, though, the Trinket will start running the program. In my case, the program just runs forever, transitioning from one pride flag to the next in a loop. I get visible flicker in the pixels, which I suspect I could alleviate with capacitors on the inputs. Some folks have recommended 1uF capacitors, but said that 10uF, while overkill, would do the job.

Programming

Brush up your C, as this is low-level programming. Nevertheless, it is an odd combination of fun and frustrating. Remember, you have no debugging tools on the Trinket.

ws2812 provides a cRGB convenience data structure and a function, ws2812_setleds function which accepts a struct cRGB *ledarray. I had not done any C programming in quite a long time, so I completely missed that struct cRGB *ledarray is a standard C way of describing struct cRGB ledarray[x], where x is unknown. This means a single contiguous block of data, one cRGB after another.

Until I figured that out, I wrote all kinds of wrong code that mostly amounted to creating arrays of pointers to cRGB structures. Now, I create my pride flags as normal cRGB arrays, and happily pass them around as struct cRGB *.

const struct cRGB genderqueer_flag[LED_COUNT * 3] = {
    { .r = 128, .g = 0,     .b = 128 },
    { .r = 128, .g = 0,     .b = 128 },
    { .r = 128, .g = 128,   .b = 128 },
    { .r = 0,   .g = 128,   .b = 0 },
    { .r = 0,   .g = 128,   .b = 0 }
};

I wrote so many bugs that trial and error was insufficient, so I wrote in some conditional compilation logic that lets me run my application directly on my host machine. For the neopixel project, I needed in particular to provide my own _delay_ms function, declarations for uint8_t and uint16_t, and wrapper for ws2812_setleds. At the top of my program I have a conditional compilation block for initial declarations:

#if __HOST__
/* headers for the host machine */

typedef unsigned char uint8_t;
struct cRGB { uint8_t g; uint8_t r; uint8_t b; };

void _delay_ms(double ms) {
    int ms_int = round(ms);
    struct timeval delay = { .tv_sec = ms_int / 1000, .tv_usec = (ms_int % 1000) * 1000000 };
    select(0, NULL, NULL, NULL, &delay);
}

#define uint8_t unsigned char
#define uint16_t unsigned short
#else
/* headers for the AVR */
#endif

In the body of my code, I provide write_leds as a wrapper around ws2812_setleds, though I realize now that I could have simply redeclared ws2812_setleds in the __HOST__ block at the start of the application.

void write_leds(const struct cRGB *leds, uint16_t count) {
#if __HOST__
    for (int i = 0; i < count; i++) {
        printf("\t{ %d, %d, %d }\n", leds[i].r, leds[i].g, leds[i].b);
    }
#else
    ws2812_setleds((struct cRGB *)leds, count);
#endif
}

The main body of my code is a simple state machine that I modeled as a machine that is either RESTING on a particular pattern or TRANSITIONING to another pattern. When transitioning I do a simple interpolation between my starting pattern and my ending pattern, so I am able to base the "current" pattern solely on the start, end, current frame number, and total number of frames.

struct state_s {
    uint8_t current;
    uint8_t frame_count;

    const struct cRGB *start;
    const struct cRGB *end;
};
uint8_t interpolate_color (uint8_t start, uint8_t end, uint8_t frame_count) {
    uint8_t res = start + ((end - start) / TRANSITION_FRAMES * (frame_count + 1));
    return res;
}

Finally, my state machine block is rather straightforward:

        switch (state.current) {
            case RESTING:
                _delay_ms(REST_TIME_MS);
                state.current = TRANSITIONING;
                break;
            case TRANSITIONING:
                if (state.frame_count < TRANSITION_FRAMES) {
                    /* code to calculate write the current pattern */
                    state.frame_count = state.frame_count + 1;
                    _delay_ms(TRANSITION_TIME_MS / TRANSITION_FRAMES);
                } else {
                    state.current = RESTING;
                }
                break;
        }

Glue it all together and I get a program that displays a pattern for five seconds, then fades to the next one over the course of five seconds.

Program Loading

The final step to actually try out all of the work is to flash the program onto the chip. My Makefile provides a main.hex file, which is the AVR image to be sent to the controller. I use avrdude to do the programming. I include in my repository a Trinket-friendly, macOS version of avrdude.conf.

Every time I try programming, I have to do it twice in a row. The first time always fails with a checksum error, and the second time succeeds. I have no idea why this happens, but I made a script that does it for me:

avrdude -c usbtiny -p attiny85 -U flash:w:main.hex
sleep 5
avrdude -c usbtiny -p attiny85 -D -U flash:w:main.hex

Note that the two avrdude commands differ. Just issuing the command with -D gives me one checksum error. Issuing the command with the -D gives a different checksum error. Issuing the two commands in this order... just works? Let me know if you know why and especially if you can tell me the correct way of running the command.

Making sure to use a data cable, plug the Trinket into a USB port. The Trinket seems to automatically go into programming mode if it detects a data cable, so just run the script above. When the programming is done, the Trinket should immediately start running the application!

More things!

I still have more that I want to do. I want to get an RGB pixel jewel and wrap it into a wearable form. I want to add a bisexual pride flag. And I want to show it off!

And then I want to start on my Halloween costume. I have a Lapis Lazuli costume that I have wanted to make for years, and now I can add lights into the design. When I do this costume, instead of using a Trinket I want to drop down to a bare AtTiny85 so I can make the electronics as compact as possible.

But more than this, I want to do more work based on a Beagle Bone or other embedded Linux platform so I can use the i2c bus to link together sensors and lights with some logic. I have already done experiments with this and I am looking forward to figuring out how to mount sensors and lights on a piece of clothing.

More references

Mental Health Care and Transgender Populations

Posted on Mon May 21 00:00:00 UTC 2018

Content Warning: open discussion of mental health care, suicide, and religion

Amelia Perry was family to me. Four months later I get by only by forgetting that she's gone.

Every day, on social media, I see people telling me and others like me that being transgender is a mental illness or that we need to "get help". Fascinatingly, many of us get a lot of help if we can afford it. Many more would get help if they had the money to afford it, but transgender people outside of tech tend to be close to destitute, so I have no idea where they are expected to "get help".

But let us for a moment say that money is no object and that we all can get help. We still cannot get care because care for mental health has been slashed so far that we have less space for psychiatric patients than we did before the Civil War. And, worse, practitioners are not being trained to deal with us, though our needs very modest and easy to honor, at least for human beings capable of listening and respecting other human beings. Further, medical practitioners are not only able to get away with refusing to treat us for ordinary medical problems, the Republican Party under Trump is pushing to ensure that it is legal everywhere to deny us service.

The people who tell me to "get help", don't actually want me to get help.

They want me to shut up and die.

The Catholic Church is Morally Bankrupt

The Catholic church controls so many hopsitals that 1/6th of patients end up in a Catholic hospital.

The Church has rejected us.

...Catholic hospitals that must follow a set of ethical and religious directives set by the U.S. Conference of Catholic Bishops. Indeed, one in six medical patients in the United States receive care at a Catholic hospital. The conference last year posted an open letter rejecting the legitimacy of transgender identities and urging parents to not accept their trans kids. The pop has denounced trans-affirming approaches as \
Summary of the stand from the Catholic Conference of Bishops

The Catholic Church has invaded or blessed the invasions of native populations on every populated continent in the world. They have conducted wholesale torture over the course of a millenia. They have hidden the ways in which priests abuse and rape parishioners. They accuse us of "ideological colonization", but we have never murdered anyone, much less entire populations, to push our ideology. We have simply tried to protect our people from harm and live normal lives in a society in which we face violent hatred.

Here are some examples of medical malpractice against transgender people, and these are simply a small number that made it into the news:

I also know at least one transgender woman who went into the hospital with broken bones after a car crash and reports that the staff stopped treaoting her as soon as they found out she was trans. They discharged her with pain medications, and she now lives with permanent pain.

If your "right" to practice your faith ends the life of human beings you are honor-bound to serve, you have committed murder, no matter the tools you used or the legality of the situation.

You who want me to shut up and die can fuck right off.

FUCK OFF.

Remember Amelia.

  • Please email me if you know the original creator of the resistance fist logo and the transgender variant. I would like to provide proper credit.

Family, Queerness, and Polyamory

Posted on Wed Feb 14 17:00:00 UTC 2018
IMG_20170225_150825.jpg

This day, the feast of St. Valentine (or whatever), is a day for lovers to get together for nice dinner, romantic dates, and possibly some great sex. I generally find it to be a rather gross mix of cisgender, heteronormative performance and unspoken expectation, in which mind reading is a required skill.

So, what happens in queer communities in which most people involved have multiple partners and lovers?

In poly circles, this usually leads to very extensive use of calendars and negotiations. or a lot of group dates. On a normal year, I would probably spend V-day itself with Daria, and then designate another day that I would spend with Cait and Leah. In that sense, this year is no different than others, except in that Daria and I are mourning, not celebrating.

Like I told most people a few months before I moved, I moved to Boston to be with Cait and Leah.

20150717-img_4736e.jpg IMG_2120.jpg savanni-leah-shipping-container

I also moved to be with Daria, Aria, and as it turns out, Julia, Jess, Amelia, Ezri, Georgia, and others. Some of these folks I knew in advance, others I met shortly after coming here. The Cambridge/Somerville area has a magical grouping in which a huge number of queer polyamorous folks who exist in a constant state of organic community, meeting and parting in ways that create a deep set of connections between us that cannot be tracked or easily described, but which form a large extended family.

For so many in the Camberville queer community (and, I, presume, the queer communities of other large cities), emphasis on consent and negotiation has created a group that has become incredibly affectionate in both sexual and non-sexual ways. We foster extremely safe spaces, and in that safety affection has been able to grow. Those who cannot deal with touch know that they will not be touched. Asexuals will not be pursued. Sexual relationships will be expected to be conducted with respect and consent. And mistakes... well, we even have room for those and informal means of alleviating harm.

Image uploaded from iOS (2).jpg

When I moved, I knew that moving to be with my girlfriends would not suffice. I needed community, and I got a large extended family.

IMG_1567.jpg Image uploaded from iOS-8.jpg

For the record, Cait, Leah, Daria, and I will be going to dinner somewhere that does not take reservations, so a litle finger-crossing is in order.

Remembering Amelia

Posted on Wed Feb 7 00:00:00 UTC 2018
IMG_1567.jpg

Tonight, I want to tell you about a dear friend. Someone very important to myself and to my partner, Daria.

Hir name was Amelia. Ze took hir own life late in the night last Tuesday. I found out as soon as I woke up Wednesday morning. A week ago today.

Amelia was gorgeous. Ze was brilliant. Ze sparkled, flirted, brightened spaces, tied innuendo into unexpected sentences, and could talk about advanced mathematics that I cannot begin to understand.

But, honestly, we never talked about advanced math or technology. Amelia was passionate about social justice. Ze came into our community during a chance run-in at a march in solidarity for immigrant's rights. Late in hir life, ze started campaigning for patient's rights in mental health. This hyper focus came about from continuous failures in her stays in various psych wards here in Boston. Some of these failures caused significant harm.

Make no mistake. While Amelia may have died at hir own hand, ze was driven to it. Multiple assaults from random men on the street who continue to feel liberty in how they treat women, followed by hospitalizations that dehumanized hir, ignored hir, and in some cases, did significantly more harm.

I miss my friend.

I miss a friendship lost.

I regret the tensions between us, the things we did not have time to heal.


IMG_2089.jpg

The day we all found out, the morning after Amelia died, seventeen people gathered for an impromptu wake.

Let that sink in. Seventeen people dropped everything to gather, weep, and reminisce, with no advance notice. Queer community is not large, but it is tightly connected and our members are loved. To quote my friend Aria:

Queer culture is grieving as a family no outsider would recognize, a haphazard landslide of lovers & friends. The ones who bring the tequila & gin, the ones who bring the casseroles, the ones who bring the cookies and apologize about the butter, the ones who make a big thai curry

While we can never change what outsiders would see, I want very much to change the definition of "outsider". While I always live visibly, I have not in the past put much effort into broadcasting that image, and it thus became possible for people ostensibly in my life to completely ignore that which they did not see. But that means that, should I die, there are many people who would have to learn about the full me only through pictures and stories. If they learn at all. Perhaps they would actively ignore.

For now, however, I want to actively try to show what a happy, healthy life outside of the norm of cisgender, heterosexual, monogamy looks like.

Haskell Exceptions and Abstractions, or Part 2 of Abstracting the Monad Stack

Posted on Fri Nov 10 17:00:00 UTC 2017

Error handling is a big deal for me. I have probably studied this more than anything else in Haskell because I find it so important to carefully handle unexpected errors. I have presented on this before, but those older presentations lacked a nuance of understanding that I have developed in the last few years.

This article carries on from my previous one, Abstracting the Monad Stack. In that article, I was unable to get all the way to exception abstraction. That this article took me so long to write demonstrates that I was correct in my assessment that this was too much for me to roll into my previous article.

Abstracting out the Exception

Code for this section

I still want to throw exceptions from my library, and my library has the HealthException type:

data HealthException = TimeSeriesExc SeriesExc
                     | UnknownException String
                     deriving (Eq, Show)


type Health r m = (MonadIO m, MonadReader r m, HasHealthContext r)

However, the higher-level context of my application frequently has a different exception type, which may well encompass exceptions from other modules:

data WebExc = WebExcHealth HealthException
            | WebExcHttp HttpException
            | AppForbidden
            | AppUnauthorized

Typically, I would have to unwrap the monad stack for my library in order to rewrap the exception class into the WebExc exception. Again, this works, but it creates tedious boilerplate that we would like to remove:

handleGetHistory :: Interval UTCTime -> WebM ([Sample Weight], [Sample TimeDistance], [Sample Steps])
handleGetHistory interval = do
    WebContext{..} <- ask
    weightRes <- getWeights interval
    timeDistanceRes <- getTimeDistance interval
    stepRes <- getSteps interval

    case (weightRes, timeDistanceRes, stepRes) of
        (Left err, _, _) -> throwError $ WebExcHealth err
        (_, Left err, _) -> throwError $ WebExcHealth err
        (_, _, Left err) -> throwError $ WebExcHealth err
        (Right weights, Right timeDistances, Right steps) -> pure (weights, timeDistances, steps)

In order to make this goal, we can do something similar to HasHealthContext, but this time for writing. Start out by building a class that describes the process of "wrapping" this exception. We must also declare such an exception in our type constraints.

class AsHealthException exc where
    _HealthException :: HealthException -> exc
    _TimeSeriesExc :: SeriesExc -> exc
    _TimeSeriesExc = _HealthException . TimeSeriesExc
    _UnknownException :: String -> exc
    _UnknownException = _HealthException . UnknownException

type Health r exc m = (MonadIO m, MonadReader r m, HasHealthContext r, MonadError exc m, AsHealthException exc)

This class declares a generic function that wraps all health exceptions, and then two dedicated functions for simply creating and wrapping a TimeSeriesExc and an UnknownException. The new components to the type constraint both declare that the calling monad must implement MonadReader, and that the exceptions being raised in the monad must support AsHealthException.

In the library code, you use these exceptions like so:

getWeights :: Health r exc m => Interval UTCTime -> m a
getWeights = throwError $ _HealthException $ UnknownException "demonstrating with _HealthException"

getTimeDistance :: Health r exc m => Interval UTCTime -> m a
getTimeDistance = throwError $ _UnknownException "demonstrating with _UnknownException"

Note how the exception is no longer explicitely called out in the type signatures for getWeights or getTimeDistance. However, that an exception is thrown is still strictly type checked and even documented in the constrant on the exc parameter in Health r exc m.

The caller must implement this class to make the wrapping transparent.

instance AsHealthException WebExc where
    _HealthException = WebExc . WebExcHealth

With this one change, we almost magically see our calling code collapse into something quite reasonable.

handleGetHistory :: Interval UTCTime -> WebM ([Sample Weight], [Sample TimeDistance], [Sample Steps])
handleGetHistory interval = do
    WebContext{..} <- ask
    weights <- getWeights interval
    timeDistances <- getTimeDistance interval
    steps <- getSteps interval
    pure (weights, timeDistances, steps)

Handle an exception inside the monad

Code for this section

Let's say that there are certain exceptions that you need to handle in place. For instance, assume that when I save a TimeDistance workout, I want to update some summary data. This is slightly contrived since in this case I would usually generate the summaries based on queries, but it serves to illustrate a point.

handleSaveTimeDistance got quite a few updates alongside handleGetHistory, so it has changed from our original example and looks like this:

handleSaveTimeDistance :: Maybe SampleID -> SetTimeDistanceParameters -> WebM (Sample TimeDistance)
handleSaveTimeDistance sampleId params =
    let workoutFromParams = undefined
        workout = workoutFromParams params
    in saveTimeDistance sampleId workout

Now I add the update function, keeping it within the Health monad for simplicity. I also add a fictitious rollback and commit functions. If written, they would assume that, like in any database setup, they are safely written to disk but in a way that does not take effect until a single monotonic commit function happens. Just for the fun of it, I'll also add a checkAuthorization function, which would be run before actually saving any data to disk.

checkAuthorization :: WebM ()
checkAuthorization = undefined

updateTimeDistanceSummary :: Health r exc m => TimeDistance -> m a
updateTimeDistanceSummary _ = undefined

commit :: Health r exc m => m ()
commit = undefined

rollback :: Health r exc m => m ()
rollback = undefined

However, this turns out to be super simple. Remember that I am working in MonadError, and so I have access to the already-familiar catchError. I have no need for anything complicated.

handleSaveTimeDistance :: Maybe SampleID -> SetTimeDistanceParameters -> WebM (Sample TimeDistance)
handleSaveTimeDistance sampleId params =
    let workoutFromParams = undefined
        workout = workoutFromParams params
    in
    catchError (do checkAuthorization
                   res <- saveTimeDistance sampleId workout
                   updateTimeDistanceSummary workout
                   commit
                   pure res)
               handler
    where
    handler err@(WebExcHealth _) = do
        rollback
        throwError err
    handler exc = throwError exc

Sprinkling in a bit of Template Haskell

Code for this section

The class declaration for AsHealthException above still looks like boilerplate, but I present it as explanation. The Lens library actually provides a function that does exactly this. Note that introducing TemplateHaskell also requires introducing additional files. Code generation from the lenses (or any other TemplateHaskell code generation) does not become available in the file in which it is declared.

{-# LANGUAGE TemplateHaskell #-}

import Control.Lens (makeClassyPrisms, prism)

data HealthException = ...

makeClassyPrisms ''HealthException

The library code must be changed just a bit. The functions on AsHealthException actually create Prisms now, a fairly complex data structure that I only barely grasp. In order to throw the exception, it must first be injected appropriately into a prism, and so the code becomes this:

getWeights :: Health r m => Interval UTCTime -> m (Either HealthException a)
getWeights = throwError $ review _HealthException $ UnknownException "demonstrating with _HealthException"

getTimeDistance :: Health r m => Interval UTCTime -> m (Either HealthException a)
getTimeDistance = throwError $ review _UnknownException "demonstrating with _UnknownException"

Client code is a little different, also, but only in the AsHealthException instance:

instance AsHealthException WebExc where
    _HealthException = prism WebExcHealth unwrap
        where unwrap (WebExcHealth exc) = Right exc
              unwrap exc = Left exc

Again, _HeathException is now a prism, as are _TimeSeriesExc and _UnknownException. This adds some extra options for unwrapping the exception, but I do not currently use, or have a good example of, such a handler.

End of a journey

This has already been a long journey in the making. At the end of it, however, you have learned how to use type constraints to effectively abstract an entire monad stack, making a significantly more reusable and simultaneously much easier to read and use.

This is probably not the only way to set up reusable monad stacks, but it is the one that I find the easiest to understand, the easiest to build, and the easiest to use. I ask you to try this kind of setup for your own code to see where it works and where it breaks. I also would like feedback on how well this worked for you and whether you have a different means of building the same kind of flexibility.

While we have covered a lot of ground, there is much more to do. In my next article I will provide a summary template of everything we have built here.

After that, it is time to start designing architectures that can handle dependency injection for mocking out resources in test, or even allowing run-time configuration of resource backends.

Abstracting the Monad Stack, Part 1

Posted on Thu Sep 21 04:00:00 UTC 2017

As a sidenote, my previous articles were all in terms of a fictitious image processing program. I am actually very interested in image processing, but that is not the code I have on hand, so for my examples, I am switching over to a health-tracking application that I've been working on for a while. I'll probably change the previous articles to reflect it. I am making this change, though, primarily because there are so many real examples for me to draw from.

Here is the application in question, and it is under active development. I have some layer refactoring to do, but the code is stable and I am mostly focusing my efforts on additional features.

Building a library monad

Code for this section

I previously talked about application monads, but today I will talk about a library monad.

Fortunately for everyone, they are almost identical to application monads, but with one twist that I want to lead you to.

In my health application, I have a library that forms the "application" and that library is meant to be wrapped around interface layers, such as the web API or a GUI. The application library is basically a set of functions that make a complete API that I can execute in the REPL.

So, assume a monad like this one:

data AppContext = App { weightSeries       :: TimeSeries Weight
                      , timeDistanceSeries :: TimeSeries TimeDistance
                      , stepSeries         :: TimeSeries Steps
                      }

-- TODO: handle time series exceptions. Make this less absurd.
data HealthException = TimeSeriesExc SeriesExc
                     | UnknownException String
                     deriving (Eq, Show)

newtype HealthM a = HealthM (ReaderT AppContext (ExceptT HealthException IO) a)
    deriving (Functor, Applicative, Monad, MonadIO, MonadError HealthException, MonadReader AppContext)

runHealthM :: AppContext -> HealthM a -> IO (Either HealthException a)
runHealthM ctx (HealthM act) = runExceptT (runReaderT act ctx)

(yes, that is a real TODO item in the code)

On it's own, this isn't bad. But the pain lies in that this is a library, and thus will likely end up in a different monad stack. As it is written, I would need to unroll this stack into IO and then re-roll it into my web stack. This is not horrible, but it is annoying. In the health application, I would do the re-rolling to glue these functions into my Web application monad, and it would look like this:

data Config = Config
data WebContext = WebContext { config :: Config, app :: AppContext }

newtype WebM a = WebM (ReaderT WebContext (ExceptT WebExc IO) a)
    deriving (Functor, Applicative, Monad, MonadIO, MonadError WebExc, MonadReader WebContext)

handleSaveTimeDistance :: Maybe SampleID -> SetTimeDistanceParameters -> WebM (Sample TimeDistance)
handleSaveTimeDistance sampleId params =
    let workoutFromParams = undefined
        workout = workoutFromParams params
    in do
    WebContext{..} <- ask
    res <- liftIO $ runHealthM app $ saveTimeDistance sampleId workout
    case res of
        Left err -> throwError $ AppExc err
        Right val -> return val

saveTimeDistance :: Maybe SampleID -> TimeDistance -> HealthM (Sample TimeDistance)

Again, this is not awful, but it is tedious. It can also become awful if I want to perform multiple operations from the library interleaved with operations from my webapp. For example, what if I want to query every series that I am storing?

handleGetHistory :: Interval UTCTime -> WebM ([Sample Weight], [Sample TimeDistance], [Sample Steps])
handleGetHistory interval = do
    WebContext{..} <- ask
    weightRes <- liftIO $ runHealthM app $ getWeights interval
    timeDistanceRes <- liftIO $ runHealthM app $ getTimeDistance interval
    stepRes <- liftIO $ runHealthM app $ getSteps interval

    case (weightRes, timeDistanceRes, stepRes) of
        (Left err, _, _) -> throwError $ AppExc err
        (_, Left err, _) -> throwError $ AppExc err
        (_, _, Left err) -> throwError $ AppExc err
        (Right weights, Right timeDistances, Right steps) -> pure (weights, timeDistances, steps)

getWeights :: Interval UTCTime -> HealthM [Sample Weight]
getTimeDistance :: Interval UTCTime -> HealthM [Sample TimeDistance]
getSteps :: Interval UTCTime -> HealthM [Sample Steps]

handleGetHistory already becomes tedious.

Rewrapping the context

Code for this section

The first, most obvious solution, is a helper function to re-wrap:

wrapEitherIO :: (exc -> WebExc) -> IO (Either exc a) -> WebM a
wrapEitherIO excTr act =
    liftIO act >>= either (throwError . excTr) pure

handleGetHistory :: Interval UTCTime -> WebM ([Sample Weight], [Sample TimeDistance], [Sample Steps])
handleGetHistory interval = do
    WebContext{..} <- ask
    weights <- wrapEitherIO AppExc $ runHealthM app $ getWeights interval
    timeDistances <- wrapEitherIO AppExc $ runHealthM app $ getTimeDistance interval
    steps <- wrapEitherIO AppExc $ runHealthM app $ getSteps interval
    pure (weights, timeDistances, steps)

And then, probably even one step further with a utility function to do the re-wrapping.

wrapEitherIO :: (exc -> WebExc) -> IO (Either exc a) -> WebM a
wrapEitherIO excTr act =
    liftIO act >>= either (throwError . excTr) pure

runHealthMInWebM :: (HealthException -> WebExc) -> AppContext -> HealthM a -> WebM a
runHealthMInWebM handler app = wrapEitherIO handler . runHealthM app

handleGetHistory :: Interval UTCTime -> WebM ([Sample Weight], [Sample TimeDistance], [Sample Steps])
handleGetHistory interval = do
    WebContext{..} <- ask
    weights <- runHealthMInWebM AppExc app $ getWeights interval
    timeDistances <- runHealthMInWebM AppExc app $ getTimeDistance interval
    steps <- runHealthMInWebM AppExc app $ getSteps interval
    pure (weights, timeDistances, steps)

This alone makes life much nicer. All of the exception checking boilerplate gets encapsulated into wrapEitherIO, and so every step of handleGetHistory gets to exist on the happy path. In many instances, I could just call this done.

Servant actually provides a typeclass for natural transformations which abstracts this away. It has a challenging type signature, but it is pretty nice and I recommend taking a look at it.

Type Constraints

Code for this section

I use type constraints as my preferred method for solving this problem. The idea behind it is that I try to have only one concrete monad stack anywhere in the application.

A "type constraint" is a mechanism by which I declare that a context must implement a particular typeclass, but that the context could be any context that implements that typeclass. A trivial example would be like this:

printSomeStuff :: (Show a, MonadIO m) => a -> m ()
printSomeStuff a = do
    liftIO $ putStrLn $ show a

This function will print out any value, so long as the value implements Show and so long as the function is called in any monad that implements MonadIO. For instance, all three of these calls to printSomeStuff are valid:

run1 :: IO ()
run1 = printSomeStuff "abcd"

run2 :: ExceptT String IO ()
run2 = printSomeStuff "abcd"

run3 :: MonadIO m => m ()
run3 = printSomeStuff "abcd"

Now, we build up on this concept, and to do so I'm going to repack all three of my get functions, this time starting from the simplest possible implementation.

saveTimeDistance :: Maybe SampleID -> TimeDistance -> AppContext -> IO (Either HealthException a)

handleSaveTimeDistance :: Maybe SampleID -> SetTimeDistanceParameters -> WebM (Sample TimeDistance)
handleSaveTimeDistance sampleId params =
    let workoutFromParams = undefined
        workout = workoutFromParams params
    in do
    WebContext{..} <- ask
    res <- liftIO $ saveTimeDistance sampleId workout app
    case res of
        Left err -> throwError $ AppExc err
        Right val -> pure val

saveTimeDistance can function in any monad that implements MonadIO, so the first thing I will do is to abstract that away:

saveTimeDistance :: (MonadIO m) => Maybe SampleId -> TimeDistance -> AppContext -> m (Either HealthException a)

handleSaveTimeDistance :: Maybe SampleID -> SetTimeDistanceParameters -> WebM (Sample TimeDistance)
handleSaveTimeDistance sampleId params =
    let workoutFromParams = undefined
        workout = workoutFromParams params
    in do
    WebContext{..} <- ask
    res <- saveTimeDistance sampleId workout app
    case res of
        Left err -> throwError $ AppExc err
        Right val -> pure val

This detaches me from a particular monad stack. This function can now be called as-is from any context that implements, hence in the above code, I no longer need to apply liftIO to saveTimeDistance. For bookkeeping, and because I am going to build upon this abstraction, I will give that type constraint a name:

type HealthM m = MonadIO m

saveTimeDistance :: HealthM m => Maybe SampleID -> TimeDistance -> AppContext -> m (Either HealthException a)

The next step requires a fairly large jump. I want to eliminate that AppContext parameter. It is required for every function in the health application, so it would be nice if I could pass it as part of a MonadReader. The naive solution would be to just do this:

type HealthM m = (MonadIO m, MonadReader AppContext m)

saveTimeDistance :: HealthM m => Maybe SampleID -> TimeDistance -> m (Either HealthException a)

Unfortunately, this actually is of detriment in the caller. If the caller has its own context in a MonadReader, that context is not likely to be the same as this one. The result is code that looks like this:

handleSaveTimeDistance :: Maybe SampleID -> SetTimeDistanceParameters -> WebM (Sample TimeDistance)
handleSaveTimeDistance sampleId params =
    let workoutFromParams = undefined
        workout = workoutFromParams params
    in do
    WebContext{..} <- ask
    res <- runReaderT (saveTimeDistance sampleId workout) app
    case res of
        Left err -> throwError $ AppExc err
        Right val -> pure val

I definitely do not want to be going in the direction of having to re-add a run function stack, but that is how this goes. The caller has to explicitely pull out the context for this call.

In order to get around this, I have to think a bit differently. I still want an implicit context of AppContext. But, really, the context could be larger so long as AppContext is present in it. So an alternate solution looks like this:

type HealthM r m = (MonadIO m, MonadReader WebContext m)

saveTimeDistance :: Health r m => Maybe SampleID -> TimeDistance -> m (Either HealthException a)
saveTimeDistance = undefined

handleSaveTimeDistance :: Maybe SampleID -> SetTimeDistanceParameters -> WebM (Sample TimeDistance)
handleSaveTimeDistance sampleId params =
    let workoutFromParams = undefined
        workout = workoutFromParams params
    in do
    res <- saveTimeDistance sampleId workout
    case res of
        Left err -> throwError $ AppExc err
        Right val -> pure val

In some ways, this looks better. The caller now can simply treat saveTimeDistance as part of WebM. But now saveTimeDistance becomes aware of WebContext, and so is beholden to a single caller. This is better, but not good enough.

What I want is a way to specify that saveTimeDistance can take any context, so long as that context provides me with a way to extract the AppContext. So, this is a constraint upon a constraint, and it ends up looking like this:

type HealthM = (MonadIO m, MonadReader r m, HasHealthContext r)

Basically, a HealthM function can take any MonadReader that provides r, so long as r "has a health context".

My library gets to declare the HasHealthContext interface. The caller needs to implement that interface for its own context.

type Health r m = (MonadIO m, MonadReader r m, HasHealthContext r)
class HasHealthContext ctx where
    hasAppContext :: ctx -> AppContext

WebContext = WebContext { config :: Config, app :: AppContext }
instance HasHealthContext WebContext where
    hasAppContext WebContext{..} = app

saveTimeDistance :: Health r m => Maybe SampleID -> TimeDistance -> m (Either HealthException a)
saveTimeDistance _ _ = do
    appCtx <- hasAppContext <$> ask
    ...

With similar improvements made to getWeights, getTimeDistance, and getSteps, handleGetHistory also gets much nicer, and that demonstrates exactly what we wanted to begin with:

handleGetHistory :: Interval UTCTime -> WebM ([Sample Weight], [Sample TimeDistance], [Sample Steps])
handleGetHistory interval = do
    WebContext{..} <- ask
    weightRes <- getWeights interval
    timeDistanceRes <- getTimeDistance interval
    stepRes <- getSteps interval

    case (weightRes, timeDistanceRes, stepRes) of
        (Left err, _, _) -> throwError $ AppExc err
        (_, Left err, _) -> throwError $ AppExc err
        (_, _, Left err) -> throwError $ AppExc err
        (Right weights, Right timeDistances, Right steps) -> pure (weights, timeDistances, steps)

getWeights :: Health r m => Interval UTCTime -> m (Either HealthException a)
getWeights = undefined

getTimeDistance :: Health r m => Interval UTCTime -> m (Either HealthException a)
getTimeDistance = undefined

getSteps :: Health r m => Interval UTCTime -> m (Either HealthException a)
getSteps = undefined

Looking forward

Not quite there yet. We still have some tedium with exception handling to do. In this system, any thrown SeriesExc must be caught and then re-wrapped in the HealthException in order for the application to typecheck and for the exception to propogate upwards. This sort of tedium likely drove the creation of extensible IO exceptions, which I view as unchecked and undocumented parts of the type signature.

So, the next step will be to abstract the exception throwing mechanism.

A Nazi Sympathizer in the White House

Posted on Wed Aug 16 16:00:00 UTC 2017

The President of the United States is a Nazi sympathizer and a white supremacist. He is a traitor to the nation.

The Republican party wants to pretend that they didn't know this. But Clinton warned us a year ago. He won anyway. 62 million people wanted a white supremacist rather than a woman.

I am but one white, trans, queer, able-bodied, priveleged voice howling in the wind. Far smarter people than me know precisely how we got to this point. Far smarter people than me know that we never really left this point.

So here are real, concrete actions that will push back this tide.

Listen to black people. Listen to queer people. They never stopped tell us how bad things are.

Black Lives Matter.

Haskell Application Monad

Posted on Fri Jul 14 16:00:00 UTC 2017

We want to get productive in Haskell very quickly. Most non-trivial applications will have configuration, connections to the outside world, can hit exceptional conditions, and benefit from having their operations logged. If your application has sensible logs at both high an low levels of detail, your devops team will thank you and your life of debugging a production application will be a happier one.

I want to get all of these things at once, and so it would be nice to provide a nearly boilerplate application stack that provides them all. I define the "application stack" as a group of attributes that contain the context and all of the common behaviors for an application. In Haskell, you do that with a monad stack, though work on extensible effects shows a great deal of promise and has been used to great effect in Purescript.

That said, I use monads and monad transformers, and I'll not explain either of them today. I feel that the best explanation is a non-trivial example implementation, which I will do in a future article, or refer you to a better tutorial.

While most of this article explains the process, the final result is this application stack, which may be all you need if you are already familiar with building monad transformer stacks.

data Context = Context { contextRoot :: FilePath } deriving Show

data AppError = AppError deriving Show

newtype AppM a = AppM (LoggingT (ReaderT Context (ExceptT AppError IO)) a)
    deriving ( Functor, Applicative, Monad, MonadIO
             , MonadError AppError, MonadReader Context, MonadLogger)

runAppM :: Context -> AppM a -> IO (Either AppError a)
runAppM ctx (AppM act) = runExceptT (runReaderT (runStderrLoggingT act) ctx)

The most basic stack

Almost every application needs IO. In Haskell it is difficult to do IO on top of anything (see MonadBaseControl for way), so I always put it at the bottom of the monad stack. A trivial application stack would look like this:

newtype AppM a = AppM (IO a) deriving (Functor, Applicative, Monad, MonadIO)

This is so trivial you will likely never do it, though it can be helpful in that it prevents confusion between your functions and system IO functions. Still, let's build out what you need to make this work.

First of all, you do want AppM to be a monad, and you will need MonadIO in order to actually run IO operations. The primary use that I have for Monads in an application is to eliminate the boilerplate involved with a lot of threading context through a series of function calls. More to the point, though, you cannot get MonadExcept, MonadReader, or MonadLogger into this stack without having Monad to begin with.

newtype AppM a = AppM (IO a)
    deriving (Functor, Applicative, Monad, MonadIO)

runAppM :: AppM a -> IO a
runAppM (AppM act) = act

runAppM is the function that connects your application stack to the Haskell IO stack. This is everything you need in order to create a stack: the stack itself and the runner. Now let's see it in action:

data Image = Image deriving Show

loadImage :: FilePath -> AppM Image
loadImage path = do 
    liftIO $ putStrLn $ "loadImage: " <> path
    pure Image
     

main :: IO ()
main = do
    res <- runAppM $ do
        img1 <- loadImage "image.png"
        img2 <- loadImage "image2.png"
        pure (img1, img2)
    print res

Injecting your context

IO a is too simple to make much sense. The whole point of having a stack is to unify a lot of effects within a common framework of behavior and with a common context. So, next we load and add a context.

In almost every circumstance, your context is read-only. This points us directly to ReaderT, since you will want to be able to ask for the context but never write back to it. Application state would seem like a thing that you would want to include, if your application stores state. I have generally found that it is easier to keep application state in something that is strictly IO, such as an IORef or a TVar. For now, we shall skip that.

So, change your stack to look like this:

data Context = Context { root :: FilePath } deriving Show

newtype AppM a = AppM (ReaderT Context IO a)
    deriving (Functor, Applicative, Monad, MonadIO, MonadReader Context)

runAppM :: Context -> AppM a -> IO a
runAppM ctx (AppM act) = runReaderT act ctx

The addition of MonadReader means that now you can call ask within your function to get back the context, and you don't have to explicitely pass the context in. The remaining functions get updated like so:

loadImage :: FilePath -> AppM Image
loadImage path = do
    Context{..} <- ask
    liftIO $ putStrLn $ "loadImage: " <> (contextRoot </> path)
    pure Image

loadContext :: IO Context
loadContext = pure $ Context { contextRoot = "/home/savanni/Pictures/" }

main :: IO ()
main = do
    ctx <- loadContext
    res <- runAppM ctx $ do
        img1 <- loadImage "image.png"
        img2 <- loadImage "image2.png"
        pure (img1, img2)
    print res

Suddenly, everything in Context is available to every function that runs in AppM. You get the local effect of global parameters while still getting to isolate them, potentially calling the same functions with different contexts within the same application.

Add exception handling and logging

Exceptions happen. The Haskell community is split between what I call explicit vs. implicit exceptions. In short, implicit exceptions are not declared in the type signature, can happen from any function, and can only be caught in IO code. Explicit exceptions are explicitely stated in the type signature and can be caught just about anywhere. I prefer them for all of my application errors. I'll give exception handling further treatment in a future article, and will show the use of explicit exceptions here.

Logging is almost always helpful for any application that is not of trivial size. And, once present, it can replace print for debugging, allowing debugging lines to remain present in the code for those cases when something starts going wrong in production.

First, the new application stack:

data AppError = AppError deriving Show

newtype AppM a = AppM (LoggingT (ReaderT Context (ExceptT AppError IO)) a)
    deriving ( Functor, Applicative, Monad, MonadIO
             , MonadError AppError, MonadReader Context, MonadLogger)

runAppM :: Context -> AppM a -> IO (Either AppError a)
runAppM ctx (AppM act) = runExceptT (runReaderT (runStderrLoggingT act) ctx)

This gets quite a bit more complicated with both the Logging and Exceptions being added. Remember that I use the term "stack" here, and each monad transformer involved represents another layer in the stack. When running the stack, you must peel off each layer in reverse order. I will illustrate with some types:

*Json> :t loadImage "img.png"
loadImage "img.png" :: AppM Image

*Json> :t unAppM $ loadImage "img.png"
unAppM $ loadImage "img.png"
  :: LoggingT (ReaderT Context (ExceptT AppError IO)) Image

*Json> :t runStderrLoggingT $ unAppM $ loadImage "img.png"
runStderrLoggingT $ unAppM $ loadImage "img.png"
  :: ReaderT Context (ExceptT AppError IO) Image

*Json> :t runReaderT (runStderrLoggingT $ unAppM $ loadImage "img.png") ctx
runReaderT (runStderrLoggingT $ unAppM $ loadImage "img.png") ctx
  :: ExceptT AppError IO Image

*Json> :t runExceptT $ runReaderT (runStderrLoggingT $ unAppM $ loadImage "img.png") ctx
runExceptT $ runReaderT (runStderrLoggingT $ unAppM $ loadImage "img.png") ctx
  :: IO (Either AppError Image)

The point of this is that in runAppM, the type of act is the entire stack, and the first thing to be called to begin unwrapping is runStderrLoggingT, then runReaderT, and finally runExceptT.

Notice, also, that the final type of runAppM has changed to IO (Either AppError a). runAppM will now return whatever exception gets thrown from within the context it is running, no matter where that exception is thrown, if that exception is thrown with throwException. Exceptions thrown with throw end up being the implicit exceptions I referred to, and those require some extra handling.

So, here is the rest of the code. In the places where I used to print output, I am now logging output. Note that the loggers require TemplateHaskell and have slightly odd syntax, but are otherwise nearly identical to print.

data Image = Image deriving Show

loadImage :: FilePath -> AppM Image
loadImage path = do
    Context{..} <- ask
    $(logInfo) (T.pack $ "loadImage: " <> (contextRoot </> path))
    pure Image

loadContext :: IO Context
loadContext = pure $ Context { contextRoot = "/home/savanni/Pictures/" }

main :: IO ()
main = do
    ctx <- loadContext
    do  res <- runAppM ctx $ do
            img1 <- loadImage "image.png"
            img2 <- loadImage "image2.png"
            pure (img1, img2)
        print res

    do  res <- runAppM ctx $ do
            img1 <- loadImage "image.png"
            throwError AppError
            img2 <- loadImage "image2.png"
            pure (img1, img2)
        print res 

This is the output from running main:

*Json> main
[Info] loadImage: /home/savanni/Pictures/image.png @(main:Json /home/savanni/src/haskell/src/Json.hs:76:7)
[Info] loadImage: /home/savanni/Pictures/image2.png @(main:Json /home/savanni/src/haskell/src/Json.hs:76:7)
Right (Image,Image)
[Info] loadImage: /home/savanni/Pictures/image.png @(main:Json /home/savanni/src/haskell/src/Json.hs:76:7)
Left AppError
*Json> 

So, the first block starting with do res <- runAppM runs to completion, returnin two images. The second block, runs loadImage for the first image, but then hits throwError and returns Left AppError, discarding the first image and not loading the second image at all.


This is nearly a application stack that I have used for more applications than I can count. Even if you need only one feature, such as exceptions, starting with a small stack hidden behind an application monad makes it very easy to add additional features as you need them, without needing to change the rest of your code. This pattern is trivial to extend, or contract, as needed, and so I think it starts every application on a good path.

Reflections

Posted on Mon Jul 10 12:00:00 UTC 2017

I left CollegeVine a little over two weeks ago. Since then, I have gotten myself really busy. For a moment tonight I am going to slow down to reflect on my time since leaving.

First of all, I had an obligatory Friday of just not doing anything. But by Saturday, I got to work. In the last two weeks I have...

  • 86 commits to 4 repositories
  • representing a new library that puts some workflow around JWTs for capability or token based authentication
  • extracting an existing module into its own library
  • converting the authentication scheme of two applications to use the capability tokens
  • build out a Javascript, React, Redux frontend for my health application, comprising of the bulk of my time, and 1000 lines of code
  • multiple interviews with three different companies and a starter interview with a fourth
  • laziness and playing during the 4th of July
  • I got engaged!!!

Poly Trans Lesbian(ish) Triad

Quite the mouthful!

I moved to Boston to be close to my partners, Cait and Leah. For the last year we have all danced around and away from the topic of marriage.

Finally, though, I decided I was ready, and so on the 3rd of July I asked the two of them to marry me. They both said yes.

Obviously we shall have no legal recognition. I don't even have much expectation of religious recognition. The only thing we will be asking for is that the community around us recognize our relationship, support us, and hold us accountable to the vows that we will make.

We will announce the date when we figure it out, but we may not even begin discussing the date until late this year.

Coding without restriction

This is rather a misnomer. I start putting restrictions on my code before I even start typing the first line. I have certain habits and discipline that come from all my years of coding. Some of those habits lead me to immediately start trying to constrain the scope of what I'm writing.

But, in fact, almost everything I worked on over the last few weeks had pre-existing code that I had to work with.

At the same time, I had incredible velocity, primarily because I understood the systems I worked in. I have a history with those systems, having grown some of them since inception. I also have at least a modicum of automated tests on everything except for my APIs. Most importantly, though, since I understand the system I have no fear of breaking unrelated components. I can make changes and deliberately break things, because I either know the code or I have written automated tests that will help me detect flaws.

Obviously, when I start my next job, I need to use absolutely every tool at my disposal to learn the next system I walk into as quickly as possible. No slow "do some projects and learn by osmosis".

  • map every data structure I can find and understand
  • find every API endpoint
  • read every automated test
  • build automated tests as soon as I find untested corners
  • pair with other people on their code
  • have people pair with me on my tasks
  • document anything that feels undocumented

Learning

When I am stressed, when I feel most pressured to get something done, I can't learn. Anything I learn must be in direct service of immediate needs, preferably to the point that I can simply copy-and-paste a new idea in and tweak it a bit to make it fit. This meant that I did not want to take the time to learn anything because I really needed to get the code working and the feature shipped.

When I relax, and have a project in mind, and a bit of extra time, I can learn a lot and move very quickly in my work. I barely knew any Javascript or React, and I knew no Redux, when I started. Yet I shocked a Javascript expert I know when she saw just how far and how quickly I could move.

This should have been obvious. It wasn't. Now I know.

Haskell

Make no mistake, I love Haskell. But I do not necessarily love the Haskell community. I have met some decent people there, but I have also seen bro-level toxicity. I think that Haskell has a lot of elitism around it, and that attracts the kind of human beings I most dislike. As such, since CollegeVine was probably the least toxic Haskell environment I have encountered, I will not seek another Haskell job until I can create it out of a team of junior programmers who are really excited to learn new things.

Moving forward

I am pretty sure that I am done with interviews with one company. I have additional interviews with two others and think that the fourth will not turn into anything. Some of the positions are really compelling. So, I have an exciting future coming up, and I hope to actually have a new employement contract signed before I have been out of work for a full month.

We shall see. But the future looks bright.

Configuring your Haskell application

Posted on Mon Jun 26 19:30:00 UTC 2017

One way or another, you are going to need to configure your Haskell application, and for that you have three major ways of doing it. I recommend choosing one and sticking to it. You can choose multiple ones, but it is important that you minimize one of them in order to keep yourself out of the mind-numbing tedium of consistently combining multiple input parameter sets and their overrides.

Your options tend to be...

  • CLI Option parsing

    I recommend this for small utilities, especially those which you are going to run frequently and with a variety of configurations.

  • Configuration files

    This is generally my preferred way of running an application. You'll still need to do a little bit with option parsing, but only enough to get a configuration. However, it can be a total pain to need to edit a file to change the configuration for a utlity, so use this for your longer-running applications.

  • Environment variables

    This is not generally how I want to configure an application, but some environments, such as Heroku, make it the easiest way.

CLI Option Parsing

The most important rule of parsing options from the CLI is...

*Don't write your own CLI parsing library.*

I have made this mistake. It is no longer on the internet. Do not do what I have done. Do this instead.

For particularly simple parameter parsing, you don't need any libraries. For example I have a tool that I use on occasion to reformat an m3u playlist for my phone. Rhythmbox exports the playlist in an m3u format, but with all paths that don't work for my Android phone. A tool like this is so simple that the only parameters to it are the input file and the output file.

In fact, the tool is so simple that it may have been better for me to accept the input data on standard in and emit the output data on standard out. Please forgive me for that, too.
import           System.Environment (getArgs)


main :: IO ()
main = do
    (source:dest:_) <- getArgs

That is the simplest way. However, you may wish to be kind to your users...

main :: IO ()
main = do
    args <- getArgs
    case args of
        (source:dest:_) -> {- do your thing! -}
        _ -> print "Run the application with the source and destination files."

This is your standby for applications with very simple parameters, and these applications are quite common. However, more complex configuration is often needed. For that, resort to Optparse-Applicative. This will give you command line options that are very similar in power to the one available in Go.

The tutorial covers basically everything, but here's a starter example:

cliParser :: Parser Config
cliParser = Config <$> option auto (long "interval" <> help "number of seconds between samples" <> value 5)
                   <*> strOption (long "log" <> help "log output file")
                   ...

main = do
    Config{..} <- execParser (info (helper <*> cliParser)
                             (fullDesc <> progDesc "description of the program"))

Look here for a summary of the functions and typeclasses involved above. The entire block around execParser is basically boilerplate code, and all of the interesting bits happen inside cliParser.

This technique is as common as mud. As an administrator, I do like to pass parameters to my applications, but I dislike services that require excessively long command lines to run. If your application requires more than four or five parameters, or if the parameters rarely change from one run to the next, look to the next section for configuration files, instead.

Configuration Files

For almost all of my configuration needs, I like to go with a file on the disk. I usually put it into a Yaml format, because that allows some complex nested configurations and saves me from needing to write a configuration parser myself.

For my example, I will demonstrate with a program that I use for my HDR processing toolchain. The program has to go through several steps, and basically it needs these parameters:

  • Do I need to align the photographs?
  • What are my input files?
  • What white balance parameters should I use for developing the files?

and so forth. These are the most important parameters. A typical file looks like this:

wb: camera
project: lake-travis-dam
sources:
- _DSC3656.dng
- _DSC3657.dng
- _DSC3658.dng
- _DSC3659.dng
- _DSC3660.dng
align: false
fanout: false

So, first I want a data structure to store this:

data WhiteBalance = Camera | Auto

data Project = Project {
      sources :: [String]
    , project :: String
    , wb :: WhiteBalance
    , align :: Bool
    , fanout :: Bool
    }
    deriving (Show)


instance Default Project where
    def = Project [] "" Camera False False

(incidentally, I like having defaults for my structures, if I can concieve of a reasonable default)

Whether Yaml or JSON, in Haskell I need a FromJSON instance for parsing this file:

instance FromJSON Project where
    parseJSON (Object obj) =
        Project <$> obj .: "sources"
                <*> obj .: "project"
                <*> obj .: "wb"
                <*> obj .: "align"
                <*> obj .: "fanout"
    parseJSON obj = fail $ show obj

instance FromJSON WhiteBalance where
    parseJSON (String str) =
        case str of
            "camera" -> pure Camera
            "auto" -> pure Auto
            _ -> fail $ "invalid wb string: " ++ T.unpack str
    parseJSON (Object obj) =
        WhiteBalance <$> obj .: "temp"
                     <*> obj .: "green"
    parseJSON obj = fail $ show obj

aside: I use fail instead of mzero or mempty because propogating out any error message at all helps immensely with debugging. I wish I could use throwError, but MonadError is not implemented for Parser.

-- now include code for reading JSON format and Yaml format

Environment Variables

While I do not particularly like using environment variables for configuration an application, Heroku and presumably some other services require their use. On the other hand, most languages treat environment variables as a simple dictionary, making them simple to retrieve. Haskell is no exception to this. The only catch is that nested structures require a little more effort to build.

Your workhorse function is System.Environment.getEnv :: String -> IO String. The function will return the value if present, or throw an IO exception if it is not present. Since you may sometimes want to make the variable optional, so, here is a function that will capture isDoesNotExistError and translate it into a Maybe:

maybeGetEnv :: String -> IO (Maybe String)
maybeGetEnv k = (Just <$> getEnv k) `catch` handleIOExc
    where
    handleIOExc exc
        | isDoesNotExistError exc = pure Nothing
        | otherwise = throw exc

Then write your configuration function like so:

import Data.List.Split (splitOn)

loadConfiguration :: IO Config
loadConfiguration = do
    p <- getEnv "PROJECT_NAME"
    s <- splitOn "," <$> getEnv "SOURCES"
    align <- maybe False read <$> maybeEnv "ALIGN_IMAGES"
    fanout <- maybe False read <$> maybeEnv "FANOUT_EXPOSURES"
    pure $ Config s p Camera align fanout

These are your three major methods for configuring an application. Many applications will permit a certain degree of hybridization between them, but I think it is best to minimize that as much as possible. For instance, a command line parameter to specify the path to a configuration file. Doing it in the general case, handling command line parameters, defaults, configuration options, and environment variables, has typically lead to a very difficult-to-use mess, and I have regretted such attempts.

Whichever method you use for passing configuration in, you'll then want to wrap that configuration up into a context for your application. I will hint more on that in my next article, on the application monad, and give it significantly more detailed treatment later on.


Questions? Comments? Feedback? Email me. I am particularly interested in places that you feel are unclear or which could use better explanation, or experiments you have run that turned out better.