Configuring your Haskell application

Savanni D'Gerinel 26 Jun, 2017

One way or another, you are going to need to configure your Haskell application, and for that you have three major ways of doing it. I recommend choosing one and sticking to it. You can choose multiple ones, but it is important that you minimize one of them in order to keep yourself out of the mind-numbing tedium of consistently combining multiple input parameter sets and their overrides.

Your options tend to be…

  • CLI Option parsing

    I recommend this for small utilities, especially those which you are going to run frequently and with a variety of configurations.

  • Configuration files

    This is generally my preferred way of running an application. You’ll still need to do a little bit with option parsing, but only enough to get a configuration. However, it can be a total pain to need to edit a file to change the configuration for a utlity, so use this for your longer-running applications.

  • Environment variables

    This is not generally how I want to configure an application, but some environments, such as Heroku, make it the easiest way.

CLI Option Parsing

The most important rule of parsing options from the CLI is…

*Don't write your own CLI parsing library.*

I have made this mistake. It is no longer on the internet. Do not do what I have done. Do this instead.

For particularly simple parameter parsing, you don’t need any libraries. For example I have a tool that I use on occasion to reformat an m3u playlist for my phone. Rhythmbox exports the playlist in an m3u format, but with all paths that don’t work for my Android phone. A tool like this is so simple that the only parameters to it are the input file and the output file.

In fact, the tool is so simple that it may have been better for me to accept the input data on standard in and emit the output data on standard out. Please forgive me for that, too.
import           System.Environment (getArgs)


main :: IO ()
main = do
    (source:dest:_) <- getArgs

That is the simplest way. However, you may wish to be kind to your users…

main :: IO ()
main = do
    args <- getArgs
    case args of
        (source:dest:_) -> {- do your thing! -}
        _ -> print "Run the application with the source and destination files."

This is your standby for applications with very simple parameters, and these applications are quite common. However, more complex configuration is often needed. For that, resort to Optparse-Applicative. This will give you command line options that are very similar in power to the one available in Go.

The tutorial covers basically everything, but here’s a starter example:

cliParser :: Parser Config
cliParser = Config <$> option auto (long "interval" <> help "number of seconds between samples" <> value 5)
                   <*> strOption (long "log" <> help "log output file")
                   ...

main = do
    Config{..} <- execParser (info (helper <*> cliParser)
                             (fullDesc <> progDesc "description of the program"))

Look here for a summary of the functions and typeclasses involved above. The entire block around execParser is basically boilerplate code, and all of the interesting bits happen inside cliParser.

This technique is as common as mud. As an administrator, I do like to pass parameters to my applications, but I dislike services that require excessively long command lines to run. If your application requires more than four or five parameters, or if the parameters rarely change from one run to the next, look to the next section for configuration files, instead.

Configuration Files

For almost all of my configuration needs, I like to go with a file on the disk. I usually put it into a Yaml format, because that allows some complex nested configurations and saves me from needing to write a configuration parser myself.

For my example, I will demonstrate with a program that I use for my HDR processing toolchain. The program has to go through several steps, and basically it needs these parameters:

  • Do I need to align the photographs?
  • What are my input files?
  • What white balance parameters should I use for developing the files?

and so forth. These are the most important parameters. A typical file looks like this:

wb: camera
project: lake-travis-dam
sources:
- _DSC3656.dng
- _DSC3657.dng
- _DSC3658.dng
- _DSC3659.dng
- _DSC3660.dng
align: false
fanout: false

So, first I want a data structure to store this:

data WhiteBalance = Camera | Auto

data Project = Project {
      sources :: [String]
    , project :: String
    , wb :: WhiteBalance
    , align :: Bool
    , fanout :: Bool
    }
    deriving (Show)


instance Default Project where
    def = Project [] "" Camera False False

(incidentally, I like having defaults for my structures, if I can concieve of a reasonable default)

Whether Yaml or JSON, in Haskell I need a FromJSON instance for parsing this file:

instance FromJSON Project where
    parseJSON (Object obj) =
        Project <$> obj .: "sources"
                <*> obj .: "project"
                <*> obj .: "wb"
                <*> obj .: "align"
                <*> obj .: "fanout"
    parseJSON obj = fail $ show obj

instance FromJSON WhiteBalance where
    parseJSON (String str) =
        case str of
            "camera" -> pure Camera
            "auto" -> pure Auto
            _ -> fail $ "invalid wb string: " ++ T.unpack str
    parseJSON (Object obj) =
        WhiteBalance <$> obj .: "temp"
                     <*> obj .: "green"
    parseJSON obj = fail $ show obj

aside: I use fail instead of mzero or mempty because propogating out any error message at all helps immensely with debugging. I wish I could use throwError, but MonadError is not implemented for Parser.

-- now include code for reading JSON format and Yaml format

Environment Variables

While I do not particularly like using environment variables for configuration an application, Heroku and presumably some other services require their use. On the other hand, most languages treat environment variables as a simple dictionary, making them simple to retrieve. Haskell is no exception to this. The only catch is that nested structures require a little more effort to build.

Your workhorse function is System.Environment.getEnv :: String -> IO String. The function will return the value if present, or throw an IO exception if it is not present. Since you may sometimes want to make the variable optional, so, here is a function that will capture isDoesNotExistError and translate it into a Maybe:

maybeGetEnv :: String -> IO (Maybe String)
maybeGetEnv k = (Just <$> getEnv k) `catch` handleIOExc
    where
    handleIOExc exc
        | isDoesNotExistError exc = pure Nothing
        | otherwise = throw exc

Then write your configuration function like so:

import Data.List.Split (splitOn)

loadConfiguration :: IO Config
loadConfiguration = do
    p <- getEnv "PROJECT_NAME"
    s <- splitOn "," <$> getEnv "SOURCES"
    align <- maybe False read <$> maybeEnv "ALIGN_IMAGES"
    fanout <- maybe False read <$> maybeEnv "FANOUT_EXPOSURES"
    pure $ Config s p Camera align fanout

These are your three major methods for configuring an application. Many applications will permit a certain degree of hybridization between them, but I think it is best to minimize that as much as possible. For instance, a command line parameter to specify the path to a configuration file. Doing it in the general case, handling command line parameters, defaults, configuration options, and environment variables, has typically lead to a very difficult-to-use mess, and I have regretted such attempts.

Whichever method you use for passing configuration in, you’ll then want to wrap that configuration up into a context for your application. I will hint more on that in my next article, on the application monad, and give it significantly more detailed treatment later on.


Questions? Comments? Feedback? Email me. I am particularly interested in places that you feel are unclear or which could use better explanation, or experiments you have run that turned out better.

Creative Commons License
Configuring your Haskell application by Savanni D’Gerinel is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Resistance

Savanni D'Gerinel 17 Jan, 2017

For the next ten days, my resistance against fascism is to sit with an enby who has just undergone surgery. The kind of person and surgery that self-proclaimed “decent folk” believe is an abomination worthy of death.

If you are one of them, fuck off.

A Quiet Walk, Interrupted

Ice Flow, 2017-01.web

Yesterday, my friend and I were out walking along the Delaware Canal Towpath in Morissville, Pennsylvania, having a lovely day. Part of the canal is still frozen, and we made a hobby of seeing if we could break the ice using whatever rocks we had to hand. The sound of an ice sheet fracturing is really unique. Not precisely like glass shattering, because the fracture and stresses race down the ice sheet, even if the break only happens in a small area. I lost count of the number of significant rocks that I hurled into the ice that simply got embedded.

We are pretty much minding our own business when an old man walking the other direction starts demanding to know where we’re going “dressed like that” and telling us “you’re not real girls”. He proceeded to hurl invectives after us for long after we could hear him. He even proclaimed just how great it must be to have wealthy parents who will support my “lifestyle”.

When I was twenty, I clawed my way nearly to financial independence to be free of my parents rules. I’ve been on my own ever since. I wanted so much to walk up and tell him precisely how much I get paid for my skills at a job that keeps me in air conditioning. The idea of shaming him into silence was almost overpowering. I am a professional in my field, not even 40, nearly at the top, and I make more money than he likely ever has.

How pathetic. Long after we could no longer hear him, we could see him still yelling at us. He had nothing better to do with his sorry excuse for a life.

It is very difficult to walk away silently. I cannot help but feel that I accomplished nothing. That there is no victory to the high road. But that perhaps there is nothing to be accomplished in my reply.

I so rarely face actual transphobia on the streets. So rarely that I vacillate between being shocked when it happens, and shocked that nobody even looks sidelong at me, even when I’m in a small town. For the rest of our walk, we kept our eyes out. We have no idea what the old man may have done. Perhaps he called the cops on us, as has happened so often to people like us. If he did, they ignored him and we continued the rest of our day unmolested.

Decent Folk

So often, the narrative is that we are a threat to “Decent Folk”. Somehow, trans queer reality is so powerful that all wholesome goodness breaks down around us.

Decent Folk assault people on the street.

Decent Folk poke their noses into other people’s private lives.

Decent Folk remain silent when death camps rise.

Decent Folk vote for a fascist, rapist, traitor because he pedals lies of prosperity.

Decent Folk vote for one who promises more power to those with power.

Decent Folk are so easily duped with fascist lies.

Defiance

What is the queer agenda? STOP HURTING OUR FRIENDS! STOP HURTING OUR CHILDREN! LET US LIVE HEALTHY PRODUCTIVE LIVES IN PEACE!

We are the ones who take in those not our kin. Teenagers thrown out of their homes for being gay, lesbian, transgender, bisexual, asexual, intersex, polyamorous. Strangers moving into a city where they can be safer.

If we could, we would walk away from your fucking “decent” culture. We would separate ourselves and build a civilization of our own. We would interact with you only to rescue the queers who emerge amongst you.

But you won’t permit that. You “Decent Folk” have all the power. So fuck you.

16 Years

Savanni D'Gerinel 1 Jan, 2017

Welcome to 2017!

In a few days I move away from Austin, likely never to return. I grew up in Round Rock. I went away for a few years, and then I returned here to begin my career and my adult life right during the dot-com crash. I’m actually a lot older than I look. Many people upon meeting me seem to assume that I’m 27 or so, when in fact I turned 38 late this year. “Wait, how old are you?”

I keenly feel the passage of time. I feel that I have not begun to approach what I wanted to accomplish by now. But, realistically, I have between 40 and 60 years left. The amount of time that I spent here… three more times. And a lot can change in 16 years.

16 years ago, I thought I was a straight man. I was married. I voted for Bush and thought the Republicans could run the country well. I was Catholic and believed the anti-abortion rhetoric, yet I somehow rejected the anti-gay rhetoric. Go figure. Though we knew a few gay men in college, it was shortly after we moved to Austin that my wife and I noticed for the first time pairs of men openly holding hands at formal “respectable” events. We began to feel a relief that this was the kind of safe city that we never really recognized we sought.

15 years ago, my wife and I decided to have a polyamorous relationship. She said that I had suggested it years earlier while we were dating. I did not remember saying that, but it felt like the kind of thing I might have. It was shortly after this, as I thought about love, romance, and relationships, that I began to believe that it was tragic that I was straight and not bisexual. I can remember being apologetic as I (very occasionally) turned a man down. And it was shortly after this that I understood that I was parting ways with the Catholic church… and I did not particularly regret that.

14 years ago, I found out how infidelity felt. Infidelity in a polyamorous relationship looks different than in a monogamous relationship, but it hurts the same. It cuts through hearts, rips out rugs, and crushes dreams.

I also learned that maybe the Republican party was actually made up of a bunch of chronic liars, and became a Democrat. Later I started to understand how violent and hateful Republicans could be. How did I never see this before? And maybe, just maybe, I shouldn’t hold the reproductive health doctrines of men who want to ban abortion but also ban all other forms of contraception and all forms of sex that carry no chance of pregnancy while simultaneously starting a war and lying to me about weapons of mass destruction!

12 years ago I joined a company that became my career for the better part of a decade. They weren’t great… in fact sometimes they were downright awful, but over time my authority became vast, as did my knowledge of everything about the business… except what was in the best interest of the business. Ya know, sometimes we techies need to be informed of the big business direction so we can make decisions intelligently.

10 years ago, with the onset of Saturn Returns, I finally figured out that I was not a man. That moment has lead me through so many changes and to so many of the people that I find so important in my life now. As a man, I would never have made any of the connections I have as an androgyne. This realization sometimes keeps me awake at night, knowing that it is by the grace of but a few words that I have in my life the love that I experience now. More rationally, a few of my current friends would have been my friends anyway, and they would have noticed my egg tendencies, and they would have aided in my hatching. I may have ended up exactly where I am now, on a different schedule.

Letting go of my own gender also let me release my expectations about my sexual orientation. Reparative therapy, especially religious-based “therapy”, is bullshit. We know this. And yet, I successfully “prayed the straight away”!

I also gave up on “til death do we part” and let my marriage end.

Five years ago, I learned photography, and I changed how I see the world. Always watching for that perfect moment. Seeing textures. Analyzing light. Understanding focus and freezing motion. The speckled shadow beneath a canopy. The shimmer of a cobweb five meters up and at least that far away.

Three years ago, I talked myself out of my first suicide attempt. In the aftermath, I evaluated my life. I saw clearly how I was wasting it on my employer’s amazingly small dreams, and I chose to spend some time quite alone. I loved living out in the woods. I hated having to drive for twenty minutes to reach the closest decent internet connection, and for an hour to reach any of my friends. But there is a lot to be said for the peace of the forest, for stars so bright as to light the ground, for rain on the metal roof a mere meter from my lofted bed… and for really cheap rent paid in cash under the table. Oh, and did I mention that my landlady also covered electricty? Pretty epic, especially since the cabin wasn’t well insulated and I had to run 2.5kW of heating that winter.

In the last two years, I have truly started to learn how black lives matter, and how little I understood my own racism in the past. I have learned about social justice, and become keenly aware of my failings. I have gained true confidence in my skills, and become comfortable in my body for the first time in my life. I have felt my socialist/anarchist heart begin to blossom as I notice the Democratic party repeatedly snatch defeat from the jaws of victory.

And, shortly after my birthday in 2014, I met the woman who has become the love of my life. She had to exercise both persistence and patience. I was wounded and avoiding romance, sex, dating. She had to convince me that a lesbian, even a trans-friendly lesbian, could be interested in an androgyne who still had and wasn’t particularly inclined to get rid of eir penis. But, she exercised that persistence, and she waited patiently, while over the course of months I fell in love and I healed. Now we talk of our sixty year plan.

I will miss Austin. I will miss the people here. I will miss all of my bike routes and the restaurants and the events. I will miss the familiarity. And I feel guilt, leaving all of you to stay and stand against the legislature.

But for this woman, where she goes my heart shall follow.

Nix Development Environments

Savanni D'Gerinel 28 Nov, 2016

nix-shell, the command that creates a subshell after evaluating any nix expression, has a lot of uses. I found it very useful in my devops work when I had multiple environments to administer, but had to use different tools for each. The shell provides excellent help in isolating my required tools to the environments involved.

A trick, though, lay in learning how to acquire those tools when the tools were not available in the nixos channel. I figured it out, and so here is the example for one of the environments I was administering. Note that I include both Linux and Darwin builds, because I wanted to offer the nix environment to my replacement at the company.

  • Packer – 0.10.1
  • Terraform – 0.7.4
  • Ansible 2
  • Python 2.7

We were deploying in Amazon AWS. I used Packer to build the custom images that we were deploying. Autoscaling works a lot better if it has a complete image that only has to be started (the Crops, i.e., the systems that can be replaced almost instantly and thus do get replaced regularly). I love Terrafrom because I was able to describe everything I was doing in AWS using the tool. Ansible is present for those systems that get reconfigured regularly (primarily the Cattle machines, things that can be rebuilt from just the devops scripts, but that I do not want to terminate). Python 2.7 is present to support Ansible, though it is sometimes convenient to have at hand.

Neither Packer nor Terraform were available in my Nix channel, so I had to build derivations for those. The process is non-obvious until it is done. Here are my scripts for them. At the time I wrote these scripts, I was running NixOS 16.03, however I still use the same scripts after having upgraded to NixOS 16.09.

nix-deps/packer.nix

{ pkgs ? import <nixpkgs> {},
  stdenv ? pkgs.stdenv }:

let
  # suggestion from @clever of #nixos
  package =
         if stdenv.system == "x86_64-linux" then "packer_0.10.1_linux_amd64.zip"
    else if stdenv.system == "x86_64-darwin" then "packer_0.10.1_darwin_amd64.zip"
    else abort "unsupported platform";
  checksum =
         if stdenv.system == "x86_64-linux" then "7d51fc5db19d02bbf32278a8116830fae33a3f9bd4440a58d23ad7c863e92e28"
    else if stdenv.system == "x86_64-darwin" then "fac621bf1fb43f0cbbe52481c8dfda2948895ad52e022e46f00bc75c07a4f181"
    else abort "unsupported platform";
in
stdenv.mkDerivation rec {
  name = "packer-${version}";
  version = "0.10.1";

  buildCommand = ''
  mkdir -p $out/bin
  unzip $src
  mv packer $out/bin/packer
  echo Installed packer to $out/bin/packer
  '';

  src = pkgs.fetchurl {
    url = "https://releases.hashicorp.com/packer/0.10.1/${package}";
    sha256 = checksum;
    name = package;
  };

  buildInputs = [ pkgs.unzip ];
}

nix-deps/terraform.nix

{ pkgs ? import <nixpkgs> {},
  stdenv ? pkgs.stdenv }:

let
  # suggestion from @clever of #nixos
  package =
         if stdenv.system == "x86_64-linux" then "terraform_0.7.4_linux_amd64.zip"
    else if stdenv.system == "x86_64-darwin" then "terraform_0.7.4_darwin_amd64.zip"
    else abort "unsupported platform";
  checksum =
         if stdenv.system == "x86_64-linux" then "8950ab77430d0ec04dc315f0d2d0433421221357b112d44aa33ed53cbf5838f6"
    else if stdenv.system == "x86_64-darwin" then "21c8ecc161628ecab88f45eba6b5ca1fbf3eb897e8bc951b0fbac4c0ad77fb04"
    else abort "unsupported platform";
in
stdenv.mkDerivation rec {
  name = "terraform-${version}";
  version = "0.7.4";

  buildCommand = ''
  mkdir -p $out/bin
  unzip $src
  mv terraform $out/bin/terraform
  echo Installed terraform to $out/bin/terraform
  '';

  src = pkgs.fetchurl {
    url = "https://releases.hashicorp.com/terraform/0.7.4/${package}";
    sha256 = checksum;
    name = package;
  };

  buildInputs = [ pkgs.unzip ];
}

The structure of each script is relatively straightforward.

  • declare that pkgs and stdenv are both required, as well as how to get them if they are absent
  • based on the OS, declare what package I want to download and the relevant checksum
  • declare the name and version of the derivation
  • create the custom build command

    In many cases, the default build commands works perfectly, but that only works for projects that have to be built with autoconfig or with Stack (and possibly some other languages). Both Terraform and Packer are binaries, and so it is necessary for me to specify the build for the derivation.

    In this case, the build is simply to unzip the downloaded package (specified in $src) and copy the executable into the destination (which has a root at $out). It is vital that the executable end up in the bin/ directory. I am not sure of the mandated directory structure of a derivation, but I know that derivations that did not include the bin/ directory would fail. I assume that they failed because there was no executable to add to the path.

  • specify precisely how to get the source package. In this case, through the fetchurl tool.
  • specify additional build inputs. These have to be somewhere in the nix namespace. pkgs.unzip refers to nixpkgs.unzip in the standard channel.

Both of the files above must go in a subdirectory. I named the subdirectory nix-deps/. Some subtle interaction will cause an infinite recursion if the two files are included in the root directory of your project.

With those present, it is time to build the nix-shell command:

shell.nix

let
  pkgs = import <nixpkgs> {};
  stdenv = pkgs.stdenv;
  terraform = import nix-deps/terraform.nix {};
  packer = import nix-deps/packer.nix {};

in stdenv.mkDerivation {
  name = "v2-devops";

  buildInputs = [ pkgs.ansible2
                  terraform
                  packer
                  pkgs.python
                  pkgs.python27Packages.alembic
                  pkgs.python27Packages.boto
                  pkgs.python27Packages.psycopg2
                  pkgs.awscli
                ];
}

The only difficult part here was for me to figure out how to import my Terraform and Packer derivations. I handle that with the import nix-deps/<package>.nix {} lines. The result of each import statement is a derivation, and so it is valid to include in buildInputs.

buildInputs again just lists the packages that must be included in this derivation. So, I included all of the packages that I use directly.

Thus, from the root directory of my devops folder, I can simply run nix-shell and have exactly the version of Terraform, Packer, Ansible, and Python that I want. This also means that I can have completely different versions for a different devops repository (I was actually administering three different clouds, all with different standards). And, possibly best of all, if I could convince my co-workers to use Nix (the tool, not the operating system), they would have had a trivial way to set up their development environments, also.

Androgyne with Surgery

Savanni D'Gerinel 2 Nov, 2016

I did not sleep well the night before. Thoughts of missing my alarm plague me the night before an important but extremely early event. Most of the time I have to be moving for a flight, but this day I needed to be at the surgical center at 5am.

_DSC2317.web

Daria had it under control. Zie woke me at 4. We were out the door at 4:30, and we arrived at the surgical center at 4:50. After that, I put my fate into everyone else’s hands as they guided me into the building and prepared my IV. Two different doctors, first my surgeon then the anesthesiologist. Strangely, the anesthesiologist did not know about spironolactone and left with the words “I have some research to do.” A few minutes later, the nurse took me to the operating room, helped me get positioned on the table, said she would go find another electric blanket for me, and then I woke up in my bed at home.

IMG_5124

I think no experience will ever match that for strangeness. I have seen pictures of “waking up”, and of getting myself into my car, but I remember absolutely nothing else.

I saw myself in the mirror some hours later. As soon as I felt up for doing so, I undressed to look in the mirror, and I nearly started crying. I felt relief beyond what I imagined, with the feeling that my body, finally, was my own.

20160911_104525

I cannot remember a time that I did not want breasts. Through all of medical transition, though, I have had to constantly ask myself whether I am an androgyne or a trans woman. Everything about my transition looks like what a trans woman would do, except that I did it slowly, and in a different order. At it’s core, how much can I alter my body before I cross into being a woman?

IMG_5319

When Washburn passed a few weeks ago, I understood in my grief that the term “mother” applies to me. “Mother” is a job. Not a gender. And it has nothing to do with biology. Even I find myself harboring ideas of biological essentialism, and even I must take time to expunge those ideas.

My transition has always been about giving myself the body I want, the one I can feel comfortable in. The things I have done to my body do not change my gender, but they have made me much more comfortable in my body.

I still do not have a gender. I am an androgyne, and I am proud of that.

_DSC2662


Dreamer, Shaper, Seeker, Maker