Haskell Application Monad

Savanni D'Gerinel 14 Jul, 2017

We want to get productive in Haskell very quickly. Most non-trivial applications will have configuration, connections to the outside world, can hit exceptional conditions, and benefit from having their operations logged. If your application has sensible logs at both high an low levels of detail, your devops team will thank you and your life of debugging a production application will be a happier one.

I want to get all of these things at once, and so it would be nice to provide a nearly boilerplate application stack that provides them all. I define the “application stack” as a group of attributes that contain the context and all of the common behaviors for an application. In Haskell, you do that with a monad stack, though work on extensible effects shows a great deal of promise and has been used to great effect in Purescript.

That said, I use monads and monad transformers, and I’ll not explain either of them today. I feel that the best explanation is a non-trivial example implementation, which I will do in a future article, or refer you to a better tutorial.

While most of this article explains the process, the final result is this application stack, which may be all you need if you are already familiar with building monad transformer stacks.

data Context = Context { contextRoot :: FilePath } deriving Show

data AppError = AppError deriving Show

newtype AppM a = AppM (LoggingT (ReaderT Context (ExceptT AppError IO)) a)
    deriving ( Functor, Applicative, Monad, MonadIO
             , MonadError AppError, MonadReader Context, MonadLogger)

runAppM :: Context -> AppM a -> IO (Either AppError a)
runAppM ctx (AppM act) = runExceptT (runReaderT (runStderrLoggingT act) ctx)

The most basic stack

Almost every application needs IO. In Haskell it is difficult to do IO on top of anything (see MonadBaseControl for way), so I always put it at the bottom of the monad stack. A trivial application stack would look like this:

newtype AppM a = AppM (IO a) deriving (Functor, Applicative, Monad, MonadIO)

This is so trivial you will likely never do it, though it can be helpful in that it prevents confusion between your functions and system IO functions. Still, let’s build out what you need to make this work.

First of all, you do want AppM to be a monad, and you will need MonadIO in order to actually run IO operations. The primary use that I have for Monads in an application is to eliminate the boilerplate involved with a lot of threading context through a series of function calls. More to the point, though, you cannot get MonadExcept, MonadReader, or MonadLogger into this stack without having Monad to begin with.

newtype AppM a = AppM (IO a)
    deriving (Functor, Applicative, Monad, MonadIO)

runAppM :: AppM a -> IO a
runAppM (AppM act) = act

runAppM is the function that connects your application stack to the Haskell IO stack. This is everything you need in order to create a stack: the stack itself and the runner. Now let’s see it in action:

data Image = Image deriving Show

loadImage :: FilePath -> AppM Image
loadImage path = do 
    liftIO $ putStrLn $ "loadImage: " <> path
    pure Image
     

main :: IO ()
main = do
    res <- runAppM $ do
        img1 <- loadImage "image.png"
        img2 <- loadImage "image2.png"
        pure (img1, img2)
    print res

Injecting your context

IO a is too simple to make much sense. The whole point of having a stack is to unify a lot of effects within a common framework of behavior and with a common context. So, next we load and add a context.

In almost every circumstance, your context is read-only. This points us directly to ReaderT, since you will want to be able to ask for the context but never write back to it. Application state would seem like a thing that you would want to include, if your application stores state. I have generally found that it is easier to keep application state in something that is strictly IO, such as an IORef or a TVar. For now, we shall skip that.

So, change your stack to look like this:

data Context = Context { root :: FilePath } deriving Show

newtype AppM a = AppM (ReaderT Context IO a)
    deriving (Functor, Applicative, Monad, MonadIO, MonadReader Context)

runAppM :: Context -> AppM a -> IO a
runAppM ctx (AppM act) = runReaderT act ctx

The addition of MonadReader means that now you can call ask within your function to get back the context, and you don’t have to explicitely pass the context in. The remaining functions get updated like so:

loadImage :: FilePath -> AppM Image
loadImage path = do
    Context{..} <- ask
    liftIO $ putStrLn $ "loadImage: " <> (contextRoot </> path)
    pure Image

loadContext :: IO Context
loadContext = pure $ Context { contextRoot = "/home/savanni/Pictures/" }

main :: IO ()
main = do
    ctx <- loadContext
    res <- runAppM ctx $ do
        img1 <- loadImage "image.png"
        img2 <- loadImage "image2.png"
        pure (img1, img2)
    print res

Suddenly, everything in Context is available to every function that runs in AppM. You get the local effect of global parameters while still getting to isolate them, potentially calling the same functions with different contexts within the same application.

Add exception handling and logging

Exceptions happen. The Haskell community is split between what I call explicit vs. implicit exceptions. In short, implicit exceptions are not declared in the type signature, can happen from any function, and can only be caught in IO code. Explicit exceptions are explicitely stated in the type signature and can be caught just about anywhere. I prefer them for all of my application errors. I’ll give exception handling further treatment in a future article, and will show the use of explicit exceptions here.

Logging is almost always helpful for any application that is not of trivial size. And, once present, it can replace print for debugging, allowing debugging lines to remain present in the code for those cases when something starts going wrong in production.

First, the new application stack:

data AppError = AppError deriving Show

newtype AppM a = AppM (LoggingT (ReaderT Context (ExceptT AppError IO)) a)
    deriving ( Functor, Applicative, Monad, MonadIO
             , MonadError AppError, MonadReader Context, MonadLogger)

runAppM :: Context -> AppM a -> IO (Either AppError a)
runAppM ctx (AppM act) = runExceptT (runReaderT (runStderrLoggingT act) ctx)

This gets quite a bit more complicated with both the Logging and Exceptions being added. Remember that I use the term “stack” here, and each monad transformer involved represents another layer in the stack. When running the stack, you must peel off each layer in reverse order. I will illustrate with some types:

*Json> :t loadImage "img.png"
loadImage "img.png" :: AppM Image

*Json> :t unAppM $ loadImage "img.png"
unAppM $ loadImage "img.png"
  :: LoggingT (ReaderT Context (ExceptT AppError IO)) Image

*Json> :t runStderrLoggingT $ unAppM $ loadImage "img.png"
runStderrLoggingT $ unAppM $ loadImage "img.png"
  :: ReaderT Context (ExceptT AppError IO) Image

*Json> :t runReaderT (runStderrLoggingT $ unAppM $ loadImage "img.png") ctx
runReaderT (runStderrLoggingT $ unAppM $ loadImage "img.png") ctx
  :: ExceptT AppError IO Image

*Json> :t runExceptT $ runReaderT (runStderrLoggingT $ unAppM $ loadImage "img.png") ctx
runExceptT $ runReaderT (runStderrLoggingT $ unAppM $ loadImage "img.png") ctx
  :: IO (Either AppError Image)

The point of this is that in runAppM, the type of act is the entire stack, and the first thing to be called to begin unwrapping is runStderrLoggingT, then runReaderT, and finally runExceptT.

Notice, also, that the final type of runAppM has changed to IO (Either AppError a). runAppM will now return whatever exception gets thrown from within the context it is running, no matter where that exception is thrown, if that exception is thrown with throwException. Exceptions thrown with throw end up being the implicit exceptions I referred to, and those require some extra handling.

So, here is the rest of the code. In the places where I used to print output, I am now logging output. Note that the loggers require TemplateHaskell and have slightly odd syntax, but are otherwise nearly identical to print.

data Image = Image deriving Show

loadImage :: FilePath -> AppM Image
loadImage path = do
    Context{..} <- ask
    $(logInfo) (T.pack $ "loadImage: " <> (contextRoot </> path))
    pure Image

loadContext :: IO Context
loadContext = pure $ Context { contextRoot = "/home/savanni/Pictures/" }

main :: IO ()
main = do
    ctx <- loadContext
    do  res <- runAppM ctx $ do
            img1 <- loadImage "image.png"
            img2 <- loadImage "image2.png"
            pure (img1, img2)
        print res

    do  res <- runAppM ctx $ do
            img1 <- loadImage "image.png"
            throwError AppError
            img2 <- loadImage "image2.png"
            pure (img1, img2)
        print res 

This is the output from running main:

*Json> main
[Info] loadImage: /home/savanni/Pictures/image.png @(main:Json /home/savanni/src/haskell/src/Json.hs:76:7)
[Info] loadImage: /home/savanni/Pictures/image2.png @(main:Json /home/savanni/src/haskell/src/Json.hs:76:7)
Right (Image,Image)
[Info] loadImage: /home/savanni/Pictures/image.png @(main:Json /home/savanni/src/haskell/src/Json.hs:76:7)
Left AppError
*Json> 

So, the first block starting with do res <- runAppM runs to completion, returnin two images. The second block, runs loadImage for the first image, but then hits throwError and returns Left AppError, discarding the first image and not loading the second image at all.


This is nearly a application stack that I have used for more applications than I can count. Even if you need only one feature, such as exceptions, starting with a small stack hidden behind an application monad makes it very easy to add additional features as you need them, without needing to change the rest of your code. This pattern is trivial to extend, or contract, as needed, and so I think it starts every application on a good path.

Creative Commons License
Haskell Application Monad by Savanni D’Gerinel is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Configuring your Haskell application

Savanni D'Gerinel 26 Jun, 2017

One way or another, you are going to need to configure your Haskell application, and for that you have three major ways of doing it. I recommend choosing one and sticking to it. You can choose multiple ones, but it is important that you minimize one of them in order to keep yourself out of the mind-numbing tedium of consistently combining multiple input parameter sets and their overrides.

Your options tend to be…

  • CLI Option parsing

    I recommend this for small utilities, especially those which you are going to run frequently and with a variety of configurations.

  • Configuration files

    This is generally my preferred way of running an application. You’ll still need to do a little bit with option parsing, but only enough to get a configuration. However, it can be a total pain to need to edit a file to change the configuration for a utlity, so use this for your longer-running applications.

  • Environment variables

    This is not generally how I want to configure an application, but some environments, such as Heroku, make it the easiest way.

CLI Option Parsing

The most important rule of parsing options from the CLI is…

*Don't write your own CLI parsing library.*

I have made this mistake. It is no longer on the internet. Do not do what I have done. Do this instead.

For particularly simple parameter parsing, you don’t need any libraries. For example I have a tool that I use on occasion to reformat an m3u playlist for my phone. Rhythmbox exports the playlist in an m3u format, but with all paths that don’t work for my Android phone. A tool like this is so simple that the only parameters to it are the input file and the output file.

In fact, the tool is so simple that it may have been better for me to accept the input data on standard in and emit the output data on standard out. Please forgive me for that, too.
import           System.Environment (getArgs)


main :: IO ()
main = do
    (source:dest:_) <- getArgs

That is the simplest way. However, you may wish to be kind to your users…

main :: IO ()
main = do
    args <- getArgs
    case args of
        (source:dest:_) -> {- do your thing! -}
        _ -> print "Run the application with the source and destination files."

This is your standby for applications with very simple parameters, and these applications are quite common. However, more complex configuration is often needed. For that, resort to Optparse-Applicative. This will give you command line options that are very similar in power to the one available in Go.

The tutorial covers basically everything, but here’s a starter example:

cliParser :: Parser Config
cliParser = Config <$> option auto (long "interval" <> help "number of seconds between samples" <> value 5)
                   <*> strOption (long "log" <> help "log output file")
                   ...

main = do
    Config{..} <- execParser (info (helper <*> cliParser)
                             (fullDesc <> progDesc "description of the program"))

Look here for a summary of the functions and typeclasses involved above. The entire block around execParser is basically boilerplate code, and all of the interesting bits happen inside cliParser.

This technique is as common as mud. As an administrator, I do like to pass parameters to my applications, but I dislike services that require excessively long command lines to run. If your application requires more than four or five parameters, or if the parameters rarely change from one run to the next, look to the next section for configuration files, instead.

Configuration Files

For almost all of my configuration needs, I like to go with a file on the disk. I usually put it into a Yaml format, because that allows some complex nested configurations and saves me from needing to write a configuration parser myself.

For my example, I will demonstrate with a program that I use for my HDR processing toolchain. The program has to go through several steps, and basically it needs these parameters:

  • Do I need to align the photographs?
  • What are my input files?
  • What white balance parameters should I use for developing the files?

and so forth. These are the most important parameters. A typical file looks like this:

wb: camera
project: lake-travis-dam
sources:
- _DSC3656.dng
- _DSC3657.dng
- _DSC3658.dng
- _DSC3659.dng
- _DSC3660.dng
align: false
fanout: false

So, first I want a data structure to store this:

data WhiteBalance = Camera | Auto

data Project = Project {
      sources :: [String]
    , project :: String
    , wb :: WhiteBalance
    , align :: Bool
    , fanout :: Bool
    }
    deriving (Show)


instance Default Project where
    def = Project [] "" Camera False False

(incidentally, I like having defaults for my structures, if I can concieve of a reasonable default)

Whether Yaml or JSON, in Haskell I need a FromJSON instance for parsing this file:

instance FromJSON Project where
    parseJSON (Object obj) =
        Project <$> obj .: "sources"
                <*> obj .: "project"
                <*> obj .: "wb"
                <*> obj .: "align"
                <*> obj .: "fanout"
    parseJSON obj = fail $ show obj

instance FromJSON WhiteBalance where
    parseJSON (String str) =
        case str of
            "camera" -> pure Camera
            "auto" -> pure Auto
            _ -> fail $ "invalid wb string: " ++ T.unpack str
    parseJSON (Object obj) =
        WhiteBalance <$> obj .: "temp"
                     <*> obj .: "green"
    parseJSON obj = fail $ show obj

aside: I use fail instead of mzero or mempty because propogating out any error message at all helps immensely with debugging. I wish I could use throwError, but MonadError is not implemented for Parser.

-- now include code for reading JSON format and Yaml format

Environment Variables

While I do not particularly like using environment variables for configuration an application, Heroku and presumably some other services require their use. On the other hand, most languages treat environment variables as a simple dictionary, making them simple to retrieve. Haskell is no exception to this. The only catch is that nested structures require a little more effort to build.

Your workhorse function is System.Environment.getEnv :: String -> IO String. The function will return the value if present, or throw an IO exception if it is not present. Since you may sometimes want to make the variable optional, so, here is a function that will capture isDoesNotExistError and translate it into a Maybe:

maybeGetEnv :: String -> IO (Maybe String)
maybeGetEnv k = (Just <$> getEnv k) `catch` handleIOExc
    where
    handleIOExc exc
        | isDoesNotExistError exc = pure Nothing
        | otherwise = throw exc

Then write your configuration function like so:

import Data.List.Split (splitOn)

loadConfiguration :: IO Config
loadConfiguration = do
    p <- getEnv "PROJECT_NAME"
    s <- splitOn "," <$> getEnv "SOURCES"
    align <- maybe False read <$> maybeEnv "ALIGN_IMAGES"
    fanout <- maybe False read <$> maybeEnv "FANOUT_EXPOSURES"
    pure $ Config s p Camera align fanout

These are your three major methods for configuring an application. Many applications will permit a certain degree of hybridization between them, but I think it is best to minimize that as much as possible. For instance, a command line parameter to specify the path to a configuration file. Doing it in the general case, handling command line parameters, defaults, configuration options, and environment variables, has typically lead to a very difficult-to-use mess, and I have regretted such attempts.

Whichever method you use for passing configuration in, you’ll then want to wrap that configuration up into a context for your application. I will hint more on that in my next article, on the application monad, and give it significantly more detailed treatment later on.


Questions? Comments? Feedback? Email me. I am particularly interested in places that you feel are unclear or which could use better explanation, or experiments you have run that turned out better.

Creative Commons License
Configuring your Haskell application by Savanni D’Gerinel is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Resistance

Savanni D'Gerinel 17 Jan, 2017

For the next ten days, my resistance against fascism is to sit with an enby who has just undergone surgery. The kind of person and surgery that self-proclaimed “decent folk” believe is an abomination worthy of death.

If you are one of them, fuck off.

A Quiet Walk, Interrupted

Ice Flow, 2017-01.web

Yesterday, my friend and I were out walking along the Delaware Canal Towpath in Morissville, Pennsylvania, having a lovely day. Part of the canal is still frozen, and we made a hobby of seeing if we could break the ice using whatever rocks we had to hand. The sound of an ice sheet fracturing is really unique. Not precisely like glass shattering, because the fracture and stresses race down the ice sheet, even if the break only happens in a small area. I lost count of the number of significant rocks that I hurled into the ice that simply got embedded.

We are pretty much minding our own business when an old man walking the other direction starts demanding to know where we’re going “dressed like that” and telling us “you’re not real girls”. He proceeded to hurl invectives after us for long after we could hear him. He even proclaimed just how great it must be to have wealthy parents who will support my “lifestyle”.

When I was twenty, I clawed my way nearly to financial independence to be free of my parents rules. I’ve been on my own ever since. I wanted so much to walk up and tell him precisely how much I get paid for my skills at a job that keeps me in air conditioning. The idea of shaming him into silence was almost overpowering. I am a professional in my field, not even 40, nearly at the top, and I make more money than he likely ever has.

How pathetic. Long after we could no longer hear him, we could see him still yelling at us. He had nothing better to do with his sorry excuse for a life.

It is very difficult to walk away silently. I cannot help but feel that I accomplished nothing. That there is no victory to the high road. But that perhaps there is nothing to be accomplished in my reply.

I so rarely face actual transphobia on the streets. So rarely that I vacillate between being shocked when it happens, and shocked that nobody even looks sidelong at me, even when I’m in a small town. For the rest of our walk, we kept our eyes out. We have no idea what the old man may have done. Perhaps he called the cops on us, as has happened so often to people like us. If he did, they ignored him and we continued the rest of our day unmolested.

Decent Folk

So often, the narrative is that we are a threat to “Decent Folk”. Somehow, trans queer reality is so powerful that all wholesome goodness breaks down around us.

Decent Folk assault people on the street.

Decent Folk poke their noses into other people’s private lives.

Decent Folk remain silent when death camps rise.

Decent Folk vote for a fascist, rapist, traitor because he pedals lies of prosperity.

Decent Folk vote for one who promises more power to those with power.

Decent Folk are so easily duped with fascist lies.

Defiance

What is the queer agenda? STOP HURTING OUR FRIENDS! STOP HURTING OUR CHILDREN! LET US LIVE HEALTHY PRODUCTIVE LIVES IN PEACE!

We are the ones who take in those not our kin. Teenagers thrown out of their homes for being gay, lesbian, transgender, bisexual, asexual, intersex, polyamorous. Strangers moving into a city where they can be safer.

If we could, we would walk away from your fucking “decent” culture. We would separate ourselves and build a civilization of our own. We would interact with you only to rescue the queers who emerge amongst you.

But you won’t permit that. You “Decent Folk” have all the power. So fuck you.

16 Years

Savanni D'Gerinel 1 Jan, 2017

Welcome to 2017!

In a few days I move away from Austin, likely never to return. I grew up in Round Rock. I went away for a few years, and then I returned here to begin my career and my adult life right during the dot-com crash. I’m actually a lot older than I look. Many people upon meeting me seem to assume that I’m 27 or so, when in fact I turned 38 late this year. “Wait, how old are you?”

I keenly feel the passage of time. I feel that I have not begun to approach what I wanted to accomplish by now. But, realistically, I have between 40 and 60 years left. The amount of time that I spent here… three more times. And a lot can change in 16 years.

16 years ago, I thought I was a straight man. I was married. I voted for Bush and thought the Republicans could run the country well. I was Catholic and believed the anti-abortion rhetoric, yet I somehow rejected the anti-gay rhetoric. Go figure. Though we knew a few gay men in college, it was shortly after we moved to Austin that my wife and I noticed for the first time pairs of men openly holding hands at formal “respectable” events. We began to feel a relief that this was the kind of safe city that we never really recognized we sought.

15 years ago, my wife and I decided to have a polyamorous relationship. She said that I had suggested it years earlier while we were dating. I did not remember saying that, but it felt like the kind of thing I might have. It was shortly after this, as I thought about love, romance, and relationships, that I began to believe that it was tragic that I was straight and not bisexual. I can remember being apologetic as I (very occasionally) turned a man down. And it was shortly after this that I understood that I was parting ways with the Catholic church… and I did not particularly regret that.

14 years ago, I found out how infidelity felt. Infidelity in a polyamorous relationship looks different than in a monogamous relationship, but it hurts the same. It cuts through hearts, rips out rugs, and crushes dreams.

I also learned that maybe the Republican party was actually made up of a bunch of chronic liars, and became a Democrat. Later I started to understand how violent and hateful Republicans could be. How did I never see this before? And maybe, just maybe, I shouldn’t hold the reproductive health doctrines of men who want to ban abortion but also ban all other forms of contraception and all forms of sex that carry no chance of pregnancy while simultaneously starting a war and lying to me about weapons of mass destruction!

12 years ago I joined a company that became my career for the better part of a decade. They weren’t great… in fact sometimes they were downright awful, but over time my authority became vast, as did my knowledge of everything about the business… except what was in the best interest of the business. Ya know, sometimes we techies need to be informed of the big business direction so we can make decisions intelligently.

10 years ago, with the onset of Saturn Returns, I finally figured out that I was not a man. That moment has lead me through so many changes and to so many of the people that I find so important in my life now. As a man, I would never have made any of the connections I have as an androgyne. This realization sometimes keeps me awake at night, knowing that it is by the grace of but a few words that I have in my life the love that I experience now. More rationally, a few of my current friends would have been my friends anyway, and they would have noticed my egg tendencies, and they would have aided in my hatching. I may have ended up exactly where I am now, on a different schedule.

Letting go of my own gender also let me release my expectations about my sexual orientation. Reparative therapy, especially religious-based “therapy”, is bullshit. We know this. And yet, I successfully “prayed the straight away”!

I also gave up on “til death do we part” and let my marriage end.

Five years ago, I learned photography, and I changed how I see the world. Always watching for that perfect moment. Seeing textures. Analyzing light. Understanding focus and freezing motion. The speckled shadow beneath a canopy. The shimmer of a cobweb five meters up and at least that far away.

Three years ago, I talked myself out of my first suicide attempt. In the aftermath, I evaluated my life. I saw clearly how I was wasting it on my employer’s amazingly small dreams, and I chose to spend some time quite alone. I loved living out in the woods. I hated having to drive for twenty minutes to reach the closest decent internet connection, and for an hour to reach any of my friends. But there is a lot to be said for the peace of the forest, for stars so bright as to light the ground, for rain on the metal roof a mere meter from my lofted bed… and for really cheap rent paid in cash under the table. Oh, and did I mention that my landlady also covered electricty? Pretty epic, especially since the cabin wasn’t well insulated and I had to run 2.5kW of heating that winter.

In the last two years, I have truly started to learn how black lives matter, and how little I understood my own racism in the past. I have learned about social justice, and become keenly aware of my failings. I have gained true confidence in my skills, and become comfortable in my body for the first time in my life. I have felt my socialist/anarchist heart begin to blossom as I notice the Democratic party repeatedly snatch defeat from the jaws of victory.

And, shortly after my birthday in 2014, I met the woman who has become the love of my life. She had to exercise both persistence and patience. I was wounded and avoiding romance, sex, dating. She had to convince me that a lesbian, even a trans-friendly lesbian, could be interested in an androgyne who still had and wasn’t particularly inclined to get rid of eir penis. But, she exercised that persistence, and she waited patiently, while over the course of months I fell in love and I healed. Now we talk of our sixty year plan.

I will miss Austin. I will miss the people here. I will miss all of my bike routes and the restaurants and the events. I will miss the familiarity. And I feel guilt, leaving all of you to stay and stand against the legislature.

But for this woman, where she goes my heart shall follow.

Nix Development Environments

Savanni D'Gerinel 28 Nov, 2016

nix-shell, the command that creates a subshell after evaluating any nix expression, has a lot of uses. I found it very useful in my devops work when I had multiple environments to administer, but had to use different tools for each. The shell provides excellent help in isolating my required tools to the environments involved.

A trick, though, lay in learning how to acquire those tools when the tools were not available in the nixos channel. I figured it out, and so here is the example for one of the environments I was administering. Note that I include both Linux and Darwin builds, because I wanted to offer the nix environment to my replacement at the company.

  • Packer – 0.10.1
  • Terraform – 0.7.4
  • Ansible 2
  • Python 2.7

We were deploying in Amazon AWS. I used Packer to build the custom images that we were deploying. Autoscaling works a lot better if it has a complete image that only has to be started (the Crops, i.e., the systems that can be replaced almost instantly and thus do get replaced regularly). I love Terrafrom because I was able to describe everything I was doing in AWS using the tool. Ansible is present for those systems that get reconfigured regularly (primarily the Cattle machines, things that can be rebuilt from just the devops scripts, but that I do not want to terminate). Python 2.7 is present to support Ansible, though it is sometimes convenient to have at hand.

Neither Packer nor Terraform were available in my Nix channel, so I had to build derivations for those. The process is non-obvious until it is done. Here are my scripts for them. At the time I wrote these scripts, I was running NixOS 16.03, however I still use the same scripts after having upgraded to NixOS 16.09.

nix-deps/packer.nix

{ pkgs ? import <nixpkgs> {},
  stdenv ? pkgs.stdenv }:

let
  # suggestion from @clever of #nixos
  package =
         if stdenv.system == "x86_64-linux" then "packer_0.10.1_linux_amd64.zip"
    else if stdenv.system == "x86_64-darwin" then "packer_0.10.1_darwin_amd64.zip"
    else abort "unsupported platform";
  checksum =
         if stdenv.system == "x86_64-linux" then "7d51fc5db19d02bbf32278a8116830fae33a3f9bd4440a58d23ad7c863e92e28"
    else if stdenv.system == "x86_64-darwin" then "fac621bf1fb43f0cbbe52481c8dfda2948895ad52e022e46f00bc75c07a4f181"
    else abort "unsupported platform";
in
stdenv.mkDerivation rec {
  name = "packer-${version}";
  version = "0.10.1";

  buildCommand = ''
  mkdir -p $out/bin
  unzip $src
  mv packer $out/bin/packer
  echo Installed packer to $out/bin/packer
  '';

  src = pkgs.fetchurl {
    url = "https://releases.hashicorp.com/packer/0.10.1/${package}";
    sha256 = checksum;
    name = package;
  };

  buildInputs = [ pkgs.unzip ];
}

nix-deps/terraform.nix

{ pkgs ? import <nixpkgs> {},
  stdenv ? pkgs.stdenv }:

let
  # suggestion from @clever of #nixos
  package =
         if stdenv.system == "x86_64-linux" then "terraform_0.7.4_linux_amd64.zip"
    else if stdenv.system == "x86_64-darwin" then "terraform_0.7.4_darwin_amd64.zip"
    else abort "unsupported platform";
  checksum =
         if stdenv.system == "x86_64-linux" then "8950ab77430d0ec04dc315f0d2d0433421221357b112d44aa33ed53cbf5838f6"
    else if stdenv.system == "x86_64-darwin" then "21c8ecc161628ecab88f45eba6b5ca1fbf3eb897e8bc951b0fbac4c0ad77fb04"
    else abort "unsupported platform";
in
stdenv.mkDerivation rec {
  name = "terraform-${version}";
  version = "0.7.4";

  buildCommand = ''
  mkdir -p $out/bin
  unzip $src
  mv terraform $out/bin/terraform
  echo Installed terraform to $out/bin/terraform
  '';

  src = pkgs.fetchurl {
    url = "https://releases.hashicorp.com/terraform/0.7.4/${package}";
    sha256 = checksum;
    name = package;
  };

  buildInputs = [ pkgs.unzip ];
}

The structure of each script is relatively straightforward.

  • declare that pkgs and stdenv are both required, as well as how to get them if they are absent
  • based on the OS, declare what package I want to download and the relevant checksum
  • declare the name and version of the derivation
  • create the custom build command

    In many cases, the default build commands works perfectly, but that only works for projects that have to be built with autoconfig or with Stack (and possibly some other languages). Both Terraform and Packer are binaries, and so it is necessary for me to specify the build for the derivation.

    In this case, the build is simply to unzip the downloaded package (specified in $src) and copy the executable into the destination (which has a root at $out). It is vital that the executable end up in the bin/ directory. I am not sure of the mandated directory structure of a derivation, but I know that derivations that did not include the bin/ directory would fail. I assume that they failed because there was no executable to add to the path.

  • specify precisely how to get the source package. In this case, through the fetchurl tool.
  • specify additional build inputs. These have to be somewhere in the nix namespace. pkgs.unzip refers to nixpkgs.unzip in the standard channel.

Both of the files above must go in a subdirectory. I named the subdirectory nix-deps/. Some subtle interaction will cause an infinite recursion if the two files are included in the root directory of your project.

With those present, it is time to build the nix-shell command:

shell.nix

let
  pkgs = import <nixpkgs> {};
  stdenv = pkgs.stdenv;
  terraform = import nix-deps/terraform.nix {};
  packer = import nix-deps/packer.nix {};

in stdenv.mkDerivation {
  name = "v2-devops";

  buildInputs = [ pkgs.ansible2
                  terraform
                  packer
                  pkgs.python
                  pkgs.python27Packages.alembic
                  pkgs.python27Packages.boto
                  pkgs.python27Packages.psycopg2
                  pkgs.awscli
                ];
}

The only difficult part here was for me to figure out how to import my Terraform and Packer derivations. I handle that with the import nix-deps/<package>.nix {} lines. The result of each import statement is a derivation, and so it is valid to include in buildInputs.

buildInputs again just lists the packages that must be included in this derivation. So, I included all of the packages that I use directly.

Thus, from the root directory of my devops folder, I can simply run nix-shell and have exactly the version of Terraform, Packer, Ansible, and Python that I want. This also means that I can have completely different versions for a different devops repository (I was actually administering three different clouds, all with different standards). And, possibly best of all, if I could convince my co-workers to use Nix (the tool, not the operating system), they would have had a trivial way to set up their development environments, also.


Dreamer, Shaper, Seeker, Maker