Mumblings and Grumblings on Computers

My Clojure Workflow, Reloaded, Reloaded

Several months ago Stuart Sierra wrote a post on the Relevance blog which I found very interesting. In it, he described a way of doing Clojure development that I wanted to try. At first glance, I was interested in it only because it gave me a good workflow for testing within the Clojure repl. I love Test-Driven development and I have always had problems practicing it well in Clojure. I had previously tried two approaches. To begin with, I tried to make do with simply running lein test in the shell when I needed to run tests. The context switch coupled with the slow JVM start up time made this approach unfeasible.

Moving on from this, I attempted to run tests in the REPL without the use of Stuart's tools.namespace library. I learned after a few hours of attempting that I was simply too dumb. I have no idea how code loading works in Clojure. I do not know how to reliably reload code and all its dependencies. I never really knew what the state of the system was at any given time. Once I started making use of the workflow suggested by Sierra I was actually able to reliably and quickly run tests in the repl. This drastically improved my feelings about Clojure development.

A lot of my problems are likely related to the fact that I don't really know what I'm doing. I understand some of the fundamentals around functional programming and LISP. But I do not understand the specific Java-ness of Clojure. I have never written Java more complex than is necessary to maintain the Braintree Java Client Library. I'm certain that for someone with more familiarity with Java, it would be very easy to start up an instance of Jetty when initializing the system and then setting up the ring handler within. I don't know how to do that. For this reason, I was unable to full commit to this technique. I had to make a compromise. Here is how I init the system and start the app.

(defn system []
  (let [db-url (or (System/getenv "DATOMIC_URL") "datomic:mem://sayc-app")]
    {:db-url db-url
     :handler (sayc-app.web/create-handler db-url)
     :web-port (or (try (Integer/parseInt (System/getenv "PORT"))
                        (catch NumberFormatException _ nil))
                   8080)
     }))

(defn start [system]
  (d/create-database (:db-url system))
  (migrate/migrate (d/connect (:db-url system)))
  (assoc system
         :server (ring/run-jetty (:handler system) {:port (or (:web-port system) 8080) :join? false})
         :db-connection (d/connect (:db-url system))))

Since I don't really understand jetty I have to rely on ring/run-jetty to initialize the server. This means there will be some hidden elements of global state. I wish I could get rid of it and I don't think this is exactly what Stuart Sierra had in mind, but it works for me.

Other than the benefit of being able to reliably and quickly run tests, there was a different huge benefit. Constructing an application this way leads you to really consider where all of your global state lies. When you initialize the database connection at such a high level, it is much harder to rely on it deep within the application. This is what led me to the approach I outlined in my previous post. When you initialize this complex system at such a high level, you have to think much harder about what needs access to what.

It is however not all sunshine and roses. This way of working is much harder than just hacking out some Clojure code. It is a quite heavyweight solution. There are parts of it that I found useful that I would like to separate out. I love the good code reloading and I love the ease with which I could run tests. However, designing an application in this way is definitely overkill for the simple toy app I've been working on. Granted, I don't think anyone would have ever thought otherwise. I'm certain that when one is waist-deep in an extermely complicated application this workflow is invaluable.

Why Are We Settling for this in Clojure?

I did a quick Github search to find instances of people passing around global database connections. It didn't take long for me to find a bunch of examples of this.

(def ^:dynamic ^java.sql.Connection *db* nil)

(def *db*
  {:subprotocol "postgresql"
  :subname "//localhost:5432/blogjr"
  :username "blogjr"


(def ^{:doc "The binding contains the database connection settings."}
  *db* {:classname "com.mysql.jdbc.Driver"})

Why are we okay with this? This isn't the post global state future we were promised. If you are using this technique, there are a limited number of options to swap out your database. You can either redefine a dynamic variable or using with-bindings. I had a conversation on Twitter recently with one of the few people I know who has actually been paid to write Clojure about this topic:

I'm quick to believe that there is an issue with what I'm about to propose. It's very likely in fact. I have never done serious work in Clojure. The largest app I've written in Clojure is maybe 300 LoC. So take this with the world's largest grain of salt (#OXYMORON #ALERT).

We're going to take advantage of one of the primitive functional programming techniques -- currying. Currying is a process whereby a function with multiple arguments is transformed into a function which takes one argument returning a function takes a single argument and so on... For example,

(defn foo [a b c] (+ a b c))

(defn foo-curried [a] (fn [b] (fn [c] (+ a b c))))

(= (foo 1 2 3) (((foo-curried 1) 2) 3))

So we write our actions like this.

(defn create-bid [db]
  (fn [request]
    (transact db tx-data)

In starting the web server, I create the database connection and initialize the web handler like so.

(defn build-routes [db]
  (defroutes routes
    (POST "/bids" [] (actions/create-bid db))
    (resources "/")))


(defn create-handler [db]
  (build-routes db)
  (-> (var routes)
      (keyword-params/wrap-keyword-params)
      (nested-params/wrap-nested-params)
      (params/wrap-params)))

I like this. I much prefer it to dealing with the global database connection. Am I missing something? Is there a reason that this isn't preferable? Is it overkill?

I likely feel uncomfortable with the status quo because there are a lot of things left in Clojure I don't understand. I don't really get what it means to reassign these global variables. I don't know what the best way is to do this. I do understand how to pass arguments to functions. I've been doing that since I first touched a Lisp in my first year of college 7 years ago. If I understood things better, I would probably feel more comfortable with this than I do, but I don't know much.

An Action's Tale

I have been working recently on a web app written in Clojure. I wanted to learn both Clojure and Bridge better and this app has helped me in both regards. I want to tell you the story of an action in this app and how I implemented it. It is a story that demonstrates one of my great struggles while I write Clojure code. As I work in Clojure, I often feel myself writing Rails code with more parentheses. I had this very problem with the action that is the subject of our story. This action parses a request, builds up a new object representing the result of a hand of Bridge and associates it with an in-progress Chicago Bridge game. As, you can see the code is very procedural. First, get the data we need, check if everything is present, then update the storage, then respond with a redirect.

(defn add-hand-to-chicago-game [request]
  (let [id (get-in request [:params :id])
        game (get @chicago-games id)
        hand (get-in request [:params :hand])]
    (if (not game) (not-found "NOT FOUND"))
    (swap! chicago-games assoc id (conj game (process-hand-params hand)))
    (redirect (str "/chicago_games/" id))))

This really bothered me. This is not the beauty of functions operating on data that I was longing to see. It got even worse once I added some validations. This version of the action doesn't even exist in the Git repo anymore so I can't show you. I hated it so much.

Finally, I bit the bullet. I broke this action apart into functions upon functions upon functions. Ahhhh, here was the beauty I was looking for...sort of. Granted, it didn't really look that beautiful. First, it's giant. I took a seven line function and ballooned it out into almost 40 lines.

(defn extract-params [request]
  {:id (get-in request [:params :id])
   :hand (get-in request [:params :hand])})

(defn enrich-with-prereqs [request]
   (assoc request :game (get @chicago-games (:id request))))

(defn validate-prereqs [prereqs]
  (assoc prereqs :errors (validate/hand (:hand prereqs))))

(defn derive-status [prereqs]
  (assoc prereqs :status
         (cond
           (nil? (:game prereqs)) 404
           (not (empty? (:errors prereqs))) 422
           :else 201)))

(defn persist [request]
  (let [new-game (conj (:game request) (process-hand-params (:hand request)))]
    (swap! chicago-games assoc (:id request) new-game)
    (assoc request :game new-game)))

(defn conditionally-persist [request]
  (if (= (:status request) 201)
    (persist request)
    request))

(defn render [request]
  (cond
    (= (:status request) 404) (not-found "NOT FOUND")
    :else (views/display-chicago-game (:id request) (:game request) (:errors request))))

(defn add-hand-to-chicago-game [request]
    (-> request extract-params enrich-with-prereqs validate-prereqs
        derive-status conditionally-persist render))

I suspect anyone who wanted to get any real work done in this language would object to this style. And I would totally agree with them. However, for this toy app, it works fine.

In thinking about this, my mind is drawn towards thinking about Ring. Ring is a Clojure middleware stack and spec. It takes a similar approach to Ruby's Rack framework. It works by applying a series of functions to the data that is the HTTP request and returning data that will be the HTTP response. In order to get the aesthetic I want, perhaps I should be pushing more of the contents of this action into the Ring stack. Rather than extracting params in the action, perhaps a param-extractor middleware which is built up with a map of pairs of verbs and actions pointing to a map representing the parameters to be extracted. At this point, the contents of the action could be changed to only validation, persistence, and rendering. I may attempt spiking this out to see what it would look like. Until then, that's the story of this action and I'm sticking to it.

Playing with Play

Two Fridays ago a challenge was posed to me and a deadline was imposed. There is a serious bathroom availability problem in my office. It is often inaccessible for cleaning at crucial times and my coworkers are forced to walk all the way across the office before finding out it is closed. I took the challenge and promised to have an app to track whether the bathroom was open by lunch time.

Bets were made and doubts were stated. Doubts primarily arising from my decision to use the Play framework with Scala. I had no experience with the framework and very little experience with the language. The doubters were correct to doubt me. I did not finish the app before lunchtime, but I did complete it and deploy it shortly after lunch. You can find it here. This experience gave me some perspective on the Play framework which I would like to share.

I use Rails all day every day, and so I encounter Play primarily as a Rails developer. The first few hours using Play with Scala felt a lot like the first few hours I spent in 2008 with Ruby on Rails. It was very friendly to the new developer. So friendly in fact that the entirety of the documentation is bundled with the framework. On the development server, you can view documentation that will take you all the way to deploying the app to Heroku. This is fabulous. For a developer working in a framework for the first time, this guidance is incredible.

Playing with the Database

Working with databases in Scala Play is very different from working in ActiveRecord in Rails. Play provides Anorm for interacting with databases. Anorm is "Anorm is not a Object Relation Manager" and they aren't Playing. Below is the code I wrote to interact with the Bathroom object.

case class Bathroom(id: Long, label: String, status: Boolean) {  
  def isClosed(): Boolean = this.status
}

object Bathroom {  
  val bathroom = {
    get[Long]("id") ~
    get[String]("label") ~
    get[Boolean]("status") map {
      case id~label~status => Bathroom(id, label, status)
    }
  }

  def all(): List[Bathroom] = DB.withConnection { implicit c =>
    SQL("SELECT * FROM bathrooms").as(bathroom *)
  }

  def create(label: String) {
    DB.withConnection { implicit c =>
      SQL("INSERT INTO bathrooms (label, status) VALUES ({label}, true)").on(
        'label -> label
      ).executeUpdate()
    }
  }

  def find(id: Long): Bathroom = {
    DB.withConnection { implicit c =>
      SQL("SELECT * FROM bathrooms WHERE id={id}").on(
        'id -> id
      ).as(bathroom *).head
    }
  }

  def save(bathroom:Bathroom) = DB.withConnection { implicit c =>
    SQL("UPDATE bathrooms set status = {status} where id = {id}").on(
      'status -> bathroom.status,
      'id -> bathroom.id
    ).executeUpdate()
  }
}

There is a lot going on here, and it is very different than how things would look in Ruby and Rails. For example, we have to define a parser which transforms database results in classes.

val bathroom = {  
  get[Long]("id") ~
  get[String]("label") ~
  get[Boolean]("status") map {
    case id~label~status => Bathroom(id, label, status)
  }
}

This is very tough for me to understand as a new person looking at this framework. I believe that I could use this if I were to write a serious app, but I have no idea how I would even begin to work on Anorm itself.

Playing with Controllers

Working in controllers in Play felt very familiar to me as a Rails developer. Look at this controller and tell me it couldn't almost be a Rails controller.

object BathroomController extends Controller {

  val bathroomForm = Form(
    "label" -> nonEmptyText
  )

  def index = Cached("homePage") {
    Action {
      Ok(views.html.index(Bathroom.all(), bathroomForm))
    }
  }

  def show(id: Long) = Action { implicit request =>
    val bathroom = Bathroom.find(id)
    Ok(views.html.bathroomShow(bathroom, BathroomStatistic.closedDurations(bathroom), BathroomStatistic.averageClosedDuration(bathroom)))
  }

  def create = Action { implicit request =>
    bathroomForm.bindFromRequest.fold(
      errors => BadRequest(views.html.index(Bathroom.all(), errors)),
      label => {
        Bathroom.create(label)
        Cache.set("homePage", None)
        Redirect(routes.BathroomController.index)
      }
    )
  }

  def toggle(id: Long) = Action { implicit request =>
    val bathroom = Bathroom.find(id)
    val event = BathroomEvent.toggleForBathroom(bathroom)
    event.save
    Bathroom.save(event.apply(bathroom))
    Cache.set("homePage", None)
    Redirect(routes.BathroomController.index)
  }
}

There are a few interesting things here.

val bathroomForm = Form(  
  "label" -> nonEmptyText
)

We declare and create a form here which will be used in the view template and to validate the submission. In the create action, we make use of the form in a way that did not make much sense to me at first.

def create = Action { implicit request =>  
  bathroomForm.bindFromRequest.fold(
    errors => BadRequest(views.html.index(Bathroom.all(), errors)),
    label => {
      Bathroom.create(label)
      Cache.set("homePage", None)
      Redirect(routes.BathroomController.index)
    }
  )
}

First, bindFromRequest is unclear because it relies on an implict variable. It is helpful to think it as bindFromRequest(request). My understanding is that bindFromRequest will pull the necessary parameters out of the request, validate the params, etc. The fold method which is invoked next takes
two arguments. The first is a function which is performed if there are errors on the form, the second is called when no errors are present. I'm not entirely sure why the name fold was chosen here, but there you go. You can find the documentation for the Form object here

Impressions of Play

Overall, I found working in Play very interesting. I was able to easily accomplish the task I set for myself, and I enjoyed working in the framework. My first encounter with it reminded me a lot of working in Rails for the first time. With the ease of use and the improved performance over Rails, I think it would be a good choice as a general web application framework.