Input/Output Redirection in the Shell

Delve deep into stdout, stderr, and pipes plus some neat tricks that you might not know.

Are you experiencing technical drift?

December 19, 2013

David Byrne, in the seminal Talking Heads song “Once in a Lifetime”, sings “How did I get here?” The meaning of those lyrics have been fodder for some great discussion, but I think there’s one interpretation which has been sorely missing from that discourse — that the narrator is a software developer. The software developer, reflecting back on his work, is wondering how he accumulated so much technical debt.

“Letting the days go by, let the water hold me down.”

The water which holds him down is an obvious reference to what can happen to a software developer over time, caught under the weight of their own codebase. But what Byrne is singing about here isn’t strictly technical debt. Technical debt is the outcome. He’s singing about something else, about “letting the days go by”, about letting your codebase slip away from you.

Byrne, although he did not know it at the time (after all, even the phrase technical debt hadn’t been invented yet) was singing about “technical drift”. Technical drift is a phenomenon that impacts a lot of well-meaning software development teams. It’s a phenomenon that occurs when a development team continuously fails to recognize and adapt to change, causing the concepts in the software’s domain and the concepts in code to start to slowly “drift” apart from one another, creating dissonance which ultimately leads to technical debt.

One of the goals of software development is to write software that most closely reflects the the domain in which we’re working. This creates more understandable code that is easy reason and discuss, both within the tech team and with the rest of the organization. But it’s important to remember that the domain is continually shifting. We’re receiving new requirements based on new information.

If we don’t adapt to these changes, a gap will appear between our domain and software, and the gap becomes our friend, technical debt.

While the signs of technical drift may be hard to spot, technical debt, the result, starts to become readily apparent.

In some cases, a software development team, or a team leader, recognizing that the gap has become too large, comes together and decides “enough is enough”. What is needed is Design —Big Design. The team stops feature development, buckles down, and refactors their code and architecture to more cleanly map to their domain.

The problem with Big Design is that it’s a “stop the world” operation. Feature development stops, and an anxious queue begins to build up waiting for the development team to free up. Furthermore, it grants false hope that such a thing might not be necessary again, but if the pattern continues, it often becomes inevitable.

How do we prevent technical drift?

If you’re not taking an active role in refactoring your application every week chances are you’re experiencing “technical drift”. Having a problem space that is continually changing necessitates continuous reworking of how your application is designed — what Agile practitioners call “iterative design”. When you’re able to do this effectively, you can round off the peaks and avoid some of the pitfalls of Big Design.

Note that the “Code” and “Domain” lines never intersect. We must accept that we do not yet have enough information to make some kinds of design decisions. But we can get close.

Understanding technical drift is more of a mindset than a particular practice, but it can help put into context why it’s important to remain vigilant, to call attention to code and domain mismatches, and to advocate for accurate and precise names wherever possible. Gaps which you’re creating may be well understood today, but as your team changes, and time progresses, that understandability suffers. Make the investment now, double down and pay attention.

Years later you don’t want to look back in horror on your code and say to yourself “my god, what have I done”.

[…]

Is it live?

July 30, 2012

TL;DR Rails 4.0 will allow you to stream arbitrary data at arbitrary intervals with Live Streaming.

HAPPY MONDAY EVERYONE!

Besides enabling multi-threading by default, one of the things I really wanted for Rails 4.0 is the ability to stream data to the client. I want the ability to treat the response object as an I/O object, and have the data I write immediately available to the client. Essentially, the ability to deliver whatever data I want, whenever I want.

Last night I merged a patch to master that allows us to do exactly that: send arbitrary data in real time to the client. In this article, I would like to show off the feature by developing a small application that automatically refreshes the page when a file is written. I’ll be working against edge Rails, specifically against this commit (hopefully then people in the future will notice if / when this article is out of date!)

Here is a video of the final product we’ll build in this article:

Response Streaming

The first thing I added was a “stream” object to the response. This object is where where data will be buffered until it is sent to the client. The stream object is meant to quack like an IO object. For example:

class MyController < ActionController::Base
  def index
    100.times {
      response.stream.write "hello world\n"
    }
    response.stream.close
  end
end

In order to maintain backwards compatibility, the above code will work, but it will not stream data to the client. It will buffer the data until the response is completed, then send everything at the same time.

Live Streaming

To make live streaming actually work, I added a module called ActionController::Live. If you mix this module in to your controller, all actions in that controller can stream data to the client in real time. We can make the above MyController example live stream by mixing in the module like so:

class MyController < ActionController::Base
  include ActionController::Live

  def index
    100.times {
      response.stream.write "hello world\n"
    }
    response.stream.close
  end
end

The code in our action stays exactly the same, but this time the data will be streamed to the client as every time we call the write method.

Webservers

Before we start on our little example project, we should talk a bit about web servers. By default, script/rails server uses WEBrick. The Rack adapter for WEBrick buffers all output in a way we cannot bypass, so developing this example with script/rails server will not work.

We could use Unicorn, but it is meant for fast responses. Unicorn will kill our connection after 30 seconds. The protocol we’re going to use actually makes this behavior irrelevant, but it’s a bit annoying to see the logs.

For this project, I think the best webserver would be either Rainbows!, Puma, or Thin. I’ve been playing with Puma a lot lately, so I’ll use it in this example.

Our application

We’re going to build an application that automatically reloads the page whenever a file is saved. You can find the final repository here.

Technology

For this project we’re going to use edge Rails and Live Streaming, along with a bit of JavaScript and Server-Sent Events. To detect file system changes, we’re going to use the rb-fsevent gem. I think this gem only works on OS X, but it should be easy to translate this project to Linux or Windows given the right library.

Server-Sent Events

If you’ve never heard of Server-Sent Events (from here on I’ll call them SSEs), it’s a feature of HTML5 that allows long polling, but is built in to the browser. Basically, the browser keeps a connection open to the server, and fires an event in JavaScript every time the server sends data. An example event looks like this:

id: 12345\n
event: some_channel\n
data: {"hello":"world"}\n\n

Messages are delimited by two newlines. The data field is the event’s payload. In this example, I’ve just embedded some JSON data in the payload. The event field is the name of the event to fire in JavaScript. The id field should be a unique id of the message. SSE does automatic reconnection; if the connection is lost, the browser will automatically try to reconnect. If an id has been provided with your messages, when the browser attempts to reconnect, it will send a header (Last-Event-ID) to the server allowing you to pick up where you left off.

We’re going to build a controller that emits SSEs and tells the browser to refresh the page.

Getting Started

The first thing we’ll do is generate a new Rails project from the Rails git repository (I keep all my git repos in ~/git):

$ cd ~/git/rails
$ ruby railties/bin/rails new ~/git/reloader --dev
$ cd ~/git/reloader

Update the Gemfile to include puma and rb-fsevent and re-bundle:

diff --git a/Gemfile b/Gemfile
index 9e075a8..51ce01c 100644
--- a/Gemfile
+++ b/Gemfile
@@ -6,6 +6,8 @@ gem 'arel',      github: 'rails/arel'
 gem 'active_record_deprecated_finders', github: 'rails/active_record_deprecated_finders'
 
 gem 'sqlite3'
+gem 'puma'
+gem 'rb-fsevent'
 
 # Gems used only for assets and not required
 # in production environments by default.

Then we’ll generate a controller for emitting SSE messages to the browser:

$ ruby script/rails g controller browser

Moving on!

Generating SSEs

I’d like an object that knows how to format messages as SSE and emits those messages to the live stream. To do this, we’ll write a small class that decorates the output stream and knows how to dump objects as SSEs:

require 'json'

module Reloader
  class SSE
    def initialize io
      @io = io
    end

    def write object, options = {}
      options.each do |k,v|
        @io.write "#{k}: #{v}\n"
      end
      @io.write "data: #{JSON.dump(object)}\n\n"
    end

    def close
      @io.close
    end
  end
end

We’ll place this file under lib/reloader/sse.rb and require it from the browser controller. In the controller, we’ll mix in ActionController::Live and try emitting some SSEs:

require 'reloader/sse'

class BrowserController < ApplicationController
  include ActionController::Live

  def index
    # SSE expects the `text/event-stream` content type
    response.headers['Content-Type'] = 'text/event-stream'

    sse = Reloader::SSE.new(response.stream)

    begin
      loop do
        sse.write({ :time => Time.now })
        sleep 1
      end
    rescue IOError
      # When the client disconnects, we'll get an IOError on write
    ensure
      sse.close
    end
  end
end

Next, update your routes.rb to point at the new controller:

Reloader::Application.routes.draw do
  get 'browser' => 'browser#index'
end

Fire up Puma in one shell:

$ puma
Puma 1.5.0 starting...
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://0.0.0.0:9292
Use Ctrl-C to stop

Then in another shell curl against the endpoint. You should see an event emitted every second. Here is my output after a few seconds:

$ curl -i http://localhost:9292/browser
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
X-UA-Compatible: IE=Edge
X-Request-Id: 76cfaa39-d23b-4eac-8337-f915410dc0de
X-Runtime: 0.430762
Transfer-Encoding: chunked

data: {"time":"2012-07-30T10:02:05-07:00"}

data: {"time":"2012-07-30T10:02:06-07:00"}

data: {"time":"2012-07-30T10:02:07-07:00"}

data: {"time":"2012-07-30T10:02:08-07:00"}

data: {"time":"2012-07-30T10:02:09-07:00"}

data: {"time":"2012-07-30T10:02:10-07:00"}

^C
$

Next we should monitor the file system.

File System Monitoring

Now we’ll update the controller to emit an event every time a file under app/assets or app/views changes. Rather than a loop in our controller, we’ll use the FSEvent object:

require 'reloader/sse'

class BrowserController < ApplicationController
  include ActionController::Live

  def index
    # SSE expects the `text/event-stream` content type
    response.headers['Content-Type'] = 'text/event-stream'

    sse = Reloader::SSE.new(response.stream)

    begin
      directories = [
        File.join(Rails.root, 'app', 'assets'),
        File.join(Rails.root, 'app', 'views'),
      ]
      fsevent = FSEvent.new

      # Watch the above directories
      fsevent.watch(directories) do |dirs|
        # Send a message on the "refresh" channel on every update
        sse.write({ :dirs => dirs }, :event => 'refresh')
      end
      fsevent.run

    rescue IOError
      # When the client disconnects, we'll get an IOError on write
    ensure
      sse.close
    end
  end
end

The controller will send an SSE named “refresh” every time a file is modified. Start Puma in one shell, then curl in a second shell, and touch a file in the third shell, you will see an event.

Curl started in one shell:

$ curl -i http://localhost:9292/browser

Touch a file in another:

$ touch app/assets/javascripts/application.js

Now the curl shell should look like this:

$ curl -i http://localhost:9292/browser
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
X-UA-Compatible: IE=Edge
X-Request-Id: 98331d36-ef7c-4d15-ad99-331149fc589b
X-Runtime: 43.307765
Transfer-Encoding: chunked

event: refresh
data: {"dirs":["/Users/aaron/git/reloader/app/assets/javascripts/"]}

Every time a file is modified under the directories we’re watching, an SSE will be sent up to the browser.

Listening with JavaScript

Next let’s add the JavaScript that will actually refresh the page. I’m going to add this directly to app/assets/javascripts/application.js. The JavaScript we’ll add simply opens an SSE connection and listens for refresh events.

jQuery(document).ready(function() {
  setTimeout(function() {
    var source = new EventSource('/browser');
    source.addEventListener('refresh', function(e) {
      window.location.reload();
    });
  }, 1);
});

Whenever a refresh event happens, the browser will reload the current page.

Parallel Requests

We need to update the configuration in development to handle multiple requests at the same time. One request for the page we’re working on, and another request for the SSE controller. Add these lines to your config/environments/development.rb but please note that they may change in the future:

  config.preload_frameworks = true
  config.allow_concurrency = true

Next we’ll see everything work together.

Trying it out!

To see the automatic refreshes in action, let’s create a test controller and view. I just want to see the automatic refreshes happen, so I’ll use the scaffolding to generate a model, view, and controller:

$ ruby script/rails g scaffold user name:string
$ rake db:migrate

Now run Puma, and navigate to http://localhost:9292/users. If you watch the developer tools, you’ll see the browser connect to /browser but the request will never finish. That is what we want: the browser listening for events on that endpoint.

If you change any file under app/assets or app/views, a message should be sent to the browser, and the browser will refresh the page.

YAY!

SSE Caveats / Features

SSEs will not work on IE (yet). If you want to use this with IE, you’ll have to find another way. SSEs will work on pretty much every other browser, including Mobile Safari.

Some webservers (notably Unicorn) cut off the request after a particular timeout. Be mindful of this when designing your application, and remember that SSE will automatically reconnect after a connection is lost.

Heroku will cut off your connections after 30 seconds. I had trouble getting the SSE to reconnect to a Heroku server, but I haven’t had time to investigate the issue.

Rails Live Streaming Caveats

Mixing the Live Streaming module in to your controller will enable every action in that controller to have a Live Streaming object. In order to make this feature work, I had to work around the Rack API and I did this by executing Live Stream actions in a new thread. This shouldn’t be a problem for most people, but I thought it would be good for people to know.

Headers cannot be written after the first call to write or close on the stream. You will get an exception if you attempt to write to the headers after those calls are made. This is because when you call write on the stream, the server will send all the headers up to the client, so writing new headers after that point is useless and probably a bug in your code.

Always make sure to close your Live Stream IO objects. If you don’t, it might mean that a socket will sit open forever.

WUT?

I thought streaming was already introduced in Rails 3.2. How is this different?

Yes, streaming templates were added to Rails 3.2. The main difference between Live Streaming and Streaming Templates is that with Streaming Template, the application developer could not choose what data was sent to the client and when. Live Streaming gives the application developer full control of what data is sent to the client and when.

Final Thoughts

I’m very excited about this feature of Rails 4. In my opinion, it is one of the most important new features. I’ve been interested in streaming data from Rails for a long time. We can use this feature to reduce latency and deliver data more quickly to clients on slow connections (e.g. cell phones), for infinite streams like chatrooms, or for cool productivity hacks like this article shows.

I hope you enjoyed this article! I think for the next demo of Live Streams, I would like to show how to reduce latency when sending JSON streams to the client. That might be fun.

<3<3<3

[…]

The Ruby Rails Aggregator