Category Archives: deployment

Deploying Elixir applications with Edeliver

We’ve been talking about deploy and releases with Elixir lately, like how to run migrations on top of a release or how to deal with environment variables. Now it’s time to discover another tool that can help us release our Elixir application. After practicing deploy and tracing through nodes with Exrm, we got more comfortable … »

Read more at the source

How to config environment variables with Elixir and Exrm

It’s very common (and highly recommended) that application keeps its configuration values separated from its version control. A way of doing this is by using ENV vars (environment variables). They’re being used for improvements mostly on maintainability. The 12-factor app manifesto explains it on its Configuration section: The twelve-factor app stores config in environment variables … »

Read more at the source

Running migration in an Exrm release

Exrm is a great tool for building releases for Elixir applications so you can deploy them on the web and even in an embedded hardware. We have been using Exrm here at Plataformatec. It is the chosen tool for deploying our projects and the contributors are doing a great job maintaining and developing new features. … »

Read more at the source

git push heroku master is asking for authentication

In this post I show you how to fix the authentication problem with heroku. If you want to deploy your branch to heroku using this command 1$ git push heroku master and git want an authentication like this 12Username for ‘https://git.heroku.com’: Password for ‘https://git.heroku.com’: then the repository is not pushed, because heroku will not allow […]

Read more at the source

The pros and cons of 4 deployment process techniques

The way of deliver your product code to your customer is commonly called “deployment”. It is an important matter because it will impact in how fast your product will respond to changes and the quality of each change. Depending on which deployment decision you take, it will impact your team and how you use your … »

Read more at the source

Ezra Zygmuntowicz — Farewell, Friend.

Yesterday, I found out that Ezra Zygmuntowicz had passed away.

Ezra and I first met in the #Caboose family. The first time we got to spend time together, in person, was at RailsConf 2006.

(photo credit: Jarkko Laine)

A few weeks later, he emailed me to ask about getting a job here at Planet Argon. We couldn’t afford him so he continued to pursue other paths… and a month later was helping found EngineYard.

(…one of those things that I still find myself asking, “what if we could have?”)

Back when Planet Argon had a hosting department, Ezra and I collaborated on various deployment best-practice projects. He was always helping our team figure out how to make deployment easier. Some of us might remember his famous nginx configuration (the closest version I could find). He was an active contributor in the Ruby on Rails Deployment group that I started. He was always around to help the community.

Ezra always seemed to find time for the community… whether on mailing lists, at conferences, commenting on our blogs, or chatting over IRC.

When Ezra and his family moved to Portland, we made several half-assed attempts to schedule time to catch up again in person. It never happened and our interactions were limited to keeping up on Facebook

His passing is a great loss to our community.

To his friends and family, I am sorry for your loss.

To his son, (should you ever discover this), your father helped pave the way for hundreds of thousands (if not millions?) of software developers around the globe—whether they know it or not. He was constantly looking for innovative ways to solve problems. His talks at conferences were always fascinating… and the rest of us would sit back and think, “where does he come up with these ideas?!”

He was a trailblazer.

He will be missed.

Thank you for everything, Ezra.

Update: If you would like to contribute to the memorial fund for Ezra’s son, Ryland, please visit this campaign on indigogo.

Read more at the source

Six days to complete the Rails hosting survey

Attention all Ruby on Rails Developers… you have six days to complete the 2014 Rails Hosting Survey.

All of the results will be shared with the community. Consider this your civic duty of the day.

http://rails-hosting.com

Read more at the source

2014 Rails Hosting Survey is now open

Hello! It’s been two years since Planet Argon last asked the Ruby on Rails community how they were deploying and hosting their applications and we thought it was time to checkin with people.

Until April 30th, you can take the survey here.

Read more at the source

2012 Rails Hosting Survey results

A few months ago we invited the Ruby on Rails community to participate in our hosting survey. We’ve since published the results and created an infographic.

To learn more, check out the Rails Hosting site.

Read more at the source

2012 Rails Hosting Survey results

A few months ago we invited the Ruby on Rails community to participate in our hosting survey. We’ve since published the results and created an infographic.

To learn more, check out the Rails Hosting site.

Read more at the source

Gold Master Testing

February 20, 2014

Eyelids closed, gold sun shines on

The world’s coated in the gold Krylon.

–Macklemore and Ryan Lewis, featuring Eighty4 Fly

At Code Climate, we feel it’s critical to deliver dependable and accurate static analysis results. To do so, we employ a variety of quality assurance techniques, including unit tests, acceptance tests, manual testing and incremental rollouts. They all are valuable, but we still had too much risk of introducing hard-to-detect bugs. To fill the gap, we’ve added a new tactic to our arsenal: gold master testing.

Gold master testing refers to capturing the result of a process, and then comparing future runs against the saved “gold master” (or known good) version to discover unexpected changes. For us, that means running full Code Climate analyses of a number of open source repos, and validating every aspect of the result. We only started doing this last week, but it’s already caught some hard-to-detect bugs that we otherwise may not have discovered until code hit production.

Why gold master testing?

Gold master testing is common when working with legacy code. Rather than trying to specify all of the logical paths through an untested module, you can feed it a varied set of inputs and turn the outputs into automatically verifying tests. There’s no guarantee the outputs are correct in this case, but at least you can be sure they don’t change (which, in some systems is even more important).

For us, given that we have a relatively reliable and comprehensive set of unit tests for our analysis code, the situation is a bit different. In short, we find gold master testing valuable because of three key factors:

The inputs and output from our analysis is extremely detailed. There are a huge number of syntactical structures in Ruby, and we derive a ton of information from them.
Our analysis depends on external code that we do not control, but do want to update from time-to-time (e.g. RubyParser)
We are extremely sensitive to any changes in results. For example, even a tiny variance in our detection of complex methods across the 20k repositories we analyze would ripple into changes of class ratings, resulting in incorrect metrics being delivered to our customers.

These add up to mean that traditional unit and acceptance testing is necessary but not sufficient. We use unit and acceptance tests to provide faster results and more localized detection of regressions, but we use our gold master suite (nicknamed Krylon) to sanity check our results against a dozen or so repositories before deploying changes.

How to implement gold master testing

The high level plan is pretty straightforward:

Choose (or randomly generate, using a known seed) a set of inputs for your module or program.
Run the inputs through a known-good version of the system, persisting the output.
When testing a change, run the same inputs through the new version of the system and flag any output variation.
For each variation, have a human determine whether or not the change is expected and desirable. If it is, update the persisted gold master records.

The devil is in the details, of course. In particular, if the outputs of your system are non-trivial (in our case a set of MongoDB documents spanning multiple tables), persisting them was a little tricky. We could keep them in MongoDB, of course, but that would not make them as accessible to humans (and tools like diff and GitHub) as a plain-test format like JSON would. So I wrote a little bit of code to dump records out as JSON:

dir = "krylon/#{slug}"
repo_id = Repo.create!(url: "git://github.com/#{slug}")
run_analysis(repo_id)
FileUtils.mkdir_p(dir)

%w[smells constants etc.].each do |coll|
  File.open("#{dir}/#{coll}.json", "w") do |f|
    docs = db[coll].find(repo_id: repo_id).map do |doc|
      round_floats(doc.except(*ignored_fields))
    end

    sorted_docs = JSON.parse(docs.sort_by(&:to_json).to_json)
    f.puts JSON.pretty_generate(sorted_docs)
  end
end

Then there is the matter of comparing the results of a test run against the gold master. Ruby has a lot of built-in functionality that makes this relatively easy, but it took a few tries to get a harness set up properly. We ended up with something like this:

dir = "krylon/#{slug}"
repo_id = Repo.create!(url: "git://github.com/#{slug}")
run_analysis(repo_id)

%w[smells constants etc.].each do |coll|
  actual_docs = db[coll].find(repo_id: repo_id).to_a
  expected_docs = JSON.parse(File.read("#{dir}/#{coll}.json"))

  actual_docs.each do |actual|
    actual = JSON.parse(actual.to_json).except(*ignored_fields)

    if (index = expected_docs.index(actual))
      # Delete the match so it can only match one time
      expected_docs.delete_at(index)
    else
      puts "Unable to find match:"
      puts JSON.pretty_generate(JSON.parse(actual.to_json))
      puts
      puts "Expected:"
      puts JSON.pretty_generate(JSON.parse(expected_docs.to_json))
      raise
    end
  end

  if expected_docs.empty?
    puts "    PASS #{coll} (#{actual_docs.count} docs)"
  else
    puts "Expected not empty after search. Remaining:"
    puts JSON.pretty_generate(JSON.parse(expected_docs.to_json))
    raise
  end
end

All of this is invoked by a couple Rake tasks:

1 2	`rake krylon:save[brynary/rack-test] # Save the results to disk rake krylon:validate[brynary/rack-test] # Validate against the gold master`

Our CI system runs the rake krylon:validate task. If it fails, someone on the Code Climate team reviews the results, and either fixes an issue or uses rake krylon:save to update the gold master.

Gotchas

In building Krylon, we ran into a few issues. They were all pretty simple to fix, but I’ll list them here to hopefully save someone some time:

Floats – Floating point numbers can not be reliably compared using the equality operator. We took the approach of rounding them to two decimal places, and that has been working so far.
Timestamps – Columns like created_at, updated_at will vary every time your code runs. We just exclude them.
Record IDs – Same as above.
Non-deterministic ordering of hash keys and arrays – This took a bit more time to track down. Sometimes Code Climate would generate hashes or arrays, but the order of those data structures was undefined and variable. We had two choices: update the Krylon validation code to allow this, or make them deterministic. We went with updating the production code to be deterministic with respect to order because it was simple.

Wrapping up

Gold master testing is not a substitute for unit tests and acceptance tests. However, it can be a valuable tool in your toolbox for dealing with legacy systems, as well as certain specialized cases. It’s a fancy name, but implementing a basic system took less than a day and began yielding benefits right away. Like us, you can start with something simple and rough, and iterate it down the road.

[…]

The Ruby Rails Aggregator