Frequently I hear speculations about story points and their relation with the development time. Questions like: “Why a 3 points card took so much time to be developed” “How long it takes to deliver an 8 points card?”, “Why the team took so long to deliver only this amount of points?” and others are frequent. However, when … »
Read more at the sourceO Papel do Líder de Produto
Qual a sua teoria favorita sobre o papel de um líder? Para mim um resumo perfeito é esta fala do Jack Welch em What is the role of a leader. Em resumo, sem a pretensão de captar a emoção de ouvi-lo: Ser o “Chief Meaning Officer”! Deixar claro para todas as pessoas à sua volta: para … »
Read more at the sourceMy Career Goals
I was going to tweet about this, but then I thought I’d have to make a bunch of tweets, and writing a blurgh post just seemed easier.
Plus I don’t really have any puns in this post, so I can’t tweet it!
My Career Goals
I think many …
Read more at the sourceESP8266 and Plantower Particle Sensor
Since forest fires have started to become a normal thing in the PNW, I’ve gotten interested in monitoring the air quality in and around my house.
I found some sensors that will measure PM2.5 which is a standard for measuring air quality.
The sensor I’m using is a PMS5003, and you can see the data sheet for it here.
I like this sensor because it supports UART, so I was able to hook it to an FTDI and read data directly from my computer.
I wanted to log the data, so I hooked it up to a Raspberry PI.
However, I decided I’d like to measure the air quality in my office, a second room in the house, and also outside.
Buying a Raspberry Pi for every sensor I purchase seems a little unreasonable, so I investigated a different solution.
I settled on the ESP8266 E-01.
This part can connect to wifi, knows how to speak UART, and is powerful enough to program directly.
My plan was to read data from the sensor, then broadcast the data via UDP and have a central Raspberry Pi collect the data and report on it.
Unfortunately, this plan has taken me many months to execute, so I’m going to write here the stuff I wish I had known when getting started.
Parts
Here are the parts I used:
Wiring
Basically I just hooked the TX / RX pins to the Plantower sensor and set the CHPD and RST pins to high.
Challenges with the ESP8266
Now I’m basically going to complain about this chip, and then I’ll post the code I used.
The first issue I ran in to is that I’m not sure what to call this thing, so searching the internet became a challenge.
It seems that “ESP8266” refers to the chip, but E-01 refers to the package?
I’m still not actually sure.
It seems there are several boards that have an ESP8266 mounted on them, but searching for ESP8266 with E01 seemed to work.
The second issue is programming the chip.
I prefer to use C when developing for embedded systems, but no matter how hard I tried, I could not get the native toolchain running on MacOS.
Finally I gave up and just used the Arduino toolchain.
Somehow, you can write programs for the ESP8266 in Arduino, but doing it in C seems impossible (on Mac anyway).
Building a circuit to program the chip seems impossible.
I found some schematics online for building a programmer, but I couldn’t get anything to work.
Instead, I ended up buying a dedicated programmer, and it seems to work well.
Power requirements are extremely finicky.
The chip wants 3.3v and at times 400mA.
If either of these criteria aren’t met, the chip won’t work.
Sometimes the chip wouldn’t do anything.
Sometimes it would start, but when it tried to do wifi it would just restart.
I ended up connecting a dedicated power supply to get the right power requirements.
The ESP8266 E-01 is not breadboard friendly.
I ended up buying some breadboard adapters so I could prototype.
CHPD and RST need to be pulled HIGH for the chip to boot.
This got me for a long time.
I was able to program the chip with the programmer, but as soon as I moved it to the breadboard, nothing worked.
In order to get the chip to actually boot, both CHPD and RST need to be pulled high.
The air quality sensor is 5v.
This isn’t too much of a problem, just kind of annoying that I really really have to use two different voltages for this task.
Picture
Here is a picture of the breadboard setup I have now:
The blue box on the right is the air quality sensor, in the middle on the breadboard is the ESP8266, and up top is the power supply.
Code
Here is the Arduino code I used:
#include <ESP8266WiFi.h> #include <WiFiUdp.h> #include <ESP8266WiFiMulti.h> #include <base64.h> #ifndef STASSID #define STASSID "WifiAPName" #define STAPSK "WifiPassword" #endif const char* ssid = STASSID; const char* password = STAPSK; ESP8266WiFiMulti WiFiMulti; WiFiUDP udp; IPAddress broadcastIp(224, 0, 0, 1); byte inputString[32]; int i = 0; int recordId = 0; void setup() { Serial.begin(9600); WiFi.mode(WIFI_STA); WiFiMulti.addAP(ssid, password); while (WiFiMulti.run() != WL_CONNECTED) { delay(500); } delay(500); } void loop() { while (Serial.available()) { inputString[i] = Serial.read(); i++; if (i == 2) { // Check for start of packet if (!(inputString[0] == 0x42 && inputString[1] == 0x4d)) { i = 0; } } if (i == 32) { i = 0; String encoded = base64::encode(inputString, 32); udp.beginPacketMulticast(broadcastIp, 9000, WiFi.localIP()); udp.print("[\"aq\",{\"mac\":\""); udp.print(WiFi.macAddress()); udp.print("\",\"record_id\":"); udp.print(recordId); udp.print(",\"record\":\""); udp.print(encoded); udp.print("\"}]"); udp.endPacket(); recordId++; } } }
I haven’t added CRC checking in this code, but it seems to work fine.
Basically it reads data from the AQ sensor, Base64 encodes the data, then broadcasts the info as JSON over UDP on my network.
Here is the client code:
require "socket" require "ipaddr" require "json" MULTICAST_ADDR = "224.0.0.1" BIND_ADDR = "0.0.0.0" PORT = 9000 if_addr = Socket.getifaddrs.find { |s| s.addr.ipv4? && !s.addr.ipv4_loopback? } p if_addr.addr.ip_address socket = UDPSocket.new membership = IPAddr.new(MULTICAST_ADDR).hton + IPAddr.new(BIND_ADDR).hton socket.setsockopt(:IPPROTO_IP, :IP_ADD_MEMBERSHIP, membership) socket.setsockopt(:IPPROTO_IP, :IP_MULTICAST_TTL, 1) socket.setsockopt(:SOL_SOCKET, :SO_REUSEPORT, 1) socket.bind(BIND_ADDR, PORT) class Sample < Struct.new(:time, :pm1_0_standard, :pm2_5_standard, :pm10_standard, :pm1_0_env, :pm2_5_env, :concentration_unit, # These fields are "number of particles beyond N um # per 0.1L of air". These numbers are multiplied by # 10, so 03um == "number of particles beyond 0.3um # in 0.1L of air" :particle_03um, :particle_05um, :particle_10um, :particle_25um, :particle_50um, :particle_100um) end loop do m, _ = socket.recvfrom(2000) record = JSON.load(m)[1] data = record["record"].unpack("m0").first unpack = data.unpack('CCnn14') crc = 0x42 + 0x4d + 28 + data.bytes.drop(4).first(26).inject(:+) unless crc != unpack.last p Sample.new(Time.now.utc, *unpack.drop(3).first(12)) end end
This code just listens for incoming data and prints it out.
I’ve posted the code here.
Conclusion
This is what I did over the long weekend!
Since the AQ sensor only uses the RX and TX pins on the ESP8266, it means I’ve got at least two more GPIO pins left.
Next I’ll add temperature and humidity sensor, then make something a bit more permanent.
Instance Variable Performance
Let’s start today’s post with a weird Ruby benchmark:
require "benchmark/ips" class Foo def initialize forward forward ? go_forward : go_backward end ivars = ("a".."zz").map { |name| "@#{name} = 5" } # define the go_forward method eval "def go_forward; #{ivars.join("; ")} end" # define the go_backward method eval "def go_backward; #{ivars.reverse.join("; ")} end" end # Heat Foo.new true Foo.new false Benchmark.ips do |x| x.report("backward") { 5000.times { Foo.new false } } x.report("forward") { 5000.times { Foo.new true } } end
This code defines a class that sets a bunch of instance variables, but the order
that the instance variables are set depends on the parameter passed in to the
constructor. When we pass true
, it defines instance variables “a” through
“zz”, and when we pass false
it defines them “zz” through “a”.
Here’s the result of the benchmark on my machine:
$ ruby weird_bench.rb
Warming up --------------------------------------
backward 3.000 i/100ms
forward 2.000 i/100ms
Calculating -------------------------------------
backward 38.491 (±10.4%) i/s - 192.000 in 5.042515s
forward 23.038 (± 8.7%) i/s - 114.000 in 5.004367s
For some reason, defining the instance variables backwards is faster than
defining the instance variables forwards. In this post we’ll discuss why. But
for now, just know that if you want performant code, always define your instance
variables backwards (just kidding, don’t do that).
How Are Instance Variables Stored?
In Ruby (specifically MRI), object instances point at an array, and instance
variables are stored in that array. Of course, we refer to instance variables
by names, not by array indexes, so Ruby keeps a map of “names to indexes” which
is stored on the class of the object.
Let’s say we have some code like this:
class Foo def initialize @a = "foo" @b = "bar" @c = "baz" @d = "hoge" end end Foo.new
Internally, the object relationship will look something like this:
The class points at a map of “names to indexes” called the “IV Index Table”.
The IV Index Table contains the names of the instance variables along with the
index of where to find that instance variable.
The instance points at the class, and also points at an array that contains the
actual values of the instance variables.
Why go to all this trouble to map instance variable names to array offsets? The
reason is that it is much faster to access an array element than look up
something from a hash. We do have to do a hash lookup to find the array
element, but instance variables have their own inline cache,
so the lookup doesn’t occur very often.
Setting Instance Variables in Slow Motion
I want to walk through exactly what happens when instance variables are set, but
we’re going to do it twice. We’ll use the code below:
class Foo def initialize @a = "foo" @b = "bar" @c = "baz" @d = "hoge" end end Foo.new Foo.new
Ruby creates the instance variable index table lazily, so it doesn’t actually
exist until the first time the code executes. The following GIF shows the
execution flow for the first time Foo.new
is called:
The first time initialize
is executed, the Foo
class doesn’t have an
instance variable index table associated with it, so when the first instance
variable @a
is set, we create a new index table, then set @a
to be index 0,
then set the value “foo” in the instance variable array at index 0.
When we see instance variable @b
, it doesn’t have an entry in the index table,
so we add a new entry that points to position 1, then set position 1 in the
array to “bar”.
This process repeats for each of the instance variables in the method.
Now lets look at what happens the second time we call Foo.new
:
This time, the class already has an instance variable index table associated
with it. When the instance variable @a
is set, it exists in the index table
with position 0, so we set “foo” to position 0 in the instance variable list.
When we see instance variable @b
, it already has an entry in the index table
with position 1, so we set “bar” to position 1 in the instance variable list.
This process repeats for each of the variables in the method.
We can actually observe the lazy creation of the index table by using
ObjectSpace.memsize_of
:
require "objspace" class Foo def initialize @a = "foo" @b = "bar" @c = "baz" @d = "hoge" end end p ObjectSpace.memsize_of(Foo) # => 520 Foo.new p ObjectSpace.memsize_of(Foo) # => 672 Foo.new p ObjectSpace.memsize_of(Foo) # => 672
The size of Foo
is smaller before we instantiate our first instance, but
remains the same size after subsequent allocations. Neat!
Lets do one more example, but with the following code:
class Foo def initialize init_all if init_all @a = "foo" @b = "bar" @c = "baz" @d = "hoge" else @c = "baz" @d = "hoge" end end end Foo.new true Foo.new false
After the first call of Foo.new true
, the Foo
class will have an instance
variable index table just like the previous examples. @a
will be associated
with position 0, @b
with position 1, and so on. But what happens on the
second allocation at Foo.new false
?
In this case, we already have an index table associated with the class, but @c
is associated with position 2 in the instance variable array, so we have to
expand the array leaving position 0 and 1 unset (internally Ruby sets them to
Qundef
). Then @d
is associated with position 3, and it is set as usual.
The important part about this is that instance variable lists must expand to the
width required for the index offset. Now lets talk about how the list expands.
Instance Variable List Allocation and Expansion
We saw how the instance variable index table is created. Now I want to spend
some time focusing on the instance variable list. This list is associated with
the instance and stores references to our actual instance variable values.
This list is lazily allocated and expands as it needs to accommodate more
values. Here is the code that figures out by how much the array should grow.
I’ve translated that function to Ruby code and added a few more comments:
def iv_index_tbl_newsize(ivup) index = ivup.index newsize = (index + 1) + (index + 1)/4 # (index + 1) * 1.25 # if the index table *wasn't* extended, then clamp the newsize down to # the size of the index table. Otherwise, use a size 25% larger than # the requested index if !ivup.iv_extended && ivup.index_table.size < newsize ivup.index_table.size else newsize end end IVarUpdate = Struct.new(:index, :iv_extended, :index_table) index_table = { a: 0, b: 1, c: 2, d: 3 } # table from our examples # We're setting `@c`, which has an index of 2. `false` means we didn't mutate # the index table. p iv_index_tbl_newsize(IVarUpdate.new(index_table[:c], false, index_table))
The return value of iv_index_tbl_newsize
is used to determine how much memory
we need for the instance variable array. As you can see, its return value is
based on the index of the instance variable, and we got that index from the
index table.
If the index table was mutated, then we’ll allow the instance variable list to
grow without bounds. But if the index table was not mutated, then we clamp
the array size to the size of the index table.
This means that the first time we allocate a particular Ruby object, it can be
larger than subsequent allocations. Again, we can use
ObjectSpace.memsize_of
to observe this behavior:
require "objspace" class Foo def initialize @a = "foo" @b = "bar" @c = "baz" @d = "hoge" end end p ObjectSpace.memsize_of(Foo.new) # => 80 p ObjectSpace.memsize_of(Foo.new) # => 72 p ObjectSpace.memsize_of(Foo.new) # => 72
The first allocation is larger because it’s the first time we’ve “seen” these
instance variables. The subsequent allocations are smaller because Ruby clamps
the instance variable array size.
Watching the Instance Variable Array Grow
Let’s do one more experiment before we get on to why the initial benchmark behaves
the way it does. Here we’re going to watch the size of the object grow as we
add instance variables (again, using ObjectSpace.memsize_of
):
require "objspace" class Foo def initialize @a = 1 p ObjectSpace.memsize_of(self) @b = 1 p ObjectSpace.memsize_of(self) @c = 1 p ObjectSpace.memsize_of(self) @d = 1 p ObjectSpace.memsize_of(self) @e = 1 p ObjectSpace.memsize_of(self) @f = 1 p ObjectSpace.memsize_of(self) @g = 1 p ObjectSpace.memsize_of(self) @h = 1 p ObjectSpace.memsize_of(self) end end puts "First" Foo.new puts "Second" Foo.new
Here’s the output from the program:
$ ruby ~/thing.rb
First
40
40
40
80
80
96
96
120
Second
40
40
40
80
80
96
96
104
You can see that as we add instance variables to the object, the object gets
bigger! Let’s make one change to the benchmark and run it again. This time
we’ll add an option that lets us define the “last” instance variable first:
require "objspace" class Foo def initialize eager_h if eager_h @h = 1 end @a = 1 p ObjectSpace.memsize_of(self) @b = 1 p ObjectSpace.memsize_of(self) @c = 1 p ObjectSpace.memsize_of(self) @d = 1 p ObjectSpace.memsize_of(self) @e = 1 p ObjectSpace.memsize_of(self) @f = 1 p ObjectSpace.memsize_of(self) @g = 1 p ObjectSpace.memsize_of(self) @h = 1 p ObjectSpace.memsize_of(self) end end puts "First" Foo.new false puts "Second" Foo.new true
Here’s the output:
$ ruby ~/thing.rb
First
40
40
40
80
80
96
96
120
Second
104
104
104
104
104
104
104
104
On the first allocation, we can observe the size of the object gradually expand
as usual. However, on the second allocation, we ask it to eagerly set @h
and
the growth pattern is totally different. In fact, it doesn’t grow at all!
Since @h
is last in our index table, Ruby immediately expands the array list
in order to set the value for the @h
slot. Since the instance variable array
is now at maximum capacity, none of the subsequent instance variable sets need
the array to expand.
Back To Our Initial Benchmark
Every time Ruby needs to expand the instance variable array, it requires calling
realloc
in order to expand that chunk of memory. We can observe calls to
realloc
using dtrace
.
class Foo def initialize forward forward ? go_forward : go_backward end ivars = ("a".."zz").map { |name| "@#{name} = 5" } # define the go_forward method eval "def go_forward; #{ivars.join("; ")} end" # define the go_backward method eval "def go_backward; #{ivars.reverse.join("; ")} end" end # Heat Foo.new true if ARGV[0] 1000.times { Foo.new false } else 1000.times { Foo.new true } end
Here I’ve rewritten the benchmark so that we can control the direction via an
environment variable. Let’s use dtrace
to measure the number of calls to
realloc
in both situations.
This case is always going forward:
$ sudo dtrace -q -n 'pid$target::realloc:entry { @ = count(); }' -c "/Users/aaron/.rbenv/versions/ruby-trunk/bin/ruby thing.rb"
dtrace: system integrity protection is on, some features will not be available
8369
This case is forward once, then reverse the rest of the time:
$ sudo dtrace -q -n 'pid$target::realloc:entry { @ = count(); }' -c "/Users/aaron/.rbenv/versions/ruby-trunk/bin/ruby thing.rb reverse"
dtrace: system integrity protection is on, some features will not be available
4369
We can see that “starting from the end” decreases the number of calls to
realloc
significantly. These increased calls to realloc
are why it’s faster
to define our instance variables forward once, then backward the rest of the
time!
I hope this was an interesting article. Please have a good day!
Read more at the sourceReview: Learn Functional Programming with Elixir
I have recently finished reading the book Learn Functional Programming with Elixir, from Ulisses Almeida, published by PragProg. I was curious about this book in particular because I wanted to know how a book that explains functional programming for people that are starting this journey would be My path to learning functional programming was not … »
Read more at the sourceWelcome Wojtek Mach to Plataformatec!
We are glad to welcome Wojtek Mach to the Research and Development department at Plataformatec. Wojtek will help us towards our Elixir efforts, contributing to Open Source, communicating with our Elixir teams across multiple projects and joining our Elixir Development Subscription team. Wojtek is a software engineer with 10 years of experience. He started working … »
Read more at the sourceTá quantos por cento pronto? Agile para PMPs e PMBok para agilistas
Polêmica! Ainda esses dias, conversava com um responsável pelo capítulo de agilidade de uma startup sobre reports de projeto sedutores e o mal que eles podem fazer. A situação é a seguinte: não habituados com a forma como as métricas de projetos ágeis são apresentadas – CFDs, histogramas de Throughput, Lead Time etc. – e … »
Read more at the sourceStop hiding the error and start fixing the problem
I’ve been working on Plataformatec for 5 years and one common mistake that I see developers making is hiding the error, instead of fixing the problem. This kind of behaviour can turn your product full of problems quickly by having a codebase with unnecessary defensive programming. Let’s explore that by taking a look at an … »
Read more at the source8% of pull requests are doomed
Today we’ll look at three terminal pull request outcomes and one way to increase velocity in
your engineering process.
Every pull request has one of three outcomes
Every pull request has costs: engineering labor, product management, and
opportunity cost, to name a few. Each also has an outcome: merged, closed
without merging, or abandoned due to inactivity.
Here’s a look at how pull requests fare across the industry:

If you group closed and inactive pull requests together (“Abandoned PRs”), you
can estimate that the average engineer abandons 8% of the pull requests they
create, which is equivalent to a loss of $24,000 per year1, or the cost of a
2018 Toyota Camry Hybrid.
(We consider pull requests that have had zero activity for more than three days
to be abandoned because our data shows a very low likelihood that PRs that go
untouched for so long get merged later.)
Achieving zero abandoned pull requests is an anti-goal, as it would require being
extremely conservative when opening them. However, a high rate of abandoned PRs can
indicate inefficiency and opportunity for improvement within an engineering
process. Reducing PR loss by 20% on a team with 10 engineers could save $48,000
per year.
How does my team stack up?
Using an anonymized, aggregated analysis of thousands of engineering
contributors, we’re able to get an understanding of how an engineering
organization compares to others in the industry:

This density plot shows that the average pull request loss rate across our
dataset is 8% (with a median of 6%). A loss rate above 11% would be in the
bottom quartile, and a loss rate below 3% would be upper quartile performance.
Improving pull request outcomes
Abandoned pull requests are, of course, a lagging indicator. You can tell because it
would be ridiculous to go to an engineering team and say, “All those PRs that
you’re closing… merge them instead!”
Potential drivers lie upstream: late changing product requirements, shifting
business priorities, unclear architectural direction and good ole’ fashioned
technical debt. If you have an issue with abandoned pull requests, soliciting
qualitative feedback is a great next step. Talk to your team. Identify something
that is impacting them and talk about how you might avoid it next time. Then,
rather than focus on the absolute value of your starting point, you can monitor
that your abandonment rate is going down over time.
After all, you’d probably rather not send a brand new Camry to the scrap yard
every year.
1 Assumes a fully loaded annual cost of $300k per developer.