I was going to tweet about this, but then I thought I’d have to make a bunch of tweets, and writing a blurgh post just seemed easier.
Plus I don’t really have any puns in this post, so I can’t tweet it!

My Career Goals

I think many …

Since forest fires have started to become a normal thing in the PNW, I’ve gotten interested in monitoring the air quality in and around my house.
I found some sensors that will measure PM2.5 which is a standard for measuring air quality.
The sensor I’m using is a PMS5003, and you can see the data sheet for it here.
I like this sensor because it supports UART, so I was able to hook it to an FTDI and read data directly from my computer.
I wanted to log the data, so I hooked it up to a Raspberry PI.
However, I decided I’d like to measure the air quality in my office, a second room in the house, and also outside.
Buying a Raspberry Pi for every sensor I purchase seems a little unreasonable, so I investigated a different solution.
I settled on the ESP8266 E-01.
This part can connect to wifi, knows how to speak UART, and is powerful enough to program directly.
My plan was to read data from the sensor, then broadcast the data via UDP and have a central Raspberry Pi collect the data and report on it.
Unfortunately, this plan has taken me many months to execute, so I’m going to write here the stuff I wish I had known when getting started.

Parts

Here are the parts I used:

• ESP8266
• Plantower PMS5003
• ESP8266 Breadboard Adapter
• ESP8266 Programmer

Wiring

Basically I just hooked the TX / RX pins to the Plantower sensor and set the CHPD and RST pins to high.

Challenges with the ESP8266

Now I’m basically going to complain about this chip, and then I’ll post the code I used.

The first issue I ran in to is that I’m not sure what to call this thing, so searching the internet became a challenge.
It seems that “ESP8266” refers to the chip, but E-01 refers to the package?
I’m still not actually sure.
It seems there are several boards that have an ESP8266 mounted on them, but searching for ESP8266 with E01 seemed to work.

The second issue is programming the chip.
I prefer to use C when developing for embedded systems, but no matter how hard I tried, I could not get the native toolchain running on MacOS.
Finally I gave up and just used the Arduino toolchain.
Somehow, you can write programs for the ESP8266 in Arduino, but doing it in C seems impossible (on Mac anyway).

Building a circuit to program the chip seems impossible.
I found some schematics online for building a programmer, but I couldn’t get anything to work.
Instead, I ended up buying a dedicated programmer, and it seems to work well.

Power requirements are extremely finicky.
The chip wants 3.3v and at times 400mA.
If either of these criteria aren’t met, the chip won’t work.
Sometimes the chip wouldn’t do anything.
Sometimes it would start, but when it tried to do wifi it would just restart.
I ended up connecting a dedicated power supply to get the right power requirements.

The ESP8266 E-01 is not breadboard friendly.
I ended up buying some breadboard adapters so I could prototype.

CHPD and RST need to be pulled HIGH for the chip to boot.
This got me for a long time.
I was able to program the chip with the programmer, but as soon as I moved it to the breadboard, nothing worked.
In order to get the chip to actually boot, both CHPD and RST need to be pulled high.

The air quality sensor is 5v.
This isn’t too much of a problem, just kind of annoying that I really really have to use two different voltages for this task.

Picture

Here is a picture of the breadboard setup I have now:

The blue box on the right is the air quality sensor, in the middle on the breadboard is the ESP8266, and up top is the power supply.

Code

Here is the Arduino code I used:

#include <ESP8266WiFi.h>
#include <WiFiUdp.h>
#include <ESP8266WiFiMulti.h>
#include <base64.h>

#ifndef STASSID
#define STASSID "WifiAPName"
#define STAPSK  "WifiPassword"
#endif

const char* ssid     = STASSID;
const char* password = STAPSK;

ESP8266WiFiMulti WiFiMulti;
WiFiUDP udp;
IPAddress broadcastIp(224, 0, 0, 1);

byte inputString[32];
int i = 0;
int recordId = 0;

void setup() {
  Serial.begin(9600);

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP(ssid, password);

  while (WiFiMulti.run() != WL_CONNECTED) {
    delay(500);
  }

  delay(500);
}

void loop() {
  while (Serial.available()) {
    inputString[i] = Serial.read();
    i++;
    if (i == 2) { // Check for start of packet
      if (!(inputString[0] == 0x42 && inputString[1] == 0x4d)) {
        i = 0;
      }
    }
    if (i == 32) {
      i = 0;
      String encoded = base64::encode(inputString, 32);
      udp.beginPacketMulticast(broadcastIp, 9000, WiFi.localIP());
      udp.print("[\"aq\",{\"mac\":\"");
      udp.print(WiFi.macAddress());
      udp.print("\",\"record_id\":");
      udp.print(recordId);
      udp.print(",\"record\":\"");
      udp.print(encoded);
      udp.print("\"}]");
      udp.endPacket();
      recordId++;
    }
  }
}

I haven’t added CRC checking in this code, but it seems to work fine.
Basically it reads data from the AQ sensor, Base64 encodes the data, then broadcasts the info as JSON over UDP on my network.

Here is the client code:

require "socket"
require "ipaddr"
require "json"

MULTICAST_ADDR = "224.0.0.1"
BIND_ADDR = "0.0.0.0"
PORT = 9000

if_addr = Socket.getifaddrs.find { |s| s.addr.ipv4? && !s.addr.ipv4_loopback? }
p if_addr.addr.ip_address

socket = UDPSocket.new
membership = IPAddr.new(MULTICAST_ADDR).hton + IPAddr.new(BIND_ADDR).hton

socket.setsockopt(:IPPROTO_IP, :IP_ADD_MEMBERSHIP, membership)
socket.setsockopt(:IPPROTO_IP, :IP_MULTICAST_TTL, 1)
socket.setsockopt(:SOL_SOCKET, :SO_REUSEPORT, 1)

socket.bind(BIND_ADDR, PORT)

class Sample < Struct.new(:time,
                          :pm1_0_standard, :pm2_5_standard, :pm10_standard,
                          :pm1_0_env,      :pm2_5_env,
                          :concentration_unit,

                          # These fields are "number of particles beyond N um
                          # per 0.1L of air". These numbers are multiplied by
                          # 10, so 03um == "number of particles beyond 0.3um
                          # in 0.1L of air"
                          :particle_03um,   :particle_05um,   :particle_10um,
                          :particle_25um,   :particle_50um,   :particle_100um)
end

loop do
  m, _ = socket.recvfrom(2000)
  record = JSON.load(m)[1]

  data = record["record"].unpack("m0").first
  unpack = data.unpack('CCnn14')
  crc = 0x42 + 0x4d + 28 + data.bytes.drop(4).first(26).inject(:+)
  unless crc != unpack.last
    p Sample.new(Time.now.utc, *unpack.drop(3).first(12))
  end
end

This code just listens for incoming data and prints it out.

I’ve posted the code here.

Conclusion

This is what I did over the long weekend!
Since the AQ sensor only uses the RX and TX pins on the ESP8266, it means I’ve got at least two more GPIO pins left.
Next I’ll add temperature and humidity sensor, then make something a bit more permanent.

Let’s start today’s post with a weird Ruby benchmark:

require "benchmark/ips"

class Foo
  def initialize forward
    forward ? go_forward : go_backward
  end

  ivars = ("a".."zz").map { |name| "@#{name} = 5" }

  # define the go_forward method
  eval "def go_forward; #{ivars.join("; ")} end"

  # define the go_backward method
  eval "def go_backward; #{ivars.reverse.join("; ")} end"
end

# Heat
Foo.new true
Foo.new false

Benchmark.ips do |x|
  x.report("backward") { 5000.times { Foo.new false } }
  x.report("forward")  { 5000.times { Foo.new true } }
end

This code defines a class that sets a bunch of instance variables, but the order
that the instance variables are set depends on the parameter passed in to the
constructor. When we pass true, it defines instance variables “a” through
“zz”, and when we pass false it defines them “zz” through “a”.

Here’s the result of the benchmark on my machine:

$ ruby weird_bench.rb
Warming up --------------------------------------
            backward     3.000  i/100ms
             forward     2.000  i/100ms
Calculating -------------------------------------
            backward     38.491  (±10.4%) i/s -    192.000  in   5.042515s
             forward     23.038  (± 8.7%) i/s -    114.000  in   5.004367s

For some reason, defining the instance variables backwards is faster than
defining the instance variables forwards. In this post we’ll discuss why. But
for now, just know that if you want performant code, always define your instance
variables backwards (just kidding, don’t do that).

How Are Instance Variables Stored?

In Ruby (specifically MRI), object instances point at an array, and instance
variables are stored in that array. Of course, we refer to instance variables
by names, not by array indexes, so Ruby keeps a map of “names to indexes” which
is stored on the class of the object.

Let’s say we have some code like this:

class Foo
  def initialize
    @a = "foo"
    @b = "bar"
    @c = "baz"
    @d = "hoge"
  end
end

Foo.new

Internally, the object relationship will look something like this:

The class points at a map of “names to indexes” called the “IV Index Table”.
The IV Index Table contains the names of the instance variables along with the
index of where to find that instance variable.

The instance points at the class, and also points at an array that contains the
actual values of the instance variables.

Why go to all this trouble to map instance variable names to array offsets? The
reason is that it is much faster to access an array element than look up
something from a hash. We do have to do a hash lookup to find the array
element, but instance variables have their own inline cache,
so the lookup doesn’t occur very often.

Setting Instance Variables in Slow Motion

I want to walk through exactly what happens when instance variables are set, but
we’re going to do it twice. We’ll use the code below:

class Foo
  def initialize
    @a = "foo"
    @b = "bar"
    @c = "baz"
    @d = "hoge"
  end
end

Foo.new
Foo.new

Ruby creates the instance variable index table lazily, so it doesn’t actually
exist until the first time the code executes. The following GIF shows the
execution flow for the first time Foo.new is called:

The first time initialize is executed, the Foo class doesn’t have an
instance variable index table associated with it, so when the first instance
variable @a is set, we create a new index table, then set @a to be index 0,
then set the value “foo” in the instance variable array at index 0.

When we see instance variable @b, it doesn’t have an entry in the index table,
so we add a new entry that points to position 1, then set position 1 in the
array to “bar”.

This process repeats for each of the instance variables in the method.

Now lets look at what happens the second time we call Foo.new:

This time, the class already has an instance variable index table associated
with it. When the instance variable @a is set, it exists in the index table
with position 0, so we set “foo” to position 0 in the instance variable list.

When we see instance variable @b, it already has an entry in the index table
with position 1, so we set “bar” to position 1 in the instance variable list.

This process repeats for each of the variables in the method.

We can actually observe the lazy creation of the index table by using
ObjectSpace.memsize_of:

require "objspace"

class Foo
  def initialize
    @a = "foo"
    @b = "bar"
    @c = "baz"
    @d = "hoge"
  end
end

p ObjectSpace.memsize_of(Foo) # => 520
Foo.new
p ObjectSpace.memsize_of(Foo) # => 672
Foo.new
p ObjectSpace.memsize_of(Foo) # => 672

The size of Foo is smaller before we instantiate our first instance, but
remains the same size after subsequent allocations. Neat!

Lets do one more example, but with the following code:

class Foo
  def initialize init_all
    if init_all
      @a = "foo"
      @b = "bar"
      @c = "baz"
      @d = "hoge"
    else
      @c = "baz"
      @d = "hoge"
    end
  end
end

Foo.new true
Foo.new false

After the first call of Foo.new true, the Foo class will have an instance
variable index table just like the previous examples. @a will be associated
with position 0, @b with position 1, and so on. But what happens on the
second allocation at Foo.new false?

In this case, we already have an index table associated with the class, but @c
is associated with position 2 in the instance variable array, so we have to
expand the array leaving position 0 and 1 unset (internally Ruby sets them to
Qundef). Then @d is associated with position 3, and it is set as usual.

The important part about this is that instance variable lists must expand to the
width required for the index offset. Now lets talk about how the list expands.

Instance Variable List Allocation and Expansion

We saw how the instance variable index table is created. Now I want to spend
some time focusing on the instance variable list. This list is associated with
the instance and stores references to our actual instance variable values.

This list is lazily allocated and expands as it needs to accommodate more
values. Here is the code that figures out by how much the array should grow.

I’ve translated that function to Ruby code and added a few more comments:

def iv_index_tbl_newsize(ivup)
  index = ivup.index
  newsize = (index + 1) + (index + 1)/4 # (index + 1) * 1.25


  # if the index table *wasn't* extended, then clamp the newsize down to
  # the size of the index table.  Otherwise, use a size 25% larger than
  # the requested index
  if !ivup.iv_extended && ivup.index_table.size < newsize
    ivup.index_table.size
  else
    newsize
  end
end

IVarUpdate = Struct.new(:index, :iv_extended, :index_table)
index_table = { a: 0, b: 1, c: 2, d: 3 } # table from our examples

# We're setting `@c`, which has an index of 2. `false` means we didn't mutate
# the index table.
p iv_index_tbl_newsize(IVarUpdate.new(index_table[:c], false, index_table))

The return value of iv_index_tbl_newsize is used to determine how much memory
we need for the instance variable array. As you can see, its return value is
based on the index of the instance variable, and we got that index from the
index table.

If the index table was mutated, then we’ll allow the instance variable list to
grow without bounds. But if the index table was not mutated, then we clamp
the array size to the size of the index table.

This means that the first time we allocate a particular Ruby object, it can be
larger than subsequent allocations. Again, we can use
ObjectSpace.memsize_of to observe this behavior:

require "objspace"

class Foo
  def initialize
    @a = "foo"
    @b = "bar"
    @c = "baz"
    @d = "hoge"
  end
end

p ObjectSpace.memsize_of(Foo.new) # => 80
p ObjectSpace.memsize_of(Foo.new) # => 72
p ObjectSpace.memsize_of(Foo.new) # => 72

The first allocation is larger because it’s the first time we’ve “seen” these
instance variables. The subsequent allocations are smaller because Ruby clamps
the instance variable array size.

Watching the Instance Variable Array Grow

Let’s do one more experiment before we get on to why the initial benchmark behaves
the way it does. Here we’re going to watch the size of the object grow as we
add instance variables (again, using ObjectSpace.memsize_of):

require "objspace"

class Foo
  def initialize
    @a = 1
    p ObjectSpace.memsize_of(self)
    @b = 1
    p ObjectSpace.memsize_of(self)
    @c = 1
    p ObjectSpace.memsize_of(self)
    @d = 1
    p ObjectSpace.memsize_of(self)
    @e = 1
    p ObjectSpace.memsize_of(self)
    @f = 1
    p ObjectSpace.memsize_of(self)
    @g = 1
    p ObjectSpace.memsize_of(self)
    @h = 1
    p ObjectSpace.memsize_of(self)
  end
end

puts "First"
Foo.new
puts "Second"
Foo.new

Here’s the output from the program:

$ ruby ~/thing.rb 
First
40
40
40
80
80
96
96
120
Second
40
40
40
80
80
96
96
104

You can see that as we add instance variables to the object, the object gets
bigger! Let’s make one change to the benchmark and run it again. This time
we’ll add an option that lets us define the “last” instance variable first:

require "objspace"

class Foo
  def initialize eager_h
    if eager_h
      @h = 1
    end
    @a = 1
    p ObjectSpace.memsize_of(self)
    @b = 1
    p ObjectSpace.memsize_of(self)
    @c = 1
    p ObjectSpace.memsize_of(self)
    @d = 1
    p ObjectSpace.memsize_of(self)
    @e = 1
    p ObjectSpace.memsize_of(self)
    @f = 1
    p ObjectSpace.memsize_of(self)
    @g = 1
    p ObjectSpace.memsize_of(self)
    @h = 1
    p ObjectSpace.memsize_of(self)
  end
end

puts "First"
Foo.new false
puts "Second"
Foo.new true

Here’s the output:

$ ruby ~/thing.rb
First
40
40
40
80
80
96
96
120
Second
104
104
104
104
104
104
104
104

On the first allocation, we can observe the size of the object gradually expand
as usual. However, on the second allocation, we ask it to eagerly set @h and
the growth pattern is totally different. In fact, it doesn’t grow at all!

Since @h is last in our index table, Ruby immediately expands the array list
in order to set the value for the @h slot. Since the instance variable array
is now at maximum capacity, none of the subsequent instance variable sets need
the array to expand.

Back To Our Initial Benchmark

Every time Ruby needs to expand the instance variable array, it requires calling
realloc in order to expand that chunk of memory. We can observe calls to
realloc using dtrace.

class Foo
  def initialize forward
    forward ? go_forward : go_backward
  end

  ivars = ("a".."zz").map { |name| "@#{name} = 5" }

  # define the go_forward method
  eval "def go_forward; #{ivars.join("; ")} end"

  # define the go_backward method
  eval "def go_backward; #{ivars.reverse.join("; ")} end"
end

# Heat
Foo.new true

if ARGV[0]
  1000.times { Foo.new false }
else
  1000.times { Foo.new true }
end

Here I’ve rewritten the benchmark so that we can control the direction via an
environment variable. Let’s use dtrace to measure the number of calls to
realloc in both situations.

This case is always going forward:

$ sudo dtrace -q -n 'pid$target::realloc:entry { @ = count(); }' -c "/Users/aaron/.rbenv/versions/ruby-trunk/bin/ruby thing.rb"
dtrace: system integrity protection is on, some features will not be available


             8369

This case is forward once, then reverse the rest of the time:

$ sudo dtrace -q -n 'pid$target::realloc:entry { @ = count(); }' -c "/Users/aaron/.rbenv/versions/ruby-trunk/bin/ruby thing.rb reverse"
dtrace: system integrity protection is on, some features will not be available


             4369

We can see that “starting from the end” decreases the number of calls to
realloc significantly. These increased calls to realloc are why it’s faster
to define our instance variables forward once, then backward the rest of the
time!

I hope this was an interesting article. Please have a good day!