Another good reason to begin documenting your Theory of The Program: it’s a way to combat the death of the author. If you have explicitly written down the intentions of some software, and what it assumed, then others cannot add assumptions onto you in the future. You are also forced into honesty about what you initially considered and how you perceived the world. (From reading’s thoughts on language and inclusiveness)

Documenting Theory Building as An Engineering Team

Recently, I was reading Marc Booker’s post about the need for documentation outside of the codebase. A thoughtful comment on the post led me to Peter Naur’s paper on Programming as Theory Building, which I recommend reading1. In it, Naur argues that the act of programming is not a practice of writing code but the work of creating a theory of the problem at hand and a theory of the system to address it. As a team exercise, we practiced documenting our theories of the OfficeLuv system and found it very rewarding.

  1. If you find it repetitive, I recommend skipping to the final couple pages where Naur talks about how best to actually apply “Theory Building” on your own. It comes after the actual paper. ↩︎

I would really love to see programmers/maintainers of influential libraries, frameworks, and languages (Redux, SwiftUI, Erlang, etc.) begin to describe their Theory Building as top-level documentation in the source code. I think several maintainers do good work to educate everyone with conference talks and blog posts (Rich Hickey, Dan Abramov, etc.), but documentation with the project would be amazing as an introduction with the codebase.

Scaling Queue Workers Efficiently with AppSignal Metrics

AppSignal asked me to write for their blog. My first post is about some techniques we’ve been using at OfficeLuv to responsively scale our queueing systems based on usage. From the introduction:

Ways I'm Able to Help

TLDR: email me at josh at

It is not enough to simply architect a system or solution. You must create a story, a path through it, from your current state to multiple end states. If you simply present a picture of end state, members do not know what to do to contribute. The path and the narrative is the actual useful solution. Architecture or solution alone is an empty vessel.

Often, people who don’t have access to the raw data expect one narrative to be drawn from data analysis. But the analysts actually in the data know that there are always many, many narratives in the data. Data is opposite a single narrative. The narratives coming from the data are also never closed or complete, as data continuously arrives. But people want and attach themselves to a single narrative.

Listening to win vs. listening to fix vs. listening to learn. The default position for engineers and executives is listening to fix. You’re employed to move things forward and solve problems. Sales’ default position is listening to win - to convert the sale, listening to sell. But good product work requires listening to learn. You can fix it later, iteratively, but first you need to soak it in with no idea on how to fix it. (While listening to Jennifer Garvey Berger.)

Code performance, reliability, security, cleanliness. These things are not achieved through big sweeps and special efforts. They are only really possible by baking the practices into every day and each decision. You can’t focus on fixing them for a week. You need to invest in them slowly, constantly, over the life of the project. (Thoughts after reading about why NetNewsWire is fast, listening to a code audit webinar, and thinking about how we’ve done so much with 2 engineers at OfficeLuv.)


This is outside my door. It is outside your door, too.

After Reading: The Making of Prince of Persia

Reading The Making of Prince of Persia journals, by Jordan Mechner, is energizing and stressful. They are the journals of a 20-something year old developer and writer. Every interaction is dramatic or consequential or fraught. Every downbeat or still moment is an existential crisis. I couldn’t read it before bed because it would keep me up at night. I finished reading it this afternoon and I was pacing back and forth during the last hundred pages.

On Writing Great Product Release Emails

If you release software, you’ll need to communicate new features and bug-fixes to your users. Eleni has written about how OfficeLuv practices writing our customer-facing blog posts announcing features before we build, but we don’t announce every feature in our customer-facing changelog. We do, however, put out an internal product release email every week. It has remained a constant pulse over our 4 years, but we’ve improved the structure and content over time.

Identifying too many aspects of ongoing product work as tech debt has caused engineering teams to over-invest too early. They fear the tech debt in the future if they don’t make the decision now that’s perfect for that orders-of-magnitude-bigger future. They think tech debt can be avoided with more tech investment, but I don’t think it works that way. You want to invest slowly and constantly, just like the stock market.

The actual nerdy economics joke there is in the second paragraph, which takes the form: Two economists are walking down the street. One of them sees a $20 bill on the ground. As she bends to pick it up, her colleague says “don’t bother, if that was a real $20 bill someone would have picked it up by now.” She replies, “no see this was left here by a consumer tech startup trying to maximize user growth; their Monthly Active Picker-Upper numbers are doubling every two months.” She picks up the $20 bill and the startup raises money at a $2 billion valuation.

Carving Tech Investment Out of Tech Debt

At OfficeLuv, we track our individual tasks into epics, like most other product teams. Also like most others, we’ve had one ongoing group called ‘Tech Debt’ since day 2. Recently, I went grooming through a bunch of the stories in there and decided to break that topic into two separate ideas, one for Tech Debt and one for Tech Investment.

In product research, people sometimes make the mistake of ‘validating’ their product idea by interviewing customers and describing the most generic version of their product. The customer already has a problem, so a product generic enough to solve all their problems sounds wonderful and they want that. But once you get down to building specifics and you haven’t solved all the problems right away, they won’t switch away from the incumbent solution because it doesn’t solve one of their problems better.
Software engineers are a similar way, in that they consider new technology often in generic terms and use cases. Oh that new tech idea? It’s better than everything you’ve been doing. Let’s adopt it everywhere. It’s the new best pattern. Then you get down to specifics of usage and the engineers have built terrible patterns of specifics off a promised generic that doesn’t exist.

I’m reading Edward Tufte’s Visual Explanations and the depth makes it a slow read. It’s like a museum. It takes ten minutes to read a page. I’m exhausted after only a few pages (rooms) and can come back for multiple days to a single topic.

The right data structure precipitates elegance and performance. “Re-think the data structure, gain better insights in the distinction between how the data is produced and how the data is consumed.”

Correct Ruby Threads and ActiveRecord Connections

Recently, we had an opportunity to trivially parallelize some ActiveRecord queries in our Ruby server. In a common response structure, we needed to both query for the actual data requested along with some meta-information about pagination, etc.

Qualities of a Smooth Queueing System

This morning, Eleni and I were reflecting on how our queueing system has evolved at OfficeLuv over the past year.