The Art of Documentation

Reading Time: 15 minutes

Hello, handsome.

Let’s talk about docs for a little while.

No, not ducks! DOCS.

Everyone laments that we don’t have more of them.

Seasoned programmers tease silver hairs from their scalp one by one, attaching each to a story about a time that docs failed to lead them—or worse, misled them.

“Senior” teams with shitty docs hire apprentices, then pat themselves on the back for “training them” by relegating them to documentation janitors (I said it).

Why do we suck at this, what does good even look like, and how do we get from here—wherever here is—to there—wherever there is?

Before we go through all that, I gotta address one thing first, and it’s going to piss a lot of people off. Get a cup of tea before you read this if you’ve ever found yourself, I dunno, ready to kneecap a stranger from Twitter over logarithmic complexity or something.

I really hate “more, more, more” documentation.

Technologists often feel guilt about having too little documentation, and that’s probably warranted—especially when the original implementers couldn’t be arsed to explain how anyone else might get it running or use it.

Occasionally, though, I see the “fix” swing really far in the opposite direction. Document everything! Four thousand word README! Headings in the wiki for every possible thing someone would want to do with this code, and two for common things!

I see two issues with that approach.

First, the brutal candor: no one wants to read all that. Writing is a skill. Like most skills that aren’t programming, it gets undervalued and discounted by most programmers, who also think they’re better at it than they are.

The amount of skill that it takes to just index that much information in an accessible way, let alone write clear and engaging content, is so far out of most programmers’ depth that I cannot, in good faith, ask them to attempt it. Writing—yes, even nonfiction writing—is an art. And the more writing there is, the more artful it needs to be for anyone to read it.

But suppose you think your team is that good. They’re not, but whatever. There’s another reason not to do this.

Second, the practical reason: no one wants to maintain all that. Remember when I got on that tear about technical debt and told you that all code, no matter how beautiful it is, requires maintenance? Remember how we concluded that therefore no feature—not even a finished feature—is free? Guess what: that’s not just code.

Documentation has that problem, too. It gets out of date. It loses accuracy. The more of it there is, the more likely that is to happen, because no one seems to have the same enthusiasm for auditing reams of documentation that they had for writing it in the first place. And remember—they mostly weren’t even that jazzed to write it in the first place. So the necessary fencepost maintenance on a whole bitranch of documentation is, frankly, never gonna happen.

Fine, Chelsea. So what’s the plan?

The plan is to:

  1. Identify when and why each piece of context might be important to another programmer or client
  2. Document, briefly, at exactly the point where that piece of context will matter
  3. Ensure that, if this piece of context ever becomes outdated, it changes itself, draws immediate attention to itself, or disappears altogether.

The secret to making that work is to think beyond the README (or the Wiki, or the Confluence, or the readthedocs), and choose the appropriate documentation channel for each piece of context rather than using one tool for all of it.

A Survey of Documentation Tools

Let’s start with the obvious ones that get used for everything. We’ll discuss which situations suit them best. We’ll gradually progress toward the documentation tools that get less attention in discussions about documentation (and why I think they’re so great).

Code Comments

This is a weird one because the programming community says it hates comments. People love to tell junior programmers that “if your code needs comments, it’s not clear enough on its own.” So, to avoid admitting defeat with a comment, programmers will just leave illegible code without them. The closest thing to a nuanced take on comments that I have heard is “use a comment to explain why something is the way it is, rather than what it is.”

As you know, I look to open source code more than I look to private code for real-life examples of collaborable practices, and the bottom line is that they’ve often got more comments than code. Take this plotting code from Pandas, for example, in which the docstrings easily make up more than half of the file. Python docstrings are a special kind of comment designed to inform api clients about what an api does, but they’re still strings consumed but otherwise ignored by the interpreter. They don’t auto-update. They function like comments.

Here’s another example. Check out the docstring-to-code ratio on this function from matplotlib:

My general rules on when to use code comments look like this:

  • Learn about the language I’m writing and follow community conventions. Many language communities include strings that the compiler ignores as conventional practice for various reasons. Examples: docstrings in Python or //Mark in iOS Development.
  • If I am doing something that no variable name can make obvious (often complicated algorithmic stuff is like this), I provide comments as headers in the logic.
  • If someone is likely to come along and see this line and think they should change it, and they shouldn’t, I’ll leave a comment explaining why to leave it like this.” They’re chiefly for code that looks wrong but has to be the way it is for some reason.

The common thread among all these: they probably shouldn’t change. The conventions of a community don’t change fast. An algorithm, though it might be tweaked, probably won’t undergo a structural change sufficient to invalidate structural headers without those headers getting removed or changed. The explanations on code usually explain the code as written. Comments work well for these purposes because, though it’s easy to change a line of code and leave a now-outdated comment, these are not cases where that should be happening.

Where do I put comments about stuff that’s definitely (or even probably) going to change? We’ll get there.

External Documentation (READMEs, ReadTheDocs, Confluence, et cetera)

I’m lumping all these together because they tend to get used for the same thing: a catchall after comments. Sometimes I see a distinction made that the README is for programmers and the others are for clients, but in practice, few teams seem to follow that distinction.

Isn’t it funny that we’re discouraged from writing comments because they could lose accuracy as the code changes, but we view documentation in a README or doc website as universally good despite being even more separate from the code?

This type of documentation is, to me, the highest risk of getting out-of-date. That said, programmers do need information about how to set up and run a piece of software if there are ever to be more maintainers than the original author. Plus, for code to help people, clients need to know how to use it. Here’s what I think goes in external documentation:

  • How to get the code up and running. Do we need to do any IDE setup? Enter any environment variables? How do I run the tests? I do not ever want to have to figure this stuff out on my own. I want a list of sequential directions, preferably with illustrations, because I want to get to the point where the code is running and I have my full suite of other tools (like tests, debuggers, and the like) available for figuring out what this code does. The standard of acceptability for me here is “another person can set themselves up to work on my code without my intervention.”
  • What the software does and how to do it. This does not have to be 4,000 words. It can be “This app is for X. In order to get it to do X, go to this URL, click the doThing button, enter your email in this field and the thingID in this field, and then hit submit. The job will take about 5 minutes. Look for an email at the email address you entered on the form to know when it’s done.” Again, my standard of acceptability is “My intervention should be unnecessary for someone new to my project to use it.”
  • A troubleshooting guide. If someone gets put on call for this software and gets woken up at 3 AM, what are the common problems, and how can they check for them? This section tends to start small and grow as use of the software reveals how it might break. This kind of section tends to be usable even when it’s long because the programmer who needs it can search for a verbatim exception name in the text and get ferried to exactly the relevant section.

That’s it. I don’t put other stuff in the external documentation. If I can figure something out from the tests or by setting a breakpoint, calling the method, and following the execution path, I’m not repeating it in the external documentation.

Automated Tests

This is where I start to document stuff that might change, because if the code’s behavior changes, the test better fail. As a reader of documentation, I like tests the best, because I can use them as little laboratories to understand how the code works by changing values in them and seeing what happens. To me, a good test suite has test names that describe, in full, the behavior of the code in question. The trick to making tests an aid rather than a burden is to write them at the appropriate level of abstraction.

  • I use feature tests to describe behavior and avoid regressions. These are at the very highest level. “When I visit this page and tap this button, I should see my personal widget list onscreen.”
  • I use integration tests to, essentially, defensively check my connections with other systems before deploying. They’re useful to identify when a collaborator is down or its API has changed, but I don’t stuff other functionality testing in here.
  • I use what I call multiclass tests (more on that here) to test systems of a few classes that should collaborate to accomplish a task. I prefer this to litanies of unit tests that, inevitably, add friction for beneficial refactors or API changes. Developer hesitation to perform changes that would reduce maintenance load because of having to change the tests are a telltale sign that the tests employ a lower level of abstraction than they should.
  • I use unit tests when I need to document and validate complicated logic contained within a single class or function. I start with the edge cases and work up to the “happy path.” I don’t put unit tests for getters, setters, or aliases, usually.

Git Commit Messages

Programmers, in my opinion, criminally underuse git commit messages because they have cargo-culted wisdom about an 80-character message limit from a crappy IBM UI that went out of circulation in the eighties. I wish any part of that last sentence were an exaggeration.

Git commit messages are the perfect place to store contextual information about how and why a programmer made a change set that might need further changes in the future. Why:

  1. A commit message is attached to the exact lines where the changes were made. Most modern IDEs have a view that allows a programmer to see the message associated with the last time any given line was changed.
  2. When a line changes, it gets included in a new change set with a new commit message. So, as the code changes, the message explaining why this line is the way it is also changes.
  3. Even if an individual line in a file changes and gets a new message, important, older changes that affect, say, the whole file will still be associated with other lines in that whole file. So a git annotation of a whole file will show relevant, timestamped, information about how this file has changed over time.

Nota Bene: The above only works under two circumstances.

  1. Programmers make well-circumscribed change sets. That means that they associate each commit with a single change and avoid throwing in distracting, extraneous, unrelated changes. The change can be large (like a structural refactor), but the change set should all relate to one thing then described (and if necessary, justified) in the commit message. I talk more about commit hygiene here.
  2. Programmers include a review of the git history in their context gathering step when they’re approaching code they don’t know. In addition to experimenting with the tests, I pop open the annotation view and take a look at all the commit messages associated with the file or files I plan to change. If the commit history is used as described, I can get a good idea from this of what has changed over time and why.

Here’s an example of some code I wrote in the annotation view in PyCharm, with my cursor hovering over one of the commits so you can see the message:

Hot take: I prefer the use of commit messages to architectural decision records. I have never seen an actual directory of architectural decision records come to any use besides appeasing CTOs. Teams can mandate that junior programmers read the ADR directory, but it’ll usually overwhelm them without teaching them much (see above point about writing skill). Teams can mandate their senior programmers read the ADR directory, but empirically, what happens here is the seniors go “I won’t, but I’ll say I did until I get caught with my pants down.” It’s a burden on the writing end and on the reading end, and most of what’s in there won’t relate to any individual change a programmer needs to make.

Pull Request Descriptions

I love pull request descriptions for their ideal timing and placement to transfer context to another programmer in an asynchronous work setting. I want someone to provide a thoughtful review of my code. In order to do that, they need to be able to run, test, and even make experimental changes to my code. Therefore, I need to provide, in my PR description, the context they might need to get to the point where they could do those things. My standard of acceptability here is “My PR description should make it possible, if I got hit by a bus, for my reviewer to take over this change set on my behalf.”

I got very specific about that over here already, so I won’t rehash what I said. That piece’s follow-up post is somehow even saltier than this post, if you can believe it. Please enjoy.

Automated Documentation

I like API documentation like Swagger or OpenAPI for automatically providing a playground for clients when I’m writing an HTTP API with endpoints that they hit and example requests they might use. I wrote a case study on that for Linode right over here, if you’re interested. I love this kind of automated documentation because, when the implementation of my API changes, the auto-generated documentation changes, too. Swagger also provides opportunities to include additional notes or information for that one weird thing about any API that needs to be customized.

When I can’t use Swagger or OpenAPI, I’ll often make a Postman collection of sample requests for my API, with actual sample request bodies, that others can run. I’ll include a step in CI to run Postman tests against that collection to ensure that all my example requests do what I say they do. Then I’ll allow that Postman Collection to supplant external API documentation.

Of course, I’m not always writing an HTTP API where those kinds of tricks work. I’ve been impressed with Apollo Studio for their work in providing living documentation for GraphQl requests. But what about plain old libraries that a client might need to use, like pandas or matplotlib for Python, collections for Swift, or GreenDAO for Android—or even the API of a programming language itself?

In those cases, I still fall back on a plain old, well-indexed directory of external documentation.

Sometimes, I just need to describe all of the available functions and all of the options those functions provide. This will definitely need maintenance, and it’s likely to get out of date. But I do my best to keep it as brief as possible, to index it well (providing meaningful titles and headers so folks can find what they need), and to keep regular maintenance on my to-do list.

And at least if I’ve employed the other documentation methods described above, I have a smaller corpus of external documentation to maintain than I would if I shoved every un-codified thought about my code base into it.

If you liked this piece, you might also like:

The technical debt series (come for the buzzwords, stay for the insights)

This piece on risk-oriented testing (also mentioned above)

This salty piece about pull request descriptions (which made the ‘best of’ list on Pocket in 2020!)

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.