This post is part of a blog series called Leveling Up: A Guide for Programmers. The series covers skills you can use to learn faster, more easily, and more strategically as a programmer.
In a recent post about warmup reading, I mentioned that I do not read academic papers for my warmup reading. I promised a deep dive on my approach to academic papers later. This is that deep dive.
Before I get into how I approach academic papers, let’s talk about when I do it—and when I don’t do it.
Academic papers are not central to my independent practice regimen.
That’s not necessarily a popular position. A lot of self-teaching resources love to focus on how to read academic papers. I think this is misleading because it makes the assumption that people need to be good at absorbing academic papers to benefit from academic literature in their industry jobs.
This is not consistent with my experience.
1. Just because it sounds smart doesn’t mean it’s accurate or relevant.
First of all, academic papers often possess biases that make them incomplete, incorrect, and incoherent resources on a topic, even in the aggregate. Examples:
- Journals only want to publish “significant” findings, so scientists often end up doing multiple comparisons and various different measurement methods to showcase the most significant-looking result they can obtain.
- Journals and dissertation defenses also both favor findings that are “new,” so scientists are forced to artificially differentiate their work from previous work, even if they’re substantiating existing claims or creating a model that’s extremely similar to an existing model. When done in iterations, this veers the academic literature in the direction of making everything seem more complicated than it is.
- For some inane reason, papers in certain fields are more likely to get published if they have complicated-looking math in them. So scientists shove tiered equations into the paper for the purpose of publication. Sometimes it doesn’t even substantiate the claim of the paper.
I’m not saying that scientists are incompetent know-it-alls. Plenty of them don’t love having to do this. But the biases in scientific publishing combined with the ‘publish or perish’ credo of the academic career track forces their hands to some degree. In addition, sometimes the teams publishing research don’t possess the statistical background to know that they have to watch for statistical power or multiple comparisons. Unfortunately the academic system does not stress statistical rigor in, for example, a medical graduate program.
The result: a lot of academic writing showcases inflated results, convoluted language, and excess math. That’s often the opposite of what I need as a practitioner to do what I aim to do: evaluate real-world implications, communicate solutions accessibly for stakeholders, and execute on them in a real-world project.
That brings me to my second point.
2. The goal of an academic paper is different from my goal as a practitioner.
Second of all, academic papers generally focus on questions of how to do something: particularly in computer science, they tend to either evaluate one approach or compare multiple approaches. As a practitioner, I don’t have to know how to do things off the top of my head.
I need to know what is available to me in my field of work, but I rarely need the ability to type it all from memory. This is not school: I am not taking a final with a #2 pencil in a pupitre. Almost all the time, I have at my disposal the most comprehensive knowledge index in history—a search engine.
I need to know what is possible so I can google how to do it later if I need to. Rather than going deep, as an industry professional or as a practitioner hoping to innovate, I recommend reading and exploring widely without going too deep.
A famous and hilarious example of a practitioner focusing on high-level stuff and trusting Google to help him with the low-level stuff:
Hello, my name is Tim. I'm a lead at Google with over 30 years coding experience and I need to look up how to get length of a python string. https://t.co/TZZeckGPyw
— Tim Dierks (@tdierks) February 26, 2017
One of Tim’s coworkers famously wrote len('string') on a post-it and stuck it to Tim’s monitor after he tweeted this. Guess what, it doesn’t matter because Tim can google “python get length of string” in two seconds. Tim is now a Director of Engineering at Google, by the way.
This is why you’re not gonna see coaching from me about how to absorb all the figures, charts, and formulae in academic papers. I don’t need all that, and in fact a lot of it is a red herring to me.
It sounds like I hate academic papers. Why do I read them at all?
There are two situations where I find it valuable to read academic papers:
1. To understand an approach in more depth that I’ve already heard about elsewhere.
If I’m interested in an approach I’m aware of but I don’t feel confident that I understand how it works or why people use it, I turn to the academic literature. I find that papers can often color my understanding with a ton of helpful context. For example, I subscribe to The Morning Paper—Adrian Colyer’s email list that delivers an interesting paper in computer science every weekday for about nine months each year. I don’t read every paper he sends out, but if the subject line suggests something interesting, I’ll absolutely take a look. What is “interesting”? To me, that’s papers that talk about a problem I’ve wrestled with before or take a deeper look at a technology that I want to use. Recent paper topics that have caught my eyeballs:
- Debugging memory leaks in web applications (I’ve done this)
- A taxonomy of quantum computing problems (sounds like a high-level view of a thing I might use someday)
- Semantics and complexity of GraphQL (I do a whole lot of API design)
2. To ground myself amidst popular misunderstandings of computer science scholarship.
As we’ve discussed, academic papers are often not written for practitioners to understand them. As a result, sometimes the practitioner community coalesces around some ideas that are…not exactly true to what the original scholars were saying.
Great example in computer science: Roy Fielding’s 2000 dissertation on REST. Programmers have strong opinions about what REST is, and they cite this paper as the progenitor of those opinions. The problem? Fielding never actually said most of that. Reading this paper helped me understand the limitations from which RESTful protocols were born and gave me a deeper, more nuanced perspective on API design.
Now that we have talked about when I read (or don’t read) academic papers, here is how I approach them:
The value of an academic paper to me, an industry professional, depends on a few questions.
I ask five questions as I am skimming a research paper:
1. What procedure are they talking about?
This is not the exact the question I use. The exact question I use is this one: can I describe the procedure they are talking about in terms of the other procedures I already know about?
Refer to the dirty secret of academia above: to get published, research has to look “new.” But very few things are com-puh-letely new: the vast, vast majority of new discoveries come from tweaking, testing, and reexamining what we already have.
I want to situate this new procedure inside my existing mind map of procedures. By explicitly drawing relationships between this new information and information I already know, I set myself up to remember this new information when I’m thinking about procedures in this general area of my mind map.
In addition, this question helps me determine if this paper matches my current level of understanding. If I don’t understand enough to draw some connections between this work and my existing knowledge, I probably need to go back and do some more research before I’m ready to learn much from this paper.
2. Does it work?
It’s worth noting that one paper does not definitively determine whether a procedure works or not. But it can be worthwhile to take a look at the results and consider whether they look promising or not for now. If it does, awesome! File that away for future applications.
And if it doesn’t? Awesome! File that away for future applications, too. An undervalued and important contribution of general research to our field is to determine what doesn’t work.
I spend the solid majority of my experimentation and refactoring time testing things out that end up not working. By getting the picture of what will not work, I am better positioned to understand the solution that ultimately does work. If I get it right on the first try, I remain unaware of what lies to the side of the path I took: which alternatives might have been viable, and which were not. I value that hard-won experience, but sometimes it’s also nice to let someone else get the gray hairs and then explain to me all the stuff they did that failed. Then I get the knowledge without spending the time myself ;).
3. What are the limitations identified in the study?
Under what conditions do the solutions explored in the paper work or not work? Sometimes the paper will go into this. I definitely look for this section. It helps me figure out when this might be applicable to my project and when it might not.
4. What limitations are not identified in the study?
Refer again to the dirty secrets of academia above: sometimes a study’s limitations get overlooked or omitted before publication. This is less likely to happen with .a study’s strengths, which is why one of my questions isn’t about strengths that failed to make the text of the study. It happens, but it’s less common.
My most important discoveries about implicit limitations in a study usually come from the methodology section. I want to know how the data was collected, measured, and compared. For the most part here I’m looking for big red flags like these and these.
In addition, I’m considering the results in broader context. Are there architectural implications to incorporating this procedure in a project? How about performance implications? Ethical implications?
Some of these questions move toward the practical application of a paper’s insights inside a project. That leads me nicely to my fifth question:
5. What more would it take for me to apply this knowledge?
I’m interested to know how I might use what I am learning in a production application. But here’s the key: I’m never going to use information based solely on academic papers as my source.
Instead, if I see an application for this information in a project, I go back and find blog posts, documentation, and software libraries that assist me. If those things do not exist and papers are the whole of it, then frankly, the procedure is not sufficiently well understood for me to be jamming it into an app that affects people’s lives.
So this is the step where I’m looking for those additional resources. Who has already done this? Who is explaining this in a straightforward way? How does that explanation influence how I would sell this information to my team? What would I have to do to incorporate the libraries into an existing project?
This critical thinking step helps me understand how this paper informs my perspective beyond making me sound smart at parties.
Conclusion
I don’t center academic papers in my independent practice regimen for two reasons. First, academic papers often showcase inflated results, convoluted language, and excess math that don’t serve my goals: to evaluate real-world implications, communicate solutions accessibly for stakeholders, and execute on them in a real-world project. Second, academic papers generally focus on questions of how to do something—which I can look up later as long as I am aware of the general approach.
That said, I find academic papers useful in two contexts: to understand an approach in depth that I’ve already heard about elsewhere, and to ground myself amidst popular misunderstandings of computer science scholarship.
When I read an academic paper, I don’t just hop in an inner tube and let the current of the paper take me where it likes. Instead, I’m going fishing: I have specific pieces of information in mind that I want to find in that paper to keep for later use. I find that this approach feels more exciting, goes faster, and helps me remember the paper better than reading alone. I use a similar approach for active reading, which is another leveling-up technique I use that we will talk about in a later blog post.
So what pieces of information am I looking for in an academic paper? I want to answer five questions:
- Can I describe the procedure they are talking about in terms of the other procedures I already know about?
- Does it work?
- What are the limitations identified in the study?
- What limitations are not identified in the study?
- What more would it take for me to apply this knowledge (blog posts, documentation, libraries)?
With these five questions answered, I am in a better position to a) recall the new information in the relevant context, b) understand how it contributes to my needs, and c) operationalize it in a real-world project.
If you liked reading about my approach to academic papers, you might also like:
This explanation by Dr. Mor Harchol-Balter of what it’s like to do a Ph.D (doctorate/doctoral degree) in Computer Science. It’s 22 pages in plain English that talks through what getting a Ph.D is like (and whether you want to do it), how to evaluate and choose school, and how she advises you to apply.
My approach to making inclusion an important part of your company culture