The Oxymoron of “Data-Driven Innovation”

Reading Time: 14 minutes

Once upon a time, during the dawn of the internet era, early web products often came from college dudes. The dudes were almost always wealthy and well-connected, but they weren’t usually market-savvy.

These people, now billionaires, have given beaucoup interviews on how they got started.

Look: I’m as concerned about survivorship bias in drawing conclusions from these interviews as you are. But one thing stands out. Asked how they got started, they all go “I was playing around and made a thing I wanted to use. Other people liked it. That was literally it.”

Compare that to the way tech companies (particularly large ones) endeavor to approach product development today.

Today’s product development wants to be “data-driven.” The focus is on quantitative methods to find product-market fit, usually by way of interviewing people who use the product, or interviewing people who represent majority groups that the org would like to see using the product. (Or, if we’re brutally honest, attempting to triangulate info about one of these two groups from product surveillance data).

Today’s product managers contrast their data-driven approach with the “I built this for me” approach we heard in interviews with founders from the 1990s. Despite its empirical strategic success, “build for yourself” now gets cast as immature and shortsighted.

What changed?

Understanding what changed reveals a false dichotomy between “build for yourself” and “build based on your focus group data.”

But before we address that false dichotomy, let’s draw some lessons from another kind of business decision-making: inclusion decisions. I get that product decisions that affect customers sounds unrelated to inclusion decisions that affect employees, but bear with me.

Often I see executives and directors approach inclusion the same way they approach other business initiatives, and then they’re surprised or frustrated when the initiatives don’t produce the public relations, employee retention, or recruiting outcomes they want.

For example, I once worked for a company that poured a lot of effort into inclusive hiring: they made an inclusion skills rubric, placed ads in “diverse” tech spaces, et cetera. Two years later, all the “diverse people” they had carefully collected had left, and the company had backfilled with almost exclusively cishet white guys.

What happened?

Let’s look at another example: Slack. The company talks a massive game about how inclusive they are. But the messaging app company not only built direct messaging across Slack channels—it was even in production before anybody (anybody being, of course, the entire internet) pointed out that the feature could hardly cater to harassers and stalkers more perfectly.

What happened?

Another example: Basecamp, the company lauded as the gold standard for tech company management a decade running, drove out about 40% of their employees over 4 days after banning political discussion at work, then saying something racist in an all-hands, then doubling down on their decisions via random blog posts on a daily basis like petulant, stubborn teenagers for…well, too long.

What happened?

What happens is, companies are approaching inclusion the way they approach a business development effort. They do “research” and focus on the highest ranked outcome or the majority outcome.

“We can’t justify that accessibility feature; too few customers asked for it.”

Or my personal favorite: “Our culture is awesome! 71% of our employees said they are comfortable expressing their ideas at work!” (Newsflash: if 71% of your employees belong to…ahem…the dominant demographic, then the results of your survey in this case are more damning than laudatory).

And in general, here’s the thing about marginalization:

Marginalized people basically never constitute the highest-ranked or majority opinion on something. It’s kinda…in the name. They’re on the margins. Not at the center.

Keeping focus on the center leads organizations to inclusion efforts that flat out do not work.

Last fall, Americans protested the curtailment of the U.S. Postal Service in favor of private options like FedEx or USPS. Wealthy folks in populous areas don’t realize the difference between the USPS and the private options, but it’s a great example of the difference between government policymaking and business decision-making.

The reason the USPS matters so much is that the USPS delivers everywhere. As in, a postal worker mounts a mule to deliver letters to the Havasupai people at the bottom of the Grand Canyon.

Yes, really. These are the actual mules (and also a horse there on the far left). PC The USPS.

Businesses focus on efficiencies—doing the things that net them the most money for the least effort. By contrast, taxpayer-funded public programs are designed and expected to cover everyone—including, and especially, the most marginalized. That’s why they’re taxpayer-funded; so they don’t face existential risk be eschewing profit-driven decision-making. Does this work perfectly? No. But I think about it a lot when people shit on the bigness and slowness of government. That bigness & slowness is supposed to create space and resources to account for the communities, that a “lean,” fast approach deliberately ignores.

Business decision-making tries to approach inclusion by citing data that centers majority opinion, which is not where ideas about inclusion come from.

“But Chelsea, businesses are profit-driven, so they HAVE to do things that way!”

-You, possibly

I don’t think so.

I think execs are accustomed to, and comfortable with, that kind of decision-making, so they default to it. That doesn’t mean they have to do it, and it doesn’t mean that it’s the best approach.

To understand why not, let’s return to the Slack connect example. That product had to be delayed for redesign, and might ultimately be recalled, over not having valued a marginalized, abuse-informed opinion in the design process. That was likely a very expensive development effort. If the thing fails, Slack lost a lot of money building it.

Or take Google’s decision to withhold Timnit Gebru’s work on their Ethical AI team, and Jeff Dean’s subsequent decision to terminate her rather than discuss the improvements she proposed. The trust that Google lost from that event comes with a massive price tag. They’ll have to work harder to recruit and they’ll have to compensate higher. Google can pay that price, but most companies couldn’t.

“Sure, Chelsea, but a couple of failures at companies that ignored the margins doesn’t prove anything.”

-You, I’m guessing

Fair. When has centering a marginalized perspective produced success and riches for a company?

Let’s start with the iPhone. People love the iPhone as an example of visionary success for a tech company. And I love it because everybody uses that example while completely miscontextualizing it.

Steve Jobs delivers the MacWorld Keynote in 2007

Most famously, the iPhone’s capacitative multi-touch screen made it possible to dream up any UI, unbounded by pesky keyboards. That feature enabled the iPhone’s ascendance to the status of a platform for independent application development. It broke the market for existing phones.

Turns out, Steve Jobs wasn’t some kind of genius visionary who independently thought that thing up. Its roots came from FingerWorks, an accessible human-computer interaction company founded with the express purpose of creating input devices for folks with compromised fine motor function. One of the founders wanted to build a way for his disabled mom to continue doing all the things she wanted to do, and that grew into the company that Apple bought in 2005. It wasn’t Apple’s idea.

FingerWorks founders John Elias and Wayne Westerman pose with the original product. PC Engadget, January 2007.

In fact, most of the things you love about your smartphone started as accessibility features.

So garden variety Pareto Principle-style business thinking doesn’t work for inclusive decisions, and companies can incur costly failures by mis-applying it, and companies can achieve momentous victories by approaching inclusion differently. Designing for the edges can produce something better for everyone.

And that’s why I cannot encourage any business, team, or org more highly to think about the corners/edges of the problem and start there to ideate solutions, rather than starting with the happy-path minimum viable case. The outcome of this approach, done thoughtfully and taken seriously, just might exceed everyone’s wildest dreams.

What does this have to do with data-driven product decisions?

Well, let’s summarize what we’ve learned so far:

1. Building for the majority group identified in summary statistics doesn’t work for inclusive decisions.

2. Orgs can incur costly failures, and even product irrelevance, by doing that.

3. Orgs can achieve momentous victories by prioritizing marginal business cases.

The catch is that building for the majority group identified in summary statistics is exactly the trap that data-driven decision making can fall into.

Back to the question:

“What changed in tech from the 1990s to now that influenced the switch from well-positioned college kids building their dreams to dads in graphic tees analyzing spreadsheets?”

What changed was who constituted the “marginal business case.” Stick with me here.

In the 1990s, personal computers as a mainstream thing were brand new. That’s when we get the paradigm-shifting “visionary” internet successes like eBay, Google, Apple, Amazon, and eventually Netflix.

You know where visionary ideas come from?

Let’s break this down, because I need you to see this.

We describe ideas as “visionary” when they break from majority logic to imagine a completely different future—one that serves currently unserved needs or poorly served needs in a completely new way.

You know who has unserved needs or poorly served needs? Marginalized people. People that the current systems ignore or rebuff.

Visionary ideas derive directly from centering people at the margins.

Read it again, please.

Now, what the hell does “centering people at the margins” have to do with wealthy, well connected college kids making shit up for fun? To understand that, we need to talk about what marginalization meant in the 1990s when the consumer web was new.

At that time, in situ “applications” like bookstores, post offices, and auctions had a service gap: location dependence. You had to be at the place. That limited access to folks with time & transportation. It limited access even for wealthy, well-connected college kids.

The location of these founders might have exacerbated the effect. See, the U.S. looked different then. The Big Stuff from a STEM perspective was still largely Boston or NYC. Silicon Valley was growing, but it wasn’t what it is now. Even today, BART lags NYC’s subway and Boston’s T.

These college kid founders largely came from in Silicon Valley—in fact, they were often the children of electrical engineers who moved from Boston to do computer hardware research for defense companies like Lockheed. So for a glimpse of time, these privileged kids experienced a marginalization that the web suddenly made addressable.

Lockheed’s lab in Silicon Valley, present day(ish). PC Satellite today.

By building for themselves, in that glimpse of time, they centered a marginal need—to access resources and communities without being co-temporal colocation.

For that glimpse of time, in other words, because everyone experienced a marginalization that the web addressed, even the most privileged person building for themselves was centering a marginalization. WAS visionary.

Nowadays, of course, since rich, well-connected white men got to build the web from their dorms and parents’ garages, they’re already centered on it. Building for already-centered people doesn’t produce visionary ideas. Tech has noticed, and so it has discounted “build for yourself.”

And THIS, my friends, is the false dichotomy.

Product teams forget the context of the “build for yourself” spirit that they discount in favor of data-driven decisions. And this has tragically obscured their potential for actual visionary work.

Here’s how: It’s not “building for yourself doesn’t shift paradigms.”

It’s “building for yourself doesn’t shift paradigms if you are already the main character.”

Product, design, and engineering teams are overwhelmingly led by people who are already main characters in tech.

So those main characters building for themselves DOES produce incrementalist, weak-willed, un-visionary work.

But!

A data-driven approach that centers mainstream opinion also produces incrementalist, weak-willed, un-visionary work!

That’s right, party people. Our best practice does the same thing as the practice that we shat on for a decade.

So what does this mean for a tech executive…or even an individual contributor?

A couple of things. Allow me to enumerate:

1. If you fit the archetype of the web’s main character, do not tell, like, a formerly incarcerated black parent not to build for themselves. “Don’t build for yourself” applies to you. Not to them. Tech systematically rebuffs their needs. They have vision you don’t have.

To generalize that example: We desperately need marginalized voices to build for themselves if we want to inject vision into our languishing internet of personalized ads decorating stalkery stadiums hosting free-for-all rhetorical cage matches.

2. Focus product development not on what you want, and not on what some weird average of your current constituency or desired constituency wants, but on what the most marginalized person needs from your thing.

On your team, this means two things.

  1. In the long term, get marginalized people into seats of power to make product decisions.

Not just because it’s nice of you. We gotta get over this idea that we’re being mensches by handing over our stolen power to unworthy or inexperienced charity cases. No. Marginalized people out-qualify mainstream existing leadership on injecting vision because only their perspectives can deliver a visionary product in an industry already optimized for mainstream perspectives.

2. In the short term, start your product discussion with “Who are we building this for?” And make sure your answer to that question is somebody people don’t typically build for.

I built my class for students who typically struggle in graduate classes. I expected the class to evaluate better for those people, and probably worse for folks who typically do well in traditional graduate classes. I was wrong. My techniques worked better for almost all my students, including the ones who tend to manage fine under typical academic paradigms. I learned a lesson and I’ll never forget it. And it’s this:

Build your medical insurance app for people who typically struggle to get or use their insurance.

Build your transit app to help folks who struggle more than others to get access to transit.

Build your [X] for people who have had it up to HERE with [X].

You’d be shocked how often your fear about losing “your base” by making this decision proves unfounded. Sure, when you establish a perspective, you might lose a few folks who just have an opposite perspective. But more often, building for the margins produces something better for everybody.

So, sure. Don’t build something for you.

But don’t let that push you to build something for no one. And don’t retreat into incrementalism based on your survey data.

Instead, if it’s vision you want, ask “Who is left out of current solutions and why?”

Advocate for them.

If you liked this piece, you might also like:

This piece about who I am, how it informs my work, and how I apply the ideas in this post as an engineer currently at Pocket.

This post about engineering for edge cases, which is the technical corollary to the product approach we’ve discussed

This piece about how blithely naming a model ‘User’ in your code base is probably bad (there, I said it)

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.