Error Analysis: what do the models get wrong?

Let’s address an elephant in the room: The overall accuracy on the models I’m about to show you is pretty high for a first pass.

A lot of the stuff I work on day-to-day starts out with an accuracy of 60-80%, and we need to get it into the high 90s. This is because my work revolves around the interpretation of legal text, with critical liability risks if the model gets certain things wrong. The larger the consequences of a misclassification, the closer a model needs to get to 100% accuracy. Another situation that might demand perfect accuracy is the use of a model on enormous data sets—the kind with millions of examples to classify, such that 97% accuracy would represent tens of thousands of mistakes.

Our use case has no catastrophic consequences of misclassification (i.e. no deaths, injuries, or crimes). We’re also not in the realm of big data: with about 11,000 examples to classify, a 1% accuracy difference equates to only 110 examples. For us, 96-97% accuracy might be accurate enough to ship as is with no further work, especially as a first pass.

Nevertheless, I’m going to demo some error analysis methods here so you can see how I would approach raising the accuracy of a model like this.

Let’s start with the logistic regression classifier.

It overwhelmingly gets the classification correct, but it struggles the most with recall of cardiologists. That tells me that some cardiologists are slipping through our classifier undetected. Let’s compare some raw numbers to get an idea of how many:

So we have a lot more false negatives than false positives with no tweaking to the out-of-box model. Returning to our conversation with the product manager at the top of this investigation, what we do now depends on the cost of a false negative fersus a false positive.

Let’s pretend, for the sake of argument, that a false positive costs us little: non-cardiologists who get our missive mostly ignore it, with a very low likelihood of unsubscribing from our client. Meanwhile, the false negative costs a lot: our client is selling a subscription for a very exciting (and expensive!) cardiology subscription product that has gotten a lot of hype from early adopters on Instagram and Youtube ;). Customer lifetime value is high, and likelihood of buying is relatively high. We want to capture all those subscriptions for our client. In that case, we’d want a very sensitive model that catches every cardiologist, plus some more false positives.

Maybe this is a small-scope change: maybe we can capture those additional cardiologists by changing the prediction probability classification threshold (the default value is 0.5). Let’s take a look at how that threshold affects our precision and recall:

Again, I’ve futzed with the zoom a bit on this graph so that the interesting part fills the viewspace.

I’m seeing an important tradeoff area near the 0.4 threshold. The cost of false negatives versus false positives determines where we want to put this tradeoff.

Out of curiosity, which threshold gives us the highest raw accuracy?

So if false positives and false negatives cost us the same, we can tweak the threshold to raise the raw accuracy of this model.

Other options at our disposal for getting a more accurate model here:¶

1. Grid searching over a range of hyperparameters (like penalty type or regularization value (C)) to tune the model to the hyperparameters that give the best accuracy.
2. Analyzing the specific cases where the model misclassified things to ask “How do these cases differ from the correctly classified cases?” and desiging features to help the model better capture those cases.

Before we do more optimization on this model, though, let’s see how the random forest classifier did.

Out of the box, this raw accuracy is higher than that of our logistic regression model, even the threshold-optimized version.

Random forest classifiers pick up accuracy from ensembling (combining the signals of those trees, a collection of weak classifiers, into one strong classifier). The tradeoff: their results, while still interpretable, are a little harder to interpret than the logistic regression ones. This is because a) you can draw a picture of the entire model as a tree chart, but it’s hard to see the whole thing at once and it’s a bit confusing with all the individual decision trees in there, and b) feature importance does not tell us whether the feature indicates or contraindicates the class. We can still make an educated guess at this information by looking at how a given feature correlates with the random forest’s predict_proba probabilities, but it’s not as automatic as it is with a logistic regression model’s coefficients.

So it’s not unusual for the random forest classifier to fit a little better, but error analysis can be a little more involved.

Let’s see where the model is getting things wrong.

This model starts off with a couple dozen fewer mistakes than the logistic regression classifier. Let’s take a look at some of those cardiologists that the model didn’t catch:

So here’s the first thing that strikes me: two of the specialties of the false positive physicians, cardiac electrophysiology and cardiac surgery, sound like they would probably perform a lot of the same procedures as a cardiologist (I looked this up—a cardiologist might refer a patient to a physician with one of those two specialties depending on their assessment of the patient’s condition).

This again begets the question: what is our campaign for? If it’s for some heart-treatment-related thing, we might actually want to add all cardiac specialties to our target list. That might give us more likely prospects in addition to knocking out some of these false positives.

I’m also seeing a relatively large number of false positives in the internal medicine department. This begets a question: how similar are the procedures performed by an internal medicine physician and a cardiologist, on averge?

We can get an answer to that by calculating the cosine similarity of the mean procedure vectors for each specialty. (Yes, this is absolutely overkill for a model with a first-pass accuracy of 97%, but now I’m interested).

Cosine similarity varies between 0 (no similarity at all) and 1 (these things are the same). So if we get the cosine similarity of a specialty with itself, we get 1 (plus some floating point fun).

All right, time to answer the question I only just realized I’ve had all my life: which physician specialties are most similar to others by procedure pattern?

This heatmap is sort of hard to see. I can put it all in the frame at once (like it is now), or I can make it big enough to read those labels. Not both. Another common way to express this information pictographically is to perform some kind of dimensionality reduction technique (in this case, for example, t-SNE would be an appropriate choice) and then display all the specialties as points in a two or three dimensional space, with more similar specialties’ points appearing close to each other. It’s also a good approach. One downside of it is that it’s confusing that the X and Y axes don’t represent dimensions. Instead, we have a lot of dimensions (in this case 3,140 dimensions) that our human brains can’t visualize all at once, so we take the salient information—distance between these points—and arrange them in a sort of neighborhood based on who should be close to whom.

See that diagonal line of black squares? Those cosine similarities are all the max value (1) because that’s where we see each specialty’s cosine similarity with itself. Light color squares show where the two specialties had no similarity. The darker the square, the more similar the procedure patterns of these two specialties. While we’re here, take a look at Cardiology relative to Cardiac Surgery and Cardiac Electrophysiology. those squares are all pretty dark, meaning those specialties do a lot of the same procedures. But this all started because we saw a bunch of internal medicine physicians labeled as cardiologists. That square is darkish, but not ultra-dark. In fact, let’s get the number:

The procedures are not totally different, but they’re not precisely similar either. So what gives? Let’s pull up the procedure records of thes 17 physicians and see if we notice anything.

We’re looking at the most performed procedures among our mislcassified internal medicine physicians. I get why they got labeled cardiologists: a lot of these are also procedures that a cardiologist would perform a lot. The top one, for example, is an ekg procedure. Internal medicine doctors do heart and circulatory ultrasounds as well as blood tests. I’ll bet those rank high in feature importance for cardiologists.*

*Admittedly feature importance in a random forest can mean the feature is either strongy indicative or strongly contraindicative, but the data patterns and our understanding of medicine both point to these being indicative.

Let’s see if I’m right:

Boom. Our most-performed procedures among internal medicine physicians falsely classified as cardiologists carry top ranks in the model.

I suspect the separating agent here, if there is one, is going to be which procedures the internal medicine doctors dont perform: that is, what makes them different from a cardiologist correctly classified as a cardiologist. We could do this by grouping top procedures for cardiologists or by looking at the top-ranked procedures in the model that internal medicine doctors don’t do and seeing if they’re different from the ones they do.

You know what I’m noticing? The procedures that both these internal medicine physicians and cardiologists do a lot are not super invasiveand not super specialized: they’re blood tests, stress tests, ultrasounds, and routine ekgs.

What gets top ranks in the model that these internal medicine doctors don’t do? Inserting a catheter into the heart for imaging. Inserting probes into the esophagus for heart ultrasounds. Doppler ultrasounds. Constant tests over long periods. These are more invasive, more specialized procedures.

So here are the features I would consider trying to engineer here:¶

If it’s really, really important for some reason that we scoop up these distinctions our model missed, I might try to generate features that capture the level of invasiveness, specialization, or diversity of the procedures physicians are performing. Internal medicine doctors handle all kinds of stuff (high diversity of procedures), but in general when it’s time to go really deep they refer to someone else (lower specialization and lower invasiveness of procedures).

So we looked at the specialties with the most false positives.

We only see one or two false positives in other specialties, so we’ll leave those alone for now and turn to the false negatives.

I’m showing the top 20 here, but I looked through about the first 200 of these. I notice that:

1. A lot of these are quite vague—for example, office visits and hospital care for varying lengths of time. A lot pf physicians do those, of many specialties.
2. The cardiologists do appear to do fewer of their most-performed procedures than, say, the internists we looked at before. The misclassified internists performed more routine ekgs—80% more, in fact—than these cardiologists, even though there are more than 5 times as many cardiologists contributing to this grouping.

These findings dovetail with the possibilities mentioned in the false positive section above: if we need to inch this model closer to 100% accuracy so badly that it’s worth a significant time investment, we could look into emphasizing those highly specialized procedures that cardiologists do relatively few of, but a different specialty would do none of.

I’m not going to jump into engineering those features now. I’ll leave it as a future option, if we need it, and move on.¶

Once we had all the features we wanted in the model, we could try some hyperparameter tuning to tweak it further.

For example, let’s see if the threshold on this one is the optimal one, the way we did with the logistic regression model.

Let’s keep running with this model and do a little more tuning:

Let’s do a grid search over some of these hyperparameters. We’ll try this out with n_estimators (the number of trees in the model), max_features (the number of features to consider at each split point), and max depth (the number of levels each tree can have). We could tune some of the others, too, if we liked, but these ones in my experience give the most bang for buck, with buck being time invested.

Random forest classifiers pick up speed from bagging (running decision tree classifiers on subsets of the data). It combines the predictions of those trees at the end (ensembling). Because each individual tree trains on a subset of the data and all those trees can train concurrently, a ramdon forest classifier often runs even faster than a single decision tree on the same dataset.

We’re going to take advantage of that speed and run this thing a whole bunch of times to compare some tuning options.

We are not planning to do this constantly. We’ll get the optimal parameters from this, then try them on our classifier.

As you can see, all that tuning snagged us a small additional accuracy (for a model that takes much longer to run, I might add). I have rarely seen this kind of tuning make an enormous difference, particularly compared to feature engineering, class consolidation, or additional data cleaning. So I’ll optimize on iteration speed at the beginning of a project with default settings, fewer trees, or less max depth, cycle on those other optimization strategies, then swing back to hyperparameter tuning at the end.

But, it’s the highest accuracy we’ve seen. Let’s run this on the test set and see how we do.

That’s our accuracy on our test set.

Time to run our model on our unlabeled data and hope that it correctly labels a lot of cardiologists (our info tells us it will). To give it a little extra chance, we can fit it on the entirety of the labeled data first.

The production_cardiology_classifications.csv file contains the ids of the unlabeled physicians and the model predictions of whether or not they are cardiologists. We can use this to plan who to contact for this campaign.

Finally, we save off our model in case we need to access it later. We might need it later for a lot of things:

• To get predictions for new data
• To use as a base line comparison while trying to improve this model with new architecture, features, or hyperparameters
• To examine as part of a bias audit or technical audit
• To keep track, in the event of multiple models exercising multiple ontologies on diffferent parts of the same table over time

Paired with this, in addition to saving our production predictions, we saved the name of the model with which we made those predictions.

For now, we’re done with our cycle on this specific model.

Recommendations regarding using this approach for other specialties¶

Earlier in this investigation, we looked at the breakdown of specialties in the labeled data. We saw that cardiology makes up by far the largest part of this data. Here’s the plot again, for reference:

The next largest slice, internal medicine, is less than half the size of cardiology. We have a lot fewer examples from which to extract information about how any other specialties’ procedure pattern differs from the others. To do this same approach on another specialty, my first choice would be to collect more data in the target specialty. If we can’t do that, then in some of these larger classes, we might have enough samples to still get a useful result.

For example, we do have almost 3,000 examples for internal medicine and more than 2,000 examples for family practice physicians.

We can work with this, though it’s likely to be worth balancing the samples to train classifiers on classes like these.

For some of these specialties, though:

We have relatively few examples of these. Out of 25,000 labeled rows, each of these specialties makes up such a teeny number. These classes are small enough that, if we absolutely must try to detect them given this training data, even the most sample-conscious classification might not catch them. We’d likely have to do some combination of clever feature engineering and/or an anomaly detection type of model to get the result we need on these.

Conclusion

We’ve done quite a bit in this series, haven’t we?

We started with a relatively large, relatively clean dataset. We looked at its size and class breakdown.

We talked about features we might use to do this classification, and we chose a starting set of features. We did some data processing to represent those features. Then we examined the similarity of the patterns in those features between labeled and unlabeled data to gain some confidence that there was a consistent pattern for our model to detect.

We tried two types of model that offer us fast iteration times, relatively high interpretability, and (in my experience) good results on feature-rich, data-rich classification problems like this one: a logistic regression model and a random forest classifiers. We talked a little bit about the attributes of those models, their speed and accuracy, and how to interpret them. Then we divided our labeled data into train, validate, and test sets, ran both models, and took a peek inside at the features they emphasized in their classifications.

Finally, we analyzed the errors: we took a look at what our models got wrong. We chose to run with the random forest classifier for our production model. We learned some more about the relationships between the classes in our data and developed hypotheses as to why the model would make the errors it made. We came up with action steps: features we could engineer to test those hypotheses and possibly improve the models. Finally, we tweaked the hyperparameters on our model to eke out a little more accuracy. We ran our final version on our test set, then trained it with the labeled data and ran it on the unlabeled data for a final product to inform our contact list.

This general approach might work for some of the other specialties, but probably not for all of them. As is often the case in data science, much of it depends on the problem we’re solving, the value we’re creating, the costs of our mistakes…

…and, of course, the data!

If you liked this piece (and series!) you might also like:

This other whole notebook of data science excitement on a completely different data question

The Framework for Feature Engineering Series (featuring our friend the MNIST dataset)

The Design Patterns for Data Science Series (where engineering practices meet data science challenges)

Exploring Numpy Vectorization (in case this post didn’t get far enough under the hood for you)