About a third of our labaled data is labeled “Cardiology.” We’ll run a random classifier 100 times and take the mean accuracy of those runs so we don’t get fooled by some random outlier that happens to be more or less accurate than random would normally be.

So our models have to get an accuracy over 60-ish percent to do better than random guessing.

Try some Models

I’m going to start with two models: a logistic regression model (used for classification) and a random forest classifier.

Why these:

• Both pretty fast
• Both relatively interpretable
• I have seen both do well on this kind of problem many times

If they both do really poorly, we can either tweak their hyperparamenters, try different features, or try a different kind of model. This is a starting point with a fast iteration loop, so we’ll get more information fast about whether we need to change something.

Once again, we end with an assertion to check the shape of our end product. I tend to do this especially when the axis parameter is involved, because in my experience with pandas it’s not consistent whether 0 or 1 refers to rows or columns.

it should be noted also: sklearn has a train_test_split function. I forewent using that function in favor of my own because:

1. train_test_split doesn’t give us a validation group to iterate on our model before testing. I want this so that I can be sure my changes are improving the model’s generalizable results rather than teaching to the test set.
2. I want easy access to which examples gave which results so I can do error analysis and look for patterns in the mistakes.

^ A note: to a data scientist, the amount of manipulation I do on that groupby might seem odd. I can do the groupby without the index resetting or column naming and get the same result. I do it because the output display with the manipulation looks nicer is easier for folks to read who are unfamiliar with the pandas library.

We have our train, validate, and test groups. It’s time to run some models on that training set.

Let’s save off these results for error analysis later.

So how does our accuracy look?

Let’s take a look at the features it highlights and see if they make sense.

We’ll use these helper methods above to match up some interpretable feature names to their weights in the models. We could theoretically pull this out into a helper class or utility file. I haven’t done that yet because:

• The methods rely very closely on the features we used, so do not want to hide that dependency for now
• Notebooks are most useful for two things: exploration and presentation. Right now, we are using this method for exploration. Eventually, we might want to use it for replication, say, of an experiment. In these cases, having this method in a helper file or even a state-aware class (that knows which features were used in this training run) might suit us better.

We’re seeing here that the five features that most strongly indicate a cardiology classification all appear to have to do with the heart. That makes sense. Let’s also look at the procedures that most strongly contraindicate a cardiology classification:

These features don’t sound anything like cardiology, so they also make sense.

Before we start any error analysis here, let’s look at the random forest classifier and see how it makes this decision.

Let’s save off these results for error analysis later.

So how does our accuracy look?

What features is the model using?

Similar to the logistic regression model, our random forest classifier seems to make decisions chiefly based on procedures that have to do with the heart (ekg stands for electrocardiogram, which is a heart evaluation device).

Feature importance in a random forest classifier doesn’t work quite the same way as weights in a logistic classifier. Rather, the feature importance tells us how much this feature contributes to the classification, in either direction. Lower numbers mean that those features contribute less, all the way down to zero (not contributing), as you see here:

That means that, if we go far enough down on the high feature importances, we’re likely to see some of the things that the logistic regression model had the lowest coefficients for, because they are very important indicators of who isn’t a cardiologist. Look:

This feature has a pretty high rank.

Gauging Accuracy

The overall accuracy on these models is pretty high for a first pass.

A lot of the stuff I work on day-to-day starts out with an accuracy of 60-80%, and we need to get it into the high 90s. This is because my work revolves around the interpretation of legal text, with critical liability risks if the model gets certain things wrong. The larger the consequences of a misclassification, the closer a model needs to get to 100% accuracy. Another situation that might demand perfect accuracy is the use of a model on enormous data sets—the kind with millions of examples to classify, such that 97% accuracy would represent tens of thousands of mistakes.

Our use case has no catastrophic consequences of misclassification (i.e. no deaths, injuries, or crimes). We’re also not in the realm of big data: with about 11,000 examples to classify, a 1% accuracy difference equates to only 110 examples. For us, 96-97% accuracy might be good enough.

Nevertheless, in the next post, we look at some examples of how we might approach raising the accuracy of a model like this.

Conclusion

In this post, we established a random baseline against which to judge the accuracy of our models on our dataset.

Then we chose two different types of model and tried them both out on our dataset.

We opened up those models and examined why they made classifications the way they did, and we talked about the difference between a weight and a feature importance.