Do we have enough data for this problem?

36,000 rows. Not all of these are labeled so we cannot train on all of them, but it’s an encouraging number for a binary classification problem, particularly if we’re talking about tabular data with scalar features that demonstrate a clear pattern.

Now let’s look at the balance of our classes—particularly how many physicians are classified as cardiologists versus other specialties. When we go to train the model we’ll take unknown specialties out of the training set, but we’ll leave them in up until then so that they get included in any feature manipulation we do.

Cardiology represents a large chunk of this pie: almost a quarter, even before we remove unknowns. This tells me that we have enough examples of this class to get a good idea of its feature patterns.

This also clues me in to what kind of model we’ll need and what kind of data we’ll feed it. In this first pass, I probably won’t rebalance the data classes since cardiology makes up about a third of the labeled data. If, instead, I were trying to identify anesthesiologists that make up such a small chunk of the data, I’d look at finding a way to get more anesthesiology data, undersampling the other classes in the training data, or oversampling the anesthesiology class (roughly in that order of preference) for my training data.

Additionally, for a class as large as cardiology, my experience tells me that we’re likely to get good results with a traditional classification model. If the class were very small, I’d timebox my attempts with a traditional classification model and consider, instead, an anomaly detection model of some stripe.

What features can we use?

For each physician we have procedure codes (along with the procedures they represent) and the number of patients on whom they have performed the procedure. We could look for a number of patterns with this data:

1. Cardiologists may perform some procedures at least once that other physicians never perform (ascertained by one hot encoding the procedures)
2. Cardiologists may perform some procedures lots of times that other physicians perform less or never (ascertained by multiplying the above by procedure count)
3. Cardiology is a very specialty form of medicine that congregates in places with large research hospitals. They may perform a different total number of procedures than other physicians (ascertained by summing all procedure counts for each physician).
4. Also because of the specialization, they may perform a more specific range of procedures than physicians in other specialties (ascertained by creating some kind of vectorization of each procedure to determine which procedures are similar to each other, and looking at the difference in the size and density of the “neighborhood” of procedures each physician performs. My first attempt at this might be tokenizing the procedure descriptions and starting with a vectorization of that).

I think that feature set number 2 out of these is likely to give us the most information about the specialty patterns for the lowest up-front time investment. So I’d like to start with that. Let’s see if we can get an idea of what a physician’s procedure breakdown might look like:

So we have here a dataframe that tells us how many of each procedure a physician has performed. For example, physician 0 has performed seven procedures, each between a dozen and two dozen times.

Let’s convert this into a dataframe with a row for each physician and a column for each procedure, with the number of times the physician has performed that procedure at the intersection.

The way I wrote this conversion takes about ten minutes to complete. We’ll take a closer look at this procedure in an upcoming post.

In a permanent project for this model, we’d want to think about how to obtain, update, and store this information. We could keep it in a csv (which would be large, with a lot of zeros), keep it in a sparse matrix separate from the physician and procedure data (which would save space but be harder to index against the physician ids), or combine all our data in some kind of database. If we do it in a row-store database like MySQL or Postgres, we run into one problem we also have with csvs: we’ll be using a lot of storage on zeros. So we might consider a column-store database like Cassandra or similar, which are built for sparse data like this. We’d want to carefully consider that though, since inserts and updates are more time-consuming for a database like that, and these counts would need to be updated every time we have new information about a physician doing a procedure. We could store sparse features in one database and dense arrays like physician id in another database, then cross-index as needed, but that comes with its own technical challenges. A discussion of the tradeoffs of how to store this data is probably out of scope for this code challenge.

So for this first pass, I stick to storing it in CSV.

Get Binary Data with Unknowns Removed

Now we’ll add our target variables to our feature dataframe. We’ll also filter down our data for training to the physicians whose specialty we know. If we leave them in, our binary classifier’s cost function will optimize on differentiating between known cardiologists and unknown cardiologists and will lean toward classifying unknown physicians as not-cardiologists.

I like to check shapes after I perform some transformations to make sure that everything worked as I expected.

We have 25,000ish labeled rows. This is still likely enough data to make headway on our classification problem.

We have about 11,000 cardiologists of unknown specialty.

Do the Labeled and Unlabeled Data Look Similar?

In order to predict anything about the unknown physicians from the ones whose specialties we know, the patterns in the two sets of data need to be similar: otherwise, the labeled data isn’t telling us about the unlabeled data. Let’s do a quick check that the procedure patterns in the labeled and unlabeled data are similar:

What we’ve done here is divide the total procedure count for each procedure by the total number of physicians for both labeled and unlabeled data. We’ll put them next to each other on a bar graph and see if we notice any big differences in which procedures were performed between the two groups.

Look for any “jagged” areas where the orange bar sticks up way over the black bar or vice versa.

I’m not in love with this color scheme, but I wanted something colorblind-friendy that also had enough contrast with each other and with the white background to easily see any jaggedness.

I played with the zoom on this chart a little so I could see things better because I was having trouble getting the interactive pan/zoom setting on matplotlib to work in the notebook. If you happen to know the trick to getting that working, I’d love to hear it.

The orange and black bars aren’t exactly the same height, but for the brunt of this data they’re pretty close. It’s still possible that these two overall patterns are the same and the patterns on a per-physician basis are completely different. That could happen if, say, all the unlabeled physicians came from hospitals and clinics where they divide up procedures completely differently than the institutions that label the physician specialties.

To ensure that that is not the case, I’d want to find out how this data was collected. Without that assurance, I’m still going to consider it unlikely enough to disregard for now based on my understanding of how the medical field works.