Hi. I’m Chelsea Troy. I’m a computer science educator at the University of Chicago. I’m writing about the techniques I use to teach my distributed course, Mobile Software Development. You can visit the teaching category to see them all.
In this post, I’ll talk about the structure I’ve put in place for collecting insights about the course and turning those into ideas about what to keep and what to change the next time the course runs.
Data Collection
In the previous piece from this series, I shared how I use surveys in my class. That’s a key component of the data collection step. There are two other data sources:
- my observations of how the class went/how students are doing
- my teaching assistant or grader’s observations of the same
When I’m lucky, things go exactly as I predicted. Sometimes they don’t:
This tweet references an activity where students worked in groups on a self-directed, test-driven data analysis feature in an Android app. I had allotted 40 minutes for this activity in class. It took an hour and 40 minutes.
The students revealed some significant improvements I could make to the exercise design right at the beginning (whoops). But they also had thoughtful discussions, researched solutions, challenged themselves to go beyond the exercise requirements, and refused to let me pull them out of group until they finished.
As more minutes ticked by, I switched to my course planning document and jettisoned other lessons I had planned that day. We’d cover them in future sessions. Even with the adjustments, it looked like we might not have time to get all the way through the exercise.
We did, just barely. Each group reached a stopping place that satisfied them with four minutes left in session. Students spent these minutes reflecting fondly on what they had just learned and how accomplished they felt. I inwardly beamed with pride at their progress, but the second class ended, I dashed to the bathroom mirror to count my new gray hairs.
Data Representation and Interpretation
Every quarter produces a few super-memorable insights like this one, but I need a way to organize student responses and experiences so that I can understand them and make adjustments.
I used to do this with exit notes. In addition to student responses on session techniques surveys, I had a shared Google doc where I would write down any of my own observations (and ideas about possible solutions) within 1-3 days after each session. I invited graders and TAs to also include their thoughts in this shared document. It didn’t work as well as I had hoped for a few reasons:
- In order to see what I had said, I needed to open each document individually at some future time, by which point I’ve probably forgotten what I said. This granular level of organization made it difficult for me to understand and adjust lesson arcs that spanned multiple sessions.
- Though they were invited, TAs and graders never once actually wrote anything in these documents, so they failed to capture the totality of the information about the session.
I needed something less granular that allowed me to change things about the course as a whole. So now I do this:
This is a course planning slide for the Spring 2020 class. The rectangles represent lessons or activities.
- Purple rectangles are for programming tutorials, yellow for group exercises, green for lectures, and blue for critical thinking/writing tasks.
- Rectangles with a white background were assigned as homework. All others happened in class.
- Red borders indicate “this set of activities took too much time, and I want to curtail them.”
- Green borders indicate “this set of activities took too much time, and I want to curtail something else to increase the amount of time that this activity receives.”
- Yellow borders indicate “I want to change something about this activity, besides curtailing it or extending its time.”
This representation allows me to move things around across sessions and see all of my options for reorganizing the course as a whole. But what about detail? A set of rectangles doesn’t show more detailed observations about what to keep or change.
This is where I take advantage of the fact that I can add comments to the rectangles:
This course planning slide covers Summer 2020, and it has a few additional visual cues that the spring one didn’t have:
- The width of the rectangles accurately represents how long the activities took, based on my notes from timing class activities and students’ reports on how long they spent on their homework assignments.
- The blue line at the bottom represented where we were in the course—I moved it down a row every time we completed a session.
Representing the time spend like this allows me to move blocks around with confidence that I won’t accidentally overburden myself or the students, or leave them bored or lacking, in a single week. TAs and graders tend to leave more comments on this document than the exit notes. I think it’s because they already have the link and don’t need to do the extra step of finding the exit notes, though I’m not sure. I also come in here and jot down stuff they said in our slack conversations or video chats.
This course planning slide gives me a manipulable map with which to plan all ten weeks of the quarter.
Making Changes
Once I have my observations, my course staff’s observations, my students’ responses, and my course planning map, it’s time to sit down and decide what to change, and how.
I’m not necessarily going to do everything the students suggest, or eliminate all of my course staff’s frustrations. Rather, I have specific goals that I want to accomplish with this course, and I look at the data with some questions in mind:
- Does this data identify a gap where the course implementation could do more to accomplish the course goals?
- What risks might we encounter, in acting on this data, of moving further away from the course goals?
I’ll share a specific example: in the spring, I had five groups of students explore five separate animation features in iOS an app that I wrote for them. I gave each group a set of questions to find the answers to, plus a modification to make to their feature.
When we regrouped, I asked each group to explain their feature to the rest of the class. I expected this exercise to give students an opportunity to teach one another, and I thought that the exploration guides would help them quickly learn enough about the feature to have something to say. In actuality, I had a really hard time getting several of the groups to talk at all, and when they did, it was one member reluctantly scrolling through their exploration guide and saying “Question 1 for us was X, and we answered Y,” and so on to the bottom. Students felt bored, and none of the groups felt confident that they had learned any of the other groups’ material.
The activity had to change. As much as I’d love for students to get presentation experience in the class, asking students to learn the core material from a relatively extemporaneous presentation from their classmates wasn’t working. I had to face the fact that my dreams for this exercise wouldn’t work out.
The next quarter, during the regroup, I lectured about all five animation features, and I directed questions at each of the groups on their specific feature. This allowed each group to demonstrate their expertise to the class within a framework that I had prepared ahead of time. We got through more material, the students learned more, and students who had applied themselves to the exercise got to demonstrate what they had learned. I have a course goal around students learning to teach and explain concepts for peers and stakeholders, but I decided that I didn’t need to ask them to do it extemporaneously. We now fulfill this objective in the class through prepared video demonstrations.
Conclusion
I have several surveys for collecting data from students throughout my course about how the homework and class activities are going; I also collect observations from myself and my course staff. Then, I represent all that data in a single diagram that allows me to pinpoint where things took too much or too little time, where they need to change, and what I’d like to keep.
As I’m making these changes, I keep the course goals in mind. What will move us toward those goals? Do we run any risks, with this activity, of turning away from course goals? These questions help me make decisions about tradeoffs where any of several choices involve giving up something that I like.
Over time, I’ll have a long-running data set that shows me how the course has changed over several runs. I don’t know what I’ll do with it yet. I’m sure I’ll update you when I do 🙂.
If you liked this piece, you might also like:
How to Jump-Start a New Programming Language, or maybe, even, gain a more concrete mental model of the one you already use!
Lessons from Space: Edge-Free Programming, which also explores an Android app’s design and the engineering choices behind it (plus, cool pictures of rockets!)
How does git detect renames?—This piece is about how git detects renames, but it’s also about how to approach questions and novel problems in programming in general.