Don’t Bring an Anecdote to a Data Fight.

So goes the meme, and it’s funny.  In a nerdy sort of academic way.  But is it good advice?

All good data fights are about competing stories.  For instance, there’s a prominent local story suggesting people are leaving Vancouver in droves over high housing costs.  Transforming this into a data fight, it quickly becomes apparent that more people are arriving than leaving – a severe blow to the story.*

But maybe it’s not all people we’re worried about – it’s the lifeblood of Vancouver that’s leaving.  People in their thirties and forties, whose careers are just taking off, but who can’t maintain a family in the tiny hell-holes this city provides.  The data actually tell us that, at least across the Metro Area (where we might expect “lifeblood” to have a useful meaning), more people in their thirties and forties are coming than leaving, it’s just that they often end up in the suburbs rather than the central city.

The data fight can keep going on its own (and I expect it will).  The stories continue to evolve accordingly.  But there’s another response to this sort of data, which goes something like:

Ok, but some of my friends have left.

This was, in effect, one of the comments I received over my recent posting about migration trends.  The standard response, of course, would be “Don’t Bring an Anecdote to a Data Fight.”  But I actually appreciate the comment, and now I’m kind of wishing I’d followed up on it by asking for more details.  Why?  I actually think it can be very valuable to bring anecdotes to a data fight.

A couple of years back, I remember first coming across news stories about people fleeing Vancouver because of housing prices.  (Ok, maybe even more than a couple of years.  I’m getting old, too old to go back and track them down).  What was striking to me about so many of these stories was the lack of verification.  When pressed, our dearly departed often described their moves more in terms of great new job opportunities, or desire to be closer to family, than in terms of housing costs.  This isn’t to say housing costs weren’t and aren’t a factor in many moves.  Indeed, they show up as a (relatively) common reason for moves provided when people are surveyed about why they move (see here for a report from the USA).  So I’d be surprised to find no one leaving Vancouver over housing prices.  But for the same reason, I’m often struck by how difficult it is to establish these kinds of motivations, even using anecdotes.  Clearly we could do a better job tracking movers in Canada!

But back to anecdotes: of what use is an anecdote in a data fight?  Let me count just a few of the uses:

1. The absence of anecdotes can be telling.  If you’ve put in a good faith effort and you can’t find an anecdote that resonates to a story of interest – one you’re trying to tell, or one you’re reading – then maybe the story just doesn’t work.  Even if the data seem compelling, if I can’t find place a particular and unique instance where the relationship they purport to show actually works, I’m going to be very skeptical.  And I should be.
2. Anecdotes can provide examples of the complexity that necessarily pertains to how well data stories might work in the real world.  For instance, they might show a story works in some circumstances, but not in others (which can help drive further data fights).  I’m very interested in anecdotes that help differentiate the people leaving Vancouver from those arriving.
3. Put slightly differently, but in an important way, anecdotes sometimes remind us about underlying heterogeneity.  Data fights are usually about trends, averages, average effects, and correspondingly dominant story lines.  But that shift is important – averages mask differences.  We should be very careful about allowing one story to dominate others if we can establish cases where alternative stories fit best.  Anecdotes that remind us of this serve an important purpose.
4. Anecdotes can call into question the validity or reliability of data.  This works best not by drawing upon a few counter-examples that call the story being told by the data into question (there are usually already baked into the data), but rather by drawing upon a few examples that call into question how the data were collected and/or interpreted.  In other words, tell me that you have friends who have left Vancouver and it won’t change my basic understanding that more people are coming than leaving.  But tell me that you have friends who left Vancouver but were still counted by the Census as being here, and I’m going to want to know more.
5. Anecdotes remind us, often in emotionally resonant terms, how important (or unimportant) some data fights might be.  Indeed, this is one of the frustrations of many quantitative social scientists: sometimes a good story is far more compelling than all of their carefully assembled data.  Bringing anecdotes to a data fight can help sell the importance of the data exemplified by the anecdote.  Politicians know this.  Social scientists should too.
6. When drawn from a data set, anecdotes also offer a way to check the data, making sure it works how we think it works.  Often this is the glory of mixed methods – qualitative & quantitative – kinds of analysis.  I don’t want to wade into any data fights without checking a few of the anecdotes contained within my data to make sure they make sense.

Anyway, the long and short of it is: please bring your anecdotes to your data fights.  Especially if it’s a fight with me.  I want to see them!  And I want details!