Is Clinton really the most honest candidate in the race?
Former executive editor of The New York Times Jill Abramson, who has investigated the Clinton family for decades, made a splash today at The Guardian with the headline, "This may shock you: Hillary Clinton is fundamentally honest," which I highly recommend you read. I ended up supporting Bernie Sanders in the Washington caucus (go Evergreen precinct!), but greatly admire Clinton and think she would make a fantastic President. Lots of other people in my state and this country, however, have been convinced by twenty-five years of smear campaigns that Clinton is the Antichrist.
In setting up her argument, Abramson cited some statistics about Clinton's track record on PolitiFact, the political fact-checking website that rates statements that politicians and pundits make on a six-point scale.
As for her statements on issues, Politifact, a Pulitzer prize-winning fact-checking organization, gives Clinton the best truth-telling record of any of the 2016 presidential candidates. She beats Sanders and Kasich and crushes Cruz and Trump, who has the biggest “pants on fire” rating and has told whoppers about basic economics that are embarrassing for anyone aiming to be president. (He falsely claimed GDP has dropped the last two quarters and claimed the national unemployment rate was as high as 35%). [my emphasis]
I've been following PolitiFact for a while and even used it to help create a truthfulness metric, which I tracked during the 2012 election. There are two main issues with PolitiFact report cards, or any fact-checking report card for that matter. First, there's always the chance that the raters are themselves biased toward newsworthiness, fairness or partisan ends. Second, and more fundamentally, it's a pretty limited amount of data even if the raters were completely unbiased. But that doesn't mean you can't learn something from it. And we're going to learn just how certain we can be that Clinton is the most truthful of the remaining presidential candidates, including her primary opponent.
So, is Hillary Clinton the most honest candidate in the race? The short and dissatisfying answer is...
Let's start by outlining the problem, which is the wide variation in the number of statements rated per politician. Current President Barack Obama is the most-rated politician on PolitiFact. Clinton is in the top 10 as of this writing. PolitiFact has rated more than 100 of Trump's and Cruz's statements apiece. Not so for either Sanders or Kasich. The number of statements matters a lot when making measurements. In the case of PolitiFact ratings, the fewer of your statements that get rated, the more likely it is that you'll have a report card that is extremely clumped into just a few categories, giving a biased picture. In addition, the more of your statements that get rated, the more we learn about your honesty beyond our prior assumptions. And most importantly, the fewer statements you've made, the more difficult it is to distinguish your honesty from other politicians.
For example, Sanders has exactly zero statements as of this writing with the ruling "Pants on Fire". Does that mean Sanders is 0% likely to tell a whopper? Probably not. But it gives us some evidence that he may not be very likely to do so. The task before us is to combine the limited evidence we have from the PolitiFact report cards with our prior knowledge about the truthfulness of these politicians to update our beliefs about the proportion of the time they make a statement that would get a given ruling. Because those proportions are unknown to us, we can only estimate them. And because we're estimating these proportions using a limited amount of evidence from data as well as limited prior knowledge, we must measure our uncertainty in those estimates.
A good way to measure uncertainty is with a probability distribution. A probability is a number between zero and one that can also be expressed as a percentage. Essentially, it describes how much you believe something to be the case (as Andrew Gelman rolls his eyes). The more likely we think it is that, say, Trump's chances are 5% of telling a statement that PolitiFact would rule as "True", the more probability we place on the value of 5%. Bayesian statistics is one way to model uncertain estimates like this while incorporating information from both the data and our prior beliefs. In Bayesian statistics, you choose a model for the likelihood of seeing a particular report card given the proportion of the time each ruling would be attained. Then you choose a model called a prior distribution that describes your preexisting beliefs about those proportions.
To give all politicians the "benefit of the doubt", and because I really don't claim to know much about their truthfulness, I choose a prior distribution that assigns equal weight to all of the possible ways in which rulings could be concentrated across the five categories. I also assume that the report card of one politician doesn't influence the report card of another. For the nerds, I'm using independent Dirichlet-multinomial models with flat priors.
Once you've built your likelihood and prior, you can combine them with fancy math to get at a posterior distribution, which represents your updated beliefs about the proportions in each ruling category for each politician. Once you've got your posterior distribution, you can graphically display your uncertainty by looking at the credible intervals f each proportion estimate, which are the values within which a large proportion (in my analysis 95%) of the posterior probability lies. And if you plot the credible intervals of different candidates next to each other, you get a sense of how easily you can distinguish one politician from the next given the evidence from the data combined with your prior beliefs. The more the credible intervals overlap, the harder it is to tell the difference between politicians.
The plot below shows 95% credible intervals for each ruling and each politician. The dots are the expected values of the proportions from the posterior distributions. From this plot we see that Trump and Cruz are predictably unlikely to be rated "True" or even "Mostly True", and more likely to be rated "False" or "Mostly False". Trump has the distinction of being the clear standout for most likely to be ruled as "Pants on Fire". Interestingly, Kasich is difficult to distinguish from the Democratic contenders and President Obama, although he is apparently more likely to set his "Pants on Fire" than all three.
On the other hand, Sanders stands out somewhat from Clinton when it comes to "Mostly True" statements, echoing the sentiment among many pundits that he's getting at the heart of some key issues, but sometimes is fast and loose with the details. And while Sanders' expected proportion of Mostly False and False statements is a little higher than Clinton's, there's not enough evidence to separate this signal from the noise. By that same token, Sanders has zero statements rated "Pants on Fire", but we don't have enough evidence to claim he's less likely to tell a whopper than Clinton, who has two statements ruled as such.
But man, that's a lot of ruling categories. Is there some way we could summarize them into one easy-to-compare metric?
Of course there is. In fact, there are several metrics we could devise. But here's what I did. First, I collapsed the "Pants on Fire" and "False" categories because I think the "Pants on Fire" category is most prone to subjective interpretation, thus partisan bias. Second, I award 0 points to False and Pants-on-Fire claims, 25 to Mostly False, 50 to Half True, 75 to Mostly True, and 100 to True. Then, for a given proposal for the proportions of a politician's statements in each category, I take the sum of these scores weighted by the associated category proportions. This metric, a number between 0 and 100, measures the overall truthfulness of the statements that a politician makes. The measurement method is very similar to what I've done before, except the ruling scores there are reversed because in the past I have comically measured how much malarkey (aka bullshit) politicians are full of.
When we compute those truthfulness scores and plot their 95% credible intervals, we see a few things. First, even the most truthful of these politicians is barely better than half truthful, or in other words barely less than half full of bullshit. But that's old news because they are politicians. Second, we see that we can't distinguish Obama, Clinton, Kasich or Sanders from one another because their credible intervals overlap extensively. Third, Cruz and Trump stand apart from these other four as being particularly egregious. Not surprisingly, Trump is in a bullshittery league of his own.
But the bottom line is this: We don't have enough data from PolitiFact alone to tell whether Clinton is the most truthful candidate in the race. Until we do, let's stop saying that, because the statement is itself bullshit.