Does this convince you that self-driving cars are safe?

A friend posed the following question:

Being difficult, I objected to the premise for two reasons:

• We’re comparing a rate (0.52 fatalities/100M miles) to a number (0 fatalities)
• Google probably drives in California, not Massachusetts

In searching for the California fatality rate, I found a beautiful data set that included miles driven and total miles on a state-by-state level. This let me elevate my statistical snark to an easy case study: I could essentially copy and paste the example for hierarchical partial pooling from the Stan documentation that was later ported to PyMC3.

To describe our approach intuitively, it is usually easier to talk about two naive approaches that are unsatisfying for different reasons:

1. No pooling: Each state, plus google, is an independent experiment. Then either robots never crash cars, or we put some prior on each experiment, and we fit 52 different models, each using 1/52nd of the data.

2. Complete pooling: We could aggregate all the miles driven by man and machine, and then just ask how surprising it is to drive 1.5 million with no crash. All driving is the same, and a mile in Wyoming is treated like a mile in NYC.

Our approach is instead called partial pooling: we assume there is a distribution of fatality rates shared by all 52 experiments, parameterized by the mean (pooled_rate), and scale (κ), but then use those parameters to estimate experiment level fatality rates (state_rate, google_rate).

Model Definition

The model is implemented in PyMC3, which also makes it easy to read, so I include the code here. See either case study above for details about modelling choices.

def car_model(miles, fatalities, google_miles=1.5, google_fatalities=0):
    with pm.Model() as model:
        pooled_rate = pm.Uniform('pooled_rate', lower=0.0, upper=1.0)
        κ_log = pm.Exponential('κ_log', lam=1.5)
        κ = pm.Deterministic('κ', tt.exp(κ_log))

        state_rate = pm.Beta('state_rate', 
                             alpha=pooled_rate*κ, 
                             beta=(1.0-pooled_rate)*κ, 
                             shape=len(fatalities))
        observed_fatalities = pm.Poisson('y', mu=state_rate*miles, observed=fatalities)

        google_rate = pm.Beta('google_rate', 
                              alpha=pooled_rate*κ, 
                              beta=(1.0-pooled_rate)*κ)
        observed_google_fatalities = pm.Poisson('y_new', 
                                                mu=google_miles*google_rate, 
                                                observed=google_fatalities)
    return model

I also include the trace plot from this model, since it is a good example of a model that a non-Hamiltonian sampler would have a lot of trouble with, but NUTS does just fine. The left hand side are histograms of parameter estimates, and the right hand side is a timeline of joint samples.

Answering the Question

The original question was whether I trust Google’s self driving cars, given the fatality rate in MA. I don’t, yet.

Also, for interest’s sake, here’s a readable chart of estimates for all fifty states and DC.

Followup!

I should have had a third objection to the question, which would be a reference for the numbers we were talking about. Actually looking those up, it turns out that as of November 28, 2017, The Verge reports Waymo (which is not google, but is sort of google) has driven 4 million miles.

How does that change our conclusions? Not much!

The average state in the dataset has 50 billion miles, so moving from 1.5 million to 4 million miles should not change the inference very much.

So really the discussion should have been about Waymo the whole time, but close enough!