It’s not often that a model gets front page attention at the Wall Street Journal.  Last Friday, WSJ placed an article on Aetna’s pricing model trouble on the front page of the Marketplace section.  According to the article, Aetna had to pull a request for increasing prices for California policyholders when they discovered “miscalculations” in the monthly premiums.  It’s not clear from the article if there were multiple errors or the magnitude of the errors.  This story comes a few months after another insurer, Wellpoint, received similar press about how external auditors found mathematical errors in their filing.

How can this happen?  You would expect that with something so important, there would be ways to make sure that “model errors” would be caught and fixed BEFORE things get filed and rate increases are announced.  (Approximately 765,000 Wellpoint and Aetna policyholders would have been affected by the “wrong” price increase had the errors not been found.)

I have not worked with either company, but I have seen several kinds of similar mistakes in pricing/cost models.  The fact that many mathematical models have mistakes is not a surprise for those who study how models are built and used.  Ray Panko, whose research focuses on spreadsheet errors, has collected data from several authors that show that 88% of spreadsheets have errors.  Furthermore, Panko has found that 1% of all spreadsheet formulas are in error.  One percent may seem small… until you consider how “big” one mistake can be.  Can you imagine how many formulas must be in Wellpoint or Aetna’s model?

I recall talking to folks at NASA about failure rates for the 10,000 components they may have on a Space Shuttle.  One tenth of one percent failure (0.1%) is still 10 components.  Some components have backups or may be non-essential, but many components are essential to the survival of the crew and the success of the mission.

Think about your last “big model”.  How many errors do you think it has?  Would you be open to letting others audit it?