The Federally Facilitated Exchange – A review after 2 days in operation

It’s been almost been two complete business days since the Federally Facilitated Exchange platform or FFE opened for business.  The FFE is designed to provide support for 34 states ( who represent 65% of the total population of the United States or 204M of 313M citizens.  (Source: July 1, 2012 projected population by state, US Census).

According the GAO, the program has spent $394M since fiscal year 2010 across 55 contracts with 77% of the $394M being spent with 10 contractors.

I’ve been somewhat vocal on the fact that I have been attempting to login to the website regularly since approximately 7:00 a.m. ET on October 1st.  I was finally able to successfully create my account at 11:29 p.m. that evening but was unable to proceed any further as I was presented with a big red error box telling me that the system was down.  I have been unable to login all day today.

Routinely I received a friend message that tells me:

We have a lot of visitors on the site right now.
Please stay on this page.

We’re working to make the experience better, and we don’t want you to lose your place in line. We’ll send you to the login page as soon as we can. Thanks for your patience!

I’m trying to be patient but I feel compelled to ask a couple questions.

1) What have the 55 contractors been doing for the past three years if not preparing for millions of people to login to the platform on day one?
2) Are these contractors qualified to build such platforms?

I’ve had it pointed out to me repeatedly that:

1) Facebook wasn’t built in a day.
2) This is a big project and no one expected it to be glitch free.
3) I’ve never done anything like this so I shouldn’t judge.

I’d like to respond to these in order:

1) Actually, Facebook was built in a day, by a guy, in a dorm room.  Sure, it was improved upon over time and didn’t have to deal with millions of simultaneous users on day one.  But if you’re going to make an argument, please be precise.  A better argument would have been, Facebook didn’t have to handle millions of users on day one – to which I would have responded no it didn’t, but it was able to and did handle millions of users after several years of operation.  It currently handles 288M total users in the US with three nines of availability 24x7x365.  Perhaps we would have learned something from them rather than trying to build something from scratch?

2) Actually I agree with this.  Sadly, no one DID expect it to work but I’m not talking about glitches.  I’m talking about fundamental core functionality failures.  On day one I couldn’t even get the landing page to load reliably – a simple HTML page – a page that the FFE program HAD to expect that millions of people would hit.  That’s a function of poor capacity planning in my view.

3) I have done things like this actually.  I was on a team of less than 100 technologists who built, from scratch, a new web based platform that connected people from all over the world to real-time data sources.  The system supported millions of users requests for streaming, real-time data from more than a dozen different sources.  The data sets where disparate structures representing wildly different kinds of things that all had to be normalized, streamed and presented to the end user is a simple to use, small interface that needed to work in any commercially available browser.  We did this in two years with FAR less than $394M and were successful on day one.  We processed millions of logins, and trillions of messages from the sources per day.  Was it as complex as the FFE – I would accept the possibility that the FFE might be a little bigger and a little more complex but not materially so.  I WILL argue that if the FFE had been a commercial effort, we would not be so forgiving.

Why do we accept so little from our government to whom we pay so much, when we require so much from our other services providers to whom we pay so little?

Creative Commons License
Erniesan Blog by Ernie Harris is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
Permissions beyond the scope of this license may be available at