With only weeks to go before the roll out of the Nevada health care exchange, developers knew they were in trouble. In August 2013 the governor was made aware of serious “readiness concerns.” In September, with the first TV commercials already on the air, developers and end users had yet to begin final testing of the site. The implementation of the site was significantly behind schedule, yet surprisingly representatives from the federal government told Nevada officials in early September that their state of readiness was about average.
In July 2012 the state of Nevada selected Xerox as the “Business Operations Solution (BOS) vendor” to design and build the Silver State Health Insurance Exchange. The contract was set to extend until December 2016 and was not to exceed $72 million dollars. The launch date of the new exchange was set by the federal government as October 1, 2013. That meant Xerox had about 15 months to get the system up and running.
But an independent verification and validation (IV&V) report dated August 30, 2013 shows that, with only four weeks to go, the project was in danger of not being ready for roll out. Under a section of the report titled “User Acceptance Testing” the report states, “As of 08/15/2013, processing applications from BOS not progressing as expected. Numerous defects and lack of coordination of data mapping between HCR-EE and BOS have been identified team putting 10/1/2013 go live date in jeopardy.” [Emphasis added]
A few sentences later the report spells out the specific reasons for concern “As of the August 28th meeting, only 22 simple applications (out of 235 scenarios) were fully executed by the BOS and HCR-EE. All of those applications failed. While the BOS and HCR-EE projects are now engaging in daily defect meetings and daily builds, IV&V believes that going live on 10/1 with full functionality is in jeopardy.”
The ability to meet the deadline was so worrisome that readiness concerns were “escalated to the Governor’s Office during August.” What followed were “executive level meetings between the Governor’s Office and the BOS/HCR-EE project teams.”
An Executive Director’s Report dated September 12, 2013 shows the situation had not improved significantly almost two weeks later. “With October 1st less than 30 days away, the team is considering the project to be at a high risk overall.”
Kevin Walsh, who has the unwieldy title of Senior Vice President and Managing Director, Eligibility and Insurance Exchange Services, Government Healthcare Solutions at Xerox, agreed to respond to questions by email. I asked him about the “high risk” status of the project and he replied “A high risk assessment for a project of this scale and importance doesn’t correlate to implied failure, but rather it ensures the appropriate level of focus by all parties.”
That may be so but the reference to “high risk” in the Director’s report is followed immediately by a link back to the IV&V report, i.e. the one which indicated twice that the go live date was “in jeopardy.” The possibility of failure in this case was not implied, it was stated outright.
One of the big milestones in the process was the Operational Readiness Review by the federal government. The Director’s report describes that review saying “The federal ORR was conducted in Carson City on September 4-5, with several federal partners traveling to join us in person. We expect to receive a formal letter with the results of the ORR within the next two weeks. Overall, they indicated that we are at an average status of implementation among the states they have reviewed thus far.”
So apparently “average status of implementation” and “high risk overall” were synonymous as of mid-September. Looking around at the general failure of the exchanges this may in fact be true. The Nevada exchange, like the federal exchange, has been up and down during its first week as attempts have been made to patch software bugs which were not caught or not addressed prior to the go live date.
One of the last documents published prior to the go live date was a status report on various elements of the exchange implementation. This report, received by the state on September 16th, indicates that serious problems with testing and system integration remained. The status update uses a color coding system–red, yellow, green–to indicate the status of various elements of the project. Of the seven elements of the project being tracked, four of them were marked in red indicating a severe risk to the success of the project. For instance, under the heading “Infrastructure” the overall schedule status is red. From the date the state received this report there were only 10 business days left before the site went public.
The Design, Development and Configuration section of the status report notes “The SDD [Site Design Document] has been rejected by the Exchange – New plan is to submit an updated document post 10/1.” Kevin Walsh at Xerox told me “Requirements for 10/1 shop, compare and enroll activities were all documented, but many of the functions not needed until 1/1/14 (the effective date of coverage) are still being worked on and will need to be part of the document before it is accepted by the client.”
The next subsections of the status report deals with “portal configuration.” It states “The teams are working to meet the code freeze date to allow UAT to commence the week of 09/20.” UAT stands for user access testing, which is the final stage of real world testing before the roll out of a new site. Code freeze refers to a break in the programming which allows features to be tested. What this indicates is that Xerox hoped final testing could begin six business days before launch.
I asked Xerox if they had met their September 20th deadline for code freeze and whether this left enough time for final testing (UAT). Kevin Walsh replied by email “the actual date was 9/23. Extensive testing by the end user community was done prior to the code freeze, but that was done in test environments. The formal UAT was intended to make sure that the production environment reflected fixes to bugs that were identified in the test environments going back as far as early August.” So final testing had just five business days. As we’ll see in a moment, not all the bugs were successfully caught before the release date.
In the final weeks before the go live date, Nevada moved their data center from Dallas to Pittsburgh. As a result System Integration Testing (SIT) was put badly behind schedule. The September 13th status report reads “SIT DELAYED – due to issues with environment prep. No ETA provided by SIT environment team. Lost 25% of available time to date (4 days) with no resolution in sight.” This is followed by a blunt admission “Not enough resources will be available in Choice DW Team to support SIT and to finalize development before go live.”
Again I turned to Kevin Walsh at Xerox who had this to say “The resource issue as of this 9/13 status report was immediately addressed and Data Warehouse (DW) testing to address 10/1 requirements was accomplished by month end. There will also need to be continued DW testing for changes made after 10/1.” So perhaps they were able to pull testing together in the remaining two weeks. But we know for certain that not every part of the exchange worked as originally intended on launch date.
One of the fundamental requirements for the exchanges is their ability to electronically transmit health enrollment information to the participating insurance carriers using an industry standard for electronic transactions known as EDI 834. As of September 13th this capability had not been implemented or tested on the Nevada exchange. The status report states “There is a block preventing the generation of an Individual EDI 834 file. This will affect 80% of our Carrier testing and is currently delaying Carrier Testing.”
Attached to the status report are meeting minutes from a meeting held by exchange director Jon Hager on September 13th. The minutes record this discussion between Hager and Jeff Drewes (who seems to be one of the developers) “Jeff noted the 834 process needs to be in place by 12/15 at the very latest. Jon noted the 834 should be the first focus immediately after 10/1.”
Apparently, the exchange’s director had given up on having a functional electronic enrollment system at launch. Because of the unresolved EDI 834 problems the current enrollment process for Nevada’s exchange is to print out completed applications and physically deliver them to the insurance carriers. Walsh explained “While enrollment transmission functionalities are being finalized, a paper-based process is getting all enrollment materials to the carriers in a secure and timely manner. We expect to have these functionalities in place in the coming weeks and to test well in advance of 12/15 to meet the needs of Nevadans buying insurance through the exchange.” December 15th is the final date on which someone can enroll online and be covered for January 1, 2014. But again, at least for the moment, the Nevada exchange is unable to exchange any enrollment records with insurance carriers except by paper.
I provided copies of the documents mentioned in this report to the software quality experts at the Consortium for IT Software Quality (CISQ). After looking them over they sent along the following response:
For a project of this magnitude, testing started very late- due to multiple issues. The testing process appears to have been slipping during the SIT (system integration test) which is indicative of major code issues (which can typically hide until you start to put the whole system together in the test environment). There are also indications of major software components not being completed in time for the SIT. This is a big red flag for a project that’s being rushed to meet an immovable deadline. As the saying goes: ‘Nine women can’t have a baby in one month’. The integration issues also suggest that the structural quality of the code was lacking any kind of oversight during the bulk of the actual software construction phase of the project. Interfaces seem to be a big issue identified PCG. Interfaces are the toughest part to work out and the software quality of those interfaces is the most typical source of ‘gotchas’ in late stages of projects.
Not all of the problems with the Nevada exchange were caused by internal scheduling. The final September status report also notes “Have not received Medicaid or CHIP plan data, lacking Bridge Plan identification.” The “owner” of this action item was listed as DWSS, the abbreviation for the Nevada Division of Welfare and Supportive Services.
September 13th seems like a very late date to be delivering plan data so I asked Xerox what caused the hold up. Kevin Walsh replied “the HHS data goes to the HIX system through DWSS, it is a complex flow. It’s not possible for us, as the end user, to determine where the delays might have been if there were any.” So it’s possible the delay was at the state level or it might have been a failure on the part of HHS.
A spokesman for the Nevada exchange did not respond to phone requests in time to be included in this story. Thursday morning the documents mentioned above–which have been available online for weeks–were down. [See update below]
The Nevada exchange did go live on October 1st and the results appear to be what you would expect. Some sample comments from the exchange’s Facebook page make clear the site did not work as smoothly as hoped:
- Can’t sign up if we can’t even apply…each day I get one step closer….today I finally got a verification code, but can’t get past the “accept terms” page. Needs to be operational!!!
- I have had to call multiple times since October 1, and I have yet to be able to access the website. The farthest I have gotten is to register. I have never been able to log in. I know there are glitches and it’s new… but this is so frustrating!
- I am getting terribly frustrated…I cannot connect to a live person on the phone not do you respond to my e-mails…I try to create an account…first I get error messages then I’m told my account is locked.
- Come on guys, really? STILL stuck on the security question. Can you PLEASE get this working
- Could you at least make some kind of news announcement at to when the health link really will be up and running. It is VERY DISCOURAGING to go to the site everyday and watch my computer try to connect to nothing.
- I have been trying since October 1st to shop for insurance, I try everyday and I still cannot get passed the security question it just erases my answer and goes no where. I am losing hope and dont know what else to do. Are people really getting though?
- This is all I get when trying to log in both yesterday and today! “You have encountered an unresolvable error. The issue has been logged and administrators have been notified. You will be notified by email when the situation is resolved.”
The Nevada exchange site has been taken down at least twice–October 3rd and October 8th–to apply software patches to fix various bugs. One site security consultant with extensive experience on major government IT projects, but who asked to remain anonymous, summed it up this way, “They are building the airplane the same time they are flying it.”
5_4_3_6_Semi_Monthly_Project_Status_Report_26_of_34_Final –
Update: CJ Bawden, spokesman for the Nevada exchange, emailed to object to my statement that the documents used in this report were “taken down” (as I’d stated above). He writes, “The entire exchange.nv.gov site was down at various times this morning. This was an issue that affected many of the Nevada State Government Agencies.” So it appears the documents were not taken down they were just down. All of the documents are once again accessible. CJ adds that Nevada, by posting these documents online, has been more transparent in this process than other states.
COMMENTS
Please let us know if you're having issues with commenting.