Kansas Health Institute | Information For Policy Makers. Health for Kansans.

TOPEKA — Information technology experts within state government are calling the computer crash that started last week and continued Wednesday at the Kansas Department of Health and Environment the biggest within living memory both for its duration and for the magnitude of the various systems disrupted because of the failure.

"This is extremely rare," said Joe Hennes, director of the Division of Information Systems and Communications (DISC). "I've worked for the state for 35 years and I've seen systems go down. But to see stuff down for three or four days, that's rare.

"When you start planning for disasters, people tend to think about a smoking hole in the ground, a terrorist attack, or a natural disaster," he said. "But this is probably a more realistic disaster than those kinds. I don't know what the cause is. Nothing has been communicated to me about the cause or what steps are being taken to fix it."

KDHE officials in initial public statements described the crash as, "experiencing connectivity issues."

But Hennes and other experts said it appeared to be something other than that.

DISC runs KANWIN the state's wide area networking service and KDHE, like other agencies, is plugged into that network.

"It is not a network problem from our standpoint," Hennes said. "Everything is fine within KANWIN. There were no alerts. Nothing showed up nor was anything reported to us. Several of us have offered to come and help. I firmly believe it is an internal (KDHE) problem because we've received no response from them.

"To be real honest with you, the relationship between DISC and KDHE hasn't been the best simply because they absolutely value their autonomy. That's just the way they operate and the way they're reacting follows right along that same line. But I wouldn't wish this on anyone. This is the worst."

The problems began at KDHE last Thursday and agency officials said today that they remained uncertain when things would return to normal.

"We should have a better idea later on this week as far as a timeframe for full functionality," said Kristi Pankratz, an agency spokesperson. "Services continue to be restored daily, even by the hour."

The KDHE problems have been a hot topic among IT and public health circles this week.

"This is a big deal. In my 10 years, this is probably the biggest happening of this kind I've seen in the state," said an IT official who asked to remain anonymous so not to appear critical of KDHE. "It's nobody that's at fault. This is just what happens."

But no one seems to remember it happening on this scale before.

The problems at KDHE have shut down several digital systems relied upon by local health departments and others for services such as immunizations and disease surveillance.

That means local health officials this week have been doing many things the old-fashioned way, relying on paper records and fax machines.

"We're trying to manage," said Lisa Horn, communications coordinator for the Lawrence-Douglas County Health Department. "You just kind of have to go back to paper and faxes and how we did things a decade ago. Luckily, a lot of the people here have been here that long and they can figure out how to get things done in the meantime. We did a lot of stuff on paper up until six or seven years ago. We're pretty much fully automated now with computers. But that wasn't always the case."

Horn said the KDHE computer problems among other things made it difficult for the department to manage its WIC program, which provides supplemental nutrition aid for low-income mothers and children. Douglas County has more than 1,500 WIC enrollees.

"When the KWIC system was down we were unable to know who was coming in for appointments, etc., which was tough," Horn said.

Also this week, in Lawrence and across the state, local health departments have been busy with immunizations for children preparing for the new school year.

With the state's immunization registry offline, that process has become a bit more complicated.

In Lawrence, Shawnee County, and other places, officials responded to the KDHE outage by asking parents to bring along paper records of their children's immunization histories.

"So far, it's not a huge issue for us," said Misty Kruger of the Shawnee County Health Agency, which earlier this week began telling parents they couldn't get their children vaccinated unless they came in with paper histories.

"Yesterday, they said they didn't have to turn anyone away," Kruger said of the agency's nurses. Everyone who showed up had a paper record.

KDHE agency email was one of the first things restored after the problems were detected last week.

But as of Wednesday, agency officials said a number of KDHE programs were still affected, including, "the Kansas Health Alert Network (KS-HAN), the State Volunteer Registry (K-SERV), the Kansas Immunization Registry (WebIZ), many Division of Environment programmatic databases, multiple applications for the Bureau of Family Health programs (WebBFH), and Oral Health Screening. Additionally, the Bureau of Public Health Informatics is currently unable to issue birth or death certificates and cannot compute or provide health statistics."

KDHE IT experts with help from at least one other state agency and vendor representatives were working "tirelessly" to solve the problems, officials said, though they were able to offer little in the way of explanation for what had gone wrong.

"We're still trying to determine what happened," said Katie Patterson-Ingles, an agency spokesperson. "The information I have is not enough to satisfy people, but that's all we have."

But a technician with close knowledge of the problem said a hardware failure had "messed up" the agency's storage area network, which at KDHE includes a number of virtual servers.

The sources said KDHE was having to restore data from backup, which in the case of KDHE would mean many terabytes of data. One terabyte equals 1,000,000,000,000 bytes of data. Restoring that much data can take many days, experts said.

The sources said the integrity and confidentiality of the agency's mountains of data in backup had not been compromised by the hardware failure or the resulting problems to the agency's storage area network.

But a full explanation of what happened and the possible damages done, the interviewed experts agreed, likely won't come until after the problems are solved, the technical forensics are complete, and state technicians begin "lessons-learned" sessions to dissect the crash and understand better how similar ones might be avoided in the future.