On February 12, Stan State’s campus network systems went down due to hardware failure. Despite backup systems becoming available the following day, these technical difficulties had many campus services unavailable for days, including the on-campus Wi-Fi that many students have been using throughout virtual learning sessions of the past year.
“In this case, it was strictly failure of the hardware,” said Geoffrey Cirullo, Deputy Chief Information Officer and Director of Technology Services at Stan State.
Cirullo oversees everything technology related, including the new data center, the Help Desk, Client Services, and Information Services which includes programmers and workflow products.
“I was the focal point during the [outage] that occurred on campus,” explained Cirullo.
Cirullo states that, although there was much inconvenience caused by the network outages, the situation could have been much worse. During the outage, the Office of Information Technology (OIT) was able to maintain availability of 80% applications necessary for students, faculty and staff. Cirullo credits a recently upgraded data center for keeping damages at a minimum.
“Building a brand new data center last summer really prepared us for this outage that occurred,” said Cirullo.
OIT experienced a similar outage over the summer as the data center was in the process of getting upgraded. As part of the new data center, an entire disaster recovery system was created. This allowed students and staff to continue to have access to the major systems while the change occurred with the data center.
“What happened with this outage is we started experiencing some intermittent hardware failures,” said Cirullo. “We struggled with them the whole week and I had my team working 7 by 24, staying up all night. That’s our life as a Systems Administrator.”
Cirullo explained they were trying to keep the system up but experienced a complete system failure on Saturday night. The decision was made to move over to the disaster recovery system. This ensured access to necessary functions to those working off campus, but they soon learned that the access on campus was limited.
“We knew that the students in housing were impacted through not being able to get Wi-Fi,” said Tawn Anderson, Director of Client Services.
Anderson and her team were able to facilitate incoming requests to help in the prioritization of necessary services that went down during the outage. The team was able to provide Wi-Fi hotspots to many of the students living in on-campus housing. “We had hotspots available for checkout that we were just about to return to the vendor. We just luckily still had them,” Anderson explained.
Nevertheless, there still was a need to work towards getting Wi-Fi systems on campus back up to full strength and accessibility. Cirullo explained that Campus Police and Campus Healthcare Services had to take priority over the rest of systems due to public safety concerns. 
Cirullo’s teams were eventually able to figure out which hardware failed through a process of elimination. Once the failed hardware was discovered, they were able to fix the necessary cards and repair disks.
“Two years ago, there’s no way that we could have survived what we went through with this,” Cirullo said. “It is a great amount of loss of service. It would have been down for about a week.”
One of the campus services that became unavailable during the outages was the campus radio station, KCSS. The station was unable to broadcast on their FM frequency 91.9. According to KCSS General Manager Dr. Greg Jacquay, the station utilizes campus Wi-Fi to get the radio signal from the station itself to the radio transmitter.
“KCSS was down for three days,” said Dr. Jacquay. “We were unable to send audio from Demergasso-Bava Hall to Bizzini Hall.”
Demergasso-Bava Hall is where KCSS Studios are located and the KCSS radio transmitter is located on top of Bizzini Hall.
“The FCC allows for a limited amount of downtime,” Jacquay explained. “Our webstream was unaffected, but the FM was down for three days.”
Jacquay said that there were no concerns of receiving a fine, but if the outage lasted seven days, he would have had to notify the Federal Communications Commission.
OIT says that they are committed to making sure that Stan State experiences less downtime. Cirullo and Anderson explained that they are in the process of upgrading and expanding to keep system downtime low.
“99.9% of the time everything is up,” said Cirullo. “We want to say that, even if this does happen again, there will be no loss of service. I want to say 99.99% of the time everything is up.”
If there are questions or concerns, Anderson and Client Services are available to support. If you are experience systems issues, please email [email protected] or call the Help Desk at 209-667-3687. The Help Desk staff can open and check the status of tickets. Tickets can also be submitted through the portal and you can chat with the Help Desk on their website, which is located on the lower right corner of the page.