When Disaster Strikes, the theme of the October 11, 2017 Technology Leadership Summit , garnered a variety of insights from participants. In this blog entry, we’ll explore the first two of five insights. These insights flow from the experience of Texas technology leaders and can help you prevent natural and man-made disasters from crushing your district’s operations.
Insight #1 – Cross-Departmental Collaboration
“Process. The process has to involve HR, Business Office, and M&O,” said David Jacobson (Lamar Consolidated ISD). The Executive Director of Technology for Round Rock ISD agreed. “It’s the processes and procedures, working with all the other customers. We have to get them to understand the importance of planning. One way is to do tabletop exercises to practice to see what we would do in the event of an event,” said Mark Gabehart (Round Rock ISD). In these situations, it is important to 1) recognize the need; 2) clarify the depth of the hole the organization is in; and 3) present a plan to never be in that hole again.
Recognize the Need
“Like Burt Reynolds in the Hooper car chase scene where every building collapses and fires blaze as Reynolds deftly navigates his vehicle through them, I felt like we were replacing obsolete equipment just as it failed,” I said as I shared my equipment replacement tale of adventure. Years of neglect had left servers with critical data ready to die. It took multiple meetings with leadership teams and other department executive directors and directors to help them understand the situation.
Clarify the Depth of the Hole
As a technology director, you need to put the needs of the district in terms non-technical people can understand. For example, here is an excerpt of the table I shared with the Superintendent’s Cabinet to help them better grasp the challenges. The Cabinet had no idea that years of under-funding technology had led them to a dangerous loss of critical business data (e.g. payroll). Staff clung to old technologies because that is what they were comfortable with, not because it was the best solution for the district.
AgeStorage RequiredDescription
| Replacement Priority
1=highest |
|||
| 1 | 11 years old | 500 gigs | Central Office Business File Server |
| 2 | 9-10 years old | 500 gigs | Moodle Course Management System |
| 2 | 8-9 years old | 200 gigs | LanSweeper |
| 2 | 7-8 years old | 500 gigs | Network Configuration Services |
| 1 | 7-8 years old | 300 gigs | SolarWinds |
| 2 | 7-8 years old | 300 gigs | MS-SQL Server |
Replacing obsolete equipment and moving physical servers into virtual servers with a storage area network (SAN) can only be the first step.
Prepare a Plan (Features Updated Links)
The next step involves an equipment replacement plan. And that is then followed by a disaster recovery and business continuity plan. In one large urban district, living from one annual budget to another meant that disaster recovery was simply impossible. The district had only one network operations center where all the district servers and data were housed. There was no alternate location and, even if data was backed up (disaster recovery), no equipment to make it operational (business continuity). In part two of this series, we will share some of the lessons Texas technology directors have learned. For now, it is important to ensure regular equipment replacement with an annual budget. In my district, wise leaders set that budget at $500,000 per year after reviewing the multi-year equipment replacement plan.
Insight #2 – Disaster Recovery Planning Resources
“There are genuine resources out there to put plans together. It’s been frustrating to find resources, but now I know about various resources. We have a disaster recovery plan, but I didn’t realize how huge the business continuity plan was. How do we continue doing business?” It’s unsettling to realize that if you have no equipment to load all your backup data into and make it work, your district can’t overcome the disaster. What’s worse, the cost of recreating a network operations center (NOC) would be exorbitant, not to mention duplicating network/internet connections to district locations.
To help you think through these issues, here are a few documents shared at the Technology Leadership Summit:
- Michael Knight (CEO/CTO of Encore Technologies, TLS17 Event Sponsor) shared his slide deck on the topic, providing several snapshots documenting the evolution of disaster recovery from multiple NOCs to the hybrid cloud.
- Frosty Walker (Chief Information Security Officer of Texas Education Agency) shared his slide deck and other resources available at the Texas Gateway, including the DIR Incident Response Team redbook.
- Sample disaster recovery plan
- David Carpenter (Huffman ISD) on I’ve Been Hacked…Now What?
- Edward Doan (Google) provided his slide deck on Google Cloud Storage, as well as was able to record a short voxercast discussing the topic.
What’s Next?
In part 2 of When Disaster Strikes, we will cover a few additional insights shared during the Technology Leadership Summit held on October 11, 2017.
