Do you suffer from CINO?!?

Do you suffer from CINO? 

What is CINO you ask?  CINO is what can happen when we try to fix something that isn’t broken.  CINO, or Change Induced Network Outage, is the glorious feeling of causing network problems on a network that prior to a change we rolled out had none.  

CINO most commonly is invited into our lives from a software update or configuration change that is typically cloaked as “proactive.”  Recovering from CINO is typically a very long road and it’s very common that the keystroke or click that launched the deluge of terror remains with us for a very long time, likely even haunting good sleep.  

Although not exactly a pass, it’s mildly comforting to know that even the biggest and best networks do it.  Facebook, Amazon, our cellular providers, etc have gone through it.  Akamai made a DNS change in 2021 that brought down websites all over the globe!  Of course, the fix 99.9% of the time is to roll back the change that was made.  

That’s a lot of grief for a little what?  Why do we do it anyway?  It’s a simple fact that making changes to the network increases our exposure to downtime.  An interesting sidebar if I may, but has any one person ever done it twice?  Certainly not three times.  Let’s look at a few reasons why it might be done and what we can do to avoid creating a CINO event!  

Both software updates and configuration changes can largely fall under the umbrella that one is attempting to do some sort of preventative maintenance.  That is making a change with the intent of protecting the device or network from potential risk. Software updates are tweaks in already deployed code that usually either target security concerns or platform stability issues.  Sometimes we don’t really have a choice in the matter.  If we’re patching a recently released security vulnerability on a public-facing interface, that’s an update that needs to be applied. 

That public-facing interface and network device are very different from your access switches though.  Configuration updates are usually tied to simple protocol tweaks to try and improve performance.  These again may be necessary, but they may also not be.  It’s important to understand what the benefit of a potential change is fully before implementing it.  Working through the 7 R’s of Change Management, regardless of how small the “update” may seem, is imperative. 

I have searched for the origin of the 7 R’s, but have been unable to find it.  If you know the origin, please let me know and I will update this post accordingly.  

RAISED – Who raised the change? 

REASON – What is the reason for the change? 

RETURN – What is the return required from the change? 

RISKS – What are the risks involved in the change? 

RESOURCES – What resources are required to deliver the change? 

RESPONSIBLE – Who is responsible for the build, test, and implementation of the change? 

RELATIONSHIP – What is the relationship between this change and other changes? 

Working fully through the 7 R’s will help determine whether an update is necessary.  And once resources are assigned, in-depth research on the updates and how the updates impact other aspects of your network must be fully flushed out.  Next is setting up a lab to test.  Then designing and testing a rollback plan.  And when it’s finally gone time, don’t forget to capture backups!  Schedule a maintenance window to complete.  And then cross your fingers and go! 

Back to the why?  Again, the device's location in the network is a significant factor.  I usually hear that a company’s security group is asking that all devices be updated to the latest, most secure code.  SOX (Sarbanes-Oxley) or PCI (Payment Card Industry) compliance standards are often mistakenly over-applied.  SOX for example in section 404 specifies ‘Management Assessment of Internal Controls’ highlighting measures around access, security, change management, and backups.  

While the section does not discuss specific IT and security requirements, it does specify password policies, wireless networks, and both logical and physical network barriers for the wired network.  Logical barriers such as VPNs or firewalls must stay updated, but access switches aren’t part of the conversation.  PCI is very similar in focusing on keeping financial transactions secure from outside threats, not local area implementations. 

Additionally, it's nearly impossible for most organizations to keep all of their access switches on a consistent code for many reasons. 

Local changes are often made to address one-off issues that an individual site might be having.  In fact,  changes are often implemented that are not needed yet remain as the engineers work through troubleshooting trying to identify a ‘fix.’.  This leads to variations in the networks and configurations.  

Another example is acquiring hardware through company expansion and acquisitions.  Whether it’s even feasible to complete updating the number of access switches an organization has is another big question.  It’s common for most Fortune 500 companies to have tens of thousands of network devices.  One of our clients for example has just over 20K devices in their network.  That’s roughly 77 devices per workday to update code if a company had a dedicated team.   

And back to the change management process required.  How long does it take your organization to fully vet a network change?  According to Gartner, the average amount of time is 2 weeks.  

Now in addition to managing and updating 77 devices per day, we need to work each through a 2-week change management window.  By the way, 184 Severity-1 ‘bugs’ were updated for the 3850 platforms in 2022.  So now we have to do 77 devices per day, with 2 weeks of vetting, 184 times last year? 

The reason companies don’t do it is that the cost doesn’t justify the ‘perceived’ risk.  The reality is that once an access switch is deployed, it is rarely touched again until it's time to upgrade it, or there is a hardware failure. CINO may occur, but exposure to its impact can be minimized by separating out only those network devices and locations with public interfaces. 

 Here at Edgeium, It’s what you deserve from an IT provider. In a market dominated by products, we sell solutions. And not just any — the right ones. For you, your business, and your goals.