Solution to prevent downtime due to bad updates in IT - OS and AV
Friday, April 23, 2010
Latest reports from across the internet about a bad update from McAfee. We have seen this kind of mess-up happening every now and then. No AntiVirus is 100% foolproof. There might be updates that cause performance issues, some delete files due to an odd signature. One reason why I always suggest clients to invest in a testbed environment so that what ever the updates are be it OS Patches, Policy changes or AV Updates. All changes go through this environment which makes it easy to spot problems and it later helps when your machines are still working fine and your competitors are not.
Below are some points I would like to mention about the deploying updates or changes of any kind.
1) When to updates: Every company has to take a call as to when they want to deploy patches or updates. They should not just go ahead and deploy updates just because the vendor has published them. A thorough testing needs to be done and the update has to pass internet audit checks. Companies are apply these patches after 8-24hours after they have been published because during this time many of them would have already installed and tested it and if it is going to open a Pandora's box you still have the lead. Some of you might disagree as too 24hrs is a long time for somebody to hack into the systems and steal data but friends. Every company has to take a decision based on CIA what is acceptable and what is not.
2) A Backup Plan: Another very important point is to always be ready with a backup plan. What if you have deployed it and something went wrong, you need to know what to do if the latest change fails and you should be able to revert back to a working condition as fast as possible.
3) Workaround Solution: Good to know a workaround solution if any for the updates being pushed to clients. This helps because sometimes workarounds are easier than deploying patches. For eq. closing a port on a firewall or a change in Group Policy.
4) Testbed Environment: A company should invest in keeping up a testbed environment which is used to deploy and test updates, patches and changes of any kind. This environment should be kept up and should be stable as to replicate the current company systems to ensure any issue with the changes are caught before they are pushed to thousands of clients.
Every company needs to take this decision, there are costs and extra processes involved but no company can afford a downtime so careful planning has to be done.
Anish
