Patching your software and information technology investments can be a daunting task. Not only does it take a lot of time, but it usually must be done after hours within a narrow service window. Plus, inventorying all your systems can be nearly impossible, as is getting accurate reporting after the task is complete. If you don’t have a good documented process for deploying patches and making sure all your systems are back up and running when complete, then you are just taking the “patch and pray” approach and hoping that it all works when you are done. Missing a key step and/or running into problems is a sure-fire way to getting yelled at by management, or worse. Patching is an essential task; doing it right won’t get you any kudos but doing it wrong will get you the worst kind of attention.
This document does not cover in detail the mature processes you need for good software patching practices, but it does provide a basic roadmap you should follow. If software patching has suddenly become a priority for your organization—and that is probably why you are reading this now—go through the following list and prepare. You should also consider engaging Patchworx personnel for some consulting and/or patching support. We have more experience in this area than just about anyone. Another option to consider is to outsource this task to us. We are very good at it and we will make sure it gets done every month, reliably, accurately, and in a narrow service window. We also smoke test your critical devices to make sure they are working properly and provide you with detailed reporting that proves the task is done. Those reports will make your managers, auditors, risk managers and CISO quite happy. You will be a hero for doing so.
Patchworx utilizes a detailed 31-step process for applying patches in each patching cycle that we keep proprietary, but the following checklist will speed you along if you decide to venture out on your own. If you have the time, patience, knowledge and adequate human resources, you will need to decide what your patch management plan needs to look like and how to best implement it.
Here is Your Software Patching Tips Checklist:
- Inventory all your servers, PCs, and laptops you plan to patch – Don’t underestimate how hard this can be with even just 100 systems. If you have thousands, it can seem impossible without the experienced personnel, tools and processes to get it done accurately.
- Inventory all your software – Most systems run dozens of different software titles. You can end up with hundreds of different software packages to patch in your enterprise. You can’t know what you need to patch until you know what you have.
- Inventory and patch your hardware-based appliances like firewalls, routers, SANs and more – This will be a whole other exercise beyond your operating systems and applications.
- Create a routine – Set a schedule at the same time every month to patch your systems. Will you do it all in one big event over a weekend? Or will you patch 20% of them at a time on five different nights? Doing the latter can mitigate impacts from unexpected problems, but it is a lot more work.
- Identify dependencies – In most enterprises you need to identify if you have application dependencies that require a certain server reboot order for everything to work properly on restart. A good example is that it is best practice to bring down a multi-tier system by starting with the presentation tier (web server), then the application tier, and lastly the database tier. Your systems should be brought up in reverse order. Read the release notes or ReadMe files to learn more about the implications of deploying a set of patches. You should also review software user forums to see if anyone else is reporting problems with the new patches.
- Deploy patches in a timely manner – It is prudent to deploy often and quickly, but unless there is an imminent threat, don’t patch in a hasty time frame. Rushing to deploy the patches without testing or checking to see if others are having problems can come back to haunt you. Check with others in your software user communities. A good rule of thumb is to apply patches 30 days from their release.
- Test your patches – As mentioned above, before applying patches to your production system, you should test the patches. You should have a test environment to do this properly. Test systems are expensive and require a whole new budget if you don’t already have one. You basically need to double up on everything, and that requires buying a lot of extra hardware and software to create your test environment. If you don’t have a budget to test, then consider outsourcing to Patchworx. We test patches before deployments.
- Do your servers require a reboot? – During the testing process, verify if the servers require a reboot or if they do so automatically. If a reboot is required, you need to plan a maintenance window. As a standard practice, maintenance windows are prudent to schedule when applying patches to production servers and PCs. Don’t let unexpected system reboots hurt your business operations or damage your databases, etc. Expect 90% of your patch deployments to require reboots.
- Smoke Testing – When you apply patches, implement smoke testing procedures. You need to do this to be certain your critical applications and services are back online and running properly upon completion of patch deployments and reboots.
- Change Management is important – This step is often overlooked. Involve all your key stakeholders in the company. These stakeholders will inform you about system or organizational needs that will affect your patch deployment processes. See, What Is Change Management and Why Is it Important?
- Notification – Let your users know when you plan to do your patch deployment. They need to know what to expect. Tell them what they should do if they have problems after the patch deployment. When patching workstations, remind the users just prior to patching to save all documents, close all applications, and logout of their workstation. Be clear to tell them, “DO NOT SHUT THE PC DOWN.”
- Have a good roll-back plan – If/when something goes wrong with the patch deployment, you need a roll-back plan. Having this plan gives you the option to quickly reverse the patches and go back to your pre-patched status. Patchworx, for example, has provisions for roll-backs because it is a mature process based upon good patching tools and procedures.
- Have a good backup of all your systems – If possible, make a snapshot image of all your servers just prior to executing the patch deployment.
- What automatically scheduled maintenance jobs do you have? – Watch out for automated jobs such as for an SQL database. If you have maintenance jobs like these, you must be sure to put them on hold. Failure to do so will really mess things up and make your life horrible.
- Automated tools or our Patchworx service – Use these resources whenever possible. Some people try to use tools like Auto-Update, but the big problem is that you cannot control when patches are applied. There is no smoke testing procedure post patching, so you won’t know if your system is back up and running properly. You also won’t have any accurate post patch deployment reporting you’ll need to show management, auditors and regulators that you got the job done successfully.
- Review your fresh patch report – This can take a lot of work to get set-up and reporting accurately. This is where most fail after a lot of hard work. You need to do this to look for patches that failed to deploy. When this happens, and it will, you need to investigate why they failed. From there, you need to create your remediation plan, reschedule and then redeploy.
- Accommodate your exceptions – Invariably you will have servers or applications that cannot be upgraded or patched. For whatever reason, you will have that critical application you use, but you won’t be allowed to update patches because it will break your application. If you are in this situation, you need to find some other method for securing those systems from the vulnerability left exposed by your inability to deploy the patches.
- Repeat – Make your patching cycles repeatable. The key here is to just make it a machine, a patching machine. That is what Patchworx is. If you find you are struggling, as most do, consider letting us do the patching for you, so you can focus on more important tasks and get some rest at night knowing you are patched. With Patchworx, you will Get Patched, Stay Patched, and Be a Hero for doing so.
Contact us for more information on our Advanced Patch Management Solution, Patchworx℠!
