Firmware Management – The Evil Mistress
Firmware management is not traditionally a topic that anyone gets excited about. In fact, it’s typically relegated to the bottom of the IT team’s list of priorities because it’s often painful and time consuming.
From a management perspective, budget spent on firmware management is very difficult to justify and ROI is nearly impossible to measure. But firmware is an evil mistress - if it’s not attended to often enough - out of date firmware can stall an OS upgrade, prevent the introduction of new hardware or worse, cause an unexpected outage of critical equipment in the data center.
We all have stories of servers, switches, storage arrays, etc. that ran fine for years until one day, the system crashed for no reason. We later discovered that there was a critical bug fix quietly released two revisions of firmware prior to the one we were running. Grrrr!
Why Firmware Has Always Been a Big Yawn 
Every product in the data center has its own unique challenges from a maintenance and support perspective. One class of devices that burns many FTE hours is server firmware management.
Servers are unique in that they typically have multiple, different components that need to be upgraded - NICs, HBAs, motherboards, storage adapters, video adapters, etc. Historically, each component required a separate download and update procedure. With chassis-based servers, there are additional items such as switch and fibre channel modules to deal with.
The firmware upgrade process with most vendors still involves a lot of very careful planning that includes:
- Research and alignment of all firmware versions across components/manufacturers to ensure compatibility
- Time spent orchestrating the order in which upgrades take place
- Careful monitoring of each individual process before moving on to the next
“Upgrading an entire server farm can take hours or more likely days.”
Now for the Exciting Part!
In 2009, Cisco introduced their Unified Computing System (UCS). In just five short years Cisco has become the # 1 provider of x86 blade servers in the Americas. There are many aspects of Cisco’s UCS Server infrastructure that turn traditional server management on its head, but the one I want to focus on today is firmware management.
I’m a big fan of the KISS principle (Keep It Super Simple). Cisco has done a great job in simplifying what has traditionally been a very time consuming, complicated and sometimes convoluted process.
Downloading Simplified
Finding all the software you need to complete firmware upgrades has been greatly streamlined with UCS. You simply log into Cisco’s support site and navigate to Downloads then Servers – Unified Computing. You then have the option of selecting from multiple current versions in the navigation tree on the left.
Once you choose the version you want, you download the UCS Infrastructure Bundle and the Software bundle for the servers in your environment. In this case, B-Series blades. After you download the bundles, you import them into the UCS firmware repository.
That’s it! That’s all you need to do to update every component in the UCS infrastructure along with the server blades and all the components within them.
Updating Simplified
In version 2.1 of UCS Manager, Cisco introduced the UCS Firmware Auto Install functionality. If you’re not familiar with Firmware Auto Install, it’s a wizard driven option for upgrading all firmware within the UCS environment. There are two wizards, one for upgrading UCS Infrastructure Firmware and the other for upgrading Server Firmware as shown below:
Of course the order in which the components should be upgraded is important so Cisco baked their best practices into the wizards to ensure that upgrades are executed in the correct order. Again, simplifying the process!
“Cisco baked their best practices into the wizards to ensure that upgrades are executed in the correct order.”
Upgrade order:
- UCS Manager
- IO Modules
- Fabric Interconnects
- Servers
As long as all the correct redundancy has been configured, every component within the UCS Infrastructure can be upgraded with no impact to production. Server upgrades obviously have to be scheduled, but you can queue them up all at once and then reboot the servers on your own schedule to complete the upgrades. After reboot, every component within the server, NICs, HBAs, CIMC, etc. will be running the updated version. It’s that easy!
“Every component within the UCS Infrastructure can be upgraded with no impact to production.”
Life Simplified!
There are obviously a few nuances and minor steps I did not include above, but I have captured the overall process of firmware management in a Cisco UCS environment. I have been supporting data center infrastructures for many years, and I never got excited about firmware management until I started working with Cisco UCS. As sad as this may sound, the fact of the matter is that maintaining firmware in UCS environment is something I look forward to. Dare I say, get excited about!
“Maintaining firmware in UCS environment is something I look forward to. Dare I say, get excited about!”
About the Author
Kevin Richards is a 20 year IT industry veteran with a background in all aspects of data center architecture, design and implementation. Kevin’s experience spans a broad range of technologies from servers and blade systems to switches, routers, firewalls and load balancers as well as SAN, NAS, and storage infrastructure. During his career he has worked with clients across all verticals including Banking & Financial Services, Pharmaceuticals, Healthcare and Manufacturing. Kevin holds certifications from VMware, NetApp and Cisco.
