In my previous post we examined the rolling upgrade process for the network infrastructure (Infiniband Gateway switches), so next in line is the upgrade process for the storage infra, i.e. the ZFS 7320 storage appliance.
Like every piece of hardware in the Exalogic, the ZFS 7320 storage appliance has high availability built-in. In this case we have two ‘storage heads’ that are working in active-passive mode to ensure high availability. This enables us to perform a rolling upgrade.
I will demonstrate that the ZFS storage heads can be upgraded in rolling fashion, without severely impacting the storage services and keeping the Exalogic and everything running on it online while doing it!
First of all we should again look at the README.html and additional documentation for patch 13113092 and carefully follow the instructions pertaining to the ZFS 7320 part.
ZFS Storage Appliance (ZFS_Storage_7320)
Upgrading the software and ILOM on each storage head involves the following steps:
1. Updating storage software to version Q3.2.1
2. Applying the ILOM update to the storage nodes
3. Updating storage software to version Q3.3
The factory default version of the ZFS storage 7320 software in Exalogic v1.0 and v18.104.22.168.0 systems is Q3.1.1. Upgrading that version directly to Q3.3 is not supported.
So apparently there are two types of firmware to be updated: storage software and ILOM software. For the first type we have to apply an intermediate update first, upgrading directly to Q3.3 should not be done.
As before, we should first check the current firmware versions. For this we should login to the service processor on the storage head ILOM interfaces.
Checking the storage software versions:
% ssh firstname.lastname@example.org Password: Oracle(R) Integrated Lights Out Manager Version 22.214.171.124 r61398 Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. -> start /SP/console Are you sure you want to start /SP/console (y/n)? y Serial console started. To stop, type ESC ( xxxxsn1:> show Properties: showcode = false showstack = false exitcoverage = false showmessage = true asserterrs = false Children: configuration => Perform configuration actions maintenance => Perform maintenance actions raw => Make raw XML-RPC calls analytics => Manage appliance analytics status => View appliance status shares => Manage shares xxxxsn1:> maintenance system updates xxxxsn1:maintenance system updates> show Updates: UPDATE DATE STATUS email@example.com,1-1.16 2010-11-1 12:46:16 previous firstname.lastname@example.org,1-1.21 2011-3-10 23:49:47 previous email@example.com,1-1.25 2011-4-29 15:48:52 current Deferred updates: The appliance is currently configured as part of a cluster. The cluster peer may have shared resources for which deferred updates are available. After all updates are completed, check both cluster peers for any deferred updates.
We can see the update history and the version we have now. We find that we have the same current version firstname.lastname@example.org,1-1.25 on storage head 2 as well, as it should be (not shown here).
This can also be done via the ZFS 7320 storage webconsole, by going to the ‘Maintenance’ section and then to the ‘SYSTEM’ tab. I will use the webconsole gui later to check the update when it has been applied, see below.
Which version is actually good ? Further down in the README we find the version to check for:
17. Select the newly downloaded version
zfssn01:maintenance system updates> select email@example.com,1-1.21
zfssn01:maintenance system updates firstname.lastname@example.org,1-1.21>
So, apparently we are already on a somewhat higher version of the storage software as is required. We’re good here.
Checking the ILOM software versions:
Furthermore, we also have to check the ILOM versions. This can be done both via the ILOM gui and the CLI. The README.html states
Upgrading ILOM on the ZFS Storage Head
Check the current ILOM version:
SP firmware 126.96.36.199
SP firmware build number: 58107
SP firmware date: Tue Aug 17 01:35:56 EDT 2010
SP filesystem version: 0.1.23
If the firmware build number is less than 61398, you need to upgrade it.
Let’s check our system :
xxxxxsn1:maintenance system updates> Serial console stopped. -> version SP firmware 188.8.131.52 SP firmware build number: 61398 SP firmware date: Tue Dec 21 17:08:53 EST 2010 SP filesystem version: 0.1.23 xxxxxsn2:> Serial console stopped. -> version SP firmware 184.108.40.206 SP firmware build number: 61398 SP firmware date: Tue Dec 21 17:08:53 EST 2010 SP filesystem version: 0.1.23
Via de ILOM webconsole you can find the version as well, via the ‘Overview’ subtab of ‘System Information’.
Looks like we are current here as well. So, as with the Gateway switches, we find that there are no updates to be applied at present because we already applied them with the october 2011 PSU 12634555. Instead I will again take the upgrade to version 220.127.116.11.0 (patch 13795376) as an example. Before we can start applying any patches however, we should first check which head is the active one, so we can do the other head first as suits our rolling upgrade process.
Login to the ILOM (if you’re not still there).
Checking storage head 1 : xxxxxsn1:> cd / xxxxxsn1:> configuration cluster xxxxxsn1:configuration cluster> show Properties: state = AKCS_OWNER description = Active (takeover completed) peer_asn = 9faf8ff1-c3a8-c090-8f4e-9871618a152e peer_hostname = xxxxxsn2 peer_state = AKCS_STRIPPED peer_description = Ready (waiting for failback) Children: resources => Configure resources
So we see that storage head1 is the active head and we should patch storage head2 first. For consistency one should also check te other the other head, but I will not show it here.
We can also see this very nicely via the 7320 storage webconsole under ‘Configuration > CLUSTER’ (figure 1) :
Figure 1 : Cluster configuration and state of the ZFS 7320 as shown in the webconsole GUI on head1.
BTW: the management webconsole can be found at https://<storage head ip>:215. The console for storage head2 (figure 2) actually shows the reverse of this view, mirrored left-to-right. Also note the FAILBACK and TAKEOVER buttons.
Figure 2 : Cluster configuration and state of the ZFS 7320 as shown in the webconsole GUI on head2.
OK, since storage head 1 is actively in control, we should upgrade head 2 first. We will again take the 18.104.22.168.0 upgrade as an example, as there are actually some updates to be done. We take “Exalogic Elastic Cloud Software Upgrade Guide, Volume 1 Release 22.214.171.124.0″ (file eecs_20_upg.pdf when unzipping the 126.96.36.199.0 patchfile p13795376_10020_Linux-x86-64.zip).
The upgrade guide states the following :
Upgrading the ZFS software to version 2011.1.1.0, and ILOM on each storage head involves the following steps:
1. Upgrading ZFS Storage 7320 Software from Version Q3.1.1 to Version Q3.2.1
2. Upgrading ILOM on ZFS Storage Heads
3. Upgrading ZFS Storage 7320 Software to Version 2011.1.1.0
So again three upgrade steps to do.
Step 1: Upgrade to storage version 2010.Q3.2.1
Further on we read:
“This section describes how to upgrade the ZFS Storage 7320 software from version 2010.Q3.1.1 (2010.08.17.1.1,1-1.16 – the factory default for Exalogic 1.0) to version 2010.Q3.2.1 (2010.08.17.2.1,1-1.21). This version is a prerequisite for installing the 2011.1.1.0 version, which is the target software version in this upgrade kit for the Sun ZFS Storage 7320 appliance.”
As mentioned earlier in section 1.2, we are already at storage version email@example.com,1-1.25, which is higher, so we only have to do steps 2 and 3, good.
Step 2: Upgrade ILOM on the ZFS Storage Heads
We have already gone through steps 1-6 of the guide, on to 7 it is:
7. Stop the system by running the following command:
xxxxxsn2:configuration cluster> Serial console stopped. -> stop /SYS Are you sure you want to stop /SYS (y/n)? y Stopping /SYS
8. To load the new FW file, you require the FTP URL. The FTP URL is of the following format: ftp://root:<root_password>@<ethernet_address_of_active_ZFS_storage_head>/<path_ to_ZFS_ILOM_update_dir>/188.8.131.52_r65138/ILOM-3_0_16_10_r65138-Sun_Fire_ X4170M2_X4270M2.pkg
Ok let’s do that, using our patcher account for FTP which we defined via the ZFS storage console before:
-> load -source ftp://patcher:mypassword@<storage VIP address>//export/common/patches/todo/
ILOM-3_0_16_10_r65138-Sun_Fire_X4170M2_X4270M2.pkg NOTE: An upgrade takes several minutes to complete. ILOM will enter a special mode to load new firmware. No other tasks can be performed in ILOM until the firmware upgrade is complete and ILOM is reset. You can choose to postpone the server BIOS upgrade until the next server poweroff. If you do not do that, you should perform a clean shutdown of the server before continuing. Are you sure you want to load the specified file (y/n)? y Preserve existing configuration (y/n)? y Delay BIOS upgrade until next server poweroff (y/n)? n ..........................................................................................
............... Firmware update is complete. ILOM will now be restarted with the new firmware. -> /sbin/reboot Connection to xxxxsn2-c closed by remote host. Connection to xxxxsn2-c closed.
OK that was the first reboot after upgrading… wait till it comes back online and login to the ILOM again:
12. Run the following command to verify the new ILOM version: -> version
Actually this is not needed as it already reports the version on login :
% ssh root@xxxxsn2-c Password: Oracle(R) Integrated Lights Out Manager Version 184.108.40.206 r65138 Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved. -> version SP firmware 220.127.116.11 SP firmware build number: 65138 SP firmware date: Wed Jun 15 08:02:17 EDT 2011 SP filesystem version: 0.1.23
Now it says we need to set the boot option to “ROM Enable” in the BIOS, I will not show this here. After doing so, it reboots once more, and we need to log back in once more to check the final status:
23. Run the following commands to check if the current head is Ready (waiting for failback), (that is, description = Ready (waiting for failback) ):
xxxxsn2 console login: root Password: Last login: Tue Mar 20 13:05:42 on console xxxxsn2:> cd / xxxxsn2:> configuration cluster xxxxsn2:configuration cluster> show Properties: state = AKCS_STRIPPED description = Ready (waiting for failback) peer_asn = a54b53a0-afba-eae1-a77a-b0013813b629 peer_hostname = xxxxsn1 peer_state = AKCS_OWNER peer_description = Active (takeover completed) Children: resources => Configure resources
OK, we have thus upgraded the ILOM software of storage head2 to version 18.104.22.168 r65138. We are now ready for step 3, upgrading the ZFS Storage 7320 Software to Version 2011.1.1.0.
In the next post, we continue where we left off to finish the upgrade of the ZFS storage by doing step 3 of the upgrade, then a switchover the active storage role to the newly upgraded head2, then upgrade head1 and finally check if eveything went OK. Of course in real life this should all be done without pause in a single excercise!
Hebt u vragen of suggesties?
De Bruyn Kopsstraat 9
2288EC Rijswijk (ZH)
+31.(0)70 319 5000