Patching Oracle Exalogic – updating the ZFS 7320 storage appliance I

Part 3a

In my previous post we examined the rolling upgrade process for the network infrastructure (Infiniband Gateway switches), so next in line is the upgrade process for the storage infra, i.e. the ZFS 7320 storage appliance.

Like every piece of hardware in the Exalogic, the ZFS 7320 storage appliance has high availability built-in. In this case we have two ‘storage heads’ that are working in active-passive mode to ensure high availability. This enables us to perform a rolling upgrade.

I will demonstrate that the ZFS storage heads can be upgraded in rolling fashion, without severely impacting the storage services and keeping the Exalogic and everything running on it online while doing it!

1.1     Checking the documentation

First of all we should again look at the README.html and additional documentation for patch 13113092 and carefully follow the instructions pertaining to the ZFS 7320 part.

ZFS Storage Appliance (ZFS_Storage_7320)

Upgrading the software and ILOM on each storage head involves the following steps:

1. Updating storage software to version Q3.2.1

2. Applying the ILOM update to the storage nodes

3. Updating storage software to version Q3.3

The factory default version of the ZFS storage 7320 software in Exalogic v1.0 and v1.0.0.1.0 systems is Q3.1.1. Upgrading that version directly to Q3.3 is not supported.

So apparently there are two types of firmware to be updated: storage software and ILOM software. For the first type we have to apply an intermediate update first, upgrading directly to Q3.3 should not be done.

1.2     Checking current versions on the storage heads.

As before, we should first check the current firmware versions. For this we should login to the service processor on the storage head ILOM interfaces.

Checking the storage software versions:

% ssh root@xxxxsn1-c.qualogy.com
Password:

Oracle(R) Integrated Lights Out Manager

Version 3.0.14.11 r61398

Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y
 
Serial console started.  To stop, type ESC (

xxxxsn1:> show
Properties:
showcode = false
showstack = false
exitcoverage = false
showmessage = true
asserterrs = false

Children:
configuration => Perform configuration actions
maintenance => Perform maintenance actions
raw => Make raw XML-RPC calls
analytics => Manage appliance analytics
status => View appliance status
shares => Manage shares

xxxxsn1:> maintenance system updates
xxxxsn1:maintenance system updates> show
Updates:

UPDATE                           DATE                      STATUS
ak-nas@2010.08.17.1.1,1-1.16     2010-11-1 12:46:16        previous
ak-nas@2010.08.17.2.1,1-1.21     2011-3-10 23:49:47        previous
ak-nas@2010.08.17.3.0,1-1.25     2011-4-29 15:48:52        current
 
Deferred updates:

The appliance is currently configured as part of a cluster. The cluster peer
may have shared resources for which deferred updates are available. After all
updates are completed, check both cluster peers for any deferred updates.

We can see the update history and the version we have now. We find that we have the same current version ak-nas@2010.08.17.3.0,1-1.25 on storage head 2 as well, as it should be (not shown here).

This can also be done via the ZFS 7320 storage webconsole, by going to the ‘Maintenance’ section and then to the ‘SYSTEM’ tab. I will use the webconsole gui later to check the update when it has been applied, see below.

Which version is actually good ? Further down in the README we find the version to check for:

17. Select the newly downloaded version

 zfssn01:maintenance system updates> select ak-nas@2010.08.17.2.1,1-1.21

zfssn01:maintenance system updates ak-nas@2010.08.17.2.1,1-1.21>

So, apparently we are already on a somewhat higher version of the storage software as is required. We’re good here.

Checking the ILOM software versions:

Furthermore, we also have to check the ILOM versions. This can be done both via the ILOM gui and the CLI. The README.html states

Upgrading ILOM on the ZFS Storage Head

Check the current ILOM version:

-> version

SP firmware 3.0.9.27

SP firmware build number: 58107

SP firmware date: Tue Aug 17 01:35:56 EDT 2010

SP filesystem version: 0.1.23

If the firmware build number is less than 61398, you need to upgrade it.

Let’s check our system :

.

xxxxxsn1:maintenance system updates>
Serial console stopped.

-> version
SP firmware 3.0.14.11
SP firmware build number: 61398
SP firmware date: Tue Dec 21 17:08:53 EST 2010
SP filesystem version: 0.1.23

xxxxxsn2:>
Serial console stopped.

-> version
SP firmware 3.0.14.11
SP firmware build number: 61398
SP firmware date: Tue Dec 21 17:08:53 EST 2010
SP filesystem version: 0.1.23

Via de ILOM webconsole you can find the version as well, via the ‘Overview’ subtab of ‘System Information’.

Looks like we are current here as well. So, as with the Gateway switches, we find that there are no updates to be applied at present because we already applied them with the october 2011 PSU 12634555. Instead I will again take the upgrade to version 2.0.0.0.0 (patch 13795376) as an example. Before we can start applying any patches however, we should first check which head is the active one, so we can do the other head first as suits our rolling upgrade process.

1.3     Checking which storage head is the active node

Login to the ILOM (if you’re not still there).

Checking storage head 1 :
xxxxxsn1:> cd /
xxxxxsn1:> configuration cluster
xxxxxsn1:configuration cluster> show
Properties:
state = AKCS_OWNER
 description = Active (takeover completed)
peer_asn = 9faf8ff1-c3a8-c090-8f4e-9871618a152e
peer_hostname = xxxxxsn2
peer_state = AKCS_STRIPPED
peer_description = Ready (waiting for failback)
Children:
resources => Configure resources

So we see that storage head1 is the active head and we should patch storage head2 first. For consistency one should also check te other the other head, but I will not show it here.

We can also see this very nicely via the 7320 storage webconsole under ‘Configuration > CLUSTER’ (figure 1) :

Figure 1 : Cluster configuration and state of the ZFS 7320 as shown in the webconsole GUI on head1.

BTW: the  management webconsole can be found at https://<storage head ip>:215. The console for storage head2 (figure 2) actually shows the reverse of this view, mirrored left-to-right. Also note the FAILBACK and TAKEOVER buttons.

Figure 2 : Cluster configuration and state of the ZFS 7320 as shown in the webconsole GUI on head2.

1.4     Upgrading storage head 2

OK, since storage head 1 is actively in control, we should upgrade head 2 first. We will again take the 2.0.0.0.0 upgrade as an example, as there are actually some updates to be done. We take “Exalogic Elastic Cloud Software Upgrade Guide, Volume 1 Release 2.0.0.0.0″ (file eecs_20_upg.pdf when unzipping the 2.0.0.0.0 patchfile p13795376_10020_Linux-x86-64.zip).

The upgrade guide states the following :

Upgrading the ZFS software to version 2011.1.1.0, and ILOM on each storage head involves the following steps:

 

1. Upgrading ZFS Storage 7320 Software from Version Q3.1.1 to Version Q3.2.1

2. Upgrading ILOM on ZFS Storage Heads

3. Upgrading ZFS Storage 7320 Software to Version 2011.1.1.0

So again three upgrade steps to do.

Step 1: Upgrade to storage version 2010.Q3.2.1

Further on we read:

“This section describes how to upgrade the ZFS Storage 7320 software from version 2010.Q3.1.1 (2010.08.17.1.1,1-1.16 – the factory default for Exalogic 1.0) to version 2010.Q3.2.1 (2010.08.17.2.1,1-1.21). This version is a prerequisite for installing the 2011.1.1.0 version, which is the target software version in this upgrade kit for the Sun ZFS Storage 7320 appliance.”

As mentioned earlier in section 1.2, we are already at storage version ak-nas@2010.08.17.3.0,1-1.25, which is higher, so we only have to do steps 2 and 3, good.

Step 2: Upgrade ILOM on the ZFS Storage Heads

We have already gone through steps 1-6 of the guide, on to 7 it is:

7. Stop the system by running the following command:

xxxxxsn2:configuration cluster>
Serial console stopped.
-> stop /SYS
Are you sure you want to stop /SYS (y/n)? y
Stopping /SYS

8. To load the new FW file, you require the FTP URL. The FTP URL is of the following format: ftp://root:<root_password>@<ethernet_address_of_active_ZFS_storage_head>/<path_ to_ZFS_ILOM_update_dir>/3.0.16.10_r65138/ILOM-3_0_16_10_r65138-Sun_Fire_ X4170M2_X4270M2.pkg

Ok let’s do that, using our patcher account for FTP which we defined via the ZFS storage console before:
 

-> load -source ftp://patcher:mypassword@<storage VIP address>//export/common/patches/todo/
13795376/Infrastructure/2.0.0.0.0/ZFS_Storage_7320/ILOM/3.0.16.10_r65138/
ILOM-3_0_16_10_r65138-Sun_Fire_X4170M2_X4270M2.pkg
NOTE: An upgrade takes several minutes to complete. ILOM
will enter a special mode to load new firmware. No
other tasks can be performed in ILOM until the
firmware upgrade is complete and ILOM is reset.
You can choose to postpone the server BIOS upgrade until the
next server poweroff. If you do not do that, you should
perform a clean shutdown of the server before continuing.
Are you sure you want to load the specified file (y/n)? y
Preserve existing configuration (y/n)? y
Delay BIOS upgrade until next server poweroff (y/n)? n
..........................................................................................
...............
Firmware update is complete.
ILOM will now be restarted with the new firmware.
-> /sbin/reboot
Connection to xxxxsn2-c closed by remote host.
Connection to xxxxsn2-c closed.

OK that was the first reboot after upgrading… wait till it comes back online and login to the ILOM again:

12. Run the following command to verify the new ILOM version: -> version

Actually this is not needed as it already reports the version on login :

% ssh root@xxxxsn2-c
Password:
Oracle(R) Integrated Lights Out Manager
Version 3.0.16.10 r65138
Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
-> version
SP firmware 3.0.16.10
SP firmware build number: 65138
SP firmware date: Wed Jun 15 08:02:17 EDT 2011
SP filesystem version: 0.1.23

Now it says we need to set the boot option to “ROM Enable” in the BIOS, I will not show this here. After doing so, it reboots once more, and we need to log back in once more to check the final status:

23. Run the following commands to check if the current head is Ready (waiting for failback), (that is, description = Ready (waiting for failback) ):

xxxxsn2 console login: root
Password:
Last login: Tue Mar 20 13:05:42 on console
xxxxsn2:> cd /
xxxxsn2:> configuration cluster
xxxxsn2:configuration cluster> show
Properties:
state = AKCS_STRIPPED
 description = Ready (waiting for failback)
peer_asn = a54b53a0-afba-eae1-a77a-b0013813b629
peer_hostname = xxxxsn1
peer_state = AKCS_OWNER
peer_description = Active (takeover completed)
Children:
resources => Configure resources

OK, we have thus upgraded the ILOM software of storage head2 to version 3.0.16.10 r65138. We are now ready for step 3, upgrading the ZFS Storage 7320 Software to Version 2011.1.1.0.

1.5     Next time

In the next post, we continue where we left off to finish the upgrade of the ZFS storage by doing step 3 of the upgrade, then a switchover the active storage role to the newly upgraded head2, then upgrade head1 and finally check if eveything went OK. Of course in real life this should all be done without pause in a single excercise!

OVER DE AUTEUR

Jos Nijhoff is an experienced Application Infrastructure consultant at Qualogy. Currently he plays a key role as technical presales for Qualogy's exclusive Exalogic partnership with Oracle for the Benelux area. Thus he keeps in close contact with Oracle presales and partner services, but maintains an independent view. He gives technical guidance and designs, reviews, manages and updates the application infrastructure before, during and after the rollout of new and existing Oracle (Fusion) Applications & Fusion Middleware implementations. Jos is also familiar with subjects like high availability, disaster recovery scenarios, virtualization, performance analysis, data security, and identity management integration with respect to Oracle applications.

Geef een reactie

Het e-mailadres wordt niet gepubliceerd. Verplichte velden zijn gemarkeerd met *

De volgende HTML tags en attributen zijn toegestaan: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Contact

Hebt u vragen of suggesties?

Mail info@qualogy.com!


De Bruyn Kopsstraat 9

2288EC Rijswijk (ZH)

The Netherlands

+31.(0)70 319 5000

  • Blog

  • Tags

  • @qualogy_news

  • @qresources

  • Reacties

  • Blijf in contact

    +31.(0)70 319 5000