The procedure for upgrading EMC XtremIO storage arrays to their latest major code release (3.0) has caused no shortage of conversation among the enterprise storage community. Granted, a large portion of that derives from competitors and marketing material which are keen to take advantage of this hurdle in the XtremIO track.
For those unfamiliar, the hurdle is the disruptive and destructive nature of the 3.0 upgrade process. To move from 2.4 to 3.0, customers must move all data from the brick(s) to another storage platform. EMC promises to provide the loaner gear to swing said data for the upgrade, but that doesn’t alleviate the infrastructure and migration impact of such a task (especially if some things are physical and without niceties like Storage vMotion).
We’ve had our share of challenges getting to this point, as you can read from prior posts, but we’re finally here. Since others are following closely behind, I thought it would be helpful to document the steps necessary to complete this upgrade (where possible, I’ll include the actual upgrade-to-3.0 tech details, but those are mostly handled by EMC).
1. Create an EMC Support Request (SR) requesting upgrade
This step is sort of a misnomer, because the XtremIO 3.0 upgrade isn’t handled by the EMC Support team. Due to the nature of swinging data, loaner hardware, etc, EMC Sales and Professional Support actually handles the process as a “free upgrade”.
2. Contact your EMC Account Team with the SR# and any environment needs (loaner storage, time frame, etc)
In my experience, the EMC Support team tried to handle this part for me (I already had loaner hardware on hand and just needed the upgrade itself), but they were unable to get through to my team, so customer contact broke the stalemate (after 12 days of Support-to-Sales emails).
3. Schedule the upgrade and perform any prerequisites (information gathering, etc)
The upgrade team typically wants 10 days lead time to schedule, and your environment may need even more, depending on maintenance windows and migration efforts.
4. Migrate data off of the XtremIO bricks to be upgraded — this means everything!
If you’re a VMware (or Hyper-V) shop, storage vMotion should reduce this pain point, especially if EMC loans you an equivalent XtremIO loaner brick. Speaking from experience, svMotion between XtremIO arrays is lightning fast.
5. Upgrade Day: Download the upgrade code and XMS
This step and those that follow may depend on your engagement. Initially, EMC wanted to schedule on-site field personnel, but since we had already been through an upgrade on a test array, I requested a remote session. If it had happened on-site, I believe the field engineer would have completed this step and possibly others.
The EMC engineer assigned to the upgrade should email an FTP link with the 3.x (in my case, 3.0.1-11) upgrade code, as well as a new XMS OVA. Download these (800MB each, roughly) to the place where you’ll host the remote session.
6. Find a spare terabyte for the new XMS
This one caught me by surprise. The new XMS OVA includes a 850GB+ thin provisioned disk. While that is indeed “thin”, the sheer possibility of it growing to 850GB caused me to create an extra datastore on my other storage to allow for the XMS to grow.
7. Deploy the new XMS
Typical OVA/OVF deployment in vSphere Client. Pick a new IP address and update DNS if you are maintaining the same XMS hostname (we had issues with reusing the same IP, even after clearing arp, etc).
8. Configure the XMS
User: xinstall, Option 4 to configure the XMS
9. Upload the upgrade code to the XMS
File: upgrade-to-3.0.1-11.tar, Target location: /images
10. Upgrade the cluster
Option 6, use filename from above, specify number of bricks, watch it roll!
11. Recreate and start the cluster
Reference fresh deployment documentation.
create-cluster cluster-name=<name> expected-number-of-bricks=<1,2,4,6> sc-mgr-ip=<ip_address>
The SE enabled encryption on our array, but this wasn’t necessary in our case, so he then disabled it. Switching modes is a cluster-stopped change, so make sure this is in the desired state before proceeding and starting the cluster. “self” enabled encryption. “disabled”, well, disables it.
modify-clusters-configuration cluster-id=1 encryption-command=switch-mode encryption-mode=<self | disabled>
11b. Start the cluster
12. Set NTP server(s) and timezone
modify-datetime ntp-servers=["<ntp_server_1>","<ntp_server_2>"] timezone=<local_tz>
Use the ‘show-timezones’ command to list them.
13. Set XMS hostname (FQDN)
14. EMC SE creates a business services case to move the cluster to the installed state
This is definitely not a customer activity as EMC has to login to their business services portal. The case they create changes to state of the cluster in the EMC Install Base (IB) database. Since ours has been in production for 9 months, I’m not quite sure what it moved “from”, but it’s installed now.
15. Reconfigure ESRS monitoring
modify-syr-notifier enable site-name="<site_name>" connection-type=esrsgw esrs-gw-host="ip_or_hostname"
The SE (who was definitely used to working with EMC PS/FE) was unclear on some of this, and we were only able to set one IP for ESRS. If someone else knows how to configure an ESRS cluster in XtremIO, please comment.
16. Edit ESRS Configuration Tool to set new XMS IP address
Since the IP address of the new XMS is different, ESRS needs to be changed accordingly. Edit this on your gateway server(s).
The upgrade required approximately 4 hours of time to complete, but probably 25-50% of this was due to the remote engineer’s unfamiliarity with a few tasks/details as well as a dinner break on his end (understandable). I’ll be remapping and provisioning shortly, but it looks good at this point.