It is rare that a new technology transforms my datacenter. Virtualization, VDI, and 10Gb converged networking
all worked to simplify my infrastructure, slash costs, and reduce downtime over
the past decade. The most recent transformation came in the form of all-flash
storage. My hope with this posting is
that I can reveal the process that my team and I went through to arrive at
storage nirvana, better known as the Pure Storage FlashArray while detailing
how the arrays have performed in my environment over the last 2 years.
I have spent years forklifting out old spinning disk only to
replace it with a new array with few (if any) performance gains. My team and I faced the daily frustration
that comes with constantly tuning storage. Trying to deliver the performance required
only to have a new application come in and kill performance starting the
process all over again. The exponential growth of virtualization was placing a
high reliance on our centralized storage infrastructure and it just couldn't
keep up.
At the start of 2011 several arrays from NetApp and EMC were
failing under the load from Databases (SQL and Oracle), Citrix VDI, Exchange, and
other virtual machines. For several
hours every evening, data audits or loads within our entire environment would bring
all our virtualized systems to a grinding halt as storage latency averaged over
300ms with peaks over 1000ms. Nearly a year was wasted battling the disk
performance Phantom Menace with the vendors, yet they still could not isolate
the root of our performance pains.
I considered adding technologies like FlashCache to our
controllers, purpose built Flash accelerator cards to our servers, or more
disks and bigger controllers to our exiting arrays. However none of these
solutions offered the IO performance required without locking us into
inflexible server based solutions or costing more than the business could
afford. I started the search for a new storage
solution that could deliver the performance and flexibility that I wanted. Hybrid arrays didn't deliver the performance
needed for our high IO write-intensive environment so I turned to all-flash
arrays as my savior.
In February 2012 after several false starts with various
all-flash vendors, I ran across a Tech Field Day presentation on YouTube. A startup
named Pure Storage was presenting the design for their all-flash array with
deep technical details. What I was
seeing looked like an easy to manage array that could dedup and compress my
data inline while providing 200K IOPS. Other vendors had strongly recommended
against turning on their dedup due to performance impacts, or their arrays performed
these tasks post-process. My team and I were energized by the prospects and decided
to put their claims to the test and within a couple weeks of reaching out to
Pure Storage I had one of their beta units running in my datacenter.
My first impressions after receiving the Flash Array were
beyond my expectations. After spending years digging through the layers and
layers of settings on traditional disk based arrays, the simple interface and
lack of random settings to configure on the array was a significant
change. Initial setup and provisioning
of storage from both the GUI and the command line was easy to learn and within
a few minutes I was up and running. It
only took a few clicks to provision LUNs to hosts. No wasted time configuring RAID, Aggregates,
Volumes, etc. or hunting for what advanced options to set for my configuration.
The simplified experience opened my eyes to how storage should be built and
managed.
I presented LUNs to a number of test systems running
Windows, Linux, and VMware. I proceeded
to throw every test I could think of against the array using SQL Server, VM
storage vMotions, Cloning VMs, and other testing scripts or tools that I could
dig up. The array performed exactly as
we were led to believe it would, if anything in most tests it exceeded our
expectations. IOPS scaled predictably with the size of IO that I was running
against the array and it could easily handle the sustained random writes from
my test applications. Dedup with over 100 test VMs cloned from our Staging
environment ranged from 4:1 to upwards of 11:1 depending on the type of data. At one point during the evaluation I ran into
issues with our Brocade Fiber switches where some paths were dropping. I was
pleasantly surprised when Pure Storage support jumped in to help troubleshoot
and resolve the issue, even though the switches were covered under NetApp
support (which wasn't being very helpful up to that point). After several
months of testing my team and I were having a hard time resisting the urge to
move production data on the array so we moved forward with purchasing our first
production FlashArray.
In August 2012 we installed our FA320 11TB array running
early GA (2.0.x) code. Within a few days
I had moved our entire Dev, QA, and Staging environments to the array and started
performance and reliability testing. I
generated as much IO load as I could then proceeded to pull power on
controllers, pull SSD’s, pull NVRAM modules, disconnect fiber, and disconnect InfiniBand
cables all in an attempt to find any weakness in the reliability of the
array. Again as with the Beta array
everything worked flawlessly. It was time to start moving production data onto
the array so that we could see if it would finally end the Phantom Menace that
was growing worse by the day.
The first project was to virtualize two small SQL
Server instances that were causing some of the problems on our old disk
storage. The very first day after cutting over these two SQL Servers running on
Pure Storage, our audit/load times on these servers plummeted from >4 hours
to just over 1 hour. Beyond the time
reduction our greatest success was the total lack of impact to other VMs on the
FlashArray during this window. Latency remained at ~1ms on the virtual machines
and <1ms on the array, no longer did we have to deal with 300+ms latency
across all our systems. With the
confidence that the array could handle our entire load without any issues I worked
over the next week to migrate all production VMs (Web, App, Infrastructure, and
Citrix) onto the array, and within 3 months my team and I had decommissioned an
old EMC Clarion used for our Oracle hosts, and all VMware storage was off our
Netapp Filer. Throughout the migration
as we added more data and increased the IO load on the array it continued to
perform flawlessly, with latency for the most part below the 1ms mark and dedup
ratios above 6:1.
Over time the array was upgraded every couple months as new
features or bug-fixes came out. During
one code upgrade that should have been non-disruptive, a Linux server running
Oracle hung due to all paths going down, again Pure Storage support stepped up
and was extremely helpful at reviewing all aspects of the Linux host storage
configuration and assisting with troubleshooting. In the end it turned out to
be a bug in the MPIO driver on the Linux host.
In 2013 we were ready to add a second Pure Storage array for
eventual use in our DR site, and as part of the purchase we also had the
opportunity to upgrade our controllers in our production array from the FA320
controllers to the FA420 controllers. In
my past experiences with controller upgrades they would lead to a major outage,
that is of course if the storage vendor didn't just force you buy a whole new
array to deliver a little more performance.
Our Pure Storage sales engineer (James) came on site and
installed the new DR array for us first and had it up and running in short
order. Since I wanted to see just how
much load two arrays could handle with them sitting side-by-side I quickly
added storage from the new array onto my test ESXi cluster while James was
unpacking the new heads for the primary array. I setup some tasks to clone and
migrate VMs back and forth between the two arrays to generate a ton of IO load
(and to see how fast I could move data between them). The timeframe for this upgrade was mid-week
with a fairly short window where the array would be moderately quiet before our
nightly database tasks started up. If the production array went down at any
time day or night it would bring down all my production systems, so running
mid-week wasn't any bigger risk. From
all our previous upgrades and fault testing I had a very high level of confidence
that the head upgrade would go off without a hitch, and true to form the head
upgrades were completed without any production impact, even with me throwing >2GBps
and >100K IOPS against the array throughout the entire upgrade process.
As with any new product there have been minor issues with
the array on occasion. However support has always been very responsive and
quick to find a resolution. They have
reached out on several occasions to notify me of proactive fixes to issues that
they are starting to see on our array allowing us to keep our production systems
running smoothly. This has resulted in
~35 minutes of total downtime the first 6 months running on the array (the move
to non-disruptive upgrades took a few revisions in the early 2.x code). In the past 18 months I have had zero
production outages as a result of an issue with the FlashArray and zero storage
related performance issues.
After more than 2 years since I first installed Pure Storage
in our datacenter we have gone through countless code updates on our arrays,
and we have even changed controller generations on our primary array. We have never lost a SSD drive the entire
time we have been running FlashArray's.
Any time we have any issues with the array or systems connected to the array,
support has been very responsive and a pleasure to work with. For the first time since I have been working
with storage arrays from various vendors I don’t have to worry about how my
storage is performing, if the next upgrade is going to take my datacenter down,
what will happen if I lose a drive or controller, or how much my next
maintenance bill will go up. My team and
I get pretty excited when talking about Pure Storage and for good reason; it
has saved countless hours managing storage in our datacenter, eliminated
storage as a performance bottleneck, lowered power requirements, reduced rack
space, and cut costs. Mostly because Pure Storage is a great company and they
are successfully changing the storage industry for the better.
