When Virtualization will mean "Management"

It's been awhile since I last blogged.  Have been busy at a new job, where I'm handling the marketing launch of our new integrated storage management product.  Surprisingly, though, it was my team leader who insisted that blogging needs to be something where you, "see an article and you write about it then".  So I'm following that advice.

I just read a CNET story, "Enterprise storage gets interesting again", which quoted a VM storage company exec as saying, "100 percent virtual deployment is a when, not an if, at this point. In the virtualized datacenters 20 years from today, all aspects of computing will be virtualized, including servers, networks and storage."  It's tough to argue this.  Analysts and consultants are putting server virtualization adoption at anywhere from 20% to 80% depending on the research and whether they were measuring systems, workloads or companies. The trend towards "virtualization as the default" is something we continue to hear from customers. 

The downside, however, is the complexity.  According to one survey, "...only about half of around 200 respondents find virtualized systems easier to manage or help them meet their SLAs."   Virtualization has become another complex layer of 3rd party management added on top of the physical IT resources in the data center.  It has spawned its own type of administrators (i.e. "VM Admins") as well as management frameworks.  And, if you really love complexity, you can even add on an additional "Cloud" layer of management (director software, portal, API, licenses, etc.), to further abstract, your abstracted management! 

Before I'm accused of being some Ludite, let me say that server virtualization has been a good trend. Similar to storage virtualization, network virtualization... heck, memory virtualization. I mean, when was the last time you had to specify which module or page of RAM to use for your application?   Virtualization shields the end-user from the details -- and dare I say complexity -- of the underlying physical resource.  And as a result, typically scalability, utilization and availability are improved.

So I agree that virtualization "is a when, not an if".  But the "when" will remain a long way off until the complexity is addressed. What is needed is for server virtualization to become part of the server management, not yet another thing to install, license and manage.  Just like how Storage arrays present volumes externally, while masking the internal complexity of cache memory, RAID levels, tiering, etc., servers will have to quickly get onto this same evolutionary path.  The "when" of prevalent server virtualization will arrive when the virtualization is embedded within the server hardware itself.

Cloud on the Top of the Hype Curve

I know there's a lot of talk out there about the Cloud and how hyped it is.  It's true.  If you look at Gartner's latest Hype cycle on Emerging technologies, you can see where they place it: literally at the top of the cycle. 

But that's OK - A lot of hype means a lot of awareness. As a vendor this adds wind in our sails and as a customer it makes it easy to find the content that will help make informed decisions.  In fact, last I checked, the term 'Cloud Computing' was yielding over 34 million hits on Google.  Sounds like lots of information for folks who want to learn more.

Another popular topic about Cloud is how long will it take to become a significant movement.  Well, if you believe the leading, trusted analysts in the space like Frank Gens from IDC (I know I do), he already puts the Cloud market -- and let's be specific -- this is the market for External Cloud services (not 'private cloud' spending) at $15.6 billion as of 2009.  That's a big Total Addressable Market.  Far bigger than many segments of IT as we know it today.   And the growth rate is expected to be 27% annually between now and 2014, with some areas such as Cloud Storage racing ahead at a 37% compound annual growth rate. 

Let's look at just Cloud Storage.  According to those same IDC numbers, it's already 9% of the market, and at the current growth rate, that would make it about $1.8bil this year.  That's almost as large as the entire SAN storage networking market, and growing far faster.  Projecting forward, by 2014 Cloud Storage spending will exceed $7 bil.  And by that time external Cloud spending will account for 10% of the IT budget. 

So some are saying Cloud is over-hyped.  Some, that it's under-hyped.  I'd say it's about right for where it is in its maturity.  The perception for some is that Cloud is just for deep file archiving, Test/Dev and limited web-based development.  In reality, early adoptors are already running 60-70% of their business apps in the cloud.  We're seeing the Government laying the groundwork for serious investments in Cloud-based services through programs such as FedRAMP.  And enterprises are swarming shows like VMworld to understand the art-of-the possible, and continue to build upon their internal 'cloud' efforts to be able to extend them towards carrier and service provider clouds.

So long as Cloud vendors continue to deliver capex and opex savings, improved efficiencies, decent quality and security, and fast time-to-solution, then the growth will continue, with or without the hype.

Newest Cloud Storage Enabler

I was fielding an inquiry from one of my engineers and thought I'd share it, as it's probably a common question, esp. as we continue to see Cloud awareness spread, while the network barriers to external Cloud service adoption remain. Enjoy.

-----Original Message-----

Sent: Monday, September 20, 2010 5:30 PM
Subject: Cirtas

Hi Mike,
I'm interested in Cirtas, because a user asked me. What do you think of this kind of storage?

-----Original Message-----

From: Mike Harding
Sent: Tuesday, September 21, 2010 9:36 AM
Subject: RE: Cirtas


This is a very valuable solution. Much of 'Cloud storage' is not useful for companies for at least a few key reasons:

1) Security -- It's the #1 inhibitor to using external cloud services. Customers are worried that their data will be compromised either in transit outside of the corporate firewall, or after it's been stored, especially within a multi-tenant hosting environment.

2) Performance -- The second biggest problem with hosting your data in the cloud is that it's far away from the applications and users. So the distance between you and your data creates latency as well as other common WAN issues such as jitter, lost packets, etc. This is why we've seen much of cloud storage being for deep archiving or uses where you don't care how long it takes to either put or retrieve your data, such as email archiving for regulatory reasons.

3) The need to change your application -- Many providers, even those using brand name enterprise-class storage hardware such as EMC Atmos, are only allowing access via a RESTful API. This means that the customer needs to write an application that uses this API in order to store and access the data, and for all intents and purposes, limits the use of that cloud storage for web application media and data.

Cirtas, which just launched publically, is a great example of what I call a Cloud Storage Enabler in that they allow customers to overcome these barriers to adopting the external cloud. Their product, Bluejet, encrypts your data so it's secure both in transit and at-rest. It accelerates the transit with data compression and deduplication. And it emulates local storage, so it looks like any NFS/CIFS target to your applications and users. Cirtas is one of a number of companies with similar Cloud Enabler solutions - you should also consider Nasuni, TwinStrata, Panzura and StorSimple.

Thanks, and good luck,

Mike Harding

Cloud Storage Optimization market

A new product category is shaping up in direct response to a new customer need.  As seen in recent stories like "What's keeping Data Storage Out of the Cloud?", companies want to use new Cloud Storage services from providers such as IBM, but they are concerned about the security, availability and cost of the required network connection.  Enter Cloud Storage Optimization.

To bridge this network gap customers face a number of sub-optimal alternatives:

1. Don't worry about it -- probably the most popular approach, which only works if you don't care about your job, or the value of the data going across the wire is so low that it's not a big deal if it takes forever for the transit, or it's hacked, or both.
2. Lease a private connection -- this is an option for 'too big to fail banks' or other major organizations where cost isn't an issue.  But for most companies, the incremental cost of the circuit eliminates the economic savings of the Cloud service.
3. Use a generic WAN optimization box -- Not a great solution as these are software-based appliances designed for lower-bandwidth branch-office connections and a broad mix of transactional data.  The Cloud Storage connection is really a SAN-like 'channel' which will be very data intensive, and will benefit from hardware-based compression and offload processing.  And similar to the private circuit, the cost of the WAN optimization appliance that supports the higher throughput needed for the Cloud storage will cost you more than your annual Cloud storage bill.

What we're seeing in response is an initial first step towards closing this market gap: a new category of Cloud Storage Optimization solutions, or Cloud Storage Gateways.  Representative companies include Cirtas, Twinstrata, Nasuni and StorSimple.  These are all start-ups who seem to be quickly gaining awareness and traction with companies and Cloud Service Providers.

The anatomy of a Cloud Storage Gateway is made of software that either resides within an x86 server (i.e. an appliance) or completely as software that can be deployed within a VM.  They typically are asymmetric (i.e. single-device) solutions often positioned as a NAS filer.  Typical capabilities include NAS-to-Cloud API emulation, WAN Optimization, Caching, In-transit Encryption and Data management features such as snapshots.  As software-based solutions they are flexible, and meant to be affordable and targeted to a more mid-market customer.  Similarly they are intended for not-overly-demanding throughput needs, as there is no purpose-built processor offload. 

For mid-market companies looking to add a Cloud tier of archival or similar offline data storage, these are products to consider.  For enterprises or companies who want to leverage Cloud storage as a nearstore alternative, you will want to wait for next-gen 'Cloud Networking' products built for high-throughput, hardware-assisted optimization, symmetric caching/network de-dupe capabilities, and that integrate with your existing network management framework.

How to Calculate an Accurate Cost of Cloud vs. Cost of Inertia

There was a very helpful article written recently on the 'hidden' costs of using external Cloud storage services.  This is important information when you are sitting down to determine the business case for moving data outside your data center.  The article correctly points out the basic cost for storage from someone like Amazon Web Services can be as little as $0.15 per GB per month, and that volume discounts can bring this down further.  However, additional features to support WORM or information lifecycle management, will increase the pricetag towards $1.00 per GB.

Some services charge to upload data, some to download, some for both.  And if you have too much to send over the wire, you are welcome to send a tape.  But that will cost you, too.

And then there's connectivity costs.  If you are already max-ed out on your internet connection, then moving data to the cloud will require incremental bandwidth.  Some enterprise-class services actually require bundled bandwidth or even direct circuits, in order to provide an SLA.

So if all this sounds daunting, let's look at your current cost of storage.  The rule of thumb is that for any IT capability, the direct cost (i.e. what you buy and deploy) is only 20% of the total cost: the rest is the cost to maintain it.  This indirect cost % appears to be increasing over time -- both a function of improved value in IT products, but also an increase in wages, facilities, energy, etc.  And this is also true for storage.

TCO components of your storage base case need to include the cost of the hardware -- either the entire cost if looking at a 3-year period, or an annual depreciation.  I'd assume you're looking mostly at Arrays, but you may also have servers involved for some supporting application.  Make sure you're comparing apples to apples with the complete outsourced offering.  If this is a decision to buy another NAS filer vs. a contract with someone like Rackspace, then you need to factor-in the complete deployed cost.  Include any installation, training, and related on-site switching expense like for instance local data migration.

Then add system software and/or array-based software licenses.  Add annual maintenance and support fees.  Add the allocated Server Administrator cost for the devices: the annual burdened wages (i.e. salary + 15% or more for taxes, benefits).  This should be a big line item: staff costs are consistently 40% or more of total costs in the data center.  Each admin can manage just so many TBs of storage.  You need to figure this out for your current environment; even if an analyst or vendor study says that one admin FTE can handle 10TB in an ideal world, what really matters is what you are running in your data center today.

We're not done: you can't forget the allocated cost of the data center space.  This includes the rent of the space, and the power, the heating and cooling, and if you need to pay for a set of hands within the facility when changes are made.   I wish I could offer a rule of thumb here, but it depends on whether you have your own data center and what 'tier' of DC it is, or whether you are using a co-lo.  There are variables in terms of wattage density per rack, efficiency of the cooling system, and probably other key variables.  For many companies their problem is that their space is either obsolete or they are out of power, or space, or both.  Anyway, this may take a little work to get to, but you need to add a cost to reflect the data center operations.

Now look at the numbers.  Assuming you are one of the lucky ones who still have the ability to add more storage internally, the business case for using the Cloud should be more compelling, especially where you are talking about a smaller amount of storage, have a smaller (i.e. less scale-efficient) operation, and especially if your needs are more temporary or at least not expected to be consistent over the life of the hardware (e.g. you don't need to use all the storage for all that period of time).

IT is definitely moving thier resources to the cloud, and it's all about the economics.  Sharpen your pencil and make sure you're taking an accurate picture of your internal vs. your expected Cloud storage costs.  Good luck!