I was having coffee with a friend and we talked about storage. His question was what I hear all the time: “Why in the &$@#() is it so expensive?” (Actually he did not say “&$@#()”. I added that for extra color.) But the logic goes like this, I can buy a 1TB drive for $100 at Best Buy. So what is the big deal? Why do I have to pay $1000,s per TB? By the end of this post you will know why.
The answer is performance. Yes, the modern disk drives are a bargain and a marvel of electro-mechanical technology. They are also hampered by old algorithms and software controls that organize the data and keep track (no pun) of all that data. Some of this has not changed since the days of PDP11’s and disk drives the size of small refrigerators. There are tables and indexes and pointers galore. There is also the issue that it is a mechanical device. Heads and platters have to move to find the data. Add to this the limits of physics. There is a speed at which the disk flies apart or the heads flex. (At this point I marvel that it works at all.)
Now we have a measure of interest which is the IOPS – the Input/output Operations Per Second. This is the basic measure of performance for a drive. It is the average number of times per second you can write a block of data to a disk or read a block of data from the drive. Now the question is: how many of these IOPS do we need and how many can we get? Let’s take the second question first. On average, a disk will have to spin 70 times to get one IOP! Remember the old algorithms? If the drive spins at 7500 rpm then we get roughly 100 IOPS per drive. Let’s suppose we want to virtualize desktops. On average, (and this varies greatly) say we need 20 IOPS per desktop. For 1000 desktops we would need 20,000 IOPS. If my storage system is built on an architecture which is dependent on 
spindles for performance, then my system will need at least 200 disk drives, not counting redundancy, backup and archiving! These are averages and can vary up and down, but the essence is the same. It answers the question so many IT professionals ask, “Why do we have so many spindles and why are they so small?”
So, what are the four questions:
- How much actual usable storage do we need?
- What level of IOPS do we need in the busy hour?
- What is the maximum throughput we need in each direction?
- What latency (in ms) can we stand?
Answer these questions for today and the end of the planning cycle before you talk to storage vendors.
ShareFEB
About the Author:
While I was at Stanford, I got to know Dr. William Schockley, the inventor of the transistor. While at Bell Labs I produced the first computer generated animation, I worked on the first packet switch and the first picture-phone. I worked with the inventors of Unix and C. I produced multiple compilers, operating systems and microcode. I worked with the Snobal language - the precursor to Perl and Ramis the precursor to SQL. The building I worked in was home for 5000 of the smartest people in the industry. We had Phd's cutting the grass. (just kidding). I never had it so good again. Today that building is empty and for sale. After the Bell System, I spent 25 years in the satellite industry. I led development teams in the VSAT industry. I started and grew early stage companies in this industry. Recently I have been involved in cloud computing startups focused on infrastructure and virtual desktops.