There is a lot going on inside your Storinator. CPUs are a cyclin', fans are turnin', data is movin', and most importantly, 45 drives are spinnin'! A steady and reliable power source is essential to make all of this happen. In our quest to create a rock-solid industrial strength server, we've spent a lot of time engineering the power system on our Storinator servers. I'm writing this post to share some of our knowledge with you.
Power as a 'Boogeyman'
I'm hoping that this blog post will help demystify the issue of power. Because a lot of system admins don't have access to gear such as clamp meters or oscilloscopes, power can be blamed for a lot of instabilities. Since power is not being measured, however, this is really a 'boogeyman' type explanation. We've spent many, many hours measuring power and looking for those proverbial glitches that are often blamed for system crashes, but we've never been able to see one. Never say never, but when troubleshooting, we've come to believe that with the robust, high-end power supplies that we use, power problems are exceedingly rare.
Specifically, what I want to share with you is typical 'current' vs 'time' plots of the Storinator during start up, steady state and operating (read/write) current draws, and to discuss how our PSU is well suited for keeping your pod up and running, while having extra juice for startups...
Power as a 'Boogeyman'
I'm hoping that this blog post will help demystify the issue of power. Because a lot of system admins don't have access to gear such as clamp meters or oscilloscopes, power can be blamed for a lot of instabilities. Since power is not being measured, however, this is really a 'boogeyman' type explanation. We've spent many, many hours measuring power and looking for those proverbial glitches that are often blamed for system crashes, but we've never been able to see one. Never say never, but when troubleshooting, we've come to believe that with the robust, high-end power supplies that we use, power problems are exceedingly rare.
Specifically, what I want to share with you is typical 'current' vs 'time' plots of the Storinator during start up, steady state and operating (read/write) current draws, and to discuss how our PSU is well suited for keeping your pod up and running, while having extra juice for startups...
Powering Hard Drives: A Primer
There are two different voltages that are fed to a hard drive, 5V and 12V. The 5V line powers all of the logic circuits in the hard drive, as well as the actuator. The actuator is an electronic device that controls the servomotor moving the hard drive head arm. The 12V line powers the motors that spin the platters. Due to the physics of a DC motor, spinning these platters up from a dead stop pulls a significant amount of current for a short period of time.
 |
| A basic look at the inside of a HDD. Of the pieces labeled in the image above, the 12V powers the motors that allow the plotters to spin around the spindle, and the 5V line handles everything else. |
When a hard drive spins up from a dead stop, it pulls a significant amount of current compared to its normal operating levels. This is due to the current needed to start the DC motors that spin the platter(s) in each drive. The DC inrush current is equal to the DC motor resistance divided into the applied voltage (12V in our case). Once the motor is spinning, it generates a back emf (electromotive force) which reduces the current draw and the hard drives operate in steady state mode.
In most hard drives, this inrush current occurs only for a short time, ~2 to 4 seconds depending on the drives (the ramp up time, and therefore maximum current draw, is set by the hard drive manufacturer and is generally configurable in firmware, but not by the user). It then becomes obvious that turning on a machine with a large amount of hard drives, and spinning them up all at once, will pull a lot of current, albeit for a very short time. If the PSU is not sized to withstand these currents, you may have problems turning on the machine. However, if one does successfully meet startup demands, power draw in steady operation will not stress the machine.
The Magic of Staggered Spinup
An obvious solution to help reduce the large 12V spin-up surge is to use staggered spin-up (i.e. to start drives one at a time with a short delay between each). We will cover this in detail in our next post.
The Magic of Staggered Spinup
An obvious solution to help reduce the large 12V spin-up surge is to use staggered spin-up (i.e. to start drives one at a time with a short delay between each). We will cover this in detail in our next post.
The Setup
To give you a look inside our machines, and what happens when you press the power button, I stripped down a Storinator to get to the power lines that run to the HDDs.
 |
| My clamp meter placed around the 12V lines that run to the HDDs |
I used a clamp current meter (MTP iC-410 AC/DC) and measured first the 12V lines and then the 5V lines that ran to the HDDs. The output of the current meter was fed through a signal conditioning circuit and then observed on an oscilloscope. To generate the plots in this post, a python script was used to pull the data off the scope and saved for analysis.
45 Seagate 4TB (Model# ST4000DM000) drives were used in a standard Storinator S45.
Our PSU is a Zippy 950W N+1 redundant supply with active-active components. Each PSU is made up of three equal modules of 475W (500W MAX), where two modules are always active powering the unit, while the third is used as back-up, should one of the other two fail.
Although available as a backup, the third module can be used to supply an extra 475W of power if the other modules are maxed out, effectively raising the DC output rating by a factor 1.5. This is what is called an active-active PSU.
The PSU is rated to supply 60A on the 5V and 64A on the 12V, and if the third modules is needed the PSU can supply up to 90A and 96A to the 5V and 12V, respectively.
This dynamic range of the PSU allows for compatibility with a wide range of hard drives, from low power desktop drives to industrious power-sucking enterprise drives. (Manufacturer specs can be found here).
Now that we have the setup and PSU specs out of the way, it's time for the fun stuff: current plots!
45 Drives Start Up
Here is the plot of the HDD current draw from the moment I press the power button, to moments before the login screen appears. The data has been processed to display both the 5V line (Red) and the 12V line (black) superimposed on the same plot.
 |
| Start up current draw of 45 HDDs from the moment the power button is pressed to moments before the login screen appears. The black trace corresponds to the current draw on the 12V line and the red trace is current draw on the 5V line |
On the 12V line, you can see the inrush current I mentioned earlier, where it peaks at 51A. There are two more quick pulses of current before the drives achieve steady state and the current draw evens out to 14A. Now in terms of the 12V line, all the hard work is pretty much over, until the machine is powered off these disks will not stop spinning and the 12V line will sit at this steady state. Boring :(
(Note: Not all drives stay in the powered-up state (an example is WD Green drives). We generally don’t recommend drives that power down and startup in our Storinators, however they may be used with caution in specialized applications to solve certain problems.)
(Note: Not all drives stay in the powered-up state (an example is WD Green drives). We generally don’t recommend drives that power down and startup in our Storinators, however they may be used with caution in specialized applications to solve certain problems.)
Now this is when the 5V line gets interesting. As the 5V line powers the logic circuits in the HDD, you can see it does not suffer from the physics of a DC motor, and therefore does not see a large inrush current. It quickly hits its somewhat steady state for start-up and you can see a little activity on line as it boots. Where things get interesting is after start-up, when you are reading and writing to the disks.
 |
| The trace is of the current draw of the 5V line under various conditions. 0-25s is the creation of a filesystem, 25-80s is a write to the RAID, and from 82-90s is the current draw falling back to idle conditions. |
The plot above is of the current draw on the 5V line over a period of 90 seconds. From 0 seconds to 25 is me creating a XFS filesystem on top of 45 drive RAID0. At 25 seconds to 80 seconds is me writing a file to the RAID, and then from 82 seconds to 90 seconds is the drives returning to idle state. It’s interesting to note that the peak current draw during the write is nearly twice as much as idle. Not pictured is the 12V line during this transfer, it stayed constant at 14A the whole time.
In our experience, most consumer drives will behave exactly like the plots seen above. A large current is drawn on the 12V line during startup with levels approximately 3x times steady state for a very short time, no more than 4 seconds. A lesser current is drawn from the 5V during reads/writes with levels peaking at 2x times steady state for the duration of the transfer.
Extending These Results to 30- and 60-Drive Models
This post talked about the power draw in terms of 45 drives in the pod, and you may be wondering about our 30 and 60 models as well. We’ve seen in the lab that the power draw of the HDDs scales linearly as you add more drives.
So, to get an idea of what the current draw for the Storinator XL60 is, multiply all the results in this post by 1.33 and for the Storinator Q30, multiply all results by 0.66. Regardless of the current levels, the plots take the same shape for all three models.
Summary
As you can see, the peak current draw on the 12V line is 51A, below the 64A limit when the third module will kick in. Once all the platters are spinning, it quickly falls to 14A.
So, to get an idea of what the current draw for the Storinator XL60 is, multiply all the results in this post by 1.33 and for the Storinator Q30, multiply all results by 0.66. Regardless of the current levels, the plots take the same shape for all three models.
Summary
As you can see, the peak current draw on the 12V line is 51A, below the 64A limit when the third module will kick in. Once all the platters are spinning, it quickly falls to 14A.
On the 5V line, startup current draw hovers around 19A then gradually falls to steady state level of 10A. When preforming read/write operations, however, current draw peaks at 20A, twice that of steady state! This current draw is well below the 60A limit of the PSU though, leaving plenty of room for all the other components.
Conclusion
Conclusion
- A large storage server must be engineered to address the large power surge that occurs on the 12V side during drive spin-up. We have put a large amount of engineering time into understanding the special demands of large storage, and have built a robust system to meet those demands.
- There are no special issues on the 5V side.
- Not all drives are created equal. Typically, drives with multiple platters are more power hungry than drives with less, and enterprise HDDs draw significantly more power than consumer level drives. We have lots of experience with the common drives in the marketplace, check our wiki, or contact us, to determine if drives are compatible. (If we don't have the specific information you need, we would be happy to do measurements for you).
No comments:
Post a Comment