Driving for performance

By | 2014/04/06

The Setup

iStock RacingI have more than the usual amount of storage at home:

  1. I work from home so I end up needing to run a home lab.
  2. I tend to be conservative about what data I throw out.
  3. I backup a lot of my data.
  4. I keep offline archives of data that doesn’t require regular backups.

Between all this, I have somewhere in the order of 100TB of raw storage floating around at home*.

Over time that’s resulted in me buying a few different storage units.

Years ago, I went for a Drobo 2, the Firewire-800/USB-2 unit capable of taking up to 4 2TB hard drives. That’s long since been retired from primary production use, but these days provides a fairly stable slab of TimeMachine storage via a Mac Mini OSX server.

I retired the Drobo 2 by replacing it with the Drobo 5S and populated it with 5 x 7200RPM 2TB drives. That gets used for multimedia – videos, iTunes library, etc.

Over time as I accumulated more data and my partner switched from having a desktop and laptop to laptop only, I eventually caved in to the notion of home NAS and purchased a Synology 1513+ with 5 x 7200RPM 3TB drives, and that’s sitting around half full now.

Finally, recently I found myself needing to refresh my virtual lab. It had been running off my Mac Pro with 3 x 2TB 7200RPM drives inside in RAID-0 for maximum speed, but the Mac Pro was just old enough that its CPUs didn’t support VT-x for ESX within Fusion. Not having the money to buy a new Mac Pro, I decided to go for an interim 27″ iMac with a Quad Core i7 processor, 32GB of RAM and most importantly, Thunderbolt2.

To make the most of Thunderbolt2 with a limited budget, I purchased the Promise Pegasus2 4-drive enclosure and populated it with 4 x 7200RPM 2TB drives.

Suddenly I found myself idly running various performance tests against all 3 RAID units as well as the fast internal SSD on the iMac and a new 4TB Seagate USB-3 hard drive connected to the machine. After running a few ad-hoc tests I decided to start capturing the information.

The Tests

I came up with 3 core tests:

  • Read/Write a 64GB file:
    • In 1MB block sizes
    • In 512KB block sizes
    • In 256KB block sizes
    • In 128KB block sizes
    • In 64KB block sizes
    • In 32KB block sizes
  • Untar/Tar a dense filesystem structure:
    • 30,458 MB spread across 65,411 files/directories
  • Create a uniform filesystem structure of 38GB:
    • 12 directories, 32 subdirectories of each parent, with 10 x 10MB files written in 4KB blocks.

These may not be a perfect/ideal set of tests, but they gave me a sufficient variance based on the sort of things I’ve spent years having to deal with, viz.:

  • Big files
  • Dense filesystems
  • Evenly distributed filesystems

With the exception of dense filesystem structure scenario, I ran each test on:

  • The Promise Pegasus2
  • The Drobo 5S with 2-drive redundancy (what it normally runs with)
  • The Drobo 5S with 1-drive redundancy
  • The iMac’s 256GB SSD
  • The Seagate 4TB external USB-3 drive
  • The Synology over gigabit LAN
  • The Synology natively

For the dense filesystem structure, I didn’t run the tests on either the iMac 256GB SSD or the Synology natively. Nothing I had could stream data as fast as the iMac SSD, so it was used as the source point for the dense filesystem tests, and the Synology had no internal storage large enough other than its own filesystem to store the tar file used, so the test would have been significantly skewed against it.

Contents

I don’t have enough drives of the same type – and each enclosure has data on it anyway – that I could afford to populate every enclosure with exactly the same type.

All drives were 7200 RPM – I learnt with Drobo some time ago that going cheap and installing 5400 RPM drives is just asking for trouble, even if  you’re not after performance.

Synology

5 x 3TB 7200 RPM Hitachi Deskstar 7K3000. Each drive is SATA III and has a 64MB cache. These are undoubtedly the most modern drives I have. The Synology is connected over 1Gbit Ethernet, and for comparison I ran as many tests as I could via the CIFS shares and locally, on the Synology itself.

Drobo S5

Leveraging Drobo’s drive agnosticism, the contents of the Drobo 5S were built up over time, so there’s:

  • 3 x Seagate Barracuda ST2000DM001 SATA-III with 64MB cache.
  • 2 x Hitachi Deskstar 7K2000 SATA-II with 32MB cache.

The Drobo S5 was attached via USB-3 directly into the main ports on the back of the iMac.

Promise Pegasus2

This is populated with my oldest drives:

  • 4 x Hitachi Deskstar 7K2000 SATA-II with 32MB cache.

The Pegasus2 was attached via Thunderbolt2 directly into one of the Thunderbolt2 ports on the back of the iMac.

iMac SSD

Apple supplied SSD SM0256F (Samsung).

External 4TB Seagate

Seagate 4TB drive, “Expansion” model, attached via USB-3 directly into the main ports on the back of the iMac.

Comparing Apples and Oranges

Clearly these aren’t all the same drive types. There’s some SATA-III and some SATA-II, there’s different cache sizes and there’s different drive manufacturers. So maybe there’s little to no point in the comparison other than my satisfying general curiosity.

But let’s keep in mind that pristine configurations don’t always happen for home or SOHO storage.

And there’s also the nagging Drobo question. In brief – I ran the tests against the Drobo in 2-drive redundancy mode initially – mainly because I forgot I’d enabled it. At the end of the tests I realised it was in 2-drive redundancy mode and that it was hardly a fair comparison, so I switched the Drobo back to 1-drive redundancy mode.

Within 30 seconds of switching back to 1-drive redundancy mode, the Drobo advised the transition was complete and I started testing. I got abysmal results throughout the testing until about 2 days later when suddenly the results shot up. The net effect is that when Drobo tells you it’s finished converting from 2-drive redundancy to 1-drive redundancy, it’s clearly not telling the whole story.

As a result, I ended up running the tests a second time for the Drobo in 1-Drive redundancy mode (with the exception of the 64GB tests, which weren’t started until after the Drobo had finished its background cleanup tasks).

Test Results

Dense Filesystem

To perform the dense filesystem test, I ran a utility of mine, generate-filesystem.pl, which goes and generates a lot of random files. These files where then tar’d up without compression – the test consisted of reading the tar file from the iMac SSD and writing directly to each drive in question.

Dense Filesystem Read-Write Test Results

 

DriveAvg Read MB/sAvg Write MB/sBest Read MB/sBest Write MB/sWorst Read MB/sWorst Write MB/s
Promise Pegasus240.79105.8160.55108.7821.15102.9
Seagate USB3 4TB51.6110.0253.34113.6550.43107.63
Drobo 5S 2 Drive Redundancy30.3537.2539.239.9222.5132.65
Drobo 5S 1 Drive Redundancy Round #112.6724.814.6430.379.4719.46
Drobo 5S 1 Drive Redundancy Round #226.8455.0828.0756.424.7252.42
Synology via IP27.4820.2831.2420.6220.0919.79

Evenly distributed filesystem

This test consisted of a script which generated 12 parent directories, 32 subdirectories of each parent, and within each subdirectory, 10 x 10MB files written with a 4KB block size.

Evenly Distributed Filesystem Performance

 

DriveAverage Write MB/sBest Write MB/sWorst Write MB/s
Promise Pegasus2281.69284.44278.26
Segate USB3 4TB131.21132.41130.17
Drobo 5S 2 Drive Redundancy29.5930.8228.34
Drobo 5S 1 Drive Redundancy Round #127.0738.1717.63
Drobo 5S 1 Drive Redundancy Round #236.0536.9934.19
Synology via IP65.167.6163.37
Synology Native188.54189.16188.24
iMac SSD658.46673.68650.85

64GB File Write, Varying Block Sizes

64GB File, 1024KB Block

 

64GB File, 512KB Block

64GB File, 256KB Block

64GB File, 128KB Block

64GB File, 64KB Block

64KB File, 32KB Block

DriveBlock SizeAverage Read MB/sAverage Write MB/sBest Read MB/sBest Write MB/sWorst Read MB/sWorst Write MB/s
Promise Pegasus21024287.82262.3308.14286.86263.84223.91
Promise Pegasus2512288.61270.03306.53285.56268.34252.91
Promise Pegasus2256285.93276.42305.02291.08262.46252.71
Promise Pegasus2128282.11264.61290280.27268.04246.81
Promise Pegasus264285.56259.7298.66285.65264.22242.96
Promise Pegasus232284.32275.61299.76298.66264.16246.52
Seagate USB3 4TB1024139.77131.05152.78148.3132.71112.38
Seagate USB3 4TB512141.48130.11150.45146.22135.71109.41
Seagate USB3 4TB256144.33129.09149.28146.18134.56109.57
Seagate USB3 4TB128144.19140.06149.99144.62134.62131.44
Seagate USB3 4TB64143.12139.32150.48146.47131.45127.68
Seagate USB3 4TB32139.38138.13144.07144.96130.76127.71
iMac 256GB SSD1024778.53664.34780.81688.59776.41640.03
iMac 256GB SSD512646.95690.43681.55695.59628.99684.81
iMac 256GB SSD256646.26686681.02691.9627.93678.75
iMac 256GB SSD128641.75684.24678.38695.5623.14674.1
iMac 256GB SSD64640.37687.3677.12690.76621.62680.42
iMac 256GB SSD32638.49686.27677.43691.78618.61681.58
Drobo 5S 2-Dr Redundancy102479.7757.11100.758.6962.9555.73
Drobo 5S 2-Dr Redundancy51274.8848.7299.9254.0858.2242.5
Drobo 5S 2-Dr Redundancy25670.8455.8298.4555.9850.1255.53
Drobo 5S 2-Dr Redundancy12868.6454.4289.4856.3153.6351.63
Drobo 5S 2-Dr Redundancy6472.5355.8478.7956.5860.6855.17
Drobo 5S 2-Dr Redundancy3274.3356.279.8157.1370.2555.22
Drobo 5S 1-Dr Redundancy102493.1393.55111.8498.5180.2990.57
Drobo 5S 1-Dr Redundancy51283.1187.84108.41103.270.0567.85
Drobo 5S 1-Dr Redundancy25683.2782.32107.9796.2367.1656.37
Drobo 5S 1-Dr Redundancy12889.4494.47118.45103.5270.1289.38
Drobo 5S 1-Dr Redundancy6489.2998.15118.49102.0774.3194.97
Drobo 5S 1-Dr Redundancy3297.2897.98129.12105.4165.192.02
Synology over IP102492.7786.3294.3389.2491.9782.76
Synology over IP51292.6488.593.8288.8591.9687.95
Synology over IP25691.8387.6392.4389.591.1386.48
Synology over IP12892.4686.1593.6788.4991.4381.74
Synology over IP6491.1387.9292.6289.1789.5687.17
Synology over IP3291.5387.7891.8989.1391.2886.73
Synology Native1024342.71209.18354.25211.41336.08206.74
Synology Native512355.68206.57366.12210.73350.46203.53
Synology Native256369.58214.69372.36219.18366.12211.41
Synology Native128384.19226.58392.43232.4372.36222.91
Synology Native64385.83236.48394.8244.54370.26229.15
Synology Native32387.17232.76397.19239.18378.82228.35

Conclusions

I want to start by spending a little time talking about Drobo. For years I’ve been a huge fan of Drobo. I stuck through it when I had hard drive after hard drive fail in the 4-drive Drobo-2. Drobo insisted that 5400 RPM drives were fine, and maybe I just had a succession of failures, but the only way I got stability was to switch to 7200 RPM drives. When I migrated from the Drobo-2 to the Drobo 5S, all I had to do was take the drives out of the 2, put them in the 5S and add another drive.

For quite some time with an older Mac Pro the Drobo was hooked up via Firewire-800 (maximum speed around 80 MB/s throughput), and I always attributed the performance I got out of it to being a symptom of that legacy connection speed. Once connected via USB-3 though, it didn’t seem to improve much. Part of that was 2-drive redundancy, but switching to 1-drive redundancy still left the Drobo wanting for speed.

Seriously wanting for speed.

Drobo sells on the basis of ease of use, and it is indeed easy to use, but it’s abundantly clear that it comes at the cost of performance and transparency. Converting from 2-drive to 1-drive redundancy is supposedly a near-instantaneous process, but it would seem to trigger some fairly serious back-end garbage collection on the part of the Drobo that you’re not told about. Due to that desire for simplicity, the Dashboard doesn’t tell you a thing – there’s no indication anything is going on at all. Not a whisper from Drobo until up to 2 days later is suddenly announces that protection is in progress, then five minutes later announces it’s complete.

That’s why for the dense and evenly distributed filesystem tests feature two Drobo 1-Drive redundancy tests – the first was performed while the unit was secretly running whatever garbage collection process it doesn’t tell you about, and the second was run after I realised what had been going on and repeated the tests.

Drobo’s secrecy extends to every aspect of its operation, something that has irked me for some time. Most of its logs – potentially the really useful stuff in diagnosing issues, is locked away in binary format. (Drobo support have told me in the past this is important, but I think the importance lays in convincing you to continue to buy extended maintenance.)

The secrecy is so mind-numbingly obtuse that when it came time to document what hard drives were in the unit, I had to power it down and remove the drives, one by one.

The Synology, the Promise – both of those have rich dashboards which give you access to essential operations information. By far the Drobo is the most user friendly unit of the three, but between the secrecy of its operations and the poor performance, that seems a very poor compromise indeed.

After all, a single USB-3 drive outperformed the Drobo regularly on the same interface. For writes that might be understandable, but I’d expect better in read situations.

Given it had my oldest hard drives in it, I was pretty impressed with the performance of the Promise Pegasus2. 4-drive RAID-5 isn’t exactly comfortable – an extra drive always works well to balance performance out, after all. It’s clearly no slouch, and offered reasonably stable performance throughout testing.

I started the tests on the basis of wanting to check out what sort of performance I could get out of the Promise Pegasus2, but I learnt two other lessons:

  • I’m done with Drobo. This is the last unit I’ll own – ease of use is no longer a good enough pay-off in return for loss of transparency or performance.
  • My next lab array will be all-SSD.

By far and away, the SSD in my iMac outstripped every other thing I tested it against. I kind of expected that – I’d previously used SATA-I and SATA-II SSDs, but the performance coming out of a single SATA-III SSD is intense.

At the time I was working on the tests, I had a brief Twitter exchange with Vaughn Stewart, who suggested I should have gone for SSDs in the Promise. My counter argument at the time was a simple one: I need more capacity than I can comfortably get with even 4 x 500GB SSDs.

From a price perspective, I don’t regret my decision – I had a limited budget, to the point where I only purchased a single extra drive to populate the Promise. 4 x 500GB SSDs undoubtedly would have given substantially greater performance, but it would have also added another $1500 to the purchase price – and that’s not including the 2.5″ to 3.5″ mount kits for each drive.

When you have to buy your own lab equipment, that’s a lot.

An EMC presentation I was at a few years ago had a simple message: people buy storage for capacity, but upgrade for performance. That was mentioned at the enterprise level, but it was running through my head every time I looked at the difference in performances between a solitary SSD and a storage unit. So I’m left with one conclusion: it’s going to apply for my lab workloads … the next time I build a lab storage unit, that Thunderbolt2 connection (or whatever it is by then) will be fully populated with SSDs.

Other than the above, I’m not spending a huge amount of time trying to put an interpretation on any of the individual results – the numbers are all presented and speak for themselves … except for the Drobo. Those numbers crawl off into a dark corner and mutter terrible things.


* (Since a chunk of that storage is in offline archives, not all of that storage is spun up at once, and once various RAID formats are taken into account, the actual amount of presented storage is less.)