Wiki Blog/2005-02-27

Homebuilt 2 Terabytes RAID5 File Server Running Linux

About two years ago, I built a computer that would save all my files (movies, music, digital photos etc.). As it always is, capacity was quickly an issue: initially, it was 40 gigabytes (an IBM drive gotten back from RMA), then I purchased a 180GXP with 180 gigabytes. Some time later, a 10 gigabytes SCSI drive (together with an Adaptec AHA-2940UW, both ripped out from an old IBM workstation) was added. Notabene, all these drives had separate file systems.

Life was better, but as the system was filling up, big files would not find place on any of the file systems, even if the total free space of them was bigger than the file.

Another thing that bothered me: my 100mbit intranet was saturated when transferring large files from or to my file server. Movies or other big files were dreadfully slow to move around.

Specifications of a new system

I decided to upgrade and made a small checklist of what would be most important:

1. Capacity should be > 1 terabyte, if possible 2 terabytes
2. Price (should be as low as possible, but not cheap)
3. Redundant storage (ie. RAID 5, as other redundant levels are more expensive)
4. Use SATA (and prevent cabling problems with parallel IDE cables; SCSI was no option, see price)
5. Use integrated SATA ports on a motherboard, if possible (extension cards were no alternative, see price)

6. Find a motherboard with these features:

   • PCI-Express (so a future expansion card would not be bandwidth-bound by the PCI or PCI-X bus)
   • AMD64 capable
   • 8 integrated SATA ports
7. Find suitable (see price) SATA backplanes with trays
8. Find a tower chassis with a suitable number of 5.24" slots to acquire the SATA backplanes

So I started my research on the web. This were the preliminary results:

Choice of motherboard: Asus A8N-SLI deluxe

Link: http://www.asus.com/prog/spec.asp?m=A8N-SLI%20Deluxe&langs=01

The NVidia NForce4 chipset is designed for the AMD64 architecture (939 package form). It features PCI-Express channels, a gigabit ethernet port, 4 SATA ports. I chose the ASUS A8N-SLI deluxe, as it has an additional Sil 3114 with 4 SATA ports. This chip (altough capable of 66mhz) is connected to the 33mhz 32bit PCI bus to the chipset and thus maxes out theoretically at 133 megabytes/s - this leaves only 33MiB/s per drive, which is a bottleneck considering the performance of the nForce4 SATA ports (see below).

Additionally, there is a secondary gigabit ethernet interface on the motherboard. I suspect this is connected to the chipset by a 33mhz 32bit PCI bus as well.

Choice of Hard drive: Maxtor DiamondMax10 300 gigabytes Maxtor

Link: http://maxtor.com/_files/maxtor/en_us/documentation/data_sheets/diamondmax_10_data_sheet.pdf

I was bound to 8 drives now because of the motherboard. The most bang for the buck (you excuse my wording) I got with 63B300S0 300 gigabytes Maxtor drives.

Choice of SATA backplanes: Sunnytek SNT-3141SATA

Link: http://www.snt.com.tw/product.php?mode=show&cid=26&pid=78

After some very deep digging, I found the Sunnytek backplanes SNT-3141SATA. They each take 4 SATA drives and use three 5.25" slots in a chassis. This meant that I would have to get a chassis with six 5.25" slots.

Choice of chassis: Silverstone TJ01S

Link: http://www.silverstonetek.com/products-tj01.htm

This chassis has the needed six 5.25" slots and has a very nice finish, along with a door to cover the 5.25" slots. I removed the door for better ventilation, though.

Choice of power supply: Thermaltake ?PurePower 560W

Link: http://www.thermaltake.com/purepower/W0023.htm

I first bought the Enermax 370W (type EG375AX-VE(G)), but it just would not work together with the main board - there were sudden power downs, most of the time I couldn't even get it to start (and this was all without any drives, just CPU, RAM, PCI graphics card). So I got a almost No-Name PSU with 420W as a temporary replacement from my dealer and that worked. It runs 24/7 now since about three weeks without problems. I installed a Thermaltake ?PurePower 560W in the meantime, which should provide more reserves, taking into account the temperatures will be significantly higher during the summer than they are now.

Choice of CPU: AMD64 3200+

This processor was the one my dealer had on stock. Software RAID consumes quite a bit of CPU and if you have a fast storage sub-system (ie. hard drives and interfaces), it might get tight in combination with gigabit ethernet, which in turn eats away some CPU resources as well when transferring data at full speed...

Setting it all up

Like I hoped, using SATA made the cabling very easy. The system is quite loud because of the fans of the SATA cages and thus belongs in a basement. You don't want to carry it around a lot neither, because it is heavy, believe me.

Operating System and RAID setup

I chose Debian as operating system and used the new installer (the current version is the release candidate RC2). For simplicity, I installed the 32bit version, even if the processor would allow a 64bit operating system and software.

Surprisingly, the installer (using the 2.6.8 Linux kernel) detected all hardware correctly and without any manual intervention. Kudos to the Debian people!

I had to set up manually the RAID system after the installation. I used mdadm.

Sensitized by the problems with my old machine, I chose to go with the Logical Volume Manager. This software allows you to add, remove hard drives almost instantly and on the fly; you can resize partitions etc. Multiple physical drives form a pool (called Logical Volume Group) of usable space. This pool can be divided into Logical Volumes. They are almost like partitions and can contain one file system. See the good intro to LVM from Red Hat.

I configured my system as follows:

+----------------------------------------+
| Ext3 File System                       |
+----------------------------------------+
| LVM Logical Volume                     |
+----------------------------------------+
| LVM Logical Volume Group               |
+----------------------------------------+
| RAID 5 array                           |
+-----------------------+----+----+------+
| physical hard disk #1 | #2 | #3 | etc. |
+-----------------------+----+----+------+

Perfomance

write performance:

me@beast:$ dd if=/dev/zero of=/storagearray/test.tmp bs=4096
1238865+0 records in
1238864+0 records out
5074386944 bytes transferred in 63.475536 seconds (79942404 bytes/sec)

me@beast:$ dd if=/dev/zero of=/storagearray/test.tmp bs=4096
1291513+0 records in
1291512+0 records out
5290033152 bytes transferred in 58.591716 seconds (90286367 bytes/sec)

me@beast:$ dd if=/dev/zero of=/storagearray/test2.tmp bs=4096
1209993+0 records in
1209992+0 records out
4956127232 bytes transferred in 61.981403 seconds (79961521 bytes/sec)

this is between 76MB/s - 86MB/s.

read performance:

me@beast:$ dd if=/storagearray/test.tmp of=/dev/null bs=4096
1291513+0 records in
1291513+0 records out
5290037248 bytes transferred in 51.320580 seconds (103078283 bytes/sec)

me@beast:$ dd if=/storagearray/test2.tmp of=/dev/null bs=4096
1209993+0 records in
1209993+0 records out
4956131328 bytes transferred in 48.381239 seconds (102439116 bytes/sec)

this is about 98MB/s.

I did as well a test of the ethernet performance: connecting a gigabit ethernet capable laptop directly to the file server, I got read performances of about 60 megabytes/s and write performances of about 35 megabytes/s. It is very well possible that there is some room for tuning here... :)

CPU load during large file operations

When reading, the dd process puts a load of about 40% to the CPU. While writing, the load is about 24%.

Problems and possible improvements

I agree that read and write performance are not stellar. The reason for this might be that it's a RAID5 array, which would also explain why write performance is lower than expected. Theoretically, read performance should scale quite well with the number of drives.

Update 2005-02-28:

LVM2 apparently affects performance (see this discussion: http://linux.msede.com/lvm_mlist/archive/2004/06/0051.html).

Through LVM, my system makes about 86MB/s:

me@beast:$ time dd of=/dev/null if=/dev/mapper/VolumeGroup1-lvol0 \
bs=1024K count=8192

8192+0 records in
8192+0 records out
8589934592 bytes transferred in 94.677439 seconds (90728421 bytes/sec)

real    1m34.686s
user    0m0.012s
sys     0m13.708s

Accessing the RAID5 array directly suddenly gives 174MB/s! This is 100% faster!

me@beast:$ time dd of=/dev/null if=/dev/md0 bs=1024K count=8192
8192+0 records in
8192+0 records out
8589934592 bytes transferred in 47.014788 seconds (182707079 bytes/sec)

real    0m47.034s
user    0m0.010s
sys     0m19.663s

Update 2005-03-01:

174MB/s is impressive for those having had only single drives, agreed. But for a 8 drive RAID 5 array, this figure should be about 150% higher, ie. around 240MB/s. See this article from the maintainer of the linux software RAID: http://cgi.cse.unsw.edu.au/~neilb/01102979338

Second Update 2005-03-01:

In an attempt to improve performance, I left out LVM2 of the loop and rebuilt the file system (ext3) directly on top of the RAID array. With this configuration even through the file system layer the read performance has improved significantly:

Write speed is still at 78MB/s:

me@beast:$ time dd of=/storage/test3 if=/dev/zero bs=1024K count=8192
8192+0 records in
8192+0 records out
8589934592 bytes transferred in 104.683379 seconds (82056337 bytes/sec)

real    1m44.812s
user    0m0.021s
sys     0m26.561s

But read speed has rocketed from 98MB/s to 166MB/s!

me@beast:$ time dd if=/storage/test2 of=/dev/null bs=1024K count=8192
8192+0 records in
8192+0 records out
8589934592 bytes transferred in 49.290285 seconds (174272366 bytes/sec)

real    0m49.372s
user    0m0.013s
sys     0m20.919s

Update 2005-03-02:

To further narrow down the reasons of sub-optimal performance, I launched concurrent reads on all of the eight drives (not through the RAID layer). The results are somewhat surprising:

• The nForce4 SATA ports performs quite well at 211MiB/s (52MiB/s per drive)
• The SiI 3114 however seems to saturate at 129MiB/s (32MiB/s per drive)

Thus total throughput is 340MB/s. To compare, non-concurrent reads give for both controllers values of 60MiB/s. Concurrent reads on all four drives connected to the nForce4 SATA ports gives a total of 245MiB/s (61MiB/s per drive).

Another point is that when booting the system (happens rarely, but still needed sometimes), one of the SATA ports (of course belonging to the SiI 3114 :) does not recognize the drive correctly. This leads the RAID software to the false assumption that the drive died and it runs in degraded mode without any redundancy. When rebooting and manually re-adding the lost drive, the array re-calculates all checksums and thus puts heavy load on all disks, which is not a very good thing. I will have to write some script to prevent the RAID array to come up after a reboot if there is a drive missing and wait for manual intervention.

Something I'll have to fix as soon as possible is that smartctl does not work with SATA without a patch of the Linux kernel (see this document: http://linux.yyz.us/sata/software-status.html#smart).

Conclusion

With a budget of about € 2700 (USD 3450) and a price of € 1.35 (USD 1.78) per gigabyte, I built a 2 terabyte file server for home use.

If you feel there is some information missing (prices of each item will follow soon) or if you have questions, please write an email to nicola.fankhauser@variant.ch.