In times when mobile phone cameras produce pictures of 2 MBytes each and decent DSLR cameras produce pictures in the range of more than 20 Mbytes each – not speaking of the various sensors around the house the question of how all of this is going to be stored is an interesting one.
Prices for mass storage is dropping for years and sized of hard disks are getting bigger and bigger. 3 Tbyte drives are fairly cheap now. Cheap enough to consider serious redundancy even for home use.
Having that home automation hobby and having very specific needs when it comes to home entertainment or even watching TV (we don’t watch live-tv…) we have a relatively huge demand for storage space. That way we are already storing over 10 Tbyte of data, fully encrypted, redundant and backed-up.
Our file server infrastructure grew with the needs over the years.
It started way back in 2003 when I set-up the first fileserver for my apartment back then. It was a fairly huge 19 inch case with 5 hard disks (100 Gbyte each). This machine was filled in 2005 and needed replacement.
We’re in IDE land back then. Because the system hardware died on me due to a power surge all the disks and a new mainboard were seated in a new case with room for a lot of disks.
One interesting detail might be that I consistently used Windows Server for that purpose.
The machine always wasn’t just a fileserver. It was smtp, imap, nntp and media server all the time. That lead to a growing demand of CPU and memory resources. It started with an 800 Mhz AMD Athlon (which died quickly) and for the next years to come I used a 2.8 Ghz Intel Pentium 4. Everything started with Windows Server 2003 – bought in the Microsoft Store when I was a Microsoft employee.
Diskspace demand kept growing and in 2009 a new case, new mainboard + memory and new disks where due.
Since 2009 a Core4Quad Q9550 with 2.8 Ghz and 16 Gbyte of Memory is the heart of our fileserver. Since we’re frequently live-transcoding video streams to feed iPads and iPhones around the house that machine has plenty of grunt to feed the demand. We can have 2 iPhones and 2 iPads playing 720p content without getting stutters. Back in 2009 we also switched to a mixed IDE and SATA setup as you can see in the picture:
Plenty of room when the new case arrived – it was getting crowded just 2 years later in 2011. Every seat was taken – which means 13 disks are in that case and 1 attached through USB.
That adds up to more than 16 Tbyte of raw storage. In 2011 we also upgraded to Windows Server 2008. We never lost a bit with that operating system, not under the heaviest load and even through serious hardware malfunctions. A lot of disks of those 13 died throughout the years: Almost 1 every 2 months was replaced – most of them through extended waranties – of course we have a spare always ready to take the place. Only one time I had to rush to a store to get a replacement drive when two disks failed short after each other. That’s why there’s that 2 Tbyte drive in the 1.5 Tbyte compound…
So it’s getting full again. Since that case isn’t really holding more disks and replacing them is getting harder because of the tight fit the idea was born to now add a bigger case but to just add a NAS/SAN which holds between 6 to 8 disks at once, comes with it’s own redundancy management and exports one big iSCSI volume.
That said a network card was added to the fileserver and a QNAP TS-859 Pro+ 8-bay appliance was bought. This one is a shiny black device which uses less power then an aditional case with extra cpu and memory would have use and after calculating through a number of combinations it’s even the cheapest solution for an 8 drive set-up.
After some intensive testing it seems that the iSCSI approach is the most robust one. Since I am just done with testing the appliance the next step is to buy drives. So stay tuned!
Source 1: http://www.qnap.com/de/index.php?lang=de&sn=375&c=292&sc=528&t=532&n=3486
which speed do you need and what do you expect from that setup? which redundancy you are selecting? which filesystem you want to use?
So many questions in one – let’s see:
1. Speed
I do not expect excessive speeds from that system. It is enough to fill a gbit line, depending on what I am doing and which disk array I am using but most of the time it is sufficient if the server delivers 10-30 megabyte a second. We’re not cutting movies on that one.
2. Redundancies
Since it grew over time we started with RAID5s. This fast got out of control and the QNAP now allows us to use RAID6. RAID6 has reached End-of-Life as well but it’s okay for what we do right now. As a matter of fact we have a separate archival backup of everything which wasn’t even part of this article – so if for some strange reason we really loose something, we restore. It never happened – yet. We lost a lot of disks over the years but replacing them was a no-problem process.
3. Filesystem
So for historical reasons everything started with NTFS 3.0 on Windows Server 2000 and quickly was migrated to Windows Server 2003 and NTFS 3.1. It remained at that level ever since. As soon as truecrypt reached a maturity level that made it useable everything was migrated to first a truecrypt containerfile on a NTFS volume. The container contained the actual used NTFS volume. And later on all disks the RAID volumes was directly encrypted and formatted with NTFS within.
Needless to say that we never lost a beat on that filesystem so there never was a reason to replace it. Even with replacing disks, everything is just seamless and effortless.
With the QNAP there will be a RAID6 which gets exported as a whole using iSCSI. Then this volume is encrypted and formatted again with NTFS.
4. Future?
Since RAID5 and RAID6 reached end-of-life with the disk drive sizes of our times I think it’s time for different approaches. But that will be part of another blog article by itself.
Hi,
great post! Could you write something about the Software you use? The streaming part would be interesting!
Streaming is handled by 4 types of software.
1. Plex
I am using plex for all the movies and music. It’s powerful enough to target different devices. If I want to play it on my iPhone it re-encodes the movie for that device. Plex is great for that.
2. AirVideo
Just on the iPad and iPhone I am using Airvideo, which is a live-transcoding application that offers direct directory access. It’s working for years now.
3. direct SAMBA/CIFS access
Of course on all those android devices and on XBMC on AppleTV or RaspberryPi I am using Samba/CIFS to access the shares and the media files there. On Android BSPlayer works great, XBMC on everything else.
XBMC is especially interesting because it implements it’s own CIFS client.
4. for music and podcast only access
For that I have two pieces of software. Since there are 5 iOS devices in the house we tend to use the Airplay and iTunes WLAN sync features. Therefore iTunes is installed on the fileserver and serves that purpose. To access the music library from any device (pcs, ios, android,..) I usually use a SubSonic setup. SubSonic is particularly great because it can deal with humongeous libraries – I once had the whole Jamendo catalog mirrored and accessible there.