Network Storage - A lot of data

Discussion in 'Network questions and troubleshooting' started by Corrupt^, Jan 4, 2017.

  1. Corrupt^

    Corrupt^ Ancient Guru

    Messages:
    6,597
    Likes Received:
    1
    GPU:
    ASUS 1080GTX STRIX
    (Network) Storage - A lot of data

    Lately I've been reading around to get a NAS to store data on (more on that later) and more importantly (even more so than being able to attach it to a network), get some redundancy in case of a HDD failure.

    Now I'm no stranger to RAID setups and NAS's in general, but the more I read about the limitations of conventional solutions, the more I realize it probably won't fit my needs.

    What I'm looking for in particular:
    • Ability to store 20 TB of data with redundancy
    • Cost efficiency (20TB, HDD expenses basically)
    • Ability to scale to a larger size in the future (40TB, 60TB, ... growth is about 2 to 4TB a year)
    • Redundancy in case 1 or 2 HDD's fail (I'm looking at 10TB+ HDD's, see P.S. 3)
    • Restoration after a HDD failure: time to rebuild, etc
    • Read performance is more important than write performance

    The reason why I'm asking is that with such an amount of data, I get the impression an old-fashioned NAS RAID setup will probably not suffice, especially RAID5 (which is bound to get an unrecoverable read error around ~12TB if I'm not mistaken).
    http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

    I have enough knowledge (provided I get some pointers to tutorials, etc) to work with Linux if need be and it made me wonder if there aren't any other solutions, other then classic RAID.

    I've also read about Drobo, but the stuff I've read isn't all that good.

    P.S. The data itself consists of generally huge files of about 1GB or more (video).
    P.S. 2: Redundancy in case of a drive failure is more important then being able to read the data over the network.
    P.S. 3: I'm aware 10TB HDD's are overtly expensive now, but I'm looking into this for 1 or 2 years down the road. Currently most of this data is backed up on a pile of 4TB HDD's in a hermetically sealed box, which is getting cumbersome.
     
    Last edited: Jan 4, 2017
  2. nhlkoho

    nhlkoho Ancient Guru

    Messages:
    7,139
    Likes Received:
    12
    GPU:
    EVGA 980ti sc+
    I'm getting ready to do this myself after I've outgrown my Qnap NAS. From what I'm reading, FreeNAS seems to be the best overall software for this. I'm new to the whole build your own NAS thing right now.
     
  3. Corrupt^

    Corrupt^ Ancient Guru

    Messages:
    6,597
    Likes Received:
    1
    GPU:
    ASUS 1080GTX STRIX
  4. Hootmon

    Hootmon Maha Guru

    Messages:
    1,222
    Likes Received:
    0
    GPU:
    MSI R9 290 Gaming 4G
    Once you go beyond conventional backup and/or RAID solution, you usually need an Enterprise-scale solution.

    The problem with video files is that the data is likely already compressed in some way, and de-duplication doesn't work very well.
     

Share This Page