How much free space to leave on an external HDD used for storage? + Corruption multiple backups?

Discussion in 'SSD and HDD storage' started by 321Boom, Nov 21, 2017.

  1. A2Razor

    A2Razor Master Guru

    Messages:
    443
    Likes Received:
    33
    GPU:
    ASUS R9 Fury X
    Sometimes people include them in the filenames themselves, yep. Those hashes can also serve as unique identifiers for files, such as for database storage and lookup of them (kindof like an image GUID - global unique identifier). Viewing images is for all intensive purposes "good enough", but the human eye won't necessarily detect miniscule damages just like most people won't see or notice a quality loss with image compression (like jpeg).

    There's probably more post processing going on with the other ports for things like dynamic contrast, edge, or color enhancement. In the worst case modern TV's have motion interpolation too which adds seriously enormous amounts of delay. --Shame that the ports don't have any options buried somewhere in the menus, yet it's good that you've found the TV is doing some stuff differently on each.

    If you wanted to use it, you'd need a kernel that supports it on each device. Though that said for a home-network situation I doubt that multihoming would be much benefit to you. mptcp is more for if you want to combine multiple connections together, or get extremely high transfer speeds over a high latency link, or achieve things like "connection redundancy" (handoffs from one connection to another).

    --You may be surprised yet there's no official Windows mptcp implementation (just like there's no official Windows "SCTP" implementation either, heh). Closest that can be done is using a Linux or BSD VM as a router run on the same machine (router inside computer, for the computer itself). There's some OpenSource projects doing just that as a short term solution to get mptcp on Windows... Apple is pretty much at the forefront in pushing mptcp (for global adoption), probably mptcp will be in the Linux and BSD mainline kernel branches before Microsoft gets onboard. (really ironic considering Microsoft is generally thought of as a technology leader)

    You'll find mptcp in quite a few commercial "Internet Bonders", yet the largest scale deployment is Apple's phones and servers. I'd personally have thought that Microsoft would have more interest in leading Internet protocol innovation ... though they're showing very little interest in doing so.

    Yep, you got it, as things are your clients "so far" are Windows machines.
    -The "d" at the end of OpenSSH there stands for "daemon". The OpenSSH project is actually both a client and a server (daemon), and you'll find that some Linux distros will throw in that "d" at the end of the service name to avoid confusion [make it clearer that it's talking of the server-component and not the client]. You can also install the SSH client and server independently, but the core openssh package tends to come with the collection of both together (more or less the complete solution / all tools you need to start a server and communicate with it).

    Windows, Linux, Android, iOS, you name it and there exists an SFTP client for it. Very widespread use, pretty much becoming a defacto standard for secure file access, even some webbrowsers getting support built in to them for it.

    NetDrive being commercial limits what OpenSource code they can "legally" use. OpenSource projects like WinFsp (which are basically FUSE' concept ported to Windows) are GPL.
    --In jist, unless a project is "L"GPL (lesser GPL) -- like "Dokan", then any works that link the GPL-code (dynamically, statically, doesn't matter -- use of that code in any way) automatically also become "GPL" (aka, public domain). In otherwords, source has to be released which means that it's difficult to protect your assets. Some licenses are more restrictive than others, but in general unless a library is something like MIT or BSD license, most companies will be pretty weary of touching them.

    ^ The desire to keep source "closed" isn't necessarily a bad thing, though it can mean reinventing the wheel when there's projects (like WinFsp) out there that already work. NetDrive is pretty stable, but it's a different project completely from the ground up.


    Anyway, getting back out of licensing those softwares all work similar (have same end goal). Whichever you pick and configure will mount an SFTP share as a 'volume' on the machine. That means you configure them once and they "do their thing". They provide what looks like a disk-drive to anything you use .. be that Windows Media Player, VLC, drawing programs, a text-editor, or Windows Explorer.

    Yes, ahem, this is for ease of storing files that you're going to work on when you're at home. Convenience, quality of life: yeah, that, maybe. (COUGH)

    "These third party clients are WinSshFs and NetDrive?" -- there's actually a ton and ton of them as you'll see when you search around, but yeah, they're all the same core concept. Mounting some remote-share with various protocols that Windows doesn't natively support.

    --It's possible to use Windows Explorer like-normal with those two clients.

    FileZilla and WinSCP are standalone programs that can transfer and synchronize files to servers. They don't mount the shares as a volume / drive, and they won't give you access to those network shares from other software.

    --They're more useful for infrequently accessed shares. For instance, say that you have a website that you might want to upload files to "occasionally", then it just may not make sense to leave mounted and you might prefer a solution like those tools.

    I'd say that the best bet is an automated scan with everything you can. If you have a machine with Avira, have that scan your files. If another machine has Malwarebytes, try scanning everything with that too, same goes for anything else including even Microsoft Security Essentials or ClamAV on the NAS itself. The more you use, the greater the chance of detection if some infection were to somehow sneak on there.

    --It'd be a good idea to tweak your AV settings in all cases because we don't necessarily know what the AntiVirus programs will recognize "as a network-share". For instance, it's always possible that they'll add detection for the third-party network mounting tools as well as Windows' native build-in mounting.

    The server itself would be fine (not infected), yet malware could still do damage ... like replacing your images with pictures of squid, infecting stored executables on the NAS, encrypting files (randsomware), etc. The more you can restrict access (eg, read-only), the safer you'll be in this regard since there'll be less files that an infected machine could touch or overwrite.

    See above, though to elaborate a bit:
    --Storing malware on the server alone does nothing beyond that [just copying files to the NAS doesn't infect the NAS or other machines necessarily] ... though any computer that can write to the NAS can also destroy data on the NAS (delete, edit, etc). This is why you'll want to give some thought to which machines should have write-access.

    The game machine only needs limited write access for transferring off videos. A media-PC would only need read-access to play movies, videos, and so on. As long as you can restrict access like that, and only do questionable things on machines that have restricted access, then your data will be very safe.

    Yep! You can do this on FreeNAS, or you could run it from Windows on your mounted volume (may be faster from the server over SSH -- due to no network-chokes). FreeBSD comes with everything you'd need to generate and check hashes for files. If you look around you'll find that alot of people have written "bash-scripts" and "one-liners" using these tools to loop over everything in a directory and write out or compare hashes to / from files.

    Example shamelessly ripped from StackOverflow (boards) of a one-liner generating hashes for a directory:
    Code:
    find ./path/to/directory/ -type f -print0  | xargs -0 sha1sum
    EDIT: (description of the above)

    Realize that this may look intimidating, but there's actually not much to this.

    Code:
    x:~$ mkdir test
    x:~$ cd test
    x:~/test$ echo "test" > 1.txt
    x:~/test$ cp 1.txt 2.txt
    x:~/test$ find ./ -type f -print0
    ./2.txt./1.txt
    x:~/test$ find ./ -type f -print0 | xargs -0
    ./2.txt ./1.txt
    x:~/test$ find ./ -type f -print0 | xargs -0 sha1sum
    4e1243bd22c66e76c2ba9eddc1f91394e57f9f83  ./2.txt
    4e1243bd22c66e76c2ba9eddc1f91394e57f9f83  ./1.txt
    
    Code:
    **ripped from the documentation**
    
    -f      regular file
    
    -print0
          True;  print  the  full file name on the standard output, followed by a null character (instead of
          the newline character that -print uses).  This allows file names that contain  newlines  or  other
          types  of  white space to be correctly interpreted by programs that process the find output.  This
          option corresponds to the -0 option of xargs.
    
    So what is all this? Well, in English version:

    mkdir test -- make a directory
    cd test -- move to that directory
    echo "test" -- writes "test" to standard output (to the console)
    > 1.txt -- pipe the output from echo to a file instead (creates a file containing the line "test")
    cp 1.txt 2.txt -- copy 1.txt to 2.txt (now we have two files in there, hooray)
    find ./ -type f -print0 -- sometimes when you want to see what you're doing it's easiest to execute it in 'parts'

    find is a command line tool to search files and directories. In this case we want to find files in "./", since that's the folder we're in right now (that newly created "test" folder).

    We get a single line output "./2.txt./1.txt" -- there's actually a character between those two that we can't see represented in the shell here, a "\0" (usually called a null terminator). That's being used instead of a newline delimiter because we asked for it with "-print0".

    "|" -- is used to feed the output of one program to another.

    " | xargs -0" -- xargs can take that null-terminator delimited output from "find" and make them in to spaced and quoted (if needed) command line arguments (basically make them ready to pass to another cmdline tool). -0 denotes that null-terminator will be used instead of newline like in "find".

    This output looks like:
    "./2.txt" "./1.txt" (two nicely formatted arguments to pass along)

    find ./ -type f -print0 | xargs -0 sha1sum -- So now all we need is to pass these in to a tool that generates hashes and takes a list of files. (like sha1sum, but could be sha256, md5, or any other tool that takes similar arguments)


    Everything seems like a flood at first, but there's also only so much that's out there too. When you first used Windows or DOS, it probably took some time getting used to as well. It's really much the same as picking up any new game, where the controls feel overwhelming and there's so many new systems and mechanics to learn. After a bit though, that all calms down when the realization hits that you're getting closer and closer to the ending.

    Playing with your file server, BSD, Linux, or any OS is much the same as playing a new videogame. Once you get the mechanics down, everything gets easier and easier with the more you experience.

    No problem as always, and you too!

    Also sorry if this seems that I'm basically preaching switching away from Windows at times with all this OpenSource stuff. I'm not even that strong an OpenSource advocate, but moreso it's just a matter of stepping out of that comfort zone just a little-bit and seeing all this awesome stuff that's offered out there (especially in the server space). Windows has its place, and so does BSD, and Linux, Android, Mac, and everything else built on them thereof [and especially OpenSource where commercial works just wouldn't happen]. They all have their strengths and things they're awesome at if you just push past the initial learning curve and see what they can do for you.

    It's like having a chocolate cake, vanilla cake, and so on. All are great, and having one flavor alone is great, but more is always better (no reason to limit yourself to just one). :)
     
    Last edited: Mar 31, 2018
    321Boom likes this.
  2. 321Boom

    321Boom Member Guru

    Messages:
    110
    Likes Received:
    10
    GPU:
    GTX980 Ti
    Thanks for confirming that. Yep, you're definitely right about the GUID part, there were times when I needed to search for the same image again (to find out who the artist was), and sometimes googling just the hash brings up the same image even from different websites that are hosting it. (The image site I save anime art from used to only use the hash as the file name before, about a year ago they updated it so the file name includes more information like: the anime/game it's from, character's name, and most importantly the artist's name, and the hash.) Here's what I'm getting at with this, all anime art I'm saving now comes with the artist name, so it's easy to find out who the artist was (so I could visit their gallery and see what other art they drew), but I've got a HUGE bunch of art which only has the hash as the file name, so finding the artist isn't as easy. As I said above, googling the hash sometimes will find the image again, so from there I could find the artist, but anything I can do to find the image again in the cases where the googling the hash won't find anything? (maybe there's some program or website that combines the hash with the GUID database or something)

    Btw, if the hash is those numbers (2c35d9e70306b3696fed52eef483e259) tacked on at the end of the file name, isn't it too short? I mean, doesn't MD5 use 128 bit, SHA uses more, so the hash should be longer? (the hash is only 32 characters), or are each 4 bits represented by 1 character? (sorry if I'm mixing things up like if bits have nothing to do with character length)

    Got it about the human eye can't detect slight signs of corruption, so it's definitely worth making a database of my hashes periodically then.

    Yes I agree, on the PC HDMI port the colours are a slightly more matted out (kind of like when you switch on Game Mode on a modern TV with a console, this also dramatically reduces input lag), so it shows that most of the 'image enhancing features' are switched off. Small trade for such a huge decrease in input lag. Yep, I read somewhere that the PC port makes the TV 'act' like a monitor rather than a TV, that's why it's so beneficial having a gaming PC hooked up to that port instead.

    Wouldn't mptcp provide a better checks than regular TCP though as we said in post 97? (so it does sound beneficial :/)

    By 'kernel that supports it on each device', you mean I need to keep this in mind when purchasing parts, or it's just something I could install later? (some way to update the kernel etc)

    Haha I know what you mean about Microsoft and the irony, for a company thought of as 'the technology leader', they sure do have their shortcomings.

    Right, thanks for the explanation. So I need to use both OpenSSH and SSHFS, or will they both be doing the same thing? Is OpenSSH built on FUSE also, or am I mixing something up here?

    Hmm thanks for the in depth explanation about licensing. Seems like lots of red tape, the more source is open, the more people could continue adding onto it/pitching in to make it better. No need to 'reinvent the wheel', yet I understand how this could be good and bad, because building something from the ground up could result in something new/innovative/better than the current options.

    So WinFsp and Dokan are another 2 alternatives to WinSshFs and NetDrive?

    'They provide what looks like a disk-drive to anything you use', in my case this will be one big 24TB volume, correct?

    Haha, of course it is :p

    Got it, that's awesome that they provide various protocols which Windows doesn't already have, and also that I'll be able to continue using Windows the way I'm used to :)

    Oh ok, got it, I don't think they're particularly something I need for my intended use with the server then? (I need access to it constantly.)

    Got it, so as many as possible :p Good to know it will be scanning both from the Windows side, and also on the BSD side.

    Noted about the network scanning features. Thanks for the heads up.

    Ok that is VERY BAD, how could you call that 'the server will be fine' lol. (it's purpose is storing data after all :p)

    2 questions regarding your quote 'The game machine only needs limited write access for transferring off videos. A media-PC would only need read-access to play movies, videos, and so on.'
    1. If the gaming rig had to be compromised, and it only has write-access to one folder (where to upload my gaming recordings, I'll move them around and organize them with the ECC desktop from there) it cannot infect any other folders? (i.e. the malware, virus, etc can't spread to the other read only folders?)

    2. My 'media-PC' is going to be my ECC desktop. I can't have that on read-only since I'll be using that as my main desktop (saving art, updating spreadsheets etc). Opinions/care to elaborate on the matter?

    Damn dood, you sure know your way around this stuff, that code was like x_X for me. Thanks for the in depth explanation of the alien language, I really appreciate you taking the time to explain it, I understood like 70% of it (thanks to your English version), but I really hope I don't have to be doing that stuff manually and could just download a one liner lol. 'Realize that this may look intimidating, but there's actually not much to this.' Lol not much to it for someone that knows what he's doing, not a newbie haha :p

    Got it, so through FreeNAS is best to avoid network chokes then. Any idea how long this hash generating will take? (i.e. long hours like a verify?)

    One thing I really didn't get, in your English version, 'cp 1.txt 2.txt -- copy 1.txt to 2.txt (now we have two files in there, hooray)', why would a copy of the file 1.txt to 2.txt be necessary, aren't we just generating hashes, so why would any copying take place?

    Haha it's slightly different from a new game, games come with nice fancy GUIs, not code :p Jokes aside, yeah I'm sure it will get easier down the line, especially when I'm implementing everything and getting to grips with everything firsthand rather than just reading about it.

    Nah don't worry about it, I completely agree with you about the whole Windows thing. As far as ease of use goes, Windows is definitely the easiest, but then there's so many things that are half-assed on Windows that I'm willing to venture to the other side, especially if it means keeping my data safer. After all, it would be a real shame having something as beautiful as a 24TB RAID10 system, but not having the correct and best measures for it :) (like ZFS, SFTP mounting etc.) Thanks again for all of your time in walking me through it. I really can't stress it enough, I understand everyone is busy with their own lives too, so I appreciate your constant support/mentoring.
     
  3. A2Razor

    A2Razor Master Guru

    Messages:
    443
    Likes Received:
    33
    GPU:
    ASUS R9 Fury X
    -Tried Google's "Search-by-Image" service, instead of searching via the Hash? Of course this assumes you still have the base images that you can re-upload. Google's search database is probably as good as you'll find far as doing this.

    -Each hex character is something called a nibble, 4-bits you're right on that part.
    -The smallest addressable (memory or file) unit on most computers is "a byte" (without using bit-shifting), or two nibbles / 8-bits. That means we could store a 128bit hash in (128 / 8) = 16 bytes.

    Unfortunately you'll notice that I called this 16-bytes and not 16-characters. Certain values of a byte, even in ASCII coding do not have a visible character assigned to them, or may have special-meaning (like 0x00 -- null terminator, newline, return, tab, break, etc). One way to get around that is to go byte by byte across the 16-bytes of the hash, and double them, converting each to a nibble-pair and then its hex-digit representation. This guarantees that there's no overlap with special characters, and that each is friendly / representable in any character-set (human readable & string storable). The null-terminator typically marks the end of a string, so that'd be a no-go if the hash contained a single byte of value '0'.

    mptcp only adds support for multiple connection endpoints and changing endpoint mid session. It doesn't add any additional checks and otherwise functions similar to TCP, which is why Internet gateways don't need extra support for it (don't have to be updated / works without change other than to the endpoints). Even a lowly consumer router NAT translation works with mptcp subflows without change (as they get treated as regular TCP connections), but mptcp was more or less designed to be this way (work with minimal changes).

    Operating System kernel. eg, Windows, Mac, Linux, FreeBSD. -- All purely software, and if Microsoft got onboard would likely just come in through a Windows Update down the road. (in however many years it takes for adoption)

    It'd be possible to implement on Windows without Microsoft getting onboard through a userland TCP stack implementation + a winsock LSP or WFP. I'm not aware of any efforts to do that (probably because it'd be really-hard). All in all it would be much-much easier if Microsoft did so themselves. To do it in user-space would be kindof like building a socksifier, only you don't use socks and instead pipe the Windows TCP-stack in to your own "built in usermode" TCP stack (yo dawg, I put TCP in your TCP so you can....), which then handles the connection using something like winpcap or a custom bridge-driver. You essentially wouldn't replace the TCP stack and would leave it there.

    Networking, yep that's a huge shortcoming of Windows. When people say that Windows isn't a good choice for servers, there's actually alot behind that other than MS hatred, cost, or favoritism.

    --Best example that I can think of is interface selection and routing rules. Say that you have two Internet connections on Windows, two-routers, two separate gateways. How do you specify which gateway should be used for an address you want to connect to? Can software even do this?

    Windows has a routing table (emphasis on "a") and you can add rules in here based on a destination IP. While that works, what if you want two connections to the same server, one out of each of your routers? Well, you just can't easily do that (without two IP's), there's "tricks" that can achieve it ... yet it gets ugly pretty fast. Linux or FreeBSD on the otherhand an application can just specify which interface it wants to bind for outbound connections "per socket". The OS's can have multiple routing tables too, one per interface, whereas Windows follows a single routing table premise.


    File-Systems and data-storage is also quickly becoming a Windows "weakness" (comparatively speaking). Even Microsoft Storage Spaces (which admittedly is a big-improvement) is still quite feature lacking compared to the choices available on the Linux & BSD side.

    OpenSSHd you'd use as your "server" that SSHFS connects to (as a client). Only the client implementations (SSHFS) are built on FUSE, because only the clients mount a network drive. The server doesn't have to mount a drive, as all files are local to it (it's just sharing them).

    The gaming machine's uploading of videos "might" be a good candidate for using those types of "one-shot" tools, since access is just going to be getting games (probably) copied on to the SSD (unless you're also going to run games off the NAS, that's cool too), and uploading your recordings. It's also much less likely that malware will try to self-propagate this way if the drive is left un-mounted. Since then to copy itself to your file-server, the malware would have to understand how to make a connection to it, eg -- not as simple as just writing files to disk [would need to get credentials to the server & have a built in SFTP client within the malware].

    Well, it all depends how locked down your access rules are for each machine. :D

    --If you only have read only access, like on a "dedicated" media-PC.. Can't write files, can't delete files, etc, not much the infection can accomplish other than destroying the media PC. So, no-problem, impossible for infection to spread itself, probably also for it to get at sensitive data as I assume there's nothing sensitive in Anime downloads or movies.


    The way I personally think of it is that your NAS is not a backup in itself. There's a common saying that you don't actually have a backup unless you have your data in "at least" two physical locations. The NAS takes care of many causes of data-loss, but malware falls more under user-error, since it's not hardware failure.

    Only true way to be completely-thorough is to be smart with your access-rights on the NAS and to also keep external (disconnected) backups on external drives. The last-resort defense. A shadow-copy (versioning) of files on the NAS can "kindof" protect against malware, but the NAS even then doesn't know the intent of the user, and you'd have to by-hand catch malware tampering with files for that to be useful.

    ----Easier said than done, takes some careful thought. Definitely will be more annoying than just giving yourself full access on all machines, though in the end very-worth-it.

    I'm presuming that malware is only interested in tampering with executable-files and not media-files. So, if all you're uploading is media-files, yes, you'd be pretty safe / the infection would be stopped at the gaming PC.

    --Of course, don't quote me on that since there have been "bugs" inside "Media Players" (I'm looking at you Microsoft, remote-code exploits) and also bad macro features (COUGH, Microsoft) that allow stuff such as "opening a web-page" embedded inside an MP3 file.... How anyone can think that's a good idea is beyond me.... Then again, we have all that garbage like MS Word macro-viruses too.

    In this case I suggest that you take extra-precautions on media-playback, especially if you use Windows Media-Player (**see above**). I consider this a low-risk personally, yet I'd get yourself a copy of Sandboxie and isolate whatever software you use for playing content (be that VLC, WMP, etc) inside a sandbox. You probably also should use Sandboxie to further secure email and web-browsing from each machine, but that's another topic.

    Hash generation and comparison are the same-speed as eachother (since both involve calculating hashes for each file). Amount of time that it takes is usually the amount of time it takes to read-back the files off the disk (choked by read-speed). So, it really depends on how much you want to verify. To verify hashes on every file on the drive implies needing to read all files on the entire drive, and that could be enormous. Meanwhile just verifying a few games might take a minute or less. Big-files will also take less time than many small files due to this being hard-disks and somewhat slower at random access vs sequential.

    If you generate hashes remotely (not on the NAS), you'll hit a network choke before you hit the drive IO choke most of the time. Though again this might not even matter because you're not running a comparison on very much.

    With only a single file, we could just run sha1sum directly without using find. Need at least two files that we can show that it goes over all files in the directory, or really just to show what the purpose of that find command was in the one-liner. ;)

    ^ This seriously.

    Being the captain' and immersing yourself in it will speed learning up tremendously, just like learning any spoken language. It's also much more fun once you get to start throwing drives in a machine and have something physical to play with. (then it's like having a toy, not studying, and the FUN [tinkering] begins)

    Windows has proven itself fantastic for the user-experience, the GUI, and "just-working" -- definitely user-friendly is something they're winning in (big-time, market share shows such).. At the same time though, you learn pretty quickly when you look at the server-side just how small Microsoft's presence has gotten there (how it's a complete 180 and a fraction of what Microsoft's share used to be). Linux pretty much took over the entire server market.

    Ironically in ease of use, you may find that Linux servers are actually easier than doing similar setups on Windows. It seems counter intuitive, but the market generally picks a winner not just in cost alone. It's kindof like when you watch "Who wants to be a millionaire" how almost always ask-the-audience turns up the right answer for how to do something, heh. But yeah, collective opinion polling by just looking at consumer choice is usually pretty telling.
     
    Last edited: Apr 12, 2018
  4. 321Boom

    321Boom Member Guru

    Messages:
    110
    Likes Received:
    10
    GPU:
    GTX980 Ti
    This is exactly what I was looking for! Thanks so much. Yep, I still have the base image/s, I just need to bring up the page I saved it from again, because there will be a tag of who the artist that drew it was on the page, and from there I could find his full gallery :) You really have no idea how much this is going to help me out. Thanks so much again.

    Awesome, thanks for the detailed explanation :) Very interesting Just one thing I didn't get 'The null-terminator typically marks the end of a string, so that'd be a no-go if the hash contained a single byte of value '0'.', By null-terminator you mean 0 correct? If 0 marks the end of a string, shouldn't it be the final character in a hash? Why are you saying it's a no-go? (There are 0s in the hash I listed: 2c35d9e70306b3696fed52eef483e259)

    So basically I'm stuck with regular TCP then for download verification? Nothing better to ensure what is getting saved from the ECC desktop directly to the NAS (as the default download location) makes it there safely?

    That does sound difficult :/ Guess I don't need to take note of any of these options since mptcp wouldn't be beneficial to my use, or is Winpcap something I could utilize?

    Thanks again for another explanation and example. While that example won't apply for my use, I could see how Linux or BSD could be superior in this aspect. I too would prefer another choice instead of Windows to handle all my data, I've read and heard that it is definitely not the safest route for servers from many sources.

    Ok, so I got that OpenSSHDd is on the server, but if SSHFS is for the client pcs, and my client pcs are going to be Windows machines, I'd need the Windows implementations of these like WinSshFs, not the actual SSHFS? (SSHFS doesn't work on Windows, it's for BSD and Unix right?)

    Hmm that does sound beneficial, especially the protections from malware, remember that the gaming rig will rarely be browsing any websites though, only times will be to update drivers or programs. So will it still be at a risk of malware/viruses? Here's something I don't fully understand, to get infected with malware or a virus, do I actually need to go into a malicious site, or just having an internet connection is enough for it to come in cause it has a 'door' to use if it needs to?

    Unfortunately I'll be connecting the gaming rig to the server more times than I originally thought. Remember when I told you I did the testing around with the HDMI ports, well anime art looks nicer on the Full 0-255 range which the gaming rig is on, so although I'll be saving the art with the ECC desktop, when I actually want to sit down and enjoy viewing it, I'll be doing that from the gaming rig. So, some questions about this:
    1. If the gaming rig is just viewing them (these folders will be set to read-only for the gaming rig), no worries that it can corrupt something while viewing since it's non-ECC right?
    2. Is there some way I can connect the gaming rig to the server with without having an internet connection where something malicious could come in from? You know maybe have my own intranet or something? LAN? Directly by UTP?
    3. If using FileZillla or WinSCP, could I also view my anime art and some gaming videos with it as if I'm using Windows normally? Or are they just to move data across from one place to another, and not function like Windows Explorer?

    There are lots of points of very good advice in this quote. I fully agree on most of the things you mentioned here. While it will be more cumbersome having to restrict access, it will be worth it for integrity's sake and safety.

    Definitely have regular offsite backups, those will be offline for most of the time so malware can't easily tamper them, and they'll be the last line of defense as we like to call it.

    I'll put more thought and careful consideration into what machine has access to which folders once everything is set up, cause yes as you stated, although annoying, very worth it.

    That's good to know. So even if I run virus and spyware/malware scans on the recordings before I upload them to the server these 'bugs' can still go undetected?

    One other thing to note, apart from gaming recordings, I'll be backing up my games into the server once the game is complete. Can't go for symlinking anymore now since I have to keep disconnecting the internet connection before gaming due to the Meltdown/Spectre updates performance hit. Games are executable-files correct, so malware could tamper with those if not the recordings/media-files, or is it the same risk and games are a different type of executable?

    No I don't use standard Windows Media Player. I use CCCP (Combined Community Codec Pack which uses MPC-HC (Media Player Classic Home Cinema)), VLC, or rarely PotPlayer when VLC has trouble with certain files (usually something in HEVC gives me trouble on VLC).

    Sandboxie sounds interesting, and you had also recommended this program before. Will I need to use Sandboxie only for media playback, or also when viewing anime art that I already have saved, saving new anime art from the web, updating spreadsheets etc? If it creates an isolated session, will it still manage to save the art to the default download location in the server? (since it's isolating itself from it?)

    That could take quite a while then, even on my current 7TB of data, let alone once the server starts filling up more and more! Well I guess I'd want to generate a hash for everything I have, makes the most sense right to make sure everything that is stored on the server stayed in 100% integrity as when I originally saved it.

    Got it, so no actual copying will be done when I'm running these one-liners on all of the data, this just happened due to the example being only one file?

    Agreed :) What you call FUN though, I call heart-wrenching terror, especially if it's in code lol, and especially till I see that I'm happy with the end result :p But you put your mind to it and start getting to grips with it. This was the same (but simpler I think) when I got into recording gameplay, at first it's x264 vs using the GPU, then all of the different features like Constant or Variable framerate, CRF (quality) and x264 encoding preset speeds (veryfast, superfast etc), Lanczos vs Bilinear vs Bicubic, etc. Was a lot to take in, and lots of testing and experimentation to be done to get the desired quality of recordings I have now, but it was all worth it in the end, and I'm sure that the server with ZFS and RAID10 (love saying that) will be sooooo worth it xD

    Indeed it is, as we said although it has it's shortcomings it's still an amazing operating system for lots of other uses. Yes I agree with you on how small the server share is with Windows, which is not surprising though when it lacks some of the nicer features Linux has.

    Lol, this is easier??? We've skipped a 100 posts talking about the subject :p Interesting to know, I thought people picked Linux more due to the extra features which Windows does not have implemented. Yep, collective opinion, the tried and tested method, all plays a part into what people will choose and opt for.

    Thanks once again for all the help and information, especially that Search-by-Image from Google, not only are you informative on data, but also on many other workarounds.
     

Share This Page