1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How much free space to leave on an external HDD used for storage? + Corruption multiple backups?

Discussion in 'SSD and HDD storage' started by 321Boom, Nov 21, 2017.

  1. A2Razor

    A2Razor Master Guru

    Likes Received:
    ASUS R9 Fury X
    Sometimes people include them in the filenames themselves, yep. Those hashes can also serve as unique identifiers for files, such as for database storage and lookup of them (kindof like an image GUID - global unique identifier). Viewing images is for all intensive purposes "good enough", but the human eye won't necessarily detect miniscule damages just like most people won't see or notice a quality loss with image compression (like jpeg).

    There's probably more post processing going on with the other ports for things like dynamic contrast, edge, or color enhancement. In the worst case modern TV's have motion interpolation too which adds seriously enormous amounts of delay. --Shame that the ports don't have any options buried somewhere in the menus, yet it's good that you've found the TV is doing some stuff differently on each.

    If you wanted to use it, you'd need a kernel that supports it on each device. Though that said for a home-network situation I doubt that multihoming would be much benefit to you. mptcp is more for if you want to combine multiple connections together, or get extremely high transfer speeds over a high latency link, or achieve things like "connection redundancy" (handoffs from one connection to another).

    --You may be surprised yet there's no official Windows mptcp implementation (just like there's no official Windows "SCTP" implementation either, heh). Closest that can be done is using a Linux or BSD VM as a router run on the same machine (router inside computer, for the computer itself). There's some OpenSource projects doing just that as a short term solution to get mptcp on Windows... Apple is pretty much at the forefront in pushing mptcp (for global adoption), probably mptcp will be in the Linux and BSD mainline kernel branches before Microsoft gets onboard. (really ironic considering Microsoft is generally thought of as a technology leader)

    You'll find mptcp in quite a few commercial "Internet Bonders", yet the largest scale deployment is Apple's phones and servers. I'd personally have thought that Microsoft would have more interest in leading Internet protocol innovation ... though they're showing very little interest in doing so.

    Yep, you got it, as things are your clients "so far" are Windows machines.
    -The "d" at the end of OpenSSH there stands for "daemon". The OpenSSH project is actually both a client and a server (daemon), and you'll find that some Linux distros will throw in that "d" at the end of the service name to avoid confusion [make it clearer that it's talking of the server-component and not the client]. You can also install the SSH client and server independently, but the core openssh package tends to come with the collection of both together (more or less the complete solution / all tools you need to start a server and communicate with it).

    Windows, Linux, Android, iOS, you name it and there exists an SFTP client for it. Very widespread use, pretty much becoming a defacto standard for secure file access, even some webbrowsers getting support built in to them for it.

    NetDrive being commercial limits what OpenSource code they can "legally" use. OpenSource projects like WinFsp (which are basically FUSE' concept ported to Windows) are GPL.
    --In jist, unless a project is "L"GPL (lesser GPL) -- like "Dokan", then any works that link the GPL-code (dynamically, statically, doesn't matter -- use of that code in any way) automatically also become "GPL" (aka, public domain). In otherwords, source has to be released which means that it's difficult to protect your assets. Some licenses are more restrictive than others, but in general unless a library is something like MIT or BSD license, most companies will be pretty weary of touching them.

    ^ The desire to keep source "closed" isn't necessarily a bad thing, though it can mean reinventing the wheel when there's projects (like WinFsp) out there that already work. NetDrive is pretty stable, but it's a different project completely from the ground up.

    Anyway, getting back out of licensing those softwares all work similar (have same end goal). Whichever you pick and configure will mount an SFTP share as a 'volume' on the machine. That means you configure them once and they "do their thing". They provide what looks like a disk-drive to anything you use .. be that Windows Media Player, VLC, drawing programs, a text-editor, or Windows Explorer.

    Yes, ahem, this is for ease of storing files that you're going to work on when you're at home. Convenience, quality of life: yeah, that, maybe. (COUGH)

    "These third party clients are WinSshFs and NetDrive?" -- there's actually a ton and ton of them as you'll see when you search around, but yeah, they're all the same core concept. Mounting some remote-share with various protocols that Windows doesn't natively support.

    --It's possible to use Windows Explorer like-normal with those two clients.

    FileZilla and WinSCP are standalone programs that can transfer and synchronize files to servers. They don't mount the shares as a volume / drive, and they won't give you access to those network shares from other software.

    --They're more useful for infrequently accessed shares. For instance, say that you have a website that you might want to upload files to "occasionally", then it just may not make sense to leave mounted and you might prefer a solution like those tools.

    I'd say that the best bet is an automated scan with everything you can. If you have a machine with Avira, have that scan your files. If another machine has Malwarebytes, try scanning everything with that too, same goes for anything else including even Microsoft Security Essentials or ClamAV on the NAS itself. The more you use, the greater the chance of detection if some infection were to somehow sneak on there.

    --It'd be a good idea to tweak your AV settings in all cases because we don't necessarily know what the AntiVirus programs will recognize "as a network-share". For instance, it's always possible that they'll add detection for the third-party network mounting tools as well as Windows' native build-in mounting.

    The server itself would be fine (not infected), yet malware could still do damage ... like replacing your images with pictures of squid, infecting stored executables on the NAS, encrypting files (randsomware), etc. The more you can restrict access (eg, read-only), the safer you'll be in this regard since there'll be less files that an infected machine could touch or overwrite.

    See above, though to elaborate a bit:
    --Storing malware on the server alone does nothing beyond that [just copying files to the NAS doesn't infect the NAS or other machines necessarily] ... though any computer that can write to the NAS can also destroy data on the NAS (delete, edit, etc). This is why you'll want to give some thought to which machines should have write-access.

    The game machine only needs limited write access for transferring off videos. A media-PC would only need read-access to play movies, videos, and so on. As long as you can restrict access like that, and only do questionable things on machines that have restricted access, then your data will be very safe.

    Yep! You can do this on FreeNAS, or you could run it from Windows on your mounted volume (may be faster from the server over SSH -- due to no network-chokes). FreeBSD comes with everything you'd need to generate and check hashes for files. If you look around you'll find that alot of people have written "bash-scripts" and "one-liners" using these tools to loop over everything in a directory and write out or compare hashes to / from files.

    Example shamelessly ripped from StackOverflow (boards) of a one-liner generating hashes for a directory:
    find ./path/to/directory/ -type f -print0  | xargs -0 sha1sum
    EDIT: (description of the above)

    Realize that this may look intimidating, but there's actually not much to this.

    x:~$ mkdir test
    x:~$ cd test
    x:~/test$ echo "test" > 1.txt
    x:~/test$ cp 1.txt 2.txt
    x:~/test$ find ./ -type f -print0
    x:~/test$ find ./ -type f -print0 | xargs -0
    ./2.txt ./1.txt
    x:~/test$ find ./ -type f -print0 | xargs -0 sha1sum
    4e1243bd22c66e76c2ba9eddc1f91394e57f9f83  ./2.txt
    4e1243bd22c66e76c2ba9eddc1f91394e57f9f83  ./1.txt
    **ripped from the documentation**
    -f      regular file
          True;  print  the  full file name on the standard output, followed by a null character (instead of
          the newline character that -print uses).  This allows file names that contain  newlines  or  other
          types  of  white space to be correctly interpreted by programs that process the find output.  This
          option corresponds to the -0 option of xargs.
    So what is all this? Well, in English version:

    mkdir test -- make a directory
    cd test -- move to that directory
    echo "test" -- writes "test" to standard output (to the console)
    > 1.txt -- pipe the output from echo to a file instead (creates a file containing the line "test")
    cp 1.txt 2.txt -- copy 1.txt to 2.txt (now we have two files in there, hooray)
    find ./ -type f -print0 -- sometimes when you want to see what you're doing it's easiest to execute it in 'parts'

    find is a command line tool to search files and directories. In this case we want to find files in "./", since that's the folder we're in right now (that newly created "test" folder).

    We get a single line output "./2.txt./1.txt" -- there's actually a character between those two that we can't see represented in the shell here, a "\0" (usually called a null terminator). That's being used instead of a newline delimiter because we asked for it with "-print0".

    "|" -- is used to feed the output of one program to another.

    " | xargs -0" -- xargs can take that null-terminator delimited output from "find" and make them in to spaced and quoted (if needed) command line arguments (basically make them ready to pass to another cmdline tool). -0 denotes that null-terminator will be used instead of newline like in "find".

    This output looks like:
    "./2.txt" "./1.txt" (two nicely formatted arguments to pass along)

    find ./ -type f -print0 | xargs -0 sha1sum -- So now all we need is to pass these in to a tool that generates hashes and takes a list of files. (like sha1sum, but could be sha256, md5, or any other tool that takes similar arguments)

    Everything seems like a flood at first, but there's also only so much that's out there too. When you first used Windows or DOS, it probably took some time getting used to as well. It's really much the same as picking up any new game, where the controls feel overwhelming and there's so many new systems and mechanics to learn. After a bit though, that all calms down when the realization hits that you're getting closer and closer to the ending.

    Playing with your file server, BSD, Linux, or any OS is much the same as playing a new videogame. Once you get the mechanics down, everything gets easier and easier with the more you experience.

    No problem as always, and you too!

    Also sorry if this seems that I'm basically preaching switching away from Windows at times with all this OpenSource stuff. I'm not even that strong an OpenSource advocate, but moreso it's just a matter of stepping out of that comfort zone just a little-bit and seeing all this awesome stuff that's offered out there (especially in the server space). Windows has its place, and so does BSD, and Linux, Android, Mac, and everything else built on them thereof [and especially OpenSource where commercial works just wouldn't happen]. They all have their strengths and things they're awesome at if you just push past the initial learning curve and see what they can do for you.

    It's like having a chocolate cake, vanilla cake, and so on. All are great, and having one flavor alone is great, but more is always better (no reason to limit yourself to just one). :)
    Last edited: Mar 31, 2018
    321Boom likes this.
  2. 321Boom

    321Boom Member Guru

    Likes Received:
    GTX980 Ti
    Thanks for confirming that. Yep, you're definitely right about the GUID part, there were times when I needed to search for the same image again (to find out who the artist was), and sometimes googling just the hash brings up the same image even from different websites that are hosting it. (The image site I save anime art from used to only use the hash as the file name before, about a year ago they updated it so the file name includes more information like: the anime/game it's from, character's name, and most importantly the artist's name, and the hash.) Here's what I'm getting at with this, all anime art I'm saving now comes with the artist name, so it's easy to find out who the artist was (so I could visit their gallery and see what other art they drew), but I've got a HUGE bunch of art which only has the hash as the file name, so finding the artist isn't as easy. As I said above, googling the hash sometimes will find the image again, so from there I could find the artist, but anything I can do to find the image again in the cases where the googling the hash won't find anything? (maybe there's some program or website that combines the hash with the GUID database or something)

    Btw, if the hash is those numbers (2c35d9e70306b3696fed52eef483e259) tacked on at the end of the file name, isn't it too short? I mean, doesn't MD5 use 128 bit, SHA uses more, so the hash should be longer? (the hash is only 32 characters), or are each 4 bits represented by 1 character? (sorry if I'm mixing things up like if bits have nothing to do with character length)

    Got it about the human eye can't detect slight signs of corruption, so it's definitely worth making a database of my hashes periodically then.

    Yes I agree, on the PC HDMI port the colours are a slightly more matted out (kind of like when you switch on Game Mode on a modern TV with a console, this also dramatically reduces input lag), so it shows that most of the 'image enhancing features' are switched off. Small trade for such a huge decrease in input lag. Yep, I read somewhere that the PC port makes the TV 'act' like a monitor rather than a TV, that's why it's so beneficial having a gaming PC hooked up to that port instead.

    Wouldn't mptcp provide a better checks than regular TCP though as we said in post 97? (so it does sound beneficial :/)

    By 'kernel that supports it on each device', you mean I need to keep this in mind when purchasing parts, or it's just something I could install later? (some way to update the kernel etc)

    Haha I know what you mean about Microsoft and the irony, for a company thought of as 'the technology leader', they sure do have their shortcomings.

    Right, thanks for the explanation. So I need to use both OpenSSH and SSHFS, or will they both be doing the same thing? Is OpenSSH built on FUSE also, or am I mixing something up here?

    Hmm thanks for the in depth explanation about licensing. Seems like lots of red tape, the more source is open, the more people could continue adding onto it/pitching in to make it better. No need to 'reinvent the wheel', yet I understand how this could be good and bad, because building something from the ground up could result in something new/innovative/better than the current options.

    So WinFsp and Dokan are another 2 alternatives to WinSshFs and NetDrive?

    'They provide what looks like a disk-drive to anything you use', in my case this will be one big 24TB volume, correct?

    Haha, of course it is :p

    Got it, that's awesome that they provide various protocols which Windows doesn't already have, and also that I'll be able to continue using Windows the way I'm used to :)

    Oh ok, got it, I don't think they're particularly something I need for my intended use with the server then? (I need access to it constantly.)

    Got it, so as many as possible :p Good to know it will be scanning both from the Windows side, and also on the BSD side.

    Noted about the network scanning features. Thanks for the heads up.

    Ok that is VERY BAD, how could you call that 'the server will be fine' lol. (it's purpose is storing data after all :p)

    2 questions regarding your quote 'The game machine only needs limited write access for transferring off videos. A media-PC would only need read-access to play movies, videos, and so on.'
    1. If the gaming rig had to be compromised, and it only has write-access to one folder (where to upload my gaming recordings, I'll move them around and organize them with the ECC desktop from there) it cannot infect any other folders? (i.e. the malware, virus, etc can't spread to the other read only folders?)

    2. My 'media-PC' is going to be my ECC desktop. I can't have that on read-only since I'll be using that as my main desktop (saving art, updating spreadsheets etc). Opinions/care to elaborate on the matter?

    Damn dood, you sure know your way around this stuff, that code was like x_X for me. Thanks for the in depth explanation of the alien language, I really appreciate you taking the time to explain it, I understood like 70% of it (thanks to your English version), but I really hope I don't have to be doing that stuff manually and could just download a one liner lol. 'Realize that this may look intimidating, but there's actually not much to this.' Lol not much to it for someone that knows what he's doing, not a newbie haha :p

    Got it, so through FreeNAS is best to avoid network chokes then. Any idea how long this hash generating will take? (i.e. long hours like a verify?)

    One thing I really didn't get, in your English version, 'cp 1.txt 2.txt -- copy 1.txt to 2.txt (now we have two files in there, hooray)', why would a copy of the file 1.txt to 2.txt be necessary, aren't we just generating hashes, so why would any copying take place?

    Haha it's slightly different from a new game, games come with nice fancy GUIs, not code :p Jokes aside, yeah I'm sure it will get easier down the line, especially when I'm implementing everything and getting to grips with everything firsthand rather than just reading about it.

    Nah don't worry about it, I completely agree with you about the whole Windows thing. As far as ease of use goes, Windows is definitely the easiest, but then there's so many things that are half-assed on Windows that I'm willing to venture to the other side, especially if it means keeping my data safer. After all, it would be a real shame having something as beautiful as a 24TB RAID10 system, but not having the correct and best measures for it :) (like ZFS, SFTP mounting etc.) Thanks again for all of your time in walking me through it. I really can't stress it enough, I understand everyone is busy with their own lives too, so I appreciate your constant support/mentoring.
  3. A2Razor

    A2Razor Master Guru

    Likes Received:
    ASUS R9 Fury X
    -Tried Google's "Search-by-Image" service, instead of searching via the Hash? Of course this assumes you still have the base images that you can re-upload. Google's search database is probably as good as you'll find far as doing this.

    -Each hex character is something called a nibble, 4-bits you're right on that part.
    -The smallest addressable (memory or file) unit on most computers is "a byte" (without using bit-shifting), or two nibbles / 8-bits. That means we could store a 128bit hash in (128 / 8) = 16 bytes.

    Unfortunately you'll notice that I called this 16-bytes and not 16-characters. Certain values of a byte, even in ASCII coding do not have a visible character assigned to them, or may have special-meaning (like 0x00 -- null terminator, newline, return, tab, break, etc). One way to get around that is to go byte by byte across the 16-bytes of the hash, and double them, converting each to a nibble-pair and then its hex-digit representation. This guarantees that there's no overlap with special characters, and that each is friendly / representable in any character-set (human readable & string storable). The null-terminator typically marks the end of a string, so that'd be a no-go if the hash contained a single byte of value '0'.

    mptcp only adds support for multiple connection endpoints and changing endpoint mid session. It doesn't add any additional checks and otherwise functions similar to TCP, which is why Internet gateways don't need extra support for it (don't have to be updated / works without change other than to the endpoints). Even a lowly consumer router NAT translation works with mptcp subflows without change (as they get treated as regular TCP connections), but mptcp was more or less designed to be this way (work with minimal changes).

    Operating System kernel. eg, Windows, Mac, Linux, FreeBSD. -- All purely software, and if Microsoft got onboard would likely just come in through a Windows Update down the road. (in however many years it takes for adoption)

    It'd be possible to implement on Windows without Microsoft getting onboard through a userland TCP stack implementation + a winsock LSP or WFP. I'm not aware of any efforts to do that (probably because it'd be really-hard). All in all it would be much-much easier if Microsoft did so themselves. To do it in user-space would be kindof like building a socksifier, only you don't use socks and instead pipe the Windows TCP-stack in to your own "built in usermode" TCP stack (yo dawg, I put TCP in your TCP so you can....), which then handles the connection using something like winpcap or a custom bridge-driver. You essentially wouldn't replace the TCP stack and would leave it there.

    Networking, yep that's a huge shortcoming of Windows. When people say that Windows isn't a good choice for servers, there's actually alot behind that other than MS hatred, cost, or favoritism.

    --Best example that I can think of is interface selection and routing rules. Say that you have two Internet connections on Windows, two-routers, two separate gateways. How do you specify which gateway should be used for an address you want to connect to? Can software even do this?

    Windows has a routing table (emphasis on "a") and you can add rules in here based on a destination IP. While that works, what if you want two connections to the same server, one out of each of your routers? Well, you just can't easily do that (without two IP's), there's "tricks" that can achieve it ... yet it gets ugly pretty fast. Linux or FreeBSD on the otherhand an application can just specify which interface it wants to bind for outbound connections "per socket". The OS's can have multiple routing tables too, one per interface, whereas Windows follows a single routing table premise.

    File-Systems and data-storage is also quickly becoming a Windows "weakness" (comparatively speaking). Even Microsoft Storage Spaces (which admittedly is a big-improvement) is still quite feature lacking compared to the choices available on the Linux & BSD side.

    OpenSSHd you'd use as your "server" that SSHFS connects to (as a client). Only the client implementations (SSHFS) are built on FUSE, because only the clients mount a network drive. The server doesn't have to mount a drive, as all files are local to it (it's just sharing them).

    The gaming machine's uploading of videos "might" be a good candidate for using those types of "one-shot" tools, since access is just going to be getting games (probably) copied on to the SSD (unless you're also going to run games off the NAS, that's cool too), and uploading your recordings. It's also much less likely that malware will try to self-propagate this way if the drive is left un-mounted. Since then to copy itself to your file-server, the malware would have to understand how to make a connection to it, eg -- not as simple as just writing files to disk [would need to get credentials to the server & have a built in SFTP client within the malware].

    Well, it all depends how locked down your access rules are for each machine. :D

    --If you only have read only access, like on a "dedicated" media-PC.. Can't write files, can't delete files, etc, not much the infection can accomplish other than destroying the media PC. So, no-problem, impossible for infection to spread itself, probably also for it to get at sensitive data as I assume there's nothing sensitive in Anime downloads or movies.

    The way I personally think of it is that your NAS is not a backup in itself. There's a common saying that you don't actually have a backup unless you have your data in "at least" two physical locations. The NAS takes care of many causes of data-loss, but malware falls more under user-error, since it's not hardware failure.

    Only true way to be completely-thorough is to be smart with your access-rights on the NAS and to also keep external (disconnected) backups on external drives. The last-resort defense. A shadow-copy (versioning) of files on the NAS can "kindof" protect against malware, but the NAS even then doesn't know the intent of the user, and you'd have to by-hand catch malware tampering with files for that to be useful.

    ----Easier said than done, takes some careful thought. Definitely will be more annoying than just giving yourself full access on all machines, though in the end very-worth-it.

    I'm presuming that malware is only interested in tampering with executable-files and not media-files. So, if all you're uploading is media-files, yes, you'd be pretty safe / the infection would be stopped at the gaming PC.

    --Of course, don't quote me on that since there have been "bugs" inside "Media Players" (I'm looking at you Microsoft, remote-code exploits) and also bad macro features (COUGH, Microsoft) that allow stuff such as "opening a web-page" embedded inside an MP3 file.... How anyone can think that's a good idea is beyond me.... Then again, we have all that garbage like MS Word macro-viruses too.

    In this case I suggest that you take extra-precautions on media-playback, especially if you use Windows Media-Player (**see above**). I consider this a low-risk personally, yet I'd get yourself a copy of Sandboxie and isolate whatever software you use for playing content (be that VLC, WMP, etc) inside a sandbox. You probably also should use Sandboxie to further secure email and web-browsing from each machine, but that's another topic.

    Hash generation and comparison are the same-speed as eachother (since both involve calculating hashes for each file). Amount of time that it takes is usually the amount of time it takes to read-back the files off the disk (choked by read-speed). So, it really depends on how much you want to verify. To verify hashes on every file on the drive implies needing to read all files on the entire drive, and that could be enormous. Meanwhile just verifying a few games might take a minute or less. Big-files will also take less time than many small files due to this being hard-disks and somewhat slower at random access vs sequential.

    If you generate hashes remotely (not on the NAS), you'll hit a network choke before you hit the drive IO choke most of the time. Though again this might not even matter because you're not running a comparison on very much.

    With only a single file, we could just run sha1sum directly without using find. Need at least two files that we can show that it goes over all files in the directory, or really just to show what the purpose of that find command was in the one-liner. ;)

    ^ This seriously.

    Being the captain' and immersing yourself in it will speed learning up tremendously, just like learning any spoken language. It's also much more fun once you get to start throwing drives in a machine and have something physical to play with. (then it's like having a toy, not studying, and the FUN [tinkering] begins)

    Windows has proven itself fantastic for the user-experience, the GUI, and "just-working" -- definitely user-friendly is something they're winning in (big-time, market share shows such).. At the same time though, you learn pretty quickly when you look at the server-side just how small Microsoft's presence has gotten there (how it's a complete 180 and a fraction of what Microsoft's share used to be). Linux pretty much took over the entire server market.

    Ironically in ease of use, you may find that Linux servers are actually easier than doing similar setups on Windows. It seems counter intuitive, but the market generally picks a winner not just in cost alone. It's kindof like when you watch "Who wants to be a millionaire" how almost always ask-the-audience turns up the right answer for how to do something, heh. But yeah, collective opinion polling by just looking at consumer choice is usually pretty telling.
    Last edited: Apr 12, 2018
  4. 321Boom

    321Boom Member Guru

    Likes Received:
    GTX980 Ti
    This is exactly what I was looking for! Thanks so much. Yep, I still have the base image/s, I just need to bring up the page I saved it from again, because there will be a tag of who the artist that drew it was on the page, and from there I could find his full gallery :) You really have no idea how much this is going to help me out. Thanks so much again.

    Awesome, thanks for the detailed explanation :) Very interesting Just one thing I didn't get 'The null-terminator typically marks the end of a string, so that'd be a no-go if the hash contained a single byte of value '0'.', By null-terminator you mean 0 correct? If 0 marks the end of a string, shouldn't it be the final character in a hash? Why are you saying it's a no-go? (There are 0s in the hash I listed: 2c35d9e70306b3696fed52eef483e259)

    So basically I'm stuck with regular TCP then for download verification? Nothing better to ensure what is getting saved from the ECC desktop directly to the NAS (as the default download location) makes it there safely?

    That does sound difficult :/ Guess I don't need to take note of any of these options since mptcp wouldn't be beneficial to my use, or is Winpcap something I could utilize?

    Thanks again for another explanation and example. While that example won't apply for my use, I could see how Linux or BSD could be superior in this aspect. I too would prefer another choice instead of Windows to handle all my data, I've read and heard that it is definitely not the safest route for servers from many sources.

    Ok, so I got that OpenSSHDd is on the server, but if SSHFS is for the client pcs, and my client pcs are going to be Windows machines, I'd need the Windows implementations of these like WinSshFs, not the actual SSHFS? (SSHFS doesn't work on Windows, it's for BSD and Unix right?)

    Hmm that does sound beneficial, especially the protections from malware, remember that the gaming rig will rarely be browsing any websites though, only times will be to update drivers or programs. So will it still be at a risk of malware/viruses? Here's something I don't fully understand, to get infected with malware or a virus, do I actually need to go into a malicious site, or just having an internet connection is enough for it to come in cause it has a 'door' to use if it needs to?

    Unfortunately I'll be connecting the gaming rig to the server more times than I originally thought. Remember when I told you I did the testing around with the HDMI ports, well anime art looks nicer on the Full 0-255 range which the gaming rig is on, so although I'll be saving the art with the ECC desktop, when I actually want to sit down and enjoy viewing it, I'll be doing that from the gaming rig. So, some questions about this:
    1. If the gaming rig is just viewing them (these folders will be set to read-only for the gaming rig), no worries that it can corrupt something while viewing since it's non-ECC right?
    2. Is there some way I can connect the gaming rig to the server with without having an internet connection where something malicious could come in from? You know maybe have my own intranet or something? LAN? Directly by UTP?
    3. If using FileZillla or WinSCP, could I also view my anime art and some gaming videos with it as if I'm using Windows normally? Or are they just to move data across from one place to another, and not function like Windows Explorer?

    There are lots of points of very good advice in this quote. I fully agree on most of the things you mentioned here. While it will be more cumbersome having to restrict access, it will be worth it for integrity's sake and safety.

    Definitely have regular offsite backups, those will be offline for most of the time so malware can't easily tamper them, and they'll be the last line of defense as we like to call it.

    I'll put more thought and careful consideration into what machine has access to which folders once everything is set up, cause yes as you stated, although annoying, very worth it.

    That's good to know. So even if I run virus and spyware/malware scans on the recordings before I upload them to the server these 'bugs' can still go undetected?

    One other thing to note, apart from gaming recordings, I'll be backing up my games into the server once the game is complete. Can't go for symlinking anymore now since I have to keep disconnecting the internet connection before gaming due to the Meltdown/Spectre updates performance hit. Games are executable-files correct, so malware could tamper with those if not the recordings/media-files, or is it the same risk and games are a different type of executable?

    No I don't use standard Windows Media Player. I use CCCP (Combined Community Codec Pack which uses MPC-HC (Media Player Classic Home Cinema)), VLC, or rarely PotPlayer when VLC has trouble with certain files (usually something in HEVC gives me trouble on VLC).

    Sandboxie sounds interesting, and you had also recommended this program before. Will I need to use Sandboxie only for media playback, or also when viewing anime art that I already have saved, saving new anime art from the web, updating spreadsheets etc? If it creates an isolated session, will it still manage to save the art to the default download location in the server? (since it's isolating itself from it?)

    That could take quite a while then, even on my current 7TB of data, let alone once the server starts filling up more and more! Well I guess I'd want to generate a hash for everything I have, makes the most sense right to make sure everything that is stored on the server stayed in 100% integrity as when I originally saved it.

    Got it, so no actual copying will be done when I'm running these one-liners on all of the data, this just happened due to the example being only one file?

    Agreed :) What you call FUN though, I call heart-wrenching terror, especially if it's in code lol, and especially till I see that I'm happy with the end result :p But you put your mind to it and start getting to grips with it. This was the same (but simpler I think) when I got into recording gameplay, at first it's x264 vs using the GPU, then all of the different features like Constant or Variable framerate, CRF (quality) and x264 encoding preset speeds (veryfast, superfast etc), Lanczos vs Bilinear vs Bicubic, etc. Was a lot to take in, and lots of testing and experimentation to be done to get the desired quality of recordings I have now, but it was all worth it in the end, and I'm sure that the server with ZFS and RAID10 (love saying that) will be sooooo worth it xD

    Indeed it is, as we said although it has it's shortcomings it's still an amazing operating system for lots of other uses. Yes I agree with you on how small the server share is with Windows, which is not surprising though when it lacks some of the nicer features Linux has.

    Lol, this is easier??? We've skipped a 100 posts talking about the subject :p Interesting to know, I thought people picked Linux more due to the extra features which Windows does not have implemented. Yep, collective opinion, the tried and tested method, all plays a part into what people will choose and opt for.

    Thanks once again for all the help and information, especially that Search-by-Image from Google, not only are you informative on data, but also on many other workarounds.

  5. A2Razor

    A2Razor Master Guru

    Likes Received:
    ASUS R9 Fury X
    > By null-terminator you mean 0 correct?
    Yep, a zero. The value 0, and not the character representation of "0".

    > If 0 marks the end of a string, shouldn't it be the final character in a hash?
    Final character of every string is typically a zero (a terminator). However ... the hash itself might contain a zero for a byte (IF it was byte-coded), thus why the hex-representation is used instead. [eg, to permit storage inside of a string -- for a text-file, on a website, and so on]

    While that "0" (as in the character "0" in a string) is legal in a hash (hex-digit zero), the representation of "0" won't actually be stored as the byte value zero inside the string {0x00, \0}. --This may sound a bit confusing at first and so it may help to look at an ASCII table or the UTF-8 encoding (of numeric values specifically). One thing that's universal of all text-encoding is that null terminator, value 0, is reserved / special meaning and this makes it hard to just store bytes in a string (unless the string is fixed-length / doesn't use a terminator).

    --If that was a string containing "just the hash", it might look like this "2c35d9e70306b3696fed52eef483e259\0" with a NULL at the end.

    Yep, as long as you're downloading using regular HTTP then you're relying on just TCP's packet checksum. Sadly this has not changed very much over time ... we've just been relying on stored hashes on websites, even today. Even under the hood of game downloaders, you'll find that most of them just use "HTTP" for the game's downloading. However, at least now some of them automate hash-checking and have a verify feature to make sure the content is intact.. --Yet ultimately this checking is no different than tools like MD5Summer in functionality (they just are fetching an extra hash-file [again over just "HTTP"] which contains hashes for each downloaded game-file)

    From the winpcap site:

    --winpcap is more of a library for developers than a standalone tool. You'll find it used in software such as Wireshark (packet capture / analysis tool), but it could also be used to implement your own TCP-stack. pcap provides an easy way to "send & receive raw packets" from a user-mode application.

    Right, you'll have to look into either win-sshfs or the commercial products that do similar. The main SSHFS project won't run under Windows (Unix like OS's only), and there's no ports of it for Cygwin either to my knowledge.

    This one is an "it depends". If your Windows machine is locked down, all ports restricted, not running software that maintains any connections to the outside (which means auto-updaters all OFF), firewall is ON / no inbound connections are allowed AT ALL. Then your PC in this "locked-down state" cannot be infected without you manually initiating an action that lets the malicious code in. That action might be insertion of media (like a USB drive), running a program, an auto-updater (which then happens to connect to a compromised server), or just browsing the web (malware getting in through "bugs in the browser or its plugins").

    I suppose the way that I'd put this is that malware is not powered by magic, and hackers aren't magical either. On most news networks and what we hear of hacking, you'd assume that they are magic powered, yet to break in to a computer or to infect a computer requires some entry point (some vulnerability / way in). On a typical Home Network (with IPv4) and Internet Sharing, there's NAT (Network Address Translation) / your router is protecting you to a degree here [from the outside], and ontop of that you have Windows' built in Firewall [for protection from the inside -- PC's on your own Local Network].

    ^ Neither being used (a router, or a local firewall) guarantees that you're fully closed up, since alot of people run servers and punch holes in the process ... though it's possible for you to configure the machine and your network in this way (with no inbound connections allowed). Another way to look at this is that your machine and home network on typical "default" configuration are akin to a bunker. Of course it's inconvenient to be trapped in a bunker with no access to the outside, and it's often desired to open that door. eg, allowing your buddies to connect to a file-server (especially over the Internet), or to launch a game-server.

    -----So, we start off fairly-safe with modern OS's [limited services running, and or bundled with a Firewall] ... yet soon enough as people install more and more software, that protection is no longer foolproof (as we've opened ourselves up). The key to safety is always opening up as little as possible.

    Right, if the folders are read-only to your gaming rig, then there's no way that your gaming PC can edit, create, or delete files there. The files would be completely safe from tamper.

    --You're talking about having your machines on your network configured such that each PC can see the NAS, and at the same time they cannot communicate to eachother? eg, having the ability to choose who each computer can talk with.

    -Gaming PC can talk to NAS "ONLY".
    -Workstation PC can talk to NAS & router.

    ^ Most "managed" switches can do this, it's a feature called per-port VLAN. You'll definitely want to skim through their manual, sometimes they call it different things (per marketing)
    --The last that I've checked, the Netgear 8Port / GS108 was the cheapest model I'd seen that had this ability.

    Only to move data sadly.

    There may be other options there for viewing Anime such as KODI, which can stream files off of pretty much any type of file server. Thanks to access control / restrictions though, mounting your anime read-only should be a non risk. Your uploads folder (for game-videos) you could opt to leave unmounted and to do those with Filezilla if you chose to. --Always possible to use a mix of both mounting and non mounted transfer.

    Yeah, lockdown that bunker. :D
    --Ultimate way to keep data safe against malware and protect against ransomware, etc, is to not allow your files to be written to in the first place. To keep everything read-only unless writing it is actually needed.

    Key thing to remember is that Virus Scanners can only find "known malware". Zero-day / new threats, and simply "under the radar" or rare-threats (that there's no information on), won't be found until they're reported and studied. The more AV software that you run, the better the odds that you're going to catch anything simply due to more eyes, more virus definitions, more people involved in investigating malware.

    Ontop of that: Sandboxie is a very good idea to secure your media-player, as Sandboxie doesn't rely on heuristics or detection / rather it relies on "isolation" or containment (not letting the media player permanently write changes to your disks). Updating your players is also a good idea since the only way for 'code' to be executed from a media-file (remember, these aren't executables) is via a vulnerability (software bug) inside the player itself. [eg, macro language like Windows Media Player and opening web-links]

    If you backup your games on completion (not just savegames), then yes that would involve executable files and definitely there's more risk there. On the otherhand, I assume that you won't be using these games on anything other than the gaming PC, and you might be able to also "check" the game's files if a distribution network [such as Steam, uPlay, Origin, etc], was used in attaining them. The hash / integrity checking of any of those download networks would actually "find and purge" an infection if it got inside of the game's files. [via re-downloading any mismatch]

    --Maybe just make it a habit after restoring a game from backup (from the NAS) to run a verify first (where it's possible to).

    The more you can tolerate using Sandboxie, the better honestly. Pretty much anything can be run and even installed directly under it, though there may be some nuisance involved in updating software run from a sandbox. There's exceptions with some games due to anti-hack / cheat-protection (and in some cases just anti-debuggers -- rarely), though most games can be installed and run from a sandbox. I suggest not doing this for anything with online play, as it risks getting banned / red-flagging cheat-detection even if it works.

    --More or less I'd say you should use Sandboxie on your web-browser, email client, and Microsoft Office (due to Microsoft's record with Macro-Languages in Word and Excel) at the least. Doing this makes you much safer in that if you're infected browsing the Internet, and if your browser is isolated .. the infection is stopped there, can't spread, and is eliminated when the browser gets closed [sandbox wiped].

    You can set allowed locations for downloading files and also manually recover them [without having set rules] with Sandboxie. Sandboxie is a form of a HIPS (Host Intrusion Prevention System) in that "you" get to dictate where programs can and cannot modify your data.

    --What sets Sandboxie apart from most HIPS software is that Sandboxie doesn't just have the ability to block access, it also can "simulate full-access" in a confined space. Full-access is simulated for files with an Overlay FileSystem (similar to OverlayFS in the Unix world), and Sandboxie does this for everything else too (memory mapped files, pipes, even administrative rights in confinement).

    That's one way to do it! Though as you get the hang of access rights, you'll probably do this by specific folders down the road. (or by both) Probably you'll be most concerned with if a PC is writing to some folder without your knowledge, since almost always this implies something afoul / malicious behavior.

    Right, I created a file and copied a file just to have "stuff" in a folder that I could generate hashes for (for the example).

    You'll find that I'm actually a noob when it comes to recording videos and doing any splicing or editing work. Yet I also haven't ever had a reason to record or edit much yet either... I think alot of it is just finding the incentive / purpose to learn what you need to really. For me, myself, it took losing ALOT of code before I started taking backups to the anal level. (RAID / NAS, external backups, online backups {on remote servers}, burned CD-backups, etc)

    Alot of this is actually what triggered my learning more and more Linux, I'm still primarily a Windows developer. Though as a whole it wasn't "Windows 10" but rather long-long before that, even before the Windows 7 days it was that desire to build exotic RAID setups that I found I just couldn't do with Windows. But yeah, once you have a reason to look at other OS's [just as with Mac in the old-days for image-editing] you move further from Windows in that you start realizing Windows just isn't best at everything.

    Today I'm at a point where I have more Linux and BSD PC's than Windows ... though really I think it's more that I just have found the beauty of each OS for what they're good at (rather than favoritism). It's hard to beat Windows for gaming, it's hard to beat Linux & BSD for servers, media-PC's, email, browsing, general workstations, etc. Don't need that many "gaming" machines, yet games are just one aspect of their use.

    --Oh yes, it's easier. HAHA! It may not be apparent, but ask just about anyone configuring a VPS or a dedicated-server in a datacenter what they'd like (for just about any server role -- {Windows, or Linux?}). Pretty close to 100% of people in the web-hosting field are going to ask for "Linux", not just because it's cheaper or that the install will use less-space, but because they'll finish faster. Install-faster, update-faster, configure-faster.

    You can install and configure Linux with Apache (for example) in literally "minutes", start to end of installation, and only occupying a few hundred megabytes of disk-space (for the entire OS). Meanwhile a Windows server with IIS? Heh, the Windows install alone [including updates] is going to take quite awhile. You'd be done setting up that Linux server before Windows even is installed and updated. ;)

    No problem, and sorry for the delay too. Things have been hectic here. Hopefully you've managed to find more originals on some of those images.
    Last edited: Apr 23, 2018
  6. 321Boom

    321Boom Member Guru

    Likes Received:
    GTX980 Ti
    Got it, thanks for the more detailed explanation, makes more sense now seeing that 0 could be a value of NULL, or a character representation.

    'as long as you're downloading using regular HTTP', does this mean there are other ways to download which are not just using regular HTTP (and will be more efficient/safer)? I'm keeping a lookout for the hashes on the websites as you stated, but these aren't available for all the data I download unfortunately, so it's not a foolproof method :/

    Can I use FTP to do my downloading? Is it safer/less chance of download corruption than regular HTTP?

    This sounds interesting, so by saying Wireshark is a packet capture/analysis tool, does that mean it will be handling the transfer of downloaded data on top of TCP? So it's a safer form/check of TCP?

    By analysis tool, is this something that regular TCP does too, or a feature implemented by Wireshark which makes it better than just replying only on TCP?

    Thanks, that cleared up some of the confusion knowing which programs go onto which OS (Windows or FreeNAS).

    If I set the gaming rig to 'no inbound connections allowed' would that mean I wouldn't be able to move any data from the server to the gaming rig (example games from the server to the SSD)? I understand that it's best to open up as little as possible, but there need to be a few things open to be able to use the PCs efficiently :/

    What do you mean 'your machine and home network on typical "default" configuration are akin to a bunker. Of course it's inconvenient to be trapped in a bunker with no access to the outside'. Does that mean I'm already set with 'no inbound connections' since I'm using the default configuration? That wouldn't make sense though because I do have access to the outside, I can connect to the internet, browse sites, etc :/

    Thanks for the confirmation and reassurance once again :)

    Yes very similar to what you stated 'having your machines on your network configured such that each PC can see the NAS, and at the same time they cannot communicate to eachother'. I need both the gaming rig, and the ECC desktop to see the NAS, but I definitely don't want the ECC desktop to see the gaming rig (or worse, the other way around, especially since the ECC desktop will have write-access to most of/all the NAS).

    The VLAN switches 'kind of' sound like what I'm aiming for, but not 100% sure they'll fulfill the purpose I have in mind, especially since I need the gaming rig to be completely without an internet connection almost all the time (it's no problem for me, I don't play online). What I'm thinking of is if I could have the gaming rig connected to the server in some way without requiring an internet connection to the gaming rig (which is why I suggested an intranet, or directly by UTP, don't know if these will work though). Remember, the gaming rig is with Windows 7, with Meltdown and Spectre patches disabled to avoid the performance hit, internet being toggled on and off with the Toggle-Internet .bat file we talked about earlier in the thread. I don't want the gaming rig having an internet connection to the outside (especially due to the Meltdown/Spectre patches disabled), but I would like it to have access to the NAS in some way (without having to Enable the Meltdown/Spectre patches again, restart the pc for the patches to take effect, then switch on the internet with the Toggle-Internet .bat file)(quite a tedious process having to do this every time when wanting to connect to the server, then disable everything and restart again when playing a game).

    In a more simplified way to look at it, I'd like the gaming rig to have no connection to the internet/outside world (it will only have an internet connection very rarely when needing to update drivers or something) but at the same time it could always 'see' the NAS as a mounted drive/external drive, so if I want to check out an old gaming recording that I stored on the NAS, or view some anime art I don't have to keep on enabling the patches again, restarting the gaming rig, etc.

    Got it, only for data transfer. Hmm that sounds like a good idea about using FileZilla to move the gaming recordings, that way there wouldn't even be a need for a folder with write-access from the gaming rig correct? Or would I still need this folder for FileZilla to have somewhere to copy the recordings to?

    I'll have to give it a try and see how it goes about mounting the anime and anime art as read-only since it could really make my life more difficult, especially when it comes to the anime art since I do move these around sometimes for better organizing. (I can't move folders if they're set to read-only right?)

    If a folder is set to read-only, can I add new files in there, or even that is prohibited (not just deleting and editing)? I have a habit of taking notes after watching an anime, and I put these notes in the corresponding folder of that anime, so it would be troublesome if I can't add these notes. (unless it's a simple process of switching from read-only to write-allowed for a couple of seconds till I put these notes in then switch back to read only).

    Since we're on the read-only subject, is the setting of which folders will be read-only something I will be setting from the NAS itself, or using the ECC desktop to set which folders are read-only (and setting for which PCs the read-only applies)?

    Also, if something is set to read-only, it wouldn't give me trouble/error messages when taking back ups with rsync and copying with TeraCopy to external drives?

    Sounds like more +1s towards Sandboxie then due to it's isolation. Seems like a very handy program to have which I will also start implementing. Thanks for suggesting it, this isn't the first time you've pointed it out, and I could see where it could be advantageous for many scenarios.

    Yep, talking about backing up the full game, since it will be English patched, and any other patches/updates installed to the game, makes it easier to just pick up and play again rather than having to install all that all over again. Yes only the gaming PC will be running the games, no need to ever execute them from the ECC desktop or NAS. Unfortunately I don't think I could check the hashes for the games, they're on cds, not a distribution network (remember they're old games). Unless there's a way to check a hash from a cd? But won't the hash differ if so since I would have added an English patch, and other update patches? (it won't be the exact same content as it was on the original cd, or am I confusing something here?).

    Awesome, even more +1s towards Sandboxie then. I'll definitely implement this, especially if it won't hinder the way I could save anime art and data (since it could be easily retrieved from the allowed locations where it could download). Makes perfect sense having an isolated session that gets wiped as soon as it's closed, so as you said, infection would be stopped there, and not get into my files. Thanks again for the suggestion.

    Thanks, this is very good advice. I think a full hash generation would still be more beneficial though to have a record of literally 100% of all the data that's stored on it (the integrity of ALL the data is important). I think doing it for specific folders is more for troubleshooting to determine from where the foul play is coming from correct?

    Yep, same here, usually there has to be something to spark that interest in order to start putting in the effort and learning. I've been playing games for 25 years, yet I felt the need to start recording 2 years ago when I picked up shmups (shmups are ridiculously difficult games, especially the Japanese bullet-hell ones, so it's nice having a recording of your achievement :)).

    Thankfully I've never lost any data in all these years, but I've always wanted a server (ease of access to your data, multiple mirrors due to RAID). Shame your experience wasn't similar :( I've heard in multiple instances that RAID setups are dealt with much better on Linux based systems, so you're not the only one suggesting this route. I understand and agree, it's not just favouritism, every OS has it's own strong suites and shortcomings, so you need to use each at what it does best :)

    Ahhh, you're a developer, no wonder most of this stuff comes naturally to you then and code doesn't scare you lol.

    Wow, I've read and heard that most people opt for Linux based system due to reliability and extra features, but didn't know it was easier as well! Two birds with one stone then choosing Linux for the NAS!

    Haha that's quite a difference having the Linux server set up by the time the Windows one is still installing! Interesting points to know. Thanks for the insight, as always.

    Sorry for the delay once again, as you said, it's really been hectic here as well. Finally found some time to sit down and reply since it's Saturday. I honestly didn't have the time to try the Search-by-Image yet, but I'm sure it will come in handy, even if it's not able to find all the images I need to bring up again, I'm sure it will be beneficial :) Thanks for all your help, and hope you have a nice relaxing weekend! :D
  7. A2Razor

    A2Razor Master Guru

    Likes Received:
    ASUS R9 Fury X
    --Sadly plain old FTP doesn't do anything for integrity checking.

    SFTP & SCP do provide integrity checking (via SSH), so those two are much safer than FTP or HTTP. HTTPS (secure) can provide decent integrity checking depending the cipher used. Though HTTPS is usually not mandatory on websites without explicit security requirements (where money is involved).

    Wireshark is for inspecting the communication of other software. It doesn't provide a protocol or do any transfers on its own, rather it lets you inspect what's happening under the hood.
    --Think of this like a debugger in the network-world.

    Outbound = connections initiated from the machine itself. (eg, acting as a client)
    Inbound = connections from the outside world in to the computer. (acting as a server)

    If a computer can only act as a client, and if there's no client software installed on the machine that doesn't "do stuff" without you giving an order ... then that machine would be untouchable & unable to be compromised unless a connection to the outside world is opened up from inside. Put another way, it could still be vulnerable, yet vulnerability would require human interaction. (left alone with no user the machine is "safe")

    Or rather, I assume that once you record videos you then drop them off on the NAS (in your storage pool), and thus there's no need for a local file-server run on the gaming-machine. When you go to edit these videos, then they're available on the NAS from your work-machine / workstation.

    The port-based VLAN on a managed switch can definitely do what you want here. That said, port-based VLAN would be an absolute hassle (nightmare) to change constantly / has no easy method to be changed on the fly in this case. [you'd have to re-flash the port configuration to the switch each time]

    --One solution to the problem of wanting to toggle Internet Access off "quickly" would be to just use two ports on the managed-switch, and assign one port as Internet Access "only", with another port as "NAS" only. You could buy a cheap 10$ NIC and shove a second network-card in the Gaming PC.

    --Another less complex method would be to delete and re-set the default gateway (as a toggle switch). Though this won't necessarily kill existing already-connected sessions, IMO.

    Here's an example what I mean by that:
    Port #1: NAS
    Port #2: Router
    Port #3: Workstation PC
    Port #4: Gaming PC (to NAS)
    Port #5: Gaming PC (to Internet)
    Port 1 VLAN rules: [1, 3, 4]
    1: yes, 2: no, 3: yes, 4: yes, 5: no, 6: no, 7: no, 8: no
    Port 2 VLAN rules: [2, 3, 5]
    1: no, 2: yes, 3: yes, 4: no, 5: yes, 6: no, 7: no, 8: no
    Port 3 VLAN rules: [1, 2, 3]
    1: yes, 2: yes, 3: yes, 4: no, 5: no, 6: no, 7: no, 8: no
    Port 4 VLAN rules: [1, 4]
    1: yes, 2: no, 3: no, 4: yes, 5: no, 6: no, 7: no, 8: no
    Port 5 VLAN rules: [2, 5]
    1: no, 2: yes, 3: no, 4: no, 5: yes, 6: no, 7: no, 8: no
    ^ This assumes an 8 port managed switch (generally the smallest that they come) with per-port VLAN tagging. The {yes / no} for each port is whether or not each port is allowed to communicate with the others. So, think of this like bridging those ports together, only this time you have control over which can send data to the others.

    EDIT / NOTE: You would need to put the NAS on a second subnet. Having two NICs on the same subnet is a no-no for Windows.

    So, you could have say:
    Code: -- {NAS} -- Gaming PC (NAS Access) -- {NAS} -- Router -- Workstation -- Gaming PC (Internet Access)
    -The NAS is a single NIC with two IP's, one on each subnet. The "Gaming PC" entries are one IP per NIC / port.

    Still need a folder with write-access for the gaming rig (to drop your videos in), yet the difference is that you won't be mounting this folder. Since you're not mounting the folder, there's no visibility (like there would be as a drive) to other software on the PC directly. This is the whole idea of security through obscurity in that it'd be much harder for any malicious code to write to the folder(s), as they would have to be programmed to be aware of servers entered in say "Filezilla".

    -- You (as the user) would of course know how to access that account & folder on the NAS to write your videos [with a tool such as Filezilla], yet malware on the machine probably has no knowledge of this. (which makes it somewhat safer if that makes sense)

    --Right, if you wanted to move around files you'd need to do it from the NAS (over an SSH terminal). Assuming that no machine has the permissions to do this graphically (drag & drop or cut & paste) then this would be a pain in the butt.

    You could change permissions pretty quick from the NAS / web-administration, but definitely not something you can do instantly or every few seconds (without that getting annoying fast). File "creation" can be allowed without deletion or modification, though sadly there's probably going to be a need to edit notes and go back / re-save already existing files.

    Different folders can have different write permissions though, so you might be able to still have different restrictions if you could live with a notes-folder (read-write) and video-folder (read-only).

    In the case of FreeNAS you'll probably do it from their accounts "web-GUI". The NAS itself won't actually have a desktop-environment (none installed / just a text-terminal), and the settings are intended to be done from a web-browser on another system [for the non-advanced stuff].

    --So I'd say: Remotely from your workstation over a web-browser.

    Nope. Backups from a read-only volume pose no problem to any tool of your choice. Copies from them with Windows Explorer, TeraCopy, xcopy, robocopy, rsync, etc, etc, will work just fine as long as you have write permissions to where you're writing. Nothing gets written in to the source-folder.

    Yeah, Sandboxie is truly awesome. It's just about the most secure sandbox isolation product on the market for Windows, worth every penny for a license, free for browser-isolation even without a license (single sandbox).

    --It may not be a full blown VirtualMachine like VMWare or VirtualBox, yet Sandboxie can isolate pretty much every known virus there is.

    Bear in mind that Sandboxie by default won't stop malware inside a Sandbox from reading data outside the Sandbox, yet the default behavior can stop malware from spreading. With manual configuration, it can also block reading folders. [eg, you can manually set folders that read-access is disallowed for] Expect some nuisances, and expect it to take some time to get used to and understand the whole isolation concept, though really really worth it.

    ^ The big VM softwares (VirtualBox and VMWare) block everything (since a guest OS can't read the disks of the host without special setup), and technically they're even more secure. You can even integrate a Linux Guest VM with a Windows Host, having Firefox run under the VM -- rendered on the host -- using something like VcXsrv, combined with PuTTy or Tunnelier (Bitvise client). Though this regulates you to Linux and is pretty inconvenient.

    Yeah, definitely both can be useful. Not always just tamper detection, although obviously great for this too. Those hash dumps of a folder are like a "snapshot in time" without taking up the space of copying everything and good when you want to see changes (deletions, creation, modification of files) after running any software-update. You'll probably find you use them not just for probing for tamper and corruption.

    --Though seriously nothing gives more peace of mind (per a sanity check) after suspecting infection to run a hashcheck of the Windows directory, user directory, etc, and find that absolutely nothing nothing has changed.

    Nah, Java, C#, Python, and other high-level languages absolutely terrify me (since they're gaining tracking in the business world). I like having full control, what can I say. :D

    Yeah, best example I can give here is that I restored an image of a Linux box from mid 2017. ~700 updates to install! Done installing in under 30 minutes, no SSD, lol. Imagine that type of update on Windows when an OS has a full year of updates to go through.... The difference of speed of updating compared to Windows has gotten absolutely insane. In Windows' defense, the Linux package systems aren't done transactional with rollback support like Windows is. On Linux distros, going back to old versions or rolling back packages usually means re-acquiring the old versions from their respective repositories.
    Last edited: May 12, 2018
  8. 321Boom

    321Boom Member Guru

    Likes Received:
    GTX980 Ti
    Right, so how can I incorporate SFTP and SCP into my browser? From what I could gather from our previous posts, Filezilla and WinSCP use SFTP, but didn't we say these are only for data transfer, so I can't be using them to browse the web with (and for saving/downloading new data)?

    Sorry I didn't quite get that and must be misunderstanding something. So Wireshark won't actually be handling my data transfers, it will just generate a log of what went wrong and right while saving new data from the web (anime art, etc)? A few posts above you stated Wireshark uses Winpcap. 'WinPcap is the industry-standard tool for link-layer network access in Windows environments: it allows applications to capture and transmit network packets bypassing the protocol stack, and has additional useful features, including kernel-level packet filtering, a network statistics engine and support for remote packet capture.' This sounds like it does handle the data transfer since it allows applications to capture and transmit network packets. (Sorry for my ignorance on certain matters, remember I'm just a gamer not a developer, so most of these things are completely new and unheard of to me).

    Got it, it's a bit clearer now :) Yes I assume for my usage I wouldn't need any inbound connections on the gaming rig then, because after moving the recorded videos to the storage pool/only folder with write access I could continue editing from the NAS.

    So even if the gaming rig is set up with no inbound connections, I could still view the data that's on the server? (wouldn't the server be considered 'a connection from the outside world?)

    So how would I go about setting up the gaming rig to not have any inbound connections?

    This is genius! That sounds like exactly what I need! So like this I could have constant access to the server from the gaming rig, while not having access to the internet, and just by pressing a button on the VLAN switch I could instantly switch access between gaming rig to NAS or gaming rig to internet, but never both at the same time! Brilliant! Thanks for the code with all the yes and no showing which connects to what, appreciate the time and explanation. Only issues I have with this are:

    1. Will having an extra network card increase DPC latency in the gaming rig? I know I'm probably being too anal, but this is very important since it's the machine that will be handling games while recording gameplay, so latency is the enemy here. The biggest offenders for latency in my system are ndis.sys and tcpip.sys (when checking with LatencyMon), as you're noticing both are network related. So, will having an extra network card increase the execution time for these 2 processes since there will be 2 network cards instead of 1? (or it's still the same process regardless of how many network cards could be installed?). Will having an extra network card add an extra process (like ndis(1).sys and tcpip(1).sys? (I know it sounds stupid, but just making 100% sure here.) Remember this is a gaming machine, so the less latency the better for added responsiveness, less spikes etc while gaming.

    2. To put the server on a separate subnet, something I need to take note of when buying parts, or something I could just configure software-wise?

    So from the above paragraph of having the gaming rig constantly mounted to the NAS with the VLAN switch, will this information about the write-accessed folder not being mounted still apply, or it is now void since the gaming rig is constantly mounted?

    Yes it makes perfect sense, I like the idea of 'security through obscurity' as you called it. I could understand how it would be more difficult for malware to access that folder needing Filezilla, rather than having easy access to it as a mounted folder. That's why it would be awesome if I could have the server mounted as read-only to the gaming rig (with no internet connection to the outside from the gaming rig via the VLAN switch), while also having the only folder with write-access (where the recording will be dropped) as a non-mountable folder. Could this be achieved?

    Uhh was afraid of that :(

    Yes there will definitely be a need to edit these notes as I'm always adding onto them with new stuff that catches my eye after repeated viewings/me finding out further information relating to the anime/art/character/game.

    I understand what you're getting at with the notes folder (read-write) and the video/anime art folders (read only), but it's not really practical for my use unfortunately :( I have loads of different folders, especially relating to anime art (sorted by character, from which game they are, etc), and it would be insanely more helpful having the note for that corresponding folder included with the art/video, rather than having to refer to the notes folder constantly to see if there even was a note to begin with! I usually name the note the same as the image, so when 'sorted by name' they both come up next to each other. I imagine it could get tedious pretty fast having to check the notes folder or using Windows search to find the note/see if I even took one.

    By 'could change permissions pretty quick from the NAS / web-administration', how quick is quick? Could I do this from the ECC desktop (through web-admin, that's the same as logging in to the NAS through a browser correct?) Seems like the best viable option :/

    The web-GUI you are referring to is the same thing I mentioned in the paragraph above (through web-admin, that's the same as logging in to the NAS through a browser)?

    To be clear, the image below, correct?
    So from the above image I could view all my folders and files as a directory, and from there select permission for each one from my ECC desktop?

    So let's say I wanted to edit a note of something that's in a read-only folder. In baby steps, I'd have to:
    Log in to the NAS through web-GUI from my ECC desktop like the above image
    Find the folder from the web-GUI
    Remove the read-only attribute from it
    Minimize the web-GUI
    Find the folder in Windows Explorer
    Edit the note
    Close Windows Explorer (I think it's better closing the folder from Explorer before changing it's permissions right?)
    Then just bring up the web-GUI again from it's minimized state and set the folder back to read-only?

    On a different note, when you said no desktop-environment, just text terminal, I'll be setting up ZFS and the RAID array using just the text terminal, or that could be done from the web-GUI?

    So when installing the extra programs for the NAS we were talking about like OpenSSHd etc, that will also be using text terminal, or through the web-GUI?

    That's very good to know, thanks. So no need to stay altering permissions when taking backups at least :)

    Yes I will definitely take the time to get used to it as it seems like a very worthwhile investment to make, both in time and money.

    'Bear in mind that Sandboxie by default won't stop malware inside a Sandbox from reading data outside the Sandbox, yet the default behavior can stop malware from spreading.' Why would it be a problem about malware reading data if it can't alter it? (I don't have any banking details, or a list of passwords stored on any of my pcs) Am I failing to see the danger here, or it's not dangerous behaviour? (It can't alter the files if they're read-only, and at best if it could just read them all it will see are gaming related stuff and family photos, nothing really personal that would put my life in jeopardy if someone else saw (unlike passwords or banking stuff))

    'With manual configuration, it can also block reading folders. [eg, you can manually set folders that read-access is disallowed for]', I assume this will come with a trade-off as usual, like some inconvenience for me to access them also?

    Why suggesting Sandboxie if VirtualBox/VMWare could block everything, therefore making them more secure? What makes Sandboxie better/higher recommended? VirtualBox/VMWare won't work for my intended use?

    Ok this sounds important and something that didn't cross my mind, so I'd generate folder hashes even for my Windows core files then, not just for my data?

    'Those hash dumps of a folder are like a "snapshot in time" without taking up the space of copying everything and good when you want to see changes after running any software-update'. This isn't clear for me (remember I've never run hash checking before), after a software update what folders should I hash check? I'm assuming you're referring to the Windows ones correct? (that's what software updates usually makes changes to, not stored data like images and games)

    Heh, so you are human after all :p Jokes aside, yep I fully agree with you on like having full control, OCD does that to you I guess, but I'm nowhere near as knowledgeable as you on the subject unfortunately :(

    Haha that is fast lol, if that was on Windows you'd be there waiting for hours! 'On Linux distros, going back to old versions or rolling back packages usually means re-acquiring the old versions', only time this would be necessary is if the new version is giving trouble or some incompatibility right? By re-acquiring, you mean simply finding the older version on their site and downloading it correct? (since Linux is free)

    I'd like to apologize for the delay once again. I really don't want you to think I've lost interest in the whole topic, and I greatly appreciate all your time, help and information. I want this server more than ever now with all of the stuff you've opened my eyes to, life just got really busy these past 2 months, don't know what happened all of a sudden :/ Miss those days where I just used to have a whole day locked in a room in front of my pcs :( Haven't had time to sit down and get a gaming session in 3 weeks! x_X (I work a 6-day week now to make matters worse). Anyway, enough of my banter, wishing you a happy gaming-packed weekend my friend! Take care, and thanks once again as always :)
    Last edited: May 27, 2018
  9. A2Razor

    A2Razor Master Guru

    Likes Received:
    ASUS R9 Fury X
    WinSCP and Filezilla are both GUI based and both have a built in file-browser. Though this browser cannot locally view files without first downloading them to a temp folder. They also can't surf the web, though they will open with sftp links.

    There are also some web-browser plugins such as FireFTP. -- if you want this integrated in the tabs of the browser and not launched externally.

    Example of the built-in file browser (Filezilla):

    WinPcap provides a set of tools, or a framework, that can be used to design a whole slew of applications. Both packet-capture (recording), and also transmission (composition). It's very possible to code your own TCP stack implementation in user-space ontop of WinPcap, and to use WinPcap in entirety for all communication, though that's not what happens with Wireshark. Wireshark just uses WinPcap for the 'packet capture'. It's intended use is to capture a log of everything that some software on your machine sends and or receives. The application being recorded is still doing all of the network communication -- Wireshark is just used to diagnose what went wrong.

    For instance, say that an application is just instantly disconnecting. Wireshark might shed some light in to why that's happening, such as if multiple corrupt packets are being received with mismatching checksums, or if packets are making it through out of order, or are forcedly being fragmented in a weird way by some "middle man" gateway.

    With the whole inbound and outbound thing, usually that's referring to the viewpoint of a firewall or {client vs server}. eg, who initiates the connection.

    Put another way:
    Acting as a server = inbound (eg, FTP server, HTTP server, etc)
    Acting as a client = outbound (surfing the web)

    Windows Firewall by default denies all inbound connections except for some rules that Microsoft creates, alot of firewalls behave this way. Unfortunately software can punch holes in Windows' firewall / add rules on their own as long as they have administrative rights... (so, Windows firewall may or may not actually provide protection)

    The stock Windows configuration on modern Windows' goal is to just allow acting as a client, unless the user (or software) opens up firewall rules for inbound. When you install games and other software, alot of them will (without asking you), create inbound rules (via registry).

    How do you stop that?
    --You could use a third-party firewall or a spare router (external router is the safest way to), or you just routinely check the Windows firewall rules from time to time to make sure no rules have been added without your consent.

    Bear in mind that you have a router restricting connections from the Internet, so this is only a concern of someone within your home connecting to your gaming machine. Of course, an infected machine within your home may also provide an attacker a way to attack from the inside (a backdoor). On public networks like at a university, I commonly recommend that people "double-NAT" and use a second router to keep their machine(s) safe.

    I have three NICs (Two Intel in a "NIC team", one AQtion 10gbit) in my main dev-system, my DPC latency is still under 50 microseconds (at peak). The largest addition for my machine is caused by the SoundCard and VideoCard. Don't think I've ever had any problems caused by the excess hardware though, certainly not the NICs.

    I doubt this will be a problem as long as you pick a good brand-name NIC. Intel, Broadcom, etc.

    Purely a software thing, all modern OS's have the ability to bind a NIC to multiple IP's / on different subnets. :)

    It would still apply from the security stance, looks like you grasped this from what followed.
    --Non mounted folders are very very unlikely to be written to by malware. Malware can still end up on them from manually being copied (by the user), yet this generally stops the spread of infection on its own, so it's always recommended where you can do things that way.

    Yes, this can be achieved easily with different users / accounts for those mounted volumes. One user (gaming machine) can have read-only access to a folder, whereas another user (workstation) can have read-write access.

    You could lock and unlock folders in a few minutes (opening the browser, changing settings, applying them, altering files, reverting them) -- though this is just not practical. Probably you'll just have to have a trusted machine that's always granted write access to the folder containing your notes.

    --IMO, if really concerned my suggestion would be doing the management of your videos archive from a tool like Filezilla. Keep the mounted volumes read-only, and rely on the security through obscurity in that no software on any machine has the direct rights to delete, touch, edit, tamper with, etc, your precious contents from the mounted network volume.

    If you use Filezilla's edit-feature, Filezilla will transfer the file automatically to a temporary folder (local), open it in an editor (such as Notepad), and when that file is "saved" it will automatically re-upload the changes to the server. This is less inconvenient than it sounds, and at least provides some level of security against malware.

    (eg, something deciding to replace your precious pictures with images of "squid" -- that actually is a real malware.....)

    Yep, that FreeNAS web-GUI is your central hub for everything. Creating your ZFS arrays, rebuilding them, creating user-accounts, setting account permissions for folders, etc, is all done from there.

    More or less, yes. As you're already realizing, that's simply too much work to do every-time. You'll have to consider some other option like the above. eg, keeping all mounting read-only, and using a read-write account in Filezilla or WinSCP.

    --Web-GUI can do this.

    All this comes pre-installed and can be enabled from the Web-GUI. FreeNAS supports Windows Shares, FTP, HTTP, even SFTP out of the box without the installation of additional software.

    VMWare is very heavy in both disk-space and memory use. You're basically running multiple Operating Systems on the same machine, each with their own space (RAM / DISK) reservation. Sandboxie on the otherhand is very lightweight, and still pretty effective. The isolation from VMWare or VirtualBox will certainly be MUCH stronger in theory, yet Sandboxie is pretty-good while having almost no performance hit, and without being too inconvenient to use.

    The danger of read-access is information like banking, email access, account-access for games, etc. You basically should try to keep some separation of roles of machines and data, so that machines that are a risk of compromise are less likely to have access to anything sensitive. This is easier said than done, so it's often best to just close off software and roles as much as possible to prevent something from slipping through the cracks (something you don't think of).

    As you become increasingly paranoid, YES. Snapshots of common points of infection (like the Windows folder) are extremely useful to look for infections. Of course, to make these snapshots and use them in any meaningful way, you have to get control over the Windows Update process -- something that's become increasingly hard with Windows 10. Microsoft, sigh ... heh.

    Being human might be up for debate, haha. OCD is definitely a good-thing though, since it provides you that drive to go deeper and deeper to the extreme ... and that always leads to a better outcomes long term it seems. In software (stability / performance), computer security, storage, or anything else really.

    Good-enough might work for some people, but good-enough certainly isn't best.

    Yep, if something is wrong with a new package (which does happen). eg, new bugs. You might have to rollback. Re-acquiring just means re-downloading and re-installing the older packages, yes.

    No worries again, as you'll notice we're all busy here. I'm split between gaming, reading the forums, playing with streaming services, work, and also working on new features / testing "YAP".

    Which speaking of "YAP", as a shameless plug: I do need more Windows 7 testers for. Since you use Win7 on your gaming rig, and you have multiple PC's, you're an ideal candidate there.
    Last edited: Jun 10, 2018
  10. DAW40

    DAW40 Master Guru

    Likes Received:
    EVGA 1060 6GB
    You cant overclock a server and these server CPU's and boards are cool for having dual CPU sockets, But at end of the day the threads are going to be slower then say a consumer workstation, since the consumer can overclock to 4Ghz plus. Where Xeon servers are usually 2.6Ghz and what not.

  11. A2Razor

    A2Razor Master Guru

    Likes Received:
    ASUS R9 Fury X
    Actually certain Xeons (in the workstation family) have unlocked multipliers (** despite not being listed as such **) and run at the same base clocks as the consumer high-end parts. Most workstation boards won't allow raising their multi, though. Chips wise, in the first gen i7 range -- the Xeon w3580 (QuadCore 45nm, comparable to the i7 975) and w3680 (HexCore 32nm) are both unlocked / can be OC'd. The E5 1650 series is also unlocked.

    --You're definitely right though that these aren't intended to be overclocked and probably shouldn't be. It makes no sense to buy a chip for stability and then risk ruining its stability.
    DAW40 likes this.
  12. DAW40

    DAW40 Master Guru

    Likes Received:
    EVGA 1060 6GB
    Yes my friend, stability would be a major issue and it doesn't help you can't change the multiplier. Thanks
  13. 321Boom

    321Boom Member Guru

    Likes Received:
    GTX980 Ti
    Would it really be necessary to browse images and videos with Filezilla? If it needs to download all the images/videos to a temp folder before I could view it I imagine it could get quite tedious when viewing anime art / gameplay recordings, mostly the anime art since I have thousands of images all in several different folders, and from your screenshot it doesn't look like it shows thumbnails. I get that it's very useful for data transfer though, especially paired with the non-mounted folder with write-access on the gaming rig for transferring the recorded videos, but can't see it being helpful from the viewing side.

    Yep something like FireFTP was what I was referring to. Know of a good alternative for Chrome though? I found one called sFTP Client, but don't really like the fact that it has 2/5 stars: https://chrome.google.com/webstore/detail/sftp-client/jajcoljhdglkjpfefjkgiohbhnkkmipm?hl=en-GB

    Right, so it sounds like Wireshark is mostly an analyzer/troubleshooter, and won't really be handling my data transfers? Will it really benefit my intended use? (remember my intended use is to have the server act like a huge external drive where everything is stored, not really going to be doing anything advanced on it, running applications from it, etc.)

    So basically all I need is another router between my main router, and my server, ECC desktop and gaming rig will all be connected to the VLAN switch which in turn will be connected to the new router?

    'this is only a concern of someone within your home connecting to your gaming machine', it doesn't matter if someone connects to the gaming rig, it's the server I'm mostly interested in keeping secured. I wouldn't like anyone from home tampering with the server either though, the lesser people have access to it, the more chance everything stays safe.

    That's good to know, thanks :) So how many NICs do I need? One extra one for the gaming rig, one for the ECC desktop, and one for the server?

    The more I think about it, the more I think that I'll end up needing to build another server a few years down the line after I finish this one. So, will I need another NIC in the ECC desktop for the 2nd server, and also an extra NIC in the gaming rig? Just asking now to make sure I pick a motherboard with enough slots for the extra NICs if so. Or can I buy a NIC with more than one port (so I can connect more than one server on the same NIC)?

    Btw, got a slight problem. . . the GTX980 Ti in the gaming rig is covering both the PCI slots (it's a micro ATX board) :/ How could I install another NIC in the gaming rig? Thinking of solutions to this apart from changing the motherboard, there are USB NICs, but these wouldn't give the same performance as a PCI NIC correct? Would this (https://www.amazon.co.uk/dp/B0726117Y9/ref=psdc_949408031_t1_B00QA1QBSS) work? It's a PCI Extender Ribbon (assuming the connecting end could fit between the motherboard and the GTX's fans).

    Good to know, thanks :)

    Got it, I really see the importance about the non-mounted folders over our posts. Thanks for all your time in explaining it and teaching me about such an important feature.

    Glad to know, thanks for confirming it :)

    In my case my trusted machine will be the ECC desktop, but the ECC desktop will also be handling the web-surfing and downloading, so that's what worries me.

    The edit-feature in Filezilla sounds brilliant! Kind of like what I need. In a way it acts like it will by pass the read-only attribute by allowing me to do changes to the notes and re-uploading that new copy to the server in place of the old document. I'm assuming the freshly updated file on the server will automatically have the read-only attribute that the previous version of the file also had.

    Thanks for so many workarounds, you really know your stuff to make this a more convenient process.

    Yes we mentioned that squid before in this thread, and that really is terrifying, so better to be safe through obscurity than risking something like that.

    Glad to hear that, at least it's with a GUI haha xD

    Yep agreed, the edit-feature from Filezilla sounds like the most viable option.

    Could easily see how useful a tool Filezilla is going to be for the server, both in handling data transfers, and in allowing changes to read-only material.

    That's awesome, sounds like a complete all-in-one package. They really thought it through properly huh.

    Got it, so being lightweight and convenience are Sandboxie's strong suits vs VMware/VirtualBox. I might have to consider VMWare or VirtualBox if their isolation is more secure though, depends on how inconvenient they are to use :/ I don't keep anything banking related on my pc, so I should be quite safe in that regard.

    Yep I believe in separation of roles where possible, I already do that now, where keeping anything gaming related on my gaming rig, and other casual stuff on my laptop.

    Oh God I hope I don't become even more paranoid haha. That is a shame about Windows 10 making it harder to have full control :( Long live Windows 7! xD

    Haha good one. Yeah I agree that OCD has it good points where it makes you strive for more, though at a cost of your mental health, energy and sanity lol.

    Good to keep this in mind. Thanks.

    Sure thing I'll be glad to help you, it's the least I can do after you've opened my eyes with all this information about the server and constant support! Sent you my contact details on PM ;)

    Take care and have a great weekend, I finally managed to sit down and get a gaming session yesterday! xD Felt so good after such a long time :D
    Last edited: Aug 3, 2018

Share This Page