Hi all,

I'm looking into possible options for a mass storage device, preferably external, to use for keeping and backing up my graphics projects and assets. I've looked at external drives, NAS and/or other external HD enclosures. I can see merit in all of them, but I'm most interested in getting one that is the most convenient to work with.

My plan is to keep "active" projects on my main system drive for the sake of saving/loading speed, but then being able to copy/move them to a backup/storage device once I'm no longer working on it, in order to keep space free on my main working drive(s). As you know, HD space can disappear quickly as projects grow. I'm thinking 3 or 4 TB should be fine. I'm most interested in convenience and reliability. Speed isn't of utmost importance, since it'll be for backup/storage, and not regular use.

Anyway, thought I'd bring my search here to see if anyone has any recommendations, or can share their own experiences they can share?

Thanks!

I keep a major back-up to back up my system ... periodically. And I have a simple external drive that I save all my downloads, projects, images and renders to ... as well as major documents. When my computer died a couple years ago ... it was a blessing to have these.

My storage setup is:

2 development systems. Same OS and software stack on both. MacOS using brew to keep things up to date. One is a laptop and the other a desktop. Both are SSD so not as much chance of failure as an HD. I don't raid locally.

1. I use rsync to keep my content and app libs in sync. The content sync also de-dupes within runtimes to hardlinks (using jdupes.). brew maintains a brewfile that can be used to restore my complete software stack on reinstall of a computer (or a new one). The rsync gets me a kind of lazy "raid" as far as I need. I can move back and forth between them with the same software setup.

2. I only keep runtimes i'm using on either computer. All runtimes are sync'd also to local and remote server. I can clone a runtime if I need it. I split them off by character generation and/or category. It's unlikely that I would use PE and toon at the same time :)

3. I use git to maintain all my projects. Each project is its own local git repo. This works well for poser since they're text files. I have a remote repo on my local server and a remote repo on a cloud VM (both servers run ubuntu). So not only are all my projects in source control, but there are 2 remote repos (one off site). I can delete any local repo I'm not using. I can clone it back from either remote. I have a non-git "working/play/experiment" directory that I use rsync to keep in sync between the two machines.

4. For binary (image) assets I have a sparse diskimage that's in slices, making it much easier for rsync to deal with. It also is sync'd between machines and servers, local and remote.

There are some scripts that handle the mounting, de-duping, unmounting, syncing, etc. I just "work-sync" or "content-sync -a". I have some utility scripts like one that creates a custom folder embedded with a cover image for the project, usually the last image in the "renders" project, or a the "images" directory, or whatever is the last image it can find. Nice to look at a directory of folder icons and have a visual as to wtf is in there :D

The local server is ubuntu, AMD in a tower case, with 12x 4TB drives, 3 as 3 way mirror (double redundant) for homes, and the rest as raid6+hot spare. Essentially, I have to lose 3 drives in the mirror or 3 drives in the raid before I lose data. And the critical (git) data is still on the remote. Just not the runtimes.

When I got a new computer (replaced laptop which crashed and burned) it was pretty trivial to bring the new one back up with everything.

In my case, I do a lot of other stuff on that ubuntu server. If I didn't, I'd use a 3 bay 3 way mirrored 12TB drives JBOD external. That would be 12TB double redundant. The cost for redundant drives is nothing compared to the cost of reinstalling, rebuilding everything.

As my dad used to say... there are two types of people: Those who have lost data and those who will lose data.

I'm both. I've blown days (probably weeks) of my life, recovering from system failures over the years. No more. Let the record show, I eventually learn.

What kind of external drive do you use? USB, like a Western Digital "My Passport"? Or something else?

Boni posted at 8:05PM Fri, 06 March 2020 - #4382685

I keep a major back-up to back up my system ... periodically. And I have a simple external drive that I save all my downloads, projects, images and renders to ... as well as major documents. When my computer died a couple years ago ... it was a blessing to have these.

Wow. That's quite the setup. Sounds like it was fun to set up.

I'm thinking of getting a personal NAS. Just not the WD "MyCloud". Tried that before, and it was awful to work with. Looks like there's some better options out there.

thoennes posted at 8:06PM Fri, 06 March 2020 - #4382705

My storage setup is:

2 development systems. Same OS and software stack on both. MacOS using brew to keep things up to date. One is a laptop and the other a desktop. Both are SSD so not as much chance of failure as an HD. I don't raid locally.

1. I use rsync to keep my content and app libs in sync. The content sync also de-dupes within runtimes to hardlinks (using jdupes.). brew maintains a brewfile that can be used to restore my complete software stack on reinstall of a computer (or a new one). The rsync gets me a kind of lazy "raid" as far as I need. I can move back and forth between them with the same software setup.

2. I only keep runtimes i'm using on either computer. All runtimes are sync'd also to local and remote server. I can clone a runtime if I need it. I split them off by character generation and/or category. It's unlikely that I would use PE and toon at the same time :)

3. I use git to maintain all my projects. Each project is its own local git repo. This works well for poser since they're text files. I have a remote repo on my local server and a remote repo on a cloud VM (both servers run ubuntu). So not only are all my projects in source control, but there are 2 remote repos (one off site). I can delete any local repo I'm not using. I can clone it back from either remote. I have a non-git "working/play/experiment" directory that I use rsync to keep in sync between the two machines.

4. For binary (image) assets I have a sparse diskimage that's in slices, making it much easier for rsync to deal with. It also is sync'd between machines and servers, local and remote.

There are some scripts that handle the mounting, de-duping, unmounting, syncing, etc. I just "work-sync" or "content-sync -a". I have some utility scripts like one that creates a custom folder embedded with a cover image for the project, usually the last image in the "renders" project, or a the "images" directory, or whatever is the last image it can find. Nice to look at a directory of folder icons and have a visual as to wtf is in there :D

The local server is ubuntu, AMD in a tower case, with 12x 4TB drives, 3 as 3 way mirror (double redundant) for homes, and the rest as raid6+hot spare. Essentially, I have to lose 3 drives in the mirror or 3 drives in the raid before I lose data. And the critical (git) data is still on the remote. Just not the runtimes.

When I got a new computer (replaced laptop which crashed and burned) it was pretty trivial to bring the new one back up with everything.

In my case, I do a lot of other stuff on that ubuntu server. If I didn't, I'd use a 3 bay 3 way mirrored 12TB drives JBOD external. That would be 12TB double redundant. The cost for redundant drives is nothing compared to the cost of reinstalling, rebuilding everything.

As my dad used to say... there are two types of people: Those who have lost data and those who will lose data.

I'm both. I've blown days (probably weeks) of my life, recovering from system failures over the years. No more. Let the record show, I eventually learn.

My main storage for daily work is a Western Digital ... it's small and convenient.