Quill-shaped cirrus cloud. Photo by W.carter.

Backing up Windows “Users” folders with Syncthing

With CrashPlan getting out of the home backup business, I found myself shopping for a new backup solution.  As replacements, CrashPlan was pushing either Carbonite or their own plan for small businesses… but either of those would have at least doubled the cost of backup.  Thus, I started to look at setting up a remote Raspberry Pi backup server myself.  A friend pointed me to BitTorrent Sync (now called Resilio Sync) which, in turn, led me to their open-source competitor, Syncthing.  Two things ultimately led me to choose Syncthing over Resilio: (1) open-source review of their security, and (2) a post by Jaime Jiménez, Running Syncthing or Resilio on a Raspberry Pi 3.  Also, as just a bonus, Synthing offered a more supported solution to also backup my legacy Windows XP gaming machine (something CrashPlan stopped allowing a couple years ago, and Resilio has outdated support for [but still currently possible]).

What this post is not:  This post is not a guide to setting up Syncthing on a Pi as a backup service.  For starters, Syncthing does not claim to be the ultimate backup service (but I figure less-than-ultimate is better than nothing).  Thus, setting up a Pi as such is something you do at your own risk and there are plenty of other posts to help you with that.  I had to read and combine several posts to build what I needed and probably wasted far too much time on it (although, most of that was in building up the initial backup, which had to be spent one way or another after CrashPlan’s bail).

So, now, finally, to the point: Getting Syncthing to behave with “Users” folders on Windows 10 (actually, Windows Vista and later)… again, at your own risk.

Everything here is as of Syncthing v0.14.45.


The easy(ish) way

The easy(ish) way is probably the way the Syncthing developers would like you to do it.  That is, if you insist on taking shortcuts like I tend to on rare occasion, then this is the less abusive of the shortcuts taken here.  This easy(ish) method is how I set up my wife’s laptop — which acts as no more than just a browser, word processor, and photo archive.

BTW, for those who are wondering what “easy(ish) versus easiest” might be, the easiest way, is to create separate sync folders for every user’s Documents, Pictures, and other folders, etc.  Easiest… but also a bit monotonous (and also the only recommended approach as far as I could find).

But, when I was using CrashPlan, I was much lazier than that and wanted similar laziness with Syncthing in the long run.  In CrashPlan, all I did at first was set it to backup the entire C:\Users folder and I was done on the simplest machines (CrashPlan probably got tired of such abuse).  Later, I revised that a bit by setting it to ignore things like the browser and antivirus caches.

With Syncthing, however, you can’t create a backup folder out of C:\Users by default (Windows will prevent creation of Syncthing’s marker folder there).  This can be worked around if you know how to force Syncthing to do so but I’m going to ignore that since it’s not a great idea anyhow — and going one level deeper into each real user’s folder is not that much harder.

So, at long-winded last, what I did was create a sync folder for C:\Users\AwesomeUserOne, for example, and whitelisted which folders under that should be synced.  (This also means you can remove the default folder Syncthing creates for you: C:\Users\AwesomeUserOne\Sync .)  The downside here, however, is that you need to remember to update the whitelist if you install a new program that creates a new folder at the whitelist level (and all other “easy” methods have a similar downside).

Note that this guide also assumes the use of SyncTrayzor, currently the most supported interface and installer on Windows.  You don’t have to have it but it makes things a bit easier to grok at first.  (I expect I’ll stop using it at some point.)

  • Folder settings (on the device to be backed up):
    • Folder Label: MyDeviceName/Users/AwesomeUserOne
      • Note: it’s easier to label sync folders like this if backing up to a Linux device (maybe use backslashes if going to a Windows device) it makes setting the destination folder path easy
    • Folder ID: leave the random string unless you really want to change it
    • Folder Path: C:\Users\AwesomeUserOne
    • Share With Devices: set this later after getting everything else working
    • Rescan Interval: 86400 seconds  (1 day)
    • Folder Type: Send Only
    • Ignore Permissions: ??? they really need better info on this (I used to set this, now I don’t)
    • File Versioning: No File Versioning (I do versioning on the backup device only)
    • Ignore Patterns (button opens new window):
      • !/Contacts
        !/Desktop
        !/Documents
        !/Music
        !/OneDrive
        !/Pictures
        !/Saved Games
        !/Videos
        *
      • Add to or remove from that list as needed.  Don’t forget the “!” before the folder name (to whitelist it from exclusion) and the “*” must be on the last line (to exclude everything else).
  • SyncTrayzor settings (File >> Settings):
    • SyncTrayzor tab:
      • Start on Login:
        • [x] Automatically start on login
    • Folders tab:
      • Bug: restart SyncTrayzor if you don’t see your folder listed
      • Set your MyDeviceName/Users/AwesomeUserOne folder to [x] Watched
    • Note that setting SyncTrayzor to watch your files appears to be a more lean setup than having Syncthing rescan your folders often.  Syncthing has it’s own hidden feature for this (which I will get into below) but it is still in testing.  Ultimately, you need to decide for yourself which of all these update methods to use.  This is just what I am choosing to use for now.
  • If all that seems stable, and the file and folder count looks correct to you, go back and set Share With Devices to your backup device.
  • Folder settings (on your backup device):
    • Rescan Interval: 0
    • Folder Type: Send & Receive  (no such thing as Receive Only)
    • Ignore Permissions: ??? they really need better info on this (I used to set this, now I don’t)
    • File Versioning: your choice (I’m using Staggered File Versioning for now)
    • Ignore Patterns (button opens new window): none

Any settings I skipped either don’t exist yet, are up to you, or otherwise not worth mentioning.


The lazier or more paranoid way

The above method, the “easy(ish)” one, backs up your original content that you probably care most about without stressing Syncthing out much.  If, however, you are like me and also want to backup some of those magical goodies that you know are stored in AppData, .*, and _flooberjoober_, etc., such as your browser bookmarks or email backups (or just want to give Syncthing a good workout) then read on.

Note that this method does not require the use of SyncTrayzor (which can still be installed and used for its other features) and instead requires the use of the “advanced” features of Syncthing to turn on their test feature for file watching.

The primary difficulty of this method was getting both the Windows device (the device to be backed up) as well as the backer server device (the aforementioned Raspberry Pi) to chill out and not run around like hyper monkeys constantly.  By that, I mean, at first Syncthing was constantly running at 20-30% CPU on both the client and the server devices (and wasn’t leaving the disk much alone either).  After all, CrashPlan didn’t do that when I set it up worse then this, so why can’t Syncthing be chill as well?

I chilled out the client end by turning off SyncTrayzor’s file watching feature for these “Users” folders and switching to Syncthing’s built-in method (which defaults to collecting changes in 10-second bundles).  But chilling out the Pi side took a lot more fiddling and will be explained more later (sort of).

Many of these settings are the same as above:

  • Folder settings (on the device to be backed up):
    • Folder Label: MyDeviceName/Users/PowerUserOne
      • Note: it’s easier to label sync folders like this if backing up to a Linux device (maybe use backslashes if going to a Windows device) it makes setting the destination folder path easy
    • Folder ID: leave the random string unless you really want to change it
    • Folder Path: C:\Users\PowerUserOne
    • Share With Devices: set this later after getting everything else working
    • Rescan Interval: 86400 seconds  (1 day)
    • Folder Type: Send Only
    • Ignore Permissions: ??? they really need better info on this (I used to set this, now I don’t)
    • File Versioning: No File Versioning (I do versioning on the backup device only)
    • Ignore Patterns (button opens new window):
      • Here’s a sample of what most of my ignore pattern looks like:
      • (?i)temp
        AppData/Local/Packages
        OneDrive/logs
        // Firefox and Waterfox caches:
        AppData/Local/**/Profiles/*.default
        // NOD32 antivirus cache:
        ESET*/updfiles
        // Some programmer tool junk:
        npm-cache
        .nuget
        // GameMaker Studio junk:
        AppData/Local/gm_ttt_*
      • Add to or remove from that as needed.  It is not meant to be complete or anywhere near it.  I only targeted the worst of what I didn’t need that is either very bulky or changes often.  I don’t use Google Chrome much so I haven’t looked for its cache.  I use Edge a bit but suspect a “temp” somewhere hold its cache (but don’t remember at the moment).  Note that the interesting Firefox settings that you might want to keep are typically in AppData\Roaming, and not AppData\Local, but you should check that for yourself to make sure the above setting is what you want.
  • Advanced Configuration (Actions >> Advanced):
    • Folder “MyDeviceName/Users/PowerUserOne” (xxxxx-xxxxx) rollup:
      • Fs Watcher Delay (seconds): 600  (10 minutes)
      • Fs Watcher Enabled: [x]  (check it)
    • Note that the default delay is 10 seconds (which is probably fine in many situations) but a much lengthier time is needed here because of how overactive some of the files in AppData can be.  Even 60 seconds is not enough time for the Pi to catch up between updates (most likely due to the external HD running relatively slow over USB 2.0).  The default to a similar setting in CrashPlan is 10 minutes, so that is what I went with here.  Thus, even when doing nothing but staring at your computer screen, about every 10 minutes the Pi gets a little busy for 30 seconds or so on this folder as it tries to update/updates a few junk files, some which never fully sync.  But I think this is fine.  My primary goal here is to keep the Pi from overheating from constant high Syncthing use (which will otherwise easily hit 80°C+ in its current not-very-well-ventilated enclosure).
    • So, yes, with this setting, you could potentially lose up to 10 minutes worth of work in a worst-case scenario.  Considering that I keep my more important work elsewhere, this is acceptable to me… and would be generally acceptable even if I kept more important work in these folders.  Again, such things are up to you to figure out for yourself.
  • If all that seems stable, and the file and folder count looks correct to you, go back and set Share With Devices to your backup device.
  • Folder settings (on your backup device):
    • Rescan Interval: 0
    • Folder Type: Send & Receive  (no such thing as Receive Only)
    • Ignore Permissions: ??? they really need better info on this (I used to set this, now I don’t)
    • File Versioning: your choice (I’m using Staggered File Versioning for now)
    • Ignore Patterns (button opens new window): none
  • Note that a Users folder backup setup like this will always appear as “Out of Sync” on the server side — unless you are really peculiar about hunting down absolutely everything to create a long Ignore Patters list to avoid the files that cannot be synced.  These are files that are pretty much always open and constantly changing.  I haven’t noticed any in the list of out of sync items that I care about.

Any settings I skipped either don’t exist yet, are up to you, or otherwise not worth mentioning.

Troubles with tweaking, testing, tribbles, etc.

Note that if you add to the Ignore Patterns and want to remove those same files from the backup server, you are in for a bit of a headache.  You may rather choose to let your server stay dirty than go through this.  (Also, I was fiddling with many things at once to try to work out the performance issues while also streamlining the Ignore Patterns and cleaning the backup files, so it is not entirely clear what caused what… but I’m pretty sure this relates only to file changes on the server and not file watch tool changes.)

During all this testing and reworking I found I had to remove the sync folder definition from Syncthing on the server — not the data on the backup disk but just the setup in Syncthing — and then re-add it.  I tried this three times with three users across two devices and found no other solution than to remove and redo the folder definition on the server.

Rebuilding a sync folder like this on the backup server usually results in a long scan of all 30,000 files again followed by a long dummy resync of all 30,000 files again (even though no data moves).  It would be nice if that could be one process instead of two (or three, as sometimes there are two dummy resyncs: a faster one and a slower one).  And, somewhere in there (between the rescan and the dummy resyncs is best, if memory serves correctly) you will also have to click the red Override Changes button on the client side — likely multiple times — and a few more times over the next day or so to get the stragglers (because these are folders that never fully sync, right?).

As a side note, this is similar to pre-copying data to your backup HD and then syncing it — it can still save a lot of sync time and data I/O on the network to do this but you will still have to wait out both the rescan and the dummy resyncs — which could be another hour or two if working with, oh, 10,000 files or more (speed of scans and such seems related to file and folder count much more so than byte volume).

Also, rebooting a Pi server can result in a similar long wait where the Syncthing service will start quickly but, for some reason, it feels the need to rescan all 300,000 files/30,000 folders/200GB which can take half an hour or more.  It appears to try to scan all folders simultaneously with the smallest folders always finishing first — which feels random and inefficient versus scanning in folder or disk order (not that I know what it is really doing under the hood, I don’t) — compounded more so by Pi’s USB 2.0 limitation.  Thus, Pi may not be ideal in this sense even though it does run efficiently enough after this scan with my small user group.

The Windows XP version of the lazier way

The equivalent to C:\Users on Windows XP is C:\Documents and Settings.  Unlike Users, however, this folder can be backed up more easily in one shebang, should you choose to be lazy and do so.  You can take either the whitelist approach with it or follow the dark side.

On the custom GUI front, SyncTrayzor doesn’t run on XP.  Syncthing-GTK sorta runs on XP but the GUI always crashes on me after a while — even the tray app will eventually crash with the same GTK error.  Also, the file watcher that comes with Syncthing-GTK (Syncthing-inotify) doesn’t run on XP.  Thus, Syncthing-GTK on XP is a tiny bit useful for installing Syncthing and setting it to autostart on boot (which are not hard to do manually if you can read instructions), unless you have better luck with the GUI not crashing than I do.

So, anyhow, using primarily the web GUI for Syncthing on XP (which works great), here’s the lazy route I went with (very similar to “the lazier or more paranoid way” described above for Windows Vista and later).

  • Folder settings (on the device to be backed up):
    • Folder Label: MyXPDeviceName/Documents_and_Settings
    • Folder ID: leave the random string unless you really want to change it
    • Folder Path: C:\Documents and Settings
    • Share With Devices: set this later after getting everything else working
    • Rescan Interval: 86400 seconds  (1 day)
    • Folder Type: Send Only
    • Ignore Permissions: ??? they really need better info on this (I used to set this, now I don’t)
    • File Versioning: No File Versioning (I do versioning on the backup device only)
    • Ignore Patterns (button opens new window):
      • (?i)temp
        Temporary Internet Files
        // Firefox and Thunderbird caches:
        Local Settings/Application Data/**/Profiles/*.default
        // NOD32 antivirus cache:
        ESET*/updfiles
      • Add to or remove from that as needed.  It is not meant to be complete or anywhere near it.  I only targeted the worst of what I didn’t need that is either very bulky or changes often.  Note that the interesting Firefox settings that you might want to keep are typically in the higher level Application Data, and not Local Settings\Application Data, but you should check that for yourself to make sure the above setting is what you want.
  • Advanced Configuration (Actions >> Advanced):
    • Folder “MyXPDeviceName/Documents_and_Settings” (xxxxx-xxxxx) rollup:
      • Fs Watcher Delay (seconds): 600  (10 minutes)
      • Fs Watcher Enabled: [x]  (check it)
    • See notes earlier on these same settings.
  • If all that seems stable, and the file and folder count looks correct to you, go back and set Share With Devices to your backup device.
  • Folder settings (on your backup device):
    • Rescan Interval: 0
    • Folder Type: Send & Receive  (no such thing as Receive Only)
    • Ignore Permissions: ??? they really need better info on this (I used to set this, now I don’t)
    • File Versioning: your choice (I’m using Staggered File Versioning for now)
    • Ignore Patterns (button opens new window): none
  • Note that a “Documents and Settings” folder backup setup like this will always appear as “Out of Sync” on the server side — unless you are really peculiar about hunting down absolutely everything to create a long Ignore Patters list to avoid the files that cannot be synced.  These are files that are pretty much always open and constantly changing.  I haven’t noticed any in the list of out of sync items that I care about.

Any settings I skipped either don’t exist yet, are up to you, or otherwise not worth mentioning.

Good night, and good backup luck.

2 thoughts on “Backing up Windows “Users” folders with Syncthing

  1. I’ve tried your whitelisting method however I’m still getting a “folder marker missing” error as syncthing doesn’t have permissions to create the .stfolder

    Do you know what permissions do I need to set on the /User/%username folder?

    Like

    1. Syncthing should be able to create a C:\User\somename\.stfolder folder. Any app can. It cannot, however, create a C:\User\.stfolder folder. Thus, I would guess something else is wrong in the Syncthing settings. I don’t recall having to create these folders manually (although, you could always try that as a test).

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s