Skip to main content

Raymii.org Logo (IEC resistor symbol)logo

Quis custodiet ipsos custodes?
Home | About | All pages | RSS Feed | Gopher

Set up your own truly secure, encrypted and shared file synchronization, aka Dropbox clone

Published: 15-10-2013 | Author: Remy van Elst | Text only version of this article


Table of Contents


TL;DR

This article describes my truly secure, encrypted file synchronization service.It used EncFS and dvcs-autosync which lets me share only the encrypted data andmount that locally to get the plaintext. It works on OS X, Linux and ARM linux.This article has setup instructions for all those platforms.

If you like this article, consider sponsoring me by trying out a Digital OceanVPS. With this link you'll get $100 credit for 60 days). (referral link)

Diagram

DiagramOverview of the solution we are building.

My data is in an EncFS encrypted folder. The unencrypted contents are availableafter unlocking the folder. The encrypted files are synced to an ssh server anto a few other machines and devices using dvcs-autosync. The encryption happenson my machines before the data leaves the to internet.

Preface

Recently I've had to stop using SpiderOak for my file backup and synchronizationacross machines. The main reason being that there is no ARM version of SpiderOakand the RAM usage was getting out of hand for me. And there still is no opensource client, sadly. However, my time with SpiderOak was good, I've paid for itand most of the time it just works fine.

But since I recently bought an ARM Laptop on which I also need my files, itbecame time to switch to another secure shared file storage. I have a fewdemands for such a service:

Then all current commercial services drop off, including SpiderOak, BittorrentSync and git-annex. This resulted in a clever combination of EncFS anddvcs-autosync. Because, in this day and age, you cannot trust any "cloud"provider with your unencrypted data. (And you can only trust those who say theydo it securely when they release there source code, wink wink Wuala/Spideroak).

Overview

I'll describe the steps and requirements needed to set this up first. Then weget started with the setup. First we'll set up the server. Then the first Linuxclient. If needed, steps are provided for adding other Linux clients. Theninstructions for OS X are provided. It is a little long, but if you want privacyand security a one time investment is required.

Requirements

Not mandatory:

Steps

So, lets get started. In about half an hour you have your own secure encryptedfile synchronization service.

Set up the SSH server

As said, you'll need an SSH server which will act as your central datarepository. Here your encrypted data will reside, and clients push and pullchanges to and from here. If you have a few laptops which are not on all thetime, this server makes sure all the clients have the most recent data.

If you don't have a VPS, InceptionHosting has good VPS servers for a niceprice. (Affiliation link). Digital Ocean also does a good job. (Alsoaffiliation link).

I won't cover the installation and setup of the server. SSH, a user account anda passwordless SSH key is all you need. Google can help you with the setup ofthat.

First install git:

apt-get install git

Now, go to your home folder and create the "repository":

cd ~git init --bare autosync.git

That's it. Now we are going to set up the clients.

rsync.net

rsync.net is a service which provides online storage available via rsync. Youcan use them instead of your own VPS server, I'll show you how in a moment. Whywould you store your data there, and why do I recommend them when this is anarticle about a truly private system? Because they use a standard open protocol,support git and don't require a local client to be installed. Dropbox forexample does. Any service which provides git/ssh access would be fine becauseyou are just dumping encrypted data there and rsync.net is one of the betterones to do that.

Add you ssh key to rsync.net:

scp ~/.ssh/backup_nopasswd.pub <user-id>@<server>.rsync.net:.ssh/authorized_keys

To add more than one key, make sure the first key is added like above, for eachsubsequent key use the following method:

 cat ~/.ssh/other_key.pub | ssh <user-id>@<server>.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc' 

See also their manual on ssh keys

Create a git repo on rsync.net:

ssh <user>@<server>.rsync.net "git init --bare autosync.git"Initialized empty Git repository in /data5/home/<user>/autosync.git/

Remember/copy that path to somewhere, we'll need it later on to add the gitremote.

That's it for now. Continue with the tutorial, when we get to the other gitparts where applicable there will be a part for rsync.net.

Read more on rsync.net git/svn support

Set up the Linux machine(s)

Install EncFS

This one is easy:

apt-get install encfs

This will automatically install and set up both FUSE and EncFS. It'll also putyou in the right user groups (FUSE).

Creating the secure EncFS folder

(Note that this is required only once on the first machine, not on all theothers added later on).

Now in your home folder, execute the following commands to create the Encfsfolders and set them up:

cd ~mkdir sharemkdir secureencfs ~/secure ~/share

The first option of the encfs command specifies the folder where all theencrypted data will be. The second options specifies the mount point where theunencrypted data will be. It will ask you a few questions. The first can beanswered with p. The second question is for the folder password. Make surethis is a long and strong password.

Output:

Creating new encrypted volume.Please choose from one of the following options: enter "x" for expert configuration mode, enter "p" for pre-configured paranoia mode, anything else, or an empty line will select standard mode.?> pParanoia configuration selected.Configuration finished.  The filesystem to be created hasthe following properties:Filesystem cipher: "ssl/aes", version 3:0:2Filename encoding: "nameio/block", version 3:0:1Key Size: 256 bitsBlock Size: 1024 bytes, including 8 byte MAC headerEach file contains 8 byte header with unique IV data.Filenames encoded using IV chaining mode.File data IV is chained to filename IV.File holes passed through to ciphertext.-------------------------- WARNING --------------------------The external initialization-vector chaining option has beenenabled.  This option disables the use of hard links on thefilesystem. Without hard links, some programs may not work.The programs 'mutt' and 'procmail' are known to fail.  Formore information, please see the encfs mailing list.If you would like to choose another configuration setting,please press CTRL-C now to abort and start over.Now you will need to enter a password for your filesystem.You will need to remember this password, as there is absolutelyno recovery mechanism.  However, the password can be changedlater using encfsctl.New Encfs Password: Verify Encfs Password: 

That's it. Try to create a file in the ~/share folder and you'll see theencrypted file show up in the ~/secure folder.

To access the folder any time later, use the above EncFS command to mount itagain.

We need to prepare the secure folder for usage with dvcs-autosync.

Make sure the folder is mounted:

encfs ~/secure ~/shareEncFS Password:

IMPORTANT! Make sure to remove the .encfs files from the secure folder beforesyncing. IF THESE FILES ARE IN THE SYNCED FOLDER, YOUR FILES ARE MUCH MOREEASIER TO BE CRACKED.*

You can also add them to the.gitignore file:

-rw-rw-r--   1 remy  remy   1.1K Aug 29 16:09 .encfs6.xmlecho .encfs6.xml >> ~/secure/.gitignore

Now continue.

Then:

cd ~/securegit initInitialized empty Git repository in /home/remy/secure/.git/date > ./dategit add dategit commit -m "Initial Commit"[master (root-commit) 3cc0fba] Initial Commit 1 file changed, 1 insertion(+) create mode 100644 date

Now make sure you have your SSH server domain name ready. For me, it issync.raymii.nl. Also have the autosync folder ready. For me this is/home/remy/autosync.git. We need this in the following command:

git remote add origin ssh://sync.raymii.nl/home/remy/autosync.git

If you are using rsync.net, you have to add an upstream like this, with the pathyou remembered from above:

git remote add origin ssh://<server>.rsync.net/data5/home/<user>/autosync.git/

This adds the SSH server as origin in git. Now push the first change:

git push -u origin master

The -u option also sets up the master branch as default and starts tracking infrom origin.

Install dvcs-autosync

On debian/Ubuntu this part is easy because you can install a package. On otherdistro's, follow the instructions here on the official repo.

So, to install the package:

apt-get install dvcs-autosync

Create an XMPP account

dvcs-autosync uses XMPP as a way to send changes to other online nodes. So, youneed a XMPP account. You can use your Google (talk) account for this, but youcan also create an account via Pidgin at services likehttp://www.swissjabber.ch/ or http://xabber.de. I have my own XMPP servernetwork, so for me that is solved. Make sure you have the username(you@xabber.de) and the password ready.

Set up dvcs-autosync

This is also quite simple. Copy the example file from here to~/.autosync and edit at least the following sections:

[autosync]path = ~/secure

Change the path to your freshly created secure folder (~/secure).

[xmpp]username = you@yourXMPPserver.tldpassword = Your_Passw0rdalsonotify = you@yourXMPPserver.tld

And change the XMPP account data. That's it. You can change more things, butthat is all explained in the config file.

Now, with all the above set up, start dvcs-autosync from the command line:

dvcs-autosync

You'll get a lot of output, which can be safely ignored when you experience noerrors:

DEBUG:jabberbot:Got presence: you@yourXMPPserver.tld/AutosyncJabberBot on MyHostName.tld (type: None, show: None, status: None, subscription: None)Could not load one of the supported DNS libraries (dnspython or pydns). SRV records will not be queried and you may need to set custom hostname/port for some servers to be accessible.INFO:root:pynotify initialized successfully, will use desktop notificationsINFO:root:Growl does not seem to be installedINFO:root:Watching path /home/remy/secureDEBUG:root:Checking/writing pidfile /home/remy/.autosync.pidWARNING:root:PID file /home/remy/.autosync.pid already exists, but no process seems to be running, removing file nowINFO:root:Using only XMPP notificationINFO:root:Ignoring files matching any of the patterns INFO:root:Adding list to inotify exclude filter: ['/home/remy/secure/.git', '/home/remy/secure/.svn', '/home/remy/secure/.hg', '/home/remy/secure/src/packages', '/home/remy/secure/src/java/openuat', '/home/remy/secure/src/csharp/sparkleshare', '/home/remy/secure/src/cpp/cross/keepassx', '/home/remy/secure/src/android/ipv6config']DEBUG:jabberbot:Registered command: helpDEBUG:jabberbot:Registered command: loginDEBUG:jabberbot:Registered command: pingDEBUG:jabberbot:Registered command: pushedDEBUG:jabberbot:Registered command: unknownDEBUG:jabberbot:Registered command: whoamiINFO:jabberbot:*** roster ***INFO:jabberbot:  ddg@gg.imINFO:jabberbot:  you@yourXMPPserver.tldINFO:jabberbot:*** roster ***INFO:jabberbot:bot connected. serving forever.

Now try to add some files, you'll see that they are automatically added:

DEBUG:root:Starting coalesce timer with 2 seconds until coalescing events for file /home/remy/secure/,fdgh4878rgHHDBa would occur (if no other changes happen in between)DEBUG:root:Resetting already active coalesce timer to new timeout of 2 seconds until coalescing events for file /home/remy/secure/,fdgh4878rgHHDBa would occurINFO:root:Coalesce event triggered for file /home/remy/secure/,fdgh4878rgHHDBaDEBUG:root:Considering file /home/remy/secure/,fdgh4878rgHHDBa, which has the following events recorded:DEBUG:root:   Event type=IN_CREATE, action=git add %sDEBUG:root:   Event type=IN_CLOSE_WRITE, action=git add %sINFO:root:Final action for file /home/remy/secure/,fdgh4878rgHHDBa: type=IN_CREATE, action=git add %sINFO:root:NOTIFICATION: Local change: Committing changes in /home/remy/secure/,fdgh4878rgHHDBa: git add %sDEBUG:root:Substituting cmd part %s with /home/remy/secure/,fdgh4878rgHHDBaWARNING:root:NOTIFICATION: Command failed: Command 'git add /home/remy/secure/,fdgh4878rgHHDBa' in '/home/remy/secure' failed.  Output:fatal: pathspec ',fdgh4878rgHHDBa' did not match any filesDEBUG:root:Substituting cmd part %s with Autocommit of file /home/remy/secure/,fdgh4878rgHHDBa changed on host localhostDEBUG:jabberbot:*** props = [u'jabber:client']DEBUG:jabberbot:*** jid = you@yourXMPPserver.tld/AutosyncJabberBot on localhostDEBUG:jabberbot:*** username = raymiiDEBUG:jabberbot:*** type = chatDEBUG:jabberbot:*** text = [Local change]: Committing changes in /home/remy/secure/,fdgh4878rgHHDBa: git add %sDEBUG:jabberbot:*** cmd = [localDEBUG:jabberbot:*** props = [u'jabber:client', u'http://jabber.org/protocol/xhtml-im']DEBUG:jabberbot:*** jid = you@yourXMPPserver.tld/AutosyncJabberBot on MyHostName.tldDEBUG:jabberbot:*** username = raymiiDEBUG:jabberbot:*** type = chat

Now you can also see with a git log that the files are added. It works!

To make sure it keeps running, add a cronjob:

crontab -e

Then add the following:

*/5 * * * * dvcs-autosync

This runs dvcs-autosync every 5 minutes. It sees when it is already running,then it does not run again.

Special steps for an ARM Chromebook

I have an ARM Chromebook with Ubuntu running in a chroot. In the chroot crondoes not run, so I have created a simple script to autostart dvcs-autosync onlogin. You can find it here on my github. Add it to your window manager toopen on login and you are also set.

Set up another Linux client

If you need to set up another Linux client, first install encfs and dvcs-autosync as explained above. Also, configure dvcs-autosync with the existingXMPP account and set up the dvcs-autosync cronjob.

Then, instead of creating an EncFS folder, clone the git repo with the encryptedEncFS data:

git clone ssh://sync.raymii.nl/home/remy/autosync.git ~/secure

If you are using rsync.net, clone the repository like this:

git clone ssh://<server>.rsync.net/data5/home/<user>/autosync.git/ ~/secure

Also remember to add another ssh key to your rsync.net account (presuming youalready have added an ssh key to your account):

 cat ~/.ssh/other_key.pub | ssh <user-id>@<server>.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc' 

Make sure to change the address and path to your own server.

Also create the ~/share folder:

mkdir `~/share`

Now you can mount the folder with EncFS:

encfs ~/secure ~/share

You can now also test if you create or remove files/folders on one LinuxMachine, they are also created or removed on the other Linux machine(s).

Prepare the OS X client

I also use OS X machines, so I want to have secure access to my files there aswell. Luckily, that is possible with the above setup. The tools required are abit more spartan in setup, but after setup is just as simple in use. You have tohave XCode and the Command Line Developer Tools installed.

Install OSXFUSE

Download the .pkg file from http://osxfuse.github.io/ and install it. This isneeded for EncFS. OSXFuse is the continuation of MacFUSE, that seems to bediscontinued. OSXFuse works on both Lion and Mountain Lion.

Install EncFS

Make sure you have installed Homebrew (from http://brew.sh/). We use homebrewto install a version of EncFS configured to use OSXFuse instead of MacFUSE. Whenbrew and OSXFuse are installed, use the following command to install EncFS:

brew install https://raw.github.com/jollyjinx/encfs.macosx/master/encfsmacosxfuse.rb

It takes a while to compile and build the required dependencies, boost forexample took 20 minutes on my 2012 Macbook Pro with an Intel Core i7.

When everything is installed and working continue to the next step.

Get the secure folder

This one is simple. Clone the git repository from the SSH server:

git clone ssh://sync.raymii.nl/home/remy/autosync.git ~/secure

If you are using rsync.net, clone the repository like this:

git clone ssh://<server>.rsync.net/data5/home/<user>/autosync.git/ ~/secure

Also remember to add another ssh key to your rsync.net account (presuming youalready have added an ssh key to your account):

 cat ~/.ssh/other_key.pub | ssh <user-id>@<server>.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc' 

Make sure to change the address and path to your own server.

Also create the ~/share folder:

mkdir ~/share

Install dvcs-autosync and dependencies

First clone the repo of dvcs-autosync:

mkdir ~/srccd ~/srcgit clone git://gitorious.org/~olivierg/dvcs-autosync/olivierg-dvcs-autosync.gitcd olivierg-dvcs-autosync

Now build dvcs-autosync:

python setup.py buildsudo python setup.py install

Now download and build xmpppy. Save the file fromhttp://downloads.sourceforge.net/project/xmpppy/xmpppy/0.5.0-rc1/xmpppy-0.5.0rc1.tar.gzto your home folder. Then extract and build it:

cd ~tar -xf xmpppy-0.5.0rc1.tar.gzcd xmpppy-0.5.0rc1sudo python setup.py install

Then download and setup MacFSEvents (inotify for OS X):

git clone https://github.com/malthe/macfsevents.gitcd macfseventssudo python setup.py install

With all the above setup you are ready to continue and set up dvcs-autosync.

Set up dvcs-autosync

This is the same as for the Linux client. Copy the example file from hereto ~/.autosync and edit at least the following sections:

[autosync]path = ~/secure

Change the path to your freshly created secure folder (~/secure).

[xmpp]username = you@yourXMPPserver.tldpassword = Your_Passw0rdalsonotify = you@yourXMPPserver.tld

And change the XMPP account data. That's it. You can change more things, butthat is all explained in the config file.

When you have set it up, start dvcs-autosync from the terminal:

dvcs-autosync

Now you should see all your files being synced. When it has caught up with allthe files, mount the EncFS folder:

encfs ~/secure ~/share

When you look in the ~/share folder now, you have all your files. Don't forgetto add a cronjob for dvcs-autosync:

crontab -e

Then add:

*/5 * * * * dvcs-autosync

With this all set up, you have your own, truly secure encrypted filesynchronization service! Well done.

Pitfalls of EncFS

EncFS does have a few disadvantages. For me they don't weight up to all theadvantages.

You can read a very good article about EncFS here. It explains all thepossibilities, but also all the pitfalls.

There has also been a extensive code audit of EncFS which resulted in someissues. Read this mailing list post to find out.

Race conditions

With multiple edits of a file, on different devices, then the file whicheverdvcs-autosync commits first is used as "master copy". The others receive an XMPPnotification, and incorporate a (5 second) wait time. When not online, as far asI've seen in the last three weeks of intensive usage, the when a file on theserver is newer it is overwritten with a pull.

If you have any questions, you can always contact me via email.

Tags: arm, articles, chromebook, cloud, debian, dropbox, dvcs-autosync, encfs, encryption, featured-two, file-synchronization, os-x, raspberry-pi, spideroak, ssh, vps