Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Folder Synchronization Utilities

Directory synchronization is the process of keeping two directory trees synchronized, such that changes made to objects in one tree are propagated automatically to the other tree.

Synchronization is different from mirroring options. You might have mirrored a company document directory to your laptop, for example, and then modified a document or two. Other people might have modified other documents in the same directory by the time you get back. With rsync, you'd need to reconcile the differences between the two directories manually or risk overwriting someone's changes. Directory synchronization tools like OFM managers (for example Total Commander) can sort out what has changed where, propagate the changed files. Specialized tools like Unison  can even merge different changes to the same file if you tell them how.

Old News ;-)

[Jun 18, 2008]  Project details for rdiff-backup

freshmeat.net

About:
rdiff-backup backs up one directory to another. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special directory so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup can also operate in a bandwidth- efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back up to a remote location, and only the differences will be transmitted. It can also handle symlinks, device files, permissions, ownership, etc., so it can be used on the entire file system.

Release focus: Major bugfixes

[May 13, 2008] Unison - file synchronization tool -- Ubuntu Geek

Unison runs on both Windows and many flavors of Unix (Solaris, Linux, OS X, etc.) systems. Moreover, Unison works across platforms,allowing you to synchronize a Windows laptop with a Unix server, for example.

Unlike simple mirroring or backup utilities, Unison can deal with updates to both replicas of a distributed directory structure.

Updates that do not conflict are propagated automatically. Conflicting updates are detected and displayed.

Unlike a distributed filesystem, Unison is a user-level program: there is no need to modify the kernel or to have superuser privileges on either host.

Unison works between any pair of machines connected to the internet, communicating over either a direct socket link or tunneling over an encrypted ssh connection. It is careful with network bandwidth, and runs well over slow links such as PPP connections. Transfers of small updates to large files are optimized using a compression protocol similar to rsync.

Unison is resilient to failure. It is careful to leave the replicas and its own private structures in a sensible state at all times,
even in case of abnormal termination or communication failures.

Unison has a clear and precise specification.

Unison is free; full source code is available under the GNU Public License.

[Jan 25, 2008] Directory Synchronize 0.92 by F. Gerbig

The tool written in Java

About: Directory Synchronize is a powerful, easy-to-configure tool to synchronize the contents of one directory with another. It can compare files by various attributes, such as size or modified-date. It can synchronize an unlimited number of directories, and has a detailed logging function.

Changes: It is now possible to keep multiple backups of changed or deleted files. Wildcards for the global log and wildcards regarding user information (name and home directory) have been implemented. Bugs regarding NullPointerExceptions in console mode and the bug that wildcards would be written back to the directory definition have been fixed.

[Dec 12, 2007] Folder Synchronizer Programs Review

programs reviewed

1. unison

   1. The developers state no new features will be added to the program.

   2. The gui is quite basic, and you will need to edit configuration files as well.

   3. uses the obscure OCaml language, so may be more difficult to customise if needed.

   4. Uses efficient rsync protocol to sync over a network.

2. xfiles

3. jfilesync

   1. needs Java 5.0, which will mean another download for most users.

   2. GUI has a few quirks - e.g. you have to click 'compare' to get it to show conflicts

   3. you can't launch a 'diff' program from within the GUI

   4. easy to use

   download here There is another program by the same name at http://home.arcor.de/mschierlm/sm-soft/jfilesyn-en.htm. The one reviewed is at the link above

4. directorysync

5. FileSync

6. foldermatch

7. HAS (Heatsoft Automatic Synchronizer)

8. Allway Sync

   1. No mailing lists, public bug database, or other user community.

   2. the only reviewed proprietary program with update detection

   3. allways sync is also known (or was previously known) as GoodSync

   4. note mispelling in name

9. MS synctoy

   1. Windows XP only

   2. detects rename

   3. uses the update detection algorithm for deletions only, not updates. Overwrites edits in the older file when there are conflicts

   4. requires .net to be installed

   5. 512Mb ram & a P4 is recommended

   6. requires you to download an additional program to 'validate' you have registered your installation of XP

   7. only allows a single pair of folders per profile

summary

There are some features that you would expect to see in these programs that are in none - for example being able to use a non re-writable media like a cd-rw (not in packet writing mode) to synchronize two machines. Other basic features (namely client server / update detection) are only in a few programs.

I think this is because it is easy to write a simple directory differencer, so many would-be shareware developers use it as 'My first Program'.

Unfortunately the simple, obvious way of doing this is the wrong way.

In both these respects it is similar to duplicate file finders.

As usual, the open-source programs might not have the pretty GUIs of the proprietary ones, but get more of the basic features right.

If more than about a dozen files are changed with each sync, then any sync program is effectively unusable without update detection. Currently the only 3 programs that support this are unison, jfilesync & AllwaySync.

Of these 3, AllwaySync does not have a client/server mode, does not have the user community of the other 2 programs (so is more likely to have unfound bugs) and is closed source.

The unison developers state no new features will be added to the program. The unison gui is quite basic, and you will need to edit configuration files as well. Unison uses the OCaml language, so may be more difficult to customise if needed.

jfilesync needs Java 5.0, which will mean another download for most users.

JFileSync has a few UI quirks but overall it works well.

So for most purposes I recommend jfilesync.

If you need to sync a large amount of data over the internet, use unison instead because of its more efficient compressed protocol.

[Dec 11, 2007] Using Unison to synchronize files between windows and solaris

This document describes how to setup Unison to perform synchronization between a windows laptop and a solaris system.

What I am trying to achieve is to use the Windows version of Unison, as compiled by Max Bowsher. This version unfortunately has a problem asking for password for the ssh account but following this document should provide an acceptable alternative.

What I do is run Unison on the laptop and make it ssh to the solaris system where the remote files are stored (and backed up).

For this to work, you will need to install a few Cygwin packages (for ssh) and manually install Unison for windows and at last, set it up so we can avoid the bug mentioned above.

[Dec 11, 2007] FileSyncingHowTo - MineZone Wiki

I've been a big fan of all the life hacks sites and tutorials that have sprung up recently. In Danny O'Brien's original Life Hacks presentation, he mentioned file synchronization as one of the tools common to the most productive users. Prior to hearing that, I had tried several times to set up file synchronization between my home and work PCs and always ran into snags.

Thanks to inspiration from the Life Hacks presentation, I finally persevered. The result is a combination of four open source tools: copSSH, putty, Unison, and WinMerge that allow me to quickly synchronize my files on a daily basis. I encountered quite a few road blocks in the process, which I will do my best to document here.

I've been using this setup successfully for months. I synchronize files once when I get to work and once before I leave and I'm able to work on all the same files, in the same directories, regardless of which PC I'm on. I can confirm that it has indeed been a productivity boon.

[PDF] File Synchronization with UNISON Introduction

[Sep 04, 2007] Keep your laptop and PC in Unison  by Federico Kereki

September 04, 2007 | Linux.com

I usually carry my laptop everywhere and do work on it, but I do also need to work at my desktop PC, so I need to synchronize the files on the two machines. I have used several sync tools in the past, ranging from simple commands like scp and rsync to utilities like Krusader, which includes a way to synchronize directories on different machines. Recently I started using Unison.

Unison can synchronize files between Unix, Linux, and Windows machines; for example, you could synchronize a Windows laptop with a Unix server with the same ease as you would synchronize two Linux machines. You can use it with any two machines connected over the Internet, and it can use an encrypted SSH connection for extra security.

Unison is freely available under the GNU GPL. Its development seems to be at a halt right now, with no one specifically on charge of maintenance or bug fixes. However, the site explains that the original developers are still using the program, and continue to maintain and support it, and (on occasion) release new bug fixes or small changes. The current stable version at the site is 2.13.16, but there's a beta version 2.27.29 (quite a jump!) and the current development, unstable, version is 2.28.15.

Since it has been around for some time, most distributions' repositories include it. I use openSUSE (both 10.1 and 10.2), and installing Unison was a breeze. A curious point: on openSUSE 10.1 I got the 2.13.16 version, but on 10.2, I got a newer 2.17.1 one. That posed a problem, because you must have the same version of the program in both machines, so I went to rpmseek.com and got the 2.17.1 version for my openSUSE 10.1 box.

WinSCP -- Directory Synchronization

WinSCP can synchronize content of local directory with remote one or vice versa or even mutually. Find the function in Commands > Synchronize. Synchronize dialog will appear, where you can select synchronization mode and configure options.

Once you confirm your options, WinSCP will collect list of differences in between the directories. If option Preview changes is enabled, you will be presented with the synchronization checklist, where you can select those synchronization actions you want to actually apply.

Files and directories excluded from transfer are excluded from synchronization too.

Before using the function make sure that WinSCP knows correct timestamps of remote files, to avoid losing data.

If you use WinSCP to synchronize your files only, you can use command-line parameter /synchronize to quickly open Synchronize dialog.

freshmeat.net Project details for DirSync

DirSync is a directory synchronizer that takes a source and destination directory as arguments and recursively ensures that the two directories are identical. It can be used to create incremental copies of large chunks of data. For example, if your file server's contents are in the directory /data, you can make a copy in a directory called /backup with the command "dirsync /data /backup." The first time you run it, all data will be copied. On subsequent runs, only the changed files are copied.

Author:
Mario Viara [contact developer]

 

Rating:	reset 1 2 3 4 5 6 7 8 9 10
8.32/10.00 (2 votes)

Homepage:
http://www.viara.cn/en/dirsync.htm
Tar/GZ:
http://www.viara.cn/download/dirsync-1_11.tar.gz
Zip:
http://www.viara.cn/download/dirsync-1_11.zip

[Nov 9, 2005] Folder Syncronization PowerToy

Rather weak, but might be OK for stall directory trees/still very buggy for large trees (say over 40K files). It simply does not work with large trees.  Does not permit regular expressions in specifing what folder you need to exclude (important for Frontpage)

[Nov 9, 2005] AASync Beta version 0.9.1 is here! Easy-to-use FTP - SFTP Folder Synchronization

Chokes on large trees...

AASync is a File Backup, Folder Syncronization utility. It is simple to use, and it allows local as well as remote backups over an FTP or SFTP connection. Makes it very easy to back up your files to a remote server or another local disk or maintain your website! The sync engine will detect the changes, and copy only the changed files for efficiency. Here are the main features:

Folder Synchronization: Local-to-Local, Local-to-Remote (any protocol) and Remote-to-Local (SFTP/SSH only). If folder sync's target is a local or remote Mac with ssh access, fork/type/creator data is preserved!

Retains fork data, type/creator and attributes on Mac-to-Mac transfers (local AND remote).

Scheduled Sync.

Ability to ignore certain files or file types.

Why AASync? What is the point, you may ask.. well here it is: Most File backup utilities can only deal with source and target folders that are mounted on the local system, like an external disk. Well, that is better than nothing, for sure, but it will not offer a lot of data protection. What does? An offsite backup meaning a daily syncronization of certain folders on your desktop to a remote host! Remote meaning a host in a different location, with which we communicate using some protocol like SFTP-SSH or FTP. AAsync will do the local sync like everybody else, but it also does Local-to-Remote and Remote-to_local sync. Remote-to_local sync is very rare, very few apps can do it, and none like AASync. AASync does NOT require any custom software to run on the remote host, all that is required is that you have a regular FTP or SFTP account, and that is it (Remote to Local sync requires SFTP or an SSH shell account)!

File Synchronization with Unison   by Erik Inge Bolso

2005-03-01 | Linux Journal

Keeping directories in sync on multiple machines can be difficult. Running Unison is one way to make the task easier.

Unison is a file-synchronization tool that runs on Linux, UNIX and Microsoft Windows. Those of you who've used IBM Lotus Notes or Intellisync Mobile Suite probably have an idea of what synchronization is good for, as compared to one-way mirroring options such as rsync. You might have mirrored a company document directory to your laptop, for example, and then modified a document or two. Other people might have modified other documents in the same directory by the time you get back. With rsync, you'd need to reconcile the differences between the two directories manually or risk overwriting someone's changes. Unison can sort out what has changed where, propagate the changed files and even merge different changes to the same file if you tell it how.

Think of Unison as two-way rsync with a bit of revision control mixed in. The most common use is keeping your local and remote home directory, or some data directory you often use in different contexts, in sync. It uses the rsync algorithm to keep network traffic down and should be tunneled through SSH over untrusted networks. No extra work is needed-simply specify ssh:// when adding a directory location. Quite a bit of extra disk space often is needed for Unison, though, because the synchronizer needs to keep track of what the files looked like on the last run.

Getting, Compiling and Installing Unison

Unison's home page is maintained at the University of Pennsylvania; the project leader, Benjamin C. Pierce, is a professor in the Department of Computer and Information Science. See the on-line Resources for the URL.

Unison isn't as widely deployed as rsync, so you might not be able to find a precompiled package for your distribution. But the binaries downloadable from the Unison home page should work for most people.

If you'd like to compile from source, you can. A few extra hoops must be jumped through, however, because Unison is programmed in OCaml, not the most common language. See Resources if there is no handy package for your distribution.

Compiling and installing Unison is simple; type make UISTYLE=xxx. The GTK user interface needs additional OCaml bindings for GTK, so I use the text interface in this article. Typing make UISTYLE=text or make UISTYLE=gtk should give you a Unison executable. Simply copy the executable to somewhere in the path on both machines you want to synchronize.

In this article, I'm using the current stable version of Unison, 2.9.1, unless otherwise noted. You need to use the latest betas if you're going to synchronize files larger than 2GB.

The developer versions tend to work well. They are what the developers run themselves on their own precious data. Sign up for the unison-hackers mailing list if you feel a bit adventurous. Jerome Vouillon, Benjamin C. Pierce and Trevor Jim tend to hang out there discussing improvements. Commit logs also float by, so you can track what is going on.

Configuring and Using Unison

Unison keeps its config and working files in a .unison directory in your home directory or wherever you want to put it. Set the UNISON environment variable to specify an alternate location.

The default configuration is stored in .unison/default.prf. Listing 1 shows a plain config file suitable for testing. Synchronizing two directories is now as simple as:

$ unison /nfsmount/dir1 /home/me/dir1

Listing 1. .unison/default.prf

# Unison preferences file
merge = diff3 -m CURRENT1 OLD CURRENT2 > NEW
backup = Name *
maxbackups = 10
log = true
logfile = /home/knan/.unison/unison.log
rshargs = -C

Unison then asks the user about any differences between the directories and offers reasonable defaults. It does take a bit of time to get used to Unison's way of thinking, however. And, Unison is no substitute for backups. Unison happily propagates back the deletion of all the files in one replica, for example, which can be a rude awakening for programmers used to CVS. For example:

rm dir1/* ; unison ssh://server/dir1 dir1

doesn't do what you expect from a:

rm dir1/*; cvs update dir1

Deleting a file is an action that is replicated on the other side upon synchronization. So, this example command removes all files in dir1 on both sides.

Once you feel comfortable, consider adding auto = true to the Unison profile. This skips questions about any non-conflicting changes but gives you a chance to back out at the end.

The Unison manual is recommended reading. It is clear and well written and explains what happens at most corner cases.

Keeping Home Directories in Sync

Once users become familiar with Unison, a common thought is to use it for keeping one's home directory in sync between machines, say, your laptop and desktop. This can be realized pretty easily. Listing 2 has a simple profile that does the job, but you probably want to extend it. Listing 2, for example, ignores MP3 files and Unison's own files and demonstrates the use of include for having common settings applied to all profiles.

Listing 2. .unison/home.prf

# Unison preferences file
root = /home/erik
root = ssh://remotehost/home/erik
# exactly two or none "root" lines
ignore = Name *.mp3
# ignore all .mp3 files anywhere
ignore = Path .unison
# ignore all files with .unison somewhere in their full path
include default
# imports settings from default.prf

Test our new profile like this:

$ unison home -testserver

And invoke it like this:

$ unison home -batch
$ unison home

 

The -batch run takes care of the easy cases without asking, backing up and logging as needed, and the second run asks you about any tricky business-like merging.

The root = lines can be omitted if you want to specify the files to be synchronized on the command line instead. The lines are equivalent to this invocation:

$ unison home /home/erik ssh://remotehost/home/erik

Merging Conflicting Changes

In order to do a three-way merge, backups must be enabled. By default, with backups disabled, Unison keeps only a checksum and metadata, such as permissions, so it has no unmodified file to reference.

In version 2.9.1 of Unison, if you choose merge for a conflict and the merge is successful without manual intervention, the changes are propagated immediately, which doesn't give you a chance to back out. So, if you have the space, I suggest leaving maxbackups at 5 or so, instead of the default 2, to leave yourself the chance of recovering from automatic mismerges. Contents of the backup directory after a merge look like this:

$ ls -1 .unison/backup/
shared.txt              merged version ("NEW")
shared.txt.1.unibck     changed remotely ("CURRENT2")
shared.txt.2.unibck     changed locally ("CURRENT1")
shared.txt.3.unibck     old version ("OLD")

As of the newest beta, 2.10.3 at the time of this writing, Unison can invoke different merge programs for different files. You might want to use 3DM to merge XML files, for example, or a database merge tool for your Berkeley databases. This functionality still is new and subject to change. It has been noted by the project leader that the merge functionality was in need of a rewrite and didn't really work too well in 2.9.1 and 2.9.20. Thus, if you intend to do much merging, you will be better off tracking the bleeding edge.

Resources for this article: www.linuxjournal.com/article/8059.

Erik Inge Bolsø is a UNIX consultant and épée fencer who lives in Molde, Norway, and has been running Linux since 1996. Another of his hobbies can be found by doing a Google search for "balrog genealogy", and he can be reached at ljcomment@tvilsom.org.

Absolutely free software .folder sync utility v.10

Folder Synchronization - The easiest way to synchronize two folders -- for Windows NT and 9x; $15.

Folder Synchronization is designed to work with Windows95, Windows98 and WindowsNT.

Feature List:

• Easy to use interface.
• Control what gets copied on single-file or summary basis.
• Control the default operations to minimize your work.
• Keep a copy of deleted files on the server.
• Optionally get a detailed log of all operations.
• Take part in a server backup concept.

What's new for Version 1.1

• Comparison of Filenames is not any more case sensitive.
• The settings are now stored independant of how you exit.

Recommended Links

Unison (file synchronizer) - Wikipedia, the free encyclopedia

unison-manual

Philip Guo - Unison File Synchronizer Liberation through Data Replication by Philip Guo

Reconcile Users' Guide Date: September 1999 Author: John H. Howard

Reconcile combines different versions of file directories, propagating all updates between them and making them identical but never losing updates at one site because of updates performed at another. Among its applications are: - Road warriors: synchronize files between laptop and home base This report describes the program in detail, including motivation, basic concepts, applications, and the program's interface.

Read the full technical report (PDF: 72.2 kB)

File Synchronization with Unison

Linux.com Keep your laptop and PC in Unison

SUSE LINUX 9.1 Administration Guide 16.4. Introduction to Unison

Perl

Reflect

reflect - A hard drive mirroring perl script. Version: 1.0 Entered-date: July 20, 1995 Description: reflect is a perl script which will assure two directory structures are identical. It's intended purpose is for backing up directories to another hard drive. It can gzip the destination files and update files that have changed. Keywords: backup, reflect, mirror, hard drive Author: niles@axp745.gsfc.nasa.gov (Rick Niles) Maintained-by: niles@axp745.gsfc.nasa.gov

Win32

Software Download File Folder Synchronization Synchronizer Backup

When you work, even just occasionally, on multiple computers, you probably have to copy files from one device to the other. Although you could keep file modifications in your mind, there is a better way to apply your skills. Let Allway Sync take care of recent changes to your files. It looks for the latest version of a file, in specified folders, and copies it to the other folder, i.e. it synchronizes them. Unlike other file synchronization software which just copies file with the most recent file modification time, Allway Sync implements a sophisticated algorithm which does not depend on system clock accuracy. Allway Sync is fast, safe, stable and easy-to-use. All you need to do is to specify the local or network folders (directories) to be synchronized and the program does the rest. If your computers are not connected, use a removable drive to synchronize them. Free download from Shareware Connection - Allway Sync synchronizes local, network or removable files and folders. Fast, safe, stable, easy-to-use and FREE!