Linux Crumbs
Simbiotic - A Cyberpunk Thriller Novel

Set Up Android Two-Way File Synchronization with GNU/Linux

Summary

This tutorial describes how to set up two-way file synchronization between a device running Android and a GNU/Linux computer. The steps use Free Software, do not use or share your data with external service providers, and work with the stock "non-rooted" Android image.

The primary purpose of creating this tutorial was to replace Palm OS Memo synchronization capability with something similar on Android. The final solution enables two-way synchronization of memos contained in text files between a folder on Android and a folder on GNU/Linux. The tutorial is also useful for anyone who wishes to synchronize file changes between Android and GNU/Linux.

This tutorial is the third in a series exploring the replacement of my ageing Palm OS device with a device running Android. The other articles are Migrating from Palm OS to Android - The Journey Begins, and Set Up Android Calendar and Contacts with DAVdroid Sync to Radicale Server on Linux.

The steps in this tutorial were developed using Android 4.4.4 (KitKat) and Kubuntu 12.04 (Precise Pangolin) GNU/Linux. The steps should be useful for other distributions with some minor modifications. For example on Fedora replace the debian package manager apt-get with the fedora package manager yum or dnf. On openSUSE use zypper.

For an alternative file sync method that does not adb-sync and developer mode on your phone and instead uses Syncthing see Set Up Android File Synchronization with GNU/Linux Using Syncthing.

Contents

How Palm OS Memo Hot-Sync Worked

When a Palm Pilot was used in conjunction with J-Pilot on GNU/Linux, the combination enabled two-way synchronization of memo creations, modifications and deletions.

This meant that if a memo was created, edited, or deleted on the Palm Pilot, then these changes would be propagated to your GNU/Linux computer the next time the device was hot-synced with J-Pilot. The reverse was also true for changes made in J-Pilot.

The hot-sync process was initiated with the following steps:

  1. Connect Palm Pilot to GNU/Linux computer with USB cable
  2. Start J-Pilot on GNU/Linux
  3. Press Hot-Sync button on Palm Pilot cable (or button in Hot-Sync app)
    Note: If the device was locked, a pass-code would need to be entered
  4. Click Sync button in J-Pilot application
  5. Let the two computers synchronize any changes since the last hot-sync

Two-way synchronization was very handy for ensuring that memos were backed-up and available in both locations.

So far I have not found anything quite as elegant for Android. Having said that, I believe I have found an adequate solution which is described in the remaining tutorial steps.

Choose Text Editors for GNU/Linux and Android

Palm OS memos are essentially text notes identified with a title (the first line) and stored under a category. Changes to memos can be synchronized between the Palm OS device and a desktop or laptop computer.

On Android we can replace this functionality by using text files for the memos. To simulate the Palm OS environment I use a folder name for the Palm OS category and a file name for the memo title. Changes in these text files can then be synchronized between Android and GNU/Linux. This means we need a text editor on Android and on GNU/Linux in order to view and edit the text files.

GNU/Linux provides many text editors, such as kate, gedit, emacs, and vi. I imagine that you already have a favourite (mine is emacs) so I leave the decision of what to use on GNU/Linux to you.

Fortunately Android also provides a broad choice of text editors. Personally I chose 920 Text Editor and installed it from F-Droid. Do feel free to test other text editors. The key thing is that the editor must be able to create, load, view, and save plain text files.

Install adb-sync on GNU/Linux

Prerequisite: The adb-sync program requires adb installed on GNU/Linux, and developer mode enabled on Android. If you have not already done this then see Installing Android Applications over USB.

Since there were no GNU/Linux packages containing adb-sync, I describe how to place the adb-sync script in your own bin folder.

Install the adb-sync file synchronization software:

  1. Download adb-sync-master.zip file.

    Clicking the link should download the adb-sync-master.zip file into your ~/Downloads folder.

  2. Open a terminal window.

    In the K Desktop Environment (KDE) this can be done with the menu option: K -> System -> Konsole (Terminal).

    TIP:   Opening a terminal in other desktop environments

    You can often open a terminal window using the following steps:

    a) Press Alt+F2 to open a run command prompt
    b) Enter "xterm" without the quotes

  3. Create a bin folder in your home folder.

    mkdir ~/bin

    The purpose of the ~/bin folder is emulate the way GNU/Linux commands are stored and discovered. The ~/bin folder should be contained in the PATH environment variable so that executable files placed in ~/bin will automatically be found. Note that the ~ (tilde) character represents the users home folder which is contained in the HOME environment variable.

    If ~/bin did not exist before, then you might need to logout and login so that executable files placed in ~/bin/ are automatically found.

    For example on login to my kubuntu 12.04 GNU/Linux distribution, if the ~/bin folder exists then ~/bin will be added to PATH. This occurs due to the following lines in my ~/.profile file.

    ----- Begin snippet from ~/.profile file -----

    # set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
  PATH="$HOME/bin:$PATH"
fi

    ----- End snippet from ~/.profile file -----

    If your distribution does not automatically find executable files placed in ~/bin then you might need to edit ~/.profile to ensure it contains lines similar to the above.

  4. Extract the zip file contents into your ~/bin folder.

    unzip ~/Downloads/adb-sync-master.zip -d ~/bin

    This will extract the contents of the zip file into an adb-sync-master folder under the ~/bin folder.

    TIP:   No unzip command?

    To install the unzip command in Ubuntu use the following command in a terminal window:

    sudo apt-get update && sudo apt-get install unzip

  5. Create a symbolic link for adb-sync in your ~/bin folder.

    ln -s ~/bin/adb-sync-master/adb-sync ~/bin

The adb-sync program is now installed and should be invokable using the adb-sync command .

Create Synchronization Folders

We need a master folder to hold all of the subfolders that we wish to synchronize. I chose ~/DroidSync as the name of my master folder. The reason for the master folder is to enable us to later use different modes of synchronization for subfolders (for example two-way or one-way synchronization).

Under the master folder we need a folder to hold all of our Palm Pilot memos. I chose Notes.

  1. Create ~/DroidSync master folder.

    mkdir ~/DroidSync

  2. Create Notes subfolder.

    mkdir ~/DroidSync/Notes

Later we will choose a location for the DroidSync file folder on Android.

Migrate Palm OS Memos from J-Pilot to Files

Now that we have a synchronization folder structure in place we can begin migrating memos into files.

  • Open J-Pilot and also a text editor on GNU/Linux.

  • For each memo in J-Pilot you wish to migrate:

    1. Ensure the memo category subfolder has been created under ~/DroidSync/Notes.

      For example if you have memo stored under the category Personal on Palm OS, then create a Personal subfolder on GNU/Linux.

      mkdir ~/DroidSync/Notes/Personal

    2. Copy the contents of the memo from J-Pilot into the copy buffer.

    3. Paste the buffer into your favourite text editor.

    4. Save the file under the title of the memo.

      For example if you have memo titled Famous Quotes under the Personal category then save the file contents under the category subfolder ~/DroidSync/Notes/Personal with the Famous-Quotes.txt file name.

    5. Repeat the above steps for each memo you wish to migrate.

Create droidsync Script

Now that we have our memos stored in folders and text files under the ~/DroidSync/Notes folder, we need to create a shell script to perform the file synchronization. Note that the script also makes a backup on GNU/Linux of the application package names installed on Android.

  1. Use your favourite GNU/Linux editor to create a new droidsync file under the ~/bin folder.

    For example:

    kate ~/bin/droidsync

  2. Copy the following droidsync script lines and paste the lines into your editor.

    ----- Begin droidsync script -----

    #!/bin/bash
#
#         Name:  droidsync
#  Description:  Synchronize folders and files between GNU/Linux
#                and Android.

#   Function to echo command and then run command
#     Usage:  echo_do eval "command-to-run"
function echo_do() {
  echo "\$ $@" | sed 's/eval //'
  "$@"
}

#   Set directories
#     On my Android sdcard0 is internal memory and sdcard1 is microSD card
DROIDDIR=/storage/sdcard1/DroidSync
LINUXDIR=~/DroidSync

#   Get package list (and test adb server connection)
echo ""
echo "***** Retrieving package list *****"
echo_do eval "adb shell pm list packages > ${LINUXDIR}/droid-packages.txt"
RETVAL=$?
if [ $RETVAL -ne 0 ]; then
  echo "ERROR:  adb server unable to connect to android device."
  echo "        Ensure (a) android device connected via USB cable,"
  echo "               (b) allow USB debugging enabled,"
  echo "               (c) device powered on at home screen, and"
  echo "               (d) acknowledge prompt 'Allow USB debugging'."
  echo "        NOTE:  Sometimes works better if device in Camera (PTP)"
  echo "               mode instead of MTP mode"
  adb kill-server
  exit 1
fi

#   Synchronize notes
echo ""
echo "***** NOTES:  Linux (clean) <--> Droid *****"
#
#   Delete/clean emacs automatic backup files (e.g. *~)
echo_do eval "find ${LINUXDIR}/Notes -name '*~' -exec rm {} \;"
#
#   Perform two-way synchronization
echo_do eval "adb-sync --two-way ${LINUXDIR}/Notes/ ${DROIDDIR}/Notes"

    ----- End droidsync script -----

  3. Choose a location for the DroidSync folder on Android.

    On my Android:

    • /storage/sdcard0 is internal memory
    • /storage/sdcard1 is a microSD card I installed

    I choose /storage/sdcard1, which is reflected in the above droidsync script by the following line:

    DROIDDIR=/storage/sdcard1/DroidSync

    If you do not have a microSD card installed, then be sure to change the DROIDDIR to /storage/sdcard0/DroidSync. For example:

    DROIDDIR=/storage/sdcard0/DroidSync

  4. Save the file and exit the editor.

  5. Make the droidsync script executable.

    chmod a+x ~/bin/droidsync

Run the droidsync Synchronization Process

Initiate the synchronization process with the following steps:

  1. Connect Android to GNU/Linux with USB cable
  2. Enter pass-code on Android if device is locked
  3. Open a terminal window on GNU/Linux
  4. Start droidsync script 
    droidsync

    If droidsync script is not automatically found then you might need to update your ~/.profile script to add ~/bin to the PATH as described in step 3 of Install adb-sync on GNU/Linux.

  5. Click Allow USB permissions if prompted on Android
  6. Let the two computers synchronize creations and/or modifications since the last synchronization

Note that file deletions are not synchronized. Instead the file is re-created on the computer from which it was deleted.

To work around this minor issue, ensure the file is deleted on both Android and GNU/Linux before running the synchronization process.

File Synchronization Set Up Complete

Congratulations! You've finished setting up file synchronization between Android and GNU/Linux.

Now files that you create or edit under the DroidSync/Notes folders on Android or GNU/Linux will be duplicated on the GNU/Linux or Android repectively whenever you run the droidsync synchronization process.

Appendix A: Other Ways to Use adb-sync

The adb-sync program is very flexible. Not only does it permit two-way synchronization, but also one way with either Android or GNU/Linux being the master and the other the slave.

One-Way Android to GNU/Linux Synchronization

In this scenario we use Android as the master source and have changes pulled to GNU/Linux.

This is useful for example to back up a spreadsheet you maintain on Android in /storage/sdcard1/DroidSync/Spreadsheets.

  1. Create /storage/sdcard1/DroidSync/Spreadsheets.

    Use a file manager in Android, such as Simple Explorer, to create /storage/sdcard1/DroidSync/Spreadsheets.

  2. Add lines to the droidsync script similar to the following:

    ----- Begin lines -----

    echo ""
echo "***** Spreadsheet:  Linux <--- Droid *****"
echo_do eval "adb-sync --reverse ${DROIDDIR}/Spreadsheets/ ${LINUXDIR}/Spreadsheets"

    ----- End lines -----

Be sure to save your Android spreadsheets in the /storage/sdcard1/DroidSync/Spreadsheets folder.

One-Way GNU/Linux to Android Synchronization

In this scenario we use GNU/Linux as the master source and have changes pushed to Android.

This is useful for example when you find and store eBooks on GNU/Linux in DroidSync/Books but wish to read these on Android.

  1. Create ~/DroidSync/Books.

    mkdir ~/DroidSync/Books

  2. Add lines to the droidsync script similar to the following:

    ----- Begin lines -----

    echo ""
echo "***** BOOKS:  Linux ---> Droid *****"
echo_do eval "adb-sync ${LINUXDIR}/Books/ ${DROIDDIR}/Books"

    ----- End lines -----

Be sure to copy your eBooks to the ~/DroidSync/Books folder.

Appendix B: Why I Chose adb-sync for File Synchronization

Usually for file synchronization duties I use tools such as unison or rsync. These tools require either a remote service or remote file system access to synchronize files and directories.

For the companion unison or rsync remote service applications on Android, these unfortunately appear to either require root privilege, or a non-free software application.

For remote file system access Android provides the Media Transfer Protocol (MTP). Fortunately several free software tools are available under GNU/Linux that provide file system access to MTP devices. Unfortunately the following two that I investigated had limitations.

  • MTPfs

    MPTfs is a FUSE file system that supports reading and writing any MTP device. Unfortunately the directory and file creation and modification dates are not correct or maintained. Directories always had the date of December 1969, and files had today's date of creation.

  • jmtpfs

    jmtpfs is a FUSE and libmtp based filesystem for accessing MTP (Media Transfer Protocol) devices. Unfortunately the directory and file creation and modification dates are not correct or maintained. In this case the directory dates behaved the same as the file dates and had today's date of creation.

In my testing with MTPfs and jmtpfs sometimes older file contents clobbered new file contents. As best I could discern, the synchronization appeared to give precedence to the larger file, and not the most recently edited file. Further hampering the use of MTP, there appears to be a bug in Android in that not all directories and files are shown using MTP until reboot.

These above limitations prevented me from using unison or rsync, and prompted me to search for another solution.

Fortunately I discovered adb-sync, which is free software that works with my non-rooted Android 4.4.4 kitkat device.

References

While researching how to migrate from Palm OS to Android I found the following references useful:

Future Articles

In a future article, I plan to cover the following topic:

  • Migrating Palm OS Keyring to Android
The Medusa Deception novel - free first in series
News   Documentation     Copyright (c) 2013-2024 Curtis Gedak

Get GParted at SourceForge.net. Fast, secure and Free Open Source software downloads Valid HTML 4.01! Correct CSS!