Using Sitecopy
Use Sitecopy to keep your websites mirrored
Posted 30.05.2006 | Updated 30.05.2006 | Contributed by Andy Mallett


I run a number of websites and know from experience that the more you keep, the harder they are to keep track of.

The situation is multiplied when you need to keep copies and website mirrors elsewhere.

A casual perusal of the internet came up with a number of open source options. I prefer to use FTP over ssh, so popular solutions like rsync were out.

The I came across the aptly named Sitecopy, which allows for FTP style file transfers.

Here's how to get it installed and running on a mirror server..

Installation

Sitecopy can be downloaded from the Sitecopy website or from the NoBlueScreens archives here [939KB].

Go to the download directory and extract the tarball..

cd   /src

tar  -zxvf  sitecopy-0.16.3.tar.gz

cd  sitecopy-0.16.3

./configure   (configure the package for your system)

make   (compile the package..)

make  install   (install programs, data files and documentation)

Configuration

1. Create a storage directory called /root/.sitecopy. Sitecopy will use this directory to contain a file which keeps track of each remote website you synchronise with.

Login as root
Type cd to change to root's homedir (/root)
From the /root directory, type:

mkdir  -m  700  .sitecopy   (note hidden dir - starts with a dot)


2. Configuration is performed via the run control file, .sitecopyrc. This file contains a set of site definitions. A unique name is assigned to every site definition, which is used on the command line to refer to the site. Create an rcfile, which will store information about the sites to administer..

From the /root directory, type:

touch  .sitecopyrc   (note hidden file - starts with a dot)
chmod 600 .sitecopyrc

The .sitecopyrc file and .sitecopy directory must both be accessible only by root - Sitecopy will not run otherwise.

Once this is done, edit the .sitecopyrc file to enter the required site details. Here is an example of my own settings which allow sitecopy to connect to this website via FTP and do the business.. 

site nobluescreens.com
server andrewmallett.net
remote /www/nobluescreens
local /usr/local/apache/htdocs/nobluescreens
username yourusername
password yourpassword
url http://www.nobluescreens.com
protocol ftp
ftp showquit
ftp  usecwd
http expect
permissions ignore
nodelete
exclude *.html


Explanation of above settings..

site nobluescreens.com - an arbitrary name give to this profile
server andrewmallett.net - name of server containing the website
remote /www/nobluescreens - name of remote directory to use
local /usr/local/apache/htdocs/nobluescreens - name of local directory to use
username yourusername - FTP username
password yourpassword - FTP password
url http://www.nobluescreens.com - url of website
protocol ftp - default protocol
ftp showquit - show quit status at end of FTP session
ftp usecwd - only upload in current dir, change dir for other uploads
http expect - can speed up the file transfer
permissions ignore - ignores file permissions
nodelete - don't delete anything on the remote server
exclude *.html - ignore files ending in .html

For more details on these settings, run man  sitecopy.

Using Sitecopy

Note that Sitecopy can be used to keep local sites or remote sites updated, either from the local webserver or from the remote webserver.

My own requirement is to mirror www.NoBlueScreens.com, by pulling these file down to a server (called Franga - don't ask) sitting on my teaching network at TAFE Tasmania. So the example shown here will reflect this functionality. Again check Sitecopy's man page for more settings.

Although it's not strictly necessary, I like to create a local copy manually of the website material I wish to mirror. So I copy all the material from NoBlueScreens.com to Franga's Apache directory /usr//local/apache/htdocs/nobluescreens.

Next change to root's home directory and then run the following two commands in order..

cd   /root
sitecopy --fetch nobluescreens.com
sitecopy --synch nobluescreens.com

It's not strictly necessary to do this from /root, but I like to as I can quickly get to the config file .sitecopyrc which also lives there. Handy in the early stages of fine tuning.

The fetch command tells Sitecopy to make a note of all files and directories on the remote server, as specified in .sitecopyrc and the synch command pulls down any files which are different, overwriting the local files. Again note I am mirroring NBS from Franga Server, so I want any changes to be propagated to this mirror server on the teaching network.

Automating Sitecopy

And that's about it. You can't run synch without running fetch first. I like to automate this job so it runs every night when things are quiet. So I have created a script and a Cron job to take care of this..

vi  /root/sitesynch.sh 

/usr/local/bin/sitecopy --fetch nobluescreens.com
/usr/local/bin/sitecopy --synch nobluescreens.com


..and add the cron job..

crontab -e 

#min hr day month weekday command

00 22 * * * /root/sitesynch.sh


Note that cron runs as csh, not bash, so it's necessary to include the full path to the sitecopy executable (top) and the script (bottom).

So now at 10pm every day, my website should be mirrored down to the teaching server. Test the script first before running it as a cron job and then test the cron job during daylight hours so you can be sure it works properly. Results are reported in root's mail.

References

http://www.lyra.org/sitecopy/
http://www.steffensiebert.de/ports/sitecopy_doc.html



- A.