FTP Replicator Design

Location Independence

Both the Web Service and Entities are written to include all functionality for both LOCAL and REMOTE locations. The functionality would then be restricted through configuration settings.

Web Services

The Web Services are written in C# and are mainly simply functions to start the appropriate entities. However there is the need for some auxiliary functions.

We need calls to start the various entities R1, R2, L2 and L3. However should a problem occur at either location, that location would initiate a roll-back and would need to inform the other location to initiate a roll-back itself.

In Asynchronous mode, the RepManager needs to get the status from the REMOTE location, so a call to get the process status is included.

We finally include a function to indicate the version of the Web Service for version purposes.

Therefore the calls are as follows:-

Python

The main system is to be written in Python, a scripting language and use a PostgreSQL database to store any information.

Run number

Each Entity is a standalone process that can be run independently. A complete run of all the entities will be called a Run. A Run can be a successful run or a run that featured a problem and initiated a roll-back. To ensure a level of security, each Run is given a unique Run number. Starting an entity without the correct Run number is considered an error and would initiate a roll-back.

L1 supplies the Run number to the run as it is the starting entity. It takes the last Run number and adds one it to produce the current Run number.

Transmission Items

We can send to types of items, standard filing system files and PostgreSQL database tables.

Files

The main function of transmitting is to take a list of files to check and if any need to be sent, to zip them into a transmission file which will be sent.

A PostgreSQL database table will hold the files that have been transmitted and the last edit time of the file. Each file to be transmitted is checked against this table.

Tables

As PostgreSQL database tables do not have any information regarding whether they have changed since the last time they were transmitted, we can only resend them every time.

Tables are added to the transmission file by performing a database backup of the table through PostgreSQL’s ‘pg_dump.exe’ program. The result from this is added to the Transmission file.

Transmission file

The transmission file will include all the above items and also a control file which contains a list of the items in he file and control information about them. This control information is the target directory for files and the target database for tables.

The transmission file is zipped up prior to transmission. This not only makes the transmission file a lot smaller, making transmission quicker, but it also provides a level of security for the receiver as a problem with transmission will make the file unlikely to be open correctly by the receiver.

Transmission Method

Transmission can be in one of two forms. FTP transmission or a simple file copy. With these two methods a number of scenarios can be catered for.

• This allows for the LOCAL and REMOTE to pass the transmission file over a common network if available.
• If neither has FTP capabilities, a third party FTP site can be used to transmit the files.
• If one of other has FTP capabilities, this can be used by one location to FTP the file to the other.

Processing

When processing a Transmission file, we first do some simple checks. Obviously being able to unzip the file is a start. Thereafter we check he number of items in the transmission against the number of files that we are expecting.

Files

For files, we first take a list of the files that are to be updated and we back them up. This is simply a case of creating a subdirectory called ‘Backup’ in the target directory and copying the target file to that.

We then copy the file from the Transmission. If something goes wrong in the processing, then the backup copy is copied back into place, so restoring the state prior to processing.

Tables

For tables, we take a similar approach by copying the target table to one with a name concatenated with ‘_BACKUP’. However with database tables, there is the additional problem that a table can have constraints and indices, so these have to be renamed also prior to processing the table.

Once the backup is done, then we use PostgreSQL’s ‘pg_restore.exe’ program to restore the table to the database.

If something does go wrong the Backup and its constraints and indices are reverted to the state prior to transmission.