FTP Replicator Specification

Setup

LOCAL System

Typically this is where iShare Maps Studio is installed and configured and where data is synchronised from other databases and the Workflow does its processing.

FTP

FTP is the central basis of communication.  The system has the flexibility to allow both the LOCAL and REMOTE to have FTP capabilities and communicate with a possible third party FTP Storage.  Alternatively, one or other of the parties can be the storage device (typically the REMOTE) and leave all FTP communication to the other. Files are compressed to speed the transmission and are checked for accuracy.

REMOTE System

Typically this is where most data is read-only and a replica of the LOCAL system.

Process

The Process of Replication is performed between the LOCAL and REMOTE by a series of ‘Entities’ which communicate with each other.  They know what there is to transmit and how to do it from a series of configuration files.

They communicate with each other by Web Service calls and log their actions so that these can be checked to ensure performance. A manager process is used to start these entities talking.  

• In Synchronise mode this will start the LOCAL and let the process precede without any further action required.
• In Asynchronous mode, whereby the REMOTE has no ability to call out, the manager process nurses the whole replication cycle.

The underlying logic of the process is seen in the ‘Entities’ and conceptual communication flow in the ‘Process’ diagrams below.

Synchronous	Asynchronous

Entities

Synchronous mode

Each of the entities has a specific role in the overall process.  If we examine the synchronous process then this is as follows.

L1 starts the process getting the previous run number and incrementing it to give us the new run number. This run number is used in all communication with the entities to verify the run.  L1 sets the LOCAL status to indicate the process has started and starts R1 at the REMOTE.  In so doing L1 will indicate the mode of communication that will be used by the REMOTE.

R1 sets the REMOTE status to indicate that the process has started.  It will build and transmit any data to be replicated.  It then calls L2 at the LOCAL, indicating the data file.

L2 will take any transmission file R1 has passed it, and process this data.  It then builds and transmits any data to be replicated.  It then calls R2 at the REMOTE specifying the data file.

R2 will process the data pass it and on completion will reset the REMOTE status to say the process has finished.  It then calls L3.

L3 will reset the LOCAL status and complete.

Asynchronous mode

In asynchronous mode, the entities perform the same job, with the exception that the REMOTE entities can no longer communicate back to the LOCAL, hence the introduction of the “RepManager”.  The RepManager is tasked with starting a LOCAL entity and then polling the REMOTE system periodically to see the status.

When the RepManager sees that the REMOTE entity is completed, it then starts the next LOCAL entity, thus managing the process.

RepManager

Whichever mode of communication is used, the RepManager is the entry point for the Replication process. The mode of communication is set in the configuration.

In Synchronous mode RepManager starts L1 and then finishes knowing that the individual entities can carry out the process by themselves.

In Asynchronous mode the RepManager starts the appropriate LOCAL entity which will start the appropriate REMOTE entitiy, thereafter the RepManager must poll the REMOTE to find out when the REMOTE entity has finished, and once finished the RepManager can then start the next LOCAL entity.

Processing

The concept of processing a transmission file is to unpack the file to a staging or temporary area.  The files to be updated are then backed up, so that if an error occurs in the updating process, these files can be restored and the application returned to its previous state.

Status

The Status of the replication will be stored in a PostgreSQL table in a reserved database.  Both the LOCAL and REMOTE systems need PostgreSQL installed.

Transmission Log

There is a transmission log that indicates the last modified time of each item being replicated.  Items requiring replication are checked against this to see if they needed inclusion in the current process.

Logs

Logs are kept throughout the system, by the LOCAL and REMOTE process entities and by the Web Service.

Errors

An important aspect of the FTP Replication is resilience.  Should the process of Replication come across a problem, rather than just report the problem, it is designed to initiate a roll-back.  This should leave the system in a state to attempt the replication the next time it is run.

When problems do occur, an appropriate e-mail message is generated.

Heartbeat

It is generally assumed that the system is healthy if no e-mail messages indicating an error are received. However one practical problem with this is that the e-mail transmission system fails, so that the replication may be failing constantly, but is unable to inform anyone about this problem.

To overcome this problem, there is a ‘Heartbeat’ functionality built into the RepManager.  This allows a simple e-mail message to be send periodically, typically every week, indicating that e-mail transmission is still possible and thus satisfying the notion that the system is healthy.