FTP Replicator

Owned by Kim Stimpson
Last updated: 02 Mar, 2017

Objective

The objective of this process is to replicate the state of a system between two locations, the LOCAL and the REMOTE.  LOCAL is so called because it is the controller of the process.  The replication is a two way process in that both parties can be replicating various parts of themselves to the other.

In general, the idea of the replication is that a location replicates data that can change at its location sending this data to the other location where this data is considered un-editable.

Considerations

Replication can work with the standard filing system and also PostgreSQL tables.  A configuration is required to indicate which items are to be included in the replication.

In order to communicate with each other there needs to be a communication web service.  This provides a mechanism to tell a location to start and to query the state of a location.

We need to ensure that transmissions are small by only sending items that need to be sent, so a transaction log is used to check whether a file has changed since it was last transmitted, thus only newer files get sent. 

PostgreSQL items are not time stamped and will therefore have to be transmitted every time.

In processing an update, the update is unpackaged into a temporary area.  A backup of the existing files are taken, and then the update applied (copy files and restore PostgreSQL items).  In the event of failure, the backup will be rolled back so that the previous state is preserved.

Transmissions are compressed to aid speed.  Data transmission uses the FTP protocol to move the files around.

  • PostgreSQL database Views are NOT replicated. We would suggest not using Views for live Layers in iShare. If you decide to use Views then you will need to manually recreate these on the External Server.
  • PostgreSQL new database Schemas are NOT replicated. If you decide to create a new schema(s) then you will need to manually recreate these on the External Server.
  • PostgreSQL entities should not contain 'error' or 'failure' as part of their name as this will cause an error in the replication.
  • PostgreSQL entities should not have duplicate names such as "Streets" and "streets" as this will cause replication to fail.
  • Prior to V5.6.2 the FTP Replicator could not handle zipped file size greater than 4GB. You will need to bear this in mind if you have a large database to replicate. If you are already using the FTP Replicator and this problem arises then please ask your Astun Consultant to provide you with the latest ReplicationPy.zip file which you can then extract over your existing files.

Other considerations are:

  • Replication should be scheduled at times when there are no other Workflow Jobs in progress, also avoiding potential server down times such as server backups.
  • Make sure any new symbols created are included in the FTP Replicator File Lists to Check including the corresponding symbols.sym file see Symbols & Styling.
  • FTP Replication may fail if the Internal and External Servers become out of sync - see FTP Replicator Sync Issues for details.

Typical situation

The typical situation where FTP Replication  is to be used will be the iShare Maps installation at a council.  All configuration changes and source data changes occur on an internal protected server to which the general public has no access.  This would be considered the LOCAL location.

There is an external server that the general public have access to as the council web site.  This is considered the REMOTE location.

FTP Replication would be set to replicate the iShare Maps directories from LOCAL to REMOTE, thus updating the web site that the general public have access to.

However, with the advent of ‘Fault Logging’, the general public now can ‘edit’ data on the REMOTE server.  So in this case, the REMOTE would replicate any of these editable entities back to the LOCAL server.  On the LOCAL server these entities should not be included in the replication and should never be edited.

Additional Considerations

For enhanced security, there is a requirement that the REMOTE server has no capacity to call the LOCAL server.  This means that the system will have to operate in one of two modes, Synchronous or Asynchronous.

Synchronous mode means each location can call the opposite location on completion.

Asynchronous mode means that LOCAL will control the whole process starting the REMOTE and then continually polling the REMOTE for completion.

The FTP Replicator can replicate only if the databases and roles are the same on both the LOCAL and REMOTE servers i.e. the name of the database to be copied from must match the name to which it is copied. The role that owns the tables to replicate must exist in both databases’ servers as well (and have table creation permissions). E.g. if a table's owner is isharedata on the LOCAL end, then it cannot be restored unless the PostgreSQL server at the REMOTE end also has a role with the same name (iShareData won't work, for example).