...

Installation & Configuration

Pre-Requisites

...

The NLPG Importer requires PyGreSQL, the PostgreSQL module for Python, which may be installed from the following location http://www.pygresql.org/. PyGreSQL is an open-source Python module that interfaces to a PostgreSQL database. 

Install & Configure Settings

...

First install and configure the NLPG Importer by doing the following:

 

• Step 1 - Extract the contents of the supplied zip file & create the folders for the NLPG Importer
• Step 2 - Create the database and add the database scripts
• Step 3 - Configure Settings

Create Job & Tasks in Studio

...

Now that you have created and configured the database ready for receiving the NLPG data you will need to do the following:

• Step 4 - Copy NLPG data to Pending folder
• Step 5 - Create & Schedule Job to Import & Build NLPG Data
  • Step 6 - Create & Run Task to Import the NLPG data
  • Step 7 - Create Data Share connection to each NLPG view
  • Step 8 - Create & Run Task to Build the Address Lookup tables

 

 You can create one more additional Job and Task which you would only run if you needed to rebuild the NLPG database from scratch:

• Step 9 - Create Job to Reset the NLPG Data
  • Step 10 - Create Task to Reset the NLPG Data

The Import Process

In order to import your NLPG data into Data Share and keep it up-to-date you will need to do the following: 

Process the NLPG data files

Files received from the NLPG hub will be copied, by the customer, into a processing folder (Pending) and an archive folder (Archive). The importer will then process each file in order of datestamp (oldest first). Once a file has been successfully processed it will be placed into a Completed folder. 

The NLPG Importer will iterate through the file line by line and will copy the contents of the line into one of four separate csv files. 

This will be done for the following record types: 

Once the file processing has finished, the importer will then attempt to copy the files into their respective tables using the Postgres 'COPY INTO' command. Each imported row will be given its own sequence number column in the database table. There will be four tables created, one for each of the record identifiers above. 

Import them into Data Share

...

A View of the latest data will be created in the NLPG Importer database for each of the four tables. 

The view for each table will be created by selecting the records with the highest sequence number per Unique Identifier. If the change type is deleted, then the record will not be included in the view. 

The Unique Identifiers for each record type are as follows:-

 

 

Record Type	Unique Identifier Field(s)
Street (11)	USRN
Street Descriptor (15)	USRN+LANGUAGE
BLPU (21)	UPRN
LPI (24)	LPI_KEY

 

 

A Data Share connection to each of the four views in the NLPG Postgres database will need to be created in iShare Studio.

 

Automate the NLPG Import process

...

In order to automate the NLPG Import process a Task will need to be created in the iShare Studio Workflow Manager to run the NLPG Importer and then a Workflow Job created to perform the following tasks:

 

1. Run the NLPG Importer
2. Download the NLPG data to Data Share
3. Rebuild the Address lookup table using PostgreSQL stored procedures

Now you can schedule the Job to run at a particular time.

...

Re-Importing the NLPG Data

...

If there is a problem with the import, it will be possible to restart the import again, starting from the first file received using the Archive folder. The process will be as follows:-

 

1. All records are deleted from each of the four tables
2. All files in the Pending folder will be deleted
3. All files are copied from the Archive folder to the Pending folder

...

The process will then continue as normal until all files have been processed.