Introduction

Standard Layers are built on OGC Standards including WMS & WFS and can be styled using the Studio SLD Style Editor, whereas Classic Layers have to first be added to the MapServer .MAP file via the MapServer Map Editor.

From iShare v6.0.0 Standard Layers can also be used in My House / My Nearest so during an upgrade to v6.0.0, it makes sense to convert any old Classic Layers to the new Standard Layers. Astun have created a Python script, which runs in a Python Virtual Environment, to help automate this process.

The converter described on this page will process an iShare Map Source, and:

• convert any Classic Layers, except those which reference Raster data, to Standard (OGC) Layers

• convert the MapServer Styles to SLD Styles

• convert any SQL in the Layer to SQL in the Map Source

• leave any existing Standard Layers in place

• create a new Map Source in a specified location

• create a corresponding new .map file in the same location, drawn from the original referenced .map file

• remove from the .map file any Layers which have been converted to Standard Layers

The resulting Map Source can either be imported into iShare, or copied over the existing source Map Source.

There are a few known issues with the conversion, and these are documented below.

Prerequisites

• Python 3.9 or later

• A Python virtual environment

• The Astun conversion script installed

Preparation and Installation

1. If Python 3.9 is not already installed:

   1. download Python 3.9 from https://www.python.org/ftp/python/3.9.11/python-3.9.11-amd64.exe

   2. right-click on the downloaded .exe file and ‘Run as Admin’ to install

   3. Choose Custom Installation

   4. When prompted for an install location, install to C:/Python39

2. Open the Windows command prompt and navigate to a suitable folder (normally D:/Astun/Tools)

3. Run C:/Python39/python -m venv mapsource-ogc in that folder to create a Python virtual environment - this will keep the Python installation separate from other installations on the machine

   Open

   

4. A mapsource-ogc folder will have been created in Astun/Tools, with a Scripts folder below it

5. Open

   

   In the command prompt, cd to the mapsource-ogc folder, navigate to the Scripts folder, and run activate.bat to activate the environment

6. The command prompt will now have a prefix (mapsource-ogc) to show that you are in the virtual environment

7. Make sure you are running a recent (18+) version of pip by running pip install --upgrade pip . To check which version of pip you are running, run the command pip --version

    

8. Download the latest version wheel file (currently 1.5.10 - mapsource_ogc-1.5.10-py2.py3-none-any.whl) from the Astun FTP site (https://download.astuntechnology.com/public/ ) and place it in the new folder that has been created for the virtual environment (e.g. D:/Astun/Tools/mapsource-ogc)

9. Install the script in the virtual environment by running pip install mapsource_ogc-1.5.10-py2.py3-none-any.whl --force-reinstall, (using the correct file name for the current version)

You should now be ready to run the conversion script.

Converting your Classic Layers

Preparation

1. Review the .map file in Studio to see if there are any issues in the list at the end of this document which are likely to affect the output - if so you will either need to fix them before the script is run or in the newly generated .map file

2. Create an output folder for your output files, to avoid overwriting any source files - for example D:\Astun\iShareGIS\TEST\WebApps\WebService\config\mapsources_ogc

Running the Script

1. In the Windows command prompt that you created earlier, run the conversion command using the format:

   update-ogc -s D:\Astun\iShareGIS\LIVE\WebApps\WebService\config\mapspources\AdoptedRoads.xml -m E:\iShareData\LIVE\_MapServerConfig\AdoptedRoads.map -w D:\Astun\iShareGIS\LIVE\Studio\config\Workflow.xml -o D:\Astun\iShareGIS\TEST\WebApps\WebService\config\mapspources_ogc\AdoptedRoads_ogc.xml

2. Note that the command line above:

   1. references a LIVE Map Source (it will not change it)

   2. must also reference the .map file used by the Map Source

   3. references the LIVE Workflow file in iShare Studio

   4. creates a new Map Source, and a new .map file with the same name as the Map Source in the same location but with a .map instead of a .xml extension

3. If there are errors or unexpected results in the output, run the command again with -l DEBUG > debug.txt 2>&1 - this will create a log file.

Script Parameters

The following parameters can be used with the script.

Conversion Approach

Initially it is suggested that the following approach is used:

1. Back up the WebApps/WebService/config/mapsources folders in LIVE and TEST iShare

2. Output the Map Source and .map file to a new folder using the script

3. Check the .map file visually, and update the reference to the map file if necessary

4. Move the new Map Source to the mapsources folder of the TEST instance of iShare (ensuring that a filename is used which doesn't overwrite existing Map Sources)

5. Move the corresponding new .map file to the _MapServerConfig folder of the TEST instance

6. In the TEST instance of Studio, import the Map Source

7. Test the Map Source in the TEST instance, and fix in Studio as necessary

8. Copy the Map Source to the mapsources folder of the LIVE instance of iShare (ensuring that a filename is used which doesn't overwrite existing Map Sources)

9. Copy the corresponding new .map file to the _MapServerConfig folder of the LIVE instance

10. In the LIVE instance of Studio, import the Map Source

11. Retest

12. If the test is successful then, in Studio, delete the original Map Source, set the Display Name of the converted Map Source to the original name, and move to the correct position

If no issues are found, the process could be streamlined by outputting the name Map Source directly to the LIVE mapsources folder, using a new name, and importing in Studio.

A batch file which will run the script by passing the filename of the Map Source XML (without the .xml extension) is included below. Paths should be edited as required before use.

Known Issues and Limitations

The following are known issues with the current version of the script - you should

• review your .map files before running the conversion to see if any of them will affect the result

• review the generated Map Source in test to check if any layers have been affected

Critical issues

These must be addressed before running the script, otherwise it will fail or individual layers will not be correctly converted.

1. Symbol definitions must be held in the symbols (.sym) file, not inline in the .map file - this should be implemented before the script is run. In addition. if the map file directly references a symbol that has an external file reference, such as in the code below, the reference will fail.

2. SYMBOL elements in the .map file must have their values quoted, otherwise, the conversion will fail:
   SYMBOL "circle"
   not
   SYMBOL circle

3. A double backslash in the map file (for example IMAGEPATH "D:\mapserver\tmp\\" ) will cause the script to fail - this should be changed to a double slash before the script is run (IMAGEPATH "D:/mapserver/tmp//" )

4. A style which retrieves colour values from query will cause the script to fail. For example:

   In the above example, we can see that the COLOR parameter is taking its value from the [mapservercolour] attribute. This configuration will produce the following error:

   The mapfile should be edited to amend this approach.

Other issues

These issues may cause layers to fail or be incorrectly converted - they can be fixed manually in the Map Source post conversion.

1. Queries containing an apostrophe in a text string (e.g. value='Children’s Centres') are correctly converted but iShare reports an error.

2. Some pattern-matching regular expression rules are not being correctly converted,

3. If Fields are specified in the Field List for a Layer but are not present in the database, the Layer will fail. Likewise, Fields used in expressions must be in a Field List.

4. If in Field Settings, Add link to field is checked, but Link using is not set, when the layer is converted to a Standard Layer, it fails to display.

5. If there is no ‘fallback’ class in the SLD, i.e. if only rules for specific values are set, MapServer creates a filter with incorrect syntax for any values with parentheses, so these values are omitted from the data returned. The fix is to add a fallback rule.

6. Labels for Point styles may be incorrectly placed, and in some cases, the Point style is not showing. The fix is to delete the SLD from the Advanced tab and recreate the Style – the color settings are retained, normally only the Label attribute setting is needed.

7. Having a select statement using double quotes for its query will cause MapServer to fail to draw the layers. For example:
   
   DATA "wkb_geometry from (select *, trim("path_type") as footpath_type from landscape.public_row) as foo using unique ogc_fid using srid=27700"
   
   the above will produce the following error in the MapServer logs:
   
   

    

   
   In that case, removing the double quotes from the DATA attribute and replacing them with single will resolve the issue:
   
   DATA 'wkb_geometry from (select *, trim("path_type") as footpath_type from landscape.public_row) as foo using unique ogc_fid using srid=27700'
   
   

    

8. It is worth checking the rendering order of your layers after the conversion and rearranging it using Studio. Have in mind that for the Classic Layers, the rendering order is coming from the map file while for the OGC Layers, the order is coming from the Map Source.