Download Modern.gov using curl

Owned by Kim Stimpson
Last updated: 26 May, 2022
2 min read

This scenario covers downloading Councillor information from Modern.gov but the same process can be applied for MPs.

  1. Download curl (v7.66.0) for Windows 64-bit from to D:\Astun\Tools

    If you cannot find a copy of curl v7.66.0 then your Astun Consultant can supply this for you. Please do not download v7.83.1 as this does not currently work.


  2. Create a Workflow Program Task 

    Name - Enter a unique name for your task e.g. Download ModernGov Councillors using curl
    Program - Use the browse button to browse to where you installed curl e.g. D:\Astun\Tools\curl\bin\curl.exe
    Additional Parameters - You need to include Additional parameters along the lines of the following where d:\astun\downloads\moderngov.xml is where the .xml will be created and https://yourcouncil.moderngov.co.uk/mgRss.aspx?f=22 should be your existing URL to Modern.gov

    -s -k -o d:\astun\downloads\moderngov.xml https://yourcouncil.moderngov.co.uk/mgRss.aspx?f=22

    curl is specified as [options / URLs] and above we are using the following options:
    -s     to run curl in silent mod
    -k     to allow insecure server connections when using SSL

    -o to write the output to a file i.e. d:\astun\downloads\moderngov.xml

  3. Now you will need edit your existing ModernGov_Councillors Data Share Connection to point to the xml output from curl. You need to edit the existing entry as this exact name is used by the SQL later.

    Location - Here you need to enter the URL to the Modern.gov data that you have just downloaded using curl e.g. d:\astun\downloads\moderngov.xml
    Xsl - Browse to the xsl transformation that you wish to use to transform the XML feed. For Modern.gov you should pick the feedxml-rss.xslt which can be found in the Studio\config folder.

  4. Now we need to download the data to the iShare Data database so, select the RSS Data Share Connection ModernGov_Councillors, which you will find under the RSS Feeds node, right click and select Synchronise.

  5. Lastly you need to amend the Job to Download Modern.gov to add the Workflow Program Task Download ModernGov Councillors using curl as the first Task in the list and make the Moderngov_couneillors [DataShare] -  [Depends on previous job] so that it looks like the following:

    1. Download ModernGov Councillors using curl
    2. Moderngov_councillors [DataShare] - [Depends on previous job]
    3. Format Modern.gov Councillors - [Depends on previous job]
    4. Modern.gov Councillors Geometry - [Depends on previous job]

  6. Save your changes.