As of the 1st of February 2015 this wiki has been deprecated.
This archive is now a read only site.
There is also a copy of the history that has been converted into a git repository at: https://github.com/IATI/IATI-Wiki-Archive.
Please do not rely on any of the information found here.

Registry Sync Tools

A series of python scripts in development by Tim Davies to flexibly manage local synchronisation of IATI xml files.

Detailed Description

This is designed to develop into a package of tools that can be used to:

  • Synchronise the latest IATI data to disk
  • Keep track of which files have changed when, and log download issues
  • Convert data between formats
  • Upload data to third party services
  • Archive files which are not longer available
    • E.g. as test-case, Oxfam went from segmented files to a single file in August 2012. A sync routine needs to be able to handle this.

Initially these are being developed as command line python scripts that could be highly configurable through command line parameters. They could be refined as modules to be imported into other code.

Documentation

Implemented so far:

  • iati_download.py - configurable script to sync files listed on the registry.

Development notes

Working with the IATI Registry API

There are a number of ways we can find the lastest changed file from the IATI Registry. See the documentation about the CKAN nightly process here. Archiver code is here.

  • From the Dataset Search with All Fields enabled we can look at extras > data_updated to see the latest last-updated-datetime element on an IATI-Activity record that the registry located for a given file.
  • From the individual data package view (e.g. http://iatiregistry.org/api/2/rest/dataset/11580e90-0893-435a-97b4-4efeafb3f867) we get a sha1 hash and last-modified data for the resource which we can compare with a copy of the file on disk
  • We can check the last-modified headers given by a HTTP server when we request a file

QR Code
QR Code Registry Sync Tools (generated for current page)