As of the 1st of February 2015 this wiki has been deprecated.
This archive is now a read only site.
There is also a copy of the history that has been converted into a git repository at: https://github.com/IATI/IATI-Wiki-Archive.
Please do not rely on any of the information found here.

Aid Information Toolkit

Linked to work on the idea of a Public Good Infrastructure we have been exploring the idea of creating a distribution of open source tools that all work together to perform key data access and use functions.

A full outline of the 'Open Aid Information Toolkit' idea can be found in this document.

This page can be used to start scoping out that work.

Desirable features / Architecture

What features would be desirable in a set of tools.

A powerful and interoperable set of scripts

  • Able to run across platforms
  • Shared configuration (e.g. for working directories etc.)
  • Able to be connected together as a pipeline of tools

Developer friendly

  • Command line tools;
  • Open source;

User friendly

  • Accessible web-based front end
  • Cloud version available
  • Easy to install and deploy

Managing provenance

  • Should make it possible to understand what has happened to data along a processing pipeline

Tools required

Registry download tool

For fetching any changed packages for a given query.

Sync Tools Development Notes

  • DataPkg is a command line tool for uploading or downloading data from CKAN
  • CKAN Client is a Python library for working with CKAN
  • OKFN IATI Tools includes scripts for downloading IATI data

Validation tool

For validating IATI XML files.

Should be able to:

  • Check for well formed data
  • Validate against the schema
  • Check for codelist values

Validation Tool Development Notes

Currency Conversion

Written up for RHOK Soton

This component should be able to read in any IATI XML file and convert transaction and budget values from one currency to another. This requires:

  • An open list of historic currency conversion rates - no comprehensive data exists, but it should be possible to use one or more of the datasets on the datahub to generate reasonably granular conversion tables for most major currencies.
    • API based solutions may be possible, but a large conversion process could involve thousands of transactions, exceeding the limits of most free APIs.
    • It would be useful to have scripts that automatically update conversion tables
  • A process to read in IATI XML files and to work out the current currency and value-date of any transaction / budget
    • Currencies may be provided in a @default-currency attribute of an iati-activity, or at the level of an individual transaction value
    • Transactions should provide a @value-date that is used for currency conversions (i.e. the currency rate from this date should be used to convert this number)
    • Note that slightly different logic might be required to handle budget and planned disbursement values.
  • A process to write the converted values back to the file
    • Ideally there would be the option to either overwrite the current currency values, changing default / element level currency in the process or to add a duplicate transaction in the new currency.

Optionally:

  • It would be useful to be able to run these conversions in parallel: for example, as a map-reduce job, to be able to convert 100s of files rapidly (there are over 1500 IATI files now, and thousands more anticipated in coming months; some processes run overnight updates of the data, and being able to do this rapidly would be beneficial).

Provenance module

Design patterns

  • We need environment variables that can be shared across scripts
  • We need a shared logging framework

Consider a model with a master script and then a set of plugins. Scripts would be executed from the master script to allow use of the same logging framework etc.

Script interface

Look at

Distribution Management

We need a light-weight structure to manage a community of contributors. Consider learning from the Open Source community.

Links, source code and development plans

See http://openhatch.org/ for ideas around supporting open source communities.


QR Code
QR Code Aid Information Toolkit (generated for current page)