As of the 1st of February 2015 this wiki has been deprecated.
This archive is now a read only site.
There is also a copy of the history that has been converted into a git repository at: https://github.com/IATI/IATI-Wiki-Archive.
Please do not rely on any of the information found here.

AidView Backend Database

AidView is powered by an eXist XML database and set of scripts written by Chris Wallace. These are working notes on the process of trying to set up a copy of the database. The backend powers the AidView API and AidView application.

Installation

Source code available on GitHub. Development version is in the Master branch, with Version 1 (production) available here.

Set up server with eXist

Starting with a Turnkey Linux Core image. (Mine = root / existpass)

  • Note, if using the core image you will need to add updated the AWS firewall rules to allow port 80 from the Turnkey management console.

Install open-jdk

apt-get install openjdk-6-jre
apt-get install openjdk-6-jdk

Fetch latest stable version of eXist from http://exist-db.org/exist/download.xml and follow the installation instructions. Then run the command line installation (your version number may vary…)

Either:

(Development version)
java -jar eXist-setup-2.0-tech-preview.jar

(Current Live version)
java -jar eXist-setup-1.4.2-rev16251.jar -p /root/eXist

Apache configuration

You will also need Apache. With mod_rewrite and mod_proxy enabled:

apt-get install apache2
a2enmod rewrite proxy proxy_http

(Note: If running CentOS you may need to follow the instructions here to allow the proxy to work /usr/sbin/setsebool -P httpd_can_network_connect 1)

Then add this to your virtual host set up: N.B. This is for eXist 2.0 (Development). For eXist 1.4 (Live) see Installation differences for version 1.4

  # IATI exist HTTP Virtual Hosy
  <VirtualHost *:80>
    ServerAdmin you@gmail.com
    ProxyRequests off
    ServerName *YOUR-SERVER-URL-HERE*
  <Proxy *>
     Allow from all
  </Proxy>
    ProxyPass / http://localhost:8080/exist/rest/db/apps/iati/
    ProxyPassReverse / http://localhost:8080/exist/rest/db/apps/iati/
    ProxyPassReverseCookiePath / /
    RewriteMap escape int:escape
    RewriteEngine on
    AllowEncodedSlashes On 
    RewriteRule ^/data$ /data/ [R]
    RewriteRule ^/$ /data/ [R]
    RewriteRule ^/data/(.*).csv$ /xquery/data.xq?_path=${escape:$1}&_format=csv [QSA,P]
    RewriteRule ^/data/(.*).json$ /xquery/data.xq?_path=${escape:$1}&_format=json [QSA,P]
    RewriteRule ^/data/(.*).xml$ /xquery/data.xq?_path=${escape:$1}&_format=xml [QSA,P]
    RewriteRule ^/data/(.*).rss$ /xquery/data.xq?_path=${escape:$1}&_format=rss [QSA,P]
    RewriteRule ^/data/(.*)$ /xquery/data.xq?_path=${escape:$1} [QSA,P]
    RewriteRule ^/admin$ /admin/ [R]
    RewriteRule ^/admin/(.*).xml$ /xquery/admin.xq?_path=${escape:$1}&_format=xml [QSA,P]
    RewriteRule ^/admin/(.*)$ /xquery/admin.xq?_path=${escape:$1} [QSA,P] 
   
  </VirtualHost>

Running eXist

Check that eXist is installed and will run by changing to the eXist bin directory and running startup.sh

cd /root/eXist/bin
./startup.sh

By default this launches a server bound to localhost, so to access I tunnel to the server

ssh -f root@[servername] -L 8080:localhost:8080 -N

I can how access eXist at http://localhost:8080

Log-in to the admin interface at http://localhost:8080/exist/admin/ to set a password (user management).

Configuring eXist

eXist version 2.*

N.B. This is for eXist 2.0 (Development). For eXist 1.4 (Live) see Installation differences for version 1.4

The IATI Backend Database scripts require some additional modules to be enabled in eXist. To enable the modules:

  • Stop your running copy of eXist
  • Change to the eXist extensions directory
cd /root/eXist/extensions
  • Copy build.properties to local.build.properties and then edit it to change the line for include.module.scheduler to 'true'
cp build.properties local.build.properties
nano local.build.properties

In local.build.properties

  # Scheduler module
  include.module.scheduler = true

  • Change to the eXist root, create the build/classes directory if it doesn't exist, and then run ./build.sh extension-modules
cd /root/eXist
mkdir build/classes
./build.sh extension-modules
  • Then edit the conf.xml file to make sure the following modules (disabled by default) are enabled:

Exist Version 2.0 Modules

<!-- XSL functions -->
            <module uri="http://www.w3.org/1999/XSL/Transform" class="org.exist.xslt.functions.XSLModule" />
            <module uri="http://exist-db.org/xquery/file" class="org.exist.xquery.modules.file.FileModule" />
            <!-- Compression module-->
            <module uri="http://exist-db.org/xquery/compression" class="org.exist.xquery.modules.compression.CompressionModule" />
            <!-- XSL functions -->
            <module uri="http://www.w3.org/1999/XSL/Transform" class="org.exist.xslt.functions.XSLModule" />
            <!-- XQDoc module: required for generating function documentation -->
            <module uri="http://exist-db.org/xquery/xqdoc" class="org.exist.xqdoc.xquery.XQDocModule" />
       <!-- included optional modules -->
            <module uri="http://exist-db.org/xquery/datetime"           class="org.exist.xquery.modules.datetime.DateTimeModule" />
            <module uri="http://exist-db.org/xquery/httpclient"         class="org.exist.xquery.modules.httpclient.HTTPClientModule" />
            <module uri="http://exist-db.org/xquery/math"               class="org.exist.xquery.modules.math.MathModule" />
            <module uri="http://exist-db.org/xquery/scheduler"          class="org.exist.xquery.modules.scheduler.SchedulerModule" />

  • Restart eXist

Install the applications

via Restore client

To get a working copy of the AidView API backend you need to 'restore' the application by using the backup file available on GitHub. N.B. The development version (eXist2.0) is in the Master branch, with Version 1 (production) available here (eXist1.4).

Once we have the files we manage the restore process by launching the eXist client program (eXist Client Shell)

  • Start the eXist admin client (also known as the eXist Client Shell).
  • Go to Tools → Restore
  • Restore the data from backup by selecting the _contents_.xml file in the root of the backup, in this case you will find this at:
  PATH_TO_BACKUP/AidView-API/_contents_.xml. 

In the restore dialogue you will need to select 'Files of Type' as 'contents.xml files' in order to make the file visible.

  • At the prompt enter username 'admin' and no password (or the password you set). The database will then restore.

or Via WebDav

To access the server via WebDav on Mac Finder, use Go → Connect to Server and connect to http://localhost:8080/exist/webdav/db with the username admin, and password you just set.

Connected via WebDav, copy the source to /db/

Configure the application

Edit the /db/apps/iati/lib/config.xqm with the hostname and IP address of your server (you can edit using the eXist client application).

Operation

Once you have configured eXist, Apache, and installed the application, you should be able to use the application.

It is divided into a front-end /data/ service, a backend /admin/ service, and a number of additional scripts, such as a 'woapi.xq' service used to run AidView.

Some documentation is provided at /admin/doc

Administration

Visit /admin/ to access the administration interface. You will be asked for the eXist admin username and password.

To load data to a corpus

A corpus is a collection of activities with it's own OLAP cubes and summary data. You can use the system to maintain separate corpus collections of activities.

From /admin/ select the 'Corpuses' option to go into an existing sorpus (or see below for details of how to create one).

To load data to a Corpus you need to first fetch a list of the latest packages from the IATI Registry. You can find links for this in the 'Tasks' section. You can either:

  • Fetch activities bit-by-bit using 'Page Refresh from CKAN'
  • or Fetch all activities with 'Full Refresh from CKAN' (this can be slow and cause heavy server load).

To create a new corpus

At present a Corpus is created by editing the code. To create a Corpus:

  • Add a directory in /db/apps/iati/data with the name of your new corpus
  • Add an 'activities' directory within this
  • Edit /db/apps/iati/config/corpusindex.xml to add a new entry for your corpus. Copy an existing record.

E.g.

	<corpus name="All" pipeline="pipeline1">
		<description>A test corpus</description>
	</corpus>

API Layer

An API layer is available at xquery/woapi.xq

Accessing Logs

You can access the logs at webapp/WEB-INF/logs/exist.log

Installation differences for version 1.4

Vhosts:

  # data.aidinfolab.org HTTP Virtual Host
<VirtualHost *:80>
  ServerAdmin kit.wallace@gmail.com
  ProxyRequests off
  ServerName data.aidinfolabs.org
 <Proxy *>
   Allow from all
 </Proxy>
  ProxyPass / http://localhost:8080/exist/rest/db/apps/iati-api/
  ProxyPassReverse / http://localhost:8080/exist/rest/db/apps/iati-api/
  ProxyPassReverseCookieDomain localhost data.aidinfolabs.org
  ProxyPassReverseCookiePath / /
  RewriteEngine on
  RewriteRule ^/$ /xquery/home.xq [P]
</VirtualHost>

Default password: daisy

eXist version 1.4* Configuration

There is no build process to configure eXist version 1.*, but modules do still need to be enabled.

Edit the conf.xml file to make sure the following modules (disabled by default) are enabled:

Exist Version 1.4* Modules

<module uri="http://exist-db.org/xquery/backups"     class="org.exist.backup.xquery.BackupModule" />           
            <module uri="http://exist-db.org/xquery/cache"       class="org.exist.xquery.modules.cache.CacheModule" />           
            <module uri="http://exist-db.org/xquery/compression" class="org.exist.xquery.modules.compression.CompressionModule" />           
            <module uri="http://exist-db.org/xquery/counter"     class="org.exist.xquery.modules.counter.CounterModule" />           
            <module uri="http://exist-db.org/xquery/datetime"    class="org.exist.xquery.modules.datetime.DateTimeModule" />            
            <module uri="http://exist-db.org/xquery/examples"    class="org.exist.xquery.modules.example.ExampleModule" />            
            <module uri="http://exist-db.org/xquery/httpclient"  class="org.exist.xquery.modules.httpclient.HTTPClientModule" />            
            <module uri="http://exist-db.org/xquery/image"       class="org.exist.xquery.modules.image.ImageModule" />
            <module uri="http://exist-db.org/xquery/jfreechart"  class="org.exist.xquery.modules.jfreechart.JFreeChartModule" /> 
            <module uri="http://exist-db.org/xquery/jndi"        class="org.exist.xquery.modules.jndi.JNDIModule" />           
            <module uri="http://exist-db.org/xquery/mail"        class="org.exist.xquery.modules.mail.MailModule" />           
            <module uri="http://exist-db.org/xquery/math"        class="org.exist.xquery.modules.math.MathModule" />           
            <module uri="http://exist-db.org/xquery/metadata"    class="org.exist.xquery.modules.metadata.MetadataModule" />           
            <module uri="http://exist-db.org/xquery/scheduler"   class="org.exist.xquery.modules.scheduler.SchedulerModule" />           
            <module uri="http://exist-db.org/xquery/simple-ql"   class="org.exist.xquery.modules.simpleql.SimpleQLModule" />           
            <module uri="http://exist-db.org/xquery/sql"         class="org.exist.xquery.modules.sql.SQLModule" />           
            <module uri="http://exist-db.org/xquery/svn"         class="org.exist.xquery.modules.svn.SVNModule" />            
            <module uri="http://exist-db.org/xquery/xslfo"       class="org.exist.xquery.modules.xslfo.XSLFOModule" />           
            <module uri="http://exist-db.org/xquery/xmldiff"     class="org.exist.xquery.modules.xmldiff.XmlDiffModule"/>            

When you install the application (for example from the 'Restore' option, using the _contents_.xml file from the git repository) you will probably see the following errors:

WARN: Failed to restore resource 'pipeline-errors-v1.xml'
from file '/home/david/Desktop/Exist_day/1.4/AidView-API/apps/iati-api/archive/pipeline-errors-v1.xml'.
Reason: failed to read resource from file /home/david/Desktop/Exist_day/1.4/AidView-API/apps/iati-api/archive/pipeline-errors-  v1.xml
WARN: Failed to restore resource 'fullB-activitySets.xml'
from file '/home/david/Desktop/Exist_day/1.4/AidView-API/apps/iati-api/archive/fullB-activitySets.xml'.
Reason: failed to read resource from file /home/david/Desktop/Exist_day/1.4/AidView-API/apps/iati-api/archive/fullB-activitySets.xml
WARN: Failed to restore resource 'Icon_External_Link.png'
from file '/home/david/Desktop/Exist_day/1.4/AidView-API/apps/iati-api/assets/Icon_External_Link.png'.
Reason: failed to read resource from file /home/david/Desktop/Exist_day/1.4/AidView-API/apps/iati-api/assets/Icon_External_Link.png

These are errors related (I think) to not being able to import the data. This data has not been included in the backup file used to restore as it is massive! So you can carry on. The structure should be in place for you to import your own data.

Administration on 1.4

Initial notes: July 2012

  • You will find an admin interface at :8080/exist/rest/ /db/apps/iati-api/xquery/admin.xq
    • Username: caprenter@gmail.com / Password: daisy
  • The fullB corpus is the only one currently initialised. You may be able to manually set up another corpus based on this model, but for now, use fullB.
  • From the admin menu choose 'fullB'
    • To load data from the registry choose 'Download Activity Set metadata from the registry'
      • This will check the registry and provide a list of files to download
      • To load from an existing fetched list of files form the registry choose 'Activity Sets' from the top of the screen
    • To load a file manually, pick 'Download Activity Set from the Web'
      • Enter the URL of the activities file into the form, and choose 'download'
        • The database will fetch and index the file

QR Code
QR Code AidView Backend Database (generated for current page)