Configuring Data Export and Purge

Overview

Onyx data consists of participant data, workstation data (experimental condition data), and instrument data (calibration data).  All three kinds of data can be exported to zip files.  Only participant data can be purged (deleted from the Onyx database)

Configuring Data Export

Three types of data can be exported:

• Participant data  (interview data)
• Workstation data  (data from experimental condition logs)
• Instrument data  (calibration data)

You configure export of all three types of data by editing the export-destinations.xml file in the WEB-INF/config directory.

An entry for each stage associated with a Participant must be added to the export-destinations.xml file if the data from that stage is to be exported. A stage can be a questionnaire or a physical measure such as Standing Height.

Data can be exported to multiple destinations. The export-destinations.xml file contains a <destination> element for each destination, and each <destination> element can contain the following:

• a <name> element that must be unique within the set of destinations
• an optional <encrypt> element to specify if and how encryption of the output file should occur. If a <destination> does not include an <encrypt> element, data exported to that destination will not be encrypted.
• a <valueset> element for each type of data (now known as an entity in the subterranean world of Magma) that will be exported to the destination; a <valueset> element, in turn, contains:
  • a <script> element that defines which data to include in or exclude from the export

The example below (taken from the export-destinations.xmlfile for the onyx-demo application) can serve as a starting point for your configuring your export destinations.  For information about how to write the script in the <javascript> element, see Magma Javascript API. 

As you can see in the example, one of the conditions that can be used in a <script> for participant data is the status of the participant interview. Four interview statuses are possible: COMPLETED, CANCELLED, CLOSED, IN_PROGRESS.

Encrypting Data Upon Export

The <encrypt> element is optional, but when specified, it can contain some parameters that affect how the encryption occurs. Four parameters can be overridden: algorithm, mode, padding and keySize. Without defining what these mean exactly, here are the possible combinations:

If you use an empty  <encrypt/> element (do not specify any parameters), these default values will be used:

To increase the keySize

Note that the maximum keySize can be extended by installing the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files. You can get these files on the Java security page and install the files following their directions.

Modifying the Location of the Keystore

The onyx keystore is required for encrypting participant data upon export. The location of the keystore file can be configured in the onyx-config.properties file.

Modifying the Export Directory

By default, the demo configuration writes export files to a directory named target under the "current working directory" (this is usually Tomcat's home directory).

You can change this output directory by editing this property in the onyx-config.properties file:

Configuring Purge of Participant Data

Participants and their data can be purged from the Onyx database. This is typically done at some point after the data has been export. Data that is selected for the purge is permanently removed from the Onyx database. The purge function is configured in a file called purge.xml in the WEB-INF/config directory.

Default purge configuration

By copying the onyx-demo application, you will have the following default version of the purge.xml file. Notice that certain elements in purge.xml resemble those in export-destinations.xml.  Both purge and export involve defining which data to exclude and which data to include in the operation, and javascripts are used to fine-tune what is included (and sometimes what is excluded).

In the default version of purge.xml:

• The first <script type="INCLUDE"> element selects data for purging based on dates. The property org.obiba.onyx.participant.purge which is defined in onyx-config.properties allows you to set a number of days. Data older than the configured number of days will be purged. The default number of days is 0.
• The second <script type="INCLUDE"> element selects data for purging based on interview status. If the interview status is CLOSED or CANCELED, and the interview was finished prior to the last export, then purge.

Modifying the default purge configuration

You can configure purge of participant data as follows:

• If your organization wants to customize the purge function rather than use the default configuration, you can modify the <script> elements in purge.xml. For information about how to write the script in the <javascript> element, see Magma Javascript API. You can also contact the Onyx development team to discuss how to fine-tune the javascripts.

• If your organization wants to use the default configuration of the purge function, edit the following line in the onyx-config.properties file to define a "purge before" date:

Search Onyx Documentation