Data Organization

This page describes how the Data Exchange files are be structured. It follows Siprocal's standard SFTP structure

Folder Structure

~/
├─inbox/
| └─ data-exchange/
|   └─ [version]/
|       └─ [TABLE]/
|           └─ latest/
|           └─ historical/
|               └─ year=[YYYY]/
|                   └─ month=[MM]/
|                       └─ day=[DD]/
├─outbox/
| └─[WORKFLOW FOLDER]/
└─HELLO_[USERNAME].txt

Version

The product version will be the top layer for all information inside the data exchange folder.

Table names

As described in Data Content, there will be 11 tables available. These will be their formatted names for the folder structure:

devices
optins
locations
installed-apps
surveys
survey-answers
campaign-details
campaign-dispatches
campaign-metrics
campaign-downloads
in-app-events

These table folders will appear underneath the version folder.

Please be aware of the distinction between point-in-time or "snapshot" and "historical tables"

`latest` and `historical` folders

Every table will have two sub folders:

`latest`

This will contain latest information for snapshot tables only.

This folder will be empty for historical tables.

historical

This will be divided into year, month, day and hold the historical data according to the expiration protocol. Both snapshot and historical tables will have information inside this folder.

NOTE: For snapshot tables the date refers to the last completed day, but the data itself might contain entries from the date when the file was processed.
For example, the snapshot for June 25th, 2023 may contain entries updated on the beginning of the 26th, since that was when the snapshot was taken.

File names

The filenames inside lastest/ and historical/year=YYYY/month=MM/day=DD/ folders will be a ULID, which has no meaning other than being lexicographically sortable by time of creation, followed by the file part, then format and compression. For example: 01ARZ3NDEKTSV4RRFFQ69G5FAV_17.csv.gz.