Browse documentation for IT-systems used by the European Environment Agency and the Eionet network.
The GioLand upload service is a platform where service providers can deliver large files containing GIS data. Deliveries are tagged with: country, theme, projection, spatial resolution, extent. Each delivery goes through a workflow of checks, resubmissions and enhancement. See this issue for the initial requirements.
Get the source code:
$ git clone https://github.com/eea/gioland.git $ cd gioland/
Customize env file:
$ cp docker/app.env.example docker/app.env $ vim docker/app.env
Start application stack:
$ docker-compose up -d $ docker-compose logs
The application expects configuration via environment variables:
DEBUG Turns on debugging behaviour if set to
on. Not secure for use in
WAREHOUSE_PATH Path to folder containing the database and uploaded files.
LOCK_FILE_PATH Path to lockfile used to synchronize access to chunked file
SENTRY_DSN URL of Sentry server to report errors.
SECRET_KEY Random secret used for Flask browser sessions.
lists of principals for that role. Principals can be in the format
BASE_URL Base URL of the application. Necessary to generate correct URLs.
UNS_LOGIN_PASSWORD Credentials for
sending notifications via UNS.
on, don’t send any UNS notifications.
LDAP_USER_DN_PATTERN Server and DN pattern for connecting to
LDAP. For example
Each service provider delivery goes through the following stages:
int(Service provider upload)
The initial upload is made in a “parcel” (think of it as a folder). Subsequent steps in the workflow each have their own parcel, where more files can be uploaded. Parcels have back-forward links so each delivery is a chain of parcels. If a workflow step (e.g. Verification check) results in a rejection, a new parcel of the previous step is created, so the chain can loop back if needed.
Each delivery is tagged with the following metadata fields, which are copied over from parcel to parcel:
The application stores data in a ZODB database in
warehouse.py module is responsible for
connecting to the database and contains the models. Uploaded files are stored
in the filesystem, under
$WAREHOUSE_PATH/parcels, where each parcel has its
own folder. Since the files are accessed from a remote machine via CIFS, for
automated GIS processing, a tree of symlinks is maintained in
$WAREHOUSE_PATH/tree, where the path is generated using the metadata fields
of each parcel.
Uploads and other workflow steps trigger notifications to relevant people. They are sent via UNS_.
.. _UNS: http://uns.eionet.europa.eu/
Service providers upload very large files (in the order of 20GB). This is done via HTTP, with the file split in 1MB chunks, and reassembled on the server. The chunks are saved in a temporary folder in the parcel.
The project owner is Alan Steel (alan.steel at eaa.europa.eu)
Other people involved in this project are:
Minimum requirements: 256MB RAM; 1 CPU
The production server needs a lot of hard disk space because raster map images are uploaded there.