Browse documentation for IT-systems used by the European Environment Agency and the Eionet network.
The Data Dictionary (DD) is used to support the delivery of environmental data by countries to Reportnet. The contents in the data dictionary are used to generate pdfs of the technical specifications for dataflows as well as Excel templates. The Data Dictionary also introduces the possibility of simple automated validation of deliveries by countries and facilitates the development of data submission interfaces.
Data Dictionary holds definitions of datasets, tables and data elements. Each of these three levels is defined by a set of attributes, the core set of which corresponds to ISO 11179 standard for describing data elements. The whole attribute set is flexible and attributes can be added / removed from/to the system.
For doing a quick-start installation, read the following instructions. They give some hints on how to quickly get DD code from github, and how to set it up as a Tomcat web application.
DD runs on Java platform, and has been tested and run on Tomcat Java Servlet Container. DD source code is built with Maven.
Please download all of these software and install them according to the instructions found at their websites: Java, Tomcat, Maven and GIT client.
The necessary versions are as follows:
Create build directory for source code and get code from GitHub
$ cd /var/local/build $ git clone https://github.com/eea/eionet.datadict.git
NB! The resulting /var/local/build/eionet.datadict directory will be denoted below as $CHECKOUT_HOME
$ cd $CHECKOUT_HOME $ mvn -Dmaven.test.skip=true clean install
Revise $CHECKOUT_HOME/custom/*.txt files for modifying the content of headers and footers in DD web pages texts and links. You will find guidelines from inside $CHECKOUT_HOME/custom/README.txt.
Create the DataDict database and the DataDict user identified by a password.
$ mysql -u root -p CREATE DATABASE DataDict; CREATE USER 'dduser-here'@'localhost' IDENTIFIED BY 'password-here'; GRANT ALL PRIVILEGES ON DataDict.* TO 'dduser-here'@'localhost';
In the env_setup/local/catalina_opts.txt file you will find all the CATALINA_OPTS that need to be configured with the specific values for your environment. Set these variables for your local tomcat installation.
Place the resulting $CHECKOUT_HOME/target/datadict.war into Tomcat’s webapps directory, and start Tomcat. At tomcat startup liquibase will create the initial DB structure.
The unit test mechanism will install its own embedded database and create the tables when you execute them. Note that the embedded database will keep running afterwards. You can run individual tests with: -Dtest=DatasetImportHandlerTest
$ cd $CHECKOUT_HOME $ mvn -Denv=unittest -Dmaven.test.skip=false test
The integration tests mechanism uses maven failsafe plugin and docker to create a dockerized test mysql database which will be used for the tests. Note that if the unit tests fail, maven will stop there and not procceed to run the integration tests. If you wish to run only the Integration tests see below.
$ cd $CHECKOUT_HOME $mvn clean verify -Denv=unittest
If you wish to skip unit tests and run only integration tests you may do so using the flag: -DskipUTs=true as shown below:
$mvn clean verify -DskipUTs=true -Denv=unittest
Build the application as already described and then create a docker image by executing the docker build command in a directory containing both the Dockerfile and the datadict.war. If no tag is provided then the image will be tagged as “latest”. The docker file can be found at the env_setup/docker_build directory.
$ mkdir docker_build $ cd docker_build $ cp $CHECKOUT_HOME/target/datadict.war . $ cp $CHECKOUT_HOME/env_setup/docker_build/* . $ docker build -t eeacms/datadict:tag-here .
[Optional] In case you need to push the image you have created to DockerHub:
$ docker push eeacms/datadict:tag-here
In the env_setup/docker_rancher directory you can find example files to be renamed and used as docker-compose.yml files for docker/rancher. Both use the docker.env file for runtime properties configuration.
Note: After running the docker-compose up or the rancher-compose up commands you have to create the DB and the corresponding user-password and restart the tomcat container.
Additional web applications are available for modifying user privileges and help texts in DD: AclAdmin tool: https://svn.eionet.europa.eu/repositories/Reportnet/acladmin/trunk HelpAdmin tool: https://svn.eionet.europa.eu/repositories/Reportnet/helpadmin/trunk
Installing these applications is done by similar steps:
An optional process has been added, so the developer can drive the logs to Graylog Server. In order to achieve this, there are 2 prerequisites :
A way to run a Graylog Server is through docker file “docker-compose.yml”. An indicative file will be the example below:
version: '2' services: some-mongo: image: "mongo:3" some-elasticsearch: image: "elasticsearch:2" command: "elasticsearch -Des.cluster.name='graylog'" graylog: image: graylog2/server:2.1.1-1 environment: GRAYLOG_PASSWORD_SECRET: somepasswordpepper GRAYLOG_ROOT_PASSWORD_SHA2: 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918 GRAYLOG_WEB_ENDPOINT_URI: http://127.0.0.1:9000/api links: - some-mongo:mongo - some-elasticsearch:elasticsearch ports: - "9000:9000" - "12201:12201"