Eurostat (eea.Eurostat)

This FME Package contains the Eurostat Reader introduced for FME 2022.1.

Eurostat is the statistical office of the European Union, based in Luxembourg (LU). It publishes official, harmonized statistics on the European Union and the euro area, offering a comparable, reliable and objective portrayal of Europe's society and economy.

A vast range of data is available for the EU as a whole, for Member States and in many cases also for candidate countries, EFTA members and other European countries, down to the level of regions and cities. All users may consult or download data and publications free of charge from the Eurostat web site. For more information on Eurostats policies and copyright rules: https://ec.europa.eu/eurostat/web/main/about-us/policies/copyright

Usage

To choose a dataflow:

• click on the down arrow on the right side.
• Select File From Web
• Browse Eurostat (eea.eurostat)
• Create or choose a web-connection.
• Fill in the parameters according to your needs.

The Eurostat Reader checks which dataflows are available. Browse through the catalogs until the desired dataflow is found. Double click the dataflow or select it and click on "ok".

The reader will allow you to pick 'one' dataflow.

FME Feature Types are defined by the dataflow chosen.

Quick Facts

Eurostat Feature Representation

*1 see: User Attributes

Eurostat Reader Parameters

Dataset / Authentication

Authentication:

• Click on the down arrow on the right side of the dataset.
• Choose "Select File From Web".
• Click on "Browse Eurostat (eea.eurostat)".

A new dialog window with the parameters will appear.

The Eurostat connection is a mandatory parameter that must be set.

• Web service: choose: Eurostat_v1 (eea.eurostat)
• Connection name: choose a name of your choice.
• Agency: The agency determines from which agency data is read. Different Agencies contain different dataflows.
• Local Cache Folder: There are a lot of dataflows to choose from. To avoid loading the list every time the information is stored as cache. This parameter lets you decide where you want to store this cache.
• Expiry Time (Seconds): This parameter determines how long the cache should be saved.
• Verify SSL Certificates: If checked, the reader will verify SSL certificates.

Once you've set up the webconnection. Click on "OK".

Choosing a Dataflow:

To select a dataflow navigating through the folders to the desired dataflow. Only 1 dataset can be chosen at a time.

Select a dataset and click on "ok".

You can use the Search function to quickly navigate to a dataflow you know the name of.

Optional extra Parameters

The Eurostat API limits filters to 1.000.000 features. Due to the size of datasets and the fact that some filters include null observations this limit is reached quite often. If this occurs the logfile say that the request was to big after returning an error. Sometimes using filters the request will force Eurostat to first create the file. It will be queued on their side and wont be added onto FME's canvas directly it will instead return an error. Reading the same data after a few minutes will result in the data being read. You can see when this occurs in the log when the result body xml includes a "queued" tag.

Filter on time

Optional. Providing a value for Start Period will ensure that data with a Time_Period greater than or equal to the given value will be read. Providing a value for End Period will ensure that data with a Time_Period less than or equal to the given value will be read. Values should correspond to the format for Time_Period for the chosen DataFlow. *1

*1 The filter used on time period is very strict and can be hard to set properly since the dataflow itself doesn't specify what it accepts in the name. To see which format of filter is allowed it is suggested to look at the Databrowser of Eurostat. Here you can search for the corresponding dataflow either by name or code. The Databrowser will then refer to the "Time Frequency" to help you choose the correct format for filtering.

Filter on first N and last N observations

Optional. This allows filtering data observations within a time series to limit the number of observations returned. Providing values will restrict the reader to only fetch the first N Observations and/or the last N Observations of the DataFlow with consideration taken to the optional Filter on time.

• This filter may be applied to data which has been filtered by dimension and / or time.
• First N Observations return the first N observations in the corresponding series
• Last N Observations return the last N observations in the corresponding series
• Both definitions may be applied e.g. to return the first and last observations in a series.
• If the filter is wrong, then a response is returned with no observation results.

Keep in mind that N observations does not equal N features. An observation is the unique combination of the codelists for a certain time period. As an example: When First N observations is set to 1. The first 3 rows of this dataflow are given as a result. This since 2013 is the first observation. But in this case Partner makes them unique for that time period. Making all these a first 1 observation.

User Attributes

Each unique dataflow can have extra attributes (e.g. freq, geo, unit, sex). These attributes refer to the series-keys in a dataflow. They are automatically exposed for each dataflow. The values of these attributes are represented as a code from unique codelists.

[*1]: OBS_FLAG is also part of the code lists. This codelist can always be present to give more information about the obersvation value and is therefore always present.

Example:

Dynamic Dataset URL

In cases when it is preferred to compose the dataset URL dynamically, it can be done as follows:

fme://eea.fme-eurostat.fme-eurostat/<fme_feature_type_name>.csv
  ?id=<eurostat_dataflow_id>
  &module=fmepy_eurostat.catalog
  &webservice=eea.eurostat.Eurostat
  &connection=<name_of_web_connection>

In the template above (linebreaks were inserted for readability), <fme_feature_type_name>, <eurostat_dataflow_id> and <name_of_web_connection> needs to be replaced with appropriate values.

In the example below the dataflow APRI_LPRC_H is specified and a connection named my-eurostat-named-connection is used:

fme://eea.fme-eurostat.fme-eurostat/APRI_LPRC_H.csv
  ?id=APRI_LPRC_H
  &module=fmepy_eurostat.catalog
  &webservice=eea.eurostat.Eurostat
  &connection=my-eurostat-named-connection

It is also possible to, instead of referring to a named connection, embed the required parameters of the web service directly in the URL:

fme://eea.fme-eurostat.fme-eurostat/APRI_LPRC_H.csv
  ?id=APRI_LPRC_H
  &module=fmepy_eurostat.catalog
  &webservice=eea.eurostat.Eurostat
  &AGENCY=ESTAT

The Web Service "Eurostat_v1" has currently only one required parameter, AGENCY:

Version Info

package.yaml

author: {email: discomap@eea.europa.eu, name: EEA}
description: Access to Eurostat statistics data
fpkg_version: 1
minimum_fme_build: 22623
name: Eurostat
package_content:
  formats:
  - {name: eurostat}
  python_packages:
  - {name: fme-eurostat}
  web_filesystems:
  - {name: Eurostat}
  web_services:
  - {name: Eurostat_v1.xml}
publisher_uid: eea
uid: eurostat
version: 1.0.1

Commit

https://github.com/eea/eea.eurostat.api.fme@133801e227d86fd790b774a22eedf2f8bd682c63

Eurostat changes

v1.0.1

Icon changed

v1.0.0

Inital Release