EEA & Eionet documentation hub

Browse documentation for IT-systems used by the European Environment Agency and the Eionet network.

Docker-based centralized logging app based on Graylog2

Prerequisites

Setting up

Pre-requisites: install docker and docker-compose.

Clone the repository

git clone https://github.com/eea/eea.docker.logcentral.git
cd eea.docker.logcentral
cp .dummy-secret.env .secret.env
cp .postfix.secret.example .postfix.secret

Configure the passwords (one time only) and start up the graylog2 app

# Configure Graylog password
vi .secret.env
# edit email configuration
vi .postfix.secret
# edit graylog email transport configuration
vi graylog.env

Choose the docker compose to run

  • docker-compose.singlenode.yml: to start graylog with a single node
  • docker-compose.multinode.yml: to start graylog with more nodes

    make a link of choosed docker-compose

    ln -sf docker-compose.yml

    Start Graylog2 app

    docker-compose up -d

Verify that the app is running by doing docker-compose ps

Now you can access the graylog2 web interface on port 80 (default):

  • http://localhost/

Ports

9000 - Graylog2 web interface 12900 - Graylog2 server API 12201 (tcp/udp) - GELF input 1514 (tcp/udp) - syslog

How to upgrade

docker-compose stop
docker-compose pull
docker-compose up -d 

How to add another node

To add another node follow the below steps.

1. Edit the “docker-compose.multinode.yml” file and add another slave node coping this code:

graylog-client-<progressive number>:
    restart: always
    image: docker.io/eeacms/graylog2:<latest tag>
    env_file:
        - .secret.env
        - graylog.env
    environment:
        - ENABLED_SERVICES=server
        - GRAYLOG_MASTER=false
    links:
        - "elasticsearch:elasticsearch"
        - "mongodb:mongodb"
        - "postfix:postfix"
    volumes:
        - /etc/localtime:/etc/localtime:ro

2. Add the new node into load balancer

Register the new stack into load balancer:

...
links:
    - graylog-master
    - graylog-client-1
    - graylog-client-<progressive_number>
...

Add new container under GRAYLOG_HOSTS var:

...
GRAYLOG_HOSTS=graylog-master,graylog-client-1,...,graylog-client-<progressive_number>
...

3. After you can stop and restart services

docker-compose stop
docker-compose up -d

How to enable LDAP security

# Go to System > Users > Configure LDAP
* LDAP Server Address - ldap2.eionet.europa.eu : 389 : StartTLS # NOTE! use the nearest ldap, e.g. ldap4.eionet.europa.eu if you deploy on the cloud.
* Search Base DN - ou=Users,o=EIONET,l=Europe
* User Search Pattern - (&(objectClass=inetOrgPerson)(uid={0}))
* Display Name attribute - cn
* Default permission group - Reader

How to set user the time zone

Since Graylog internally processes and stores messages in the UTC timezone, it is important to set the correct timezone for each user.

Even though the system defaults are often enough to display correct times, in case your team is spread across different timezones, each user can be assigned and change their respective timezone setting. You can find the current timezone settings for the various components on the System -> Overview page of your Graylog web interface.

To change your Timezone, go to System -> Users and edit the user’s preferences

How to add a new GELF UDP input

# Go to System > Input > GELF UDP > Launch new input
* Check global input
* title - your chioice e.g. "GELF UDP"
* bind address - leave the default
* port - 12201
* receive buffer size - leave the default

Structure

  • fluentd: A fluentd log collector instance listening for syslog messages
  • web: A nginx instance exposing the web interfaces used to analyze logs
  • graylog: A graylog2 instance used for storing and analyzing logs
  • demo/ a set of scripts to generate logs to be collected by the system defined in this repo

Testing that it works

  • docker-compose up
  • cd demo/
  • ./gen_syslog.py

Testing traceback logging

To log full tracebacks applications have to be set to use a GELFHandler. An example of such application can be found in:

./demo/gen_gelf_tracebacks.py

Note: These tracebacks will only be viewable in the graylog2 interface

  • cd demo
  • If running for the first time
    • virtualenv sandbox
    • ```source sandbox/bin/activate
    • ```pip install -r requirements.txt
  • ./gen_gelf_tracebacks.py

Ports forwarded on the local machine

  • 9000 - the web interface on the nginx server
  • 5140, 1514 - the Syslog UDP, TCP port listening for syslog messages
  • 12201 - the GELF port listening for GELF messages
  • 12900 - Graylog api port

Development tips

If you want to modify something in the base image follow these steps:

Handling data and updates

NOTE: Do not run docker-compose rm unless you know what you are doing. This will drop the data volume containing the settings and the stored logs.

Correct update procedure should follow these steps:

  • If the services in docker-compose changed, create a copy of the docker-compose file: cp docker-compose.yml docker-compose-old.yml
  • Get the latest config from git git pull origin master
  • Pull the latest builds for the given tags: docker-compose pull
  • Stop the services defined in the old docker-compose file: docker-compose -f docker-compose-old.yml stop
  • Optionally backup your data using something similar with docker run --volumes-from eeadockerlogcentral_data_1 someimage $BACKUP_COMMAND
  • Start the freshly pulled services: docker-compose up
  • Remove the backup docker-compose file: rm docker-compose-old.yml

Note: The copy is needed as services can be renamed or removed during the git pull, making docker-compose stop ignore the other running services.

Common issues

Problem: After graylog container is restart it will stop and restart over and over again.

Fix: Enter graylog container and delete /opt/graylog2-web-interface/RUNNING_PID file