Email alerts for problems with Dockerized services using Elasticsearch and ElastAlert

In Managing Docker Logs with Elasticsearch and Kibana (dockerized, of course), we set up Elasticsearch to store logs from Docker containers, and Kibana to analyze them.

Now having logs in Kibana is great, but what if your services run into an issue? You can, of course, find them by actively looking at Kibana.

However, that’s probably not what you need to sleep calm at night. You will soon find yourself anxiously refreshing Kibana every five minutes just to see if anything’s wrong. You will want a more pro-active alerting approach.

Introducing ElastAlert

What’s great about your current setup is: You have all your logs stored in a searchable index. That makes it relatively easy to identify issues in them.

ElastAlert leverages that strength. It can periodically scan through your logs stored in Elasticsearch. When it finds something in the logs that looks like an issue, it sends out alerts.

Filters and Alerts

So what’s considered an issue? You can freely define that using filters in ElastAlert. The most simple example: Everything that’s logged to STDERR is an issue.

How will you be alerted about issues? Again, you can define your alert channels pretty freely. Email? Slack? A Jira ticket? All of that works.

Let’s set it up

We’ve already set up a docker-compose.yml in the last article. We’ll simply go ahead and add ElastAlert to it:

volumes:
  elasticsearch:

networks:
  elk:

services:
  elasticsearch:
    image: elasticsearch:6.5.3
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    networks:
      - elk
  kibana:
    image: kibana:6.5.3
    networks:
      - elk
  nginx_for_elk:
    image: nginx
    ports:
      - 9200:9200
      - 5601:5601
    networks:
      - elk
    volumes:
      - type: bind
        source: /home/my-user/elk-stack/nginx.conf
        target: /etc/nginx/nginx.conf
      - type: bind
        source: /home/my-user/elk-stack/.htpasswd
        target: /etc/nginx/.htpasswd
  elastalert:
    image: bitsensor/elastalert:latest
    volumes:
      - type: bind
        source: /home/my-user/elk-stack/elastalert/config/elastalert.yaml
        target: /opt/elastalert/config.yaml
      - type: bind
        source: /home/my-user/elk-stack/elastalert/config/config.json
        target: /opt/elastalert-server/config/config.json
      - type: bind
        source: /home/my-user/elk-stack/elastalert/rules
        target: /opt/elastalert/rules
      - type: bind
        source: /home/my-user/elk-stack/elastalert/config/smtp-auth.yml
        target: /opt/elastalert/smtp-auth.yml
    networks:
      - elk

As you can see, we simply added another service and defined a bunch of configuration files that will be mounted into the container.

As before, we’re assuming that all config files are located under /home/my-user/elk-stack. Change the paths as needed.

Configuring ElastAlert

So let’s go through the different configuration files we’re referencing: elastalert/config/elastalert.yaml defines how exactly ElastAlert will run:

es_host: elasticsearch
es_port: 9200
es_username: USERNAME
es_password: PASSWORD
writeback_index: elastalert_status
rules_folder: rules
run_every:
  seconds: 5

buffer_time:
  minutes: 1


alert_time_limit:
  days: 2

ElastAlert stores its status information in Elasticsearch itself, that’s why we define the es_ parameters. Use the username and password you defined in the previous article.

You can leave the other options unchanged:

• writeback_index tells ElastAlert where to store its status information,
• rules_folder sets the folder in which rules are defined (i.e.: what’s an issue and how will you be alerted),
• runs_every makes ElastAlert test if rules apply every 5 seconds,
• alert_time_limit stops sending out alerts if the same rule has applied for two days without a change.

elastalert/config/config.json lets you set up basic service parameters for ElastAlert itself:

{
  "appName": "elastalert-server",
  "port": 3030,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "es_username": "USERNAME",
  "es_password": "PASSWORD",
  "writeback_index": "elastalert_status"
}

Change USERNAME and PASSWORD and leave the rest unchanged.

Writing a Rule

Now that we have ElastAlert configured, we’ll want to add a rule that identifies issues and alerts us when they occur.

We’ll take the most primitive rule: Whenever one of our services has anything logged to STDERR, send out an alert.

Let’s fill elastalert/rules/error.yaml:

# Rule name
name: Alert on any error
# Fill in USERNAME and PASSWORD here
es_host: "elasticsearch"
es_port: 9200
es_username: USERNAME
es_password: PASSWORD

# Elasticsearch Index to search for issues. As we are using
# filebeat for our log file delivery to Elasticsearch, all logs 
# will be in indices called filebeat-something.
index: filebeat*
timestamp_field: "@timestamp"

type: any
realert: 
  seconds: 0
# Alert whenever something was logged to STDERR
filter:
- query:
    query_string:
      query: "stream:\"stderr\""
# Aggregate alerts by the Docker container name, 
# so you will receive one alert per container. It will 
# contain all errors logged in the last five minutes.
# This avoids huge amounts of emails in case one of
# your services logs a lot of errors.
aggregation:
  minutes: 5
aggregation_key: "docker.container.name"
# Send out the alert via email.
alert:
  - email
# Use the container name as the email subject
alert_subject_args:
  - docker.container.name
alert_subject:
  - "Error on {0}"
# Recipient address for alert emails
email:
  - "your-email-address@example.com"
# For email delivery, we need to define an SMTP connection. Enter
# your SMTP host name, port and whether TLS/SSL is to be used
# (if not, it'll fall back to STARTSSL).
# We define the SMTP user data in smtp-auth.yml.
smtp_host: "smtp.example.com"
smtp_port: 25
smtp_ssl: false  
smtp_auth_file: "/opt/elastalert/smtp-auth.yml"

Now the last step is to store the SMTP user data in elastalert/smtp-auth.yml:

user: SMTP_USERNAME
password: SMTP_PASSWORD

Obviously, replace SMTP_USERNAME and SMTP_PASSWORD with your SMTP credentials.

Start the Service

Congratulations, you’re all set up. A simple docker-compose restart will set up your brand new ElastAlert service and connect it with Elasticsearch. As soon as one of your Docker services logs an error to STDERR, ElastAlert will find it, and after five minutes, you’ll conveniently receive an email with the details. And over are the times you needed to check Kibana every five minutes.