Self-Hosted Alerting with Alerta and ntfy

Overview

Overview

When you run infrastructure long enough, you end up with alerts coming from multiple sources: Prometheus, Grafana, custom scripts, hardware sensors, and more. Managing these across different channels (email, Slack, PagerDuty) becomes unwieldy and often expensive for self-hosted setups.

This post walks through building a self-hosted alerting solution using two powerful tools:

  • Alerta — an alert aggregation and deduplication server
  • ntfy — a simple push notification service with support for actions

The goal: consolidate all your alerts into one place, then route them to any device via ntfy's flexible notification system with clickable action buttons.

Architecture

 1┌─────────────┐     ┌─────────────┐     ┌─────────────┐
 2│  Prometheus │     │   Grafana   │     │   Custom    │
 3│             │     │             │     │   Scripts   │
 4└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
 5       │                   │                   │
 6       └───────────────────┼───────────────────┘
 7 8                   ┌───────────────┐
 9                   │    Alerta     │
10                   │  (Aggregator) │
11                   └───────┬───────┘
12                           │ Webhook
1314                   ┌───────────────┐
15                   │     ntfy      │
16                   │  (Notifs +    │
17                   │   Actions)    │
18                   └───────┬───────┘
1920            ┌──────────────┼──────────────┐
21            ▼              ▼              ▼
22        Phone         Desktop        Browser

Setting Up ntfy

ntfy is a POSIX-friendly notification server that can self-host or use their cloud service. For this setup, we'll assume you have a self-hosted instance.

Basic Authentication

For token-based authentication, use the Authorization header:

1curl -H "Authorization: Bearer tk_YOUR_TOKEN_HERE" \
2  -d "Alert message here" \
3  https://ntfy.example.com/topic-name

Rich Notifications with Actions

The real power comes from ntfy's ability to send notifications with clickable action buttons:

 1curl -X POST "https://ntfy.example.com/power" \
 2  -H "Authorization: Bearer tk_YOUR_TOKEN_HERE" \
 3  -H "Title: Battery Overcharge Alert" \
 4  -H "Priority: 5" \
 5  -H "Tags: battery,rotating_light,warning" \
 6  -H "Actions: \
 7http, Shut off battery, https://ops.example.com/actions/battery/shutdown, method=POST, body='{\"topic\":\"power\",\"action\":\"shutdown\"}'; \
 8http, Turn on AC, https://ops.example.com/actions/ac/start, method=POST, body='{\"topic\":\"power\",\"action\":\"ac_on\"}'; \
 9http, Silence 12h, https://ops.example.com/actions/alerts/silence, method=POST, body='{\"topic\":\"power\",\"duration\":\"12h\"}'" \
10  -d "⚡ Battery voltage exceeded safe threshold (62.4 V > 58.8 V). Charging relay latched; thermal at 44.2 °C. Immediate action recommended."

Action header format:

1Actions: http,<button label>,<url>,method=<METHOD>,body='<JSON_BODY>[,<other options>]

Multiple actions are separated by semicolons. The method=POST and body= options let you execute actions directly from the notification.

Priority Levels

Priority Level Use Case
1 Min Debug info, low urgency
2 Low Non-urgent updates
3 Default Normal alerts
4 High Important alerts requiring ack
5 Emergency Critical alerts, makes noise

Self-Hosting ntfy with User Authentication

When self-hosting ntfy in a production environment, you'll want to set up proper user authentication and access control rather than relying on open access.

Docker Compose Configuration

 1version: "3.9"
 2services:
 3  ntfy:
 4    image: binwiederhier/ntfy:latest
 5    container_name: ntfy
 6    restart: unless-stopped
 7    command: "serve"
 8    environment:
 9      TZ: Australia/Adelaide
10      NTFY_BASE_URL: https://ntfy.example.com
11      NTFY_BEHIND_PROXY: "true"
12    volumes:
13      - ./config:/etc/ntfy
14      - ./cache:/var/cache/ntfy
15      - ./attachments:/var/lib/ntfy/attachments
16    ports:
17      - "8080:80"
18    healthcheck:
19      test: ["CMD", "wget", "-qO-", "http://localhost/health"]
20      interval: 30s
21      timeout: 5s
22      retries: 5

Server Configuration

Create config/server.yml with user authentication and access control:

 1# ntfy server config
 2base-url: https://ntfy.example.com
 3behind-proxy: true
 4
 5# HTTP listeners (TLS handled by your reverse proxy)
 6listen-http: ":80"
 7
 8# Persistence
 9cache-file: /var/cache/ntfy/cache.db
10attachments-dir: /var/lib/ntfy/attachments
11debug: true
12
13# Auth & topics
14enable-signups: false        # only admin-created users
15web-root: "."                # serve built-in web UI
16
17# Housekeeping
18attachment-total-size-limit: "2G"
19attachment-file-size-limit: "50M"
20attachment-expiry-duration: "168h"   # 7 days
21
22# User authentication
23auth-file: "/var/lib/ntfy/user.db"
24auth-default-access: "deny-all"
25
26# User accounts (bcrypt hashed passwords)
27auth-users:
28  - "admin:$2a$10$EXAMPLE_HASH:admin"
29  - "alerta:$2a$10$EXAMPLE_HASH:user"
30  - "jetspotter:$2a$10$EXAMPLE_HASH:user"
31
32# API tokens (format: user:token:description)
33auth-tokens:
34  - "alerta:tk_3gd7d2yftt4b8ixyfe9mnmro88o:Alerta token"
35  - "jetspotter:tk_94fzusxnkfg80yd6ej4gn6jvdsv:Jetspotter token"
36
37# Access control (format: user:topic:access)
38auth-access:
39  - "alerta:*:rw"
40  - "jetspotter:planes:rw"

Understanding the Configuration

User vs Token Authentication

ntfy supports two authentication mechanisms:

  1. User accounts (auth-users) — Username/password pairs for logging into the web UI or using basic auth with nopy (the ntfy CLI). The password hash format is bcrypt ($2a$10$...).

  2. API tokens (auth-tokens) — Long-lived tokens for programmatic access. Format is user:token:description. These are what you use in the Authorization: Bearer header when sending notifications.

Access Control

The auth-access section controls which users can access which topics:

Format Meaning
user:*:rw Read/write access to all topics
user:topic:rw Read/write access to specific topic
user:topic:r Read-only access to specific topic
user:topic:- Deny access to topic

Default Access Policy

auth-default-access: "deny-all" means users can only access topics explicitly listed in auth-access. This is the safest default for a production environment.

Generating Password Hashes

To create a bcrypt hash for a new user, you can use Python:

1import bcrypt
2password = "your-password-here"
3hash = bcrypt.hashpw(password.encode(), bcrypt.gensalt(rounds=10))
4print(hash.decode())

Or use the ntfy CLI if you have access to a running instance.

Adding a New User

  1. Generate a bcrypt password hash
  2. Add the user to auth-users:
    1auth-users:
    2  - "newuser:$2a$10$YOUR_HASH_HERE:user"
    
  3. Optionally grant topic access:
    1auth-access:
    2  - "newuser:alerts:rw"
    
  4. Restart the ntfy container

Adding a New Token

Tokens are simpler to add — just append to auth-tokens:

1auth-tokens:
2  - "alerta:tk_3gd7d2yftt4b8ixyfe9mnmro88o:Alerta token"
3  - "service-x:tk_NEW_TOKEN_HERE:Service X token"
4  - "service-x:alerts:rw"  # Grant access to alerts topic

Generating Tokens via CLI

You can also generate tokens directly from the ntfy container:

1docker exec -it ntfy ntfy token generate

This outputs a fresh token like tk_za8ww8krqrw84tctwfs9n7axcf89f. You then add it to auth-tokens with the appropriate user and description, and grant the user access to the relevant topic in auth-access.

Connecting Alerta to ntfy

With authentication configured, your Alerta ntfy plugin needs a token that has write access to your alert topic. Make sure your auth-access includes something like:

1auth-access:
2  - "alerta:alerts:rw"    # Alerta can publish to 'alerts' topic

Then in your Alerta config:

1NTFY_URL = "https://ntfy.example.com/alerts"
2NTFY_TOKEN = "tk_3gd7d2yftt4b8ixyfe9mnmro88o"

Setting Up Alerta with the ntfy Plugin

Installing Alerta

1pip install git+https://github.com/alerta/alerta.git@master
2pip install strenum pymongo

Creating the ntfy Plugin

The ntfy plugin for Alerta is available at the Alerta GitHub repo, but you can also create a custom one. Create a directory for your plugin:

1mkdir -p /code/ntfy_plugin

Dockerfile for Alerta

 1# syntax=docker/dockerfile:1
 2FROM python:3.12-slim
 3
 4WORKDIR /code
 5
 6RUN pip install --no-cache-dir --upgrade pip setuptools wheel && \
 7    pip install --no-cache-dir \
 8      git+https://github.com/alerta/alerta.git@master \
 9      strenum \
10      pymongo
11
12COPY ntfy_plugin/ /code/ntfy_plugin/
13RUN cd /code/ntfy_plugin && python3 setup.py install
14
15ENV ALERTA_SVR_CONF_FILE=/etc/alertad.conf
16
17USER alerta
18
19CMD ["alertad", "run", "--host", "0.0.0.0", "--port", "8080"]

Alerta Server Configuration

Create /etc/alertad.conf with your settings:

 1BASE_URL = 'https://alerta-api.example.com'
 2
 3CORS_ORIGINS = [
 4    'https://alerta.example.com'
 5]
 6
 7DEBUG = True
 8DATE_FORMAT_SHORT_TIME = 'HH:mm'
 9DATE_FORMAT_MEDIUM_DATE = 'EEE d MMM yyyy HH:mm'
10DATE_FORMAT_LONG_DATE = 'dd/MM/yyyy HH:mm:ss.sss'
11
12ALLOWED_ENVIRONMENTS = ['Production', 'Development', 'Testing']
13DEFAULT_ENVIRONMENT = 'Production'
14
15SIGNUP_ENABLED = False
16
17ADMIN_USERS = ['admin@example.com']
18DEFAULT_ADMIN_ROLE = 'admin'
19USER_DEFAULT_SCOPES = ['read', 'write:alerts']
20
21PLUGINS = ["ntfy"]
22
23AUTH_REQUIRED = True
24
25# ntfy plugin settings
26NTFY_URL = "https://ntfy.example.com/your-topic"
27NTFY_TOKEN = "tk_YOUR_NTFY_TOKEN"
28NTFY_MIN_SEVERITY = "major"

Docker Compose Setup

 1version: "3.9"
 2
 3services:
 4  mongo:
 5    image: mongo:6.0
 6    container_name: alerta-mongo
 7    restart: unless-stopped
 8    volumes:
 9      - mongo_data:/data/db
10
11  alerta:
12    build:
13      context: .
14      dockerfile: Dockerfile
15    depends_on:
16      - mongo
17    restart: unless-stopped
18    ports:
19      - "8088:8080"
20    environment:
21      DATABASE_URL: mongodb://mongo:27017/alerta
22      DEBUG: "True"
23      NTFY_URL: https://ntfy.example.com/your-topic
24      NTFY_TOKEN: tk_YOUR_NTFY_TOKEN
25      NTFY_MIN_SEVERITY: major
26      ALERTA_SVR_CONF_FILE: /etc/alertad.conf
27    volumes:
28      - ./alertad.conf:/etc/alertad.conf:ro
29
30volumes:
31  mongo_data:

Start with:

1docker compose up -d

Sending Alerts to Alerta

Alerta accepts alerts via a simple JSON webhook. Here's how to send an alert:

 1curl -X POST http://localhost:8088/alert \
 2  -H "Content-Type: application/json" \
 3  -H "Authorization: Key YOUR_ALERTA_API_KEY" \
 4  -d '{
 5    "resource": "db-server-01",
 6    "event": "DiskFull",
 7    "environment": "Production",
 8    "service": ["Database"],
 9    "severity": "critical",
10    "text": "Disk usage at 95% on /var",
11    "tags": ["storage", "filesystem"],
12    "origin": "custom-script",
13    "correlate": ["DiskFull", "DiskOK"]
14  }'

Alert Fields

Field Required Description
resource Yes Resource name (hostname, service, etc.)
event Yes Event name (DiskFull, CPUHigh, etc.)
environment Yes Production, Development, or Testing
service Yes Array of affected services
severity Yes normal, warning, minor, major, critical
text Yes Human-readable alert description
tags No Array of tags for categorization
origin No Source of the alert
correlate No Related events (for deduplication)

Severity Levels

Severity Description
normal Return to normal condition
warning Warning condition
minor Minor problem
major Major problem
critical Critical, immediate action needed

Managing Alerta

Accessing the Alerta Shell

For administrative tasks, access the Alerta MongoDB shell:

1mongosh --host localhost:27017 alerta

Common Database Operations

List all users:

1db.users.find().pretty()

List all API keys:

1db.keys.find().pretty()

Create a new API key:

1db.keys.insertOne({
2  key: "YOUR_NEW_API_KEY",
3  user: "username",
4  scopes: ["admin", "write", "read"],
5  text: "Description of key purpose",
6  expireTime: new Date("2026-12-31")
7})

Check collections:

 1show collections
 2// Output:
 3// alerts
 4// codes
 5// customers
 6// groups
 7// heartbeats
 8// keys
 9// metrics
10// perms
11// states
12// users

Putting It Together: Complete Alert Flow

Here's the complete flow from alert source to notification:

1. Your monitoring script detects an issue:

 1#!/usr/bin/env python3
 2import requests
 3import time
 4
 5def send_alert(resource, event, severity, text):
 6    payload = {
 7        "resource": resource,
 8        "event": event,
 9        "environment": "Production",
10        "service": ["WebApp"],
11        "severity": severity,
12        "text": text,
13        "tags": ["automation"],
14        "origin": "monitoring-script"
15    }
16    
17    response = requests.post(
18        "http://alerta-server:8088/alert",
19        headers={
20            "Content-Type": "application/json",
21            "Authorization": f"Key {requests.get('http://alerta-server:8088/api/key').text}"
22        },
23        json=payload
24    )
25    return response.ok
26
27# Example: High disk usage alert
28send_alert(
29    resource="web-server-01",
30    event="HighDiskUsage",
31    severity="warning",
32    text="Disk usage at 85% on /var/log"
33)

2. Alerta receives and deduplicates the alert

Alerta correlates alerts based on resource, event, and environment. Duplicate alerts within the correlation window are merged.

3. ntfy receives the webhook and sends notification

The Alerta ntfy plugin forwards the alert to your ntfy topic, which pushes to all subscribed devices with the appropriate priority and any configured actions.

ntfy Clients and Subscriptions

Mobile App

The ntfy Android and iOS apps let you subscribe to topics and receive push notifications with actions directly on your phone.

Desktop

For desktop, you can use:

  • Browser extension
  • ntfy desktop client
  • Or just keep the web interface open

Subscribing Without Authentication

For public topics, subscribe without auth:

1ntfy sub https://ntfy.example.com/your-topic

With Authentication

For private topics:

1ntfy sub https://ntfy.example.com/your-topic -u "your-access-token"
2
3
4
5Happy monitoring!
6
7
8Written by OpenCode :)