Self-Hosted Alerting with Alerta and ntfy
Overview
Overview
When you run infrastructure long enough, you end up with alerts coming from multiple sources: Prometheus, Grafana, custom scripts, hardware sensors, and more. Managing these across different channels (email, Slack, PagerDuty) becomes unwieldy and often expensive for self-hosted setups.
This post walks through building a self-hosted alerting solution using two powerful tools:
- Alerta — an alert aggregation and deduplication server
- ntfy — a simple push notification service with support for actions
The goal: consolidate all your alerts into one place, then route them to any device via ntfy's flexible notification system with clickable action buttons.
Architecture
1┌─────────────┐ ┌─────────────┐ ┌─────────────┐
2│ Prometheus │ │ Grafana │ │ Custom │
3│ │ │ │ │ Scripts │
4└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
5 │ │ │
6 └───────────────────┼───────────────────┘
7 ▼
8 ┌───────────────┐
9 │ Alerta │
10 │ (Aggregator) │
11 └───────┬───────┘
12 │ Webhook
13 ▼
14 ┌───────────────┐
15 │ ntfy │
16 │ (Notifs + │
17 │ Actions) │
18 └───────┬───────┘
19 │
20 ┌──────────────┼──────────────┐
21 ▼ ▼ ▼
22 Phone Desktop Browser
Setting Up ntfy
ntfy is a POSIX-friendly notification server that can self-host or use their cloud service. For this setup, we'll assume you have a self-hosted instance.
Basic Authentication
For token-based authentication, use the Authorization header:
1curl -H "Authorization: Bearer tk_YOUR_TOKEN_HERE" \
2 -d "Alert message here" \
3 https://ntfy.example.com/topic-name
Rich Notifications with Actions
The real power comes from ntfy's ability to send notifications with clickable action buttons:
1curl -X POST "https://ntfy.example.com/power" \
2 -H "Authorization: Bearer tk_YOUR_TOKEN_HERE" \
3 -H "Title: Battery Overcharge Alert" \
4 -H "Priority: 5" \
5 -H "Tags: battery,rotating_light,warning" \
6 -H "Actions: \
7http, Shut off battery, https://ops.example.com/actions/battery/shutdown, method=POST, body='{\"topic\":\"power\",\"action\":\"shutdown\"}'; \
8http, Turn on AC, https://ops.example.com/actions/ac/start, method=POST, body='{\"topic\":\"power\",\"action\":\"ac_on\"}'; \
9http, Silence 12h, https://ops.example.com/actions/alerts/silence, method=POST, body='{\"topic\":\"power\",\"duration\":\"12h\"}'" \
10 -d "⚡ Battery voltage exceeded safe threshold (62.4 V > 58.8 V). Charging relay latched; thermal at 44.2 °C. Immediate action recommended."
Action header format:
1Actions: http,<button label>,<url>,method=<METHOD>,body='<JSON_BODY>[,<other options>]
Multiple actions are separated by semicolons. The method=POST and body= options let you execute actions directly from the notification.
Priority Levels
| Priority | Level | Use Case |
|---|---|---|
| 1 | Min | Debug info, low urgency |
| 2 | Low | Non-urgent updates |
| 3 | Default | Normal alerts |
| 4 | High | Important alerts requiring ack |
| 5 | Emergency | Critical alerts, makes noise |
Self-Hosting ntfy with User Authentication
When self-hosting ntfy in a production environment, you'll want to set up proper user authentication and access control rather than relying on open access.
Docker Compose Configuration
1version: "3.9"
2services:
3 ntfy:
4 image: binwiederhier/ntfy:latest
5 container_name: ntfy
6 restart: unless-stopped
7 command: "serve"
8 environment:
9 TZ: Australia/Adelaide
10 NTFY_BASE_URL: https://ntfy.example.com
11 NTFY_BEHIND_PROXY: "true"
12 volumes:
13 - ./config:/etc/ntfy
14 - ./cache:/var/cache/ntfy
15 - ./attachments:/var/lib/ntfy/attachments
16 ports:
17 - "8080:80"
18 healthcheck:
19 test: ["CMD", "wget", "-qO-", "http://localhost/health"]
20 interval: 30s
21 timeout: 5s
22 retries: 5
Server Configuration
Create config/server.yml with user authentication and access control:
1# ntfy server config
2base-url: https://ntfy.example.com
3behind-proxy: true
4
5# HTTP listeners (TLS handled by your reverse proxy)
6listen-http: ":80"
7
8# Persistence
9cache-file: /var/cache/ntfy/cache.db
10attachments-dir: /var/lib/ntfy/attachments
11debug: true
12
13# Auth & topics
14enable-signups: false # only admin-created users
15web-root: "." # serve built-in web UI
16
17# Housekeeping
18attachment-total-size-limit: "2G"
19attachment-file-size-limit: "50M"
20attachment-expiry-duration: "168h" # 7 days
21
22# User authentication
23auth-file: "/var/lib/ntfy/user.db"
24auth-default-access: "deny-all"
25
26# User accounts (bcrypt hashed passwords)
27auth-users:
28 - "admin:$2a$10$EXAMPLE_HASH:admin"
29 - "alerta:$2a$10$EXAMPLE_HASH:user"
30 - "jetspotter:$2a$10$EXAMPLE_HASH:user"
31
32# API tokens (format: user:token:description)
33auth-tokens:
34 - "alerta:tk_3gd7d2yftt4b8ixyfe9mnmro88o:Alerta token"
35 - "jetspotter:tk_94fzusxnkfg80yd6ej4gn6jvdsv:Jetspotter token"
36
37# Access control (format: user:topic:access)
38auth-access:
39 - "alerta:*:rw"
40 - "jetspotter:planes:rw"
Understanding the Configuration
User vs Token Authentication
ntfy supports two authentication mechanisms:
-
User accounts (
auth-users) — Username/password pairs for logging into the web UI or using basic auth withnopy(the ntfy CLI). The password hash format is bcrypt ($2a$10$...). -
API tokens (
auth-tokens) — Long-lived tokens for programmatic access. Format isuser:token:description. These are what you use in theAuthorization: Bearerheader when sending notifications.
Access Control
The auth-access section controls which users can access which topics:
| Format | Meaning |
|---|---|
user:*:rw |
Read/write access to all topics |
user:topic:rw |
Read/write access to specific topic |
user:topic:r |
Read-only access to specific topic |
user:topic:- |
Deny access to topic |
Default Access Policy
auth-default-access: "deny-all" means users can only access topics explicitly listed in auth-access. This is the safest default for a production environment.
Generating Password Hashes
To create a bcrypt hash for a new user, you can use Python:
1import bcrypt
2password = "your-password-here"
3hash = bcrypt.hashpw(password.encode(), bcrypt.gensalt(rounds=10))
4print(hash.decode())
Or use the ntfy CLI if you have access to a running instance.
Adding a New User
- Generate a bcrypt password hash
- Add the user to
auth-users:1auth-users: 2 - "newuser:$2a$10$YOUR_HASH_HERE:user" - Optionally grant topic access:
1auth-access: 2 - "newuser:alerts:rw" - Restart the ntfy container
Adding a New Token
Tokens are simpler to add — just append to auth-tokens:
1auth-tokens:
2 - "alerta:tk_3gd7d2yftt4b8ixyfe9mnmro88o:Alerta token"
3 - "service-x:tk_NEW_TOKEN_HERE:Service X token"
4 - "service-x:alerts:rw" # Grant access to alerts topic
Generating Tokens via CLI
You can also generate tokens directly from the ntfy container:
1docker exec -it ntfy ntfy token generate
This outputs a fresh token like tk_za8ww8krqrw84tctwfs9n7axcf89f. You then add it to auth-tokens with the appropriate user and description, and grant the user access to the relevant topic in auth-access.
Connecting Alerta to ntfy
With authentication configured, your Alerta ntfy plugin needs a token that has write access to your alert topic. Make sure your auth-access includes something like:
1auth-access:
2 - "alerta:alerts:rw" # Alerta can publish to 'alerts' topic
Then in your Alerta config:
1NTFY_URL = "https://ntfy.example.com/alerts"
2NTFY_TOKEN = "tk_3gd7d2yftt4b8ixyfe9mnmro88o"
Setting Up Alerta with the ntfy Plugin
Installing Alerta
1pip install git+https://github.com/alerta/alerta.git@master
2pip install strenum pymongo
Creating the ntfy Plugin
The ntfy plugin for Alerta is available at the Alerta GitHub repo, but you can also create a custom one. Create a directory for your plugin:
1mkdir -p /code/ntfy_plugin
Dockerfile for Alerta
1# syntax=docker/dockerfile:1
2FROM python:3.12-slim
3
4WORKDIR /code
5
6RUN pip install --no-cache-dir --upgrade pip setuptools wheel && \
7 pip install --no-cache-dir \
8 git+https://github.com/alerta/alerta.git@master \
9 strenum \
10 pymongo
11
12COPY ntfy_plugin/ /code/ntfy_plugin/
13RUN cd /code/ntfy_plugin && python3 setup.py install
14
15ENV ALERTA_SVR_CONF_FILE=/etc/alertad.conf
16
17USER alerta
18
19CMD ["alertad", "run", "--host", "0.0.0.0", "--port", "8080"]
Alerta Server Configuration
Create /etc/alertad.conf with your settings:
1BASE_URL = 'https://alerta-api.example.com'
2
3CORS_ORIGINS = [
4 'https://alerta.example.com'
5]
6
7DEBUG = True
8DATE_FORMAT_SHORT_TIME = 'HH:mm'
9DATE_FORMAT_MEDIUM_DATE = 'EEE d MMM yyyy HH:mm'
10DATE_FORMAT_LONG_DATE = 'dd/MM/yyyy HH:mm:ss.sss'
11
12ALLOWED_ENVIRONMENTS = ['Production', 'Development', 'Testing']
13DEFAULT_ENVIRONMENT = 'Production'
14
15SIGNUP_ENABLED = False
16
17ADMIN_USERS = ['admin@example.com']
18DEFAULT_ADMIN_ROLE = 'admin'
19USER_DEFAULT_SCOPES = ['read', 'write:alerts']
20
21PLUGINS = ["ntfy"]
22
23AUTH_REQUIRED = True
24
25# ntfy plugin settings
26NTFY_URL = "https://ntfy.example.com/your-topic"
27NTFY_TOKEN = "tk_YOUR_NTFY_TOKEN"
28NTFY_MIN_SEVERITY = "major"
Docker Compose Setup
1version: "3.9"
2
3services:
4 mongo:
5 image: mongo:6.0
6 container_name: alerta-mongo
7 restart: unless-stopped
8 volumes:
9 - mongo_data:/data/db
10
11 alerta:
12 build:
13 context: .
14 dockerfile: Dockerfile
15 depends_on:
16 - mongo
17 restart: unless-stopped
18 ports:
19 - "8088:8080"
20 environment:
21 DATABASE_URL: mongodb://mongo:27017/alerta
22 DEBUG: "True"
23 NTFY_URL: https://ntfy.example.com/your-topic
24 NTFY_TOKEN: tk_YOUR_NTFY_TOKEN
25 NTFY_MIN_SEVERITY: major
26 ALERTA_SVR_CONF_FILE: /etc/alertad.conf
27 volumes:
28 - ./alertad.conf:/etc/alertad.conf:ro
29
30volumes:
31 mongo_data:
Start with:
1docker compose up -d
Sending Alerts to Alerta
Alerta accepts alerts via a simple JSON webhook. Here's how to send an alert:
1curl -X POST http://localhost:8088/alert \
2 -H "Content-Type: application/json" \
3 -H "Authorization: Key YOUR_ALERTA_API_KEY" \
4 -d '{
5 "resource": "db-server-01",
6 "event": "DiskFull",
7 "environment": "Production",
8 "service": ["Database"],
9 "severity": "critical",
10 "text": "Disk usage at 95% on /var",
11 "tags": ["storage", "filesystem"],
12 "origin": "custom-script",
13 "correlate": ["DiskFull", "DiskOK"]
14 }'
Alert Fields
| Field | Required | Description |
|---|---|---|
| resource | Yes | Resource name (hostname, service, etc.) |
| event | Yes | Event name (DiskFull, CPUHigh, etc.) |
| environment | Yes | Production, Development, or Testing |
| service | Yes | Array of affected services |
| severity | Yes | normal, warning, minor, major, critical |
| text | Yes | Human-readable alert description |
| tags | No | Array of tags for categorization |
| origin | No | Source of the alert |
| correlate | No | Related events (for deduplication) |
Severity Levels
| Severity | Description |
|---|---|
| normal | Return to normal condition |
| warning | Warning condition |
| minor | Minor problem |
| major | Major problem |
| critical | Critical, immediate action needed |
Managing Alerta
Accessing the Alerta Shell
For administrative tasks, access the Alerta MongoDB shell:
1mongosh --host localhost:27017 alerta
Common Database Operations
List all users:
1db.users.find().pretty()
List all API keys:
1db.keys.find().pretty()
Create a new API key:
1db.keys.insertOne({
2 key: "YOUR_NEW_API_KEY",
3 user: "username",
4 scopes: ["admin", "write", "read"],
5 text: "Description of key purpose",
6 expireTime: new Date("2026-12-31")
7})
Check collections:
1show collections
2// Output:
3// alerts
4// codes
5// customers
6// groups
7// heartbeats
8// keys
9// metrics
10// perms
11// states
12// users
Putting It Together: Complete Alert Flow
Here's the complete flow from alert source to notification:
1. Your monitoring script detects an issue:
1#!/usr/bin/env python3
2import requests
3import time
4
5def send_alert(resource, event, severity, text):
6 payload = {
7 "resource": resource,
8 "event": event,
9 "environment": "Production",
10 "service": ["WebApp"],
11 "severity": severity,
12 "text": text,
13 "tags": ["automation"],
14 "origin": "monitoring-script"
15 }
16
17 response = requests.post(
18 "http://alerta-server:8088/alert",
19 headers={
20 "Content-Type": "application/json",
21 "Authorization": f"Key {requests.get('http://alerta-server:8088/api/key').text}"
22 },
23 json=payload
24 )
25 return response.ok
26
27# Example: High disk usage alert
28send_alert(
29 resource="web-server-01",
30 event="HighDiskUsage",
31 severity="warning",
32 text="Disk usage at 85% on /var/log"
33)
2. Alerta receives and deduplicates the alert
Alerta correlates alerts based on resource, event, and environment. Duplicate alerts within the correlation window are merged.
3. ntfy receives the webhook and sends notification
The Alerta ntfy plugin forwards the alert to your ntfy topic, which pushes to all subscribed devices with the appropriate priority and any configured actions.
ntfy Clients and Subscriptions
Mobile App
The ntfy Android and iOS apps let you subscribe to topics and receive push notifications with actions directly on your phone.
Desktop
For desktop, you can use:
- Browser extension
- ntfy desktop client
- Or just keep the web interface open
Subscribing Without Authentication
For public topics, subscribe without auth:
1ntfy sub https://ntfy.example.com/your-topic
With Authentication
For private topics:
1ntfy sub https://ntfy.example.com/your-topic -u "your-access-token"
2
3
4
5Happy monitoring!
6
7
8Written by OpenCode :)