Prometheus

Overview

Prometheus is a leading open-source monitoring and alerting toolkit. Easyalert integrates with Prometheus via Alertmanager webhooks, receiving alert notifications when your alerting rules fire.

This integration receives webhooks from Alertmanager, not directly from Prometheus. Make sure you have Alertmanager configured.

Requirements

Prometheus and AlertManager installed
Easyalert account and active tenant
Access to AlertManager configuration file

Setup Instructions

Create Integration in Easyalert

Go to Integrations page from left menu
Click Add Integration button
Select Prometheus as Source Type
Enter a name (e.g., Kubernetes Prometheus)
Click Create to save
Copy the generated Webhook URL

Example: https://api.easyalert.io/api/v1/webhooks/ingest/wh_abc123...

Configure Alertmanager

Edit your alertmanager.yml file:

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'easyalert'

receivers:
  - name: 'easyalert'
    webhook_configs:
      - url: 'YOUR_WEBHOOK_URL_HERE'
        send_resolved: true
        http_config:
          follow_redirects: true

Reload Alertmanager

Apply the configuration:

curl -X POST http://alertmanager:9093/-/reload

Test the Integration

Trigger a test alert and verify it appears in Easyalert

Alertmanager Configuration

Route Parameters

Parameter	Description	Recommended Value
`group_by`	Labels to group by	`['alertname', 'severity']`
`group_wait`	Wait time after first alert	`30s`
`group_interval`	Interval for adding new alerts to group	`5m`
`repeat_interval`	Resend interval for same alert	`4h`

Multi-Tenant Configuration (for MSPs)

Route different customers to different webhooks:

route:
  receiver: "default"
  routes:
    - match:
        customer: "acme"
      receiver: "acme-easyalert"
    - match:
        customer: "beta"
      receiver: "beta-easyalert"

receivers:
  - name: "acme-easyalert"
    webhook_configs:
      - url: "https://api.easyalert.io/api/v1/webhooks/ingest/wh_acme..."
        send_resolved: true

  - name: "beta-easyalert"
    webhook_configs:
      - url: "https://api.easyalert.io/api/v1/webhooks/ingest/wh_beta..."
        send_resolved: true

Field Mapping

Easyalert automatically maps Alertmanager fields:

Alertmanager Field	Easyalert Field
`labels.alertname`	Title
`annotations.summary`	Title (fallback)
`annotations.description`	Description
`status`	Status (firing → problem, resolved → ok)
`labels.severity`	Severity
`labels.instance`	Host
`labels.job`	Service
`fingerprint`	Event ID
All labels	Tags

Severity Mapping

Prometheus Label	Easyalert Severity
`critical`	`critical`
`error`	`high`
`warning`	`warning`
`info`	`info`

Status Handling

Alertmanager Status	Easyalert Status	Action
firing	Problem	Creates/updates incident
resolved	OK	Resolves incident

Set send_resolved: true in your webhook config to automatically resolve incidents when alerts clear.

Label → Tag Conversion

All labels from AlertManager are automatically converted to tags. This allows you to use any label for routing.

Example Conversion

Labels in alert:

labels:
  alertname: HighCPUUsage
  severity: warning
  customer: acme
  team: backend
  environment: production

Available tags in Easyalert:

tags.alertname = "HighCPUUsage"
tags.severity = "warning"
tags.customer = "acme"
tags.team = "backend"
tags.environment = "production"

Routing Examples

Escalation Routing:

tags.customer equals "acme" → Acme Corp Policy
tags.team equals "database" → DBA Team Policy
tags.environment equals "production" → Critical Policy

Notification Rules:

tags.severity equals "critical" → call + sms + email
tags.environment equals "staging" → email only

Fingerprint and Duplicate Detection

AlertManager generates a unique fingerprint for each alert. Easyalert uses this fingerprint to detect duplicate alerts.

How It Works

When alert arrives, fingerprint is checked
Same fingerprint + firing status = duplicate (skip)
Same fingerprint + resolved status = incident is closed
Different fingerprint = new incident

repeat_interval Setting

repeat_interval determines how often AlertManager resends the same alert:

route:
  repeat_interval: 4h # Resend every 4 hours

Thanks to duplicate detection, even if AlertManager resends the same alert, a new incident won’t be created.

Alert Rule Examples

Host/VM Monitoring

groups:
  - name: host-alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
          team: infrastructure
          customer: acme
        annotations:
          summary: "High CPU usage: {{ $labels.instance }}"
          description: 'CPU usage above 80% (current: {{ $value | printf "%.1f" }}%)'

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        labels:
          severity: warning
          team: infrastructure
        annotations:
          summary: "High memory usage: {{ $labels.instance }}"
          description: "Memory usage above 85%"

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 10m
        labels:
          severity: critical
          team: infrastructure
        annotations:
          summary: "Critical disk space: {{ $labels.instance }}"
          description: "Remaining disk space below 15%"

Kubernetes Alerts

groups:
  - name: kubernetes-alerts
    rules:
      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 3
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Pod crash loop: {{ $labels.namespace }}/{{ $labels.pod }}"
          description: "Pod restarted more than 3 times in last 15 minutes"

      - alert: DeploymentReplicasMismatch
        expr: kube_deployment_status_replicas_available != kube_deployment_spec_replicas
        for: 10m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Deployment replica mismatch: {{ $labels.deployment }}"
          description: "Expected: {{ $value }}, Active: different"

Database Alerts

groups:
  - name: database-alerts
    rules:
      - alert: PostgreSQLDown
        expr: pg_up == 0
        for: 1m
        labels:
          severity: critical
          team: database
        annotations:
          summary: "PostgreSQL down: {{ $labels.instance }}"
          description: "PostgreSQL instance unreachable"

      - alert: PostgreSQLHighConnections
        expr: sum(pg_stat_activity_count) > 100
        for: 5m
        labels:
          severity: warning
          team: database
        annotations:
          summary: "High connection count: {{ $labels.instance }}"
          description: "Active connections: {{ $value }}"

Deployment Examples

Docker Compose

version: "3.8"
services:
  alertmanager:
    image: prom/alertmanager:latest
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    command:
      - "--config.file=/etc/alertmanager/alertmanager.yml"

Kubernetes ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
    route:
      receiver: 'easyalert'
    receivers:
      - name: 'easyalert'
        webhook_configs:
          - url: 'WEBHOOK_URL'
            send_resolved: true

Troubleshooting

Alerts not being received

Check AlertManager logs for webhook errors 2. Verify the webhook URL is accessible from Alertmanager 3. Test connectivity: curl -X POST YOUR_WEBHOOK_URL -d '{}' 4. Check firewall rules between Alertmanager and internet 5. Test config with amtool

Labels not becoming tags

Verify alert rules include required labels 2. Check that labels aren’t being dropped by Alertmanager routes 3. Review group_by settings

Resolved alerts not clearing incidents

Verify send_resolved: true in webhook config 2. Check that fingerprint matches between firing and resolved 3. Review Alertmanager routing to ensure resolved alerts reach webhook

Duplicate incidents

Check group_by configuration 2. Verify fingerprint is consistent 3. Review group_interval and repeat_interval settings

Connection refused

Check firewall/network settings 2. Verify AlertManager can reach Easyalert API 3. Test URL access from AlertManager server

Test Command

# Send test alert to AlertManager
curl -X POST http://localhost:9093/api/v2/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "customer": "test"
    },
    "annotations": {
      "summary": "Test alert",
      "description": "This is a test alert"
    }
  }]'

Best Practices

Use Consistent Severity Labels

Standardize on severity: critical|warning|info across all alert rules for consistent mapping.

Include Good Annotations

Add meaningful summary and description annotations to help responders understand the alert.

Enable Resolved Notifications

Always set send_resolved: true to automatically resolve incidents.

Add Runbook Links

Include runbook_url annotation for quick access to remediation steps.

Use Labels for Routing

Add labels like customer, team, environment for escalation routing in Easyalert.

Getting started

Analytics

Synthetics

Management

Integrations

Synthetics

Configuration

Access Management

Overview

Requirements

Setup Instructions

Alertmanager Configuration

Route Parameters

Multi-Tenant Configuration (for MSPs)

Field Mapping

Severity Mapping

Status Handling

Label → Tag Conversion

Example Conversion

Routing Examples

Fingerprint and Duplicate Detection

How It Works

repeat_interval Setting

Alert Rule Examples

Host/VM Monitoring

Kubernetes Alerts

Database Alerts

Deployment Examples

Docker Compose

Kubernetes ConfigMap

Troubleshooting

Test Command

Best Practices

​Overview

​Requirements

​Setup Instructions

​Alertmanager Configuration

​Route Parameters

​Multi-Tenant Configuration (for MSPs)

​Field Mapping

​Severity Mapping

​Status Handling

​Label → Tag Conversion

​Example Conversion

​Routing Examples

​Fingerprint and Duplicate Detection

​How It Works

​repeat_interval Setting

​Alert Rule Examples

​Host/VM Monitoring

​Kubernetes Alerts

​Database Alerts

​Deployment Examples

​Docker Compose

​Kubernetes ConfigMap

​Troubleshooting

​Test Command

​Best Practices

Overview

Requirements

Setup Instructions

Alertmanager Configuration

Route Parameters

Multi-Tenant Configuration (for MSPs)

Field Mapping

Severity Mapping

Status Handling

Label → Tag Conversion

Example Conversion

Routing Examples

Fingerprint and Duplicate Detection

How It Works

repeat_interval Setting

Alert Rule Examples

Host/VM Monitoring

Kubernetes Alerts

Database Alerts

Deployment Examples

Docker Compose

Kubernetes ConfigMap

Troubleshooting

Test Command

Best Practices