> ## Documentation Index
> Fetch the complete documentation index at: https://docs.easyalert.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Prometheus

> Receive alerts from Prometheus Alertmanager

## Overview

Prometheus is a leading open-source monitoring and alerting toolkit. Easyalert integrates with Prometheus via Alertmanager webhooks, receiving alert notifications when your alerting rules fire.

<Info>
  This integration receives webhooks from Alertmanager, not directly from
  Prometheus. Make sure you have Alertmanager configured.
</Info>

***

## Requirements

* **Prometheus** and **AlertManager** installed
* **Easyalert account** and active tenant
* Access to AlertManager configuration file

***

## Setup Instructions

<Steps>
  <Step title="Create Integration in Easyalert">
    1. Go to **Integrations** page from left menu
    2. Click **Add Integration** button
    3. Select **Prometheus** as **Source Type**
    4. Enter a name (e.g., `Kubernetes Prometheus`)
    5. Click **Create** to save
    6. Copy the generated **Webhook URL**

    > **Example:** `https://api.easyalert.io/api/v1/webhooks/ingest/wh_abc123...`
  </Step>

  <Step title="Configure Alertmanager">
    Edit your `alertmanager.yml` file:

    ```yaml theme={null}
    global:
      resolve_timeout: 5m

    route:
      group_by: ['alertname', 'severity']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: 'easyalert'

    receivers:
      - name: 'easyalert'
        webhook_configs:
          - url: 'YOUR_WEBHOOK_URL_HERE'
            send_resolved: true
            http_config:
              follow_redirects: true
    ```
  </Step>

  <Step title="Reload Alertmanager">
    Apply the configuration:

    ```bash theme={null}
    curl -X POST http://alertmanager:9093/-/reload
    ```
  </Step>

  <Step title="Test the Integration">
    Trigger a test alert and verify it appears in Easyalert
  </Step>
</Steps>

***

## Alertmanager Configuration

### Route Parameters

| Parameter         | Description                             | Recommended Value           |
| ----------------- | --------------------------------------- | --------------------------- |
| `group_by`        | Labels to group by                      | `['alertname', 'severity']` |
| `group_wait`      | Wait time after first alert             | `30s`                       |
| `group_interval`  | Interval for adding new alerts to group | `5m`                        |
| `repeat_interval` | Resend interval for same alert          | `4h`                        |

### Multi-Tenant Configuration (for MSPs)

Route different customers to different webhooks:

```yaml theme={null}
route:
  receiver: "default"
  routes:
    - match:
        customer: "acme"
      receiver: "acme-easyalert"
    - match:
        customer: "beta"
      receiver: "beta-easyalert"

receivers:
  - name: "acme-easyalert"
    webhook_configs:
      - url: "https://api.easyalert.io/api/v1/webhooks/ingest/wh_acme..."
        send_resolved: true

  - name: "beta-easyalert"
    webhook_configs:
      - url: "https://api.easyalert.io/api/v1/webhooks/ingest/wh_beta..."
        send_resolved: true
```

***

## Field Mapping

Easyalert automatically maps Alertmanager fields:

| Alertmanager Field        | Easyalert Field                          |
| ------------------------- | ---------------------------------------- |
| `labels.alertname`        | Title                                    |
| `annotations.summary`     | Title (fallback)                         |
| `annotations.description` | Description                              |
| `status`                  | Status (firing → problem, resolved → ok) |
| `labels.severity`         | Severity                                 |
| `labels.instance`         | Host                                     |
| `labels.job`              | Service                                  |
| `fingerprint`             | Event ID                                 |
| All labels                | Tags                                     |

***

## Severity Mapping

| Prometheus Label | Easyalert Severity |
| ---------------- | ------------------ |
| `critical`       | `critical`         |
| `error`          | `high`             |
| `warning`        | `warning`          |
| `info`           | `info`             |

***

## Status Handling

| Alertmanager Status | Easyalert Status | Action                   |
| ------------------- | ---------------- | ------------------------ |
| firing              | Problem          | Creates/updates incident |
| resolved            | OK               | Resolves incident        |

<Tip>
  Set `send_resolved: true` in your webhook config to automatically resolve
  incidents when alerts clear.
</Tip>

***

## Label → Tag Conversion

**All labels** from AlertManager are automatically converted to tags. This allows you to use any label for routing.

### Example Conversion

Labels in alert:

```yaml theme={null}
labels:
  alertname: HighCPUUsage
  severity: warning
  customer: acme
  team: backend
  environment: production
```

Available tags in Easyalert:

```
tags.alertname = "HighCPUUsage"
tags.severity = "warning"
tags.customer = "acme"
tags.team = "backend"
tags.environment = "production"
```

### Routing Examples

**Escalation Routing:**

```
tags.customer equals "acme" → Acme Corp Policy
tags.team equals "database" → DBA Team Policy
tags.environment equals "production" → Critical Policy
```

**Notification Rules:**

```
tags.severity equals "critical" → call + sms + email
tags.environment equals "staging" → email only
```

***

## Fingerprint and Duplicate Detection

AlertManager generates a unique **fingerprint** for each alert. Easyalert uses this fingerprint to detect duplicate alerts.

### How It Works

1. When alert arrives, fingerprint is checked
2. Same fingerprint + firing status = duplicate (skip)
3. Same fingerprint + resolved status = incident is closed
4. Different fingerprint = new incident

### repeat\_interval Setting

`repeat_interval` determines how often AlertManager resends the same alert:

```yaml theme={null}
route:
  repeat_interval: 4h # Resend every 4 hours
```

Thanks to duplicate detection, even if AlertManager resends the same alert, a new incident won't be created.

***

## Alert Rule Examples

### Host/VM Monitoring

```yaml theme={null}
groups:
  - name: host-alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
          team: infrastructure
          customer: acme
        annotations:
          summary: "High CPU usage: {{ $labels.instance }}"
          description: 'CPU usage above 80% (current: {{ $value | printf "%.1f" }}%)'

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        labels:
          severity: warning
          team: infrastructure
        annotations:
          summary: "High memory usage: {{ $labels.instance }}"
          description: "Memory usage above 85%"

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 10m
        labels:
          severity: critical
          team: infrastructure
        annotations:
          summary: "Critical disk space: {{ $labels.instance }}"
          description: "Remaining disk space below 15%"
```

### Kubernetes Alerts

```yaml theme={null}
groups:
  - name: kubernetes-alerts
    rules:
      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 3
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Pod crash loop: {{ $labels.namespace }}/{{ $labels.pod }}"
          description: "Pod restarted more than 3 times in last 15 minutes"

      - alert: DeploymentReplicasMismatch
        expr: kube_deployment_status_replicas_available != kube_deployment_spec_replicas
        for: 10m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Deployment replica mismatch: {{ $labels.deployment }}"
          description: "Expected: {{ $value }}, Active: different"
```

### Database Alerts

```yaml theme={null}
groups:
  - name: database-alerts
    rules:
      - alert: PostgreSQLDown
        expr: pg_up == 0
        for: 1m
        labels:
          severity: critical
          team: database
        annotations:
          summary: "PostgreSQL down: {{ $labels.instance }}"
          description: "PostgreSQL instance unreachable"

      - alert: PostgreSQLHighConnections
        expr: sum(pg_stat_activity_count) > 100
        for: 5m
        labels:
          severity: warning
          team: database
        annotations:
          summary: "High connection count: {{ $labels.instance }}"
          description: "Active connections: {{ $value }}"
```

***

## Deployment Examples

### Docker Compose

```yaml theme={null}
version: "3.8"
services:
  alertmanager:
    image: prom/alertmanager:latest
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    command:
      - "--config.file=/etc/alertmanager/alertmanager.yml"
```

### Kubernetes ConfigMap

```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
    route:
      receiver: 'easyalert'
    receivers:
      - name: 'easyalert'
        webhook_configs:
          - url: 'WEBHOOK_URL'
            send_resolved: true
```

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Alerts not being received">
    1. Check AlertManager logs for webhook errors 2. Verify the webhook URL is
       accessible from Alertmanager 3. Test connectivity: `curl -X POST
           YOUR_WEBHOOK_URL -d '{}'` 4. Check firewall rules between Alertmanager and
       internet 5. Test config with `amtool`
  </Accordion>

  <Accordion title="Labels not becoming tags">
    1. Verify alert rules include required labels 2. Check that labels aren't
       being dropped by Alertmanager routes 3. Review `group_by` settings
  </Accordion>

  <Accordion title="Resolved alerts not clearing incidents">
    1. Verify `send_resolved: true` in webhook config 2. Check that fingerprint
       matches between firing and resolved 3. Review Alertmanager routing to ensure
       resolved alerts reach webhook
  </Accordion>

  <Accordion title="Duplicate incidents">
    1. Check `group_by` configuration 2. Verify fingerprint is consistent 3.
       Review `group_interval` and `repeat_interval` settings
  </Accordion>

  <Accordion title="Connection refused">
    1. Check firewall/network settings 2. Verify AlertManager can reach
       Easyalert API 3. Test URL access from AlertManager server
  </Accordion>
</AccordionGroup>

### Test Command

```bash theme={null}
# Send test alert to AlertManager
curl -X POST http://localhost:9093/api/v2/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "customer": "test"
    },
    "annotations": {
      "summary": "Test alert",
      "description": "This is a test alert"
    }
  }]'
```

***

## Best Practices

<AccordionGroup>
  <Accordion title="Use Consistent Severity Labels">
    Standardize on `severity: critical|warning|info` across all alert rules for
    consistent mapping.
  </Accordion>

  <Accordion title="Include Good Annotations">
    Add meaningful `summary` and `description` annotations to help responders
    understand the alert.
  </Accordion>

  <Accordion title="Group Related Alerts">
    Use `group_by` to prevent notification storms from related alerts.
  </Accordion>

  <Accordion title="Enable Resolved Notifications">
    Always set `send_resolved: true` to automatically resolve incidents.
  </Accordion>

  <Accordion title="Add Runbook Links">
    Include `runbook_url` annotation for quick access to remediation steps.
  </Accordion>

  <Accordion title="Use Labels for Routing">
    Add labels like `customer`, `team`, `environment` for escalation routing in
    Easyalert.
  </Accordion>
</AccordionGroup>