> ## Documentation Index
> Fetch the complete documentation index at: https://docs.easyalert.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Grafana

> Receive alerts from Grafana unified alerting and legacy alerts

## Overview

Grafana is a popular open-source platform for monitoring and observability. Easyalert supports both **Grafana Unified Alerting (8+)** and **Legacy Alerting (7.x)** formats.

<Info>
  Grafana 8+ uses Unified Alerting by default. Legacy alerting is deprecated but
  still supported for older installations.
</Info>

***

## Requirements

* **Grafana 8.0+** (for Unified Alerting) or **Grafana 7.x** (Legacy)
* **Easyalert account** and active tenant
* **Admin** or **Editor** permissions in Grafana

***

## Setup Instructions

<Steps>
  <Step title="Create Integration in Easyalert">
    1. Go to **Integrations** page from left menu
    2. Click **Add Integration** button
    3. Select **Grafana** as **Source Type**
    4. Enter a name (e.g., `Grafana Production`)
    5. Click **Create** to save
    6. Copy the generated **Webhook URL**

    > **Example:** `https://api.easyalert.io/api/v1/webhooks/ingest/wh_abc123...`
  </Step>

  <Step title="Create Contact Point (Unified Alerting)">
    1. Go to **Alerting → Contact points**
    2. Click **Add contact point** button
    3. Fill in the form:

    | Field           | Value                   |
    | --------------- | ----------------------- |
    | **Name**        | `Easyalert`             |
    | **Integration** | Select `Webhook`        |
    | **URL**         | Your copied Webhook URL |
    | **HTTP Method** | `POST`                  |

    4. Click **Test** button to verify connection
    5. Click **Save contact point**
  </Step>

  <Step title="Configure Notification Policy">
    1. Go to **Alerting → Notification policies**
    2. Edit **Default policy** or add **New nested policy**:

    | Field               | Value                   |
    | ------------------- | ----------------------- |
    | **Contact point**   | Select `Easyalert`      |
    | **Group by**        | `alertname`, `severity` |
    | **Group wait**      | `30s`                   |
    | **Group interval**  | `5m`                    |
    | **Repeat interval** | `4h`                    |
  </Step>

  <Step title="Test the Integration">
    Use the **Test** button in the contact point configuration to send a test alert
  </Step>
</Steps>

***

## Legacy Alerting Configuration (Grafana 7.x)

For Grafana 7.x and earlier with legacy alerting:

<Steps>
  <Step title="Create Notification Channel">
    1. Go to **Alerting → Notification channels** 2. Click **Add channel** 3.
       Select **webhook** as the type
  </Step>

  <Step title="Configure Webhook">
    * **Name**: Easyalert - **URL**: Your Easyalert webhook URL - **HTTP
      Method**: POST - **Send on all alerts**: Yes - **Include image**: Optional
  </Step>
</Steps>

***

## Creating Alert Rules

**Labels** you define in alert rules become **tags** in Easyalert.

### Add Labels (IMPORTANT)

In **Section 3: Add annotations and labels**, add labels for routing:

| Label         | Value                   | Description              |
| ------------- | ----------------------- | ------------------------ |
| `severity`    | `warning` or `critical` | Alert severity           |
| `customer`    | `AcmeCorp`              | Customer name (for MSPs) |
| `team`        | `backend`               | Responsible team         |
| `environment` | `production`            | Environment              |

### Add Annotations

| Annotation    | Example Value                              |
| ------------- | ------------------------------------------ |
| `summary`     | `High CPU usage on {{ $labels.instance }}` |
| `description` | `CPU usage is {{ $values.A }}%`            |

***

## Field Mapping

### Unified Alerting

| Grafana Field             | Easyalert Field                          |
| ------------------------- | ---------------------------------------- |
| `labels.alertname`        | Title                                    |
| `annotations.summary`     | Title (fallback)                         |
| `annotations.description` | Description                              |
| `status`                  | Status (firing → problem, resolved → ok) |
| `labels.severity`         | Severity                                 |
| `labels.instance`         | Host                                     |
| `labels.job`              | Service                                  |
| `fingerprint`             | Event ID                                 |

### Legacy Alerting

| Grafana Field | Easyalert Field |
| ------------- | --------------- |
| `ruleName`    | Title           |
| `message`     | Description     |
| `state`       | Status          |
| `ruleUrl`     | URL             |
| `evalMatches` | Metric details  |

***

## Severity Mapping

| Grafana Severity | Easyalert Severity |
| ---------------- | ------------------ |
| critical         | Critical           |
| error            | High               |
| warning          | Warning            |
| info             | Info               |

### Legacy State Mapping

| Grafana State | Easyalert Status  |
| ------------- | ----------------- |
| alerting      | Problem           |
| ok            | OK                |
| pending       | Problem (warning) |
| no\_data      | Problem (warning) |

***

## Label → Tag Conversion

**All labels** from Grafana are automatically available as **tags** in Easyalert.

### Example Conversion

Labels in Alert Rule:

```
alertname = HighCPUUsage
severity = critical
customer = acme
team = backend
environment = production
```

Available Tags in Easyalert:

```
tags.alertname = "HighCPUUsage"
tags.severity = "critical"
tags.customer = "acme"
tags.team = "backend"
tags.environment = "production"
```

### Routing Examples

**Escalation Routing:**

```
tags.customer equals "acme" → Acme Corp Policy
tags.team equals "database" → DBA Team Policy
tags.environment equals "production" → Critical Policy
```

**Notification Rules:**

```
tags.severity equals "critical" → call + sms + email
tags.severity equals "warning" → sms + email
```

***

## Fingerprint and Duplicate Detection

Grafana generates a unique **fingerprint** for each alert. Easyalert uses this fingerprint to:

1. Not create new incident if same alert comes again
2. Auto-close incident when `resolved` status arrives

***

## Example Alert Rules

### CPU Alert

```yaml theme={null}
# Query (Prometheus)
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Condition
IS ABOVE 80

# Labels
severity: warning
team: infra
customer: acme

# Annotations
summary: High CPU usage on {{ $labels.instance }}
```

### Memory Alert

```yaml theme={null}
# Query
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Condition
IS ABOVE 85

# Labels
severity: critical
team: infra
```

***

## Grafana-Specific Features

### Dashboard Links

Grafana includes helpful links in alerts:

| Field          | Description                     |
| -------------- | ------------------------------- |
| `dashboardURL` | Link to the dashboard           |
| `panelURL`     | Direct link to the panel        |
| `silenceURL`   | Quick link to silence the alert |
| `generatorURL` | Link to the alert rule          |

These are captured and displayed in the incident details.

***

## Test

```bash theme={null}
curl -X POST "YOUR_WEBHOOK_URL" \
  -H "Content-Type: application/json" \
  -d '{
    "receiver": "Easyalert",
    "status": "firing",
    "orgId": 1,
    "alerts": [{
      "status": "firing",
      "labels": {
        "alertname": "TestAlert",
        "severity": "warning",
        "customer": "test",
        "team": "backend"
      },
      "annotations": {
        "summary": "Test Alert",
        "description": "This is a test"
      },
      "startsAt": "2024-12-03T10:00:00Z",
      "fingerprint": "test-123"
    }],
    "externalURL": "http://grafana:3000"
  }'
```

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Alerts not being received">
    1. Test the contact point from Grafana UI 2. Check Grafana server logs for
       webhook errors 3. Verify network connectivity to Easyalert 4. Ensure
       notification policy routes to correct contact point
  </Accordion>

  <Accordion title="Labels not becoming tags">
    1. Check Labels section in alert rule 2. Verify labels are properly defined
    2. Review webhook samples in Easyalert
  </Accordion>

  <Accordion title="Resolved alerts not working">
    1. Verify "Disable Resolved Message" is NOT checked in contact point 2.
       Check notification policy includes resolved alerts 3. Verify alert rule has
       proper conditions for resolution
  </Accordion>

  <Accordion title="Missing dashboard links">
    1. Ensure `externalURL` is configured in Grafana 2. Check Grafana root URL
       settings
  </Accordion>

  <Accordion title="Legacy vs Unified format issues">
    1. Check which alerting system is enabled in Grafana 2. Easyalert
       auto-detects format, but verify payload structure 3. Review webhook samples
       for actual format being sent
  </Accordion>
</AccordionGroup>

***

## Best Practices

<AccordionGroup>
  <Accordion title="Use Unified Alerting">
    Migrate to Unified Alerting for better features and consistency with
    Prometheus/Alertmanager format.
  </Accordion>

  <Accordion title="Include Runbook URLs">
    Add `runbook_url` annotation to help responders quickly find remediation
    steps.
  </Accordion>

  <Accordion title="Organize with Folders">
    Use Grafana folders to organize alerts and include folder name in labels.
  </Accordion>

  <Accordion title="Configure External URL">
    Set Grafana's external URL properly so dashboard links work correctly.
  </Accordion>

  <Accordion title="Use Labels for Routing">
    Add labels like `customer`, `team`, `environment` for escalation routing in
    Easyalert.
  </Accordion>
</AccordionGroup>