mirror of https://github.com/zulip/zulip.git
integrations: Update New Relic integration.
Newrelic has introduced a new update. This commit updates the exisiting integration by adding support for new types of payloads and cleaning up legacy codes and fixtures. Fixes #29729. Co-authored-by: Pieter CK <pieterceka123@gmail.com> Co-authored-by: Lauryn Menard <lauryn@zulip.com>
This commit is contained in:
Giovanni Silva
Tim Abbott
parent
424e5d6ad0
commit
70d559cafa
Binary file not shown.
|
Before Width: | Height: | Size: 4.3 KiB After Width: | Height: | Size: 3.1 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 774 B After Width: | Height: | Size: 356 B |
Binary file not shown.
|
Before Width: | Height: | Size: 47 KiB After Width: | Height: | Size: 71 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 36 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 18 KiB |
|
|
@ -793,11 +793,7 @@ DOC_SCREENSHOT_CONFIG: dict[str, list[BaseScreenshotConfig]] = {
|
|||
"mention": [ScreenshotConfig("webfeeds.json")],
|
||||
"nagios": [BaseScreenshotConfig("service_notify.json")],
|
||||
"netlify": [ScreenshotConfig("deploy_building.json")],
|
||||
"newrelic": [
|
||||
ScreenshotConfig("incident_active_new.json", "001.png"),
|
||||
ScreenshotConfig("incident_acknowledged_new.json", "002.png"),
|
||||
ScreenshotConfig("incident_closed_new.json", "003.png"),
|
||||
],
|
||||
"newrelic": [ScreenshotConfig("incident_activated_new_default_payload.json", "001.png")],
|
||||
"opencollective": [ScreenshotConfig("one_time_donation.json")],
|
||||
"opsgenie": [ScreenshotConfig("addrecipient.json", image_name="000.png")],
|
||||
"pagerduty": [ScreenshotConfig("trigger_v2.json")],
|
||||
|
|
|
|||
|
|
@ -1,27 +1,78 @@
|
|||
New Relic can send messages to a Zulip channel for incidents.
|
||||
# Zulip New Relic integration
|
||||
|
||||
Get Zulip notification for New Relic incidents.
|
||||
|
||||
{start_tabs}
|
||||
|
||||
1. {!create-channel.md!}
|
||||
|
||||
1. {!create-an-incoming-webhook.md!}
|
||||
|
||||
1. {!generate-integration-url.md!}
|
||||
1. {!generate-webhook-url-basic.md!}
|
||||
|
||||
1. On [New Relic](https://one.newrelic.com),
|
||||
select **Alerts & AI**.
|
||||
1. In New Relic, go to the **Alerts** menu, and select **Destinations**.
|
||||
Choose **Webhook** in the **Add a destination** section.
|
||||
|
||||
1. Navigate to **Notification channels**.
|
||||
1. Set a **Webhook name**, such as `Zulip`. Set the **Endpoint URL** to
|
||||
the URL generated above. Click **Save destination**.
|
||||
|
||||
1. Create a new notification channel. Select channel type of **Webhook**, choose a name (e.g., "Zulip"), enter the webhook url created earlier as **Base Url**.
|
||||
1. In the **Alerts** menu, select **Workflows**. Click on
|
||||
**+ Add a Workflow**.
|
||||
|
||||
1. It should look like:
|
||||
|
||||
1. Set your workflow name, and filter the trigger conditions. In the
|
||||
**Notify** section, choose **Webhook**. In the **Edit notification
|
||||
message** menu, select the destination for Zulip created above.
|
||||
|
||||
1. The webhook works with the default payload, click **Create channel**.
|
||||
1. In the **Payload** section, you can configure the payload for this
|
||||
workflow. The default payload template is sufficient to get the
|
||||
integration working, but using the message template below will enable
|
||||
the integration to notify you of any **acknowledged** New Relic
|
||||
incidents. To include additional custom fields, refer to
|
||||
[configuration options](#configuration-options):
|
||||
|
||||
1. After creating the channel send a test notification to make sure it works.
|
||||
{
|
||||
{% raw %}
|
||||
"id": {{ json issueId }},
|
||||
"issueUrl": {{ json issuePageUrl }},
|
||||
"title": {{ json annotations.title.[0] }},
|
||||
"priority": {{ json priority }},
|
||||
"totalIncidents": {{json totalIncidents}},
|
||||
"state": {{ json state }},
|
||||
"createdAt": {{ createdAt }},
|
||||
"updatedAt": {{ updatedAt }},
|
||||
"alertPolicyNames": {{ json accumulations.policyName }},
|
||||
"alertConditionNames": {{ json accumulations.conditionName }},
|
||||
"owner": {{ json owner }},
|
||||
"zulipCustomFields": {}
|
||||
{% endraw %}
|
||||
}
|
||||
|
||||
1. Click **Send test notification** to receive a test notification. Select
|
||||
**Save message**, and click **Activate Workflow**.
|
||||
|
||||
{end_tabs}
|
||||
|
||||
{!congrats.md!}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### Configuration options
|
||||
|
||||
* With New Relic's [custom payload feature][1], you can include custom
|
||||
fields in your Zulip notifications by configuring a `zulipCustomFields`
|
||||
dictionary in your notification payload template. The keys of
|
||||
`zulipCustomFields` will be displayed in the Zulip notification
|
||||
message, so we recommend that they be human-readable and descriptive.
|
||||
The values of the dictionary can be strings, integers, booleans, or
|
||||
lists of the those same data types.
|
||||
|
||||
### Related documentation
|
||||
|
||||
* [**New Relic webhook integration**][2]
|
||||
|
||||
* [**New Relic message templates**][1]
|
||||
|
||||
{!webhooks-url-specification.md!}
|
||||
|
||||
[1]: https://docs.newrelic.com/docs/alerts-applied-intelligence/notifications/message-templates/
|
||||
[2]: https://docs.newrelic.com/docs/alerts/get-notified/notification-integrations/#webhook
|
||||
|
|
|
|||
|
|
@ -0,0 +1,11 @@
|
|||
{
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK",
|
||||
"title": "PIETER-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'",
|
||||
"priority": "CRITICAL",
|
||||
"totalIncidents": 1,
|
||||
"state": "CREATED",
|
||||
"createdAt": 1713592289021,
|
||||
"updatedAt": 1713592289021,
|
||||
"alertPolicyNames": ["Golden Signals"],
|
||||
"alertConditionNames": ["High CPU"]
|
||||
}
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
{
|
||||
"id": "13bbcdca-f0b6-470d-b0be-b34583c58869",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK",
|
||||
"title": "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'",
|
||||
"priority": "HIGH",
|
||||
"impactedEntities": [
|
||||
"PIETER-UBUNTU"
|
||||
],
|
||||
"totalIncidents": 1,
|
||||
"state": "ACTIVATED",
|
||||
"trigger": "USER_ACTION",
|
||||
"isCorrelated": "false",
|
||||
"createdAt": 1713769949493,
|
||||
"updatedAt": 1713770077412,
|
||||
"sources": [
|
||||
"newrelic"
|
||||
],
|
||||
"alertPolicyNames": [
|
||||
"Golden Signals"
|
||||
],
|
||||
"alertConditionNames": [
|
||||
"Storage on Host Exceeded Threshold"
|
||||
],
|
||||
"workflowName": "issue workflow"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"id": "3576f543-dc3c-4d97-9f16-5c81f35195cb",
|
||||
"details": "Violation description test.",
|
||||
"alertPolicyNames": ["Test policy name"],
|
||||
"condition_name": "Server Down",
|
||||
"createdAt": 1605133931151,
|
||||
"state": "acknowledged",
|
||||
"owner": "Alice",
|
||||
"issueUrl": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"incident_id": 1234,
|
||||
"details": "Violation description test.",
|
||||
"policy_name": "Test policy name",
|
||||
"condition_name": "Server Down",
|
||||
"timestamp": 1605133931151,
|
||||
"current_state": "acknowledged",
|
||||
"owner": "Alice",
|
||||
"incident_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
{
|
||||
"id": "13bbcdca-f0b6-470d-b0be-b34583c58869",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK",
|
||||
"title": "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'",
|
||||
"priority": "HIGH",
|
||||
"totalIncidents": 1,
|
||||
"state": "ACTIVATED",
|
||||
"createdAt": 1713769949493,
|
||||
"updatedAt": 1713770077412,
|
||||
"alertPolicyNames": [
|
||||
"Golden Signals"
|
||||
],
|
||||
"alertConditionNames": [
|
||||
"Storage on Host Exceeded Threshold"
|
||||
],
|
||||
"owner": "Pieter Cardillo Kwok",
|
||||
"zulipCustomFields": {
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
{
|
||||
"id": "c5faa7e6-7b54-402d-af79-f99601e0278c",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/4420147/issues/c5faa7e6-7b54-402d-af79-f99601e0278c?notifier=WEBHOOK",
|
||||
"title": "zulip_app query result is > 1.0 for 1 minutes on 'Zulip Server Low Storage'",
|
||||
"priority": "CRITICAL",
|
||||
"impactedEntities": [
|
||||
"zulip_app"
|
||||
],
|
||||
"totalIncidents": 1,
|
||||
"state": "ACTIVATED",
|
||||
"trigger": "STATE_CHANGE",
|
||||
"isCorrelated": false,
|
||||
"createdAt": 1713755131352,
|
||||
"updatedAt": 1713769708699,
|
||||
"sources": [
|
||||
"newrelic"
|
||||
],
|
||||
"alertPolicyNames": [
|
||||
"Golden Signals"
|
||||
],
|
||||
"alertConditionNames": [
|
||||
"Zulip Server Low Storage"
|
||||
],
|
||||
"workflowName": "testWorkflow"
|
||||
}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
{
|
||||
"id": "13bbcdca-f0b6-470d-b0be-b34583c58869",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK",
|
||||
"title": "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'",
|
||||
"priority": "HIGH",
|
||||
"totalIncidents": 1,
|
||||
"state": "ACTIVATED",
|
||||
"createdAt": 1713769949493,
|
||||
"updatedAt": 1713769949494,
|
||||
"alertPolicyNames": [
|
||||
"Golden Signals"
|
||||
],
|
||||
"alertConditionNames": [
|
||||
"Storage on Host Exceeded Threshold"
|
||||
],
|
||||
"owner": "N/A",
|
||||
"zulipCustomFields": {
|
||||
"Your custom payload": "somedata123"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"id": "8ceed342-f305-4bfa-adb8-97ba93f5dd26",
|
||||
"details": "Violation description test.",
|
||||
"alertPolicyNames": ["Test policy name"],
|
||||
"condition_name": "Server Down",
|
||||
"createdAt": 1605133931151,
|
||||
"state": "activated",
|
||||
"owner": "",
|
||||
"issueUrl": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
{
|
||||
"id": "95a9344a-2590-48ce-8d83-07e28b6d22c6",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK",
|
||||
"title": "main_app-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'",
|
||||
"priority": "CRITICAL",
|
||||
"impactedEntities": ["main_app-UBUNTU"],
|
||||
"totalIncidents": 1,
|
||||
"state": "CLOSED",
|
||||
"trigger": "INCIDENT_CLOSED",
|
||||
"isCorrelated": false,
|
||||
"createdAt": 1713766590228,
|
||||
"updatedAt": 1713766657383,
|
||||
"sources": ["newrelic"],
|
||||
"alertPolicyNames": ["Golden Signals"],
|
||||
"alertConditionNames": ["High CPU"],
|
||||
"workflowName": "DBA Team workflow"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"id": "f0d98b28-bf9d-49e7-b9d0-ac7cbb52e73a",
|
||||
"details": "Violation description test.",
|
||||
"alertPolicyNames": ["Test policy name"],
|
||||
"condition_name": "Server Down",
|
||||
"createdAt": 1605133931151,
|
||||
"state": "closed",
|
||||
"owner": "",
|
||||
"issueUrl": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"incident_id": 1234,
|
||||
"details": "Violation description test.",
|
||||
"policy_name": "Test policy name",
|
||||
"condition_name": "Server Down",
|
||||
"timestamp": 1605133931151,
|
||||
"current_state": "closed",
|
||||
"owner": "",
|
||||
"incident_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
{
|
||||
"id": "13bbcdca-f0b6-470d-b0be-b34583c58869",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK",
|
||||
"title": "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'",
|
||||
"priority": "HIGH",
|
||||
"totalIncidents": 1,
|
||||
"state": "CLOSED",
|
||||
"createdAt": 1713769949493,
|
||||
"updatedAt": 1713770135419,
|
||||
"alertPolicyNames": [
|
||||
"Golden Signals"
|
||||
],
|
||||
"alertConditionNames": [
|
||||
"Storage on Host Exceeded Threshold"
|
||||
],
|
||||
"owner": "Pieter Cardillo Kwok",
|
||||
"zulipCustomFields": {
|
||||
"Your custom payload": "somedata123"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
{
|
||||
"id": "208c5e92-f250-40be-b7c2-50508c268c15",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK",
|
||||
"title": "MAIN-APP-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'",
|
||||
"priority": "CRITICAL",
|
||||
"impactedEntities": [
|
||||
"MAIN-APP-UBUNTU"
|
||||
],
|
||||
"totalIncidents": 1,
|
||||
"state": "CREATED",
|
||||
"trigger": "INCIDENT_ADDED",
|
||||
"isCorrelated": false,
|
||||
"createdAt": 1713767789495,
|
||||
"updatedAt": 1713767789495,
|
||||
"sources": [
|
||||
"newrelic"
|
||||
],
|
||||
"alertPolicyNames": [
|
||||
"Golden Signals"
|
||||
],
|
||||
"alertConditionNames": [
|
||||
"High CPU"
|
||||
],
|
||||
"workflowName": "DBA Team workflow"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"id": "8114ada3-572e-4550-a310-12375371669e",
|
||||
"details": "Violation description test.",
|
||||
"alertPolicyNames": ["Test policy name"],
|
||||
"condition_name": "Server Down",
|
||||
"createdAt": 1605133931151,
|
||||
"state": "created",
|
||||
"owner": "",
|
||||
"issueUrl": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"id": "208c5e92-f250-40be-b7c2-50508c268c15",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK",
|
||||
"title": "PIETER-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'",
|
||||
"priority": "CRITICAL",
|
||||
"totalIncidents": 1,
|
||||
"state": "CREATED",
|
||||
"createdAt": 1713767789495,
|
||||
"updatedAt": 1713767789495,
|
||||
"alertPolicyNames": ["Golden Signals"],
|
||||
"alertConditionNames": ["High CPU"],
|
||||
"owner": "John Doe",
|
||||
"zulipCustomFields":{
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
{
|
||||
"id": "13bbcdca-f0b6-470d-b0be-b34583c58869",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK",
|
||||
"title": "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'",
|
||||
"priority": "HIGH",
|
||||
"impactedEntities": [
|
||||
"PIETER-UBUNTU"
|
||||
],
|
||||
"totalIncidents": 1,
|
||||
"state": "ACTIVATED",
|
||||
"trigger": "STATE_CHANGE",
|
||||
"isCorrelated": "false",
|
||||
"createdAt": 1713769949493,
|
||||
"updatedAt": 1713769949494,
|
||||
"sources": [
|
||||
"newrelic"
|
||||
],
|
||||
"alertPolicyNames": [
|
||||
"Golden Signals"
|
||||
],
|
||||
"alertConditionNames": [
|
||||
"Storage on Host Exceeded Threshold"
|
||||
],
|
||||
"workflowName": "issue workflow",
|
||||
"owner": "N/A",
|
||||
"zulipCustomFields": {
|
||||
"Your custom payload": "somedata123",
|
||||
"custom status 1": true,
|
||||
"Custom list 1": ["SSD", 2000, false, null, 13.33],
|
||||
"Custom field 1": null,
|
||||
"Custom field 2": 9000
|
||||
}
|
||||
}
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
{
|
||||
"id": "e04156e4-4cac-4f39-9d27-75d361e40a6d",
|
||||
"createdAt": 1605133931151,
|
||||
"state": "activated"
|
||||
}
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
{
|
||||
"timestamp": 1605133931151,
|
||||
"current_state": "open"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"id": "3881eb6c-6d57-447e-beaf-b2f610f014b0",
|
||||
"details": "Violation description test.",
|
||||
"alertPolicyNames": ["Test policy name"],
|
||||
"condition_name": "Server Down",
|
||||
"createdAt": "1969-12-31 23:59:55",
|
||||
"state": "acknowledged",
|
||||
"owner": "",
|
||||
"issueUrl": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"incident_id": 1234,
|
||||
"details": "Violation description test.",
|
||||
"policy_name": "Test policy name",
|
||||
"condition_name": "Server Down",
|
||||
"timestamp": "1969-12-31 23:59:55",
|
||||
"current_state": "open",
|
||||
"owner": "",
|
||||
"incident_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
{
|
||||
"id": "95a9344a-2590-48ce-8d83-07e28b6d22c6",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK",
|
||||
"title": "main_app-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'",
|
||||
"priority": "CRITICAL",
|
||||
"totalIncidents": 1,
|
||||
"state": "CLOSED",
|
||||
"createdAt": "1713766657asdasd383",
|
||||
"updatedAt": "1713766657383",
|
||||
"alertPolicyNames": ["Golden Signals"],
|
||||
"alertConditionNames": ["High CPU"]
|
||||
}
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"incident_id": 1234,
|
||||
"details": "Violation description test.",
|
||||
"policy_name": "Test policy name",
|
||||
"condition_name": "Server Down",
|
||||
"timestamp": 1605133931151,
|
||||
"owner": "Alice",
|
||||
"incident_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"id": "279751d5-8ad8-41d4-adb8-9e895c58b606",
|
||||
"details": "Violation description test.",
|
||||
"alertPolicyNames": ["Test policy name"],
|
||||
"condition_name": "Server Down",
|
||||
"createdAt": 1605133931151,
|
||||
"owner": "Alice",
|
||||
"issueUrl": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"id": "6eba9b54-898f-43eb-8892-e41ddca4a10f",
|
||||
"details": "Violation description test.",
|
||||
"alertPolicyNames": ["Test policy name"],
|
||||
"condition_name": "Server Down",
|
||||
"state": "acknowledged",
|
||||
"owner": "",
|
||||
"issueUrl": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"incident_id": 1234,
|
||||
"details": "Violation description test.",
|
||||
"policy_name": "Test policy name",
|
||||
"condition_name": "Server Down",
|
||||
"current_state": "open",
|
||||
"owner": "",
|
||||
"incident_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"incident_id": 1234,
|
||||
"details": "Violation description test.",
|
||||
"policy_name": "Test policy name",
|
||||
"condition_name": "Server Down",
|
||||
"timestamp": 1605133931151,
|
||||
"current_state": "open",
|
||||
"owner": "",
|
||||
"incident_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
{
|
||||
"id": "95a9344a-2590-48ce-8d83-07e28b6d22c6",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK",
|
||||
"title": "main_app-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'",
|
||||
"priority": "CRITICAL",
|
||||
"totalIncidents": 1,
|
||||
"state": "CLOSED",
|
||||
"createdAt": 1713766590228,
|
||||
"updatedAt": 1713766657383,
|
||||
"alertPolicyNames": ["Golden Signals"],
|
||||
"alertConditionNames": ["High CPU"],
|
||||
"zulipCustomFields": {
|
||||
"Your custom payload": "somedata123",
|
||||
"custom status 1": true,
|
||||
"Custom list 1": ["SSD", 2000, false, null, 13.33],
|
||||
"Custom field 1": null,
|
||||
"Custom field 2": 9000
|
||||
}
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"id": "1d1fff5c-c723-4bf6-8458-bc63b253279e",
|
||||
"details": "Violation description test.",
|
||||
"alertPolicyNames": ["Test policy name"],
|
||||
"condition_name": "Server Down",
|
||||
"createdAt": 1605133931151,
|
||||
"state": "hello world",
|
||||
"owner": "Alice",
|
||||
"issueUrl": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"incident_id": 1234,
|
||||
"details": "Violation description test.",
|
||||
"policy_name": "Test policy name",
|
||||
"condition_name": "Server Down",
|
||||
"timestamp": 1605133931151,
|
||||
"current_state": "hello world",
|
||||
"owner": "Alice",
|
||||
"incident_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
{
|
||||
"id": "95a9344a-2590-48ce-8d83-07e28b6d22c6",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK",
|
||||
"title": "main_app-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'",
|
||||
"priority": "CRITICAL",
|
||||
"impactedEntities": ["main_app-UBUNTU"],
|
||||
"totalIncidents": 1,
|
||||
"state": "CLOSED",
|
||||
"trigger": "INCIDENT_CLOSED",
|
||||
"isCorrelated": false,
|
||||
"createdAt": 1713766590228,
|
||||
"updatedAt": "123123123129312381923",
|
||||
"sources": ["newrelic"],
|
||||
"alertPolicyNames": ["Golden Signals"],
|
||||
"alertConditionNames": ["High CPU"],
|
||||
"workflowName": "DBA Team workflow"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"id": "37d07862-e156-480f-86f9-7e8e9b7ed4f4",
|
||||
"details": "Violation description test.",
|
||||
"alertPolicyNames": ["Test policy name"],
|
||||
"condition_name": "Server Down",
|
||||
"createdAt": 160513393115100000,
|
||||
"state": "open",
|
||||
"owner": "",
|
||||
"issueUrl": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"incident_acknowledge_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234/acknowledge",
|
||||
"incident_id": 1234,
|
||||
"details": "Violation description test.",
|
||||
"policy_name": "Test policy name",
|
||||
"condition_name": "Server Down",
|
||||
"timestamp": 160513393115100000,
|
||||
"current_state": "open",
|
||||
"owner": "",
|
||||
"incident_url": "https://alerts.newrelic.com/accounts/2941966/incidents/1234"
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"id": "13bbcdca-f0b6-470d-b0be-b34583c58869",
|
||||
"issueUrl": "https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK",
|
||||
"title": "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'",
|
||||
"priority": "HIGH",
|
||||
"totalIncidents": 1,
|
||||
"state": "ACTIVATED",
|
||||
"trigger": "STATE_CHANGE",
|
||||
"createdAt": 1713769949493,
|
||||
"updatedAt": 1713769949494,
|
||||
"alertPolicyNames": [
|
||||
"Golden Signals"
|
||||
],
|
||||
"alertConditionNames": [
|
||||
"Storage on Host Exceeded Threshold"
|
||||
],
|
||||
"owner": "N/A",
|
||||
"zulipCustomFields": {
|
||||
"Invalid fields 1": ["SSD", 2000, false, null, {}],
|
||||
"Invalid field 2": {},
|
||||
"Is valid": true
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
{
|
||||
"unrecognized1": "https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK",
|
||||
"unrecognized2": "PIETER-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'",
|
||||
"unrecognized3": "CRITICAL",
|
||||
"updatedField1": 1,
|
||||
"updatedField2": "CREATED",
|
||||
"updatedField3": 1713592289021,
|
||||
"changedField1": 1713592289021,
|
||||
"changedField2": ["Golden Signals"],
|
||||
"changedField3": ["High CPU"]
|
||||
}
|
||||
|
|
@ -6,253 +6,371 @@ class NewRelicHookTests(WebhookTestCase):
|
|||
URL_TEMPLATE = "/api/v1/external/newrelic?stream={stream}&api_key={api_key}"
|
||||
WEBHOOK_DIR_NAME = "newrelic"
|
||||
|
||||
# The following 9 unit tests are for the old format
|
||||
# corresponding json fixtures were renamed to have the "_old" trailing
|
||||
# These tests and fixtures are to be deleted when old notifications EOLed
|
||||
|
||||
def test_open_old(self) -> None:
|
||||
expected_topic_name = "Test policy name (1234)"
|
||||
def test_incident_activated_new_default_payload(self) -> None:
|
||||
expected_topic_name = "zulip_app query result is > 1.0 for 1 minutes on 'Zulip S..."
|
||||
expected_message = """
|
||||
[Incident](https://alerts.newrelic.com/accounts/2941966/incidents/1234) **opened** for condition: **Server Down** at <time:2020-11-11 22:32:11.151000+00:00>
|
||||
``` quote
|
||||
Violation description test.
|
||||
:red_circle: **[zulip_app query result is > 1.0 for 1 minutes on 'Zulip Server Low Storage'](https://radar-api.service.newrelic.com/accounts/4420147/issues/c5faa7e6-7b54-402d-af79-f99601e0278c?notifier=WEBHOOK)**
|
||||
|
||||
```quote
|
||||
**Priority**: CRITICAL
|
||||
**State**: ACTIVATED
|
||||
**Updated at**: <time: 2024-04-22 07:08:28.699000+00:00 >
|
||||
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `Zulip Server Low Storage`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 03:05:31.352000+00:00 >
|
||||
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_opened_old",
|
||||
"incident_activated_new_default_payload",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_closed_old(self) -> None:
|
||||
expected_topic_name = "Test policy name (1234)"
|
||||
def test_incident_activated_new_provided_base_payload(self) -> None:
|
||||
expected_topic_name = "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Sto..."
|
||||
expected_message = """
|
||||
[Incident](https://alerts.newrelic.com/accounts/2941966/incidents/1234) **closed** for condition: **Server Down**
|
||||
""".strip()
|
||||
:orange_circle: **[PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'](https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK)**
|
||||
|
||||
self.check_webhook(
|
||||
"incident_closed_old",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
```quote
|
||||
**Priority**: HIGH
|
||||
**State**: ACTIVATED
|
||||
**Updated at**: <time: 2024-04-22 07:12:29.494000+00:00 >
|
||||
|
||||
def test_acknowledged_old(self) -> None:
|
||||
expected_topic_name = "Test policy name (1234)"
|
||||
expected_message = """
|
||||
[Incident](https://alerts.newrelic.com/accounts/2941966/incidents/1234) **acknowledged** by **Alice** for condition: **Server Down**
|
||||
""".strip()
|
||||
```
|
||||
|
||||
self.check_webhook(
|
||||
"incident_acknowledged_old",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
```spoiler :file: Incident details
|
||||
|
||||
def test_not_recognized_old(self) -> None:
|
||||
with self.assertRaises(AssertionError) as e:
|
||||
self.check_webhook(
|
||||
"incident_state_not_recognized_old",
|
||||
"",
|
||||
"",
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertIn(
|
||||
"The newrelic webhook requires current_state be in [open|acknowledged|closed]",
|
||||
e.exception.args[0],
|
||||
)
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `Storage on Host Exceeded Threshold`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 07:12:29.493000+00:00 >
|
||||
- **Your custom payload**: somedata123
|
||||
|
||||
def test_missing_fields_old(self) -> None:
|
||||
expected_topic_name = "Unknown Policy (Unknown ID)"
|
||||
expected_message = """
|
||||
[Incident](https://alerts.newrelic.com) **opened** for condition: **Unknown condition** at <time:2020-11-11 22:32:11.151000+00:00>
|
||||
``` quote
|
||||
No details.
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_default_fields_old",
|
||||
"incident_activated_new_provided_base_payload",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_missing_current_state_old(self) -> None:
|
||||
with self.assertRaises(AssertionError) as e:
|
||||
def test_incident_closed_default_payload(self) -> None:
|
||||
expected_topic_name = "main_app-UBUNTU query result is > 2.0 for 1 minutes on 'H..."
|
||||
expected_message = """
|
||||
:red_circle: **[main_app-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'](https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK)**
|
||||
|
||||
```quote
|
||||
**Priority**: CRITICAL
|
||||
**State**: CLOSED
|
||||
**Updated at**: <time: 2024-04-22 06:17:37.383000+00:00 >
|
||||
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `High CPU`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 06:16:30.228000+00:00 >
|
||||
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_missing_current_state_old",
|
||||
"",
|
||||
"",
|
||||
"incident_closed_default_payload",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertIn(
|
||||
"The newrelic webhook requires current_state be in [open|acknowledged|closed]",
|
||||
e.exception.args[0],
|
||||
)
|
||||
|
||||
def test_missing_timestamp_old(self) -> None:
|
||||
with self.assertRaises(AssertionError) as e:
|
||||
def test_incident_closed_provided_base_payload(self) -> None:
|
||||
expected_topic_name = "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Sto..."
|
||||
expected_message = """
|
||||
:orange_circle: **[PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'](https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK)**
|
||||
|
||||
```quote
|
||||
**Priority**: HIGH
|
||||
**State**: CLOSED
|
||||
**Updated at**: <time: 2024-04-22 07:15:35.419000+00:00 >
|
||||
**Acknowledged by**: Pieter Cardillo Kwok
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `Storage on Host Exceeded Threshold`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 07:12:29.493000+00:00 >
|
||||
- **Your custom payload**: somedata123
|
||||
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_missing_timestamp_old",
|
||||
"",
|
||||
"",
|
||||
"incident_closed_provided_base_payload",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertIn(
|
||||
"The newrelic webhook requires timestamp in milliseconds", e.exception.args[0]
|
||||
|
||||
def test_incident_acknowledged_default_payload(self) -> None:
|
||||
expected_topic_name = "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Sto..."
|
||||
expected_message = """
|
||||
:orange_circle: **[PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'](https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK)**
|
||||
|
||||
```quote
|
||||
**Priority**: HIGH
|
||||
**State**: ACTIVATED
|
||||
**Updated at**: <time: 2024-04-22 07:14:37.412000+00:00 >
|
||||
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `Storage on Host Exceeded Threshold`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 07:12:29.493000+00:00 >
|
||||
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_acknowledged_default_payload",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_malformatted_time_old(self) -> None:
|
||||
def test_incident_acknowledged_provided_base_payload(self) -> None:
|
||||
expected_topic_name = "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Sto..."
|
||||
expected_message = """
|
||||
:orange_circle: **[PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'](https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK)**
|
||||
|
||||
```quote
|
||||
**Priority**: HIGH
|
||||
**State**: ACTIVATED
|
||||
**Updated at**: <time: 2024-04-22 07:14:37.412000+00:00 >
|
||||
**Acknowledged by**: Pieter Cardillo Kwok
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `Storage on Host Exceeded Threshold`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 07:12:29.493000+00:00 >
|
||||
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_acknowledged_provided_base_payload",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_incident_created_default_payload(self) -> None:
|
||||
expected_topic_name = "MAIN-APP-UBUNTU query result is > 2.0 for 1 minutes on 'H..."
|
||||
expected_message = """
|
||||
:red_circle: **[MAIN-APP-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'](https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK)**
|
||||
|
||||
```quote
|
||||
**Priority**: CRITICAL
|
||||
**State**: CREATED
|
||||
**Updated at**: <time: 2024-04-22 06:36:29.495000+00:00 >
|
||||
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `High CPU`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 06:36:29.495000+00:00 >
|
||||
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_created_default_payload",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_incident_created_provided_base_payload(self) -> None:
|
||||
expected_topic_name = "PIETER-UBUNTU query result is > 2.0 for 1 minutes on 'Hig..."
|
||||
expected_message = """
|
||||
:red_circle: **[PIETER-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'](https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK)**
|
||||
|
||||
```quote
|
||||
**Priority**: CRITICAL
|
||||
**State**: CREATED
|
||||
**Updated at**: <time: 2024-04-22 06:36:29.495000+00:00 >
|
||||
**Acknowledged by**: John Doe
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `High CPU`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 06:36:29.495000+00:00 >
|
||||
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_created_provided_base_payload",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_incident_default_base_with_zulip_custom_fields(self) -> None:
|
||||
expected_topic_name = "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Sto..."
|
||||
expected_message = """
|
||||
:orange_circle: **[PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'](https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK)**
|
||||
|
||||
```quote
|
||||
**Priority**: HIGH
|
||||
**State**: ACTIVATED
|
||||
**Updated at**: <time: 2024-04-22 07:12:29.494000+00:00 >
|
||||
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `Storage on Host Exceeded Threshold`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 07:12:29.493000+00:00 >
|
||||
- **Your custom payload**: somedata123
|
||||
- **Custom status 1**: True
|
||||
- **Custom list 1**: SSD, 2000, False, None, 13.33
|
||||
- **Custom field 1**: None
|
||||
- **Custom field 2**: 9000
|
||||
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_default_base_with_zulip_custom_fields",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_incident_provided_base_with_zulip_custom_fields(self) -> None:
|
||||
expected_topic_name = "main_app-UBUNTU query result is > 2.0 for 1 minutes on 'H..."
|
||||
expected_message = """
|
||||
:red_circle: **[main_app-UBUNTU query result is > 2.0 for 1 minutes on 'High CPU'](https://radar-api.service.newrelic.com/accounts/1/issues/0ea2df1c-adab-45d2-aae0-042b609d2322?notifier=SLACK)**
|
||||
|
||||
```quote
|
||||
**Priority**: CRITICAL
|
||||
**State**: CLOSED
|
||||
**Updated at**: <time: 2024-04-22 06:17:37.383000+00:00 >
|
||||
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `High CPU`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 06:16:30.228000+00:00 >
|
||||
- **Your custom payload**: somedata123
|
||||
- **Custom status 1**: True
|
||||
- **Custom list 1**: SSD, 2000, False, None, 13.33
|
||||
- **Custom field 1**: None
|
||||
- **Custom field 2**: 9000
|
||||
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_provided_base_with_zulip_custom_fields",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_incident_with_invalid_zulip_custom_fields(self) -> None:
|
||||
expected_topic_name = "PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Sto..."
|
||||
expected_message = """
|
||||
:orange_circle: **[PIETER-UBUNTU query result is > 1.0 for 5 minutes on 'Storage on Host Exceeded Threshold'](https://radar-api.service.newrelic.com/accounts/4420147/issues/13bbcdca-f0b6-470d-b0be-b34583c58869?notifier=WEBHOOK)**
|
||||
|
||||
```quote
|
||||
**Priority**: HIGH
|
||||
**State**: ACTIVATED
|
||||
**Updated at**: <time: 2024-04-22 07:12:29.494000+00:00 >
|
||||
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
|
||||
- **Alert policies**: `Golden Signals`
|
||||
- **Conditions**: `Storage on Host Exceeded Threshold`
|
||||
- **Total incidents**: 1
|
||||
- **Incident created at**: <time: 2024-04-22 07:12:29.493000+00:00 >
|
||||
- **Invalid fields 1**: *Value is not a supported data type*
|
||||
- **Invalid field 2**: *Value is not a supported data type*
|
||||
- **Is valid**: True
|
||||
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_with_invalid_zulip_custom_fields",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_missing_essential_fields_default_payload(self) -> None:
|
||||
expected_topic_name = "New Relic incident alerts"
|
||||
expected_message = """
|
||||
:danger: A New Relic [incident](https://one.newrelic.com/alerts-ai) updated
|
||||
|
||||
**Warning**: Unable to use the default notification format because at least one expected field was missing from the incident payload. See [New Relic integration documentation](/integrations/doc/newrelic).
|
||||
|
||||
**Missing fields**: `issueUrl`, `title`, `priority`, `totalIncidents`, `state`, `createdAt`, `updatedAt`, `alertPolicyNames`, `alertConditionNames`
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"missing_essential_fields_default_payload",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_malformatted_time(self) -> None:
|
||||
with self.assertRaises(AssertionError) as e:
|
||||
self.check_webhook(
|
||||
"incident_malformatted_time_old",
|
||||
"incident_malformed_timestamp",
|
||||
"",
|
||||
"",
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertIn("The newrelic webhook expects time in milliseconds.", e.exception.args[0])
|
||||
|
||||
def test_time_too_large_old(self) -> None:
|
||||
def test_time_too_large(self) -> None:
|
||||
with self.assertRaises(AssertionError) as e:
|
||||
self.check_webhook(
|
||||
"incident_time_too_large_old",
|
||||
"",
|
||||
"",
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertIn("The newrelic webhook expects time in milliseconds.", e.exception.args[0])
|
||||
|
||||
# The following 10 unit tests are for the new format
|
||||
# One more test than the old format as we have 4 states instead of 3 in the old
|
||||
# corresponding json fixtures have "_new" trailing in the name
|
||||
|
||||
def test_activated_new(self) -> None:
|
||||
expected_topic_name = "Test policy name (8ceed342-f305-4bfa-adb8-97ba93f5dd26)"
|
||||
expected_message = """
|
||||
[Incident](https://alerts.newrelic.com/accounts/2941966/incidents/1234) **active** for condition: **Server Down** at <time:2020-11-11 22:32:11.151000+00:00>
|
||||
``` quote
|
||||
Violation description test.
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_active_new",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_created_new(self) -> None:
|
||||
expected_topic_name = "Test policy name (8114ada3-572e-4550-a310-12375371669e)"
|
||||
expected_message = """
|
||||
[Incident](https://alerts.newrelic.com/accounts/2941966/incidents/1234) **created** for condition: **Server Down**
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_created_new",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_closed_new(self) -> None:
|
||||
expected_topic_name = "Test policy name (f0d98b28-bf9d-49e7-b9d0-ac7cbb52e73a)"
|
||||
expected_message = """
|
||||
[Incident](https://alerts.newrelic.com/accounts/2941966/incidents/1234) **closed** for condition: **Server Down**
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_closed_new",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_acknowledged_new(self) -> None:
|
||||
expected_topic_name = "Test policy name (3576f543-dc3c-4d97-9f16-5c81f35195cb)"
|
||||
expected_message = """
|
||||
[Incident](https://alerts.newrelic.com/accounts/2941966/incidents/1234) **acknowledged** by **Alice** for condition: **Server Down**
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_acknowledged_new",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_not_recognized_new(self) -> None:
|
||||
with self.assertRaises(AssertionError) as e:
|
||||
self.check_webhook(
|
||||
"incident_state_not_recognized_new",
|
||||
"",
|
||||
"",
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertIn(
|
||||
"The newrelic webhook requires state be in [created|activated|acknowledged|closed]",
|
||||
e.exception.args[0],
|
||||
)
|
||||
|
||||
def test_missing_fields_new(self) -> None:
|
||||
expected_topic_name = "Unknown Policy (e04156e4-4cac-4f39-9d27-75d361e40a6d)"
|
||||
expected_message = """
|
||||
[Incident](https://alerts.newrelic.com) **active** for condition: **Unknown condition** at <time:2020-11-11 22:32:11.151000+00:00>
|
||||
``` quote
|
||||
No details.
|
||||
```
|
||||
""".strip()
|
||||
|
||||
self.check_webhook(
|
||||
"incident_default_fields_new",
|
||||
expected_topic_name,
|
||||
expected_message,
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
def test_missing_state_new(self) -> None:
|
||||
with self.assertRaises(AssertionError) as e:
|
||||
self.check_webhook(
|
||||
"incident_missing_state_new",
|
||||
"",
|
||||
"",
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertIn(
|
||||
"The newrelic webhook requires state be in [created|activated|acknowledged|closed]",
|
||||
e.exception.args[0],
|
||||
)
|
||||
|
||||
def test_missing_timestamp_new(self) -> None:
|
||||
with self.assertRaises(AssertionError) as e:
|
||||
self.check_webhook(
|
||||
"incident_missing_timestamp_new",
|
||||
"",
|
||||
"",
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertIn(
|
||||
"The newrelic webhook requires timestamp in milliseconds", e.exception.args[0]
|
||||
)
|
||||
|
||||
def test_malformatted_time_new(self) -> None:
|
||||
with self.assertRaises(AssertionError) as e:
|
||||
self.check_webhook(
|
||||
"incident_malformatted_time_new",
|
||||
"",
|
||||
"",
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertIn("The newrelic webhook expects time in milliseconds.", e.exception.args[0])
|
||||
|
||||
def test_time_too_large_new(self) -> None:
|
||||
with self.assertRaises(AssertionError) as e:
|
||||
self.check_webhook(
|
||||
"incident_time_too_large_new",
|
||||
"incident_time_too_large",
|
||||
"",
|
||||
"",
|
||||
content_type="application/json",
|
||||
|
|
|
|||
|
|
@ -1,53 +1,158 @@
|
|||
# Webhooks for external integrations.
|
||||
|
||||
from django.core.exceptions import ValidationError
|
||||
from django.http import HttpRequest, HttpResponse
|
||||
from django.utils.translation import gettext as _
|
||||
|
||||
from zerver.decorator import webhook_view
|
||||
from zerver.lib.exceptions import JsonableError
|
||||
from zerver.lib.response import json_success
|
||||
from zerver.lib.typed_endpoint import JsonBodyPayload, typed_endpoint
|
||||
from zerver.lib.validator import (
|
||||
WildValue,
|
||||
check_float,
|
||||
check_int,
|
||||
check_list,
|
||||
check_none_or,
|
||||
check_string,
|
||||
check_string_fixed_length,
|
||||
check_string_in,
|
||||
check_union,
|
||||
)
|
||||
from zerver.lib.webhooks.common import check_send_webhook_message, unix_milliseconds_to_timestamp
|
||||
from zerver.models import UserProfile
|
||||
|
||||
# Newrelic planned to upgrade Alert Notification Channels to Workflows and Destinations
|
||||
# https://discuss.newrelic.com/t/plan-to-upgrade-alert-notification-channels-to-workflows-and-destinations/188205
|
||||
# This view will handle both old and new format but will keep it easy to delete the old code
|
||||
# once it is EOLed by the end of June, 2023
|
||||
MISSING_FIELDS_NOTIFICATION = """
|
||||
:danger: A New Relic [incident]({url}) updated
|
||||
|
||||
# Once old is EOLed, delete the OPEN_TEMPLATE
|
||||
OPEN_TEMPLATE = """
|
||||
[Incident]({incident_url}) **opened** for condition: **{condition_name}** at <time:{iso_timestamp}>
|
||||
``` quote
|
||||
**Warning**: Unable to use the default notification format because at least one expected field was missing from the incident payload. See [New Relic integration documentation](/integrations/doc/newrelic).
|
||||
|
||||
**Missing fields**: {formatted_missing_fields}
|
||||
"""
|
||||
|
||||
NOTIFICATION_TEMPLATE = """
|
||||
{priority_symbol} **[{title}]({incident_url})**
|
||||
|
||||
```quote
|
||||
**Priority**: {priority}
|
||||
**State**: {state}
|
||||
**Updated at**: {time_updated}
|
||||
{owner}
|
||||
```
|
||||
|
||||
```spoiler :file: Incident details
|
||||
{details}
|
||||
```
|
||||
""".strip()
|
||||
"""
|
||||
|
||||
ACTIVE_TEMPLATE = """
|
||||
[Incident]({incident_url}) **active** for condition: **{condition_name}** at <time:{iso_timestamp}>
|
||||
``` quote
|
||||
{details}
|
||||
```
|
||||
""".strip()
|
||||
NOTIFICATION_DETAILS = """
|
||||
- **Alert policies**: {alert_policy}
|
||||
- **Conditions**: {conditions}
|
||||
- **Total incidents**: {total_incidents}
|
||||
- **Incident created at**: {time_created}
|
||||
"""
|
||||
|
||||
DEFAULT_TEMPLATE = (
|
||||
"""[Incident]({incident_url}) **{status}** {owner}for condition: **{condition_name}**""".strip()
|
||||
)
|
||||
ALL_EVENT_TYPES = ["CREATED", "ACTIVATED", "CLOSED"]
|
||||
|
||||
TOPIC_TEMPLATE = """{policy_name} ({incident_id})""".strip()
|
||||
PRIORITIES = {
|
||||
"CRITICAL": ":red_circle:",
|
||||
"HIGH": ":orange_circle:",
|
||||
"MEDIUM": ":yellow:",
|
||||
"LOW": ":blue_circle:",
|
||||
}
|
||||
|
||||
# Once old is EOLed, delete old and keep new
|
||||
OLD_EVENT_TYPES = ["closed", "acknowledged", "open"]
|
||||
NEW_EVENT_TYPES = ["created", "activated", "acknowledged", "closed"]
|
||||
ALL_EVENT_TYPES = list(set(OLD_EVENT_TYPES).union(set(NEW_EVENT_TYPES)))
|
||||
DEFAULT_NEWRELIC_URL = "https://one.newrelic.com/alerts-ai"
|
||||
|
||||
|
||||
EXPECTED_FIELDS = [
|
||||
"issueUrl",
|
||||
"title",
|
||||
"priority",
|
||||
"totalIncidents",
|
||||
"state",
|
||||
"createdAt",
|
||||
"updatedAt",
|
||||
"alertPolicyNames",
|
||||
"alertConditionNames",
|
||||
]
|
||||
|
||||
|
||||
def get_timestamp_string(payload: WildValue, event_type: str) -> str:
|
||||
# This function is intended to be used only for the "updatedAt"
|
||||
# and "createdAt" fields. Theoretically, neither field can be
|
||||
# None at any time.
|
||||
unix_time = payload[event_type].tame(check_union([check_int, check_string]))
|
||||
timestamp = str(unix_milliseconds_to_timestamp(unix_time, "newrelic"))
|
||||
return f"<time: {timestamp} >"
|
||||
|
||||
|
||||
def parse_payload(payload: WildValue) -> dict[str, str]:
|
||||
priority = payload["priority"].tame(check_string_in(PRIORITIES.keys()))
|
||||
priority_symbol = PRIORITIES.get(priority, ":alert:")
|
||||
conditions_list = payload.get("alertConditionNames", ["Unknown condition"]).tame(
|
||||
check_list(check_string)
|
||||
)
|
||||
conditions = ", ".join([f"`{c}`" for c in conditions_list])
|
||||
policy_list = payload.get("alertPolicyNames", ["Unknown policy"]).tame(check_list(check_string))
|
||||
alert_policy = ", ".join([f"`{p}`" for p in policy_list])
|
||||
|
||||
owner = payload.get("owner").tame(check_none_or(check_string))
|
||||
acknowledged = ""
|
||||
if owner and owner != "N/A":
|
||||
acknowledged = f"**Acknowledged by**: {owner}"
|
||||
|
||||
message_context: dict[str, str] = {
|
||||
"title": payload["title"].tame(check_string),
|
||||
"incident_url": payload.get("issueUrl", DEFAULT_NEWRELIC_URL).tame(check_string),
|
||||
"total_incidents": str(payload["totalIncidents"].tame(check_int)),
|
||||
"state": payload["state"].tame(check_string_in(ALL_EVENT_TYPES)),
|
||||
"time_created": get_timestamp_string(payload, "createdAt"),
|
||||
"time_updated": get_timestamp_string(payload, "updatedAt"),
|
||||
"priority": priority,
|
||||
"priority_symbol": priority_symbol,
|
||||
"conditions": conditions,
|
||||
"alert_policy": alert_policy,
|
||||
"owner": acknowledged,
|
||||
}
|
||||
|
||||
return message_context
|
||||
|
||||
|
||||
def format_zulip_custom_fields(payload: WildValue) -> str:
|
||||
body_custom_field_detail: str = ""
|
||||
zulip_custom_fields = payload.get("zulipCustomFields", {})
|
||||
|
||||
for key, value in zulip_custom_fields.items():
|
||||
custom_field_name = key.capitalize()
|
||||
try:
|
||||
details = value.tame(
|
||||
check_none_or(
|
||||
check_union(
|
||||
[
|
||||
check_int,
|
||||
check_float,
|
||||
check_string,
|
||||
check_list(
|
||||
check_none_or(check_union([check_int, check_float, check_string]))
|
||||
),
|
||||
]
|
||||
)
|
||||
)
|
||||
)
|
||||
if isinstance(details, list):
|
||||
custom_field_detail = ", ".join([f"{detail}" for detail in details])
|
||||
else:
|
||||
custom_field_detail = f"{details}"
|
||||
|
||||
custom_field_message = f"- **{custom_field_name}**: {custom_field_detail}\n"
|
||||
body_custom_field_detail += custom_field_message
|
||||
except ValidationError:
|
||||
invalid_field_message = (
|
||||
f"- **{custom_field_name}**: *Value is not a supported data type*\n"
|
||||
)
|
||||
body_custom_field_detail += invalid_field_message
|
||||
return body_custom_field_detail
|
||||
|
||||
|
||||
def check_for_expected_fields(payload: WildValue) -> list[str]:
|
||||
return [key for key in EXPECTED_FIELDS if key not in payload]
|
||||
|
||||
|
||||
@webhook_view("NewRelic", all_event_types=ALL_EVENT_TYPES)
|
||||
|
|
@ -58,112 +163,21 @@ def api_newrelic_webhook(
|
|||
*,
|
||||
payload: JsonBodyPayload[WildValue],
|
||||
) -> HttpResponse:
|
||||
# Handle old format
|
||||
# Once old is EOLed, delete if block and keep else block
|
||||
if not payload.get("id").tame(check_none_or(check_string_fixed_length(36))):
|
||||
info = {
|
||||
"condition_name": payload.get("condition_name", "Unknown condition").tame(check_string),
|
||||
"details": payload.get("details", "No details.").tame(check_string),
|
||||
"incident_url": payload.get("incident_url", "https://alerts.newrelic.com").tame(
|
||||
check_string
|
||||
),
|
||||
"incident_acknowledge_url": payload.get(
|
||||
"incident_acknowledge_url", "https://alerts.newrelic.com"
|
||||
).tame(check_string),
|
||||
"status": payload.get("current_state", "None").tame(check_string),
|
||||
"iso_timestamp": "",
|
||||
"owner": payload.get("owner", "").tame(check_string),
|
||||
}
|
||||
|
||||
unix_time = payload.get("timestamp").tame(
|
||||
check_none_or(check_union([check_string, check_int]))
|
||||
missing_fields = check_for_expected_fields(payload)
|
||||
if missing_fields:
|
||||
formatted_missing_fields = ", ".join([f"`{fields}`" for fields in missing_fields])
|
||||
content = MISSING_FIELDS_NOTIFICATION.format(
|
||||
url=DEFAULT_NEWRELIC_URL,
|
||||
formatted_missing_fields=formatted_missing_fields,
|
||||
)
|
||||
if unix_time is None:
|
||||
raise JsonableError(_("The newrelic webhook requires timestamp in milliseconds"))
|
||||
|
||||
info["iso_timestamp"] = str(unix_milliseconds_to_timestamp(unix_time, "newrelic"))
|
||||
|
||||
# Add formatting to the owner field if owner is present
|
||||
if info["owner"] != "":
|
||||
info["owner"] = "by **{}** ".format(info["owner"])
|
||||
|
||||
# These are the three promised current_state values
|
||||
if info["status"].lower() == "open":
|
||||
content = OPEN_TEMPLATE.format(**info)
|
||||
elif info["status"].lower() == "acknowledged":
|
||||
content = DEFAULT_TEMPLATE.format(**info)
|
||||
elif info["status"].lower() == "closed":
|
||||
content = DEFAULT_TEMPLATE.format(**info)
|
||||
else:
|
||||
raise JsonableError(
|
||||
_("The newrelic webhook requires current_state be in [open|acknowledged|closed]")
|
||||
)
|
||||
|
||||
topic_info = {
|
||||
"policy_name": payload.get("policy_name", "Unknown Policy").tame(check_string),
|
||||
"incident_id": payload.get("incident_id", "Unknown ID").tame(
|
||||
check_union([check_string, check_int])
|
||||
),
|
||||
}
|
||||
topic_name = TOPIC_TEMPLATE.format(**topic_info)
|
||||
|
||||
check_send_webhook_message(request, user_profile, topic_name, content, info["status"])
|
||||
topic = "New Relic incident alerts"
|
||||
check_send_webhook_message(request, user_profile, topic, content)
|
||||
return json_success(request)
|
||||
|
||||
# Handle new format
|
||||
else:
|
||||
info = {
|
||||
"condition_name": payload.get("condition_name", "Unknown condition").tame(check_string),
|
||||
"details": payload.get("details", "No details.").tame(check_string),
|
||||
"incident_url": payload.get("issueUrl", "https://alerts.newrelic.com").tame(
|
||||
check_string
|
||||
),
|
||||
"incident_acknowledge_url": payload.get(
|
||||
"incident_acknowledge_url", "https://alerts.newrelic.com"
|
||||
).tame(check_string),
|
||||
"status": payload.get("state", "None").tame(check_string),
|
||||
"iso_timestamp": "",
|
||||
"owner": payload.get("owner", "").tame(check_string),
|
||||
}
|
||||
|
||||
unix_time = payload.get("createdAt").tame(
|
||||
check_none_or(check_union([check_string, check_int]))
|
||||
)
|
||||
if unix_time is None:
|
||||
raise JsonableError(_("The newrelic webhook requires timestamp in milliseconds"))
|
||||
|
||||
info["iso_timestamp"] = str(unix_milliseconds_to_timestamp(unix_time, "newrelic"))
|
||||
|
||||
# Add formatting to the owner field if owner is present
|
||||
if info["owner"] != "":
|
||||
info["owner"] = "by **{}** ".format(info["owner"])
|
||||
|
||||
# These are the three promised state values
|
||||
if info["status"].lower() == "activated":
|
||||
content = ACTIVE_TEMPLATE.format(**info)
|
||||
elif info["status"].lower() == "acknowledged":
|
||||
content = DEFAULT_TEMPLATE.format(**info)
|
||||
elif info["status"].lower() == "closed":
|
||||
content = DEFAULT_TEMPLATE.format(**info)
|
||||
elif info["status"].lower() == "created":
|
||||
content = DEFAULT_TEMPLATE.format(**info)
|
||||
else:
|
||||
raise JsonableError(
|
||||
_(
|
||||
"The newrelic webhook requires state be in [created|activated|acknowledged|closed]"
|
||||
)
|
||||
)
|
||||
|
||||
policy_names_list = payload.get("alertPolicyNames", []).tame(check_list(check_string))
|
||||
if policy_names_list:
|
||||
policy_names_str = ",".join(policy_names_list)
|
||||
else:
|
||||
policy_names_str = "Unknown Policy"
|
||||
topic_info = {
|
||||
"policy_name": policy_names_str,
|
||||
"incident_id": payload.get("id", "Unknown ID").tame(check_string),
|
||||
}
|
||||
topic_name = TOPIC_TEMPLATE.format(**topic_info)
|
||||
|
||||
check_send_webhook_message(request, user_profile, topic_name, content, info["status"])
|
||||
message_context = parse_payload(payload)
|
||||
incident_details = NOTIFICATION_DETAILS.format(**message_context)
|
||||
incident_details += format_zulip_custom_fields(payload)
|
||||
content = NOTIFICATION_TEMPLATE.format(details=incident_details, **message_context)
|
||||
topic = message_context["title"]
|
||||
check_send_webhook_message(request, user_profile, topic, content, message_context["state"])
|
||||
return json_success(request)
|
||||
|
|
|
|||
Loading…
Reference in New Issue