Skip to content
  • There are no suggestions because the search field is empty.

Hetrix Down Notifications

Internet Down Alerts — Tier 1 Support Standard Operating Procedure

  • Document ID: SOP-T1-02
  • Version: 2.0
  • Owner: Tier 1 Support Associates
  • Approver: John Martinez — T1 Manager
  • Applies To: Tier 1 Support Associates
  • Related SOPs: SOP-T1-01 — Customer-Reported Connectivity Issues
    Carrier Escalation SOP

1. Purpose

This procedure guides Tier 1 Support Associates in creating and managing tickets when Hetrix monitoring reports that a circuit or IP is down.

2. Scope

This procedure applies to all Hetrix downtime alerts received in HubSpot (under “T1 All Ticket Views” or “All Ticket Views”), via Outlook email.

3. Roles & Responsibilities

  • Tier 1 Support Associate: Monitors alert channels, verifies downtime duration in Hetrix, creates and manages the HubSpot ticket, and coordinates with the carrier through resolution.
  • T1 Manager: Approves this SOP and is the escalation point for carrier non-response.
  • CC Distribution: Jarred Morgan and Ryan Dowd on every notification email; Mick Campbell additionally for HGV customers only.

4. Prerequisites

Non-Technical Prerequisites

  • Active HubSpot account with ticket creation and template access.
  • Access to the Hetrix dashboard (https://hetrixtools.com/dashboard/).
  • Awareness of the CC distribution rule (Jarred Morgan and Ryan Dowd always; Mick Campbell for HGV customers only).

Technical Prerequisites

  • Ability to search Hetrix uptime monitors by IP address.
  • Ability to search and associate HubSpot deals to a ticket.
  • Access to the Carrier Escalation SOP for non-responsive carriers.

5. Procedure

STEP 1 — Confirm Downtime Duration

Once a notification is received in HubSpot, check the IP target in Hetrix to determine how long it has been down. Access the Hetrix dashboard and locate the IP under Uptime Monitors, using the search field to find it directly.

Decision threshold

IP down Hubspot Ticket is 10 minutes or longer → create an Internet Down Notification ticket in HubSpot.
IP down Hubspot Ticket recovers in under 10 minutes → close the ticket in HubSpot without further action.

STEP 2 — Create the Ticket

In HubSpot, click the plus (+) icon at the top left and select Ticket.

Use the following format for the ticket name and description:

Ticket naming format

Internet Down Notification — Customer Name — Carrier Name — Circuit Address

If a customer has multiple affected circuits or IPs, create a separate ticket for each one — do not combine them.

Fill out the remaining ticket fields:

  • Is this an Internal Ticket: No
  • Source: Tier 1 Support
  • Create Date: today's date

Click Create. On the right panel, add the associated customer or company.

You will be prompted to send a notification email. Copy the required recipients, then insert the Hetrix Down Notification template (Insert → Templates → search “Hetrix down notification”).

Required CC on the notification email

Always CC: Jarred Morgan (jarred.morgan@hypernetworks.com), Ryan Dowd (ryan.dowd@hypernetworks.com)
CC Mick Campbell (mick.campbell@hypernetworks.com) only when the customer is an HGV account.

Hetrix down notification template

Hi (Insert Company/Account Name) Team,
We hope you are doing well.
Our monitoring system has detected that your internet circuit listed below is currently experiencing a service interruption.
Circuit Details: Carrier / Circuit ID / Network IP / Gateway IP / Location Address / Site Name
To help us determine the cause of the outage, could you please confirm: any scheduled maintenance at the site, whether users are experiencing connectivity issues, whether commercial power is available, and whether the router/modem is powered on with all cables securely connected.
If power is available, please reboot the (Insert Carrier) router/modem by disconnecting power for 60 seconds, then reconnecting. Let us know the outcome once it finishes rebooting.
If the issue persists, please provide: Local Contact Name, Local Contact Number, Site Operating Hours.
Best Regards,

After sending, fill in on the right panel: Date and Time Ticket Opened with Carrier (must match the exact date/time the initial email was sent), Create Date, and Reason for Opening.

STEP 3 — Log the Deal Information

In the Notes section, record all information associated with the deal being worked, then click Create Note.

  • Deal: —
  • Deal Link: —
  • Address: —
  • Site Name: —
  • Circuit ID: —
  • MAC Address: —
  • Network IP: —
  • Gateway: —
  • Subnet: —
  • Carrier Account #: —
  • Carrier & No.: —
  • Security PIN: —
  • Note
  • Every carrier ticket number, dispatch phone number, and monitoring update must also be logged in the Notes section as it happens — not only at ticket creation.

STEP 4 — Associate the Deal

Search for the deal using the deal number at the top of the page, or the deal link saved in your note. Copy the deal name.

Return to the ticket, click the ticket name, then click Add in the associations panel. Paste the deal name, select it, click Next, then Save.

STEP 5 — Contact the Carrier

Contact the carrier by phone or email to inquire about the reason for disruption and relay every carrier update to the customer by email.

Record the carrier ticket number in the Notes section. Carrier ticket numbers are for internal T1 reference only and must not be shared with the customer.

Priority handling

Deals with a data center location must be escalated immediately upon confirmation.

For Lumen circuits, the carrier ticket can be escalated by phone or through the Lumen portal (credentials are maintained separately — see Carrier Portal Access SOP).

STEP 6 — Confirm and Monitor

If the customer confirms the circuit is up or online, proceed with 24 hours of monitoring to confirm service stability before closing.

If the customer is still experiencing issues, call the carrier for further investigation and possible dispatch.

If the customer does not respond, or the local contact is unreachable, monitor the circuit and work the 72-, 48-, and 24-hour no-response templates (available in the HubSpot template library) before closing.

If there is still no response after 72 hours, send the ticket closure email and close the ticket.

6. Escalation Criteria

  • Data center location → escalate to the carrier immediately, ahead of standard queue order.
  • Carrier fails to respond or provide updates → escalate using the Carrier Escalation SOP.
  • Customer reports the issue persists after carrier confirmation → return to the carrier for further investigation and possible dispatch.

7. Do's and Don'ts

Do

  • Check how long the IP has been down in Hetrix before creating anything. The automatically generated ticket by Hetrix in Hubspot must be 10mins before creating a ticket.
  • Create a separate ticket for each circuit when a customer has more than one affected.
  • Set the date and time the ticket was opened with the carrier to match exactly when the initial email was sent to the customer.
  • Copy Jarred Morgan and Ryan Dowd on the notification email, and Mick Campbell only for HGV customers.
  • Escalate deals with a data center location right away.

Don't

  • Don't create a ticket if the IP came back up in less than ten minutes in HubSpot. Close it instead.
  • Don't close the ticket as soon as the customer confirms the circuit is up. Run the 24 hours of monitoring first.
  • Don't close a non-responsive ticket before 72 hours. Work the 72-, 48-, and 24-hour follow-ups first.

8. Revision History

  • v1.1: Original knowledge-base article format. Do's/Don'ts section stated a 15-minute threshold, which conflicted with the 10-minute threshold in the procedure steps.
  • v2.0: Restructured into standard SOP format. Corrected the threshold inconsistency: the Do's/Don'ts section now states 10 minutes, consistent with Step 1 of the procedure. No other procedural content changed.
  • v2.1: Clarified the 10mins threshold is the age of the automatically generated ticket in Hubspot by Hetrix.