SOP-T1-02: Monitoring Down Notifications

Document ID: SOP-T1-02

Version: 2.2

Owner: Tier 1 Support Associates

Approver: John Martinez — T1 Manager

Applies To: Tier 1 Support Associates

Related SOPs: SOP-T1-01 — Customer-Reported Connectivity Issues Carrier Escalation SOP

1. Purpose

This procedure guides Tier 1 Support Associates in creating and managing tickets when the monitoring tool reports that a circuit or IP is down.

2. Scope

This procedure applies to all monitoring downtime alerts received in the ticketing system (under “T1 All Ticket Views” or “All Ticket Views”), via email.

3. Roles & Responsibilities

Tier 1 Support Associate: Monitors alert channels, verifies downtime duration in the monitoring tool, creates and manages the ticket, and coordinates with the carrier through resolution.

T1 Manager: Approves this SOP and is the escalation point for carrier non-response.

CC Distribution: The standard internal CC list on every notification email; the Key Account Manager additionally for designated key accounts only. Names and addresses are kept in the T1 Internal Contact Sheet.

4. Prerequisites

Non-Technical Prerequisites

• Active ticketing system account with ticket creation and template access.
• Access to the monitoring dashboard (link stored with the credentials in the team password manager).
• Awareness of the CC distribution rule (standard internal CC list always; Key Account Manager for designated key accounts only).

Technical Prerequisites

• Ability to search uptime monitors in the monitoring tool by IP address.
• Ability to search and associate deal records to a ticket.
• Access to the Carrier Escalation SOP for non-responsive carriers.

5. Procedure

STEP 1 — Confirm Downtime Duration

Once a notification is received in the ticketing system, check the IP target in the monitoring tool to determine how long it has been down. Access the monitoring dashboard and locate the IP under Uptime Monitors, using the search field to find it directly.

Decision threshold

Auto-generated down ticket is 10 minutes old or older → create an Internet Down Notification ticket.

IP recovers before the auto-generated ticket reaches 10 minutes → close the auto-generated ticket without further action.

STEP 2 — Create the Ticket

In the ticketing system, click the plus (+) icon at the top left and select Ticket.

Use the following format for the ticket name and description:

Ticket naming format

Internet Down Notification — Customer Name — Carrier Name — Circuit Address

If a customer has multiple affected circuits or IPs, create a separate ticket for each one — do not combine them.

Fill out the remaining ticket fields:

• Is this an Internal Ticket: No
• Source: Tier 1 Support
• Create Date: today's date

Click Create. On the right panel, add the associated customer or company.

You will be prompted to send a notification email. Copy the required recipients, then insert the Monitoring Down Notification template (Insert → Templates → search “down notification”).

Required CC on the notification email

Always CC: the standard internal CC list (see T1 Internal Contact Sheet)

CC the Key Account Manager only when the customer is a designated key account.

Monitoring down notification template

Hi (Insert Company/Account Name) Team,

We hope you are doing well.

Our monitoring system has detected that your internet circuit listed below is currently experiencing a service interruption.

Circuit Details: Carrier / Circuit ID / Network IP / Gateway IP / Location Address / Site Name

To help us determine the cause of the outage, could you please confirm: any scheduled maintenance at the site, whether users are experiencing connectivity issues, whether commercial power is available, and whether the router/modem is powered on with all cables securely connected.

If power is available, please reboot the (Insert Carrier) router/modem by disconnecting power for 60 seconds, then reconnecting. Let us know the outcome once it finishes rebooting.

If the issue persists, please provide: Local Contact Name, Local Contact Number, Site Operating Hours.

Best Regards,

After sending, fill in on the right panel: Date and Time Ticket Opened with Carrier (must match the exact date/time the initial email was sent), Create Date, and Reason for Opening.

STEP 3 — Log the Deal Information

In the Notes section, record all information associated with the deal being worked, then click Create Note.

Deal: —

Deal Link: —

Address: —

Site Name: —

Circuit ID: —

MAC Address: —

Network IP: —

Gateway: —

Subnet: —

Carrier Account #: —

Carrier & No.: —

Security PIN: —

Note

Every carrier ticket number, dispatch phone number, and monitoring update must also be logged in the Notes section as it happens — not only at ticket creation.

STEP 4 — Associate the Deal

Search for the deal using the deal number at the top of the page, or the deal link saved in your note. Copy the deal name.

Return to the ticket, click the ticket name, then click Add in the associations panel. Paste the deal name, select it, click Next, then Save.

STEP 5 — Contact the Carrier

Contact the carrier by phone or email to inquire about the reason for disruption and relay every carrier update to the customer by email.

Record the carrier ticket number in the Notes section. Carrier ticket numbers are for internal T1 reference only and must not be shared with the customer.

Priority handling

Deals with a data center location must be escalated immediately upon confirmation.

For Lumen circuits, the carrier ticket can be escalated by phone or through the Lumen portal (credentials are maintained separately — see Carrier Portal Access SOP).

STEP 6 — Confirm and Monitor

If the customer confirms the circuit is up or online, proceed with 24 hours of monitoring to confirm service stability before closing.

If the customer is still experiencing issues, call the carrier for further investigation and possible dispatch.

If the customer does not respond, or the local contact is unreachable, monitor the circuit and work the 72-, 48-, and 24-hour no-response templates (available in the ticketing system's template library) before closing.

If there is still no response after 72 hours, send the ticket closure email and close the ticket.

6. Escalation Criteria

• Data center location → escalate to the carrier immediately, ahead of standard queue order.
• Carrier fails to respond or provide updates → escalate using the Carrier Escalation SOP.
• Customer reports the issue persists after carrier confirmation → return to the carrier for further investigation and possible dispatch.

7. Do's and Don'ts

Do

• Check how long the IP has been down in the monitoring tool before creating anything. The ticket automatically generated by the monitoring tool must be at least 10 minutes old before you create a working ticket.
• Create a separate ticket for each circuit when a customer has more than one affected.
• Set the date and time the ticket was opened with the carrier to match exactly when the initial email was sent to the customer.
• Copy the standard internal CC list on the notification email, and the Key Account Manager only for designated key accounts.
• Escalate deals with a data center location right away.

Don't

• Don't create a ticket if the IP came back up in less than ten minutes in the ticketing system. Close it instead.
• Don't close the ticket as soon as the customer confirms the circuit is up. Run the 24 hours of monitoring first.
• Don't close a non-responsive ticket before 72 hours. Work the 72-, 48-, and 24-hour follow-ups first.

8. Revision History

v1.1: Original knowledge-base article format. Do's/Don'ts section stated a 15-minute threshold, which conflicted with the 10-minute threshold in the procedure steps.

v2.0: Restructured into standard SOP format. Corrected the threshold inconsistency: the Do's/Don'ts section now states 10 minutes, consistent with Step 1 of the procedure. No other procedural content changed.

v2.1: Clarified that the 10-minute threshold is the age of the ticket automatically generated by the monitoring tool.

v2.2: Prepared for public-facing publication: removed customer names, internal staff names and email addresses, third-party vendor names and contact details, and internal tool names (replaced with neutral terms such as “ticketing system” and “monitoring tool”). Contact details now live in the T1 Internal Contact Sheet. No procedural steps changed.