UCSB Systems Health Dashboard
All Systems Operational

About This Site

Welcome to UCSB Systems Health Dashboard. This site provides information on the health of resources available to UCSB, including the option of subscribing to email or text messages advising you when a Major Incident takes place. For assistance/information on this site, please contact the Core IT Services Service Desk at (805) 893-5000. For assistance or questions on any specific system, please contact the Service Provider responsible for the support. The contact will be provided within the description of the resource.

Departmental Systems Health Dashboards:
Administrative & Residential IT: https://status.arit.ucsb.edu/
Enterprise Technology Services: https://status.ets.ucsb.edu/
Instructional Development: http://status.id.ucsb.edu/
Student Affairs: https://status.sa.ucsb.edu/

Campus Business Applications ? Operational
BARC ? Operational
Data Warehouse ? Operational
Electronic Timekeeping (Kronos) ? Operational
ESPRESSO Applications ? Operational
Gateway ? Operational
Gateway Management Console (GMC) ? Operational
Hyperion ? Operational
OnLine General Ledger (OLGL) ? Operational
PeopleSoft Financials ? Operational
UC Learning Center ? Operational
UCPATH ? Operational
UCPath Operational Data Store ? Operational
Student Business Applications Operational
GOLD (Gaucho Online Data) ? Operational
GauchoSpace ? Operational
GauchoPrint ? Operational
Administrative Services Applications ? Operational
WebTMA ? Operational
MetaSys ? Operational
Lenel (Campus Door Access) Operational
Messaging, Collaboration, and Communication Services ? Operational
Box Box Drive Operational
Box Box Sync Operational
Box Uploads/Downloads Operational
Box Storage Services ? Operational
Box WebDAV Operational
Campus (UCSB) Connect Email ? Operational
Google Apps Gmail ? Operational
Google Apps Groups ? Operational
Google Apps Hangouts ? Operational
Google Calendar ? Operational
Google Drive ? Operational
Other Google Services ? Operational
Pantheon Operations Customer Sites ? Operational
Pantheon Operations Dashboard Operational
Pantheon Operations Spinup Operations ? Operational
Pantheon Operations Workflow Operations ? Operational
Smartsheet Reporting ? Operational
Smartsheet SSO (Single Sign-On) ? Operational
Smartsheet Web Application ? Operational
Smartsheet System Generated Emails ? Operational
Smartsheet Integrations ? Operational
Smartsheet Resource Management ? Operational
Smartsheet Mobile/API ? Operational
Smartsheet Search Functionality ? Operational
Smartsheet Connectors ? Operational
Zoom Web Conferencing Services ? Operational
Core IT Email Forwarding (smtp.ucsb.edu) ? Operational
Campus Identity and Authentication Services ? Operational
Identity ? Operational
LDAP ? Operational
SSO ? Operational
Duo Core Authentication Service Operational
Duo Admin Panel Operational
Duo Push Delivery Operational
Duo Phone Call Delivery Operational
Duo SMS Message Delivery Operational
Duo Cloud PKI Operational
Campus Technical Services ? Operational
Campus Network ? Operational
GoAnywhere ? Operational
Mainframe ? Operational
North Hall Data Center ? Operational
ServiceNow ? Operational
monitor.ets.ucsb.edu ? Operational
StatusPage Administration Operational
UCSB StatusPage Admin Message ? Operational
Facilities Management Services Operational
Campus Potable Water System ? Operational
Campus Electrical System ? Operational
Library Services Operational
Library Main Website ? Operational
Library Catalog (Alma/Primo) Operational
Journal Access (EZProxy) ? Operational
Library Digital Repository (ADRL) ? Operational
Course Reserves (Ares) ? Operational
Public Computers Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Due to maintenance on our PeopleSoft Financials system, Gateway and all Espresso applications, with the exception of the Online GL, will be down early this Sunday morning, April 18th from 1am until 8am.
Posted on Apr 16, 15:43 PDT
Past Incidents
Apr 16, 2021

No incidents reported today.

Apr 15, 2021
Resolved - This incident has been resolved.
Apr 15, 14:02 PDT
Investigating - We are aware of a possible issue with VPN. The responsible service operators have been notified and an investigation is proceeding.
Apr 15, 13:48 PDT
Resolved - The OLGL is now available
Apr 15, 08:29 PDT
Investigating - We are aware of a possible issue with OnLine General Ledger (OLGL). The responsible service operators have been notified and an investigation is proceeding.
Apr 15, 08:17 PDT
Apr 14, 2021

No incidents reported.

Apr 13, 2021
Resolved - The problem with Google Docs has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google and we are making continuous improvements to make our systems better.
Apr 13, 12:34 PDT
Investigating - We're investigating reports of an issue with Google Docs. We will provide more information shortly. The affected users are able to access Google Docs, but are seeing error messages, high latency, and/or other unexpected behavior.
Apr 13, 12:01 PDT
Apr 12, 2021

No incidents reported.

Apr 11, 2021

No incidents reported.

Apr 10, 2021

No incidents reported.

Apr 9, 2021

No incidents reported.

Apr 8, 2021
Resolved - The issue with site creation on Pantheon has been resolved.
Apr 8, 13:38 PDT
Investigating - Pantheon support is looking into an issue where drupal website creation is malfunctioning for UCSB.
Mar 19, 13:31 PDT
Apr 7, 2021
Resolved - RCA - DNS issue impacting multiple Microsoft services (Tracking ID GVY5-TZZ)
Summary of Impact: Between 21:21 UTC and 22:00 UTC on 1 Apr 2021, Azure DNS experienced a service availability issue. This resulted in customers being unable to resolve domain names for services they use, which resulted in intermittent failures accessing or managing Azure and Microsoft services. Due to the nature of DNS, the impact of the issue was observed across multiple regions. Recovery time varied by service, but the majority of services recovered by 22:30 UTC.

Root Cause: Azure DNS servers experienced an anomalous surge in DNS queries from across the globe targeting a set of domains hosted on Azure. Normally, Azure's layers of caches and traffic shaping would mitigate this surge. In this incident, one specific sequence of events exposed a code defect in our DNS service that reduced the efficiency of our DNS Edge caches. As our DNS service became overloaded, DNS clients began frequent retries of their requests which added workload to the DNS service. Since client retries are considered legitimate DNS traffic, this traffic was not dropped by our volumetric spike mitigation systems. This increase in traffic led to decreased availability of our DNS service.

Mitigation: The decrease in service availability triggered our monitoring systems and engaged our engineers. Our DNS services automatically recovered themselves by 22:00 UTC. This recovery time exceeded our design goal, and our engineers prepared additional serving capacity and the ability to answer DNS queries from the volumetric spike mitigation system in case further mitigation steps were needed. The majority of services were fully recovered by 22:30 UTC. Immediately after the incident, we updated the logic on the volumetric spike mitigation system to protect the DNS service from excessive retries.

Next Steps: We apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):

Repair the code defect so that all requests can be efficiently handled in cache.
Improve the automatic detection and mitigation of anomalous traffic patterns.
Apr 7, 16:58 PDT
Investigating - https://status.azure.com/en-us/status
Azure DNS - Investigating

We are currently investigating reports of an issue affecting Azure DNS. More information will be provided as it is known.

This message was last updated at 21:59 UTC on 01 April 2021

Warning DNS issues - Investigating

Engineering is investigating an issue with DNS that is impacting several downstream Azure services.

This message was last updated at 22:07 UTC on 01 April 2021
Apr 1, 15:20 PDT
Apr 6, 2021

No incidents reported.

Apr 5, 2021
Resolved - Issue with SSO has been identified and resolved.
Apr 5, 15:50 PDT
Investigating - We are experiencing issues with the timekeeping logon screen. Upon logging in users are directed to https://ucsb.kronos.net/wfc/navigator/logonWithUID

This issue is currently being investigated.
Apr 5, 15:17 PDT
Resolved - Database activity on GauchoSpace has returned to normal patterns, without any service degradation. We will continue to monitor the systems for any issues.
Apr 5, 12:20 PDT
Update - Load spikes on the database server for GauchoSpace have mostly subsided, although there may still be periodic degradation. We adjusted the frequency of an external service pushing updates to GauchoSpace, but are still investigating the root cause of the issue.
Apr 2, 14:25 PDT
Update - We are still seeing load spikes affecting the database, including a recurring issue causing a brief outage from approximately 7:36 to 7:41 each morning this week. We are continuing to investigate the cause of the issue and will update when we have identified the source.
Mar 31, 09:51 PDT
Investigating - We are experiencing load spikes on the database server for GauchoSpace, resulting in periodic service disruptions. We are investigating the underlying issue.
Mar 29, 10:53 PDT
Apr 4, 2021

No incidents reported.

Apr 3, 2021

No incidents reported.

Apr 2, 2021