UCSB Systems Health Dashboard
Update - We continue to work on this issue, and have some more details to share:
- Users of all device types and operating systems can be impacted - MacOS, Windows, iOS, Android, and ChromeOS have all been reported impacted.
- All reports seem to originate from users of the Cox network and specifically in Isla Vista.
- Cox and Duo have both been engaged to help in resolving this issue.
- The typically effective workaround is to move your device onto a non-Cox network, such as the UCSB VPN, a wifi hotspot, or disabling wifi on a mobile device so that it falls back to its cellular connection.
- If you have this issue, the IT Support desks should be able to help you get connected, if temporarily:
--- Housing Resident IT support is available at 805-893-5555
--- Student IT Support is available at 805-893-4357 or https://www.lsit.ucsb.edu/get-help
--- Faculty and Staff IT Support can be reached at 805-893-5000 or https://ithelp.ucsb.edu

- There is another unrelated Duo issue where the prompt fails to load that looks similar but has another cause and impacts only Apple devices. If you have an Apple device it is worth checking out this article to see if it helps resolve your issue: https://help.duo.com/s/article/3710

Sep 28, 2022 - 12:02 PDT
Investigating - IT support desks across campus are aware of multiple cases where some Apple devices (Macs and iPhones) are not properly receiving the Duo prompt and thereby are unable to complete 2 factor authentication. The Duo prompt may appear blank or as a an empty grey box or might only be partially loaded. Engineering teams are investigating, possible workarounds may include changing to a different network or resetting your network connection.
Sep 15, 2022 - 16:28 PDT

About This Site

Welcome to UCSB Systems Health Dashboard. This site provides information on the health of resources available to UCSB, including the option of subscribing to email or text messages advising you when a Major Incident takes place. For assistance/information on this site, please contact the Core IT Services Service Desk at (805) 893-5000. For assistance or questions on any specific system, please contact the Service Provider responsible for the support. The contact will be provided within the description of the resource.

Departmental Systems Health Dashboards:
Administrative & Residential IT: https://status.arit.ucsb.edu/
Instructional Development: http://status.id.ucsb.edu/
Student Affairs: https://status.sa.ucsb.edu/
Library: https://status.library.ucsb.edu/

Campus Business Applications ? Operational
BARC ? Operational
Concur ? Operational
Data Warehouse ? Operational
Electronic Timekeeping (Kronos) ? Operational
ESPRESSO Applications ? Operational
Gateway ? Operational
Gateway Management Console (GMC) ? Operational
Hyperion ? Operational
OnLine General Ledger (OLGL) ? Operational
PeopleSoft Financials ? Operational
UC Learning Center ? Operational
UCPATH ? Operational
UCPath Operational Data Store ? Operational
Docusign ? Operational
Student Business Applications Operational
GOLD (Gaucho Online Data) ? Operational
GauchoSpace ? Operational
GauchoSpace Archive (Summer 2015-Fall 2021) Operational
GauchoPrint ? Operational
Administrative Services Applications ? Operational
WebTMA ? Operational
MetaSys ? Operational
Lenel (Campus Door Access) Operational
Messaging, Collaboration, and Communication Services ? Operational
Box Box Drive Operational
Box Box Sync Operational
Box Uploads/Downloads Operational
Box Storage Services ? Operational
Box WebDAV Operational
Campus (UCSB) Connect Email ? Operational
Google Apps Gmail ? Operational
Google Apps Groups ? Operational
Google Apps Hangouts ? Operational
Google Calendar ? Operational
Google Drive ? Operational
Other Google Services ? Operational
Pantheon Operations Customer Sites ? Operational
Pantheon Operations Dashboard Operational
Pantheon Operations Spinup Operations ? Operational
Pantheon Operations Workflow Operations ? Operational
Smartsheet Reporting ? Operational
Smartsheet SSO (Single Sign-On) ? Operational
Smartsheet Web Application ? Operational
Smartsheet System Generated Emails ? Operational
Smartsheet Integrations ? Operational
Smartsheet Resource Management ? Operational
Smartsheet Mobile/API ? Operational
Smartsheet Search Functionality ? Operational
Smartsheet Connectors ? Operational
Zoom Web Conferencing Services ? Operational
Core IT Email Forwarding (smtp.ucsb.edu) ? Operational
Google Apps Google Meet Operational
Campus Identity and Authentication Services ? Partial Outage
Identity ? Operational
LDAP ? Operational
SSO ? Operational
Duo Core Authentication Service Partial Outage
Duo Admin Panel Operational
Duo Push Delivery Operational
Duo Phone Call Delivery Operational
Duo SMS Message Delivery Operational
Duo Cloud PKI Operational
Campus Technical Services ? Operational
Campus Network ? Operational
GoAnywhere ? Operational
Mainframe ? Operational
North Hall Data Center ? Operational
ServiceNow ? Operational
monitor.ets.ucsb.edu ? Operational
Azure ? Operational
AWS ? Operational
CITS Service Desk ? Operational
StatusPage Administration Operational
UCSB StatusPage Admin Message ? Operational
Facilities Management Services Operational
Campus Potable Water System ? Operational
Campus Electrical System ? Operational
Library Services Operational
Library Main Website ? Operational
Library Catalog (Primo) Operational
Journal Access (EZProxy) ? Operational
Library Digital Repository (ADRL) ? Operational
Course Reserves (Ares) ? Operational
Public Computers Operational
GauchoPrint - Library Printers ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Sep 29, 2022

No incidents reported today.

Sep 28, 2022
Resolved - As of 1:43 PM PDT, error rates and latencies for invoked on API Gateway endpoints in the US-WEST-2 Region are now at normal levels. The issue began at 9:20 AM PDT when error rates and latencies for API Gateway began to increase. Error rates began to improve at 10:38 AM PDT, when engineers took action to reduce contention within the subsystem that handles request processing for API Gateway. Error rates continued to improve until 1:10 PM PDT, when engineers applied a mitigation to resolve the contention within the affected subsystem. These actions accelerated recovery, and by 1:43 PM PDT, error rates and latencies had returned to normal levels. Affected AWS services have now recovered as well. The issue has been resolved and the service is operating normally.
Sep 28, 16:15 PDT
Update - Latest update from Amazon:

Increased Invoke Error Rates
11:33 AM PDT We continue to work on resolving the elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. We continue to see a significant improvement in error rates, starting at 10:40 AM PDT, but are not seeing full recovery yet. The issue is caused by contention within the subsystem that is responsible for request processing within the API Gateway service. Engineers are engaged and have applied traffic filters as a precautionary measure, while they work to identify the root cause and resolve the issue. Engineers continue to work to reduce contention within the affected subsystem, which we believe will resolve the elevated error rates and latencies. Customers with applications that use API Gateway, or customers invoking Lambda functions via API Gateway, will be experiencing elevated error rates and latencies as a result of this issue. The AWS services listed below are also experiencing elevated error rates as a result of this issue. While we have seen improvements in error rates since 10:40 AM PDT, recovery has stalled and we do not have a clear ETA on full recovery. For customers that have dependencies on API Gateway and are experiencing error rates, we do not have any mitigations to recommend to address the issue on the customer side. We do expect error rates to continue to improve as contention with the affected subsystem resides, and will provide further updates as recovery progresses.

Sep 28, 12:11 PDT
Investigating - The is a problem with the Amazon API Gateway in US-West-2 beginning at 9:40am and ongoing. Multiple services are impacted - EC2, Connect, Workspaces, and many others. Amazon is actively working to resolve the issue, and UCSB engineers are monitoring the situation.
Sep 28, 11:19 PDT
Resolved - [RESOLVED] Increased Invoke Error Rates
2:05 PM PDT As of 1:43 PM PDT, error rates and latencies for invoked on API Gateway endpoints in the US-WEST-2 Region are now at normal levels. The issue began at 9:20 AM PDT when error rates and latencies for API Gateway began to increase. Error rates began to improve at 10:38 AM PDT, when engineers took action to reduce contention within the subsystem that handles request processing for API Gateway. Error rates continued to improve until 1:10 PM PDT, when engineers applied a mitigation to resolve the contention within the affected subsystem. These actions accelerated recovery, and by 1:43 PM PDT, error rates and latencies had returned to normal levels. Affected AWS services have now recovered as well. The issue has been resolved and the service is operating normally.

Sep 28, 14:42 PDT
Update - AWS is still researching the cause of this incident (related to the general AWS outage). In the meanwhile, and if you require services from the CITS Service Desk, please go to our portal at ithelp.ucsb.edu to create a self-service ticket or use the online chat feature (located at the top right on the same portal page) to initiate an online chat with one of our Service Desk technicians. We apologize for the inconvenience and appreciate your patience.
Sep 28, 11:44 PDT
Investigating - The Call Center (AWS Connect) for the Service Desk and the Campus Operator phone number are both down. AWS engineers are working on the problem.
Sep 28, 09:54 PDT
Sep 27, 2022

No incidents reported.

Sep 26, 2022

No incidents reported.

Sep 25, 2022

No incidents reported.

Sep 24, 2022

No incidents reported.

Sep 23, 2022

No incidents reported.

Sep 22, 2022

No incidents reported.

Sep 21, 2022

No incidents reported.

Sep 20, 2022

No incidents reported.

Sep 19, 2022

No incidents reported.

Sep 18, 2022

No incidents reported.

Sep 17, 2022

No incidents reported.

Sep 16, 2022
Completed - Scheduled maintenance has been completed and GauchoSpace is now available.
Sep 16, 11:38 PDT
In progress - Scheduled maintenance has concluded and GauchoSpace is now available.
Sep 16, 06:00 PDT
Scheduled - GauchoSpace will be unavailable on Friday, September 16 from 6:00am to 2:00pm for end-of-quarter updates and maintenance.

Find out about the transition from GauchoSpace to Canvas here: https://www.canvas.ucsb.edu/transition

Sep 8, 10:18 PDT
Sep 15, 2022