The 535 UTM experienced a dataplane failure due to a feature that was not enabled during a software update. The software itself should have enabled the option, so it's not clear why that did not happen. The feature was manually enabled on both UTM systems based on TAC engineering advice.
Posted Feb 05, 2024 - 16:37 PST
Update
Network maintenance will occur on the morning of Feb. 1st, 4:00-6:30am. During this time, there will be several network service disruptions ranging in duration from 3-10 minutes.
Between 4:00-5:00am, the SOC will apply software updates to the campus UTM firewalls. The purpose of this update is to address a bug which results in high disk utilization.
Between 5:00-6:30am, the NOC will reboot the North Hall core router, followed by the Public Safety core router. The primary purpose is to reset the internal hardware state of the North Hall router, which has demonstrated problematic behavior. In particular, when the North Hall UTM was active the core router listed client MAC addresses as flapping (rapidly moving) between ports in a manner which is not possible given the topology.
We regret having to do this with little notice, however it is our hope that keeping this to early morning hours will mitigate most of the impact to our clients.
Thank you for your patience.
Posted Jan 31, 2024 - 14:49 PST
Identified
The problem was due to a temporary failure of the primary campus firewall. That firewall has been restarted and appears to be operating properly, and the vendor's technical support team has been contacted. Traffic will be migrated back to the primary firewall between 6:00-6:15pm, with an anticipated disruption to campus traffic of approximately three minutes.
Posted Jan 26, 2024 - 16:25 PST
Investigating
There was a campus network outage for several minutes starting at approximately 2:10pm. The disruption occurred when the network connections across the primary campus firewall failed and traffic was re-routed across a backup firewall. The cause of the connection failure is under investigation.
Posted Jan 26, 2024 - 14:39 PST
This incident affected: Campus Technical Services (Campus Network).