Event Description: mywebhosting and associated services are unreachable
Customer Impact: Websites are not loading and some DNS resolution may be delayed.
Additional Information: Engineers have reseated all cable #2:09PM. Engineers are still on-site trouble-shooting affected equipment #1:30PM. Engineers are trouble-shooting affected equipment #1:00PM Engineers are investigating high-latency #12:30PM Engineers are investigating #12:12PM
]]>Latest message, May 12, 2021 8:28 AM
Title: Issue affecting viewing email content in Outlook
User Impact: Users may be unable to view email message content within Outlook.
More info: Initial reports indicate that Outlook on the web is unaffected and users with access to Outlook on the web can view email messages there while we work on a solution.
Current status: We've finished the deployment of our fix and we've confirmed via monitoring and user reports that the issue has been mitigated. Users will be required to restart their applications for the changes to take effect. Please reference EX255650 for further details.
Previous status: We're gathering and analyzing data in an effort to isolate the cause of impact. Initial reports indicate that impact is specific to the Outlook client and users with access to other protocols, such as Outlook on the web or the Outlook mobile app, can view message content in those platforms as a potential workaround.
Scope of impact: This issue could affect any user attempting to view an email message in the Outlook client.
Next update by: Wednesday, May 12, 2021, 9:30 AM (5/11/2021, 9:30 PM UTC)
]]>User impact: Users may be unable to access multiple Microsoft 365 and Azure services, including the Service Health Dashboard.
More info: Initial reports indicate that primary impact is to Microsoft Teams; however, other services including Exchange Online, Intune and Yammer are also impacted.
Current status: Microsoft have resolved this issue
]]>Last updated: September 29, 2020 1:53 PM
Status: Investigating
User Impact: Users may be unable to access multiple Microsoft 365 services.
More info: Users would be unable to access Outlook.com, Microsoft Teams including Teams Live Events, and Office.com. Additionally, Power Platform and PowerApps properties are affected by this incident.
Existing customer sessions are not impacted and any user who is logged in to an existing session would be able to continue their sessions.
Final status: After extended monitoring of our service, we have confirmed that the issue has been fully mitigated and is running within optimal service levels. If you remain impacted please engage with your support representatives to further assist with any additional issues.
Scope of impact: Any user may experience access problems for Microsoft 365 services.
Start time: Tuesday, September 29, 2020, 10:25 AM (9/28/2020, 9:25 PM UTC)
Next update by: Tuesday, September 29, 2020, 2:00 PM (1:00 AM UTC)
]]>Event Description: A DDOS attack is currently affecting critical infrastructure
Customer Impact: Unable to make or receive calls
Additional Information: Engineers are working on this urgently to resolve
Update Currently resolved however we are monitoring closely.
Reference:
]]>Event Description: myfolders performance issues affecting sync services.
Customer Impact: Syncing may take a long time to complete.
Additional Information: Optimus Systems have identified the cause and resolving issues with the sync service to resume normal operation.
Resolution The fix has been applied and syncing is resuming normally.
Reference:
]]>This is now resolved
Error when attempting to access Microsoft 365 services User
Users may have received an error when attempting to access Microsoft 365 services.
Users may have received the following error message: "AADSTS90033: A transient error had occurred. Please try again." Microsoft Teams - Teams Live Events were impacted by this issue.
Final status: Microsoft redistributed traffic within the affected infrastructure and have validated that service has been restored. Preliminary root cause: An increase in traffic within a portion of infrastructure responsible for Microsoft 365 access had caused an issue in which users were unable to authenticate to Microsoft 365 services.
Microsoft incident ID MO216274
]]>Event Description: Error when attempting to access Microsoft 365 services
Customer Impact: Users may receive an error when attempting to access Microsoft 365 services
Additional Information: Users may the following error message: "AADSTS90033: A transient error has occurred. Please try again". Users hosted in the Australia or New Zealand region may be affected by this event. Microsoft are aware of the issue and issued a global notification ticket and are analysing system logs to determine the source of the issue
Microsoft incident ID MO216274
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 780123277
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 780123277
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>Reference: T20200129.0053
]]>Event Description: Engineering have been made aware of the issue within the URL's of the notifications being incorrect and are treating it with top priority. This means clicking release in your quarantine emails my not work until the DNS issue is resolved. You can still log in directly to the portal using https://mymailsecurity.cloud-protect.net and release your messages in there.
Customer Impact: Unable to release quarantined messages in the mymailsecurity spam digest.
Additional Information: We will continue to remain in contact with engineering and will provide you with updates in regards to the issue with the notifications throughout the day.
Reference: T20200129.0053
]]>Reference: 486347
]]>Reference: 486347
]]>Reference: 486347
]]>Event Description: Issue with DNS queries from OpenDNS. OpenDNS is a security filtering product.
Customer Impact: Unable to browse internet resources.
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>Reference:
]]>Reference:
]]>Event Descriptions: Calls to and from the Spark network are failing over all voice platforms
Engineers are working with the vendor to get the issue resolved
Customer Impact: Inbound calls from the 09 region will have one way audio
Additional Information:
Reference:
]]>HTTP 200 - OK
Reference 778643986
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 778643986
]]>HTTP 200 - OK
Reference 778643986
]]>HTTP 200 - OK
Reference 776440686
]]>HTTP 200 - OK
Reference 780123277
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 780123277
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 778643986
]]>Reference: MO196220
]]>Users may experience access issues with the Microsoft 365 Admin Center, Exchange Online, SharePoint Online, Microsoft Teams, Skype for Business, and Yammer.
]]>Event Description: We've identified a problem that may be preventing customers from accessing the Microsoft 365 admin center and multiple services. We’re working to restore access to all services and we'll provide more information when we’ve isolated the root cause.
Additional Information: https://status.office.com/
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 780123277
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 780123277
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>Reference: 726389
]]>Reference: 726389
]]>Event Description: An issue affecting some outbound emails is occurring. This results in a bounce-back message.
Customer Impact: Intermittent sent messages may result in a bounce-back message indicating the message was not delivered.
Reference: 726389
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 780123277
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 780123277
]]>HTTP 200 - OK
Reference 780123277
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 780123277
]]>HTTP 200 - OK
Reference 780123277
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 780123277
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 778643986
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 778643986
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 778643986
]]>HTTP 200 - OK
Reference 781902655
]]>HTTP 200 - OK
Reference 780123277
]]>Event Description: HTTP 404 - Not Found
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 780123277
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 778643986
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>Reference: 486271
]]>Event Description: Our automated systems have detected poor call quality on some calls. Engineers are investigating
Customer Impact: Customers may experience poor call quality on some calls.
Reference: 486271
]]>HTTP 200 - OK
Reference 778643986
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 778643986
]]>HTTP 200 - OK
Reference 778643986
]]>HTTP 200 - OK
Reference 776440686
]]>HTTP 200 - OK
Reference 780123277
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 780123277
]]>HTTP 200 - OK
Reference 781902655
]]>HTTP 200 - OK
Reference 780123277
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 778643986
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 780123277
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>HTTP 200 - OK
Reference 778643986
]]>HTTP 200 - OK
Reference 780123277
]]>HTTP 200 - OK
Reference 776440686
]]>HTTP 200 - OK
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 781902655
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 778643986
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 776440686
]]>Event Description: Connection Timeout
Additional Information: Further updates will be provided as the issue is investigated.
Reference 780123277
]]>Internal Server Error
Reference 9999999999
]]>Event Description: Office 365 for mail and all associated services including Skype for Business and Microsoft Teams is currently experiencing a major outage.
Customer Impact: All of these services are either down, or experiencing significant performance problems
]]>Start Date/Time: 05/11/2018 09:20 - Resolution Date/Time: 05/11/2018 16:15 - Duration: 6 hours 55 minutes
Event Description: [Restored] Voice services were offline.
Customer Impact: Customers would have experienced incoming and outbound calling failures.
Summary of Incident: Optimus Systems have several parts to our voice network, the affected part of the network was: CVS - Cloud Voice Switch - This is the hosts our inter connections, which terminates calls to and from landlines and mobiles. All calls in and out of CVS were affected and down. Most of our customers leverage CVS in some part as it connects at the root level back to our PBX instances.
Timeline of Incident:
Post event actions:
Reference: #486252
]]>Reference: #486252
]]>Reference: #486252
]]>Reference: #486252
]]>Reference: #486252
]]>Reference: #486252
]]>Reference: #486252
]]>Reference: #486252
]]>Reference: #486252
]]>Event Description: Voice Services offline
Customer Impact: Calls are failing to complete.
Additional Information: Engineers Investigating
Reference: #486252
]]>Start Date/Time: 18/10/2018 16:07 - Resolution Date/Time: 18/10/2018 16:42 - Duration: 35 minutes
Event Description: [Restored] Degradation of Voice Services - Inbound/Outbound
Customer Impact: Some customers would have experienced issues with both incoming and outbound calls.
Summary of Incident: Following on from the voice outage yesterday afternoon, below is a timeline and summary of the outage:
Our monitoring system indicated that an upstream interconnect Session Border Controller (SBC) was under high load. At this time multiple tests were run to investigate the issue.
SIP Trunk registrations and calls started dropping for customers and an urgent escalation was raised. The upstream interconnect provider made the decision to restart their SBC, which appeared to resolve the issue, while we re-routed services to a secondary interconnect simultaneously.
Post event actions: Our engineers are reviewing SIP trunk SRV DNS records to ensure calls and registrations failover to our secondary interconnects quicker and will be implementing a change control process to action this.
Time line:
15:56 – First notification received of high load on host – engineers start investigating. Multiple tests calls are placed which are all successful.
16:00 – Notification received that the service was up.
16:07 – Continued notification of degraded service.
16:09 – Registrations on SIP Trunk servers start reducing and some calls are failing.
16:17 – Advised via Help Desk of customer calls
16:20 – Case raised with primary interconnect who restarted their SBC in attempt to resolve issue.
16:22 – Engineers believe the restart of the SBC resolved issue. Continued monitoring the situation.
16:35 - Engineers re-routed SIP Trunk calls to a secondary interconnect.
16:42 – Calls and registrations monitored as successful.
16:44 – Issue confirmed resolved and extended monitoring underway.
]]>Reference: 486247
]]>Reference: 486247
]]>Event Description: Degradation of Voice Services with upstream carriers.
Customer Impact: Customer may experience issues with both incoming and outbound calls.
Additional Information: Engineers are investigating the issue
Reference: 486247
]]>Event Description: CVS Time of day routing is having issues. Voice engineers are investigating.
Customer Impact: Some inbound calls may not be connecting.
Additional Information: Work around - Change the time of day setting to Forced On. Contact the Optimus Systems Service Desk if you are experiencing any issues.
Update 9:41AM 1/10/2018 - The issue is now resolved, status is set to monitoring.
** Resolved ** 10:21AM 1/10/2018 - The issue is now fully resolved.
Reference: INC2018101
]]>This is the final update on this voice outage and a detailed incident report will be reported here within the next 24 hours.
Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Event Description: SIP Trunk outage is occurring with no inbound and outbound calls being connected.
Customer Impact: Inbound and Outbound calls are failing.
Additional Information: Our upstream carriers are continuing to investigate the underlying cause of the issues which appear to be load related. They have not seen any evidence of a denial of service attack at this stage. We are continuing to work with the carrier to try to identify additional causes of the massive load and working on increasing resources.
Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Reference: 486203
]]>Event Description: Degradation of Voice Services
Customer Impact: A number of calls both inbound and outbound are failing to connect.
Additional Information: Our upstream carriers are looking into this issue currently.
Reference: 486203
]]>Event Description: We are investigating a mobile calling issue where some calls are failing, specifically 2degrees numbers
Customer Impact: Low level of mobile calls are failing. A fault has been raised with our up stream carrier.
Additional Information: Engineers are investigating
Reference: 486191
Update: 29/05/2018 14:40hrs We have been working with 2Degrees to understand the changes made in their network that have caused this issue. Currently the call connection rate is back to expected levels and engineers are keeping a close observation on calling levels.
]]>Event Description: Some internet connectivity components are currently degraded. An upstream provider has performed upgrades across various circuits and are looking to roll back.
Customer Impact: Some websites, emails or other internet services are slow or fail to connect.
Additional Information:
Reference: 635618
Update: 9:12AM Issue has been identified and resolution is being implemented to all connections.
Update: 9:40AM Resolution has been implemented to all connections. Issue is in monitor state. Please raise any further issues with the helpdesk.
Update: 11:10AM No further issues have been experienced, this issue is marked as resolved.
]]>Event Description: Unplanned - Currently having issues with Time based call flows (Time of the Day) Callflows in some mycloudpbx instances.
Customer Impact: Some calls may not route correctly based on the Time of Day settings.
Additional Information: Further updates will be provided as the issue is investigated.
Reference OPT081801
Update: 10:15AM A workaround has been implemented and call flows are now performing normally. Investigation is still on-going.
Update: 5:40PM The issue has been identified and is currently being worked on for resolution.
Update: 8:45AM 9/01/2018 This issue is still being investigated - the workaround is still in place, no issues should currently be experienced.
Update: 9:00AM 9/01/2018 The issue is now resolved, all time of day call flows will now perform normally, the workaround has been removed.
]]>Event Description: Connectivity Issues
Customer Impact: No connectivity to subscribed services
Additional Information: Engineers are investigating currently
Reference: OPT17121
UPDATE 15:29 1/12/17: Services have been restored and are under observation by our engineers, the cause of the outage is still under investigation.
RESOLVED 21:00 1/12/17: Investigations revealed an issue occurred with one of our core routers after being subjected to a DDoS attack. Steps were taken to re-distribute the traffic elsewhere and service was restored. In addition to this, the manufacturer released specific features to better deal with this going forward and therefore a change process has been proposed to implement new firmware.
]]>Event Description: Inbound/Outbound calls on SIP Trunks are failing to connect due to an outage with an upstream carrier.
Customer Impact: Customers may be experiencing a loss of Voice services affecting both incoming and outgoing calls.
Additional Information: Engineers are working on a resolution, more updates to follow shortly.
Reference: 486109
UPDATE 11:44 5/9/17: Services appear to be restored and are under observation by our engineers, the cause of the outage is still under investigation.
]]>Event Description: The services are not responding meaning web services aren't accessible.
Customer Impact: Web services and file synchronisation aren't functional.
Additional Information: Engineers are working on finding the root cause and resolving it.
Reference: OPT1034
Update: 17:15hrs Our engineers discovered the faulty module and have resolved the issue. We will continue to monitor this issue.
]]>Event Description: We have observed an increasing amount of international network issues to our Datacentre resources in the following subnets:
The issue presents as mini 30-60 second traffic outages from international locations.
Early investigations with our backbone internet provider indicate that this issue has been reported by multiple ISP's and that the issue is isolated to inbound international traffic that is transiting the backbone connection points that exit LAX in the United States (West Coast)
Customer Impact: All other sources of inbound internet browsing/traffic remain functional and ISP's using the West Coast link should either have fail-over routing in place or the connection will retry in 30 seconds and continue operation.
Additional Information: We are continuing to investigate this and reach a resolution as soon as possible
Reference: DC17030
Update 14/082017 10:45am: Our upstream backhaul providers have found degraded connectivity on one of their border routers in Los Angeles, to alleviate this they have disabled the affected device and re-routed some transit data at the POP which has stabilised connectivity to/from the US.
]]>UPDATE 10:55AM - Engineers are still investigating however the issue doesn't appear to be continuing at this point.
UPDATE 11:25AM - Engineers have identified cause of issue with an interconnect for an upstream connection. Upstream provider has resolved this issue. NOC continue to monitor.
UPDATE 1:15PM - NOC have monitored this with no further issues. Issue is resolved.
]]>29/03/2017 16:40 - Update: While engineers are continuing to investigate the issue, we have seen some of the calls getting through. Further updates will be sent out once we have confirmation from the Upstream carrier on the progress.
29/03/2017 17:55 - Resolved: Our upstream carrier confirms the issue has now been resolved. Our engineers are continuing to monitor the situation closely. An incident report with further details to follow within 48hrs
]]>15/02/2017 11:00 - Diagnosed An issue with the MySQL database packages has been detected and a socket error has been encountered, which has replicated through the cluster. Now the issue has been investigated an appropriate fix will be applied, tested and the outcome reported here shortly.
15/02/2017 11:15 - Resolved Engineers have restored services and will continue to monitor. An incident report detailing root cause of the issue will be supplied within next 72 hours.
]]>13/02/2017 11:47 - Update The issue has been isolated to an upstream carrier's infrastructure. We are working with them under urgency to restore services.
13/02/2017 11:57 - Update Some calls are now completing, however intermittent issues continue. We are still working with the upstream carrier.
13/02/2017 12:07 - Update Most calls are now completing successfully both inbound and outbound. We are continuing to work on this.
13/02/2017 12:17 - Resolved Engineers restored services and will continue monitor the issue. An incident report detailing root cause of the issue will be supplied within next 24 hours.
]]>07/02/2017 11:54 - Resolved Engineers have investigated and found the clustered Apache service was over utilizing resources across the cluster after a security patch was applied. This was killed and restarted. Access has been tested and is functional. We will watch this specific event in hyper-care mode for the next 2 hours before confirming a complete resolution.
09/02/2017 10:38 - Closed We have monitored this closely and have identified the specific process inside the clustered Apache service that was causing problems and had development patch the issue.
]]>16/12/2013 20:28 - Resolved Have investigated and found a hung apache process effecting a few connections. This was killed and restarted. Access has been tested and is functional. Monitoring will continue.
]]>DSL connections may be affected due to broadband provider backhaul issues. Chorus have advised a number of network elements are down due to the earthquake affecting a large part of the New Zealand. Connections may fluctuate during the day due to Chorus re-routing traffic and working on returning services.
All Optimus Systems network elements are in working order, and we are not experiencing any quake related issues on the network.
]]>Update 23:35 SSL Certificate has been updated and tested.
]]>We are currently experiencing network degradation specifically international connectivity that is currently causing network constraints and connectivity issues to our servers. Our network engineers are working to resolve this asap. We apologise for any inconvenience or impact this is having.
Update: 4:57pm Network engineers have managed to mitigate the network issues now and continue to monitor the situation.
]]>