News   Apr 25, 2024
How It Started
 107     0 
News   Apr 24, 2024
Latest Iteration of 50 Eglinton West Proposes 46- and 58-Storey Towers
 1K     1 
News   Apr 24, 2024
Sustainability Month: Mass Timber Construction Showing Long-Term Economic Benefits
 1.6K     1 

Telecoms in Canada

It's times like this I'm happy I don’t depend on a cell phone and always carry or have cash on hand. My cell service is down but my Bell landline is working perfectly.

www.blogto.com

Here are all the ways Toronto is falling apart because of the Rogers internet outage

It's absolute mayhem across the city this morning as many services in Toronto have been rendered useless thanks to a huge internet outage from Roge...
www.blogto.com www.blogto.com

www.blogto.com

Chaos breaks out in Toronto as Rogers internet outage takes down debit machines

Interac services are down all across Toronto this morning thanks to a massive, nationwide Rogers internet outage, which means that nobody can use t...
www.blogto.com www.blogto.com
 
gabe said:
The dangers of a near monopoly. ATMs, Non cash payment abilities, 911 services, internet and mobile from almost all of Canada are down!

www.ctvnews.ca

Rogers wireless services 'starting to recover,' company says

Rogers said Friday evening it is beginning to see its wireless services return online following a nearly day-long nationwide network outage.
www.ctvnews.ca www.ctvnews.ca




https://twitter.com/x/status/1545398324852826112
Click to expand...
While I get the point here, this shouldn't happen whether monopoly or not.

I am really interested in the explanation as to how the entire network was impacted, and it's for such a long period of time. My internet is still not back after now after 9 hours.
Obviously, a sophisticated cyber attack is one thing, but if it isn't actually that, what the hell did they do operationally that would allow for the entire network to blow up like this? Why is it taking so long to restore service? I have to wonder about their operational risk management here, (again assuming this wasn't an outside attack of extremely high scale and sophistication). This kind of catastrophic failure shouldn't be possible. There should be multiple redundant operational and safety protocols to prevent anything like this from occuring.

I hope they are summoned to a parliamentary committee to explain.
 
12 hours and counting....

No word from Rogers what's happening, just the same copy and paste pre-written “apology” from last year’s major outage. This is beyond ridiculous how this keeps happening.
 
gabe said:
12 hours and counting....

No word from Rogers what's happening, just the same copy and paste pre-written “apology” from last year’s major outage. This is beyond ridiculous how this keeps happening.
Click to expand...
My internet just came back a few minutes ago.
 
I experienced how unwise it is to use a single supplier, or in my case TekSavvy, a reseller. My business partner and I use Rogers directly or indirectly at home and at our office for internet, internet phone and cell service. What a nightmare. I looked at Bell offerings. Their landlines are rather expensive and their internet tops at 10 Mbps at the Manulife Centre... I have no alternative to my current service.
 
Jonny5 said:
While I get the point here, this shouldn't happen whether monopoly or not.

I am really interested in the explanation as to how the entire network was impacted, and it's for such a long period of time. My internet is still not back after now after 9 hours.
Obviously, a sophisticated cyber attack is one thing, but if it isn't actually that, what the hell did they do operationally that would allow for the entire network to blow up like this? Why is it taking so long to restore service? I have to wonder about their operational risk management here, (again assuming this wasn't an outside attack of extremely high scale and sophistication). This kind of catastrophic failure shouldn't be possible. There should be multiple redundant operational and safety protocols to prevent anything like this from occuring.

I hope they are summoned to a parliamentary committee to explain.
Click to expand...

So the exact details of how they blew themselves up are not yet known............but the gist is becoming clear.........

At 445am yesterday they moved to update their BGPs (Border Gateway Protocols)........and.........well.......that didn't go well.

This is the same issues the blew up Facebook/Insta awhile back.

Here's a decent little primer:


Below is a timelapse of Rogers disappearing from the internet........

https://www.reddit.com/r/Rogers/comments/vuk17t
 
This is why I split my internet and cellphone between Rogers and Bell.
If one is out, I have the other as backup.

Rogers had been pretty good for me the last 10 years, until yesterday at least. Oh well
 
Northern Light said:
So the exact details of how they blew themselves up are not yet known............but the gist is becoming clear.........

At 445am yesterday they moved to update their BGPs (Border Gateway Protocols)........and.........well.......that didn't go well.

This is the same issues the blew up Facebook/Insta awhile back.

Here's a decent little primer:


Below is a timelapse of Rogers disappearing from the internet........

https://www.reddit.com/r/Rogers/comments/vuk17t
Click to expand...
It seems their updates/upgrades are taking lessons from Microsoft.

Ours came back around 2200. Our Internet is a third-party ISP on the Rogers network but our TV is satellite and we still have a hard landline. While some of the media was confusing, it seems that 911 wasn't directly affected, but people couldn't call it via a Rogers-connected device (well, duh).

I do think the government has a regulatory role here. Bored thumbs aside, so much of our critical infrastructure, economy and perhaps security has been surrendered to online platforms and I think it is reasonable to expect that these networks be robust and have better level of redundancy. While I understand some areas are different, when our power goes down, our Internet drops immediately. It is unreasonable to expect that businesses pay for two parallel services just in case.
 
lenaitch said:
It seems their updates/upgrades are taking lessons from Microsoft.

Ours came back around 2200. Our Internet is a third-party ISP on the Rogers network but our TV is satellite and we still have a hard landline. While some of the media was confusing, it seems that 911 wasn't directly affected, but people couldn't call it via a Rogers-connected device (well, duh).

I do think the government has a regulatory role here. Bored thumbs aside, so much of our critical infrastructure, economy and perhaps security has been surrendered to online platforms and I think it is reasonable to expect that these networks be robust and have better level of redundancy. While I understand some areas are different, when our power goes down, our Internet drops immediately. It is unreasonable to expect that businesses pay for two parallel services just in case.
Click to expand...
Something interesting I hadn't considered noted in a Reddit forum by a person who claims to know an insider is that there was mass confusion for the first few hours as many key Rogers executives and staff could not be reached because, naturally, they all use Rogers phones and internet. The normal procedure would be to initiate a "fire call" as soon as things went down at 5:00 a.m.; in operations parlance, a fire call is where you are authorised to wake up important people up by phoning them no matter the time of day because the matter is so urgent, but this failed as those people could not receive phone calls or emails. Even when those people finally woke up Friday morning and realised what was happening and made the necessary arrangements to get internet access on their phone, they encountered problems like needing to be physically present at remote data centres to help diagnose the issue as their Rogers laptops couldn't connect to the company network, and the internal diagnostic tools on the Rogers network were inaccessible, so it took hours of travel time to have key people move to the correct data centres to get direct access to the network systems and figure out what to do.
 
Last edited:
Jonny5 said:
Something interesting I hadn't considered noted in a Reddit forum by a person who claims to know an insider is that there was mass confusion for the first few hours as many key Rogers executives and staff could not be reached because, naturally, they all use Rogers phones and internet. The normal procedure would be to initiate a "fire call" as soon as things went down at 5:00 a.m.; in operations parlance, a fire call is where you are authorised to wake up important people up by phoning them no matter the time of day because the matter is so urgent, but this failed as those people could not receive phone calls or emails. Even when those people finally woke up Friday morning and realised what was happening and made the necessary arrangements to get internet access on their phone, they encountered problems like needing to be physically present at remote data centres to help diagnose the issue as their Rogers laptops couldn't connect to the company network, and the internal diagnostic tools on the Rogers network were inaccessible, so it took hours of travel time to have key people move to the correct data centres to get direct access to the network systems and figure out what to do.
Click to expand...
This is kind of absurd. My employer uses redundant data connections at all major premises. We go so far as ensuring the connections enter the building at different places, so one guy with a backhoe can't cut both lines. We were ticking along fine on Friday beside the fact that 25% of our WFH workforce had to make an unplanned trip to the office.

I get that Rogers is a telco and wants to use their own services, but it is frankly nuts that they don't have redundancy for their internal corporate network.
 
Jonny5 said:
Something interesting I hadn't considered noted in a Reddit forum by a person who claims to know an insider is that there was mass confusion for the first few hours as many key Rogers executives and staff could not be reached because, naturally, they all use Rogers phones and internet.
Click to expand...
I actually think that's funny in a perverse sort of way.
afransen said:
This is kind of absurd. My employer uses redundant data connections at all major premises. We go so far as ensuring the connections enter the building at different places, so one guy with a backhoe can't cut both lines. We were ticking along fine on Friday beside the fact that 25% of our WFH workforce had to make an unplanned trip to the office.

I get that Rogers is a telco and wants to use their own services, but it is frankly nuts that they don't have redundancy for their internal corporate network.
Click to expand...
Good disaster management plan by your employer, but it's easier for large organizations. Some, particularly on another forum I follow, are not a fan the suggestion that government regulation is needed to ensure network strength and redundancy, saying it should be up to individuals. It's a lot tougher for a small business to afford or justify two network connections 'just in case'. Of course, they could always revert to cash.
 
You must log in or register to post here.

Back
Top