• Get involved.
    We want your input!
    Apply for Membership and join the conversations about everything related to broadcasting.

    After we receive your registration, a moderator will review it. After your registration is approved, you will be permitted to post.
    If you use a disposable or false email address, your registration will be rejected.

    After your membership is approved, please take a minute to tell us a little bit about yourself.
    https://www.radiodiscussions.com/forums/introduce-yourself.1088/

    Thanks in advance and have fun!
    RadioDiscussions Administrators

Spotify Is Down

I just thought about that the other day. I noticed the local IHeart station ad's mostly promote only the Iheart app, and not the station itself. I feel like that is really shooting yourself in the foot.
 
These are some of the main reasons why I will always hang on to a huge music library, and to a lesser extent a decent library of Blu-ray and DVDs. It's also nice to physically have the media. Most of what I have will likely be usable longer than I will be alive! I have had very liitle experience with 'disc rot', be it CD, any movie disc formats, or computer discs. Maybe a handful (less than a dozen) in 20+ years.
 
I was wondering what was going on yesterday. I took a 2+ hour road trip yesterday, and was getting frustrated trying to listen to pull up one of my Spotify mixes in the car (via my phone and Bluetooth). Data displayed, but endless buffering. Switched to the Audacy app instead (WXRT Chicago Blues and Soul, which I deemed appropriate as I was driving through Illinois). No problems with it.
 
Each of these services likely has tons of servers in their data centers, storage arrays to hold all their music files, redundancy up the gazoo, but their single points of failure are likely to be in their data connections (fiber optic lines) to the world, the equipment they use to control those connections (routers, switches, load balancers, etc.), and the DNS listings that facilitate connecting into them from wherever a listener is. They're dead in the water if one or more of those goes offline, or gets corrupted, until the equipment can be repaired or replaced, or the "defective" DNS listing can get corrected and propagated out to the world. Unfortunately, those issues take time to diagnose.
 
Spotify certainly doesn't have a monopoly on the this. Outages happen to everyone, even radio stations when a transmitter goes down, or a whole cluster when the software goes down as we recently saw from iHeart's NYC Wheatstone crash.

I've personally experienced a Netflix outage, wireless carrier outages, ISP outages, have seen multiple websites and services get taken out by an AWS outage, and many others. Every single one of these incidents is an emergency of the highest priority for all affected, and the engineering teams rush to fix them ASAP.

Things break. It happens.
 
Weiserguy said:
Each of these services likely has tons of servers in their data centers, storage arrays to hold all their music files, redundancy up the gazoo, but their single points of failure are likely to be in their data connections (fiber optic lines) to the world, the equipment they use to control those connections (routers, switches, load balancers, etc.), and the DNS listings that facilitate connecting into them from wherever a listener is. They're dead in the water if one or more of those goes offline, or gets corrupted, until the equipment can be repaired or replaced, or the "defective" DNS listing can get corrected and propagated out to the world. Unfortunately, those issues take time to diagnose.
Click to expand...
Heck, even my lowly website server in Dallas has connections to two different main internet "lines". They have huge UPS arrays, generators and everything else to keep things running. In 5 years, I have lost less than 2 hours.

But that one loss was caused by a software update that did not install correctly and crashed everything. My option is to mirror the site at two server "farms" but that would increase my cost by around $4 thousand a year, which is more than I want to spend to prevent the loss of just a few hours in 5 years.

Using my example, Spotify and others like them must have decided how much to invest in redundancy, based on both lost revenue and damage to their image. At some point, like at radio stations, we say "it's not worth it".

For decades, most Los Angeles FMs that were on Mt Wilson had no auxiliary site elsewhere. They might have had an auxiliary transmitter or two, even an auxiliary antenna. But no back-up site. Then a forest fire nearly consumed the whole hill and everybody built backups as they realized that a major fire would involve several months of rebuild time... at best.
 
Last edited:
Saul the Iguana Killer said:
These are some of the main reasons why I will always hang on to a huge music library, and to a lesser extent a decent library of Blu-ray and DVDs. It's also nice to physically have the media. Most of what I have will likely be usable longer than I will be alive! I have had very liitle experience with 'disc rot', be it CD, any movie disc formats, or computer discs. Maybe a handful (less than a dozen) in 20+ years.
Click to expand...
Same here. I have a large music library of CDs, etc plus my own music server. When a friend asked me if I still had music, I said "Yuuup!" I don't use music streaming and own my own music source.
 
DavidEduardo said:
Heck, even my lowly website server in Dallas has connections to two different main internet "lines". They have huge UPS arrays, generators and everything else to keep things running. In 5 years, I have lost less than 2 hours.

But that one loss was caused by a software update that did not install correctly and crashed everything. My option is to mirror the site at two server "farms" but that would increase my cost by around $4 thousand a year, which is more than I want to spend to prevent the loss of just a few hours in 5 years.
Click to expand...
The term that applies here is recovery time objective, or RTO. It’s a calculation of how much time you can afford to be down and how long it will take to restore service. It’s basically a risk vs. benefit calculation, which is what you described above.


DavidEduardo said:
Using my example, Spotify and others like them must have decided how much to invest in redundancy, based on both lost revenue and damage to their image. At some point, like at radio stations, we say "it's not worth it".
Click to expand...

That said, start-ups are notorious for not making these calculations and just winging it and not even considering the need for RTOs and risk calculations. Not to get political here, but you see that same mindset in what Musk’s crew is doing with federal government computer systems. Their approach is all improvised and seat-of-the-pants, the antithesis of “waterfall”-style planning and execution, and all very much part of Silicon Valley start-up “culture”. No wonder so much of what it produces doesn’t work well.
 
Mark Roberts said:
The term that applies here is recovery time objective, or RTO. It’s a calculation of how much time you can afford to be down and how long it will take to restore service. It’s basically a risk vs. benefit calculation, which is what you described above.




That said, start-ups are notorious for not making these calculations and just winging it and not even considering the need for RTOs and risk calculations. Not to get political here, but you see that same mindset in what Musk’s crew is doing with federal government computer systems. Their approach is all improvised and seat-of-the-pants, the antithesis of “waterfall”-style planning and execution, and all very much part of Silicon Valley start-up “culture”. No wonder so much of what it produces doesn’t work well.
Click to expand...
People aren't likely to cancel a service like Spotify if they're down for a few hours, once. I didn't even notice, as the downtime fell in the middle of my workplace's Easter closure period, so I was at the coast doing things other than working and commuting and didn't have music on. (Spotify is European-based, and the long Easter break may have been a factor in the issues.)

If the service became unreliable, and it was down a couple of times a month, people might start to churn across to rivals, but a single period of downtime is generally forgotten as soon as the service comes back up. On the rare occasions my home internet goes kaput, it's frustrating but as soon as the link comes back up, I'm back doing whatever I was doing and I've forgotten about it.
 
DavidEduardo said:
Heck, even my lowly website server in Dallas has connections to two different main internet "lines". They have huge UPS arrays, generators and everything else to keep things running. In 5 years, I have lost less than 2 hours.
Click to expand...
I have no issue with the rest of what you wrote, David, but the above needs some further clarification. Your server sounds like it resides within a server farm, which is a data center that either houses servers that customers like you provide to them, or they lease you a server (or a fractional share of one, using virtualization hardware/software like VMware). A place like that probably does have multiple internet lines. The key consideration, though, is where do those lines go? A professionally-designed, professionally-run, commercial service has likely provisioned fiber optic lines from different providers, routing via different paths from different starting points (like a cable headend, or a Telco central office, or a CLEC's high speed data hub). So if one fails, the others are likely not affected. But individuals running a server or two out of their house or small office don't always realize this, so if they order a second line as a backup from the same supplier, and the routing is the same to the same supplier location, and they don't realize that if one fails, they both fail. (Perfect example: a backhoe is repairing a water main break, and cuts a fiber trunk line that contains both circuits. Nothing gets back online until the water main is first repaired, then the trunk needs to be repaired or replaced, fiber by fiber.)
 
Status
This thread has been closed due to inactivity. You can create a new thread to discuss this topic.


Back
Top Bottom