• Get involved.
    We want your input!
    Apply for Membership and join the conversations about everything related to broadcasting.

    After we receive your registration, a moderator will review it. After your registration is approved, you will be permitted to post.
    If you use a disposable or false email address, your registration will be rejected.

    After your membership is approved, please take a minute to tell us a little bit about yourself.
    https://www.radiodiscussions.com/forums/introduce-yourself.1088/

    Thanks in advance and have fun!
    RadioDiscussions Administrators

KCBS Tech Trouble

michael hagerty

michael hagerty

Airing the CBS News 24-7 feed on AM and FM. Streaming is a loop citing “network issues”. Alice (the only other station in the cluster I can receive in Folsom) is on the air. Sounds like the KCBS studios had a massive IT failure. Happened to us ten or so years ago at KFBK.
 
Last edited:
Audacy's new Infinity Network distributes CBS News Radio. Don't know who distributes 24/7. They had a big snowstorm in NYC, so that may be part of the problem. Or it could be an uplink issue.
 
Update: 1:30 pm. KCBS still not back on AM, FM or streaming. The CBS News 24-7 feed continues to air raw, while the stream continues to run the disclaimer loop ("Our local studios are experiencing a network outage affecting the availability of this stream. We apologize for the inconvenience and appreciate your patience as we work to resolve this issue as soon as possible. Thank you.").

Nothing on KCBS' social media as yet regarding what the problem is.

 
Back in the bad old analog days, didn't most of the O & O's have "backup" equipment and someone who knew how to turn it on? I get not everybody has a spare 50kw AM transmitter but every time something really bad happened to the control board or room at every station I worked at last century you could go to the patch panel and use the production studio.

Now days with just about everything digital except for the transmitter output one would think they would have "copies" of everything on a couple of PCs just in case the servers have issues.
 
michael hagerty said:
Airing the CBS News 24-7 feed on AM and FM. Streaming is a loop citing “network issues”. Alice (the only other station in the cluster I can receive in Folsom) is on the air. Sounds like the KCBS studios had a massive IT failure. Happened to us ten or so years ago at KFBK.
Click to expand...
It’s interesting. In the digital world, stations can be brought to its knees on a failure like that. I guess patch cords and traditional boards are gone.
 
Michael Rivers Kramer said:
It’s interesting. In the digital world, stations can be brought to its knees on a failure like that. I guess patch cords and traditional boards are gone.
Click to expand...
In the digital world there are certain "single points of failure". A core router crash is a frequently cited one, and not just in broadcasting. That's why ISPs often have redundancy built in, such as a pair of core routers sharing the load, but with either one having enough spare capacity to assume the full load if either glitches. In the same way that many stations keep an auxiliary transmitter in hot standby or ready to fire up on a moment's notice. Redundant UPS's is another example.
 
So what I've been told second-hand from a KCBS employee is that there was construction on one of the upper floors of 88 Kearney and it hit something that killed power in the building. KCBS went to generators, but the generators stopped at some point.

I'm asking for more details, since Alice @ 97.3 stayed on the air. Maybe the individual stations in the cluster are on independent generators---or KCBS doesn't share their generators?

@Weiserguy, did they explain it at all when they came back?
 
Disaster recovery isn't as simple as analog/patch bays good, digital/servers/switches bad.

One thing that's become easier with today's technology is to be able to recover from something like the power issue that appears to have taken the Audacy SF studios off line today.

For example: the automation system I represent, Myriad, offers a configuration that replicates an on-premise automation system with an identical automation system running in the cloud, whether at an off-site location or on a cloud platform.

Because our system integrates playout, scheduling and (if desired) news, everything constantly stays updated between local and cloud. Power goes down? If you have a laptop with a USB mic and the batteries are good, you could bring up the cloud version in a browser window and keep right on rolling, as long as you have the ability in your system to switch that cloud feed to everywhere it has to go (in this case, to the AM and FM transmitter sites and the stream and apps).

Or someone could take control from a sister station (KNX?) or from a home studio - as long as you think it through ahead of time and make plans to allow that to happen.

The cloud isn't a perfect answer, obviously - you're still at the mercy of Internet connections that can be iffy or gone if there's a massive natural disaster. If you're really serious about disaster recovery, you can set up all kinds of backup paths, but that takes lots of planning and money.

It's interesting to note that KCBS has a full backup studio and small newsroom at the AM site in Novato, but of course that's only of use once news people can get all the way up to Novato, and that appears not to have been invoked today.

And of course every disaster recovery plan is only as useful as the training that staffers get for how to use the tools that are provided. The station where I worked for 20 years built out a DR newsroom/studio at one of its transmitter sites a few years ago, but I don't think any of the staff there now even knows where that site is, how to access it in an emergency, how to switch it on to the air or how to use what's there if they get there.
 
Much smaller, simpler operation but there is a country classic station in KY that I fixed up a laptop with Zara on it that the owner can take to the transmitter site and he has 2 days of music.

Back to Michael Post. The first week I was at KIUL they were getting ready to spend all kinds of money fixing the buried tank for the backup generator that they thought had some water in it. If they had to dig it up there would have been the soil samples and extra cost to haul off the contaminated soil plus inspections. They got lucky because after some kind of test is was determined that the diesel fuel had "gelled". The old generator did really work because one Saturday Morning the lights flashed twice and it came on for 15 minutes.

Back in 2009 the Cumulus stations in Macon GA were off for an extended time due to a "breaker failure" that was difficult to replace.
 
fybush said:
And of course every disaster recovery plan is only as useful as the training that staffers get for how to use the tools that are provided. The station where I worked for 20 years built out a DR newsroom/studio at one of its transmitter sites a few years ago, but I don't think any of the staff there now even knows where that site is, how to access it in an emergency, how to switch it on to the air or how to use what's there if they get there.
Click to expand...

And there's the example of short-sightedness taking down even good contingency planning.

Somehow, I am not surprised at what Scott describes. I once had a nice set up at a music-formatted FM station that I did middays at and doubled as the chief operator. If the program feed was lost, there was a CD pre-loaded in a player at the transmitter site that would auto-start -- not all that difficult to engineer, actually -- which could play in repeat mode until someone could get there and resume some semblance of regular programming, with an inexpensive club mixer and a couple of CD players. (The basic on-air library was duplicated onto a bunch of CD-R discs and I would update the current and recurrent disc on my weekly visits to the transmitter site for FCC-required inspections.)

Occasionally, I would force the loss of the program feed to check if the backup would start, and it always did.

A few years after I left, I had occasion to visit my former station. No one knew the backup was there. The weekly inspection requirement had ended and the chief operator was a contract employee, so I didn't get to ask if the equipment was still there. I asked the new GM what the backup plan was if the studios went offline, and he had no clue.

Doesn't do you any good to have backup facilities if no one knows how to activate them.
 
What exactly were they airing? Because if you listen to the CBS News Radio station on the Audacy app it is now the audio of the CBS News 24-7 tv streaming channel.
 
Weiserguy said:
In the digital world there are certain "single points of failure". A core router crash is a frequently cited one, and not just in broadcasting. That's why ISPs often have redundancy built in, such as a pair of core routers sharing the load, but with either one having enough spare capacity to assume the full load if either glitches. In the same way that many stations keep an auxiliary transmitter in hot standby or ready to fire up on a moment's notice. Redundant UPS's is another example.
Click to expand...
First, I'm surprised that the longstanding backup arrangement with KNX wasn't put into effect. This suggests a loss of connectivity.

Broadcasters need to think of their systems that directly affect broadcast the way that utilities think of their operational systems: behind at least a second layer of firewalls within the main corporate network, maintaining full redundancy of systems, prioritizing availability, and strictly limiting who has administrative access. (Cybersecurity and business continuity go hand in hand.)
michael hagerty said:
So what I've been told second-hand from a KCBS employee is that there was construction on one of the upper floors of 88 Kearney and it hit something that killed power in the building. KCBS went to generators, but the generators stopped at some point.
Click to expand...
An interesting data point is that 88 Kearny is across Post from the underground PG&E transformer that blew up at least ten years ago (I don't remember exactly when, but it was after my office moved out of 120 Kearny). There were a couple of injuries to people walking by on the sidewalk, injuries that resulted from that explosion.

TheBigA said:
BTW They haven't updated their social media sites in several days.
Click to expand...
I noticed that, too. Staffing issue, perhaps?
 
Mark Roberts said:
First, I'm surprised that the longstanding backup arrangement with KNX wasn't put into effect. This suggests a loss of connectivity.
Click to expand...

Maybe, when they started feeding KNX overnights with KCBS' programming, the backup configuration got screwed up and they were unable to activate it?
 
michael hagerty said:
So what I've been told second-hand from a KCBS employee is that there was construction on one of the upper floors of 88 Kearney and it hit something that killed power in the building. KCBS went to generators, but the generators stopped at some point.

I'm asking for more details, since Alice @ 97.3 stayed on the air. Maybe the individual stations in the cluster are on independent generators---or KCBS doesn't share their generators?

@Weiserguy, did they explain it at all when they came back?
Click to expand...
No, she was very vague. Just something like "a technical problem".
 
K.M. Richards said:
Maybe, when they started feeding KNX overnights with KCBS' programming, the backup configuration got screwed up and they were unable to activate it?
Click to expand...
KCBS had no power. They’d have to get KNX’s feed to the transmitter in Novato. It appears the emergency feed at the transmitter is the CBS News 24-7 feed.

If I were KCBS, I’d change that tomorrow, so that if the studio fails, KNX airs, and KNX can then mention as often as is prudent that they are being heard in both cities because of whatever issue in SF.
 
michael hagerty said:
If I were KCBS, I’d change that tomorrow, so that if the studio fails, KNX airs, and KNX can then mention as often as is prudent that they are being heard in both cities because of whatever issue in SF.
Click to expand...

Apparently, Mike, you and I are smarter than the management at Audacy.
 
michael hagerty said:
KCBS had no power. They’d have to get KNX’s feed to the transmitter in Novato.
Click to expand...
And to the FM transmitter west of Sausalito. So that's a challenge that didn't exist in the days when each station was on AM only.

michael hagerty said:
If I were KCBS, I’d change that tomorrow, so that if the studio fails, KNX airs, and KNX can then mention as often as is prudent that they are being heard in both cities because of whatever issue in SF.
Click to expand...

Which is how it used to work. There is an aircheck from August 26, 1981 where that's exactly what happened - the day before, construction workers drilled into a gas main at Battery and Sacramento, and a geyser of gas spewed into the air. One Embarcadero Center, where KCBS was at the time, was among the Financial District locations evacuated as a result. They didn't get back into the studio until 5 am on the 26th. Much more recently, KNX was a beneficiary of the backup arrangement when it had to evacuate its studios and simulcast KCBS. I can't readily remember or find the year when that happened, but it was in the 2010s, if I recall correctly.
 


Back
Top Bottom