Troubleshoot Yourself Or Have Someone Else?

[Kelly A]

In another thread a poster brought up something that over the past ten years or so I've noticed has become a thing with many 'Broadcast Engineer's' that involve responsibilities of the job:
During a prior gig, and even today, there were several very seasoned union engineers that worked for me. We had this annoying problem with slave clocks in one particular TV studio that would randomly 'glitch' for a second or two. It happened enough that the talent was frequently complaining. I had tasked at least three of my 'engineers' to fix the problem. After several weeks of asking them what the status of the troubleshooting was; one of the seasoned 'engineer's' said they had switched to the backup master clock system, which of course didn't fix the problem. Their next chosen step was to order-up a loaner master clock, then ship ours back to the factory for repairs. At that point, I'd had enough. I gathered the three of them to discuss what steps they'd taken to troubleshoot the problem. The answer was essentially; call the manufacturer for support, and look for failed equipment to ship back to the manufacturer. Of course to me, that doesn't count as troubleshooting. We then went then went through some simple assumptions: 'Would a problem with the building master clock system only effect one studio?' (Following blank stares of 'gulp-uh-oh').
Next I went to the tool storage area and pulled out a dusty oscilloscope. We walked down to the studio with the clock problem and I connected the oscilloscope to one of the prompter slave clock inputs. We waited about ten minutes watching a solid 10Mhz waveform on the scope until a off the screen square wave glitch at the same time the other clocks went to 8888. 'Gentlemen, we have an electrical-something near the SMPTE clock feed that when switching, is causing an inductive pulse into the time code wire'. My three engineers were staring at their shoes. Next I started tracing the cable path down the hall and found it laying against an electrical box that contained electrical contactor's used for starting air conditioning fans. Moved the cable away from the box, reconnected the oscilloscope, and no glitching clocks.
I didn't say it out loud, but in my head I was asking: 'What the Hell do we pay you guy's for??'
Still find myself asking the same sort of questions today: 'Did you download system logs to check for anything unusual during a particular time?' Did you get out a volt ohm meter to check for opens or shorts? Did you get out or download a schematic and confirm power supply voltages?
It seems to me it's time for some self-reflection: If you mainly rely on others to troubleshoot or repair the equipment in a facility, can you be considered an engineer? Or are have you become a shipping and receiving employee?

 

[Harvey_Dogg]

Those of us who make repairs or troubleshoot are all old dogs pf a dying breed. We now live in a "just throw it away" world.

 

[Greg Strickland]

Kelly A- That's why you were/are the chief/senior engineer or engineering manager. Presumably the engineers who worked from you learned something as well. In that facility you needed them too, correct?

On the troubleshooting part- a facility like that probably had the gear, shop and staff to troubleshoot. The part about using logic and common sense to isolate a problem is skill set. Some may have it naturally, some may learn it, and some may not be destined for this occupation.

The other engineers I work with are all very good at what they do. Ideas and projects are discussed, good ideas are improved into excellent ideas, errors are caught and solved, and we are all better for it.

 

[PTBoardOp94]

In order to troubleshoot a piece of equipment, first you need to have some understanding of how the equipment works.

If the manufacturer doesn't provide any "theory of operation" or "troubleshooting" guidance in the manual, I'm not sure where you'd even begin.

Obviously, a broadcast engineer should have experience with most of the equipment in a broadcast facility. But not necessarily with every piece of equipment, or with every fault.

So if this clock didn't have reasonable guidance in the manufacturer documentation, and the junior engineers had not seen a problem like this, I sympathize with them. Starting problems as a reverse engineering project on a black box sucks.

 

[Greg Strickland]

PTBoardOp94- In Kelly A's situation making the correct determination did not require technical skill. Nothing to do with the manufacturer or the documentation.

It was a task of logic. If one light in the house is flickering but the other lights are fine, do you call the electric utility to report an outage?

Of course, you might make an emotional video of your experience dealing with it, the changes you went through and post it on social media. 😉

 

[Kelly A]

Those of us who make repairs or troubleshoot are all old dogs pf a dying breed. We now live in a "just throw it away" world.

And to me, the throwaway world excuse is a cop out that I've heard all too often. Even systems that ultimately require entire motherboard replacement, still require troubleshooting to determine what the fault is. Even worse, is throwing parts at the problem, hoping the device will eventually work as expected. Personally, I think it boils down to the throwaway line of thinking as a reason to be lazy. The problem is; technical folks in the broadcast industry are a dying breed because many have allowed themselves to become lazy shipping clerks instead of keeping up on the technology that they're paid to install and maintain.

 

[PTBoardOp94]

PTBoardOp94- In Kelly A's situation making the correct determination did not require technical skill. Nothing to do with the manufacturer or the documentation.

It was a task of logic. If one light in the house is flickering but the other lights are fine, do you call the electric utility to report an outage?

It was a task of logic, but only if you know how the equipment works well enough to apply a logical analysis.

If the equipment's operation is unknown, a "task of logic" is really just a guessing game.

 

[Kelly A]

Obviously, a broadcast engineer should have experience with most of the equipment in a broadcast facility. But not necessarily with every piece of equipment, or with every fault.

I'd say first they need to understand simple troubleshooting techniques. And when I say simple: Step 1. Has anyone changed anything associated with the piece of equipment or system lately? That includes software updates, wiring changes, additions. If not, move to Step 2.
Step 2. When did this problem start happening? Is the problem unique to a particular user or shift? Goes to potential user training or user error. Many times you need to ask the witness several times and several different ways to get a correct description of the problem.
Step 3. If the device is completely inoperative, does it power-up? If not, you may have to get a screwdriver and check fuses plus power supply voltages with a simple, inexpensive volt ohmmeter. If, like the slave clocks in my example; the problem was intermittent but happened to all the slave clocks at the same time in that studio, what is the common thread that all the slave clocks share?
It's really Troubleshooting 101. You start at the far end of a problem and work through until the problem is found. If one doesn't have the patience to do the basics, it's easy to argue they're in the wrong business.

So if this clock didn't have reasonable guidance in the manufacturer documentation, and the junior engineers had not seen a problem like this, I sympathize with them. Starting problems as a reverse engineering project on a black box sucks.

In my example, these aren't junior engineers. These are all men at least in their 50's. I wouldn't be as concerned if they were inexperienced, and I could help them potentially learn. But where I work(ed) was a large network with union-represented professionals, who were well compensated accordingly. If you're paying people that much, this isn't a vocational training center.

In my example, there were no black boxes involved. The problem was isolated to one studio only, yet the master clock system worked for three of the four TV studios. In other words, the issue was isolated to one geographic space and the slave clocks in that space. The common thread? All the clocks in that studio were fed from a single wire that ran 100' down the hall to a distribution amp located in the studio.

 

[Kelly A]

Kelly A- That's why you were/are the chief/senior engineer or engineering manager. Presumably the engineers who worked from you learned something as well. In that facility you needed them too, correct?

The thing is; when one has that amount of tenure and experience, is union represented, and considered a professional in their field, they shouldn't need training to know the basics. Sure, I needed them to do their jobs. As a manager, I do mine.

On the troubleshooting part- a facility like that probably had the gear, shop and staff to troubleshoot. The part about using logic and common sense to isolate a problem is skill set. Some may have it naturally, some may learn it, and some may not be destined for this occupation.

I agree, but in my example, and even what I deal with today, most don't bother getting out a simple volt ohmmeter, let alone an oscilloscope. The first step is: call the manufacturer. Second step is: see if there are loaners available. Third step is: ship the device for repair.

 

[secondchoice]

I get your disappointment. My question why was the electrical contractor allowed to put AC circuits in a data cable rack? Didn't somebody check his (her) work and see data and AC cables together? Were the 3 (techs) ever informed that someone was messing with "their" stuff. When I work for Lucent, the power was usually in another cable rack and had a metal shielding if it was AC. To be honest, installed wiring going south is rare might have fooled me unless I knew someone had be working in the cable rack.

 

[Kelly A]

I get your disappointment. My question why was the electrical contractor allowed to put AC circuits in a data cable rack?

The other way around. The SMPTE time code cable to that studio ran above a dropped ceiling and over some HVAC ducting, including an electrical contactor which started a fan in that duct. Once moved away from the box with the contactor, the clocks stopped glitching.

 

[secondchoice]

Sorry I just assumed an operation that can afford had 3 techs would have Data cable racks or trays to keep the wiring organized. Had the clock cable been in communications / data wiring tray there would not have been a problem. Loose cables look bad too. I would suggest every opportunity (remodel, studio rebuild, new equipment etc.) have dedicated areas or routes for AC and Data to keep them separated physically if you can't budget racks or trays.

 

[Mikey Radio]

Sorry I just assumed an operation that can afford had 3 techs would have Data cable racks or trays to keep the wiring organized. Had the clock cable been in communications / data wiring tray there would not have been a problem. Loose cables look bad too. I would suggest every opportunity (remodel, studio rebuild, new equipment etc.) have dedicated areas or routes for AC and Data to keep them separated physically if you can't budget racks or trays.

I still remember watching a "tech" install a VERY costly 20,000 lumen projector and some related electronics, mounted atop a decorative enclosure that was designed to raise the projector to the required height, and finished so it would blend in with the venue. He hooked everything up, tested it, the image was bright and incredibly clear. Then, rather than cutting the extra length off the power and signal cables and terminating them properly, then dressing the cables as a professional would be expected to do, he took several meters of extra power and signal cables from the various gear, coiled it all up together, and dumped it into the bottom of the enclosure.

Needless to say, within a day or 2 the image on the screen was crap and they brought in another tech to finish the install properly.

 

[Kelly A]

Sorry I just assumed an operation that can afford had 3 techs would have Data cable racks or trays to keep the wiring organized. Had the clock cable been in communications / data wiring tray there would not have been a problem. Loose cables look bad too. I would suggest every opportunity (remodel, studio rebuild, new equipment etc.) have dedicated areas or routes for AC and Data to keep them separated physically if you can't budget racks or trays.

But that's the thing: Most of your average run of the mill "broadcast engineers" wouldn't have even thought running a low voltage data source near an electrical relay/contactor would be a problem. Has nothing to do with budget, but more with competence and technique.

 

[OLDRFGUY]

But we also have a management failure here. Guys you tasked with finding and fixing the problem didn't. Whether they didn't know how, whether they didn't care, isn't the point.

When the manager can't get results out of his/her subordinates, they are not managing.

When the manager does the task they delegated to someone else, they are not managing.

MANAGEMENT BY DEFINITION IS GETTING THINGS DONE THRU OTHERS. I learned this the first day of management school.

I have seen a lot of this in broadcasting. An engineer will get promoted and will have no management skills and so comedies like the one described then ensue. Of course, broadcasting - particularly radio - is endemic with bad and mis - management.
And I have worked for guys who were fine engineers who were then promoted into management and were TERRIBLE.
Of course, sales people are usually promoted into management in radio, and they seldom have the necessary skillset either. Indeed in most radio stations the only person with any real management skill is the business or HR person.

Of course, to be an engineer requires a skill set and technical experience. Management is a profession with a different skill set. An engineering manager must have BOTH. One can earn a master's degree in management - there is a LOT of information there.

By the way, one of the FEW really good engineering MANAGERS I ever worked for had a very clever interview technique for hiring people: He would ask them if they had taken foreign language courses. In my interview with him when he asked me, I replied that I had several years of Latin. He immediately hired me without asking any further "technical" background questions. He explained himself: Latin is a highly structured and formal language. It forces one to think LOGICALLY. And a major part of any troubleshooting or design procedure is LOGIC. We got on fine.

I would enquire of the 'manager' above who had the problem getting the subordinates to troubleshoot: Who hired these people?
What was the process? Another problem in radio and TeeVee is engineering managers who hire the wrong folx because they don't know how to hire (again, a management failure). But then they make things worse by not correcting the problems by either:

1) Getting the person up to speed technically with training and mentoring OR

2) If that doesn't work for that individual, cutting them loose and moving on.

Too many broadcast managers can't hire, and when they do hire someone, if that person doesn't work out, they won't cut 'em loose and move on because they fear it'll make 'em look bad. This is also something you learn in management school.

 

[frankberry]

We had a situation at a TV station (which I won't identify) where the Chief Engineer planned a new audio control room.
Rather than pulling the electrical power from the same distribution center as all other broadcast equipment, he chose to get the power from the "office" distribution center. The result was massive hum problems caused by ground loops. Eventually, I moved the source of the power to the "broadcast" distribution center and the issues disappeared.

 

[Kelly A]

But we also have a management failure here. Guys you tasked with finding and fixing the problem didn't. Whether they didn't know how, whether they didn't care, isn't the point.

When the manager can't get results out of his/her subordinates, they are not managing.

When the manager does the task they delegated to someone else, they are not managing.

MANAGEMENT BY DEFINITION IS GETTING THINGS DONE THRU OTHERS. I learned this the first day of management school.

I have seen a lot of this in broadcasting. An engineer will get promoted and will have no management skills and so comedies like the one described then ensue. Of course, broadcasting - particularly radio - is endemic with bad and mis - management.
And I have worked for guys who were fine engineers who were then promoted into management and were TERRIBLE.
Of course, sales people are usually promoted into management in radio, and they seldom have the necessary skillset either. Indeed in most radio stations the only person with any real management skill is the business or HR person.

I don't understand your point. So are you saying that if an 'engineer' makes no attempt to troubleshoot a problem, instead shipping the device to the manufacturer, or contractor for repair, that it's a management problem?

Of course, to be an engineer requires a skill set and technical experience. Management is a profession with a different skill set. An engineering manager must have BOTH. One can earn a master's degree in management - there is a LOT of information there.

But, as in my case, I've been both. In my example; there shouldn't be the need for me to show seasoned engineers how to connect and use an oscilloscope to troubleshoot a problem. And even after seeing a result, none of them even put any thought into what could be causing the glitch seen on the scope.

I would enquire of the 'manager' above who had the problem getting the subordinates to troubleshoot: Who hired these people?
What was the process? Another problem in radio and TeeVee is engineering managers who hire the wrong folx because they don't know how to hire (again, a management failure). But then they make things worse by not correcting the problems by either:

Again, these are seasoned, union-represented engineers. Blaming management for incompetence or lazyness seems misplaced.

1) Getting the person up to speed technically with training and mentoring

So seasoned 50+ year old union represented engineers need to be trained on how to use an oscilloscope?

2) If that doesn't work for that individual, cutting them loose and moving on.

That seems easy, but do you know how hard it is to find competent replacements?

 

[OLDRFGUY]

I agree a seasoned engineer should know how to use a scope. (My 7 year old granddaughter plays with a TEK 2235 and a microphone.) This problem seemed obvious to me to be a communication glitch of some sort and not a master clock issue.

Maybe firing somebody over this sort of think is overkill. But it IS a managment response. So is offering training. So is a conversation about the troubleshooting process. Other responses come to mind. It's the MANAGER's call on how to respond. BUT THAT'S MANAGEMENT.
As is hiring new folx.

And: Any problem in a department with a manager is BY DEFINITION the manager's problem. That's part of management. And in theory that's why you get paid more to manage. As I pointed out, MANY radio stations and radio companies are BADLY MANAGED.
I was not trying to criticize the individual who made the post, but rather the situation which may or may not have really been under their control. But if you got the title on the door, you carry the responsibility.

In my experience (just my own personal view) when I manage people I use what I learned in school about management. And sometimes it sux. That didn't work very well at a couple of radio companies, so I went on down the road after a while. I ain't bankrupt. They have been. Sure, that's not the only reason, but it was a factor.