Troubleshooting often involves conflicting symptoms and scenarios

I’ve always regarded debugging and troubleshooting as the most challenging of all hands-on engineering skills. It’s not formally taught; it is usually learned through hands-on experience (often the hard way) and almost every case is different. And it’s a long list of why debugging and troubleshooting are often so difficult.

In some cases, there’s the “aha” moment when the problem is clearly identified and knocked down, but in many other cases, you are “pretty sure” you’ve got the problem but not completely so.

Note that I distinguish between debugging and troubleshooting. The former is when you are working on a breadboard or prototype that is not working and perhaps has never fully worked; it’s in the design phase. The latter is when a tested, solid product with some track record and field exposure misbehaves or fails in use. Each has its own starting points and constraints, but the terms are used interchangeably by many people.

Every engineer or technician has his or her own horror story of an especially challenging situation. It’s especially frustrating when there is no direct, consistent one-to-one link between observed symptoms and root cause(s). There are multiple cause/effect scenarios:

Clarity: The single-problem, single-effect situation—generally, the easiest to deal with.
Causality: A single problem with multiple causes, where one problem (often not directly visible) triggers a second, more visible one.
Correlation: Two apparent problems, with one common cause—or maybe the observed symptoms are unrelated? It’s also easy to have the assumption that correlation implies causality, but that is often not the case.
Coincidence: Two apparent problems that appear linked but really have no link at all.
Confusion: A problem with contradictory explanations, where the explanation addresses one aspect but does not explain the others.
Consistent: The problem is intermittent with no consistent set of circumstances that cause it to occur.

My recent dilemma

Whatever the cause(s) of faults, the most frustrating situation for engineers is where the problem is presumably fixed, but no clear cause (or causes) is found. This happened to me recently with my home heating system, which heats up water for domestic use and for radiator heating. It has one small pump sending heated water to a storage tank and a second small pump sending it to radiators; both pumps do not run at the same time.

One morning, I saw that we lost heat and hot water, so I checked the system (just four years old) and saw that the service-panel circuit breaker with a dedicated line had tripped.

A tripped breaker is generally bad news. My first thought was that perhaps there had been some AC-line glitch during the night, but all other sensitive systems in the house—PCs, network interfaces, and plug-in digital clocks—were fine. Perhaps some solar flare or cosmic particles had targeted just this one AC feed? Very unlikely. I reset the breaker and the system ran for about an hour, then the breaker tripped again.

I called the service team that had installed the system, they came over and they, too, were mystified. The small diagnostic panel display on the system said all was fine. They noted that my thermostat was a 50-year-old mechanical unit, similar to the classic 1953 round Honeywell unit, designed by Henry Dreyfus and now in the permanent display at Cooper Hewitt/Smithsonian Design Museum in New York (Figure 1). These two-wire units, with their bimetallic strip and glass-enclosed mercury-wetted switch, are extremely reliable; millions are still in use after many decades.

Figure 1 You have to start somewhere: The first step was to take out a possible but unlikely source of the problem. So, the mercury-wetted metallic-strip thermostat (above) similar to the classic Honeywell unit was replaced with a simple PRO1 T701 electronic equivalent (below). Sources: Cooper Hewitt Museum

While failure of these units is rare, technicians suggested replacing it “just in case.” I said, sure, “why not?” and replaced it with a simple, non-programmable, non-connected electronic unit that emulates the functions of the mechanical/mercury one.

But we knew that was very unlikely to be the actual problem, and the repair technicians could not envision any scenario where a thermostat—on 24-V AC loop with contact closure to energize a mechanical or solid-state relay to call for heat—could induce a circuit breaker to trip. Maybe the original thermostat contacts were “chattering” excessively, thus inducing the motor to cycle on and off rapidly? Even so, that shouldn’t trip a breaker.

Once again, the system ran for about an hour and then the breaker tripped. The techs spend some time adjusting the system’s hot water and heating water pumps; each has a small knob that selects among various operating modes.

Long-story short: the “fixed” system has been working fine for several weeks. But…and it’s a big “but” …they never did actually find a reason for the circuit-breaker tripping. Even if the pumps were not at their optimum settings, that should not cause an AC-line breaker to trip. And why would the system run for several years without a problem.

What does it all mean?

From an engineering perspective, that’s the most frustrating outcome. Now, even though the system is running, it still has me in that “somewhat worried” mental zone. A problem that should not have occurred did occur several times, but now it has gone away for no confirmed reason.

There’s not much that can be done to deal with non-reproducible problems such as this one. Do I need an AC-line monitor, as perhaps that’s the root cause? What sort of other long-term monitoring instrumentation is available for this heating system? How long would you have it “baby-sit” the system?

Perhaps there was an intermittent short circuit in the system’s internal AC wiring that caused the breaker to trip, and the act of opening the system enclosure and moving things around made the intermittent go away? We can only speculate.

Right now, I’m trying to put this frustrating dilemma out of my mind, but it’s not easy. Online troubleshooting guides are useless, as they have generic flowcharts asking, “Is the power on?” “Are the cables and connectors plugged in and solid?”

Perhaps I’ll instead re-read the excellent book, “DeBugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems” by David J. Agans (Figure 2). Although my ability and desire to poke, probe, and swap parts of a home heating system are close to zero.

Figure 2 This book on systematic debugging of electronic designs and products (and software) has many structured and relevant tactics for both beginners and experienced professionals. Source: Digital Library—Association for Computing Machinery

Or perhaps the system just wanted some personal, hands-on attention after four years of faithful service alone in the basement.

Have you ever had a frustrating failure where you poked, pushed, checked, measured, swapped parts, and did more, with the problem eventually going away—yet you really have no idea what the problem was? How did you handle it? Did you accept it and move on or pursue the mystery further?

Troubleshooting often involves conflicting symptoms and scenarios

Become a member

Become a member

Become a subscriber

Become a sponsor