Watchdog versus the truck

One of the first jobs I had, when I first got out of college, was for a company that designed and manufactured monitors for large trucks, the kind used in mining operations. This company was a small entity with around 25 employees and a couple of engineers. The main product was a monitor that sat on the dash of these trucks and watched over things like oil pressure, coolant temperature, and level, hydraulic pressure, etc. Variations of this monitor had 4, 5, or 6 indicator lights that lit if the monitored point went out of spec. An alarm also sounded, and the truck was shut down by a relay connection.

Do you have a memorable experience solving an engineering problem at work or in your spare time? Tell us your Tale

Another engineer and I decided it was time to bring this analog monitor into the microprocessor era. The idea was to monitor the same functions, but only have one indicator light with an LCD showing the issue. Along with the alarming function, we could also add more information on the LCD, like temperatures and pressure readings. It wasn’t a very complex design. At that time, micros didn’t typically have watchdog circuits, so we added one of the few external watchdogs available at the time. Our concern was that some transient would throw the micro off course, and we wanted the watchdog to reset the monitor in that case. The 24-V input voltage and all sensor inputs had some level of transient suppression (but, after several decades, I have forgotten what the circuit consisted of).

We completed a design, and it worked very well on the bench. Next, we hit it with various transients that we could generate. Not having access to any transient test equipment, we had to invent some methods to test this. Worse, we had no specs or general information on what kind of transients these trucks can experience, but we ourselves were satisfied that it was ready for a beta test.

After testing, we sent the monitor to a local mining company to have it installed on a working truck. We also sent a harness system with leads long enough to get to the sensors located around the truck. The company called us after they got the monitor mounted on the dash and all the sensors wired to the harness, so a visit was scheduled to test the monitor on a running truck.

I need to stop at this point to describe the truck. It was a 175-ton dump truck. There are bigger trucks now, but it was very large for the time. Picture tires 10 feet high and a 1600 HP diesel/electric generator system powering electric motors turning each wheel. The driver’s cab was about 18 feet off the ground and was reached using an attached ladder. The driver and the two of us climbed this ladder to begin the test.

To add to the pressure, there were a dozen or so managers and workers on the ground watching the tests. The mining company managers gave the go-ahead to begin. The driver started the truck (quit a roar)—the monitor fired up, and the LCD began showing the status of the monitored points… great!

After a few seconds, the truck shut down… not great. We looked at each other—a few seconds later, the truck roared alive again—monitor working—a couple more seconds, the truck shuts down—a few seconds later, the truck restarts, etc., etc., etc.

After a half dozen of these cycles, we told the driver to shut the truck down. We couldn’t tie up the million-dollar truck any longer, so we could not do any more investigation. We packed up our equipment and left with our heads down.

Back at the shop, we talked through what went on. We concluded that the monitor’s micro was disrupted by an unknown transient. The watchdog then discovered the code running amok and tripped the shutdown relay. The watchdog then rebooted the micro, resetting the relay, which allowed the truck to restart itself.

One of the major design issues was that some sensors required tens of feet of wire and were unshielded single leads (most sensors used chassis ground). These single wires (or should I call them antennas) could have been close to various relays and electric actuators on the truck, or worse yet, near the cabling used for the generator-to-motor system. Also, the watchdog, which did discover the issue, did not fulfill its function—it allowed the truck to restart.

This is where “Tales from the Cube” articles tell us how they fixed the issue by adding a larger resistor, fixing a bad solder joint, or reworking a reversed diode. In this tale, there is no happy ending. The boss didn’t want to continue with the project, and I’m sure the customer was not impressed. The project was cancelled. So why did I write this up?

I thought it was a good example of what can happen on engineering projects—sometimes they fail (moving from the lab to the field often exposes design issues), and sometimes you don’t get a chance to fix the design. Young engineers should understand this and not be disenchanted when it does. Don’t let it get you down. Remember, we learn a lot by failure.

Shortly after this project, we got the opportunity to design a full, micro-based dashboard for a large articulated truck. One of the things we designed was a fiber-optic cable data-transfer system to the back portion of the truck. This minimized the length of sensor wires, providing antennas for the transient. In this design, the system worked flawlessly.

Damian Bonicatto is a consulting engineer with decades of experience in embedded hardware, firmware, and system design. He holds over 30 patents.

Phoenix Bonicatto is a freelance writer.

Related Content

• When a ring isn’t really a ring
• In the days of old, when engineers were bold
• Software sings the cold-weather blues
• Going against the grain dust

The post Watchdog versus the truck appeared first on EDN.