Debugging intermittent Comcast, part 1: Scenario-setting

Inconsistency is a fundamental bane of troubleshooting. So it goes at work…and also at home.

At some point(s) in your engineering career, have you ever had the “pleasure” of inheriting the ongoing development and maintenance for a poorly-at-best documented project whose original leader was no longer available for advisement? Me too; I’ve dealt with handoff aftereffects for more than a dozen years so far with no definitive end in sight, albeit with increasing clarity as time passes. But in the case study described today, my project is personal, not professional.

When my then-fiancée (now-wife) and I bought the home we now live in, the previous owner had recently passed away and the original builders/owners were also no longer available for consultation (the husband is also deceased, I believe; I don’t have any contact-or-other info about his wife). So, aside from a pile of user manuals, receipts and other paperwork, I was pretty much on my own in sorting out the plumbing, mechanical, electrical, and other aspects of the residence.

An incomplete history

Judging from reputation per conversations with neighbors, not to mention the thick speaker wiring still installed in various rooms’ walls and baseboards, along with a nice set of B&W outdoor speakers, the prior owner was an enthusiastic audiophile. And judging from the sizeable living room plasma television, not to mention the projection system downstairs, also a videophile (as well as, I suspect, a “techie”).

So, it was no surprise to be told by the realtor that the home was already pre-wired for Comcast (a.k.a. Xfinity, the company’s brand for consumer products and services) broadband and television. That said, the only residence-specific detail I was aware of, and mentioned by the realtor only in passing, was that the prior owner “had another line of service installed” at some point. Hold that though.

When I moved in, I transferred my existing Comcast service from my previous Colorado residence, a process that to the best of my recollection was quick and painless. Coax cable runs on top of (plus shallowly under) the ground, as well as being attached to exterior walls at three of the four sides of the home, along with shielded, outdoor-rated Ethernet cable (all of which I’ve mentioned before).

The technician showed me which coax feed I should plug my cable modem into. And with that, for the next decade-plus and beyond relocating that strand’s interior end to my furnace room, which became the home’s networking nexus, and upgrading my service tier during the COVID-19 pandemic, ignorance was bliss.

I’ve played around a bit with MoCA in attempting to wired-extend the LAN to two guest bedrooms downstairs, versus running new Ethernet feeds or just relying on Wi-Fi. Beyond that, I kept the various wall-mounted coax connectors scattered around the house capped with terminators, since I was instead leveraging Windows Media Center over Ethernet to distribute TV service from a networked CableCard triple-tuner device and a Windows 7-based PC to various Xbox 360-based clients.

A missed-map mishap

Eventually, I expanded from a single-router setup to a multi-node mesh network, still using wired Ethernet for the backbone. All this time, the mysterious plastic box labeled “Comcast” in one corner of the property nagged at me.

As did the realtor’s cryptic “prior owner had another line of service installed” insight. But in the spirit of “if it works, don’t touch it,” I held my curiosity at bay…until last October, when necessity first forced my hand. In preparation for the nearby emergency access road construction that began shortly thereafter, the community’s water and sanitation service upgraded my neighborhood’s hydrants, so they could alternatively act as conveniently-located refill sites for water trucks. Unfortunately, in the process of replacing the hydrant across the street from me, they inadvertently also cut the thick coax feed(s) that Comcast-serviced my residence as well as that of my next-door neighbor.

After alerting Comcast to the fact that I was having a problem that was theirs, not mine, to solve (I can’t count the number of times I’ve heard the words “when’s the last time you replaced your cable modem” in the last near-year), they sent out a diagnostics tech, who confirmed the cut.

The community’s water and sanitation service washed their hands of the issue, since it turns out Comcast had seemingly neglected to add the wiring to publicly accessible maps post-initial installation. And it took another couple of dates to get a different Comcast technician out to spend nearly a day digging up the wiring, re-splicing it back together, and re-burying it.

Broadband and television service came back up immediately…but only for about 24 hours; then they went down again, this time only for a couple of hours (that day, at least). I’d never experienced Comcast flakiness before, so it wasn’t much of stretch to associate not only a correlation but a more definitive causation between what I was now experiencing and the day-prior repair attempt.

I’d already realized that a simple cable splice, versus a comprehensive full-run replacement, would result in at least a modicum of SNR loss. I deduced the incremental signal degradation had “pushed” our apparently already-marginal service “over a cliff”, at least periodically.

What symptoms did I notice whenever I was having Comcast service issues, aside from the fact that LAN devices would lose Internet connectivity and sometimes the router itself would also go down (since the Google Nest system is cloud-managed)? The second LED from the top of my Netgear CM1100 cable modem, referencing downstream connectivity, would perpetually blink, with those below it non-illuminated.

And speaking of the cable modem, sometimes even when I was online, the third LED down, referencing upstream connectivity, would still blink (versus its usual steady-illuminated state). That all said, unless broadband connectivity went down for an extended period, and sometimes even if it did, television service often still remained “up”.

Convincing Comcast of the validity of my ongoing issues—not to mention the company’s ongoing responsibility for solving them—was a different matter. The first tech that came out had given me his business card with an invitation to reach out if I continued having issues. Every time I texted him—I did strive to practice at least some restraint—he’d remote-check my modem and CableCard status and report back no issues logged on his end, including no T2, T3 or T4 timeouts.

There was also seasonal variability to the unreliability (assumed associations: ambient temperature and moisture). As autumn turned to winter, the outages thankfully became less frequent, at least for a while (again, hold that thought). So, I eventually gave up trying to get someone back out on Comcast’s dime and switched over to cellular hotspots whenever broadband started flaking out.

Learnings and lingering mysteries from this portion of the story:

  • The Comcast “tap” that services me and my next-door neighbor, it turns out, is at the very end of the line that runs through our neighborhood. Service-drop alerts sent to Comcast when customers’ cable modems, set-top boxes, and the like go offline only automatically result in a “truck roll” when a critical mass of customers are simultaneously affected. I don’t remember the exact number, but I think the tech said a half-dozen or thereabouts. Otherwise, Comcast logs the situation as a potential issue (similar to what happens if a customer reports an outage over the phone or in-app) but by default assumes a power outage or other unrelated-to-Comcast problem is the root cause of the service glitch. This is why I had to work so hard on my end to get the repair going in the first place.

  • You might have noticed in the earlier photos I took last fall that there are two thick coax feeds that had gotten severed. Particularly given that ours was the last line “tap”, I was baffled as to why there wasn’t just one cut cable. Descriptions sometimes refer to such lines as “loops” or “rings”, which initially explanation-contented me. But I more recently came up with what I think is a more likely answer. Again, hold that thought.

Another year, another cut cable

Fast-forward to June of this year. We were still getting occasional outages, but only a couple of times a month, most of them lasting only a few minutes each, and typically happening in the mid- to late-afternoons (again, with an ambient temperature association). But one day, broadband was up-and-down (lather, rinse and repeat) for several hours straight. I decided to take a break from work and go for a hike around the neighborhood, wherein I came across a bunch of Comcast trucks.

Apparently, an Xcel Energy boring machine digging a pathway under the road in preparation for running electrical cabling had once again sliced through a Comcast feed, this one servicing an entire street’s worth of customers next to ours. And although we didn’t solidly lose service where I lived, the technician I spoke with indicated that as part of the repair, they were re-tuning all the area’s line RF amplifiers, thereby explaining why our service was also up-and-down for so long that afternoon. Here’s what a few of the neighborhood RF amps look like:

Including this one near the cut location, which for some reason, ended up with its top still off (or is this a broader area-servicing fiber coax transceiver node, readers? Let me know in the comments!

And then there’s this unidentified (again, readers?) hunk of equipment right next to it, also surrounded by Comcast “flags”, in this case with its cover still intact albeit ajar.

Unfortunately, this re-tuning apparently further suppressed the ongoing SNR at my residence, because beginning the very next day I started experiencing more frequent outages again, this time roughly every other day. I eventually rang up Comcast again and scheduled a technician call for the next day. That evening, when service came back, Comcast called me back and tried its best to convince me to cancel my appointment, but I strongly declined the offer.

I’ll continue the story in part 2 of this series, scheduled for publication next week. Until then, I as-always welcome your thoughts in the comments!

—Brian Dipert is the associate editor, as well as a contributing editor, at EDN.

Related Content

The post Debugging intermittent Comcast, part 1: Scenario-setting appeared first on EDN.

MCUs strengthen security in IoT and control systems