I just added a new item to my list of dumb things to check. (Maybe I should call it my "things to check when you think you've tried everything").
The new item is: * Is it the first of the month? Maybe there is a billing problem and something got shut off.
A few Sunday's ago my Verizon FIOS connection went dead. Not the entire connection, just the internet. The TV still worked (I don't have the phone service)
I did the usual debugging steps but nothing worked. Realizing it was the first of the month, I thought "hmm.... could this be a billing issue?"
I called tech support and entered my account number. After 6 different prompts (!) rather than sending me to tech support the computer voice said, "Hmmm... it appears your problem can't be solved by technical support. Please hold on for a moment." (yes, the synthesized voice actually said, "hmmm"). After a few rings I got a message, "Sorry, the finance department is closed. We re-open Monday at yadda yadda yadda."
August 1st, you see, is a Sunday. With no way to reach technical support (no matter what I tried I got redirected) I was sunk until Monday. Luckily I wasn't oncall.
On Monday I called again. Since the finance department is only open hours that I'm at work, I was hoping that their procedure wouldn't require me to have physical access to my stuff at home.
Finance said that there's nothing wrong with my account and my service should be fine. As a testament to this, they pointed out that my TV service works, so obviously I wasn't shut off.
She also pointed out that I have a credit of $41.32. That number sounded familiar. Ah yes, the prior month when there really was a billing problem that number came up. However, that got resolved.
What was the chance that when the problem was resolved last time there was still a flag sitting around somewhere that meant "disconnect this customer on the first of the next billing cycle". Such a bug wouldn't appear until the first of the next month, eh? Knowing that TV and ISP subsystems are probably entirely different provisioning systems that get fed from sources, how likely is it that they got out of sync? How likely is it that when the last billing problem got resolved the flag was flipped for TV service but the request to flip it for the ISP service got lost by a buggy batch job somewhere? (My hunch: someone forgot to include unittests for the weird case of a customer having FIOS with no phone service.)
I was about to give up when I realized that the phone system had transfered me to her. Thus, some database somewhere thought I had a payment problem on my account. I pointed this out to her and she couldn't deny my logic. I could hear her typing. Suddenly she said, "Oh, that shouldn't be.. um... how could that be?" I could hear her typing a bit more.
Then she said, "Ok, things are fixed here. Now I need to transfer you to tech support so they can make sure that things are activated." I thanked her profusely.
To make a long story short, the tech support people were confused, also claimed that my account was fine, and I had to explain that things really weren't fine.
Of course, since I was at work and not at home I couldn't reboot my router (their "go to" solution for most problems, and rightly so). However, I could point out that I couldn't ping my router. They could ping their end. They could reboot their end.
Using a little verbal jujitsu, I asked them if they could reboot my router remotely. They said they could even when the system was otherwise down. But wait... he coudn't access it through that out-of-band channel! Ah, this holds promise.
Verizon FIOS installs a little box in the basement that is essentially a UPS so that during a power outage you can still make phone calls. In that box is a bit of network equipment that can only be rebooted by removing the battery from the UPS or by having Verizon reboot it remotely. He suggested I pull out the battery when I get home and call back if that didn't fix it. I asked him to reboot remotely (I didn't want to have to call back and explain the entire situation to a new tech).
After the reboot, he could see my router, but he could also see that their DHCP server wasn't giving me an address. This, good friends, was another instance of "the flag" causing problems.
He had to tweek some stuff (I hope he meant re-pushing a database) and soon it was doing DHCP right. Moments later I could ping my home router.
Success!
End note: I'm writing this blog post on Aug 22 but I'm scheduling it to not appear until August 31. Since September 1st is a Wednesday the billing department will be open if "the flag" haunts me again... though from midnight to 9am I'll be dead in the water. I just tried calling tech support and was transfered to the finance department.
Since you were dead-in-the-water, with no way to reach technical assistance (since they kept re-routing you to a closed desk), were you able to get SLA credit for the outage?