TL;DR: Even if you're 100% sure you know what the problem you're trying to solve is, don't act like you're sure. Also, this story is mostly pictures, just read it.
Two small points to will help you understand this story.
- "Incident Response" is a little Slack bot we built to help our company if/when there is an incident to respond to. One of its functions is to contact people who aren't present. It will call/text every 4 minutes and alert the channel every 3rd time if the attempts are still ongoing
- The "Incident Response" bot is part of the larger "Fishbot" Slack bot that we have.
Ok, let's go.
2:07 PM March 2nd 2017
At this point: Hmm. That was weird. Oh well.
This doesn't make any sense, but it will probably stop
At least my co-workers have a sense of humor about it. I'll delete all the
Page records in the database, because the bot only sends messages if it sees one of those.
Well that didn't work, must be slack then.
Well it can't be Slack, it's happening every 12 minutes like you would expect it to. But I've deleted all the
Page records and that's the only way for this to happen.
Ooh! Good idea! That will fix it for sure!
I have tried everything. I know that it has to be a problem with the app on Heroku somehow
No really, I've tried everything, I am doomed to live the rest of my days forever haunted by the words "Continuing to try to contact"
I knew in the deepest parts of my heart that the problem had to be some sort of rogue version of the bot running on Heroku somewhere just stuck in a loop. If only I could do the right combination of deploying commented-out code or restarting the dynos. There was absolutely nothing else that could be the cause.
Except that there was.
Oh. My. God.
I was so sure I knew what the problem was. I didn't know how to solve that problem quite yet, but at least I definitely knew what it was.
I was wrong.
I spent an inordinate amount of time and effort working on perfectly good, valid solutions to a problem that didn't exist.
By telling myself that I knew beyond a shadow of a doubt what the problem was, I limited the possible solutions. Any possible solution I tried had to fit what I "knew" the problem was.
This was all for a very insignificant, simple issue. Imagine how much time, how many resources, how many companies are wasted trying to solve problems that don't exist.
If you get the problem wrong, it doesn't matter how hard you work, you will make no progress. You might even make things worse.