Here’s the key takeaway from this post: use the scientific method to deal with bugs, rather than relying only on your intuition—never assume “ghosts” are influencing your code, infrastructure, libraries, or even making fun of you through the Internet.
Has this ever happened to you?
You’re a developer sitting at your desk. Suddenly, a fellow developer with less experience reaches you and almost shamefully admits:
“Hey, it’s the third day I’m working on this bug… it just doesn’t work, I don’t know why… I’ve tried everything! It seems like the code has a life of its own. Nothing makes it work. I’m almost giving up. Can you give me a hand?”
You say “of course!”, and you start a pair programming session. And then, just 10 minutes later, you figure out a path that should have been followed to solve the problem, which the other person never even considered. Why?
It’s because you’re a more experienced developer, right? Well, not always…
Do you believe ghosts are haunting your code?
Sometimes, being faster at solving coding problems isn’t due to experience. Often, it’s because slower devs have no method to find and crush bugs—they rely solely on intuition, expecting a solution to come out of the blue.
When a solution takes too long to magically emerge, the very next step for the slow dev is to blame intermittent and inexplicable problems with the code.
And, if they believe in the inexplicable, then I can tell you that they do believe in “ghosts in the code”.
It’s as if these ghosts are trying to fool them. And so they start to treat their code not as something purely deterministic, but rather as a haunted spell that produces random outcomes depending on spurious things.
Here are some hints that may indicate a concealed faith in “ghosts”. Beware of them whenever you’re talking to another programmer:
- Blaming other programmers whenever they have a bug they can’t fix.
- If the bug happens in production, blaming the infrastructure without exploring the bug’s probable causes.
- When you ask what they tried, they say: “I’ve tried everything”, and when you ask “what do you mean by everything?” They can only list one or two things.
- We can’t forget the classic, “I have nothing to do with the bug because it works in my local environment”.
- You start trying things the programmer has supposedly already tried and the results are different from what they said was happening (and you notice them becoming nervous during the exercise).
These are just a few symptoms, we could list dozens more. Many factors can contribute to having a superstitious mindset about computer code. I can assert that lack of experience is not the major factor. Rather, it’s the lack of a proper methodology for problem-solving.
We should do programming with a scientific approach
One can program in many different ways. However, the most effective way, especially when you face a persistent bug, is to follow a scientific approach. This is tricky sometimes, but it can be done by following scientific steps methodically, such as these:
- Understand the most you can about the problem; study not only the problem, but also its possible sources.
- Make one or more hypotheses about what’s happening.
- Investigate each hypothesis separately, taking notes about what you discover.
- Conjecture a solution and try it out.
- See if the results are as expected in all the cases; if not, figure out why.
- Use the acquired knowledge to repeat the process until the problem is solved. In each iteration, as more information is acquired, you’ll narrow down the branches of possible causes and get closer to the solution.
If you employ this process thoroughly, taking notes on each step and following the flow until the end, you’ll definitely get closer to the solution, because at each iteration, one of the two will happen:
- Wrong/bad conjectures about the error are eliminated, or
- You delve deeper in the investigative trail leading directly to the solution.
Most programmers I’ve known do not follow this process, unfortunately. For me, it seems like they believe ghosts eventually cause problems, and miracles will come from another dimension to fix them—as if they literally expect ideas to solve problems to magically appear in their minds.
I do admit that solutions can emerge intuitively sometimes, yet not without the loss of time and the gain of gray hair. So here’s my advice to you: don’t blame ghosts, heisenbugs, or any other fleeting entity—instead, remind yourself that programming is a science, and should be approached as such.
If you want to learn more about how to test hypotheses, check out Computer Science Unleashed. It will give you a pleasant introduction to data science in general, and also to computer networking and cybersecurity. These topics are super important for programmers, and this book makes them a breeze!
Two stories to illustrate my claim
Let me tell you two stories from my personal experience of dealing with programmers that believed in ghosts:
The ghost in the tests
A few weeks ago, a software engineer sent me a message with a confusing question. I couldn’t even understand what was being asked, so a meeting was called to clarify it. He was a bit desperate, saying lots about the tests, explaining it was working on his machine but not on Jenkins, and saying that “when I comment out the new tests I’m adding, everything works perfectly”.
I asked him to calm down while I studied the case. The first thing I did was to read the error log messages on Jenkins. Alas, they were saying a different thing: a configuration error… but I kept listening to him.
He said he had been working on that issue for the last 2-3 days, and he had already tried everything, but nothing worked. I said his hypothesis didn’t make sense—it wasn’t logical. I asked:
“Have you tried to run that code on Jenkins removing each one of your tests to figure out exactly which test causes the error?” (he was adding 4 tests to the codebase).
He said yes. But when I looked at Jenkins’s history, I couldn’t find such tries there! And I knew he wasn’t lying. This story supports my view that
when people believe in ghosts, they also start to think (really, they believe it) that they’ve tried things that they did not.
So instead of confronting him, I simply said, politely:
“Do you mind if we try that again together?”
So we removed each of the tests, until none was there, and… boom, the error was still there! Conclusion: the error wasn’t related to his code at all, but to a configuration problem with Jenkins, exactly as the logs were saying. He could have tried every single change in his code—nothing would have solved it. This is the big catch: when you believe in ghosts, they will always lead you in the wrong direction. So stop believing ghosts before you’re in too deep.
The ghost in the library
Recently, an old client called me late at night, saying they had a problem no developer could solve, and they were sure the problem was in some code I had written for them many years ago.
I tried to calm him down and explain that their ideas seemed inconsistent, but words weren’t enough. I ended up joining a live meeting with ten other developers who had been discussing the problem for over three hours. They said to me almost in unison:
“We think it is a problem with the library. The Python requests library is not working as expected.”
So I said: “Have you updated the library? Did you check if any bugs were recently found in the library?”
“No, and no.”
What? They were blaming a library that had not been updated, that was used by thousands of people worldwide… and yet they believed they were facing a problem nobody else had found? It couldn’t be. So I asked right away:
“Have you tried to do that call using curl instead of requests just to make sure it is in fact a problem with the requests library? Does that work as expected in all the contexts this code is being executed?”
Silence. After about 30 seconds (really, it took that long) one of them said:
“No, but it makes no sense to try that, because the problem’s in the library.” He then gave me some irrational reasons supporting his claim. Using the same principle from the last story, I simply said, politely:
“No problem, I understand you, but can we do this test just to make sure it’s indeed a problem with the requests library? I can help you easily do the test, just send me the full request and I’ll translate it to curl here in a minute…”
“Well, it makes no sense to test that, but OK, let’s do it!”—said one of them.
curl has an advantage: it can easily be run by different people in different contexts. So 15 minutes after I had joined the meeting, boom… we ran the request with curl, and yet the same problem happened! Meaning, the problem wasn’t the requests library.
They were chasing a ghost and, of course, ghosts don’t exist. They lost days investigating a hypothesis without conducting even a basic test to check if the hypothesis had any merit. This unscientific approach only lead them to needlessly blame others while getting no closer to solving the bug.
Had they used the scientific method of making hypotheses, testing them, and eliminating bad conjectures, they’d have progressed as fast as I did when I joined their meeting.
I’m confident I came out on top in these stories not because I’m special, or because I have superpowers. My skills as a developer are just above average. Instead, what made me find the problem faster was my scientific problem-solving mindset. All programmers need to have this type of mindset if they want to excel at their jobs.
What about your own experiences?
Have you been through situations like those? Did you relate to the ghost believers? Tell me your story in the comments, I’d love to learn from you! ❤️
I can’t tell you how I was unconsciously nodding while reading each line and every line of this blog post. Exactly those kind of situations I was pretty much experiencing every day in my work for the past years and it sometimes got rather hard for me not mistake this kind behavior as laziness.
I indeed believe that having this kind of thinking process while approaching problems does make a huge difference, so learning it should be a top priority for everyone. Now I got a reference to share in such occasions ;)
Thank you for sharing the article! Indeed, I had the same sensation when I was reading it for the first time myself :)
That’s good to hear this post has reverberated in you like that.
I think that, as you said, we need to avoid mistaking it for laziness and try to “teach” the thinking process to the guys who work with us, that’s the only way to prevent those situations from happening in the future again.
I’m happy you like it and that you are going to share it in the future.
Thanks!
It is embarrassing to say that in the dead of night, when I have been tirelessly working away for hours on end, I often find myself believing in such ghosts. Only to find the next morning that the only thing causing my demise was myself.