Tuesday, September 6, 2016

Root cause analysis haters, obedience lovers and myopic players

Oh, throwing money and people at a problem doesn't solve it for good?
We've all been there and seen that. If you haven't yet, then sooner or later you'll find yourself(or someone else) in that position.
By the way, I get it that sometimes you just need more people to put in more working hours and get shit done. However, I want to talk about situations in which quantity doesn't help and when you should think different.

During my consulting career I've seen countless number of times managers putting more people to work on a specific problem because it was still there or coming back after a while(regression). So they usually fell on a terrific idea, that they need to hire more people to work on it. So they had and have told new-hires to do the same thing all over again, but they were so close-minded that they didn't even think that the problem may just not be there. It didn't come as a surprise, that after a few months when checking something else,  I've noticed that the problem was still recurring.

What were they doing wrong? They were just great in scratching the surface, but terrible at diving to the bottom of a stack. This is what happens when one is a finger pointing person who doesn't like root case analysis(RCA) and at the same time doesn't let other people do it.
How many times you were told by boss to do something and you did it just because of his authority, not because that it made any sense at all.
Sounds like a school right? Sure it does, it's the same generation of people.
This is how they were raised and that's the only thing they know. If you used to be told for over 18 years to follow orders and behave the same way others do, you'll have hard time being creative.
So it's not that I'm blaming individuals, I'm more blaming the system, but in a workplace you look at the person, and you see a person, not things which made him what he now is.

Why many people tend to not like RCA? Because it's hard. It requires to get out of comfort zone and face the problem on the ground which isn't in one's sweet spot. It requires to get hands dirty, it requires to have a decent background in many fields, but the toughest part is that it requires to be open minded.
Yes, technical work is hard, but allowing yourself to be creative if you're not used to it, is bloody uncomfortable. People like getting orders and executing on them, because it's easy when someone provides you a step by step guide on what to do. You just do it and feel comfortable, because you know you won't fail if you follow exactly the same things others do. Yeah, it means doing average, but we're all afraid of failing, so why take a risk right?

The other reason is that people are myopic and focus on short-term game. They will do something over and over again instead of focusing on long-term plan. Why? Some people just like repetitive work, so they are fine coming back to the same regression after each release of a product.
Some employees are leaving their job soon and they don't care. Others were put in the wrong position at the company and for some there's no excuse, there are simply stupid.
There is one more type of people which I've met and you probably have too - those who like regressions, because thanks to them they always have something they can work on to show off results and brag about how many problems they solved. And no one asks them if they solved problems for a long-term. "Data-driven" stats are looking good, so why bother?
This is ridiculous, there are many more problems out there to be solved, but some people just don't get it or don't want to get it.

At the end of the day, whatever the reason is, it's a waste of money and human potential. People doing dumb things are wasting organization's money, and myopic managers are wasting human potential.

If you're working with a manager whose excuse is always "we don't have time to figure out the details of this", and the problem keeps coming back, then here is what can work. Or it may not work, but will make him need to figure out different excuse and then you know if a problem is the problem, or that guy is the problem.
What works for some managers is showing them on real examples how much time they wasted fixing 20 exactly same bugs and how could they've saved all that time by spending just a few hours on introducing simple framework or dedicating one person to do this once for good. Show them the statistics, show them numbers and propose real life solution. They rarely deny it, because it would make them look silly that they refused data driven solution while they brag about stats and data all the time.
But be careful if you go there. If you proposed a solution then you need to execute and keep the promise.
If you tell you can do something, jump in and do it, because most likely you won't get second chance if you fail. Spending time doing something, you're spending company's money, so you need to be effective.

If the excuse is "this problem is so tough, we've tried to fix it, I had entire team to sit on it and it's impossible" then audit the process and a way in which you're approaching the problem.
Are you doing creative work and trying new approaches? If you hear a new approach coming from a back of the room, do you listen?
Keep trying to figure it out, because it's really rare to fall on unsolvable problem. Don't create a status quo around it just because "this happens in all companies I worked, so this is normal".
Maybe you just used to approach it in a wrong way all the time. Maybe the process is broken or might be that you don't have qualified people around you.
People thing is real. Some managers just don't have people capable of doing creative work. They hire masters of obedience - because they're easier to manage than creative troublemakers - to later cry about low quality of work they produce.

If you're sitting in management position and always demanding hot patches and quick fixes, don't expect things to be solved in long term and please don't blame devs.  This is something that needs to come from management to the bottom, not the other way around. Focus on "how can we ensure this doesn't happen again" instead of "just get this out of my sight for now". Let people be creative, tell them what's the problem but don't frame them with your way of thinking. Let others solve it, allow others to have different perspective than yours.
There is a great trick that works for human beings. Have you ever heard or seen that if someone didn't know something is impossible, he actually made it possible? Don't trap people in a stiff mindset that something is just is the way it is. It maybe is for you, for someone else it may be not.
When I was starting my security bug hunting journey years ago, people used to tell me "those big brand have top security experts, there is no way you can bring more value to the table". So I got framed in that mindset and indeed I wasn't finding any security flaws for a while. When I was ready to quit I fell on a blogpost of some kid who found a really trivial flaw in Facebook. I was pretty much WTF and decided to try out a few crazy things, even though I was almost sure it's impossible to be that easy.
Guess what? I started finding vulnerabilities and each next of them embraced "it's possible" and I was finding more and more bugs. There were more people like me out there who didn't care about "it's impossible", and since then big corporations spent tens of millions of dollars on bug bounty programs and now the narrative is - "it's possible, you can hack Facebook and Google, and we'll even pay you for it!".
It's all thanks to people who showed the world that it is possible. Because if everyone is saying - it's not there, it's impossible, then there are big chances that it actually may be there because no one went there and checked.

Oh, and don't be afraid of authority. It means almost nothing these days. I've seen countless number of big brands saying they are not hackable, just to cry a few months later that they suffered from a breach. Anyone remembers Oracle's CSO claiming that their products have amazing security assurance process and they have magnificent security posture? Yeah, hundreds of security bugs have been discovered since then by people who don't care about "impossible".



TL;DR?
"Insanity is doing the same thing over and over again and expecting different results." - Einstein
"Whether you think you can, or you think you can't--you're right." - Henry Ford

2 comments:

  1. Well written. Why would anyone have a strong opposition to root cause analyses? That seems self defeating.

    ReplyDelete
    Replies
    1. Thanks for commenting and providing your feedback Neil! You're right, it's absolutely self-defeating, but this is how things roll in some organisations.
      Why would one not like RCA? Because it requires hard work and exposes to potential failure. It's easier to delegate the task or cover it with short fix than spend weeks digging into the problem and learning about underlying architecture and stuff.
      Also some people know they're there for a while, so they just cash the check, make myopic decisions to cover the crap and make everything seem looking great at low cost.
      Some people just are like that, it's harsh truth but our industry is flooded with people who don't care about legacy nor they are interested in doing the work that can expose them to critics, so it's easier to apply one more patch and remain being perceived as an expert and life saver.

      Delete