You Want IT When?

What’s your worst fear as a software professional? My worst fear is changing jobs and joining a team that is in crisis. You’ve probably been part of a team, heard about a team or witnessed another team in your company deliver to one of those unrealistic schedules.  Often times, the project starts off right, but entropy slowly builds as changes are introduced to the schedule with the velocity of an open fire hydrant.  It doesn’t start that way: it builds with a crescendo.  It’s not a case of replacing the old water with new water; it’s a case of filling a pool that’s filled 98% of capacity while watching the water spill over the sides and drain into the streets.

It’s not the fear of working hard, long hours, or getting my hands dirty that concerns me - in fact I find crisis invigorating - it’s whether I’ll be empowered to make the changes required to right the ship that concerns me most.  There’s nothing like the feeling of dread when participating in an endeavor that controls your life because the leadership is making mistaken decisions that you believe are the root cause of your undesirable circumstances: the kind of decisions that dig the project into a deeper and deeper hole.

Well it happened to me when I started a new job as a Software Development Manager in 1999.  When I arrived at the office on that first Monday, I was the first one to arrive, and when the first veteran employee arrived, he greeted me with the traditional first hellos and said, “yeah, the guys will be strolling in late today, they just worked through the weekend.”  I knew instantly that things were going to be interesting, and I was not to be disappointed.

I was recently tagged by Scott Sehlhorst of Tyner Blain to write about a personal success story as he requested it: “How I helped make a successful product!”  So I’d like share as briefly as I can, a personal success story that I’m most proud to have accomplished with a great bunch of professionals - smart, talented, and just great people to have as colleagues and now have as friends. 

The History 

As some of you may already know from my previous postings, I can get a little long winded, and an entire book can be written about this one, so I will do my best to keep it short.  Most importantly, I’d like you all to take away from this that I’m eating my own dog food.  This project is the inspiration for this blog, and all of the practices I will describe on this blog I have used on this project, so I won’t go into a lot of detail about the practices as they will eventually be described as time passes.  Whatever you may conclude about these practices, they have been field tested under extremely demanding circumstances, and I attribute them to the success of this project, and I believe the team members would all agree.  Software project success is about putting the right tools in the hands of talented people.  Now, here’s the rest of the story.

Fortunately, my first assignment was not to work on that project as it gave me time to know the people and learn the culture.  I had a smaller first assignment that was partly related, but it had its own timelines and requirements.  The Development Manager of the project in crises had the office next to mine.  We worked together at my previous employer.  We talked about his project. I asked, “when is the release date?”  “December,” he replied.  “You think you’re going to make it,” I asked? “Yeah, it’ll be tough, but I think we should be okay,” he replied.  Yup, he had an acute case of denial.

Here we were in September after many months of work, the schedule already slipped once (maybe twice, I’m not sure), and many features were still under development, he wasn’t alarmed. Neither his words nor his behaviors gave the slightest hint that he felt there was trouble ahead.  After the conversation, I went back to my office and started measuring the project and assessing the data.  My conclusion, there is no way this project is going out the door on that date unless the plan is to ship something that doesn’t work.  This is the benefit of metrics when you see the data, you can easily match the data against past projects to draw comparisons.  Also, there are independent conclusions that can be drawn from the data: like how many defects are required to find and fix to deliver a stable release, how long it will take to find and fix them, and how stable is the code base.   There was no mistaken it; this project was in trouble.

The project finally delivered in March or April of the following year, and it was a disaster.  Many, if not all of the customers, could not use it successfully.  We lost customers and revenues declined significantly.  At one customer, the application brought down a service provider’s entire network.  They stoped using the product forever.  Need I say more?

Triage

After a number of months transpired and proving my bone fides on another assignment, I was given the opportunity to right the ship, or as some thought, a death sentence. One of the other more experienced managers at the company told me flat out I was going to fail.  There was no saving this product according to him, and it wasn’t even worth trying.  I didn’t agree. One thing that was obvious to me at the time was that the executive management was going to empower me to do what I thought was right to correct things.  Here’s what I did to turn it around.  I’m not going to go into too much detail because I plan to cover them in more depth in future articles.

1. Developed a schedule using size estimates, and I tracked and updated the schedule weekly.  There was a schedule for the crisis project, but it was never opened nor tracked once it was written.  It may seem obvious to some who are reading this post, but my experience has been that most schedules are written at the beginning of a project, rarely looked at again and especially never updated again.  If you going to keep a team focused on achieving an aggressive date or any date, there is no shortcutting this practice.
2. Wrote a weekly status report on the project where I reported on metrics and the conclusions I was drawing from them.  Also included in the status report was an assessment on how we were progressing to the written schedule and what the short term goals were.  Surprisingly many of the team members were reading the status report, but I would have created it even if they didn’t because it solidified what I needed to have the team focus on to achieve our goals.
3. Measured the vital signs of the project weekly and analyzed the data using graphs.  During the QA phase, I measured daily.  The essay titled “An Objective Method for Navigating a Project Successfully” identifies the practices that I used during the QA and Defect fixing phase.  As a manager, having a suite of metrics that allow you to get a handle on the pulse of the project is like a doctor treating a heart attack victim in the emergency room.  There’s no way to treat the patient properly without the data.  This is what a good set of metrics does for the manager of a software project.  Here’s what I’m getting from the metrics: the pace of development, the progress towards the schedule, the stability of the code base, the quality of the product, the number of defects needed to find and fix before the product can be released with high quality, and how the team is progressing on those measures.
4. Set release criteria by which a release decision would be made.  This was a quality objective identified by what else? Metrics!  This was a real-time application that processes data 24 x 7 with adaptive performance algorithms requiring 30 days of data before the algorithms kick in.  Some of the release criteria were as follows: a candidate build needed to withstand continuous automated stability testing 24 x 7 for 30 days, the final build required a minimum 14 days continuous automated stability testing, less than 10 defects identified in the last 2 weeks of the release, and all high and critical priority defects fixed.  These were the dominant quality criteria but not the only.
5. Revamped the performance test suite.  Working with the product mangers, we identified and documented detailed performance requirements.   The QA performance test lead developed the performance test specification that we all reviewed.  We identified proper ambient user and network activity required during the performance testing.  We did not stop performance testing once the system met the specification, we continued to ratchet up the stress testing until we found the breaking points, and we fixed all issues that destabilized the system — basically any core dump was fixed.  The system had to fail gracefully.
6. Raised the quality standards of the team.  A defect that was not easily reproducible was not easily permitted to be closed.  If necessary, I worked with the developers to reproduce the defects.  Most defects labeled not reproducible are defects waiting to be found by the customer.
7. Assigned defects for fixing to all qualified members of the software team, not just the developer who worked in the area. I got tremendous pushback from the team on this one.  Some said it was wrong because it doesn’t make the developer who introduced the defect accountable.  I explained that I wanted the team accountable to the release not just the code they wrote.  This had a few benefits not only to the team but the individual as well: one, it created a deep bench of broad knowledge of the code base that accelerated defect fixing, two, developers had a great opportunity to prove their mettle, and three, developers by broadening their knowledge of the code base became more valuable to the team and provided them with more opportunity.
8. Held weekly cross functional status meeting with the team leads.  The purpose of the meeting was as follows: review and report on status, record actions, follow-up on open actions, review weekly objectives, and identify problems that needed to be solved.  I recorded formal meeting minutes, distributed them to the team, and used them as the working document for managing the project.  I can’t tell you how often I hear managers say they have no time to record minutes.  Recording status meeting minutes saves time when the right information is discussed in the meeting and recorded in the minutes: recording and tracking actions is paramount.  I can’t tell you how many times action owners have said, “I forgot I was supposed to do that.” If you don’t record it and track it, you lose it.  If it’s important, losing it adversely impacts your project.
9. Slipped the first release approximately 6 months because it did not meet the quality criteria.   I talk about this in “No Pain, No Gain.”  The release wasn’t meeting the established quality criteria, so the only choice was to slip the release.  Using metrics, the problem was assessed early, but it wasn’t easy to assess a new date as I did not have a technique for predicting how residual defects would influence the current code base.  My models assume the previous release is a high quality release.  Obviously, that wasn’t true here.  After this release, the team was hitting the release date with a schedule slip of 2 to 4 weeks on 6 to 9 month schedules with many releases slipping 2 or less weeks.

Summary 

There were many more practices that I changed or introduced for this project that had benefits to the team and the project, and I plan to discuss them more deeply in future posts.  Remember this was the same team of engineers that delivered a product that had extremely poor quality, and we were losing customers and revenue as a result.  What I found is that the right set of practices and processes when skillfully employed release the fullest potential of very talented people, though none of this would have been possible without the support and dedication of a great team of professionals.  I believe the team came to appreciate that as well — especially since they got their lives back.

The ultimate confirmation of one’s success is what your customers are telling you.  Here’s what one of the sales managers said to me a short time after the release.  He said, “Thank you Bill for releasing a product that the sales team can sell again.”  But the complement that the team is most proud of is when a large European TeleCom customer came back to us after a lengthy trial and complemented the team with “this was the first product from any TeleCom vendor that ever made it through our trials flawlessly.”  We won the contract, and revenues began growing again.  Quality and process do pay.

Who I Tagged

Glen B. Allemen who writes Herding Cats.  Ideas, comments, and references about project management, tools, processes, and field experiences.

Meme Source

Scott Sehlhorst who writes Tyner Blain.  Scott started the meme in his post “Software Product Succes Stories.”