There are no big problems, there are just a lot of little problems. – Henry Ford

Most reasonable people would agree that the smartest way to solve a problem is to break it into smaller pieces.  Sometimes, though, when we’re close to the situation we need to find a way to take a step back and look at the problem differently.

As this baby found out, taking on the whole watermelon was never going to work, so (with a little help) he took a step back (figuratively, because he seems like a pre-walker) and approached it differently:

Recently, one of my teams was facing a challenge.  They were making frequent changes to some complex code and each change would be very time consuming.  .There were many different places where this challenge was presenting itself.  Pretty quickly into the brainstorming session, someone came up with a solution that would eliminate about 60% of the places where this problem was happening – in other words, 60% of the problem would be eliminated resulting in a big productivity gain.  Awesome, right?

I then watched something that struck me….a room full of smart people spent almost an hour shooting holes in this solution because of “the other 40%”.  The group went back and forth and concluded that they couldn’t solve the problem and that everyone would go back and think about it – then get back together in a week.

I jumped in with the idea that they invest the small effort to implement the solution that would fix 60% of their problems and then look at the other 40% as a second problem to solve.  While this was pretty obvious, this group was pretty reluctant until I explained what I thought was the path they were on:

• Another week was going to pass with them running into this efficiency problem (and someone quickly calculated that the effort for the quick fix was going be less time than they were going to save within a week or two)
• They probably weren’t going to find a solution during that week and when they got back together, they’d just re-hash what they already talked about
• If they think they came up with a solution that covers everything, they’d probably realize while  implementing the solution that it really didn’t cover everything and as they “peeled the onion”, the work would grow and take longer and cost more than they expected
• In the rare event they came up with an all encompassing solution, it would probably be too big to justify doing anytime soon and they’d be living with the pain for a long time
• Then, as time goes on, the problem will grow and the solution will become that much more difficult

I also explained that after we implemented the solution that knocked out the 60%, the remaining problem would become clearer and could probably be broken down even further.

• Why do so many people on software projects forget to break their problems down into smaller pieces? First, they “know too much”.   I once had a CFO tell me that he could solve bigger software development problems than the average programmer because he didn’t know as much as they did.  When I heard this, I thought it was ridiculous, but he insisted that they would allow all of the things they know get in the way of solving the root problem.  He said that he could assume away a lot of the variables and make them constants, while the programmer would try to solve the problem with too many variables.
• Sometimes people are afraid that there is a solution to the whole problem and that they’ll look bad when someone else comes along with that solution and plays Monday morning quarterback on your work.
• Sometimes solving the smaller problem results in having to go back and do some rework when you get to one of the subsequent problems.
• There’s always the “hero complex” –people want to solve “hard problems”, not the “simple problems”
• Let’s face it, most people on development teams grew up hearing that you want to think everything through, get it right the first time and eliminate all that wasted rework.  It’s part of our DNA.

Now, most companies are moving to iterative or agile methodologies and have realized that it’s OK to make quick decisions without all of the information.  It’s OK to have to go back and rework some stuff.   Implementing something is better than coming up with the most elegant solution that never gets implemented.  Predictability is important to companies and it’s a lot easier to predict smaller pieces of work.

I’d much rather solve simple problems than not solve the real complex ones.   This is the same approach I try to take with everything (SOA, Agile development, incremental legacy system replacement, or eating watermelon – to name a few).

Tom Famularo

FAST

“Simplicity means the achievement of maximum effect with minimum means.” – Albert Einstein

In the book “Rework”, by Jason Fried, he boasts how his company’s software has far less features than its competitors.  When I first read that, I found it odd, but it got me thinking.

There are many reasons why I think larger software packages or systems are “on their way out” in favor of a combination of smaller components, calculation services, and rules engines orchestrated together.  The area I’d like to highlight here is related to “feature bloat” in those software larger packages.

Having been a Product Manager for various Life Insurance software products and reviewing many others,  I would estimate that at least 50% of the overall functionality is not needed or used by the average customer (and up to 75% of the domain specific functionality).  Over the past 20 years, one of the objectives for many insurers has been to consolidate legacy systems onto a single platform.  As a result, the vendors were forced into supporting all different types of products and business processes.   In order to be considered for such consolidation projects, vendors had to be able to support traditional product, universal life, annuities – fixed and variable.  Then, when vendors needed to expand they either went into other markets (health, disability, LTC ) or geographies (Asia-Pac, Europe, South America).  This caused their systems to grow more and more.  In addition, Policy Administration wasn’t enough, companies had to expand their breadth of functionality to the front office, underwriting, claims, billing, and so on.  In addition, over the years, they’ve chased whatever the cool feature of the day was or whatever some programmer developed over any given weekend.

To say the least, for these vendors to keep up in the features and functions race, the size of the systems grew immensely.  This is not even taking into consideration the vendors who were acquired and their software was “merged into a larger enterprise”.

On the surface, it sounds like a good thing to have all of this functionality in one place.  So, why is all of this a problem?

1.      The systems were initially architected for a purpose that wasn’t different than accommodating everything it ended up accommodating.  So each step of the way, more and more code was layered on top its foundation (similar to a foundation that wasn’t big enough for the building sitting on top of it)

2.      Each of the major additions usually resulted in some re-architecture or software pattern changes that left the system with a lot of inconsistent ways of doing things.

3.      In addition to all of that extra functionality that isn’t really needed, there’s always a ton of customization required to meet the customer’s real needs.  As a result of navigating through all of this extra code and configuration (the aforementioned feature bloat), the cost, effort, and risk of doing the customization is extremely high.  One could argue, it’s 50/50 as to whether it’s just easier to start from scratch.

4.      The end result of all of the customization (configuration or code) wrapped around all of the code in this large system is very difficult and costly to maintain.  So, you’re now stuck with dealing with this problem for the life of the software.

5.      Performance usually suffers from having to perform more operations than necessary to support the business as well as a larger memory footprint.  The answer is usually “add more servers” – which then blows up the infrastructure costs (both hardware and additional software licenses)

6.      Tons of inconsistent code and inter-dependencies makes it difficult to use lower cost resources (off-shore or less experienced) and usually results in requiring expert developers or significant ties to the vendor.

7.      The vendors have to keep up with their diverse customer group and spread their investments pretty thinly.  In addition, the hottest new topics are where the majority of the investment dollars end up going – which leaves the average customer not getting a good return on the recurring license/maintenance they’re paying the vendor…which probably doesn’t matter because the combination of the software architecture and all of the customizations make real upgrades too difficult anyway.

By moving to a model with smaller components and services, companies can meet their immediate business needs with a lot less code, configuration hardware, and headaches.  It allows them to have an evolutionary expansion plan where they’re adding only what they need.  The net result is smaller implementations, lower cost maintenance and more technology choices over the long run.

Tom Famularo

FAST

The recently released “2011 US Insurance CIO Survey” from Celent provides a great snapshot of where insurance CIOs are focused in a number of areas.   While cloud computing is clearly emerging, it’s interesting that of the surveyed respondents:

• No companies reported that cloud computing is “in wide use”
• 38% of companies have cloud computing “in limited use”
• 52% “will investigate/pilot in 2011”

Obviously, it’s still in the early stages and it will take a few years for the model to develop and for cloud computing to be fully exploited.  Because the benefits are so great, I believe that companies and providers will quickly start to find new and interesting ways to tap into the vast potential of “the cloud”.

In our case, I can’t see how we could have gotten FAST off the ground without the Amazon Cloud.  In 2010, we paid Amazon between $1,500 and $2,000 each month – about $20,000 annualized.  What do we get for that?

• 6 Windows servers.  Each has 100 GB of disk space, dual core processors, and 7.5 GB of memory
• 2 Linux servers.  Each has 150 GB  of disk space, single processor, and 2 GB of memory

If we need a new server, it literally takes minutes.  We pay for what we need, when we need it.  We don’t have an MIS department.  We don’t have a server room.

Doing this “the old way”, we would have spent vast sums on hardware and building out a server room.  And we would have had to hire experts to configure and manage the servers.  Even if we had done this “on the cheap”, we would have spent $250,000 to $300,000 on hardware and staff (compared to about $20,000). As a startup, it’s almost impossible to replicate the quality and stability of the Amazon environment at any price.  Having lived through that in the past, there’s a tremendous drain on management and on staff productivity as you iterate toward a “rock solid” environment.

In other words, we paid Amazon less than 10% of what it would have cost to do it ourselves. THAT’S 90% LESS for something better.

So, does this have any implication for the insurance industry more broadly?  It sure does.  While the option of putting everything “up on the cloud” would not be practical today for a large insurer, as the industry moves to a more SOA-based approach it is practical to deploy discrete web services or components on the cloud.   The potential is obvious and the benefits could be extraordinary.

John Gorman

FAST

“The worst programs are the ones where the programmers doing the original work don’t lay a solid foundation, and then they’re not involved in the program in the future”. – Bill Gates

Have you ever looked back at something you wrote or created years later and wished you could do it all over again?  While you may be proud of what you originally did, most of the times you can think of many things you would do differently.  Most software architects and developers can relate to this.

While the desire to rework things you’ve already done exists in most facets of business (or even life in general), the issue is magnified with software.  There are several reasons for this:

• First, a significant amount of time lapses from creation to completion. Beyond initial completion of the software, the problem gets worse over its lifespan.
• Typically, the first things written are the foundational items.
  • There is usually pressure to “see something” working.
  • Technology choices, architecture patterns, coding standards, and the exact way that certain things will operate are all decided in the first few months.
  • You may or may not have the right people making these decisions. The wrong people will create problems that will be with you forever. The right people
    are sometimes too in touch with the latest and greatest technologies and philosophies and also are probably their own biggest critics in the end.
  • Decisions are sometimes also too much of a function of the times. With the rapid change in technology, over any 5-10 year period, this could mean big differences between how it was done and how it should be done “today.”
• Business Needs Change
  • It’s not practical to consider all future potential business needs up front, but in most cases throughout the life of a software product, the requirements will change
  • Many times, the changes of requirements are big enough that require sweeping global changes to the software and it is very costly and risky.

The irony is that the compounding of this problem over the years is usually the thing that ends the life of a software product, but there’s rarely a time to justify going back and changing it.  Having managed software products and divisions at both small and large companies, the one thing that is a common is that there is more work in the backlog than can ever get done with the budgets and schedules.

If I had a nickel for every time a developer came to me and said that they wanted to make some global change to the system because it was the right thing to do, I’d have a lot of nickels.  95 times out of 100, their ideas for changes would make the product better.  Unfortunately because of the circumstances, 90 out of those 95 times I couldn’t justify doing what they wanted to do.  The costs of making those global changes are enormous.  The risks to customers, sales, and internal schedules are even worse.  The further along the product is in its lifecycle, the higher the cost and risk.  In the rare event that we could make a case for something, the cost and time had a big impact and it ended up being so disruptive to the product that only future customers could benefit from it (i.e. the already impractical notion of an upgrade became impossible).

The argument that most good architects and developers make is “that’s why we need to get it right the first time” – hard to argue with that.  Unfortunately, that sometimes leads to the old “analysis paralysis” and it’s still unlikely that you won’t be looking at it a few years later with many things you wish were different.

I’ve had the good fortune (and pretty rare distinction) of being part of creating several new software products from scratch.  While there are things that you learn on each one that you apply to the next, you still fall into the trap of continuously wanting to rewrite portions of it.

In my latest endeavor, I’ve tried to tackle this problem head-on.  We’ve architected things in such a way where every piece of code is designed to be replaceable (and backward compatible) in a very cost/time efficient way (largely through built-in automation).  We have a small team whose single focus is to constantly look at the code, design patterns, and technology choices and rapidly make those sweeping changes.  In fact, many of those ubiquitous changes to the actual code can be made through external configuration.  This approach provides a pretty practical solution to this problem that seems to have no real solution.

I’ll write more detail about our solution for this later, but wanted to get something out there to see if others have closely observed this same pattern/problem and/or if others have found solutions that don’t require a lot of compromise or a lot of money.

Tom Famularo

FAST

“Documents create illusions of agreement…100 different people can read the same words, but in their heads, they’re imagining 100 different things” – Jason Fried, “Rework”

This is part of my series of blogs regarding how iterative development methodologies like Agile solve practical problems that have been haunting software implementation projects for years.

Too often, too much time and money are  spent, the customer is frustrated, and the wrong things are built into a software application.  I’ve seen projects at many companies where more than 50% of the effort was done after the development team was “code complete.”  While I have many opinions on this broad topic, I am going to narrow my focus to  the top reasons why the notion of gathering business requirements through traditional means is flawed, as well as some tips for getting around this.

I have to apologize up front that most of my writing here describes why this problem happens and my tips for mitigation are pretty brief.  My thinking is that this is one of those cases where recognition of the problem being a big part of the solution.

People Develop Software for People

One of the first things to remember is that we are building something abstract (software for businesses), not a physical thing that we can represent in drawings (like bridges or rocket ships).  Much of the functionality in business software is open to interpretation of people.  Because so many people involved with building or implementing software are analytical with an engineering mindset,  they want everything to fit into nice algorithms with a set of true and false facts.  Computers are after all just a set of 1’s and 0’s, right?  However, people are not.  There are too many variables related to a person for their software “needs” to be boiled down to “the right answer.”

Software “Requirements”

The typical process that people have been implementing for years seems to make sense:

1. A team of Business Analysts interviews a series of users, managers and other BAs to figure out what the company needs to do to run their business
2. The BAs go back to their hotel room or some other remote location and compile all of their notes and sketch out their thoughts
3. Those thoughts will then get turned into pages and pages of documents that contain process flows, use cases, descriptions of requirements and much more
4. The BAs then mail the documents back to their counterparts for review, get some feedback and incorporate
5. Then comes judgment day – the sign-off process…the customer then signs off and the requirements are “locked down.”

NOTE:  In some cases, there is a “liaison” between the end customer and the interviewer (BA).

So, what could possibly go wrong with this process?  It is “tried and true.”  It all comes down to one significant flaw and that is everything is open to interpretation and we then run into the “human factor.”  The interviewer and interviewee(s) have a different frame of reference to draw upon.  All of their experiences leading up to those key interviews drive their perspectives. Typically, the interviewer is thinking about the “new system” and the interviewee is thinking about the same system they’ve been using for the past 20 years.

We’ve all seen many discussions where you can tell two people are saying the same thing, but they mean something different.  When those parties have completely different frames of reference, they hear the words and process them the way it makes sense to them.  So, that’s why we put it in writing.  Right?  Now we can be sure that we both mean the same thing.  Unfortunately, we humans can process ideas in oral communication about 5 times faster than we can when reading.  The benefits of body language, facial expressions, and inflection are lost.  So, in the traditional system implementation process, the BA writes what he or she was verbalizing during meetings and the client now understands even less.  However, for so many valid reasons, the client does not want to slow down the process, so he or she only asks a few questions and provide some limited feedback.  The feedback is then interpreted by the original author – the BA.  This process continues until an “agreement” is reached – which doesn’t mean that what is in each other’s heads is the same.  As they say, “there are many ways to skin a cat.”  This holds true for software implementation.

Detailed Design

My favorite part comes in next…detailed design documents are then written telling the client what is going to change within an application.  In a vendor situation, the client really doesn’t know the details of what is already in that application.  So, you then ask the client to “sign off” on the delta of something where the starting point is vague to the client and you can only hope that you have an exact interpretation of the end state.

Development Time

Developers are notorious for not being the best readers.  Typically, they were on the opposite end of the spectrum from the English majors in college.  Reading text that has been written for the end user to understand is counter to the make-up of someone who wants to read numbers, technical jargon, and instructions.  So, naturally, they are interpreting what someone else has already interpreted.  Rarely do they get it right   No surprises here – this is what everyone expected – which is why we have the functional test after the unit test.  Note:  In many cases, the developers aren’t trusted to do the work on their own, so they often have a Tech Lead in between.  Yet, another person to translate.

The Functional Test

First, many organizations skip this and go straight to a QA person to review what the programmer did.  In those cases, there is yet another person to interpret this requirement.  For those who have implemented this step, they’ve noticed some benefit.  Unfortunately, most of the time, when the original BA tests the functionality, it doesn’t operate the way they thought it should.

Hint:  This part actually works. The BA will then communicate to the developer, in small chunks, quite often sitting with them, to tell them what isn’t correct.  The developer, then makes the changes and hands it back.  They iterate until the functionality works.  So, at least now, we’ve eliminated one more level of interpretation…what is developed now works to the interpretation of the BA.

Summary

From the time the customer indicated what they wanted to the time they actually see something, several months (in some cases over a year) have passed.  In that time, many things could’ve happened:    turnover in client personnel, reprioritization of business objectives, frustration with project delays and cost overruns, or, simply, now that they see what was built, they realize it just isn’t what they wanted.

I’ve been involved in too many cases of conflict resolution after the fact.  The development team built something different than the customer thought they were going to get.  In most cases, when I look at the requirements, I can see both “sides” of the story.  Is this anyone’s fault?  Usually, it is not.  It was a flawed process in the first place.

Unfortunately, the amount of time lost and dollars spent in this process were significantly more than they needed to be.  Most experts would agree that so many of the problems related to software projects are related to “bad requirements,” but why do so many people still follow this same process?

Why Iterative Development Works

Here are some pretty simple ways that an iterative development process can reduce the cost, timeline, and frustration meter.  I am not going to describe the details of those processes.  You can pick up any book or just google on Agile.

• Interpretation discrepancies are found much sooner in the process (before significant amount of time is spent headed the wrong direction)
• Since so many requirements can be supported different ways, the customer might even prefer the misinterpreted version that was quickly built – more likely when it’s sooner rather than later and more discrete rather than when it’s combined with tons of other changes
• Developers have more of a sense of ownership when they are closer to the customer and are more likely to be working as part of the team
• Functionality is broken down into smaller parts (“stories”) that can be prioritized.  Therefore, not all pieces the customer thought they needed are really needed (this is a cornerstone of Agile and is worth its own topic for a later date)
• Less people have to learn and internalize the requirements

If your organization is struggling with some of the things I describe above (whether with internal development or with vendors), then I encourage you to consider switching over to an iterative development methodology like Agile.

Tom Famularo
FAST