Questions, only questions

Today I'm full of questions. Software development and programming is hard. Is a close encounter with layers and layers of complexity. Once we master one level, we open a whole new level. To tackle it one must employ multiple techniques. But they always involve trade-offs. How to balance programming using standard and maybe cumbersome idioms vs more elegant but more obscure idioms? What if the programming style is too personal and strange to others? Shall we settle for the lowest common denominator ? How do we balance the conflict between something harder to understand at first but easier to extend and maintain against something more familiar but with exponential complexity growth after each change? How do we maintain the "conceptual integrity" (F. Brooks) through the life of the system? As the project complexity grows, the need for sophisticated techniques increases. (I'm coming to the conclusion that if a developer can't comprehend some level of complexity, shouldn't be allowed to participate in the project without close mentoring). Refactoring is necessary because entropy. Every code base starts to degrade and loose integrity as evolves. I wonder if there's a code entropy metric... At what point a refactoring is good enough? When do you stop? At least your architecture should be stable, or you didn't validate it. I guess complexity is in the eye of the beholder. I found that a system composed of smaller simple pieces with complex interactions is easier to evolve than bigger pieces with simple interactions because each piece is internally complex and the limited interactions means its harder to combinate them in new ways. Of course, gratuitous complexity should be avoided at all costs. Sometimes I look for a "zen" approach to coding. At the end, I always take a pragmatic approach and find a trade-off that balances the forces. Isn't coding about that? Each line you write is a choice you need to make... At some point we need to use "introspection" and "metaprogramming" but not the programming techniques: introspection on how we approach problem solving and programming and metaprogramming on what we can do to program better.

Project (and people) over Process

Thinking in my previous post with the RUP sample documents (or artifacts), I don't want to perpetuate the myth of RUP being a document-driven heavyweight process. Using templates involves a trade-off: you have a common format for the documents and guides you to the questions you need to ask (who? why?). On the other hand, the template comes in a "one-size-fits-all" format, and you need to adapt the template contents to your project (but to be fair, RUP includes a small project vision template).
The process of discovering what should be in a document is as important (if not more) as the document itself, you’ll find what should be in the document and what shouldn’t be while gaining insight in the problem. Specially true for analysis and/or design artifacts
The goal for every artifact is communication, unless you reached the point when it can be automatically translated into an executable form. And remember:

• You DON'T need to fill every template that a process provides.
• You DON'T need to follow blindly every activity that a process describes.
• You NEED to adapt the process to your project.
• You NEED to follow the process that makes sense for your project in the context of your organization.

(Yes, if you have a process group, you’ll need to convince them…. And yes, that includes not always doing test first, not always pair program, etc…)

RUP sample documents: Inception

Here are sample RUP documents I created for a training I gave:

The requirements set:

BOSBusinessCase.pdf

BOSVision.pdf

BOSGlossary.pdf

BOSUCOrderBizcochos.pdf

BOSUCRetrieveConsolidatedOrder.pdf

The management set:

BOSSoftwareDevelopmentPlan.pdf

BOSRiskList.pdf

The most important ones are the vision document and the use cases. One thing I really like about the vision document is the problem statement, defining clearly the problem to be solved by the system.

One important thing I see missing in many “processes” is an explicit focus on finding what the user needs (can be very different of what the user wants), clearly stating the problem to be solved helps a lot in that direction.

Also you need to have some alternatives identified. How many projects are started without a clear problem to solve? Sometimes they're successful, but you must be VERY lucky to pull this off.

Also the alternatives are important: developing good software is very difficult: sometimes, something like excel is good enough!

Basically, it boils down to clearly defining the problem and have the alternatives in mind instead of jumping blindly into a development project.

BTW, the problem statement is not exclusive of RUP, they're just requirements management good practices. Even is included in the IEEE 830 SRS standard.

My restatement of "No silver bullet"

(this is my interpretation today)
The essence of software development is capture the domain information and translate it to a computer for execution.
("simple" enough, eh?)
"The most radical possible solution for constructing software is not to construct it at all" (Frederick P. Brooks)

Waterfall, RUP, XP... and one process to bring them all, and in the darkness bind them

What happens if you have a big project to execute and your organization mandates Waterfall? Of course, you can try to follow the mandated process, and hope everything goes fine (bad news: chances are not).

As a rule, you must ALWAYS adopt/adapt the process to fit your project (and not the other way around)

A big project will be less risky using RUP, but if you can't convince the outside groups (typically management, QA, overzealous SPEG/SPI group, maybe the customers too) to use it, you can use RUP and make it look like a waterfall from the outside (but ONLY form the outside). BTW you should try to convince those groups that waterfall only looks good on paper and modern processes can yield better results, but it takes time, and probably you need to start right now with the project. How do you map RUP and Waterfall? Don't do the classic mistake of Inception=Requirements, Elaboration=Design, Construction=Coding & Unite testing, etc... you will be in trouble when at the end of Inception, the outside groups will ask for the complete requirements for review!

The trick here is map the end of Elaboration to the end Requirements phase, so you can collect all the requirements but also implement the critical ones. You can map the end of the Design phase to somewhere in the middle of Construction (or at the end if you can get away with it). When you're done with Construction, you can release your code to the outside QA team to do whatever they want. Probably is a good idea to have customer involvement early so he can try the system and provide valuable feedback (that's the whole idea!).

You will end with an oddly shaped (in time/effort) “Waterfall-looking” schedule but still all the advantages of an architecture centric iterative incremental methodology like RUP. And if you can't live without doing XP and will complain if you can't do it, you can safely do XP inside each RUP iteration...

BTW, by no means TRANSITION=TESTING!!!!

What's your project score?

The Standish group's CHAOS report has a table of success factors:

Success Criteria	Points
1. User Involvement	19
2. Executive Management Support	16
3. Clear Statement of Requirements	15
4. Proper Planning	11
5. Realistic Expectations	10
6. Smaller Project Milestones	9
7. Competent Staff	8
8. Ownership	6
9. Clear Vision & Objectives	3
10. Hard-Working, Focused Staff	3
TOTAL	100

The study is not new, but still is valid. This are the critical factors for a project success.
Mapping it to your current project, will let you focus on the areas that will improve your chances...
What is your current project score?

Software Engineering Radio

I really like this site. Software engineering podcasts in english from a german team of experienced software developers and architects. I really enjoy (almost) all of the episodes, some refreshed concepts (like architecture) and others gave me an idea of new topics (I'm fascinated with MDSD/DSL stuff, but still I couldn't try it). Some of the topics covered are Architecture, Patterns, Model Driven Software development, Agile process, SOA, Dynamic languages, etc (and very good interviews too).

The podcasts are intended as a reference resource, not technology specific or a news podcast.

My only complaint is that I have my MP3 player totally full because of their podcasts (guess I should get a bigger one) :-D

Bottom line: totally recommended, don't miss it... go now!

RUP and risk management

I must admit that I like the Unified Process. The more non-trivial the project is, the more I think is applicable.
Typically, RUP is presented as a use-case driven, architecture first, iterative-incremental process divided in four phases: inception, elaboration, construction, and transition.
Somehow, using a manufacturing metaphor (I know, I know... usually are the worst ones, and can't go too far), the project can be divided into 2 major stages: "engineering" (inception and elaboration) and "production" (construction and transition).
The good thing is "engineering" aims to reduce significantly the risk level of the project. The biggest uncertainties at the beginning of a project are usually related to requirements (and scope) or technical ones, and they have a big impact because they’re the major sources of cost and time overruns. The traditional approach like the “waterfall” process is to pinpoint all the requirements and then complete the design to the detail (what the “agilists” call “Big Requirements Up Front” and “Big Design Up Front”). I will not beat the “Waterfall” straw man, it has its niche (albeit a lot smaller than most people think). Other option is ignore them and hope that the team is “agile” enough to respond to the risks when they materialize.
In the case of RUP, there’s an explicit intention to “defuse” those risks, by exposing and validating the assumptions as soon as possible:

• the inception phase main goal is to discover what it should be done (usually identifying most of the use cases, and detailing the most critical ones)
• the elaboration phase main goal is to define how it should be done, by defining the architecture of the system (based on the critical use cases)

After those phases are completed, you will have the architecture of the system, feedback from the most critical use cases, a clear view of the scope, and no technical surprises awaiting. At this point you should be able to have an accurate estimation of the total project effort, and the total risk exposure of the project should be significantly lower...