Saturday, September 23, 2006

Design is back. Long live design

It is very encouraging to read from the experiences of agile practitioners that they think up-front design (Architecture and framework) is too important to be left implicit in any methodology. I always believed in up-front design and evolutionary design facilitated by effective refactoring and automated testing, being core parts of any methodology. There are complimentary to each other, rather than forcing a programmer to use one over another. You dismiss one; you have a very lousy software at hand.

Martin Fowler also advocates nicely in making use of best of both worlds. It makes more sense now.
Being an Agile Architect
We found that a year down the road that there was still a lot missing. Too many "infrastructure" or "framework" things were blowing by in this delirious rush to crunch out 100% business functionality every week no matter what. Someone with Architecture experience needs to perform explicitly the role of Agile Architect and restore these capabilities to a team

Sam's post on agile architecture has led to the death of my last vestiges of my denial in regards to a need for some formal recognition of software design on an agile team.

The absence of explicit software design activities on our team has lead to some degradation of software design that is leading the code down the slippery slope to opaqueness and to the initial indications of software entropy. We're trying to add a bit more formality to software design and software design guidance into our process. Recognizing a software designer's role on the team tasked with up-front design and continuous software design quality assurance is part of this effort.

It pays to be a coward Extreme Programmer


survic said...

Do not kill me, I am just a messenger:

1.Bad architecture? Yes, but you have a product, don’t you? It is much better than you have nothing!

2. More on its human side: Even it is true that we need to design the architecture; because developers know nobody is capable of doing it other than themselves; but they are not “entitled” to do that; so, they ridicule the idea in public so that they can do it in private, piece by piece, during coding. As for the time they need for the design, because they have denied the existence of the design, they “merge” it with “requirement gathering” and "coding" -- it is not totally honest, but it works, because they do not need that much time to do that, this leads to system side.

3. The system side: Like families, all bad systems are very different; but all good ones are alike. There is only one architecture, a silver bullet (OK, if it is not, let’s believe it is!). One size fit all. As a result, there is no “design”, just learn it, and learning can certainly be fast and easy, then, and simply follow it. So, although sounds too good to be true, and paradoxical, we do can have a “best practice” (silver bullet) design, and we spend only very little time on design. Yes, you do can eat the cake and have the cake that same time – if you are capable of creating cakes faster enough.

survic said...

The classic XP’s doctrine: use a metaphor as the architecture. “Metaphor” means a silver bullet -- well-known, “best practice”.

Vikas said...

Never. :) You are more knowledgeable than me about Agile Methodology. Feel free to correct me.

I am doing more brain-storming rather than debating this issue with you

Context: Project with duration more 6 months with team of more than or equal to 4 developers.

1. At least we will have some product.

I agree. We have a developed a product with no up front design. We have used automated unit testing and continuous integration. So end product would be a multi-tiered, multi-layered, object oriented architected solution.

But how many times I started coding something then I hit the roadblock and I have to jettison the current approach and restart coding. But when I did some up-front design backed by proof of concept, I rarely failed. I think that this must be true for lot of other developers
I do see some failed stories or some stories missing the deadline.

2. Covert design

There will lack of communication in development team unless there is some kind of brotherhood among developers.
No appreciation of design your effort from management. Mort and Elvis will be getting the same treatment from management or may be Mort may be more valuable.
May be I am wrong on this one. In Agile Methodology, every developer has to be expert. So if you are selected on agile team, you are already recognized. If you survive on an agile team, it is appreciation in itself.

3. You mean that by end of development, there will be architecture and framework which will be a silver bullet/one shoe that fits all. You know that there is nothing like that. Even the best framework that one can find in the market needs to be tear apart to adjust to peculiarities of a project (transactional vs. non-transactional, high volume transactional vs. low volume transactional and so on)

I think that whole problem is getting expert programmers in a good market. If a client is hiring all expert developers, why should he didn’t get a product that meets his business requirement and have solid architecture.

Up-front Deign is being done covertly or passed to next phase. Since in current phase, one may not time to revisit the design while working on business valued stories.

Survic ,
Would you make Design as implicit activity or explicit if you were in project like that and you could influence the Software Development Process?

survic said...

The problem is not the process, but time. Most of the time, projects are time-boxed. You can add people, you can even cut some features, but you cannot add time.

Also, for a 4 month project, you got to have a first cut in the first 2 month. So, for a 4-month project, you have two weeks design time. That is it. We can refactor, of course ;-)

survic said...

you mentioned 6 month project, so, give it three weeks. That is about it. Otherwise, it is too risky.

If -- always true – there are new technologies, then, that includes the learning ("researching") time. So, there is no choice but refactoring or passing the "lesson learned" to next projects.

I guess what I am saying is that I will ask for design time, but I will begin to do coding while I design. So, basically, I will ask for the freedom of “non-linear” progress in the first two or even three months, so that I can do big refactoring when necessary. On the other hand, “design” is very quick 5-pages writing and two days discussions and then learning/researching/prototyping. No M$ Word or visio or even class designers. Those things are bad -- the difference between good design and bad design.

survic said...

You mentioned 6 month project, so, give it three weeks. That is about it. Otherwise, it is too risky.

If -- always true – there are new technologies, then, that includes the learning ("researching") time. So, there is no choice but refactoring or passing the "lesson learned" to next projects.

I guess what I am saying is that I will ask for design time, but I will begin to do coding while I design. So, basically, I will ask for the freedom of “non-linear” progress in the first two or even three months, so that I can do big refactoring when necessary. On the other hand, “design” is very quick 5-pages writing and two days discussions and then learning/researching/prototyping. No M$ Word or visio or even the "class designers". Those things are bad -- the difference between good design and bad design.

I guess that is another aspect why XP is saying “code is design": in XP, designing is also mostly coding (it is just more trial-error and relatively big refactoring), it is not "picture drawing".

Vikas said...

Hi Survic,
I agree 100% with your views on four month long projects.
I agree with your broad outlines about 6 month long projects.

What about projects with duration of 1 year or more than that?

I am watching whether your feelings about UML and documents will change or not. :)

survic said...

If one project last more than one year without deliverable, that is a problem project in the very beginning. Release often, so, no matter how you stretch it, more than one year without release is not right.

No release is bad for developers. You may say, management is offering some free research time, why you do not take it? Management/clients change. As professionals, we need to take care of our clients (management is, in a sense, a client). It is not good for any business project that last longer than a year without release. If it is not good for clients, it is not good for developers – one way or another, eventually it is developers who take the blame, believe. At least, the project is gone, all your work is gone (except the salary; but we also work for work, don’t we?).

Cut it into pieces, and release something. I am not talking about MOON or MAR stuff; that kind of thing perhaps is too inter-related, no way to modularize. However, for all/any business applications, I do not believe that they cannot be cut into pieces.

As for pictures, I use them for reverse engineering to create nice pictures for demo only. Any other usage, and any enthusiasm about it, I treated it as a red flag. I am open-minded, so, I guess I should sun-set this for 3 years.

I used UML in UP/RUP processes. It always gives me bad taste. Other than UML, UP/RUP is fine. Then, you may say, what is left in UP/RUP without UML – you got it – XP ;-)

All boils down to two things:

a. Know a few “silver bullets” (“metaphors”), so that you can get things started quickly.

b. Refactoring techniques, including big refactoring techniques. Big refactoring should be avoided if possible, however, small-big is a continuum.

As you can see, either of them needs UML.

As for the entity model ("domain model" per se), we talked about it before: to the extent that we need some pictures, then, we have ER model – we will have that anyway.

Of course, ER is UML nowadays, so, in that sense, we cannot avoid UML.

Vikas said...

Hi Survic,
One more articulated post from you.

I agree with you. Documenting 100% is as bad as writing automated unit for front end. Documentation is excellent for reverse engineering or expressing or documenting something interesting about the system. You need documentation also from manager’s perspective, because programmers change jobs. If you lose a programmer who had done design/analysis work, you lose all the work done by him because of no documentation.

Why do say UML left a bad taste in mouth

1. Because of Visio‘s bad support for UML (only 1.0)?
2. UML is not expressive enough?
3. UML is tough to prepare rich Models?

survic said...

Reverse engineering is a good thing.

Even “forward engineering” is a good thing – if we use it as a productivity tool by programmers (a variation of code generation). I guess the problem is the implication of “forward engineering” or “UML programming” -- the underlying assumption is that architects design the UML; developers run them and fill in the rest – it is not the reality now; so, do not assume it. If you do, then, it is disaster.

As long as we treat UML as what it is: pretty pictures, then, we are fine -- as long as we know that the more serious side of architecture design is to try things out. Note that it is not aimlessly trying -- I have seen that also: purposelessly prototyping forever! The best strategy is to pick a few well-known architectures, and assess them in the light of the requirements, and tear them apart, and put them together again – and of course, those “tearing apart” and “putting together” require trial-error/truth.

As for the “bad taste” and “red flag”: It is more the underlying assumption mentioned before than real technical limitations of UML. The problem happens even in Java’s context, where the visio-like things (Rose, etc.) are much better – as I said, that is actually the problem -- dummy visio quickly let you know that it is just pictures -- I actually like visio for that: honesty.

Again, as I said, I will certainly revisit this stand in about three years: perhaps at the time, we can do UML visual programming -- it is both the UML spec and the IDE support.

survic said...

I read Martin Fowler’s article again. I have to admit that he (his book) taught me the parallel between database and OO analysis(from our recent discussion, I realized that it is interesting that this insight is actually not the “mainstream OO”!) and how to do refactoring. I did not like his “enterprise patterns", though, too late and too much renaming.

However, the article you linked is pretty good. I remember I read it before, but it is worth it to re-read it.

I noticed that he is not sure about “metaphor”. It seems that my interpretation (“metaphor means well-known silver bullet architectures”) is a pretty slick one to defend classic extreme XP. It is interesting, I do not know where I got the idea, or, my brain just somehow connected the dots, trying to reconcile with the reality, and “invented” the idea?!

Another thing is about design patterns. He is implying that since developers use design patterns, so, they are doing design. I agree with him that we should use design patterns. However, my defense for classic extreme XP is that “design patterns” for modern languages (Java/C#/VB.Net) is a misnomer: it is “coding patterns”. As a result, programmers should learn and use design pattern ubiquitously while at the same time honestly say that they do not design, they just code!

Of course, all those sound like word playing/twisting. However, sometimes it is interesting to observe how we use those concepts/words.

As for UML, his non-extreme stand reminded me that, in addition to reverse engineering, I do not oppose using hand-written UML: it is always good to draw a picture when necessary. However, I also noticed that I do not even use hand-written UML that much anymore, because there are only that many design patterns, just mention the name is sufficient, isn’t it!

Vikas said...

Rocky explains his general antipathy towards Agile Methodologies,guid,5b6a0b92-00d8-4135-969c-002fff8dd971.aspx

Vikas said...

Refactoring is Martin Folwer’s best work so far besides Analysis Pattern(though it is little dated). I agree with your assessment about Enterprise Pattern book. I didn’t learn anything new from that book as compared to GoF’s design pattern book. I do read his blog but hardly understand anything. His blog is quite cryptic as compared to his book. I read this article some times back; it really makes lot of sense to me now.
I am also okay with the idea of using hand-written XML , only problem is that I can re-read even my handwriting after some days.

When ever I model, I try to follow UML notations. UML notations can be more expressive than block diagrams in some cases. But I don’t restrict myself to UML notations, I use very thing that I think, will be useful to express my thoughts.

I am glad that I ask you to elaborate your statement. Once you elaborate, I find out that our views are so same.

As you said , you go to extremes. While it is good to find the weakness of technology when you take to extremes, it also hurts. I am sure that there must be decision makes who get excited by the new ideas and take things to extreme. It is good to educate them with all weakness of new idea/technology/concept and force them to use it pragmatic. Pragmatism pays.

As you said, Great minds think alike.

You must have guessed by now that my focus is on Unified Process, Long-term projects. That is where money and stability is. It must have been more to do with my consulting background.

survic said...

I read Rocky’s post, and the link in his post. Google is the way to go, isn’t it?!

The thing is, interestingly, you wonder, if you can only categorize the google way into two categories: XP way, or, well, let’s say, non-XP way, which category you put it in?

I guess, it is the trick here, isn’t it: XP somehow brain-washed us to think it this way, XP way or non-XP way. In reality, it is the developers, the team, not the Process or, the “Methodology”.

However, isn’t it also the core of the original message from the classic XP?

You pointed out one thing: consulting background, especially consulting for design phase. By itself, it is more in line with UP per se, not XP per se. I guess that must be a factor for not feeling comfortable with XP?

However, also as you pointed out, pragmatism, results. XP is all about results (ironically, it has its embarrassing record in its birth record, man, it is not a perfect world. It is humorous to point the out, but also cruel though. But it is good also, so that we all know, “all depends”). UP/RUP can be configured to keep things running, keep the momentum, and not get cancelled (release often is the ultimate way to avoid being cancelled), and deliver the final product(s) smoothly – if you do that, as I can see obviously did, as any experienced developer would, then, it is UP configured as XP/Agile, or agile (lower case “a”), or, just configured plain right and good.

Just to be fair to XP, techniques of extensive unit testing (not UI though, that is too much; also, not most of the property methods, thast is also too much) and refactoring techniques -- those are brought in, put on the table, and made visible, by XP. Before XP, those things are too “low level” to be even widely discussed seriously.

Also, when UP/RUP/UML was at its marketing peak and create huge enthusiasm, XP came out and provided a much needed balance and saved a lot of projects. It sounds paradoxical: the extreme programming moderated the unified process.

Vikas said...

survic wrote

As a result, programmers should learn and use design pattern ubiquitously while at the same time honestly say that they do not design, they just code!

As I mentioned in my previous post, there is nothing called coders in arena of oo programming. Even if you have architect/designer, still programmer has to take tens/hundreds of design decisions while coding.

How to sell Refactoring to Non-technical Managers?

Refactoring is a disciplined technique of making changes to software code structure, altering its internal structure without affecting its ...