Agile web development - Mind the Gap!

I will try to bring forward the inherent complexity of modern web development and explain the gap that I have experienced exist between technical development and design.

Web development is in many areas distinct different from conventional software development. In few other software development disciplines are such a variety of professions involved more or less directly in the prosess.

Agile software development methodologies are mostly coined at software development companies, or consultancies with such a long and trusting customer relationship that they are almost to consider as a subsidiary. For most however, the level of trust needed to run agile methodology to the letter must be built over time. 

Most agile software development methodologies like scrum almost exclusively revolve around the development iterations or sprints. In most of the project I run, this methodology only supports a part of the job a web development project contains. An important part without doubt.

Scrum and XP use User Stories to communicate the need of the customer, and the customer keeps these stories in prioritized order according to business need. Most times developers take rather eagerly to these development methodologies, as they supply autonomy for developers and value customer participation. Surprise: Most developers actually want to understand what the customer needs :)

Customers seem to like the idea that they on a regular basis can view for them selves how progress is coming along. And most customers like the fact that a backlog is not a technical document but is business focused. There is already to much "magicy", intangible about the development part that most customers feels falls rather far outside their field of both competence and interest. And I guess no one wants to feel their doing something their incompetent at when spending several MNOK for a new website. The backlog is something they can understand, and they can verify when stories get done. Combined with wireframes that tell how these textually written features will function on a website things get tangible! I for one like that. 

In my world however there is a bit more to creating successful web projects than development from a backlog, no matter how well it conforms to agile methodology. Websites are all about sticking out and catch a user attention span that ranges in just a few seconds. Its about organizing information and services in such a manner that the goals of the customer is met. This is a science of its own, and there are several methodologies to go about creating a good user experience. Scrum is not one of them, nor was it ever supposed to be.

In short; Web development projects tend, in my experience, to have two distinct fazes where the first faze is about creating the backlog and prototype. The other part that is all about iterations and development, and being agile... is it too late in the project lifecycle?

The linking of these two worlds of web development is something I still have no luck finding literature that address. The closest I get is the general idea in Agile that everyone is part of the team, and can do all tasks in the team. It is not feasible for a number of reasons.

One reason is that due to the completly difference in competences between designers and developers means that they cannot do each others job, a key aspect of the team. 

Secondly they also need to do work before the developers know what to make. Meaning they need to be done, before the developers can do sprint planning. This indicates that they should be part of the specification team (part of the product owner team). I have tried both. 

In most projects the customer wants to complete the prototype before development starts in a nice and waterfallish (not to mention costly) way. In most cases the interaction designers (who make the prototype) have never heard of a backlog. This results in a situation where the only input the developers get is the prototype. This leads to frustrated developers making speeches as agile conferances with the titles such as "Prototypes are evil". Of course training can amend some of this lack of understanding. I am also aware that others may have differing experiences. I urge you then to please post them for others to learn.

The designers (information, interaction, graphical) concept developers etc are often used to complete their job before the developers get a go. The big specification up front is then present (a sin in agile thinking).

To start the agile process at this late stage when the big specification already are in place, is of course possible. Moreover it is better than moving into a black box type of setting. If for no other reason it makes it possible for me as a PM to get a good solid grasp on actual progress, as opposed to an over optimistic (or over pessimistic) guess. It is possible to identify problems with the initial design (there always are), and do something about it in order to find a solution that is a bit more cost-benefit oriented for example.

In web development especially this "problematic" design faze is critically important for success. No Backlog in the world will suffice to replace it, nor will a prototype ever be a backlog. So there is only one road left to walk. The gap has to be closed somehow, as it brings with it a huge risk in fix priced projects, or causes the customer to pay to more than needed, and spend more internal resources than necessary.  

More to come on how I go about closing the gap and create lean agile projects delivering solid ROI to customers, both successes and failures :)