State of the art SW R&D

Practice makes perfect and successful System design happens in the trenches close to the action. Methods and tools that does not originate from involvement in the day to day struggle simply lack credibility. Technology has to withstand the test of implementation details and unexpected complications.

We plot a course, set a heading but have to be able to deviate as plans break down when we navigate in uncharted waters. Experience let us read and explore the environment and compare with on our previous journeys. The Agile challenge is to work with open ended designs that can tolerate shifts and take advantage of opportunities. This requires a whole new take on architecture and methodology.

Beyond lean

How do we stay lean without starving innovation and progress. Computer science is an immature engineering field compared to mechanics and electronics. The innovation rate is higher than in any other engineering field. Developer skills erodes fast in this environment.

Tuning and trimming down administration offers gains and but an upper limit. The SW industry progresses in  productivity leaps. The human spark is ever present to adapt and exploit the environment. SW engineers have to be a new breed of engineers. They study and evaluate SW innovations from a productivity perspective.

Read more

Sustained agile development 

Agile methods emphasize collaboration and demonstration of working software. It optimizes the feature flow and prevents over-budget, over-engineered systems. It is healthy to start small and purposeful and seize opportunities as they appear. 

In the other hand it is futile to polish a lab-grade proof of concept SW into a product. Promises and hope gradually turn into to a maintenance nightmare and quality liability. The agile road is winding and the roadside is full of decommissioned SW that broke down due to architectural debt.

The failure of traditional methods was their dependency on arbitrary estimates as input. The complex reality amplify the input errors causing forecasts to break down. Project turns into crisis management. Sustained agile development requires that problems are prevented to grow out of proportion. Systematic use of performance indicators let us forecast problem areas before they become major problems.   

Architectural debt is a concept SW developers are familiar with. It is built up as every new function makes the next a little more complicated to add; it builds up feature resistance. It is critical to invest in architecture at the right time. Eventually products gets starved when new functions takes to long to reach the market and revenue drops.

Feature resistance is proportional to complexity in the form of code dependencies. The ability to handle this is subjective to the skill level of the people working it. Complexity is not a single system metric, it is unevenly evenly distributed in making some parts more complex than others. Simple sub-systems have a larger set of developer candidates that can work on it.

The way to handle this is to systematically measure the complexity and performance on subsystem level.

 

Agile Architecture

 It is an important principle of the agile manifesto to never compromise accountability through coercion or dictation. That clause actually invalidates all un-invited external influences, in reality agile often relies on a few individuals ability to design viable solutions "on the fly". This does not scale well.

SW craftsmen or teams left on their own evolve their own arbitrary solutions to common problems, thereby reinventing the wheel. This causes variations and an inconsistency throughout the system, besides the teams lack resources and incentive to create robust reusable solutions.

Self similar systems have similar solutions to similar problems, in a deeper sense than a code standard look alike. Self similar code is simple to understand and is therefore developer neutral. It is vital that a major evaluation criteria in the agile feedback back loop has self similarity as a performance indicator.

Traditional architecture based on requirement elaboration is incompatible with Agile!  Dictation would compromise accountability. Agile favors collaboration and views requirements as weak and shifting entities. These are highly unsuitable as a foundation of anything. The days of the omnipotent architect that dictates and governs all with the power of requirements, is over.

The agile architecture is NOT a product of requirement elaboration. In the Agile team centric world the architect is a service provider that brings solutions to the production line. The incentive is that it makes them more productive ! It is the DNA of the system instead of a detailed description of the final result. This can create arbitrarily complex systems by weaving and transforming relatively simple parts.

The agile architecture accommodates features and requirements, and most will fit out of the box. The architecture is in a since an evolutionary envelope in which the application(s) evolve. This envelope is expanded when it fails, or is going to fail, to accommodate a new feature. This creative process is a pull force as developers extends the architecture by promoting local solutions. The architect harvests, incubates and syndicates these without depleting the team resources.

Architecture is an integral part of the development process of every system. It provides both the scaffolds and the building blocks of a design. By managing complexity we can gain productivity.

Read more

 

Scale up

Every person added to a team slows down the others a little. It is a fundamental property of human social behavior. Even the most socially challenged programmers have need to coordinate their work. The only way to defeat this hydra is to reduce the need for coordination by creating silos. The most common way it is to set up self sufficient teams organized according to the structural architecture.

The problem with to small units is that there is not sufficient jobs to support specialization. The team members are to some extent expected to be universal programmers, which is a rare talent. The days of the matrix organization with specialist roles diluted across many projects is not viable any more.

Talents soon tire of burning features off the backlog, while the scrum master oils up the hamster wheel. There is very little computer science challenges in feature implementation. The real challenges are found in the abstract components and possibly few hard feature problems. 

Craftsmanship and product understanding takes time to develop. Knowhow is always in short demand and under pressure. This makes patience with entry level staff at best short. Simply assigning new members a complete feature is not viable. Breaking out and deskilling tasks within a feature or story allows us to concentrate the resources on qualified tasks. Rather than diluting specialists between projects we augment their output with junior resources. This is an essential shift to make SW production and maintenance cost efficient in an agile environment.

A single feature contains several levels of inner difficulties, once we figure out how to solve it, the rest may be trivial. Sr - Jr cooperation frees up bottleneck key skills, allows everyone to contribute and increases the talents output.  Working in pairs allow the programmers to divide work between themselves just as Rembrandt did in his days.

Read more