one major problem JS and vanguard is NOT systems thinking. At best it is a hard systems approach derived from Deming/Shewart. The approach utilises analytical tools within a bounded deterministic system (activities) of work. The approach contains no ST concepts and ideas.

Thanks. I get you. I agree about outcome measures. I think the opposite though about “qualitative data” (employee satisfaction), and have found it a very useful tool.

I’ll read the case study in more detail. I don’t see how the quantitative data was used to “guide” improvement. I don’t see evidence that a relationship between cause and effect had been established.

I’m thinking about attending the conference in Atlanta, in which case I’ll get to talk to David in person perhaps.

Thanks for clarifying your thoughts.

Paul.

We seem to be talking past each other and reaching the point of diminishing returns – so one final attempt to clarify what I’m saying

1. Process or activity measures are not helpful.
2. Result or outcome measures are helpful.

Lead time or quality are the result or outcome of a process and not the process itself. Therefore, they are useful in helping a team improve its process.

While customer or employee satisfaction data would be interesting, I don’t see how they can help guide a team to look for *where* they can improve their process.

Karl

Again, I’m not trying to dam Davids work. From my own experience, I have found it very difficult to correlate cause and effect when using process metrics to measure software. This doesn’t mean that what happened at the BBC wasn’t effective. It just makes it very hard to put it down to any one specific cause. David is clear that it is the result of measuring and lean thinking. This is obviously a subjective opinion.

The social sciences have the same problem. How to assess a system that is so complex that it is hard to quantitatively relate cause and effect. What they do is collate qualitative data from as many people as possible and look for statistical significance.

Such qualitative data would be irrefutable (for example 20% of the developers strongly agreed that measuring cycle time had a positive effect on performance is an irrefutable statement).

Paul.

Customer and team surveys would have been interesting, but I don’t think their omission is a valid reason for rejecting the case study as invalid.

Karl

I wasn’t trying to dam Davids work. I’m merely pointing out that his case study doesn’t pass as scientific proof.

I still think a survey of the customers and dev team members to assertion their impressions of the approach taken would add significant weight.

I guess you disagree.

Paul.

Just read through David Joyces case study, and it makes the same mistake virtually all studies like this make. The focus is on process metrics. So David as decided what things in the process are indicators of success and has measured those. Of course there is an assumption here. Also there is a tendency in practice to “get what you measure”, which means that your process metrics will tend to improve over time. We know nothing about the things that were not measured, and whether they could be adversely effecting the overall result.

What is more important (an also more elusive) to measure is the actual result. So a result metric would be return on investment say, or customer satisfaction. These are the things that the organisation is looking to get back in return.

So what we are really interested in is the “value” of the process improvement initiative from the perspective of the organisation an its customers. It would have been great if David had performed a survey of the stake-holders and customers involved and got them to assess the “value add” from their perspective. A similar survey amongst the development team would have been useful too.

Whilst this would provide only qualitative data, it at least could be deemed to be results based, rather then process based.

I’m sure Mr Seddon talks about the difference in his book.

Regards,

Paul.