Theory of Constraints and Agile Project Management

Recently on the LinkedIn forum PM Toolbox, Arash Sadati asked for comments about tool support for how Goldratt's Theory of Constraints (TOC) or Critical Chain Project Management can be integrated as part of the Agile Project Management. Here are my thoughts on the subject.

One thing all agile projects share is that - because of a conscious removal of dependencies between features wherever possible - they will be resource constrained rather than critical path constrained (a generalisation but broadly valid). Methods like Scrum-XP try to avoid specific role constraints (e.g. we can't make progress because we don't have a BA or a GUI expert or a middleware expert available) by specifying only 3 role types (Product Owner, Scrum Master, Team Member). So the team as a whole must to some extent be generalists (or at least prepared to learn / fill in specific roles) to reduce the risk of role constraints. In any real application of Scrum of course such role constraints may be real and teams will have to pay attention to them. The default condition though is worth understanding because it's a means to keep the effect of role constraints to a minimum.

FDD defines many more roles and I think as a result there is a greater risk of role constraints. Availability of specific class owners for example is a typical problem which Palmer and Felsing discuss in their book on FDD. Flexibility, especially in small teams, for people to wear multiple role "hats" is crucial. The really interesting observation in comparing agile to waterfall is the requirements process where detailed specification of features is ideally "just in time". David Anderson and others point out that it is a serious waste of precious resources to allow too much work to be done on future requirements when the critical constraint is the development team.

When we were specifying the xProcess product which is designed to support any priority-driven process (any agile process in other words), we did want to be able to give visibility to problems arising from role or resource constraints. Patterns of tasks can be defined in custom processes that require specific or general roles - or even allocation to specific team members - so that forecasting can take into account not only the total development effort available, but in the cases where specialised roles are required. It can therefore show the impact of limited availability of critical roles. Custom reports can be generated to show resource utilisation, and where these show less than 100% utilisation, the situation is likely to be one where specific roles are "critical", requiring other roles to wait for them.

FDD sub-processes in context

Like most agile processes FDD doesn't pretend to cover every aspect of the software development life-cycle. Just the essentials for establishing a common approach. In particular FDD doesn't explicitly state how and when releases of software occur during or following an FDD iteration.

Here's a diagram that attempts to put the releases in context and also shows the distinction between the first 3 sub-processes which define the project (or iteration), and the multiple instances of the 4th and 5th sub-processes which develop the project (iteration). There can be just one release at the end of an FDD iteration or (more normally?) several releases which deploy the developed software.

What's different about Folders?

When you're deciding on the structure of project patterns for a new process, often the first thing to consider is how to represent the different hierarchies of elements that make up projects.

In FDD for example there is the hierarchy of features, normally described as being Features, which are contained in Feature Sets, which are contained in Business Areas or Major Feature Sets. This logical grouping of functionality is independent of priority and time-ordering. It may well correspond to the structure of the user manual for example or the Functional Specification should such a document be required. It doesn't give us any time view. On the other hand the hierarchy from Releases -> containing Timeboxes -> containing Features is a time-based structure. Similarly the hierarchy based on the five subprocesses of FDD also will have a time correspondence: The Definition Stage -> containing FDD#1 Develop Overall Model, FDD#2 Build Feature List, and FDD#3 Plan by Feature; The Build Stage -> containing many instances (for each feature) of FDD#4 Design by Feature and FDD#5 Build by Feature; and the Release/Deployment Stage following completion of a release.

Each of these hierachies is an important view of the tasks and artifacts in the project. However when defining a process in xProcess you must decide which hierarchy will define where a task "lives". This is the parent-child hierarchy of tasks and is analogous to a directory structure which tells you where a file is located. All the other relevant hierachies can be represented with Folders which contain explicit or implicit (defined by rule) references to the tasks. This means that different features may appear in different Timeboxes and Releases, and indeed different subtasks of a feature may appear in different folders (design and build for example may appear in different timeboxes; build and certify a feature may appear in different stages). Since Folders may contain Folders in xProcess as many hierarchies can be provided as are required to understand the process, each to several containment layers if this helps understand the view.

So Folders provide an alternative hierarchy of tasks to the main parent-child structure. Their membership is defined either explicitly (because the process has put them there or the user has dragged them into the Folder) or implicitly (because the task has a matching category to the category of the Folder). In some methods it is sensible to use the parent-child hierarchy for time-based structure (Basic Scrum 3.0.1 follows this scheme for example). In others using a more static structure such as Business Areas/Feature Sets in FDD or a Work Breakdown Structure in traditional methods, gives the preferred structure for parents-children with more dynamic hierachies being represented with Folders.

Process patterns: what are "composites"?

The key to defining the task structure in a process definition is the task pattern. Task patterns consist of prototype tasks, optionally contained in a prototype project. Here's an example:

This is the task pattern for a "Feature" in the FDD process and it results in the creation of a set of subtasks and artifacts when the Feature pattern is "instantiated". This is quite a simple pattern and it doesn't include any composite tasks, which are structures to allow multiple tasks, optional tasks and iterations of tasks (repeated tasks that follow on one after the other). It is composites that allow arbitary process diagrams, drawn say using UML's activity diagram format, to be translated into xProcess processes. Here's an example of a task pattern, this time shown in the Hierarchy Diagram view, which does contain a composite. It's the "Feature Set" pattern from the same process.

The composite task in this example is called "Features: $Name$" (this is a pattern so the $Name$ will be substituted by the user supplied name when the pattern is instantiated). It contains a choice of 3 patterns: "Defect", "Feature" and "Task". Any of these patterns may be instantiated multiple times within the composite task.

Composite tasks are parent tasks, in other words they do (or will) contain child tasks, and they also define constraints on what type of child tasks can be created through the patterns they contain. In the latest version of xProcess composites can be instances (only one pattern can be instantiated in the composite), collections (an arbitrary number of patterns can be instantiated in them) or iterations (similar to collections but each pattern will be constrained to start after previously instantiated tasks have completed). Composites contain one or more patterns which define the options available when creating new tasks in the composite. In the example above, Defects, Features or Tasks can all be instantiated within the composite.

So the composite mechanism gives the Process Engineer the ability to define task patterns with loops, sequences and optional paths. More or less any pattern that can be represented in a flow chart or activity diagram can be created in an xProcess pattern by using composites. Composites go further though. Because of xProcess' support for top-down estimation, composites have their own estimates and resource requirements so that they can be scheduled even before the detailed decisions about how many tasks, iterations or options have been made.

To understand how these top-down estimates work in practice, see this blog article: Latest Patterns for Scrum which shows how the top-down estimate for a Sprint is used for scheduling resources, up to the point that all the backlog items have been created within it.

Footnote Question (for the techies among you!):
Are all parent tasks also composites as well as vice versa?
Well firstly all composites are parent tasks. Even if they as yet contain no child tasks they are classified as parent tasks because they allow task patterns to be instantiated in them. On the other hand parent tasks are only composites if they contain a set of patterns. This set defines the patterns that can be instantiated in the composite /parent task. So a parent task without such a set of patterns is not a composite and (in version 2.x) any non-hidden task pattern maybe instantiated in it.
[Note in version 3 this behaviour is likely to be changed, effectively removing any distinction between parents and composites. If the parent task does not contain any patterns, no patterns will be able to be instantiated in it. This will allow Process Engineers to create project patterns where, for example, certain tasks have a fixed set of children which cannot be added to.]

Searching the blog by topic

Blogspot offers some useful facilities for finding blog entries by topic. Below most entries there are a series of labels. Clicking on any of these will bring up other related articles on the same topic. Try it our with the labels below this post.

Latest patterns for FDD

The release of 2.8 this week sees some interesting updates to the FDD patterns. Like the Scrum updates for xProcess, the new FDD process uses "composite tasks" to define the structure of FDD projects, allowing them to be expanded top down as the details of feature sets and features are defined.

This hierarchy diagram shows the structure of the FDD project pattern. For those familiar with the method, the 5 stage process is visible in the top-level tasks:

1. Develop an Overall Model
2. Build a Features List
3. Plan by Feature and
   
4. combined with 5. Design by Feature - Build by Feature.

Note that the darker blue nodes are patterns, so these can be further expanded in the hierarchy diagram or viewed in the graphical pattern editor.

Here for example is the hierarchy view of the Feature Set pattern, which groups together a set of related features into a deliverable package which can be scheduled for a particular release. Note that here the "Features" composite task, as well as containing Features, may also group Defects (work on correcting issues from previous releases) and Tasks (any other work related to the Feature Set).

The Feature pattern has already been the subject of a number of postings on this blog (see for example...). The pattern diagram shown here uses a quite traditional view of the tasks required to deliver a feature with the emphasis on Specify (the "design by feature" part) and Develop (the "build by feature" part). We can see more detail of this task pattern by using "Go Into" in the pattern editor. Here the specific artifacts and quality gateways that are associated with each task, as well as the number and types of roles required to carry out the task, can be reviewed and modified.
Clearly other ways of breaking these tasks down are possible, and some of the FDD literature (for example Palmer and Felsing's book Feature Driven Development) provide alternative schemes that place stronger emphasis for example on design and code walkthroughs rather than the test-driven approach implied here.

The beauty of xProcess is that the task patterns are all easily and graphically editable so you can make this method match exactly what you want the FDD teams to carry out. As all plans, tasks, artifacts, processes and time records are stored in the xProcess versioned data stores, the compliance with your process can be monitored at any time, either to improve the process patterns where teams have discovered better ways of working, or to improve the teams' approach by following the best practice captured in your process patterns.

FDD has a three-level hierarchy of functionality: Features, Feature Sets and Major Feature Sets (also referred to as Business Areas). So far we've seen where Features and Feature Sets appear within the hierarchy of a project. Major Feature Sets (MFS's) are handled slightly differently. MFS's are created by an xProcess pattern in a similar way to the other patterns we've looked at, but they result in categorized folders rather than tasks which are part of the project hierarchy. This is so that the Feature Sets are visible within the Gantt charts and release schedules. Because of the scope of major Feature Sets, they generally do not have such a clearly delineated start and end date and so it would not make the project schedule clearer to include them in that way. So opening a MFS folder shows you all the feature sets and features in that category. You can then review and prioritize the features in just this one business area. Here's an example of a MFS in a particular project with its corresponding Gantt chart (click on the diagram to see more detail).
Finally in this brief review of xProcess FDD, the pattern for a Release is also instructive. Again a prioritized folder is used for the Release pattern, rather than a task in the project task hierarchy (see the Scrum method for an example of a process that uses this alternative). By default just one release will be created in the project (with a target close to the end of the project). However the Project Manager can create as many additional interim releases as he requires. The "scope" of the release is defined by moving the required features/feature sets/tasks into the release folder. The scheduler of xProcess uses the implied and defined priorities of these tasks to calculate completion dates and provide alerts if targets are unlikely to be met. As the project progresses, the scheduler uses the input from team members completing and possibly re-estimating the required effort for tasks to give immediate visibility of targets' status and costs providing all stakeholders of the FDD projects timely and detailed information to support decision making and further planning.

New project Pattern for FDD

The project pattern for the Basic FDD process, which is delivered with xProcess is changing in version 2.7.2. Here's a preview.


This puts the emphasis back on the familiar 5 stages of FDD at the top level (Develop an Overall Model, Build a Features List, Plan by Feature, Design by Feature and Design by Feature). The first 3 stages are included in Timebox 00 which covers the start up period of the project and Features and Feature Sets are created as subtasks of combined final stages: Design by Feature - Design by Feature. Major Feature Sets are modeled as Folders with membership defined by Category.

3 Types of Requirement

Priority-driven processes only work if you can model the process around task patterns for a single requirement. There is little point in prioritizing say, coding over testing, or design over specification, since all these activities must take place at some stage, even if they happen inside an activity that is called something else. But are all requirements the same (in the sense of using the same processes) or are some requirements more the same than others?!

Recently I was asked to advise on the requirements capture process for an organization wanting to apply agile techniques in a controlled environment. I came up with some slightly different names for types of requirement, though I think the concepts will be familiar to you if you've analyzed other software development methods. The requirements types (shortened to PCF) are as follows:

• Business problem statements (Problems)
• Solution contraint statements (Constraints)
• Solution feature statements (Features)

I've written elsewhere about FDD, which provides a very good process for managing and prioritizing features. However often projects applying processes like FDD do not pay enough attention to the other types of requirement that precede features.

Problems, for example, should identify issues in terms of a measurable aspect of the current solution and the degree to which a performance improvement would overcome it. Constraints express aspects of the solution that the current designers are not expected to change (though they should also state the rationale behind the constraint, and how and by whom it may be changed if justification exists). Only by considering all of these three types of requirement can the essential requirements of a system be captured.