I say that because a number of researchers have invested time in studying the maintainability of software as an after-the-fact question.  These researchers typically focus on the wide variety of complexity measurements that have been developed, e.g., Halstead’s Volume measurement, COCOMO Cost Factors, Lines of Code, Module Cohesion, Cyclomatic Complexity, Comments Volume, Average Lengths of Variable Names(!)  I’m sure there are lessons to be learned from these things but they are not strictly aimed at being prescriptive.  If you’re interested in this topic I can refer you to Defining and Measuring Maintainability by R. Cheaito et al.

(Well, you may be curious: A study for the Canadian Space Agency reported that longer variable names are more maintainable.  Another study reported that both low and high comment volumes lead to more difficult maintenance, the latter possibly indicating lack of focus and tending to impede code reading comprehension.)

So anyway, what do you do to create a maintainable system?  A number of researchers have broken the answers down into two parts:

• Characteristics of the software

• Characteristics of the processes used to create and maintain the software

You may scratch your head a little at the second point.  But one model for thinking about this was described in Maintaining Maintainability by Ramage and Bennet.  They describe the various actors in the software process as managers, users, business processes, tacit knowledge, designers, maintainers and the software itself.  These authors argue that, from the organization’s point of view, the maintainability of the system is heavily influenced by the relationships between the various actors.  Poor relationships between users and maintainers results in lower quality maintainability.  Similarly, the maintainers often rely on designers for an understanding of concepts and intentions in the system design, and a poor relationship there will also cause maintainability to suffer.  The authors of this approach offer a formula for measuring organizational maintainability.

The other perspective on how processes relate to maintainability concerns how adherence to maintainability guidelines and quality checking are implemented in the organization.  A complete model of this includes a maintainability quality model, a set of source code guidelines, a source analysis tool, the analysis results profile of the source code, and a quality engineer to interpret those results and the source code itself.

Next: characteristics of software that improve maintainability

• Continuous process improvement
• Continuous product flow via reasonably sized projects  constantly moving through the pipeline
• Planning for uncertainty by building adaptation to change into the process and plan
• Focusing on the software being usable and well-directed toward its intended purpose
• Creating software that is maintainable
• Creating projects that have frequent deliveries and opportunities for feedback
• Improving quality by early defect detection, thus minimizing unplanned rework

Some of these techniques may be self-explanatory and others require more thought and consideration.  For the first, Continuous Process Improvement, the fundamental requirement is that you’ve got to be measuring something to know if you’re improving it.  More importantly, deciding what to measure is the result of something even more important: you’ve got to decide what is most critical.  In the remainder of this post I will give an example of a metric for continuous process improvement.

A good place to start for AS/400 RPG maintenance shops is to do some Pareto analysis on what source code is most frequently changed.  Pareto analysis basically leads to the “80-20” rule and is accomplished by listing items in descending order by the key measurement (e.g., lines of code changed in the last three years) and having another column showing cumulative figures.  Analyzing the result often reveals that, for example, “the top ten programs account for over half of our source changes.”  A spreadsheet might look something like this (LOCM = Lines Of Code Modified/Added/Deleted):

The information to do this analysis can be obtained in various ways.  Rough figures can be obtained by analyzing the dates in your QRPGSRC or QRPGLESRC files.  Depending on your maintenance practices this may or may not give you counts of deleted lines.  If you have a change management system you may be able to obtain statistics from that.

If this analysis is too difficult, a more coarse-grained approach, if you have a change management system, is to measure the number of checkouts or promotions.

Understanding this information can help you identify opportunities for improving your throughput or quality.  I would start with the top program, “CS150R”, and determine what projects and fixes had driven so many modifications.  Were there continuous change requests from users?  If so, perhaps there are some things that could be soft-coded to reduce future changes.  Were there a lot of defects?  If so, perhaps the program could be improved with some refactoring. Or perhaps I need to be more careful about who I assign to work on this program.  Or maybe we need to do some training or documentation about this program.

This is an example of a very basic analysis that most RPG shops can execute with minimal effort, but good potential payback.  Another very valuable analysis is to look at the defect rate per program, if you have that information available.

What is interesting about this is that it shows the ever growing heap of software that must be maintained.  The duration of software maintenance on an application easily exceeds the duration of its original development.

Why is this important?  Two reasons:  1) when planning new software development I think it should be becoming very clear that organizations must build solid maintenance plans into the development plan.  In particular, maximizing maintainability of the software; and 2) AS/400 iSeries IT management needs to educate execute management about the long-term cost of software and seek to maximize the long-term value of the software through effective maintenance.

Can the analysis process that maintenance programmers engage in, so called program comprehension, be measured?  Is it important to measure it?  If it can be measured, is it then possible to improve it?

Some studies have shown that program comprehension takes from 40-60% of maintenance programmers’ time.  That indicates pretty clearly that this is worth measuring and improving if there is a way.  But is there a way?  There seems to be remarkably little research on this question despite the fact this is arguably the single most costly task in the software lifecycle (i.e., it consumes 40-60% of maintenance programmers time, and maintenance programming consumes 80-90% of lifecycle costs.)

Let me first define what I am including under the title “program comprehension” for purposes of this post:

• Building mental models of the program’s control flow and data flow, sufficient to carry out a given maintenance task, i.e., comprehending the program
• Locating the code that needs to be changed for the task
• Defining the changes for the task and performing the impact analysis for those changes

Can these things really be measured? I don’t know if there’s a direct way to measure these things, but I think that there may in fact be a way to get some related useful measurements of the aggregate process without excessive effort if adequate code analysis tools are available.  My basic idea is this:

• Measure the quality endpoint after the code is completed, in other words, count the post-coding defects found – whether found by review, QA or in production.
• Calculate the complexity of:
  • The code that was added, changed or deleted (deleted code might need to be weighted downward)
  • The complexity of the existing code at the changed locations; this should indicate how complex was it to analyze.  This is the calculation that I believe factors in the “program comprehension” aspect.

It would take some experimenting to determine the best complexity measurement for this purpose, but let’s say that for starters you simply used McCabe’s Cyclomatic Complexity measurement within the function(s) where the change was made.  In addition to this I think you would have to also include a fan-in metric to give weight to the backward program slices that needed to be considered for the change.  Additionally, a computation for forward program slices needs to be considered to give weight to the impact analysis of the changes.  More generally, however, this may simply require some sort of measurement of both the backward and forward program slices.

I know I’m throwing a lot in the pot there, but in theory this is all automatable with a good tool (my own tool, Codelyzer, for example, can do a substantial portion of this for RPG.)   The point here is to find a way to measure the work that is done in programming maintenance.  Having such a measurement is a key first step towards identifying opportunities for improvement.  I think a measurement of this type begins to fit the actual mental process and effort a programmer requires to make a change.  Other people, such as Capers Jones, has propose extending the use of function points into maintenance measurement, and in fact has proposed “micro function points.”  As much as greatly admire the vast amount of original analysis and work he has done, I have to disagree with him on this point.  Here’s an example of why I disagree:

While this is a relatively simple example I think the point is clear – analyzing code that is be added, changed or deleted requires analyzing the conditional logic that applies, and this logic can get quite complex and time-consuming to analyze.  That is why I say the locations of the code changes must be factored into to any attempt to measure maintenance work effort.  This conditional logic applies within the function, or subroutine, as well to all the calls of that subroutine, i.e., calculating the backward program slices.

While this is certainly not a complete prescription for measuring maintenance work I think it does begin to describe what a solution might look like.

There are a number of problems associated with estimating maintenance work (by which I mean enhancement projects as opposed to production support fixes.)

A summary of these problems includes:

1. In some cases analyzing the programming changes to be made can comprise half or more of the total programming work.  How can reasonable estimates be made without doing half the work?
2. There are no comprehensive measurements of maintenance work.  New development estimates can be made fairly reliable through the use of function points, but there are several problems with the use of function points for estimating maintenance work.  When can function points be useful, and what can be used when they are not appropriate?
3. There are no reliable means of factoring program complexity into the estimating equation without, again, completing the analysis work.  A change in one location of a highly complex function maybe be simple, but a change in a different place might be difficult – how can this be measured and factored in?
4. Absent the ideal analysis tool, there is no clear means of identifying the full set of required regression test cases.
5. Programming estimates are always affected by the knowledge and skill of the particular programmer doing the work, but maintenance work is even more highly leveraged by these factors.  In particular, domain knowledge of the business, application and program are critical, as well as knowledge of programming patterns used in the application, and the history of changes to the application.  How can the knowledge of these be measured and factored into estimates?
6. For some types of maintenance work there is no apparent way to measure the amount of work that was performed – this impedes the effort to measure productivity and improve future estimates.  To make a difficult example, say the maintenance work that a programmer did consisted of reorganizing a function of 100 statements so that it executed correctly and more quickly.  The programmer just reorganized and sequenced the same set of statements – how could this be measured?

Following are my comments on the first of these problems – no complete solutions here, and sometimes just more questions.  As someone said, the closer you look, the more you don’t know.

1. In some cases analyzing the programming changes to be made can comprise half or more of the total programming work.  How can reasonable estimates be made without doing half the work?

This problem needs to be broken down for finer grained analysis.  In his book “Estimating Software Costs” Capers Jones defines five types of maintenance work:

• Code additions without internal changes
• Additions with internal changes
• Replacing an existing feature with a new feature
• Adding new features accompanied by scattered updates
• Scattered updates, deletions and concurrent changes by multiple programmers

I think the most difficult estimating challenges occur in the types that involve changes to existing code.  Estimating work that is essentially new code added on to the application can mostly be estimated simply as new work with interface and regression testing requirements.  Mr. Jones suggests that the difficulty of working with existing code can be alleviated – sometimes – by the use of specialized tools.

So the most challenging problem here is how to estimate changes to existing code.  As I said earlier, the first challenge of that is to locate all the code to be changed – and this can be a significant portion of the work.  How can this be estimated?  Without an ideal reverse engineering tool that is full of domain knowledge that can map change requirements to specific code, I don’t know of a way around doing this analysis work.  I suspect the best solution to this problem has to do with optimizing the project processes and management expectations around this unavoidable fact of maintenance life.  But it is it provably unavoidable?  That’s an open question as far as I know.

More on the sub-problems of maintenance estimates in the next post.

Let me first clarify the term “maintenance” as it is very ambiguous – in this context what I am referring to is software “enhancement”, aka, “adaptive” maintenance according to the ISO 14764 specification.

Here are the key unsolved software maintenance problems as I see them:

Estimating – how can a modification be estimated without first thoroughly locating and understanding all relevant code in the existing application?  Doing so is, of course, often a large portion of the work.

Tracking, measuring and improving key tasks – many studies have shown that program comprehension can take as much as half of a maintenance programmer’s time.  Program comprehension is mostly an internal mental task: how can it be tracked, measured and improved?  Or even estimated?  If this can’t be tracked, measure, improved or estimated, well, there’s half the budget right there beyond the control of management.

Methodology – ISO 12207 is a specification for the complete software life cycle.  It contains a process for software maintenance which breaks down into further tasks and activities.  However, there is no mention, again, of program comprehension – half the work of maintenance programmers.  What would a methodology look like that included this key activity?

Part 2 of this questions is whether an Agile-like methodology can work with maintenance projects.  I have written on this subject before, here, which also connects to further posts I have written.

Training – Software maintenance is a specialized skill, and by all accounts more difficult than new development.  What would an ideal training curriculum contain for maintenance programmers?  One of vLegaci’s offerings is maintenance training, but I regard this as a work in progress.  I have searched extensively for some theoretical grounding for this topic and have found very little.

Regression testing – There are two problems here, 1) how can you automatically scope the required regression testing, and 2) in the absence of a well-documented system, how can you execute thorough regression tests inexpensively?

IDE/IME – IME is my name for Interactive Maintenance Environment.  Perhaps other platforms have the maintenance equivalent of an IDE, but I am not aware of them.  There are many features that could be very valuable to maintenance programmers.

Defining the Efficacy of Available Resources – following are some things that effect work estimates – how do you quantify them for estimating purposes: quality of documentation, relevant knowledge and skill level of maintenance programmers, coverage of existing regression tests, actual maintainability of target code, etc.

I will tackle each of these subjects in following posts.

Consider first the goals of measuring these processes.  The goals of measuring such work are that the measurements should be:

• Translatable into process improvements and increased value delivery
• Cost effective to capture
• Demonstrably consistent
• Automated wherever feasible
• Traceable, in two respects: from summary to original point of capture, and from one point in a project to another
• Parallel, that is, measurements should line up for the same time periods, projects, resources, etc

Let me emphasize the first goal: without increases in productivity, quality or value all the other goals have little meaning.

In his book, “Applied Software Measurements, Global Analysis of Productivity and Quality” by Capers Jones, the author expresses some exasperation that in over 50 years the IT community has not yet produced a standardized “chart of accounts” for software project activities.  I think the concept of a standard chart of accounts for project tasks makes a great deal of sense, and, even if the IT world as a whole does not standardize, surely a given IT organization should do so.

In his book he references two charts of accounts that have been developed, one by an organization with which he is associated, Software Productivity Research (SPR), and one by IBM.  For most business IT organizations (referred to in the book as “MIS”) the SPR chart of accounts seems more appropriate.  It contains 25 top level tasks, which SPR has broken down into as many as 150 subordinate tasks.

Here is the SPR chart of accounts of project tasks:

1. 1. Requirements
   2. Prototyping
   3. System architecture
   4. Project planning
   5. Initial analysis and design
   6. Detail design and specification
   7. Design reviews and inspection
   8. Coding
   9. Reusable code acquisition
   10. Purchased software acquisition
   11. Code reviews and inspections
   12. Independent verification and validation
   13. Configuration control
   14. Integration
   15. User documentation
   16. Unit testing
   17. Function testing
   18. Integration testing
   19. System testing
   20. Field testing
   21. Acceptance testing
   22. Independent testing
   23. Quality assurance
   24. Installation and user training
   25. Project management

Mr. Jones makes the point that most projects do not contain activities for all these tasks, and in fact many MIS projects routinely succeed with 6 to 12 tasks (whereas military projects, for example, tend to contain 15 to 25 tasks.)  And again note that these tasks can have a number of subtasks.

The data that the author suggests should be collected for each of these tasks is the following:

• The size of the staff for each activity
• The total effort for each activity
• The total cost for each activity
• The schedule for each activity
• The overlap or concurrency of activity schedules
• The deliverables or work products from each activity

These activities and measurements form the structure for the initial estimates and project plan, and then the tracking of actuals against the plan.

Another key aspect of this is that function point metrics are captured for each project.  These then become the basis for subsequent analysis of activity results against function points to enable improvement of future estimating and planning.

What About Agile Projects?

Given the distinct nature of the Agile methodology a different chart of accounts is proposed:

1. 1. Initial planning
   2. Test case development for initial iteration
   3. Scrum session for testing
   4. Development of initial iteration
   5. Scrum sessions for development
   6. Internal testing of initial iteration
   7. Scrum session for internal testing
   8. User documentation of initial iteration
   9. Scrum session for user documentation
   10. Release of initial iteration
   11. Scrum session for release
   12. User acceptance of initial iteration
   13. Scrum session for acceptance test

For those not familiar, in Agile projects the above activities represent one iteration of a 2-4 week development cycle that will be iterated until the project conclusion; a scrum session is a quick discussion by all team members of 1) what has been done since yesterday, 2) what is to be done today, and 3) what impediments exist, if any.

In this book, which as I have said before is a truly excellent reference of actual empirical data from over 12,000 software projects, the author does some in depth analysis of the results of different project methodologies.  His analysis concludes that different methodologies are more successful for different sized projects.  I will write more on this in a later post, but a quick summary is that for small projects Agile is highly successful, and for larger projects TSP/PSP and CMM are more successful.  The author goes on to point out that if Agile is used for larger projects then additional tasks need to be added for such things as change control, integration and so on.

A key question, not addressed in this book, is whether Agile can be made to work with maintenance enhancement projects.  I have previously written two posts on this question, Part 1 and Part 2.

But What About Maintenance (Enhancement) Projects?

Notably lacking from either chart of accounts are tasks for analyzing the existing system or program comprehension, though these are often lumped under initial or detailed design.  This is problematic in that many studies have shown that program comprehension typically consumes half of a programmer’s time on maintenance or enhancement work.  How can this be measured?  Estimated?  This seems to be one of the largest remaining unsolved problems in software engineering.  I will write more on this in a future post, but for now I will say that the beginning steps to solving these problems are the implementation for the team of 1) consistent, visible methods and 2) tools that enable tracking of program research.

Caveat: selection of which measurements to implement should be based on the anticipated value they contribute to software maintenance process improvement efforts or general business goals.

1. Early and continuous delivery of valuable software

In my own experience this can be very problematic for sizeable legacy enhancement projects – it is possible, but I think only in the right circumstances.  What are the objectives of this anyway?  Why is it important?

Two key objectives in any project are:

1. Minimize risk
2. Maximize speed and volume of value delivery

The second point is clear enough, make the rate of value delivery as high as possible.  A graph of value delivered by the project should be sloping down over time.  This ensures that if the project is discontinued for any reason, the most value pieces got out the door, plus it maximizes the rate of ROI.

The first point, minimize risk, is a little more complicated.  In a presentation by Alan Shalloway, CEO of Net Objectives, at Agile 2008, he commented that there are five possible reasons that mistakes are made in software development:

-          Developers didn’t write the code properly

-          Developers misunderstood what the customer wanted

-          The customer realized later they misspoke

-          The customer spoke properly but realized after they saw the program that they asked for the wrong thing

-          The customer spoke properly and got what they originally wanted, but now they have a better idea

Risk for a project is minimized by reducing both the occurrence of these mistakes and their impact when they do occur.  One of the key benefits of the first Agile point, “early and continuous delivery”, is that it reduces the impact of these mistakes by discovering them as quickly as possible.

So that is the thinking behind this concept, now, the question: is this possible in legacy enhancement projects?

New development projects typically realize this goal by stubbing off modules in early iterations and completing them in later iterations.  A map of when different pieces of functionality are delivered might end up looking something like this:

Is this process of stubbing off functionality possible in enhancement projects?  I believe the answer is

-     usually yes, when the new functionality is realized as new interfaces to new modules, representing new edges of the application – akin to new development, though called by old code, and

-          usually no, when the new functionality is realized by modifying the internals of multiple interconnected existing modules.

Modules in this context are AS/400 programs or RPG subroutines or procedures.  I qualify these both by “usually” as I think the truth is more fine-grained than this level of analysis – but this is a good start – more in the future.

I think this is a very important point to have clarity on as it needs to be considered when planning projects and setting expectations with the organization.

• Users
• Processes
• Documentation
• Availability of software authors for consultations
• Skills of maintenance programmers
• Tools available to aid the maintenance process
• The software itself

For each of these factors measurements can be defined, captured and analyzed to improve the affect on the overall maintenance operation.  In some cases the measurements by themselves have little value, but gain value only when used in ratios with other measurements.  Not all measurements are necessarily of value to a given organization – management needs to exercise judgment as to which measurements may be most useful.

Another way, of course, to determine what should be measured is  to look at the goals of software measurement.  I had outlined one set of possible goals in an earlier post, here.  For this this exercise, however, I will use the perspective described above.

Over the next few posts I will drill down into each of these areas and suggest possible measurements to be used.