Does Documentation Help Maintenance?

One of the truisms of IT is that there’s never enough documentation.  But ask yourself, your scientific self, does documentation really help software maintenance, and if so, what kind exactly and how?

These kinds of questions are very difficult to answer as it’s very difficult and costly to arrange controlled experiments.  Some researchers, however, have managed to execute at least small scale experiments and one such example was reported in “The Impact of UML Documentation on Software Maintenance: An Experimental Evaluation” ($) by Erik Arisholm, et al.   As is common in such studies, senior computer science students were used for the tests.  This is not an ideal industrial study, but it’s about as good as it gets for conducting actual experiments.

The experiment was actually conducted twice, once in Oslo and once in Ottawa.  The purpose was to determine if the existence of UML documentation resulted in software maintenance being done a) more quickly and b) more correctly.  One factor, which the researchers covered both ways, was whether the time required to update the UML documentation to reflect the program changes was included in the total.  Results were reported both including and excluding that time.

The primary findings of the researchers were:

• The existence of UML documentation reduced the time spent programming on average by 20-25%
• If the time required to update the UML documentation was included, then the total times were equal, whether or not UML documentation existed.
• The software changes that were made were about 25% more accurate if UML documentation existed

Given that updating documentation to reflect maintenance is probably a requirement, it can be said that the net gain, from a programming point of view, is that the quality of code changes is measurably better.

A few other interesting findings:

• Expert programmers were helped by the documentation much more than novice programmers
• Familiarity with the UML tool was a significant factor in the results – when UML was reviewed and maintained with a simpler tool, in this case Visio, the programmers were able to review and update the UML documentation more quickly
• Advanced features of UML that were used in the documentation tended to degrade its efficacy – probably due to confusion
• Large diagrams also resulted in poor comprehension as they were judged too difficult to read and master

The authors also point out that the efficacy of UML as opposed to other modeling schemes, such as OML or text-centric documentation is far from established.

I want to point out that this study was focused on the effectiveness of UML for programmers doing maintenance work.  UML documentation may have other important uses in an organization as well.

I think it is unclear at this point in time what is the most cost-effective means of documenting a system, either for an organization as a whole or from a narrower programming perspective, however, the use of UML is not a bad bet.