Show Me The Documentation

Lump is poorly documented. Lump is open source. Lump is not a good example of poor open source software documentation.

Some jackass Microsoft-certified IT weenie who can't find his own ass with two hands, a flashlight, and the Microsoft Knowledgebase at his beck and call may some day use Lump as an example of how open source software has crap documentation and crap usability for users, all of whom demand Features! Features! Features! (cue Steve Ballmer dance). This jackass is making the wrong comparison.


Instead of comparing Lump with . . . I don't know. Does Microsoft have an equivalent? What about Adobe? No matter. Let us assume that there is some Weblog oriented CMS out there, closed source, that anyone gives a shit about. I know, it is a bit of a stretch, but let's say it exists. Maybe it is part of Steam's market diversification plan, and called Pile, so it's a Steaming Pile. My point is that a comparison between Lump and Pile is the wrong comparison to make.

Let us assume instead that Lump has a closed source software equivalent created by someone named Lee Peregrine, just so we have a proper comparison to make. Lump and its hypothetical equivalent have some things in common. The first thing they have in common is that each was created for the sole purpose of serving the needs of its creators. The second is that it lacks sophistication. The third is that it is poorly documented. The difference -- the only real difference for purposes of this discussion -- is that unlike the creation of the closed source equivalent's creator, I decided to make Lump's source code available to the public. Period.

Another example of a more-correct comparison than Lump and Pile here is something more like a comparison between FreeBSD and Microsoft Windows. As far as I've been able to determine thus far, FreeBSD is the single best user-documented modern, general purpose operating system in existence. Microsoft Windows, meanwhile, is one of the worst. While there is certainly a huge mass of documentation related to MS Windows, about half of it seems to be of dubious usefulness, and about half of it is either totally unuseful or useful only for extremely limited circumstances to solve extremely rare problems even though presented as a general solution to a very common problem.

I do not even want to consider the notorious horrorshow of Oracle documentation.

Part of the problem with Microsoft documentation is that it is developed the same way as its software -- with little or no input from users, a sad state of affairs that is applicable even to Microsoft's open source software. This bears repeating:

Many of the problems with Microsoft's documentation and programs is that Microsoft does its development with a markedly dismissive attitude toward user input, even when the software is open source.

Open Source Not Immune

This is not to say that open source software is immune to bad documentation. As I pointed out in The Lure Of Features, Ruby sometimes runs afoul of poor documentation:

As it turns out, it was easier to write my RSS feed generator from scratch than to figure out how to use Ruby's RSS core module. Yes, really. The problem, you see, is that documentation for the RSS library is shit, and all the examples on the Web for how to use it to build an RSS feed are every bit as anemic and sad as the actual documentation. I'm not entirely sure most of the people writing these howtos have even used it. I think most of them have probably just copied the same old thing they found somewhere else, word for word, and changed a few details to thinly veil the plagiarism.

By contrast, I have seen indications that Microsoft's .NET libraries tend to be very thoroughly documented. Then again, if I was looking at Ruby from the outside and had only as much experience with it as I have with C# (next to nothing), I would have said the same thing about Ruby -- "I have seen indications that Ruby's core libraries tend to be very thoroughly documented."

An occasional acquaintance of mine, Curtis "Ovid" Poe, said:

Plenty of questionable assertions (ha!) are made about tests and some of these assertions are just plain wrong. For example, tests are not documentation. Tests show what the code does, not what it's supposed to do. More importantly, they don't show why your code is doing stuff and if the business has a different idea of what your code should do, your passing tests might just plain be wrong.

Despite this, there seems to be a (hopefully small) subculture within the Ruby community that claims that any library with complete test coverage is "well-documented" code. This is only slightly more offensive to someone who actually needs documentation than claiming that code with comments in it is "well-documented". As Ovid's commentary suggests, good documentation explains not only what it is, but how it is used -- and, in addition to that, why. By contrast, unit tests only explain what it isn't, and they do not explain it in plain English (or Japanese or whatever).

I find attitudes like those of people who regard a test suite as "documentation" particularly odious, not only because it's frustrating to be told something is well-documented when it so obviously is not, but also because it stands in the way of proper documentation being written.

This is not something I encountered when I spent a few minutes looking for documentation for the RSS library, but it is something I have seen in the past. It needs to be identified, exposed to the light of day, and thoroughly denounced so people will get on with the business of actually documenting their software and libraries. Anybody who claims that a test suite is sufficient documentation for a language's core library should be publicly embarrassed for that claim and voted off the island before (s)he influences some credulous newcomer.

Testing is great, mind you. I have no problem with it. I believe that, the vast majority of the time, software development projects should have extensive test suites. I believe, in fact, that even some rinky-dink project like Lump should have a test suite. I also believe that to truly appreciate the benefits of helpful software development processes and tools, we must be honest with ourselves about those benefits. There is No Silver Bullet, and if we kid ourselves that something is a silver bullet it will -- no matter how great it might be when used appropriately -- become more of a hindrance than help in many cases, as it is misapplied when some other aproach to solving a given problem should be taken at the time.

By the same token, I will not turn a blind eye to the documentation weaknesses in the open source world, despite the fact that I still believe that open source software tends toward better documentation given similar conditions. MS Windows and FreeBSD have similar effective lifespans and are deployed across a similar range of use cases, but FreeBSD has far better documentation. That does not mean that the poor documentation practices for some open source software should be ignored. If we ignore problems, we cannot reasonably expect to fix them.

I expect to start doing a better job of documenting Lump just as soon as it becomes a bit more stable -- if not sooner. If you need it to be sooner, letting me know you need it to be sooner is a good way to make it sooner, because I tend to listen to that sort of thing. I recommend using the issue tracker at Lump's BitBucket project repository, and being specific about what you need when submitting an issue. For the moment, unless someone tells me otherwise, I am likely to believe that other priorities for Lump development rank higher than documenting it for other users that I do not believe exist. In the meantime, then, expect it to lack thorough documentation, and please refrain from using it as an example of how open source software is poorly documented. Its closed source equivalent would, of course, be even less well documented, to say nothing of being unavailable, because where I simply do not expect anyone else to want to use Lump right now, it would equivalently be unlikely that random people would be allowed to use the closed source equivalent.

Show Me The Docs

Linus Torvalds famously said "Talk is cheap. Show me the code."

I would like to extend that to documentation, here. If someone complains that there is a lack of documentation for something, there is obviously something wrong. It may be that the documentation does not exist. It may be that the documentation is insufficient for the person's needs. It may be that the documentation, no matter how complete and helpful, is not easily enough discoverable.

The solution to this is not to claim your documentation is at least better than something else's documentation, or that it is well documented because of a test suite, or that someone is stupid for not being able to find the documentation. The solution sure as hell is not to point to Lump as an example of bad documentation. The solution is to "show me the documentation."

Talk is cheap. Show me the documentation.


I suppose someone might call me a hypocrite for making such a fuss about the importance of documentation while failing thus far to document Lump. I could perhaps defend myself by pointing out that Lump has, as of this writing, existed for less than a week -- but I do not consider that much of a defense, given the fact I just said "Talk is cheap. Show me the documentation."

Fair enough.

I do not consider it hypocrisy, personally. I have a better argument than that: I am not excusing a lack of documentation that others say they need, or that I expect others to need. If someone points out to me that Lump is poorly documented, I will readily admit that person's statement is correct. More to the point,if someone claims reasonable need for documentation, I would even be inclined to write it. If you go back and read the end of the previous section in this very blogstrapping entry, you will see that I even asked people to prompt me if they need documentation so I can revise my estimation of how important it is to document Lump.

This is why I do not consider myself a hypocrite in this matter. You may still think I am. If so, however, I would like you to keep this in mind:

Even if I am a hypocrite, that does not make what I said wrong.