The Apache Way, Redux

The Apache Software Foundation (ASF) is a globally-recognized source of quality software and collaborative methodology. But the growth of the foundation did not come without some cost. Nowadays, the ASF has to be concerned with things that were undreamed of back when the organization was a handful of coders collaborating via email. Now the Foundation has intellectual property to safeguard and protect; press and public relations to handle; multi-axis repository access control to manage; two orders of magnitude more mailing lists; and four orders of magnitude more mail; and numerous other challenging tasks that have little or nothing to do with writing code.

The result has been the growth of bureaucracy — but since there are no bureaus and the term “bureaucracy” often connotes intentional interference and delays in processing, let’s use a more light-hearted term and call the necessary, non-coding support tasks administrivia instead. Of course, the tasks aren’t trivial, but who wants to read about bureaucracy? Reality is so uninteresting.

Back when the Apache Group was just a dozen or so developers working on a single codebase, life seemed pretty simple. Compared to today, it possibly was simple. Each developer worked on the code primarily out of self-interest, adding features and fixing bugs that affected him individually. As often as not, each member had overlapping interests with another, and each was eager to see one another’s changes.

As the size of the development team grew, interests became more and more diverse, often with fewer people engaged on any particular topic than before. And as companies became involved with the ASF, more and more of the developers coded because they were paid to do so and not due to personal motives.

Meanwhile, the Apache Software Foundation itself was forming and growing, and the Apache HTTP Server was no longer the only thing “Apache.” Projects approached the ASF because of its license, or the environment, or sometimes because of the Apache “brand.” The increasing number of distinct projects was accompanied by an ever-growing diversity of development — and a decrease of overlapping interests. Figure One shows the growth of projects, committers, and members.

It became clear in 2002 that the Foundation had started to lose its cohesive vision as a consequence of its growth and diversification. The observation prompted a long and vigorous discussion, fondly remembered as “The Great Reorg Discussion,” or sometimes “The Great Reorg Flamewar.” One result was the creation of the Apache Incubator project (described previously here). However, the discussion itself and its consequences demonstrate one of the greatest challenges facing the Apache Software Foundation: The problem of scale.

Growing Upwards

The criteria used to nominate and elect new members to the Foundation are completely subjective. However, one common theme is personal acquaintance with the individual’s work. A member who participates almost exclusively in the Apache Formatting Objects Processor project (http://xmlgraphics.apache.org/fop/), for instance, is unlikely to nominate someone from the Web server project, unless he has personal experience of the quality of the individual’s contributions. In other words, members tend to nominate contributors whose interests are close to their own.

As the Foundation grows, it becomes more and more common that a member-nominee is completely unknown to the majority of the membership. Sometimes people resolve this using a “web of trust” model: “I know and trust Chris. If Chris says Jean is worthy of membership, I’m willing to take Chris’ word for it and vote in favor.” Recently, some have objected to this practice, since it’s following the same path that led to “The Great Reorg Discussion.” Others contend that the dilution of membership is probably only an issue if one postulates that the member ranks must grow to keep pace with other types of growth. Debate continues, yet there is universal agreement that membership must be capable of growth.

While a good deal more energy goes into discussions (and processing) of administrivia now than in the past, it’s unclear that it’s proportionally greater. Nevertheless, the sheer amount of it can have a deleterious effect on some of the things that helped the Foundation become what it is.

For example, gone are the days when one could just hack away on whatever you liked with no consideration other than making it work. Now there is the ever-present specter of “Will this infringe on anyone’s intellectual property?” When working on shared or imported code, one is faced with the question “Am I licensed to do this?” (which is really the intellectual property question in just another guise). When writing announcements, talking to a journalist, or when posting email, there is a concern that a statement will be misinterpreted or will be taken out of context.

These administrivial matters can act as sand in the gears of a free-wheeling organization like the Apache Software Foundation. Such details can slow things down and become irritants. Like sand, administrivia gets in everywhere.

A Victim of its Own Success?

So how can the ASF maintain its innovative momentum despite the impedance of the overhead accreted by its own success?

To keep sand out of the machine, you could enclose the works in a protective housing, but that’s the opposite of a visible and open machine. Perhaps mudflaps might be workable?

In a way, that’s one of the functions of the Foundation’s Public Relations Committee: to handle some of the administrivia so that the individual projects don’t have to. The PRC, as it’s called, is tasked with being the “front man” for the Foundation. It liaises with journalists, analysts, the press in general, and basically anyone who has a question about non-technical aspects of the ASF.

Each project is supposed to involve the PRC if it’s contacted directly about something that falls within the PRC’s purview, such as a journalist wanting an interview or a participating company wanting to issue a press release that mentions Apache packages. This doesn’t always work, in large part because the PRC is still relatively new and, being administrivial in nature, hasn’t necessarily engaged the attention of the software people.

One of the PRC’s most important tasks is protecting the Apache brands and marks, defining when and how those assets may be used, and licensing them appropriately to qualified organizations. Protection includes examining unlicensed uses of the marks and requesting correction if they’re being used improperly. A notice from the ASF might be as simple as an email message, yet occasionally (rarely) it requires a formal Cease & Desist notification from the Foundation’s counsel.

Not only does the PRC handle a lot of the administrivial needs of the Foundation, but it tries to do so in a uniform manner. That is, if Analyst A contacts the ASF about Project X, and Analyst B wants to talk about Project Y, going through the PRC means that both are handled the same way. Likewise, if company Aleph wants to use one of the Apache logos, and company Zed asks if it can use another one, the PRC can apply a uniform policy. If these requests went to individuals or individual projects, the results might be different. The answer might be “no” due to conservative uncertainty instead of “yes” as dictated by PRC policy.

A Well-Oiled Machine

In addition to proofing against sand, the machine must also remain efficient. Because the primary role of the Apache Software Foundation is software development, it is important to shuttle administrivia away from the development process as much as possible. Procedures escort the most common types of requests through the Foundation. For example, it’s recommended that project leaders and developers forward gritty messages to the PRC instead of handling them directly.

One attempt to keep ideas flowing and permit development untrammelled by excessive red tape is the relatively new Apache Labs project (http://labs.apache.org/). From the project’s Web page:

Apache Labs is a place for innovation where committers of the foundation can experiment with new ideas. The aim is to provide the necessary resource[s]to promote and maintain innovative power within the Apache community without the burden of community building. Research efforts are intrinsically associated with a higher mortality rate and don’t satisfy the established incubation and development practices. Labs provides a place for the Apache committer community to collaborate on such efforts, without discrimination of purpose, medium or implementation technology.

You might notice two distinct oddities about this project: the phrase “without the burden of community building,” and that participation is limited to individuals who have already been granted commit access to one or more other projects.

Clearly the focus is on experimenting with technology, rather than on putting together groups of people to perform the experiment. The limitation to existing committers means that work committed to the labs’ repository is already covered by a Contribution Licence Agreement and that particular hurdle of IP clearing is already past. Any sand introduced to this process will come from inside the ASF rather than from outside.

The Apache Labs project is different from ‘traditional’ ASF projects in another way: merit acquired elsewhere in the Foundation is directly transferrable to the Labs project. If you have commit access to any other project, you can have it to the lab repository.

The Apache Labs project is still quite new, and how effective it is and how much traction it gains is still a matter of speculation. It’s an attempt to harness the creative talents of people who are already proven innovators. What happens to a lab once its goals are accomplished — say, once the experiment is shown to be a success or failure — is a matter yet to be dealt with. A successful experiments is likely to be incorporated into an existing project, or perhaps form the nucleus of new ones.

It’s also unclear if the Labs project will be a success. Some are concerned that the Labs might become a small-scale SourceForge, with experiments spawned but then abandoned due to lack of interest. Unlike a full-fledged Apache project, though, an experiment in the Labs can succeed with only a single interested individual; normal ASF projects require a minimum of three.

The Labs project is aimed at fostering new ideas and experiments because that’s where the concern about potential moribundity lies. Innovation within existing projects appears to be largely unaffected by the administrivia problem, although there are the occasional sandstorms when large chunks of code are contributed or sometimes when company involvement becomes an issue. (This might happen when a company first has some of its employees associate with a project because it meets the company’s needs, but occasionally it gets raised when a company finds out that some of its employees are involved at the ASF and the company had no clue about it.)

Although the Apache Incubator might be viewed as a means of encouraging and managing innovation, the very fact that it’s the gateway to the ASF from the outside means that it necessarily involves a lot of the administrivial details that can interfere. Ideally, the Incubator filters the sand away from the process of innovation and development, but it’s not always as clean as one might hope. Of course, in its educational role, the Incubator should be introducing new participants to what automation and streamlining mechanisms have been developed for dealing with the grit, so that they won’t have to flounder in it looking for ways to handle it.

One way in which the Incubator proved susceptible to creeping grit was in its very function: within the last year, some people noticed that things coming into the Incubator far outnumbered those leaving, indicating possible procedural constipation. One way in which the innovation process _did_ work, however, was in addressing this problem once noticed: more people paid attention to moving things through the incubation process, and tools were developed to simplify and automate parts of the process. Since graduation from the Incubator doesn’t happen automatically, a one-time solution won’t work– now that it has been noticed, people will need to continue revisiting the status of incubating projects to make sure the constipation doesn’t inadvertently happen again.

One of the very area the Apache Software Foundation will innovate, intentionally or otherwise, is in scaling and “keeping the pace.” It seems to be generally accepted that organizations follow a sort of “Peter Principle,” growing until further growth is choked by the accumulated red tape. Since the ASF doesn’t regard growth as a goal, attention will focus not on how to maintain growth, but on how to deal with the gritty administrivia that accumulates. Getting rid of the red tape, or reducing it or controlling it or whatever, will help existing processes. If it also happens to allow continued growth of the Foundation’s work and involvement, that’ll be a nice fringe benefit.

This is perhaps a difficult point, so let me haul out and dust off another analogy: the unintentional success of the Apache HTTP Server project. After the first three years of its existence, the server software had achieved 50% market share worldwide. That was nice, but it had never been a goal. To paraphrase remarks about the phenomenon, “We never set out to rule the world– it just sort of happened. We just want to code.” Similarly, if getting rid of or streamlining red tape lets the Foundation continue growing, that’s all well and good. But the goal is to sweep the sand away and let the ASF keep doing what it’s doing.

Led By Evolution

The Apache Software Foundation is like a lot of the code developed under its aegis: its development is more by evolution than by design. “See a need, fill a need.” is a much more common approach than anticipating a future need.

Obstacles like administrivia tend to build up until someone consciously notices them, at which point they may start to be addressed. Not necessarily, though — sometimes there’s an additional intermediate stage, and nothing will happen until someone gets sufficiently irritated. It can be like being in a room of people watching television; someone will comment that the program is rubbish, and everyone else will smile and nod and agree, but the channel won’t be changed until someone gets really fed up.

It’s really not as burdensome as it might sound, however. Since almost all changes are made by people who have a vested interest in seeing them made, they tend to be handled more quickly than they might if they were assigned to an uncaring support technician for resolution. In addition, the fixers bring to the solution process an understanding of _exactly_ what problem needs to be addressed, so it’s not as though the wrong thing will get fixed or the problem will be entirely hand-waved away as a non-issue.

The possibility of the Foundation becoming paralyzed by the consequences of its own success has not gone unnoticed by people in the organization. Some steps to keep things active and moving freely have already been taken, as described above. How successful they’ll be remains to be seen, but if they don’t pan out as hoped, the people who proposed them will come up with new ideas to try. Then there’s always the strong possibility that others will be attracted to finding solutions. Although its participants tend to be software developers and not experts in business process, they are nevertheless very bright and inventive. Whatever solutions they come up with will work, of that I have no doubt– and I think their answers will often be quite surprising.