Softpanorama May the source be with you, but remember the KISS principle ;-)
	Contents	Bulletin	Scripting in shell and Perl	Network troubleshooting	History	Humor

Softpanorama Classic Books

"No scene from prehistory is quite so vivid as that of the mortal struggles of great beasts in the tar pits. The fiercer the struggle, the more entangling the tar, and no beast is so strong or so skillful but that he ultimately sinks. "

The Mythical Man-Month. Essays on Software Engineering by Frederick Brooks Jr. Anniversary Edition  Paperback, 322 pages Published by Addison-Wesley Pub Co in July 1995.

The book was first published in 1975 and was based on Fred Brooks experience with the design of  operating systems for famous IBM/360 series.  It is interesting that OS/360 was generally a failure -- badly designed and in many respects inferior OS.

But hardware of System/360 was really great -- a masterpiece of engineering.

Actually both hardware and descendants of operating systems designed for those machines are still  sold by IBM (IBM mainframe business).

While OS/360 was inferior from the very beginning it was a huge success as hardware was a huge success (rising tide lifts all boats). And despite being a failed, inferior architecture it survived. Some really good peaces of System/360 software suits did not faire equally well. For example, PL/1 an innovative programming language that pioneered the use of exceptions (and inspired C) is dead. Despite having initial good complier and later two really brilliantly written compilers each of which is still a masterpiece of software engineering.  At the same time ugly JCL is still used.

Despite not so good accomplishments as the head of the project, Brooks have written a really brilliant book. Most of the material in the book is as relevant today as it was when originally written. Understandably, though, part of the material is outdated. In the 20th Anniversary Edition, rather than update the original text, which is considered "classical," Brooks has wisely decided to add update chapters. These discuss the issues presented while shedding fresh light from the 90s on them. Personally, I recommend reading the relevant parts of Chapter 18 after reading each previous chapter. Chapter 18 is titled, "Propositions of The Mythical Man-Month: True or False?", and it contains updated information about each of the chapters from 1 to 17.

The book itself still did not lost its value due to the fact that it contains unique observations about large scale software development, the observations that can't be found anywhere else(for example Steve McConnell is just a consultant who never headed large projects). Brooks touches things that are typical. but not highly visible in any complex engineering project. He is my compilation of key points from the book:

• The talent of the manager and the key architect are the crucial factors in the success of the large project. 
   
• Planning of the length of the project in fixed units such as man-months or developer hours suffer from multiple defects. Such planning does not objectively reflect project size and complexity and due to those two factors  man-months or man-hours are not factorable, associative, or commutative. 1 programmer * 12 months does not equal 12 programmers * 1 month. The performance of programming teams, in other words, does not "scale" in a linear fashion any more than the performance of multi-processor computer systems. He found, in fact, that when you throw additional programmers at a project that is late, you are only likely to make it more late. The way to get a project back on schedule is to remove promised-but-not-yet-completed features, rather than multiplying worker bees.

  When you stop to think about it, this phenomenon is easy to understand. There is an inescapable overhead to yoking up programmers in parallel. The members of the team must "waste time" attending meetings, drafting project plans, exchanging EMAIL, negotiating interfaces, enduring performance reviews, and so on. In any team of more than a few people, at least one member will be dedicated to "supervising" the others, while another member will be dedicated to housekeeping functions such as managing builds, updating Gantt charts, and coordinating everyone's calendar. At Microsoft, there will be at least one team member that just designs T-shirts for the rest of the team to wear. And as the team grows, there is a combinatorial explosion such that the percentage of effort devoted to communication and administration becomes larger and larger.

• The slipping from schedule is gradual and almost unnoticeable. This is reflected in his famous maxim: "How do disastrously late projects get that way? One day at a time." (Brooks applied this phrase to the software field long before it became a 12-step self-help mantra.) Most slippage is the offspring of “termites, not tornados.”
   
• Sequential constraints: Some projects take an irreducible amount of time because one part of the task needs to be finished before the next can be begun — or, as Brooks puts it, “The bearing of a child takes nine months, no matter how many women are assigned.”
   
• Coding isn’t what takes the most time: Brooks’s rule of thumb is that actually writing the code takes 1/6 of a project’s time — with architecture development taking another third and the remaining half devoted to testing and debugging.
   
• Programmers vary wildly in individual productivity: Wildly as in the highly productive programmer may well be able to produce ten times more useful code in the same amount of time than his average colleague.
   
• The goal of “conceptual integrity” or unity of design: The best programs and software systems exhibit a sense of consistency and elegance, as if they are shaped by a single creative mind, even when they are in fact the product of many. The fiendish problem is less how to create that sense in the first place than in how to preserve it against all the other pressures that come to bear throughout the process of writing software.  Brooks says the most important aspect of the design of a system is to ensure its conceptual integrity i.e. a consistent set of design ideas. "We're done not when there's nothing more to add, but when there's nothing more to take away".  In order to achieve this Brooks suggests the need for a system architect
   
• The “second system effect”: Brooks noted that an engineer or team will often make all the necessary compromises necessary to ship their first product. But if given a change to redesign it they can fail, unable to resist too ambitious extension of functionality/features/interface, etc.
   
• Talent really matters and makes the difference.  Paradoxically the design of System/360 OS was a screw up. Brooks traces this screw up to the desire to add new peoples into late project. Which only makes it even later. This is due to the fact that a in order to be productive new players need to learn the rope and that actually distract those people who are already in the project. Also the biggest that team the more complex communication structure becomes which in turn leads to errors and delays. big part of software development is communication and adding people makes that communication more complicated than it previously was, therefore meaning it takes longer to get things done. The popular metaphor a baby can't be produced any quicker by having 9 women rather than just one  expresses very similar reservations.
   
• Incompleteness and inconsistencies of ideas only becomes clear late during implementation – pretty much putting a dagger into the idea that we can define everything up front and then code it just like that.
   
• An interesting idea about creating a surgical team, with the key person responsible for the success of the "operation" and other providing support in various functions. Only few the most experienced people should do the coding. But testing and other support functions such as configuration management, backups, writing tools, etc should be provided as a services from other members of staff.
   
• Brooks pioneered the notion that productivity increases gained by using high level languages.  If developer can use higher level language they will be more productive and the number of bugs will be less, especially the most difficult to debug architectural blunders. That is vey important advice that stresses the important of usage of scripting languages. 
   
• The idea of only performing system debugging when each individual component actually works is something which should be obvious but is often not followed. If we know a component doesn't work on its own then we can guarantee it is not going to work when we try to integrate it with other components so the exercise seems slightly pointless to me. Common sense advice I think!
   
• Speaking of code reuse Brooks points out that the key here is the perceived cost of finding a component to reuse that is important. If the API call cannot be easily found programmers tend to reimplement this functionality again and again.
   
• Brooks talks of the need to have dynamic documentation that reflects the stage of the project.  He stated that documentation is a very sharp tool and it should be paid proper attention. He used a project workbook to do this and I think the modern day equivalent would be the project wiki. The idea of creating self documenting programs to help minimize the documentation that needs to be written is also covered.
   
• The need to progressively refine the system by growing it based on customer feedback rather than building it.  That does not mean that the initial architecture is unimportant, but that a lot more can be achieved by stripping the initial design from any niceties and implementing the core functionality first. He was one of the first to describe the limitations of the waterfall model are described, but also warns that prototyping approach has its pitfalls too ("the effect of the second system").

Anniversary edition contains in addition to the original book his influential 1986 essay "No Silver Bullet"  which was originally an invited paper for the IFIP '86 conference in Dublin, and later published in Computer magazine. In this paper, Brooks speculated that no technology will be found, within ten years of its publication (in 1986), which will enhance the process of software development by an order of magnitude. Nine years later, in retrospect, Brooks sadly notes that he was right. Scripting languages changed this a little bit but still this remains true as of November 2009.

Frederick P. Brooks

Frederick P. Brooks, Jr. is the "father of the IBM System 360". While manager of the 360 project it was Dr. Brooks who specified that a byte would consist of 8 bits and that word consists of 4 bytes. Whether or not you agree with his decision, it's hard to argue that this has not had a huge impact on the computer field.

In his time Machiavelli also was a senior civil servant in the Republic of Florence. Later the Medici returned to Florence, the Republic was replaced with an absolute rule and Machiavelli was dismissed from his job. Out of favors, out of job, he found himself under house arrest. The only thing to do was to write a book with his reflections on the process of gaining and maintaining power. Do you think the new rulers were stupid enough to hire a clever civil servant of the previous regime? So people like Fred Brooks and Machiavelli are probably very clever but not clever enough. Fouche and Talleyrand, for that matter, never wrote books -- they were men of action.

No book on software project management has been so influential and so timeless as The Mythical Man-Month. Even now,  20 years after the initial publication it's still a very important.

Who is this Fred Brooks?

He was born in 1931 in Durham, North Carolina and grew up in nearby Greenville, NC.  He was interested in computers during his teenage years, and went on to double major in math and physics at Duke University.  At Harvard, he studied for his Ph.D. under Howard Aiken, the inventor of the early Harvard computers.  He joined IBM in 1956, working in Poughskeepie and Yorktown, NY and in 1957 Brooks and Dura Sweeney patented a Stretch interrupt system for the IBM Stretch computer that introduced most features of today's interrupt systems. 

In the early 1960s he was the leader of the development team for the IBM 360 Series.  The 360 quickly became the most popular mainframe computer on the market.  In 1964, Brooks left IBM to found the Computer Science department at the University of North Carolina at Chapel Hill and to become a professor of Computer Science.  He has chaired the department for the past twenty years. 

Throughout his career, Brooks has authored two books in the field of Computer Science, The Mythical Man-Month: Essays on Software Engineering and Computer Architecture: Concepts and Evolution.

The Mythical-Man Month is undoubtedly his most famous work in which he compiled essays about his work on the IBM 360 Series.  It also contains the famous “Silver Bullet Essay” where he likens his work on the 360 Series to a werewolf that can only be killed by a “Silver Bullet.”  Brooks was also known for his famous aphorisms, his most popular being Brooks’ Law: “Adding manpower to a late software project makes it later”   Many of his quotes are applicable in a wider context, such as “Complexity is the fatal foe” (P 344). 

Brooks is still a professor of Computer Science at UNC today.

See also

• Faculty Biography Frederick P. Brooks Jr. (UNC-CH Computer Science)
• The Mythical Man-Month - Wikipedia, the free encyclopedia
• Fred Brooks - Wikipedia, the free encyclopedia

Old News ;-)

[Aug 11, 2012] WHEN A BAZAAR IS NO LONGER A BAZAAR

Retrospective Notes

First-time readers of this rant should be aware that it was written well before either the Halloween Documents or AOL/Sun acquisition of Netscape. For all I know, at this writing, some other relevant event is erupting. Sun's quasi-open-source licensing of Java might even be considered germane.

In any case: I disclaim any powers of prophecy that might be inferred from some skewed reading of what follows. That's a little like reading the Book of Revelations as predicting Monica Lewinsky (the Whore of Babble-On, forsooth!) and this Y2K fiasco (yea, verily, thy Systems will Tremble and their Bowels will Loosen before the Wrath of the Two-Thousand-Year-Old Lamb.)

I also want people to know: I like what's happening with Open Source, and I'm glad that it has such an articulate and intelligent spokesman in Eric Raymond. My expressions of doubt--both serious and tongue-in-cheek--shouldn't be taken as disrespect for the process or the person.

I wrote this for fun. So please read it that way.

I also disclaim any influence on events. No, I don't know if Mr. Villaincantelope-or-whatever-his-name-is found some kind of backhanded inspiration in this piece. It's quite enough that Eric Raymond mentioned Fred Brooks' attribution of Bazaar-like practices to Microsoft; I entirely trust Microsoft employees to take even the most mud-besmeared ball and run with it, good little team-players that they are. I just doubt that they could even find a ball under all the mud you'll see here.

With all that clarified....let's MUDWRESTLE!

The Browser Wars Continued: Children's Crusade vs. Shi'ite Jihad?

Eric Raymond is going to be a Netceleb. Marc Andreesen virtually guarantees this. The developer consensus within Netscape will propel the author of "The Cathedral and The Bazaar" almost to household-name level of public visibility. Netscape will publish its browser source, Agoric processes will save the day, Microsoft's tidal advance will be stemmed.

Or will it?

I have mused on Eric Raymond's choice of "Cathedral" and "Bazaar" as juxtaposed metaphors for the two different styles. Historically, the one is Christian, the other Islamic.

Much could be made of this.

Allow me.

Religious Architecture and Social Archetypes

In Christian countries, the Cathedral was the source of ecclesiastical power, and to some extent it remains so. As a political power, however, its role has gone from nearly central to nearly non-existent. Nobody is building Cathedrals anymore, so that pork-barrel is long empty. And nothing has taken its place.

In Islamic countries, the bazaar - which is seldom, if ever, co-located with any related job-producing construction projects -- remains the center of ecclesiastical power. In self-styled Islamic Republics, the bazaar is therefore the center of power, period.

Perhaps the theocratic overtones in the choice of the term Cathedral was Eric Raymond's allusion to what many see as the doctrinaire and sanctimonious rumblings from RMS the Pope, versus the more freewheeling and (in strict GNU terms) easily-corrupted style of Linux development. And perhaps in choosing the term "bazaar" Raymond meant only to suggest something more social and economic and mundane; something that was, architecturally, at least, more low-to-the-ground. He refers to Eric Drexler's notions of Agoric systems, saying that he thought of entitling the essay "The Cathedral and the Agora." So maybe none of this has anything to do with the taint of theocracy.

But maybe everyone should (re)read God and Golem, a book written Norbert Wiener, the coiner of the term "cybernetics." In all things computer-related, ecclesiarchs are not far off.

If you want evidence that the bazaar style is easily corrupted and/or prone to its own problems of self-destructive, dogma-inspired holy wars, you need look no further than Microsoft itself.

Yes, Microsoft.

The Gospel According to Fred

Eric Raymond apparently has glanced as far afield as Redmond in seeking precedents, though apparently with only a shudder and a giggle so far. "For Further Reading" leads off with this:

I quoted several bits from Frederick P. Brooks's classic The Mythical Man-Month because, in many respects, his insights have yet to be improved upon.... The new edition is wrapped up by an invaluable 20-years-later retrospective in which Brooks forthrightly admits to the few judgements in the original text which have not stood the test of time. I first read the retrospective after this paper was substantially complete, and was surprised to discover that Brooks attributes bazaar-like practices to Microsoft!

A careful reading of Brooks's 25-year retrospective turns up another, crueler, irony. One of the few points where Brooks concedes a probable error of judgment is in his original insistence that all the information about OS/360 development be made available to everyone on the project.

Information should be free. At IBM. In the mid-60s.

Well. How clueful.

But now he says he was wrong?

That Stained-Glass Window is a Module - MY Module

At some point during this famously-late project, Brooks' policy of providing all information to every programmer meant filling wide shelves in every office with the official Project Notebook, and providing updates to it constantly. This, Brooks now feels, was a disaster.

He should, he says now, have gone along with what was, at the time, a novel software engineering concept: modularization through "Information Hiding". Presumably this would have been implemented by restricting the flow of information within the development organization itself.

It surprises me that Brooks wouldn't now consider, after the fact, that promoting an ethic (or at least, an aesthetic) of modularization in an open organization would be superior to enforcing (perhaps by prohibition) laws of modularization in a closed one.

One suspects that Brooks's Cathedral dreams never died. References to medieval architecture and Christian spirituality abound in his book. To have made OS/360 a true Cathedral - why, he'd be an IBM Cardinal today! A Saint, even! (Or CEO emeritus, anyway.) It seems he now feels that, besides tolerating poor architecture, he erred in leaving out a crucial Cathedral-building management technique: tolerance of internal trade secrets.

Medieval craft guilds were notoriously secretive about their processes, and Cathedrals provided not only pork barrels during lean times, but perhaps an early version of the R&D Tax Credit. But what damn good is that subsidy if your research failures become widely known while your successes aren't safe from pilfering by other guilds?

Poor Fred. Devout IBM Christian to the end.

He missed a trick. To be Pope in a Cathedral-as-pork-barrel economy, you have to be wily, political, amoral. You have to be the broker-of-secrets, which means you have to have secrets worth having.

You don't have to be actually evil, but it sure helps!

Death to the Infidels! (Uh, You're on Our Side, Right?)

Now we have FTP sites via the Internet and patchfiles distributed on newsgroups, rather than the OS/360 Project Notebook maintained by armies of documentation clerks. But apart from that, I'm hard put to see a major difference in social organization.

Yes, this freedom-of-information on the OS/360 project was IBM-internal, but so what? There were literally thousands of people working on OS/360 at one point, quite a sizeable community. Maybe larger than the number of people who have ever contributed a line of code to GNU/Linux. Why would such a community look outside itself? After all, even outside, except for a few niches, IBM was the world of computing. Information wants to be free, I suppose, but programmers want to be paid. OS/360 programmers had everything they could have wanted. Nice salaries. A prestigious host organization. A large community of similarly-spirited co-workers. Access to all the information they needed to do their jobs (and much more.) Not to mention that there were - by virtue of Brook's early, and now openly regretted, decision to triage "conceptual integrity" in the interests of getting everybody busy - lots of opportunities for creative (if not terribly original) work.

What a revelation. Fred Brooks, Bazaar Model developer. Way back in the mid-60s. But now, more amazingly, a lapsed Bazaar Model developer (if only by conflating information-hiding in software with information-hiding in organizations.) Who'd've thunk it?

I find it very persuasive that Microsoft is, internally, a bazaar- model development organization, a la OS/360. Knowing what we now know about how the Bazaar Model works, it helps explain Microsoft's enduring success. Not to mention all the feeping creaturism.

Comparative Religion: Refining the Taxonomy

To reiterate: In Islamic Republics, the center of powerful ideas is the community of religious scholars and ecclesiarchs, which locates itself in the center of economic power in those countries, and sets the terms of trade within those countries to a great extent. In the extreme cases (Iran), the insularity and self-righteousness runs to truly fanatical extremes.

Too bad the Mullahs aren't in Janet Reno's jurisdiction, huh?

It might, from this point onward, become necessary to make some finer distinctions in our use of the term "Bazaar Model". Prior to what may be a prematurely-characterized Netscape Enlightenment, we can clearly distinguish these two:

1. Fundamentalist Shi'ite Bazaar Model

Microsoft is the paragon now, but it seems to have independently re-evolved Project OS/360 characteristics.

In my brief, ill-fated, stint at Taligent, the same mentality - and the same OS/360 diseases - seemed prevalent. Parallels abound -- new processor architecture, incoherent OS architecture (too many architects), too many projects, little outsourcing, and...IBM was involved! (Hey, maybe Taligent was all just an IBM plot to weaken Apple! Who else would know better than IBM that it could never work? But...nah. Never attribute to malice what can be perfectly well explained by stupidity.)

I suspect the same problems will recur in any corporate-captive programming community that reaches a certain size with a common focus. SGI? Sybase? Both have repeated the history of OS/360 in recent years. Oracle's NC thrust appears to have been a similarly hubristic move.

Ayatollahs? Hoo boy, do they ever have Ayatollahs. "We want 100% of the market." How will it all work out?

Easy. Just remember: "God is great."

2. Baptist Church-Social Bazaar Model

Linux, notably, but also the BSD sects. There is religiousity in the air, but - within decent Protestant limits - a tolerance for diversity.

Commercial vendors of a size modest enough to be admitted to the community (Red Hat, Debian, etc.), are well-regarded, by and large. Main Street is OK - anything bigger is suspect.

Most people still need day jobs, though. How will it all work out?

Easy. Just remember: "God will provide."

But there's a stir in the churchyard. Somebody's arriving.

With Netscape throwing its hat into the ring (or is the free software community throwing its hat into theirs?) ...well, as Eric Raymond says, "this is the Big Time."

Or is it?

Pizza! Pepsi! Stock Options! Hallelujah!

What does it mean, exactly, for the vendor of a popular piece of commercial software to show up at the Baptist Church Social, and declare itself ready for immersion in the waters of the Holy Spirit - provided it's good for business? Especially a vendor that has been "Embracing and Extending" HTML as feverishly (and stupidly) as those Shi'ite Fundamentalists in Redmond?

Yep, you guessed it. You get

3. The Marc Andreesen Reformed Church of the Almighty Dollar Church-Social Bazaar Model

All bazaar booths are allocated by Netscape.

All bazaar merchandise must be tagged "brought to you by Netscape." Special branding irons are used in the case of cotton candy.

Modest-sized vendors who don't sign license agreements with weasel-wording will be strapped into the "dunk bozo" seat, and the top-intellectual property law firms of the Valley will take turns pitching the baseballs to see who can tank them.

Most people still need day jobs, though. How will it all work out?

Easy. Just remember:

"God is cool."

Now let us pray.

[Aug 22, 2010] Some lesser-known truths about programming

This is a rehash of some Brooks ideas.  Some comments are really good...

August 17, 2010 | Dot Mac

My experience as a programmer  has taught me a few things about writing software. Here are some things that people might find surprising about writing code:

• A programmer spends about 10-20% of his time writing code, and most programmers write about 10-12 lines of code per day that goes into the final product, regardless of their skill level. Good programmers spend much of the other 90% thinking, researching, and experimenting to find the best design. Bad programmers spend much of that 90% debugging code by randomly making changes and seeing if they work.
  “A great lathe operator commands several times the wage of an average lathe operator, but a great writer of software code is worth 10,000 times the price of an average software writer.” –Bill Gates
• A good programmer is ten times more productive than an average programmer. A great programmer is 20-100 times more productive than the average. This is not an exaggeration – studies since the 1960′s have consistently shown this. A bad programmer is not just unproductive – he will not only not get any work done, but create a lot of work and headaches for others to fix.
• Great programmers spend very little of their time writing code – at least code that ends up in the final product. Programmers who spend much of their time writing code are too lazy, too ignorant, or too arrogant to find existing solutions to old problems. Great programmers are masters at recognizing and reusing common patterns. Good programmers are not afraid to refactor (rewrite) their code constantly to reach the ideal design. Bad programmers write code which lacks conceptual integrity, non-redundancy, hierarchy, and patterns, and so is very difficult to refactor. It’s easier to throw away bad code and start over than to change it.
• Software obeys the laws of entropy, like everything else. Continuous change leads to software rot, which erodes the conceptual integrity of the original design. Software rot is unavoidable, but programmers who fail to take conceptual integrity into consideration create software that rots so so fast that it becomes worthless before it is even completed. Entropic failure of conceptual integrity is probably the most common reason for software project failure. (The second most common reason is delivering something other than what the customer wanted.) Software rot slows down progress exponentially, so many projects face exploding timelines and budgets before they are killed.
• A 2004 study found that most software projects (51%) will fail in a critical aspect, and 15% will fail totally. This is an improvement since 1994, when 31% failed.
• Although most software is made by teams, it is not a democratic activity. Usually, just one person is responsible for the design, and the rest of the team fills in the details.
• Programming is hard work. It’s an intense mental activity. Good programmers think about their work 24/7. They write their most important code in the shower and in their dreams. Because the most important work is done away from a keyboard, software projects cannot be accelerated by spending more time in the office or adding more people to a project.

Edward D. Weinberger:

Much of the content of this blog was originally posted in the classic THE MYTHICAL MAN MONTH, by Fredric Brooks. He was the guy that headed the IBM project to build the first modern operating system, OS/360, on IBM Mainframes in the 1960′s, so he clearly knew his stuff. He is convinced of the vast difference between the best and the average.

Believe it or not, the reason why managers make more than programmers is well captured by the comic strip DILBERT. Though Dilbert is technically adept, he clearly does not understand the business world. He therefore needs the “adult supervision” of the pointy-haired boss. Admittedly, the boss is an idiot, with absolutely NO technical savvy; however, he does understand the business world, including the importance of marketing.

And one other thing. Programmers are paid for more than productivity, which is why productive programmers are not necessarily paid more than others. The technology is changing so fast that the guy who may be a mediocre user of a hot technology gets paid more than the best COBOL programmer in the world, simply because nobody cares much about COBOL any more.

Maintenance Man:

A great developer may be worth a lot more than an average one. Might also be 100 times as productive. Then why again is the great one getting paid the same salary as the average one?

jambox:

@MaintenanceMan Too many companies allow their software products to be managed by non-technical managers. They simply do not know who is good and who isn’t! They also don’t know good software from bad and evaluate performance based on speed of delivery most of all. They’re also acutely vulnerable to bafflegibber.

Lou:

Thank you; good article. I recently worked (past tense) at a place that thought you could in fact speed software delivery by adding extra programmers and extra hours per programmer. Programming is not manual labor. Physical bodies can continue laboring long after the mind shuts off – whereas in programming, when the mind shuts off, you may code so ineffectively that you make negative progress. Bless the software managers that understand the pacing and rhythm of development.

@Maintenance Man, that’s a good question. I think it’s partly due to a lack of understanding on the part of managers and companies. As soon as it’s easy to quantify the increased value of a good programmer vs a bad one, the salary gap will increase.

Paul W. Homer:

 Nice. Although I’m still a little unsure of the first two points. In the short run, I think a great programmer is not all that much more productive than an average one, but I definitely believe that if you factor in time and the amount of code that actually stays alive the difference is huge. Also, I could agree with the 10-12 lines per day, if you are talking averages. I know a lot of programmers who (without cutting and pasting) can produce thousands of lines in a very intense week, but who then slack shortly afterwards.

Jeff Dege:

 I’m seeing that “good programmers are X-times more productive than bad programmers” meme, again, and again without a description of the shapes of the curve. This leaves people with misconceptions.

Productivity is not a normal distribution, it’s a Rayleigh distribution. If good programmers are ten times better than bad programmers, the distribution will be such that good programmers are twice as good as average programmers, and average programmers are five times as good as bad programmers.

The curve is skewed, and the median is above the mean. (Meaning that most programmers are better than average, as odd as that sounds.)

J:

I love reading something like this and then hear people talk about how this just proves how great they are at programming. No one ever seems to read and say, “Hmm … I wonder if I’m the guy that is a tenth as productive as good programmers?” On another note, although I agree with some generalizations, I think it is dangerous to put programmers in three categories: bad, good, and great. There is a lot of wiggle room where people don’t fit nicely in one of these categories. Some people are also good at some things and not at others. There are also people that have all the skills, but just don’t have experience yet. Everyone needs to make mistakes and learn from them, and that just takes some time.

Rosstafarian:

Except that average programmers are not just a multiple better than bad programmers. Somewhere just south of average, poor programmers contribute less than the increased communication overhead needed to include them on the team. A little below that, you have really poor programmers whose typical coding change makes the system worse and either requires time by alert developers to fix the problems they create or dramatically increases the risk of failure of the effort if it’s not detected in time.

In my experience, the best programmers routinely achieve goals that average programmers don’t understand even when the result is later explained to them. In terms of pure productivity, the difference is something like 10-15x between the best and the simply good. I firmly believe that there is a transition to negative value that occurs well into “average” territory but which is only apparent to a few enlightened members of management.

Gabe da Silveira:

You’re making the same mistake though. It is some combination of talent and experience that makes one great at anything. There’s absolutely no reason to believe that great programmers were somehow better than average from the outset… they could have had breakthroughs throughout their journey. Likewise the most promising candidate could get lazy in a cush job or maybe lose interest in advancing their skills.

Ulf Wiger:

@Alfred: In my experience, software companies (esp larger ones) need many different kinds of programmers. The trick is to find your niche and figure out how you can best contribute.

The software industry as a whole needs inventors, finishers, motivators, maintainers, … and also project managers and line managers who are (or at least have been) skilled enough at programming. Some people love working for years maintaining and improving a particular product; others wouldn’t be caught dead doing that, but want to innovate, prove a concept, and then move on.

The trick for every company is to find the right mix, and dependable programmers who are Good Enough are extremely important. I have plenty of good war stories about brilliant programmers, but they are best told over beer…

Matt J.

When I see that much enthusiastic agreement, it raises my suspicions. The majority is rarely right.

And sure enough, when I look more closely at this, I see lots of problems covered up with sententious authority. I am glad to see that doctor doom caught one of them, but there are more.

I will focus on only one, the point about it not being democratic. This is true, BUT: if you let only one person do all the design, you feed frustration on the rest of the team, and then what do you do if the one designer is hit by the proverbial bus? It is better to admit that the design process cannot be entirely democratic, but let the one designer bounce his ideas of other members of the team. This solves both problems by giving them a hand in the design also, and by distributing knowledge of the design among several people, so that if he is hit by a bus, he is replaceable without as much loss.

For the same reason, it is important that that one designer know how to share with the team. Pick someone who is bright, but autocratic, and you will ruin the team and the project.

Weinberger had an interesting correction to the article too, but he overestimates the PHB’s command of the business world. The real problem of modern day business is that the PHB who is technically ignorant is nearly as ignorant of the business world, too! That is WHY Scott Adams has the recurring line about ‘manager’ really coming from an ancient word for “mastodon dung”. That is WHY he reminds us of the managers so dumb, they didn’t even know how to use voicemail. He didn’t make that example up, either. It comes from real life.

Nor is it really that new. The reasons such gross incompetence is tolerated in the overpaid management class of society was covered very well by Thorstein Veblen in his “The Theory of the Leisure Class”. If you really want to understand why managers are so destructive, why they will never admit that Fred Brooks was right, then read this book.

[Nov 5, 2009] Scott Rosenberg’s Wordyard » Blog Archive » Code Reads #1 The Mythical Man-Month

The book introduces a whole slew of concepts at the heart of our understanding of how we build software:

• How do disastrously late projects get that way? “One day at a time.” (Brooks applied this phrase to the software field long before it became a 12-step self-help mantra.) Most slippage is the offspring of “termites, not tornados.”
• Sequential constraints: Some projects take an irreducible amount of time because one part of the task needs to be finished before the next can be begun — or, as Brooks puts it, “The bearing of a child takes nine months, no matter how many women are assigned.”
• Coding isn’t what takes the most time: Brooks’s rule of thumb is that actually writing the code takes 1/6 of a project’s time — with planning taking another third and the remaining half devoted to testing and debugging.
• Programmers vary wildly in individual productivity: Wildly as in the highly productive programmer may well be able to produce ten times more useful stuff, however you choose to measure it, than his colleague.
• The goal of “conceptual integrity” or unity of design: The best programs and software systems exhibit a sense of consistency and elegance, as if they are shaped by a single creative mind, even when they are in fact the product of many. The fiendish problem is less how to create that sense in the first place than in how to preserve it against all the other pressures that come to bear throughout the process of writing software.
• The “second system effect”: Brooks noted that an engineer or team will often make all the compromises necessary to ship their first product, then founder on the second. Throughout project number one, they stored up pet features and extras that they couldn’t squeeze into the original product, and told themselves, “We’ll do it right next time.” This “second system” is “the most dangerous a man ever designs.”

The Mythical Man Month Book Review at Mark Needham

Pretty much since I started working at ThoughtWorks 2 1/2 years ago I've been told that this is a book I have to read and I've finally got around to doing so.

Maybe it's not that surprising but my overriding thought about the book is that just about every mistake that we make in software development today is covered in this book!

What did I learn?

• The title of the book and the second chapter of the book refers to the situation that surely everyone who has ever worked on a software development project is aware of – if a project is late then adding new people onto it will make it even later. This is due to the fact that a big part of software development is communication and adding people makes that communication more complicated than it previously was, therefore meaning it takes longer to get things done. My colleague Francisco has a nice post describing the ways that adding people can slow down a development team. The idea that a baby can't be produced any quicker by having 9 women rather than just one is a particularly common metaphor used to explain this.
• Incompleteness and inconsistencies of ideas only becomes clear during implementation – pretty much putting a dagger into the idea that we can define everything up front and then code it just like that. This is certainly the area that the agile and lean approaches look to change and certainly the earlier we can try out different ideas by using approaches such as set based concurrent engineering the more quickly we can end up with a useful solution.
• An interesting idea about creating a surgical team ,with a few very experienced people doing the majority of the coding and being assisted by other members of the team, is suggested as being a successful route to delivering software. It sounds quite different to the teams that I have worked on where everyone on the team is involved although the objectives behind it seem valid – reducing the communication points and ensuring the conceptual integrity of the solution. Uncle Bob recently wrote about this describing these teams as master craftsman teams but it sounds as if this would require quite a radical shift in the recruiting strategies of organisations. Dave Hoover also has an interesting post on this subject but he takes the angle of building apprentices on teams like this.
• This seems closely linked to another idea about team composition described later on in the book which speaks of the need for a team to have a technical director and a producer – the technical director sounds to be quite similar to Toyota's idea of the Chief Engineer and they would be technically in charge while the producer (Iteration Manager?) is in charge of everything else. The underlying idea here is that we don't just have one person in charge of a team, there are two distinct and important roles.
• Brooks says the most important aspect of the design of a system is to ensure its conceptual integrity i.e. a consistent set of design ideas. In order to achieve this Brooks suggests the need for a system architect – while I agree with this idea I think it is more a role and maybe one that can be done by the Tech Lead on a project. The Poppendieck's also talk of the need for conceptual integrity in Lean Software Development. The point here is to create a system which is easy to use both in terms of function to conceptual complexity. I am reminded of a Dan North quote at this stage: "We're done not when there's nothing more to add, but when there's nothing more to take away"
• The productivity increases gained by using high level languages are mentioned – the underlying idea being that using these allow us to avoid an entire level of exposure to error. I think this makes sense and as an example I think the introduction of functional collection parameters into C# 3.0 will lead to a reduction in the amount of time spent debugging loop constructs since we no longer have to use these so frequently.
• When talking about object oriented programming Brooks speaks of the need to design objects which describe the concepts of the client.
   

  If we design large grained classes that address concepts our clients are already working with, they can understand and question the design as it grows, and they can cooperation in the design of test cases.

  In other words…Domain Driven Design! Reading this part of the book very much reminded me of Phil Will's QCon presentation where he spoke of the way that the business and software development teams at the Guardian were able to collaborate to drive the design of the domain model for their new website.

• The idea of only performing system debugging when each individual component actually works is something which should be obvious but is often not followed. If we know a component doesn't work on its own then we can guarantee it is not going to work when we try to integrate it with other components so the exercise seems slightly pointless to me. Common sense advice I think!
• Speaking of code reuse Brooks points out that the key here is the perceived cost of finding a component to reuse that is important – this ties in nicely with an idea from Dan Bergh Johnsson's QCon presentation

  Your API has 10-30 seconds to direct a programmer to the right spot before they implement it [the functionality] themselves

• Brooks talks of the need to have documentation for our projects – he uses a project workbook to do this and I think the modern day equivalent would be the project wiki. The idea of creating self documenting programs to help minimise the documentation that needs to be written is also covered. The importance of how we name concepts in our code is especially important in this area.
• The need to progressively refine the system by growing it rather than building it is suggested later on in the book – the limitations of the waterfall model are described and the approaches of agile/lean are pretty much described – building frequently, getting it working end to end, rapid prototyping and so on.

In Summary

I really enjoyed reading this book and seeing how a lot of the ideas in more modern methodologies were already known about in the 1980s and aren't in essence new ideas.

I'd certainly recommend this book.

[Dec 15, 2006] Computerworld - The mythical open source miracle by Neil McAllister

Actually Spolsky does not understand the role of scripting languages. But hi is right of target with his critique of OO. Object oriented programming is no silver bullet.

Dec 14, 2006 (InfoWorld) Joel Spolsky is one of our most celebrated pundits on the practice of software development, and he's full of terrific insight. In a recent blog post, he decries the fallacy of "Lego programming" -- the all-too-common assumption that sophisticated new tools will make writing applications as easy as snapping together children's toys. It simply isn't so, he says -- despite the fact that people have been claiming it for decades -- because the most important work in software development happens before a single line of code is written.

By way of support, Spolsky reminds us of a quote from the most celebrated pundit of an earlier generation of developers. In his 1987 essay "No Silver Bullet," Frederick P. Brooks wrote, "The essence of a software entity is a construct of interlocking concepts ... I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation ... If this is true, building software will always be hard. There is inherently no silver bullet."

As Spolsky points out, in the 20 years since Brooks wrote "No Silver Bullet," countless products have reached the market heralded as the silver bullet for effortless software development. Similarly, in the 30 years since Brooks published " The Mythical Man-Month" -- in which, among other things, he debunks the fallacy that if one programmer can do a job in ten months, ten programmers can do the same job in one month -- product managers have continued to buy into various methodologies and tricks that claim to make running software projects as easy as stacking Lego bricks.

Don't you believe it. If, as Brooks wrote, the hard part of software development is the initial design, then no amount of radical workflows or agile development methods will get a struggling project out the door, any more than the latest GUI rapid-development toolkit will.

And neither will open source. Too often, commercial software companies decide to turn over their orphaned software to "the community" -- if such a thing exists -- in the naive belief that open source will be a miracle cure to get a flagging project back on track. This is just another fallacy, as history demonstrates.

In 1998, Netscape released the source code to its Mozilla browser to the public to much fanfare, but only lukewarm response from developers. As it turned out, the Mozilla source was much too complex and of too poor quality for developers outside Netscape to understand it. As Jamie Zawinski recounts, the resulting decision to rewrite the browser's rendering engine from scratch derailed the project anywhere from six to ten months.

This is a classic example of the fallacy of the mythical man-month. The problem with the Mozilla code was poor design, not lack of an able workforce. Throwing more bodies at the project didn't necessarily help; it may have even hindered it. And while implementing a community development process may have allowed Netscape to sidestep its own internal management problems, it was certainly no silver bullet for success.

The key to developing good software the first time around is doing the hard work at the beginning: good design, and rigorous testing of that design. Fail that, and you've got no choice but to take the hard road. As Brooks observed all those years ago, successful software will never be easy. No amount of open source process will change that, and to think otherwise is just more Lego-programming nonsense.

McGee's Musings

I'm working on my weekly InfoWorld column (this one will run in print and online on March 8) and I'm referencing an essay from Frederick Brooks (of "Mythical Man-Month" fame) entitled "No Silver Bullet: Essence and Accidents of Software Engineering."

You just have to read this. I've read it many times before and referenced it in a column on web services two years ago, but the essay continues to amaze me. Although it was written eighteen years ago, the content still rings true. Just a sample:

The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract in that such a conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed.

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet.

The Mythical Man-Month Essays on Software Engineering REVIEWED BY TAL COHEN

From A Second Look at the Cathedral and Bazaar by Nikolai Bezroukov,
http://firstmonday.org/issues/issue4_12/bezroukov/index.html 

One of the most indefensible ideas of CatB (i.e. Raymond) is that Brooks' Law is non-applicable in the Internet-based distributed development environment as exemplified by Linux

The real problem with the CatB statement is that due to the popularity of CatB this statement could discourage OSS community from reading and studying The Mythical Man-Month, one of the few
computer science books that has remained current decades after its initial publication.

Actually the term "Brooks' Law" is usually formulated as "Adding manpower to a late software project makes it later".

I believe that the illusion of the non-applicability of "mythical man-month postulate" and Brooks' law is limited only to projects for which a fully functional prototype already exists and most or all architectural problems are solved...
 

Table of Contents

Recommended Links

The Mythical Man-Month - Wikipedia, the free encyclopedia -- pretty extensive review of Brooks' masterpiece...

Mythical Man-Month quotes

The mythical man-month (anniversary ed.) ACM ref list

ERCB The Mythical Man-Month, Anniversary Edition

Brooks' Law and open source The more the merrier

ONLamp.com The Mythical Man-Month Revisited -- a very superficial, but funny paper...

Amazon.com- Books- The Mythical Man-Month- Essays on Software ...

Publisher page

• Mythical Man-Month, The- Essays on Software Engineering ...

Reviews

• Blog
• Slashdot Review The Mythical Man Month Essays on Software Engineering
• Ian Alexander's Reviews of Books on Requirements Engineering and Related Subjects - Brooks on Mythical Man-Month
• FabTime Book Reviews The Mythical Man-Month by Frederick Brooks, Jr.

Conceptual Integrity

Fred Brooks

• Faculty Biography Frederick P. Brooks Jr. (UNC-CH Computer Science)
• Fred Brooks - Wikipedia, the free encyclopedia

Reviews

The Mythical Man-Month

An interesting review by Lane Core Jr.

Few books on software project management have been as influential and timeless as The Mythical Man-Month. With a blend of software engineering facts and thought-provoking opinions, Fred Brooks offers insight for anyone managing complex projects. [outside back cover, Twentieth Anniversary Edition, with four new chapters; Addison-Wesley, 1995]

Chapter 2: The Mythical Man-Month

Good cooking takes time. If you are made to wait, it is to serve you better, and to please you. Menu of Restaurant Antoine, New Orleans [page 13]

More software projects have gone awry for lack of calendar time than for all other causes combined. Why is this cause of disaster so common?

First, our techniques of estimating are poorly developed. More seriously, they reflect an unvoiced assumption which is quite untrue, i.e., that all will go well.

Second, our estimating techniques fallaciously confuse effort with progress, hiding the assumption that men and months are interchangeable.

Third, because we are uncertain of our estimates, software managers often lack the courteous stubbornness of Antoine's chef.

Fourth, schedule progress is poorly monitored. Techniques proven and routine in other engineering disciplines are considered radical innovations in software engineering.

Fifth, when schedule slippage is recognized, the natural (and traditional) response is to add manpower. Like dousing a fire with gasoline, this makes matters worse, much worse. More fire requires more gasoline, and thus begins a regenerative cycle which ends in disaster. [page 14]

Optimism

In many creative activities the medium of execution is intractable. Lumber splits; paints smear; electrical circuits ring. These physical limitations of the medium constrain the ideas that may be expressed, and they also create unexpected difficulties in the implementation. [page 15]

Computer programming, however, creates with an exceedingly tractable medium. The programmer builds from pure thought-stuff: concepts and very flexible representations thereof. Because the medium is tractable, we expect few difficulties in implementation; hence our pervasive optimism. Because our ideas are faulty, we have bugs; hence our optimism is unjustified. [page 15]

The Man-Month

The second fallacious thought mode is expressed in the very unit of effort used in estimating and scheduling: the man-month. Cost does indeed vary as the product of the number of men and the number of months. Progress does not. Hence the man-month as a unit for measuring the size of a job is a dangerous and deceptive myth. It implies that men and months are interchangeable.

When a task cannot be partitioned because of sequential constraints, the application of more effort has no effect on the schedule. The bearing of a child takes nine months, no matter how many women are assigned. Many software tasks have this characteristic because of the sequential nature of debugging. [page 17; emphasis in original]

Systems Test

In examining conventionally scheduled projects, I have found that few allowed one-half of the projected schedule for testing, but that most did indeed spend half of the actual schedule for that purpose. Many of these were on schedule until and except in system testing. [page 20]

Gutless Estimating

Observe that for the programmer, as for the chef, the urgency of the patron may govern the scheduled completion of the task, but it cannot govern the actual completion. An omelette, promised in two minutes, may appear to be progressing nicely. But when it has not set in two minutes, the customer has two choices--wait or eat it raw. Software customers have had the same choices.

The cook has another choice; he can turn up the heat. The result is often an omelette nothing can save--burned in one part, raw in another. [page 21]

Regenerative Schedule Disaster

Oversimplifying outrageously, we state Brooks's Law:

Adding manpower to a late software project makes it later.

This then is the demythologizing of the man-month. The number of months of a project depends upon its sequential constraints. The maximum number of men depends upon the number of independent subtasks. From these two quantities one can derive schedules using fewer men and more months. (The only risk is product obsolescence.) One cannot, however, get workable schedules using more men and fewer months. More software projects have gone awry for lack of calendar time than for all other causes combined. [pages 25-26; emphasis in original]

Chapter 11: Plan to Throw One Away

Two Steps Forward and One Step Back

The fundamental problem with program maintenance is that fixing a defect has a substantial (20-50 percent) chance of introducing another. So the whole process is two steps forward and one step back.

Why aren't defects fixed more cleanly? First, even a subtle defect shows itself as a local failure of some kind. In fact it often has system-wide ramifications, usually nonobvious. Any attempt to fix it with minimum effort will repair the local and obvious, but unless the structure is pure or the documentation very fine, the far-reaching effects of the repair will be overlooked. Second, the repairer is usually not the man who wrote the code, and often he is a junior programmer or trainee. [page 122]

Chapter 14: Hatching a Catastrophe

How does a project get to be a year late?... One day at a time. [page 153]

But the day-by-day slippage is harder to recognize, harder to prevent, harder to make up. Yesterday a key man was sick, and a meeting couldn't be held. Today the machines are all down, because lightning struck the building's power transformer. Tomorrow the disk routines won't start testing, because the first disk is a week late from the factory. Snow, jury duty, family problems, emergency meetings with customer, executive audits--the list goes on and on. Each one only postpones some activity by a half-day or a day. And the schedule slips, one day at a time. [page 154]

Milestones or Millstones?

For picking the milestones there is only one relevant rule. Milestones must be concrete, specific, measurable events, defined with knife-edge sharpness. Coding, for a counterexample, is "90 percent finished" for half of the total coding time. Debugging is "99 percent complete" most of the time. "Planning complete" is an event one can proclaim almost at will. [page 154]

Review  by Orville R. Weyrich, Jr.

I met this book shortly after its first publication, while I was a graduate student in chemistry who had just recently mastered the art of programming a Texas Instruments TI-59 calculator. I realized the limitations of the hand-held calculator when I managed to program some chemical analyses which required days of dedicated calculator-time to complete. At that point, I asked for an IBM-360 mainframe account and taught myself programming. Several thousand lines of Fortran-G later, I was ready for bigger and better things, and found The Mythical Man Month in the university bookstore.

Perhaps it is too melodramatic to say that The Mythical Man Month changed my life, so suffice it to say that after reading the book, I went on to get a job programming a PDP-11 for Knoxville Utility Board. After I got my degree in chemistry, I changed my career to software engineering and never looked back. Now almost twenty years later, after three years as Assistant Professor of Computer Science specializing in software engineering and a decade in private industry, this book still has a special place in my heart.

With the melodrama now dispatched, let me say that I wholeheartedly recommend this book for any manager of a software engineering project, and especially for anyone involved in a Year-2000 project.

A few more comments follow, arranged by chapter.

Chapter 1: The Tar Pit

Programming in-the-large seems to be like the tar pits that entrapped the prehistoric creatures such as the giant sloth, whose bones today grace the science library at the University of Georgia. Brooks writes:

 

Large-system programming has over the past decade been such a tar-pit, and many great and powerful beasts [including the IBM 360 project team] have thrashed violently in it. Most have emerged with running systems -- few have met goals, schedules, and budgets .... No one thing seems to cause the difficulty -- any particular paw can be pulled away. But the accumulation of simultaneous and interacting factors brings slower and slower motion. 

Chapter 2: The Mythical Man-Month

Develops the justification for "Brooks's Law"

 

Adding manpower to a late software project makes it later

Chapter 3: The Surgical Team

Following the lead of Harlan Mills, Brooks constructs an analogy between the roles necessary for a successful software engineering project, and the roles necessary for a surgical team. For many years, I have related best to the roles of "toolsmith" and "tester." 

Chapter 4: Aristocracy, Democracy, and System Design

In 1972 I was privileged to tour France as part of a joint expedition of the French and Religion departments of my college. Although I was there as a student of French, visits to many a Cathedral were on the itinerary due to the influence of the religion department. Brooks manages to make my experience relevant to software engineering:
 

Most European cathedrals show differences in plan or architectural style between parts built in different generations by different builders. The later builders were tempted to "improve" upon the designs of the earlier ones, to reflect the changes in fashion and differences in individual taste ... and the result proclaims the pridefulness of the builders as much as the glory of God.

Brooks goes on to relate this observation to the need for conceptual integrity in the design of a software system, and makes observations on how this can be achieved. 

Chapter 5: The Second System Effect

Beware the software engineer who has just completed his first successful system -- his next project runs the risk of dying of "featuritis". 

Chapter 6: Passing the Word

Say what you mean and mean what you say -- on the necessity of making the documentation match the implementation. 

Chapter 7: Why Did the Tower of Babel Fail?

Using additional analogies, Brooks points out that "The job least well done by project managers is to utilize the technical genius who is not strong on management talent." He then proceeds to illustrate his point with an example of how to do it right, taken from the fictional account of Heinlein's The Man Who Sold the Moon. 

Chapter 16: No Silver Bullet

This chapter, which was first published in IEEE Computer, [April, 1987] after the first edition of The Mythical Man Month, is included in the revised edition. It argues that CASE tools are useful adjuncts to software engineering, but are not a panacea.

Epilogue

I will give Frederick P. Brooks, Jr. the last word:
 

The tar pit of software engineering will continue to be sticky for a long time to come. Once can expect the human race to continue attempting systems just within or just beyond our reach; and software systems are perhaps the most intricate and complex of man's handiworks. The management of this complex craft will demand our best use of new languages and systems, our best adaptation of proven engineering management methods, liberal doses of common sense, and a God-given humility to recognize our fallibility and limitations.

Review The Mythical Man-Month Essays on Software Engineering

Review Discussion Mythical Man Month

Probably the biggest insight I took from the book (i.e. They talk about the "Mythical Man Month") is that the what kills timelines software projects more than anything else is communication overhead. Reducing the communication requirements between groups working on software is the single most important thing you can do during software development
----------------
Eric Raymond's "loophole" to Brooks law - primary development does not scale, debugging does.
-----------------
From A Second Look at the Cathedral and Bazaar by Nikolai Bezroukov, http://firstmonday.org/issues/issue4_12/bezroukov/index.html

One of the most indefensible ideas of CatB (i.e. Raymond) is that Brooks' Law is non-applicable in the Internet-based distributed development environment as exemplified by Linux The real problem with the CatB statement is that due to the popularity of CatB this statement could discourage OSS community from reading and studying The Mythical Man-Month, one of the few computer science books that has remained current decades after its initial publication. Actually the term "Brooks' Law" is usually formulated as "Adding manpower to a late software project makes it later". I believe that the illusion of the non-applicability of "mythical man-month postulate" and Brooks' law is limited only to projects for which a fully functional prototype already exists and most or all architectural problems are solved
----------------
Open Source vs. Brook (i.e. more progs later projects).
----------------------
Now I am reading the newly published Extreme Programming Explained by Kent Beck. He describes a fear of touching another programmer's code as a factor in slowing down many projects. I first read MMM over twenty years ago. Since then, I have been looking for any sign that any manager I worked for had even heard of the book. So far, no joy. Which is really scary, considering who my present employer
-------------------------------------
Brook also managed the development of the IBM 360 Operating System." Note that while manager of the 360 project it was Dr. Brooks who specified that a byte would consist of 8 bits. Whether or not you agree with his decision, it's hard to argue that this has not had a huge impact on the computer field. Tanner Lovelace UNC Chapel Hill

My 1 1/2 Cents

Wow! Nobody seems to have noticed a minor detail... IBM OS 360 looks to have been a major screw up.
And that's the lesson: If you screw up with style, then people will applaud you anyway... Especially if you are a computer scientist. (:-)

After all, computer science is after the first principles of the game, not just the "Implementation business"...

---

Mr. Fred Brooks must have been a politician in his time. Being in command for several thousand programmers must be more complex than the Job of a General.

And that leads us to Mr. Machiavelli's work "The Prince". In his time, Mr. Machiavelli was a seniour politician in the Republic of Florence. Later the Medicis returned to Florence, the Republic was junked and so was Mr. Machiavelli. Out of favors, out of Job, he found himself in his Palazo under house arrest. The only thing to do was to write a book with his reflections on the process of gaining and maintaining power.
And what? He is sending his writings to the new powers. Do you think the Medicis are stupid enough to hire an all to clever civil servant of the previous regime? Bingo.

So people like Fred Brooks and Machiavelli are probably very clever but not clever enough. Fouche, for that matter, never wrote books - he was a man of action.

MMM vs. OSS advocado.com discussion

Off the top of my head..., posted 13 Nov 2000 by rillian

For example, according to Brooks, opening development up wide should result in basically an infinite time-to-completion for a project, since all effort gets sucked up in inter-developer training and communication.

I think this one is easily explained by the different organizational structures. When a commercial development team hires someone, there's an expectation on both sides now that they're "inside the wall" that someone must be supported and involved and given work to do. Else, why did you hire them? The lower chemical potential of open development means those doing the work are more-or-less free to ignore the training of new contributors, who must learn on their own. With no schedule, or a recognizance of the value of an educational budget, this is no problem.

Microsoft chimes in to remark that the reason OSS has worked is that anarchy is OK at producing copies of systems which are already fully architected. The "taillights are easy to follow" principle.

The advantage of "chasing taillights" is you don't have to do much work to achieve a shared vision among the contributors. It's that vision that is hardest to pass on. I understand taking advantage of this is one of the reasons behind GNOME's emulation of Microsoft products. But the effect applies just as well to designs just over the horizon. How often have you read a new project description and said "Yes, I see what they want to do"?

Others have remonstrated that OSS projects aren't really bazaars, but autocratic cathedrals run by the elect, and so are perfectly understandable.

It is true that one effective model is to have circles of contributors, where new contributors are "trained" by a circle of more experienced developers, who in turn defer to core members. But the term "autocratic cathedral" poorly captures the complexity of the situation. This model is only autocratic in that--for reasons of resource efficiency, lack of distinct vision and respect for their technical competence--everyone follows the decisions made by the core members. And one is free to move between these roles as time and interest permit. In other words, not autocratic at all.

Perhaps (to further mix metaphors) this is a surgical team, drawing their new members from recent graduates of a custom-tailored university degree program, crossed with the social freedom one has in starting/joining a minor shared-interest club in a large city. I don't think there's much interest to be found in calling that "traditional top-down design" and understood.

On a more philosophical note, stephan's point about historical context is a valuable one, and applies just as well to the The Cathedral and the Bazaar. We shouldn't expect to find directly applicable rules in seminal works of this sort. Initial theories often have limited scope or are found to be wrong on some (even many) details. But the central idea still has value, and is either incorporated in later theories, or develops into a sophisticated field in its own right, sharing only the name and central premise of the original. (Darwin is a good example here.) We should keep this in mind and not be surprised when bold claims don't apply to our situation.

As a great friend used to say, "I think we can complicate this a bit." :-)

Mythical Man-Month

The essays in this book are concise, clear, and eminently readable. Brooks has a way of forcing you to question your implicit assumptions. In the title chapter on schedule slippage, he drives the reader humorously, but relentlessly, to this conclusion:

Adding manpower to a late software project makes it later.

In another essay, Brooks points out:

Chemical engineers learned long ago that a process that works in the laboratory cannot be implemented in a factory in one step. An intermediate step called the pilot plant is necessary....In most [software] projects, the first system is barely usable. It may be too slow, too big, awkward to use, or all three. There is no alternative but to start again, smarting but smarter, and build a redesigned version in which these problems are solved.... Delivering the throwaway to customers buys time, but it does so only at the cost of agony for the user, distraction for the builders while they do the redesign, and a bad reputation for the product that the best redesign will find hard to live down. Hence, plan to throw one away; you will, anyhow.

The 20th anniversary edition includes all of the chapters of the original book, plus a lot of new material. The chapter "No Silver Bullet--Essence and Accident in Software Engineering" was first published in 1986. Brooks asserted that no single software engineering development would produce an order-of-magnitude improvement in programming productivity within ten years. While this paper caused a lot of rebuttal in the software engineering community, Brooks was right--there has been "No Silver Bullet."

The other chapters discuss why this is so and Brooks points out one of his mistakes in the first edition--"David Parnas Was Right, and I Was Wrong About Information Hiding." Brooks states, "I am now convinced that information hiding, today often embodied in object programming, is the only way of raising the level of software design."

If you create, maintain, manage, or are involved in part of the software engineering process, you must read The Mythical Man Month. From the hard-nosed advice to the enjoyable anecdotes, you will enjoy this book.

Notes on Fred Brooks' 'The Mythical Man-Month'

Excerpts

InformIT Mythical Man-Month, The Essays on Software Engineering, Anniversary Edition, 2nd Edition

Preface

To my surprise and delight, The Mythical Man-Month continues to be popular after twenty years. Over 250,000 copies are in print. People often ask which of the opinions and recommendations set forth in 1975 I still hold, and which have changed, and how. Whereas I have from time to time addressed that question in lectures, I have long wanted to essay it in writing.

Peter Gordon, now a Publishing Partner at Addison-Wesley, has been working with me patiently and helpfully since 1980. He proposed that we prepare an Anniversary Edition. We decided not to revise the original, but to reprint it untouched (except for trivial corrections) and to augment it with more current thoughts.

Chapter 16 reprints "No Silver Bullet: Essence and Accidents of Software Engineering," a 1986 IFIPS paper that grew out of my experience chairing a Defense Science Board study on military software. My co-authors of that study, and our executive secretary, Robert L. Patrick, were invaluable in bringing me back into touch with real-world large software projects. The paper was reprinted in 1987 in the IEEE Computer magazine, which gave it wide circulation.

"No Silver Bullet" proved provocative. It predicted that a decade would not see any programming technique which would by itself bring an order-of-magnitude improvement in software productivity. The decade has a year to run; my prediction seems safe. "NSB" has stimulated more and more spirited discussion in the literature than has The Mythical Man-Month. Chapter 17, therefore, comments on some of the published critique and updates the opinions set forth in 1986.

In preparing my retrospective and update of The Mythical Man-Month, I was struck by how few of the propositions asserted in it have been critiqued, proven, or disproven by ongoing software engineering research and experience. It proved useful to me now to catalog those propositions in raw form, stripped of supporting arguments and data. In hopes that these bald statements will invite arguments and facts to prove, disprove, update, or refine those propositions, I have included this outline as Chapter 18.

Chapter 19 is the updating essay itself. The reader should be warned that the new opinions are not nearly so well informed by experience in the trenches as the original book was. I have been at work in a university, not industry, and on small-scale projects, not large ones. Since 1986, I have only taught software engineering, not done research in it at all. My research has rather been on virtual reality and its applications.

In preparing this retrospective, I have sought the current views of friends who are indeed at work in software engineering. For a wonderful willingness to share views, to comment thoughtfully on drafts, and to re-educate me, I am indebted to Barry Boehm, Ken Brooks, Dick Case, James Coggins, Tom DeMarco, Jim McCarthy, David Parnas, Earl Wheeler, and Edward Yourdon. Fay Ward has superbly handled the technical production of the new chapters.

I thank Gordon Bell, Bruce Buchanan, Rick Hayes-Roth, my colleagues on the Defense Science Board Task Force on Military Software, and, most especially, David Parnas for their insights and stimulating ideas for, and Rebekah Bierly for technical production of, the paper printed here as Chapter 16. Analyzing the software problem into the categories of essence and accident was inspired by Nancy Greenwood Brooks, who used such analysis in a paper on Suzuki violin pedagogy.

Addison-Wesley's house custom did not permit me to acknowledge in the 1975 Preface the key roles played by their staff. Two persons' contributions should be especially cited: Norman Stanton, then Executive Editor, and Herbert Boes, then Art Director. Boes developed the elegant style, which one reviewer especially cited: "wide margins, and imaginative use of typeface and layout." More important, he also made the crucial recommendation that every chapter have an opening picture. (I had only the Tar Pit and Rheims Cathedral at the time.) Finding the pictures occasioned an extra year's work for me, but I am eternally grateful for the counsel.

Deo soli gloria or Soli Deo Gloria -- To God alone be the glory.

Chapel Hill, N.C., F.
 

Why is programming fun- - The Mythical Man Month

Why is programming fun? What delights may its practitioner expect as his reward?

First is the sheer joy of making things. As the child delights in his mud pie, so the adult enjoys building things, especially things of his own design. I think this delight must be an image of God's delight in making things, a delight shown in the distinctiveness of each leaf and each snowflake.

Second is the pleasure of making things that are useful to other people. Deep within, we want others to use our work and to find it helpful. In this respect the programming system is not essentially different from the child's first clay pencil holder "for Daddy's office."

Third is the fascination of fashioning complex puzzle-like objects of interlocking moving parts and watching them work in subtle cycles, playing out the consequences of principles built in from the beginning. The programmed computer has all the fascination of the pinball machine or the jukebox mechanism, carried to the ultimate.

Fourth is the joy of always learning, which springs from the nonrepeating nature of the task. In one way or another the problem is ever new, and its solver learns something: sometimes practical, sometimes theoretical, and sometimes both.

Finally, there is the delight of working in such a tractable medium. The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures. (As we shall see later, this tractability has its own problems.)

Yet the program construct, unlike the poet's words, is real in the sense that it moves and works, producing visible outputs separately from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be.

Programming then is fun because it gratifies creative longings built deep within us and delights sensibilities we have in common with all men.

Chapter 4

This great church is an incomparable work of art. There is neither aridity nor confusion in the tenets it sets forth.

If is the zenith of a style, the work of artists who had understood and assimilated all their predecessors' successes, in complete possession of the techniques of their times, but using them without indiscreet display nor gratuitous feats of skill.

It was Jean d'Orbais who undoubtedly conceived the general plan of the building, a plan which was respected, at least in its essential elements, by his successors. This is one of the reasons for the extreme coherence and unity of the edifice.

Conceptual Integrity
Most European cathedrals show differences in plan or architectural style between parts built in different generations by different builders. The later builders were tempted to improve upon the designs of the earlier ones, to reflect both changes in fashion and differences in individual taste. So the peaceful Norman transept abuts and contradicts the soaring Gothic nave, and the result proclaims the pridefulness of the builders as much as the glory of God.

Against these, the architectural unity of Reims stands in glorious contrast. The joy that stirs the beholder comes as much from the integrity of the design as from any particular excellences. As the guidebook tells, this integrity was achieved by the self-abnegation of eight generations of builders, each of whom sacrificed some of his ideas so that the whole might be of pure design. The result proclaims not only the glory of God, but also His power to salvage fallen men from their pride.

Even though they have not taken centuries to build, most programming systems reflect conceptual disunity far worse than that of cathedrals. Usually this arises not from a serial succession of master designers, but from the separation of design into many tasks done by many men.

I will contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas. In this chapter and the next two, we will examine the consequences of this theme for programming system design:

• How is conceptual integrity to be achieved?
• Does not this argument imply an elite, or aristocracy of architects, and a horde of plebeian implementers whose creative talents and ideas are suppressed?
• How does one keep the architects from drifting off into the blue with unimplementable or costly specifications?
• How does one ensure that every trifling detail of an architectural specification gets communicated to the implementer, properly understood by him, and accurately incorporated into the product?

Achieving Conceptual Integrity
The purpose of a programming system is to make a computer easy to use. To do this, it furnishes languages and various facilities that are in fact programs invoked and controlled by language features. But these facilities are bought at a price: the external description of a programming system is ten to twenty times as large as the external description of the computer system itself. The user finds it far easier to specify any particular function, but there are far more to choose from, and far more options and formats to remember.

Ease of use is enhanced only if the time gained in functional specification exceeds the time lost in learning, remembering, and searching manuals. With modern programming systems this gain does exceed the cost, but in recent years the ratio of gain to cost seems to have fallen as more and more complex functions have been added. I am haunted by the memory of the ease of use of the IBM 650, even without an assembler or any other software at all.

Because ease of use is the purpose, this ratio of function to conceptual complexity is the ultimate test of system design. Neither function alone nor simplicity alone defines a good design.

This point is widely misunderstood. Operating System/360 is hailed by its builders as the finest ever built, because it indisputably has the most function. Function, and not simplicity, has always been the measure of excellence for its designers. On the other hand, the Time-Sharing System for the PDP-10 is hailed by its builders as the finest, because of its simplicity and the spareness of its concepts. By any measure, however, its function is not even in the same class as that of OS/360. As soon as ease of use is held up as the criterion, each of these is seen to be unbalanced, reaching for only half of the true goal.

For a given level of function, however, that system is best in which one can specify things with the most simplicity and straightforwardness. Simplicity is not enough. Mooers's TRAC language and Algol 68 achieve simplicity as measured by the number of distinct elementary concepts. They are not, however, straightforward. The expression of the things one wants to do often requires involuted and unexpected combinations of the basic facilities. It is not enough to learn the elements and rules of combination; one must also learn the idiomatic usage, a whole lore of how the elements are combined in practice. Simplicity and straightforwardness proceed from conceptual integrity. Every part must reflect the same philosophies and the same balancing of desiderata. Every part must even use the same techniques in syntax and analogous notions in semantics. Ease of use, then, dictates unity of design, conceptual integrity.

Aristocracy and Democracy
Conceptual integrity in turn dictates that the design must proceed from one mind, or from a very small number of agreeing resonant minds.

Schedule pressures, however, dictate that system building needs many hands. Two techniques are available for resolving this dilemma. The first is a careful division of labor between architecture and implementation. The second is the new way of structuring programming implementation teams discussed in the previous chapter.

The separation of architectural effort from implementation is a very powerful way of getting conceptual integrity on very large projects. I myself have seen it used with great success on IBM's Stretch computer and on the System/360 computer product line. I have seen it fail through lack of application on Operating System/360.

By the architecture of a system, I mean the complete and detailed specification of the user interface. For a computer this is the programming manual. For a compiler it is the language manual. For a control program it is the manuals for the language or languages used to invoke its functions. For the entire system it is the union of the manuals the user must consult to do his entire job.

The architect of a system, like the architect of a building, is the user's agent. It is his job to bring professional and technical knowledge to bear in the unalloyed interest of the user, as opposed to the interests of the salesman, the fabricator, etc.

Architecture must be carefully distinguished from implementation. As Blaauw has said, "Where architecture tells what happens, implementation tells how it is made to happen." He gives as a simple example a clock, whose architecture consists of the face, the hands, and the winding knob. When a child has learned this architecture, he can tell time as easily from a wristwatch as from a church tower. The implementation, however, and its realization, describe what goes on inside the case--powering by any of many mechanisms and accuracy control by any of many.

In System/360, for example, a single computer architecture is implemented quite differently in each of some nine models. Conversely, a single implementation, the Model 30 data flow, memory, and microcode, serves at different times for four different architectures: a System/360 computer, a multiplex channel with up to 224 logically independent subchannels, a selector channel, and a 1401 computer.

The same distinction is equally applicable to programming systems. There is a U.S. standard Fortran IV. This is the architecture for many compilers. Within this architecture many implementations are possible: text-in-core or compiler-in-core, fast-compile or optimizing, syntax-directed or ad-hoc. Likewise any assembler language or job-control language admits of many implementations of the assembler or scheduler.

Now we can deal with the deeply emotional question of aristocracy versus democracy. Are not the architects a new aristocracy, an intellectual elite, set up to tell the poor dumb implementers what to do? Has not all the creative work been sequestered for this elite, leaving the implementers as cogs in the machine? Won't one get a better product by getting the good ideas from all the team, following a democratic philosophy, rather than by restricting the development of specifications to a few?

As to the last question, it is the easiest. I will certainly not contend that only the architects will have good architectural ideas. Often the fresh concept does come from an implementer or from a user. However, all my own experience convinces me, and I have tried to show, that the conceptual integrity of a system determines its ease of use. Good features and ideas that do not integrate with a system's basic concepts are best left out. If there appear many such important but incompatible ideas, one scraps the whole system and starts again on an integrated system with different basic concepts.

As to the aristocracy charge, the answer must be yes and no. Yes, in the sense that there must be few architects, their product must endure longer than that of an implementer, and the architect sits at the focus of forces which he must ultimately resolve in the user's interest. If a system is to have conceptual integrity, someone must control the concepts. That is an aristocracy that needs no apology.

No, because the setting of external specifications is not more creative work than the designing of implementations. It is just different creative work. The design of an implementation, given an architecture, requires and allows as much design creativity, as many new ideas, and as much technical brilliance as the design of the external specifications. Indeed, the cost-performance ratio of the product will depend most heavily on the implementer, just as ease of use depends most heavily on the architect.

There are many examples from other arts and crafts that lead one to believe that discipline is good for art. Indeed, an artist's aphorism asserts, "Form is liberating." The worst buildings are those whose budget was too great for the purposes to be served. Bach's creative output hardly seems to have been squelched by the necessity of producing a limited-form cantata each week. I am sure that the Stretch computer would have had a better architecture had it been more tightly constrained; the constraints imposed by the System/360 Model 30's budget were in my opinion entirely beneficial for the Model 75's architecture.

Similarly, I observe that the external provision of an architecture enhances, not cramps, the creative style of an implementing group. They focus at once on the part of the problem no one has addressed, and inventions begin to flow. In an unconstrained implementing group, most thought and debate goes into architectural decisions, and implementation proper gets short shrift.

This effect, which I have seen many times, is confirmed by R. W. Conway, whose group at Cornell built the PLIC compiler for the PLII language. He says, "We finally decided to implement the language unchanged and unimproved, for the debates about language would have taken all our effort."

What Does the Implementer Do While Waiting?
It is a very humbling experience to make a multimillion-dollar mistake, but it is also very memorable. I vividly recall the night we decided how to organize the actual writing of external specifications for OS/360. The manager of architecture, the manager of control program implementation, and I were threshing out the plan, schedule, and division of responsibilities.

The architecture manager had 10 good men. He asserted that they could write the specifications and do it right. It would take ten months, three more than the schedule allowed.

The control program manager had 150 men. He asserted that they could prepare the specifications, with the architecture team coordinating; it would be well-done and practical, and he could do it on schedule Furthermore, if the architecture team did it, his 150 men would sit twiddling their thumbs for ten months.

To this the architecture manager responded that if I gave the control program team the responsibility, the result would not in fact be on time, but would also be three months late, and of much lower quality. I did, and it was. He was right on both counts. Moreover, the lack of conceptual integrity made the system far more costly to build and change, and I would estimate that it added a year to debugging time.

Many factors, of course, entered into that mistaken decision; but the overwhelming one was schedule time and the appeal of putting all those 150 implementers to work. It is this siren song whose deadly hazards I would now make visible.

When it is proposed that a small architecture team in fact write all the external specifications for a computer or a programming system, the implementers raise three objections:

• The specifications will be too rich in function and will not reflect practical cost considerations.
• The architects will get all the creative fun and shut out the inventiveness of the implementers.
• The many implementers will have to sit idly by while the specifications come through the narrow funnel that is the architecture team.

The first of these is a real danger, and it will be treated in the next chapter. The other two are illusions, pure and simple. As we have seen above, implementation is also a creative activity of the first order. The opportunity to be creative and inventive in implementation is not significantly diminished by working within a given external specification, and the order of creativity may even be enhanced by that discipline. The total product will surely be.

The last objection is one of timing and phasing. A quick answer is to refrain from hiring implementers until the specifications are complete. This is what is done when a building is constructed.

In the computer systems business, however, the pace is quicker, and one wants to compress the schedule as much as possible. How much can specification and building be overlapped?

As Blaauw points out, the total creative effort involves three distinct phases: architecture, implementation, and realization. It turns out that these can in fact be begun in parallel and proceed simultaneously.

In computer design, for example, the implementer can start as soon as he has relatively vague assumptions about the manual, somewhat clearer ideas about the technology, and well-defined cost and performance objectives. He can begin designing data flows, control sequences, gross packaging concepts, and so on. He devises or adapts the tools he will need, especially the record-keeping system, including the design automation system.

Meanwhile, at the realization level, circuits, cards, cables, frames, power supplies, and memories must each be designed, refined, and documented. This work proceeds in parallel with architecture and implementation.

The same thing is true in programming system design. Long before the external specifications are complete, the implementer has plenty to do. Given some rough approximations as to the function of the system that will be ultimately embodied in the external specifications, he can proceed. He must have well-defined space and time objectives. He must know the system configuration on which his product must run. Then he can begin designing module boundaries, table structures, pass or phase breakdowns, algorithms, and all kinds of tools. Some time, too, must be spent in communicating with the architect.

Meanwhile, on the realization level there is much to be done also. Programming has a technology, too. If the machine is a new one, much work must be done on subroutine conventions, supervisory techniques, searching and sorting algorithms.

Conceptual integrity does require that a system reflect a single philosophy and that the specification as seen by the user flow from a few minds. Because of the real division of labor into architecture, implementation, and realization, however, this does not imply that a system so designed will take longer to build. Experience shows the opposite, that the integral system goes together faster and takes less time to test. In effect, a widespread horizontal division of labor has been sharply reduced by a vertical division of labor, and the result is radically simplified communications and improved conceptual integrity.

No Silver Bullet Essence and Accidents of Software Engineering

From The Mythical Man-Month, Anniversary Edition, pages 7-8.

No Silver Bullet Essence and Accidents of Software Engineering -- Famous paper by F. Brooks

Fashioning complex conceptual constructs is the essence; accidental tasks arise in representing the constructs in language. Past progress has so reduced the accidental tasks that future progress now depends upon addressing the essence.

Of all the monsters that fill the nightmares of our folklore, none terrify more than werewolves, because they transform unexpectedly from the familiar into horrors. For these, one seeks bullets of silver that can magically lay them to rest.

The familiar software project, at least as seen by the non-technical manager, has something of this character; it is usually innocent and straightforward, but is capable of becoming a monster of missed schedules, blown budgets, and flawed products. So we hear desperate cries for a silver bullet--something to make software costs drop as rapidly as computer hardware costs do.

But, as we look to the horizon of a decade hence, we see no silver bullet. There is no single development, in either technology or in management technique, that by itself promises even one order-of-magnitude improvement in productivity, in reliability, in simplicity. In this article, I shall try to show why, by examining both the nature of the software problem and the properties of the bullets proposed.

Skepticism is not pessimism, however. Although we see no startling breakthroughs--and indeed, I believe such to be inconsistent with the nature of software--many encouraging innovations are under way. A disciplined, consistent effort to develop, propagate, and exploit these innovations should indeed yield an order of-magnitude improvement. There is no royal road, but there is a road.

The first step toward the management of disease was replacement of demon theories and numerous theories by the germ theory. That very step, the beginning of hope, in itself dashed all hopes of magical solutions. It told workers that progress would be made stepwise, at great effort, and that a persistent, unremitting care would have to be paid to a discipline of cleanliness. So it is with software engineering today.

Does it have to be hard? --Essential difficulties

Not only are there no silver bullets now in view, the very nature of software makes it unlikely that there will be any--no inventions that will do for software productivity, reliability, and simplicity what electronics, transistors, and large-scale integration did for computer hardware.

We cannot expect ever to see two fold gains every two years.

First, one must observe that the anomaly is not that software progress is so slow, but that computer hardware progress is so fast. No other technology since civilization began has seen six orders of magnitude in performance-price gain in 30 years. In no other technology can one choose to take the gain in either improved performance or in reduced costs. These gains flow from the transformation of computer manufacture from an assembly industry into a process industry.

Second, to see what rate of progress one can expect in software technology, let us examine the difficulties of that technology. Following Aristotle, I divide them into essence, the difficulties inherent in the nature of software, and accidents, those difficulties that today attend its production but are not inherent.

The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract in that such a conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed.

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet.

Let us consider the inherent properties of this irreducible essence of modern software systems: complexity, conformity, changeability, and invisibility.

Complexity. Software entities are more complex for their size than perhaps any other human construct because no two parts are alike (at least above the statement level). If they are, we make the two similar parts into a subroutine--open or closed. In this respect, software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound.

Digital computers are themselves more complex than most things people build: They have very large numbers of states. This makes conceiving, describing, and testing them hard. Software systems have orders-of-magnitude more states than computers do.

Likewise, a scaling-up of a software entity is not merely a repetition of the same elements in larger sizes; it is necessarily an increase in the number of different elements. In most cases, the elements interact with each other in some nonlinear fashion, and the complexity of the whole increases much more than linearly.

The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence. For three centuries, mathematics and the physical sciences made great strides by constructing simplified models of complex phenomena, deriving properties from the models, and verifying those properties by experiment. This paradigm worked because the complexities ignored in the models were not the essential properties of the phenomena. It does not work when the complexities are the essence.

Many of the classic problems of developing software products derive from this essential complexity and its nonlinear increases with size. From the complexity comes the difficulty of communication among team members, which leads to product flaws, cost overruns, and schedule delays. From the complexity comes the difficulty of enumerating, much less understanding, all the possible states of the program, and from that comes the unreliability. From complexity of function comes the difficulty of invoking function, which makes programs hard to use. From complexity of structure comes the difficulty of extending programs to new functions without creating side effects. From complexity of structure come the unvisualized states that constitute security trapdoors.

Not only technical problems, but management problems as well come from the complexity. It makes overview hard, thus impeding conceptual integrity. It makes it hard to find and control all the loose ends. It creates the tremendous learning and understanding burden that makes personnel turnover a disaster.

Conformity. Software people are not alone in facing complexity. Physics deals with terribly complex objects even at the "fundamental" level [BJC:]. The physicist labors on, however, in a firm faith that there are unifying principles to be found, whether in quarks or in unified field theory. Einstein argued that there must be simplified [BJC: undecipherable] of nature, because God is not capricious or arbitrary.

No such faith comforts the software engineer. Much of the complexity that he must master is arbitrary complexity, forced without rhyme or reason by the many human institutions and systems to which his interfaces must conform. These differ from interface to interface, and from time to time, not because of necessity but only because they were designed by different people, rather than by God.

In many cases, the software must conform because it is the most recent arrival on the scene. In others, it must conform because it is perceived as the most conformable. But in all cases, much complexity comes from conformation to other interfaces; this complexity cannot be simplified out by any redesign of the software alone.

Quotes

• Good cooking takes time. If you are made to wait, it is to serve you better, and to please you. Menu of Restaurant Antoine, New Orleans. [page 13]
• More software projects have gone awry for lack of calendar time than for all other causes combined. Why is this cause of disaster so common?
• First, our techniques of estimating are poorly developed. More seriously, they reflect an unvoiced assumption which is quite untrue, i.e., that all will go well.
• Second, our estimating techniques fallaciously confuse effort with progress, hiding the assumption that men and months are interchangeable.
• Third, because we are uncertain of our estimates, software managers often lack the courteous stubbornness of Antoine's chef.
• Fourth, schedule progress is poorly monitored. Techniques proven and routine in other engineering disciplines are considered radical innovations in software engineering.
• Fifth, when schedule slippage is recognized, the natural (and traditional) response is to add manpower. Like dousing a fire with gasoline, this makes matters worse, much worse. More fire requires more gasoline, and thus begins a regenerative cycle which ends in disaster. [page 14]
• In many creative activities the medium of execution is intractable. Lumber splits; paints smear; electrical circuits ring. These physical limitations of the medium constrain the ideas that may be expressed, and they also create unexpected difficulties in the implementation. [page 15]
• Computer programming, however, creates with an exceedingly tractable medium. The programmer builds from pure thought-stuff: concepts and very flexible representations thereof. Because the medium is tractable, we expect few difficulties in implementation; hence our pervasive optimism. Because our ideas are faulty, we have bugs; hence our optimism is unjustified. [page 15]
• The second fallacious thought mode is expressed in the very unit of effort used in estimating and scheduling: the man-month. Cost does indeed vary as the product of the number of men and the number of months. Progress does not. Hence the man-month as a unit for measuring the size of a job is a dangerous and deceptive myth. It implies that men and months are interchangeable.
• When a task cannot be partitioned because of sequential constraints, the application of more effort has no effect on the schedule. The bearing of a child takes nine months, no matter how many women are assigned. Many software tasks have this characteristic because of the sequential nature of debugging. [page 17; emphasis in original]
• In examining conventionally scheduled projects, I have found that few allowed one-half of the projected schedule for testing, but that most did indeed spend half of the actual schedule for that purpose. Many of these were on schedule until and except in system testing. [page 20]
• Observe that for the programmer, as for the chef, the urgency of the patron may govern the scheduled completion of the task, but it cannot govern the actual completion. An omelette, promised in two minutes, may appear to be progressing nicely. But when it has not set in two minutes, the customer has two choices--wait or eat it raw. Software customers have had the same choices.
• The cook has another choice; he can turn up the heat. The result is often an omelette nothing can save--burned in one part, raw in another. [page 21]
• Adding manpower to a late software project makes it later.
• This then is the demythologizing of the man-month. The number of months of a project depends upon its sequential constraints. The maximum number of men depends upon the number of independent subtasks. From these two quantities one can derive schedules using fewer men and more months. (The only risk is product obsolescence.) One cannot, however, get workable schedules using more men and fewer months. More software projects have gone awry for lack of calendar time than for all other causes combined. [pages 25-26; emphasis in original]
• The fundamental problem with program maintenance is that fixing a defect has a substantial (20-50 percent) chance of introducing another. So the whole process is two steps forward and one step back.
• Why aren't defects fixed more cleanly? First, even a subtle defect shows itself as a local failure of some kind. In fact it often has system-wide ramifications, usually nonobvious. Any attempt to fix it with minimum effort will repair the local and obvious, but unless the structure is pure or the documentation very fine, the far-reaching effects of the repair will be overlooked. Second, the repairer is usually not the man who wrote the code, and often he is a junior programmer or trainee. [page 122]
• How does a project get to be a year late?... One day at a time. [page 153]
• But the day-by-day slippage is harder to recognize, harder to prevent, harder to make up. Yesterday a key man was sick, and a meeting couldn't be held. Today the machines are all down, because lightning struck the building's power transformer. Tomorrow the disk routines won't start testing, because the first disk is a week late from the factory. Snow, jury duty, family problems, emergency meetings with customer, executive audits--the list goes on and on. Each one only postpones some activity by a half-day or a day. And the schedule slips, one day at a time. [page 154]
• For picking the milestones there is only one relevant rule. Milestones must be concrete, specific, measurable events, defined with knife-edge sharpness. Coding, for a counterexample, is "90 percent finished" for half of the total coding time. Debugging is "99 percent complete" most of the time. "Planning complete" is an event one can proclaim almost at will. [page 154]

Humor

Discussion Mythical Man Month

My 1 1/2 Cents

Wow! Nobody seems to have noticed a minor detail... IBM OS 360 looks to have been a major screw up. And that's the lesson: If you screw up with style, then people will applaud you anyway... Especially if you are a computer scientist. (:-)

After all, computer science is after the first principles of the game, not just the "Implementation business"...

Random Findings

Brooks on optimism of programmers

All programmers are optimists. Perhaps this modern sorcery especially attracts those who believe in happy endings and fairy godmothers. Perhaps the hundreds of nitty frustrations drive away all but those who habitually focus on the end goal. Perhaps it is merely that computers are young, programmers are younger, and the young are always optimists. But however the selection process works, the result is indisputable:
"This time it will surely run," or "I just found the last bug."

-- Frederick Brooks, Jr., The Mythical Man Month

Brooks' Law and open source The more the merrier

Etc

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg  : C++ Humor : ARE YOU A BBS ADDICT? : Object oriented programmers of all nations :  C Humor : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor :  PseudoScience Related Humor : Networking Humor  : Shell Humor: Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : The Most Comprehensive Collection of Editor-related Humor : Microsoft plans to buy Catholic Church : Education Humor : IBM Humor :  Assembler-related Humor :  VIM Humor Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor : Best Russian Programmer Humor : Russian Musical Humor : The Perl Purity Test :  Politically Incorrect Humor : GPL-related Humor : OFM Humor : IDS Humor : Real Programmers Humor : Scripting Humor : Web Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor :

The Last but not Least

Copyright © 1996-2013 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine. This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting hosting of this site with different providers to distribute and speed up access. Currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: January 21, 2013