Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Fighting Software Overcomplexity and Relevance of the KISS Principle in SE

Bloat is caused by feature creep at every layer, not just programming layer or feature set layer. Complex software systems definitely can be build, but never can be fully comprehended.

Only by breaking a project down into manageable parts can we hope to interact with it in an effective manner. Furthermore, the inertia against changes is much less severe when they are small and simple. this idea was expressed in different ways in such popular quotes as Ockham’s razor, Einstein’s statement about the simplicity of theories or simply reciting the KISS (Keep It Simple Stupid) mantra. But independently of what quote about the value of simplicity we prefer, reducing complexity is of  paramount importance in software.

But this is known to be a tremendously difficult task that requires a lot of innovation and self-discipline.  One fruitful approach is raising the level of your implementation language and using scripting language along with lower level language instead of using just one.

Also adding features until you convert your product into a Christmas tree is easy and natural strategy, almost irresistible...  Actually that's why I value scripting languages so highly: they provide a novel way to reduce (actually not exactly reduce, but at least hide) the complexity permitting the developer to operate at the higher level of abstractions. And it permit creating more complex and powerful systems that have shorter and thus more maintainable code.

Some very good analogies are used to explain the principles, with my favorite being the broken window tale. The basic story is simple: abandoned buildings (or automobiles on the street) remain untouched until a window is broken. Left un-repaired, this sends a message that the object is fair game so within a very short time, vandals destroy the rest. The same thing happens in software development. Once a sub par feature is passed as acceptable, the signal to everyone is clear, and the quality of the remaining work suffers.

This is especially true for general purpose libraries. Once they became "too generally purpose" they became useless. Good example is glib 2.x vs glib 1.x:  glib-1.2.10 is 1/2 of uclibc in size. glib-2.2.2 is 2 times uclibc. For times growth  with very little useful functionality premium. Here is an interesting post on this topic from Tim Hockin:

On Sun, Oct 31, 2004 at 01:11:07AM +0300, Denis Vlasenko wrote:
> I am not a code genius, but want to help.
>
> Hmm probably some bloat-detection tools would be helpful,
> like "show me source_lines/object_size ratios of fonctions in
> this ELF object file". Those with low ratio are suspects of
> excessive inlining etc.

The problem with apps of this sort is the multiple layers of abstraction.

Xlib, GLib, GTK, GNOME, Pango, XML, etc.

No one wants to duplicate effort (rightly so).  Each of these libs tries to do EVERY POSSIBLE thing.  They all end up bloated.  Then you have to link them all in.  You end up bloated.  Then it is very easy to rely on
those libs for EVERYTHING, rather thank actually thinking.

So you end up with the mindset of, for example, "if it's text it's XML". You have to parse everything as XML, when simple parsers would be tons faster and simpler and smaller.

Bloat is cause by feature creep at every layer, not just the app.

Youck.
 

Over the history of software development (or at least since advent of IBM mainframes) bigger software was often equated with being better software. Commercial companies like Adobe (Acrobat 6 is really horrible bloatware, probably the champion of the field), IBM (Webshere, Tivoli, you name it), Microsoft (Windows XP; although it can be trimmed to barebones) and Oracle have a stake in producing big complex software. At the same time products like Excel while big are surprisingly flexible and robust. I would not call Excel 2003 bloatware but it definitely several times bigger then, say, Excel 97 which has approximately 80% of Excel 2003 functionality.

Linux suffered from the same disease (especially in Red Hat and Suse distributions) and in the level of bloat Linux is pretty close to Windows.

Regarding the eternal vicious circle of bloated software faster processors ->  even more bloated software  -> even faster processors, do any of you honestly suspect there might be some kind of liason between the parties concerned in order to perpetuate these largely unnecessary upgrades :-)

I have 2 Windows PC's: PC A  (Celeron 1.3, 512M of RAM runs Win2003 / Office 2003 and is ~3 years old. PC B (Celeron 1.2, 512M of RAM runs Windows 98 and Office 97 (dual booted with Red Hat 8). I don't have any problems moving data between the two systems at all. Aside from Excel I do not see relevant increases in functionality of the MS Office applications.

There may be a conspiracy between the hardware manufactures and software designers in order to design software to use up these CPU cycles and eat RAM like there's no tomorrow...

I also have several Linux PCs: an old Compaq laptop (133MHz, 96M of RAM) that is running Caldera 1.2 and newer Celeron 1.2/512M of ram desktop running Red Hat 8. Sometimes I honestly think that the laptop is faster ;-). I know Gnome is a horrible desktop manager, but still to achieve this effect you need to be very talented bloatware designer.

Tradeoffs Connected with the Simplicity

There is no free lunch. Many pressures tend to make programs more complicated (and therefore more expensive and buggy). One is technical machismo. Programmers are bright people who are (justly) proud of their ability to handle complexity and juggle abstractions. Often they compete with their peers to see who can build the most intricate and beautiful complexities. Just as often, their ability to design outstrips their ability to implement and debug, and the result is an expensive failure.

Often (at least in the commercial software world) excessive complexity comes from project requirements that are based on the marketing fad of the month rather than the reality of what customers want or software can actually deliver. Also complexity in commercial products is a time-tested defense against competitors. Many a good design has been smothered under marketing's pile of “check-list features” — features which, few customers benefit from. But here you can trap competition pretty nicely: usually competitors feel that they has to compete with chrome by adding more chrome. They forget that chrome tend to benefit the first comer. And that along tend to protect you from all, but the most talented competitors who can transcend this "more chrome" strategy and concentrate on better functionality and compatibility. For everybody else massive bloat naturally diminishes compatibility and leads to incompatibilities that segment the field; your former competitor suddenly moves into a different niche or just die because he does not have the same resources as you. Look at Quattro Pro and Word Perfect as two interesting examples.

The only way to avoid these traps is to encourage a software culture that actively resists bloat and complexity — an engineering tradition that puts a high value on simple solutions, looks for ways to break program systems up into small cooperating pieces, and reflexively fights attempts to gussy up programs with a lot of chrome (or, even worse, to design programs around the chrome). This tradition is associated with Unix and we need a conscious efforts to preserve it despite many Windows-emulators that now operated in Linux world.

Notes: Those pages are written by people for whom English is not a native language. Some amount of grammar and spelling errors should be expected. This is a Spartan WHYFF (We Help You For Free) site. It cannot replace the best teachers and the best books. The site contain some obsolete pages as it develops like a living tree... Some links on older pages are broken. Please try to use Google, Open directory, etc. to find a replacement link (see HOWTO search the WEB for details). We would appreciate if you can mail us a correct link. Search Amazon by keywords:     Open directory Research Index

Old News ;-)

grumpOps

Ahem! [Permalink]

Fri Jul 22 13:56:52 EDT 2005
Category [Internet Politics]

This was sent to me by a colleague. From "S4—The System Standards Stockholm Syndrome" by John G. Waclawsky, Ph.D.:

The “Stockholm Syndrome” describes the behavior of some hostages. The “System Standards Stockholm Syndrome” (S4) describes the behavior of system standards participants who, over time, become addicted to technology complexity and hostages of group thinking.

Read the whole thing over at BCR.

And while this particularly picks on the ITU types, it should hit close to home to a whole host of other "endeavors".

 

What is complexity

Complexity has turned out to be very difficult to define. The dozens of definitions that have been offered all fall short in one respect or another, classifying something as complex which we intuitively would see as simple, or denying an obviously complex phenomenon the label of complexity. Moreover, these definitions are either only applicable to a very restricted domain, such as computer algorithms or genomes, or so vague as to be almost meaningless. Edmonds (1996) gives a good review of the different definitions and their shortcomings, concluding that complexity necessarily depends on the language that is used to model the system. Still, I believe there is a common, "objective" core in the different concepts of complexity. Let us go back to the original Latin word complexus, which signifies "entwined", "twisted together". This may be interpreted in the following way: in order to have a complex you need two or more components, which are joined in such a way that it is difficult to separate them. Similarly, the Oxford Dictionary defines something as "complex" if it is "made of (usually several) closely connected parts". Here we find the basic duality between parts which are at the same time distinct and connected. Intuitively then, a system would be more complex if more parts could be distinguished, and if more connections between them existed. More parts to be represented means more extensive models, which require more time to be searched or computed. Since the components of a complex cannot be separated without destroying it, the method of analysis or decomposition into independent modules cannot be used to develop or simplify such models. This implies that complex entities will be difficult to model, that eventual models will be difficult to use for prediction or control, and that problems will be difficult to solve. This accounts for the connotation of difficult, which the word "complex" has received in later periods.

The aspects of distinction and connection determine two dimensions characterizing complexity. Distinction corresponds to variety, to heterogeneity, to the fact that different parts of the complex behave differently. Connection corresponds to constraint, to redundancy, to the fact that different parts are not independent, but that the knowledge of one part allows the determination of features of the other parts. Distinction leads in the limit to disorder, chaos or entropy, like in a gas, where the position of any gas molecule is completely independent of the position of the other molecules. Connection leads to order or negentropy, like in a perfect crystal, where the position of a molecule is completely determined by the positions of the neighbouring molecules to which it is bound. Complexity can only exist if both aspects are present: neither perfect disorder (which can be described statistically through the law of large numbers), nor perfect order (which can be described by traditional deterministic methods) are complex. It thus can be said to be situated in between order and disorder, or, using a recently fashionable expression, "on the edge of chaos".

The simplest way to model order is through the concept of symmetry, i.e. invariance of a pattern under a group of transformations. In symmetric patterns one part of the pattern is sufficient to reconstruct the whole. For example, in order to reconstruct a mirror-symmetric pattern, like the human face, you need to know one half and then simply add its mirror image. The larger the group of symmetry transformations, the smaller the part needed to reconstruct the whole, and the more redundant or "ordered" the pattern. For example, a crystal structure is typically invariant under a discrete group of translations and rotations. A small assembly of connected molecules will be a sufficient "seed", out of which the positions of all other molecules can be generated by applying the different transformations. Empty space is maximally symmetric or ordered: it is invariant under any possible transformation, and any part, however small, can be used to generate any other part.

It is interesting to note that maximal disorder too is characterized by symmetry, not of the actual positions of the components, but of the probabilities that a component will be found at a particular position. For example, a gas is statistically homogeneous: any position is as likely to contain a gas molecule as any other position. In actuality, the individual molecules will not be evenly spread. But if we look at averages, e.g. the centers of gravity of large assemblies of molecules, because of the law of large numbers the actual spread will again be symmetric or homogeneous. Similarly, a random process, like Brownian motion, can be defined by the fact that all possible transitions or movements are equally probable.

Complexity can then be characterized by lack of symmetry or "symmetry breaking", by the fact that no part or aspect of a complex entitity can provide sufficient information to actually or statistically predict the properties of the others parts. This again connects to the difficulty of modelling associated with complex systems.

(1996) notes that the definition of complexity as midpoint between order and disorder depends on the level of representation: what seems complex in one representation, may seem ordered or disordered in a representation at a different scale. For example, a pattern of cracks in dried mud may seem very complex. When we zoom out, and look at the mud plain as a whole, though, we may see just a flat, homogeneous surface. When we zoom in and look at the different clay particles forming the mud, we see a completely disordered array. The paradox can be elucidated by noting that scale is just another dimension characterizing space or time (Havel, 1995), and that invariance under geometrical transformations, like rotations or translations, can be similarly extended to scale transformations (homotheties).

Havel (1995) calls a system "scale-thin" if its distinguishable structure extends only over one or a few scales. For example, a perfect geometrical form, like a triangle or circle, is scale-thin: if we zoom out, the circle becomes a dot and disappears from view in the surrounding empty space; if we zoom in, the circle similarly disappears from view and only homogeneous space remains. A typical building seen from the outside has distinguishable structure on 2 or 3 scales: the building as a whole, the windows and doors, and perhaps the individual bricks. A fractal or self-similar shape, on the other hand, has infinite scale extension: however deeply we zoom in, we will always find the same recurrent structure. A fractal is invariant under a discrete group of scale transformations, and is as such orderly or symmetric on the scale dimension. The fractal is somewhat more complex than the triangle, in the same sense that a crystal is more complex than a single molecule: both consist of a multiplicity of parts or levels, but these parts are completely similar.

To find real complexity on the scale dimension, we may look at the human body: if we zoom in we encounter complex structures at least at the levels of complete organism, organs, tissues, cells, organelles, polymers, monomers, atoms, nucleons, and elementary particles. Though there may be superficial similarities between the levels, e.g. between organs and organelles, the relations and dependencies between the different levels are quite heterogeneous, characterized by both distinction and connection, and by symmetry breaking.

We may conclude that complexity increases when the variety (distinction), and dependency (connection) of parts or aspects increase, and this in several dimensions. These include at least the ordinary 3 dimensions of spatial, geometrical structure, the dimension of spatial scale, the dimension of time or dynamics, and the dimension of temporal or dynamical scale. In order to show that complexity has increased overall, it suffices to show, that - all other things being equal - variety and/or connection have increased in at least one dimension.

The process of increase of variety may be called differentiation, the process of increase in the number or strength of connections may be called integration. We will now show that evolution automatically produces differentiation and integration, and this at least along the dimensions of space, spatial scale, time and temporal scale. The complexity produced by differentiation and integration in the spatial dimension may be called "structural", in the temporal dimension "functional", in the spatial scale dimension "structural hierarchical", and in the temporal scale dimension "functional hierarchical".

It may still be objected that distinction and connection are in general not given, objective properties. Variety and constraint will depend upon what is distinguished by the observer, and in realistically complex systems determining what to distinguish is a far from trivial matter. What the observer does is picking up those distinctions which are somehow the most important, creating high-level classes of similar phenomena, and neglecting the differences which exist between the members of those classes (Heylighen, 1990). Depending on which distinctions the observer makes, he or she may see their variety and dependency (and thus the complexity of the model) to be larger or smaller, and this will also determine whether the complexity is seen to increase or decrease.

For example, when I noted that a building has distinguishable structure down to the level of bricks, I implicitly ignored the molecular, atomic and particle structure of those bricks, since it seems irrelevant to how the building is constructed or used. This is possible because the structure of the bricks is independent of the particular molecules out of which they are built: it does not really matter whether they are made out of concrete, clay, plaster or even plastic. On the other hand, in the example of the human body, the functioning of the cells critically depends on which molecular structures are present, and that is why it is much more difficult to ignore the molecular level when building a useful model of the body. In the first case, we might say that the brick is a "closed" structure: its inside components do not really influence its outside appearance or behavior (Heylighen, 1990). In the case of cells, though, there is no pronounced closure, and that makes it difficult to abstract away the inside parts.

Although there will always be a subjective element involved in the observer's choice of which aspects of a system are worth modelling, the reliability of models will critically depend on the degree of independence between the features included in the model and the ones that were not included. That degree of independence will be determined by the "objective" complexity of the system. Though we are in principle unable to build a complete model of a system, the introduction of the different dimensions discussed above helps us at least to get a better grasp of its intrinsic complexity, by reminding us to include at least distinctions on different scales and in different temporal and spatial domains.

References:

• Heylighen F. (1997): "The Growth of Structural and Functional Complexity during Evolution", in: F. Heylighen & D. Aerts (eds.) (1997): "The Evolution of Complexity" (Kluwer, Dordrecht). (in press)
• Edmonds B. (1996): " is Complexity?", in: F. Heylighen & D. Aerts (eds.), The Evolution of Complexity (Kluwer, Dordrecht).
• Havel I. (1995): "Scale Dimensions in Nature", International Journal of General Systems 23 (2), p. 303-332.
• Heylighen F. (1990): "Relational Closure: a mathematical concept for distinction-making and complexity analysis", in: Cybernetics and Systems '90 , R. Trappl (ed.), (World Science Publishers), p. 335-342.

See also:

• The Growth of Complexity
• Links on Complexity and Self-Organization
• Complex, Adaptive Systems

The Growth of Complexity

blind variation and selective retention tend to produce increases in both structural and functional complexity of evolving systems

---

At least since the days of Darwin, evolution has been associated with the increase of complexity: if we go back in time we see originally only simple systems (elementary particles, atoms, molecules, unicellular organisms) while more and more complex systems appear in later stages. However, from the point of view of classical evolutionary theory there is no a priori reason why more complicated systems would be preferred by natural selection. Evolution tends to increase fitness, but fitness can be achieved as well by very complex as by very simple systems. For example, according to some theories, viruses, the simplest of living systems, are degenerated forms of what were initially much more complex organisms. Since viruses live as parasites, using the host organisms as an environment that provides all the resources they need to reproduce themselves, maintaining a metabolism and reproductory systems of their own is just a waste of resources. Eventually, natural selection will eliminate all superfluous structures, and thus partially decrease complexity.

Complexity increase for individual (control) systems

The question of why complexity of individual systems appears to increase so strongly during evolution can be easily answered by combining the traditional cybernetic idea of the "Law of Requisite Variety" and a concept of coevolution, as used in the evolutionary "Red Queen Principle".

Ashby's Law of Requisite Variety states that in order to achieve complete control, the variety of actions a control system should be able to execute must be at least as great as the variety of environmental perturbations that need to be compensated. Evolutionary systems (organisms, societies, self-organizing processes, ...) obviously would be fitter if they would have greater control over their environments, because that would make it easier for them to survive and reproduce. Thus, evolution through natural selection would tend to increase control, and therefore internal variety. Since we may assume that the environment as a whole has always more variety than the system itself, the evolving system would never be able to achieve complete control, but it would at least be able to gather sufficient variety to more or less control its most direct neighbourhood. We might imagine a continuing process where the variety of an evolving system A slowly increases towards but never actually matches the infinite variety of the environment.

However, according to the complementary principles of selective variety and of requisite constraint, Ashby's law should be restricted in its scope: at a certain point further increases in variety diminish rather than increase the control that system A has over its environment. A will asymptotically reach a trade-off point, depending on the variety of perturbations in its environment, where requisite variety is in balance with requisite constraint. For viruses, the balance point will be characterised by a very low variety, for human beings by a very high one.

This analysis assumes that the environment is stable and a priori given. However, the environment of A itself consists of evolutionary systems (say B, C, D...), which are in general undergoing the same asymptotic increase of variety towards their trade-off points. Since B is in the environment of A, and A in the environment of B, the increase in variety in the one will create a higher need (trade-off point) in variety for the other, since it will now need to control a more complex environment. Thus, instead of an increase in complexity characterised by an asymptotic slowing down, we get a positive feedback process, where the increase in variety in one system creates a stronger need for variety increase in the other. The net result is that many evolutionary systems that are in direct interaction with each other will tend to grow more complex, and this with an ever increasing speed.

As an example, in our present society, individuals and organizations tend to gather more knowledge and more resources, increasing the range of actions they can take, since this will allow them to cope better with the possible problems appearing in their environment. However, if the people you cooperate or compete with (e.g. colleagues) become more knowledgeable and resourceful, you too will have to become more knowledgeable and resourceful in order to respond to the challenges they pose to you. The result is an ever faster race towards more knowledge and better tools, creating the "information explosion" we all know so well.

The present argument does not imply that all evolutionary systems will increase in complexity: those (like viruses, snails or mosses) that have reached a good trade-off point and are not confronted by an environment putting more complex demands on them will maintain their present level of complexity. But it suffices that some systems in the larger ecosystem are involved in the complexity race to see an overall increase of available complexity.

Complexity increase for global (eco)systems

The resoning above explains why individual systems will on average tend to increase in complexity. However, the argument can be extended to show how complexity of the environment as a whole increases. Let us consider a global system, consisting of a multitude of co-evolving subsystems. The typical example would be an ecosystem, where the subsystems are organisms belonging to different species.

Now, it is well-documented by ecologists and evolutionary biologists that ecosystems tend to become more complex: the number of different species increases, and the number of dependencies and other linkages between species increases. This has been observed as well over the geological history of the earth, as in specific cases such as island ecologies which initially contained very few species, but where more and more species arose by immigration or by differentiation of a single species specializing on different niches (like the famous Darwin's finches on the Galapagos islands).

As is well explained by E.O. Wilson in his "The Diversity of Life", not only do ecosystems contain typically lots of niches that will eventually be filled by new species, there is a self-reinforcing tendency to create new niches. Indeed, a hypothetical new species (let's call them "bovers") occupying a hitherto empty niche, by its mere presence creates a set of new niches. Different other species can now specialize in somehow using the resources produced by that new species, e.g. as parasites that suck the bover's blood or live in its intestines, as predators that catch and eat bovers, as plants that grow on the bovers excrements, as furrowers that use abandoned bover holes, etc. etc. Each of those new species again creates new niches, that can give rise to even further species, and so on, ad infinitum. These species all depend on each other: take the bovers away and dozens of other species may go extinct.

This principle is not limited to ecosystems or biological species: if in a global system (e.g. the inside of a star, the primordial soup containing different interacting chemicals, ...) a stable system of a new type appears through evolution (e.g. a new element in a star, or new chemical compound), this will in general create a new environment or selector. This means that different variations will either be adapted to the new system (and thus be selected) or not (and thus be eliminated). Elimination of unfit systems may decrease complexity, selection of fit systems is an opportunity for increasing complexity, since it makes it possible for systems to appear which were not able to survive before. For example, the appearance of a new species creates an opportunity for the appearance of species-specific parasites or predators, but it may also cause the extinction of less fit competitors or prey.

However, in general the power for elimination of other systems will be limited in space, since the new system cannot immediately occupy all possible places where other systems exist. E.g. the appearance of a particular molecule in a pool of "primordial soup" will not affect the survival of molecules in other pools. So, though some systems in the neighbourhood of the new system may be eliminated, in general not all systems of that kind will disappear. The power for facilitating the appearance of new systems will similarly be limited to a neighbourhood, but that does not change the fact that it increases the overall variety of systems existing in the global system. The net effect is the creation of a number of new local environments or neighbourhoods containing different types of systems, while other parts of the environment stay unchanged. The environment as a whole becomes more differentiated and, hence, increases its complexity.

Sun Inner Circle

What's preventing businesses from realizing better utilization rates is an outbreak of the 1:1:1 ratio — one application per operating environment per server. While this ratio is effective for meeting peak load targets, it's off the mark for achieving IT efficiency. The more things that need to be managed, the more time-consuming and expensive that infrastructure becomes. Clearly, this approach to managing the infrastructure doesn't scale effectively — a big problem when saving IT dollars is the primary goal.

The old paradigm of managing infrastructure resources is largely to blame for the current system bloat. Traditionally, organizations have invested in people to manage this legacy of 1:1:1. So as the business grew, people-management costs significantly increased — a practice that is prohibitively expensive.

Many IT managers also thought having dedicated server environments was a more reliable way to ensure performance and availability while mitigating security risks. But allowing each department to control its own resources has exacerbated the problem of doing more with more, rather than doing more with less. Finally, to meet workload requirements, IT managers often looked to peak workloads as the barometer for system needs. Yet basing server needs on peak usage levels is costly and inefficient, as normal loads typically require just a fraction of those resources and not all applications will peak at the same time.

An obvious way to combat these costly practices is through server consolidation. But simply getting more applications onto fewer servers is not enough. Effective server consolidation is contingent on maintaining the confidence of IT managers that applications will have the resources they need to meet performance levels. It's also important that applications housed on the same server can be isolated to avoid fault propagation.

Said another way, server consolidation is only valuable if IT managers have the same assurance that performance levels will be on par with levels achieved by the 1:1:1 ratio and the confidence that one application will not adversely impact the security or availability of other applications co-hosted on the same server. There is a way to make this a reality. By implementing the technique of virtualization into your data center utilization strategy, you can achieve all the benefits of the 1:1:1 setup while simultaneously reducing IT expenses.

Deadly Sins - common start-up errors Entrepreneur - Find Articles

Also be aware of the other side of the same coin: excessive debt and overhead. Debt can destroy your start-up, so stick to your business plan, and don't let appetite exceed budget or planned expenditures. "Too many entrepreneurs bring the infrastructure 'bloat' from their previous corporate careers to their start-up," says Pierce Johnson, founder of Chicago-based Johnson Technologies Inc. (which he sold last year to eSkye Solutions). "Most of the failed companies I know added too many employees too soon."

Basic rule: Stick to the business plan, and if a particular expenditure isn't budgeted there, forget about it.

'Smart Growth' Innovating to Meet the Needs of the Market without Feeding the Beast of Complexity - Knowledge@Wharton

Managing Complexity

Wilson points out that complexity can be an organizational drag, consuming resources, diluting focus and impacting profitability. In that way, it can be a drag on innovation efforts. But conversely, he notes, it is important to understand how the current innovation system helps or hinders the issue of complexity. "In many situations, the innovation system itself can be one of the drivers -- a poor innovation system can lead to clutter and complexity," he says.

Next, companies must get a grip on what causes that complexity. "Is it a lack of customer knowledge, or poor understanding of the economics of the situation?" asks Wilson. Additionally, he says, they need to get an accurate picture of the real effects of complexity.

There are corrective strategies for complexity, he notes. "One of them is to reduce complexity in your portfolio or in your processes. But reducing your portfolio is only one strategy, and it may not be the right strategy for your organization."

Another strategy, says Wilson, is to "make your complexity more approachable for the customer and make the choices digestible." Indeed, there exist ways to empower the customer to comfortably deal with the full range of a company's offerings.

Wharton marketing professor Barbara Kahn says discovering that golden mean of how much is not too much is the trick. "If it's too much, they won't deal with it; if it's too little, then they may be able to deal with it," she says of customers' buying patterns.

That's where customer expertise comes into play, according to Kahn. "One of the factors that makes [a higher number of offerings possible] is expertise," she says. "The more people become experts, the more they articulate their preferences -- and the more they have a consumption vocabulary and know what the relevant attributes are, the more variety they will be able to take." She also suggests "arranging [product] assortment in such a way that consumers just see what it is they want and they don't have to see all that they don't want. Websites are really good at that."

Kahn likens the process of empowering customers with how salad bars help patrons navigate a mind-boggling range of options. "If you thought of all the different kinds of salads that you could make, and you presented [customers] all the different options, people wouldn't be able to deal with that -- there would be too much variety," she says. "But if you do it the way [restaurants] do with salad bars, and divide salads up into attributes ... they can deal with that variety because they can deal with those different attributes."

... ... ...

Companies that take the quick route to de-proliferate their offerings in an attempt to reduce complexity might end up returning to the same situation two years later, according to Wilson. That could lead to another danger, he says, of "cutting too shallow or too often." He warns companies not to underestimate customers' memory of portfolio changes. "The last thing you want to do is reduce some of the complexity, and then two years later tell the customer, "We didn't do it properly the last time; we're doing it again."

[Nov 11, 2006] Knowledge@Wharton Newsletter Special Report October 25, 2006

Special Section 'Smart Growth': Innovating to Meet the Needs of the Market without Feeding the Beast of Complexity As companies struggle to innovate in today's competitive environment, they need to continually guard against adding to their "clutter" -- the creeping impact of complexity on efficiency and cost-competitiveness. In this three-part special report, experts from Wharton and George Group Consulting discuss how management can approach this problem by thinking "ambidextrously" -- that is, focusing on innovation and broad exploration while minimizing the impact of clutter on operational processes and costs. Also, in the accompanying podcast (with transcript), Mike McCallister, CEO of Humana, discusses balancing innovation and complexity in the health care industry. http://knowledge.wharton.upenn.edu/special_section.cfm?specialID=58

Part I: Innovation vs. Proliferation: Getting to the Heart of the Customer How can companies innovate without falling into the trap of needless proliferation in their products or services? The key, according to Wharton faculty and experts from George Group Consulting, is understanding unmet and unarticulated consumer needs while aligning innovation processes to those insights. http://knowledge.wharton.upenn.edu/article/1585.cfm#part1

Part III: Getting a Grip on the Costs of Complexity Determining the financial impacts of innovation-related complexity begins with taking a close look at existing operations to understand the actual cost incurred and value generated at each step in the process -- all the way from idea generation through product development, manufacturing, marketing and customer support, among other back-office functions. http://knowledge.wharton.upenn.edu/article/1585.cfm#part3

[Sep 19, 2006] EiffelWorld Column by Dr. Bertrand Meyer

May 2005 (EiffelWorld) The power of simplicity (May 2005)

The best defense against falling prey to technology fashion is to be skeptical of complex solutions. Is the complexity warranted? Sometimes it is, but often it's just a smokescreen to hide the existence of simple and effective answers. Take the basic idea of object technology: to use the power of software modeling techniques -- essentially, abstract data types -- to describe systems of just any kind.

The idea was there from the beginning, and Eiffel took it to its full development thanks to Design by Contract (and multiple inheritance, genericity, deferred classes, Uniform Access, More...

[Mar 11, 2005] The Fishbowl / Catching a Silver Bullet

A storm in a teacup was launched last week by an ONLamp.com article making wild claims about Ruby on Rails:

What would you think if I told you that you could develop a web application at least ten times faster with Rails than you could with a typical Java framework? You can—without making any sacrifices in the quality of your application! How is this possible?

I’m not going to be drawn into the Rails vs The World debate. Rails may be wonderful. It may make me significantly more efficient than I would be coding in Webwork. and Java. I’ve not tried it beyond throwing together a toy application, and I’m going to withhold judgement until I’ve done something serious with it. But I can categorically say that Rails is not going to make me ten times more efficient. When I encounter this sort of hyperbole, I always find myself returning to the words of Fred Brooks:

But, as we look to the horizon of a decade hence, we see no silver bullet. There is no single development, in either technology or in management technique, that by itself promises even one order-of-magnitude improvement in productivity, in reliability, in simplicity. — Fred Brooks, No Silver Bullet

The core of Brooks’ argument concerns complexity. Writing software is a complex business, and it essentially comes down to the combination of two types of complexity: Essential complexity is the complexity inherent in the problem being solved. Accidental complexity is the complexity that derives from the environment that the problem is being solved in.

Consider an impossibly perfect tool that reduces accidental complexity to zero. For this magical tool to give you a ten-fold increase in productivity, that would have to mean that you are spending 90% of your time fighting your current tools, and only 10% of your time solving the problem you are coding.

I’ve been in one or two truly pathological environments where I’ve felt like this (usually involving EJB 1.1), but you have to do a lot of concerted work applying layer upon layer of antipatterns to get the ratio that high.

This is precisely why demonstrations are to be taken with a grain of salt. Any task that can be performed in a demo must necessarily have an essential complexity close to zero: it’s a solved problem before the demo even begins. So if one vendor’s demonstration takes a tenth of the time as another vendor’s, all that time is accidental complexity.

And even the accidental complexity of a demo is usually of a totally different nature to that you’d encounter on a real project: the kind of complexity that accrues around a task that can be completed in the course of a lecture is vastly different to that which is encountered in a year-long multiple-developer project.

to tie pre-built web services together with the click of two buttons. Configuration is a much more significant overhead for a 45 minute “from scratch” or “here’s something I prepared earlier” demo than it is over the lifetime of a real project.

It’s like those debates about optimisation. “I just made this loop 10 times faster!” “Great, except we only call that method once a day.”

A few years ago, a marketroid visited my then employer to give a demo of a recently re-launched IDE. The demo was very slick, showing how you could throw together an EJB project or a SOAP interface in a few clicks of the mouse.

Then, we started asking about the product’s refactoring support, and the answer was “Oh, we don’t have that.” It was at this point the developers in the audience switched off. The IDE might have made it really easy to set up a web application skeleton or add new actions, but in any reasonably complex task you’re going to be spending most of your time dealing with code. Refactoring tools only become necessary when you are iteratively refining your code: something you do constantly as you move towards the solution to the essential challenges of the programming task, but totally irrelevant to a slick, pre-packaged demo.

This isn’t to say that improving processes, using more powerful languages and so on can’t give you a significant advantage. If something makes you even ten percent more efficient, that’s a huge advantage to have over your competition. But if anyone promises to make you ten times more efficient, they’re discrediting themselves before they start, which is a disservice to all involved.

[July 04, 2005] Q&A Internet Pioneer Looks Ahead - Computerworld Q&A: An Internet Pioneer Looks Ahead Leonard Kleinrock predicts 'really smart' handhelds, but warns of out-of-control complexity.

July 04, 2005 (Computerworld) You have warned that we are "hitting a wall of complexity." What do you mean? We once arrogantly thought that any man-made system could be completely understood, because we created it. But we have reached the point where we can't predict how the systems we design will perform, and it's inhibiting our ability to do some really interesting system designs. We are allowing distributed control and intelligent agents to govern the way these systems behave. But that has its own dangers; there are cascading failures and dependencies we don't understand in these automatic protective mechanisms.

Will we see catastrophic failures of complex systems, like the Internet or power grid? Yes. The better you design a system, the more likely it is to fail catastrophically. It's designed to perform very well up to some limit, and if you can't tell how close it is to this limit, the collapse will occur suddenly and surprisingly. On the other hand, if a system slowly erodes, you can tell when it's weakening; typically, a well-designed system doesn't expose that.

So, how can complex systems be made more safe and reliable? Put the protective control functions in one portion of the design, one portion of the code, so you can see it. People, in an ad hoc fashion, add a little control here, a little protocol there, and they can't see the big picture of how these things interact. When you are willy-nilly patching new controls on top of old ones, that's one way you get unpredictable behavior.

[Jan 4, 2005] Discussion of Open Source Software, A Moderated by Bill Thomas. contains interesting thought of feather bloat problem in open source.

... ... ...

Why Do Research on Open Source Software?

Bill Thomas: Why do you think that open source software warrants research at this time?

Chuck Weinstock: One reason is that it appears that the community is treating open source software as the next silver bullet. We all know that silver bullets very seldom find their target, and the community moves on to the next silver bullet.

Pat Place: I would add that there is a substantial amount of software available these days that is open source. If you are interested in building systems out of existing components—be they open source or any other form of source—you need to understand at least the risks as well as the benefits of doing so. If we can say anything that helps, then I think that's a good thing.

Scott Hissam: The phenomenon that is happening with the Linux environment is getting everybody's attention to open source software—more attention than has ever been paid to it before, at least in the media. People are enamored and believe that Linux is a successful, stable development environment, and that somehow every piece of open source software that they get is going to be just as stable and just as reliable as the Linux platform—if you believe that it is as stable and reliable as it is touted to be in the press.

Jed Pickel: Open source has been around for a long time—probably more than 20 years. I think one of the reasons why it's getting so much attention now is because commercial interests are developing it. That's why we’re seeing the media interest.

Is There a Community of Developers?

Dan Plakosh: You can release your source code, but I’m not sure people really know what to do with it. I released open source software two years ago and I’ve had very few people dive into developing it.

Place: It gets even worse when you get something that is hundreds of thousands of lines or a million lines. The historical aspect is interesting because you can look back at the history of programs that were open source, or close to open source, with lots of people helping and providing fixes. You can start to see what was successful about them and what is different about what is becoming the current open source movement, which I honestly believe is going to lead to disaster. I can provide anecdotal evidence of examples where you've got something like a tcsh [an expanded version of the original C shell for the UNIX operating system] which is not that complicated a program, but has features and peculiarities that are so weird that you'd never even want them. And yet somebody has said, "Oh, I'll just go and stick this thing into the system." For example, in tcsh, if you have time displayed as part of your prompt and it happens to hit the hour, it'll go "ding" instead of printing the time. I mean, this is insanity: feature upon feature upon feature that leads to code that's got more junk in it than you can possibly be interested in. It ends up becoming ultimately unmaintainable code.

Hissam: But I would say that the tcsh example that you gave is an unbounded development activity that nobody really paid attention to. Nobody really cared about it and that's why it got unwieldy. Not every piece of open source software is developed in that way.

Place: That's exactly true. I think that's the key to the difference between those things that have been and will be successful and those things that will not be successful. Somebody or some very small group of people have a very clear idea as to what that system is going to be, what it's going to do, and how it's going to be architected. And they keep it that way.

Weinstock: Some people refer to those people as the "arbiters of good taste."

Place: That's the phrase that was used primarily about the original UNIX developers. They were arbiters of good taste. With all of the stuff that people from all the universities shipped to them back in the mid-1970s and early ’80s, they decided what went into the source and what was not in the source. For the longest time with Linux, Linus Torvalds [the Finnish graduate student who created the original Linux operating system] was the person who did that. He had a vision as to what it was going to be. That seems to be drifting out. Linux is perhaps losing some of that arbiter-of-good-taste quality.

Pickel: On your tcsh example: there are plenty of examples of closed source software having very similar things. For example, one commercial product is such that if you hit certain key sequences, you can end up with a flight simulator—which is a little bit different from tcsh beeping at the end of a line. The difference, though, is that should the community come across that item in tcsh, and feel like it needs to be removed, and there are enough people who agree with that, then it would be removed from tcsh. You can easily go and change that if it bothered you enough.

Place: Absolutely. I've done that because I wanted tcsh to be as small as possible and I’ve used it as small as possible.

Hissam: So you removed a whole bunch of things out of tcsh that you didn't like. Right? Now let’s say the next version of tcsh comes out and you want to adopt it.

Place: I have developed the version of the shell that has the capabilities I need. If anything does come out in tcsh that I'm interested in, I might take that as a patch file and patch my source with those changes. But I'm not taking all their stuff again.

Weinstock: You now own the problem.

Place: Yes, absolutely. I willingly accept that. Of course, the advantage is that it was open source. I could choose to take on the risk and build something that was what I wanted.

Who Has Accountability for Open Source Software?

Thomas: It seems that no one has any accountability with open source software. It’s strictly "buyer beware."

Pickel: I would disagree with that. Let's go back to the tcsh example again, because I think that's a good one in that a person maintains it and is accountable for listening to the users. Pat didn't speak up in this case. He decided to split off his own version and now he's accountable for that.

Someone wants the X widget or the Y widget, so they just go and put that fix in, and you get this loss of sanity.

Place: If you look at the actual source code, you'll see all of these different names of people who've added this and added that. The risk I see with open source is that all of these features are getting thrown into a basically good system. Someone wants the X widget or the Y widget, so they just go and put that fix in and you get this loss of a sense of stability of the project—this loss of sanity. tcsh going "ding" is kind of stupid. It goes to my notion of good taste; it's below the cut line.

Pickel: What you're describing is an example of open source working in the most optimal sense in that there are people who have different goals from a project than you do. You decide to split off your own version; that's open source working right there.

Hissam: It depends on what your own goals are. If your own goals are to keep up with the latest and greatest, then him vectoring off his own version—he's stuck.

Place: That's disaster if you want to keep up with the latest.

Plakosh: I don't think people develop open source software with the intent that people will take it and go off and build their own products from it. I think it's more so that people will contribute to mature whatever piece of software that they're doing.

Place: The freedom for anyone to make a change leads to the fact that the product will never mature because it will always be in a state of flux.

Plakosh: There is not the freedom for anybody to make a change. I really think you're looking at isolated cases. For example, take Linux. The majority of people who use Linux never look at the source. In the majority of most open source products, I would bet that people do not look at the source. They don't care. They don't recompile it. They don't want to have anything to do with it. It's only the people who are working toward the development of Linux who are looking at the source, or occasionally someone who has the in-depth knowledge finds a bug and looks at the source. They may fix it but they may also submit it to one of the Linux working groups to have it corrected.

What does it mean to be accountable?

Hissam: Let’s go back to the earlier question about accountability. We disagreed about whether anyone is accountable. What does it mean to be accountable? It means that there's liability on the part of somebody.

Place: I wouldn't even go as far as that. Dan has released source. He's put his name up saying, "I think this is a good piece of source." In an open source project, other than a couple of special cases, there's a substantial body of existing code that gets released and then people can work on it, rather than working on stuff from scratch. I see a potential split there. Take Linux. Who is accountable for Linux these days? Does Linus put his name on it saying, "I think this is all good source code"? I don't think so anymore.

Hissam: No. The worst thing that can happen to the people who are "accountable" for Linux is that the world turns their back on it. But concerning accountability: I think the answer is that no one is accountable outright and I think it is buyer beware.

Weinstock: So that's why people go to places like [Linux vendor] Red Hat instead of just downloading it off the Web.

Hissam: Because they want to hand money to somebody. They want to be able to say, "Give me this and give me that."

Weinstock: Red Hat also sells support. You can go just buy a Red Hat CD and you get nothing with it other than the CD. But you can also go to them and get support for Linux.

Pickel: That's how Linux has made it into the corporate world: doing support.

Plakosh: I've dealt with support before and support is not typically all that it’s cracked up to be. Support is usually geared toward people who have problems in reading the documentation or who don't understand things. Linux got into the commercial domain mainly due to the attractiveness of it being free and being somewhat reliable.

What Are the Benefits and Drawbacks of Developing Open Source?

Thomas: Let's backtrack a little bit here. What would you say are the benefits and drawbacks of developing software in an open source environment, from the standpoint of the developer?

Weinstock: There are different ways of looking at that. Why would I want to participate in the development or why would I want to put myself more out there for free...

Place: I'll tell you at least one person's motivation for the latter. For the last two years, he's been unable to further his software at all, so since it was previously freely available, he said, "Okay, let's make this an open source project in the official open source project way. We'll get people who have ideas and have some suggestions for developing this further, and/or who have bug fixes to be able to maintain this thing."

Hissam: So, would you say that the rate of change on this project has increased or decreased, and have those changes been dramatic?

Place: Well, it's certainly increased. There's also one place where you can get an official source version that has the bug fixes in it, which you couldn't do previously.

Thomas: Would you say that putting out a program with open source code is a way of testing the market for it?

Pickel: Exactly. That's another good point that I wanted to make. One of the interesting things about open source is that you build on other people's software. When you release something, you never quite know how other people are going to make use of it. You learn quickly that way because they give you immediate feedback and contribute changes. It's a great way to figure out market demand.

Hissam: That would be a benefit. But if that evolution is unchecked, you're going to get the tcsh phenomenon. It's almost like a cancer: cancerous features.

Pickel: You choose the branch of the code that most suits your goals at a given time.

Weinstock: But that presents the consumer with a real problem, right? Which branch? What happens to the uneducated consumer who doesn’t have a basis for picking a branch?

Pickel: They go to places like Red Hat.

Place: If you want a version of BSD [a popular version of UNIX; BSD stands for Berkeley Software Distribution], which one do you pick right now? There are three versions of BSD that are all based upon 4.4, BSD Light, which was the last release. So which one do you choose?

Weinstock: Getting back to what you said about consumers going to Red Hat because they don't know how to make that choice: That's fine for an open source project where there is a Red Hat, but my guess is that most of them don't have a Red Hat. I mean, how do I know which Emacs to choose, for instance?

Plakosh: The only reason you have companies like Red Hat out there is because the distribution package for Linux is so large and so complicated—or at least it's viewed that way by the consumer. For a small piece of software, you're not going to have these distributors.

Pickel: Going back to your point about what to do if there is no Red Hat: I think these companies are out there for the purpose of infiltrating the corporate world—getting this kind of software into the corporate world. The techies and the geeks aren't necessarily interested in a Red Hat, though they may use it because they don't necessarily care to package all the software. But there are projects that don't have corporate backing behind them, or a very organized way of going about things. They're just not going to make it to as large an audience. They won't make it into the corporate world quite as easily.

Hissam: Every techie and geek on the planet right now, and every open source activity, started with dreams of IPOs [initial public offerings of stock]. They want to be the next millionaire. They want to start the next company.

Pickel: If you look at the past year or so, that might be one of the motivations behind open source software: people think they're going to make a killing off it. If you look over the past 20 years, there haven’t necessarily been financial reasons. One of the ways you get paid for leading a successful open source project is by getting your name out there, by getting well known, by becoming the guy who started that project.

Is Open Source Right for the Department of Defense?

Weinstock: Let’s talk about open source as applied to our Department of Defense client. What are the advantages of developing something using the open source model? When applied to the DoD, the notoriety factor is probably not important to them.

Place: There's another question that I should like to raise with respect to DoD customers. What systems do you envision the DoD would like to build with open source? Is it the weapons systems? Is it the payroll systems? How many people out there are interested in the payroll system?

Weinstock: It would seem to me that we're probably talking about subsystems first of all. Pieces of systems.

Place: So we’re talking back at the level of things like the operating system (OS), the database, the underlying components—the bits that we take for granted. That's one of the issues, when we talk about DoD customers. We need to understand that they're not going to build systems with this stuff.

Weinstock: But they will build systems that contain parts.

Place: Then the question is, which parts? Clearly it's going to be exactly those things—the OS parts, the database parts, the GUI [graphical user interface] parts.

What Are the Criteria for Success in Open Source Development?

Hissam: We can go back to the premise that these large organizations, be they DoD or not, think that they can get something done with access to a large, talented pool of engineers. There's some belief that they can get access to a lot of peer reviews, beta testers, people out there to look at their software and make it better and get it done quicker. That seems to be the running belief. If that's a model of success in Linux, then it must be true for every piece of open source software. But we should debunk that. Past performance should not be used as an indicator of future performance. That's one of the reasons that the Software Engineering Institute has to start looking at the processes that are used in open source development. What are the criteria that have to be there in order for it to be successful?

Past performance should not be used as an indicator of future performance.

Place: There are instances of projects that have been very successful. I would claim that Linux certainly has been one of them. It has achieved a level of reliability. The BSDs are reasonably successful, and some other open source things are reasonably successful. One of the common themes I've seen through either open or freely available source projects over the last 20 years is that there has been a substantial body of software—i.e., the system is basically there—before its release. People are bug fixing rather than developing new features, so that you've got a system with a structure and a design, and other people are now fixing the things that "he forgot" or that "he got wrong."

Weinstock: That suggests that it should start off with a user base or some sort of base of people who care about it.

Place: You certainly need people to care about it, and the people who care about this typically are the users.

The user base of Linux was people who developed software, not users of software, per se.

Plakosh: But if you look at how Linux kicked off, it kicked off by being more or less the toy of software engineers to build upon. It came with a lot of people's research projects. There were a lot of people looking at this functionality and that functionality. The user base of Linux was people who developed software, not users of software, per se.

Place: That’s a good point. The other thing, in thinking about what has been successful, is that the initial release was something that was a well-designed system.

What Are the Advantages to the User?

Thomas: Let's talk a little bit about open source from the user's perspective. What are the advantages to using open source software?

Hissam: Let me start off by cutting to the chase. It's a two-edged sword. The advantages are that users can get the latest and greatest and the fastest fixes. The disadvantages are that they have to get the latest and greatest and the fastest fixes. They might spend 75% of their time using a product and 25% of their time upgrading the product.

Just because a product is out in open source doesn't mean that I, as a user, have to track it.

Plakosh: That's not necessarily true. Just because a product is out in open source doesn't mean that I, as a user, have to track it. There are a lot of internal releases that I don't need to worry about or that I may not want to worry about. That perspective is somewhat from the mentality of the world that we live in, developing software.

What Are the Security Implications of Using Open Source Software?

Pickel: From a security perspective, you could look at it from a couple of standpoints. You really have to pay close attention to the software because if the community becomes aware of a vulnerability, then they're going to exploit it. So you need to beat them.

When you were talking about the double-edged sword, I realized that it also applies to the perspective of the developers in that there's an advantage to having people working on your software, but you also have to be ready to deal with them. If you have a lot of demand, and a lot of people developing your software, it's going to be tough to deal with them.

If the community becomes aware of a vulnerability, then they're going to exploit it.

Place: You raised the issue of security. What trust do you place in open source software, given that it's changing so rapidly? How much analysis can you do on the 5,000 fixes that came in last week?

Weinstock: Do you use Linux in a secure environment?

Pickel: Absolutely. Actually, I run Linux on all my machines. I'm not going to look at every single line of code. I'm not going to look at every update. But the thing is that there are people out there who are.

Weinstock: You hope. Do you believe that every nook and cranny of Linux has been looked at with that in mind?

Pickel: Not necessarily. But I believe that a lot more nooks and crannies have been looked at than are looked at in closed source environments.

Plakosh: That's an interesting statement because I would tend to bet that there is a difference between theory and practice. In theory, you would think that you're releasing source and you would have people combing over the code looking for security holes and trying to fix them. In practice, I don't think that's necessarily the case. In theory, it sounds great: the more people have it, the more people are looking at the source code and the more people are going to try to find bugs or security holes so that they can try to fix them. That sounds great. In practice, I think the only people who are trying to do that are maybe people who are trying to break into a system.

Pickel: Exactly. And they're part of the community. If they identify a hole and start exploiting it, people will notice that.

Hissam: You're saying that even the bad guys, in a sense...

Pickel: ...they're part of your development.

Place: But you only find out after the fact.

Pickel: That's still a better environment than closed source.

Plakosh: Actually, that's not necessarily much better because some of these holes that you can find in open source code you never would have found if it was closed source. They never would have existed.

Hissam: If I were a hacker, I could get the latest distribution from Red Hat, go to my garage, close the doors, get a lot of Twinkies and Coke, and start mulling over it until I can find a hack. Then I can just turn on my modem and go attack somebody who's using Linux.

Pickel: Certainly, there's some potential of administrators not noticing. This is an issue that we deal with every day in the CERT® Coordination Center. It's quite common that somebody will break into a site and the site administrators don't know how it happened. But when you deal with the sophisticated administrator, you can usually track down what program was the source, the problem. Then, maybe there's a new vulnerability in that. So, a lot of times, we look through the vulnerabilities, get in contact with vendors, and have the problem fixed. So there, in that situation, a few people were compromised. But the community as a whole is now operating on more secure software.

Place: Then there is the difficulty of getting customers to upgrade with the patch. Don't underestimate the number of people who are behind the curve.

Thomas: Is it any better in a closed source situation?

Hissam: Closed or open source, when a vulnerability is discovered, people have to react. I think the other thing that is interesting is that a hacker can become very intimate with some of the underlying protocols that are used. There may be that somewhere in the code it says, "You'd better check for null value here and the packet header for this because if you don't you could lock up the machine." The hacker says, "Wow! I hadn't thought of that before. If I tried this against Windows, I wonder what it would do?" Just by reading the source code, they could learn some very sophisticated, obscure attack.

What Comparisons Can Be Made Between Open Source and COTS?

Weinstock: There's also a big push to use COTS [commercial off-the-shelf] software, and widely using COTS raises all sorts of problems. At least some of the same arguments that apply to COTS apply to open source software.

Place: You might get an instability argument more so with open source than with COTS.

Plakosh: You'll get quality arguments too.

Weinstock: But it's the argument that "you own the solution if you try to modify it in any way."

Hissam: Let’s say a contractor is using open source in a government program and they're having some success. Then they run into a roadblock. They decide, "All I have to do is change this one line of code and we'll save the government millions of dollars." And they do it. Now let’s say the open source community doesn't want to adopt that change because it's very specific to whatever the government is doing. Now the government—by virtue of a contractor—is in the business of maintaining and competing with that open source.

Plakosh: That may or may not be true. I think one of the best advantages to me as a software developer is to take somebody else's open source and save development time, if it does do that for me. And I don’t intend to track it in the future.

Weinstock: Yes, but suppose there's a security compromise that's found in some future version and the four-star says, "That rolls back to the version you modified seven years ago—or six months ago." Now you've got to find someone who's even smart enough to put the changes in.

Plakosh: No you don’t. You've taken over maintenance of that. That’s fine, and you move on.

Weinstock: And the technician who worked on it seven years ago is still there, on the payroll, and there ready to help?

Plakosh: No. But we’re just talking about another method of software reuse. You're equating it to a product and having to track versions, rather than someone looking at open source software and saying, "Man, I could use these features. Let me rip them out and use them." You’ve taken the code from an open source product and reused it elsewhere.

Weinstock: But you’ve lost a supposed benefit of open source, which is that vast community of developers.

Plakosh: But if you weren’t interested in that, it doesn’t matter. You’ve gained. You didn't have to write that. You didn't lose anything. That's the benefit to a lot of people who are using open source. They don't have to reinvent the wheel. Somebody's already invented it. Yeah, I have to maintain it. Yeah, I'd better take a look at what I'm getting. Yeah, I'd better check the quality of it.

Weinstock: I'm not disagreeing with you. That's certainly a valid, good thing about open source. But if the whole world had that view of open source, there wouldn't be open source. All I'm saying is that it's sort of outside the spirit of open source as I see it.

Plakosh: What's the spirit of open source? Open source has two motives. One is that you want other people to work on your code; you want resources. So you want to obtain free resources.

Weinstock: Right. And you taking the code and going you own way and not feeding back to the community does not accomplish that.

Plakosh: But that's one motive. It’s for your own—how can I put it—personal glory, corporate gain, whatever. The second advantage is for other people just to advance technology and to advance people's growth. People give things out so that some other people can learn how to do something. For other people, it fosters new ideas. That's why I give out source code. I don't do it for my personal gain. I do it so that other people can look at it and use it for whatever they want to use it for.

Weinstock: That's from a developer's perspective.

Plakosh: I think it's great if someone reuses and finds benefit from something that I wrote, and if it saves them some time. Just like if I'm going to write a piece of software, I go out looking. I'm not going to try writing everything from scratch when I know that there is software that I can lift.

Pickel: The real benefit of open source, in my opinion, is that you build on things that are already available. You're furthering technology, and then people build on what you've done.

Place: You get the function you want as well. That's the other thing. If you're buying from Chuck's House of Software, you get what Chuck wants to sell you, not what you want.

Pickel: But you do have to be a developer to get that. I've been a user/developer of these things for a long time. If you're not a developer, you don't get it. I suspect that one of the results of this is that there will be more people who are developers out there. It's going to convince more people to look at the source code.

[Dec 27, 2004] Forth An underview

Forth is not just a language, its more of a philosophy for solving problems. This can be summarised with the acronym K.I.S.S. (Keep It Simple and Stupid). To quote from Jerry Boutelle (owner of Nautilus Systems in Santa Cruz, California) when asked "How does using Forth affect your thinking?" replied:

Forth has changed my thinking in many ways. Since learning Forth I've coded in other languages, including assembler, Basic and Fortran. I've found that I use the same kind of decomposition we do in Forth, in the sense of creating words and grouping them together. For example, in handling strings I would define subroutines analogous to Forth's CMOVE, -TRAILING, FILL, etc. More fundamentally, Forth has reaffirmed my faith in simplicity. Most people go out and attack problems with complicated tools. But simpler tools are available and more useful. I try to simplify all the aspects of my life. There's a quote I like from Tao Te Ching by the Chinese philosopher Lao Tzu: "To attain knowledge, add things every day; to obtain wisdom, remove things every day".

[Oct 20, 2004] Charley Reese Simplify

If we as a species are going to survive, we are going to have to learn to live simpler lives. By that I mean consume less stuff. The world's poor are already living simpler lives, and not by their own choice, so it's up to us in the industrialized countries to set the example.

OK, I know this sounds preachy and far-fetched, not to mention being highly unlikely to influence anybody. Nevertheless, sooner by choice or later by necessity, we will have to recognize that we are, if we continue the present trend and lifestyle, going to consume our own planet. Our ancestors will look mighty funny one day clinging to the solar system's only orbiting trash dump while trying to choose between garbage and cannibalism as a source of food.

Consult any almanac and look at the exorbitant rate at which we are pumping oil, mining coal and other minerals, cutting forests, catching fish and dousing the land with ever-increasing amounts of fertilizers, pesticides and herbicides. There's no question we are just now beginning to run short of a lot of natural resources. The price of oil is just one example of what's in store for us unless we curb our appetites.

Ravenous consumption was rather all right when the world population was only a billion, and few of them wealthy enough to afford much stuff. The Industrial Revolution changed all that. Utilizing fossil-fuel energy, it did raise the standard of living, and people began breeding ever more prolifically. Today, there are 6 billion people, and practically every one of them aspires to consume at the Donald Trump level. Cheap electronics make sure that nobody is ignorant of how the fat cats live. Even in the Amazon jungle, they watch "Baywatch.''

Europe, Russia, the United States and Japan have long been consuming at a rapid rate, and now two more giants are coming on line, so to speak, as India and China develop their massive economies, which is to say their appetites for energy and commodities. Then there are the so-called Asian tigers — Malaysia, the Philippines, Indonesia and Korea — all determined to raise their standard of living to the level of the West.

Well, we'd get nowhere asking anyone to remain poor as a conservation measure. What the world needs is a new lifestyle of elegant simplicity, so that people will learn to aspire to a few well-made items that can be used and passed on instead of junk, which is discarded as soon as it begins to wear or break down.

I include myself in criticism of overconsumption. I fancy myself on the low end of consumption. I care nothing for jewelry, clothes, fancy cars or furniture. The latter two things I tend to keep until they fall apart. But I have a weakness for books. There are books all over my little condo — five bookshelves, one covering a whole wall to the ceiling, and more books stacked on coffee tables, end tables and the floor.

On one little shelf between my dining/living room and the kitchen, I can see six sandstone coasters, a plastic timer, a bottle of glass cleaner, two candy dishes, a plastic globe, a rack for hanging bananas, a flashlight, four candles, a plaster-of-Paris Nefertiti, a bottle of Tabasco, a plastic watering jug, 11 cookbooks, a pipe rack with pipes and tobacco, a plate with a portrait of Robert E. Lee, and a kerosene lamp. That's one stinking little shelf. What do I really need? Maybe the flashlight and the Tabasco sauce. I haven't smoked a pipe in years and never cook anything more elaborate than fried eggs and baloney sandwiches.

Let's face it: Most of us, even us lower-middle-class types, lack consumption discipline. We get led astray by the singing sirens — new, more, bigger and upgraded. We need to seriously cut back, lest our grandchildren inherit a used-up, worn-out planet. And not just us — the whole world must reduce consumption, though of course about a fifth of the people still need basic food and housing.

Let us all try to simplify by decluttering and then avoid recluttering. Good luck to us all. We'll need it.

The Old Joel on Software Forum - Software Bloat and Moore's Law

On the topic of software bloat, it would help to distinguish between size bloat and processor cycle bloat.  With regards to size, the more features you pile on, the bigger your code footprint will become.  Just as mentioned by Joel, the cheaper disk space becomes, size bloat becomes more and more negligible.  Processor cycle bloat is typically brought about by software layering.  To get something done, you end up calling layers and layers of software.... most of which add little to what you need to do.  I come from a c/c++ background so yes, I did knock VB, Perl, and other scripting languages around for a while with regards to their performance aspect.  However, I am finally recognizing that all the above languages have become so pervasive in the programming world.

In the miniscule world where microseconds matter, size and speed matter.  This used to be the perception and programming accumen used to be how tight you can keep your code.  You tell me what the new perception is.

[Sep 10, 2004] Where is IT Going In The Next 5 Years - Code Bloat

I'm going to start with a tip before I rant.  The Acrobat Reader speedup tool available here ( http://www.tnk-bootblock.co.uk/ )really works. It disables loading a unnecessary plug-ins at startup.

<rant>
Why is this tool even necessary?  This proves that code bloat exists everywhere, not just at MS.  Why in the world would you need to load 75 plug-ins by default at startup - in a simple reader?  This is just pure lazy progamming. 

I understand the pressures of time-to-market quick delivery.  I also understand that there is a lot of overhead involved when you are trying to develop re-usable OO code versus hand-crafting an application.  This type of thing is inexcusable however.  IMHO Adobe is worse than Microsoft lately in producing applications that give you time for a beer break while you're waiting for them to load.  AutoCad and other are not necessarily speed demons either.

I've read estimates that the final Longhorn, running full Aero, Indigo and WinFS will need a base machine with 1GB of RAM to run decently.

Linux may be cleaner and more stable , but if you go look at recommended systems, the requirements aren't that much different than Windows.

There's got to be a better way.
</rant>

Oops, I have to start a program up - might as well go have a beer....

Tarwn (Programmer) Sep 10, 2004 Uh oh, you brought up Linux vs Windows :) I'd argue the point on requirements but I am sure someone else will bring it up. I agree on insane bloat though, I'm noticing it everywhere. Load the new ATI hardware drivers and the .Net-based control panel will eat up some more windows loading time as well as huge chunks of memory...it's a settings util, why would a settings util be forced to load on startup? Obviously I need the drivers, but please, I shouldn't be forced to hack my registray just to make the settings console stop eating my memory... Firefox: I love the browser, I hate the fact that is is sitting on 67Mb of RAM right now... STEAM: (for half-life players) Choke, cough cough...the release version wasn't even adequate to call a beta version but that aside, we have hidden loadup when the system starts and 23-24Mb while running in the taskbar (dunno about normal background, I killed that reg entry :P) MS Word...Outlook still takes foever to start up, word takes a litle while, so what is MS Word doing with 22Mb of my memory? MS Outlook: Apparently not everything is covered by MS Words 22Mb, so here's another 13Mb for my mail program... WinVNC: Now this is what I'm talking about...4Mb. VNC is running as a server and using less then 1/5th of the memory that MS Word is....and I haven't opened anything since rebooting except Outlook, Editplus (1mb), and Firefox... So yeah, I agree that bloat is everywhere...but machines keep getting faster and RAM keeps getting cheaper so companies feel justified in cutting corners to bang out bloated software, the bloating probably isn't even an effect of time limits anymore, probably bloated by design (or lack thereof)...

About Bradbury Software

Bradbury Software, LLC was founded in 1999 by Nick Bradbury. Nick is the creator of the HTML editor HomeSite, which was acquired by Allaire in 1996 and is now owned by Macromedia. After leaving Allaire in 1998, Nick continued his love of acronyms by creating the CSS/xHTML editor TopStyle and the RSS/Atom reader FeedDemon.

Our mission is to provide fast, efficient, reliable software that exceeds peoples' expectations.

...yadda, yadda, yadda.

Okay, almost every software company has a similar mission statement, and in most cases it's a hollow sentiment that flies in the face of how it really does business. How many times have you bought software that was so buggy that you thought you must've accidentally installed a pre-beta version? And how many hours have you wasted downloading bloated software that made your high-powered system run like a narcoleptic slug on downers?

If you're like us, you're tired of this. So, rather than bore you with our mission, we'll tell you...

Our Promises to our Customers

1. Our products will go through extensive beta testing involving dozens of external testers. We will not release our software until these testers tell us it's ready.
2. If serious bugs are reported in the current version of our software, we will not make you wait (and pay!) for the next version to see them fixed. Fixing bugs in the current version will always take priority over the release of the next version.
3. We will build our software based on the needs of our customers. We maintain an online forum where we take feature requests, and when it comes time to work on the next version, we enable you to vote for which of these features you want to see.
4. Our software will always be system-friendly. TopStyle and FeedDemon are perfect examples of this. They install no shared DLLs, ActiveX controls or other system files. Since they're self-contained, you can install them without fear of them interfering with your system or with other applications.
5. We will keep our software fast and compact. Too many products are extremely slow and bloated beyond reason, filling your hard drive and wasting system resources. TopStyle and FeedDemon load very quickly so you can start using them immediately, and they're also surprisingly compact.

[July 16, 2002] Light methodologies value simplicity over complexity - Builder UK Tom Mochal, Builder.com

Light methodologies rely on quickly iterative design cycles to fulfill their promise of rapid development and smart solutions. But how quick can you be if you’re using plodding design tools or wading through reams of cumbersome, overwrought code?

A key aspect of light methodologies is their need for simplicity. All light methodologies value simplicity over complexity whenever possible. Use that one tool that satisfies 80 percent of your needs instead of adopting three tools to cover 100 percent of your wish list. If you can use 50 lines of simple code to substitute for 25 lines of elegant code that only you can understand, go with the 50 lines. When you design an application, do it as cleanly and simply as possible.

Simple design and coding

The overall design of your application needs to be simple and flexible. Avoid design decisions that are perfect for your first iterations but then don’t allow you to add features and functions in later iterations. This can happen if you tie program components too closely together, instead of maintaining a level of independence.

You also don’t want to overengineer a solution. If you’re building 100 reports for your application, you probably need some sort of user library structure to keep track of the reports and what they do. But if your solution requires just 10 reports, drop the library.

Your code needs to be simple to review and to understand by others who follow you. If you look at the total life cycle of an application, only about 20 percent of cost is spent during the development phase. The remaining 80 percent is spent in the support and maintenance phases. If you build a no-frills application, the code might run in production for 10 years or more. Simple and straightforward code written up front allows easier learning curves, error fixes, and enhancements over the entire life cycle.

Simple program documentation
Writing documentation is the bane of many programmers. First of all, many programmers are great with computer languages but aren’t very strong with English. Secondly, programmers tend to write their comments and notes for themselves, not someone else who will need to understand them.

Light methodologies tend to advocate essential documentation, but no more. This minimalist approach recognises the inherent limitations of documentation.

Take program documentation, for instance. If you‘re trying to track down problems in code, you’re not going to be able to find the problem in a programmer’s manual. The only place you’ll find the bug is in the code itself. Even if the customer asks you to investigate how a feature works, you typically can’t rely on an external programmer’s manual. The only way to know for sure is to check the code. So having a stand-alone programmer’s manual that describes the code probably doesn’t make sense.

On the other hand, the code itself should have plenty of comments. These comments shouldn’t reflect the obvious but instead should point out creative techniques or describe major sections of code that enable certain features and functions.

Programmers might also be asked to assist with users’ manuals and help features. Again, you should convey a basic understanding to others that are not involved in the project. In many cases, large users' manuals are created but only certain parts are ever referred to again. Work with your customer to anticipate the basic documentation needed and build at that level. The more extravagant the documentation, the more content will never be referred to again.

Simple specifications
All of us have heard about the 80/20 rule. Perhaps 80 percent of an application’s business logic can be coded in 20 percent of the total development and testing time. Light methodologies rely on users really accepting the 80/20 philosophy. It’s true that you don’t want any sequence of user logic to result in errors or an application failure. However, you may not need to create an elegant recovery strategy for every possible input combination. For obscure error combinations, maybe it’s acceptable to simply point out to users that they have made a mistake and need to start the transaction again.

In the same way, users sometimes ask for every feature and function that they might possibly need over time. The better approach is to look at must-have requirements and then build the application to those specifications. In many cases, extra features will not be utilised often. If some borderline features do prove to be absolutely necessary, they can be added into future iterations or as enhancements after the project is complete.

Rank priorities as low, medium, and high, and then agree that no low-priority work will be incorporated in the project. You can note the requirement to show that it was considered, but there’s always something more important to work on than a speculative feature that will not be needed when the application goes live. Again, if the requirement is important, write it down, but as an item to be considered later, not to work on at this time.

It’s as easy as you make it
Light methodologies tend to falter when applied to very large and very complex projects, which require more rigor and structure. On the other hand, sometimes we make projects larger and more complex than they need to be. When you are working with your customer on a development project, try to always think simply. Think about implementing the basic requirements, in a simplistic manner, rather than trying to create a solution that meets 120 percent of the business needs. You may have heard the saying that “better is the enemy of good.” You can always make things better and better, but your sponsor will be more than happy with a good solution that is delivered on time and on schedule.

The Old Joel on Software Forum - Programmer Folkways

The recent threads posted on quality of life balance, software overcomplexity (which diminishes job satisfaction), and the repetitive nature of SW practice, impels me to comment.

Basically, it seems that when topics like this arise, the opinions that most programmers contribute all tend toward pretty much the  same set of conclusions and values. (note, I didn't say "all". The insightful few who question the status quo are usually torn to ribbons and personally attacked.)

My conclusion is that most programmers are personally half satisfied to miserable, but due to peer or self imposed pressure decided a long time ago that beating their heads against a brick wall was the only honorable thing to do. It's kind of a Spartan code of ethics, among an occupational group that never seems to have any profile with anyone outside the field.

Basically, it's the falling on a sword and impaling yourself while nobody else gives a sh*t.

And the sameness of most people's opinions forces me to conclude that we're a bunch of robots. Most of us adopt the thinking of our age group.

A few gems that always come up:

• Career "should be"