Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Obscurantism in Information Technology

Nicholas Carr's "IT does not matter" Fallacy and the Commodization of IT technology

Version 0.22

• Introduction
• Anti-historicism
  • Frivolous Treatment of IT history
  • Poor forecasting trick record
• Limited Understanding of Enterprise IT
  • False Attribution of Problems Connected with Dilbertalization of IT to IT Itself
  • Superficial understanding of enterprise IT costs structure
  • Entry level of understanding software support  issues
  • Discounting the value of IQ inherent in a local datacenter staff
• Idealization of "in the cloud" computing
  • Bandwidth communism
  • Sweeping under the carpet mainframe-style problems with external providers of IT services
  • Issues of security and trust
• Competing technologies
  • Virtualization as a better solution of the "excess capacity" and "commodization" of IT functions
  • Datacenter in the box
• Conclusions

Introduction

The slogan "IT does not matter" is catchy, provocative,  and wrong.  In other words it fits perfectly well the definition of the troll :-)

In 2003 Nikolas Carr (WOW, this guy somehow managed to get a Wikipedia entry !) declared that information technology inevitably going the way of the railroads, the telegraph and electricity, which all became, in economic terms, just ordinary factors of production, or "commodity inputs."   He also claimed that datacenters disappear and happy users connected to "in the cloud" service provider via Internet (software as service) will take place of programmers and system administrators. Datacenters as we know them will disappear. 

In his notorious article "IT does not matter", published in the May 2003 edition of the Harvard Business Review he stated:

From a strategic standpoint, they became invisible; they no longer mattered. That is exactly what is happening to information technology today, and the implications for corporate IT management are profound.

In this speculative discourse, Carr offers us a vision of the future direction of computing.  But in many ways his vision if cloudy. Carr outright dismissal of the relevance of current IT and waxing lyrical on "in the cloud" service providers smells utopism. However, using "in the cloud service" providers (or WAN-based distributed computing in more technical terms), has a lot more pitfalls than many people realize, especially when they're under the influence of vendors' cozy brochures. Due to lack of qualification (Carr can approach IT only as a sociologist; he does not have real understanding of technologies involved)  his article is a masterpiece of self-deluding, self-serving obscurantism applied to information technology.  I am using the term "obscurantism" here in the meaning close to one of the meanings of the term provided by Wikipedia :

In the 19th and 20th centuries "obscurantism" became a polemical term accusing authors of writing in a deliberately vague and abstruse style in order to hide their vacuousness: the writer's ignorance is obscured.

And unfortunately it was rather uncritically met by mainstream press. This page is a late attempt to analyze his hypothesis (or fallacy to be exact) and call a spade a spade.

In best obscurantist traditions Carr uses pretty weak and questionable analogies in his attempts to make the case that IT technology does not matter all that much and the local datacenters are doomed.  He uses very superficial analogies with railways and electrical grid but deliberately ignores the obvious basic fact that the IT technology is very different with the complexity exceeding anything else created by making and unmatched flexibility which alone ensures that race to new heights (and new dimensions of competitive advantage for enterprise) is endless and basically unpredictable. His utopian vision of dominance of remote datacenters (in the form "in the cloud" service providers) and local users who in some mystic way became InfoTech specialists is not supported by the last five years of IntoTech development, which were dominated by the processes connected with increased the power of local computing devices (virtualized servers, desktops and laptops with virtual machines running two or more OSes simultaneously, virtual appliances, wireless connections as the standard laptop connectivity channel, blades, etc).

While WAN connectivity dramatically improved due to ubiquity of fiber lines and Internet became more diverse and powerful in recent years in no way that suggest the dramatic switch to "utility computing" is under way and that the best and/or most economical way to implement now standard (mature) levels of IT technology (services as Carr calls them) is to implement them "in the cloud" (bandwidth communism). Carr predictions also looks far from being plausible from several other standpoints, which we will discuss below.  Utility computing is a viable trend but it has its limitations and as such is far from being dominant and defining trend of the last five years. Nothing suggest dramatic acceleration of this trend in the future.  A lot of issues are still unresolved and costs advantages in many cases are questionable. Recent developments of server technology (virtualization) as well as laptop technology actually suggest quite an opposite direction: local computing can help and even increase its share (datacenter in the box concept).  Mobile trailers that are now produced as "portable datacenters" are probably the most dramatic illustration of this trend (Microsoft, Sun).  They run in direction completely opposite to Carr cloudy vision.  Due to those factors I think that the rumors about datacenter demise spread by the author are premature.

IT technology now serves as both the right of entry, and the key to success for almost any business you can think off.  In a way IT technology strategic importance is not importance of the set of tools or protocols (which now can be standard like Windows, Microsoft Office,  TCP/IP, DNS, SMTP, etc) but importance of the superstructure, architecture of integration of those tools. In InfoTech like in Lego it is quite possible to implement completely non-standard, unique system from standard parts.   And this flexibility of IT serves as a catalyst for implementing other innovations (GPS on trucks, RFID in packaging,  ecommerce and internet auctions systems, etc).  In this sense the strategic importance of IT even increased as technologies became more diverse and mature.

In the same way the fact that companies can purchase the same hardware does not mean that strategic advantage disappear. Here, like in cooking, ingredients might be the same, but quality of a dish can be completely different. Mismanaged, fashion driven and/or badly though out hardware acquisitions can drive the company into serious problems and even doom it to a failure. the same in true about enterprise software acquisitions as was the case with several misguided SAP/R3 implementations, which brought down large, multi-billion companies.  Even moderately "mis-designed" or "mis-stress-tested" enterprise software can put the company on the ropes, as we all saw recently in airlines: inferior IT system can wipe the profits for the year due to a single large storm.

Due to the timing (end of dot-com bubble deflation) Carr oversimplified and sensationalist hypothesis became the fig leaf for justifying outsourcing. Carr himself soon became "traveling snake oil salesman" of sort. I think that the resurgence of religious fundamentalism and  "IT does not matter" fallacy has common foundation -- modern culture (not necessary only American, despite a long tradition of anti-intellectualism in the country) devalues knowledge and rationalism. While junk-science and a celebrity-focused culture might be financed by defenders of status quo, with "IT denialism" the reality is more complex. Carr understood all too well that promoters of miracle diets (corporate IT diet in this case) do well in the current environment and he just could not resist the temptation to milk the cow (although judging from the level of discounting of used books both his subsequent books proved to be paperweights)... 

There is little objective evidence that "IT does not matter" fallacy has typical right-wing backers. Still this is a clear example of obscurantism in action and in essence is not that different from similar more politically charged types of obscurantism like Neo-Creationism. An important point is that the impact of anti-intellectualism is much greater today than the 1800's when science and technology had much less to offer.  The ability of obscurantism to move and find a fertile ground in IT is in itself a very interesting and telling development. 

While dismissing the current structure of IT based on local datacenters and IT staff as an obsolete Carr provided very little future guidance. His simplistic analogies  between IT and railways and between IT and electricity grid looks too superficial for any informed IT professional. Unlike railways or electricity networks the InfoTech system are the most complex artifacts ever created by mankind.  Yes, the most complex artifacts created by man. And that fact alone is enough to dispel the myth Carr tries to propagate. While move to service providers might take place in some limited form, the hypothesis that they will become dominant form of supplying IT services  and users can overtake IT professionals is too far fetched.  As for "in the cloud service providers" this is the return to the mainframe era on a new level and we all remember how much IBM was hated at this time and how much energy and money people have spent trying to free themselves from this "central provider of services" trap.And to what extent adoption of PC were driven by the desire to send a particular "central service provider" to hell. 

His extrapolation of Google success on the whole industry is extremely naive taken into account level of complexity inherent in the industry. He also ignores the fact that Google record is far from being stellar and the company experienced serious difficulties in many attempts to extent its success beyond search (including his attempt to unseat Microsoft Office dominance). Even the most successful  of  the "in the cloud" services -- Web hosting -- is far from  a paradise and people often are opting out of it as soon as their site had grown beyond certain size and/or complexity. First they switch to accounts with shell access, then to virtual machine and then to hosted server. In the later case the value of whole exercise for large company is doubtful.  Yes I know about a couple of companies (actually eventually eaten by competitors) who paid "arm and leg" for outsourced Web infrastructure. But in cases that I know that was more sign of incompetence of IT brass in particular companies, then the sign of coming new times (in one case a company paid approximately $20K per month for pretty straightforward Windows-based corporate web site).  I doubt that Amazon or eBay will outsource their Web hosing to somebody.  It's pretty funny that, as I noted before,  Carr's vision of "in the cloud" service providers is very similar to previous mainframes and we all know the horrors of mainframe infrastructure which was decimated by PC revolution. That, by the way, means that in a very deep sense our hero is a reactionary figure which fits obscurantist label perfectly well. 

At the same time Carr is a gifted writer and astute observer who in his writings helped to reveal the level of degradation and bureaucratization (what is often called "dilbertization of IT") that exists in current datacenters today. Here is an apt quote from one Amazon review:

The key question is, "is it as bad as Carr reports?" I can share my experience as a consultant who has worked some of the largest US and global corporations by answering "unfortunately, yes". I've seen the symptoms Carr cites in one engagement after another. It is not the fault of technology. I've witnessed the implementation of technical solutions that should have added value to business operations, yet were so mismanaged by IT that the solutions never came close to the projected ROI that justified their acquisition and implementation. Indeed, this book is similar to a collection of anti-patterns -- common bad practices -- which, sadly, reflect a typical IT department.

But in no way that "dirt-digging" can justification the decimation of current IT and replacement by the fantacy of "in the cloud" specialized services providers.  First of all as with any utopia he does not discuss shortcomings of "in the cloud" solutions, which are plenty. 

IT, with all warts and problems, has real strategic not tactical or commodity value in many (but not all) large businesses. Moreover as lower level of IT became readily available  and more standardized the advantage just moves to higher level. For example, TCP/IP connectivity for example was a huge competitive advantage of the USA 1980th and 1990th). Later this advantage modes to higher level (http based protocols, search, eCommerce). Then in a side move TCP/IP moved into wireless.  It is still evolving and it might be that now IP6 can give to some companies a competitive advantage (for example, better security, as hackers are by-and-large IPv4 bound).  Just imagine how much money large enterprises are wasting on all those expensive and semi-useless Firewall 1, F5 and BlueCoat appliances ;-). 

Of course, there is a lot of problem with IT as it exists in large corporations.  It is often inefficient, wasteful and not very friendly if not outright hostile to business needs.  It looks like Carr provided more or less objective and insightful analysis of the problems with corporate IT (I actually enjoy reading his blog) and offered unscientific, snail-oil salesman style explanation/solution to those serious problems, a simplistic fallacy (aka "vision") of "in the cloud" paradise based of simplistic and misplaced  railways/electrical generation-based analogy. I would like to stress that being unscientific does not matter in case some false idea is needed by some political forces and "Carr hypothesis" served perfectly well as an ideological justification of IT outsourcing. 

But while it is easy (and absolutely appropriate) to criticize corporate IT for inefficiency and excessive bureaucratization, you should not throw a child with the dirty bathwater, which Carr definitely does:

"In the long run, the IT department is unlikely to survive, at least not in its familiar form," Carr writes. "It will have little left to do once the bulk of business computing shifts out of private data centers and into the cloud. Business units and even individual employees will be able to control the processing of information directly, without the need for legions of technical people."

I would argue that "in the cloud" paradise looks more like software demo then reality ;-). It turns out that for the last five years there was relatively little demand for "in the cloud" service providers and this technology competes with several other, first of all with virtual appliances and "datacenter in the box". I see are several fundamental problems with Carr views:

Anti-historicism

• Unability to Grasp the Complexity of Modern Software
• Frivolous Treatment of IT history
• Poor forecasting trick record

Unability to Grasp the Complexity and Diversity of Modern IT Infrastructure

First of all Carr approach is anti-historic. IT already experienced several dramatic transformations. Also complexity if II systems has no precedents in human history and as such analogies with railways and electrical greed are deeply and irrevocably flawed as they does not capture the key characteristic of the technology. Carr analogies related to railway and electrical grid are flawed because software systems are the most complex artifacts known. The level of complexity of a large computer IT infrastructure is simply not compatible with the complexity of railway network and electrical grid (which are pretty complex artifacts in its own rights; the problem is that software systems are not complex they are super complex).  Here quantity turns into quality.

Not everybody understands the level of complexity inherent in software systems and, as Andrei Ershov noted in his famous article Aesthetics and the human factor in programming (based on presentation to AFIPS Spring Joint Computer Conference in 1972),  the common gap in understanding typical for management  presuppose that claims of software professionals and IT professional in general for a special status will always be disputed.

And as he noted even in 1972 programmers and system administrators felt that they are gradually slipping into the paws of managers, who try to male the work planned, measurable, uniform and faceless.  But despite all those attempts programmers (as well as network and system administrators) survived because complexity of those systems make such talent a very scare commodity.

The second example of his anti-historical approach is that Carr profess a very narrow and static definition of IT.  During the last 50 years IT evolved in many different directions and now is differentiated between a dozen or so sub-fields which are different to the point at which people from one area often do not understand technology of the other.  For example, data base engineering is one major part of IT, networking services is another. Web services is example of a composite field which involves and interacts with both networking infrastructure, server infrastructures and data bases. There is little common between them, they use different technologies (few networking engineers know Perl, Java and SQL). Moreover the level of specialization is such  that even within a single server field there is quite a bit of specialization between personnel who service Windows server and those who service Unix servers.

This complexity and diversity also limits the possibility of the deployment of non-professionals. People who do not have specific talents cannot be taught programming for the same reason not everybody can ne taught to play a violin, guitar or clarinet. You need a specific abilities and they are pretty rate. For example programming talent is pretty special and might actually better correlates with writing talent and musical talents then with mathematical talent; not everybody who can write a program is a programmer as in no way I consider graphomans to be writers.  Of course, some programmers are simultaneously gifted mathematicians (like Donald Knuth).

This complexity also influence the economic benefits of software deployment in a very interesting way. For example, the architectural talent becomes essential for the success of any given software or large infrastructure project. Differences in costs of project complex project run by professional team leaded by talented architect and teams of the run-of-a-mill people without any architectural vision are quite drastic.

Lack of local IT talent instantly makes a company a donor in the hands of ruthless external supplies and contractors. Consultants (especially large consultant firms) can help but they also can become part of the problem due to the problem of loyalty. We all know what happened with medicine when doctors were allowed to be bribed by pharmaceutical companies. This situation which is aptly called "Viva Viagra" and which useless or outright dangerous drags like Vioxx were allowed to became blockbusters were replicated in IT: myth about independence of IT consultants is just a myth (and moreover, some commercial IDS and EMS systems are not that different from Vioxx ;-).  Cases when a company which lost IT talent (for example due to outsourcing) was overcharged two or even ten times for a particular project are not that uncommon. And believe me clueless companies can be persuaded to buy equivalent of Vioxx in software or hardware (appliances) quite easily, much like clueless patients. Such cases are pretty common; they are just too embarrassing for company brass so few people (or nobody) are fired and the whole mess usually is swiped under the carpet.

Anybody with long experience in IT organizations can tell stories when a particularly stupid or arrogant manager  (arrogant managers never trust their own staff)  in charge of a functional IT unit costs company millions of dollars in additional expenses due to overpaid consultants,  unnecessary hardware and enterprise software cost he/she bravely bought. Add to that exorbitant maintenance fees and that small victory of a particular manager can turn into non-strategic but pretty costly disadvantage for the company.  Despite aura of respectability of the industry software salesmen are much closer to used car salesmen then people unrelated to IT presuppose.  If those people feel a weakness, that's it: the company is cooked. In some way the atmosphere also remind the atmosphere in large investment banks during recent financial crisis, the atmosphere similar to high demand cult with suppressed independence of all actors from top to boom and where clients are openly despised and viewed as legitimate prey.  

Also just the level of complexity of software systems (including even such common as MS Office) means that Carr's idea the users can replace InfoTech professionals is simply naive. Yes, in some cases regular user can easily  accomplish simple tasks (and most of tasks a typical user faces are simple; that one of the reason of huge success of MS Office) without help from the expert. But in such cases Ms Word is downgraded to a variant of typewriter. Also Mongol horde approach can work in some cases: instead of using capabilities of software you can add low-paid low-trained people and with some additional management effort get similar net result.  But in other cases such an approach can be an unmitigated disaster. This Mongol horde analogy is of cause stretched but the difference in capabilities if a user and seasoned professional are so substantial that they cannot be ignored. 

The limitations of a regular user can be easily understood by evaluation the average level of mastery of MS Word or Excel. Quite frankly it is deplorable. A regular MS Word user cannot productively use probably 80% of MS Word functionality. Moreover out of 20% of features that he/she uses probably 10% is used incorrectly due to lack of training and education (styles is one such feature).   For Excel the situation is even worse as this is more complex and very capable product. This lack of knowledge translates into lower productivity and as such costs money. First of all because additional software is bought and maintained for the tasks that with some level of sophistication can be accomplished by plain vanilla Office. Knowledge is power and power is money.  

For more complex enterprise tools like Lotus Notes, LAMP stack (Linux-Apache-MySQL-PHP/Perl/Python), etc that situation is even less user-friendly and differences of productivity and used capabilities and achieved results of a regular user and seasoned professional more drastic. 

Complexity of software products also means that without specialists users tend to overbuy software. If we are talking about a large firm that can in minor way affect the bottom line as enterprise license for some packages costs half-million dollars or more.  For example, it is not uncommon for a large company to have overlapping enterprise packages for monitoring hardware systems and network equipment each costing, say $200K a year.  Such duplication can be avoided but if and only if there is adequate IT talent of the floor. And with high level of talent the enterprise can use open source products that might provide enough functionality for the particular enterprise. In a way the lesser is the level of IT talent, the more money the enterprise needs to spend on enterprise software and hardware  

The last but not least: it is not enough to understands a particular software technology or protocol. Real professionalism starts with the understanding of the limits of applicability. This understanding is clearly lacking in Carr view of service providers, which definitely have their place and importance but are generally overrated by Carr as a universal solution to IT problems. In fact, while solving some existing problem, they simultaneously are actually a source of new problems. As we will see below this model have server serious limitations. Among them are "bandwidth communism" and re-introduction of  mainframe-style mentality with all its warts. 

Frivolous Treatment of IT history

In his writings Carr presuppose that this is the first time IT faces with a radical transformation (switch to "in the cloud" providers in his vision). But IT the history is much more complex and does not fit this paradigm. Unlike railways or electrical grids, IT already experienced several radical transformations with the last major one being the demise of  "mainframe era"  dump terminals and the emergence of PC-based corporate IT.  Actually the IT history is even more complex and can be split on half-dozen stages (with end of which meaning a small revolution):

1. Batch computing dominance (1950-1960).
2. The era of timesharing (late 1960s - early 70th) mainframe-based time-sharing services became kings of the hill in computing centers of their day.  This version of transformation can be called "batch processing does not matter".
3. Minicomputer revolutions (late 1970th-early 1980th), when mainframe were first undermined by DEC computers and Unix
4. PC revolution (early 1980th-late 1990th) which hundred of PC companies were first created and later most of PC companies perished in the wave of consolidation. This stage of development of IT might well have been called "mainframes does not matter" at it put the final nail in "mainframe dominance" coffin. Mainframes as a class of computers survived and even later experience some kind of limited Renaissance (VM/Linux).
5. WEB revolution (early 1990th) emergence of WEB-based infrastructure, portals (Yahoo) and search engines in early 90th (AltaVista, etc). Start of dot-com boom and emergence of linux and open source. 
6. Dot-com bust was the third bust and can be called "Internet matters less that previously thought". During dot-com bust many companies went bust because they try to provide Internet services which nobody needed. That included a lot of utility computing pioneers as well as Internet infrastructure providers. 
7. Reemergence of collaborative computing (stating from 2000) with new collaborative tools like blogs and Wikis as well as Internet search engines like Google. 

As you can see there is very little similarity between IT history and railway history or history of electrical networks. They are artifacts of different complexity.  IT history does not support  Carr simplistic hypothesis of an imminent  waterfall-style transformation from local datacenter into services with Internet connected service providers.  There will definitely be new transformations but so far it is early to tell what form they will take. 

Poor forecasting track record

If we try to time his predictions then his performance as a forecaster was so far not impressive. He wrote his paper in 2003 almost in the bottom of IT slump. But IT recovered from slump and expanded in 2003-2007.  In a way he behaved like a  typical "doom and gloom" stock forecaster who missed the upswing of 2003-2007 and judging from his books he does not care to check his former forecasts --  he is too busy writing new books.

But truth be told in 2004-2007 IT employment in corporations (and corporate IT datacenters) have grown. For example in 2007 alone the staffing has grown 7-8%;  2008 might be a down year due to general economic slowdown and huge layoffs in financial sector (for example, UK banks will eliminate 10K IT jobs). But this is nothing to do with Carr's "in the cloud" vision.

In the same timeframe the rise of "cloud based" services was impressive (especially for WEB hosting providers) but so far failed to doom the local datacenters.  During those years Google did not add anything strategically important to its search. Web based applications he developed were not received very well and linger in relative obscurity. Meantime Microsoft increase its hold with Office 2003 and Office 2007.  That strongly suggest that people were voting with their money for the locally installed software during this period and there are some limits of what web applications can provide (limits which are gradually expensing, but at the page similar to the page of improvement of regular "installed on the hard drive" applications). 

Also the current switch to Web based mail by regular users cannot be treated like a success of "cloud" computing. In my opinion this is just a reflection of limited capabilities of SMTP protocol which make simple Web client more or less adequate for average user. SMTP does not allow sending large files as attachments (or more correctly the same is limited on mail hubs), interaction with  the calendar and ToDo lists is pretty simple so the problem of limited bandwidth does not come into play the way it degrade experience with Flicr and YouTube..  Still capabilities of best Web-based email clients are poor in comparison with best corporate email clients like Outlook and Lotus Notes. It is also telling that people who need advanced capabilities usually use both local and remote clients. Here we can talk not about replacement but "peaceful coexistence".

As for typical Office applications "displacement of desktop applications" currently looks like a failure.  There are simply not very many takers for them yet.  And laptops power have grown to such extent that to replicate their functions with dumb devices (network terminals)  on remote servers is a very questionable (and expensive) idea indeed. I for example doubt that Google can ever provide me the computing power I use in my Dell D620 laptop in Excel and FrontPage. And that factor alone limits the applicability of  "cloud computing" for me and many other users, although I am not against experimenting with Microsoft remote information backup/synchronization services when they will be available.

All in all, at lease for five years since 2003 his prediction can be classified as false (or premature), although the current financial crisis might negatively effect IT employment, but for reasons that have nothing to do with the attractiveness of the cloud-based services and all this blah-blah-blah about electrical utilities analogy.  Trends that we can observe are more complex then simplistic move to "cloud-based" service providers.

In this limited (but not short by IT standards) timeframe Carr proved to be a  false prophet.

Sending the concept of a datacenter with a dedicated staffing to the dustbin of history  because it cannot deliver the impossible is foolish at best. At the same time software applications capabilities are limited mainly by human imagination and existing generation of computer hardware. Since human creativity is so vast in potential and computer hardware is still evolving by leaps and bounds, it would be foolish to think of software technology as being mature. It still quickly evolves and each decade introduced new hardware and elements in the corporate IT infrastructure. As long as software continue to evolve  IT will always be strategic. One example are track companies who missed the benefits of GPS-based positioning and wireless communication.  Carr strategic advice to keep a tight grip on the IT budget and “innovate when risks are low” is by definition a defensive strategy that is suitable for "old or dying" companies, not from the new hungry for market share players. The key for success is the vision what areas of It can serve as a catalyst for new business processes or enhance existing processes.  Currently the potential for this role of "catalyst" for custom WEB applications, new types of advertizing, specialized monitoring services (like truck tracking) is quite substantial. 

Andre Gide once said “Believe those who are seeking the truth; doubt those who find it.”

Limited Understanding of Enterprise IT

First and foremost Nicholas G. Carr does not understand the technology he is writing about (Carr holds a B.A. from Dartmouth College and an M.A., in English literature, from Harvard University). At best he can approach IT as a sociologist, not as a professional.  Such a sociological approach is completely legitimate and useful but prone to oversimplifications and overgeneralizations. We can discuss several issues connected with oversimplifications and overgeneralizations here:

• False Attribution of Problems Connected with Dilbertalization of IT to IT Itself
• Superficial understanding of enterprise IT costs structure
• Entry level of understanding software support  issues
• Discounting the value of IQ inherent in a local datacenter staff

I would like to stress that IMHO Carr has an extremely week understanding of networking issues, especially the cost of high bandwidth wide area network connections and technical challenges of providing them reliably on 24*7 basis for a large enterprise customers such as, say, SAP/R3 users.

False Attribution of Problems Connected with Dilbertalization of IT

Bureaucratization of datacenter is also an old problem which became prominent early in mainframe era. Here is an appropriate definition from  Todd's Humor Archive (reproduced with minor variations):

Computing Center [n], is an organization whose functions are

1. To impede wherever possible the development and usefulness of computing in the company or University.
2. To gain the lion's share of funding, spend it largely on obsolete, bloated and otherwise inappropriate IT Solutions, and convince the businesses/campuses wherever possible to spend funds on the same.
3. To oppose vigorously any new, useful and popular technology for three years or more until nearly everyone on the business/campuses and elsewhere in the world is using it, then to adopt that technology and immediately attempt to centralize and gain complete and sole control of it

The fact that "Dilbertalised IT" of many enterprise datacenters is horrible as a business enabler is nothing new and does not mean that IT itself is useless.  IT mismanagement erodes business profits, shareholder value, and business operational efficiency. As one Amazon reader recollects this is not the first time IT faced those problems:

When I started out we "MIS" professionals were the priests and priestesses who worked our magic in glass rooms. We were merely arrogant then. Life was simpler and some vendors worked closely with us. IBM, which is my main background, had a reputation for never letting their customers fail. That is not to say that their recommendations and solutions always translated into business value for their customers, but rarely did they result in disasters either. As time went on though MIS became IS, then IT. Systems grew more complex, proprietary systems gave way to interoperability, then open systems, and new vendors started arriving in droves. Innovation fanned the flames of complexity, and IT remained arrogant, but began focusing so much on the technology (and trying to keep up with it) that they lost sight of business needs. Methods devolved into chaos and the chasm between IT and the business widened to the point where IT was in some cases counter-productive to business needs.

Some IT departments are dead because of bureaucratization, when good workers leave and the vast majority of IT staff became utterly incompetent, lack the necessary skills, both technical and business to effectively and efficiently perform their duties. Those problems has nothing to do with and cannot be solved with the utility computing.  Such "death of IT" is more common that most people assume.

Still Carr did a reasonably good job in exposing IT warts and this part of his writing has some sociological value and might partially explain the success of the article and books, the success that permitted him to milk this cash cow for several years. There is no doubt that the process of dilbertization of IT went too far.  For political reason, his claims have found resonance with bean counters and some people in positions of power. In a way he provided plausible and useful justification for outsourcing everything and everybody in IT  ;-).  That paradox permits him to survive and prosper in the chosen role.  Such metamorphose was not always appreciated by readers. Here are some Amazon reviews on his book that contain apt observations about "snake oil seller" part of Nicholas Carr success: 

Twisty language beguiles the easily amused, May 30, 2004

By A Customer
It's star time. While filling in for a 'let go' editor of the Harvard Business Review (HBR), a business writer with no personal involvement or experience in IT uses prime-time pages of HBR to conjure up a British tabloid piece that raises him to IT stardom.

Watch the movie, 'Being There' to catch what's going on with this follow-on book from the smash hit, "IT Doesn't Matter" in the May 2004 issue of HBR. Although a formula for business failure, this is a shoe-in for the Hype Award for Selling Books. Peter Sellers, ...and Chance the Gardener...take notes.

Klyde Hartler, Frankfurt, Germany
 

Written by an author who never managed an IT organization, May 26, 2007

Although Nicholas Carr has some good eye opening arguments, most of it is based on theory, and not practice. As usual, although such arguments are worthy of thought and debate, it should never be taken at its face value as the new paradigm.

For example, Nicholas Carr makes an argument there are huge amount of IT "waste" because of excessive disk capacity and CPU capabilities. People who have worked in IT for any reasonable period of time knows that excess capacity rarely remains wasted, especially when many organizations experience 50%+ growth in disk storage needs every year. The same thing goes for the CPU and memory capacity. Having excessive hardware capacity for the future needs is called "capacity planning" and not a waste. Anyone who has been taken off guard (and paid a dear price with costly downtime) due to lack of disk storage, memory, or CPU horsepower knows the value of having some excessive hardware capacity. Minimizing the risk of critical downtime is hardly a "waste".

Some of Nicholas Carr's argument, admittedly, is at least somewhat, if not mostly, true. For example, he argues that IT, particularly its infrastructure, offers little competitive advantage because IT has become a commodity. This is true of SAN, OS, RDBMS, email, routers, switches, and servers (among others). If one's competitors have better and faster storage, for example, all one needs to do is buy the same storage from the same vendor at the same price to negate that competitive advantage.

Nicholas Carr does not address two important factors enough in the competitive forces of IT:

1)The power of innovation. Although competitive advantage via better hardware is becoming non-existent, a superior code written by a good developer is much more difficult to duplicate. Carr seems to dismiss the idea of innovation in IT since it is a "commodity". This is only partially true.
/
2)The competitive advantage of having superior IT personnel. The quality of the Knowledge Worker has become a key component, and perhaps the main key component, of competitive forces. In an era where it is increasingly difficult to gain competitive advantage, the quality of a company's Knowledge Worker is becoming more crucial. Despite the increasing commoditization of IT, the quality of IT depends largely on the quality of the IT personnel.

Excellent questions with inferior answers,
July 8, 2004 `Does IT Matter' is a difficult book to rate.

As to the questions it raises, it deserves 5 stars. But its answers, are two-star, at best.

By way of analogy, most bomb threats are bogus, but each one must be treated as if it were genuine. With that, in his new book Does IT Matter?, Nicholas Carr throws a bomb, and it turns out to be a dud.

Carr's book is an outgrowth of his article "IT Doesn't Matter," which appeared in the May 2003 issue of the Harvard Business Review. His hypothesis is that the strategic importance of IT has diminished. Carr views IT as a commodity, akin to electricity.

He also compares IT to the railroad infrastructure. In the early days, railroads that had their own tracks had a huge advantage, but once the rails become ubiquitous and open, that advantage went away.

Carr feels that since all companies can purchase the same hardware and software, any strategic advantage is obviated. It's true that the core functions of IT (processing, network transport, storage, etc.) are affordable and available to all, but there's still huge strategic advantage to be gained in how they're implemented.

It's much like two airlines that purchase the same model of airplane. If one airline streamlines and optimizes operations, trains its staff and follows standard operating procedures, it can expect to make a profit. If the other has operational inefficiencies, labor problems and other setbacks, it could lose money. The airplane is identical, but the outcome is not.

Carr is correct in that there have been some huge IT outlays of dubious value. But to say that IT is simply the procurement of hardware and software is to be blind to the fact that hardware and software are but two of the myriad components of IT.

To use the railroad metaphor, hardware and off-the-shelf software are the rails of IT; how they are designed and implemented is what provides their strategic value. Carr views IT as completely evolved. But the reality is that although IT has matured, it still is in a growth mode. The IT of today is vastly different from the IT of both 1999 and 2009.

Carr's view that most innovations within IT will tend to enhance the reliability and efficiency of IT rather than provide a competitive advantage is in direct opposition to what is said by every CIO I have met.

YOU BET IT DOES.,
May 29, 2004

Happened to pick up and browse through this paperweight at the airport and patted myself for not having bought it.

For one thing, in Carr's world, information technology managers are fools that drain dollars to the tune of 2 trillion every year, slavishly upgrading to whatever new thing vendors want to sell. This is a VERY narrow definition of organizational "IT". The world of technology is much wider and ever-expanding, making a direct or indirect impact into our daily lives, both personal and commercial. So the scope of this book is a bit hog-tied.

For another, these petty cavils are not new. Paul Strassman (Paul Strassman Inc.) has been saying the same things for a while. Morgan Stanley's star analyst Stephen Roach broke into the same bungled song and dance in the early 90s. Even more recently, "the bubble burst, I told you" has been recently toted as the safely wise justification. One could think of the many financial bubbles that have burst to disastrous consequences, but that's for another day.

Is IT unproductive? If the answer to that is yes (which it is not) then there are two possible explanations. That IT investments are ineffective (the author's take) or that we're not measuring productivity right. Nobel laureate MIT professor Robert Solow, with his work on "productivity paradox", has proven that the latter may be just as true as the ineffectiveness of tech spending.

Come to think of it, consider this:

- Without IT, the author would not have the website that promotes this hackneyed refrain.
- Nor would we have the Amazon.com website which allows hundreds of readers to commit the mistake of spending money on his book.
- Or on its PDF ebook version.
- IT was used in the the typesetting of this very book.
- IT is used by his publisher to manage all their accounts, including the royalties the author will get.
- IT is used to power almost every record in the university and the office that pays the author his monthly salary (Harvard)
- It is IT in the car that he drives to work that parks automatically or that aids his navigation
- ...and so on...

Overall, the book didn't catch my attention for more than 15 minutes. Which is about the time I've spent on writing the above.

Superficial understanding of enterprise IT costs structure

In a big manufacturing company with multiple local datacenters the total cost of all IT operations is often less then 1% of total costs (in financial companies it is more, reaching 3%-10%). So anybody who tries to speak about advantages of other solution need to explain why current costs are excessive and why we need to jump into bandwagon of "in the cloud" service providers (which might BTW be more expensive and provide lesser quality/agility then local services).  In big corporation economy of scale already fully realized so move to the cloud is questionable. If total costs are under 1% to justify the mess that occurs is a really difficult task. The same thinking is applicable to the return on investment in reducing those costs.  In no way modern enterprise IT is a resource hog.

At the same time race to the bottom in IT costs is self-destructing. There is some limit in cost-cutting after which attempts to diminish IT costs and cut people backfire and actually lead to increased costs. One problem here is the corporate brass's inability to separate wheat from shaft. The real competitive value of a good IT is not only the infrastructure and smooth operation of network and corporate applications; it also provides valuable protection for the company from ruthless and energetic charlatans (aka consultants) who often try to sell the company some new fashionable snake oil  in best big-pharma traditions (IT analogs of  Vioxx and other useless and/or dangerous medications, produced to generate profit not to solve real problems :-).  Outsourcing also destroys loyalty and that has its own (often huge) costs. It is not uncommon that disenfranchised IT staff stops being the filter against aggressive marketing and soon the company will successfully complete the acquisition of several ISS sensors and related infrastructure or other useless Vioxx-style appliances. SAP/R3 is also a mixed blessing in many environments leading to increased not decreased costs and flexibility. Actually to deploy SAP/R3 in unsuitable environment is one of the best way to destroy the company. SOX games also can be played pretty destructively for uninformed and clueless brass after no critical mass of IT IQ left the company. See my IT Outsourcing/Offshoring Skeptic: Fighting Outsourcing Myths article for detail.

That means that it might be hard to use IT to gain a strategic advantage (first of all you need talented people as IT is just a technology), but absence of IT talent puts the company in the cost disadvantage. A lot of firms that tried to outsource IT discovered this dimension of the strategic value of  IT rather quickly. Because IT expenditures remains high and businesses can be held hostage by poorly designed IT system, there's enormous need for technically astute, hard-nosed IT professionals who can navigate the companies between Scylla of IT vendors and Haribda of  IT outsources.  Dilbertalization of IT inevitably leads to  high additional costs, costs that exceed to the cost of keeping local IT staff.  

It looks like Carr does not understand what "excessive capacity" means in IT and how much in costs overhead large enterprise is suffering from it. This is a complex topic and I will take elementary example that shows that naive laments about unused capacity are without merit. First of all there is a dynamic of increasing of load of the server during life time so a server that has 3-5% CPU load at the beginning of the life cycle can have 20% load at the end (three-five years later). Also the cost of this supposal redundant capacity is negligible and adding capacity at the beginning is a good insurance against very costly hardware upgrades at the end of the life-cycle.

For example entry level Dell PowerEdge 1950 server with two 2.33GHz CPU , 8G of RAM and four 80G 15K RPM drives costs around $4K and PowerEdge 2950 server with two 3.0GHz dual core CPUs, 32G RAM and 6 146G, 15K RPM drives is approximately $8K.  The difference is just $4K which is rounding error in most datacenter cost structures.  So it is often wise to over specify the hardware using this as an insurance against adverse developments as losses are minor, but potential benefits are great (for example what if we underestimated the load of company will buy another one which double the user population and the load of the server).

Moreover with blades and virtualization (for example using XEN) the load theoretically can be moved dynamically from one server to another within the datacenter so "cloud based" provider even in principle does not have any "capacity utilization" advantages over local deployment. Specialization advantages of a service provider still exists but can be deal with by outsourcing maintenance of the servers and OS to the third party.

Also "economy of scale" argument applied to service providers can be misleading. Large enteprise IT already enjoys significant economy of scale and further centralization might not add anything significant. Let's discuss a hypothetical deployment of very simple and relatively smooth for deployment in the cloud case of a single web server for corporate Intranet. Let's assume that this server supports just http and some minimal video streaming services (corporate, vendors and business partners video materials) so bandwidth is not  a big problem.  Now let's compare the cost of getting the same server in a cloud and associated costs:

• Local datacenter deployment. The total hardware costs (one Dell server with the configuration listed above) are between $5K and $10K for three years period (that includes 3 year hardware warranty from Dell). Cost of bandwidth is zero (the server is on local network).  The cost of  basic OS support which includes provision of patches is $350 a year (we assume Suse 10).
   
• Remote "dedicated server" deployment. PowerEdge 1950 with much weaker (and cheaper) configuration (2G of RAM and 2 *146G Harddrive) is approximately $600 a month and PE2950 server with 4G of RAM and 4*146GB drives is $1200 a month (see Web Hosting Dedicated Hosting Dedicated Server Upgrades) . That means that you have  to pay  $17K for software maintenance for PE1950 or 32K for maintenance of PE2950 per three year.  A typical Unix admin in enterprise environment usually service approximately 50 servers and has cost of ($70K * 3 /50)= $4.2 per server per three year.  We did not count possible additional cost of bandwidth in remote deployment and the value of having local knowledgable people on staff "("goodwill").

The question arise does remote deployment provides any real savings even if we factor in additional building rent/electricity/ air-conditioning, etc related costs?
 

Entry level of understanding software support  issues

Carr naively consider users capable and willing of learning complex software products and enjoying keeping data "in the cloud".  My experience completely contradicts both of those claim.

First of all learning complex software is far beyond many "regular users". For example, skills of most users of MS Word (probably including Mr. Carr) are entry-level and they cannot use even 10% of capabilities of MS Word.  At least not on the level of creating and using complex macros (which requires VBA skills).  Often not even on the level of sing style sheets. This is another facet of  complexity problem, where quantity turns into the quality. 

Now about the joy of keeping all your data "in the cloud". From an end user perspective, it doesn’t make a difference if a server glitch is caused by a web host or a software service provider. Your data is in the cloud, and not on your local PC. If the cloud evaporates you can’t get to your data. If the service or site goes down, you can’t get to your data.

While software as a service might represent a licensing savings, there might be a better way to achieve the middle ground and lot to lose all advantages of local storage of data. For example by using some kind P2P infrastructure automatically synchronized so that you can view/edit data locally.  Data should not be held hostage until the user can get back to the cloud.  In this sense Microsoft's Microsoft Live Mesh might be a step in right direction as it provides useful middle grown by synchronizing data across multiple computers belonging to a users (home/office or office1/office2, etc).

Ignoring potential disconnect and communication problems with service providers

Large companies have become very adept at establishing remote teams in places like India, Russia, etc. Due to their ability to offer higher wages and better environment they are able to attract local talent and run challenging research and development projects. Often, though, these outposts become disconnected from their parent companies because they fail to establish rapport with key participants in the metropolia. Foreign prophets are usually ignored.  There is something fundamental in this "tragedy of local talent" and communication is still a problem even in the age of videoconferences. Without "warm-body" personal contacts it is difficult to build long-term trust based relationships.

Many organizations who thought outsourcing IT was the key to success miserably failed. Whose who did not failed lost competitive advantage, experienced the demoralizing effect of  vendor lockdown and QOC hokey pokey which just simply made no sense.

That means that client of service providers will be implicitly pushed to lowest common denominator and cannot utilize local expertise, even if such exists. They face "helpdesk level" people and instead of benefitting from specialized provider are often proposed wrong solutions to misdiagnosed problems.  my experience with WEB providers suggests that trivial problems like an error in DNS record or wrong permissions can became real production issues.

Service provide can evole software independently of wishes of some of the customers. That means that customers who are not satified with the direction taken need iether to adapt or abandon the service.

The quality of backups is another problem that is more complex then it looks in advertising materials. Often there is some kind of "vendor lock" that stimulates users to keep everything on the remote server, because remote application they are using does not provide a friendly way to backup data locally.  In such cases situations where a lot of data were lost due to the crash are not that uncommon.  Here is one relevant blog entry (The Problems and Challenges with Software as a Service  Gear Diary):

Gear Diary is a WordPress enabled site, so many team members use the online WYSIWYG editor to create and edit content. It saves drafts, allows you to upload (and even watermark) graphics/pictures, and is, for all intent and purposes, an online word processor, much like Google Docs.  When the site went belly up, most of the content headed south the border as well.  Most team members had not saved a local copy of their work…which got me thinking…

One of the biggest and hottest trends I’ve been hearing a lot about lately is software as a service, a la Google Docs, Office Live, etc. If you take the Gear Diary site issue as a point of reference, and apply software as a service (which is basically what WordPress is acting as), you get an interesting and fairly destructive situation.  WordPress doesn’t offer any kind of method of saving its documents locally, or in a format that can be read (or edited) by any other local application. Despite the fact that WordPress creates HTML documents, all data stays on the server.

If you bump into a server issue, i.e. you go down, your data gets lost. It happened to Gear Diary. It can happen to any user that uses a software as a service app. what bothers me more, is that unless there’s a specific viewer or offline editing tool for the document type, the data is useless.  Further, if the app doesn’t allow you to save data locally, an off line viewer isn’t going to do much good anyway.

Many users here (those that wor of WordPress and saving it as a text or HTML file. That at least gets the data out and saved to your local hard drive.  However, it doesn’t address disaster recovery on the client side (which was one of the big draws, aside from cost savings and the lack of deployment problems…all you need in most cases is a compatible browser…).

Discounting the value of IQ inherent in a local datacenter staff

Much of the value of software is limited to the level of skills of IT personnel. For example I am sure that Carr can be classified as an entry level user of MS Word. But he probably does not even suspect the level of functionality that MS Word has and will never uses 90% of this functionality effectively downgrading MS Word to entry level product. Only software specialist can delve into the complexity of modern software packages and survive  ;-). That means that just having talented IT personnel on the floor is in itself a comparative advantage.  End users will never be able to learn reasonably advanced features of complex packages like MS Word (which need, at least, good understanding of VBA, but probably much more).  Therefore they need help of IT specialists and good external specialist are very expensive and not the easy to find person who simultaneously knows the business in which application is running.   In the current complex world  with few sources of lasting comparatives advantages, "peopleware" is one of the few that cannot be replicated easily. Carr notion that users can implement IT system is simply ridiculous. Typical user does not abilities to master MS Word or Excel (and to be frank, few professional can fully  master them too ;-).  The level of complexity of modern IT software products and IT systems in general is just staggering: this are the most complex system created by men. And "experts" like Carr propose to delegate understanding and architecture of such system to users or (intermediated by their own revenue stream) providers. When should I, as an IT manager make targeted strategic investments, as opposed to overall upgrades? Should I put on my rose-colored academic glasses and move servers to Solaris or should I follow the trend and try to convert my server farms to Linux and virtualizes all instances of Windows on VMware. Or I would be better off using blades ? Should I outsource my development to India or some other lower cost country or critical mass of developers is critical to the company success ? IT  Primitive solutions of complex issues that face people in this industry could be tough to swallow.

For example if I supported particular complex application in a particular company and then switch sides and move from my company to some application provider I essentially can dictate the conditions (and believe me that will not be favorable conditions, why they should be ?) as nobody can match my level of understanding of the particular area. And such milking of companies due to lack of local IQ is not that uncommon.  Actually the best situation for me as a sale rep is to face as a company representative some bizarre over-promoted secretary who due to her "non-technical virtues" was promoted to the head of IT security or, better, to the level of vice-president of IT.  As she is afraid to demonstrate her own incompetence and does not trust her own people she can serve as a Trojan house for pretty lucrative deals. Usually there are very few talented and dedicated people in IT which can be the architects of good systems. But IT brass usually cannot distinguish them from regular ladder climbers and sociopaths. So in a way the cost of employing a lot of IT staff is the cost connected with the inherent inability by higher management to separate the wheat from the shaft in modern bureaucratic organizations.

Superior architecture and software code are difficult to duplicate and they does provide huge strategic advantage. Another often neglected side of strategic advantage of IT is the  protection provided by local staff from greedy vendors and service providers. For example, protection form SOX excesses and greedy accounting firms is definitely an advantage, strategic or not: without it  accounting firms can milk weakened by loss of talent enterprises for many millions additional dollars. And all in the name of better security  :-).  Protection from snake oil salesman who promise miracle diets for the IT datacenter (including Nicholas Carr ;-) is yet another advantage...

Nearly all large enterprises run Office applications on Intel-based PC under Windows.  But that does not mean that some companies cannot find an advantage to use Mac or Linux. Yes, hardware is now so cheap that just about everyone can afford it. But to afford does not means to use efficiently. While your competitors have access to the same online information you do ability to absorb and creatively use this information is quite different.  It's not clear that ubiquitous, standardized technology can help any particular firm stand out. But such benefits are not automatic.  If they do not have capable people (or worse have a PHB as the head of "PC standardization" project)  they will definitely overpay for it and underutilize it (incompetence of IT personnel definitely double IBM or Dell revenue stream) .  But as we move to the datacenter level and especially datacenter architecture, the level of standardization is much less and issues are more complex. For example cases when company can benefit from adopting blades are different from cases when company can adopt VMware and without proper level of knowledge no vendor will spell the difference for you: each will push its own product. So while standardization in the small is evident, standardization at large is not.  

The fact that Lego consist of the same peaces does not limit the amount of things you can create from it. The same it true with IT.  Carr's  vision is that we are about to shift to massive, centralized utility computing plants in which individuals will not need to carry laptops. simplified terminals are sufficient.  The current trend is just an opposite: laptop are becoming more and more powerful and versatile.

Also the current level of complexity if IT system presuppose need to be specialized and trained in the field for years if not decades.

And if we take an opposite extreme "complete dilbertalization" of IT is  a source of huge comparative disadvantage as many outsourcing efforts have shown to surprised brass.  Bad apples in IT can really provide huge damage to the enterprise as several failures of SAP/R3 implementations (some of which resulted in bankruptcies of firesale of companies) had shown.

 In the worst case scenario the company loses critical mass of knowledge and became a hostage of unscrupulous outsourcer and greedy vendors. Also the ability to pioneer some new higher level integration approaches or improve the logistics infrastructure can be a source of comparative advantage.
 

Idealization of "in the cloud" computing

Fiber networks increases bandwidth substantially and revitalized distributed computing. But there is a big difference whether you distribute over LAN or WAN.  WANT is much tougher case. With all the tremendous progress of Internet available bandwidth does not increase as quickly as computing power. Nowhere close, and it never has. If anything, due to increased scrutiny and "shaping" by ISPs (they are not a charity and need to be profitable)  bandwidth "per user" might recently start decreasing as such resource hogs as YouTube and video programming distribution services (movies on demand) are becoming more and more prominent. Ability of P2P services to clog the Net in the most inopportune moment  now is well established and is  a real curse for  university networks. 

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

That means that unless you pay for the quality of service "in the cloud" computing model stands on a very shaky ground. Reliable 24x7 bandwidth is not free and contrary to Carr's views substantial amount of traffic with the remote datacenter cannot be achieved via public channels. But buying it be extremely expensive: for mid-side companies it is usually as expensive as keeping everything in house. That makes problematic "in the cloud" approach to any service where disruptions or low bandwidth in certain times of the day can lead to substantial monetary losses.  Businesses perceive "in the cloud" solutions as brittle, inflexible, and, due to concentration of profit margins, hostile to their business needs. So much for the improvement of the business model in comparison with traditional datacenters. 

Actually services itself are not that cheap so economy in switching to service provider for large organizations might  simply be not sufficient. As for May, 2008 the number of providers are in dozens with only few established vendors (WEB site providers, Amazon Web Services, Goggle App Engine and soon Microsoft Live Mesh ). Spectrum of available services is very limited and currently is far from ability to replace real datacenter. Web site providers is the most viable and proved to be  the cost savers for small web site owners. Advantages are quickly diminishing with the complexity of website, though. Among Amazon Web services only S3 storage currently can be called a successful service. Microsoft Live Mesh pragmatically is a data synchronization service. It presuppose existence of local computers and initially provides mostly syncing files between multiple instances of local computers belonging to the same user.

Most existing services are not cheap and became expensive as you scale them up.  For example  Joyent provides hosting of WEB applications on virtual machines with minimal cost of $45 per month which OK price for test machine. But if you want just one "real" CPU (which is desirable for production instance) costs increase to $250 a month.  In the latter case one year costs is equal of the cost of pretty nice locally installed Dell 1U server with outsourced 3 year vendor-based 24x7 hardware support. If Web application works strictly for the company and not for external customers this is a wash. Even for small businesses Verizon Broadband provides $40 a month service (includes 250MB of free online backup). T1 line is approximately $250-$300 a month.  So web server configured as VMware virtual appliance is not that different from hosted service and might be also a wash.  

All-in-all it is not very clear what market share "in the cloud" services deserve and whether they are modern variant of Japanese 5th generation computing or not.  Currently, there are a lot more places to run software than there used to be, thanks to the proliferation of powerful laptops, mobile phones like iPhone and other devices.

From social standpoint Carr also looks very unconvincing. Service providers mind their own interests first. also large service provider is the same bureaucratic monster as a typical datacenter that share with the latter all the spectrum of dilbertalization problems. Large customers experience "vendor lock-in" working with service providers as it involves significant effort to adapt ton both sides. So walk-out is a less viable option that one can assume on pure theoretical backgrounds.

Bandwidth communism

Promoting remote outsourced datacenter, Carr forget to calculate the cost of bandwidth.  Yes fiber-optic changed landscape making remote services more viable. But the devil is in detail. For example file sharing for a large company over WAN is still bad idea as bandwidth is insufficient and costly. Also any enterprise making bet of 24x7 availability of public bandwidth for vital corporate services looks slightly suicidal.  All-in-all his "grand utility computing" vision ("bandwidth communism") is a trap in which Nicholas Carr falls due to the lack of understanding of networking. 

The situation is different for different services. for example for mail outsoucers like Postini, this problem is not relevant due to the properties of the SMTP protocol: they can communicate via regaular Internet. The same is true to DNS services providers, webmail and instant messaging. CRM is also pretty close. But for ERP, file sharing and WAN based backup it is  different.

There is free lunch and providing high speed networking services over WAN is a very challenging engineering task to say the least. The cost of bandwidth also puts natural limits on service providers growth as local networks are usually much cheaper and faster. Achieving 1Gbit/s speed on LAN is extremely cheap while it is quite expensive over WAN. There are also other limiting factors:  

1. Limited bandwidth at the point of connection of provider to the Internet. Every provider is connected to the Internet via a pipe and that pipe is only so big. For example OC-1 and OC-3 have their upper limit of 51.84Mbit/s and 155.2 Mibit/s  correspondingly.  Funny the upper speed of OC-3 (which is pretty expensive) is only slightly higher that 100Mbit/s which become the lowest common denominator for LANs.  Large service providers typically use OC-46 with speed up to 2488.32 Mbit/s which is similar to the speed of gigabit Ethernet.  10 Gigabit Ethernet is the fastest commonly available network standard for LANs.  It is still emerging technology with only 1 million ports shipped in 2007.  It might be eventually used in modified form for WANs too.   Anyway as bandwidth is limited and shared between multiple customers the spike in activity of one customer might negatively affect others.  Networking problems at the provider level affect all customers and recovery period might lead to spikes of activity.
    
2. Local vs remote storage of data. Recent enterprise level hardrives (Cheetah 15K)  have speed up to 164 MB/sec (Megabytes, not megabits).  From the speed and cost point of view the ability to keep data/programs local is a big technological advantage.  It might be that the only viable role for remote providers is syncronization with local data ;-). Example of this approach is Microsoft's Live Mesh
    
3. Interdependence of customers on the Net level. This is jut another incarnation of "tragedy of commons" problem. Bandwidth hogs like game, P2P, music and video enthusiasts experience do not care a dime about your SLA and can easily put a company that uses public links into disadvantage any time of the day something new and exiting HD movie was released. Also  providers are not willing to sacrifice their revenue to accommodate "free-riders.": as soon as usage of bandwidth cuts into profits it is punishable and no amount of rhetoric about "Internet freedom" can change that. enterprise customer can suffer from the effort of providers to manage free-riding. That means the corporation which moved services to the cloud competes with various bandwidth hogs who do not want to scarifies any ground and ready to go quite far to satisfy their real or perceived needs.  My university experience suggest that corporate users will definitely experience Internet clogging in the form of sluggish download speeds, slow response times and frustration with data-heavy services that become much less useful and/or enjoyable. See for example Time Warner Cable Vs. The Horde.
4. Competition for the resource at ISP level.  For any successful service providing all the necessary bandwidth is costly and cuts into margins.  Recently Amazon faced the situation when bandwidth required for its Elastic Compute Cloud (EC2) is higher then by all of Amazon.com’s global websites combined. You can read between lines how that affect profitability:

Adoption of Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3) continues to grow. As an indicator of adoption, bandwidth utilized by these services in fourth quarter 2007 was even greater than bandwidth utilized in the same period by all of Amazon.com’s global websites combined.

Web services providers which offer customers unlimited bandwidth are banking on the fact that the majority of their customers will not use much of their bandwidth. This is essentially a marketing trick.  As soon as you exceed a fraction of what is promised they may well kick you out.  People who tried to implement software , mp3 or video sharing services on low cost ISP accounts realized that very soon. See for example references that I collected under "Unlimited bandwidth myth".  Web neutrality does not mean the tragedy of commons is not applicable.   As Bernardo A. Huberman, Rajan M. Lukose noted:

Because the Internet is a public good and its numerous users are not charged in proportion to their use, it appears rational for individuals to consume bandwidth greedily while thinking that their actions have little effect on the overall performance of the Internet. Because every individual can reason this way, the whole Internet's performance can degrade considerably, which makes everyone worse off. An analysis of the congestions created by such dilemmas predicts that they are intermittent in nature with definite statistical properties leading to short-lived spikes in congestion. Internet latencies were measured over a wide range of conditions and locations and were found to confirm these predictions, thus providing a possible microscopic mechanism for the observed intermittent congestions of the Internet.

So a company which will try to implement Web based streaming of say corporate video conference via cloud is up to nasty surprises unless it paid "arm and leg" for dedicated lines  to its headquarters and other major locations (which make the whole idea much less attractive in comparison with the local datacenter). The ability to stream video of any considerable quality in real-time between two (or more!) arbitrary points in the network is not really something that can be easily done over the current Internet.

The main point to make is that a reliable WAN network connectivity cost a lot of money  is difficult to achieve. This problem is unavoidable if your major components are "in the cloud" (in WAN). Also in the "free internet" enterprises are starting to compete for bandwidth with streaming media (films over Internet). The latter proved to be a huge resource hog and quality of a typical Internet connection now fluctuates widely during the day. That means that in order to achieve respectable quality of service for bandwidth intensive applications enterprises need to buy dedicated WAN connections. That is a very expensive habit to say the least. I once was involved in moving SAP/R3 instance from USA to Europe (just from one continent to another) and to achieve reasonable latency for requests coming from the USA was not that easy and definitely not cheap.  The cost of  high bandwidth transatlantic connection was the major part of additional costs and had eaten almost all savings from the centralization. The same is true about any WAN connection: reliable high-bandwidth WAN connections are expensive.  Moreover the reliability needs to be carefully monitored as anybody who was responsible for company WAN can attest.

Sweeping under the carpet mainframe-style problems with external providers of IT services

Fiber optic lines made "in the cloud" computing possible. But that does not mean that this services are a "universal bottle opener". Any new service along with solving old problems creates new. First of all in case you use external providers loyalty of staff disappears and for any complex service you face "all or nothing situation": If service works everything is wonderful, but if it does not troubleshooting is extremely difficult and fixing the problem is almost impossible even  if you understands what is happening -- infrastructure belongs to other organization. Anybody who used Web hosting (the most successful example of such services) can attest that this is a wonderful service as long as it works. But if you have a problem you have a big problem: local staff has no loyalty to the particular organization and competition does not work as another provider can be as bad or even worse; so switching brings you nothing but additional headache and losses.  Even elementary problem can take several months to be resolved and I am not joking I experience it myself.  Oversubscription which leads to highly loaded servers and insufficient network bandwidth is another common problem.  There are also problems related to the "race to the bottom" in such services: the main differentiator becomes price and to attract new customers Web providers often make claims that are untrue (unlimited bandwidth is one typical example). As a result naive customers who believe in such claims are burned. 

What he is proposing is a remote centralized and outsourced datacenter that provides services via "cloud" using high speed fiber links. Some services, for example with limited I/O (both ways) are more suitable for this utility deployment mode. Along with email and Web hosting, payroll is one such service that proved to beneficial to decentralize. But most I/O intensive enterprise processing is more efficiently done on a local level. Also issues of quality, loyalty, knowledge of the business are not automatically solved by utility model.  Contrary to simplistic description and assumptions typical for writers like Carr utility model inherently suffers from mainframe warts and thus present huge challenges in this area. For example anybody who used low cost Web-hosting provider can attest that interests of providers run contrary to the interests of advanced uses and as such often stifle technology adoption to those technologies that are supported by a particular provider. Also provision of the adequate bandwidth (and responses times) can be a shaky area too especially in rush period like 9-12 EST. Security is another challenge: break-in into a large provider affects all its customers. Claiming that Web provider are a total solution for any organization is like saying that just because the Big 4 accounting firms exist with the army of  accountants, tax specialists and so on, organizations can do away with internal accountants altogether.  The hybrid platforms, such as Saleforce.com's application hosting capabilities is also an interesting proposition. Still there is a lot of disappointments with this model as exemplified with the following characterization of "cloud-based" services and outsourcing: 'We thought that we are like a farmer shooing a fat  rabbit, but it turned out that we are a slow moving target and the rabbit is shooing us." This quote suggests that  providers of such services as well as outsourcers in the future might have difficulties to shake money loose,  as IBM  and Sun already discovered.

Issues of security and trust.

I think Carr's understanding of data security is "cloudy" at best. Most companies want control over their data. Outside data storage could be also compromised by disgruntled employee or random hacker. Moreover the company that uses the service provider has no control on social atmosphere at the service providers (who probably will feel tremendous pressures to cut costs and have atmosphere of a sweet shop -- much like outsourcers). And nobody wants to give their intellectual property to somebody else.

Stealing electricity doesn't do the kind of damage that stealing  payroll data with SSNs does.

In a way Nicholas Carr lives in the dream world of unpatented, unprotected information.  Information that does not matter. The real world is different.

As I mentioned before  breach into a ISP affects all its customers and PR losses can be tremendous, up to and including going out of business.   Mass intrusions happened with several WEB services providers. Compete disregard of security happens in WEB providers world more often then Carr would like to assume and actually can be attributed to misdirected cost cutting efforts.  Putting lip stick on the pig after the fact with "Hacker Proof" certificates etc does not solve the problem: infrastructure is complex, user population is diverse and overall security is as good as the  weakest link. 

IT managers also should think twice before assuming that consultants and systems integrators are always acting as objective advisers when recommending best-of-breed products and technologies, experts say. Many companies  [clients] still view consultants as neutral and bringing the best solution, but that's often not the case. They don't view it as a problem until a large project blows up in their face.

Recently, however, the ties linking vendors and service providers have grown tighter. Consultants and vendors have been investing in one another and launching joint companies.  Cisco Systems Inc., for example, invested $1 billion in KPMG Consulting LLC when it launched in January as a separate company from KPMG LLP.

As I mentioned above it's a myth to think you can bring in a systems integrator at the beginning and delegate responsibility for architecture to the integrator.  Just as the companies may be forming alliances with specific vendors, they also have varying levels of experience and skills in different technologies. And having final "résumé control"—in other words, being able to handpick the specialists from consultancies and integrators that will work on a project might be a limited remedy. The last thing you want is a consultant to come in and not be familiar with the technology you're implementing and use the company as a testbed. Among questions to ask

• What types of vendor alliances are in place?
• Does the consultant personally or his firm have an equity stake in the vendors?
• What commission or compensation does the consultant or his firm receive for selling vendors' products?
• How many consultants of the firm the consultant represents were trained on a vendor product through a given alliance?
• Does the consultant firm have any exclusive relationship with vendors? How many alliances does the consultant firm have in a specific category and how do they differ?
• What measures should be taken to limit influence of alliances on the strategy and technology recommendations

Competing technologies

While "in the cloud" services providers are good for certain tasks (with web providers as the most prominent example; mail filtering (Positni) is another one), they are not so good for complex enterprise applications (for example SAP/R3) for Office applications and many other things. Web services via Internet will become more important but they do not represent the dominant path and do not exclude progress (probably more fast) in other areas that keep local datacenter strong and healthy and IT staff busy. There are two competing technology that make Carr predictions in 2008 even more naive then they looked in 2003. 

•  The second is switch to running applications on virtualized appliances instead of real servers (or "datacenter of virtual machines). 
• Higher level standardization as exemplified by "datacenter in a box": the technology for commodization of datacenters completely different from the "in the cloud" providers. There are a lot of common services in a typical datacenter that can be "pre-deployed" by vendors like Sun and Microsoft in mega-appliance fashion. 

Conclusions

Carr initial article and subsequent books are highly problematic, suffering from several fundamental flows that we discussed above. The central idea of the article "IT does not matter" is simply a fallacy.  

"In the cloud" service providers is just one new development in IT space and it faces serious difficulties ahead.  Potentially more important are virtualization and standardization of datacenter infrastructure (right now represented by "datacenter in the box").

Hagel and Brown in their book The Only Sustainable Edge suggest that competitive advantage is the relentless building of distinctive capability on the edges (frontiers), both within an organization, and the networks it operates in. That means that organizations have to choose what they are going to excel in, as it is not possible for any single organization to excel in everything. Something needs to be scarifies to stay in focus. So while some parts of IT can be outsourced or done by service provider, other are not and those parts will always require IT staff organized in some form of  datacenter (datacenter=hardware+software+IT professionals). 

The key problem is IT is its dilbertalization and outsourcing everything to service provider just move this problem to service provider level. Moving services to remote "in the cloud" service providers (or even local outsourcers) does not solve this problem as the latter also are not immune from the disease.

There are some clear analogy between old "mainframe-based' datacenter and new service provider approach. And user hostility to IBM in the era of mainframes was a symptom of underling problems. Something similar might happen with remote service providers attractiveness. In addition one of the major problems in utilizing remote WAN-based service providers is bandwidth problem and the cost of solving this problem depends on the type of service. That means that some services like email and Web hosting are inherently more suitable to outsourcing to providers in the cloud then others (application streaming, etc).  For high bandwidth services there is no free lunch -- you need to pay for bandwidth in order to get respectable reliability.  

The entire discussion about IT and its impact on competitive advantage boils down to availability of smart, highly trained professionals who are capable of broad strategic thinking and able to connect seemingly disparate ideas, systems and protocols. Without them there can be no competitive advantage. But you cannot employ just smart people. So some kind of IT organization is necessary although it might be quite different then the current structure.

Notes: Those pages are written by people for whom English is not a native language. Some amount of grammar and spelling errors should be expected. This is a Spartan WHYFF (We Help You For Free) site. It cannot replace the best teachers and the best books. The site contain some obsolete pages as it develops like a living tree... Some links on older pages are broken. Please try to use Google, Open directory, etc. to find a replacement link (see HOWTO search the WEB for details). We would appreciate if you can mail us a correct link. Search Amazon by keywords:     Open directory Research Index

Old News ;-)

[May 11, 2008] Q&A Author Nicholas Carr on the Terrifying Future of Computing

Carr should probably think about career of humorist...

12.20.07 | Wired

Wired: What's left for PCs?

Carr: They're turning into network terminals.

Wired: Just like Sun Microsystems' old mantra, "The network is the computer"?

Carr: It's no coincidence that Google CEO Eric Schmidt cut his teeth there. Google is fulfilling the destiny that Sun sketched out

The IT department is dead, author argues

 NetworkWorld.com

Carr is best known for a provocative Harvard Business Review article entitled "Does IT Matter?"

... ... ...

With his new book, Carr is likely to engender even more wrath among CIOs and other IT pros.

"In the long run, the IT department is unlikely to survive, at least not in its familiar form," Carr writes. "It will have little left to do once the bulk of business computing shifts out of private data centers and into the cloud. Business units and even individual employees will be able to control the processing of information directly, without the need for legions of technical people."

Carr's rationale is that utility computing companies will replace corporate IT departments much as electric utilities replaced company-run power plants in the early 1900s.

Carr explains that factory owners originally operated their own power plants. But as electric utilities became more reliable and offered better economies of scale, companies stopped running their own electric generators and instead outsourced that critical function to electric utilities.

Carr predicts that the same shift will happen with utility computing. He admits that utility computing companies need to make improvements in security, reliability and efficiency. But he argues that the Internet, combined with computer hardware and software that has become commoditized, will enable the utility computing model to replace today’s client/server model.

"It has always been understood that, in theory, computing power, like electric power, could be provided over a grid from large-scale utilities — and that such centralized dynamos would be able to operate much more efficiently and flexibly than scattered, private data centers," Carr writes.

Carr cites several drivers for the move to utility computing. One is that computers, storage systems, networking gear and most widely used applications have become commodities.

He says even IT professionals are indistinguishable from one company to the next. "Most perform routine maintenance chores — exactly the same tasks that their counterparts in other companies carry out," he says.

Carr points out that most data centers have excess capacity, with utilization ranging from 25% to 50%. Another driver to utility computing is the huge amount of electricity consumed by data centers, which can use 100 times more energy than other commercial office buildings.

"The replication of tens of thousands of independent data centers, all using similar hardware, running similar software, and employing similar kinds of workers, has imposed severe economic penalties on the economy," he writes. "It has led to the overbuilding of IT assets in every sector of the economy, dampening the productivity gains that can spring from computer automation."

Carr embraces Google as the leader in utility computing. He says Google runs the largest and most sophisticated data centers on the planet, and is using them to provide services such as Google Apps that compete directly with traditional client/server software from vendors such as Microsoft.

"If companies can rely on central stations like Google's to fulfill all or most of their computing requirements, they'll be able to slash the money they spend on their own hardware and software — and all the dollars saved are ones that would have gone into the coffers of Microsoft and the other tech giants," Carr says.

Other IT companies that Carr highlights in the book for their innovative approaches to utility computing are: Salesforce.com, which provides CRM software as a service; Amazon, which offers utility computing services called Simple Storage Solution (S3) and Elastic Compute Cloud (EC2) with its excess capacity; Savvis, which is a leader in automating the deployment of IT; and 3Tera, which sells a software program called AppLogic that automates the creation and management of complex corporate systems.

... ... ...

Carr offers a grimmer future for IT professionals. He envisions a utility computing era where "managing an entire corporate computing operation would require just one person sitting at a PC and issuing simple commands over the Internet to a distant utility."

He not only refers to the demise of the PC, which he says will be a museum piece in 20 years, but to the demise of the software programmer, whose time has come to an end.

Carr gives several examples of successful Internet companies including YouTube, Craigslist, Skype and Plenty of Fish that run their operations with minimal IT professionals. YouTube had just 60 employees when it was bought by Google in 2006 for $1.65 billion. Craigslist has a staff of 22 to run a Web site with billions of pages of content. Internet telephony vendor Skype supports 53 million customers with only 200 employees. Meanwhile, Internet dating site Plenty of Fish is a one-man shop.

"Given the economic advantages of online firms — advantages that will grow as the maturation of utility computing drives the costs of data processing and communication even lower —traditional firms may have no choice but to refashion their own businesses along similar lines, firing many millions of employees in the process," Carr says.

IT professionals aren't the only ones to suffer demise in Carr's eyes. He saves his most dire predictions for the fate of journalists.

Comments

Blending a few useful points into inflammatory utopia/dystopia

Submitted by JimB (not verified) on Mon, 01/07/2008 - 4:32pm.

 

Carr's vision is either utopian or dystopian, depending on how you look at it, but either way, it mixes a few likely trends with lots of naive wishful thinking, unsound logic, and sophomoric shock value.

The likely trends include greater commoditiza