May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Mental Overload

News Main page Recommended Links Email Overload Information Overload Etc

College those days can be much less fun than it used to be. This is mainly due to mental overload. Life became complicated, full of annoying, time-consuming distractions like email and cell phones. Students are spinning too many plates. Teachers are often cruel (or simply detached from teaching and too engaged in research) and just push too much information without taking into account the ability to absorb it by students.   Trying to sip this amount of  information from the fire hose is a difficult and challenging task :-).

Students ability to think clearly, and learn effectively, are adversely affected by their permanent information overload. Often, students feel the need to study harder when faced with  learning difficulties at the end of semester. It might actually have adverse effect and increase learning difficulties. This is a time to use mentors to full extent or try to mentor yourself as this can help to break the "saturation with information" that many experience.

The key fact is that the ability to learn and retain information can be compromised  at the end of semester by   mental overload which seems to be a prerequisite for passing the courses being taken. It was noticed long ago that the ability of overworked students to express themselves clearly in both written and spoken English is somewhat  impaired. Even in early 1980s there were cases when a telephone conversation with a Physics grad student in a Californian University had to be abandoned as the person in question could not communicate fluently. It is also noticeable that overworked students cannot sequence or recall simple facts like names, addresses, and telephone numbers with accuracy. Making his behavior close to person who is over 60 and whose memory start to deteriorate. This memo from Wash college might help:

In today’s world, mental overload is a fact of life. Fortunately, by applying some simple techniques from the computer world, you can avoid some of the costly consequences of a too full brain!


A too-full computer can:
· give you error messages
· run slower
· take longer to process information
· crash

A too-full brain can cause you to:
· make mistakes
· forget to do something
· let things slip through the cracks
· become sluggish
· loose creativity
· become unproductive
· procrastinate
· become indecisive
· get stressed out
· experience a total mental break down

Does excessive multi-tasking like happens to college students make us stupid?  The answer is tentative yes. Whenever we drop one task to perform another, we face "resumption costs" -- the time and energy it takes to orient ourselves when we return to the original task. It's true that interweaving two lengthy tasks can take less total time than performing the tasks separately. But there's a price:

Top updates

Softpanorama Switchboard
Google Search


Old News ;-)

Disadvantages of hard work.


If you have arrived at this page, you may well be asking "why?". In fact, "Why does someone want to write about the disadvantages of hard work, when we are all told incessantly how beneficial it is?"

I conducted an experiment. I entered the terms

"hard work" disadvantages
into (try it for yourself) and found over 21,000 hits.

I then rephrased my question and entered

"disadvantages of hard work"

into Google and got precisely zero hits. No-one on the entire web, it would seem, has written this phrase. Why not? Clearly it is "culturally verboten".

I was motivated to ask this question of the search engine, as, after many years of teaching in the University sector, I have met a significant number of people who I consider have been significantly damaged as individuals by subscribing to the "hard work is necessary" hypothesis.

So let us put the record straight here, and spell out some of the advantages of working just sufficiently to satisfy the various criteria of emotional and spiritual need, the demands of the job, the necessity of keeping body supplied with food clothing and shelter, and the social requirements of interacting with others.

Case histories

Among the people I have observed who subscribe to the "hard work is good" hypothesis are several University academics whose ability to think clearly, and administer effectively, are adversely affected by their permanent state of tiredness. Often, these folk feel the need to intervene when it is inappropriate. People like this generally are unhappy with the status quo, and feel that any change or intervention is bound to be for the better.

Among the students I have met, there are significant numbers whose ability to learn and retain information, let alone process it effectively, have been compromised by years of being forced to acquire unnecessary skills and learn unnecessary facts; I maintain this has actually physically damaged their brains, and that an enlightened court of law would award them damages against their educational institutions. Often, this kind of mental overload seems to be a prerequisite for admission to the course being taken.

At Berkeley (Uni Calif) in the 1960s I noticed that the ability of overworked students to express themselves clearly in spoken English was severely impaired. This was confirmed in the early 1980s when a telephone conversation with a Physics grad student in a Californian University had to be abandoned as the person in question could not communicate fluently. It is also noticeable that overworked students cannot sequence or recall simple facts like names, addresses, and telephone numbers with accuracy. Neither can they spell accurately or proof read what they have written. They also try to "rote learn" ineffectually, as they cannot repeat accurately what they have just seen, read, or heard.

Among the medics I have met, there are a significant number, likewise, who "do what they do, regardless" - thus if you go to a physician you get dosed up with drugs; to a surgeon, you get cut open; in fact, each specialist tries to fit your ailment into his own field of competence. This activity is unrelated to the needs of the case.

Among the politicians I have known, the greatest damage to society is caused by those people who regard themselves as the greatest "movers and shakers". Moreover, there is a class of commentator that regards the activity of "moving and shaking" to be intrinsically beneficial, without regard to the end effects.

Choice in the marketplace

Much of the excessive pressure to work harder, to produce more for less, and to drive staff harder is justified by the mantra "choice for the consumer". It is a psychological observation that given excessive choice, the majority of people have extreme difficulty in exercising it and arriving at a rational purchasing decision. Supermarkets should note this. It is far easier to choose from a limited range of goods than from acres of produce spread out among miles of shelving.

The same observation applies to the motivation of students on modular degree courses. Excessive choice leads to a shallow educational experience. It is also somewhat demotivating for the student. I am often asked to delimit my course materials so that the student knows what is not to be covered in the exam tests.

Feedback regulation

There is a report at that the brain (specifically, the left pre-frontal cortex) undergoes structural changes on long exposure (many years) to stress such as overwork. This makes the brain's owner more disposed to see the negative side of events, rather than the positive. One can see a certain amount of self-regulation here, for positive disposition in a person predisposes him/her to work harder. We can also identify the scientific reasons for negative reactions to excessive perceived stress and the onset of depressive illness caused directly by being subjected to a heavy workload.

Optimum range of workload

It is apparent that most people have a range of demand that they can tolerate, or even feel comfortably happy with. Below the lower limit they feel discontented and underutilised, and above the upper limit they seek to shed work and may even become bad-tempered. An attribute of people who rise to high positions within their organisations is that they are very tolerant of a wide range of work demands; they find occupations for themselves if lightly loaded, and they are benign under pressure, even if it is unreasonable. For this reason, they are candidates for promotion.

Shared views

The tenor of this argument is shared by Prince Charles in a report in the Guardian newspaper on Tuesday 13th Sept 2005.

[ Mar 09, 2007] A very good memo on mental overload from Washington College

Trying to sip information from the fire hose is a difficult and challenging task :-). This memo might help.

In today’s world, mental overload is a fact of life. Fortunately, by applying some simple techniques from the computer world, you can avoid some of the costly consequences of a too full brain!


A too-full computer can:
· give you error messages
· run slower
· take longer to process information
· crash

A too-full brain can cause you to:
· make mistakes
· forget to do something
· let things slip through the cracks
· become sluggish
· loose creativity
· become unproductive
· procrastinate
· become indecisive
· get stressed out
· experience a total mental break down

[ Mar 09, 2007] Mental overload by Katherine Lewis

Does excessive multi-tasking like happens to college students make us stupid?  The answer is tentative yes:
Multi-tasking may be too much for the brain to handle
Friday, March 09, 2007

We feel so efficient, listening to a teleconference while sorting e-mail and eating lunch at the same time. But experts warn that instead of completing three tasks in the space of one, we're really spending more time to achieve mediocre results.

"Research that's looked at multi-tasking shows that you can't do it well -- no one can," said Kristin Byron, assistant professor of management at Syracuse University. "You're fighting the way your brain works."

The brain acts on just one task at a time. What we perceive as simultaneous multi-tasking is really rapid switching back and forth to keep different tasks going -- even if one is as simple as deciding to lift the sandwich for another bite.

It's like the classic vaudeville act of spinning plates. Your brain can set a task in motion, then another, and then another, before returning to pick up the first task, explained David Strayer, a psychology professor at the University of Utah in Salt Lake City.

"If the demands of any given task aren't too taxing, you can get two, three, four plates going up, but at some point you're going to reach a threshold when they're going to crash."

You may avoid driving while talking on a cell phone because of the physical challenge of holding both phone and steering wheel. But Strayer's research shows hands-free cell phone use is just as dangerous while driving. The risk comes in toggling between the two mental demands.

Moreover, subjects in a recent study scored significantly lower on IQ tests they took while driving. "When your attention is taken away from a task, you are not going to perform it as smartly," Strayer said.

So does multi-tasking make us stupid?

It's not an outlandish conclusion. A 2005 study sponsored by Hewlett-Packard found the average worker lost 10 IQ points when interrupted by ringing telephones and incoming e-mails -- about equal to the cost of missing an entire night of sleep.

"Interruptions are time-consuming, and they are dangerous in the sense that they can lead to errors," said David E. Meyer, a psychology professor at the University of Michigan in Ann Arbor. "You are trying to feed information through various kinds of processing channels in the brain which have limited capacity and are really only available for one thing at a time."

Whenever we drop one task to perform another, we face "resumption costs" -- the time and energy it takes to orient ourselves when we return to the original task. It's true that interweaving two lengthy tasks can take less total time than performing the tasks separately. But there's a price.

"A lot of tasks we have to do, there are little moments of gaps which you can steal for another task," said Hal Pashler, psychology professor at the University of California in San Diego. "The interesting hidden cost ... is that (we) may be strikingly unable to recollect what happened."

That's because the free moments in each task -- such as waiting for a partner to respond in a conversation -- appear to be used to store or consolidate memories. If we talk on the phone while checking e-mail, it's at the expense of downtime our brains need.

"The conversation plus the e-mail may take less of your life, but the cost is that tomorrow you may not know exactly what you said," Pashler said.

Thus, if you try to take in new material or facts while multi-tasking, you'll have a tougher time learning, said Russell A. Poldrack, psychology professor at the University of California in Los Angeles.

Does all this mean we should never check our Blackberries while waiting in line at the grocery store? Or even sip a cup of coffee while listening to a conference speaker? After all, multi-tasking is woven into the fabric of modern life. More than 85 percent of people multi-task, and 67 percent believe they do it well, according to a survey by Apex Performance, a Charlotte, N.C., training firm.

Fortunately, the experts give us some slack. "You can't say in every situation it would be better to always focus on one task," Poldrack said.

If you're a stock trader who has to respond quickly to a lot of information, it makes sense to monitor multiple televisions and computer screens at once, he said. It may not matter that the next day you're hazy about which news anchor said what.

Certain physical actions, like walking or eating, are so hard-wired that they don't tax our brains much. There's certainly no harm in combining simple, low-stakes tasks, like folding laundry and watching television. And if background music energizes you to finish your work, that may outweigh the cost of your mind shifting between listening and crafting a report, Poldrack said.

Similarly, talking to an adult passenger doesn't hurt your driving the way talking on a cell phone does, Strayer has found. That's because the person in your car is attuned to the driving environment, and will pause the conversation when a tricky maneuver approaches.

To the degree that tasks rely on similar processes, they are more likely to interfere with each other. For instance, talking on the phone and writing an e-mail is hard, because both involve language, Poldrack said.

The answer is to choose carefully when you take on more than one job at once. For high-priority or complex tasks, you might want to shut down your e-mail, turn off the phone and close your office door. Apex Performance founder Louis Czoka even recommends that clients shut their eyes to focus on a teleconference.

***** Informing Ourselves To Death

The great English playwright and social philosopher George Bernard Shaw once remarked that all professions are conspiracies against the common folk. He meant that those who belong to elite trades -- physicians, lawyers, teachers, and scientists -- protect their special status by creating vocabularies that are incomprehensible to the general public.

This process prevents outsiders from understanding what the profession is doing and why -- and protects the insiders from close examination and criticism. Professions, in other words, build forbidding walls of technical gobbledegook over which the prying and alien eye cannot see.

We may well ask ourselves, then, is there something that the masters of computer technology think they are doing for us which they and we may have reason to regret? I believe there is, and it is suggested by the title of my talk, "Informing Ourselves to Death." In the time remaining, I will try to explain what is dangerous about the computer, and why. And I trust you will be open enough to consider what I have to say. Now, I think I can begin to get at this by telling you of a small experiment I have been conducting, on and off, for the past several years. There are some people who describe the experiment as an exercise in deceit and exploitation but I will rely on your sense of humor to pull me through.

Here's how it works: It is best done in the morning when I see a colleague who appears not to be in possession of a copy of The New York Times. "Did you read The Times this morning?," I ask. If the colleague says yes, there is no experiment that day. But if the answer is no, the experiment can proceed. "You ought to look at Page 23," I say. "There's a fascinating article about a study done at Harvard University." "Really? What's it about?" is the usual reply. My choices at this point are limited only by my imagination. But I might say something like this: "Well, they did this study to find out what foods are best to eat for losing weight, and it turns out that a normal diet supplemented by chocolate eclairs, eaten six times a day, is the best approach. It seems that there's some special nutrient in the eclairs -- encomial dioxin -- that actually uses up calories at an incredible rate."

Another possibility, which I like to use with colleagues who are known to be health conscious is this one: "I think you'll want to know about this," I say. "The neuro-physiologists at the University of Stuttgart have uncovered a connection between jogging and reduced intelligence. They tested more than 1200 people over a period of five years, and found that as the number of hours people jogged increased, there was a corresponding decrease in their intelligence. They don't know exactly why but there it is."

I'm sure, by now, you understand what my role is in the experiment: to report something that is quite ridiculous -- one might say, beyond belief. Let me tell you, then, some of my results: Unless this is the second or third time I've tried this on the same person, most people will believe or at least not disbelieve what I have told them. Sometimes they say: "Really? Is that possible?" Sometimes they do a double-take, and reply, "Where'd you say that study was done?" And sometimes they say, "You know, I've heard something like that."

Now, there are several conclusions that might be drawn from these results, one of which was expressed by H. L. Mencken fifty years ago when he said, there is no idea so stupid that you can't find a professor who will believe it. This is more of an accusation than an explanation but in any case I have tried this experiment on non-professors and get roughly the same results. Another possible conclusion is one expressed by George Orwell -- also about 50 years ago -- when he remarked that the average person today is about as naive as was the average person in the Middle Ages. In the Middle Ages people believed in the authority of their religion, no matter what. Today, we believe in the authority of our science, no matter what.

And something else, which once was our friend, turned against us, as well. I refer to information. There was a time when information was a resource that helped human beings to solve specific and urgent problems of their environment. It is true enough that in the Middle Ages, there was a scarcity of information but its very scarcity made it both important and usable. This began to change, as everyone knows, in the late 15th century when a goldsmith named Gutenberg, from Mainz, converted an old wine press into a printing machine, and in so doing, created what we now call an information explosion. Forty years after the invention of the press, there were printing machines in 110 cities in six different countries; 50 years after, more than eight million books had been printed, almost all of them filled with information that had previously not been available to the average person. Nothing could be more misleading than the idea that computer technology introduced the age of information. The printing press began that age, and we have not been free of it since.

But what started out as a liberating stream has turned into a deluge of chaos. If I may take my own country as an example, here is what we are faced with: In America, there are 260,000 billboards; 11,520 newspapers; 11,556 periodicals; 27,000 video outlets for renting tapes; 362 million TV sets; and over 400 million radios. There are 40,000 new book titles published every year (300,000 world-wide) and every day in America 41 million photographs are taken, and just for the record, over 60 billion pieces of advertising junk mail come into our mail boxes every year. Everything from telegraphy and photography in the 19th century to the silicon chip in the twentieth has amplified the din of information, until matters have reached such proportions today that for the average person, information no longer has any relation to the solution of problems.

The tie between information and action has been severed. Information is now a commodity that can be bought and sold, or used as a form of entertainment, or worn like a garment to enhance one's status. It comes indiscriminately, directed at no one in particular, disconnected from usefulness; we are glutted with information, drowning in information, have no control over it, don't know what to do with it.

And there are two reasons we do not know what to do with it. First, as I have said, we no longer have a coherent conception of ourselves, and our universe, and our relation to one another and our world. We no longer know, as the Middle Ages did, where we come from, and where we are going, or why. That is, we don't know what information is relevant, and what information is irrelevant to our lives. Second, we have directed all of our energies and intelligence to inventing machinery that does nothing but increase the supply of information. As a consequence, our defenses against information glut have broken down; our information immune system is inoperable. We don't know how to filter it out; we don't know how to reduce it; we don't know to use it. We suffer from a kind of cultural AIDS.

Now, into this situation comes the computer. The computer, as we know, has a quality of universality, not only because its uses are almost infinitely various but also because computers are commonly integrated into the structure of other machines. Therefore it would be fatuous of me to warn against every conceivable use of a computer. But there is no denying that the most prominent uses of computers have to do with information. When people talk about "information sciences," they are talking about computers -- how to store information, how to retrieve information, how to organize information. The computer is an answer to the questions, how can I get more information, faster, and in a more usable form? These would appear to be reasonable questions. But now I should like to put some other questions to you that seem to me more reasonable. Did Iraq invade Kuwait because of a lack of information? If a hideous war should ensue between Iraq and the U.S., will it happen because of a lack of information? If children die of starvation in Ethiopia, does it occur because of a lack of information? Does racism in South Africa exist because of a lack of information? If criminals roam the streets of New York City, do they do so because of a lack of information?

Or, let us come down to a more personal level: If you and your spouse are unhappy together, and end your marriage in divorce, will it happen because of a lack of information? If your children misbehave and bring shame to your family, does it happen because of a lack of information? If someone in your family has a mental breakdown, will it happen because of a lack of information?

I believe you will have to concede that what ails us, what causes us the most misery and pain -- at both cultural and personal levels -- has nothing to do with the sort of information made accessible by computers. The computer and its information cannot answer any of the fundamental questions we need to address to make our lives more meaningful and humane. The computer cannot provide an organizing moral framework. It cannot tell us what questions are worth asking. It cannot provide a means of understanding why we are here or why we fight each other or why decency eludes us so often, especially when we need it the most. The computer is, in a sense, a magnificent toy that distracts us from facing what we most needed to confront -- spiritual emptiness, knowledge of ourselves, usable conceptions of the past and future. Does one blame the computer for this? Of course not. It is, after all, only a machine. But it is presented to us, with trumpets blaring, as at this conference, as a technological messiah.

Through the computer, the heralds say, we will make education better, religion better, politics better, our minds better -- best of all, ourselves better. This is, of course, nonsense, and only the young or the ignorant or the foolish could believe it. I said a moment ago that computers are not to blame for this. And that is true, at least in the sense that we do not blame an elephant for its huge appetite or a stone for being hard or a cloud for hiding the sun. That is their nature, and we expect nothing different from them. But the computer has a nature, as well. True, it is only a machine but a machine designed to manipulate and generate information. That is what computers do, and therefore they have an agenda and an unmistakable message.

The message is that through more and more information, more conveniently packaged, more swiftly delivered, we will find solutions to our problems. And so all the brilliant young men and women, believing this, create ingenious things for the computer to do, hoping that in this way, we will become wiser and more decent and more noble. And who can blame them? By becoming masters of this wondrous technology, they will acquire prestige and power and some will even become famous. In a world populated by people who believe that through more and more information, paradise is attainable, the computer scientist is king. But I maintain that all of this is a monumental and dangerous waste of human talent and energy. Imagine what might be accomplished if this talent and energy were turned to philosophy, to theology, to the arts, to imaginative literature or to education? Who knows what we could learn from such people -- perhaps why there are wars, and hunger, and homelessness and mental illness and anger.

As things stand now, the geniuses of computer technology will give us Star Wars, and tell us that is the answer to nuclear war. They will give us artificial intelligence, and tell us that this is the way to self-knowledge. They will give us instantaneous global communication, and tell us this is the way to mutual understanding. They will give us Virtual Reality and tell us this is the answer to spiritual poverty. But that is only the way of the technician, the fact-mongerer, the information junkie, and the technological idiot.

Here is what Henry David Thoreau told us: "All our inventions are but improved means to an unimproved end." Here is what Goethe told us: "One should, each day, try to hear a little song, read a good poem, see a fine picture, and, if it is possible, speak a few reasonable words." And here is what Socrates told us: "The unexamined life is not worth living." And here is what the prophet Micah told us: "What does the Lord require of thee but to do justly, and to love mercy and to walk humbly with thy God?" And I can tell you -- if I had the time (although you all know it well enough) -- what Confucius, Isaiah, Jesus, Mohammed, the Buddha, Spinoza and Shakespeare told us. It is all the same: There is no escaping from ourselves. The human dilemma is as it has always been, and we solve nothing fundamental by cloaking ourselves in technological glory.

Reuters Business Briefing - Information Overload

WADDINGTON, P. Dying for information: an investigation of information overload in the UK and world-wide. 1996. London: Reuters Business Information. A non-conclusive study that can be easily critiqued, but still provide an interesting observations.

During the survey, one in four of over 1,300 managers surveyed, admitted to actually suffering ill health as a result of the amount of information they handle. Ironically, half agreed high levels of information were required in order to perform effectively. In the survey, 48% of managers predicted the Internet would play a primary role in aggravating the problem further in 1998.
Effects of Information Overload

According to the survey, the results of `Infoglut` include:

Psychologist Dr.David Lewis, an internationally known Psychologist, Consultant and Lecturer, analysed the findings of the survey and commented:

"Information Fatigue Syndrome – having too much information – can be as dangerous as having too little. Among other problems, it can lead to a paralysis of analysis, making it far harder to find the right solutions or make the best decisions."

 Details of survey findings

The 350 page report, entitled:

`Dying for Information? An investigation into the effects of information overload in the U.K and World-wide`,

Is based on a survey of 1,313 junior, middle and senior managers in the U.K, U.S, Australia, Hong Kong and Singapore.

 Key Findings

Key Findings (continued)

Information Overload  by David Brodwin, Sandra Bernardi  Upside magazine

The information age has lived up to its name: It takes ages to sift through all the information that is now available to organizations. The overabundance of information has created a time-consuming and difficult challenge for busy executives and professionals searching for that key piece of data in an increasingly competitive world.

Content owners (primary publishers like McGraw-Hill, CMP Publications, Ziff-Davis and others) want to extract as much value from their ownership as possible, and not give away half or more of the revenue to the information repackagers that now stand between them and the end-customer. As the balance between print-based revenue and electronic revenue shifts, primary publishers are scrutinizing these issues more seriously than they did in the past, when electronic revenue was meager compared with the robust print business.

HarperEdge Data Smog

It's Monday morning, and you're just getting in to work. You pass the pile of faxes on the fax machine, then grab the stack of mail from your mailbox, wondering why you still get so much paper in this electronic era. At your desk, you note your company's stock price as it scrolls by on your screensaver, click some Web links to get last night's sports scores and this morning's headlines, and, finally, check your email. There are more than 100 messages. Some are useless marketing spams; some are mildly interesting newsletters; and buried near the end of the queue, there's an urgent message from your boss. If only you had a faster modem at home, you could have been in touch over the weekend and avoided the backlog...

Welcome to the information glut. According to Data Smog, a new book by David Shenk, information overload is more than an inconvenience; it's responsible for higher levels of stress, decreasing educational standards, and numerous other social problems. Traditional mass media are partially to blame, but the Internet--with its vast storehouse of knowledge and promise of instant communication--contributes to the avalanche. While the book's alarmism may seem a bit extreme, we've all experienced the paralysis that comes with having too much information, too many options, and no time to make a rational decision, let alone take a one-hour lunch.

So how can you cut through all the junk? We can't turn off your fax machine or convince you to throw out your cellular phone. However, if you want to get the most out of your computer and the Internet--without being buried by too much information--read on.

Information Overload

 Interesting IEEE Article

On the 3rd December 1997 a one day colloquium was held at The Institute of Electrical Engineers in London to address the problems of information overload. The primary objective of the colloquium is to consider how to overcome this problem from a technical perspective, that is examining a number of I.T. stategies such as Data Warehousing and the use of Intelligent Agents.

A paper entitled, "Information Overload - myth or reality?", suggests that there may be alternative non-technical reasons why information overload exists. The paper briefly examine the rise in status of information, the stresses of modern working life, the reality and consequences of information overload.

[Feb 2, 1999 ] In search of knowledge

Professors at Harvard Business School are famous for their case study style of teaching, in which they shine a bright, critical light on real-world companies and their strategies. Until recently, however, HBS itself easily could have served as an object lesson in how not to manage massive amounts of unstructured data. After years of recording classroom lectures, HBS and its users were awash in a sea of thousands of hours of unstructured videotape. Though available to researchers, the videotape brain trust went essentially untapped, since users couldn't easily navigate the content to locate relevant information.

"Nothing was indexed or easily accessed," recalls Larry Bouthillier, head of multimedia production at HBS. "We were able to do some keyword searches using transcriptions of the audio, but it was hard for people to find stuff that we didn't know we had."

Bouthillier's dilemma is shared by hundreds of companies, many of which have been struggling for years to create enterprise data warehouses designed to give top decision makers access to all the data generated by key operational systems. After finally getting those large decision-support systems into production, however, these companies are finding that the new data warehouses hold, at most, between 10 percent to 15 percent of all the data used daily across an enterprise. The other 85 percent to 90 percent is unstructured data--documents, images, text files, video, audio and other types of content that don't fit neatly into the rows, columns and access methods used to manage most data generated by transaction-oriented systems. Yet, for many companies, including HBS, that unstructured information can be every bit as important as the data from traditional operational systems such as financial and manufacturing applications.

As a result, many IT managers are hunting for tools that can manage unstructured data in much the same robust way with which they are already able to manage structured data. That means capturing it in electronic form, checking it for quality, indexing it in ways that allow users to find pertinent information quickly and providing easy access to it from the Web.

"What's needed is a way to help users get through the huge overload of unstructured data that already exists in their organizations," says Ralph Sprague, a professor in decision sciences at the University of Hawaii, in Honolulu. "We need a lot more than keyword search engines and document management systems. We need a way to catalog and understand the semantically rich content that exists in unstructured data such as images, video and documents."

The good news is that IT managers are finding a wide range of tools that do just that. A multitude of vendors--including makers of content management systems, search engines and even data warehouses--are beginning to extend their products to support advanced management of unstructured data types. In some companies, such tools are becoming the foundation for knowledge management. But there's also some bad news: Many of these tools are still immature. As such, they're often narrowly focused on one type of unstructured data such as documents or video. Many also lack support for critical standards such as XML (Extensible Markup Language).

Information overload

Even so, this new class of software is attracting the attention of IT managers anxious to provide users with a way to navigate through a vast and growing sea of unstructured data. HBS' Bouthillier is one. Three years ago, Bouthillier began making videotapes of the school's lectures available to researchers via streaming video running on five Sun Microsystems Inc. SPARC servers and accessed through a switched Ethernet network. When that setup failed to deliver easy access, Bouthillier took another stab at getting all of that video under control.

His solution: Video Cataloger, an indexing and retrieval engine from Virage Inc., of San Mateo, Calif., that, in effect, lets users query large amounts of video data by subject or image type. Video Cataloger scans video, looking for hints about its content based on displayed text information, audio, time codes and images. Then the product creates an index that can be used to search for specific video content. Hits are shown in the form of freeze-framed images of storyboards of the video.

Bouthillier has begun to allow students and researchers to query the video data over the school's intranet from Web browsers. The only drawback: It takes time to scan each video into the system.

And HBS isn't the only organization being overwhelmed with unstructured data. Far from going away, documents, text files, video and other types of unstructured data are proliferating in most organizations. Gartner Group Inc., of Stamford, Conn., estimates that U.S. companies produce 5.5 billion documents annually. And most of them -- 59 percent -- are still being accessed and retrieved manually.

... In response, many organizations are attempting to build portals on their intranets that, among other things, can act as central repositories for documents, images and other types of unstructured data. That's what professional audio workstation maker Digidesign, a division of Avid Technology Inc., is doing. In the process of attempting to get ISO 9002 quality certification, managers at the 300-person global company early this year discovered that it was practically impossible to find all the information about how Digidesign goes about building its products.

"Most of it was located on a multitude of servers in dozens of different file and security formats," says Bill Schwartz, Digidesign's operations project manager, in Palo Alto, Calif. "At a minimum, we needed to get the documents that described our policies and procedures into one place so that anyone in the company, anywhere in the world, could access it. And we were talking here about thousands of documents."

Recommended Links

Softpanorama Top Visited

Softpanorama Recommended


FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  


Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy


War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes


Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law


Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least

Copyright © 1996-2015 by Dr. Nikolai Bezroukov. was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case is down currently there are two functional mirrors: (the fastest) and


The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: February 19, 2014