|May the source be with you, but remember the KISS principle ;-)|
|Contents||Bulletin||Scripting in shell and Perl||Network troubleshooting||History||Humor|
Version 1.3 (Aug 28, 2017)
|Hybrid Cloud as Alternative to "Pure" Cloud Computing||Continuous delivery -- running on the blade edge||Infrastructure as Code|
|Unix Configuration Management Tools||Unix System Monitoring||Enterprise Job schedulers||Questionable costs efficiency of pure cloud||Lock in related problems||Cloud Mythology|
|Preventing Vendors From Playing The Blame Game||Bandwidth communism||Problem of loyalty||Issues of security and trust in "pure" cloud environment||Typical problems with IT infrastructure||Heterogeneous Unix server farms|
|Dictionary of corporate bullshit||Dispelling IT Management Myths||Fundamental Absurdity of IT Management||Conway Law||Brooks law||Managing Managers Expectations|
|Troubleshooting Remote Autonomous Servers||Configuring Low End Autonomous Servers||Review of Remote Management Systems||Virtual Software Appliances||Real Insights into Architecture Come Only From Actual Programming||Is Google evil ?|
|High Demand Cults Leaders Practices||Groupthink||Pollyanna creep||Machiavellians Manipulators Tricks||Disciplined Minds||Belief-coercion in high demand cults|
|Bosos or Empty Suits (Aggressive Incompetent Managers)||The Peter Principle||Sysadmin Horror Stories||Humor||Etc|
|The term cult usually refers to a
social group defined
by its religious,
or its common interest
in a particular personality, object or
goal. The term itself is controversial
and it has divergent definitions in both
and it also has been an ongoing source of contention among scholars across several fields of study.
sociological classifications of religious movements, a cult is a social group with
deviant or novel
beliefs and practices...
... ... ...
...In 1990 Lucy Patrick commented: "Although we live in a democracy, cult behavior manifests itself in our unwillingness to question the judgment of our leaders, our tendency to devalue outsiders and to avoid dissent. We can overcome cult behavior, he says, by recognizing that we have dependency needs that are inappropriate for mature people, by increasing anti-authoritarian education, and by encouraging personal autonomy and the free exchange of ideas."phenomenon that occurs within a group of people in which the desire for harmony or conformity in the group results in an irrational or dysfunctional decision-making outcome. Group members try to minimize conflict and reach a consensus decision without critical evaluation of alternative viewpoints by actively suppressing dissenting viewpoints, and by isolating themselves from outside influences.
Groupthink requires individuals to avoid raising controversial issues or alternative solutions, and there is loss of individual creativity, uniqueness and independent thinking. The dysfunctional group dynamics of the "in-group" produces an "illusion of invulnerability" (an inflated certainty that the right decision has been made). Thus the "in-group" significantly overrates its own abilities in decision-making and significantly underrates the abilities of its opponents (the "out-group"). Furthermore, groupthink can produce dehumanizing actions against the "out-group".
Antecedent factors such as group cohesiveness, faulty group structure, and situational context (e.g., community panic) play into the likelihood of whether or not groupthink will impact the decision-making process.
You’ve probably heard the hype about DevOps: It helps you deliver products faster, improves your profitability, ensures continuous integration, removes roadblocks from your releases and gives you a competitive advantage. And that such companies as Netflix and Amazon supposedly practice it (and in case of Netflix heavily advertize).
Those claims alert all seasoned IT professionals because they are very similar claims about previous IT fads that now are semi-forgotten. There were at least half dozen previous attempt to achieve nirvana in the corporate datacenter and, especially, enterprise software development, starting from "verification revolution" initiated by Edsger W. Dijkstra, who probably was one of first "cult leader" style figures in history of programming.
We can dig even deer and trace this to "structured programming" revolutions with its pseudo religious crusade against goto statements (see important historically article by Donald Knuth Structured programming with goto statements cs.sjsu.edu). Verification hoopla actually destroyed careers of several talented computer scientists such as David Gries (of Compiler Construction for Digital Computers fame; also participated in creation PL-C compiler designed by Richard W. Conway and Thomas R. Wilcox ) who jumped into this bandwagon. It also damaged long term viability of achievements of Niklaus Wirth ( a very talented language developer, who participated in creation of Algol 60 and later developed Pascal and Modula), also known for his Wirth's law: "Software is getting slower more rapidly than hardware becomes faster."
Dismal quality of Netflix web interface actually instantly razes several question about "What those guys are doing?". Like in Amazon this is essentially an index to the product database. But if Amazon at least tries to implement several most revenant types of searches, Netflix does not. For rare movies it is rather challenging to find what you interested in, unless you know the exact title ;-). Movies are not consistently indexed by director and major stars. The way they deal with reviews is actually sophomoric.
Amazon is much better, but in certain areas it still has "very average" quality. For Amazon one especially bad oversight is that the reviews are indexed only in one dimension (number of stars), not by "reviewer reputation" and more advanced criteria. As a result, many reviews are "fake" (bought, or "friends reviews" produced by people who are not actual customers). That distorts that rating of the products, in some cases considerably. Recently they tried to compensate this by "verified purchase" criteria, but without much success, as in most cases the book can be returned. While some fake reviews are detectable by the total number of reviews posted (often this number is one ;-), in many cases it is not possible easily to distinguish "promotional campaign" from actual reviews. Shrewd companies can subsidized purchases in exchange for positive reviews. Amazon cloud has its uses, but is rather expensive.
You probably heard a little bit about so called High Demand Cults
Remember ... A group does not have to be religious to be cultic in behavior. High demand groups can be commercial, political and psychological. Be aware, especially if you are a bright, intelligent and idealistic person. The most likely person to be caught up in this type of behavioral system is the one who says “I won’t get caught. It will never happen to me. I am too intelligent for that sort of thing.”
The following statements, compiled by Dr. Michael Langone, editor of Cultic Studies Journal, often characterize manipulative groups. Comparing these statements to the group with which you or a family member is involved may help you determine if this involvement is cause for concern.
- Love bombing: Fake display of excessively zealous, unquestioning commitment and adulation to new members. Expensive gift, etc. This is a typical practice of sociopaths and many cult leaders are sociopaths.
- Isolation: The group leader instills a polarized, "we-they" mentality and try to isolate members from their previous contacts. Members are encouraged or required to work with and/or socialize only with group members. Severing of Ties with Past, Family, Friends, Goals, and Interests - Especially if they are negative towards or impede the goals of the group.
- High demand on members time, intensity of contacts: Intense involvement in cult activities work along with isolation from others. Behaviour is closely prescribed and carefully supervised. Members are expected to devote inordinate amounts of time to the group and group activities.
- Manipulation: The group's leadership induces guilt feelings in members in order to control them. The pity play is high in their list of manipulation techniques. It's okay to pity someone who has gone through difficult times, but if you find yourself feeling sorry for someone's sad story, make sure the story is true. The pity play should serves as a valuable warning sign that you are dealing with a manipulator, in this case a female sociopaths.
- Brainwashing: Questioning, doubt, and dissent are discouraged or even punished with a bouts of anger or (fake) withdrawal. Special techniques are used to suppress doubts about illegal or questionable or amoral practices in the group or its leader(s) and put dissidents in line. Special literature is distributed and indoctrination campaign is launched. Much like pedophiles "groom" children by pushing or encouraging them to watch porno movies and books with explicit content. If group leader requires special favors from members of the group (for example sex with female members) this all is masked under some artificial pretext like "exercise in liberation" or "exercise in compassion".
- Dictate and micromanagement: The group's leader practice micromanagement and dictates – sometimes in great detail – how members should think, act, what to wear and feel.
- Instilling amorality, end justifies the means mentality: Any action or behaviour is justifiable as long as it furthers the group's goals. The group (leader) becomes absolute truth and is above all man-made laws.
My initial impression that former cultists come face with a multiplicity of losses, accompanied by a deep, and sometimes debilitating, sense of anguish. See for example interviews with defector from Mormonism on YouTube
... ... ....
My hope upon initiating this research was to provide a link between cult leaders and corporate psychopath and demonstrate that cult leaders practices (that are more or less well understood and for which extensive literature exists) have a strong predictive power for the behavior of a corporate psychopath. We should not focus just on the acute and long-term distress accompanying reporting to corporate psychopath.
Here are some psychological mechanisms used:
- Control of the Environment and Communication The control of human communication is the most basic feature of the high demand cult environment. This is the control of what the individual sees, hears, reads, writes, experiences and expresses. It goes even further than that, and controls the individuals communication with himself - his own thoughts.
- The Mystique of the Organization. This seeks to provoke specific patterns of behaviour and emotion in such a way that these will appear to have arisen spontaneously from within the environment. For the manipulated person this assumes a near-mystical quality. This is not just a power trip by the manipulators. They have a sense of “higher purpose” and see themselves as being the “keepers of the truth.” By becoming the instruments of their own mystique, they create a mystical aura around the manipulating institution - the Party, the Government, the Organization, etc. They are the chosen agents to carry out this mystical imperative.
- Everything is black & white. Pure and impure is defined by the ideology of the organization. Only those ideas, feelings and actions consistent with the ideology and policy are good. The individual conscience is not reliable. The philosophical assumption is that absolute purity is attainable and that anything done in the name of this purity is moral. By defining and manipulating the criteria of purity and conducting an all-out war on impurity (dissension especially) the organization creates a narrow world of guilt and shame. This is perpetuated by an ethos of continuous reform, the demand that one strive permanently and painfully for something which not only does not exist but is alien to the human condition.
- Absolute “Truth” . Their “truth” is the absolute truth. It is sacred - beyond questioning. There is a reverence demanded for the leadership. They have ALL the answers. Only to them is given the revelation of “truth”.
- Thought terminating clichés. Everything is compressed into brief, highly reductive, definitive-sounding phrases, easily memorized and easily expressed. There are “good” terms which represents the groups ideology and “evil” terms to represent everything outside which is to be rejected. Totalist language is intensely divisive, all-encompassing jargon, unmercifully judging. To those outside the group this language is tedious - the language of non-thought. This effectively isolates members from outside world. The only people who understand you are other members. Other members can tell if you are really one of them by how you talk.
What are the hallmarks of unhealthy, aberrant churches? The key indicator is control oriented leadership, ministers who have a need to "lord it over the flock." Abusive leaders demand submission and unquestioning loyalty. The person who raises uncomfortable questions or does not "get with the program" is cast aside. Guilt, fear, and intimidation are used to manipulate and control vulnerable members, especially those who have been taught to believe that questioning their pastor is comparable to questioning God.
Junk science (and DevOps firmly belongs to this category) is and always was based on cherry-picked evidence which has carefully been selected or edited to support a pre-selected "truth". Facts that do not fit the agenda are suppressed (Groupthink). Apocalyptic yelling are also very typical. Same for Pollyanna creep. Deployment is typically top-down. Corporate management is used as an enforcement branch (corporate Lysenkoism)
Here are some signs of "junk science" (Seven Eight Warning Signs of Junk Science):
Here is a non-exclusive list of
seveneight symptoms to watch out for:
- Science by press release. It’s never, ever a good sign when ‘scientists’ announce dramatic results before publishing in a peer-reviewed journal. When this happens, we generally find out later that they were either self-deluded or functioning as political animals rather than scientists. This generalizes a bit; one should also be suspicious of, for example, science first broadcast by congressional testimony or talk-show circuit.
- Rhetoric that mixes science with the tropes of eschatological panic. When the argument for theory X slides from “theory X is supported by evidence” to “a terrible catastrophe looms over us if theory X is true, therefore we cannot risk disbelieving it”, you can be pretty sure that X is junk science. Consciously or unconsciously, advocates who say these sorts of things are trying to panic the herd into stampeding rather than focusing on the quality of the evidence for theory X.
- Rhetoric that mixes science with the tropes of moral panic. When the argument for theory X slides from “theory X is supported by evidence” to “only bad/sinful/uncaring people disbelieve theory X”, you can be even more sure that theory X is junk science. Consciously or unconsciously, advocates who say these sorts of things are trying to induce a state of preference falsification in which people are peer-pressured to publicly affirm a belief in theory X in spite of private doubts.
- Consignment of failed predictions to the memory hole. It’s a sign of sound science when advocates for theory X publicly acknowledge failed predictions and explain why they think they can now make better ones. Conversely, it’s a sign of junk science when they try to bury failed predictions and deny they ever made them.
- Over-reliance on computer models replete with bugger factors that aren’t causally justified.. No, this is not unique to climatology; you see it a lot in epidemiology and economics, just to name two fields that start with ‘e’. The key point here is that simply fitting historical data is not causal justification; there are lots of ways to dishonestly make that happen, or honestly fool yourself about it. If you don’t have a generative account of why your formulas and coupling constants look the way they do (a generative account which itself makes falsifiable predictions), you’re not doing science – you’re doing numerology.
- If a ‘scientific’ theory seems tailor-made for the needs of politicians or advocacy organizations, it probably has been. Real scientific results have a cross-grained tendency not to fit transient political categories. Accordingly, if you think theory X stinks of political construction, you’re probably right. This is one of the simplest but most difficult lessons in junk-science spotting! The most difficult case is recognizing that this is happening even when you agree with the cause.
- Past purveyers of junk science do not change their spots. One of the earliest indicators in many outbreaks of junk science is enthusiastic endorsements by people and advocacy organizations associated with past outbreaks. This one is particularly useful in spotting environmental junk science, because unreliable environmental-advocacy organizations tend to have long public pedigrees including frequent episodes of apocalyptic yelling. It is pardonable to be taken in by this the first time, but foolish by the fourth and fifth.
- Refusal to make primary data sets available for inspection. When people doing sound science are challenged to produce the observational and experimental data their theories are supposed to be based on, they do it. (There are a couple of principled exceptions here; particle physicists can’t save the unreduced data from particle collisions, there are too many terabytes per second of it.) It is a strong sign of junk science when a ‘scientist’ claims to have retained raw data sets but refuses to release them to critics.
If we are talking about DevOps as a software development methodology it is similar to Agile. The latter was a rotten attempt to reshuffle a set of old ideas for fun and profit (some worthwhile, some not so much) into attractive marketable technocult and milk the resulting hoopla with books, conferences, consulting, etc.
In 2017 only few seasoned software developers believe that Agile is more then self-promotion campaign of group of unscrupulous and ambitious people, who appointed themselves as the high priests of this cult. Half-life of such "cargo cult" programming methodologies is usually a decade (Agile became fashionable around 1996). Now it is well past hype stage of the software methodology cycle.
Another relevant concept as for discussion of DevOps hoopla is Cargo Cult science. Cargo cult science comprises practices that have the semblance of being scientific, but do not in fact follow the scientific method. The term was first used by physicist Richard Feynman during his 1974 commencement address at the California Institute of Technology. Talented software developers are often superstars who can follow methods not suitable for "mere mortals" like continuous delivery.
I think Bill Joy (of BSD Unix, csh, NFS, Java, vi editor fame ) was such a software superstar when he produced BSD tapes. He actually created vi editor using terminal over 1200 baud modem (Bill Joy's greatest gift to man – the vi editor • The Register) -- which is excruciatingly slow. the same is true for Donald Knuth who created singlehandedly a Fortran complier during a summer, when he was still a student. And Ken Thompson--who is the father of Unix. And Larry Wall who created Perl while being almost blind on one eye. But that does not mean that there practices are scalable. Brooks book "Mythical man-month" is still relevant today as it was at the moment of publication.
The biggest mistake you can make as a manager or a large and important software project is to delegate the key functions in the design to mediocre people. You can't replace software talent with the organization, although organization helps. any attempt to claim otherwise are cargo cult science. In other word tin software engineering there is no replacement for displacement.
In reality nothing is new under the sun in software development. DevOps rehashed the ideas many of which are at least a decade old, with some that are at least 30 years old (Unix configuration management, version control). And the level of discussion is often lower than the level on which those ideas are discussed in The Mythical Man-Month, which was published in 1975 (another sign of "junk science").
What is more important that all under the surface of all those lofty goals is the burning desire of companies brass to use DevOps as another smoke screen for outsourcing. Yet another justification for "firing a lot of people."
|Under the surface of all those lofty goals is the burning desire of companies brass to use DevOps as another smoke screen for outsourcing. Yet another justification for "firing a lot of people."|
As in any cult, there are some grains of rationality in DevOps, along with poignant critique of the status quo. Which works well to attract the followers. Overhyping some ideas about how to cope with the current, unsatisfactory for potential followers situation in enterprise IT is another sign of a technocult. Some of those ideas are, at least on superficial level, pretty attractive. Otherwise such a technocult can't attract followers. As a rule, techno-cults generally emerge in time of huge dissatisfaction and strive by proposing "salvation" from the current difficulties:
...think this is one of the main reason why we see this DevOps movement, we are many that see this happen in many organizations, the malfunctioning organization and failing culture that can’t get operation and development to work towards common goals. In those organizations some day development give up and take care of operation by them self and let the operation guys take care of the old stuff.
Nobody can deny that there a lot of problems in corporate datacenters those days. bureaucratization is rampant and that stifle few talented people who still did not escape this environment, or switched to writing open source software during working hours, because achieving anything within the constrain of the existing organization is simply impossible ;-)
But along the this small set of rational (and old) ideas there a set of completely false, even bizarre ideas and claims, which make it a cult.. There is also fair share of Pollyanna creep. Again, it is important to understand that part of the promotion campaign success (with money for it mostly from companies who benefit from outsourcing) is connected with the fact that corporate IT brass realized than DevOps can serve well as a smote screen for another round of outsourcing of "ops". (Ulf Månsson about infrastructure )
This creates new way of working, one good example is the cultural change at Nokia Entertainment UK, presented at the DevOps conference in Göteborg, by inclusion going from 6 releases/year and 50 persons working with releases to 246 releases/year with only 4 persons, see http://www.slideshare.net/pswartout/devopsorg-how-we-are-including-almost-everyone. That story was impressive.
Pretention that this is a new technology is artfully created by inventing a new language ripe with new obscure terms. Another variant of Newspeak. And this is very important to understand. It is this language that allow to package bizarre and unproven ideas into the cloak of respectability.
Primitivism of thinking and unfounded claims of this new IT fashion (typical half-life of IT fad is less then a decade; for example, who now remembers all this verification hoopla and books published on the topic) are clearly visible in advocacy papers such as Comparing DevOps to traditional IT Eight key differences - DevOps.com. Some of the claims are clearly suspect, and smell of "management consultant speak" (an interesting variety of corporate bullshit). For example:
Traditional IT is fundamentally a risk averse organization. A CIO’s first priority is to do no harm to the business. It is the reason why IT invests in so much red tape, processes, approvals etc. All focused on preventing failure. And yet despite all these investments, IT has a terrible track record – 30% of new projects are delivered late, 50% of all new enhancements are rolled back due to quality issues, and 40% of the delay is caused by infrastructure issues.
A DevOps organization is risk averse too but they also understand that failure is inevitable. So instead of trying to eliminate failure they prefer to choose when and how they fail. They prefer to fail small, fail early, and recover fast. And they have built their structure and process around it. Again the building blocks we have referred to in this article – from test driven development, daily integration, done mean deployable, small batch sizes, cell structure, automation etc. all reinforce this mindset.
Note a very disingenuous claim "IT has a terrible track record – 30% of new projects are delivered late, 50% of all new enhancements are rolled back due to quality issues, and 40% of the delay is caused by infrastructure issues." I think all previous fashionable methodologies used the same claim, because the mere complexity of large software projects inevitably leads to overruns and doubling estimated time and money is a good software engineering methodology known since publishing of Mythical man month :-)
As for another disingenuous claim "So instead of trying to eliminate failure they prefer to choose when and how they fail. They prefer to fail small, fail early, and recover fast." this is not a realistic statement. It is a fake. In case of a SNAFU you can't predict the size of the failure. Just look at history of Netflix, Amazon or Google Gmail failures. Tell this mantra "They prefer to fail small, fail early, and recover fast" to customers of Google or Amazon during the next outage
Note also such criticism is carefully swiped under the carpet and definition of DevOps evolves with time to preserve its attractiveness to new members ( DevOps is Dead! Long Live DevOps! - DevOps.com ). In other words the correct definition of DevOps is "it is a very good thing" ;-). For example:
Some are seekers on the quest for the one, true DevOps. They were misled. I’m here to say: Give it up. Whatever you find at the end of that journey isn’t it. That one true DevOps is dead. Dead and buried. The search is pointless or, kind of worse: The search misses the point altogether.
DevOps began as a sort of a living philosophy: about inclusion rather than exclusion, raises up rather than chastises, increases resilience rather than assigns blame, makes smarter and more awesome rather than defines process and builds bunkers. At any rate, it was also deliberately never strictly defined.
In 2010, it seemed largely about Application Release Automation (ARA), monitoring, configuration management and a lot of beginning discussion about culture and teams. By 2015, a lot of CI/CD, containers and APIs had been added. The dates are rough but my point is: That’s all still there but now DevOps discussions today also include service design and microservices. Oh, and the new shiny going by the term “serverless.” It is about all of IT as it is naturally adapting.
The key objection to DevOps is that reliance on super-platforms such as Amazon cloud or Microsoft Asure could, in the future potentially intellectually capture the organization and remaining sysadmins (and IT staff in general) , who are increasingly distant from “nuts and bolt" of operating system and hardware and operate in what essentially is proprietary environment of particular vendor. That converts Unix sysadmin into a flavor on Windows sysadmin with less nice GUI. In other words they need put all trust in the platforms and detached themselves from "nut and bolts" levels. That means that, in a way, they became as dependent on those platforms as opiates addicts on their drags.
The also creates a set of DevOps promoters such as cloud providers, who want to become "gatekeepers", binding users to their technology. Those gatekeepers once they became non-displaceable makes sure that the organization lost the critical mass of technical IQ in "low level" (operating system level) and can't abandon them without much pain.
At this point they start to use this dependency to their advantage. Typically they try to estimate customers "willingness to pay" and, as a result, gradually increase the price of their services. IBM was great practitioner of this fine art in the past. That's why everybody hated it. VMware is also proved to be quite adept in this art.
Not that Amazon cloud is cheap even today iether. It is not, even is we calculate not only hardware and electricity savings, but also (and rather generously as in one sysadmin for 50 servers) manpower saving they provide. It is cheaper to run your own hardware with internal cloud then Amazon unless you have a huge peaks.
The same is even more true for VMware. If in the particular organization VMware used as virtualization platform for Linux (which due to the being open source allows para- virtualization, instead of full virtualization that VMware implements) , then to talk about savings is possible only when one is sufficiently drunk. Not that such savings do not exist. They do, but lion share of them is going directly to VMware, not to the organization which deploy this platform.
The danger is that you are basically willingly allowed to capture yourself and willing to be part of this ecosystem that is controlled by one single "gatekeeper." Such a decision creates an environment in which switching costs can be immense. That's why there is such a competition amount three top players in cloud provider space for the new enterprise customers. The first who grab a particular customer is the one to control and can milk such a customer for a long, long time. Classic DevOps advocates response that "you should have negotiated better" is false because people not have enough information when they enter negotiations, and it's too late when they finally will "get it."
Like Agile before DevOps emphasizes a culture of common goals (this time between "operations" and developers with the idea of merging them like in good old times -- the initial name was NoOps; which is pretty idiotic idea taking into account the complexity of the current IT infrastructure) and get things done together, presenting itself as new IT culture.
But details are fuzzy and contradictory. and that strongly suggests that like many similar IT "fashions" before it DevOps mean "a good thing". While it seems the IT management is rushing to embrace the concept of DevOps (because it justifies further outsourcing under the smokescreen of new terms), nobody agrees on what it actually means. And that creates some skepticism.
DevOps paints a picture of two cultures (" operations" and "developers" ) once at odds, now working together in harmony. But first of all there are weak and strong developers. There are weak and strong Unix sysadmins. That can't be changed with some fashionable chair reshuffling. What actually happen is that more talented members of the team get additional workload. That's why some critics claim that DevOps kills developers. Meaning "talented developers". And it certainly can be done but it is easier said then done.
The fact that DevOps is somehow connected with Agile also suggests that it is yet another snake oil initiative. With a bunch of talented an unprincipled salesmen who benefit from training courses, consulting gigs, published books, conferences, and other legal ways to extract money from lemmings.
When you read sentences like "DevOps is an environment where an agile relationship will take place between operations and development teams. "( https://www.quora.com/What-is-devops-and-why-is-it-important ) you quickly understand what type of people can benefit from DevOps.
DevOps is presented by adherents as all singing, all dancing universal solution to all problems of mankind, or, at least, of problems in the data center. Ignoring the fact that there is no "techno cure" for large datacenter problems, because those problem are not only technological in nature, but also reflect complex mix of sociological (the curse of overcomplexity is one of those; see The Collapse of Complex Societies) and, especially, balance of power between various groups within the data center.
Such as corporate management, developers and operation staff. Which create pretty interesting mix from sociological point of view and simultaneously creates a set of internal conflict and constant struggle for power between various strata of the datacenter ecosystem. From this point of view DevOps clearly represents political victory for developers, and management at the expense of everybody else.
In a way, this is a replay under new name old Sun idea "network is computer". Nothing too new and nothing really existing. That does not mean that there are no rational elements in DevOps hoopla, but those key elements exist for a long time. And in no way they are connected to DevOps and can (some probably should) be already deployed in really "upscale" datacenters.
And they are probably not very suitable for "less upscale" datacenters because you simply can not productively use technology platform that is far above technical capabilities of your staff. And if you try at the end of the day you will suffer unpredictable set of negative consequences. And they are not only limited to huge cost overruns (which you usually can swipe under the carpet). Which are somewhat similar to those predicted by the first two Softpanorama Laws of Computer Security:
As DevOps carries certain political dimensions (connection to the neoliberal transformation of the society, with its fake "cult of creativity", rise of role (and income) of the top 1% and decline of IT "middle class"), outsourcing and additional layoffs are probably the most prominent result of introduction of DevOps into real datacenters. Of course those "side effects" are never mentioned aloud in DevOps hoopla) problems.
|As DevOps carries certain political dimensions (connection to the neoliberal transformation of the society, with its fake "cult of creativity", rise of top 1% and decline of middle class), outsourcing and additional layoffs are probably the most prominent result of introduction of DevOps into real datacenters. Of course those "side effects" are never mentioned aloud in DevOps hoopla) problems.|
The whole idea that by adding some tools of questionable quality (like Puppet) and ideological brainwashing you can return datacenter to the good old days when a single person was a masters of all trades -- a developer, a system administrator and a tester is completely unrealistic. That will never happen. 60th and early 70th are not reproducible in the in the modern datacenter.
The maximum you can achieve (and that's pretty complex undertaking) is to remove some artificial bureaucratic barriers and make developer more productive because he can control more of the whole process not to bump with his head into bureaucratic obstacles when he need to make some changes in the environment in addition to his application.
But it is very important to understand that there are good developers and bad developers. Reckless power hungry developer, who will utilize those new opportunities 100%, is a great danger. By removing bureaucratic obstacles you also remove some safeguards against such developers. There is no free lunch.
|The whole idea that by adding some tools of questionable quality (like Puppet) and ideological brainwashing you can return to good old days when a single person was a masters of all trades -- a developer, a system administrator and a tester is completely unrealistic. That will never happen. The maximum you can achieve (and that's pretty complex undertaking) is to remove some bureaucratic barriers and make developer more productive because he can control more of the whole process not to bump with his head into bureaucratic obstacles when he need to make some changes in the environment in addition to his application. But there are good developers and bad developers. Reckless developer is a great danger. By removing bureaucratic obstacles you also remove safeguards against such developers. There is no free lunch.|
In other words the idea of even partial elimination of specialization between developers and operation staff is a dangerous illusion taking into account the complexity of modern operating systems. Giving a talented developer his own sysadmin and tester makes sense. But making him his own sysadmin and tester is borderline idiotism. Talented developers is a very rare commodity and their use should be highly optimized. Re-reading Mythical Man-month helps to get proper line of thinking about this problem.
The idea of removing extra red tape also has some validity, but here you are facing uphill battle against entrenched IT management interests, which try to isolate itself from people in the trenches with additional management layers (and those "redundant" management layers is kind of cancer of IT departments, because those middlemen tend to replicate like rabbits in uncontrollable fashion). High level IT management want technoserfs to work like slaves on this techno plantation to get their bonuses. And they want to achieve that without direct contact with those "badly smelling people" creating those middle management positions that isolate them. But in order to pay those people the herd of technoserfs needs to thinned, as private profits and bonuses for the brass here comes before efficiency :-) Generally the question of parasitic rents within IT organization is a very interesting topic, but it is beyond the scope of this paper.
As large datacenters and large software development problem are inherently complex. So going over budget and taking twice longer than it should in large software project is more or less typical course of events. A larger danger here is compete fiasco, like in the past happened with too enthusiastic idiots who implemented SAP in their companies. This type of failures actually can crash the whole company and put it out of business.
Any claims that those fundamental issues can be somehow eliminated with some cosmetic measures ("magic dust") is a typical snake oil salesmen staff. Software development talent, especially high level software development talent that encompass also strong architectural vision, is a very scarce commodity. In you have no such developers, that chances of successful completion of a complex project are slim. No superficial remedies can help to solve this tremendously difficult problem. Huge blunders of incompetent high level IT managers is also a reality.
In short the key problem of modern datacenter and modern software applications is lack of top software talent both on the level of software developers and high IT management. As well as mind boggling complexity of both the operating systems and applications. They are probably the most complex artifacts created by mankind. And that means that it is unclear whether the better solution lies in increasing the complexity or decreasing it. I tend to think that the latter might be a better path forward (KISS principle). Which puts me on the collision course with DevOps hoopla. Your mileage may vary.
Like any repackaging effort ,DevOps presents old, exiting technologies as something revolutionary. They are not. Among measures that DevOps "movement" advocates several old ideas make sense if implemented without fanaticism. And they could be implemented before the term DevOps (initially NoOps) was invented. That includes:
Some of those (Docker) definitely makes more sense than others because preoccupation with VMware as the only solution for virtualization in enterprise environment is unhealthy from multiple points of view.
But all of them are adding, not removing complexity. Just the source of it different (in case of Docker you also can somewhat compensate increase of the complexity due to the switch to virtual environment by the fact that you can compartmentalize most critical applications in the separate "zones (using Solaris terminology -- Docker is essentially reimplementation of Solaris zones in Linux).
May be the truth lies somewhere in between and selected and "in moderation" implementation of some of those technology in the datacenter can be really beneficial. But excessive zeal can hurt not help. In this sense presence of newly converted fanatics is a recipe for additional problems. if not a disaster.
I actually saw the situations, when implementation of DevOps brought corporate IT to a screeching halt, almost complete paralysis when nothing but maintaining (with great difficulties) of the status quo was achieved for the first two years. Outsourcing in those cases played major role as new staff needs large amount of time and effort to understand the infrastructure and processes and attempt to "standardize" some servers and services often fail dismally due to complex interrelations between them and applications in place. While the current situation is definitely unsatisfactory from many points of view, the first principle should be "do no harm". In many cases it might sense to announce the switch to DevOps, force people to prepare to it (especially discard old servers and unused applications) and then cancel the implementation. You probably can probably achieve 80% of positive effect this way, avoiding 80% of negative effects. Moving datacenter to a new location can also tremendously help and can be used instead of DevOps implementation :-)
While the current situation in a typical large datacenter is definitely unsatisfactory from many points of view, the first principle should be "do no harm".
Again, the main problem with DevOps hoopla is that, if implemented in full scale, it substantially increases the complexity of environment. Both virtual machines and configuration management tools provide additional "levels of indirection" which makes troubleshooting more complex and causes of failures more varied and complex. That represents problems even for seasoned Unix sysadmins to say nothing for poor developers who are thrown into this water and asked to swim.
Sometimes DevOps methodology implemented in modest scope does provide some of the claimed benefits (and some level of configuration management in a large datacenter is a must). The question is what if the optimal solution for this and how not to overdo it. But in a typical DevOps implementation this question somehow does not arise and it just degenerates into another round of centralization and outsourcing which make the situation worse. Often much worse to the level when It really became completely dysfunctional. I saw this affect of DevOps implementation in large corporations.
So it should be evaluated on case by case basic, not as panacea. As always much depends of the talent of the people who try to implement it. Also change in a large datacenter is exceedingly difficult and often degenerates into what can be called one step forward, two steps back". For example, learning such tools as Puppet or Chief requires quite a lot of effort for rather questionable return on investment as complexity precludes full utilization of the tool and it is downsized to basic staff. So automation using them is a mixed blessing.
Similarly move to lightweight VM (and virtual servers in general) which is a part of DevOps hype are easier to deploy, but load management of multiple servers running on the same box is an additional and pretty complex task. Also VMware that dominates VM scene is expensive (which mean that the lion share of saving are going to VMware, not to the enterprise which deploys it ;-) and is a bad VM for Linux. Linux needs para-virtualization not full CPU virtualization that VMware, which was designed for Windows, offers (with some interesting optimization tweaks). Docker, which is a rehash of the idea of Solaris zones is a better deal, but is a pretty new for Linux technology and it has its own limitations. Often severe as this is a light weight VM.
Cost wise DevOps (typically implemented with the help of outsourcers) is a rather expensive proposition which provides both higher complexity and lower reliability (outsourcing costs are low usually only at the beginning as most vendors perform "bait and switch" underbidding their services and then try to compensate this by inflating operating costs, whenever possible, especially in areas were local "know how" disappeared.
So benefits, is they exist, are not automatic, but are dependent of the qualification of people implementing this initiative.
VM problems while relatively rare still are unwelcome addition to the set of existing problem; also if server gos down all VM gores down with it). Using VM is very similar to replacing enterprise servers with desktops, but at a higher cost (disposable desktops is an interesting alternative to VMware VMs cost wise). The cost of "decent" desktop now is one tens of the cost of a "decent" enterprise server ($6K-$8K) but 10 desktops can definitely do more computation wise.
Complexity of some tools such as Puppet also lessen reliability ("Alice in Wonderland" situations) and costs.
DevOps serve as a smoke screen for further outsourcing and moving from a traditional data center to the cloud deployment model. Those activities are considered by corporate brass as another way to cut costs (and the costs of IT in most large manufacturing corporations are already about 1% or less; so there is not much return on this investment anyway).
From 1992 to 2012 Data Centers already experienced technological reorganization, which might be called Intel revolution, which dramatically increased role of Intel computers in the datacenter, introduced new server form factors such as blades and new storage technologies such as SAN and NAS. Virtualization became common in Windows world due to proliferation of VMware instances. Faster internet and wireless technologies allowed more distributed workforce and ability for people work part of the week from home. Smartphones now exceed the power of 1996 desktop. Moreover there was already a distinct trend of the consolidation of datacenters within the large companies.
As the result many services, such as email and to lesser extent file storage are already provided via internal company cloud from central servers. At the same time it became clear that along with technical challenges "cloud services" present huge threat to security and privacy. The driving force behind the cloud is the desire to synchronize and access data from several devices that people now own (desktop, laptop, smartphone, tablets. In other words to access your own data from multiple devices. The first such application, ability to view corporate e-mail from the cell phone, essentially launched Blackberry smartphones into prominence.
In view of those changes managing datacenter remotely became a distinct possibility. That's why DevOps serves for higher management as a kind of "IT outsourcing manifesto". But with outsourcing the problem o loyalty comes into forefront.
DevOps was a perverse reaction to the problem in companies who now need to maintain thousand of servers, such as Netflix, Amazon, Facebook and Google. People wanted some technological panacea and it fills the niche. Providing a false solution to the real and complex problems. So DevOps is intrinsically connected with attempts of automation of system administration using Unix configuration management tools. BTW that does not mean that Puppet or Chief are the proper way to do that (the king might still be naked ;-). They just bask in DevOps hype . And try to sustain and maintain it for fun and profit.
The second part is an attempt to break isolated and fossilized silos in which developers and operation staff live in large corporation. Sometimes with almost without interaction. So in a way this as an over-reaction to a dysfunctional ops organization (as Adrian Cockcroft admitted in his comment.), which actually clearly demonstrates that he does not understand how complex organization operated (his idea of abolishing operations is just an attempt to grab the power by "developer class") and never read the book by Charles Perrow (Complex Organizations A Critical Essay). In complex organization technological issues re always intersect with power struggle and effects of neoliberalism on IT and corporate environment in general. Blowback of excessive outsourcing of IT is hitting large corporations and is a part of problem that IT organizations experience now. DevOps in this sense is a part of the problem and not a part of the solution. You will never read about this this in "DevOps hoopla" style of books that proliferate (Amazon lists around hundred of books on this topic; most junk). In reality globalization while solving one set of problem creates another and her IT is both the victim and the part of this "march of neoliberalism" over globe:
Information and communications technologies are part of the infrastructure of globalization in finance, capital mobility and transnational business. Major changes in the international economic landscape are intertwined and contemporary accelerated globalization is in effect a package deal that includes informatization (applications of information technology), flexibilization (changes in production and labour associated with post-Fordism), financialization (the growing importance of financial instruments and services) and deregulation or liberalization (unleashing market forces). This package effect contributes to the dramatic character of the changes associated with globalization, which serves as their shorthand description.
A good early overview why DevOps emerged (when it was still called NoOps) was made in 2012 by Mile Loukidis, who wrote historically important paper of the subject.
Adrian Cockcroft’s article about NoOps at Netflix ignited a controversy that has been smoldering for some months. John Allspaw’s detailed response to Adrian’s article makes a key point: What Adrian described as “NoOps” isn’t really. Operations doesn’t go away. Responsibilities can, and do, shift over time, and as they shift, so do job descriptions. But no matter how you slice it, the same jobs need to be done, and one of those jobs is operations. What Adrian is calling NoOps at Netflix isn’t all that different from Operations at Etsy. But that just begs the question: What do we mean by “operations” in the 21st century? If NoOps is a movement for replacing operations with something that looks suspiciously like operations, there’s clearly confusion. Now that some of the passion has died down, it’s time to get to a better understanding of what we mean by operations and how it’s changed over the years.
At a recent lunch, John noted that back in the dawn of the computer age, there was no distinction between dev and ops. If you developed, you operated. You mounted the tapes, you flipped the switches on the front panel, you rebooted when things crashed, and possibly even replaced the burned out vacuum tubes. And you got to wear a geeky white lab coat. Dev and ops started to separate in the ’60s, when programmer/analysts dumped boxes of punch cards into readers, and “computer operators” behind a glass wall scurried around mounting tapes in response to IBM JCL. The operators also pulled printouts from line printers and shoved them in labeled cubbyholes, where you got your output filed under your last name.
The arrival of minicomputers in the 1970s and PCs in the ’80s broke down the wall between mainframe operators and users, leading to the system and network administrators of the 1980s and ’90s. That was the birth of modern “IT operations” culture. Minicomputer users tended to be computing professionals with just enough knowledge to be dangerous. (I remember when a new director was given the root password and told to “create an account for yourself” … and promptly crashed the VAX, which was shared by about 30 users). PC users required networks; they required support; they required shared resources, such as file servers and mail servers. And yes, BOFH (“Bastard Operator from Hell”) serves as a reminder of those days. I remember being told that “no one” else is having the problem you’re having — and not getting beyond it until at a company meeting we found that everyone was having the exact same problem, in slightly different ways. No wonder we want ops to disappear. No wonder we wanted a wall between the developers and the sysadmins, particularly since, in theory, the advent of the personal computer and desktop workstation meant that we could all be responsible for our own machines.
But somebody has to keep the infrastructure running, including the increasingly important websites. As companies and computing facilities grew larger, the fire-fighting mentality of many system administrators didn’t scale. When the whole company runs on one 386 box (like O’Reilly in 1990), mumbling obscure command-line incantations is an appropriate way to fix problems. But that doesn’t work when you’re talking hundreds or thousands of nodes at Rackspace or Amazon. From an operations standpoint, the big story of the web isn’t the evolution toward full-fledged applications that run in the browser; it’s the growth from single servers to tens of servers to hundreds, to thousands, to (in the case of Google or Facebook) millions. When you’re running at that scale, fixing problems on the command line just isn’t an option. You can’t afford letting machines get out of sync through ad-hoc fixes and patches. Being told “We need 125 servers online ASAP, and there’s no time to automate it” (as Sascha Bates encountered) is a recipe for disaster.
The response of the operations community to the problem of scale isn’t surprising. One of the themes of O’Reilly’s Velocity Conference is “Infrastructure as Code.” If you’re going to do operations reliably, you need to make it reproducible and programmatic. Hence virtual machines to shield software from configuration issues. Hence Puppet and Chef to automate configuration, so you know every machine has an identical software configuration and is running the right services. Hence Vagrant to ensure that all your virtual machines are constructed identically from the start. Hence automated monitoring tools to ensure that your clusters are running properly. It doesn’t matter whether the nodes are in your own data center, in a hosting facility, or in a public cloud. If you’re not writing software to manage them, you’re not surviving.
Furthermore, as we move further and further away from traditional hardware servers and networks, and into a world that’s virtualized on every level, old-style system administration ceases to work. Physical machines in a physical machine room won’t disappear, but they’re no longer the only thing a system administrator has to worry about. Where’s the root disk drive on a virtual instance running at some colocation facility? Where’s a network port on a virtual switch? Sure, system administrators of the ’90s managed these resources with software; no sysadmin worth his salt came without a portfolio of Perl scripts. The difference is that now the resources themselves may be physical, or they may just be software; a network port, a disk drive, or a CPU has nothing to do with a physical entity you can point at or unplug. The only effective way to manage this layered reality is through software.
So infrastructure had to become code. All those Perl scripts show that it was already becoming code as early as the late ’80s; indeed, Perl was designed as a programming language for automating system administration. It didn’t take long for leading-edge sysadmins to realize that handcrafted configurations and non-reproducible incantations were a bad way to run their shops. It’s possible that this trend means the end of traditional system administrators, whose jobs are reduced to racking up systems for Amazon or Rackspace. But that’s only likely to be the fate of those sysadmins who refuse to grow and adapt as the computing industry evolves. (And I suspect that sysadmins who refuse to adapt swell the ranks of the BOFH fraternity, and most of us would be happy to see them leave.) Good sysadmins have always realized that automation was a significant component of their job and will adapt as automation becomes even more important. The new sysadmin won’t power down a machine, replace a failing disk drive, reboot, and restore from backup; he’ll write software to detect a misbehaving EC2 instance automatically, destroy the bad instance, spin up a new one, and configure it, all without interrupting service. With automation at this level, the new “ops guy” won’t care if he’s responsible for a dozen systems or 10,000. And the modern BOFH is, more often than not, an old-school sysadmin who has chosen not to adapt.
James Urquhart nails it when he describes how modern applications, running in the cloud, still need to be resilient and fault tolerant, still need monitoring, still need to adapt to huge swings in load, etc. But he notes that those features, formerly provided by the IT/operations infrastructures, now need to be part of the application, particularly in “platform as a service” environments. Operations doesn’t go away, it becomes part of the development. And rather than envision some sort of uber developer, who understands big data, web performance optimization, application middleware, and fault tolerance in a massively distributed environment, we need operations specialists on the development teams. The infrastructure doesn’t go away — it moves into the code; and the people responsible for the infrastructure, the system administrators and corporate IT groups, evolve so that they can write the code that maintains the infrastructure. Rather than being isolated, they need to cooperate and collaborate with the developers who create the applications. This is the movement informally known as “DevOps.”
Amazon’s EBS outage last year demonstrates how the nature of “operations” has changed. There was a marked distinction between companies that suffered and lost money, and companies that rode through the outage just fine. What was the difference? The companies that didn’t suffer, including Netflix, knew how to design for reliability; they understood resilience, spreading data across zones, and a whole lot of reliability engineering. Furthermore, they understood that resilience was a property of the application, and they worked with the development teams to ensure that the applications could survive when parts of the network went down. More important than the flames about Amazon’s services are the testimonials of how intelligent and careful design kept applications running while EBS was down. Netflix’s ChaosMonkey is an excellent, if extreme, example of a tool to ensure that a complex distributed application can survive outages; ChaosMonkey randomly kills instances and services within the application. The development and operations teams collaborate to ensure that the application is sufficiently robust to withstand constant random (and self-inflicted!) outages without degrading.
On the other hand, during the EBS outage, nobody who wasn’t an Amazon employee touched a single piece of hardware. At the time, JD Long tweeted that the best thing about the EBS outage was that his guys weren’t running around like crazy trying to fix things. That’s how it should be. It’s important, though, to notice how this differs from operations practices 20, even 10 years ago. It was all over before the outage even occurred: The sites that dealt with it successfully had written software that was robust, and carefully managed their data so that it wasn’t reliant on a single zone. And similarly, the sites that scrambled to recover from the outage were those that hadn’t built resilience into their applications and hadn’t replicated their data across different zones.
In addition to this redistribution of responsibility, from the lower layers of the stack to the application itself, we’re also seeing a redistribution of costs. It’s a mistake to think that the cost of operations goes away. Capital expense for new servers may be replaced by monthly bills from Amazon, but it’s still cost. There may be fewer traditional IT staff, and there will certainly be a higher ratio of servers to staff, but that’s because some IT functions have disappeared into the development groups. The bonding is fluid, but that’s precisely the point. The task — providing a solid, stable application for customers — is the same. The locations of the servers on which that application runs, and how they’re managed, are all that changes.
One important task of operations is understanding the cost trade-offs between public clouds like Amazon’s, private clouds, traditional colocation, and building their own infrastructure. It’s hard to beat Amazon if you’re a startup trying to conserve cash and need to allocate or deallocate hardware to respond to fluctuations in load. You don’t want to own a huge cluster to handle your peak capacity but leave it idle most of the time. But Amazon isn’t inexpensive, and a larger company can probably get a better deal taking its infrastructure to a colocation facility. A few of the largest companies will build their own datacenters. Cost versus flexibility is an important trade-off; scaling is inherently slow when you own physical hardware, and when you build your data centers to handle peak loads, your facility is underutilized most of the time. Smaller companies will develop hybrid strategies, with parts of the infrastructure hosted on public clouds like AWS or Rackspace, part running on private hosting services, and part running in-house. Optimizing how tasks are distributed between these facilities isn’t simple; that is the province of operations groups. Developing applications that can run effectively in a hybrid environment: that’s the responsibility of developers, with healthy cooperation with an operations team.
The use of metrics to monitor system performance is another respect in which system administration has evolved. In the early ’80s or early ’90s, you knew when a machine crashed because you started getting phone calls. Early system monitoring tools like HP’s OpenView provided limited visibility into system and network behavior but didn’t give much more information than simple heartbeats or reachability tests. Modern tools like DTrace provide insight into almost every aspect of system behavior; one of the biggest challenges facing modern operations groups is developing analytic tools and metrics that can take advantage of the data that’s available to predict problems before they become outages. We now have access to the data we need, we just don’t know how to use it. And the more we rely on distributed systems, the more important monitoring becomes. As with so much else, monitoring needs to become part of the application itself. Operations is crucial to success, but operations can only succeed to the extent that it collaborates with developers and participates in the development of applications that can monitor and heal themselves.
Success isn’t based entirely on integrating operations into development. It’s naive to think that even the best development groups, aware of the challenges of high-performance, distributed applications, can write software that won’t fail. On this two-way street, do developers wear the beepers, or IT staff? As Allspaw points out, it’s important not to divorce developers from the consequences of their work since the fires are frequently set by their code. So, both developers and operations carry the beepers. Sharing responsibilities has another benefit. Rather than finger-pointing post-mortems that try to figure out whether an outage was caused by bad code or operational errors, when operations and development teams work together to solve outages, a post-mortem can focus less on assigning blame than on making systems more resilient in the future. Although we used to practice “root cause analysis” after failures, we’re recognizing that finding out the single cause is unhelpful. Almost every outage is the result of a “perfect storm” of normal, everyday mishaps. Instead of figuring out what went wrong and building procedures to ensure that something bad can never happen again (a process that almost always introduces inefficiencies and unanticipated vulnerabilities), modern operations designs systems that are resilient in the face of everyday errors, even when they occur in unpredictable combinations.
In the past decade, we’ve seen major changes in software development practice. We’ve moved from various versions of the “waterfall” method, with interminable up-front planning, to “minimum viable product,” continuous integration, and continuous deployment. It’s important to understand that the waterfall and methodology of the ’80s aren’t “bad ideas” or mistakes. They were perfectly adapted to an age of shrink-wrapped software. When you produce a “gold disk” and manufacture thousands (or millions) of copies, the penalties for getting something wrong are huge. If there’s a bug, you can’t fix it until the next release. In this environment, a software release is a huge event. But in this age of web and mobile applications, deployment isn’t such a big thing. We can release early, and release often; we’ve moved from continuous integration to continuous deployment. We’ve developed techniques for quick resolution in case a new release has serious problems; we’ve mastered A/B testing to test releases on a small subset of the user base.
All of these changes require cooperation and collaboration between developers and operations staff. Operations groups are adopting, and in many cases, leading in the effort to implement these changes. They’re the specialists in resilience, in monitoring, in deploying changes and rolling them back. And the many attendees, hallway discussions, talks, and keynotes at O’Reilly’s Velocity conference show us that they are adapting. They’re learning about adopting approaches to resilience that are completely new to software engineering; they’re learning about monitoring and diagnosing distributed systems, doing large-scale automation, and debugging under pressure. At a recent meeting, Jesse Robbins described scheduling EMT training sessions for operations staff so that they understood how to handle themselves and communicate with each other in an emergency. It’s an interesting and provocative idea, and one of many things that modern operations staff bring to the mix when they work with developers.
What does the future hold for operations? System and network monitoring used to be exotic and bleeding-edge; now, it’s expected. But we haven’t taken it far enough. We’re still learning how to monitor systems, how to analyze the data generated by modern monitoring tools, and how to build dashboards that let us see and use the results effectively. I’ve joked about “using a Hadoop cluster to monitor the Hadoop cluster,” but that may not be far from reality. The amount of information we can capture is tremendous, and far beyond what humans can analyze without techniques like machine learning.
Likewise, operations groups are playing a huge role in the deployment of new, more efficient protocols for the web, like SPDY. Operations is involved, more than ever, in tuning the performance of operating systems and servers (even ones that aren’t under our physical control); a lot of our “best practices” for TCP tuning were developed in the days of ISDN and 56 Kbps analog modems, and haven’t been adapted to the reality of Gigabit Ethernet, OC48* fiber, and their descendants. Operations groups are responsible for figuring out how to use these technologies (and their successors) effectively. We’re only beginning to digest IPv6 and the changes it implies for network infrastructure. And, while I’ve written a lot about building resilience into applications, so far we’ve only taken baby steps. There’s a lot there that we still don’t know. Operations groups have been leaders in taking best practices from older disciplines (control systems theory, manufacturing, medicine) and integrating them into software development.
And what about NoOps? Ultimately, it’s a bad name, but the name doesn’t really matter. A group practicing “NoOps” successfully hasn’t banished operations. It’s just moved operations elsewhere and called it something else. Whether a poorly chosen name helps or hinders progress remains to be seen, but operations won’t go away; it will evolve to meet the challenges of delivering effective, reliable software to customers. Old-style system administrators may indeed be disappearing. But if so, they are being replaced by more sophisticated operations experts who work closely with development teams to get continuous deployment right; to build highly distributed systems that are resilient; and yes, to answer the pagers in the middle of the night when EBS goes down. DevOps.
Photo: Taken at IBM’s headquarters in Armonk, NY. By Mike Loukides.
PSYCHOLOGY OF SPIRITUAL SECTS. The psychological dynamics underlying the creation and growth of spiritual movements.
The main features
Which are the features of psychological influence most common to spiritual movements ?
- 1. Outlook/aspiration of members
- 2. Authority of leader/founder
- 3. Shared belief in scripture, revelation, or sayings.
- 4. Uniqueness of the movement.
- 5. Salvation through baptism, initiation, or conversion.
- 6. Belief in prediction/prophecy
- 7. Belief versus intellect/Secrets
- 8. Common practice of a ritual
- 9. Sacrifice. Financial secrecy, favours to the rich
- 10. Unquestioning leadership, reprehensible behaviour amongst members.
- 11. Fear of expulsion
1. Type of members.
There are many types of members, each with its own motivation.
- First of all there are the ones who have been moved to do so out of a genuine spiritual aspiration, prepared to sacrifice.
- Other members may have been searching for a truth behind the dreadful aspects of life around them and an escape therefrom.
- People who wrestle with a psychological problem seek (unconsciously) a key or solution to it in a particular teaching or discipline.
- Then there are those that did so out of a feeling of isolation and loneliness. Lack of true companionship may have made them long to be taken up in a fraternity of kindred souls.
- People will be drawn to a brotherhood whose aims are within their reach of comprehension and nearest to their hearts. A religious background may make them veer towards a sect that accomodates nostalgia, like the Jehova's Witnesses.
- For younger people it may be an escape from having to enter into a harsh materialistic society. They are in an idealistic life-cycle, prepared to give up their study and career to a worthy cause.
Some older people can never make up their minds and wander from one cause to another.
- Finally there are those who wrestle with a father/mother complex. The childhood's need for such archetypal figure may persist in later life. It is transferred subsequently to the master, who then holds sway over him/her.
The weaker the individual's independance, the more will he be tied to the group. Members who understand group-mechanisms, prepared to cope with them in order to direct their attention to the spirit, will benefit most as they are selective in picking up the cream of what is given and taking the rest with a grain of salt.
New religious movements arise usually around a father/mother figure who has gained authority after receiving a special revelation, communication, truth or insight. His charisma will vouchsafe loyal followers, even if his lifestyle may give rise to severe doubts to some. He may boost his prestige by claiming to follow the footsteps of a an esteemed spiritual teacher, represent an esoteric tradition, be of noble descent, or channel the wisdom of a great mind. (Eckankar's Paul Twitchell is the last in the lineage of 970 "Eckmasters")
He/she represents an archetype in members' subconscious minds. That of a wise father, or mother. As such he/she will have a compelling influence on followers who project their father/mother complex on him/her.
Alternatively women may fall in love with the leader, worship him, exert themselves to cater after his wishes and whims. They will try to stay in his viscinity, make themselves indispensable and slowly take control of the movement. Jealousy amongst them will make things even worse and split the ranks.
The psychological make-up of a guru may be generalized as follows:
- Difficult youth. Self-chosen isolation at childhood. Introvert. Therefore not used to share inner life with friends.
- Narcissistically absorbed in inner mental processes.
- Undergone a traumatic experience.
- Authoritarian attitude. Élitist and anti-democratic.
- Attracting disciples rather than friends on account of the fact that they never learnt to exchange thoughts in childhood.
Jeffrey Masson (see below) has this to say about gurus:
Every guru claims to know something you cannot know by yourself or through ordinary channels. All gurus promise access to a hidden reality if only you will follow their teaching, accept their authority, hand your life over to them. Certain questions are off limits. There are things you cannot know about the guru and the guru's personal life. Every doubt about the guru is a reflection of your own unworthiness, or the influence of an external evil force. The more obscure the action of the guru, the more likely it is to be right, to be cherished. Ultimately you cannot admire the guru, you must worship him. You must obey him, you must humble yourself, for the greater he is, the less you are - until you reach the inner circle and can start abusing other people the way your guru abused you. All this is in the very nature of being a guru.
Nature seems to instill in a person, faced with a mission, great task, or challenge, a feeling of superiority, unsurmountable optimism, and enormous self-esteem, bordering on an inflated ego, to accomplish what is needed. This drive is reminiscent of the reckless impetus of the adolescent. Having reached maturity a person may feel "chosen" - impelled to forge ahead with vigor and inspire others. Undaunted in the face of obstacles and criticism, it is as if a cloak of invulnerability is laid on his/her shoulders.
Similarly an artist may be driven by a compulsion to express an inner content. He will be prepared to sacrifice everything to give way to his creative impulse. Fortunately his sacrifice does not involve more than the people immediately around him.
Not so with the leader. The number of his followers may grow to considerable proportions. Nature is not concerned whether his sense of superiority has any real foundation. The inflated ego is more or less instinctively driven towards a goal.
Although attaining heights no one would have thought conceivable of that person, when the hour of truth has come events may prove that he has overreached himself, disregarded good advice, or lost complete sense of reality. The result may be either catastrophe, or the uncritical followers may be saddled up with a heritage built on quicksand - on a flight of fancy without actual foundation.
This applies to many fields of human endeavour (Hitler), but specially in the treacherous domain of the spirit.
Discipline - nauseaThe teacher may come to the conclusion that unless his followers change fundamentally - undergo a catharsis, or transformation - they will never be able to move forward. He/she regards them as being "asleep" (Jesus, Gurdjieff). Unless drastic measures are employed they will not wake up. To jolt them out of their complacency great sacrifices are demanded. Jesus asked a rich young man to give up all his worldly possessions (S.Matthew 19:21) before following him. Masters in Zen Buddhism, or Gurdjieff, made novices undergo a harsh regime in order to crack open and attain a different state of mind.
This I can have no quarrel with, if it is done against a background of compassion. If the unselfish motive disappears, or commercial considerations become dominant, the harsh discipline may become morbid and degrading. Having lost his dedication the teacher may become nauseated by the mentality and sheepishness of his followers, and in cases derive a sadistic delight in tormenting them.
In recent years reports are brought out about sexual violation of members by guru's, leaders and....bishops! Another example of authority being abused.
The path of a guru is like that of a razor's edge. He may so easily succumb to the temptation of exploiting the power he has attained over his followers. Financial irresponsibility, abuse of followers, reprehensible sexual behaviour......... mass suicide, it is all within his reach once he has overstepped boundaries.
During his lifetime the leader will act as a moderator and steer the movement. He will re-interprete his teachings as he sees fit from the responses he receives. The death of the founder marks a turning point. His teachings will become inflexible, as no one dares to temper with them as he did himself. The élan disappears, rigidity takes over, unless another figure arises that leads the movement in a different direction, for better or for worse (St.Paul).
The more secret(ive) the leader's sayings the better. Pronouncements are characterized by great certainty and authority as if it were the word of God. In some cases it is presented as such. By his special way of delivery and presentation it may escape the audience that similar wisdom may be found in any book on spirituality nowadays found in the bookshop around the corner.
Whether the guru bases his wise words on actual experience or on hearsay is difficult to ascertain. In general it may be said that the more mystifying his teachings the stronger their appeal. After all it is beyond reason and should appeal only to the heart.
An exception should be made for true mystical literature based on inner experience which can hardly be expected to appeal to the intellect, but be appreciated intuitively, especially by those who had similar experiences.
Members may adopt fresh meanings to words, talk to each other in a jargon that the outsider can hardly follow (group-speak). The result being an inability to relate in speech, or explain new concepts to the outsider (Fourth Way).
(This may be best understood in other fields: help-programs of software, pop-up windows, warning-messages, not to speak of manuals for installing hardware, drawn up by boffins, are a nightmare to most users!)
Another characteristic is to lift out of context one aspect of religious truth and make it absolute. Such key truth will overshadow all other aspects of faith.
It may be:
etc. When this occurs other significant facets of faith are pushed to the background.
- prophecy, as we have seen with Jehova's Witnesses, Adventists and countless sects;
- baptism at an adult age, important to Baptists;
- charity in the case of the Salvation Army (but who could blame them for this?);
- channeling in New Age/Spiritualistic movements,
Such partial truths are often heralded as the result of a search for knowledge. The motto "Knowledge is power" is used to suggest that the statements are objective, scientific, or historical facts. Actually they cannot stand the touchstone of the merest critical scrutiny.
Authorities may be paraded to back-up such claims. They have either never been heard of, cannot be considered impartial, or their pronouncements have been lifted out of context. The discussion about the veracity of evolution is full of such red herrings.
4. Uniqueness of the movement
Movements will extol usually their superiority over others. After all there should be a strong reason to select that particular group. Some present themselves as being the sole way towards salvation, being God's chosen people. Others make a promise of a benefit that is only reserved for members of that sect. To avert attention some pride themselves of not having a teaching, or for their openness and democratic rules.
In short new movements will advance a variety of reasons for their uniqueness. Herewith a few:
- Our holy book is the word of God.
- Never before has mankind been offered this discipline/interpretation/insight of our leader.
- Do not pour new wine in old bottles.
- God's, or a celestial, new message to mankind for this particular time.
- Impending disaster (pollution, atomic explosion, collision) calls for drastic measures. Salvation only reserved for faithful members.
- Fresh interpretation of holy book thanks to fresh insight/revelation of founder.
- Esoteric tradition, once accessible to adepts only, now revealed.
- New channel for teachings of esoteric lodge/brotherhood.
- New doctrine/insight based on latest scientific discoveries, reveals truth.
- Only those following this particular work on self, discipline, or belief, will reach eternity, be released from earth's satanic attraction, cycle of rebirths, etc.
- Preparatory group to make way for the coming of the new messiah/world-teacher/avatar
Noteworthy is the vehemency with which groups stress differences between each other. The closer movements share an outlook the more virulent the attacks on their rivals become, seemingly more than on groups which follow a completely different belief.
Eric Hoffer writes in his 'The True Believer': "true believers of various hues ....view each other with mortal hatred and are ready to fly at each other's throat..."
This manifests itself specially when groups split. In Christianity one could not steep low enough to attack other followers of Christ, who held a slightly different opinion. It resulted in persecution of heretics, burning of early Christian literature, and disastrous wars.
Despite their peaceful appearance relatively new spiritual movements like Theosophy, Rosicrucianism, etc., following splits, exert themselves in accusations against former comrades.
Attacks against belief in paranormal phenomena, for instance by CSICOP, are reminiscent of the zeal of a Christian crusade, be it that they have their roots in humanism and its desperately clinging to a rationalistic/materialistic outlook on life current at the beginning of this century. Consequently the groups of these 'evangelists of rational enlightenment' have similar behavioural patterns and vehemency as sects.
5. Probation and conversion
Certain sects are only too eager to accept individuals. They may have high entrance fees. Or their members are swayed by zeal to convert.
Many movements will put up a barrier by means of an initiation to test the applicant's fitness to become part of the group. Henceforth they will play an important pioneer-part in the foretold future. Having reached such coveted stage members will not fail to follow what they are being told for fear of expulsion.
The new member may undergo a conversion, gaining a completely new insight in the meaning of life, see it in a way the sect does. His previous life with all its relationships has become meaningless. He may have turned himself inside out by a confession of his previous "sins". His conversion is marked by a feeling of peace, happiness and transcendence.
6. Failure of predictions
Common belief in a prophecy will be a strong binding force. One of the principal attractions of the first Christian sects was that they offered salvation from a threatening disaster. That being the end of the world. Only the baptized would await a glorious future. Sects like the Jehova's Witnesses have taken over this succesful formula.
Christians have had to come up with all sorts of arguments to explain away the unfulfilled prediction of their founder regarding the end of the world: "This generation shall not pass away, till all these things be accomplished." (S.Matthew 24:34). One of the lame excuses being that this prediction concerns the fall of Jersusalem only. However, all prophecies in the New Testament in this respect suggest that the impending doom was to be expected in their lifetime.
Jehova's Witnesses have taken the risk of being more specific in their predictions. Older members, who built their faith on them, have had the humiliating experience of having had to explain away various times in their lives the failure of the outcome of their forewarnings.
But predictions are not limited to the religious faiths. The New Age movements use this shared belief in portents as well. For more than sixty years an imminent landing of UFO's has been predicted. Various cults claimed in vain to be their first contactees.
In other movements the second coming of Christ was a main feature (Benjamin Creme). In Theosophy a Messenger was expected from 1975 onward.
The uncritical believers in Edgar Cayce's trance sayings put weight on his predictions of cataclysms (photo Edgar Cayce).
Nostradamus' (photo) obscure astrological foresayings have captured the minds of people for centuries. Each time his verses were interpreted again to suit the circumstances. In hindsight some of his quatrains seem to have relevance to the catastrophe of the destroyed World Trade Center. Quatrains I, 87 - IX, 92 and X, 59 may refer to skyscrapers in New York involved in a terrible explosion.
Sociologists have observed that, failure of prediction results in quite the opposite effect on believers. Contrary to what one would expect it may cause a rally amongst members. Failure is blamed on a misunderstanding, or a faux pas by members. To counteract ridicule they tend to stick together more than ever.
Of course there is a limit. According to a social survey, when predictions fail to materialize three times in a row members are bound to stop, reflect and draw conclusions.
The shattering of such false hopes comes as a severe blow and may mark the beginning of the end of a movement.
One wonders in this respect how many members of the People Forever International sect promoting physical immortality for its followers would have to die before their groups would break up in disappointment. (Since I wrote this ten years ago I have been informed that indeed members have died and the movement broke up in 1998!)
Yet, we see from the Jehova's Witnesses that skilful manoeuvring may off-set unfulfilled prophecies.
To what extremes such believes can lead shows the mass suicidal action of the Heaven's Gate sect and later in Uganda. Such tragic endings are the result of various contributing factors, which are beyond the scope of this article.
7. Belief versus intellect/Secrets
Often disciplines followed in spiritual movements have the effect of a lowering of the threshold to the unconscious mind. Suggestion will begin to play an important part. Precepts are being experienced as the truth, sacrosanct and sure. There is no element of doubt anymore about assumptions and speculation, although actually they lack any factual foundation.
Absolute belief that the Bible is God's word is the cornerstone of most orthodox Christian sects. In Islam the Koran is supposed to contain the word of Allah.
Intellectual analysis of faith is tentamount to heresy.
The ideal breeding ground for convictions are mass-gatherings. During mass gatherings, such as congresses, members are stirred up to an euphoria, the effect of which may linger on for weeks. This is the precise period of time for leaders, or committees, to announce fresh sectarian measures, postulate incredible notions/prophecies, call for further sacrifices, etc. etc. It will all be accepted unquestioningly. Only at a later date, when the euphoria has worn off, will one start to wonder about what was decided.
Spiritual movements often hide a corpse in their closet. It may be a part of the history of the movement, details about the hidden life of the leader, or a once revered figure. Things may have been written by them one does not like to be reminded of. A fight, quarrel, full of vehemence and hatred, may have led to a split.
There are so many examples that a long list could be drawn up of the many concealed secrets of spiritual groups.
Whereas in most movements the works of the leader are almost known by heart, Jehova's Witnesses hardly know of the existence of the seven volumes of writings Studies in the Scriptures of their founder Charles Taze Russell (1852-1916). Some of his opinions are such cause of embarrassment that they are not deemed worth reading nowadays.
Eventually a renegate member will reveal such secrets in writing. Frantic denials and counter accusations by those in charge presently will follow almost automatically. These are usually accepted in gratitude by devotees, who cannot get over the shock of such revelations.
8. Common practice, work and ritual
Communal singing, ritual and (incomprehensible) practices (Freemasonry) are strong binding factors. The more irrational they are, the better. Others are a special food regime, the change of name, clothing, or a common aversion.
Joint work for the benefit of the group gives the feeling of a common endeavour and unites the participants. So does proselytization in the streets, menial work of construction and renovation of premises. There is a thin line between true participation and exploitation, however.
Dubious was the practice, common in the seventies, to incite members to criticize one of them to an extent that he/she would break down under the weight of often absurd allegations and insults, resulting in a brainwash effect.
9. Sacrifices, financial secrecy, favours to the rich.
Finances are always a ticklish matter. Human groups always wish to grow. Finances are important. Accountability is often not considered appropriate. Danger arises that members of the inner circle become lax in expenditure of members' contributions. Ambitious schemes call for a constant need for funding. This is the ideal breeding ground for favours to wealthy members. Those who contribute generously stand more chance to be taken in confidence and admitted to the inner circles. Often, as a proof of loyalty, extraordinary sums of money are demanded.
Degrees of initiation may be dependent on one's years of loyalty to the group. In Eckankar up to 8 degrees are given. However, if one fails to pay membership's fees for some time, degrees of initiation may be stripped off again.
Next to financial contributions members will often be expected to offer services to the group. However, if they also have to work for practically nothing in commercial enterprises it becomes dubious. Movements that gather wealth at the expense of their members are questionable. Seldom or never requests for return of contributions/investments are honoured.
10. Unquestioning leadership, reprehensible behaviour amongst members
Man in a herd may not show the best side of his nature. Unconscious drives may reign his/her behaviour. This is applicable especially in circumstances that man strives for the spiritual. Heshe may tend to show split-personality behaviour. On one hand the spiritual personality which is supposed to have come to terms with his animal nature. It is wise, friendly and compassionate on the outside. In the shadows lurks the personality that has been forced into the background, still ridden with all the expulsed human frailties. In moments of weakness it will see its chance to play hideous tricks. It will do so without being noticed by the person involved. The result being: uncharitable behaviour, envy, malicious gossip, hypocrisy, harsh words, insensitivity, unfounded criticism and even worse, not expected from such charismatic figure. It is one of the main reasons for people leaving a particular group in great disappointment.
It is not often realized that, like other human groups, spiritual movements behave like organisms. Group-psychological processes manifest which are sometimes not unlike those in primitive societies. There is the pecking order, the alpha members, and also the group-instinct directed against similar groups. Aggression goes unnoticed and is tolerated when an acceptable common goal is provided. For instance hostility against an individual outside the group, or a critical member inside. This has the effect of strengthening ties within the group like in the animal world.
If leadership loses contact with its members it will have to exert greater discipline. Deviating opinions cannot be tolerated anymore. Persons who hold them are seen as traitors. Acting against them, preferably in secret, is the only way out for the leadership to avert this danger. Members may disappear suddenly without the reasons becoming known, much to the surprise of those left behind. For such machinations in Theosophy read Emily Lutyens: "Candles in the Sun".
Spiritual newsgroups on Internet provide illustration of (un)concious nastiness being ventilated under the veil of anonimity. Messages are often rife with diatribe, personal attacks and misunderstanding. Many of such contributors have no interest at all in the matters discussed. Yet even in closed newsgroups, only open to subscribers, complaints about the tone of communications are being aired.
11. Fear of exclusion
The stronger members are tied to a group, the more the fear of exclusion lurks. They may have invested their life's savings in the work (Scientology), paid a percentage of their income, failed to conclude their study, or make a career, or sacrificed a succesful one.
In many cases a member will have alienated himself from family and friends. He has been told to cut ties with the past. (In the Attleboro cult followers are advised to burn photographs that remind them of bygone days). No wonder his or her sudden conversion, accompanied by fanatism and urge to proselytize, has shied away former friends and relatives.
There is no way left but to seek comfort and understanding with members of the spiritual group.
Isolation is sometimes intentionally sought. Formerly, in the Bhagavan Shri Rajneesh movement, members went about in red/orange dresses and wore mala's with a photo of their master, so setting themselves aside from the mundane world.
The Hare Krishna movement goes even further. Groups of members go out into the streets in their oriental dresses for song and dance routines. However, in most movements the alienation is far more subtle and the natural outcome of an adverse attitude towards the materialism of society.
The true nature of the so-called friendships within the group will only be revealed after a devotee has left the fold. Members have seen this happen, but did not give it a thought at the time, because it happened to someone else. But when they undergo the same fate themselves they will feel the humiliation of being ignored, not being greeted anymore, marriage gone - even not being recognized by one's own children anymore.
The outcast feels thrown in an abyss. He is cut off from social contacts, his life in pieces. The magnitude of this desperate experience should not be under-estimated. The renegade will feel deep shame. He may have confessed in the group intimate secrets, which are now being ridiculed by his former so-called friends. The expulsee, deeply hurt, may become embittered and even enter into a suicidal mental state.
Those readers who have been a member of a movement may recognize some of the above psychological mechanisms. The first reaction of non-members may be to vow never to enter a group. Let us bear in mind, however, that it should be considered a challenge to face these obstacles for the benefit that may result from association with kindred spirits.
A prerequisite is that these conditions are being noticed, looked in the eye, and not denied. The closer people live together, the more group-tensions will build up. Even in reputable circles as Freudian psycho-analytical associations they occur. Few communes are granted a long life as a result of one or more of the pitfalls summarized above. Headquarters, contrary to expectations, are known to be hotbeds of gossip, mutual repulsion and cynism.
So, do not be disheartened and join a group of your liking. After all people who marry also see wrecked marriages all around them, yet go ahead intent on a happy union in mutual trust, without regard to the outcome.
Involvement with other people will lead to personal growth if the consequences are anticipated. The more one stands on one's own feet the more benefit will arise from cooperating with others. It should be borne in mind that the saying "It is better to give, than to receive" is not merely a moral precept. (Read my precepts for living)
Please remember that there are hundreds of movements and that it has not been my intention to summarize them all, or to level any form of criticism at one of them. Indicating the psychological mechanisms operative in some, or all of them, has been my main theme.
On a separate page I have gone into the mysterious presence-phenomenon arising between people who meet in harmony.
In conclusion one may take heed of Krishnamurti's words in 1929 when he refused to become a 'World Teacher' of an organisation set up for him:
I maintain that Truth is a pathless land, and you cannot approach it by any path whatsoever, by any religion, by any sect. Truth being limitless, unconditioned, cannot be organised, nor should any organisation be formed to lead or coerce people along any particular path. If you first understand that, then you will see how impossible it is to organize a belief. A belief is purely an individual matter, and you cannot and must not organize it. If you do, it becomes dead, crystallised; it becomes a creed, a sect, a religion, to be imposed on others.
© Michael Rogge, 2011
Oct 26, 2017 | www.amazon.com
Mohammad B. Abdulfatah on February 10, 2003Programming Malpractice Explained: Justifying Chaoswiredweird HALL OF FAME TOP 1000 REVIEWER on May 24, 2004
To fairly review this book, one must distinguish between the methodology it presents and the actual presentation. As to the presentation, the author attempts to win the reader over with emotional persuasion and pep talk rather than with facts and hard evidence. Stories of childhood and comradeship don't classify as convincing facts to me.
A single case study-the C3 project-is often referred to, but with no specific information (do note that the project was cancelled by the client after staying in development for far too long).
As to the method itself, it basically boils down to four core practices:
- Always have a customer available on site.
- Unit test before you code.
- Program in pairs.
- Forfeit detailed design in favor of incremental, daily releases and refactoring.
If you do the above, and you have excellent staff on your hands, then the book promises that you'll reap the benefits of faster development, less overtime, and happier customers. Of course, the book fails to point out that if your staff is all highly qualified people, then the project is likely to succeed no matter what methodology you use. I'm sure that anyone who has worked in the software industry for sometime has noticed the sad state that most computer professionals are in nowadays.
However, assuming that you have all the topnotch developers that you desire, the outlined methodology is almost impossible to apply in real world scenarios. Having a customer always available on site would mean that the customer in question is probably a small, expendable fish in his organization and is unlikely to have any useful knowledge of its business practices.
Unit testing code before it is written means that one would have to have a mental picture of what one is going to write before writing it, which is difficult without upfront design. And maintaining such tests as the code changes would be a nightmare.
Programming in pairs all the time would assume that your topnotch developers are also sociable creatures, which is rarely the case, and even if they were, no one would be able to justify the practice in terms of productivity. I won't discuss why I think that abandoning upfront design is a bad practice; the whole idea is too ridiculous to debate.
Both book and methodology will attract fledgling developers with its promise of hacking as an acceptable software practice and a development universe revolving around the programmer. It's a cult, not a methodology, were the followers shall find salvation and 40-hour working weeks.
Experience is a great teacher, but only a fool would learn from it alone. Listen to what the opponents have to say before embracing change, and don't forget to take the proverbial grain of salt.
Two stars out of five for the presentation for being courageous and attempting to defy the standard practices of the industry. Two stars for the methodology itself, because it underlines several common sense practices that are very useful once practiced without the extremity.eXtreme buzzwordingA customer on May 2, 2004
Maybe it's an interesting idea, but it's just not ready for prime time.
Parts of Kent's recommended practice - including aggressive testing and short integration cycle - make a lot of sense. I've shared the same beliefs for years, but it was good to see them clarified and codified. I really have changed some of my practice after reading this and books like this.
I have two broad kinds of problem with this dogma, though. First is the near-abolition of documentation. I can't defend 2000 page specs for typical kinds of development. On the other hand, declaring that the test suite is the spec doesn't do it for me either. The test suite is code, written for machine interpretation. Much too often, it is not written for human interpretation. Based on the way I see most code written, it would be a nightmare to reverse engineer the human meaning out of any non-trivial test code. Some systematic way of ensuring human intelligibility in the code, traceable to specific "stories" (because "requirements" are part of the bad old way), would give me a lot more confidence in the approach.
The second is the dictatorial social engineering that eXtremity mandates. I've actually tried the pair programming - what a disaster. The less said the better, except that my experience did not actually destroy any professional relationships. I've also worked with people who felt that their slightest whim was adequate reason to interfere with my work. That's what Beck institutionalizes by saying that any request made of me by anyone on the team must be granted. It puts me completely at the mercy of anyone walking by. The requisite bullpen physical environment doesn't work for me either. I find that the visual and auditory distraction make intense concentration impossible.
I find revival tent spirit of the eXtremists very off-putting. If something works, it works for reasons, not as a matter of faith. I find much too much eXhortation to believe, to go ahead and leap in, so that I will eXperience the wonderfulness for myself. Isn't that what the evangelist on the subway platform keeps saying? Beck does acknowledge unbelievers like me, but requires their exile in order to maintain the group-think of the X-cult.
Beck's last chapters note a number of exceptions and special cases where eXtremism may not work - actually, most of the projects I've ever encountered.
There certainly is good in the eXtreme practice. I look to future authors to tease that good out from the positively destructive threads that I see interwoven.A work of fictionA customer on November 14, 2002
The book presents extreme programming. It is divided into three parts:
(1) The problem
(2) The solution
(3) Implementing XP.
The problem, as presented by the author, is that requirements change but current methodologies are not agile enough to cope with this. This results in customer being unhappy. The solution is to embrace change and to allow the requirements to be changed. This is done by choosing the simplest solution, releasing frequently, refactoring with the security of unit tests.
The basic assumption which underscores the approach is that the cost of change is not exponential but reaches a flat asymptote. If this is not the case, allowing change late in the project would be disastrous. The author does not provide data to back his point of view. On the other hand there is a lot of data against a constant cost of change (see for example discussion of cost in Code Complete). The lack of reasonable argumentation is an irremediable flaw in the book. Without some supportive data it is impossible to believe the basic assumption, nor the rest of the book. This is all the more important since the only project that the author refers to was cancelled before full completion.
Many other parts of the book are unconvincing. The author presents several XP practices. Some of them are very useful. For example unit tests are a good practice. They are however better treated elsewhere (e.g., Code Complete chapter on unit test). On the other hand some practices seem overkill. Pair programming is one of them. I have tried it and found it useful to generate ideas while prototyping. For writing production code, I find that a quiet environment is by far the best (see Peopleware for supportive data). Again the author does not provide any data to support his point.
This book suggests an approach aiming at changing software engineering practices. However the lack of supportive data makes it a work of fiction.
I would suggest reading Code Complete for code level advice or Rapid Development for management level advice.Not Software Engineering.Philip K. Ronzone on November 24, 2000
Any Engineering discipline is based on solid reasoning and logic not on blind faith. Unfortunately, most of this book attempts to convince you that Extreme programming is better based on the author's experiences. A lot of the principles are counterintuitive and the author exhorts you just try it out and get enlightened. I'm sorry but these kind of things belong in infomercials not in s/w engineering.
The part about "code is the documentation" is the scariest part. It's true that keeping the documentation up to date is tough on any software project, but to do away with documentation is the most ridiculous thing I have heard.
It's like telling people to cut of their noses to avoid colds. Yes we are always in search of a better software process. Let me tell you that this book won't lead you there.The "gossip magazine diet plans" style of programming.A customer on August 11, 2003
This book reminds me of the "gossip magazine diet plans", you know, the vinegar and honey diet, or the fat-burner 2000 pill diet etc. Occasionally, people actually lose weight on those diets, but, only because they've managed to eat less or exercise more. The diet plans themselves are worthless. XP is the same - it may sometimes help people program better, but only because they are (unintentionally) doing something different. People look at things like XP because, like dieters, they see a need for change. Overall, the book is a decently written "fad diet", with ideas that are just as worthless.Hackers! Salvation is nigh!!Ian K. VINE VOICE on February 27, 2015
It's interesting to see the phenomenon of Extreme Programming happening in the dawn of the 21st century. I suppose historians can explain such a reaction as a truly conservative movement. Of course, serious software engineering practice is hard. Heck, documentation is a pain in the neck. And what programmer wouldn't love to have divine inspiration just before starting to write the latest web application and so enlightened by the Almighty, write the whole thing in one go, as if by magic? No design, no documentation, you and me as a pair, and the customer too. Sounds like a hacker's dream with "Imagine" as the soundtrack (sorry, John).
The Software Engineering struggle is over 50 years old and it's only logical to expect some resistance, from time to time. In the XP case, the resistance comes in one of its worst forms: evangelism. A fundamentalist cult, with very little substance, no proof of any kind, but then again if you don't have faith you won't be granted the gift of the mystic revelation. It's Gnosticism for Geeks.
Take it with a pinch of salt.. well, maybe a sack of salt. If you can see through the B.S. that sells millions of dollars in books, consultancy fees, lectures, etc, you will recognise some common-sense ideas that are better explained, explored and detailed elsewhere.Long have I hated this bookAnil Philip on March 24, 2005
Kent is an excellent writer. He does an excellent job of presenting an approach to software development that is misguided for anything but user interface code. The argument that user interface code must be gotten into the hands of users to get feedback is used to suggest that complex system code should not be "designed up front". This is simply wrong. For example, if you are going to deploy an application in the Amazon Cloud that you want to scale, you better have some idea of how this is going to happen. Simply waiting until your application falls over and fails is not an acceptable approach.
One of the things I despise the most about the software development culture is the mindless adoption of fads. Extreme programming has been adopted by some organizations like a religious dogma.
Engineering large software systems is one of the most difficult things that humans do. There are no silver bullets and there are no dogmatic solutions that will make the difficult simple.not found - the silver bullet
Maybe I'm too cynical because I never got to work for the successful, whiz-kid companies; Maybe this book wasn't written for me!
This book reminds me of Jacobsen's "Use Cases" book of the 1990s. 'Use Cases' was all the rage but after several years, we slowly learned the truth: Uses Cases does not deal with the architecture - a necessary and good foundation for any piece of software.
Similarly, this book seems to be spotlighting Testing and taking it to extremes.
'the test plan is the design doc'
Not True. The design doc encapsulates wisdom and insight
a picture that accurately describes the interactions of the lower level software components is worth a thousand lines of code-reading.
Also present is an evangelistic fervor that reminds me of the rah-rah eighties' bestseller, "In Search Of Excellence" by Peters and Waterman. (Many people have since noted that most of the spotlighted companies of that book are bankrupt twenty five years later).
- - in a room full of people with a bully supervisor (as I experienced in my last job at a major telco) innovation or good work is largely absent.
- - deploy daily - are you kidding? to run through the hundreds of test cases in a large application takes several hours if not days. Not all testing can be automated.
- - I have found the principle of "baby steps", one of the principles in the book, most useful in my career - it is the basis for prototyping iteratively. However I heard it described in 1997 at a pep talk at MCI that the VP of our department gave to us. So I dont know who stole it from whom!
Lastly, I noted that the term 'XP' was used throughout the book, and the back cover has a blurb from an M$ architect. Was it simply coincidence that Windows shares the same name for its XP release? I wondered if M$ had sponsored part of the book as good advertising for Windows XP! :)
Feb 01, 2028 | www.itskeptic.orgThe change madness is getting worse with every passing year .
The demands for change being placed on corporate IT are plain ridiculous. As a consequence we are breaking IT. In pursuit of absurd project commitments we are eating ourselves .
And the hysteria reaches fever pitch as people extrapolate trends into the future linearly or worse still exponentially. This is such bad scientific thinking that it shouldn't be worthy of debate, but the power of critical thought is a scarce resource
Rob England (The IT Skeptic) -> Dierdre Popov , March 7, 2013 3:43 AMRob England (The IT Skeptic) -> Simon Kent , February 28, 2013 5:06 AM
A broken management and governance system, a broken value system, and a broken culture.
But even in the best and healthiest organisations, there are plenty of rogues; psychopaths (and milder sociopaths) who are never going to care about anyone but themselves. They soar in management (and they're drawn to the power); they look good to all measures and controls except a robust risk management system - it is the last line of defense.Fustbariclation , February 27, 2013 10:03 PM
...I'm saying there is a real limit to how fast humans can change: how fast we can change our behaviours, our attitudes, our processes, our systems. We need to accept that the technology is changing faster than society, our IT sector, our organisations, our teams, ourselves can change.
I'm saying there is a social and business backlash already to the pace of change. We're standing in the ruins of an economy that embraced fast change.
I'm saying there are real risks to the pace of change, and we currently live in a culture that thinks writing risks down means you can then ignore them, or that if you can't ignore them you can always hedge them somehow.
We have to slow down a bit. perhaps "Slow IT" is the wrong name but it was catchy. I'm not saying go slooooow. We've somehow sustained a pretty impressive pace for decades. But clearly it can't go much faster, if at all, and all these demands that it must go faster are plain silly. It just can't. There's bits falling off, people burning out, smoking shells of projects everywhere.
I'm not saying stop, but I am saying ease off a little, calm down, stop panicking, stop this desperate headlong rush. You are right Simon that mindfulness is a key element: we all need time to think. Let the world keep up.Wdpowel , March 14, 2013 10:06 AM
Yes, Rob, short-termism is certainly bad news, and rushing to achieve short-term goals without thinking about them in the larger context is a good indication of disaster ahead.
It's easy to mistake activity for progress.
Much of the zeitgeist that drives the frenzy you describe is generated by vendors especially those with software in their portfolio. Software has more margin that hardware or service. As a result they have more marketing budget. With that budget they invest and spend a lot of time and effort to figure out exactly how to generate the frenzy with a new thing that you must have. They have to do this to keep market interest in the products. That is actually what their job is.
The frenzy is deliberately and I would say almost scientifically engineered by very very bright marketing people in software vendors. Savvy IT organizations are aware of that distinction and maintain their focus on enabling their business to be successful. IT as Utility, On Demand, SOA, Cloud, ..... Software vendors will not and should not stop doing that - that is what keeps them in business and generates profits that enable new innovation. The onus is on the buyer to understand that whatever the latest technology is, does not provide the answer for how they will improve business performance. Improving business performance is the burden that only the organization can bear.
Aug 29, 2017 | insights.hpe.comThe quickie guide to continuous delivery in DevOps
In today's world, you have to develop and deliver almost in the same breath. Here's a quick guide to help you figure out which continuous delivery concepts will help you breathe easy, and which are only hot air. Developers are always under pressure to produce more and release software faster, which encourages the adoption of new concepts and tools. But confusing buzzwords obfuscate real technology and business benefits, particularly when a vendor has something to sell. That makes it hard to determine what works best -- for real, not just as a marketing phrase -- in the continuous flow of build and deliver processes. This article gives you the basics of continuous delivery to help you sort it all out.
To start with, the terms apply to different parts of the same production arc, each of which are automated to different degrees:
- Continuous integration means frequently merging code into a central repository . "Frequently" means usually several times a day. Each merge triggers an automated "build and test" instance, a process sometimes called continuous build . But by either name, continuous integration and continuous build do nothing in terms of delivery or deployment. They're about code management, not what happens to the code afterward.
- Continuous delivery refers to the automation of the software release process , which includes some hands-on effort by developers. Usually, developers approve or initiate the deployment, though there can be other manual steps as well.
- Continuous deployment is continuous delivery with no manual steps for developers. The whole thing is automated, and it requires not so much as a nod from humans.
With continuous deployment, "a developer's job typically ends at reviewing a pull request from a teammate and merging it to the master branch," explains Marko Anastasov in a blog post . "A continuous integration/continuous deployment service takes over from there by running all tests and deploying the code to production, while keeping the team informed about [the] outcome of every important event."
However, knowing the terms and their definitions isn't enough to help you determine when and where it is best to use each. Because, of course, every shop is different.
It would be great if the market clearly distinguished between concepts and tools and their uses, as they do with terms like DevOps. Oh, wait.
"DevOps is a concept, an idea, a life philosophy," says Gottfried Sehringer, chief marketing officer at XebiaLabs , a software delivery automation company. "It's not really a process or a toolset, or a technology."
But, alas, industry terms are rarely spelled out that succinctly. Nor are they followed with hints and tips on how and when to use them. Hence this guide, which aims to help you learn when to use what.Choose your accelerator according to your need for speed
But wait -- Isn't speed the key to all software development? These days, companies routinely require their developers to update or add features once per day, week, or month. This was unheard of back in the day, even in the era of agile software development .
That's not the end of it; some businesses push for software updates to be faster still. "If you work for Amazon, it might be every few seconds," says Sehringer.
Even if you're a deity of software bug-squashing, how can you -- or any developer or operations specialist -- deliver high-quality, "don't break anything" code when you have to build and release that fast? Everyone has their own magic bullet. "Agile -- " cries one crowd. " Continuous build -- " yells another. " Continuous integration -- " cheers a third.
Let's just cut to the chase on all that, shall we?
"Just think of continuous as 'automated,'" says Nate Berent-Spillson, senior delivery director at Nexient , a software services provider. "Automation is driving down cost and the time to develop and deploy."
Well, frack, why don't people just say automation?
Add to the idea of automation the concepts of continuous build, continuous delivery, continuous everything, which are central to DevOps, and we find ourselves talking in circles. So, let's get right to sorting all that out.
... ... ...Rinse. Repeat, repeat, repeat, repeat (the point of automation in DevOps)
Automation has obvious returns on investment. "You can make sure it's good in pre-production and push it immediately to production without breaking anything, and then just repeat, repeat, repeat, over and over again," says Sehringer.
In other words, you move delivery through all the steps in a structured, repeatable, automated way to reduce risk and increase the speed of releases and updates.
In an ideal world, you would push a button to release every few seconds," Sehringer says. But this is not an ideal world, and so people plug up the process along the way.
A company may need approval for an application change from its legal department. "Some companies are heavily regulated and may need additional gates to ensure compliance," notes Sehringer. "It's important to understand where these bottlenecks are." The ARA software should improve efficiencies and ensure the application is released or updated on schedule.
"Developers are more familiar with continuous integration," he says. "Application release automation is more recent and thus less understood."
... ... ...
Pam Baker has written hundreds of articles published in leading technology, business and finance publications including InformationWeek, Institutional Investor magazine, CIO.com, NetworkWorld, ComputerWorld, IT World, Linux World, and more. She has also authored several analytical studies on technology, eight books -- the latest of which is Data Divination: Big Data Strategies -- and an award-winning documentary on paper-making. She is a member of the National Press Club, Society of Professional Journalists and the Internet Press Guild.
Aug 28, 2017 | developers.slashdot.org
Anonymous Coward writes:xbytor ( 215790 ) , Sunday August 27, 2017 @04:00PM ( #55093989 ) Homepage
Re: Score: , Insightful)
Yeah, this is an incredibly low quality article. It doesn't specify what it means by what AI should do, doesn't specify which type of AI, doesn't specify why AI should be used, etc. Junk article.
It's basically a bullshit bingo post where someone repeats a buzzword without any knowledge of the material behind it.buzzwords Score: , Funny)cyber-vandal ( 148830 ) writes:
> a new paradigm shift.
I stopped reading after this.Re: buzzwordssycodon ( 149926 ) , Sunday August 27, 2017 @04:10PM ( #55094039 )
Not enough leveraging core competencies through blue sky thinking and synergistic best of breed cloud machine learning for you?Same Old Thing Score: , Insightful)AmazingRuss ( 555076 ) writes:
Continuous integration Prototyping Incremental development Rapid application development Agile development Waterfall development Spiral development
Now, introducing, "Continuous Delivery"...or something.
Here is the actual model, a model that will exist for the next 1,000 years.
1. Someone (or something) gathers requirement. 2. They get it wrong. 3. They develop the wrong thing that doesn't even work they way they thought it should 4. The project leader is canned 5. The software is implemented by an outside vendor, with all the flaws. 6. The software operates finally after 5 years of modifications to both the software and the workflows (to match the flaws in the software). 7. As soon as it's all running properly and everyone is trained, a new project is launched to redo it, "the right way". 8. Goto 1Re:ColdWetDog ( 752185 ) writes:
If everyone is stupid, no one is.Re:Graydyn Young ( 2835695 ) writes:
No no. We got rid of line numbers a long time ago.Re:Tablizer ( 95088 ) writes:
+1 DepressingAI meets Hunger GamesTheStickBoy ( 246518 ) writes:
It's a genetic algorithm where YOU are the population being flushed out each cycle.Re:alvinrod ( 889928 ) , Sunday August 27, 2017 @04:15PM ( #55094063 )Here is the actual model, a model that will exist for the next 1,000 years.
1. Someone (or something) gathers requirement. 2. They get it wrong. 3. They develop the wrong thing that doesn't even work they way they thought it should 4. The project leader is canned 5. The software is implemented by an outside vendor, with all the flaws. 6. The software operates finally after 5 years of modifications to both the software and the workflows (to match the flaws in the software). 7. As soon as it's all running properly and everyone is trained, a new project is launched to redo it, "the right way". 8. Goto 1
You just accurately described a 6 year project within our organization....and it made me cry Does this model have a name? an urban dictionary name? if not it needs one.Re:buzzwords Score: , Insightful)phantomfive ( 622387 ) writes:
Yeah, maybe there's something useful in TFA, but I'm not really inclined to go looking based on what was in the summary. At no point, did the person being quoted actually say anything of substance.
It's just buzzword soup with a dash of new technologies thrown in.
Five years ago they would have said practically the same words, but just talked about utilizing the cloud instead of AI.
I'm also a little skeptical of any study published by a company looking to sell you what the study has just claimed to be great. That doesn't mean its a complete sham, but how hard did they look for other explanations why some companies are more successful than others?Re:Anonymous Coward writes:
At first I was skeptical, but I read some online reviews of it, and it looks pretty good [slashdot.org]. All you need is some AI and everything is better.I smell Bullshit Bingo...93 Escort Wagon ( 326346 ) writes:
that's all, folks...Meeting goalspetes_PoV ( 912422 ) , Sunday August 27, 2017 @05:56PM ( #55094339 )
I notice the targets are all set from the company's point of view... including customer satisfaction. However it's quite easy to meet any goal, as long as you set it low enough.
Companies like Comcast or Qwest objectively have abysmal customer satisfaction ratings; but they likely meet their internal goal for that metric. I notice, in their public communications, they always use phrasing along the lines of "giving you an even better customer service experience" - again, the trick is to set the target low andcontinuous delivery == constant change ( Score: , Insightful)SethJohnson ( 112166 ) writes:
This might be good for developers, but it's a nightmare for the poor, bloody, customers.
Any professional outfit will test a new release (in-house or commercial product) thoroughly before letting it get anywhere close to an environment where their business is at stake.
This process can take anywhere from a day or two to several months, depending on the complexity of the operation, the scope of the changes, HOW MANY (developers note: not if any ) bugs are found and whether any alterations to working practices have to be introduced.
So to have developers lob a new "release" over the wall at frequent intervals is not useful, it isn't clever, nor does it save (the users) any money or speed up their acceptance. It just costs more in integration testing, floods the change control process with "issues" and means that when you report (again, developers: not if ) problems, it is virtually impossible to describe exactly which release you are referring to and even more impossible for whoever fixes the bugs to produce the same version to fix and then incorporate those fixes into whatever happens to be the latest version - that hour. Even more so when dozens of major corporate customers are ALL reporting bugs with each new version they test.Re:Herkum01 ( 592704 ) writes:
Any professional outfit will test a new release (in-house or commercial product) thoroughly before letting it get anywhere close to an environment where their business is at stake. This process can take anywhere from a day or two to several months, depending on the complexity of the operation, the scope of the changes, HOW MANY (developers note: not if any) bugs are found and whether any alterations to working practices have to be introduced.
I wanted to chime in with a tangible anecdote to support yourRe:JohnFen ( 1641097 ) writes:
I can sympathize with that few, of it appearing to have too many developers focused upon deployment/testing then actual development.
However, I come at it from the other side, the developers just push new development out and production support is responsible for addressing the mess, it is horrible, there is too much disconnect between developers and their resulting output creating consistent outages. The most successful teams follow the mantra "Eat your own dog food" , developers who support the crap they pushRe:AmazingRuss ( 555076 ) writes:This might be good for developers
It's not even good for developers."a new paradigm shift."sethstorm ( 512897 ) writes:
Another one?Let's hope not.bobm ( 53783 ) writes:
AI is enough of a problem, why make it worse?According to one studyangel'o'sphere ( 80593 ) writes:
One study, well then I'm sold.
But do you know who likes Continuous Delivery? Not the users. The users hate stuff changing for the sake of change, but trying to convince management seems an impossible task.Re:Junta ( 36770 ) writes:
Why should users not like it? If you shop on amazon you don't know if a specific feature you notice today came there via continuous delivery or a more traditional process.Re:angel'o'sphere ( 80593 ) writes:
The crux of the problem is that we (in these discussions and the analysts) describe *all* manner of 'software development' as the same thing. Whether it's a desktop application, an embedded microcontroller in industrial equipment, a web application for people to get work done, or a webapp to let people see the latest funny cat video.
Then we start talking past each other, some of us terrified what 'continious delivery' means in the context of software in the microcontroller of a health care device, others tRe:JohnFen ( 1641097 ) writes:
Well, 'continuous delievery' is a term with a defined meaning. And releasing phone apps with unwanted UI/functionality in rapid succession is not part of that definition. Continuous delievery basically only is the next logical step after continuous integration. You deploy the new functionallity automatically (or with a click of a button) when certain test criteria are met. Usually on a subset of your nodes so only a subset of your customers sees it. If you have crashes on those nodes or customer complaints youRe:Junta ( 36770 ) writes:You deploy the new functionallity automatically (or with a click of a button) when certain test criteria are met. Usually on a subset of your nodes so only a subset of your customers sees it. If you have crashes on those nodes or customer complaints you roll back.
Why do you consider this to be a good thing? It's certainly not for those poor customers who were chosen to be involuntary beta testers, and it's also not for the rest of the customers who have to deal with software that is constantly changing underneath them.Re:manu0601 ( 2221348 ) writes:'continuous delievery' is a term with a defined meaning. And releasing phone apps with unwanted UI/functionality in rapid succession is not part of that definition.
It is a natural consequence of a continuous delivery, emphasis on always evolving and changing and that the developer is king and no one can question developer opinion. Developer decides it should move, it moves. No pesky human testers to stand up and say 'you confused the piss out of us' to make them rethink it. No automatic test is going to capture 'confuse the piss out of the poor non-developer users'.If you have crashes on those nodes or customer complaints you roll back.
Note that a customer with a choice is likely to just go somewhere else rather than use your software.AI written paper4wdloop ( 1031398 ) writes:
I suspect that article was actually written by an AI. That would explain why it makes so little sense to human mind.IT what?Comrade Ogilvy ( 1719488 ) writes:
IT in my company does network, Windows, Office and Virus etc. type of work. Is this what they talk about? Anyway, it's been long outsourced to IT (as in "Indian" technology)...For some businesses maybe but...JohnFen ( 1641097 ) writes:
I recently interviewed at a couple of the new fangled big data marketing startups that correlate piles of stuff to help target ads better, and they were continuously deploying up the wazoo. In fact, they had something like zero people doing traditional QA.
It was not totally insane at all. But they did have a blaze attitude about deployments -- if stuff don't work in production they just roll back, and not worry about customer input data being dropped on the floor. Heck, they did not worry much about daRe:Njovich ( 553857 ) writes:But they did have a blaze attitude about deployments -- if stuff don't work in production they just roll back, and not worry about customer input data being dropped on the floor.
It's amazing how common this attitude has become. It's aggressively anti-customer, and a big part of the reason for the acceleration of the decline of software quality over the past several years.Noangel'o'sphere ( 80593 ) writes:
You want your deployment system to be predictable, and as my old AI professor used to say, intelligent means hard to predict. You don't want AI for systems that just have to do the exact same thing reliably over and over again.Summary sounds retardedJunta ( 36770 ) writes:
A continuous delivery pipeline has as much AI as a nematode has natural intelligence ... probably even less.In other words...JohnFen ( 1641097 ) writes:
Analyst who understands neither software development nor AI proceeds to try to sound insightful about both.All I know isjmcwork ( 564008 ) writes:
All I know is that, as a user, rapid-release or continuous delivery has been nothing but an enormous pain in the ass to me and I wish it would die the horrible death it deserves already.Every morning: git update; make install
As long as customers are comfortable with doing this, I do not see a problem. Now, that will require that developers keep making continuous,
Jun 16, 2017 | opensource.com
...At Duke University's Office of Information Technology (OIT), we began looking at containers as a way to achieve higher density from the virtualized infrastructure used to host websites. Virtual machine (VM) sprawl had started to become a problem. We favored separating each client's website onto its own VM for both segregation and organization, but steady growth meant we were managing more servers than we could handle. As we looked for ways to lower management overhead and make better use of resources, Docker hit the news, and we began to experiment with containerization for our web applications.
For us, the initial investigation of containers mirrors a shift toward a DevOps culture.Where we started
When we first looked into container technology, OIT was highly process driven and composed of monolithic applications and a monolithic organizational structure. Some early forays into automation were beginning to lead the shift toward a new cultural organization inside the department, but even so, the vast majority of our infrastructure consisted of "pet" servers (to use the pets vs. cattle analogy). Developers created their applications on staging servers designed to match production hosting environments and deployed by migrating code from the former to the latter. Operations still approached hosting as it always had: creating dedicated VMs for individual services and filing manual tickets for monitoring and backups. A service's lifecycle was marked by change requests, review boards, standard maintenance windows, and lots of personal attention.A shift in culture
As we began to embrace containers, some of these longstanding attitudes toward development and hosting began to shift a bit. Two of the larger container success stories came from our investigation into cloud infrastructure. The first project was created to host hundreds of R-Studio containers for student classes on Microsoft Azure hosts, breaking from our existing model of individually managed servers and moving toward "cattle"-style infrastructure designed for hosting containerized applications.
The other was a rapid containerization and deployment of the Duke website to Amazon Web Services while in the midst of a denial-of-service attack, dynamically creating infrastructure and rapidly deploying services.
The success of these two wildly nonstandard projects helped to legitimize containers within the department, and more time and effort was put into looking further into their benefits and those of on-demand and disposable cloud infrastructure, both on-premises and through public cloud providers.
It became apparent early on that containers lived within a different timescale from traditional infrastructure. We started to notice cases where short-lived, single-purpose services were created, deployed, lived their entire lifecycle, and were decommissioned before we completed the tickets created to enter them into inventory, monitoring, or backups. Our policies and procedures were not able to keep up with the timescales that accompanied container development and deployment.
In addition, humans couldn't keep up with the automation that went into creating and managing the containers on our hosts. In response, we began to develop more automation to accomplish usually human-gated processes. For example, the dynamic migration of containers from one host to another required a change in our approach to monitoring. It is no longer enough to tie host and service monitoring together or to submit a ticket manually, as containers are automatically destroyed and recreated on other hosts in response to events.
Some of this was in the works for us already-automation and container adoption seem to parallel one another. At some point, they become inextricably intertwined.
As containers continued to grow in popularity and OIT began to develop tools for container orchestration, we tried to further reinforce the "cattle not pets" approach to infrastructure. We limited login of the hosts to operations staff only (breaking with tradition) and gave all hosts destined for container hosting a generic name. Similar to being coached to avoid naming a stray animal in an effort to prevent attachment, servers with generic names became literally forgettable. Management of the infrastructure itself became the responsibility of automation, not humans, and humans focused their efforts on the services inside the containers.
Containers also helped to usher continuous integration into our everyday workflows. OIT's Identity Management team members were early adopters and began to build Kerberos key distribution centers (KDCs) inside containers using Jenkins, building regularly to incorporate patches and test the resulting images. This allowed the team to catch breaking builds before they were pushed out onto production servers. Prior to that, the complexity of the environment and the widespread impact of an outage made patching the systems a difficult task.Embracing continuous deployment
Since that initial use case, we've also embraced continuous deployment. There is a solid pattern for every project that gets involved with our continuous integration/continuous deployment (CI/CD) system. Many teams initially have a lot of hesitation about automatically deploying when tests pass, and they tend to build checkpoints requiring human intervention. However, as they become more comfortable with the system and learn how to write good tests, they almost always remove these checkpoints.
Within our container orchestration automation, we use Jenkins to patch base images on a regular basis and rebuild all the child images when the parent changes. We made the decision early that the images could be rebuilt and redeployed at any time by automated processes. This meant that any code included in the branch of the git repository used in the build job would be included in the image and potentially deployed without any humans involved. While some developers initially were uncomfortable with this, it ultimately led to better development practices: Developers merge into the production branch only code that is truly ready to be deployed.
This practice facilitated rebuilding container images immediately when code is merged into the production branch and allows us to automatically deploy the new image once it's built. At this point, almost every project using the automatic rebuild has also enabled automated deployment.Looking ahead
Today the adoption of both containers and DevOps is still a work in progress for OIT.
Internally we still have to fight the entropy of history even as we adopt new tools and culture. Our biggest challenge will be convincing people to break away from the repetitive break-fix mentality that currently dominates their jobs and to focus more on automation. While time is always short, and the first step always daunting, in the long run adopting automation for day-to-day tasks will free them to work on more interesting and complex projects.
Thankfully, people within the organization are starting to embrace working in organized or ad hoc groups of cross-discipline members and developing automation together. This will definitely become necessary as we embrace automated orchestration and complex systems. A group of talented individuals who possess complementary skills will be required to fully manage the new environments.
Dec 19, 2016 | theregister.co.uk
Get real – it's not all about developers and DevOps
Listen to some DevOps evangelists talk, and you would get the impression that IT operations teams exist only to serve the needs of developers. Don't get me wrong, software development is a good competence to have in-house if your organisation depends on custom applications and services to differentiate its business.
As an ex-developer, I appreciate the value of being able to deliver something tailored to a specific need, even if it does pain me to see the shortcuts too often taken nowadays due to ignorance of some of the old disciplines, or an obsession with time-to-market above all else.
But before this degenerates into an 'old guy' rant about 'youngsters today', let's get back to the point that I really want to make.
All of the hype around software and developers, which tends to significantly skew even the DevOps discussion, runs the risk of creating the perception that IT ops is just a necessary evil. Indeed, some go so far as to make the case for a 'NoOps' world in which the public cloud magically takes care of everything downstream once developers have 'innovated' and 'created'.
This kind of view comes about from people looking through the wrong end of the telescope. Turn the thing around and look up close at what goes on in the world of ops, and you get a much better sense of perspective. Teams operating in this space are not just there to deploy the next custom software release and make sure it runs quickly and robustly - in fact that's often a relatively small part of what they do.
This becomes obvious when you recognize how much stuff runs in an Enterprise IT landscape - software packages enabling core business processes, messaging, collaboration and workflow platforms keeping information flowing, analytics environments generating critical business insights, and desktop and mobile estates serving end user access needs - to name but a few.Vital operations
There's then everything required to deal with security, data protection, compliance and other aspects of risk. Apart from the odd bit of integration and tailoring work - the need for which is diminishing with modern 'soft-coded', connector-driven solutions - very little of all this has anything to do with development and developers.
A big part of the rationale for modernising your application landscape and migrating to the latest flexible and open software packages and platforms is to eradicate the need for coding wherever you can. Code is expensive to build and maintain, and the same can often be achieved today through software switches, policy-driven workflow, drag-and-drop interface design, and so on. Sensible IT teams only code when they absolutely have to.
And coming back to operations, you are sadly mistaken if you think that the public cloud makes all challenges and requirements go away. If anything, the piecemeal adoption of cloud services has made things more complex and unpredictable from an integration and management perspective.
There are all kinds of valid reasons to keep an application sitting on your own infrastructure anyway - regulation, performance, proximity to dependent solutions and data, etc. Then let's not forget the simple fact that running things in the cloud is often more expensive over the longer term.
Against this background, an 'appropriate' level of custom development and the selective use of cloud services will be the way forward for most organisations, all underpinned by a well-run data centre environment acting as the hub for hybrid delivery. This is the approach that tends to be taken by the most successful enterprise IT teams, and the element that makes particularly high achievers stand out is agile and effective IT operations.
This isn't just to support any DevOps agenda you might have; it is demonstrably a key enabler across the board. Of course if you work in operations, you will know already intuitively know all this. But if you want some ammunition to spell it out to others who need enlightenment, take a look at our research report entitled IT Ops and a Digital Business Enabler; more than just keeping the lights on . This is based on input from 400 Senior European IT professionals. ®Paul SmithI think this is one fad that has run its course. If nothing else, the one thing that cloud has brought to the software world is the separation of software from the environment it runs in, and since the the Ops side of DevOps is all about the integration of the platform and software, what you end up with in a cloudy world is a lot of people looking for a new job.Anonymous CowardJohn 104
For decades developers have been ignored by infrastructure vendors because the decision makers buying infrastructure sit in the infrastructure teams. Now with the cloud etc vendors realize they will lose supporters within these teams.
So instead - infrastructure vendors target developers to become their next fanboys.
E.g. Dear developer, you won't need to speak to your infrastructure admins anymore to setup a development environment. Now you can automate, orchestrate the provisioning of your containerized development environment at the push of a button. Blah blah blah, but you have to buy our storage.
I remember the days when every DBA wanted RAID10 just because thats what the whitepaper recommended. By that time storage technology had long moved on, but the DBA still talked about Full Stripe Writes.
Now with DevOps you'll have Developers influencing infrastructure decisions, because they just learned about snapshots. And yes - it has to be all flash - and designed from the ground up by millenials that eat avocado.Re: DevOps was never supposed to replace OperationsJohn 104
Yes, DevOps isn't about replacing Ops. But try telling that to the powers that be. It is sold and seen as a cost cutting measure.
As for devs learning Ops and vice versa, there are very few on both sides who really understand what it takes to do the others job. I have a very high regard for Devs, but when it comes to infra, they are, as a whole, very incompetent. Just like I'm incompetent in Dev. can't have one without the other. I feel that in time, the pendulum will swing away from cloud as execs and accountants realize how it isn't really saving any money.
The real question is: Will there be any qualified operations engineers available or will they all have retired out or have found work elsewhere. It isn't easy to be an ops engineer, takes a lot of experience to get there, and qualified candidates are hard to come by. Let's face it, in today's world, its a dying breed.Very Nice
Nice of you to point out what us in Ops have known all along. I'm afraid it will fall on deaf ears, though. Until the executives who constantly fall for the new shiny are made to actually examine business needs and processes and make business decisions based on said.
Our laughable move to cloud here involved migrating off of on prem Exchange to O365. The idea was to free up our operations team to allow us to do more in house projects. Funny thing is, it takes more management of the service than we ever did on premises. True, we aren't maintaining the Exchange infra, but now we have SQL servers, DCs, ADFS, etc, to maintain in the MS cloud to allow authentication just to use the product. And because mail and messaging is business critical, we have to have geographically disparate instances of both. And the cost isn't pretty. Yay cloud.
May 17, 2017 | theregister.co.uk
Linus Torvalds believes the technology industry's celebration of innovation is smug, self-congratulatory, and self-serving. The term of art he used was more blunt: "The innovation the industry talks about so much is bullshit," he said. "Anybody can innovate. Don't do this big 'think different'... screw that. It's meaningless.
In a deferential interview at the Open Source Leadership Summit in California on Wednesday, conducted by Jim Zemlin, executive director of the Linux Foundation, Torvalds discussed how he has managed the development of the Linux kernel and his attitude toward work.
"All that hype is not where the real work is," said Torvalds. "The real work is in the details."
Torvalds said he subscribes to the view that successful projects are 99 per cent perspiration, and one per cent innovation.
As the creator and benevolent dictator of the open-source Linux kernel , not to mention the inventor of the Git distributed version control system, Torvalds has demonstrated that his approach produces results. It's difficult to overstate the impact that Linux has had on the technology industry. Linux is the dominant operating system for servers. Almost all high-performance computing runs on Linux. And the majority of mobile devices and embedded devices rely on Linux under the hood.
The Linux kernel is perhaps the most successful collaborative technology project of the PC era. Kernel contributors, totaling more than 13,500 since 2005, are adding about 10,000 lines of code, removing 8,000, and modifying between 1,500 and 1,800 daily, according to Zemlin. And this has been going on – though not at the current pace – for more than two and a half decades.
"We've been doing this for 25 years and one of the constant issues we've had is people stepping on each other's toes," said Torvalds. "So for all of that history what we've done is organize the code, organize the flow of code, [and] organize our maintainership so the pain point – which is people disagreeing about a piece of code – basically goes away."
The project is structured so people can work independently, Torvalds explained. "We've been able to really modularize the code and development model so we can do a lot in parallel," he said.
Technology plays an obvious role but process is at least as important, according to Torvalds.
"It's a social project," said Torvalds. "It's about technology and the technology is what makes people able to agree on issues, because ... there's usually a fairly clear right and wrong."
But now that Torvalds isn't personally reviewing every change as he did 20 years ago, he relies on a social network of contributors. "It's the social network and the trust," he said. "...and we have a very strong network. That's why we can have a thousand people involved in every release."
The emphasis on trust explains the difficulty of becoming involved in kernel development, because people can't sign on, submit code, and disappear. "You shoot off a lot of small patches until the point where the maintainers trust you, and at that point you become more than just a guy who sends patches, you become part of the network of trust," said Torvalds.
Ten years ago, Torvalds said he told other kernel contributors that he wanted to have an eight-week release schedule, instead of a release cycle that could drag on for years. The kernel developers managed to reduce their release cycle to around two and half months. And since then, development has continued without much fuss.
"It's almost boring how well our process works," Torvalds said. "All the really stressful times for me have been about process. They haven't been about code. When code doesn't work, that can actually be exciting ... Process problems are a pain in the ass. You never, ever want to have process problems ... That's when people start getting really angry at each other." ®
May 17, 2017 | theregister.co.uk12 May 2017 at 14:56, Trevor Pott Infrastructure as code is a buzzword frequently thrown out alongside DevOps and continuous integration as being the modern way of doing things. Proponents cite benefits ranging from an amorphous "agility" to reducing the time to deploy new workloads. I have an argument for infrastructure as code that boils down to "cover your ass", and have discovered it's not quite so difficult as we might think.
... ... ...
None of this is particularly surprising. When you have an environment where each workload is a pet , change is slow, difficult, and requires a lot of testing. Reverting changes is equally tedious, and so a lot of planning goes into making sure than any given change won't cascade and cause knock-on effects elsewhere.
In the real world this is really the result of two unfortunate aspects of human nature. First: everyone hates doing documentation, so it's highly unlikely that in an unstructured environment every change from the last refresh was documented. The second driver of chaos and problems is that there are few things more permanent than a temporary fix.
When you don't have the budget for the right hardware, software or services you make do. When something doesn't work you "innovate" a solution. When that breaks something, you patch it. You move from one problem to the next, and if you're not careful, you end up with something so fragile that if you breathe on it, it falls over. At this point, you burn it all down and restart from scratch.
This approach to IT is fine - if you have 5, 10 or even 50 workloads. A single techie can reasonably be expected to keep that all in their head, know their network and solve any problems they encounter. Unfortunately, 50 workloads is today restricted to only the smallest of shops. Everyone else is juggling too many workloads to be playing the pets game any more.
Most of us use some form of desired state solution already. Desired state solutions basically involve an OS agent that gets a config from a centralized location and applies the relevant configuration to the operating system and/or applications. Microsoft's group policy can be considered a really primitive version of this, with System Center being a more powerful but miserable to use example. The modern friendly tools being Puppet, Chef, Saltstack, Ansible and the like.
Once you have desired state configs in place we're no longer beating individual workloads into shape, or checking them manually for deviation from design. If all does what it says on the tin, configurations are applied and errors thrown if they can't be. Usually there is some form of analysis software to determine how many of what is out of compliance. This is a big step forward.
... ... ...
This article is sponsored by HPE.
May 16, 2017 | soleil.wikia.comThe Technocult, also known as the Machine cult is the semi-offical name given by The Church of the Crossed Heart to followers of the Mechanicum faith who supply and maintain virtually all of the church's technology, engineering and industry.
Although they serve with the Church of the Crossed Heart they have their own version of worship that differs substantially in theology and ritualistic forms from that of The Twelve Angels . Instead the Technocult worships a deity they call the Machine god or Omnissiah. The Technocult believes that knowledge is divine and comes only form the Omnissiah thus making any objects that demonstrate the application of knowledge , i.e machinery, or contain it (books) holy in the eyes/optical implants of the Techcult. The Technocult regard organic flesh as weak and imperfect, with the Rot being veiwed as a divine message from the Omnissah demonstrating its weakness, thus making its removal and replacement by mechanical, bionic parts a sacred process that brings them closer to their god with many of its older members having very little of their original bodies remaining.
The date of the cults formation is unknown, or a closely guarded secret...
May 16, 2017 | www.allaboutagile.com
1. Saying you're doing Agile just cos you're doing daily stand-ups. You're not doing agile. There is so much more to agile practices than this! Yet I'm surprised how often I've heard that story. It really is remarkable.
... ... ....
3. Thinking that agile is a silver bullet and will solve all your problems. That's so naiive, of course it won't! Humans and software are a complex mix with any methodology, let alone with an added dose of organisational complexity. Agile development will probably help with many things, but it still requires a great deal of skill and there is no magic button.
... ... ...
8. People who use agile as an excuse for having no process or producing no documentation. If documents are required or useful, there's no reason why an agile development team shouldn't produce them. Just not all up-front; do it as required to support each feature or iteration. JFDI (Just F'ing Do It) is not agile!
David, 23 February 2010 at 1:21 am
So agree on number 1. Following "Certified" Scrum Master training (prior to the exam requirement), a manager I know now calls every regular status meeting a "scrum", regardless of project or methodology. Somehow the team is more agile as a result.
Ironically he pulled up another staff member for "incorrectly" using the term retrospective.
Andy Till, 23 February 2010 at 9:28 am
I can think of far worse, how about pairing with the guy in the office who is incapable of compromise?
Steve Watson, 13 May 2010 at 10:06 am
I like number 9 as I find with testing people think that they no longer need to write proper test cases and scripts – a list of confirmations on a user story will do. Well, if its a simple change I guess you can dispense with test scripts, but if its something more complex then there is no reason NOT to write scripts. If you have a reasonably large team of people who could execute the tests, they can follow the test steps and validate against the expected results. It also means that you can sensibly lump together test cases and cover them with one test.
If you dont think about how you will execute them and just tackle them one by one off the confirmations list, you miss the opportunity to run one test and cover many separate cases, saving time.
I always find test scripts useful if someone different re-runs a test, as they then follow the same process as before. This is why we automate regression so the tests are executed the same each time.
John Quincy, 24 October 2011 at 12:02 am
I am not a fan of agile. Unless you have a small group of developers who are in perfect sync with each other at all times, this "one size fits all" methodology is destructive and downright dangerous. I have personally witnessed a very good company go out of business this year because they transformed their development shop from a home-grown iterative methodology to SCRUM. The team was required to abide by the SCRUM rules 100%. They could not keep up with customer requirements and produced bug filled releases that were always late. These developers went from fun, friendly, happy people (pre-SCRUM) [who NEVER missed a date] to bitter, sarcastic, hard to be around 'employees'. When the writing was on the wall a couple of months back, the good ones got the hell out of there, and the company could not recover.
Some day, I'm convinced that Beck through Thomas will proclaim that the Agile Manifesto was all a big practical joke that got out of control.
This video pretty much lays out the one and only reason why management wants to implement Agile:
grumpasaurus, 9 February 2014 at 4:30 pm
It's a cycle of violence when a project claims to be Agile just because of standups and iterations and don't think about resolving the core challenges they've had to begin with. People are left still battling said challenges and then say that Agile sucks.
Jun 06, 2014 | www.infoq.comRachel Shannon-Solomon suggests that most enterprises are not ready for DevOps, while Gene Kim says that they must make themselves ready if they want to survive.
Rachel Shannon-Solomon, a venture associate at At Work-Bench, has recently written a blog post for The Wall Street Journal entitled DevOps Is Great for Startups, but for Enterprises It Won't Work-Yet , arguing that while DevOps is appealing to startups, there are important stumbling blocks in the path of DevOps adoption within the enterprise.
While acknowledging that large companies such as Google and Facebook benefit from implementing DevOps, and that "there is no lack of appetite to experiment with DevOps practices" within "Fortune 500s and specifically financial services firms", Shannon-Solomon remarks that "there are few true change agents within enterprise IT willing to affect DevOps implementations."
Shehas come to this conclusion basedon "conversations with startup founders, technology incumbents offering DevOps solutions, and technologists within large enterprises."
Shannon-Solomon brings four arguments to support her position:
- Siloed structures and organizational change . According to Shannon-Solomon, enterprises create siloed structures between teams because that's how large organizations maximize value results, and one of DevOps'main purpose is to bring thosesilos down. Sometimes a large company relies on another vendor for operational support, and such vendors "tend to bring their own degree of siloing." Also, it is expensive for an enterprise to invest into "holistic solutions to facilitate DevOps" after they "invested heavily on integration for existing solutions."
- Buy vs. build. The tools needed to implement a DevOps culture are lacking. While some of the tools can be provided by vendors and others can be created within the enterprise, a process which takes a long period of time, "there is a marathon of organizational change and restructuring that must occur before such tools could ever be bought or built."
- Vendors of DevOps solutions acknowledge that when selling to the enterprise, they are trying to sell a cultural revolution. According to Shannon-Solomon, it is hard to introduce DevOps to development teams because Vendors need to first win over individual developers with the efficiency of their solution, for example, by initially eschewing procurement and offering a sandbox environment directly to developers to test out the environment.
Selling DevOps toolkits to the enterprise means facing the well-documented challenges of navigating procurement and a mindset currently more primed to build than buy DevOps tools.
- Return on investment . Shannon-Solomon cites a senior IT professional working at an investment bank saying that his company "has not been very successful at tracking DevOps projects occurring within individual business units using homegrown tools, and any evaluation of a project's success has been limited to anecdotal assessments and perceived results."
Shannon-Solomonends her post wondering "how long will it be until enterprises are forced to accept that they must accelerate their experiments with DevOps" and hoping that "more individual change agents within large organizations may emerge" in the future.
Jun 05, 2013 | Slashdot
"What developers see as iterative and flexible, users see as disorganized and never-ending.
This article discusses how some experienced developers have changed that perception. '... She's been frustrated by her Agile experiences - and so have her clients.
"There is no process. Things fly all directions, and despite SVN [version control] developers overwrite each other and then have to have meetings to discuss why things were changed. Too many people are involved, and, again, I repeat, there is no process.' The premise here is not that Agile sucks - quite to the contrary - but that developers have to understand how Agile processes can make users anxious, and learn to respond to those fears. Not all those answers are foolproof.
For example: 'Detailed designs and planning done prior to a project seems to provide a "safety net" to business sponsors, says Semeniuk. "By providing a Big Design Up Front you are pacifying this request by giving them a best guess based on what you know at that time - which is at best partial or incorrect in the first place." The danger, he cautions, is when Big Design becomes Big Commitment - as sometimes business sponsors see this plan as something that needs to be tracked against.
"The big concern with doing a Big Design up front is when it sets a rigid expectation that must be met, regardless of the changes and knowledge discovered along the way," says Semeniuk.' How do you respond to user anxiety from Agile processes?"
Agile summed up (Score:5, Funny)
Agile is to proper software engineering what Red Bull Flugtag is to proper aeronautic engineering....
Re: doesn't work
As TFA points out, that always works fine when your requirements are *all* known an are completely static. That rarely happens in most fields.
Even in the ones where it does it's usually just management having the balls to say "No, you can give us the next bunch of additions and changes when this is delivered, we agreed on that". It frequently ends up delivering something less than useful.
Re: doesn't work (Score:5, Insightful)
The problem with Agile is that it gives too much freedom to the customer to change their mind late in the project and make the developers do it all over again.
Re: doesn't work (Score:4, Insightful)
...but they can be trusted to say what is most important to them at the time.
No they can't. If you are delivering to customer requests you will always be a follower and never succeed. You need to anticipate what the customers need. As with the I guess made up quote attributed to Henry Ford, "If I listened to my customers I'd have been trying to make faster horses." Whether he said it or not, the statement is true. Customers know what they have and just want it to be faster/better/etc you need to find out what they really need.
Re: doesn't work (Score:5, Insightful)
It frequently is. It doesn't matter what methodology you use -- if you change major features/priorities at the last minute it will cost multiple times as much. Yet frequently customers expect it to be cheap because "we're agile". And by accepting that change will happen you don't push the customers to make important decisions early, ensuring that major changes will happen, instead of just being possible.
Re: doesn't work (Score:5, Interesting)
"Proper software engineering" doesn't work.
You're right, but you're going to the other extreme. The problem with all methodologies, or processes, or whatever today's buzzword is, is that too many people want to practice them in their purest form. Excessive zeal in using any one approach is the enemy of getting things done.
On a sufficiently large project, some kind of upfront design is necessary. Spending too much time on it or going into too much detail is a waste though. Once you start to implement things, you'll see what was overlooked or why some things won't work as planned. If you insist on spinning back every little change to a monstrously detailed Master Design Document, you'll move at a snail's pace. As much as I hate the buzzword "design patterns", some pattern is highly desirable. Don't get bent out of shape though when someone has a good reason for occasionally breaking that pattern or, as you say, you'll wind up with 500 SLOC's to add 2+2 in the approved manner.
Lastly, I agree that there is no substitute for good engineers who actually talk to and work with each other. Also don't require that every 2 bit decision they make amongst themselves has to be cleared, or even communicated, to the highest levels. If you don't trust those people to make intelligent decisions (including about when things do have to be passed up) then you've either got the wrong people or a micromanagement fetish. Without good people you'll never get anything decent done, but with good people you still need some kind of organization.
The problem the article refers to about an upfront design being ironclad promises is tough. Some customers will work with you, and others will get their lawyers and "systems" people to waste your time complaining about every discrepancy, without regard to how important it is. Admittedly bad vendors will try and screw their customers with "that doesn't matter" to excuse every screw-up and bit of laziness. For that reason I much prefer working on in-house projects, where "sure we could do exactly what we planned" gets balanced with the cost and other tradeoffs.
The worst example of those problems is defense projects. As someone I used to work with said: In defense everything has to meet spec, but it doesn't have to work. In the commercial world specs are flexible, but it has to work.
If you've ever worked in that atmosphere you'll understand why every defense project costs a trillion dollars. There is absolutely no willingness to make tradeoffs as the design progresses and you find out what's practical and necessary and what's not. I'm not talking about meeting difficult requirements if they serve a purpose (that's what you're paid for) but being unwilling to compromise on any spec that somebody at the beginning of the project pulled out of their posterior and obviously doesn't need to be so stringent. An elephant is a mouse built to government specifications.
Ok, you can get such things changed, but it requires 10 hours from program managers for every hour of engineering. Conversely, don't even think about offering a feature or capability that will be useful and easy to implement, but is not in the spec. They'll just start writing additional specs to define it and screw you by insisting you meet those.
As you might imagine, I'm very happy to be back in the commercial world.
Re: doesn't work (Score:2, Interesting)
You've fallen into the trap of using their terminology. As soon as 'the problem' is defined in terms of 'upfront design', you've already lost half the ideological battle.
'The problem' (with methodology) is that people want to avoid the difficult work of thinking hard about the business/customer's problem and coming up with solutions that meet all their needs. But there isn't a substitute for thinking hard about the problem and almost certainly never will be.
The earlier you do that hard thinking about the customer's problems that you are trying to solve the cheaper, faster and better quality the result will be. Cheaper? Yes, because bugfixing that is done later in the project is a lot more expensive (as numerous software engineering studies have shown) Faster? Yes, because there's less rework. (Also, since there is usually a time = money equivalency, you can't have it done cheap unless it is also done fast. Higher quality? Yes, because you don't just randomly stumble across quality. Good design trumps bad design every single time.
... ... ...
Re: doesn't work (Score:4, Interesting)
Until the thing is built or the software is shipped there are many options and care should be taken that artificial administrative constraints don't remove too many of them.
Exactly, and as someone who does both hardware and software I can tell you that that's better understood by Whoever Controls The Great Spec in hardware than in software. Hardware is understood to have physical constraints, so not every change is seen as the result of a screw-up. It's a mentality.
I'll also admit that there is a tendency to get sloppy in software specs because it is easier to make changes. Hardware, with the need to order materials, have things fabbed, tape out a chip, whatever, imposes a certain discipline that's lacking when you know you can change the source code at anytime. Being both, I'm not saying this is because hardware engineers are virtuous and software engineers are sloppy, but because engineers are human (at least some of them).
Re: doesn't work (Score:2)
This is my evidence that "proper software engineering" *can* work. The fact that most businesses (and their customers) are willing to save money by accepting less from their software is not the fault of software engineering. We could and did build buildings much faster than we do today, if you are willing to make more mistakes and pay more in human lives. If established industries and their customers began demanding software at that higher standard and were willing to pay for it like it was real engineering, then maybe it would happen more often.
Impressive stuff, and not unique to the space shuttle. Fly-by-wire systems are the same way. You're talking DO-178B [wikipedia.org] Level A stuff. It works, and it's very very expensive. If it was only 10x the cost of normal software development I'd be amazed. I agree that way too much software is poorly planned and implemented crap, and part of the reason is that nobody wants realistic cost estimates or to make the difficult decisions about what it's supposed to do up-front. But what you're talking about is aerospace quality. You couldn't afford a car or even a dishwasher made to those standards.
Re: doesn't work (Score:3)
260 people maintaining 420,000 lines of code, written to precise externally provided specifications that change once every few years.
This is fine for NASA, but if you want something that does roughly what you need before your competitors come up with something better, you'd better find some better programmers.
May 15, 2017 | blog.appdynamics.com
In light of all the hype, we have created a DevOps parody Series – DevOps: Fact or Fiction .
For those of you who did not see, in October we created an entirely separateblog(inspired by this ) – however decided that it is relevant enough to transform into a series on the AppDynamics Blog . The series will point out the good, the bad, and the funny about IT and DevOps. Don't take anything too seriously – it's nearly 100% stereotypes : ).
Stay tuned for more DevOps: Fact or Fiction to come. Here we go
Apr 15, 2014 | jeffknupp.comHow 'DevOps' is Killing the Developer
There are two recent trends I really hate: DevOps and the notion of the "full-stack" developer. The DevOps movement is so popular that I may as well say I hate the x86 architecture or monolithic kernels. But it's true: I can't stand it. The underlying cause of my pain? This fact: not every company is a start-up, though it appears that every company must act as though they were. DevOps
"DevOps" is meant to denote a close collaboration and cross-pollination between what were previously purely development roles, purely operations roles, and purely QA roles. Because software needs to be released at an ever-increasing rate, the old "waterfall" develop-test-release cycle is seen as broken. Developers must also take responsibility for the quality of the testing and release environments.
The increasing scope of responsibility of the "developer" (whether or not that term is even appropriate anymore is debatable) has given rise to a chimera-like job candidate: the "full-stack" developer. Such a developer is capable of doing the job of developer, QA team member, operations analyst, sysadmin, and DBA. Before you accuse me of hyperbole, go back and read that list again. Is there any role in the list whose duties you wouldn't expect a "full-stack" developer to be well versed in?
Where did these concepts come from? Start-ups, of course (and the Agile methodology). Start-ups are a peculiar beast and need to function in a very lean way to survive their first few years. I don't deny this . Unfortunately, we've taken the multiple technical roles that engineers at start-ups were forced to play due to lack of resources into a set of minimum qualifications for the role of "developer".Many Hats
Imagine you're at a start-up with a development team of seven. You're one year into development of a web applications that X's all the Y's and things are going well, though it's always a frantic scramble to keep everything going. If there's a particularly nasty issue that seems to require deep database knowledge, you don't have the liberty of saying "that's not my specialty," and handing it off to a DBA team to investigate. Due to constrained resources, you're forced to take on the role of DBA and fix the issue yourself.
Now expand that scenario across all the roles listed earlier. At any one time, a developer at a start-up may be acting as a developer, QA tester, deployment/operations analyst, sysadmin, or DBA. That's just the nature of the business, and some people thrive in that type of environment. Somewhere along the way, however, we tricked ourselves into thinking that because, at any one time, a start-up developer had to take on different roles he or she should actually be all those things at once.
If such people even existed , "full-stack" developers still wouldn't be used as they should. Rather than temporarily taking on a single role for a short period of time, then transitioning into the next role, they are meant to be performing all the roles, all the time . And here's what really sucks: most good developers can almost pull this off.The Totem Pole
Good developers are smart people. I know I'm going to get a ton of hate mail, but there is a hierarchy of usefulness of technology roles in an organization. Developer is at the top, followed by sysadmin and DBA. QA teams, "operations" people, release coordinators and the like are at the bottom of the totem pole. Why is it arranged like this?
Because each role can do the job of all roles below it if necessary.
Start-ups taught us this. Good developers can be passable DBAs, if need be. They make decent testers, "deployment engineers", and whatever other ridiculous term you'd like to use. Their job requires them to know much of the domain of "lower" roles. There's one big problem with this, and hopefully by now you see it: It doesn't work in the opposite direction.
A QA person can't just do the job of a developer in a pinch, nor can a build-engineer do the job of a DBA. They never acquired the specialized knowledge required to perform the role. And that's fine. Like it or not, there are hierarchies in every organization, and people have different skill sets and levels of ability. However, when you make developers take on other roles, you don't have anyone to take on the role of development!
An example will make this more clear. My dad is a dentist running his own practice. He employs a secretary, hygienist, and dental assistant. Under some sort of "DentOps" movement, my dad would be making appointments and cleaning people's teeth while trying to find time to drill cavities, perform root canals, etc. My dad can do all of the other jobs in his office, because he has all the specialized knowledge required to do so. But no one, not even all of his employees combined, can do his job.
Such a movement does a disservice to everyone involved, except (of course) employers. What began as an experiment aimed at increasing software quality has become a farce, where the most talented employees are overworked (while doing less, less useful work) and lower-level positions simply don't exist.
And this is the crux of the issue. All of the positions previously held by people of various levels of ability are made redundant by the "full-stack" engineer. Large companies love this, as it means they can hire far fewer people to do the same amount of work. In the process, though, actual development becomes a vanishingly small part of a developer's job . This is why we see so many developers that can't pass FizzBuzz: they never really had to write any code. All too common a question now, can you imagine interviewing a chef and asking him what portion of the day he actually devotes to cooking?Jack of All Trades, Master of None
If you are a developer of moderately sized software, you need a deployment system in place. Quick, what are the benefits and drawbacks of the following such systems: Puppet, Chef, Salt, Ansible, Vagrant, Docker. Now implement your deployment solution! Did you even realize which systems had no business being in that list?
We specialize for a reason: human beings are only capable of retaining so much knowledge. Task-switching is cognitively expensive. Forcing developers to take on additional roles traditionally performed by specialists means that they:
- aren't spending their time developing
- need to keep up with an enormous domain of knowledge
- are going to burn out
What's more, by forcing developers to take on "full-stack" responsibilities, they are paying their employees far more than the market average for most of those tasks. If a developer makes 100K a year, you can pay four developers 100K per year to do 50% development and 50% release management on a single, two-person task. Or, simply hire a release manager at, say, 75K and two developers who develop full-time. And notice the time wasted by developers who are part time release-managers but don't always have releases to manage.Don't Kill the Developer
The effect of all of this is to destroy the role of "developer" and replace it with a sort of "technology utility-player". Every developer I know got into programming because they actually enjoyed doing it (at one point). You do a disservice to everyone involved when you force your brightest people to take on additional roles.
Not every company is a start-up. Start-ups don't make developers wear multiple hats by choice, they do so out of necessity. Your company likely has enough resource constraints without you inventing some. Please, don't confuse "being lean" with "running with the fewest possible employees". And for God's sake, let developers write code!
Enno • 2 years agoSome background... I started life as a dev (30years ago), have mostly been doing sysadmin and project tech lead sorts of work for the last 15. I've always assumed the DevOps movement was resulting in sub-par development and sub-par sysadmin/ops precisely because people were timesharing their concerns.Zakaria ANBARI -> Enno • 2 years ago
But what it does bring to the party is a greater level of awareness of the other guys problems. There's nothing quite like being rung out of bed at 3am to motivate a developer to improve his products logging to make supporting it easier. Similarly the admin exposed to the vagaries of promoting things into production in a supportable, repeatable, deterministic manner quickly learns to appreciate the issues there. So DevOps has served a purpose and has offered benefits to the organisations that signed on for it.
But, you're right. Pure DevOps is no more efficient or sensible than pure Agile (or the pure Extreme Programming or the pure Structured Programming that preceeded it). The problem is purists and ideological zealotry not the particular brand of religion in question. Insistence on adherence to dogma is the problem as it prevents adoption of flexible, 'fit for purpose' solutions. Exposure to all the alternatives is good. Insisting that one hammer is ideal for every sort of nail we have and ever will have is not.totally agree with you !DevOps Reaper • 2 years agoI'm very disappointed to see this kind of rubbish. It's this type of egocentric thinking and generalization that the developer is an omniscient deity requiring worshiping and pampering that prevents DevOps from being successful. Based on the tone and your perspective it sounds like you've been doing DevOps wrong.spunky brewster -> DevOps Reaper • a month ago
A developer role alone is not the linchpin that keeps DevOps humming - instead it's the respect that each team member holds for each discipline and each team member's area of expertise, the willingness of the entire team to own the product, feature delivery and operational stability end to end, to leverage each others skills and abilities, to not blame Dev or Ops or QA for failure, and to share knowledge.
There are developers who have a decent set of skills outside of development in QA, Operations, DB Admin, Networking, etc. Equally so, there are operations engineers who have a decent set of skills outside of operations in QA, Development, DB Admin, networking, etc. Extend this to QA and other disciplines. What I have never seen is one person who can perform all those jobs outside of their main discipline with the same level of professionalism, experience and acumen that each of those roles require to do it well at an Enterprise/World Class level.
If you're a developer doing QA and operations, you're doing it because you have to, but there should be no illusion that you're as good in alternate roles as someone trained and experienced in those disciplines. To do so is a disservice to yourself and your organization that signs your paycheck. If you're in this situation and you'd prefer making a difference rather than spewing complains, I would recommend talking to your manager and above about changing their skewed vision of DevOps. If they aren't open to communication, collaboration, experimentation and continual improvement, then their DevOps vision is dysfunctional and they're not supporting DevOps from the top down. Saying your DevOps and not doing it is *almost* more egregious than saying the developer is the top of a Totem Pole of existence.he prefaced it with 'crybabies please ignore' It's his opinion. That everyone but the lower totem pole people agree with so.. agree to disagree. I also don't think being at the bottom of the totem pole is a big f'in deal. If you're getting paid.. embrace it! So many other ways to enjoy life! The top dog people have all the pressure and die young! 99% of the people on earth dont know the difference between one nerd and another. And other nerds are always going to be egomaniacs who will find some way to justify their own superiority no matter what your achievements. So this kind of posturing is a waste of time.Pramod • 2 years agoAmen to that!!carlivar • 2 years agoI think there's a problem with your definition of DevOps. It doesn't mean developers have to be "full-stack" or do ops stuff. And it doesn't mean "act like a startup." It simply means, at its basis, that Developers and Operations work well together and do not have any communication barriers. This is why I hate DevOps as a title or department, because DevOps is a culture.StillMan -> carlivar • 2 years ago
Let's take your DentOps example. The dentist has 3 support staff. What if they rarely spoke to the dentist? What if they were on different floors of the building? What if the dentist wrote an email about how teeth should be cleaned and wasn't available to answer questions or willing to consider feedback? What if once in a while the dentist needed to understand enough about the basics of appointment scheduling to point out problems with the system? Maybe appointments are being scheduled too close together. Would the patients get backed up throughout the day because that's the secretary's problem? Of course not. Now we'd be getting into a more accurate analogy to DevOps. If anything a dentist's office is ALREADY "DentOps" and the whole point of DevOps is to make the dev/ops interaction work in a logical culture that other industries (like dentists) already use!I would tend to agree with some of that. Being able to trouble shoot network issues using monitoring tools like Fiddler is a good thing to be aware of. I can also see a lot of companies using it as a way to make one person do everything. Moreover, there are probably folks out there that perpetuate that behavior by taking on the machismo argument.spunky brewster -> carlivar • a month ago
By saying that if I can do it that you should be able to do it too or else you're not as good of a developer as I am. I have never heard anyone outright claim this, but I've seen this attitude time and time again from ambitious analysts looking to get a leg up, a pay raise, and a way to template their values on the rest of the team. One of the first things that you're taught as a dev is that you can't hope to know it all.
Your responsibility first and foremost as a developer is the stability and reliability of your code and the services that you provide. In some industries this is literally a matter of life and death(computers in your car, mission critical medical systems). It doesn't work for everyplace.Ole Hauris -> Sørensen • 2 years ago
I wouldn't want to pay a receptionist 200k a year like a dentist though. Learn to hire better receptionists. Even a moderately charming woman can create more customer loyalty, and cheaper, than the best dentist in the world. I want my dentist to keep quiet and have a steady hand. I want my receptionist to engage me and acknolwedge my existence.
I want my secretary to be a multitasking master. I want my dentist not to multitask at all - OUCH!Good points, I tend to agree. I prefer to think of DevOps as more of a full-stack team concept. Applying the full-stack principle at the individual levels is not sustainable, as you point out.ceposta Ole Hauris Sørensen • 2 years ago +1.
The full-stack DevOps team will have team members with primary skills in either of the traditional specialties, and will, over time, develop decent secondary skills. But the value is not in people constantly content switching - that actually kills efficiency. The value is in developers understanding and developing an open relationship with testing and operations - and vice versa. And this cooperation is inhibited by putting people in separate teams with conflicting goals. DevOps in practice is not a despecialization. It's bringing the specialists together.Julio • 2 years ago
The more isolated or silo'd developers become, the less they realize what constitutes delivering software, and the more problems are contributed to the IT process of test/build/release/scale/monitor, etc. Writing code is a small fraction of that delivery process. I've written about the success of devops and microservices that touches on this stuff because they're highly related. The future success of devops/microservices/cloud/etc isn't related to technology insofar as it is culture: http://blog.christianposta....
Thanks for the post!Cody • 2 years ago
Interesting points were raised. Solid arguments. I identified with this text and my current difficulties as developer.Great article and you're definitely describing one form of dysfunctional organisation where DevOps, Agile, Full Stack, and every other $2 word has been corrupted to become a cost cutting justification; cramming more work onto people who aren't skilled for it, and eho end up not having any time to do what they were hired as experts for!Zac Smith • 6 months ago
But I'd also agree with other posters that it's a little developer centric. I'm a terrible programmer and a great DBA. I can tell you most programmers who try to be DBAs are equally terrible. It's definitely not "doing the job of the receptionist" 😄
And we shouldn't forget what DevOps is meant to be about; teams making sure nobody gets called at night to fix each other's messes. That means neither developers with shitty deployments straight to production nor operations letting the disks silently fill because "if it ain't C: it ain't our problem."I know of 0 developers that can manage a network of any appreciable scale.spunky brewster Zac Smith • a month ago
In cloud and large enterprise networks, if there were a totem (which there isn't) using your methodology would place the dev under the network engineer. Their software implements the protocol and configuration intent of the NE. Good thing the whole concept is a pile of rubbish. I think you fell into the trap you called out which is thinking at limited scale.It's true. We can all create LAN's at home but I wouldn't dare f with a corporate network and risk shutting down amazon for a day. Which seems to happen quite a bit.... maybe they're DEVOPPING a bit too much.David Rawk Zac Smith • 6 months agoI tend to agree.. they are not under, but beside.. Both require a heap of skill and that includes coding.. but vastly different code.Wilmer • 9 months agoJeff Knupp is to one side of the spectrum. DevOps Reaper is to the other side.ישראל פרוכטר Wilmer • 8 months ago
Enno is more attune to what is really going on. So I won't repeat any of those arguments.
However I will ask you to put me in a box. What am I?
I graduated as a Computer Engineer (hybrid between Electrical Engineering and Computer Science). I don't say that anymore as companies have no idea as to what that means. So I called myself a Digital Electronics and Software Engineer for a while. The repeated question was all too often: "So what are you, software or hardware?"
I spent my first few years working down from board design, writing VHDL and Verilog, to embedded software in C and C++, then algorithms in optimization with the CUDA framework in C, with C++ wrappers and C# for the logic tier. Then worked another few year in particle physics with C++ compute engines with x86 assembly declarations for speed and C# for WPF UIs.
After that I went to work for a wind turbines company as system architect where is was mostly embedded and programming ARM Cortex microprocessors, high power electronics controls, custom service and diagnostics tools in C#. Real-time web based dashboards with Angular, Bootrap, and the likes for a good looking web app.
Nowadays I'm working with mobile first web applications that have a massive backend to power them. It is mostly a .NET stack form Entity Framework, to .NET WebAPI, to Angular power font ends. This company is not a start up but it is a small company. Therefore I wear the many hats. I introduced the new software life cycle with includes continuous integration and continuous deployment. Yes, I manage build servers, build tools, I develop, I'm QA, I'm a tester, I'm a DBA., I'm the deployment and configuration manager.
If you are wondering I have resorted to start calling a full stack developer. It has that edgy sound that companies like to hear. I'm still a young developer. I've only been developing for 10 years.
In my team we are all "Jack of all Trades" and "Masters of Many". We switch tasks and hats because it is fun and keep everyone from getting bored/stuck. Our process is called "Best practices that work for this team".
So, I think of myself as a software engineer. I think I'm a developer. I think I'm DevOps, I think I'm QA.I join you with the lack of title, we don't need those titles (only when HR people are involved, and we need to kind of fake our persona anyhow...)Matt King • a year agoLets start with that DevOps didn't come from startups. It came from Boeing mainly, and a few other major blue chip IT shops, investing heavily in systems management technology around the turn of the century. The goal at the time was simply to change the ratio of servers to IT support personnel, and the re-thinking and re-organizing of development and operations into one organization with one set of common goals. The 'wearing many hats' thing you discuss is a feature of startups, but that feature is independent of siloed or integrated organizations.BosnianDolphin • 2 years ago
I prefer the 'sportzing' analogy of basketball and football. Football has specialist teams that are largely functionally independent because they focus on distinct goals. Basketball has specialist positions, but the whole team is focused on the same goals. I'm not saying one sport is better than the other. I am saying the basketball mentality works better in the IT environment. Delivering the product or service to the customer is the common goal that everyone should be thinking about, and how the details of their job fits into that overall picture. It sounds like to me you are really saying "Hey, its my job and only my job to think about how it all fits together and works"
Secondly, while it is pretty clear that the phrase 'full stack engineer' is about as useful as "Cloud Computing", your perspective that somehow developers are the 'top' of the tree able to do any job is very mistaken. There are key contributors from every specialty who have that ability, and more useful names for them are things like "10x", or "T-shaped". Again, you are describing a real situation, but correlating it with unrelated associations. It is just as likely, and just as valuable, to find an information architect who can also code, or a systems admin that can also diagnose database performance, or an electrician that can also hang sheetrock. Those people do fit your analogy of 'being on top', because they are not siloed and stovepiped into just their speciality.
The DevOps mindset fosters this way of thinking, instead of the old and outdated specialist way of thinking you are defending. Is it possible your emotional reaction is fear based against the possibility that your relative value will decrease if others start thinking outside their boxes?
Interesting to note that Agile also started at Boeing, but 10 years earlier. I live in the startup world of Seattle, but know my history and realize that much of what appears new is actually just 'new to you'(or me), and that most of cutting edge technology and thinking is just combining ideas from other industries in new ways.Safespace Scooter • 2 years ago
Agree on most points, but nobody needs DBA - unless it is some massive project. DBA people should pick up new skills fast.The problem is that developers are trained to crank out code and hope that QA teams will find problems, many times not even sure how to catch holes. DevOps trains people to think critically and do both. It isn't killing developers, it is making them look like noobs while phasing them out.strangedays Safespace Scooter • a year agoYeah, good luck with that attitude. Your company's gonna have a good'ole time looking for and keeping new developer talent. Because as we all know, smart people love working with dummies. I'd love to see 'your QA' team work on our 'spatial collision algorithm' and make our devs "look like noob". You sound like most middle management schmucks.Manachi • 2 years agoFantastic article! I was going to start honing in on the points I particularly agree with but all of it is just spot on. Great post.Ralli Soph • 9 days agoFunniest article so far on full stack. It's a harsh reality for devs, because were asked to do everything, know everything, so how can you really believe QA or DBA can do the job of someone like that? There is a crazy amount of hours a fullstack dev invests in aquiring that kind of knowledge, not to mention some people are also talented at their job. Imagine trying to tell the QA to do that? Maybe for a few hours someone can be a backup just in case something happens, but really it's like replacing the head surgeon.spunky brewster • a month agoThe best skill you can learn in your coding career is your next career. Noone wants a 45 year old coder.hrmilo • 3 months ago
I see so much time wasted learning every new thing when you should just be plugging away to get the job done, bank the $$, and move on. All your accumulated skills will be worthless in a decade or so, and your entire knowledge useless in 2 decades. My ability to turn a wrench is what's keeping me from the poor house. And I have a engineering degree from UIUC! I also don't mind. Think about a 100 week as a plumber with OT in a reasonably priced neighborhood, vs a coder. Who do you think is making more? Now I'm not saying you cant survive into your 50's programming, but typically they get retired forcefully, and permanently.. by a heart attack!
But rambling aside.. the author makes a good point and i think is the future of big companies in tech. The current model is driven by temporary factors. Ideally you'd have a specialized workforce. But I think that as a programmer you are in constant fear of being obsolete so you don't want to be pigeon-holed. It's just not mathematically possible to have that 10,000 hour mastery in 50 different areas.. unless you are Bill Murray in Groundhog Day.
A developer who sees himself at the top of a pyramid. Not surprising, your myopic and egotistical view. I laugh at people who code a few SELECT statements and think they can fill the DBA role. HA HA HA. God, the arrogance. "Well it worked on my machine." - How many sys admins have heard this out of a developers mouth. Unfortunately, projects get stuck with supporting such issues because that very ego has led the developer too far down the road to turn back. They had no common sense or modesty to call on the knowledge of their Sys Ops team to help design the application. I interview jobs candidates all the time calling themselves full stack simply because they compliment their programming language of choice with a mere smattering of knowledge in client-side technologies and can write a few SQL queries. Most developers have NO PERCEPTION of the myriad intricacies it takes to get an application from their unabated desktop with its FULL ADMIN perms and "unlimited resources", through a staging/QA environment, and eventually to the securely locked down production system with limited and perhaps shared or hosted limited resources. Respect of your support teams, communication and coordination, and the knowledge that you do not know it all. THAT'S being Full Stack and DevOps sir.spunky brewster hrmilo • a month agothere's always that one query that noone can do in a way that takes less than 2 hours until u pass it off to a real DBA.. its the 80/20 rule basically. I truly dont believe 'full stack' exists. It's an illusion. There's always something that suffers.Ivan Gavrilyuk • 6 months ago I
The real problem is smart people are in such demand we're forced to adapt to this tribal pre-civilization hodgepodge. Once the industry matures, it'll disappear. Until then they will think they re-invented the wheel.
'm confused here. DevOps roles are strictly automation focused, at least according to all job specifications I see on the internet. They don't need any development skills at all. To me it looks like a new term for what we used to call IT Operations, but more scripting/automation focused. DevOps engineer will need to know Puppet, Chef, Ansible, OS management, public cloud management, know how to set up monitoring, logging and all that stuff usual sysadmin used to do but in the modern world. In fact I used to apply for DevOps roles but quickly changed my mind as it turned out no companies need a person wearing many hats, it has absolutely nothing to do with creating software. Am I wrong?
Mario Bisignani Ivan Gavrilyuk • 4 months agoIt depends on what you mean with development skills. Have you ever tried to automate the deployment of a large web application? In fact the scripts that automate the deployment of large scalable web applications are pretty complex softwares which require in-depth thinking and should follow all the important principles a good developers should know: components isolation, scalability, maintainability, extensibility, etc..Valentin Georgiev • 10 months ago1k% agree with this article!TechZilla • a year agoSuccessful DevOps doesn't mean a full stack developer does it all, that's only true for a broken company that succeeded despite bad organization. For example, Twitter's Dev only culture is downright sick, and ONLY works because they are in the tech field. Mind you, I still believe personally that it works for them DESPITE its unbalanced structure. In other words, bad DevOps means the Dev has no extra resources and just more requirements, yea that sucks!....Adi Chiru • a year ago
BUT, on the flip,
Infrastructure works with QA/Build to define supportable deployment standards, they gotta learn all the automatic bits and practice using them. Now Devs have to package all their applications properly, in the formats supported by QABuild's CI and Repositories (that 'working just fine' install script definitly doesn't count). BUT the Dev's get pre-made CI-ready examples, and if needed, code-migration assistance from the QA/Build team. Pretty soon they learn how to package that type of app, like a J2EE Maven EAR, or a Webdeploy to IIS.... and the rest should be hadled for them, as automaticlly as possible by the proactive operations teams.
Make sense? This is how its supposed to work, It sounds like your left alone in a terrible Dev Only/heavy world. The key to DevOps that is great, and everybody likes, vs. more work... is having a very balanced work flow between the teams, and making sure the pass-off points are VERY well defined. Essentially it requires management that cut the responsibility properly, so they have a shared interest in collaborating. In a Dev heavy organization, the Devs can just throw garbage over the wall, and operations has to react to constant problems... they start to hate each other and ..... Dev managers get the idea that they can cut out ops if they do "DevOps", so then they throw it all at you like right now.Max Börebäck • a year ago
I see in this post so much rubbish and narrow mindedness, so much of the exact stuff that is killing any type of companies. In the last 10 years I had many roles that required of me, as a system engineer, to come in and straighten out all kind of really bad compromises developers did just to make stuff work.
The role never shows the level of intelligence or capabilities. I've seen so many situations in the last 10 years when smart people with the wrong attitude and awareness are too smart for anyone's good and limited people still providing more value than a very smart ones acting as if he is too smart to even have a conversation about anything.
This post is embarrassing for you Jeff, I am sorry for you man.... you just don't get it!A developer do not have to do full stack, the developer can continue with development, but has to adopt some things for packaging,testing, and how it is operated.Peperud • a year ago
Operations can continue with operations, but has to know how things are built and packaged.
Developers and operations needs to share things like use the same application server for example. Developer needs to understand how it is operated to make sure that the code is written in a proper way. Operations needs to adopt in the need for fast delivery and be able to support a controlled way of deploying daily into production.
Here is a complementing post I have around the topic
http://bit.ly/1r3iVffVery much agree with you Jeff. I've been thinking along these lines for awhile now...Lenny Joseph • a year agoI will share my experience , I started off my career teaching Programming which included database programming (Oracle-pl/sql, SQL Server-transact sql) which gave good insights into database internals which landed me into DBA world for last 10 years . During these 10 years where i have worked in technology companies regarded as top-notch , I have seen very smart Developers writing excellent application codes but missing out on writing optimized piece to interact with the database. Hence, I think each Job has a scale and professionals of any group can not do what the top professionals of other group can do. I have seen Developers with fairly good database internals knowledge and I have dbas writing code for their automation which can compares well with features of some commercial database products like TOAD. So , generalization like this does not hold.ceposta • a year agoBTW.. "efficiency" is not the goal... "DG • a year ago
DevOps and the Myth of Efficiency, Part I
http://blog.christianposta....The idea that there is a hierachy of usefulness is bunk. Most developers are horrible at operations because they dislike it. Most sysadmins and DBAs are horrible at coding because they dislike it. People gravitate to what interests them and a disinterested person does a much poorer job than an interested one. DevOps aims to combine roles by removing barriers, but there are costs to quality that no one likes to talk about. Using your hierarchy example most doctors could obtain their RN but they would not make good nurses.Lana Boltneva • 2 years agoSo true! I offer you to check this 6 best practices in DevOps too http://intersog.com/blog/ag...Jonathan McAllister • 2 years agoThis is an excellent article on the general concepts of DevOps and the DevOps movement. It helps to identify the cultural shifts required to facilitate proper DevOps implementations. I also write about DevOps.. I authored a book on implementing CI, CD and DevOps related functions within an organization and it was recently published. The book is aptly titled Mastering Jenkins ( http://www.masteringjenkins... ) and aims to codify not only the architectural implementations and requirements of DevOps but the cultural shift needed to propery advocate for the proper adoption of DevOps practices. Let me know what you think.Chris Kavanagh • 2 years agoI agree. Although I'm not in the business (yet), I will be soon. What I've noticed just playing around with Vagrant and Chef, Puppet, Ansible is the great amount of time to try and master just one of these provisioners. I can't imagine being responsible for all these roles you spoke of in the article. How can one possibly master all of them, and be good at any of them?Sarika Mehta • 2 years agohmmm.... users & business see as one application.... for them how it was developed, deployed does not matter.... IT is an enabler by definition... so DevOps is mostly about that... giving one view to the customer; quick changes, stable changes, stable application....Hanut Singh • 2 years ago
Frankly, DevOps is not about developers or testers... it is about the right architecture, right framework... developers/testers anyways do what is there in the script... DevOps is just a new scripts to them.
For right DevOps, you need right framework. architecture for the whole of the program; you need architecture which is built end to end and not in silos...Quite the interesting read. Having worked as a "Full Stack" Developer , I totally agree with you. Well done sir. My hat is tipped.Masood • 2 years agoSoftware Developer write code that business/customer useDaniel • 2 years ago
Test Developer write test code to test SUT
Release Developer write code to automate release process
Infrastructure developer write code to create infrastructure automatically.
Performance Developer writes code to performance test the SUT
Security Developer writes code to scan the SUT for security
Database Developer write code for DB
So which developer are you thinking DevOps going to kill?
Today's TDD world, a developer (it could be anyone above) needs to get out of their comfort zone to makes sure they write a testable, releasable, deployable, performable, security complaint and maintainable code.
DevOps brings all this roles together to collaborate and deliver.You deserve an award.....Mash -> Dick • 2 years agoWhy wouldn't they be? What are the basic responsibilities that make for a passable DBA and which of those responsibilities cannot be done by a good developer? Say a good developer has just average experience writing stored procs, analyzing query performance, creating (or choosing not to create, for performance reasons) indexes, constraints and triggers, configuring database access rights, setting up regular backups, regular maintenance (ex. rebuilding indexes to avoid fragmentation)... just to name a few.SuperQ Mash • a year ago
I'm sure there's several responsibilities that DBA's have that developers would have very little to no experience in, but we're talking about making for a passable DBA. Developers may not be as good at the job as someone who specializes in it for a living, but the author's wording seems to have been chosen very carefully.
Yup, I see lots of people trying to defend the DBA as a thing, just like people keep trying to defend the traditional sysadmin as a thing. I started my career as a sysadmin in the 90s, but times have changed and I don't call myself a sysadmin anymore, because that's not what I do.
Now I'm a Systems Engineer/SRE. My mode of working isn't slamming software together, but engineering automation to do it for me.
But I also do QA, Data storage performance analysis, networking, and [have a] deep knowledge of the applications I support.
May 15, 2017 | www.upguard.comDevOps and I sort of have a love/hate relationship. DevOps is near and dear to our heart here at UpGuard and there are plenty of things that I love about it . Love it or hate it, there is little doubt that it is here to stay. I've enjoyed a great deal of success thanks to agile software development and DevOps methods, but here are 10 things I hate about DevOps!
#1 Everyone thinks it's about Automation.
#2 "True" DevOps apparently have no processes - because DevOps takes care of that.
#3 The Emergence of the "DevOps' DevOp", a pseudo intellectual loudly spewing theories about distantly unrelated fields that are entirely irrelevant and are designed to make them feel more intelligent and myself more inadequate:
"The Copenhagen interpretation certainly applies to DevOps"
"I'm modeling the relationship between Dev and Ops using quantum entanglement, with a focus on relative quantum superposition - it's the only way to look at it. Why aren't you?"
#4 Enterprise Architects. They used to talk about the "Enterprise Continuum". Now they talk about "The Delivery Continuum" or the "DevOps Continuum". How about talking about the business guys?
#5 Heroes abound with tragic statements like "It took 3 days to automate everything.. it's great now!" - Clearly these people have never worked in a serious enterprise.
#6 No-one talks about Automation failure...it's everywhere. i.e Listen for the words "Pockets of Automation". Adoption of technology, education and adaptation of process is rarely mentioned (or measured).
#7 People constantly pointing to Etsy, Facebook & Netflix as DevOps. Let's promote the stories of companies that better represent the market at large.
#8 Tech hipsters discounting, or underestimating, Windows sysadmins. There are a lot of them and they better represent the Enterprise than many of the higher profile blowhards.
#9 The same hipsters saying their threads have filled up with DevOps tweets where there were none before.
#10 I've never heard of a Project Manager taking on DevOps. I intend on finding one.
What do you think - did I miss anything? Rants encouraged ;-) Please add your comments.
May 15, 2017 | testingthemind.wordpress.comDevOps. The latest software development fad. Now you can be Agile, use Continuous Delivery, and believe in DevOps.
Continuous Delivery (CD), the act of small, frequent, releases was defined in detail by Jez Humble and Dave Farley in their book – Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. The approach makes a lot of sense and encourages a number of healthy behaviors in a team. For example, frequent releases more or less have to be small. Small releases are easier to understand, which in turn increases our chances of building good features, but also our chances of testing for the right risks. If you do run into problems during testing then it's pretty easy to work out the change that caused them, reducing the time to debug and fix issues.
Unfortunately, along with all the good parts of CD we have a slight problem. The book focused on the areas which were considered to be the most broken, and unfortunately that led to the original CD description implying "Done" meant the code was shipped to production. As anyone who has ever worked on software will know, running code in production also requires a fair bit of work.
So, teams started adopting CD but no one was talking about how the Ops team fitted into the release cycle. Everything from knowing when production systems were in trouble, to reliable release systems was just assumed to be fully functional, and unnecessary for explanation.
To try to plug the gap DevOps rose up.
Now, just to make things even more confusing. Dave Farley later said that not talking about Ops was an omission and CD does include the entire development and release cycle, including running in production. So DevOps and CD have some overlap there.
DevOps does take a slightly different angle on the approach than CD. The emphasis for DevOps is on the collaboration rather than the process. Silos should be actively broken down to help developers understand systems well enough to be able to write good, robust and scalable code.
So far so good.
The problem is we now have teams saying they're doing DevOps. By that they mean is they make small, frequent, releases to production AND the developers are working closely with the Ops team to get things out to production and to keep them running.
Sounds good. So what's the problem?
Well, the problem is the name. We now have a term "DevOps" to describe the entire build, test, release approach. The problem is when you call something DevOps anyone who doesn't identify themselves as a dev or as Ops automatically assumes they're not part of the process.
Seriously, go and ask your designers what they think of DevOps. Or how about your testers. Or Product Managers. Or Customer Support.
And that's a problem.
We've managed to take something that is completely dependant on collaboration, and trust, and name it in a way that excludes a significant number of people. All of the name suggestions that arise when you mention this are just ridiculous. DevTestOps? BusinessDevTestOps? DesignDevOps? Aside from just being stupid names these continue to exclude anyone who doesn't have these words in their title.
So do I hate DevOps? Well no, not the practice. I think we should always be thinking about how things will actually work in production. We need an Ops team to help us do that so it makes total sense to have them involved in the process. Just take care with that name.
Is there a solution? Well, in my mindwe're still talking about collaboration above all else. Thinking about CD as "Delivery on demand" also makes more sense to me. We, the whole team, should be ready to deliver working software to the customer when they want it. By being aware of the confusion, and exclusion that some of these names create we can hopefully bring everyone into the project before it's too late.
May 15, 2017 | www.gartner.com
Hype Cycle for DevOps, 2016
DevOps initiatives include a range of technologies and methodologies spanning the software delivery process. IT leaders and DevOps practitioners should proactively understand the readiness and capabilities of technology to identify the most appropriate choices for their specific DevOps initiative.Table of Contents
- What You Need to Know
- The Hype Cycle
- The Priority Matrix
- On the Rise
- DevOps Toolchain Orchestration
- DevOps Toolchain
- Mobile DevOps
- Web-Scale Operations
- Continuous Delivery
- Lean IT
- User and Entity Behavior Analytics
- Continuous Experience
- Application Release Automation
- Mediated APIs
- At the Peak
- Web-Scale Development
- Container Management
- Web-Scale Application Architecture
- Enterprise-Class Agile Development
- Software-Defined Data Center
- Continuous Configuration Automation
- Behavior-Driven Development
- Sliding Into the Trough
- Citizen Developers
- Configuration Auditing
- Software-Defined Networking
- Application Testing Services
- Climbing the Slope
- Application Performance Monitoring Suites
- Test Data Management
- Hype Cycle Phases, Benefit Ratings and Maturity Levels
- Gartner Recommended Reading
May 15, 2017 | en.wikipedia.org
The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win (2013) is the third book by Gene Kim. The business novel tells the story of an IT manager who has ninety days to rescue an over-budget and late IT initiative, code-named The Phoenix Project. The book was co-authored by Kevin Behr and George Spafford and published by IT Revolution Press in January 2013.
The novel is thought of as the modern day version of The Goal by Eliyahu M. Goldratt. The novel describes the problems that almost every IT organization faces, and then shows the practices of how to solve the problems, improve the lives of those who work in IT and be recognized for helping the business win. The goal of the book is to show that a truly collaborative approach between IT and business is possible.
The novel tells the story of Bill, the IT manager at Parts Unlimited. The company's new IT initiative, code named Phoenix Project, is critical to the future of Parts Unlimited, but the project is massively over budget and very late. The CEO wants Bill to report directly to him and fix the mess in ninety days or else Bill's entire department will be outsourced. With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow, streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.
The book has been called a "must read" for IT professionals and quickly reached #1 in its Amazon.com categories. The Phoenix Project was featured on 800 CEO Reads Top 25: What Corporate America Is Reading for June, 2013. InfoQ stated, "This book will resonate at one point or another with anyone who's ever worked in IT." Jeremiah Shirk, Integration & Infrastructure Manager at Kansas State University, said of the book: "Some books you give to friends, for the joy of sharing a great novel. Some books you recommend to your colleagues and employees, to create common ground. Some books you share with your boss, to plant the seeds of a big idea. The Phoenix Project is all three." Other reviewers were more skeptical, including the IT Skeptic "Fictionalising allows you to paint an idealised picture, and yet make it seem real, plausible... Sorry but it is all too good to be true... none of the answers are about people or culture or behaviour. They're about tools and techniques and processes."  Jez Humble (author of Continuous Delivery) said "unlike real life, there aren't many experiments in the book that end up making things worse..."
May 15, 2017 | dzone.com
In a recent webinar, XebiaLabs VP of DevOps Strategy Andrew Phillips sat down with Atos Global Thought Leader in DevOps Dick van der Sar to separate the facts from the fiction. Their findings: most myths come attached with a small piece of fact and vice versa.1. DevOps Is Developers Doing Operations: Myth
An integral part of DevOps' automation component involves a significant amount of code. This causes people to believe Developers do most of the heavy lifting in the equation. In reality, what ends up happening is due to the amount of infrastructure as Code, Ops begin to look a lot like Dev.2. Projects Are Dead: Myth
Projects are an ongoing process of evolving systems and failures. To think they can just be handed off to maintenance forever after completion is simply incorrect. This is only true for tightly scoped software needs, including systems built for specific events. When you adopt DevOps and Agile, you are replacing traditional project-based approaches with a focus on product lifecycles.3. DevOps Doesn't Work in Complex Environments: Myth
DevOps is actually made to thrive in complex environments. The only instance in which it doesn't work is when unrealistic and/or inappropriate goals are set for the enterprise. Complex environments typically suffer due to lack of communication about the state of, and changes to, the interconnected systems. DevOps, on the other hand, encourages communication and collaboration that prevent these issues from arising.4. It's Hard to Sell DevOps to the Business: Myth
The benefits to DevOps are closely tied benefiting the business. However, that's hard to believe when you pitch adopting DevOps as a plan to "stop working on features and sink a lot of your money into playing with shiny new IT tech." Truth is, DevOps is going to impact the entire enterprise. This may be the source of resistance, but as long as you find the balance between adoption and disruption, you will experience a successful transition.5. Agile Is for Lazy Engineers: Myth
DevOps prides itself on eliminating unnecessary overhead. Through automation, your enterprise can see a reduction in documentation, meetings, and even manual tasks, giving team members more time to focus on more important priorities. You know your team is running successfully if their productivity increases.
Nonetheless, DevOps does not come without its own form of "boring" processes, including test plans or code audits. Agile may eliminate waste but that doesn't include the tedious yet necessary aspects.6. If You Can't Code, You Have No Chance in DevOps: Fact
This is only afact because the automation side of DevOps is all Infrastructure as Code (IaC). This typically requires some sort of software development skill such as modularization, automated testing, and Continuous Integration (CI) as IaC. Regardless of scale, automating anything will require, at the very least, software development skills.7. Managers Disappear: Myth
Rather than disappear, managers take a different role with DevOps. In fact, they are still a necessity to the team. Managers are tasked with the responsibility of keeping the entire DevOps team on track. Classic management tasks may seem to disappear but only because the role is changing to be more focused on empowerment.8. DevOps or Die: Fact!
Many of today's market leaders already have some sort of advanced DevOps structure in place. As industries incorporate IT further into their business, we will begin to see DevOps as a basic necessity to the modern business and those that can't adapt will simply fall behind.
That being said, you shouldn't think of DevOps as the magic invincibility potion that will keep your enterprise failure free. Rather, DevOps can prevent many types of failure, but there will always be environment specific threats unique to every organization that DevOps can't rescue you from. Discover how to optimize your DevOps workflows with our cloud-based automated testing infrastructure, brought to you in partnership with Sauce Labs .
May 15, 2017 | vassit.co.ukOut of these misunderstandings several common myths have been created. Acceptance of these myths misleads business further.
Here are some of the most common myths and the facts that debunk them.
Myth 1: DevOps needs agile.
Although DevOps and agile are terms frequently used together, they are a long way away from being synonymous with one another. Agile development refers to a method of software delivery that builds software incrementally, whereas DevOps refers not only to a method of delivery but to a culture, which when adopted, results in many business benefits , including faster software delivery.
DevOps processes can help to compliment agile development, but it is not reliant on agile and can support a range of operation models such as
- Waterfall – where build processes can be optimised and accelerated, and automation can be implemented.
- Agile – where heightened communication between development and operations increases end-product quality.
- Hybrid approach – where speed, quality and compliance are all increased.
For optimum results, full adoption of the DevOps philosophy is necessary.Myth 2: DevOps can't work with legacy.
DevOps is often regarded as a modern concept that helps forward-thinking businesses innovate. Although this is true, it can also help those organisations with long-established, standard IT practices. In fact, with legacy applications there are usually big advantages to DevOps adoption.
Managing legacy care and bringing new software to market quickly; blending stability and agility, is a frequently encountered problem in this new era of digital transformation. Bi-modal IT is an approach where Mode 1 refers to legacy systems focussed on stability, and Mode 2 refers to agile IT focussed on rapid application delivery. DevOps principles are often included exclusively within Mode 2, but automation and collaboration can also be used with success within Mode 1 to increase delivery speed whilst ensuring stability.Myth 3: DevOps is only for continuous delivery.
DevOps doesn't (necessarily) imply continuous delivery. The aim of a DevOps culture is to increase the delivery frequency of an organisation, often from quarterly/monthly to daily releases or more, and improve their ability to respond to changes in the market.
While continuous delivery relies heavily on automation and is aimed at agile and lean thinking organisations, unlike DevOps it is not reliant on a shared culture which enhances collaboration. Gartner summed up the distinction with a report that stated that: "DevOps is not a market, but a tool-centric philosophy that supports a continuous delivery value chain."Myth 4: DevOps requires new tools.
As with the implementation of any new concept or idea, a common misconception about DevOps adoption is that new toolsets, and skills are required. Though the provision of appropriate and relevant tools can aid adoption, organisations are by no means required to replace tools and processes they use to produce software.
DevOps enables organisations to deliver new capabilities more easily, and bring new software into production more rapidly in order to respond to market changes. It is not strictly reliant on new tools to get this job done.Myth 5: DevOps is a skill.
The rapid growth of the DevOps movement has resulted in huge demand for professionals who are skilled within the methodology. However, this fact is often misconstrued to suggest that DevOps is itself a skill – this is not the case.
DevOps is a culture – one that needs to be fully adopted throughout an entire organisation for optimum results, and one that is best supported with appropriate and relevant tools.Myth 6: DevOps is software.
Understanding that DevOps adoption can be better facilitated with software is important, however, maybe more so is understanding that they are not one and the same. Although it is true that there is a significant amount of DevOps software available on the market today, purchasing a specific ad-hoc DevOps product, or even suite of products, will not make your business 'DevOps'.
The DevOps methodology is the communication, collaboration and automation of your development and operations functions, and as described above, is required to be adopted by an entire organisation to achieve optimum results. The software and tools available will undoubtedly reduce the strain of adoption on your business but conscious adoption is required for your business to fully reach the potential that DevOps offers.Conclusion
Like any new and popular term, people have somewhat confused and sometimes contradictory or partial impressions of what DevOps is and how it works.
DevOps is a philosophy which enables businesses to automate their processes and work more collaboratively to achieve a common goal and deliver software more rapidly.
At VASSIT we help organisations to successfully implement DevOps, click here to learn how we made DevOps a reality at TSB bank Sabadell
[Mar 20, 2017] It sucks to be right - blog dot lusis
These are just a few things that jumped out at me (and annoyed me)
However, there are teams at Netflix that do traditional Operations, and teams that do DevOps as well.
Ops is ops is ops. No matter what you call it, Operations is operations.
Notice that we didn't use the typical DevOps tools Puppet or Chef to create builds at runtime
There's no such thing as a "DevOps tool". People were using CFengine, Puppet and Chef long before DevOps was even a term. These are configuration management tools. In fact Adrian has even said they use Puppet in their legacy datacenter:
yet he seems to make the distinction between the ops guys there and the "devops" guys (whatever those are).
There is no ops organization involved in running our cloud…
Just because you outsourced it, doesn't mean it doesn't exist. Oh and it's not your cloud. It's Amazon's.
Reading between the lines
Actually this doesn't take much reading between the lines. It's out there in plain sight:
In reality we had the usual complaints about how long it took to get new capacity, the lack of consistency across supposedly identical systems, and failures in Oracle, in the SAN and the networks, that took the site down too often for too long.
We tried bringing in new ops managers, and new engineers, but they were always overwhelmed by the fire fighting needed to keep the current systems running.
This is largely because the people making decisions are development managers, who have been burned repeatedly by configuration bugs in systems that were supposed to be identical.
The developers used to spend hours a week in meetings with Ops discussing what they needed, figuring out capacity forecasts and writing tickets to request changes for the datacenter.
There is no ops organization involved in running our cloud, no need for the developers to interact with ops people to get things done, and less time spent actually doing ops tasks than developers would spend explaining what needed to be done to someone else.
I'm glad to see this spelled out in such detail. This is what I've been telling people semi-privately for a while now. Because Netflix had such a terrible experience with its operations team, they went to the opposite extreme and disintermediated them.
Imagine you were scared as a kid by a clown. Now imagine you have kids of your own. You hate clowns. You had a bad experience with clowns. But it's your kid's birthday party so here you are making baloon animals, telling jokes and doing silly things to entertain the kids.Just because you aren't wearing makeup doesn't make you any less of a clown. You're doing clown shit. Through the eyes of the kids, you're a clown. Deal with it. Netflix is still doing operations. What should be telling and frightening to operations teams everywhere is this:
The Netflix response to poorly run operations that can't service the business is going to become the norm and not the exception. Evolve or die.
Please note that I don't lay all the blame on the Netflix operations team. I would love to hear the flipside of this story from someone who was there originally when the streaming initiative started. It would probably be full of stories we've heard before - no resources, misalignment of incentives and a whole host of others.
Adrian, thank you for writing the blog post. I hope it serves as a warning to those who come. Hopefully someday you'll be able to see a clown again and not get scared ;)
[Mar 20, 2017] What we mean by "operations," and how it's changed over the years by Mike Loukides
June 7, 2012
Adrian Cockcroft's article about NoOps at Netflix ignited a controversy that has been smouldering for some months. John Allspaw's detailed response to Adrian's article makes a key point: What Adrian described as "NoOps" isn't really. Operations doesn't go away. Responsibilities can, and do, shift over time, and as they shift, so do job descriptions. But no matter how you slice it, the same jobs need to be done, and one of those jobs is operations. What Adrian is calling NoOps at Netflix isn't all that different from Operations at Etsy. But that just begs the question: What do we mean by "operations" in the 21st century? If NoOps is a movement for replacing operations with something that looks suspiciously like operations, there's clearly confusion. Now that some of the passion has died down, it's time to get to a better understanding of what we mean by operations and how it's changed over the years.
At a recent lunch, John noted that back in the dawn of the computer age, there was no distinction between dev and ops. If you developed, you operated. You mounted the tapes, you flipped the switches on the front panel, you rebooted when things crashed, and possibly even replaced the burned out vacuum tubes. And you got to wear a geeky white lab coat. Dev and ops started to separate in the '60s, when programmer/analysts dumped boxes of punch cards into readers, and "computer operators" behind a glass wall scurried around mounting tapes in response to IBM JCL. The operators also pulled printouts from line printers and shoved them in labeled cubbyholes, where you got your output filed under your last name.
The arrival of minicomputers in the 1970s and PCs in the '80s broke down the wall between mainframe operators and users, leading to the system and network administrators of the 1980s and '90s. That was the birth of modern "IT operations" culture. Minicomputer users tended to be computing professionals with just enough knowledge to be dangerous. (I remember when a new director was given the root password and told to "create an account for yourself" … and promptly crashed the VAX, which was shared by about 30 users). PC users required networks; they required support; they required shared resources, such as file servers and mail servers. And yes, BOFH ("Bastard Operator from Hell") serves as a reminder of those days. I remember being told that "no one" else is having the problem you're having - and not getting beyond it until at a company meeting we found that everyone was having the exact same problem, in slightly different ways. No wonder we want ops to disappear. No wonder we wanted a wall between the developers and the sysadmins, particularly since, in theory, the advent of the personal computer and desktop workstation meant that we could all be responsible for our own machines.
But somebody has to keep the infrastructure running, including the increasingly important websites. As companies and computing facilities grew larger, the fire-fighting mentality of many system administrators didn't scale. When the whole company runs on one 386 box (like O'Reilly in 1990), mumbling obscure command-line incantations is an appropriate way to fix problems. But that doesn't work when you're talking hundreds or thousands of nodes at Rackspace or Amazon. From an operations standpoint, the big story of the web isn't the evolution toward full-fledged applications that run in the browser; it's the growth from single servers to tens of servers to hundreds, to thousands, to (in the case of Google or Facebook) millions. When you're running at that scale, fixing problems on the command line just isn't an option. You can't afford letting machines get out of sync through ad-hoc fixes and patches. Being told "We need 125 servers online ASAP, and there's no time to automate it" (as Sascha Bates encountered) is a recipe for disaster.
The response of the operations community to the problem of scale isn't surprising. One of the themes of O'Reilly's Velocity Conference is "Infrastructure as Code." If you're going to do operations reliably, you need to make it reproducible and programmatic. Hence virtual machines to shield software from configuration issues. Hence Puppet and Chef to automate configuration, so you know every machine has an identical software configuration and is running the right services. Hence Vagrant to ensure that all your virtual machines are constructed identically from the start. Hence automated monitoring tools to ensure that your clusters are running properly. It doesn't matter whether the nodes are in your own data center, in a hosting facility, or in a public cloud. If you're not writing software to manage them, you're not surviving.
Furthermore, as we move further and further away from traditional hardware servers and networks, and into a world that's virtualized on every level, old-style system administration ceases to work. Physical machines in a physical machine room won't disappear, but they're no longer the only thing a system administrator has to worry about. Where's the root disk drive on a virtual instance running at some colocation facility? Where's a network port on a virtual switch? Sure, system administrators of the '90s managed these resources with software; no sysadmin worth his salt came without a portfolio of Perl scripts. The difference is that now the resources themselves may be physical, or they may just be software; a network port, a disk drive, or a CPU has nothing to do with a physical entity you can point at or unplug. The only effective way to manage this layered reality is through software.
So infrastructure had to become code. All those Perl scripts show that it was already becoming code as early as the late '80s; indeed, Perl was designed as a programming language for automating system administration. It didn't take long for leading-edge sysadmins to realize that handcrafted configurations and non-reproducible incantations were a bad way to run their shops. It's possible that this trend means the end of traditional system administrators, whose jobs are reduced to racking up systems for Amazon or Rackspace. But that's only likely to be the fate of those sysadmins who refuse to grow and adapt as the computing industry evolves. (And I suspect that sysadmins who refuse to adapt swell the ranks of the BOFH fraternity, and most of us would be happy to see them leave.) Good sysadmins have always realized that automation was a significant component of their job and will adapt as automation becomes even more important. The new sysadmin won't power down a machine, replace a failing disk drive, reboot, and restore from backup; he'll write software to detect a misbehaving EC2 instance automatically, destroy the bad instance, spin up a new one, and configure it, all without interrupting service. With automation at this level, the new "ops guy" won't care if he's responsible for a dozen systems or 10,000. And the modern BOFH is, more often than not, an old-school sysadmin who has chosen not to adapt.
James Urquhart nails it when he describes how modern applications, running in the cloud, still need to be resilient and fault tolerant, still need monitoring, still need to adapt to huge swings in load, etc. But he notes that those features, formerly provided by the IT/operations infrastructures, now need to be part of the application, particularly in "platform as a service" environments. Operations doesn't go away, it becomes part of the development. And rather than envision some sort of uber developer, who understands big data, web performance optimization, application middleware, and fault tolerance in a massively distributed environment, we need operations specialists on the development teams. The infrastructure doesn't go away - it moves into the code; and the people responsible for the infrastructure, the system administrators and corporate IT groups, evolve so that they can write the code that maintains the infrastructure. Rather than being isolated, they need to cooperate and collaborate with the developers who create the applications. This is the movement informally known as "DevOps."
Amazon's EBS outage last year demonstrates how the nature of "operations" has changed. There was a marked distinction between companies that suffered and lost money, and companies that rode through the outage just fine. What was the difference? The companies that didn't suffer, including Netflix, knew how to design for reliability; they understood resilience, spreading data across zones, and a whole lot of reliability engineering. Furthermore, they understood that resilience was a property of the application, and they worked with the development teams to ensure that the applications could survive when parts of the network went down. More important than the flames about Amazon's services are the testimonials of how intelligent and careful design kept applications running while EBS was down. Netflix's ChaosMonkey is an excellent, if extreme, example of a tool to ensure that a complex distributed application can survive outages; ChaosMonkey randomly kills instances and services within the application. The development and operations teams collaborate to ensure that the application is sufficiently robust to withstand constant random (and self-inflicted!) outages without degrading.
On the other hand, during the EBS outage, nobody who wasn't an Amazon employee touched a single piece of hardware. At the time, JD Long tweeted that the best thing about the EBS outage was that his guys weren't running around like crazy trying to fix things. That's how it should be. It's important, though, to notice how this differs from operations practices 20, even 10 years ago. It was all over before the outage even occurred: The sites that dealt with it successfully had written software that was robust, and carefully managed their data so that it wasn't reliant on a single zone. And similarly, the sites that scrambled to recover from the outage were those that hadn't built resilience into their applications and hadn't replicated their data across different zones.
In addition to this redistribution of responsibility, from the lower layers of the stack to the application itself, we're also seeing a redistribution of costs. It's a mistake to think that the cost of operations goes away. Capital expense for new servers may be replaced by monthly bills from Amazon, but it's still cost. There may be fewer traditional IT staff, and there will certainly be a higher ratio of servers to staff, but that's because some IT functions have disappeared into the development groups. The bonding is fluid, but that's precisely the point. The task - providing a solid, stable application for customers - is the same. The locations of the servers on which that application runs, and how they're managed, are all that changes.
One important task of operations is understanding the cost trade-offs between public clouds like Amazon's, private clouds, traditional colocation, and building their own infrastructure. It's hard to beat Amazon if you're a startup trying to conserve cash and need to allocate or deallocate hardware to respond to fluctuations in load. You don't want to own a huge cluster to handle your peak capacity but leave it idle most of the time. But Amazon isn't inexpensive, and a larger company can probably get a better deal taking its infrastructure to a colocation facility. A few of the largest companies will build their own datacenters. Cost versus flexibility is an important trade-off; scaling is inherently slow when you own physical hardware, and when you build your data centers to handle peak loads, your facility is underutilized most of the time. Smaller companies will develop hybrid strategies, with parts of the infrastructure hosted on public clouds like AWS or Rackspace, part running on private hosting services, and part running in-house. Optimizing how tasks are distributed between these facilities isn't simple; that is the province of operations groups. Developing applications that can run effectively in a hybrid environment: that's the responsibility of developers, with healthy cooperation with an operations team.
The use of metrics to monitor system performance is another respect in which system administration has evolved. In the early '80s or early '90s, you knew when a machine crashed because you started getting phone calls. Early system monitoring tools like HP's OpenView provided limited visibility into system and network behavior but didn't give much more information than simple heartbeats or reachability tests. Modern tools like DTrace provide insight into almost every aspect of system behavior; one of the biggest challenges facing modern operations groups is developing analytic tools and metrics that can take advantage of the data that's available to predict problems before they become outages. We now have access to the data we need, we just don't know how to use it. And the more we rely on distributed systems, the more important monitoring becomes. As with so much else, monitoring needs to become part of the application itself. Operations is crucial to success, but operations can only succeed to the extent that it collaborates with developers and participates in the development of applications that can monitor and heal themselves.
Success isn't based entirely on integrating operations into development. It's naive to think that even the best development groups, aware of the challenges of high-performance, distributed applications, can write software that won't fail. On this two-way street, do developers wear the beepers, or IT staff? As Allspaw points out, it's important not to divorce developers from the consequences of their work since the fires are frequently set by their code. So, both developers and operations carry the beepers. Sharing responsibilities has another benefit. Rather than finger-pointing post-mortems that try to figure out whether an outage was caused by bad code or operational errors, when operations and development teams work together to solve outages, a post-mortem can focus less on assigning blame than on making systems more resilient in the future. Although we used to practice "root cause analysis" after failures, we're recognizing that finding out the single cause is unhelpful. Almost every outage is the result of a "perfect storm" of normal, everyday mishaps. Instead of figuring out what went wrong and building procedures to ensure that something bad can never happen again (a process that almost always introduces inefficiencies and unanticipated vulnerabilities), modern operations designs systems that are resilient in the face of everyday errors, even when they occur in unpredictable combinations.
In the past decade, we've seen major changes in software development practice. We've moved from various versions of the "waterfall" method, with interminable up-front planning, to "minimum viable product," continuous integration, and continuous deployment. It's important to understand that the waterfall and methodology of the '80s aren't "bad ideas" or mistakes. They were perfectly adapted to an age of shrink-wrapped software. When you produce a "gold disk" and manufacture thousands (or millions) of copies, the penalties for getting something wrong are huge. If there's a bug, you can't fix it until the next release. In this environment, a software release is a huge event. But in this age of web and mobile applications, deployment isn't such a big thing. We can release early, and release often; we've moved from continuous integration to continuous deployment. We've developed techniques for quick resolution in case a new release has serious problems; we've mastered A/B testing to test releases on a small subset of the user base.
All of these changes require cooperation and collaboration between developers and operations staff. Operations groups are adopting, and in many cases, leading in the effort to implement these changes. They're the specialists in resilience, in monitoring, in deploying changes and rolling them back. And the many attendees, hallway discussions, talks, and keynotes at O'Reilly's Velocity conference show us that they are adapting. They're learning about adopting approaches to resilience that are completely new to software engineering; they're learning about monitoring and diagnosing distributed systems, doing large-scale automation, and debugging under pressure. At a recent meeting, Jesse Robbins described scheduling EMT training sessions for operations staff so that they understood how to handle themselves and communicate with each other in an emergency. It's an interesting and provocative idea, and one of many things that modern operations staff bring to the mix when they work with developers.
What does the future hold for operations? System and network monitoring used to be exotic and bleeding-edge; now, it's expected. But we haven't taken it far enough. We're still learning how to monitor systems, how to analyze the data generated by modern monitoring tools, and how to build dashboards that let us see and use the results effectively. I've joked about "using a Hadoop cluster to monitor the Hadoop cluster," but that may not be far from reality. The amount of information we can capture is tremendous, and far beyond what humans can analyze without techniques like machine learning.
Likewise, operations groups are playing a huge role in the deployment of new, more efficient protocols for the web, like SPDY. Operations is involved, more than ever, in tuning the performance of operating systems and servers (even ones that aren't under our physical control); a lot of our "best practices" for TCP tuning were developed in the days of ISDN and 56 Kbps analog modems, and haven't been adapted to the reality of Gigabit Ethernet, OC48* fiber, and their descendants. Operations groups are responsible for figuring out how to use these technologies (and their successors) effectively. We're only beginning to digest IPv6 and the changes it implies for network infrastructure. And, while I've written a lot about building resilience into applications, so far we've only taken baby steps. There's a lot there that we still don't know. Operations groups have been leaders in taking best practices from older disciplines (control systems theory, manufacturing, medicine) and integrating them into software development.
And what about NoOps? Ultimately, it's a bad name, but the name doesn't really matter. A group practicing "NoOps" successfully hasn't banished operations. It's just moved operations elsewhere and called it something else. Whether a poorly chosen name helps or hinders progress remains to be seen, but operations won't go away; it will evolve to meet the challenges of delivering effective, reliable software to customers. Old-style system administrators may indeed be disappearing. But if so, they are being replaced by more sophisticated operations experts who work closely with development teams to get continuous deployment right; to build highly distributed systems that are resilient; and yes, to answer the pagers in the middle of the night when EBS goes down. DevOps.
Adrian Cockcroft's Blog Ops, DevOps and PaaS (NoOps) at Netflix
March 19, 2012
There has been a sometimes heated discussion on twitter about the term NoOps recently, and I've been quoted extensively as saying that NoOps is the way developers work at Netflix. However, there are teams at Netflix that do traditional Operations, and teams that do DevOps as well. To try and clarify things I need to explain the history and current practices at Netflix in chunks of more than 140 characters at a time.
When I joined Netflix about five years ago, I managed a development team, building parts of the web site. We also had an operations team who ran the systems in the single datacenter that we deployed our code to. The systems were high end IBM P-series virtualized machines with storage on a virtualized Storage Area Network. The idea was that this was reliable hardware with great operational flexibility so that developers could assume low failure rates and concentrate on building features. In reality we had the usual complaints about how long it took to get new capacity, the lack of consistency across supposedly identical systems, and failures in Oracle, in the SAN and the networks, that took the site down too often for too long.
At that time we had just launched the streaming service, and it was still an experiment, with little content and no TV device support. As we grew streaming over the next few years, we saw that we needed higher availability and more capacity, so we added a second datacenter. This project took far longer than initial estimates, and it was clear that deploying capacity at the scale and rates we were going to need as streaming took off was a skill set that we didn't have in-house. We tried bringing in new ops managers, and new engineers, but they were always overwhelmed by the fire fighting needed to keep the current systems running.
Netflix is a developer oriented culture, from the top down. I sometimes have to remind people that our CEO Reed Hastings was the founder and initial developer of Purify, which anyone developing serious C++ code in the 1990's would have used to find memory leaks and optimize their code. Pure Software merged with Atria and Rational before being swallowed up by IBM. Reed left IBM and formed Netflix. Reed hired a team of very strong software engineers who are now the VPs who run developer engineering for our products. When we were deciding what to do next Reed was directly involved in deciding that we should move to cloud, and even pushing us to build an aggressively cloud optimized architecture based on NoSQL. Part of that decision was to outsource the problems of running large scale infrastructure and building new datacenters to AWS. AWS has far more resources to commit to getting cloud to work and scale, and to building huge datacenters. We could leverage this rather than try to duplicate it at a far smaller scale, with greater certainty of success. So the budget and responsibility for managing AWS and figuring out cloud was given directly to the developer organization, and the ITops organization was left to run its datacenters. In addition, the goal was to keep datacenter capacity flat, while growing the business rapidly by leveraging additional capacity on AWS.
Over the next three years, most of the ITops staff have left and been replaced by a smaller team. Netflix has never had a CIO, but we now have an excellent VP of ITops Mike Kail (@mdkail), who now runs the datacenters. These still support the DVD shipping functions of Netflix USA, and he also runs corporate IT, which is increasingly moving to SaaS applications like Workday. Mike runs a fairly conventional ops team and is usually hiring, so there are sysadmin, database,, storage and network admin positions. The datacenter footprint hasn't increased since 2009, although there have been technology updates, and the over-all size is order-of-magnitude a thousand systems.
As the developer organization started to figure out cloud technologies and build a platform to support running Netflix on AWS, we transferred a few ITops staff into a developer team that formed the core of our DevOps function. They build the Linux based base AMI (Amazon Machine Image) and after a long discussion we decided to leverage developer oriented tools such as Perforce for version control, Ivy for dependencies, Jenkins to automate the build process, Artifactory as the binary repository and to construct a "bakery" that produces complete AMIs that contain all the code for a service. Along with AWS Autoscale Groups this ensured that every instance of a service would be totally identical. Notice that we didn't use the typical DevOps tools Puppet or Chef to create builds at runtime. This is largely because the people making decisions are development managers, who have been burned repeatedly by configuration bugs in systems that were supposed to be identical.
By 2012 the cloud capacity has grown to be order-of-magnitude 10,000 instances, ten times the capacity of the datacenter, running in nine AWS Availability zones (effectively separate datacenters) on the US East and West coast, and in Europe. A handful of DevOps engineers working for Carl Quinn (@cquinn - well known from the Java Posse podcast) are coding and running the build tools and bakery, and updating the base AMI from time to time. Several hundred development engineers use these tools to build code, run it in a test account in AWS, then deploy it to production themselves. They never have to have a meeting with ITops, or file a ticket asking someone from ITops to make a change to a production system, or request extra capacity in advance. They use a web based portal to deploy hundreds of new instances running their new code alongside the old code, put one "canary" instance into traffic, if it looks good the developer flips all the traffic to the new code. If there are any problems they flip the traffic back to the previous version (in seconds) and if it's all running fine, some time later the old instances are automatically removed. This is part of what we call NoOps. The developers used to spend hours a week in meetings with Ops discussing what they needed, figuring out capacity forecasts and writing tickets to request changes for the datacenter. Now they spend seconds doing it themselves in the cloud. Code pushes to the datacenter are rigidly scheduled every two weeks, with emergency pushes in between to fix bugs. Pushes to the cloud are as frequent as each team of developers needs them to be, incremental agile updates several times a week is common, and some teams are working towards several updates a day. Other teams and more mature services update every few weeks or months. There is no central control, the teams are responsible for figuring out their own dependencies and managing AWS security groups that restrict who can talk to who.
Automated deployment is part of the normal process of running in the cloud. The other big issue is what happens if something breaks. Netflix ITops always ran a Network Operations Center (NOC) which was staffed 24x7 with system administrators. They were familiar with the datacenter systems, but had no experience with cloud. If there was a problem, they would start and run a conference call, and get the right people on the call to diagnose and fix the issue. As the Netflix web site and streaming functionality moved to the cloud it became clear that we needed a cloud operations reliability engineering (CORE) team, and that it would be part of the development organization. The CORE team was lucky enough to get Jeremy Edberg (@jedberg - well know from running Reddit) as its initial lead engineer, and also picked up some of the 24x7 shift sysadmins from the original NOC. The CORE team is still staffing up, looking for Site Reliability Engineer skill set, and is the second group of DevOps engineers within Netflix. There is a strong emphasis on building tools too make as much of their processes go away as possible, for example they have no run-books, they develop code instead,
To get themselves out of the loop, the CORE team has built an alert processing gateway. It collects alerts from several different systems, does filtering, has quenching and routing controls (that developers can configure), and automatically routes alerts either to the PagerDuty system (a SaaS application service that manages on call calendars, escalation and alert life cycles) or to a developer team email address. Every developer is responsible for running what they wrote, and the team members take turns to be on call in the PagerDuty rota. Some teams never seem to get calls, and others are more often on the critical path. During a major production outage con call, the CORE team never make changes to production applications, they always call a developer to make the change. The alerts mostly refer to business transaction flows (rather than typical operations oriented Linux level issues) and contain deep links to dashboards and developer oriented Application Performance Management tools like AppDynamics which let developers quickly see where the problem is at the Java method level and what to fix,
The transition from datacenter to cloud also invoked a transition from Oracle, initially to SimpleDB (which AWS runs) and now to Apache Cassandra, which has its own dedicated team. We moved a few Oracle DBAs over from the ITops team and they have become experts in helping developers figure out how to translate their previous experience in relational schemas into Cassandra key spaces and column families. We have a few key development engineers who are working on the Cassandra code itself (an open source Java distributed systems toolkit), adding features that we need, tuning performance and testing new versions. We have three key open source projects from this team available on github.com/Netflix. Astyanax is a client library for Java applications to talk to Cassandra, CassJmeter is a Jmeter plugin for automated benchmarking and regression testing of Cassandra, and Priam provides automated operation of Cassandra including creating, growing and shrinking Cassandra clusters, and performing full and incremental backups and restores. Priam is also written in Java. Finally we have three DevOps engineers maintaining about 55 Cassandra clusters (including many that span the US and Europe), a total of 600 or so instances. They have developed automation for rolling upgrades to new versions, and sequencing compaction and repair operations. We are still developing our Cassandra tools and skill sets, and are looking for a manager to lead this critical technology, as well as additional engineers. Individual Cassandra clusters are automatically created by Priam, and it's trivial for a developer to create their own cluster of any size without assistance (NoOps again). We have found that the first attempts to produce schemas for Cassandra use cases tend to cause problems for engineers who are new to the technology, but with some familiarity and assistance from the Cloud Database Engineering team, we are starting to develop better common patterns to work to, and are extending the Astyanax client to avoid common problems.
In summary, Netflix stil does Ops to run its datacenter DVD business. we have a small number of DevOps engineers embedded in the development organization who are building and extending automation for our PaaS, and we have hundreds of developers using NoOps to get their code and datastores deployed in our PaaS and to get notified directly when something goes wrong. We have built tooling that removes many of the operations tasks completely from the developer, and which makes the remaining tasks quick and self service. There is no ops organization involved in running our cloud, no need for the developers to interact with ops people to get things done, and less time spent actually doing ops tasks than developers would spend explaining what needed to be done to someone else. I think that's different to the way most DevOps places run, but its similar to other PaaS enviroments, so it needs it's own name, NoOps. [Update: the DevOps community argues that although it's different, it's really just a more advanced end state for DevOps, so lets just call it PaaS for now, and work on a better definition of DevOps].
John Allspaw12:20 PMAdrian Cockcroft1:51 PM
Looks like your comments can't be more than 4,096 chars on this blog. :)
So here's my comment: https://gist.github.com/2140086Thanks John. I agree with some of what you point out. Netflix in effect over-reacted to a dysfunctional ops organization. I think there are several other organizations who would recognize our situation, and would also find a need to over-react to make the solution stick.Edward Capriolo 12:28 PM
Your definition of DevOps seems far broader than the descriptions and definitions I can find by googling or looking on Wikipedia. I don't recognize what we do in those definitions - since they are so focused on the relationship between a Dev org and an Ops org, so someone should post an updated definition to Wikipedia or devops.com.
Until then maybe I'll just call it NetflOps or botchagalops.I have a loaded NoOps question for you :) I am very interesting in understanding how a decentralized environments he said-she said issues get solved. For example, I know netflix uses horizontally scalable rest layers as integration points.Adrian Cockcroft 1:44 PM
Suppose one team/application is having an intermittent problem/bug with another team/application. Team 1 opens a issue. Team 2 reads investigates closes the issue as not a problem. Team 1 double checks and reasserts the issue is Team 2.
In a decentralized environment how is this road block cleared?
As an ops person I spend a good deal of time chasing down problems very external to me. I accept this as an ops person. Since developers are pressed into ops how much time will a developer spend on another teams reported problems. Will team 1 forgo there own scrum deadlines this week because team 2 sucks up all their time reporting bogus problems?We aren't decentralized. So in the scenario you mention everyone gets in a room and figures it out, or we just end up with a long email thread if it's less serious. APM tools help pinpoint what is going on at the request level down to Java code. Once we have root cause someone files a Jira to fix the issue. There is a manager rotation for centrally prioritizing and coordinating response to major outages. (I'm on duty this week, it comes up every few months.)@sixfootdad 2:44 PM
We have a few people who have "mad wireshark skills" to debug network layer problems, but that's infrequent and I'm hopeful that boundary.com will come up with better tools in this space.
We don't follow a rigid methodology or fixed release deadlines, we ship code frequently enough that delays aren't usually a big issue, and we have a culture of responsible adults so we communicate and respect each others needs across teams. The infrequent large coordinating events like a new country launch are dealt with by picking one manager to own the overall big picture and lead the coordination.Adrian - Great article! I'm always fascinated to read & hear how folks have solved problems that plague a lot of IT organizations.Adrian Cockcroft 4:50 PM
I've got a question about something in your reply above: "we have a culture of responsible adults so we communicate and respect each others needs across teams".
I've found that, time and time again, the most difficult thing about organizational change is the people. How does one go about hiring "responsible adults"? I know that it might sound like a silly or a flippant question, but seriously -- I've lost count of how many times grown up folks act like childish, selfish, spoiled brats.My views on culture may not be much help - read http://perfcap.blogspot.com/2011/12/how-netflix-gets-out-of-way-of.html to see my explanation of how Netflix does things differently.Paul Kelly 1:37 AM
Culture is very hard to create or modify but easy to destroy. This is because everyone has to buy into it for it to be effective, and then every manager has to hire only people who are compatible with the culture, and also get rid of people who turn out not to fit in, even if they are doing good work.
So the short answer is start a new company from scratch with the culture you want, and pay a lot of attention to who you hire. I don't think it is possible to do a culture shift if there are more than a roomful of people involved.Part of getting a "culture of responsible adults" together is partly down to "culture" - although it helps to have mature sensible individuals, fostering that also means avoiding finger-pointing and blame.Adrian Cockcroft 4:53 PM
The more defensive people are made to feel, the more likely they are to start throwing tantrums when under pressure. A culture where you can put your hand up and say: "I got that wrong, how can we put it right?" gets better results in the long term than one where you might be fired or disciplined for a genuine mistake.
I always wondered how evil geniuses like Ernst Blofeld recruit when getting it wrong means you might end up in the shark tank...Yes, incident reviews that don't involve blame and finger-pointing are also key. Making the same mistake several times, trying to hide your mistakes, or clear lapses of judgement can't be tolerated though.Jason6:02 PMGreat article Adrian. I have a question. Is a consecutive IP space important? Since AWS EIP doesn't guarantee consecutive addresses, I've wondered if this mattered to app developers.Adrian Cockcroft 11:16 AM
Anything that could have been done by subnet is out the window. For example if you wanted to do port sweeps of your network blocks for an audit, perform penetration testing, or parsing logs by IPs. I suppose this could be done pragmatically but was curious about your experiences. Does it matter?If the network topology matters you can use VPC to manage it. Also if you are a big enough customer to have an enterprise level support contract with AWS and use a lot of EIPs it is possible to get them allocated in contiguous blocks.manul 9:10 AMSome thoughts by me on the topic of NoOps and DevOps and the future of operations and the need for operations http://imansson.wordpress.com/2012/03/21/35/Sudsy11:44 AMI'm curious about your statement "Notice that we didn't use the typical DevOps tools Puppet or Chef to create builds at runtime. This is largely because the people making decisions are development managers, who have been burned repeatedly by configuration bugs in systems that were supposed to be identical."Adrian Cockcroft12:09 PM
If systems are built from the same puppet manifests, what kind of configuration bugs can occur? Also, how is the alternative method you choose any less likely to cause the same problems?Puppet is overkill for what we end up doing. We have a base AMI that is always identical, we install an rpm on it or untar a file once, rebake the AMI and we are done. That AMI is then turned into running systems using the AWS autoscaler. It's more reliable because there are fewer things to go wrong at boot time, no dependencies on repo's or orchestration tools, we know that the install had completely succeeded the same way on every instance before the instance booted.TimFraser1:18 PM
Adrian, Last time we talked you mentioned that you were not 100% transitioned to AWS, but were - looks like this was achieved, with the exception for the core DVD Business.
As many companies are talking about Hyrbrid cloud as a target state, and Netflix went through the transition from Pvt/Managed Ops to Public-AWS/NoOps, can you talk about the interim state you were in and what development and operations use cases you optimized around during transition – ala, what were the Hybrid models and principles you all followed – to move from Ops to NoOps. Did you attempt to keep these teams and tools separate or did you all try to create a transition strategy that allowed you to hedge the bets with AWS and the 'all-in strategy" to possibly come back if needed etc..
What Ops Governance and Dev Tooling approach did you take in the state? Specifically around cloud abstraction layers to ease the Access management, support, tooling, elasticity needs. Can you shed some light on the thinking and approach you took while you were in mid-state of transition?
Also can you comment on the how much do you govern and drive the development and deployment approach so you can unify the continuous Integration and Continuous deployment tools so that you can reduce the chaos in this space?
Adrian Cockcroft12:15 PM
If you look at the presentations on slideshare.net/adrianco I have discussed in some detail what the transition strategy and tools looked like. We continued to run the old code in the datacenter as we gradually moved functionality away from it. The old DC practices were left intact as we moved developers to the cloud one group at a time.
It seems to me that rather than NoOps, this is an outsourcing of ops. I assume that the issues you had with Ops came from the need to control cost, scale etc? If not, please would you clarify?
If it was, how has moving to AWS solved your problems? Is it that AWS can scale more readily based on experience to date? Was cost control an issue before, and is it now, in terms of Ops
Given that NoOps = outsourcing, I can see that managing the relationship with the service provider becomes vital for you?Thanks in advance, interesting stuff!
DevOps Stackify 8:43 PM
Dev and admin teams struggle these days with keeping up with agile development. DevOps helps by breaking down some of the walls. But one of the biggest challenges is getting the entire development team involved and not just 1 or 2 people who help do deployments.
The entire team needs visibility to the production server environments to help support and troubleshoot applications.