Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Accidental Shutdowns/Reboot Blunders

News Sysadmin Horror Stories Recommended Links Performing the operation on a wrong server Simple Unix Backup Tools Unix mv command
Missing backup Rush/absence of testing Creative uses of rm  Pure stupidity  Multiple sysadmin working on the same box  Locking yourself out
Safe-rm Typical Errors In Using Find Tips Unix History Humor Etc

Accidental reboot is a serious blunder, especially if you rebooted production box in the middle of the day. This is one of CLM (career limiting moves) for any administrator.  Typically his happens with remote sessions when the administrator opens a shell to one box, then open one to another box. Then became distracted, after which returns to the wrong terminal window and send the wrong command to the wrong machine. Nobody is perfect.

The rule before issuing any reboot command should be classic "Stop. Think. Click". You also can redefine reboot to provide a warning with the name of the box and pause.

Stop. Think. Click

Such script are usually called molly-guard. Originally used of the plexiglass covers improvised for the BRS on an IBM 4341 mainframe after a programmer's toddler daughter (named Molly) tripped it twice in one day. Later generalized to covers over stop/reset switches on disk drives and networking equipment.

Ubuntu has a shell script called molly-guard which guards against accidental shutdowns-reboots

NAME

       molly-guard - guard against accidental shutdowns/reboots

SYNOPSIS

       shutdown [-hV] [--molly-guard-do-nothing] [-- script_options]

       halt [-hV] [--molly-guard-do-nothing] [-- script_options]

       reboot [-hV] [--molly-guard-do-nothing] [-- script_options]

       poweroff [-hV] [--molly-guard-do-nothing] [-- script_options]

DESCRIPTION

       molly-guard attempts to prevent you from accidentally shutting down or
       rebooting machines. It does this by injecting a couple of checks before
       the existing commands: halt, reboot, shutdown, and poweroff. This
       happens via scripts with the same names in /usr/sbin, so it only works
       if you have /usr/sbin before /sbin in your PATH!

       Before molly-guard invokes the real command, all scripts in
       /etc/molly-guard/run.d/ have to run and exit successfully; else, it
       aborts the command.  run-parts(1) is used to process the directory.

       molly-guard passes any script_options to the scripts, and also
       populates the environment with the following variables:

       ∑   MOLLYGUARD_CMD - the actual command invoked by the user.

       ∑   MOLLYGUARD_DO_NOTHING - set to 1 if this is a demo-run.

       ∑   MOLLYGUARD_SETTINGS - the path to a shell script snippet which
           scripts can source to obtain settings.

       molly-guard prints the contents of /etc/molly-guard/messages.d/COMMAND
       or /etc/molly-guard/messages.d/default to the console, if either
       exists. This is due to /etc/molly-guard/run.d/10-print-message.

GUARDING SSH SESSIONS

       molly-guard was primarily designed to shield SSH connections. This
       functionality (which should arguably be provided by the openssh-server
       package) is implemented in /etc/molly-guard/run.d/30-query-hostname.

       This script first tests whether the command is being executed from a
       tty which has been created by sshd. It also checks whether the variable
       SSH_CONNECTION is defined. If any of these tests are successful, test
       script queries the user for the machineīs hostname, which should be
       sufficient to prevent the user from doing something by accident.

       You can pass the --pretend-ssh script option to molly-guard to pretend
       that those tests succeeds. Alternatively, setting ALWAYS_QUERY_HOSTNAME
       in /etc/molly-guard/rc causes the script to always query.

       The following situations are still UNGUARDED. If you can think of ways
       to protect against those, please let me know!

       ∑   running sudo within screen or screen within sudo; sudo eats the
           SSH_CONNECTION variable, and screen creates a new pty.

       ∑   executing those command in a remote terminal window, that is a
           XTerm started on a remote machine but displaying on the local X
           server.

       You have been warned. You can use the --molly-guard-do-nothing switch
       to prevent anything from happening, e.g.  halt
       --molly-guard-do-nothing.

OPTIONS

       --molly-guard-do-nothing
           Cause molly-guard to print the command which would be executed,
           after processing all scripts, instead of executing it.

       -h, --help
           Display usage information.

       -V, --version
           Display version information.

SEE ALSO

       shutdown(8), halt(1), reboot(8), poweroff(8).

LEGALESE

       molly-guard is copyright by martin f. krafft. Andrew Ruthven came up
       with the idea of using the scripts directory and submitted a patch,
       which I modified a bit.

       This manual page was written by martin f. krafft <madduck@madduck.net>.

       Permission is granted to copy, distribute and/or modify this document
       under the terms of the Artistic License 2.0

COPYRIGHT

       Copyright © 2008 martin f. krafft

 

NEWS CONTENTS

Old News

Accidental shutdown

Re: Accidental shutdown

by Todd A. Jacobson 2009-05-21T20:46:53+00:00.

On Thu, May 21, 2009 at 12:31:47AM +0100, Bhasker C V wrote:
> I can rename and shell wrap the binaries poweroff/shutdown/reboot but
> that would not be a clean method and I am sure there should be much
> better way than that.
Nope. You could disable the reboot command in your sudoers file, but
that isn't going to prevent you from rebooting the wrong machine if you
really make an effort.
You might also consider editing sudoers to change the sudo password
prompt to include the hostname of the box you're on, so that you're less
likely to issue commands to the wrong box if you're paying attention.
However, the real problem here is that you're assuming Linux should
protect you from yourself. It won't; part of being a power user is not
running privileged commands without exercising due care. With power
comes responsibility!
As has been said before: "*nix is user friendly. It's just picky about
who its friends are!"
--
"Oh, look: rocks!"
-- Doctor Who, "Destiny of the Daleks"

by Scott Giffordon 2009-05-21T20:51:53+00:00.

Bhasker C V writes:
> Is there a method to prevent accidental powerdown of a linux box ?
> or atleast alert ?
If you get in the habit of running "shutdown -r +1" instead of
"reboot", it will warn users for 1 minute before shutting down the
server. That should give you enough time to run "shutdown -c" to
cancel the shutdown if you realize it's on the wrong machine.
Hope this helps,
-----Scott.

[Jun 16, 2010] Prevent Accidental Shutdown/Reboot in Ubuntu

Jan 15, 2010 | Linux Today

blackhole

Re: Solution looking for a problem
> How exactly does someone "accidentally" issue a shutdown or reboot command? ... Failing that highly likely scenario, this is someone shopping around a solution for a problem that doesn't really exist. Give me a break.

I haven't checked out the actual package in question, but based on the fact that (according to the posted output) it notes the connection is via SSH and asks for a hostname, I would say the author of the article did not articulate well what the purpose of the package is. The purpose appears to be to avoid shutting down the *wrong* computer when connecting remotely.

I've never had that problem, but more than once I've shut down the local computer when I intended to shut down a remote computer. I think after the second time (after I stopped swearing!) I created aliases for halt and reboot that first query with a message like: Really halt {hostname} [yn]?

Marco

Re: Solution looking for a problem
Re: How exactly does someone "accidentally" issue a shutdown or reboot command?

I've done it while I was distracted, open a shell to one box, then open one to another box. Go to lunch. Forget which shell you are using and send the wrong command to the wrong machine. Nobody is perfect.

[Jun 14, 2010] IT Resource Center forums - greatest blunders

Michael Steele

    When I was first starting out I worked for a Telecom as an 'Application Administrator' and I sat in a small room with a half a dozen other admins and together we took calls from users as their calls escalated up from tier I support. We were tier II in a three tier organization.

    A month earlier someone from tier I confused a production server with a test server and rebooted it in the middle of the day. These servers were remotely connected over a large distance so it can be confusing. Care is needed before rebooting.

    The tier I culprit took a great deal of abuse for this mistake and soon became a victim of several jokes. An outage had been caused in a high availability environment which meant management, interviews, reports; It went on and on and was pretty brutal.

    And I was just as brutal as anyone.

    Their entire organization soon became victimize by everyone from our organization. The abuse traveled right up the management tree and all participated.

    It was hilarious, for us.

    Until I did the same thing a month later.

    There is nothing more humbling then 2000 people all knowing who you are for the wrong reason and I have never longed for anonymity more.

    Now I alway do a 'uname' or 'hostname' before a reboot, even when I'm right in front of it.

[Jun 9, 2010] horror - University of Cambridge Computing Service - Unix Support

(3) At the same institution, we were running a system software that had a serious bug where if anyone had logged out ungracefully, the system wouldn't let any more users onto the system and users who were logged on couldn't execute any new commands. (The newest release of the software later on did fix this bug.) I had to reboot the machine to restore the system to a sane state. I did a wall <<EOF We need to shutdown blah blah... EOF and then shutdown. Well, I should've waited since at the precise moment, one of our users was doing a once-a-year massive conversion of our financial data (talk about bad luck). I had shutdown in the middle of a very long disk write and thus, data was lost. We did recover that data and life went on.

Moral: make damn sure that *no one* is doing anything on your system before you reboot, even if other users are vociferously clamoring for you to reboot.

My 10 UNIX Command Line Mistakes by Vivek Gite

with 90 comments

Anyone who has never made a mistake has never tried anything new. -- Albert Einstein.

Here are a few mistakes that I made while working at UNIX prompt. Some mistakes caused me a good amount of downtime. Most of these mistakes are from my early days as a UNIX admin.

... ... ...

Rebooted Solaris Box

On Linux killall command kill processes by name (killall httpd). On Solaris it kill all active processes. As root I killed all process, this was our main Oracle db box:
killall process-name

Selected Comments

UnixEagle

Rebooted the wrong box

While adding alias to main network interface I ended up changing the main IP address, the system froze right away and I had to call for a reboot

Instead of appending text to Apache config file, I overwritten itís contents

Firewall lockdown while changing the ssh port

Wrongfully run a script contained recursive chmod and chown as root on / caused me a downtime of about 12 hours and a complete re-install

Some mistakes are really silly, and when they happen, you donít believe your self you did that, but every mistake, regardless of itís silliness, should be a learned lesson.

If you did a trivial mistake, you should not just overlook it, you have to think of the reasons that made you did it, like: you didnít have much sleep or your mind was confused about personal life or Ö..etc.

I like Einsteinís quote, you really have to do mistakes to learn.

Recommended Links

molly-guard - Wiktionary

Prevent Accidental Shutdown-Reboot in Ubuntu Linux Adda

Stop. Think. Click



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: November 05, 2016