Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Skepticism and critical thinking is not panacea, but can help to understand the world better

Scriptorama: A Slightly Skeptical View
on Scripting Languages

News Introduction Recommended Books Recommended Links Programming Languages Design Recommended Papers Scripting Languages for Java vs. Pure Java
Software Engineering Anti-OO John Ousterhout Larry Wall Shell Giants Software Prototyping Software Life Cycle Models
Shells AWK Perl Perl Warts and Quirks Python PHP Javascript
Ruby Tcl/Tk R programming language Rexx Lua S-lang JVM-based scripting languages
Pipes Regex Program understanding Brooks law KISS Principle Software Prototyping Unix Component Model
Programming as a Profession Programming style Language design and programming quotes History Humor Random Findings Etc

This is the central page of the Softpanorama WEB site because I am strongly convinced that the development of scripting languages, not the replication of the efforts of BSD group undertaken by Stallman and Torvalds is the central part of open source. See Scripting languages as VHLL for more details.

 
Ordinarily technology changes fast. But programming languages are different: programming languages are not just technology, but what programmers think in.

They're half technology and half religion. And so the median language, meaning whatever language the median programmer uses, moves as slow as an iceberg.

Paul Graham: Beating the Averages

Libraries are more important that the language.

Donald Knuth


Introduction

A fruitful way to think about language development is to consider it a to be special type of theory building. Peter Naur suggested that programming in general is theory building activity in his 1985 paper "Programming as Theory Building". But idea is especially applicable to compilers and interpreters. What Peter Naur failed to understand was that design of programming languages has religious overtones and sometimes represent an activity, which is pretty close to the process of creating a new, obscure cult ;-). Clueless academics publishing junk papers at obscure conferences are high priests of the church of programming languages. Some, like Niklaus Wirth and Edsger W. Dijkstra, (temporary) reached the status close to (false) prophets :-).

On a deep conceptual level building of a new language is a human way of solving complex problems. That means that complier construction in probably the most underappreciated paradigm of programming of large systems. Much more so then greatly oversold object-oriented programming. OO benefits are greatly overstated.

For users, programming languages distinctly have religious aspects, so decisions about what language to use are often far from being rational and are mainly cultural.  Indoctrination at the university plays a very important role. Recently they were instrumental in making Java a new Cobol.

The second important observation about programming languages is that language per se is just a tiny part of what can be called language programming environment. The latter includes libraries, IDE, books, level of adoption at universities,  popular, important applications written in the language, level of support and key players that support the language on major platforms such as Windows and Linux and other similar things.

A mediocre language with good programming environment can give a run for the money to similar superior in design languages that are just naked.  This is  a story behind success of  Java and PHP. Critical application is also very important and this is a story of success of PHP which is nothing but a bastardatized derivative of Perl (with all the most interesting Perl features surgically removed ;-) adapted to creation of dynamic web sites using so called LAMP stack.

Progress in programming languages has been very uneven and contain several setbacks. Currently this progress is mainly limited to development of so called scripting languages.  Traditional high level languages field is stagnant for many decades.  From 2000 to 2017 we observed the huge sucess of Javascript; Python encroached in Perl territory (including genomics/bioinformatics) and R in turn start squeezing Python in several areas. At the same time Ruby despite initial success remained niche language.  PHP still holds its own in web-site design.

Some observations about scripting language design and  usage

At the same time there are some mysterious, unanswered question about factors that help the particular scripting language to increase its user base, or fail in popularity. Among them:

Nothing succeed like success

Those are difficult questions to answer without some way of classifying languages into different categories. Several such classifications exists. First of all like with natural languages, the number of people who speak a given language is a tremendous force that can overcome any real of perceived deficiencies of the language. In programming languages, like in natural languages nothing succeed like success.

The second interesting category is number of applications written in particular language that became part of Linux or, at least, are including in standard RHEL/FEDORA/CENTOS or Debian/Ubuntu repository.

The third relevant category is the number and quality of books for the particular language.

Complexity Curse

History of programming languages raises interesting general questions about the limit of complexity of programming languages. There is strong historical evidence that a language with simpler core, or even simplistic core Basic, Pascal) have better chances to acquire high level of popularity.

The underlying fact here probably is that most programmers are at best mediocre and such programmers tend on intuitive level to avoid more complex, more rich languages and prefer, say, Pascal to PL/1 and PHP to Perl. Or at least avoid it on a particular phase of language development (C++ is not simpler language then PL/1, but was widely adopted because of the progress of hardware, availability of compilers and not the least, because it was associated with OO exactly at the time OO became a mainstream fashion).

Complex non-orthogonal languages can succeed only as a result of a long period of language development (which usually adds complexly -- just compare Fortran IV with Fortran 99; or PHP 3 with PHP 5 ) from a smaller core. Attempts to ride some fashionable new trend extending existing popular language to this new "paradigm" also proved to be relatively successful (OO programming in case of C++, which is a superset of C).

Historically, few complex languages were successful (PL/1, Ada, Perl, C++), but even if they were successful, their success typically was temporary rather then permanent  (PL/1, Ada, Perl). As Professor Wilkes noted   (iee90):

Things move slowly in the computer language field but, over a sufficiently long period of time, it is possible to discern trends. In the 1970s, there was a vogue among system programmers for BCPL, a typeless language. This has now run its course, and system programmers appreciate some typing support. At the same time, they like a language with low level features that enable them to do things their way, rather than the compiler’s way, when they want to.

They continue, to have a strong preference for a lean language. At present they tend to favor C in its various versions. For applications in which flexibility is important, Lisp may be said to have gained strength as a popular programming language.

Further progress is necessary in the direction of achieving modularity. No language has so far emerged which exploits objects in a fully satisfactory manner, although C++ goes a long way. ADA was progressive in this respect, but unfortunately it is in the process of collapsing under its own great weight.

ADA is an example of what can happen when an official attempt is made to orchestrate technical advances. After the experience with PL/1 and ALGOL 68, it should have been clear that the future did not lie with massively large languages.

I would direct the reader’s attention to Modula-3, a modest attempt to build on the appeal and success of Pascal and Modula-2 [12].

Complexity of the compiler/interpreter also matter as it affects portability: this is one thing that probably doomed PL/1 (and later Ada), although those days a new language typically come with open source compiler (or in case of scripting languages, an interpreter) and this is less of a problem.

Here is an interesting take on language design from the preface to The D programming language book 9D language failed to achieve any significant level of popularity):

Programming language design seeks power in simplicity and, when successful, begets beauty.

Choosing the trade-offs among contradictory requirements is a difficult task that requires good taste from the language designer as much as mastery of theoretical principles and of practical implementation matters. Programming language design is software-engineering-complete.

D is a language that attempts to consistently do the right thing within the constraints it chose: system-level access to computing resources, high performance, and syntactic similarity with C-derived languages. In trying to do the right thing, D sometimes stays with tradition and does what other languages do, and other times it breaks tradition with a fresh, innovative solution. On occasion that meant revisiting the very constraints that D ostensibly embraced. For example, large program fragments or indeed entire programs can be written in a well-defined memory-safe subset of D, which entails giving away a small amount of system-level access for a large gain in program debuggability.

You may be interested in D if the following values are important to you:

The role of fashion

At the initial, the most difficult stage of language development the language should solve an important problem that was inadequately solved by currently popular languages.  But at the same time the language has few chances to succeed unless it perfectly fits into the current software fashion. This "fashion factor" is probably as important as several other factors combined. With the notable exclusion of "language sponsor" factor.  The latter can make or break the language.

Like in woman dress fashion rules in language design.  And with time this trend became more and more pronounced.  A new language should simultaneously represent the current fashionable trend.  For example OO-programming was a visit card into the world of "big, successful languages" since probably early 90th (C++, Java, Python).  Before that "structured programming" and "verification" (Pascal, Modula) played similar role.

Programming environment and the role of "powerful sponsor" in language success

PL/1, Java, C#, Ada, Python are languages that had powerful sponsors. Pascal, Basic, Forth, partially Perl (O'Reilly was a sponsor for a short period of time)  are examples of the languages that had no such sponsor during the initial period of development.  C and C++ are somewhere in between.

But language itself is not enough. Any language now need a "programming environment" which consists of a set of libraries, debugger and other tools (make tool, lint, pretty-printer, etc). The set of standard" libraries and debugger are probably two most important elements. They cost  lot of time (or money) to develop and here the role of powerful sponsor is difficult to underestimate.

While this is not the necessary condition for becoming popular, it really helps: other things equal the weight of the sponsor of the language does matter. For example Java, being a weak, inconsistent language (C-- with garbage collection and OO) was pushed through the throat on the strength of marketing and huge amount of money spend on creating Java programming environment. 

The same was partially true for  C# and Python. That's why Python, despite its "non-Unix" origin is more viable scripting language now then, say, Perl (which is better integrated with Unix and has pretty innovative for scripting languages support of pointers and regular expressions), or Ruby (which has support of coroutines from day 1, not as "bolted on" feature like in Python).

Like in political campaigns, negative advertizing also matter. For example Perl suffered greatly from blackmail comparing programs in it with "white noise". And then from withdrawal of O'Reilly from the role of sponsor of the language (although it continue to milk that Perl book publishing franchise ;-)

People proved to be pretty gullible and in this sense language marketing is not that different from woman clothing marketing :-)

Language level and success

One very important classification of programming languages is based on so called the level of the language.  Essentially after there is at least one language that is successful on a given level, the success of other languages on the same level became more problematic. Higher chances for success are for languages that have even slightly higher, but still higher level then successful predecessors.

The level of the language informally can be described as the number of statements (or, more correctly, the number of  lexical units (tokens)) needed to write a solution of a particular problem in one language versus another. This way we can distinguish several levels of programming languages:

 "Nanny languages" vs "Sharp razor" languages

Some people distinguish between "nanny languages" and "sharp razor" languages. The latter do not attempt to protect user from his errors while the former usually go too far... Right compromise is extremely difficult to find.

For example, I consider the explicit availability of pointers as an important feature of the language that greatly increases its expressive power and far outweighs risks of errors in hands of unskilled practitioners.  In other words attempts to make the language "safer" often misfire.

Expressive style of the languages

Another useful typology is based in expressive style of the language:

Those categories are not pure and somewhat overlap. For example, it's possible to program in an object-oriented style in C, or even assembler. Some scripting languages like Perl have built-in regular expressions engines that are a part of the language so they have functional component despite being procedural. Some relatively low level languages (Algol-style languages) implement garbage collection. A good example is Java. There are scripting languages that compile into common language framework which was designed for high level languages. For example, Iron Python compiles into .Net.

Weak correlation between quality of design and popularity

Popularity of the programming languages is not strongly connected to their quality. Some languages that look like a collection of language designer blunders (PHP, Java ) became quite popular. Java became  a new Cobol and PHP dominates dynamic Web sites construction. The dominant technology for such Web sites is often called LAMP, which means Linux - Apache -MySQL- PHP. Being a highly simplified but badly constructed subset of Perl, kind of new Basic for dynamic Web sites construction PHP provides the most depressing experience. I was unpleasantly surprised when I had learnt that the Wikipedia engine was rewritten in PHP from Perl some time ago, but this fact quite illustrates the trend.

So language design quality has little to do with the language success in the marketplace. Simpler languages have more wide appeal as success of PHP (which at the beginning was at the expense of Perl) suggests. In addition much depends whether the language has powerful sponsor like was the case with Java (Sun and IBM) as well as Python (Google).

Progress in programming languages has been very uneven and contain several setbacks like Java. Currently this progress is usually associated with scripting languages. History of programming languages raises interesting general questions about "laws" of programming language design. First let's reproduce several notable quotes:

  1. Knuth law of optimization: "Premature optimization is the root of all evil (or at least most of it) in programming." - Donald Knuth
  2. "Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp." - Phil Greenspun
  3. "The key to performance is elegance, not battalions of special cases." - Jon Bentley and Doug McIlroy
  4. "Some may say Ruby is a bad rip-off of Lisp or Smalltalk, and I admit that. But it is nicer to ordinary people." - Matz, LL2
  5. Most papers in computer science describe how their author learned what someone else already knew. - Peter Landin
  6. "The only way to learn a new programming language is by writing programs in it." - Kernighan and Ritchie
  7. "If I had a nickel for every time I've written "for (i = 0; i < N; i++)" in C, I'd be a millionaire." - Mike Vanier
  8. "Language designers are not intellectuals. They're not as interested in thinking as you might hope. They just want to get a language done and start using it." - Dave Moon
  9. "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay
  10. "Programs must be written for people to read, and only incidentally for machines to execute." - Abelson & Sussman, SICP, preface to the first edition

Please note that one thing is to read language manual and appreciate how good the concepts are, and another to bet your project on a new, unproved language without good debuggers, manuals and, what is very important, libraries. Debugger is very important but standard libraries are crucial: they represent a factor that makes or breaks new languages.

In this sense languages are much like cars. For many people car is the thing that they use get to work and shopping mall and they are not very interesting is engine inline or V-type and the use of fuzzy logic in the transmission. What they care is safety, reliability, mileage, insurance and the size of trunk. In this sense "Worse is better" is very true. I already mentioned the importance of the debugger. The other important criteria is quality and availability of libraries. Actually libraries are what make 80% of the usability of the language, moreover in a sense libraries are more important than the language...

A popular belief that scripting is "unsafe" or "second rate" or "prototype" solution is completely wrong. If a project had died than it does not matter what was the implementation language, so for any successful project and tough schedules scripting language (especially in dual scripting language+C combination, for example TCL+C) is an optimal blend that for a large class of tasks. Such an approach helps to separate architectural decisions from implementation details much better that any OO model does.

Moreover even for tasks that handle a fair amount of computations and data (computationally intensive tasks) such languages as Python and Perl are often (but not always !) competitive with C++, C# and, especially, Java.

The second important observation about programming languages is that language per se is just a tiny part of what can be called language programming environment. the latter includes libraries, IDE, books, level of adoption at universities, popular, important applications written in the language, level of support and key players that support the language on major platforms such as Windows and Linux and other similar things. A mediocre language with good programming environment can give a run for the money to similar superior in design languages that are just naked. This is a story behind success of Java. Critical application is also very important and this is a story of success of PHP which is nothing but a bastardatized derivative of Perl (with all most interesting Perl features removed ;-) adapted to creation of dynamic web sites using so called LAMP stack.

History of programming languages raises interesting general questions about the limit of complexity of programming languages. There is strong historical evidence that languages with simpler core, or even simplistic core has more chanced to acquire high level of popularity. The underlying fact here probably is that most programmers are at best mediocre and such programmer tend on intuitive level to avoid more complex, more rich languages like, say, PL/1 and Perl. Or at least avoid it on a particular phase of language development (C++ is not simpler language then PL/1, but was widely adopted because OO became a fashion). Complex non-orthogonal languages can succeed only as a result on long period of language development from a smaller core or with the banner of some fashionable new trend (OO programming in case of C++).

Programming Language Development Timeline

Here is modified from Byte the timeline of Programming Languages (for the original see BYTE.com September 1995 / 20th Anniversary /)

Forties

ca. 1946


1949

Fifties


1951


1952

1957


1958


1959

Sixties


1960


1962


1963


1964


1965


1966


1967



1969

Seventies


1970


1972


1974


1975


1976


1977


1978


1979

Eighties


1980


1981


1982


1983


1984


1985


1986


1987


1988


1989

Nineties


1990


1991


1992


1993


1994


1995


1996


1997


2000


2006

2007 

2008:

2009:

2011

2017:

Special note on Scripting languages

Scripting helps to avoid OO trap that is pushed by
  "a hoard of practically illiterate researchers
publishing crap papers in junk conferences."

Despite the fact that scripting languages are really important computer science phenomena, they are usually happily ignored in university curriculums.  Students are usually indoctrinated (or in less politically correct terms  "brainwashed")  in Java and OO programming ;-)

This site tries to give scripting languages proper emphasis and  promotes scripting languages as an alternative to mainstream reliance on "Java as a new Cobol" approach for software development. Please read my introduction to the topic that was recently converted into the article: A Slightly Skeptical View on Scripting Languages.  

The tragedy of scripting language designer is that there is no way to overestimate the level of abuse of any feature of the language.  Half of the programmers by definition is below average and it is this half that matters most in enterprise environment.  In a way the higher is the level of programmer, the less relevant for him are limitations of the language. That's why statements like "Perl is badly suitable for large project development" are plain vanilla silly. With proper discipline it is perfectly suitable and programmers can be more productive with Perl than with Java. The real question is "What is the team quality and quantity?".  

Scripting is a part of Unix cultural tradition and Unix was the initial development platform for most of mainstream scripting languages with the exception of REXX. But they are portable and now all can be used in Windows and other OSes.

List of Softpanorama pages related to scripting languages

Standard topics

Main Representatives of the family Related topics History Etc

Different scripting languages provide different level of integration with base OS API (for example, Unix or Windows). For example Iron Python compiles into .Net and provides pretty high level of integration with Windows. The same is true about Perl and Unix: almost all Unix system calls are available directly from Perl. Moreover Perl integrates most of Unix API in a very natural way, making it perfect replacement of shell for coding complex scripts. It also have very good debugger. The latter is weak point of shells like bash and ksh93

Unix proved that treating everything like a file is a powerful OS paradigm. In a similar way scripting languages proved that "everything is a string" is also an extremely powerful programming paradigm.

Unix proved that treating everything like a file is a powerful OS paradigm. In a similar way scripting languages proved that "everything is a string" is also extremely powerful programming paradigm.

There are also several separate pages devoted to scripting in different applications. The main emphasis is on shells and Perl. Right now I am trying to convert my old Perl lecture notes into a eBook Nikolai Bezroukov. Introduction to Perl for Unix System Administrators.

Along with pages devoted to major scripting languages this site has many pages devoted to scripting in different applications.  There are more then a dozen of "Perl/Scripting tools for a particular area" type of pages. The most well developed and up-to-date pages of this set are probably Shells and Perl. This page main purpose is to follow the changes in programming practices that can be called the "rise of  scripting," as predicted in the famous John Ousterhout article Scripting: Higher Level Programming for the 21st Century in IEEE COMPUTER (1998). In this brilliant paper he wrote:

...Scripting languages such as Perl and Tcl represent a very different style of programming than system programming languages such as C or Java. Scripting languages are designed for "gluing" applications; they use typeless approaches to achieve a higher level of programming and more rapid application development than system programming languages. Increases in computer speed and changes in the application mix are making scripting languages more and more important for applications of the future.

...Scripting languages and system programming languages are complementary, and most major computing platforms since the 1960's have provided both kinds of languages. The languages are typically used together in component frameworks, where components are created with system programming languages and glued together with scripting languages. However, several recent trends, such as faster machines, better scripting languages, the increasing importance of graphical user interfaces and component architectures, and the growth of the Internet, have greatly increased the applicability of scripting languages. These trends will continue over the next decade, with more and more new applications written entirely in scripting languages and system programming languages used primarily for creating components.

My e-book Portraits of Open Source Pioneers contains several chapters on scripting (most are in early draft stage) that expand on this topic. 

The reader must understand that the treatment of the scripting languages in press, and especially academic press is far from being fair: entrenched academic interests often promote old or commercially supported paradigms until they retire, so change of paradigm often is possible only with the change of generations. And people tend to live longer those days... Please also be aware that even respectable academic magazines like Communications of ACM and IEEE Software often promote "Cargo cult software engineering" like Capability Maturity (CMM) model.

Dr. Nikolai Bezroukov


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

2007 2006 2005 2004 2003 2002 2001 2000 1999

[May 24, 2019] How to send keystrokes from one computer to another by USB?

Notable quotes:
"... On a different note, have you considered a purely software/network solution such as TightVNC ? ..."
Aug 05, 2018 | stackoverflow.com

Ask Question


Yehonatan ,Aug 5, 2018 at 6:34

Is there a way to use one computer to send keystrokes to another by usb ?

What i'm looking to do is to capture the usb signal used by a keyboard (with USBTrace for example) and use it with PC-1 to send it to PC-2. So that PC-2 recognize it as a regular keyboard input.

Some leads to do this would be very appreciated.

Lucas ,Jan 16, 2011 at 19:18

What you essentially need is a USB port on PC-1 that will act as a USB device for PC-2.

That is not possible for the vast majority of PC systems because USB is an asymmetric bus, with a host/device (or master/slave, if you wish) architecture. USB controllers (and their ports) on most PCs can only work in host mode and cannot simulate a device.

That is the reason that you cannot network computers through USB without a special cable with specialized electronics.

The only exception is if you somehow have a PC that supports the USB On-The-Go standard that allows for a USB port to act in both host and device mode. USB-OTG devices do exist, but they are usually embedded devices (smartphones etc). I don't know if there is a way to add a USB-OTG port to a commodity PC.

EDIT:

If you do not need a keyboard before the OS on PC-2 boots, you might be able to use a pair of USB Bluetooth dongles - one on each PC. You'd have to use specialised software on PC-1, but it is definitely possible - I've already seen a possible implementation on Linux , and I am reasonably certain that there must be one for Windows. You will also need Bluetooth HID drivers on PC-2, if they are not already installed.

On a different note, have you considered a purely software/network solution such as TightVNC ?

bebbo ,Sep 20, 2017 at 18:14

There is a solution: https://github.com/Flowm/etherkey

This uses a network connection from your computer to the raspi which is connected to a teensy (usb developer board) to send the key strokes.

This solution is not an out-of-the-box product. The required skill is similar to programming some other devices like arduino. But it's a complete and working setup.

Yehonatan ,Jan 25, 2011 at 5:51

The cheapest options are commercial microcontrollers (eg arduino platform, pic, etc) or ready built usb keyboard controllers (eg i-pac, arcade controllers,etc)

Benoit-Pierre DEMAINE ,Oct 27, 2017 at 17:17

SEARCH THIS PROGRAM:

TWedge: Keyboard Wedge Software (RS232, Serial, TCP, Bluetooth)

then, MAKE YOUR OWN CONNECTION CABLE WITH:

(usb <-> rs232) + (NULL MODEM) + (rs232 <-> usb)

Connect 2 computer, write your own program to send signal to your (usb <-> rs232) unit, then you can control another computer under the help of TWedge.

> ,

The above mentionned https://github.com/Flowm/etherkey is one way. The keyboard is emulated from an rPi, but the principle can be used from PC to PC (or Mac to Whatever). The core answer to your question is to use an OTG-capable chip, and then you control this chip via a USB-serial adapter.

https://euer.krebsco.de/a-software-kvm-switch.html uses a very similar method, using an Arduino instead of the Teensy.

The generic answer is: you need an OTG capable, or slave capable device: Arduino, Teensy, Pi 0 (either from Rapberry or Orange brands, both work; only the ZERO models are OTG capable), or, an rPi-A with heavy customisation (since it does not include USB hub, it can theoretically be converted into a slave; never found any public tutorial to do it), or any smartphone (Samsung, Nokia, HTC, Oukitel ... most smartphones are OTG capable). If you go for a Pi or a phone, then, you want to dig around USB Gadget. Cheaper solutions (Arduino/Teensy) need custom firmware.

[May 03, 2019] RedHat local repository and offline updates

Aug 03, 2018 | stackoverflow.com

My company just bought a two redhat license for two physical machines , the machines wont be accessible via internet , so we have an issue here regarding the updates , patches , ... etc .

i am thinking of configuring a local repository to be accessible via internet and have all the necessary updates but there is a problem here that i have only two licenses , is it applicable if i activate the local repository for the updates and one of my two service machines , or is there any other way like if there is some sort of offline package that i can download it separately from redhat and update my machines without internet access ?

thanks in advance

XXX

You have several options:

See How can we regularly update a disconnected system (A system without internet connection)? for details.

[Mar 20, 2019] How do I troubleshoot a yum repository problem that has an error No package available error?

Mar 20, 2019 | unix.stackexchange.com

Kiran ,Jan 2, 2017 at 23:57

I have three RHEL 6.6 servers. One has a yum repository that I know works. The other two servers I will refer to as "yum clients." These two are configured to use the same yum repository (the first server described). When I do yum install httpd on each of these two yum client servers, I get two different results. One server prepares for the installation as normal and prompts me with a y/n prompt. The second server says

No package httpd available.

The /etc/yum.conf files on each of the two servers is identical. The /etc/yum.repos.d/ directories have the same .repo files. Why does one yum client not see the httpd package? I use httpd as an example. One yum client cannot install any package. The other yum client can install anything. Neither have access to the Internet or different servers the other one does not have access to.

XXX

If /etc/yum.conf is identical on all servers, and that package is not listed there in exclude line, check if the repo is enabled on all the servers.

Do

grep enabled /etc/yum.repos.d/filename.repo

and see if it is set to 0 or 1.

value of enabled needs to be set to 1, for yum to use that repo.

If repo is not enabled, you can edit the repo file, and change the enable to 1, or try to run yum with enablerepo switch, to enable it for that operation.

Try to run yum like this.

yum --enablerepo=repo_name install package_name

[Mar 20, 2019] How to I print to STDERR only if STDOUT is a different destination?

Mar 14, 2013 | stackoverflow.com

squiguy, Mar 14, 2013 at 19:06

I would like Perl to write to STDERR only if STDOUT is not the same. For example, if both STDOUT and STDERR would redirect output to the Terminal, then I don't want STDERR to be printed.

Consider the following example (outerr.pl):

#!/usr/bin/perl

use strict;
use warnings;

print STDOUT "Hello standard output!\n";
print STDERR "Hello standard error\n" if ($someMagicalFlag);
exit 0

Now consider this (this is what I would like to achieve):

bash $ outerr.pl
Hello standard output!

However, if I redirect out to a file, I'd like to get:

bash $ outerr.pl > /dev/null
Hello standard error

and similary the other way round:

bash $ outerr.pl 2> /dev/null
Hello standard output!

If I re-direct both out/err to the same file, then only stdout should be displayed:

bash $ outerr.pl > foo.txt 2>&1
bash $ cat foo.txt
Hello standard output!

So is there a way to evaluate / determine whether OUT and ERR and are pointing to the same "thing" (descriptor?)?

tchrist ,Mar 15, 2013 at 5:07

On Unix-style systems, you should be able to do:
my @stat_err = stat STDERR;
my @stat_out = stat STDOUT;

my $stderr_is_not_stdout = (($stat_err[0] != $stat_out[0]) ||
                            ($stat_err[1] != $stat_out[1]));

But that won't work on Windows, which doesn't have real inode numbers. It gives both false positives (thinks they're different when they aren't) and false negatives (thinks they're the same when they aren't).

Jim Stewart ,Mar 14, 2013 at 20:59

You can do that (almost) with -t:
-t STDERR

will be true if it is a terminal, and likewise for STDOUT.

This still would not tell you what terminal, and if you redirect to the same file, you may stilll get both.

Hence, if

-t STDERR && ! (-t STDOUT) || -t STDOUT && !(-t STDERR)

or shorter

-t STDOUT ^ -t STDERR  # thanks to @mob

you know you're okay.

EDIT: Solutions for the case that both STDERR and STDOUT are regular files:

Tom Christianson suggested to stat and compare the dev and ino fields. This will work in UNIX, but, as @cjm pointed out, not in Windows.

If you can guarantee that no other program will write to the file, you could do the following both in Windows and UNIX:

  1. check the position the file descriptors for STDOUT and STDERR are at, if they are not equal, you redirected one of them with >> to a nonempty file.
  2. Otherwise, write 42 bytes to file descriptor 2
  3. Seek to the end of file descriptor 1. If it is 42 more than before, chances are high that both are redirected to the same file. If it is unchanged, files are different. If it is changed, but not by 42, someone else is writing there, all bets are off (but then, you're not in Windows, so the stat method will work).

[Mar 17, 2019] Translating Perl to Python

Mar 17, 2019 | stackoverflow.com

21


John Kugelman ,Jul 1, 2009 at 3:29

I found this Perl script while migrating my SQLite database to mysql

I was wondering (since I don't know Perl) how could one rewrite this in Python?

Bonus points for the shortest (code) answer :)

edit : sorry I meant shortest code, not strictly shortest answer

#! /usr/bin/perl

while ($line = <>){
    if (($line !~  /BEGIN TRANSACTION/) && ($line !~ /COMMIT/) && ($line !~ /sqlite_sequence/) && ($line !~ /CREATE UNIQUE INDEX/)){

        if ($line =~ /CREATE TABLE \"([a-z_]*)\"(.*)/){
                $name = $1;
                $sub = $2;
                $sub =~ s/\"//g; #"
                $line = "DROP TABLE IF EXISTS $name;\nCREATE TABLE IF NOT EXISTS $name$sub\n";
        }
        elsif ($line =~ /INSERT INTO \"([a-z_]*)\"(.*)/){
                $line = "INSERT INTO $1$2\n";
                $line =~ s/\"/\\\"/g; #"
                $line =~ s/\"/\'/g; #"
        }else{
                $line =~ s/\'\'/\\\'/g; #'
        }
        $line =~ s/([^\\'])\'t\'(.)/$1THIS_IS_TRUE$2/g; #'
        $line =~ s/THIS_IS_TRUE/1/g;
        $line =~ s/([^\\'])\'f\'(.)/$1THIS_IS_FALSE$2/g; #'
        $line =~ s/THIS_IS_FALSE/0/g;
        $line =~ s/AUTOINCREMENT/AUTO_INCREMENT/g;
        print $line;
    }
}

Some additional code was necessary to successfully migrate the sqlite database (handles one line Create table statements, foreign keys, fixes a bug in the original program that converted empty fields '' to \' .

I posted the code on the migrating my SQLite database to mysql Question

Jiaaro ,Jul 2, 2009 at 10:15

Here's a pretty literal translation with just the minimum of obvious style changes (putting all code into a function, using string rather than re operations where possible).
import re, fileinput

def main():
  for line in fileinput.input():
    process = False
    for nope in ('BEGIN TRANSACTION','COMMIT',
                 'sqlite_sequence','CREATE UNIQUE INDEX'):
      if nope in line: break
    else:
      process = True
    if not process: continue
    m = re.search('CREATE TABLE "([a-z_]*)"(.*)', line)
    if m:
      name, sub = m.groups()
      line = '''DROP TABLE IF EXISTS %(name)s;
CREATE TABLE IF NOT EXISTS %(name)s%(sub)s
'''
      line = line % dict(name=name, sub=sub)
    else:
      m = re.search('INSERT INTO "([a-z_]*)"(.*)', line)
      if m:
        line = 'INSERT INTO %s%s\n' % m.groups()
        line = line.replace('"', r'\"')
        line = line.replace('"', "'")
    line = re.sub(r"([^'])'t'(.)", r"\1THIS_IS_TRUE\2", line)
    line = line.replace('THIS_IS_TRUE', '1')
    line = re.sub(r"([^'])'f'(.)", r"\1THIS_IS_FALSE\2", line)
    line = line.replace('THIS_IS_FALSE', '0')
    line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')
    print line,

main()

dr jimbob ,May 20, 2018 at 0:54

Alex Martelli's solution above works good, but needs some fixes and additions:

In the lines using regular expression substitution, the insertion of the matched groups must be double-escaped OR the replacement string must be prefixed with r to mark is as regular expression:

line = re.sub(r"([^'])'t'(.)", "\\1THIS_IS_TRUE\\2", line)

or

line = re.sub(r"([^'])'f'(.)", r"\1THIS_IS_FALSE\2", line)

Also, this line should be added before print:

line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')

Last, the column names in create statements should be backticks in MySQL. Add this in line 15:

  sub = sub.replace('"','`')

Here's the complete script with modifications:

import re, fileinput

def main():
  for line in fileinput.input():
    process = False
    for nope in ('BEGIN TRANSACTION','COMMIT',
                 'sqlite_sequence','CREATE UNIQUE INDEX'):
      if nope in line: break
    else:
      process = True
    if not process: continue
    m = re.search('CREATE TABLE "([a-z_]*)"(.*)', line)
    if m:
      name, sub = m.groups()
      sub = sub.replace('"','`')
      line = '''DROP TABLE IF EXISTS %(name)s;
CREATE TABLE IF NOT EXISTS %(name)s%(sub)s
'''
      line = line % dict(name=name, sub=sub)
    else:
      m = re.search('INSERT INTO "([a-z_]*)"(.*)', line)
      if m:
        line = 'INSERT INTO %s%s\n' % m.groups()
        line = line.replace('"', r'\"')
        line = line.replace('"', "'")
    line = re.sub(r"([^'])'t'(.)", "\\1THIS_IS_TRUE\\2", line)
    line = line.replace('THIS_IS_TRUE', '1')
    line = re.sub(r"([^'])'f'(.)", "\\1THIS_IS_FALSE\\2", line)
    line = line.replace('THIS_IS_FALSE', '0')
    line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')
    if re.search('^CREATE INDEX', line):
        line = line.replace('"','`')
    print line,

main()

Brad Gilbert ,Jul 1, 2009 at 18:43

Here is a slightly better version of the original.
#! /usr/bin/perl
use strict;
use warnings;
use 5.010; # for s/\K//;

while( <> ){
  next if m'
    BEGIN TRANSACTION   |
    COMMIT              |
    sqlite_sequence     |
    CREATE UNIQUE INDEX
  'x;

  if( my($name,$sub) = m'CREATE TABLE \"([a-z_]*)\"(.*)' ){
    # remove "
    $sub =~ s/\"//g; #"
    $_ = "DROP TABLE IF EXISTS $name;\nCREATE TABLE IF NOT EXISTS $name$sub\n";

  }elsif( /INSERT INTO \"([a-z_]*)\"(.*)/ ){
    $_ = "INSERT INTO $1$2\n";

    # " => \"
    s/\"/\\\"/g; #"
    # " => '
    s/\"/\'/g; #"

  }else{
    # '' => \'
    s/\'\'/\\\'/g; #'
  }

  # 't' => 1
  s/[^\\']\K\'t\'/1/g; #'

  # 'f' => 0
  s/[^\\']\K\'f\'/0/g; #'

  s/AUTOINCREMENT/AUTO_INCREMENT/g;
  print;
}

Mickey Mouse ,Jun 14, 2011 at 15:48

all of scripts on this page can't deal with simple sqlite3:
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE Filename (
  FilenameId INTEGER,
  Name TEXT DEFAULT '',
  PRIMARY KEY(FilenameId) 
  );
INSERT INTO "Filename" VALUES(1,'');
INSERT INTO "Filename" VALUES(2,'bigfile1');
INSERT INTO "Filename" VALUES(3,'%gconf-tree.xml');

None were able to reformat "table_name" into proper mysql's `table_name` . Some messed up empty string value.

Sinan Ünür ,Jul 1, 2009 at 3:24

I am not sure what is so hard to understand about this that it requires a snide remark as in your comment above. Note that <> is called the diamond operator. s/// is the substitution operator and // is the match operator m// .

Ken_g6 ,Jul 1, 2009 at 3:22

Based on http://docs.python.org/dev/howto/regex.html ...
  1. Replace $line =~ /.*/ with re.search(r".*", line) .
  2. $line !~ /.*/ is just !($line =~ /.*/) .
  3. Replace $line =~ s/.*/x/g with line=re.sub(r".*", "x", line) .
  4. Replace $1 through $9 inside re.sub with \1 through \9 respectively.
  5. Outside a sub, save the return value, i.e. m=re.search() , and replace $1 with the return value of m.group(1) .
  6. For "INSERT INTO $1$2\n" specifically, you can do "INSERT INTO %s%s\n" % (m.group(1), m.group(2)) .

hpavc ,Jul 1, 2009 at 12:33

Real issue is do you know actually how to migrate the database? What is presented is merely a search and replace loop.

> ,

Shortest? The tilde signifies a regex in perl. "import re" and go from there. The only key differences are that you'll be using \1 and \2 instead of $1 and $2 when you assign values, and you'll be using %s for when you're replacing regexp matches inside strings.

[Mar 16, 2019] Translating Perl to Python - Stack Overflow

Mar 16, 2019 | stackoverflow.com

Translating Perl to Python Ask Question 21


John Kugelman ,Jul 1, 2009 at 3:29

I found this Perl script while migrating my SQLite database to mysql

I was wondering (since I don't know Perl) how could one rewrite this in Python?

Bonus points for the shortest (code) answer :)

edit : sorry I meant shortest code, not strictly shortest answer

#! /usr/bin/perl

while ($line = <>){
    if (($line !~  /BEGIN TRANSACTION/) && ($line !~ /COMMIT/) && ($line !~ /sqlite_sequence/) && ($line !~ /CREATE UNIQUE INDEX/)){

        if ($line =~ /CREATE TABLE \"([a-z_]*)\"(.*)/){
                $name = $1;
                $sub = $2;
                $sub =~ s/\"//g; #"
                $line = "DROP TABLE IF EXISTS $name;\nCREATE TABLE IF NOT EXISTS $name$sub\n";
        }
        elsif ($line =~ /INSERT INTO \"([a-z_]*)\"(.*)/){
                $line = "INSERT INTO $1$2\n";
                $line =~ s/\"/\\\"/g; #"
                $line =~ s/\"/\'/g; #"
        }else{
                $line =~ s/\'\'/\\\'/g; #'
        }
        $line =~ s/([^\\'])\'t\'(.)/$1THIS_IS_TRUE$2/g; #'
        $line =~ s/THIS_IS_TRUE/1/g;
        $line =~ s/([^\\'])\'f\'(.)/$1THIS_IS_FALSE$2/g; #'
        $line =~ s/THIS_IS_FALSE/0/g;
        $line =~ s/AUTOINCREMENT/AUTO_INCREMENT/g;
        print $line;
    }
}

Some additional code was necessary to successfully migrate the sqlite database (handles one line Create table statements, foreign keys, fixes a bug in the original program that converted empty fields '' to \' .

I posted the code on the migrating my SQLite database to mysql Question

Jiaaro ,Jul 2, 2009 at 10:15

Here's a pretty literal translation with just the minimum of obvious style changes (putting all code into a function, using string rather than re operations where possible).
import re, fileinput

def main():
  for line in fileinput.input():
    process = False
    for nope in ('BEGIN TRANSACTION','COMMIT',
                 'sqlite_sequence','CREATE UNIQUE INDEX'):
      if nope in line: break
    else:
      process = True
    if not process: continue
    m = re.search('CREATE TABLE "([a-z_]*)"(.*)', line)
    if m:
      name, sub = m.groups()
      line = '''DROP TABLE IF EXISTS %(name)s;
CREATE TABLE IF NOT EXISTS %(name)s%(sub)s
'''
      line = line % dict(name=name, sub=sub)
    else:
      m = re.search('INSERT INTO "([a-z_]*)"(.*)', line)
      if m:
        line = 'INSERT INTO %s%s\n' % m.groups()
        line = line.replace('"', r'\"')
        line = line.replace('"', "'")
    line = re.sub(r"([^'])'t'(.)", r"\1THIS_IS_TRUE\2", line)
    line = line.replace('THIS_IS_TRUE', '1')
    line = re.sub(r"([^'])'f'(.)", r"\1THIS_IS_FALSE\2", line)
    line = line.replace('THIS_IS_FALSE', '0')
    line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')
    print line,

main()

dr jimbob ,May 20, 2018 at 0:54

Alex Martelli's solution above works good, but needs some fixes and additions:

In the lines using regular expression substitution, the insertion of the matched groups must be double-escaped OR the replacement string must be prefixed with r to mark is as regular expression:

line = re.sub(r"([^'])'t'(.)", "\\1THIS_IS_TRUE\\2", line)

or

line = re.sub(r"([^'])'f'(.)", r"\1THIS_IS_FALSE\2", line)

Also, this line should be added before print:

line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')

Last, the column names in create statements should be backticks in MySQL. Add this in line 15:

  sub = sub.replace('"','`')

Here's the complete script with modifications:

import re, fileinput

def main():
  for line in fileinput.input():
    process = False
    for nope in ('BEGIN TRANSACTION','COMMIT',
                 'sqlite_sequence','CREATE UNIQUE INDEX'):
      if nope in line: break
    else:
      process = True
    if not process: continue
    m = re.search('CREATE TABLE "([a-z_]*)"(.*)', line)
    if m:
      name, sub = m.groups()
      sub = sub.replace('"','`')
      line = '''DROP TABLE IF EXISTS %(name)s;
CREATE TABLE IF NOT EXISTS %(name)s%(sub)s
'''
      line = line % dict(name=name, sub=sub)
    else:
      m = re.search('INSERT INTO "([a-z_]*)"(.*)', line)
      if m:
        line = 'INSERT INTO %s%s\n' % m.groups()
        line = line.replace('"', r'\"')
        line = line.replace('"', "'")
    line = re.sub(r"([^'])'t'(.)", "\\1THIS_IS_TRUE\\2", line)
    line = line.replace('THIS_IS_TRUE', '1')
    line = re.sub(r"([^'])'f'(.)", "\\1THIS_IS_FALSE\\2", line)
    line = line.replace('THIS_IS_FALSE', '0')
    line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')
    if re.search('^CREATE INDEX', line):
        line = line.replace('"','`')
    print line,

main()

Brad Gilbert ,Jul 1, 2009 at 18:43

Here is a slightly better version of the original.
#! /usr/bin/perl
use strict;
use warnings;
use 5.010; # for s/\K//;

while( <> ){
  next if m'
    BEGIN TRANSACTION   |
    COMMIT              |
    sqlite_sequence     |
    CREATE UNIQUE INDEX
  'x;

  if( my($name,$sub) = m'CREATE TABLE \"([a-z_]*)\"(.*)' ){
    # remove "
    $sub =~ s/\"//g; #"
    $_ = "DROP TABLE IF EXISTS $name;\nCREATE TABLE IF NOT EXISTS $name$sub\n";

  }elsif( /INSERT INTO \"([a-z_]*)\"(.*)/ ){
    $_ = "INSERT INTO $1$2\n";

    # " => \"
    s/\"/\\\"/g; #"
    # " => '
    s/\"/\'/g; #"

  }else{
    # '' => \'
    s/\'\'/\\\'/g; #'
  }

  # 't' => 1
  s/[^\\']\K\'t\'/1/g; #'

  # 'f' => 0
  s/[^\\']\K\'f\'/0/g; #'

  s/AUTOINCREMENT/AUTO_INCREMENT/g;
  print;
}

Mickey Mouse ,Jun 14, 2011 at 15:48

all of scripts on this page can't deal with simple sqlite3:
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE Filename (
  FilenameId INTEGER,
  Name TEXT DEFAULT '',
  PRIMARY KEY(FilenameId) 
  );
INSERT INTO "Filename" VALUES(1,'');
INSERT INTO "Filename" VALUES(2,'bigfile1');
INSERT INTO "Filename" VALUES(3,'%gconf-tree.xml');

None were able to reformat "table_name" into proper mysql's `table_name` . Some messed up empty string value.

Sinan Ünür ,Jul 1, 2009 at 3:24

I am not sure what is so hard to understand about this that it requires a snide remark as in your comment above. Note that <> is called the diamond operator. s/// is the substitution operator and // is the match operator m// .

Ken_g6 ,Jul 1, 2009 at 3:22

Based on http://docs.python.org/dev/howto/regex.html ...
  1. Replace $line =~ /.*/ with re.search(r".*", line) .
  2. $line !~ /.*/ is just !($line =~ /.*/) .
  3. Replace $line =~ s/.*/x/g with line=re.sub(r".*", "x", line) .
  4. Replace $1 through $9 inside re.sub with \1 through \9 respectively.
  5. Outside a sub, save the return value, i.e. m=re.search() , and replace $1 with the return value of m.group(1) .
  6. For "INSERT INTO $1$2\n" specifically, you can do "INSERT INTO %s%s\n" % (m.group(1), m.group(2)) .

hpavc ,Jul 1, 2009 at 12:33

Real issue is do you know actually how to migrate the database? What is presented is merely a search and replace loop.

> ,

Shortest? The tilde signifies a regex in perl. "import re" and go from there. The only key differences are that you'll be using \1 and \2 instead of $1 and $2 when you assign values, and you'll be using %s for when you're replacing regexp matches inside strings.

[Mar 16, 2019] Regex translation from Perl to Python - Stack Overflow

Mar 16, 2019 | stackoverflow.com

Regex translation from Perl to Python Ask Question 1


royskatt ,Jan 30, 2014 at 14:45

I would like to rewrite a small Perl programm to Python. I am processing text files with it as follows:

Input:

00000001;Root;;
00000002;  Documents;;
00000003;    oracle-advanced_plsql.zip;file;
00000004;  Public;;
00000005;  backup;;
00000006;    20110323-JM-F.7z.001;file;
00000007;    20110426-JM-F.7z.001;file;
00000008;    20110603-JM-F.7z.001;file;
00000009;    20110701-JM-F-via-summer_school;;
00000010;      20110701-JM-F-yyy.7z.001;file;

Desired output:

00000001;;Root;;
00000002;  ;Documents;;
00000003;    ;oracle-advanced_plsql.zip;file;
00000004;  ;Public;;
00000005;  ;backup;;
00000006;    ;20110323-JM-F.7z.001;file;
00000007;    ;20110426-JM-F.7z.001;file;
00000008;    ;20110603-JM-F.7z.001;file;
00000009;    ;20110701-JM-F-via-summer_school;;
00000010;      ;20110701-JM-F-yyy.7z.001;file;

Here is the working Perl code:

#filename: perl_regex.pl
#/usr/bin/perl -w
while(<>) {                                                           
  s/^(.*?;.*?)(\w)/$1;$2/;                                            
  print $_;                                                           
}

It call it from the command line: perl_regex.pl input.txt

Explanation of the Perl-style regex:

s/        # start search-and-replace regexp
  ^       # start at the beginning of this line
  (       # save the matched characters until ')' in $1
    .*?;  # go forward until finding the first semicolon
    .*?   # go forward until finding... (to be continued below)
  )
  (       # save the matched characters until ')' in $2
    \w    # ... the next alphanumeric character.
  )
/         # continue with the replace part
  $1;$2   # write all characters found above, but insert a ; before $2
/         # finish the search-and-replace regexp.

Could anyone tell me, how to get the same result in Python? Especially for the $1 and $2 variables I couldn't find something alike.

royskatt ,Jan 31, 2014 at 6:18

Python regular expression is very similar to Perl's, except:

Use re.sub to replace.

import re
import sys

for line in sys.stdin: # Explicitly iterate standard input line by line
    # `line` contains trailing newline!
    line = re.sub(r'^(.*?;.*?)(\w)', r'\1;\2', line)
    #print(line) # This print trailing newline
    sys.stdout.write(line) # Print the replaced string back.

royskatt ,Jan 31, 2014 at 16:36

The replace instruction for s/pattern/replace/ in python regexes is the re.sub(pattern, replace, string) function, or re.compile(pattern).sub(replace, string). In your case, you will do it so:
_re_pattern = re.compile(r"^(.*?;.*?)(\w)")
result = _re_pattern.sub(r"\1;\2", line)

Note that $1 becomes \1 . As for perl, you need to iterate over your lines the way you want to do it (open, inputfile, splitlines, ...).

[Mar 13, 2019] amp html - Convert img to amp-img - Stack Overflow

Mar 13, 2019 | stackoverflow.com

> ,

Which is the default way to convert an <img> to a <amp-img> ?

I explain myself: In the site that I'm converting to AMP I have lot of images without widht and height e.g.:

<img src="/img/image.png" alt="My image">

If I not specify the layout, the layout="container" is set by default and the most of the images throw the following error:

amp-img error: Layout not supported for: container

In the other hand, the most of the images don't fit with the responsive layout, which is recommended by Google for most of the cases

I have been checking the types of layout on the documentation:

But any of them seems to fit with an image that have to be shown as its real size, not specifying width or height.

So, in that case, which is the equivalent in AMP?

,

As you are saying you have multiple images, it's better you use the ' layout="responsive" ', with that, you will make your images responsive atleast.

Now regarding the Width and Height . They are must.

If you read the purpose of AMP, one of them is to make the pages ' Jumping/Flickering Free Content ', which happens if there is no width mentioned for Images.

By Specifying the Width, the Browser (mobile browser), can calculate the precise space and keep it for that Image and show the Content after that. In that way, there wont' be any flickering of the content, as the page and images are loaded.

Regarding the re-writing of your HTML, one tip I can provide is, you can write some small utility with PHP, Python or Node JavaScript, which can actually read the source image, calculate their dimensions and replace your IMG tags.

Hope this helps and wish you good luck for your AMP powered site :-)

[Mar 10, 2019] How do I detach a process from Terminal, entirely?

Mar 10, 2019 | superuser.com

stackoverflow.com, Aug 25, 2016 at 17:24

I use Tilda (drop-down terminal) on Ubuntu as my "command central" - pretty much the way others might use GNOME Do, Quicksilver or Launchy.

However, I'm struggling with how to completely detach a process (e.g. Firefox) from the terminal it's been launched from - i.e. prevent that such a (non-)child process

For example, in order to start Vim in a "proper" terminal window, I have tried a simple script like the following:

exec gnome-terminal -e "vim $@" &> /dev/null &

However, that still causes pollution (also, passing a file name doesn't seem to work).

lhunath, Sep 23, 2016 at 19:08

First of all; once you've started a process, you can background it by first stopping it (hit Ctrl - Z ) and then typing bg to let it resume in the background. It's now a "job", and its stdout / stderr / stdin are still connected to your terminal.

You can start a process as backgrounded immediately by appending a "&" to the end of it:

firefox &

To run it in the background silenced, use this:

firefox </dev/null &>/dev/null &

Some additional info:

nohup is a program you can use to run your application with such that its stdout/stderr can be sent to a file instead and such that closing the parent script won't SIGHUP the child. However, you need to have had the foresight to have used it before you started the application. Because of the way nohup works, you can't just apply it to a running process .

disown is a bash builtin that removes a shell job from the shell's job list. What this basically means is that you can't use fg , bg on it anymore, but more importantly, when you close your shell it won't hang or send a SIGHUP to that child anymore. Unlike nohup , disown is used after the process has been launched and backgrounded.

What you can't do, is change the stdout/stderr/stdin of a process after having launched it. At least not from the shell. If you launch your process and tell it that its stdout is your terminal (which is what you do by default), then that process is configured to output to your terminal. Your shell has no business with the processes' FD setup, that's purely something the process itself manages. The process itself can decide whether to close its stdout/stderr/stdin or not, but you can't use your shell to force it to do so.

To manage a background process' output, you have plenty of options from scripts, "nohup" probably being the first to come to mind. But for interactive processes you start but forgot to silence ( firefox < /dev/null &>/dev/null & ) you can't do much, really.

I recommend you get GNU screen . With screen you can just close your running shell when the process' output becomes a bother and open a new one ( ^Ac ).


Oh, and by the way, don't use " $@ " where you're using it.

$@ means, $1 , $2 , $3 ..., which would turn your command into:

gnome-terminal -e "vim $1" "$2" "$3" ...

That's probably not what you want because -e only takes one argument. Use $1 to show that your script can only handle one argument.

It's really difficult to get multiple arguments working properly in the scenario that you gave (with the gnome-terminal -e ) because -e takes only one argument, which is a shell command string. You'd have to encode your arguments into one. The best and most robust, but rather cludgy, way is like so:

gnome-terminal -e "vim $(printf "%q " "$@")"

Limited Atonement ,Aug 25, 2016 at 17:22

nohup cmd &

nohup detaches the process completely (daemonizes it)

Randy Proctor ,Sep 13, 2016 at 23:00

If you are using bash , try disown [ jobspec ] ; see bash(1) .

Another approach you can try is at now . If you're not superuser, your permission to use at may be restricted.

Stephen Rosen ,Jan 22, 2014 at 17:08

Reading these answers, I was under the initial impression that issuing nohup <command> & would be sufficient. Running zsh in gnome-terminal, I found that nohup <command> & did not prevent my shell from killing child processes on exit. Although nohup is useful, especially with non-interactive shells, it only guarantees this behavior if the child process does not reset its handler for the SIGHUP signal.

In my case, nohup should have prevented hangup signals from reaching the application, but the child application (VMWare Player in this case) was resetting its SIGHUP handler. As a result when the terminal emulator exits, it could still kill your subprocesses. This can only be resolved, to my knowledge, by ensuring that the process is removed from the shell's jobs table. If nohup is overridden with a shell builtin, as is sometimes the case, this may be sufficient, however, in the event that it is not...


disown is a shell builtin in bash , zsh , and ksh93 ,

<command> &
disown

or

<command> &; disown

if you prefer one-liners. This has the generally desirable effect of removing the subprocess from the jobs table. This allows you to exit the terminal emulator without accidentally signaling the child process at all. No matter what the SIGHUP handler looks like, this should not kill your child process.

After the disown, the process is still a child of your terminal emulator (play with pstree if you want to watch this in action), but after the terminal emulator exits, you should see it attached to the init process. In other words, everything is as it should be, and as you presumably want it to be.

What to do if your shell does not support disown ? I'd strongly advocate switching to one that does, but in the absence of that option, you have a few choices.

  1. screen and tmux can solve this problem, but they are much heavier weight solutions, and I dislike having to run them for such a simple task. They are much more suitable for situations in which you want to maintain a tty, typically on a remote machine.
  2. For many users, it may be desirable to see if your shell supports a capability like zsh's setopt nohup . This can be used to specify that SIGHUP should not be sent to the jobs in the jobs table when the shell exits. You can either apply this just before exiting the shell, or add it to shell configuration like ~/.zshrc if you always want it on.
  3. Find a way to edit the jobs table. I couldn't find a way to do this in tcsh or csh , which is somewhat disturbing.
  4. Write a small C program to fork off and exec() . This is a very poor solution, but the source should only consist of a couple dozen lines. You can then pass commands as commandline arguments to the C program, and thus avoid a process specific entry in the jobs table.

Sheljohn ,Jan 10 at 10:20

  1. nohup $COMMAND &
  2. $COMMAND & disown
  3. setsid command

I've been using number 2 for a very long time, but number 3 works just as well. Also, disown has a 'nohup' flag of '-h', can disown all processes with '-a', and can disown all running processes with '-ar'.

Silencing is accomplished by '$COMMAND &>/dev/null'.

Hope this helps!

dunkyp

add a comment ,Mar 25, 2009 at 1:51
I think screen might solve your problem

Nathan Fellman ,Mar 23, 2009 at 14:55

in tcsh (and maybe in other shells as well), you can use parentheses to detach the process.

Compare this:

> jobs # shows nothing
> firefox &
> jobs
[1]  + Running                       firefox

To this:

> jobs # shows nothing
> (firefox &)
> jobs # still shows nothing
>

This removes firefox from the jobs listing, but it is still tied to the terminal; if you logged in to this node via 'ssh', trying to log out will still hang the ssh process.

,

To disassociate tty shell run command through sub-shell for e.g.

(command)&

When exit used terminal closed but process is still alive.

check -

(sleep 100) & exit

Open other terminal

ps aux | grep sleep

Process is still alive.

[Mar 10, 2019] How to run tmux/screen with systemd 230 ?

Aug 02, 2018 | askubuntu.com

MvanGeest ,May 10, 2017 at 20:59

I run 16.04 and systemd now kills tmux when the user disconnects ( summary of the change ).

Is there a way to run tmux or screen (or any similar program) with systemd 230? I read all the heated disussion about pros and cons of the behavious but no solution was suggested.

(I see the behaviour in 229 as well)

WoJ ,Aug 2, 2016 at 20:30

RemainAfterExit=

Takes a boolean value that specifies whether the service shall be considered active even when all its processes exited. Defaults to no.

jpath ,Feb 13 at 12:29

The proper solution is to disable the offending systemd behavior system-wide.

Edit /etc/systemd/logind.conf ( you must sudo , of course) and set

KillUserProcesses=no

You can also put this setting in a separate file, e.g. /etc/systemd/logind.conf.d/99-dont-kill-user-processes.conf .

Then restart systemd-logind.service .

sudo systemctl restart systemd-logind

sarnold ,Dec 9, 2016 at 11:59

Based on @Rinzwind's answer and inspired by a unit description the best I could find is to use TaaS (Tmux as a Service) - a generic detached instance of tmux one reattaches to.
# cat /etc/systemd/system/tmux@.service

[Unit]
Description=tmux default session (detached)
Documentation=man:tmux(1)

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/tmux new-session -d -s %I
ExecStop=/usr/bin/tmux kill-server
KillMode=none

[Install]
WantedBy=multiplexer.target

# systemctl start tmux@instanceone.service
# systemctl start tmux@instancetwo.service
# tmux list-sessions

instanceone: 1 windows (created Sun Jul 24 00:52:15 2016) [193x49]
instancetwo: 1 windows (created Sun Jul 24 00:52:19 2016) [193x49]

# tmux attach-session -t instanceone

(instanceone)#

Robin Hartmann ,Aug 2, 2018 at 20:23

You need to set the Type of the service to forking , as explained here .

Let's assume the service you want to run in screen is called minecraft . Then you would open minecraft.service in a text editor and add or edit the entry Type=forking under the section [Service] .

> ,

According to https://unix.stackexchange.com/a/287282/117599 invoking tmux using
systemd-run --user --scope tmux

should also do the trick.

[Mar 10, 2019] linux - How to attach terminal to detached process

Mar 10, 2019 | unix.stackexchange.com

Ask Question 86


Gilles ,Feb 16, 2012 at 21:39

I have detached a process from my terminal, like this:
$ process &

That terminal is now long closed, but process is still running and I want to send some commands to that process's stdin. Is that possible?

Samuel Edwin Ward ,Dec 22, 2018 at 13:34

Yes, it is. First, create a pipe: mkfifo /tmp/fifo . Use gdb to attach to the process: gdb -p PID

Then close stdin: call close (0) ; and open it again: call open ("/tmp/fifo", 0600)

Finally, write away (from a different terminal, as gdb will probably hang):

echo blah > /tmp/fifo

NiKiZe ,Jan 6, 2017 at 22:52

When original terminal is no longer accessible...

reptyr might be what you want, see https://serverfault.com/a/284795/187998

Quote from there:

Have a look at reptyr , which does exactly that. The github page has all the information.
reptyr - A tool for "re-ptying" programs.

reptyr is a utility for taking an existing running program and attaching it to a new terminal. Started a long-running process over ssh, but have to leave and don't want to interrupt it? Just start a screen, use reptyr to grab it, and then kill the ssh session and head on home.

USAGE

reptyr PID

"reptyr PID" will grab the process with id PID and attach it to your current terminal.

After attaching, the process will take input from and write output to the new terminal, including ^C and ^Z. (Unfortunately, if you background it, you will still have to run "bg" or "fg" in the old terminal. This is likely impossible to fix in a reasonable way without patching your shell.)

manatwork ,Nov 20, 2014 at 22:59

I am quite sure you can not.

Check using ps x . If a process has a ? as controlling tty , you can not send input to it any more.

9942 ?        S      0:00 tail -F /var/log/messages
9947 pts/1    S      0:00 tail -F /var/log/messages

In this example, you can send input to 9947 doing something like echo "test" > /dev/pts/1 . The other process ( 9942 ) is not reachable.

Next time, you could use screen or tmux to avoid this situation.

Stéphane Gimenez ,Feb 16, 2012 at 16:16

EDIT : As Stephane Gimenez said, it's not that simple. It's only allowing you to print to a different terminal.

You can try to write to this process using /proc . It should be located in /proc/ pid /fd/0 , so a simple :

echo "hello" > /proc/PID/fd/0

should do it. I have not tried it, but it should work, as long as this process still has a valid stdin file descriptor. You can check it with ls -l on /proc/ pid /fd/ .

See nohup for more details about how to keep processes running.

Stéphane Gimenez ,Nov 20, 2015 at 5:08

Just ending the command line with & will not completely detach the process, it will just run it in the background. (With zsh you can use &! to actually detach it, otherwise you have do disown it later).

When a process runs in the background, it won't receive input from its controlling terminal anymore. But you can send it back into the foreground with fg and then it will read input again.

Otherwise, it's not possible to externally change its filedescriptors (including stdin) or to reattach a lost controlling terminal unless you use debugging tools (see Ansgar's answer , or have a look at the retty command).

[Mar 10, 2019] linux - Preventing tmux session created by systemd from automatically terminating on Ctrl+C - Stack Overflow

Mar 10, 2019 | stackoverflow.com

Preventing tmux session created by systemd from automatically terminating on Ctrl+C Ask Question -1


Jim Stewart ,Nov 10, 2018 at 12:55

Since a few days I'm successfully running the new Minecraft Bedrock Edition dedicated server on my Ubuntu 18.04 LTS home server. Because it should be available 24/7 and automatically startup after boot I created a systemd service for a detached tmux session:

tmux.minecraftserver.service

[Unit]
Description=tmux minecraft_server detached

[Service]
Type=forking
WorkingDirectory=/home/mine/minecraftserver
ExecStart=/usr/bin/tmux new -s minecraftserver -d "LD_LIBRARY_PATH=. /home/mine/minecraftser$
User=mine

[Install]
WantedBy=multi-user.target

Everything works as expected but there's one tiny thing that keeps bugging me:

How can I prevent tmux from terminating it's whole session when I press Ctrl+C ? I just want to terminate the Minecraft server process itself instead of the whole tmux session. When starting the server from the command line in a manually created tmux session this does work (session stays alive) but not when the session was brought up by systemd .

FlKo ,Nov 12, 2018 at 6:21

When starting the server from the command line in a manually created tmux session this does work (session stays alive) but not when the session was brought up by systemd .

The difference between these situations is actually unrelated to systemd. In one case, you're starting the server from a shell within the tmux session, and when the server terminates, control returns to the shell. In the other case, you're starting the server directly within the tmux session, and when it terminates there's no shell to return to, so the tmux session also dies.

tmux has an option to keep the session alive after the process inside it dies (look for remain-on-exit in the manpage), but that's probably not what you want: you want to be able to return to an interactive shell, to restart the server, investigate why it died, or perform maintenance tasks, for example. So it's probably better to change your command to this:

'LD_LIBRARY_PATH=. /home/mine/minecraftserver/ ; exec bash'

That is, first run the server, and then, after it terminates, replace the process (the shell which tmux implicitly spawns to run the command, but which will then exit) with another, interactive shell. (For some other ways to get an interactive shell after the command exits, see e. g. this question – but note that the <(echo commands) syntax suggested in the top answer is not available in systemd unit files.)

FlKo ,Nov 12, 2018 at 6:21

I as able to solve this by using systemd's ExecStartPost and tmux's send-keys like this:
[Unit]
Description=tmux minecraft_server detached

[Service]
Type=forking
WorkingDirectory=/home/mine/minecraftserver
ExecStart=/usr/bin/tmux new -d -s minecraftserver
ExecStartPost=/usr/bin/tmux send-keys -t minecraftserver "cd /home/mine/minecraftserver/" Enter "LD_LIBRARY_PATH=. ./bedrock_server" Enter

User=mine

[Install]
WantedBy=multi-user.target

[Mar 01, 2019] Creating symlinks instead of /bin /sbin /lib and /lib64 directories in RHEL7

That change essentially means that /usr should be on the root partition, not on a separate partition which with the current sizes of harddrive is a resobale requirement.
Notable quotes:
"... On Linux /bin and /usr/bin are still separate because it is common to have /usr on a separate partition (although this configuration breaks in subtle ways, sometimes). In /bin is all the commands that you will need if you only have / mounted. ..."
Mar 01, 2019 | unix.stackexchange.com

balki ,May 2, 2015 at 6:17

What? no /bin/ is not a symlink to /usr/bin on any FHS compliant system. Note that there are still popular Unixes and Linuxes that ignore this - for example, /bin and /sbin are symlinked to /usr/bin on Arch Linux (the reasoning being that you don't need /bin for rescue/single-user-mode, since you'd just boot a live CD).

/bin

contains commands that may be used by both the system administrator and by users, but which are required when no other filesystems are mounted (e.g. in single user mode). It may also contain commands which are used indirectly by scripts

/usr/bin/

This is the primary directory of executable commands on the system.

essentially, /bin contains executables which are required by the system for emergency repairs, booting, and single user mode. /usr/bin contains any binaries that aren't required.

I will note, that they can be on separate disks/partitions, /bin must be on the same disk as / . /usr/bin can be on another disk - although note that this configuration has been kind of broken for a while (this is why e.g. systemd warns about this configuration on boot).

For full correctness, some unices may ignore FHS, as I believe it is only a Linux Standard, I'm not aware that it has yet been included in SUS, Posix or any other UNIX standard, though it should be IMHO. It is a part of the LSB standard though.

LawrenceC ,Jan 13, 2015 at 16:12

/sbin - Binaries needed for booting, low-level system repair, or maintenance (run level 1 or S)

/bin - Binaries needed for normal/standard system functioning at any run level.

/usr/bin - Application/distribution binaries meant to be accessed by locally logged in users

/usr/sbin - Application/distribution binaries that support or configure stuff in /sbin.

/usr/share/bin - Application/distribution binaries or scripts meant to be accessed via the web, i.e. Apache web applications

*local* - Binaries not part of a distribution; locally compiled or manually installed. There's usually never a /local/bin but always a /usr/local/bin and /usr/local/share/bin .

JonnyJD ,Jan 3, 2013 at 0:17

Some kind of "update" on this issue:

Recently some Linux distributions are merging /bin into /usr/bin and relatedly /lib into /usr/lib . Sometimes also (/usr)/sbin to /usr/bin (Arch Linux). So /usr is expected to be available at the same time as / .

The distinction between the two hierarchies is taken to be unnecessary complexity now. The idea was once having only /bin available at boot, but having an initial ramdisk makes this obsolete.

I know of Fedora Linux (2011) and Arch Linux (2012) going this way and Solaris is doing this for a long time (> 15 years).

xenoterracide ,Jan 17, 2011 at 16:23

On Linux /bin and /usr/bin are still separate because it is common to have /usr on a separate partition (although this configuration breaks in subtle ways, sometimes). In /bin is all the commands that you will need if you only have / mounted.

On Solaris and Arch Linux (and probably others) /bin is a symlink to /usr/bin . Arch also has /sbin and /usr/sbin symlinked to /usr/bin .

Of particular note, the statement that /bin is for "system administrator" commands and /usr/bin is for user commands is not true (unless you think that bash and ls are for admins only, in which case you have a lot to learn). Administrator commands are in /sbin and /usr/sbin .

[Feb 21, 2019] The rm='rm -i' alias is an horror

Feb 21, 2019 | superuser.com

The rm='rm -i' alias is an horror because after a while using it, you will expect rm to prompt you by default before removing files. Of course, one day you'll run it with an account that hasn't that alias set and before you understand what's going on, it is too late.

... ... ...

If you want save aliases, but don't want to risk getting used to the commands working differently on your system than on others, you can to disable rm like this
alias rm='echo "rm is disabled, use remove or trash or /bin/rm instead."'

Then you can create your own safe alias, e.g.

alias remove='/bin/rm -irv'

or use trash instead.

[Feb 21, 2019] What is the minimum I have to do to create an RPM file?

Feb 21, 2019 | stackoverflow.com

webwesen ,Jan 29, 2016 at 6:42

I just want to create an RPM file to distribute my Linux binary "foobar", with only a couple of dependencies. It has a config file, /etc/foobar.conf and should be installed in /usr/bin/foobar.

Unfortunately the documentation for RPM is 27 chapters long and I really don't have a day to sit down and read this, because I am also busy making .deb and EXE installers for other platforms.

What is the absolute minimum I have to do to create an RPM? Assume the foobar binary and foobar.conf are in the current working directory.

icasimpan ,Apr 10, 2018 at 13:33

I often do binary rpm per packaging proprietary apps - also moster as websphere - on linux. So my experience could be useful also a you, besides that it would better to do a TRUE RPM if you can. But i digress.

So the a basic step for packaging your (binary) program is as follow - in which i suppose the program is toybinprog with version 1.0, have a conf to be installed in /etc/toybinprog/toybinprog.conf and have a bin to be installed in /usr/bin called tobinprog :

1. create your rpm build env for RPM < 4.6,4.7
mkdir -p ~/rpmbuild/{RPMS,SRPMS,BUILD,SOURCES,SPECS,tmp}

cat <<EOF >~/.rpmmacros
%_topdir   %(echo $HOME)/rpmbuild
%_tmppath  %{_topdir}/tmp
EOF

cd ~/rpmbuild
2. create the tarball of your project
mkdir toybinprog-1.0
mkdir -p toybinprog-1.0/usr/bin
mkdir -p toybinprog-1.0/etc/toybinprog
install -m 755 toybinprog toybinprog-1.0/usr/bin
install -m 644 toybinprog.conf toybinprog-1.0/etc/toybinprog/

tar -zcvf toybinprog-1.0.tar.gz toybinprog-1.0/
3. Copy to the sources dir
cp toybinprog-1.0.tar.gz SOURCES/

cat <<EOF > SPECS/toybinprog.spec
# Don't try fancy stuff like debuginfo, which is useless on binary-only
# packages. Don't strip binary too
# Be sure buildpolicy set to do nothing
%define        __spec_install_post %{nil}
%define          debug_package %{nil}
%define        __os_install_post %{_dbpath}/brp-compress

Summary: A very simple toy bin rpm package
Name: toybinprog
Version: 1.0
Release: 1
License: GPL+
Group: Development/Tools
SOURCE0 : %{name}-%{version}.tar.gz
URL: http://toybinprog.company.com/

BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root

%description
%{summary}

%prep
%setup -q

%build
# Empty section.

%install
rm -rf %{buildroot}
mkdir -p  %{buildroot}

# in builddir
cp -a * %{buildroot}


%clean
rm -rf %{buildroot}


%files
%defattr(-,root,root,-)
%config(noreplace) %{_sysconfdir}/%{name}/%{name}.conf
%{_bindir}/*

%changelog
* Thu Apr 24 2009  Elia Pinto <devzero2000@rpm5.org> 1.0-1
- First Build

EOF
4. build the source and the binary rpm
rpmbuild -ba SPECS/toybinprog.spec

And that's all.

Hope this help

> ,

As an application distributor, fpm sounds perfect for your needs . There is an example here which shows how to package an app from source. FPM can produce both deb files and RPM files.

[Feb 21, 2019] perl - How to prompt for input and exit if the user entered an empty string - Stack Overflow

Feb 20, 2019 | stackoverflow.com

NewLearner ,Mar 12, 2012 at 3:22

I'm new to Perl and I'm writing a program where I want to force the user to enter a word. If the user enters an empty string then the program should exit.

This is what I have so far:

print "Enter a word to look up: ";

chomp ($usrword = <STDIN>);

DVK , Nov 19, 2015 at 19:11

You're almost there.
print "Enter a word to look up: ";
my $userword = <STDIN>; # I moved chomp to a new line to make it more readable
chomp $userword; # Get rid of newline character at the end
exit 0 if ($userword eq ""); # If empty string, exit.

Pondy , Jul 6 '16 at 22:11

File output is buffered by default. Since the prompt is so short, it is still sitting in the output buffer. You can disable buffering on STDOUT by adding this line of code before printing...
select((select(STDOUT), $|=1)[0]);

[Feb 11, 2019] Resuming rsync on a interrupted transfer

May 15, 2013 | stackoverflow.com

Glitches , May 15, 2013 at 18:06

I am trying to backup my file server to a remove file server using rsync. Rsync is not successfully resuming when a transfer is interrupted. I used the partial option but rsync doesn't find the file it already started because it renames it to a temporary file and when resumed it creates a new file and starts from beginning.

Here is my command:

rsync -avztP -e "ssh -p 2222" /volume1/ myaccont@backup-server-1:/home/myaccount/backup/ --exclude "@spool" --exclude "@tmp"

When this command is ran, a backup file named OldDisk.dmg from my local machine get created on the remote machine as something like .OldDisk.dmg.SjDndj23 .

Now when the internet connection gets interrupted and I have to resume the transfer, I have to find where rsync left off by finding the temp file like .OldDisk.dmg.SjDndj23 and rename it to OldDisk.dmg so that it sees there already exists a file that it can resume.

How do I fix this so I don't have to manually intervene each time?

Richard Michael , Nov 6, 2013 at 4:26

TL;DR : Use --timeout=X (X in seconds) to change the default rsync server timeout, not --inplace .

The issue is the rsync server processes (of which there are two, see rsync --server ... in ps output on the receiver) continue running, to wait for the rsync client to send data.

If the rsync server processes do not receive data for a sufficient time, they will indeed timeout, self-terminate and cleanup by moving the temporary file to it's "proper" name (e.g., no temporary suffix). You'll then be able to resume.

If you don't want to wait for the long default timeout to cause the rsync server to self-terminate, then when your internet connection returns, log into the server and clean up the rsync server processes manually. However, you must politely terminate rsync -- otherwise, it will not move the partial file into place; but rather, delete it (and thus there is no file to resume). To politely ask rsync to terminate, do not SIGKILL (e.g., -9 ), but SIGTERM (e.g., pkill -TERM -x rsync - only an example, you should take care to match only the rsync processes concerned with your client).

Fortunately there is an easier way: use the --timeout=X (X in seconds) option; it is passed to the rsync server processes as well.

For example, if you specify rsync ... --timeout=15 ... , both the client and server rsync processes will cleanly exit if they do not send/receive data in 15 seconds. On the server, this means moving the temporary file into position, ready for resuming.

I'm not sure of the default timeout value of the various rsync processes will try to send/receive data before they die (it might vary with operating system). In my testing, the server rsync processes remain running longer than the local client. On a "dead" network connection, the client terminates with a broken pipe (e.g., no network socket) after about 30 seconds; you could experiment or review the source code. Meaning, you could try to "ride out" the bad internet connection for 15-20 seconds.

If you do not clean up the server rsync processes (or wait for them to die), but instead immediately launch another rsync client process, two additional server processes will launch (for the other end of your new client process). Specifically, the new rsync client will not re-use/reconnect to the existing rsync server processes. Thus, you'll have two temporary files (and four rsync server processes) -- though, only the newer, second temporary file has new data being written (received from your new rsync client process).

Interestingly, if you then clean up all rsync server processes (for example, stop your client which will stop the new rsync servers, then SIGTERM the older rsync servers, it appears to merge (assemble) all the partial files into the new proper named file. So, imagine a long running partial copy which dies (and you think you've "lost" all the copied data), and a short running re-launched rsync (oops!).. you can stop the second client, SIGTERM the first servers, it will merge the data, and you can resume.

Finally, a few short remarks:

JamesTheAwesomeDude , Dec 29, 2013 at 16:50

Just curious: wouldn't SIGINT (aka ^C ) be 'politer' than SIGTERM ? – JamesTheAwesomeDude Dec 29 '13 at 16:50

Richard Michael , Dec 29, 2013 at 22:34

I didn't test how the server-side rsync handles SIGINT, so I'm not sure it will keep the partial file - you could check. Note that this doesn't have much to do with Ctrl-c ; it happens that your terminal sends SIGINT to the foreground process when you press Ctrl-c , but the server-side rsync has no controlling terminal. You must log in to the server and use kill . The client-side rsync will not send a message to the server (for example, after the client receives SIGINT via your terminal Ctrl-c ) - might be interesting though. As for anthropomorphizing, not sure what's "politer". :-) – Richard Michael Dec 29 '13 at 22:34

d-b , Feb 3, 2015 at 8:48

I just tried this timeout argument rsync -av --delete --progress --stats --human-readable --checksum --timeout=60 --partial-dir /tmp/rsync/ rsync://$remote:/ /src/ but then it timed out during the "receiving file list" phase (which in this case takes around 30 minutes). Setting the timeout to half an hour so kind of defers the purpose. Any workaround for this? – d-b Feb 3 '15 at 8:48

Cees Timmerman , Sep 15, 2015 at 17:10

@user23122 --checksum reads all data when preparing the file list, which is great for many small files that change often, but should be done on-demand for large files. – Cees Timmerman Sep 15 '15 at 17:10

[Jan 29, 2019] Do journaling filesystems guarantee against corruption after a power failure

Jan 29, 2019 | unix.stackexchange.com

Nathan Osman ,May 6, 2011 at 1:50

I am asking this question on behalf of another user who raised the issue in the Ubuntu chat room.

Do journaling filesystems guarantee that no corruption will occur if a power failure occurs?

If this answer depends on the filesystem, please indicate which ones do protect against corruption and which ones don't.

Andrew Lambert ,May 6, 2011 at 2:51

There are no guarantees. A Journaling File System is more resilient and is less prone to corruption, but not immune.

All a journal is is a list of operations which have recently been done to the file system. The crucial part is that the journal entry is made before the operations take place. Most operations have multiple steps. Deleting a file, for example might entail deleting the file's entry in the file system's table of contents and then marking the sectors on the drive as free. If something happens between the two steps, a journaled file system can tell immediately and perform the necessary clean up to keep everything consistent. This is not the case with a non-journaled file system which has to look at the entire contents of the volume to find errors.

While this journaling is much less prone to corruption than not journaling, corruption can still occur. For example, if the hard drive is mechanically malfunctioning or if writes to the journal itself are failing or interrupted.

The basic premise of journaling is that writing a journal entry is much quicker, usually, than the actual transaction it describes will be. So, the period between the OS ordering a (journal) write and the hard drive fulfilling it is much shorter than for a normal write: a narrower window for things to go wrong in, but there's still a window.

Further reading

Nathan Osman ,May 6, 2011 at 2:57

Could you please elaborate a little bit on why this is true? Perhaps you could give an example of how corruption would occur in a certain scenario. – Nathan Osman May 6 '11 at 2:57

Andrew Lambert ,May 6, 2011 at 3:21

@George Edison See my expanded answer. – Andrew Lambert May 6 '11 at 3:21

psusi ,May 6, 2011 at 17:58

That last bit is incorrect; there is no window for things to go wrong. Since it records what it is about to do before it starts doing it, the operation can be restarted after the power failure, no matter at what point it occurs during the operation. It is a matter of ordering, not timing. – psusi May 6 '11 at 17:58

Andrew Lambert ,May 6, 2011 at 21:23

@psusi there is still a window for the write to the journal to be interrupted. Journal writes may appear atomic to the OS but they're still writes to the disk. – Andrew Lambert May 6 '11 at 21:23

psusi ,May 7, 2011 at 1:57

@Amazed they are atomic because they have sequence numbers and/or checksums, so the journal entry is either written entirely, or not. If it is not written entirely, it is simply ignored after the system restarts, and no further changes were made to the fs so it remains consistent. – psusi May 7 '11 at 1:57

Mikel ,May 6, 2011 at 6:03

No.

The most common type of journaling, called metadata journaling, only protects the integrity of the file system, not of data. This includes xfs , and ext3 / ext4 in the default data=ordered mode.

If a non-journaling file system suffers a crash, it will be checked using fsck on the next boot. fsck scans every inode on the file system, looking for blocks that are marked as used but are not reachable (i.e. have no file name), and marks those blocks as unused. Doing this takes a long time.

With a metadata journaling file system, instead of doing an fsck , it knows which blocks it was in the middle of changing, so it can mark them as free without searching the whole partition for them.

There is a less common type of journaling, called data journaling, which is what ext3 does if you mount it with the data=journal option.

It attempts to protect all your data by writing not just a list of logical operations, but also the entire contents of each write to the journal. But because it's writing your data twice, it can be much slower.

As others have pointed out, even this is not a guarantee, because the hard drive might have told the operating system it had stored the data, when it fact it was still in the hard drive's cache.

For more information, take a look at the Wikipedia Journaling File System article and the Data Mode section of the ext4 documentation .

SplinterReality ,May 6, 2011 at 8:03

+1 for the distinction between file system corruption and data corruption. That little distinction is quite the doozy in practice. – SplinterReality May 6 '11 at 8:03

boehj ,May 6, 2011 at 10:57

Excuse my utter ignorance, but doesn't data=journal as a feature make no sense at all? – boehj May 6 '11 at 10:57

psusi ,May 6, 2011 at 18:11

Again, the OS knows when the drive caches data and forces it to flush it when needed in order to maintain a coherent fs. Your data file of course, can be lost or corrupted if the application that was writing it when the power failed was not doing so carefully, and that applies whether or not you use data=journal. – psusi May 6 '11 at 18:11

user3338098 ,Aug 1, 2016 at 16:30

@psusi doesn't matter how careful the program is in writing the data, plenty of hard drives silently corrupt the data on READING stackoverflow.com/q/34141117/3338098user3338098 Aug 1 '16 at 16:30

psusi ,Aug 21, 2016 at 3:22

@user3338098, drives that silently corrupt data are horribly broken and should not ever be used, and are an entirely different conversation than corruption caused by software doing the wrong thing. – psusi Aug 21 '16 at 3:22

camh ,May 6, 2011 at 3:26

A filesystem cannot guarantee the consistency of its filesystem if a power failure occurs, because it does not know what the hardware will do.

If a hard drive buffers data for write but tells the OS that it has written the data and does not support the appropriate write barriers, then out-of-order writes can occur where an earlier write has not hit the platter, but a later one has. See this serverfault answer for more details.

Also, the position of the head on a magnetic HDD is controlled with electro-magnets. If power fails in the middle of a write, it is possible for some data to continue to be written while the heads move, corrupting data on blocks that the filesystem never intended to be written.

Nathan Osman ,May 6, 2011 at 6:43

Isn't the drive's firmware smart enough to suspend writing when retracting the head? – Nathan Osman May 6 '11 at 6:43

camh ,May 6, 2011 at 7:54

@George: It's going to depend on the drive. There's a lot out there and you don't know how well your (cheap) drive does things. – camh May 6 '11 at 7:54

psusi ,May 6, 2011 at 18:05

The hard drive tells the OS if it uses a write behind cache, and the OS takes measures to ensure they are flushed in the correct order. Also drives are designed so that when the power fails, they stop writing. I have seen some cases where the sector being written at the time of power loss becomes corrupt because it did not finish updating the ecc ( but can be easily re-written correctly ), but never heard of random sectors being corrupted on power loss. – psusi May 6 '11 at 18:05

jlliagre ,May 6, 2011 at 8:35

ZFS, which is close but not exactly a journaling filesystem, is guaranteeing by design against corruption after a power failure.

It doesn't matter if an ongoing write is interrupted in the middle as in such case, its checksum will be certainly incorrect so the block will be ignored. As the file system is copy on write, the previous correct data (or meta-data) is still on disk and will be used instead.

sakisk ,May 6, 2011 at 10:13

The answer is in most cases no:

Nathan Osman ,May 6, 2011 at 16:35

What events could lead to a corrupt journal? The only thing I could think of was bad sectors - is there anything else? – Nathan Osman May 6 '11 at 16:35

sakisk ,May 7, 2011 at 13:21

That's right, hardware failures are the usual case. – sakisk May 7 '11 at 13:21

[Jan 29, 2019] Split string into an array in Bash

May 14, 2012 | stackoverflow.com

Lgn ,May 14, 2012 at 15:15

In a Bash script I would like to split a line into pieces and store them in an array.

The line:

Paris, France, Europe

I would like to have them in an array like this:

array[0] = Paris
array[1] = France
array[2] = Europe

I would like to use simple code, the command's speed doesn't matter. How can I do it?

antak ,Jun 18, 2018 at 9:22

This is #1 Google hit but there's controversy in the answer because the question unfortunately asks about delimiting on , (comma-space) and not a single character such as comma. If you're only interested in the latter, answers here are easier to follow: stackoverflow.com/questions/918886/ – antak Jun 18 '18 at 9:22

Dennis Williamson ,May 14, 2012 at 15:16

IFS=', ' read -r -a array <<< "$string"

Note that the characters in $IFS are treated individually as separators so that in this case fields may be separated by either a comma or a space rather than the sequence of the two characters. Interestingly though, empty fields aren't created when comma-space appears in the input because the space is treated specially.

To access an individual element:

echo "${array[0]}"

To iterate over the elements:

for element in "${array[@]}"
do
    echo "$element"
done

To get both the index and the value:

for index in "${!array[@]}"
do
    echo "$index ${array[index]}"
done

The last example is useful because Bash arrays are sparse. In other words, you can delete an element or add an element and then the indices are not contiguous.

unset "array[1]"
array[42]=Earth

To get the number of elements in an array:

echo "${#array[@]}"

As mentioned above, arrays can be sparse so you shouldn't use the length to get the last element. Here's how you can in Bash 4.2 and later:

echo "${array[-1]}"

in any version of Bash (from somewhere after 2.05b):

echo "${array[@]: -1:1}"

Larger negative offsets select farther from the end of the array. Note the space before the minus sign in the older form. It is required.

l0b0 ,May 14, 2012 at 15:24

Just use IFS=', ' , then you don't have to remove the spaces separately. Test: IFS=', ' read -a array <<< "Paris, France, Europe"; echo "${array[@]}" – l0b0 May 14 '12 at 15:24

Dennis Williamson ,May 14, 2012 at 16:33

@l0b0: Thanks. I don't know what I was thinking. I like to use declare -p array for test output, by the way. – Dennis Williamson May 14 '12 at 16:33

Nathan Hyde ,Mar 16, 2013 at 21:09

@Dennis Williamson - Awesome, thorough answer. – Nathan Hyde Mar 16 '13 at 21:09

dsummersl ,Aug 9, 2013 at 14:06

MUCH better than multiple cut -f calls! – dsummersl Aug 9 '13 at 14:06

caesarsol ,Oct 29, 2015 at 14:45

Warning: the IFS variable means split by one of these characters , so it's not a sequence of chars to split by. IFS=', ' read -a array <<< "a,d r s,w" => ${array[*]} == a d r s w – caesarsol Oct 29 '15 at 14:45

Jim Ho ,Mar 14, 2013 at 2:20

Here is a way without setting IFS:
string="1:2:3:4:5"
set -f                      # avoid globbing (expansion of *).
array=(${string//:/ })
for i in "${!array[@]}"
do
    echo "$i=>${array[i]}"
done

The idea is using string replacement:

${string//substring/replacement}

to replace all matches of $substring with white space and then using the substituted string to initialize a array:

(element1 element2 ... elementN)

Note: this answer makes use of the split+glob operator . Thus, to prevent expansion of some characters (such as * ) it is a good idea to pause globbing for this script.

Werner Lehmann ,May 4, 2013 at 22:32

Used this approach... until I came across a long string to split. 100% CPU for more than a minute (then I killed it). It's a pity because this method allows to split by a string, not some character in IFS. – Werner Lehmann May 4 '13 at 22:32

Dieter Gribnitz ,Sep 2, 2014 at 15:46

WARNING: Just ran into a problem with this approach. If you have an element named * you will get all the elements of your cwd as well. thus string="1:2:3:4:*" will give some unexpected and possibly dangerous results depending on your implementation. Did not get the same error with (IFS=', ' read -a array <<< "$string") and this one seems safe to use. – Dieter Gribnitz Sep 2 '14 at 15:46

akostadinov ,Nov 6, 2014 at 14:31

not reliable for many kinds of values, use with care – akostadinov Nov 6 '14 at 14:31

Andrew White ,Jun 1, 2016 at 11:44

quoting ${string//:/ } prevents shell expansion – Andrew White Jun 1 '16 at 11:44

Mark Thomson ,Jun 5, 2016 at 20:44

I had to use the following on OSX: array=(${string//:/ }) – Mark Thomson Jun 5 '16 at 20:44

bgoldst ,Jul 19, 2017 at 21:20

All of the answers to this question are wrong in one way or another.

Wrong answer #1

IFS=', ' read -r -a array <<< "$string"

1: This is a misuse of $IFS . The value of the $IFS variable is not taken as a single variable-length string separator, rather it is taken as a set of single-character string separators, where each field that read splits off from the input line can be terminated by any character in the set (comma or space, in this example).

Actually, for the real sticklers out there, the full meaning of $IFS is slightly more involved. From the bash manual :

The shell treats each character of IFS as a delimiter, and splits the results of the other expansions into words using these characters as field terminators. If IFS is unset, or its value is exactly <space><tab><newline> , the default, then sequences of <space> , <tab> , and <newline> at the beginning and end of the results of the previous expansions are ignored, and any sequence of IFS characters not at the beginning or end serves to delimit words. If IFS has a value other than the default, then sequences of the whitespace characters <space> , <tab> , and <newline> are ignored at the beginning and end of the word, as long as the whitespace character is in the value of IFS (an IFS whitespace character). Any character in IFS that is not IFS whitespace, along with any adjacent IFS whitespace characters, delimits a field. A sequence of IFS whitespace characters is also treated as a delimiter. If the value of IFS is null, no word splitting occurs.

Basically, for non-default non-null values of $IFS , fields can be separated with either (1) a sequence of one or more characters that are all from the set of "IFS whitespace characters" (that is, whichever of <space> , <tab> , and <newline> ("newline" meaning line feed (LF) ) are present anywhere in $IFS ), or (2) any non-"IFS whitespace character" that's present in $IFS along with whatever "IFS whitespace characters" surround it in the input line.

For the OP, it's possible that the second separation mode I described in the previous paragraph is exactly what he wants for his input string, but we can be pretty confident that the first separation mode I described is not correct at all. For example, what if his input string was 'Los Angeles, United States, North America' ?

IFS=', ' read -ra a <<<'Los Angeles, United States, North America'; declare -p a;
## declare -a a=([0]="Los" [1]="Angeles" [2]="United" [3]="States" [4]="North" [5]="America")

2: Even if you were to use this solution with a single-character separator (such as a comma by itself, that is, with no following space or other baggage), if the value of the $string variable happens to contain any LFs, then read will stop processing once it encounters the first LF. The read builtin only processes one line per invocation. This is true even if you are piping or redirecting input only to the read statement, as we are doing in this example with the here-string mechanism, and thus unprocessed input is guaranteed to be lost. The code that powers the read builtin has no knowledge of the data flow within its containing command structure.

You could argue that this is unlikely to cause a problem, but still, it's a subtle hazard that should be avoided if possible. It is caused by the fact that the read builtin actually does two levels of input splitting: first into lines, then into fields. Since the OP only wants one level of splitting, this usage of the read builtin is not appropriate, and we should avoid it.

3: A non-obvious potential issue with this solution is that read always drops the trailing field if it is empty, although it preserves empty fields otherwise. Here's a demo:

string=', , a, , b, c, , , '; IFS=', ' read -ra a <<<"$string"; declare -p a;
## declare -a a=([0]="" [1]="" [2]="a" [3]="" [4]="b" [5]="c" [6]="" [7]="")

Maybe the OP wouldn't care about this, but it's still a limitation worth knowing about. It reduces the robustness and generality of the solution.

This problem can be solved by appending a dummy trailing delimiter to the input string just prior to feeding it to read , as I will demonstrate later.


Wrong answer #2

string="1:2:3:4:5"
set -f                     # avoid globbing (expansion of *).
array=(${string//:/ })

Similar idea:

t="one,two,three"
a=($(echo $t | tr ',' "\n"))

(Note: I added the missing parentheses around the command substitution which the answerer seems to have omitted.)

Similar idea:

string="1,2,3,4"
array=(`echo $string | sed 's/,/\n/g'`)

These solutions leverage word splitting in an array assignment to split the string into fields. Funnily enough, just like read , general word splitting also uses the $IFS special variable, although in this case it is implied that it is set to its default value of <space><tab><newline> , and therefore any sequence of one or more IFS characters (which are all whitespace characters now) is considered to be a field delimiter.

This solves the problem of two levels of splitting committed by read , since word splitting by itself constitutes only one level of splitting. But just as before, the problem here is that the individual fields in the input string can already contain $IFS characters, and thus they would be improperly split during the word splitting operation. This happens to not be the case for any of the sample input strings provided by these answerers (how convenient...), but of course that doesn't change the fact that any code base that used this idiom would then run the risk of blowing up if this assumption were ever violated at some point down the line. Once again, consider my counterexample of 'Los Angeles, United States, North America' (or 'Los Angeles:United States:North America' ).

Also, word splitting is normally followed by filename expansion ( aka pathname expansion aka globbing), which, if done, would potentially corrupt words containing the characters * , ? , or [ followed by ] (and, if extglob is set, parenthesized fragments preceded by ? , * , + , @ , or ! ) by matching them against file system objects and expanding the words ("globs") accordingly. The first of these three answerers has cleverly undercut this problem by running set -f beforehand to disable globbing. Technically this works (although you should probably add set +f afterward to reenable globbing for subsequent code which may depend on it), but it's undesirable to have to mess with global shell settings in order to hack a basic string-to-array parsing operation in local code.

Another issue with this answer is that all empty fields will be lost. This may or may not be a problem, depending on the application.

Note: If you're going to use this solution, it's better to use the ${string//:/ } "pattern substitution" form of parameter expansion , rather than going to the trouble of invoking a command substitution (which forks the shell), starting up a pipeline, and running an external executable ( tr or sed ), since parameter expansion is purely a shell-internal operation. (Also, for the tr and sed solutions, the input variable should be double-quoted inside the command substitution; otherwise word splitting would take effect in the echo command and potentially mess with the field values. Also, the $(...) form of command substitution is preferable to the old `...` form since it simplifies nesting of command substitutions and allows for better syntax highlighting by text editors.)


Wrong answer #3

str="a, b, c, d"  # assuming there is a space after ',' as in Q
arr=(${str//,/})  # delete all occurrences of ','

This answer is almost the same as #2 . The difference is that the answerer has made the assumption that the fields are delimited by two characters, one of which being represented in the default $IFS , and the other not. He has solved this rather specific case by removing the non-IFS-represented character using a pattern substitution expansion and then using word splitting to split the fields on the surviving IFS-represented delimiter character.

This is not a very generic solution. Furthermore, it can be argued that the comma is really the "primary" delimiter character here, and that stripping it and then depending on the space character for field splitting is simply wrong. Once again, consider my counterexample: 'Los Angeles, United States, North America' .

Also, again, filename expansion could corrupt the expanded words, but this can be prevented by temporarily disabling globbing for the assignment with set -f and then set +f .

Also, again, all empty fields will be lost, which may or may not be a problem depending on the application.


Wrong answer #4

string='first line
second line
third line'

oldIFS="$IFS"
IFS='
'
IFS=${IFS:0:1} # this is useful to format your code with tabs
lines=( $string )
IFS="$oldIFS"

This is similar to #2 and #3 in that it uses word splitting to get the job done, only now the code explicitly sets $IFS to contain only the single-character field delimiter present in the input string. It should be repeated that this cannot work for multicharacter field delimiters such as the OP's comma-space delimiter. But for a single-character delimiter like the LF used in this example, it actually comes close to being perfect. The fields cannot be unintentionally split in the middle as we saw with previous wrong answers, and there is only one level of splitting, as required.

One problem is that filename expansion will corrupt affected words as described earlier, although once again this can be solved by wrapping the critical statement in set -f and set +f .

Another potential problem is that, since LF qualifies as an "IFS whitespace character" as defined earlier, all empty fields will be lost, just as in #2 and #3 . This would of course not be a problem if the delimiter happens to be a non-"IFS whitespace character", and depending on the application it may not matter anyway, but it does vitiate the generality of the solution.

So, to sum up, assuming you have a one-character delimiter, and it is either a non-"IFS whitespace character" or you don't care about empty fields, and you wrap the critical statement in set -f and set +f , then this solution works, but otherwise not.

(Also, for information's sake, assigning a LF to a variable in bash can be done more easily with the $'...' syntax, e.g. IFS=$'\n'; .)


Wrong answer #5

countries='Paris, France, Europe'
OIFS="$IFS"
IFS=', ' array=($countries)
IFS="$OIFS"

Similar idea:

IFS=', ' eval 'array=($string)'

This solution is effectively a cross between #1 (in that it sets $IFS to comma-space) and #2-4 (in that it uses word splitting to split the string into fields). Because of this, it suffers from most of the problems that afflict all of the above wrong answers, sort of like the worst of all worlds.

Also, regarding the second variant, it may seem like the eval call is completely unnecessary, since its argument is a single-quoted string literal, and therefore is statically known. But there's actually a very non-obvious benefit to using eval in this way. Normally, when you run a simple command which consists of a variable assignment only , meaning without an actual command word following it, the assignment takes effect in the shell environment:

IFS=', '; ## changes $IFS in the shell environment

This is true even if the simple command involves multiple variable assignments; again, as long as there's no command word, all variable assignments affect the shell environment:

IFS=', ' array=($countries); ## changes both $IFS and $array in the shell environment

But, if the variable assignment is attached to a command name (I like to call this a "prefix assignment") then it does not affect the shell environment, and instead only affects the environment of the executed command, regardless whether it is a builtin or external:

IFS=', ' :; ## : is a builtin command, the $IFS assignment does not outlive it
IFS=', ' env; ## env is an external command, the $IFS assignment does not outlive it

Relevant quote from the bash manual :

If no command name results, the variable assignments affect the current shell environment. Otherwise, the variables are added to the environment of the executed command and do not affect the current shell environment.

It is possible to exploit this feature of variable assignment to change $IFS only temporarily, which allows us to avoid the whole save-and-restore gambit like that which is being done with the $OIFS variable in the first variant. But the challenge we face here is that the command we need to run is itself a mere variable assignment, and hence it would not involve a command word to make the $IFS assignment temporary. You might think to yourself, well why not just add a no-op command word to the statement like the : builtin to make the $IFS assignment temporary? This does not work because it would then make the $array assignment temporary as well:

IFS=', ' array=($countries) :; ## fails; new $array value never escapes the : command

So, we're effectively at an impasse, a bit of a catch-22. But, when eval runs its code, it runs it in the shell environment, as if it was normal, static source code, and therefore we can run the $array assignment inside the eval argument to have it take effect in the shell environment, while the $IFS prefix assignment that is prefixed to the eval command will not outlive the eval command. This is exactly the trick that is being used in the second variant of this solution:

IFS=', ' eval 'array=($string)'; ## $IFS does not outlive the eval command, but $array does

So, as you can see, it's actually quite a clever trick, and accomplishes exactly what is required (at least with respect to assignment effectation) in a rather non-obvious way. I'm actually not against this trick in general, despite the involvement of eval ; just be careful to single-quote the argument string to guard against security threats.

But again, because of the "worst of all worlds" agglomeration of problems, this is still a wrong answer to the OP's requirement.


Wrong answer #6

IFS=', '; array=(Paris, France, Europe)

IFS=' ';declare -a array=(Paris France Europe)

Um... what? The OP has a string variable that needs to be parsed into an array. This "answer" starts with the verbatim contents of the input string pasted into an array literal. I guess that's one way to do it.

It looks like the answerer may have assumed that the $IFS variable affects all bash parsing in all contexts, which is not true. From the bash manual:

IFS The Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read builtin command. The default value is <space><tab><newline> .

So the $IFS special variable is actually only used in two contexts: (1) word splitting that is performed after expansion (meaning not when parsing bash source code) and (2) for splitting input lines into words by the read builtin.

Let me try to make this clearer. I think it might be good to draw a distinction between parsing and execution . Bash must first parse the source code, which obviously is a parsing event, and then later it executes the code, which is when expansion comes into the picture. Expansion is really an execution event. Furthermore, I take issue with the description of the $IFS variable that I just quoted above; rather than saying that word splitting is performed after expansion , I would say that word splitting is performed during expansion, or, perhaps even more precisely, word splitting is part of the expansion process. The phrase "word splitting" refers only to this step of expansion; it should never be used to refer to the parsing of bash source code, although unfortunately the docs do seem to throw around the words "split" and "words" a lot. Here's a relevant excerpt from the linux.die.net version of the bash manual:

Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion , tilde expansion , parameter and variable expansion , command substitution , arithmetic expansion , word splitting , and pathname expansion .

The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and pathname expansion.

You could argue the GNU version of the manual does slightly better, since it opts for the word "tokens" instead of "words" in the first sentence of the Expansion section:

Expansion is performed on the command line after it has been split into tokens.

The important point is, $IFS does not change the way bash parses source code. Parsing of bash source code is actually a very complex process that involves recognition of the various elements of shell grammar, such as command sequences, command lists, pipelines, parameter expansions, arithmetic substitutions, and command substitutions. For the most part, the bash parsing process cannot be altered by user-level actions like variable assignments (actually, there are some minor exceptions to this rule; for example, see the various compatxx shell settings , which can change certain aspects of parsing behavior on-the-fly). The upstream "words"/"tokens" that result from this complex parsing process are then expanded according to the general process of "expansion" as broken down in the above documentation excerpts, where word splitting of the expanded (expanding?) text into downstream words is simply one step of that process. Word splitting only touches text that has been spit out of a preceding expansion step; it does not affect literal text that was parsed right off the source bytestream.


Wrong answer #7

string='first line
        second line
        third line'

while read -r line; do lines+=("$line"); done <<<"$string"

This is one of the best solutions. Notice that we're back to using read . Didn't I say earlier that read is inappropriate because it performs two levels of splitting, when we only need one? The trick here is that you can call read in such a way that it effectively only does one level of splitting, specifically by splitting off only one field per invocation, which necessitates the cost of having to call it repeatedly in a loop. It's a bit of a sleight of hand, but it works.

But there are problems. First: When you provide at least one NAME argument to read , it automatically ignores leading and trailing whitespace in each field that is split off from the input string. This occurs whether $IFS is set to its default value or not, as described earlier in this post. Now, the OP may not care about this for his specific use-case, and in fact, it may be a desirable feature of the parsing behavior. But not everyone who wants to parse a string into fields will want this. There is a solution, however: A somewhat non-obvious usage of read is to pass zero NAME arguments. In this case, read will store the entire input line that it gets from the input stream in a variable named $REPLY , and, as a bonus, it does not strip leading and trailing whitespace from the value. This is a very robust usage of read which I've exploited frequently in my shell programming career. Here's a demonstration of the difference in behavior:

string=$'  a  b  \n  c  d  \n  e  f  '; ## input string

a=(); while read -r line; do a+=("$line"); done <<<"$string"; declare -p a;
## declare -a a=([0]="a  b" [1]="c  d" [2]="e  f") ## read trimmed surrounding whitespace

a=(); while read -r; do a+=("$REPLY"); done <<<"$string"; declare -p a;
## declare -a a=([0]="  a  b  " [1]="  c  d  " [2]="  e  f  ") ## no trimming

The second issue with this solution is that it does not actually address the case of a custom field separator, such as the OP's comma-space. As before, multicharacter separators are not supported, which is an unfortunate limitation of this solution. We could try to at least split on comma by specifying the separator to the -d option, but look what happens:

string='Paris, France, Europe';
a=(); while read -rd,; do a+=("$REPLY"); done <<<"$string"; declare -p a;
## declare -a a=([0]="Paris" [1]=" France")

Predictably, the unaccounted surrounding whitespace got pulled into the field values, and hence this would have to be corrected subsequently through trimming operations (this could also be done directly in the while-loop). But there's another obvious error: Europe is missing! What happened to it? The answer is that read returns a failing return code if it hits end-of-file (in this case we can call it end-of-string) without encountering a final field terminator on the final field. This causes the while-loop to break prematurely and we lose the final field.

Technically this same error afflicted the previous examples as well; the difference there is that the field separator was taken to be LF, which is the default when you don't specify the -d option, and the <<< ("here-string") mechanism automatically appends a LF to the string just before it feeds it as input to the command. Hence, in those cases, we sort of accidentally solved the problem of a dropped final field by unwittingly appending an additional dummy terminator to the input. Let's call this solution the "dummy-terminator" solution. We can apply the dummy-terminator solution manually for any custom delimiter by concatenating it against the input string ourselves when instantiating it in the here-string:

a=(); while read -rd,; do a+=("$REPLY"); done <<<"$string,"; declare -p a;
declare -a a=([0]="Paris" [1]=" France" [2]=" Europe")

There, problem solved. Another solution is to only break the while-loop if both (1) read returned failure and (2) $REPLY is empty, meaning read was not able to read any characters prior to hitting end-of-file. Demo:

a=(); while read -rd,|| [[ -n "$REPLY" ]]; do a+=("$REPLY"); done <<<"$string"; declare -p a;
## declare -a a=([0]="Paris" [1]=" France" [2]=$' Europe\n')

This approach also reveals the secretive LF that automatically gets appended to the here-string by the <<< redirection operator. It could of course be stripped off separately through an explicit trimming operation as described a moment ago, but obviously the manual dummy-terminator approach solves it directly, so we could just go with that. The manual dummy-terminator solution is actually quite convenient in that it solves both of these two problems (the dropped-final-field problem and the appended-LF problem) in one go.

So, overall, this is quite a powerful solution. It's only remaining weakness is a lack of support for multicharacter delimiters, which I will address later.


Wrong answer #8

string='first line
        second line
        third line'

readarray -t lines <<<"$string"

(This is actually from the same post as #7 ; the answerer provided two solutions in the same post.)

The readarray builtin, which is a synonym for mapfile , is ideal. It's a builtin command which parses a bytestream into an array variable in one shot; no messing with loops, conditionals, substitutions, or anything else. And it doesn't surreptitiously strip any whitespace from the input string. And (if -O is not given) it conveniently clears the target array before assigning to it. But it's still not perfect, hence my criticism of it as a "wrong answer".

First, just to get this out of the way, note that, just like the behavior of read when doing field-parsing, readarray drops the trailing field if it is empty. Again, this is probably not a concern for the OP, but it could be for some use-cases. I'll come back to this in a moment.

Second, as before, it does not support multicharacter delimiters. I'll give a fix for this in a moment as well.

Third, the solution as written does not parse the OP's input string, and in fact, it cannot be used as-is to parse it. I'll expand on this momentarily as well.

For the above reasons, I still consider this to be a "wrong answer" to the OP's question. Below I'll give what I consider to be the right answer.


Right answer

Here's a naοve attempt to make #8 work by just specifying the -d option:

string='Paris, France, Europe';
readarray -td, a <<<"$string"; declare -p a;
## declare -a a=([0]="Paris" [1]=" France" [2]=$' Europe\n')

We see the result is identical to the result we got from the double-conditional approach of the looping read solution discussed in #7 . We can almost solve this with the manual dummy-terminator trick:

readarray -td, a <<<"$string,"; declare -p a;
## declare -a a=([0]="Paris" [1]=" France" [2]=" Europe" [3]=$'\n')

The problem here is that readarray preserved the trailing field, since the <<< redirection operator appended the LF to the input string, and therefore the trailing field was not empty (otherwise it would've been dropped). We can take care of this by explicitly unsetting the final array element after-the-fact:

readarray -td, a <<<"$string,"; unset 'a[-1]'; declare -p a;
## declare -a a=([0]="Paris" [1]=" France" [2]=" Europe")

The only two problems that remain, which are actually related, are (1) the extraneous whitespace that needs to be trimmed, and (2) the lack of support for multicharacter delimiters.

The whitespace could of course be trimmed afterward (for example, see How to trim whitespace from a Bash variable? ). But if we can hack a multicharacter delimiter, then that would solve both problems in one shot.

Unfortunately, there's no direct way to get a multicharacter delimiter to work. The best solution I've thought of is to preprocess the input string to replace the multicharacter delimiter with a single-character delimiter that will be guaranteed not to collide with the contents of the input string. The only character that has this guarantee is the NUL byte . This is because, in bash (though not in zsh, incidentally), variables cannot contain the NUL byte. This preprocessing step can be done inline in a process substitution. Here's how to do it using awk :

readarray -td '' a < <(awk '{ gsub(/, /,"\0"); print; }' <<<"$string, "); unset 'a[-1]';
declare -p a;
## declare -a a=([0]="Paris" [1]="France" [2]="Europe")

There, finally! This solution will not erroneously split fields in the middle, will not cut out prematurely, will not drop empty fields, will not corrupt itself on filename expansions, will not automatically strip leading and trailing whitespace, will not leave a stowaway LF on the end, does not require loops, and does not settle for a single-character delimiter.


Trimming solution

Lastly, I wanted to demonstrate my own fairly intricate trimming solution using the obscure -C callback option of readarray . Unfortunately, I've run out of room against Stack Overflow's draconian 30,000 character post limit, so I won't be able to explain it. I'll leave that as an exercise for the reader.

function mfcb { local val="$4"; "$1"; eval "$2[$3]=\$val;"; };
function val_ltrim { if [[ "$val" =~ ^[[:space:]]+ ]]; then val="${val:${#BASH_REMATCH[0]}}"; fi; };
function val_rtrim { if [[ "$val" =~ [[:space:]]+$ ]]; then val="${val:0:${#val}-${#BASH_REMATCH[0]}}"; fi; };
function val_trim { val_ltrim; val_rtrim; };
readarray -c1 -C 'mfcb val_trim a' -td, <<<"$string,"; unset 'a[-1]'; declare -p a;
## declare -a a=([0]="Paris" [1]="France" [2]="Europe")

fbicknel ,Aug 18, 2017 at 15:57

It may also be helpful to note (though understandably you had no room to do so) that the -d option to readarray first appears in Bash 4.4. – fbicknel Aug 18 '17 at 15:57

Cyril Duchon-Doris ,Nov 3, 2017 at 9:16

You should add a "TL;DR : scroll 3 pages to see the right solution at the end of my answer" – Cyril Duchon-Doris Nov 3 '17 at 9:16

dawg ,Nov 26, 2017 at 22:28

Great answer (+1). If you change your awk to awk '{ gsub(/,[ ]+|$/,"\0"); print }' and eliminate that concatenation of the final ", " then you don't have to go through the gymnastics on eliminating the final record. So: readarray -td '' a < <(awk '{ gsub(/,[ ]+/,"\0"); print; }' <<<"$string") on Bash that supports readarray . Note your method is Bash 4.4+ I think because of the -d in readarray – dawg Nov 26 '17 at 22:28

datUser ,Feb 22, 2018 at 14:54

Looks like readarray is not an available builtin on OSX. – datUser Feb 22 '18 at 14:54

bgoldst ,Feb 23, 2018 at 3:37

@datUser That's unfortunate. Your version of bash must be too old for readarray . In this case, you can use the second-best solution built on read . I'm referring to this: a=(); while read -rd,; do a+=("$REPLY"); done <<<"$string,"; (with the awk substitution if you need multicharacter delimiter support). Let me know if you run into any problems; I'm pretty sure this solution should work on fairly old versions of bash, back to version 2-something, released like two decades ago. – bgoldst Feb 23 '18 at 3:37

Jmoney38 ,Jul 14, 2015 at 11:54

t="one,two,three"
a=($(echo "$t" | tr ',' '\n'))
echo "${a[2]}"

Prints three

shrimpwagon ,Oct 16, 2015 at 20:04

I actually prefer this approach. Simple. – shrimpwagon Oct 16 '15 at 20:04

Ben ,Oct 31, 2015 at 3:11

I copied and pasted this and it did did not work with echo, but did work when I used it in a for loop. – Ben Oct 31 '15 at 3:11

Pinaki Mukherjee ,Nov 9, 2015 at 20:22

This is the simplest approach. thanks – Pinaki Mukherjee Nov 9 '15 at 20:22

abalter ,Aug 30, 2016 at 5:13

This does not work as stated. @Jmoney38 or shrimpwagon if you can paste this in a terminal and get the desired output, please paste the result here. – abalter Aug 30 '16 at 5:13

leaf ,Jul 17, 2017 at 16:28

@abalter Works for me with a=($(echo $t | tr ',' "\n")) . Same result with a=($(echo $t | tr ',' ' ')) . – leaf Jul 17 '17 at 16:28

Luca Borrione ,Nov 2, 2012 at 13:44

Sometimes it happened to me that the method described in the accepted answer didn't work, especially if the separator is a carriage return.
In those cases I solved in this way:
string='first line
second line
third line'

oldIFS="$IFS"
IFS='
'
IFS=${IFS:0:1} # this is useful to format your code with tabs
lines=( $string )
IFS="$oldIFS"

for line in "${lines[@]}"
    do
        echo "--> $line"
done

Stefan van den Akker ,Feb 9, 2015 at 16:52

+1 This completely worked for me. I needed to put multiple strings, divided by a newline, into an array, and read -a arr <<< "$strings" did not work with IFS=$'\n' . – Stefan van den Akker Feb 9 '15 at 16:52

Stefan van den Akker ,Feb 10, 2015 at 13:49

Here is the answer to make the accepted answer work when the delimiter is a newline . – Stefan van den Akker Feb 10 '15 at 13:49

,Jul 24, 2015 at 21:24

The accepted answer works for values in one line.
If the variable has several lines:
string='first line
        second line
        third line'

We need a very different command to get all lines:

while read -r line; do lines+=("$line"); done <<<"$string"

Or the much simpler bash readarray :

readarray -t lines <<<"$string"

Printing all lines is very easy taking advantage of a printf feature:

printf ">[%s]\n" "${lines[@]}"

>[first line]
>[        second line]
>[        third line]

Mayhem ,Dec 31, 2015 at 3:13

While not every solution works for every situation, your mention of readarray... replaced my last two hours with 5 minutes... you got my vote – Mayhem Dec 31 '15 at 3:13

Derek 朕會功夫 ,Mar 23, 2018 at 19:14

readarray is the right answer. – Derek 朕會功夫 Mar 23 '18 at 19:14

ssanch ,Jun 3, 2016 at 15:24

This is similar to the approach by Jmoney38, but using sed:
string="1,2,3,4"
array=(`echo $string | sed 's/,/\n/g'`)
echo ${array[0]}

Prints 1

dawg ,Nov 26, 2017 at 19:59

The key to splitting your string into an array is the multi character delimiter of ", " . Any solution using IFS for multi character delimiters is inherently wrong since IFS is a set of those characters, not a string.

If you assign IFS=", " then the string will break on EITHER "," OR " " or any combination of them which is not an accurate representation of the two character delimiter of ", " .

You can use awk or sed to split the string, with process substitution:

#!/bin/bash

str="Paris, France, Europe"
array=()
while read -r -d $'\0' each; do   # use a NUL terminated field separator 
    array+=("$each")
done < <(printf "%s" "$str" | awk '{ gsub(/,[ ]+|$/,"\0"); print }')
declare -p array
# declare -a array=([0]="Paris" [1]="France" [2]="Europe") output

It is more efficient to use a regex you directly in Bash:

#!/bin/bash

str="Paris, France, Europe"

array=()
while [[ $str =~ ([^,]+)(,[ ]+|$) ]]; do
    array+=("${BASH_REMATCH[1]}")   # capture the field
    i=${#BASH_REMATCH}              # length of field + delimiter
    str=${str:i}                    # advance the string by that length
done                                # the loop deletes $str, so make a copy if needed

declare -p array
# declare -a array=([0]="Paris" [1]="France" [2]="Europe") output...

With the second form, there is no sub shell and it will be inherently faster.


Edit by bgoldst: Here are some benchmarks comparing my readarray solution to dawg's regex solution, and I also included the read solution for the heck of it (note: I slightly modified the regex solution for greater harmony with my solution) (also see my comments below the post):

## competitors
function c_readarray { readarray -td '' a < <(awk '{ gsub(/, /,"\0"); print; };' <<<"$1, "); unset 'a[-1]'; };
function c_read { a=(); local REPLY=''; while read -r -d ''; do a+=("$REPLY"); done < <(awk '{ gsub(/, /,"\0"); print; };' <<<"$1, "); };
function c_regex { a=(); local s="$1, "; while [[ $s =~ ([^,]+),\  ]]; do a+=("${BASH_REMATCH[1]}"); s=${s:${#BASH_REMATCH}}; done; };

## helper functions
function rep {
    local -i i=-1;
    for ((i = 0; i<$1; ++i)); do
        printf %s "$2";
    done;
}; ## end rep()

function testAll {
    local funcs=();
    local args=();
    local func='';
    local -i rc=-1;
    while [[ "$1" != ':' ]]; do
        func="$1";
        if [[ ! "$func" =~ ^[_a-zA-Z][_a-zA-Z0-9]*$ ]]; then
            echo "bad function name: $func" >&2;
            return 2;
        fi;
        funcs+=("$func");
        shift;
    done;
    shift;
    args=("$@");
    for func in "${funcs[@]}"; do
        echo -n "$func ";
        { time $func "${args[@]}" >/dev/null 2>&1; } 2>&1| tr '\n' '/';
        rc=${PIPESTATUS[0]}; if [[ $rc -ne 0 ]]; then echo "[$rc]"; else echo; fi;
    done| column -ts/;
}; ## end testAll()

function makeStringToSplit {
    local -i n=$1; ## number of fields
    if [[ $n -lt 0 ]]; then echo "bad field count: $n" >&2; return 2; fi;
    if [[ $n -eq 0 ]]; then
        echo;
    elif [[ $n -eq 1 ]]; then
        echo 'first field';
    elif [[ "$n" -eq 2 ]]; then
        echo 'first field, last field';
    else
        echo "first field, $(rep $[$1-2] 'mid field, ')last field";
    fi;
}; ## end makeStringToSplit()

function testAll_splitIntoArray {
    local -i n=$1; ## number of fields in input string
    local s='';
    echo "===== $n field$(if [[ $n -ne 1 ]]; then echo 's'; fi;) =====";
    s="$(makeStringToSplit "$n")";
    testAll c_readarray c_read c_regex : "$s";
}; ## end testAll_splitIntoArray()

## results
testAll_splitIntoArray 1;
## ===== 1 field =====
## c_readarray   real  0m0.067s   user 0m0.000s   sys  0m0.000s
## c_read        real  0m0.064s   user 0m0.000s   sys  0m0.000s
## c_regex       real  0m0.000s   user 0m0.000s   sys  0m0.000s
##
testAll_splitIntoArray 10;
## ===== 10 fields =====
## c_readarray   real  0m0.067s   user 0m0.000s   sys  0m0.000s
## c_read        real  0m0.064s   user 0m0.000s   sys  0m0.000s
## c_regex       real  0m0.001s   user 0m0.000s   sys  0m0.000s
##
testAll_splitIntoArray 100;
## ===== 100 fields =====
## c_readarray   real  0m0.069s   user 0m0.000s   sys  0m0.062s
## c_read        real  0m0.065s   user 0m0.000s   sys  0m0.046s
## c_regex       real  0m0.005s   user 0m0.000s   sys  0m0.000s
##
testAll_splitIntoArray 1000;
## ===== 1000 fields =====
## c_readarray   real  0m0.084s   user 0m0.031s   sys  0m0.077s
## c_read        real  0m0.092s   user 0m0.031s   sys  0m0.046s
## c_regex       real  0m0.125s   user 0m0.125s   sys  0m0.000s
##
testAll_splitIntoArray 10000;
## ===== 10000 fields =====
## c_readarray   real  0m0.209s   user 0m0.093s   sys  0m0.108s
## c_read        real  0m0.333s   user 0m0.234s   sys  0m0.109s
## c_regex       real  0m9.095s   user 0m9.078s   sys  0m0.000s
##
testAll_splitIntoArray 100000;
## ===== 100000 fields =====
## c_readarray   real  0m1.460s   user 0m0.326s   sys  0m1.124s
## c_read        real  0m2.780s   user 0m1.686s   sys  0m1.092s
## c_regex       real  17m38.208s   user 15m16.359s   sys  2m19.375s
##

bgoldst ,Nov 27, 2017 at 4:28

Very cool solution! I never thought of using a loop on a regex match, nifty use of $BASH_REMATCH . It works, and does indeed avoid spawning subshells. +1 from me. However, by way of criticism, the regex itself is a little non-ideal, in that it appears you were forced to duplicate part of the delimiter token (specifically the comma) so as to work around the lack of support for non-greedy multipliers (also lookarounds) in ERE ("extended" regex flavor built into bash). This makes it a little less generic and robust. – bgoldst Nov 27 '17 at 4:28

bgoldst ,Nov 27, 2017 at 4:28

Secondly, I did some benchmarking, and although the performance is better than the other solutions for smallish strings, it worsens exponentially due to the repeated string-rebuilding, becoming catastrophic for very large strings. See my edit to your answer. – bgoldst Nov 27 '17 at 4:28

dawg ,Nov 27, 2017 at 4:46

@bgoldst: What a cool benchmark! In defense of the regex, for 10's or 100's of thousands of fields (what the regex is splitting) there would probably be some form of record (like \n delimited text lines) comprising those fields so the catastrophic slow-down would likely not occur. If you have a string with 100,000 fields -- maybe Bash is not ideal ;-) Thanks for the benchmark. I learned a thing or two. – dawg Nov 27 '17 at 4:46

Geoff Lee ,Mar 4, 2016 at 6:02

Try this
IFS=', '; array=(Paris, France, Europe)
for item in ${array[@]}; do echo $item; done

It's simple. If you want, you can also add a declare (and also remove the commas):

IFS=' ';declare -a array=(Paris France Europe)

The IFS is added to undo the above but it works without it in a fresh bash instance

MrPotatoHead ,Nov 13, 2018 at 13:19

Pure bash multi-character delimiter solution.

As others have pointed out in this thread, the OP's question gave an example of a comma delimited string to be parsed into an array, but did not indicate if he/she was only interested in comma delimiters, single character delimiters, or multi-character delimiters.

Since Google tends to rank this answer at or near the top of search results, I wanted to provide readers with a strong answer to the question of multiple character delimiters, since that is also mentioned in at least one response.

If you're in search of a solution to a multi-character delimiter problem, I suggest reviewing Mallikarjun M 's post, in particular the response from gniourf_gniourf who provides this elegant pure BASH solution using parameter expansion:

#!/bin/bash
str="LearnABCtoABCSplitABCaABCString"
delimiter=ABC
s=$str$delimiter
array=();
while [[ $s ]]; do
    array+=( "${s%%"$delimiter"*}" );
    s=${s#*"$delimiter"};
done;
declare -p array

Link to cited comment/referenced post

Link to cited question: Howto split a string on a multi-character delimiter in bash?

Eduardo Cuomo ,Dec 19, 2016 at 15:27

Use this:
countries='Paris, France, Europe'
OIFS="$IFS"
IFS=', ' array=($countries)
IFS="$OIFS"

#${array[1]} == Paris
#${array[2]} == France
#${array[3]} == Europe

gniourf_gniourf ,Dec 19, 2016 at 17:22

Bad: subject to word splitting and pathname expansion. Please don't revive old questions with good answers to give bad answers. – gniourf_gniourf Dec 19 '16 at 17:22

Scott Weldon ,Dec 19, 2016 at 18:12

This may be a bad answer, but it is still a valid answer. Flaggers / reviewers: For incorrect answers such as this one, downvote, don't delete! – Scott Weldon Dec 19 '16 at 18:12

George Sovetov ,Dec 26, 2016 at 17:31

@gniourf_gniourf Could you please explain why it is a bad answer? I really don't understand when it fails. – George Sovetov Dec 26 '16 at 17:31

gniourf_gniourf ,Dec 26, 2016 at 18:07

@GeorgeSovetov: As I said, it's subject to word splitting and pathname expansion. More generally, splitting a string into an array as array=( $string ) is a (sadly very common) antipattern: word splitting occurs: string='Prague, Czech Republic, Europe' ; Pathname expansion occurs: string='foo[abcd],bar[efgh]' will fail if you have a file named, e.g., food or barf in your directory. The only valid usage of such a construct is when string is a glob. – gniourf_gniourf Dec 26 '16 at 18:07

user1009908 ,Jun 9, 2015 at 23:28

UPDATE: Don't do this, due to problems with eval.

With slightly less ceremony:

IFS=', ' eval 'array=($string)'

e.g.

string="foo, bar,baz"
IFS=', ' eval 'array=($string)'
echo ${array[1]} # -> bar

caesarsol ,Oct 29, 2015 at 14:42

eval is evil! don't do this. – caesarsol Oct 29 '15 at 14:42

user1009908 ,Oct 30, 2015 at 4:05

Pfft. No. If you're writing scripts large enough for this to matter, you're doing it wrong. In application code, eval is evil. In shell scripting, it's common, necessary, and inconsequential. – user1009908 Oct 30 '15 at 4:05

caesarsol ,Nov 2, 2015 at 18:19

put a $ in your variable and you'll see... I write many scripts and I never ever had to use a single eval – caesarsol Nov 2 '15 at 18:19

Dennis Williamson ,Dec 2, 2015 at 17:00

Eval command and security issues – Dennis Williamson Dec 2 '15 at 17:00

user1009908 ,Dec 22, 2015 at 23:04

You're right, this is only usable when the input is known to be clean. Not a robust solution. – user1009908 Dec 22 '15 at 23:04

Eduardo Lucio ,Jan 31, 2018 at 20:45

Here's my hack!

Splitting strings by strings is a pretty boring thing to do using bash. What happens is that we have limited approaches that only work in a few cases (split by ";", "/", "." and so on) or we have a variety of side effects in the outputs.

The approach below has required a number of maneuvers, but I believe it will work for most of our needs!

#!/bin/bash

# --------------------------------------
# SPLIT FUNCTION
# ----------------

F_SPLIT_R=()
f_split() {
    : 'It does a "split" into a given string and returns an array.

    Args:
        TARGET_P (str): Target string to "split".
        DELIMITER_P (Optional[str]): Delimiter used to "split". If not 
    informed the split will be done by spaces.

    Returns:
        F_SPLIT_R (array): Array with the provided string separated by the 
    informed delimiter.
    '

    F_SPLIT_R=()
    TARGET_P=$1
    DELIMITER_P=$2
    if [ -z "$DELIMITER_P" ] ; then
        DELIMITER_P=" "
    fi

    REMOVE_N=1
    if [ "$DELIMITER_P" == "\n" ] ; then
        REMOVE_N=0
    fi

    # NOTE: This was the only parameter that has been a problem so far! 
    # By Questor
    # [Ref.: https://unix.stackexchange.com/a/390732/61742]
    if [ "$DELIMITER_P" == "./" ] ; then
        DELIMITER_P="[.]/"
    fi

    if [ ${REMOVE_N} -eq 1 ] ; then

        # NOTE: Due to bash limitations we have some problems getting the 
        # output of a split by awk inside an array and so we need to use 
        # "line break" (\n) to succeed. Seen this, we remove the line breaks 
        # momentarily afterwards we reintegrate them. The problem is that if 
        # there is a line break in the "string" informed, this line break will 
        # be lost, that is, it is erroneously removed in the output! 
        # By Questor
        TARGET_P=$(awk 'BEGIN {RS="dn"} {gsub("\n", "3F2C417D448C46918289218B7337FCAF"); printf $0}' <<< "${TARGET_P}")

    fi

    # NOTE: The replace of "\n" by "3F2C417D448C46918289218B7337FCAF" results 
    # in more occurrences of "3F2C417D448C46918289218B7337FCAF" than the 
    # amount of "\n" that there was originally in the string (one more 
    # occurrence at the end of the string)! We can not explain the reason for 
    # this side effect. The line below corrects this problem! By Questor
    TARGET_P=${TARGET_P%????????????????????????????????}

    SPLIT_NOW=$(awk -F"$DELIMITER_P" '{for(i=1; i<=NF; i++){printf "%s\n", $i}}' <<< "${TARGET_P}")

    while IFS= read -r LINE_NOW ; do
        if [ ${REMOVE_N} -eq 1 ] ; then

            # NOTE: We use "'" to prevent blank lines with no other characters 
            # in the sequence being erroneously removed! We do not know the 
            # reason for this side effect! By Questor
            LN_NOW_WITH_N=$(awk 'BEGIN {RS="dn"} {gsub("3F2C417D448C46918289218B7337FCAF", "\n"); printf $0}' <<< "'${LINE_NOW}'")

            # NOTE: We use the commands below to revert the intervention made 
            # immediately above! By Questor
            LN_NOW_WITH_N=${LN_NOW_WITH_N%?}
            LN_NOW_WITH_N=${LN_NOW_WITH_N#?}

            F_SPLIT_R+=("$LN_NOW_WITH_N")
        else
            F_SPLIT_R+=("$LINE_NOW")
        fi
    done <<< "$SPLIT_NOW"
}

# --------------------------------------
# HOW TO USE
# ----------------

STRING_TO_SPLIT="
 * How do I list all databases and tables using psql?

\"
sudo -u postgres /usr/pgsql-9.4/bin/psql -c \"\l\"
sudo -u postgres /usr/pgsql-9.4/bin/psql <DB_NAME> -c \"\dt\"
\"

\"
\list or \l: list all databases
\dt: list all tables in the current database
\"

[Ref.: https://dba.stackexchange.com/questions/1285/how-do-i-list-all-databases-and-tables-using-psql]


"

f_split "$STRING_TO_SPLIT" "bin/psql -c"

# --------------------------------------
# OUTPUT AND TEST
# ----------------

ARR_LENGTH=${#F_SPLIT_R[*]}
for (( i=0; i<=$(( $ARR_LENGTH -1 )); i++ )) ; do
    echo " > -----------------------------------------"
    echo "${F_SPLIT_R[$i]}"
    echo " < -----------------------------------------"
done

if [ "$STRING_TO_SPLIT" == "${F_SPLIT_R[0]}bin/psql -c${F_SPLIT_R[1]}" ] ; then
    echo " > -----------------------------------------"
    echo "The strings are the same!"
    echo " < -----------------------------------------"
fi

sel-en-ium ,May 31, 2018 at 5:56

Another way to do it without modifying IFS:
read -r -a myarray <<< "${string//, /$IFS}"

Rather than changing IFS to match our desired delimiter, we can replace all occurrences of our desired delimiter ", " with contents of $IFS via "${string//, /$IFS}" .

Maybe this will be slow for very large strings though?

This is based on Dennis Williamson's answer.

rsjethani ,Sep 13, 2016 at 16:21

Another approach can be:
str="a, b, c, d"  # assuming there is a space after ',' as in Q
arr=(${str//,/})  # delete all occurrences of ','

After this 'arr' is an array with four strings. This doesn't require dealing IFS or read or any other special stuff hence much simpler and direct.

gniourf_gniourf ,Dec 26, 2016 at 18:12

Same (sadly common) antipattern as other answers: subject to word splitting and filename expansion. – gniourf_gniourf Dec 26 '16 at 18:12

Safter Arslan ,Aug 9, 2017 at 3:21

Another way would be:
string="Paris, France, Europe"
IFS=', ' arr=(${string})

Now your elements are stored in "arr" array. To iterate through the elements:

for i in ${arr[@]}; do echo $i; done

bgoldst ,Aug 13, 2017 at 22:38

I cover this idea in my answer ; see Wrong answer #5 (you might be especially interested in my discussion of the eval trick). Your solution leaves $IFS set to the comma-space value after-the-fact. – bgoldst Aug 13 '17 at 22:38

[Jan 28, 2019] regex - Safe rm -rf function in shell script

Jan 28, 2019 | stackoverflow.com

community wiki
5 revs
,May 23, 2017 at 12:26

This question is similar to What is the safest way to empty a directory in *nix?

I'm writing bash script which defines several path constants and will use them for file and directory manipulation (copying, renaming and deleting). Often it will be necessary to do something like:

rm -rf "/${PATH1}"
rm -rf "${PATH2}/"*

While developing this script I'd want to protect myself from mistyping names like PATH1 and PATH2 and avoid situations where they are expanded to empty string, thus resulting in wiping whole disk. I decided to create special wrapper:

rmrf() {
    if [[ $1 =~ "regex" ]]; then
        echo "Ignoring possibly unsafe path ${1}"
        exit 1
    fi

    shopt -s dotglob
    rm -rf -- $1
    shopt -u dotglob
}

Which will be called as:

rmrf "/${PATH1}"
rmrf "${PATH2}/"*

Regex (or sed expression) should catch paths like "*", "/*", "/**/", "///*" etc. but allow paths like "dir", "/dir", "/dir1/dir2/", "/dir1/dir2/*". Also I don't know how to enable shell globbing in case like "/dir with space/*". Any ideas?

EDIT: this is what I came up with so far:

rmrf() {
    local RES
    local RMPATH="${1}"
    SAFE=$(echo "${RMPATH}" | sed -r 's:^((\.?\*+/+)+.*|(/+\.?\*+)+.*|[\.\*/]+|.*/\.\*+)$::g')
    if [ -z "${SAFE}" ]; then
        echo "ERROR! Unsafe deletion of ${RMPATH}"
        return 1
    fi

    shopt -s dotglob
    if [ '*' == "${RMPATH: -1}" ]; then
        echo rm -rf -- "${RMPATH/%\*/}"*
        RES=$?
    else
        echo rm -rf -- "${RMPATH}"
        RES=$?
    fi
    shopt -u dotglob

    return $RES
}

Intended use is (note an asterisk inside quotes):

rmrf "${SOMEPATH}"
rmrf "${SOMEPATH}/*"

where $SOMEPATH is not system or /home directory (in my case all such operations are performed on filesystem mounted under /scratch directory).

CAVEATS:

SpliFF ,Jun 14, 2009 at 13:45

I've found a big danger with rm in bash is that bash usually doesn't stop for errors. That means that:
cd $SOMEPATH
rm -rf *

Is a very dangerous combination if the change directory fails. A safer way would be:

cd $SOMEPATH && rm -rf *

Which will ensure the rf won't run unless you are really in $SOMEPATH. This doesn't protect you from a bad $SOMEPATH but it can be combined with the advice given by others to help make your script safer.

EDIT: @placeybordeaux makes a good point that if $SOMEPATH is undefined or empty cd doesn't treat it as an error and returns 0. In light of that this answer should be considered unsafe unless $SOMEPATH is validated as existing and non-empty first. I believe cd with no args should be an illegal command since at best is performs a no-op and at worse it can lead to unexpected behaviour but it is what it is.

Sazzad Hissain Khan ,Jul 6, 2017 at 11:45

nice trick, I am one stupid victim. – Sazzad Hissain Khan Jul 6 '17 at 11:45

placeybordeaux ,Jun 21, 2018 at 22:59

If $SOMEPATH is empty won't this rm -rf the user's home directory? – placeybordeaux Jun 21 '18 at 22:59

SpliFF ,Jun 27, 2018 at 4:10

@placeybordeaux The && only runs the second command if the first succeeds - so if cd fails rm never runs – SpliFF Jun 27 '18 at 4:10

placeybordeaux ,Jul 3, 2018 at 18:46

@SpliFF at least in ZSH the return value of cd $NONEXISTANTVAR is 0placeybordeaux Jul 3 '18 at 18:46

ruakh ,Jul 13, 2018 at 6:46

Instead of cd $SOMEPATH , you should write cd "${SOMEPATH?}" . The ${varname?} notation ensures that the expansion fails with a warning-message if the variable is unset or empty (such that the && ... part is never run); the double-quotes ensure that special characters in $SOMEPATH , such as whitespace, don't have undesired effects. – ruakh Jul 13 '18 at 6:46

community wiki
2 revs
,Jul 24, 2009 at 22:36

There is a set -u bash directive that will cause exit, when uninitialized variable is used. I read about it here , with rm -rf as an example. I think that's what you're looking for. And here is set's manual .

,Jun 14, 2009 at 12:38

I think "rm" command has a parameter to avoid the deleting of "/". Check it out.

Max ,Jun 14, 2009 at 12:56

Thanks! I didn't know about such option. Actually it is named --preserve-root and is not mentioned in the manpage. – Max Jun 14 '09 at 12:56

Max ,Jun 14, 2009 at 13:18

On my system this option is on by default, but it cat't help in case like rm -ri /* – Max Jun 14 '09 at 13:18

ynimous ,Jun 14, 2009 at 12:42

I would recomend to use realpath(1) and not the command argument directly, so that you can avoid things like /A/B/../ or symbolic links.

Max ,Jun 14, 2009 at 13:30

Useful but non-standard command. I've found possible bash replacement: archlinux.org/pipermail/pacman-dev/2009-February/008130.htmlMax Jun 14 '09 at 13:30

Jonathan Leffler ,Jun 14, 2009 at 12:47

Generally, when I'm developing a command with operations such as ' rm -fr ' in it, I will neutralize the remove during development. One way of doing that is:
RMRF="echo rm -rf"
...
$RMRF "/${PATH1}"

This shows me what should be deleted - but does not delete it. I will do a manual clean up while things are under development - it is a small price to pay for not running the risk of screwing up everything.

The notation ' "/${PATH1}" ' is a little unusual; normally, you would ensure that PATH1 simply contains an absolute pathname.

Using the metacharacter with ' "${PATH2}/"* ' is unwise and unnecessary. The only difference between using that and using just ' "${PATH2}" ' is that if the directory specified by PATH2 contains any files or directories with names starting with dot, then those files or directories will not be removed. Such a design is unlikely and is rather fragile. It would be much simpler just to pass PATH2 and let the recursive remove do its job. Adding the trailing slash is not necessarily a bad idea; the system would have to ensure that $PATH2 contains a directory name, not just a file name, but the extra protection is rather minimal.

Using globbing with ' rm -fr ' is usually a bad idea. You want to be precise and restrictive and limiting in what it does - to prevent accidents. Of course, you'd never run the command (shell script you are developing) as root while it is under development - that would be suicidal. Or, if root privileges are absolutely necessary, you neutralize the remove operation until you are confident it is bullet-proof.

Max ,Jun 14, 2009 at 13:09

To delete subdirectories and files starting with dot I use "shopt -s dotglob". Using rm -rf "${PATH2}" is not appropriate because in my case PATH2 can be only removed by superuser and this results in error status for "rm" command (and I verify it to track other errors). – Max Jun 14 '09 at 13:09

Jonathan Leffler ,Jun 14, 2009 at 13:37

Then, with due respect, you should use a private sub-directory under $PATH2 that you can remove. Avoid glob expansion with commands like 'rm -rf' like you would avoid the plague (or should that be A/H1N1?). – Jonathan Leffler Jun 14 '09 at 13:37

Max ,Jun 14, 2009 at 14:10

Meanwhile I've found this perl project: http://code.google.com/p/safe-rm/

community wiki
too much php
,Jun 15, 2009 at 1:55

If it is possible, you should try and put everything into a folder with a hard-coded name which is unlikely to be found anywhere else on the filesystem, such as ' foofolder '. Then you can write your rmrf() function as:
rmrf() {
    rm -rf "foofolder/$PATH1"
    # or
    rm -rf "$PATH1/foofolder"
}

There is no way that function can delete anything but the files you want it to.

vadipp ,Jan 13, 2017 at 11:37

Actually there is a way: if PATH1 is something like ../../someotherdirvadipp Jan 13 '17 at 11:37

community wiki
btop
,Jun 15, 2009 at 6:34

You may use
set -f    # cf. help set

to disable filename generation (*).

community wiki
Howard Hong
,Oct 28, 2009 at 19:56

You don't need to use regular expressions.
Just assign the directories you want to protect to a variable and then iterate over the variable. eg:
protected_dirs="/ /bin /usr/bin /home $HOME"
for d in $protected_dirs; do
    if [ "$1" = "$d" ]; then
        rm=0
        break;
    fi
done
if [ ${rm:-1} -eq 1 ]; then
    rm -rf $1
fi

,

Add the following codes to your ~/.bashrc
# safe delete
move_to_trash () { now="$(date +%Y%m%d_%H%M%S)"; mv "$@" ~/.local/share/Trash/files/"$@_$now"; }
alias del='move_to_trash'

# safe rm
alias rmi='rm -i'

Every time you need to rm something, first consider del , you can change the trash folder. If you do need to rm something, you could go to the trash folder and use rmi .

One small bug for del is that when del a folder, for example, my_folder , it should be del my_folder but not del my_folder/ since in order for possible later restore, I attach the time information in the end ( "$@_$now" ). For files, it works fine.

[Jan 17, 2019] How do I launch the default web browser in Perl on any operating system

Jan 17, 2019 | stackoverflow.com

The second hit on "open url" at search.cpan brings up Browser::Open:

use Browser::Open qw( open_browser );

my $url = 'http://www.google.com/';
open_browser($url);

If your OS isn't supported, send a patch or a bug report.

--cjm

More at Stack Overflow More at Stack Overflow

[Jan 10, 2019] linux - How does cat EOF work in bash - Stack Overflow

Notable quotes:
"... The $sql variable now holds the new-line characters too. You can verify with echo -e "$sql" . ..."
"... The print.sh file now contains: ..."
"... The b.txt file contains bar and baz lines. The same output is printed to stdout . ..."
Jan 10, 2019 | stackoverflow.com

How does "cat << EOF" work in bash? Ask Question 454


hasen ,Mar 23, 2010 at 13:57

I needed to write a script to enter multi-line input to a program ( psql ).

After a bit of googling, I found the following syntax works:

cat << EOF | psql ---params
BEGIN;

`pg_dump ----something`

update table .... statement ...;

END;
EOF

This correctly constructs the multi-line string (from BEGIN; to END; , inclusive) and pipes it as an input to psql .

But I have no idea how/why it works, can some one please explain?

I'm referring mainly to cat << EOF , I know > outputs to a file, >> appends to a file, < reads input from file.

What does << exactly do?

And is there a man page for it?

Dennis Williamson ,Mar 23, 2010 at 18:28

That's probably a useless use of cat . Try psql ... << EOF ... See also "here strings". mywiki.wooledge.org/BashGuide/InputAndOutput?#Here_StringsDennis Williamson Mar 23 '10 at 18:28

hasen ,Mar 23, 2010 at 18:54

@Dennis: good point, and thanks for the link! – hasen Mar 23 '10 at 18:54

Alex ,Mar 23, 2015 at 23:31

I'm surprised it works with cat but not with echo. cat should expect a file name as stdin, not a char string. psql << EOF sounds logical, but not othewise. Works with cat but not with echo. Strange behaviour. Any clue about that? – Alex Mar 23 '15 at 23:31

Alex ,Mar 23, 2015 at 23:39

Answering to myself: cat without parameters executes and replicates to the output whatever send via input (stdin), hence using its output to fill the file via >. In fact a file name read as a parameter is not a stdin stream. – Alex Mar 23 '15 at 23:39

The-null-Pointer- ,Jan 1, 2018 at 18:03

@Alex echo just prints it's command line arguments while cat reads stding(when piped to it) or reads a file that corresponds to it's command line args – The-null-Pointer- Jan 1 '18 at 18:03

kennytm ,Mar 23, 2010 at 13:58

This is called heredoc format to provide a string into stdin. See https://en.wikipedia.org/wiki/Here_document#Unix_shells for more details.

From man bash :

Here Documents

This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen.

All of the lines read up to that point are then used as the standard input for a command.

The format of here-documents is:

          <<[-]word
                  here-document
          delimiter

No parameter expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word . If any characters in word are quoted, the delimiter is the result of quote removal on word , and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. In the latter case, the character sequence \<newline> is ignored, and \ must be used to quote the characters \ , $ , and ` .

If the redirection operator is <<- , then all leading tab characters are stripped from input lines and the line containing delimiter . This allows here-documents within shell scripts to be indented in a natural fashion.

Xeoncross ,May 26, 2011 at 22:51

I was having the hardest time disabling variable/parameter expansion. All I needed to do was use "double-quotes" and that fixed it! Thanks for the info! – Xeoncross May 26 '11 at 22:51

trkoch ,Nov 10, 2015 at 17:23

Concerning <<- please note that only leading tab characters are stripped -- not soft tab characters. This is one of those rare case when you actually need the tab character. If the rest of your document uses soft tabs, make sure to show invisible characters and (e.g.) copy and paste a tab character. If you do it right, your syntax highlighting should correctly catch the ending delimiter. – trkoch Nov 10 '15 at 17:23

BrDaHa ,Jul 13, 2017 at 19:01

I don't see how this answer is more helpful than the ones below. It merely regurgitates information that can be found in other places (that have likely already been checked) – BrDaHa Jul 13 '17 at 19:01

Vojtech Vitek ,Feb 4, 2014 at 10:28

The cat <<EOF syntax is very useful when working with multi-line text in Bash, eg. when assigning multi-line string to a shell variable, file or a pipe. Examples of cat <<EOF syntax usage in Bash: 1. Assign multi-line string to a shell variable
$ sql=$(cat <<EOF
SELECT foo, bar FROM db
WHERE foo='baz'
EOF
)

The $sql variable now holds the new-line characters too. You can verify with echo -e "$sql" .

2. Pass multi-line string to a file in Bash
$ cat <<EOF > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
EOF

The print.sh file now contains:

#!/bin/bash
echo $PWD
echo /home/user
3. Pass multi-line string to a pipe in Bash
$ cat <<EOF | grep 'b' | tee b.txt
foo
bar
baz
EOF

The b.txt file contains bar and baz lines. The same output is printed to stdout .

edelans ,Aug 22, 2014 at 8:48

In your case, "EOF" is known as a "Here Tag". Basically <<Here tells the shell that you are going to enter a multiline string until the "tag" Here . You can name this tag as you want, it's often EOF or STOP .

Some rules about the Here tags:

  1. The tag can be any string, uppercase or lowercase, though most people use uppercase by convention.
  2. The tag will not be considered as a Here tag if there are other words in that line. In this case, it will merely be considered part of the string. The tag should be by itself on a separate line, to be considered a tag.
  3. The tag should have no leading or trailing spaces in that line to be considered a tag. Otherwise it will be considered as part of the string.

example:

$ cat >> test <<HERE
> Hello world HERE <-- Not by itself on a separate line -> not considered end of string
> This is a test
>  HERE <-- Leading space, so not considered end of string
> and a new line
> HERE <-- Now we have the end of the string

oemb1905 ,Feb 22, 2017 at 7:17

this is the best actual answer ... you define both and clearly state the primary purpose of the use instead of related theory ... which is important but not necessary ... thanks - super helpful – oemb1905 Feb 22 '17 at 7:17

The-null-Pointer- ,Jan 1, 2018 at 18:05

@edelans you must add that when <<- is used leading tab will not prevent the tag from being recognized – The-null-Pointer- Jan 1 '18 at 18:05

JawSaw ,Oct 28, 2018 at 13:44

your answer clicked me on "you are going to enter a multiline string" – JawSaw Oct 28 '18 at 13:44

Ciro Santilli 新疆改造中心 六四事件 法轮功 ,Jun 9, 2015 at 9:41

POSIX 7

kennytm quoted man bash , but most of that is also POSIX 7: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04 :

The redirection operators "<<" and "<<-" both allow redirection of lines contained in a shell input file, known as a "here-document", to the input of a command.

The here-document shall be treated as a single word that begins after the next and continues until there is a line containing only the delimiter and a , with no characters in between. Then the next here-document starts, if there is one. The format is as follows:

[n]<<word
    here-document
delimiter

where the optional n represents the file descriptor number. If the number is omitted, the here-document refers to standard input (file descriptor 0).

If any character in word is quoted, the delimiter shall be formed by performing quote removal on word, and the here-document lines shall not be expanded. Otherwise, the delimiter shall be the word itself.

If no characters in word are quoted, all lines of the here-document shall be expanded for parameter expansion, command substitution, and arithmetic expansion. In this case, the in the input behaves as the inside double-quotes (see Double-Quotes). However, the double-quote character ( '"' ) shall not be treated specially within a here-document, except when the double-quote appears within "$()", "``", or "${}".

If the redirection symbol is "<<-", all leading <tab> characters shall be stripped from input lines and the line containing the trailing delimiter. If more than one "<<" or "<<-" operator is specified on a line, the here-document associated with the first operator shall be supplied first by the application and shall be read first by the shell.

When a here-document is read from a terminal device and the shell is interactive, it shall write the contents of the variable PS2, processed as described in Shell Variables, to standard error before reading each line of input until the delimiter has been recognized.

Examples

Some examples not yet given.

Quotes prevent parameter expansion

Without quotes:

a=0
cat <<EOF
$a
EOF

Output:

0

With quotes:

a=0
cat <<'EOF'
$a
EOF

or (ugly but valid):

a=0
cat <<E"O"F
$a
EOF

Outputs:

$a
Hyphen removes leading tabs

Without hyphen:

cat <<EOF
<tab>a
EOF

where <tab> is a literal tab, and can be inserted with Ctrl + V <tab>

Output:

<tab>a

With hyphen:

cat <<-EOF
<tab>a
<tab>EOF

Output:

a

This exists of course so that you can indent your cat like the surrounding code, which is easier to read and maintain. E.g.:

if true; then
    cat <<-EOF
    a
    EOF
fi

Unfortunately, this does not work for space characters: POSIX favored tab indentation here. Yikes.

David C. Rankin ,Aug 12, 2015 at 7:10

In your last example discussing <<- and <tab>a , it should be noted that the purpose was to allow normal indentation of code within the script while allowing heredoc text presented to the receiving process to begin in column 0. It is a not too commonly seen feature and a bit more context may prevent a good deal of head-scratching... – David C. Rankin Aug 12 '15 at 7:10

Ciro Santilli 新疆改造中心 六四事件 法轮功 ,Aug 12, 2015 at 8:22

@DavidC.Rankin updated to clarify that, thanks. – Ciro Santilli 新疆改造中心 六四事件 法轮功 Aug 12 '15 at 8:22

Jeanmichel Cote ,Sep 23, 2015 at 19:58

How should i escape expension if some of the content in between my EOF tags needs to be expanded and some don't? – Jeanmichel Cote Sep 23 '15 at 19:58

Jeanmichel Cote ,Sep 23, 2015 at 20:00

...just use the backslash in front of the $Jeanmichel Cote Sep 23 '15 at 20:00

Ciro Santilli 新疆改造中心 六四事件 法轮功 ,Sep 23, 2015 at 20:01

@JeanmichelCote I don't see a better option :-) With regular strings you can also consider mixing up quotes like "$a"'$b'"$c" , but there is no analogue here AFAIK. – Ciro Santilli 新疆改造中心 六四事件 法轮功 Sep 23 '15 at 20:01

Andreas Maier ,Feb 13, 2017 at 12:14

Using tee instead of cat

Not exactly as an answer to the original question, but I wanted to share this anyway: I had the need to create a config file in a directory that required root rights.

The following does not work for that case:

$ sudo cat <<EOF >/etc/somedir/foo.conf
# my config file
foo=bar
EOF

because the redirection is handled outside of the sudo context.

I ended up using this instead:

$ sudo tee <<EOF /etc/somedir/foo.conf >/dev/null
# my config file
foo=bar
EOF

user9048395

add a comment ,Jun 6, 2018 at 0:15
This isn't necessarily an answer to the original question, but a sharing of some results from my own testing. This:
<<test > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
test

will produce the same file as:

cat <<test > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
test

So, I don't see the point of using the cat command.

> ,Dec 19, 2013 at 21:40

Worth noting that here docs work in bash loops too. This example shows how-to get the column list of table:
export postgres_db_name='my_db'
export table_name='my_table_name'

# start copy 
while read -r c; do test -z "$c" || echo $table_name.$c , ; done < <(cat << EOF | psql -t -q -d $postgres_db_name -v table_name="${table_name:-}"
SELECT column_name
FROM information_schema.columns
WHERE 1=1
AND table_schema = 'public'
AND table_name   =:'table_name'  ;
EOF
)
# stop copy , now paste straight into the bash shell ...

output: 
my_table_name.guid ,
my_table_name.id ,
my_table_name.level ,
my_table_name.seq ,

or even without the new line

while read -r c; do test -z "$c" || echo $table_name.$c , | perl -ne 
's/\n//gm;print' ; done < <(cat << EOF | psql -t -q -d $postgres_db_name -v table_name="${table_name:-}"
 SELECT column_name
 FROM information_schema.columns
 WHERE 1=1
 AND table_schema = 'public'
 AND table_name   =:'table_name'  ;
 EOF
 )

 # output: daily_issues.guid ,daily_issues.id ,daily_issues.level ,daily_issues.seq ,daily_issues.prio ,daily_issues.weight ,daily_issues.status ,daily_issues.category ,daily_issues.name ,daily_issues.description ,daily_issues.type ,daily_issues.owner

[Jan 03, 2019] Using Lua for working with excel - Stack Overflow

Jan 03, 2019 | stackoverflow.com

Using Lua for working with excel Ask Question 2


Animesh ,Oct 14, 2009 at 12:04

I am planning to learn Lua for my desktop scripting needs. I want to know if there is any documentation available and also if there are all the things needed in the Standard Lib.

uroc ,Oct 14, 2009 at 12:09

You should check out Lua for Windows -- a 'batteries included environment' for the Lua scripting language on Windows

http://luaforwindows.luaforge.net/

It includes the LuaCOM library, from which you can access the Excel COM object.

Try looking at the LuaCOM documentation, there are some Excel examples in that:

http://www.tecgraf.puc-rio.br/~rcerq/luacom/pub/1.3/luacom-htmldoc/

I've only ever used this for very simplistic things. Here is a sample to get you started:

-- test.lua
require('luacom')
excel = luacom.CreateObject("Excel.Application")
excel.Visible = true
wb = excel.Workbooks:Add()
ws = wb.Worksheets(1)

for i=1, 20 do
    ws.Cells(i,1).Value2 = i
end

Animesh ,Oct 14, 2009 at 12:26

Thanks uroc for your quick response. If possible, please let me know of any beginner tutorial or atleast some sample code for using COM programming via Lua. :) – Animesh Oct 14 '09 at 12:26

sagasw ,Oct 16, 2009 at 1:02

More complex code example for lua working with excel:
require "luacom"

excel = luacom.CreateObject("Excel.Application")

local book  = excel.Workbooks:Add()
local sheet = book.Worksheets(1)

excel.Visible = true

for row=1, 30 do
  for col=1, 30 do
    sheet.Cells(row, col).Value2 = math.floor(math.random() * 100)
  end
end


local range = sheet:Range("A1")

for row=1, 30 do
  for col=1, 30 do
    local v = sheet.Cells(row, col).Value2

    if v > 50 then
        local cell = range:Offset(row-1, col-1)

        cell:Select()
        excel.Selection.Interior.Color = 65535
    end
  end
end

excel.DisplayAlerts = false
excel:Quit()
excel = nil

Another example, could add a graph chart.

require "luacom"

excel = luacom.CreateObject("Excel.Application")

local book  = excel.Workbooks:Add()
local sheet = book.Worksheets(1)

excel.Visible = true

for row=1, 30 do
  sheet.Cells(row, 1).Value2 = math.floor(math.random() * 100)
end

local chart = excel.Charts:Add()
chart.ChartType = 4  --  xlLine

local range = sheet:Range("A1:A30")
chart:SetSourceData(range)

Incredulous Monk ,Oct 19, 2009 at 4:17

A quick suggestion: fragments of code will look better if you format them as code (use the little "101 010" button). – Incredulous Monk Oct 19 '09 at 4:17

[Jan 01, 2019] mc - How can I set the default (user defined) listing mode in Midnight Commander- - Unix Linux Stack Exchange

Jan 01, 2019 | unix.stackexchange.com

Ask Question 0

papaiatis ,Jul 14, 2016 at 11:51

I defined my own listing mode and I'd like to make it permanent so that on the next mc start my defined listing mode will be set. I found no configuration file for mc.

,

You have probably Auto save setup turned off in Options->Configuration menu.

You can save the configuration manually by Options->Save setup .

Panels setup is saved to ~/.config/mc/panels.ini .

[Dec 05, 2018] How to make putty ssh connection never to timeout when user is idle?

Dec 05, 2018 | askubuntu.com

David MZ ,Feb 13, 2013 at 18:07

I have a Ubuntu 12.04 server I bought, if I connect with putty using ssh and a sudoer user putty gets disconnected by the server after some time if I am idle How do I configure Ubuntu to keep this connection alive indefinitely?

das Keks ,Feb 13, 2013 at 18:24

If you go to your putty settings -> Connection and set the value of "Seconds between keepalives" to 30 seconds this should solve your problem.

kokbira ,Feb 19 at 11:42

?????? "0 to turn off" or 30 to turn off????????? I think he must put 0 instead of 30! – kokbira Feb 19 at 11:42

das Keks ,Feb 19 at 11:46

No, it's the time between keepalives. If you set it to 0, no keepalives are sent but you want putty to send keepalives to keep the connection alive. – das Keks Feb 19 at 11:46

Aaron ,Mar 19 at 20:39

I did this but still it drops.. – Aaron Mar 19 at 20:39

0xC0000022L ,Feb 13, 2013 at 19:29

In addition to the answer from "das Keks" there is at least one other aspect that can affect this behavior. Bash (usually the default shell on Ubuntu) has a value TMOUT which governs (decimal value in seconds) after which time an idle shell session will time out and the user will be logged out, leading to a disconnect in an SSH session.

In addition I would strongly recommend that you do something else entirely. Set up byobu (or even just tmux alone as it's superior to GNU screen ) and always log in and attach to a preexisting session (that's GNU screen and tmux terminology). This way even if you get forcibly disconnected - let's face it, a power outage or network interruption can always happen - you can always resume your work where you left. And that works across different machines. So you can connect to the same session from another machine (e.g. from home). The possibilities are manifold and it's a true productivity booster. And not to forget, terminal multiplexers overcome one of the big disadvantages of PuTTY: no tabbed interface. Now you get "tabs" in the form of windows and panes inside GNU screen and tmux .

apt-get install tmux
apt-get install byobu

Byobu is a nice frontend to both terminal multiplexers, but tmux is so comfortable that in my opinion it obsoletes byobu to a large extent. So my recommendation would be tmux .

Also search for "dotfiles", in particular tmux.conf and .tmux.conf on the web for many good customizations to get you started.

Rajesh ,Mar 19, 2015 at 15:10

Go to PuTTy options --> Connection
  1. Change the default value for "Seconds between keepalives(0 to turn off)" : from 0 to 600 (10 minutes) --This varies...reduce if 10 minutes doesn't help
  2. Check the "Enable TCP_keepalives (SO_KEEPALIVE option)" check box.
  3. Finally save setting for session

,

I keep my PuTTY sessions alive by monitoring the cron logs
tail -f /var/log/cron

I want the PuTTY session alive because I'm proxying through socks.

[Dec 05, 2018] How can I scroll up to see the past output in PuTTY?

Dec 05, 2018 | superuser.com

Ask Question up vote 3 down vote favorite 1

user1721949 ,Dec 12, 2012 at 8:32

I have a script which, when I run it from PuTTY, it scrolls the screen. Now, I want to go back to see the errors, but when I scroll up, I can see the past commands, but not the output of the command.

How can I see the past output?

Rico ,Dec 13, 2012 at 8:24

Shift+Pgup/PgDn should work for scrolling without using the scrollbar.

> ,Jul 12, 2017 at 21:45

If shift pageup/pagedown fails, try this command: "reset", which seems to correct the display. – user530079 Jul 12 '17 at 21:45

RedGrittyBrick ,Dec 12, 2012 at 9:31

If you don't pipe the output of your commands into something like less , you will be able to use Putty's scroll-bars to view earlier output.

Putty has settings for how many lines of past output it retains in it's buffer.


before scrolling

after scrolling back (upwards)

If you use something like less the output doesn't get into Putty's scroll buffer


after using less

David Dai ,Dec 14, 2012 at 3:31

why is putty different with the native linux console at this point? – David Dai Dec 14 '12 at 3:31

konradstrack ,Dec 12, 2012 at 9:52

I would recommend using screen if you want to have good control over the scroll buffer on a remote shell.

You can change the scroll buffer size to suit your needs by setting:

defscrollback 4000

in ~/.screenrc , which will specify the number of lines you want to be buffered (4000 in this case).

Then you should run your script in a screen session, e.g. by executing screen ./myscript.sh or first executing screen and then ./myscript.sh inside the session.

It's also possible to enable logging of the console output to a file. You can find more info on the screen's man page .

,

From your descript, it sounds like the "problem" is that you are using screen, tmux, or another window manager dependent on them (byobu). Normally you should be able to scroll back in putty with no issue. Exceptions include if you are in an application like less or nano that creates it's own "window" on the terminal.

With screen and tmux you can generally scroll back with SHIFT + PGUP (same as you could from the physical terminal of the remote machine). They also both have a "copy" mode that frees the cursor from the prompt and lets you use arrow keys to move it around (for selecting text to copy with just the keyboard). It also lets you scroll up and down with the PGUP and PGDN keys. Copy mode under byobu using screen or tmux backends is accessed by pressing F7 (careful, F6 disconnects the session). To do so directly under screen you press CTRL + a then ESC or [ . You can use ESC to exit copy mode. Under tmux you press CTRL + b then [ to enter copy mode and ] to exit.

The simplest solution, of course, is not to use either. I've found both to be quite a bit more trouble than they are worth. If you would like to use multiple different terminals on a remote machine simply connect with multiple instances of putty and manage your windows using, er... Windows. Now forgive me but I must flee before I am burned at the stake for my heresy.

EDIT: almost forgot, some keys may not be received correctly by the remote terminal if putty has not been configured correctly. In your putty config check Terminal -> Keyboard . You probably want the function keys and keypad set to be either Linux or Xterm R6 . If you are seeing strange characters on the terminal when attempting the above this is most likely the problem.

[Dec 01, 2018] Lua editors WoWWiki FANDOM powered by Wikia

Dec 01, 2018 | wowwiki.wikia.com

[Nov 13, 2018] Resuming rsync partial (-P/--partial) on a interrupted transfer

Notable quotes:
"... should ..."
May 15, 2013 | stackoverflow.com

Glitches , May 15, 2013 at 18:06

I am trying to backup my file server to a remove file server using rsync. Rsync is not successfully resuming when a transfer is interrupted. I used the partial option but rsync doesn't find the file it already started because it renames it to a temporary file and when resumed it creates a new file and starts from beginning.

Here is my command:

rsync -avztP -e "ssh -p 2222" /volume1/ myaccont@backup-server-1:/home/myaccount/backup/ --exclude "@spool" --exclude "@tmp"

When this command is ran, a backup file named OldDisk.dmg from my local machine get created on the remote machine as something like .OldDisk.dmg.SjDndj23 .

Now when the internet connection gets interrupted and I have to resume the transfer, I have to find where rsync left off by finding the temp file like .OldDisk.dmg.SjDndj23 and rename it to OldDisk.dmg so that it sees there already exists a file that it can resume.

How do I fix this so I don't have to manually intervene each time?

Richard Michael , Nov 6, 2013 at 4:26

TL;DR : Use --timeout=X (X in seconds) to change the default rsync server timeout, not --inplace .

The issue is the rsync server processes (of which there are two, see rsync --server ... in ps output on the receiver) continue running, to wait for the rsync client to send data.

If the rsync server processes do not receive data for a sufficient time, they will indeed timeout, self-terminate and cleanup by moving the temporary file to it's "proper" name (e.g., no temporary suffix). You'll then be able to resume.

If you don't want to wait for the long default timeout to cause the rsync server to self-terminate, then when your internet connection returns, log into the server and clean up the rsync server processes manually. However, you must politely terminate rsync -- otherwise, it will not move the partial file into place; but rather, delete it (and thus there is no file to resume). To politely ask rsync to terminate, do not SIGKILL (e.g., -9 ), but SIGTERM (e.g., pkill -TERM -x rsync - only an example, you should take care to match only the rsync processes concerned with your client).

Fortunately there is an easier way: use the --timeout=X (X in seconds) option; it is passed to the rsync server processes as well.

For example, if you specify rsync ... --timeout=15 ... , both the client and server rsync processes will cleanly exit if they do not send/receive data in 15 seconds. On the server, this means moving the temporary file into position, ready for resuming.

I'm not sure of the default timeout value of the various rsync processes will try to send/receive data before they die (it might vary with operating system). In my testing, the server rsync processes remain running longer than the local client. On a "dead" network connection, the client terminates with a broken pipe (e.g., no network socket) after about 30 seconds; you could experiment or review the source code. Meaning, you could try to "ride out" the bad internet connection for 15-20 seconds.

If you do not clean up the server rsync processes (or wait for them to die), but instead immediately launch another rsync client process, two additional server processes will launch (for the other end of your new client process). Specifically, the new rsync client will not re-use/reconnect to the existing rsync server processes. Thus, you'll have two temporary files (and four rsync server processes) -- though, only the newer, second temporary file has new data being written (received from your new rsync client process).

Interestingly, if you then clean up all rsync server processes (for example, stop your client which will stop the new rsync servers, then SIGTERM the older rsync servers, it appears to merge (assemble) all the partial files into the new proper named file. So, imagine a long running partial copy which dies (and you think you've "lost" all the copied data), and a short running re-launched rsync (oops!).. you can stop the second client, SIGTERM the first servers, it will merge the data, and you can resume.

Finally, a few short remarks:

JamesTheAwesomeDude , Dec 29, 2013 at 16:50

Just curious: wouldn't SIGINT (aka ^C ) be 'politer' than SIGTERM ? – JamesTheAwesomeDude Dec 29 '13 at 16:50

Richard Michael , Dec 29, 2013 at 22:34

I didn't test how the server-side rsync handles SIGINT, so I'm not sure it will keep the partial file - you could check. Note that this doesn't have much to do with Ctrl-c ; it happens that your terminal sends SIGINT to the foreground process when you press Ctrl-c , but the server-side rsync has no controlling terminal. You must log in to the server and use kill . The client-side rsync will not send a message to the server (for example, after the client receives SIGINT via your terminal Ctrl-c ) - might be interesting though. As for anthropomorphizing, not sure what's "politer". :-) – Richard Michael Dec 29 '13 at 22:34

d-b , Feb 3, 2015 at 8:48

I just tried this timeout argument rsync -av --delete --progress --stats --human-readable --checksum --timeout=60 --partial-dir /tmp/rsync/ rsync://$remote:/ /src/ but then it timed out during the "receiving file list" phase (which in this case takes around 30 minutes). Setting the timeout to half an hour so kind of defers the purpose. Any workaround for this? – d-b Feb 3 '15 at 8:48

Cees Timmerman , Sep 15, 2015 at 17:10

@user23122 --checksum reads all data when preparing the file list, which is great for many small files that change often, but should be done on-demand for large files. – Cees Timmerman Sep 15 '15 at 17:10

[Nov 08, 2018] How to find which process is regularly writing to disk?

Notable quotes:
"... tick...tick...tick...trrrrrr ..."
"... /var/log/syslog ..."
Nov 08, 2018 | unix.stackexchange.com

Cedric Martin , Jul 27, 2012 at 4:31

How can I find which process is constantly writing to disk?

I like my workstation to be close to silent and I just build a new system (P8B75-M + Core i5 3450s -- the 's' because it has a lower max TDP) with quiet fans etc. and installed Debian Wheezy 64-bit on it.

And something is getting on my nerve: I can hear some kind of pattern like if the hard disk was writing or seeking someting ( tick...tick...tick...trrrrrr rinse and repeat every second or so).

In the past I had a similar issue in the past (many, many years ago) and it turned out it was some CUPS log or something and I simply redirected that one (not important) logging to a (real) RAM disk.

But here I'm not sure.

I tried the following:

ls -lR /var/log > /tmp/a.tmp && sleep 5 && ls -lR /var/log > /tmp/b.tmp && diff /tmp/?.tmp

but nothing is changing there.

Now the strange thing is that I also hear the pattern when the prompt asking me to enter my LVM decryption passphrase is showing.

Could it be something in the kernel/system I just installed or do I have a faulty harddisk?

hdparm -tT /dev/sda report a correct HD speed (130 GB/s non-cached, sata 6GB) and I've already installed and compiled from big sources (Emacs) without issue so I don't think the system is bad.

(HD is a Seagate Barracude 500GB)

Mat , Jul 27, 2012 at 6:03

Are you sure it's a hard drive making that noise, and not something else? (Check the fans, including PSU fan. Had very strange clicking noises once when a very thin cable was too close to a fan and would sometimes very slightly touch the blades and bounce for a few "clicks"...) – Mat Jul 27 '12 at 6:03

Cedric Martin , Jul 27, 2012 at 7:02

@Mat: I'll take the hard drive outside of the case (the connectors should be long enough) to be sure and I'll report back ; ) – Cedric Martin Jul 27 '12 at 7:02

camh , Jul 27, 2012 at 9:48

Make sure your disk filesystems are mounted relatime or noatime. File reads can be causing writes to inodes to record the access time. – camh Jul 27 '12 at 9:48

mnmnc , Jul 27, 2012 at 8:27

Did you tried to examin what programs like iotop is showing? It will tell you exacly what kind of process is currently writing to the disk.

example output:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
 1033 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [flush-8:0]
   10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]

Cedric Martin , Aug 2, 2012 at 15:56

thanks for that tip. I didn't know about iotop . On Debian I did an apt-cache search iotop to find out that I had to apt-get iotop . Very cool command! – Cedric Martin Aug 2 '12 at 15:56

ndemou , Jun 20, 2016 at 15:32

I use iotop -o -b -d 10 which every 10secs prints a list of processes that read/wrote to disk and the amount of IO bandwidth used. – ndemou Jun 20 '16 at 15:32

scai , Jul 27, 2012 at 10:48

You can enable IO debugging via echo 1 > /proc/sys/vm/block_dump and then watch the debugging messages in /var/log/syslog . This has the advantage of obtaining some type of log file with past activities whereas iotop only shows the current activity.

dan3 , Jul 15, 2013 at 8:32

It is absolutely crazy to leave sysloging enabled when block_dump is active. Logging causes disk activity, which causes logging, which causes disk activity etc. Better stop syslog before enabling this (and use dmesg to read the messages) – dan3 Jul 15 '13 at 8:32

scai , Jul 16, 2013 at 6:32

You are absolutely right, although the effect isn't as dramatic as you describe it. If you just want to have a short peek at the disk activity there is no need to stop the syslog daemon. – scai Jul 16 '13 at 6:32

dan3 , Jul 16, 2013 at 7:22

I've tried it about 2 years ago and it brought my machine to a halt. One of these days when I have nothing important running I'll try it again :) – dan3 Jul 16 '13 at 7:22

scai , Jul 16, 2013 at 10:50

I tried it, nothing really happened. Especially because of file system buffering. A write to syslog doesn't immediately trigger a write to disk. – scai Jul 16 '13 at 10:50

Volker Siegel , Apr 16, 2014 at 22:57

I would assume there is rate general rate limiting in place for the log messages, which handles this case too(?) – Volker Siegel Apr 16 '14 at 22:57

Gilles , Jul 28, 2012 at 1:34

Assuming that the disk noises are due to a process causing a write and not to some disk spindown problem , you can use the audit subsystem (install the auditd package ). Put a watch on the sync calls and its friends:
auditctl -S sync -S fsync -S fdatasync -a exit,always

Watch the logs in /var/log/audit/audit.log . Be careful not to do this if the audit logs themselves are flushed! Check in /etc/auditd.conf that the flush option is set to none .

If files are being flushed often, a likely culprit is the system logs. For example, if you log failed incoming connection attempts and someone is probing your machine, that will generate a lot of entries; this can cause a disk to emit machine gun-style noises. With the basic log daemon sysklogd, check /etc/syslog.conf : if a log file name is not be preceded by - , then that log is flushed to disk after each write.

Gilles , Mar 23 at 18:24

@StephenKitt Huh. No. The asker mentioned Debian so I've changed it to a link to the Debian package. – Gilles Mar 23 at 18:24

cas , Jul 27, 2012 at 9:40

It might be your drives automatically spinning down, lots of consumer-grade drives do that these days. Unfortunately on even a lightly loaded system, this results in the drives constantly spinning down and then spinning up again, especially if you're running hddtemp or similar to monitor the drive temperature (most drives stupidly don't let you query the SMART temperature value without spinning up the drive - cretinous!).

This is not only annoying, it can wear out the drives faster as many drives have only a limited number of park cycles. e.g. see https://bugs.launchpad.net/ubuntu/+source/hdparm/+bug/952556 for a description of the problem.

I disable idle-spindown on all my drives with the following bit of shell code. you could put it in an /etc/rc.boot script, or in /etc/rc.local or similar.

for disk in /dev/sd? ; do
  /sbin/hdparm -q -S 0 "/dev/$disk"
done

Cedric Martin , Aug 2, 2012 at 16:03

that you can't query SMART readings without spinning up the drive leaves me speechless :-/ Now obviously the "spinning down" issue can become quite complicated. Regarding disabling the spinning down: wouldn't that in itself cause the HD to wear out faster? I mean: it's never ever "resting" as long as the system is on then? – Cedric Martin Aug 2 '12 at 16:03

cas , Aug 2, 2012 at 21:42

IIRC you can query some SMART values without causing the drive to spin up, but temperature isn't one of them on any of the drives i've tested (incl models from WD, Seagate, Samsung, Hitachi). Which is, of course, crazy because concern over temperature is one of the reasons for idling a drive. re: wear: AIUI 1. constant velocity is less wearing than changing speed. 2. the drives have to park the heads in a safe area and a drive is only rated to do that so many times (IIRC up to a few hundred thousand - easily exceeded if the drive is idling and spinning up every few seconds) – cas Aug 2 '12 at 21:42

Micheal Johnson , Mar 12, 2016 at 20:48

It's a long debate regarding whether it's better to leave drives running or to spin them down. Personally I believe it's best to leave them running - I turn my computer off at night and when I go out but other than that I never spin my drives down. Some people prefer to spin them down, say, at night if they're leaving the computer on or if the computer's idle for a long time, and in such cases the advantage of spinning them down for a few hours versus leaving them running is debatable. What's never good though is when the hard drive repeatedly spins down and up again in a short period of time. – Micheal Johnson Mar 12 '16 at 20:48

Micheal Johnson , Mar 12, 2016 at 20:51

Note also that spinning the drive down after it's been idle for a few hours is a bit silly, because if it's been idle for a few hours then it's likely to be used again within an hour. In that case, it would seem better to spin the drive down promptly if it's idle (like, within 10 minutes), but it's also possible for the drive to be idle for a few minutes when someone is using the computer and is likely to need the drive again soon. – Micheal Johnson Mar 12 '16 at 20:51

,

I just found that s.m.a.r.t was causing an external USB disk to spin up again and again on my raspberry pi. Although SMART is generally a good thing, I decided to disable it again and since then it seems that unwanted disk activity has stopped

[Nov 08, 2018] Determining what process is bound to a port

Mar 14, 2011 | unix.stackexchange.com
I know that using the command:
lsof -i TCP

(or some variant of parameters with lsof) I can determine which process is bound to a particular port. This is useful say if I'm trying to start something that wants to bind to 8080 and some else is already using that port, but I don't know what.

Is there an easy way to do this without using lsof? I spend time working on many systems and lsof is often not installed.

Cakemox , Mar 14, 2011 at 20:48

netstat -lnp will list the pid and process name next to each listening port. This will work under Linux, but not all others (like AIX.) Add -t if you want TCP only.
# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:24800           0.0.0.0:*               LISTEN      27899/synergys
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      3361/python
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      2264/mysqld
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      22964/apache2
tcp        0      0 192.168.99.1:53         0.0.0.0:*               LISTEN      3389/named
tcp        0      0 192.168.88.1:53         0.0.0.0:*               LISTEN      3389/named

etc.

xxx , Mar 14, 2011 at 21:01

Cool, thanks. Looks like that that works under RHEL, but not under Solaris (as you indicated). Anybody know if there's something similar for Solaris? – user5721 Mar 14 '11 at 21:01

Rich Homolka , Mar 15, 2011 at 19:56

netstat -p above is my vote. also look at lsof . – Rich Homolka Mar 15 '11 at 19:56

Jonathan , Aug 26, 2014 at 18:50

As an aside, for windows it's similar: netstat -aon | more – Jonathan Aug 26 '14 at 18:50

sudo , May 25, 2017 at 2:24

What about for SCTP? – sudo May 25 '17 at 2:24

frielp , Mar 15, 2011 at 13:33

On AIX, netstat & rmsock can be used to determine process binding:
[root@aix] netstat -Ana|grep LISTEN|grep 80
f100070000280bb0 tcp4       0      0  *.37               *.*        LISTEN
f1000700025de3b0 tcp        0      0  *.80               *.*        LISTEN
f1000700002803b0 tcp4       0      0  *.111              *.*        LISTEN
f1000700021b33b0 tcp4       0      0  127.0.0.1.32780    *.*        LISTEN

# Port 80 maps to f1000700025de3b0 above, so we type:
[root@aix] rmsock f1000700025de3b0 tcpcb
The socket 0x25de008 is being held by process 499790 (java).

Olivier Dulac , Sep 18, 2013 at 4:05

Thanks for this! Is there a way, however, to just display what process listen on the socket (instead of using rmsock which attempt to remove it) ? – Olivier Dulac Sep 18 '13 at 4:05

Vitor Py , Sep 26, 2013 at 14:18

@OlivierDulac: "Unlike what its name implies, rmsock does not remove the socket, if it is being used by a process. It just reports the process holding the socket." ( ibm.com/developerworks/community/blogs/cgaix/entry/ ) – Vitor Py Sep 26 '13 at 14:18

Olivier Dulac , Sep 26, 2013 at 16:00

@vitor-braga: Ah thx! I thought it was trying but just said which process holds in when it couldn't remove it. Apparently it doesn't even try to remove it when a process holds it. That's cool! Thx! – Olivier Dulac Sep 26 '13 at 16:00

frielp , Mar 15, 2011 at 13:27

Another tool available on Linux is ss . From the ss man page on Fedora:
NAME
       ss - another utility to investigate sockets
SYNOPSIS
       ss [options] [ FILTER ]
DESCRIPTION
       ss is used to dump socket statistics. It allows showing information 
       similar to netstat. It can display more TCP and state informations  
       than other tools.

Example output below - the final column shows the process binding:

[root@box] ss -ap
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
LISTEN     0      128                    :::http                    :::*        users:(("httpd",20891,4),("httpd",20894,4),("httpd",20895,4),("httpd",20896,4)
LISTEN     0      128             127.0.0.1:munin                    *:*        users:(("munin-node",1278,5))
LISTEN     0      128                    :::ssh                     :::*        users:(("sshd",1175,4))
LISTEN     0      128                     *:ssh                      *:*        users:(("sshd",1175,3))
LISTEN     0      10              127.0.0.1:smtp                     *:*        users:(("sendmail",1199,4))
LISTEN     0      128             127.0.0.1:x11-ssh-offset                  *:*        users:(("sshd",25734,8))
LISTEN     0      128                   ::1:x11-ssh-offset                 :::*        users:(("sshd",25734,7))

Eugen Constantin Dinca , Mar 14, 2011 at 23:47

For Solaris you can use pfiles and then grep by sockname: or port: .

A sample (from here ):

pfiles `ptree | awk '{print $1}'` | egrep '^[0-9]|port:'

rickumali , May 8, 2011 at 14:40

I was once faced with trying to determine what process was behind a particular port (this time it was 8000). I tried a variety of lsof and netstat, but then took a chance and tried hitting the port via a browser (i.e. http://hostname:8000/ ). Lo and behold, a splash screen greeted me, and it became obvious what the process was (for the record, it was Splunk ).

One more thought: "ps -e -o pid,args" (YMMV) may sometimes show the port number in the arguments list. Grep is your friend!

Gilles , Oct 8, 2015 at 21:04

In the same vein, you could telnet hostname 8000 and see if the server prints a banner. However, that's mostly useful when the server is running on a machine where you don't have shell access, and then finding the process ID isn't relevant. – Gilles May 8 '11 at 14:45

[Nov 08, 2018] How to find which process is regularly writing to disk?

Notable quotes:
"... tick...tick...tick...trrrrrr ..."
"... /var/log/syslog ..."
Jul 27, 2012 | unix.stackexchange.com

Cedric Martin , Jul 27, 2012 at 4:31

How can I find which process is constantly writing to disk?

I like my workstation to be close to silent and I just build a new system (P8B75-M + Core i5 3450s -- the 's' because it has a lower max TDP) with quiet fans etc. and installed Debian Wheezy 64-bit on it.

And something is getting on my nerve: I can hear some kind of pattern like if the hard disk was writing or seeking someting ( tick...tick...tick...trrrrrr rinse and repeat every second or so).

In the past I had a similar issue in the past (many, many years ago) and it turned out it was some CUPS log or something and I simply redirected that one (not important) logging to a (real) RAM disk.

But here I'm not sure.

I tried the following:

ls -lR /var/log > /tmp/a.tmp && sleep 5 && ls -lR /var/log > /tmp/b.tmp && diff /tmp/?.tmp

but nothing is changing there.

Now the strange thing is that I also hear the pattern when the prompt asking me to enter my LVM decryption passphrase is showing.

Could it be something in the kernel/system I just installed or do I have a faulty harddisk?

hdparm -tT /dev/sda report a correct HD speed (130 GB/s non-cached, sata 6GB) and I've already installed and compiled from big sources (Emacs) without issue so I don't think the system is bad.

(HD is a Seagate Barracude 500GB)

Mat , Jul 27, 2012 at 6:03

Are you sure it's a hard drive making that noise, and not something else? (Check the fans, including PSU fan. Had very strange clicking noises once when a very thin cable was too close to a fan and would sometimes very slightly touch the blades and bounce for a few "clicks"...) – Mat Jul 27 '12 at 6:03

Cedric Martin , Jul 27, 2012 at 7:02

@Mat: I'll take the hard drive outside of the case (the connectors should be long enough) to be sure and I'll report back ; ) – Cedric Martin Jul 27 '12 at 7:02

camh , Jul 27, 2012 at 9:48

Make sure your disk filesystems are mounted relatime or noatime. File reads can be causing writes to inodes to record the access time. – camh Jul 27 '12 at 9:48

mnmnc , Jul 27, 2012 at 8:27

Did you tried to examin what programs like iotop is showing? It will tell you exacly what kind of process is currently writing to the disk.

example output:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
 1033 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [flush-8:0]
   10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]

Cedric Martin , Aug 2, 2012 at 15:56

thanks for that tip. I didn't know about iotop . On Debian I did an apt-cache search iotop to find out that I had to apt-get iotop . Very cool command! – Cedric Martin Aug 2 '12 at 15:56

ndemou , Jun 20, 2016 at 15:32

I use iotop -o -b -d 10 which every 10secs prints a list of processes that read/wrote to disk and the amount of IO bandwidth used. – ndemou Jun 20 '16 at 15:32

scai , Jul 27, 2012 at 10:48

You can enable IO debugging via echo 1 > /proc/sys/vm/block_dump and then watch the debugging messages in /var/log/syslog . This has the advantage of obtaining some type of log file with past activities whereas iotop only shows the current activity.

dan3 , Jul 15, 2013 at 8:32

It is absolutely crazy to leave sysloging enabled when block_dump is active. Logging causes disk activity, which causes logging, which causes disk activity etc. Better stop syslog before enabling this (and use dmesg to read the messages) – dan3 Jul 15 '13 at 8:32

scai , Jul 16, 2013 at 6:32

You are absolutely right, although the effect isn't as dramatic as you describe it. If you just want to have a short peek at the disk activity there is no need to stop the syslog daemon. – scai Jul 16 '13 at 6:32

dan3 , Jul 16, 2013 at 7:22

I've tried it about 2 years ago and it brought my machine to a halt. One of these days when I have nothing important running I'll try it again :) – dan3 Jul 16 '13 at 7:22

scai , Jul 16, 2013 at 10:50

I tried it, nothing really happened. Especially because of file system buffering. A write to syslog doesn't immediately trigger a write to disk. – scai Jul 16 '13 at 10:50

Volker Siegel , Apr 16, 2014 at 22:57

I would assume there is rate general rate limiting in place for the log messages, which handles this case too(?) – Volker Siegel Apr 16 '14 at 22:57

Gilles , Jul 28, 2012 at 1:34

Assuming that the disk noises are due to a process causing a write and not to some disk spindown problem , you can use the audit subsystem (install the auditd package ). Put a watch on the sync calls and its friends:
auditctl -S sync -S fsync -S fdatasync -a exit,always

Watch the logs in /var/log/audit/audit.log . Be careful not to do this if the audit logs themselves are flushed! Check in /etc/auditd.conf that the flush option is set to none .

If files are being flushed often, a likely culprit is the system logs. For example, if you log failed incoming connection attempts and someone is probing your machine, that will generate a lot of entries; this can cause a disk to emit machine gun-style noises. With the basic log daemon sysklogd, check /etc/syslog.conf : if a log file name is not be preceded by - , then that log is flushed to disk after each write.

Gilles , Mar 23 at 18:24

@StephenKitt Huh. No. The asker mentioned Debian so I've changed it to a link to the Debian package. – Gilles Mar 23 at 18:24

cas , Jul 27, 2012 at 9:40

It might be your drives automatically spinning down, lots of consumer-grade drives do that these days. Unfortunately on even a lightly loaded system, this results in the drives constantly spinning down and then spinning up again, especially if you're running hddtemp or similar to monitor the drive temperature (most drives stupidly don't let you query the SMART temperature value without spinning up the drive - cretinous!).

This is not only annoying, it can wear out the drives faster as many drives have only a limited number of park cycles. e.g. see https://bugs.launchpad.net/ubuntu/+source/hdparm/+bug/952556 for a description of the problem.

I disable idle-spindown on all my drives with the following bit of shell code. you could put it in an /etc/rc.boot script, or in /etc/rc.local or similar.

for disk in /dev/sd? ; do
  /sbin/hdparm -q -S 0 "/dev/$disk"
done

Cedric Martin , Aug 2, 2012 at 16:03

that you can't query SMART readings without spinning up the drive leaves me speechless :-/ Now obviously the "spinning down" issue can become quite complicated. Regarding disabling the spinning down: wouldn't that in itself cause the HD to wear out faster? I mean: it's never ever "resting" as long as the system is on then? – Cedric Martin Aug 2 '12 at 16:03

cas , Aug 2, 2012 at 21:42

IIRC you can query some SMART values without causing the drive to spin up, but temperature isn't one of them on any of the drives i've tested (incl models from WD, Seagate, Samsung, Hitachi). Which is, of course, crazy because concern over temperature is one of the reasons for idling a drive. re: wear: AIUI 1. constant velocity is less wearing than changing speed. 2. the drives have to park the heads in a safe area and a drive is only rated to do that so many times (IIRC up to a few hundred thousand - easily exceeded if the drive is idling and spinning up every few seconds) – cas Aug 2 '12 at 21:42

Micheal Johnson , Mar 12, 2016 at 20:48

It's a long debate regarding whether it's better to leave drives running or to spin them down. Personally I believe it's best to leave them running - I turn my computer off at night and when I go out but other than that I never spin my drives down. Some people prefer to spin them down, say, at night if they're leaving the computer on or if the computer's idle for a long time, and in such cases the advantage of spinning them down for a few hours versus leaving them running is debatable. What's never good though is when the hard drive repeatedly spins down and up again in a short period of time. – Micheal Johnson Mar 12 '16 at 20:48

Micheal Johnson , Mar 12, 2016 at 20:51

Note also that spinning the drive down after it's been idle for a few hours is a bit silly, because if it's been idle for a few hours then it's likely to be used again within an hour. In that case, it would seem better to spin the drive down promptly if it's idle (like, within 10 minutes), but it's also possible for the drive to be idle for a few minutes when someone is using the computer and is likely to need the drive again soon. – Micheal Johnson Mar 12 '16 at 20:51

,

I just found that s.m.a.r.t was causing an external USB disk to spin up again and again on my raspberry pi. Although SMART is generally a good thing, I decided to disable it again and since then it seems that unwanted disk activity has stopped

[Nov 08, 2018] How to split one string into multiple variables in bash shell? [duplicate]

Nov 08, 2018 | stackoverflow.com
This question already has an answer here:

Rob I , May 9, 2012 at 19:22

For your second question, see @mkb's comment to my answer below - that's definitely the way to go! – Rob I May 9 '12 at 19:22

Dennis Williamson , Jul 4, 2012 at 16:14

See my edited answer for one way to read individual characters into an array. – Dennis Williamson Jul 4 '12 at 16:14

Nick Weedon , Dec 31, 2015 at 11:04

Here is the same thing in a more concise form: var1=$(cut -f1 -d- <<<$STR) – Nick Weedon Dec 31 '15 at 11:04

Rob I , May 9, 2012 at 17:00

If your solution doesn't have to be general, i.e. only needs to work for strings like your example, you could do:
var1=$(echo $STR | cut -f1 -d-)
var2=$(echo $STR | cut -f2 -d-)

I chose cut here because you could simply extend the code for a few more variables...

crunchybutternut , May 9, 2012 at 17:40

Can you look at my post again and see if you have a solution for the followup question? thanks! – crunchybutternut May 9 '12 at 17:40

mkb , May 9, 2012 at 17:59

You can use cut to cut characters too! cut -c1 for example. – mkb May 9 '12 at 17:59

FSp , Nov 27, 2012 at 10:26

Although this is very simple to read and write, is a very slow solution because forces you to read twice the same data ($STR) ... if you care of your script performace, the @anubhava solution is much better – FSp Nov 27 '12 at 10:26

tripleee , Jan 25, 2016 at 6:47

Apart from being an ugly last-resort solution, this has a bug: You should absolutely use double quotes in echo "$STR" unless you specifically want the shell to expand any wildcards in the string as a side effect. See also stackoverflow.com/questions/10067266/tripleee Jan 25 '16 at 6:47

Rob I , Feb 10, 2016 at 13:57

You're right about double quotes of course, though I did point out this solution wasn't general. However I think your assessment is a bit unfair - for some people this solution may be more readable (and hence extensible etc) than some others, and doesn't completely rely on arcane bash feature that wouldn't translate to other shells. I suspect that's why my solution, though less elegant, continues to get votes periodically... – Rob I Feb 10 '16 at 13:57

Dennis Williamson , May 10, 2012 at 3:14

read with IFS are perfect for this:
$ IFS=- read var1 var2 <<< ABCDE-123456
$ echo "$var1"
ABCDE
$ echo "$var2"
123456

Edit:

Here is how you can read each individual character into array elements:

$ read -a foo <<<"$(echo "ABCDE-123456" | sed 's/./& /g')"

Dump the array:

$ declare -p foo
declare -a foo='([0]="A" [1]="B" [2]="C" [3]="D" [4]="E" [5]="-" [6]="1" [7]="2" [8]="3" [9]="4" [10]="5" [11]="6")'

If there are spaces in the string:

$ IFS=$'\v' read -a foo <<<"$(echo "ABCDE 123456" | sed 's/./&\v/g')"
$ declare -p foo
declare -a foo='([0]="A" [1]="B" [2]="C" [3]="D" [4]="E" [5]=" " [6]="1" [7]="2" [8]="3" [9]="4" [10]="5" [11]="6")'

insecure , Apr 30, 2014 at 7:51

Great, the elegant bash-only way, without unnecessary forks. – insecure Apr 30 '14 at 7:51

Martin Serrano , Jan 11 at 4:34

this solution also has the benefit that if delimiter is not present, the var2 will be empty – Martin Serrano Jan 11 at 4:34

mkb , May 9, 2012 at 17:02

If you know it's going to be just two fields, you can skip the extra subprocesses like this:
var1=${STR%-*}
var2=${STR#*-}

What does this do? ${STR%-*} deletes the shortest substring of $STR that matches the pattern -* starting from the end of the string. ${STR#*-} does the same, but with the *- pattern and starting from the beginning of the string. They each have counterparts %% and ## which find the longest anchored pattern match. If anyone has a helpful mnemonic to remember which does which, let me know! I always have to try both to remember.

Jens , Jan 30, 2015 at 15:17

Plus 1 For knowing your POSIX shell features, avoiding expensive forks and pipes, and the absence of bashisms. – Jens Jan 30 '15 at 15:17

Steven Lu , May 1, 2015 at 20:19

Dunno about "absence of bashisms" considering that this is already moderately cryptic .... if your delimiter is a newline instead of a hyphen, then it becomes even more cryptic. On the other hand, it works with newlines , so there's that. – Steven Lu May 1 '15 at 20:19

mkb , Mar 9, 2016 at 17:30

@KErlandsson: done – mkb Mar 9 '16 at 17:30

mombip , Aug 9, 2016 at 15:58

I've finally found documentation for it: Shell-Parameter-Expansionmombip Aug 9 '16 at 15:58

DS. , Jan 13, 2017 at 19:56

Mnemonic: "#" is to the left of "%" on a standard keyboard, so "#" removes a prefix (on the left), and "%" removes a suffix (on the right). – DS. Jan 13 '17 at 19:56

tripleee , May 9, 2012 at 17:57

Sounds like a job for set with a custom IFS .
IFS=-
set $STR
var1=$1
var2=$2

(You will want to do this in a function with a local IFS so you don't mess up other parts of your script where you require IFS to be what you expect.)

Rob I , May 9, 2012 at 19:20

Nice - I knew about $IFS but hadn't seen how it could be used. – Rob I May 9 '12 at 19:20

Sigg3.net , Jun 19, 2013 at 8:08

I used triplee's example and it worked exactly as advertised! Just change last two lines to <pre> myvar1= echo $1 && myvar2= echo $2 </pre> if you need to store them throughout a script with several "thrown" variables. – Sigg3.net Jun 19 '13 at 8:08

tripleee , Jun 19, 2013 at 13:25

No, don't use a useless echo in backticks . – tripleee Jun 19 '13 at 13:25

Daniel Andersson , Mar 27, 2015 at 6:46

This is a really sweet solution if we need to write something that is not Bash specific. To handle IFS troubles, one can add OLDIFS=$IFS at the beginning before overwriting it, and then add IFS=$OLDIFS just after the set line. – Daniel Andersson Mar 27 '15 at 6:46

tripleee , Mar 27, 2015 at 6:58

FWIW the link above is broken. I was lazy and careless. The canonical location still works; iki.fi/era/unix/award.html#echotripleee Mar 27 '15 at 6:58

anubhava , May 9, 2012 at 17:09

Using bash regex capabilities:
re="^([^-]+)-(.*)$"
[[ "ABCDE-123456" =~ $re ]] && var1="${BASH_REMATCH[1]}" && var2="${BASH_REMATCH[2]}"
echo $var1
echo $var2

OUTPUT

ABCDE
123456

Cometsong , Oct 21, 2016 at 13:29

Love pre-defining the re for later use(s)! – Cometsong Oct 21 '16 at 13:29

Archibald , Nov 12, 2012 at 11:03

string="ABCDE-123456"
IFS=- # use "local IFS=-" inside the function
set $string
echo $1 # >>> ABCDE
echo $2 # >>> 123456

tripleee , Mar 27, 2015 at 7:02

Hmmm, isn't this just a restatement of my answer ? – tripleee Mar 27 '15 at 7:02

Archibald , Sep 18, 2015 at 12:36

Actually yes. I just clarified it a bit. – Archibald Sep 18 '15 at 12:36

[Nov 08, 2018] How to split a string in shell and get the last field

Nov 08, 2018 | stackoverflow.com

cd1 , Jul 1, 2010 at 23:29

Suppose I have the string 1:2:3:4:5 and I want to get its last field ( 5 in this case). How do I do that using Bash? I tried cut , but I don't know how to specify the last field with -f .

Stephen , Jul 2, 2010 at 0:05

You can use string operators :
$ foo=1:2:3:4:5
$ echo ${foo##*:}
5

This trims everything from the front until a ':', greedily.

${foo  <-- from variable foo
  ##   <-- greedy front trim
  *    <-- matches anything
  :    <-- until the last ':'
 }

eckes , Jan 23, 2013 at 15:23

While this is working for the given problem, the answer of William below ( stackoverflow.com/a/3163857/520162 ) also returns 5 if the string is 1:2:3:4:5: (while using the string operators yields an empty result). This is especially handy when parsing paths that could contain (or not) a finishing / character. – eckes Jan 23 '13 at 15:23

Dobz , Jun 25, 2014 at 11:44

How would you then do the opposite of this? to echo out '1:2:3:4:'? – Dobz Jun 25 '14 at 11:44

Mihai Danila , Jul 9, 2014 at 14:07

And how does one keep the part before the last separator? Apparently by using ${foo%:*} . # - from beginning; % - from end. # , % - shortest match; ## , %% - longest match. – Mihai Danila Jul 9 '14 at 14:07

Putnik , Feb 11, 2016 at 22:33

If i want to get the last element from path, how should I use it? echo ${pwd##*/} does not work. – Putnik Feb 11 '16 at 22:33

Stan Strum , Dec 17, 2017 at 4:22

@Putnik that command sees pwd as a variable. Try dir=$(pwd); echo ${dir##*/} . Works for me! – Stan Strum Dec 17 '17 at 4:22

a3nm , Feb 3, 2012 at 8:39

Another way is to reverse before and after cut :
$ echo ab:cd:ef | rev | cut -d: -f1 | rev
ef

This makes it very easy to get the last but one field, or any range of fields numbered from the end.

Dannid , Jan 14, 2013 at 20:50

This answer is nice because it uses 'cut', which the author is (presumably) already familiar. Plus, I like this answer because I am using 'cut' and had this exact question, hence finding this thread via search. – Dannid Jan 14 '13 at 20:50

funroll , Aug 12, 2013 at 19:51

Some cut-and-paste fodder for people using spaces as delimiters: echo "1 2 3 4" | rev | cut -d " " -f1 | revfunroll Aug 12 '13 at 19:51

EdgeCaseBerg , Sep 8, 2013 at 5:01

the rev | cut -d -f1 | rev is so clever! Thanks! Helped me a bunch (my use case was rev | -d ' ' -f 2- | rev – EdgeCaseBerg Sep 8 '13 at 5:01

Anarcho-Chossid , Sep 16, 2015 at 15:54

Wow. Beautiful and dark magic. – Anarcho-Chossid Sep 16 '15 at 15:54

shearn89 , Aug 17, 2017 at 9:27

I always forget about rev , was just what I needed! cut -b20- | rev | cut -b10- | revshearn89 Aug 17 '17 at 9:27

William Pursell , Jul 2, 2010 at 7:09

It's difficult to get the last field using cut, but here's (one set of) solutions in awk and perl
$ echo 1:2:3:4:5 | awk -F: '{print $NF}'
5
$ echo 1:2:3:4:5 | perl -F: -wane 'print $F[-1]'
5

eckes , Jan 23, 2013 at 15:20

great advantage of this solution over the accepted answer: it also matches paths that contain or do not contain a finishing / character: /a/b/c/d and /a/b/c/d/ yield the same result ( d ) when processing pwd | awk -F/ '{print $NF}' . The accepted answer results in an empty result in the case of /a/b/c/d/eckes Jan 23 '13 at 15:20

stamster , May 21 at 11:52

@eckes In case of AWK solution, on GNU bash, version 4.3.48(1)-release that's not true, as it matters whenever you have trailing slash or not. Simply put AWK will use / as delimiter, and if your path is /my/path/dir/ it will use value after last delimiter, which is simply an empty string. So it's best to avoid trailing slash if you need to do such a thing like I do. – stamster May 21 at 11:52

Nicholas M T Elliott , Jul 1, 2010 at 23:39

Assuming fairly simple usage (no escaping of the delimiter, for example), you can use grep:
$ echo "1:2:3:4:5" | grep -oE "[^:]+$"
5

Breakdown - find all the characters not the delimiter ([^:]) at the end of the line ($). -o only prints the matching part.

Dennis Williamson , Jul 2, 2010 at 0:05

One way:
var1="1:2:3:4:5"
var2=${var1##*:}

Another, using an array:

var1="1:2:3:4:5"
saveIFS=$IFS
IFS=":"
var2=($var1)
IFS=$saveIFS
var2=${var2[@]: -1}

Yet another with an array:

var1="1:2:3:4:5"
saveIFS=$IFS
IFS=":"
var2=($var1)
IFS=$saveIFS
count=${#var2[@]}
var2=${var2[$count-1]}

Using Bash (version >= 3.2) regular expressions:

var1="1:2:3:4:5"
[[ $var1 =~ :([^:]*)$ ]]
var2=${BASH_REMATCH[1]}

liuyang1 , Mar 24, 2015 at 6:02

Thanks so much for array style, as I need this feature, but not have cut, awk these utils. – liuyang1 Mar 24 '15 at 6:02

user3133260 , Dec 24, 2013 at 19:04

$ echo "a b c d e" | tr ' ' '\n' | tail -1
e

Simply translate the delimiter into a newline and choose the last entry with tail -1 .

Yajo , Jul 30, 2014 at 10:13

It will fail if the last item contains a \n , but for most cases is the most readable solution. – Yajo Jul 30 '14 at 10:13

Rafael , Nov 10, 2016 at 10:09

Using sed :
$ echo '1:2:3:4:5' | sed 's/.*://' # => 5

$ echo '' | sed 's/.*://' # => (empty)

$ echo ':' | sed 's/.*://' # => (empty)
$ echo ':b' | sed 's/.*://' # => b
$ echo '::c' | sed 's/.*://' # => c

$ echo 'a' | sed 's/.*://' # => a
$ echo 'a:' | sed 's/.*://' # => (empty)
$ echo 'a:b' | sed 's/.*://' # => b
$ echo 'a::c' | sed 's/.*://' # => c

Ab Irato , Nov 13, 2013 at 16:10

If your last field is a single character, you could do this:
a="1:2:3:4:5"

echo ${a: -1}
echo ${a:(-1)}

Check string manipulation in bash .

gniourf_gniourf , Nov 13, 2013 at 16:15

This doesn't work: it gives the last character of a , not the last field . – gniourf_gniourf Nov 13 '13 at 16:15

Ab Irato , Nov 25, 2013 at 13:25

True, that's the idea, if you know the length of the last field it's good. If not you have to use something else... – Ab Irato Nov 25 '13 at 13:25

sphakka , Jan 25, 2016 at 16:24

Interesting, I didn't know of these particular Bash string manipulations. It also resembles to Python's string/array slicing . – sphakka Jan 25 '16 at 16:24

ghostdog74 , Jul 2, 2010 at 1:16

Using Bash.
$ var1="1:2:3:4:0"
$ IFS=":"
$ set -- $var1
$ eval echo  \$${#}
0

Sopalajo de Arrierez , Dec 24, 2014 at 5:04

I would buy some details about this method, please :-) . – Sopalajo de Arrierez Dec 24 '14 at 5:04

Rafa , Apr 27, 2017 at 22:10

Could have used echo ${!#} instead of eval echo \$${#} . – Rafa Apr 27 '17 at 22:10

Crytis , Dec 7, 2016 at 6:51

echo "a:b:c:d:e"|xargs -d : -n1|tail -1

First use xargs split it using ":",-n1 means every line only have one part.Then,pring the last part.

BDL , Dec 7, 2016 at 13:47

Although this might solve the problem, one should always add an explanation to it. – BDL Dec 7 '16 at 13:47

Crytis , Jun 7, 2017 at 9:13

already added.. – Crytis Jun 7 '17 at 9:13

021 , Apr 26, 2016 at 11:33

There are many good answers here, but still I want to share this one using basename :
 basename $(echo "a:b:c:d:e" | tr ':' '/')

However it will fail if there are already some '/' in your string . If slash / is your delimiter then you just have to (and should) use basename.

It's not the best answer but it just shows how you can be creative using bash commands.

Nahid Akbar , Jun 22, 2012 at 2:55

for x in `echo $str | tr ";" "\n"`; do echo $x; done

chepner , Jun 22, 2012 at 12:58

This runs into problems if there is whitespace in any of the fields. Also, it does not directly address the question of retrieving the last field. – chepner Jun 22 '12 at 12:58

Christoph Böddeker , Feb 19 at 15:50

For those that comfortable with Python, https://github.com/Russell91/pythonpy is a nice choice to solve this problem.
$ echo "a:b:c:d:e" | py -x 'x.split(":")[-1]'

From the pythonpy help: -x treat each row of stdin as x .

With that tool, it is easy to write python code that gets applied to the input.

baz , Nov 24, 2017 at 19:27

a solution using the read builtin
IFS=':' read -a field <<< "1:2:3:4:5"
echo ${field[4]}

[Nov 08, 2018] How do I split a string on a delimiter in Bash?

Notable quotes:
"... Bash shell script split array ..."
"... associative array ..."
"... pattern substitution ..."
"... Debian GNU/Linux ..."
Nov 08, 2018 | stackoverflow.com

stefanB , May 28, 2009 at 2:03

I have this string stored in a variable:
IN="bla@some.com;john@home.com"

Now I would like to split the strings by ; delimiter so that I have:

ADDR1="bla@some.com"
ADDR2="john@home.com"

I don't necessarily need the ADDR1 and ADDR2 variables. If they are elements of an array that's even better.


After suggestions from the answers below, I ended up with the following which is what I was after:

#!/usr/bin/env bash

IN="bla@some.com;john@home.com"

mails=$(echo $IN | tr ";" "\n")

for addr in $mails
do
    echo "> [$addr]"
done

Output:

> [bla@some.com]
> [john@home.com]

There was a solution involving setting Internal_field_separator (IFS) to ; . I am not sure what happened with that answer, how do you reset IFS back to default?

RE: IFS solution, I tried this and it works, I keep the old IFS and then restore it:

IN="bla@some.com;john@home.com"

OIFS=$IFS
IFS=';'
mails2=$IN
for x in $mails2
do
    echo "> [$x]"
done

IFS=$OIFS

BTW, when I tried

mails2=($IN)

I only got the first string when printing it in loop, without brackets around $IN it works.

Brooks Moses , May 1, 2012 at 1:26

With regards to your "Edit2": You can simply "unset IFS" and it will return to the default state. There's no need to save and restore it explicitly unless you have some reason to expect that it's already been set to a non-default value. Moreover, if you're doing this inside a function (and, if you aren't, why not?), you can set IFS as a local variable and it will return to its previous value once you exit the function. – Brooks Moses May 1 '12 at 1:26

dubiousjim , May 31, 2012 at 5:21

@BrooksMoses: (a) +1 for using local IFS=... where possible; (b) -1 for unset IFS , this doesn't exactly reset IFS to its default value, though I believe an unset IFS behaves the same as the default value of IFS ($' \t\n'), however it seems bad practice to be assuming blindly that your code will never be invoked with IFS set to a custom value; (c) another idea is to invoke a subshell: (IFS=$custom; ...) when the subshell exits IFS will return to whatever it was originally. – dubiousjim May 31 '12 at 5:21

nicooga , Mar 7, 2016 at 15:32

I just want to have a quick look at the paths to decide where to throw an executable, so I resorted to run ruby -e "puts ENV.fetch('PATH').split(':')" . If you want to stay pure bash won't help but using any scripting language that has a built-in split is easier. – nicooga Mar 7 '16 at 15:32

Jeff , Apr 22 at 17:51

This is kind of a drive-by comment, but since the OP used email addresses as the example, has anyone bothered to answer it in a way that is fully RFC 5322 compliant, namely that any quoted string can appear before the @ which means you're going to need regular expressions or some other kind of parser instead of naive use of IFS or other simplistic splitter functions. – Jeff Apr 22 at 17:51

user2037659 , Apr 26 at 20:15

for x in $(IFS=';';echo $IN); do echo "> [$x]"; doneuser2037659 Apr 26 at 20:15

Johannes Schaub - litb , May 28, 2009 at 2:23

You can set the internal field separator (IFS) variable, and then let it parse into an array. When this happens in a command, then the assignment to IFS only takes place to that single command's environment (to read ). It then parses the input according to the IFS variable value into an array, which we can then iterate over.
IFS=';' read -ra ADDR <<< "$IN"
for i in "${ADDR[@]}"; do
    # process "$i"
done

It will parse one line of items separated by ; , pushing it into an array. Stuff for processing whole of $IN , each time one line of input separated by ; :

 while IFS=';' read -ra ADDR; do
      for i in "${ADDR[@]}"; do
          # process "$i"
      done
 done <<< "$IN"

Chris Lutz , May 28, 2009 at 2:25

This is probably the best way. How long will IFS persist in it's current value, can it mess up my code by being set when it shouldn't be, and how can I reset it when I'm done with it? – Chris Lutz May 28 '09 at 2:25

Johannes Schaub - litb , May 28, 2009 at 3:04

now after the fix applied, only within the duration of the read command :) – Johannes Schaub - litb May 28 '09 at 3:04

lhunath , May 28, 2009 at 6:14

You can read everything at once without using a while loop: read -r -d '' -a addr <<< "$in" # The -d '' is key here, it tells read not to stop at the first newline (which is the default -d) but to continue until EOF or a NULL byte (which only occur in binary data). – lhunath May 28 '09 at 6:14

Charles Duffy , Jul 6, 2013 at 14:39

@LucaBorrione Setting IFS on the same line as the read with no semicolon or other separator, as opposed to in a separate command, scopes it to that command -- so it's always "restored"; you don't need to do anything manually. – Charles Duffy Jul 6 '13 at 14:39

chepner , Oct 2, 2014 at 3:50

@imagineerThis There is a bug involving herestrings and local changes to IFS that requires $IN to be quoted. The bug is fixed in bash 4.3. – chepner Oct 2 '14 at 3:50

palindrom , Mar 10, 2011 at 9:00

Taken from Bash shell script split array :
IN="bla@some.com;john@home.com"
arrIN=(${IN//;/ })

Explanation:

This construction replaces all occurrences of ';' (the initial // means global replace) in the string IN with ' ' (a single space), then interprets the space-delimited string as an array (that's what the surrounding parentheses do).

The syntax used inside of the curly braces to replace each ';' character with a ' ' character is called Parameter Expansion .

There are some common gotchas:

  1. If the original string has spaces, you will need to use IFS :
    • IFS=':'; arrIN=($IN); unset IFS;
  2. If the original string has spaces and the delimiter is a new line, you can set IFS with:
    • IFS=$'\n'; arrIN=($IN); unset IFS;

Oz123 , Mar 21, 2011 at 18:50

I just want to add: this is the simplest of all, you can access array elements with ${arrIN[1]} (starting from zeros of course) – Oz123 Mar 21 '11 at 18:50

KomodoDave , Jan 5, 2012 at 15:13

Found it: the technique of modifying a variable within a ${} is known as 'parameter expansion'. – KomodoDave Jan 5 '12 at 15:13

qbolec , Feb 25, 2013 at 9:12

Does it work when the original string contains spaces? – qbolec Feb 25 '13 at 9:12

Ethan , Apr 12, 2013 at 22:47

No, I don't think this works when there are also spaces present... it's converting the ',' to ' ' and then building a space-separated array. – Ethan Apr 12 '13 at 22:47

Charles Duffy , Jul 6, 2013 at 14:39

This is a bad approach for other reasons: For instance, if your string contains ;*; , then the * will be expanded to a list of filenames in the current directory. -1 – Charles Duffy Jul 6 '13 at 14:39

Chris Lutz , May 28, 2009 at 2:09

If you don't mind processing them immediately, I like to do this:
for i in $(echo $IN | tr ";" "\n")
do
  # process
done

You could use this kind of loop to initialize an array, but there's probably an easier way to do it. Hope this helps, though.

Chris Lutz , May 28, 2009 at 2:42

You should have kept the IFS answer. It taught me something I didn't know, and it definitely made an array, whereas this just makes a cheap substitute. – Chris Lutz May 28 '09 at 2:42

Johannes Schaub - litb , May 28, 2009 at 2:59

I see. Yeah i find doing these silly experiments, i'm going to learn new things each time i'm trying to answer things. I've edited stuff based on #bash IRC feedback and undeleted :) – Johannes Schaub - litb May 28 '09 at 2:59

lhunath , May 28, 2009 at 6:12

-1, you're obviously not aware of wordsplitting, because it's introducing two bugs in your code. one is when you don't quote $IN and the other is when you pretend a newline is the only delimiter used in wordsplitting. You are iterating over every WORD in IN, not every line, and DEFINATELY not every element delimited by a semicolon, though it may appear to have the side-effect of looking like it works. – lhunath May 28 '09 at 6:12

Johannes Schaub - litb , May 28, 2009 at 17:00

You could change it to echo "$IN" | tr ';' '\n' | while read -r ADDY; do # process "$ADDY"; done to make him lucky, i think :) Note that this will fork, and you can't change outer variables from within the loop (that's why i used the <<< "$IN" syntax) then – Johannes Schaub - litb May 28 '09 at 17:00

mklement0 , Apr 24, 2013 at 14:13

To summarize the debate in the comments: Caveats for general use : the shell applies word splitting and expansions to the string, which may be undesired; just try it with. IN="bla@some.com;john@home.com;*;broken apart" . In short: this approach will break, if your tokens contain embedded spaces and/or chars. such as * that happen to make a token match filenames in the current folder. – mklement0 Apr 24 '13 at 14:13

F. Hauri , Apr 13, 2013 at 14:20

Compatible answer

To this SO question, there is already a lot of different way to do this in bash . But bash has many special features, so called bashism that work well, but that won't work in any other shell .

In particular, arrays , associative array , and pattern substitution are pure bashisms and may not work under other shells .

On my Debian GNU/Linux , there is a standard shell called dash , but I know many people who like to use ksh .

Finally, in very small situation, there is a special tool called busybox with his own shell interpreter ( ash ).

Requested string

The string sample in SO question is:

IN="bla@some.com;john@home.com"

As this could be useful with whitespaces and as whitespaces could modify the result of the routine, I prefer to use this sample string:

 IN="bla@some.com;john@home.com;Full Name <fulnam@other.org>"
Split string based on delimiter in bash (version >=4.2)

Under pure bash, we may use arrays and IFS :

var="bla@some.com;john@home.com;Full Name <fulnam@other.org>"
oIFS="$IFS"
IFS=";"
declare -a fields=($var)
IFS="$oIFS"
unset oIFS

IFS=\; read -a fields <<<"$var"

Using this syntax under recent bash don't change $IFS for current session, but only for the current command:

set | grep ^IFS=
IFS=$' \t\n'

Now the string var is split and stored into an array (named fields ):

set | grep ^fields=\\\|^var=
fields=([0]="bla@some.com" [1]="john@home.com" [2]="Full Name <fulnam@other.org>")
var='bla@some.com;john@home.com;Full Name <fulnam@other.org>'

We could request for variable content with declare -p :

declare -p var fields
declare -- var="bla@some.com;john@home.com;Full Name <fulnam@other.org>"
declare -a fields=([0]="bla@some.com" [1]="john@home.com" [2]="Full Name <fulnam@other.org>")

read is the quickiest way to do the split, because there is no forks and no external resources called.

From there, you could use the syntax you already know for processing each field:

for x in "${fields[@]}";do
    echo "> [$x]"
    done
> [bla@some.com]
> [john@home.com]
> [Full Name <fulnam@other.org>]

or drop each field after processing (I like this shifting approach):

while [ "$fields" ] ;do
    echo "> [$fields]"
    fields=("${fields[@]:1}")
    done
> [bla@some.com]
> [john@home.com]
> [Full Name <fulnam@other.org>]

or even for simple printout (shorter syntax):

printf "> [%s]\n" "${fields[@]}"
> [bla@some.com]
> [john@home.com]
> [Full Name <fulnam@other.org>]
Split string based on delimiter in shell

But if you would write something usable under many shells, you have to not use bashisms .

There is a syntax, used in many shells, for splitting a string across first or last occurrence of a substring:

${var#*SubStr}  # will drop begin of string up to first occur of `SubStr`
${var##*SubStr} # will drop begin of string up to last occur of `SubStr`
${var%SubStr*}  # will drop part of string from last occur of `SubStr` to the end
${var%%SubStr*} # will drop part of string from first occur of `SubStr` to the end

(The missing of this is the main reason of my answer publication ;)

As pointed out by Score_Under :

# and % delete the shortest possible matching string, and

## and %% delete the longest possible.

This little sample script work well under bash , dash , ksh , busybox and was tested under Mac-OS's bash too:

var="bla@some.com;john@home.com;Full Name <fulnam@other.org>"
while [ "$var" ] ;do
    iter=${var%%;*}
    echo "> [$iter]"
    [ "$var" = "$iter" ] && \
        var='' || \
        var="${var#*;}"
  done
> [bla@some.com]
> [john@home.com]
> [Full Name <fulnam@other.org>]

Have fun!

Score_Under , Apr 28, 2015 at 16:58

The # , ## , % , and %% substitutions have what is IMO an easier explanation to remember (for how much they delete): # and % delete the shortest possible matching string, and ## and %% delete the longest possible. – Score_Under Apr 28 '15 at 16:58

sorontar , Oct 26, 2016 at 4:36

The IFS=\; read -a fields <<<"$var" fails on newlines and add a trailing newline. The other solution removes a trailing empty field. – sorontar Oct 26 '16 at 4:36

Eric Chen , Aug 30, 2017 at 17:50

The shell delimiter is the most elegant answer, period. – Eric Chen Aug 30 '17 at 17:50

sancho.s , Oct 4 at 3:42

Could the last alternative be used with a list of field separators set somewhere else? For instance, I mean to use this as a shell script, and pass a list of field separators as a positional parameter. – sancho.s Oct 4 at 3:42

F. Hauri , Oct 4 at 7:47

Yes, in a loop: for sep in "#" "ł" "@" ; do ... var="${var#*$sep}" ...F. Hauri Oct 4 at 7:47

DougW , Apr 27, 2015 at 18:20

I've seen a couple of answers referencing the cut command, but they've all been deleted. It's a little odd that nobody has elaborated on that, because I think it's one of the more useful commands for doing this type of thing, especially for parsing delimited log files.

In the case of splitting this specific example into a bash script array, tr is probably more efficient, but cut can be used, and is more effective if you want to pull specific fields from the middle.

Example:

$ echo "bla@some.com;john@home.com" | cut -d ";" -f 1
bla@some.com
$ echo "bla@some.com;john@home.com" | cut -d ";" -f 2
john@home.com

You can obviously put that into a loop, and iterate the -f parameter to pull each field independently.

This gets more useful when you have a delimited log file with rows like this:

2015-04-27|12345|some action|an attribute|meta data

cut is very handy to be able to cat this file and select a particular field for further processing.

MisterMiyagi , Nov 2, 2016 at 8:42

Kudos for using cut , it's the right tool for the job! Much cleared than any of those shell hacks. – MisterMiyagi Nov 2 '16 at 8:42

uli42 , Sep 14, 2017 at 8:30

This approach will only work if you know the number of elements in advance; you'd need to program some more logic around it. It also runs an external tool for every element. – uli42 Sep 14 '17 at 8:30

Louis Loudog Trottier , May 10 at 4:20

Excatly waht i was looking for trying to avoid empty string in a csv. Now i can point the exact 'column' value as well. Work with IFS already used in a loop. Better than expected for my situation. – Louis Loudog Trottier May 10 at 4:20

, May 28, 2009 at 10:31

How about this approach:
IN="bla@some.com;john@home.com" 
set -- "$IN" 
IFS=";"; declare -a Array=($*) 
echo "${Array[@]}" 
echo "${Array[0]}" 
echo "${Array[1]}"

Source

Yzmir Ramirez , Sep 5, 2011 at 1:06

+1 ... but I wouldn't name the variable "Array" ... pet peev I guess. Good solution. – Yzmir Ramirez Sep 5 '11 at 1:06

ata , Nov 3, 2011 at 22:33

+1 ... but the "set" and declare -a are unnecessary. You could as well have used just IFS";" && Array=($IN)ata Nov 3 '11 at 22:33

Luca Borrione , Sep 3, 2012 at 9:26

+1 Only a side note: shouldn't it be recommendable to keep the old IFS and then restore it? (as shown by stefanB in his edit3) people landing here (sometimes just copying and pasting a solution) might not think about this – Luca Borrione Sep 3 '12 at 9:26

Charles Duffy , Jul 6, 2013 at 14:44

-1: First, @ata is right that most of the commands in this do nothing. Second, it uses word-splitting to form the array, and doesn't do anything to inhibit glob-expansion when doing so (so if you have glob characters in any of the array elements, those elements are replaced with matching filenames). – Charles Duffy Jul 6 '13 at 14:44

John_West , Jan 8, 2016 at 12:29

Suggest to use $'...' : IN=$'bla@some.com;john@home.com;bet <d@\ns* kl.com>' . Then echo "${Array[2]}" will print a string with newline. set -- "$IN" is also neccessary in this case. Yes, to prevent glob expansion, the solution should include set -f . – John_West Jan 8 '16 at 12:29

Steven Lizarazo , Aug 11, 2016 at 20:45

This worked for me:
string="1;2"
echo $string | cut -d';' -f1 # output is 1
echo $string | cut -d';' -f2 # output is 2

Pardeep Sharma , Oct 10, 2017 at 7:29

this is sort and sweet :) – Pardeep Sharma Oct 10 '17 at 7:29

space earth , Oct 17, 2017 at 7:23

Thanks...Helped a lot – space earth Oct 17 '17 at 7:23

mojjj , Jan 8 at 8:57

cut works only with a single char as delimiter. – mojjj Jan 8 at 8:57

lothar , May 28, 2009 at 2:12

echo "bla@some.com;john@home.com" | sed -e 's/;/\n/g'
bla@some.com
john@home.com

Luca Borrione , Sep 3, 2012 at 10:08

-1 what if the string contains spaces? for example IN="this is first line; this is second line" arrIN=( $( echo "$IN" | sed -e 's/;/\n/g' ) ) will produce an array of 8 elements in this case (an element for each word space separated), rather than 2 (an element for each line semi colon separated) – Luca Borrione Sep 3 '12 at 10:08

lothar , Sep 3, 2012 at 17:33

@Luca No the sed script creates exactly two lines. What creates the multiple entries for you is when you put it into a bash array (which splits on white space by default) – lothar Sep 3 '12 at 17:33

Luca Borrione , Sep 4, 2012 at 7:09

That's exactly the point: the OP needs to store entries into an array to loop over it, as you can see in his edits. I think your (good) answer missed to mention to use arrIN=( $( echo "$IN" | sed -e 's/;/\n/g' ) ) to achieve that, and to advice to change IFS to IFS=$'\n' for those who land here in the future and needs to split a string containing spaces. (and to restore it back afterwards). :) – Luca Borrione Sep 4 '12 at 7:09

lothar , Sep 4, 2012 at 16:55

@Luca Good point. However the array assignment was not in the initial question when I wrote up that answer. – lothar Sep 4 '12 at 16:55

Ashok , Sep 8, 2012 at 5:01

This also works:
IN="bla@some.com;john@home.com"
echo ADD1=`echo $IN | cut -d \; -f 1`
echo ADD2=`echo $IN | cut -d \; -f 2`

Be careful, this solution is not always correct. In case you pass "bla@some.com" only, it will assign it to both ADD1 and ADD2.

fersarr , Mar 3, 2016 at 17:17

You can use -s to avoid the mentioned problem: superuser.com/questions/896800/ "-f, --fields=LIST select only these fields; also print any line that contains no delimiter character, unless the -s option is specified" – fersarr Mar 3 '16 at 17:17

Tony , Jan 14, 2013 at 6:33

I think AWK is the best and efficient command to resolve your problem. AWK is included in Bash by default in almost every Linux distribution.
echo "bla@some.com;john@home.com" | awk -F';' '{print $1,$2}'

will give

bla@some.com john@home.com

Of course your can store each email address by redefining the awk print field.

Jaro , Jan 7, 2014 at 21:30

Or even simpler: echo "bla@some.com;john@home.com" | awk 'BEGIN{RS=";"} {print}' – Jaro Jan 7 '14 at 21:30

Aquarelle , May 6, 2014 at 21:58

@Jaro This worked perfectly for me when I had a string with commas and needed to reformat it into lines. Thanks. – Aquarelle May 6 '14 at 21:58

Eduardo Lucio , Aug 5, 2015 at 12:59

It worked in this scenario -> "echo "$SPLIT_0" | awk -F' inode=' '{print $1}'"! I had problems when trying to use atrings (" inode=") instead of characters (";"). $ 1, $ 2, $ 3, $ 4 are set as positions in an array! If there is a way of setting an array... better! Thanks! – Eduardo Lucio Aug 5 '15 at 12:59

Tony , Aug 6, 2015 at 2:42

@EduardoLucio, what I'm thinking about is maybe you can first replace your delimiter inode= into ; for example by sed -i 's/inode\=/\;/g' your_file_to_process , then define -F';' when apply awk , hope that can help you. – Tony Aug 6 '15 at 2:42

nickjb , Jul 5, 2011 at 13:41

A different take on Darron's answer , this is how I do it:
IN="bla@some.com;john@home.com"
read ADDR1 ADDR2 <<<$(IFS=";"; echo $IN)

ColinM , Sep 10, 2011 at 0:31

This doesn't work. – ColinM Sep 10 '11 at 0:31

nickjb , Oct 6, 2011 at 15:33

I think it does! Run the commands above and then "echo $ADDR1 ... $ADDR2" and i get "bla@some.com ... john@home.com" output – nickjb Oct 6 '11 at 15:33

Nick , Oct 28, 2011 at 14:36

This worked REALLY well for me... I used it to itterate over an array of strings which contained comma separated DB,SERVER,PORT data to use mysqldump. – Nick Oct 28 '11 at 14:36

dubiousjim , May 31, 2012 at 5:28

Diagnosis: the IFS=";" assignment exists only in the $(...; echo $IN) subshell; this is why some readers (including me) initially think it won't work. I assumed that all of $IN was getting slurped up by ADDR1. But nickjb is correct; it does work. The reason is that echo $IN command parses its arguments using the current value of $IFS, but then echoes them to stdout using a space delimiter, regardless of the setting of $IFS. So the net effect is as though one had called read ADDR1 ADDR2 <<< "bla@some.com john@home.com" (note the input is space-separated not ;-separated). – dubiousjim May 31 '12 at 5:28

sorontar , Oct 26, 2016 at 4:43

This fails on spaces and newlines, and also expand wildcards * in the echo $IN with an unquoted variable expansion. – sorontar Oct 26 '16 at 4:43

gniourf_gniourf , Jun 26, 2014 at 9:11

In Bash, a bullet proof way, that will work even if your variable contains newlines:
IFS=';' read -d '' -ra array < <(printf '%s;\0' "$in")

Look:

$ in=$'one;two three;*;there is\na newline\nin this field'
$ IFS=';' read -d '' -ra array < <(printf '%s;\0' "$in")
$ declare -p array
declare -a array='([0]="one" [1]="two three" [2]="*" [3]="there is
a newline
in this field")'

The trick for this to work is to use the -d option of read (delimiter) with an empty delimiter, so that read is forced to read everything it's fed. And we feed read with exactly the content of the variable in , with no trailing newline thanks to printf . Note that's we're also putting the delimiter in printf to ensure that the string passed to read has a trailing delimiter. Without it, read would trim potential trailing empty fields:

$ in='one;two;three;'    # there's an empty field
$ IFS=';' read -d '' -ra array < <(printf '%s;\0' "$in")
$ declare -p array
declare -a array='([0]="one" [1]="two" [2]="three" [3]="")'

the trailing empty field is preserved.


Update for Bash≥4.4

Since Bash 4.4, the builtin mapfile (aka readarray ) supports the -d option to specify a delimiter. Hence another canonical way is:

mapfile -d ';' -t array < <(printf '%s;' "$in")

John_West , Jan 8, 2016 at 12:10

I found it as the rare solution on that list that works correctly with \n , spaces and * simultaneously. Also, no loops; array variable is accessible in the shell after execution (contrary to the highest upvoted answer). Note, in=$'...' , it does not work with double quotes. I think, it needs more upvotes. – John_West Jan 8 '16 at 12:10

Darron , Sep 13, 2010 at 20:10

How about this one liner, if you're not using arrays:
IFS=';' read ADDR1 ADDR2 <<<$IN

dubiousjim , May 31, 2012 at 5:36

Consider using read -r ... to ensure that, for example, the two characters "\t" in the input end up as the same two characters in your variables (instead of a single tab char). – dubiousjim May 31 '12 at 5:36

Luca Borrione , Sep 3, 2012 at 10:07

-1 This is not working here (ubuntu 12.04). Adding echo "ADDR1 $ADDR1"\n echo "ADDR2 $ADDR2" to your snippet will output ADDR1 bla@some.com john@home.com\nADDR2 (\n is newline) – Luca Borrione Sep 3 '12 at 10:07

chepner , Sep 19, 2015 at 13:59

This is probably due to a bug involving IFS and here strings that was fixed in bash 4.3. Quoting $IN should fix it. (In theory, $IN is not subject to word splitting or globbing after it expands, meaning the quotes should be unnecessary. Even in 4.3, though, there's at least one bug remaining--reported and scheduled to be fixed--so quoting remains a good idea.) – chepner Sep 19 '15 at 13:59

sorontar , Oct 26, 2016 at 4:55

This breaks if $in contain newlines even if $IN is quoted. And adds a trailing newline. – sorontar Oct 26 '16 at 4:55

kenorb , Sep 11, 2015 at 20:54

Here is a clean 3-liner:
in="foo@bar;bizz@buzz;fizz@buzz;buzz@woof"
IFS=';' list=($in)
for item in "${list[@]}"; do echo $item; done

where IFS delimit words based on the separator and () is used to create an array . Then [@] is used to return each item as a separate word.

If you've any code after that, you also need to restore $IFS , e.g. unset IFS .

sorontar , Oct 26, 2016 at 5:03

The use of $in unquoted allows wildcards to be expanded. – sorontar Oct 26 '16 at 5:03

user2720864 , Sep 24 at 13:46

+ for the unset command – user2720864 Sep 24 at 13:46

Emilien Brigand , Aug 1, 2016 at 13:15

Without setting the IFS

If you just have one colon you can do that:

a="foo:bar"
b=${a%:*}
c=${a##*:}

you will get:

b = foo
c = bar

Victor Choy , Sep 16, 2015 at 3:34

There is a simple and smart way like this:
echo "add:sfff" | xargs -d: -i  echo {}

But you must use gnu xargs, BSD xargs cant support -d delim. If you use apple mac like me. You can install gnu xargs :

brew install findutils

then

echo "add:sfff" | gxargs -d: -i  echo {}

Halle Knast , May 24, 2017 at 8:42

The following Bash/zsh function splits its first argument on the delimiter given by the second argument:
split() {
    local string="$1"
    local delimiter="$2"
    if [ -n "$string" ]; then
        local part
        while read -d "$delimiter" part; do
            echo $part
        done <<< "$string"
        echo $part
    fi
}

For instance, the command

$ split 'a;b;c' ';'

yields

a
b
c

This output may, for instance, be piped to other commands. Example:

$ split 'a;b;c' ';' | cat -n
1   a
2   b
3   c

Compared to the other solutions given, this one has the following advantages:

If desired, the function may be put into a script as follows:

#!/usr/bin/env bash

split() {
    # ...
}

split "$@"

sandeepkunkunuru , Oct 23, 2017 at 16:10

works and neatly modularized. – sandeepkunkunuru Oct 23 '17 at 16:10

Prospero , Sep 25, 2011 at 1:09

This is the simplest way to do it.
spo='one;two;three'
OIFS=$IFS
IFS=';'
spo_array=($spo)
IFS=$OIFS
echo ${spo_array[*]}

rashok , Oct 25, 2016 at 12:41

IN="bla@some.com;john@home.com"
IFS=';'
read -a IN_arr <<< "${IN}"
for entry in "${IN_arr[@]}"
do
    echo $entry
done

Output

bla@some.com
john@home.com

System : Ubuntu 12.04.1

codeforester , Jan 2, 2017 at 5:37

IFS is not getting set in the specific context of read here and hence it can upset rest of the code, if any. – codeforester Jan 2 '17 at 5:37

shuaihanhungry , Jan 20 at 15:54

you can apply awk to many situations
echo "bla@some.com;john@home.com"|awk -F';' '{printf "%s\n%s\n", $1, $2}'

also you can use this

echo "bla@some.com;john@home.com"|awk -F';' '{print $1,$2}' OFS="\n"

ghost , Apr 24, 2013 at 13:13

If no space, Why not this?
IN="bla@some.com;john@home.com"
arr=(`echo $IN | tr ';' ' '`)

echo ${arr[0]}
echo ${arr[1]}

eukras , Oct 22, 2012 at 7:10

There are some cool answers here (errator esp.), but for something analogous to split in other languages -- which is what I took the original question to mean -- I settled on this:
IN="bla@some.com;john@home.com"
declare -a a="(${IN/;/ })";

Now ${a[0]} , ${a[1]} , etc, are as you would expect. Use ${#a[*]} for number of terms. Or to iterate, of course:

for i in ${a[*]}; do echo $i; done

IMPORTANT NOTE:

This works in cases where there are no spaces to worry about, which solved my problem, but may not solve yours. Go with the $IFS solution(s) in that case.

olibre , Oct 7, 2013 at 13:33

Does not work when IN contains more than two e-mail addresses. Please refer to same idea (but fixed) at palindrom's answerolibre Oct 7 '13 at 13:33

sorontar , Oct 26, 2016 at 5:14

Better use ${IN//;/ } (double slash) to make it also work with more than two values. Beware that any wildcard ( *?[ ) will be expanded. And a trailing empty field will be discarded. – sorontar Oct 26 '16 at 5:14

jeberle , Apr 30, 2013 at 3:10

Use the set built-in to load up the $@ array:
IN="bla@some.com;john@home.com"
IFS=';'; set $IN; IFS=$' \t\n'

Then, let the party begin:

echo $#
for a; do echo $a; done
ADDR1=$1 ADDR2=$2

sorontar , Oct 26, 2016 at 5:17

Better use set -- $IN to avoid some issues with "$IN" starting with dash. Still, the unquoted expansion of $IN will expand wildcards ( *?[ ). – sorontar Oct 26 '16 at 5:17

NevilleDNZ , Sep 2, 2013 at 6:30

Two bourne-ish alternatives where neither require bash arrays:

Case 1 : Keep it nice and simple: Use a NewLine as the Record-Separator... eg.

IN="bla@some.com
john@home.com"

while read i; do
  # process "$i" ... eg.
    echo "[email:$i]"
done <<< "$IN"

Note: in this first case no sub-process is forked to assist with list manipulation.

Idea: Maybe it is worth using NL extensively internally , and only converting to a different RS when generating the final result externally .

Case 2 : Using a ";" as a record separator... eg.

NL="
" IRS=";" ORS=";"

conv_IRS() {
  exec tr "$1" "$NL"
}

conv_ORS() {
  exec tr "$NL" "$1"
}

IN="bla@some.com;john@home.com"
IN="$(conv_IRS ";" <<< "$IN")"

while read i; do
  # process "$i" ... eg.
    echo -n "[email:$i]$ORS"
done <<< "$IN"

In both cases a sub-list can be composed within the loop is persistent after the loop has completed. This is useful when manipulating lists in memory, instead storing lists in files. {p.s. keep calm and carry on B-) }

fedorqui , Jan 8, 2015 at 10:21

Apart from the fantastic answers that were already provided, if it is just a matter of printing out the data you may consider using awk :
awk -F";" '{for (i=1;i<=NF;i++) printf("> [%s]\n", $i)}' <<< "$IN"

This sets the field separator to ; , so that it can loop through the fields with a for loop and print accordingly.

Test
$ IN="bla@some.com;john@home.com"
$ awk -F";" '{for (i=1;i<=NF;i++) printf("> [%s]\n", $i)}' <<< "$IN"
> [bla@some.com]
> [john@home.com]

With another input:

$ awk -F";" '{for (i=1;i<=NF;i++) printf("> [%s]\n", $i)}' <<< "a;b;c   d;e_;f"
> [a]
> [b]
> [c   d]
> [e_]
> [f]

18446744073709551615 , Feb 20, 2015 at 10:49

In Android shell, most of the proposed methods just do not work:
$ IFS=':' read -ra ADDR <<<"$PATH"                             
/system/bin/sh: can't create temporary file /sqlite_stmt_journals/mksh.EbNoR10629: No such file or directory

What does work is:

$ for i in ${PATH//:/ }; do echo $i; done
/sbin
/vendor/bin
/system/sbin
/system/bin
/system/xbin

where // means global replacement.

sorontar , Oct 26, 2016 at 5:08

Fails if any part of $PATH contains spaces (or newlines). Also expands wildcards (asterisk *, question mark ? and braces [ ]). – sorontar Oct 26 '16 at 5:08

Eduardo Lucio , Apr 4, 2016 at 19:54

Okay guys!

Here's my answer!

DELIMITER_VAL='='

read -d '' F_ABOUT_DISTRO_R <<"EOF"
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"
NAME="Ubuntu"
VERSION="14.04.4 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.4 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
EOF

SPLIT_NOW=$(awk -F$DELIMITER_VAL '{for(i=1;i<=NF;i++){printf "%s\n", $i}}' <<<"${F_ABOUT_DISTRO_R}")
while read -r line; do
   SPLIT+=("$line")
done <<< "$SPLIT_NOW"
for i in "${SPLIT[@]}"; do
    echo "$i"
done

Why this approach is "the best" for me?

Because of two reasons:

  1. You do not need to escape the delimiter;
  2. You will not have problem with blank spaces . The value will be properly separated in the array!

[]'s

gniourf_gniourf , Jan 30, 2017 at 8:26

FYI, /etc/os-release and /etc/lsb-release are meant to be sourced, and not parsed. So your method is really wrong. Moreover, you're not quite answering the question about spiltting a string on a delimiter.gniourf_gniourf Jan 30 '17 at 8:26

Michael Hale , Jun 14, 2012 at 17:38

A one-liner to split a string separated by ';' into an array is:
IN="bla@some.com;john@home.com"
ADDRS=( $(IFS=";" echo "$IN") )
echo ${ADDRS[0]}
echo ${ADDRS[1]}

This only sets IFS in a subshell, so you don't have to worry about saving and restoring its value.

Luca Borrione , Sep 3, 2012 at 10:04

-1 this doesn't work here (ubuntu 12.04). it prints only the first echo with all $IN value in it, while the second is empty. you can see it if you put echo "0: "${ADDRS[0]}\n echo "1: "${ADDRS[1]} the output is 0: bla@some.com;john@home.com\n 1: (\n is new line) – Luca Borrione Sep 3 '12 at 10:04

Luca Borrione , Sep 3, 2012 at 10:05

please refer to nickjb's answer at for a working alternative to this idea stackoverflow.com/a/6583589/1032370 – Luca Borrione Sep 3 '12 at 10:05

Score_Under , Apr 28, 2015 at 17:09

-1, 1. IFS isn't being set in that subshell (it's being passed to the environment of "echo", which is a builtin, so nothing is happening anyway). 2. $IN is quoted so it isn't subject to IFS splitting. 3. The process substitution is split by whitespace, but this may corrupt the original data. – Score_Under Apr 28 '15 at 17:09

ajaaskel , Oct 10, 2014 at 11:33

IN='bla@some.com;john@home.com;Charlie Brown <cbrown@acme.com;!"#$%&/()[]{}*? are no problem;simple is beautiful :-)'
set -f
oldifs="$IFS"
IFS=';'; arrayIN=($IN)
IFS="$oldifs"
for i in "${arrayIN[@]}"; do
echo "$i"
done
set +f

Output:

bla@some.com
john@home.com
Charlie Brown <cbrown@acme.com
!"#$%&/()[]{}*? are no problem
simple is beautiful :-)

Explanation: Simple assignment using parenthesis () converts semicolon separated list into an array provided you have correct IFS while doing that. Standard FOR loop handles individual items in that array as usual. Notice that the list given for IN variable must be "hard" quoted, that is, with single ticks.

IFS must be saved and restored since Bash does not treat an assignment the same way as a command. An alternate workaround is to wrap the assignment inside a function and call that function with a modified IFS. In that case separate saving/restoring of IFS is not needed. Thanks for "Bize" for pointing that out.

gniourf_gniourf , Feb 20, 2015 at 16:45

!"#$%&/()[]{}*? are no problem well... not quite: []*? are glob characters. So what about creating this directory and file: `mkdir '!"#$%&'; touch '!"#$%&/()[]{} got you hahahaha - are no problem' and running your command? simple may be beautiful, but when it's broken, it's broken. – gniourf_gniourf Feb 20 '15 at 16:45

ajaaskel , Feb 25, 2015 at 7:20

@gniourf_gniourf The string is stored in a variable. Please see the original question. – ajaaskel Feb 25 '15 at 7:20

gniourf_gniourf , Feb 25, 2015 at 7:26

@ajaaskel you didn't fully understand my comment. Go in a scratch directory and issue these commands: mkdir '!"#$%&'; touch '!"#$%&/()[]{} got you hahahaha - are no problem' . They will only create a directory and a file, with weird looking names, I must admit. Then run your commands with the exact IN you gave: IN='bla@some.com;john@home.com;Charlie Brown <cbrown@acme.com;!"#$%&/()[]{}*? are no problem;simple is beautiful :-)' . You'll see that you won't get the output you expect. Because you're using a method subject to pathname expansions to split your string. – gniourf_gniourf Feb 25 '15 at 7:26

gniourf_gniourf , Feb 25, 2015 at 7:29

This is to demonstrate that the characters * , ? , [...] and even, if extglob is set, !(...) , @(...) , ?(...) , +(...) are problems with this method! – gniourf_gniourf Feb 25 '15 at 7:29

ajaaskel , Feb 26, 2015 at 15:26

@gniourf_gniourf Thanks for detailed comments on globbing. I adjusted the code to have globbing off. My point was however just to show that rather simple assignment can do the splitting job. – ajaaskel Feb 26 '15 at 15:26

> , Dec 19, 2013 at 21:39

Maybe not the most elegant solution, but works with * and spaces:
IN="bla@so me.com;*;john@home.com"
for i in `delims=${IN//[^;]}; seq 1 $((${#delims} + 1))`
do
   echo "> [`echo $IN | cut -d';' -f$i`]"
done

Outputs

> [bla@so me.com]
> [*]
> [john@home.com]

Other example (delimiters at beginning and end):

IN=";bla@so me.com;*;john@home.com;"
> []
> [bla@so me.com]
> [*]
> [john@home.com]
> []

Basically it removes every character other than ; making delims eg. ;;; . Then it does for loop from 1 to number-of-delimiters as counted by ${#delims} . The final step is to safely get the $i th part using cut .

[Nov 08, 2018] Utilizing multi core for tar+gzip-bzip compression-decompression

Nov 08, 2018 | stackoverflow.com

Ask Question up vote 163 down vote favorite 67


user1118764 , Sep 7, 2012 at 6:58

I normally compress using tar zcvf and decompress using tar zxvf (using gzip due to habit).

I've recently gotten a quad core CPU with hyperthreading, so I have 8 logical cores, and I notice that many of the cores are unused during compression/decompression.

Is there any way I can utilize the unused cores to make it faster?

Warren Severin , Nov 13, 2017 at 4:37

The solution proposed by Xiong Chiamiov above works beautifully. I had just backed up my laptop with .tar.bz2 and it took 132 minutes using only one cpu thread. Then I compiled and installed tar from source: gnu.org/software/tar I included the options mentioned in the configure step: ./configure --with-gzip=pigz --with-bzip2=lbzip2 --with-lzip=plzip I ran the backup again and it took only 32 minutes. That's better than 4X improvement! I watched the system monitor and it kept all 4 cpus (8 threads) flatlined at 100% the whole time. THAT is the best solution. – Warren Severin Nov 13 '17 at 4:37

Mark Adler , Sep 7, 2012 at 14:48

You can use pigz instead of gzip, which does gzip compression on multiple cores. Instead of using the -z option, you would pipe it through pigz:
tar cf - paths-to-archive | pigz > archive.tar.gz

By default, pigz uses the number of available cores, or eight if it could not query that. You can ask for more with -p n, e.g. -p 32. pigz has the same options as gzip, so you can request better compression with -9. E.g.

tar cf - paths-to-archive | pigz -9 -p 32 > archive.tar.gz

user788171 , Feb 20, 2013 at 12:43

How do you use pigz to decompress in the same fashion? Or does it only work for compression? – user788171 Feb 20 '13 at 12:43

Mark Adler , Feb 20, 2013 at 16:18

pigz does use multiple cores for decompression, but only with limited improvement over a single core. The deflate format does not lend itself to parallel decompression. The decompression portion must be done serially. The other cores for pigz decompression are used for reading, writing, and calculating the CRC. When compressing on the other hand, pigz gets close to a factor of n improvement with n cores. – Mark Adler Feb 20 '13 at 16:18

Garrett , Mar 1, 2014 at 7:26

The hyphen here is stdout (see this page ). – Garrett Mar 1 '14 at 7:26

Mark Adler , Jul 2, 2014 at 21:29

Yes. 100% compatible in both directions. – Mark Adler Jul 2 '14 at 21:29

Mark Adler , Apr 23, 2015 at 5:23

There is effectively no CPU time spent tarring, so it wouldn't help much. The tar format is just a copy of the input file with header blocks in between files. – Mark Adler Apr 23 '15 at 5:23

Jen , Jun 14, 2013 at 14:34

You can also use the tar flag "--use-compress-program=" to tell tar what compression program to use.

For example use:

tar -c --use-compress-program=pigz -f tar.file dir_to_zip

ranman , Nov 13, 2013 at 10:01

This is an awesome little nugget of knowledge and deserves more upvotes. I had no idea this option even existed and I've read the man page a few times over the years. – ranman Nov 13 '13 at 10:01

Valerio Schiavoni , Aug 5, 2014 at 22:38

Unfortunately by doing so the concurrent feature of pigz is lost. You can see for yourself by executing that command and monitoring the load on each of the cores. – Valerio Schiavoni Aug 5 '14 at 22:38

bovender , Sep 18, 2015 at 10:14

@ValerioSchiavoni: Not here, I get full load on all 4 cores (Ubuntu 15.04 'Vivid'). – bovender Sep 18 '15 at 10:14

Valerio Schiavoni , Sep 28, 2015 at 23:41

On compress or on decompress ? – Valerio Schiavoni Sep 28 '15 at 23:41

Offenso , Jan 11, 2017 at 17:26

I prefer tar - dir_to_zip | pv | pigz > tar.file pv helps me estimate, you can skip it. But still it easier to write and remember. – Offenso Jan 11 '17 at 17:26

Maxim Suslov , Dec 18, 2014 at 7:31

Common approach

There is option for tar program:

-I, --use-compress-program PROG
      filter through PROG (must accept -d)

You can use multithread version of archiver or compressor utility.

Most popular multithread archivers are pigz (instead of gzip) and pbzip2 (instead of bzip2). For instance:

$ tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 paths_to_archive
$ tar --use-compress-program=pigz -cf OUTPUT_FILE.tar.gz paths_to_archive

Archiver must accept -d. If your replacement utility hasn't this parameter and/or you need specify additional parameters, then use pipes (add parameters if necessary):

$ tar cf - paths_to_archive | pbzip2 > OUTPUT_FILE.tar.gz
$ tar cf - paths_to_archive | pigz > OUTPUT_FILE.tar.gz

Input and output of singlethread and multithread are compatible. You can compress using multithread version and decompress using singlethread version and vice versa.

p7zip

For p7zip for compression you need a small shell script like the following:

#!/bin/sh
case $1 in
  -d) 7za -txz -si -so e;;
   *) 7za -txz -si -so a .;;
esac 2>/dev/null

Save it as 7zhelper.sh. Here the example of usage:

$ tar -I 7zhelper.sh -cf OUTPUT_FILE.tar.7z paths_to_archive
$ tar -I 7zhelper.sh -xf OUTPUT_FILE.tar.7z
xz

Regarding multithreaded XZ support. If you are running version 5.2.0 or above of XZ Utils, you can utilize multiple cores for compression by setting -T or --threads to an appropriate value via the environmental variable XZ_DEFAULTS (e.g. XZ_DEFAULTS="-T 0" ).

This is a fragment of man for 5.1.0alpha version:

Multithreaded compression and decompression are not implemented yet, so this option has no effect for now.

However this will not work for decompression of files that haven't also been compressed with threading enabled. From man for version 5.2.2:

Threaded decompression hasn't been implemented yet. It will only work on files that contain multiple blocks with size information in block headers. All files compressed in multi-threaded mode meet this condition, but files compressed in single-threaded mode don't even if --block-size=size is used.

Recompiling with replacement

If you build tar from sources, then you can recompile with parameters

--with-gzip=pigz
--with-bzip2=lbzip2
--with-lzip=plzip

After recompiling tar with these options you can check the output of tar's help:

$ tar --help | grep "lbzip2\|plzip\|pigz"
  -j, --bzip2                filter the archive through lbzip2
      --lzip                 filter the archive through plzip
  -z, --gzip, --gunzip, --ungzip   filter the archive through pigz

> , Apr 28, 2015 at 20:41

This is indeed the best answer. I'll definitely rebuild my tar! – user1985657 Apr 28 '15 at 20:41

mpibzip2 , Apr 28, 2015 at 20:57

I just found pbzip2 and mpibzip2 . mpibzip2 looks very promising for clusters or if you have a laptop and a multicore desktop computer for instance. – user1985657 Apr 28 '15 at 20:57

oᴉɹǝɥɔ , Jun 10, 2015 at 17:39

This is a great and elaborate answer. It may be good to mention that multithreaded compression (e.g. with pigz ) is only enabled when it reads from the file. Processing STDIN may in fact be slower. – oᴉɹǝɥɔ Jun 10 '15 at 17:39

selurvedu , May 26, 2016 at 22:13

Plus 1 for xz option. It the simplest, yet effective approach. – selurvedu May 26 '16 at 22:13

panticz.de , Sep 1, 2014 at 15:02

You can use the shortcut -I for tar's --use-compress-program switch, and invoke pbzip2 for bzip2 compression on multiple cores:
tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 DIRECTORY_TO_COMPRESS/

einpoklum , Feb 11, 2017 at 15:59

A nice TL;DR for @MaximSuslov's answer . – einpoklum Feb 11 '17 at 15:59

,

If you want to have more flexibility with filenames and compression options, you can use:
find /my/path/ -type f -name "*.sql" -o -name "*.log" -exec \
tar -P --transform='s@/my/path/@@g' -cf - {} + | \
pigz -9 -p 4 > myarchive.tar.gz
Step 1: find

find /my/path/ -type f -name "*.sql" -o -name "*.log" -exec

This command will look for the files you want to archive, in this case /my/path/*.sql and /my/path/*.log . Add as many -o -name "pattern" as you want.

-exec will execute the next command using the results of find : tar

Step 2: tar

tar -P --transform='s@/my/path/@@g' -cf - {} +

--transform is a simple string replacement parameter. It will strip the path of the files from the archive so the tarball's root becomes the current directory when extracting. Note that you can't use -C option to change directory as you'll lose benefits of find : all files of the directory would be included.

-P tells tar to use absolute paths, so it doesn't trigger the warning "Removing leading `/' from member names". Leading '/' with be removed by --transform anyway.

-cf - tells tar to use the tarball name we'll specify later

{} + uses everyfiles that find found previously

Step 3: pigz

pigz -9 -p 4

Use as many parameters as you want. In this case -9 is the compression level and -p 4 is the number of cores dedicated to compression. If you run this on a heavy loaded webserver, you probably don't want to use all available cores.

Step 4: archive name

> myarchive.tar.gz

Finally.

[Nov 02, 2018] Working with data streams on the Linux command line by David Both The Linux Philosophy for SysAdmins And Everyone Who Wants To Be One by David Both. It is well worth $32.

Notable quotes:
"... This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface." ..."
Oct 30, 2018 | opensource.com
Author's note: Much of the content in this article is excerpted, with some significant edits to fit the Opensource.com article format, from Chapter 3: Data Streams, of my new book, The Linux Philosophy for SysAdmins .

Everything in Linux revolves around streams of data -- particularly text streams. Data streams are the raw materials upon which the GNU Utilities , the Linux core utilities, and many other command-line tools perform their work.

As its name implies, a data stream is a stream of data -- especially text data -- being passed from one file, device, or program to another using STDIO. This chapter introduces the use of pipes to connect streams of data from one utility program to another using STDIO. You will learn that the function of these programs is to transform the data in some manner. You will also learn about the use of redirection to redirect the data to a file.

The Linux Terminal

I use the term "transform" in conjunction with these programs because the primary task of each is to transform the incoming data from STDIO in a specific way as intended by the sysadmin and to send the transformed data to STDOUT for possible use by another transformer program or redirection to a file.

The standard term, "filters," implies something with which I don't agree. By definition, a filter is a device or a tool that removes something, such as an air filter removes airborne contaminants so that the internal combustion engine of your automobile does not grind itself to death on those particulates. In my high school and college chemistry classes, filter paper was used to remove particulates from a liquid. The air filter in my home HVAC system removes particulates that I don't want to breathe.

Although they do sometimes filter out unwanted data from a stream, I much prefer the term "transformers" because these utilities do so much more. They can add data to a stream, modify the data in some amazing ways, sort it, rearrange the data in each line, perform operations based on the contents of the data stream, and so much more. Feel free to use whichever term you prefer, but I prefer transformers. I expect that I am alone in this.

Data streams can be manipulated by inserting transformers into the stream using pipes. Each transformer program is used by the sysadmin to perform some operation on the data in the stream, thus changing its contents in some manner. Redirection can then be used at the end of the pipeline to direct the data stream to a file. As mentioned, that file could be an actual data file on the hard drive, or a device file such as a drive partition, a printer, a terminal, a pseudo-terminal, or any other device connected to a computer.

The ability to manipulate these data streams using these small yet powerful transformer programs is central to the power of the Linux command-line interface. Many of the core utilities are transformer programs and use STDIO.

In the Unix and Linux worlds, a stream is a flow of text data that originates at some source; the stream may flow to one or more programs that transform it in some way, and then it may be stored in a file or displayed in a terminal session. As a sysadmin, your job is intimately associated with manipulating the creation and flow of these data streams. In this post, we will explore data streams -- what they are, how to create them, and a little bit about how to use them.

Text streams -- a universal interface

The use of Standard Input/Output (STDIO) for program input and output is a key foundation of the Linux way of doing things. STDIO was first developed for Unix and has found its way into most other operating systems since then, including DOS, Windows, and Linux.

" This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."

-- Doug McIlroy, Basics of the Unix Philosophy

STDIO

STDIO was developed by Ken Thompson as a part of the infrastructure required to implement pipes on early versions of Unix. Programs that implement STDIO use standardized file handles for input and output rather than files that are stored on a disk or other recording media. STDIO is best described as a buffered data stream, and its primary function is to stream data from the output of one program, file, or device to the input of another program, file, or device.

There are three STDIO data streams, each of which is automatically opened as a file at the startup of a program -- well, those programs that use STDIO. Each STDIO data stream is associated with a file handle, which is just a set of metadata that describes the attributes of the file. File handles 0, 1, and 2 are explicitly defined by convention and long practice as STDIN, STDOUT, and STDERR, respectively.

STDIN, File handle 0 , is standard input which is usually input from the keyboard. STDIN can be redirected from any file, including device files, instead of the keyboard. It is not common to need to redirect STDIN, but it can be done.

STDOUT, File handle 1 , is standard output which sends the data stream to the display by default. It is common to redirect STDOUT to a file or to pipe it to another program for further processing.

STDERR, File handle 2 . The data stream for STDERR is also usually sent to the display.

If STDOUT is redirected to a file, STDERR continues to be displayed on the screen. This ensures that when the data stream itself is not displayed on the terminal, that STDERR is, thus ensuring that the user will see any errors resulting from execution of the program. STDERR can also be redirected to the same or passed on to the next transformer program in a pipeline.

STDIO is implemented as a C library, stdio.h , which can be included in the source code of programs so that it can be compiled into the resulting executable.

Simple streams

You can perform the following experiments safely in the /tmp directory of your Linux host. As the root user, make /tmp the PWD, create a test directory, and then make the new directory the PWD.

# cd /tmp ; mkdir test ; cd test

Enter and run the following command line program to create some files with content on the drive. We use the dmesg command simply to provide data for the files to contain. The contents don't matter as much as just the fact that each file has some content.

# for I in 0 1 2 3 4 5 6 7 8 9 ; do dmesg > file$I.txt ; done

Verify that there are now at least 10 files in /tmp/ with the names file0.txt through file9.txt .

# ll
total 1320
-rw-r--r-- 1 root root 131402 Oct 17 15:50 file0.txt
-rw-r--r-- 1 root root 131402 Oct 17 15:50 file1.txt
-rw-r--r-- 1 root root 131402 Oct 17 15:50 file2.txt
-rw-r--r-- 1 root root 131402 Oct 17 15:50 file3.txt
-rw-r--r-- 1 root root 131402 Oct 17 15:50 file4.txt
-rw-r--r-- 1 root root 131402 Oct 17 15:50 file5.txt
-rw-r--r-- 1 root root 131402 Oct 17 15:50 file6.txt
-rw-r--r-- 1 root root 131402 Oct 17 15:50 file7.txt
-rw-r--r-- 1 root root 131402 Oct 17 15:50 file8.txt
-rw-r--r-- 1 root root 131402 Oct 17 15:50 file9.txt

We have generated data streams using the dmesg command, which was redirected to a series of files. Most of the core utilities use STDIO as their output stream and those that generate data streams, rather than acting to transform the data stream in some way, can be used to create the data streams that we will use for our experiments. Data streams can be as short as one line or even a single character, and as long as needed.

Exploring the hard drive

It is now time to do a little exploring. In this experiment, we will look at some of the filesystem structures.

Let's start with something simple. You should be at least somewhat familiar with the dd command. Officially known as "disk dump," many sysadmins call it "disk destroyer" for good reason. Many of us have inadvertently destroyed the contents of an entire hard drive or partition using the dd command. That is why we will hang out in the /tmp/test directory to perform some of these experiments.

Despite its reputation, dd can be quite useful in exploring various types of storage media, hard drives, and partitions. We will also use it as a tool to explore other aspects of Linux.

Log into a terminal session as root if you are not already. We first need to determine the device special file for your hard drive using the lsblk command.

[root@studentvm1 test]# lsblk -i
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 60G 0 disk
|-sda1 8:1 0 1G 0 part /boot
`-sda2 8:2 0 59G 0 part
|-fedora_studentvm1-pool00_tmeta 253:0 0 4M 0 lvm
| `-fedora_studentvm1-pool00-tpool 253:2 0 2G 0 lvm
| |-fedora_studentvm1-root 253:3 0 2G 0 lvm /
| `-fedora_studentvm1-pool00 253:6 0 2G 0 lvm
|-fedora_studentvm1-pool00_tdata 253:1 0 2G 0 lvm
| `-fedora_studentvm1-pool00-tpool 253:2 0 2G 0 lvm
| |-fedora_studentvm1-root 253:3 0 2G 0 lvm /
| `-fedora_studentvm1-pool00 253:6 0 2G 0 lvm
|-fedora_studentvm1-swap 253:4 0 10G 0 lvm [SWAP]
|-fedora_studentvm1-usr 253:5 0 15G 0 lvm /usr
|-fedora_studentvm1-home 253:7 0 2G 0 lvm /home
|-fedora_studentvm1-var 253:8 0 10G 0 lvm /var
`-fedora_studentvm1-tmp 253:9 0 5G 0 lvm /tmp
sr0 11:0 1 1024M 0 rom

We can see from this that there is only one hard drive on this host, that the device special file associated with it is /dev/sda , and that it has two partitions. The /dev/sda1 partition is the boot partition, and the /dev/sda2 partition contains a volume group on which the rest of the host's logical volumes have been created.

As root in the terminal session, use the dd command to view the boot record of the hard drive, assuming it is assigned to the /dev/sda device. The bs= argument is not what you might think; it simply specifies the block size, and the count= argument specifies the number of blocks to dump to STDIO. The if= argument specifies the source of the data stream, in this case, the /dev/sda device. Notice that we are not looking at the first block of the partition, we are looking at the very first block of the hard drive.

[root@studentvm1 test]# dd if=/dev/sda bs=512 count=1
�c�#�м���؎���|�#�#���!#��8#u
��#���u��#�#�#�|���t#�L#�#�|���#�����?t��pt#���y|1��؎м ��d|<�t#��R�|1��D#@�D��D#�##f�#\|f�f�#`|f�\
�D#p�B�#r�p�#�K`#�#��1��������#a`���#f��u#����f1�f�TCPAf�#f�#a�&Z|�#}�#�.}�4�3}�.�#��GRUB GeomHard DiskRead Error
�#��#�<u��ܻޮ�###��� ������ �_U�1+0 records in
1+0 records out
512 bytes copied, 4.3856e-05 s, 11.7 MB/s

This prints the text of the boot record, which is the first block on the disk -- any disk. In this case, there is information about the filesystem and, although it is unreadable because it is stored in binary format, the partition table. If this were a bootable device, stage 1 of GRUB or some other boot loader would be located in this sector. The last three lines contain data about the number of records and bytes processed.

Starting with the beginning of /dev/sda1 , let's look at a few blocks of data at a time to find what we want. The command is similar to the previous one, except that we have specified a few more blocks of data to view. You may have to specify fewer blocks if your terminal is not large enough to display all of the data at one time, or you can pipe the data through the less utility and use that to page through the data -- either way works. Remember, we are doing all of this as root user because non-root users do not have the required permissions.

Enter the same command as you did in the previous experiment, but increase the block count to be displayed to 100, as shown below, in order to show more data.

[root@studentvm1 test]# dd if=/dev/sda1 bs=512 count=100
##33��#:�##�� :o�[:o�[#��S�###�q[#
#<�#{5OZh�GJ͞#t�Ұ##boot/bootysimage/booC�dp��G'�*)�#A�##@
#�q[
�## ## ###�#���To=###<#8���#'#�###�#�����#�' �����#Xi �#��` qT���
<���
� r���� ]�#�#�##�##�##�#�##�##�##�#�##�##�#��#�#�##�#�##�##�#��#�#����# � �# �# �#

�#
�#
�#

�#
�#
�#

�#
�#
�#100+0 records in
100+0 records out
51200 bytes (51 kB, 50 KiB) copied, 0.00117615 s, 43.5 MB/s

Now try this command. I won't reproduce the entire data stream here because it would take up huge amounts of space. Use Ctrl-C to break out and stop the stream of data.

[root@studentvm1 test]# dd if=/dev/sda

This command produces a stream of data that is the complete content of the hard drive, /dev/sda , including the boot record, the partition table, and all of the partitions and their content. This data could be redirected to a file for use as a complete backup from which a bare metal recovery can be performed. It could also be sent directly to another hard drive to clone the first. But do not perform this particular experiment.

[root@studentvm1 test]# dd if=/dev/sda of=/dev/sdx

You can see that the dd command can be very useful for exploring the structures of various types of filesystems, locating data on a defective storage device, and much more. It also produces a stream of data on which we can use the transformer utilities in order to modify or view.

The real point here is that dd , like so many Linux commands, produces a stream of data as its output. That data stream can be searched and manipulated in many ways using other tools. It can even be used for ghost-like backups or disk duplication.

Randomness

It turns out that randomness is a desirable thing in computers -- who knew? There are a number of reasons that sysadmins might want to generate a stream of random data. A stream of random data is sometimes useful to overwrite the contents of a complete partition, such as /dev/sda1 , or even the entire hard drive, as in /dev/sda .

Perform this experiment as a non-root user. Enter this command to print an unending stream of random data to STDIO.

[student@studentvm1 ~]$ cat /dev/urandom

Use Ctrl-C to break out and stop the stream of data. You may need to use Ctrl-C multiple times.

Random data is also used as the input seed to programs that generate random passwords and random data and numbers for use in scientific and statistical calculations. I will cover randomness and other interesting data sources in a bit more detail in Chapter 24: Everything is a file.

Pipe dreams

Pipes are critical to our ability to do the amazing things on the command line, so much so that I think it is important to recognize that they were invented by Douglas McIlroy during the early days of Unix (thanks, Doug!). The Princeton University website has a fragment of an interview with McIlroy in which he discusses the creation of the pipe and the beginnings of the Unix philosophy.

Notice the use of pipes in the simple command-line program shown next, which lists each logged-in user a single time, no matter how many logins they have active. Perform this experiment as the student user. Enter the command shown below:

[student@studentvm1 ~]$ w | tail -n +3 | awk '{print $1}' | sort | uniq
root
student
[student@studentvm1 ~]$

The results from this command produce two lines of data that show that the user's root and student are both logged in. It does not show how many times each user is logged in. Your results will almost certainly differ from mine.

Pipes -- represented by the vertical bar ( | ) -- are the syntactical glue, the operator, that connects these command-line utilities together. Pipes allow the Standard Output from one command to be "piped," i.e., streamed from Standard Output of one command to the Standard Input of the next command.

The |& operator can be used to pipe the STDERR along with STDOUT to STDIN of the next command. This is not always desirable, but it does offer flexibility in the ability to record the STDERR data stream for the purposes of problem determination.

A string of programs connected with pipes is called a pipeline, and the programs that use STDIO are referred to officially as filters, but I prefer the term "transformers."

Think about how this program would have to work if we could not pipe the data stream from one command to the next. The first command would perform its task on the data and then the output from that command would need to be saved in a file. The next command would have to read the stream of data from the intermediate file and perform its modification of the data stream, sending its own output to a new, temporary data file. The third command would have to take its data from the second temporary data file and perform its own manipulation of the data stream and then store the resulting data stream in yet another temporary file. At each step, the data file names would have to be transferred from one command to the next in some way.

I cannot even stand to think about that because it is so complex. Remember: Simplicity rocks!

Building pipelines

When I am doing something new, solving a new problem, I usually do not just type in a complete Bash command pipeline from scratch off the top of my head. I usually start with just one or two commands in the pipeline and build from there by adding more commands to further process the data stream. This allows me to view the state of the data stream after each of the commands in the pipeline and make corrections as they are needed.

It is possible to build up very complex pipelines that can transform the data stream using many different utilities that work with STDIO.

Redirection

Redirection is the capability to redirect the STDOUT data stream of a program to a file instead of to the default target of the display. The "greater than" ( > ) character, aka "gt", is the syntactical symbol for redirection of STDOUT.

Redirecting the STDOUT of a command can be used to create a file containing the results from that command.

[student@studentvm1 ~]$ df -h > diskusage.txt

There is no output to the terminal from this command unless there is an error. This is because the STDOUT data stream is redirected to the file and STDERR is still directed to the STDOUT device, which is the display. You can view the contents of the file you just created using this next command:

[student@studentvm1 test]# cat diskusage.txt
Filesystem Size Used Avail Use% Mounted on
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 1.2M 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mapper/fedora_studentvm1-root 2.0G 50M 1.8G 3% /
/dev/mapper/fedora_studentvm1-usr 15G 4.5G 9.5G 33% /usr
/dev/mapper/fedora_studentvm1-var 9.8G 1.1G 8.2G 12% /var
/dev/mapper/fedora_studentvm1-tmp 4.9G 21M 4.6G 1% /tmp
/dev/mapper/fedora_studentvm1-home 2.0G 7.2M 1.8G 1% /home
/dev/sda1 976M 221M 689M 25% /boot
tmpfs 395M 0 395M 0% /run/user/0
tmpfs 395M 12K 395M 1% /run/user/1000

When using the > symbol to redirect the data stream, the specified file is created if it does not already exist. If it does exist, the contents are overwritten by the data stream from the command. You can use double greater-than symbols, >>, to append the new data stream to any existing content in the file.

[student@studentvm1 ~]$ df -h >> diskusage.txt

You can use cat and/or less to view the diskusage.txt file in order to verify that the new data was appended to the end of the file.

The < (less than) symbol redirects data to the STDIN of the program. You might want to use this method to input data from a file to STDIN of a command that does not take a filename as an argument but that does use STDIN. Although input sources can be redirected to STDIN, such as a file that is used as input to grep, it is generally not necessary as grep also takes a filename as an argument to specify the input source. Most other commands also take a filename as an argument for their input source.

Just grep'ing around

The grep command is used to select lines that match a specified pattern from a stream of data. grep is one of the most commonly used transformer utilities and can be used in some very creative and interesting ways. The grep command is one of the few that can correctly be called a filter because it does filter out all the lines of the data stream that you do not want; it leaves only the lines that you do want in the remaining data stream.

If the PWD is not the /tmp/test directory, make it so. Let's first create a stream of random data to store in a file. In this case, we want somewhat less random data that would be limited to printable characters. A good password generator program can do this. The following program (you may have to install pwgen if it is not already) creates a file that contains 50,000 passwords that are 80 characters long using every printable character. Try it without redirecting to the random.txt file first to see what that looks like, and then do it once redirecting the output data stream to the file.

$ pwgen -sy 80 50000 > random.txt

Considering that there are so many passwords, it is very likely that some character strings in them are the same. First, cat the random.txt file, then use the grep command to locate some short, randomly selected strings from the last ten passwords on the screen. I saw the word "see" in one of those ten passwords, so my command looked like this: grep see random.txt , and you can try that, but you should also pick some strings of your own to check. Short strings of two to four characters work best.

$ grep see random.txt
R=p)'s/~0}wr~2(OqaL.S7DNyxlmO69`"12u]h@rp[D2%3}1b87+>Vk,;4a0hX]d7see;1%9|wMp6Yl.
bSM_mt_hPy|YZ1<TY/Hu5{g#mQ<u_(@8B5Vt?w%i-&C>NU@[;zV2-see)>(BSK~n5mmb9~h)yx{a&$_e
cjR1QWZwEgl48[3i-(^x9D=v)seeYT2R#M:>wDh?Tn$]HZU7}j!7bIiIr^cI.DI)W0D"'vZU@.Kxd1E1
z=tXcjVv^G\nW`,y=bED]d|7%s6iYT^a^Bvsee:v\UmWT02|P|nq%A*;+Ng[$S%*s)-ls"dUfo|0P5+n Summary

It is the use of pipes and redirection that allows many of the amazing and powerful tasks that can be performed with data streams on the Linux command line. It is pipes that transport STDIO data streams from one program or file to another. The ability to pipe streams of data through one or more transformer programs supports powerful and flexible manipulation of data in those streams.

Each of the programs in the pipelines demonstrated in the experiments is small, and each does one thing well. They are also transformers; that is, they take Standard Input, process it in some way, and then send the result to Standard Output. Implementation of these programs as transformers to send processed data streams from their own Standard Output to the Standard Input of the other programs is complementary to, and necessary for, the implementation of pipes as a Linux tool.

STDIO is nothing more than streams of data. This data can be almost anything from the output of a command to list the files in a directory, or an unending stream of data from a special device like /dev/urandom , or even a stream that contains all of the raw data from a hard drive or a partition.

Any device on a Linux computer can be treated like a data stream. You can use ordinary tools like dd and cat to dump data from a device into a STDIO data stream that can be processed using other ordinary Linux tools.

Topics Linux Command line

David Both is a Linux and Open Source advocate who resides in Raleigh, North Carolina. He has been in the IT industry for over forty years and taught OS/2 for IBM where he worked for over 20 years. While at IBM, he wrote the first training course for the original IBM PC in 1981. He has taught RHCE classes for Red Hat and has worked at MCI Worldcom, Cisco, and the State of North Carolina. He has been working with Linux and Open Source Software for almost 20 years. David has written articles for...

[Oct 29, 2018] Getting all the matches with 'grep -f' option

Perverted example, but interesting question.
Oct 29, 2018 | stackoverflow.com

Arturo ,Mar 24, 2017 at 8:59

I would like to find all the matches of the text I have in one file ('file1.txt') that are found in another file ('file2.txt') using the grep option -f, that tells to read the expressions to be found from file.

'file1.txt'

a

a

'file2.txt'

a

When I run the command:

grep -f file1.txt file2.txt -w

I get only once the output of the 'a'. instead I would like to get it twice, because it occurs twice in my 'file1.txt' file. Is there a way to let grep (or any other unix/linux) tool to output a match for each line it reads? Thanks in advance. Arturo

RomanPerekhrest ,Mar 24, 2017 at 9:02

the matches of the text - some exact text? should it compare line to line? – RomanPerekhrest Mar 24 '17 at 9:02

Arturo ,Mar 24, 2017 at 9:04

Yes it contains exact match. I added the -w options, following your input. Yes, it is a comparison line by line. – Arturo Mar 24 '17 at 9:04

Remko ,Mar 24, 2017 at 9:19

Grep works as designed, giving only one output line. You could use another approach:
while IFS= read -r pattern; do
    grep -e $pattern file2.txt
done < file1.txt

This would use every line in file1.txt as a pattern for the grep, thus resulting in the output you're looking for.

Arturo ,Mar 24, 2017 at 9:30

That did the trick!. Thank you. And it is even much faster than my previous grep command. – Arturo Mar 24 '17 at 9:30

ar7 ,Mar 24, 2017 at 9:12

When you use
grep -f pattern.txt file.txt

It means match the pattern found in pattern.txt in the file file.txt .

It is giving you only one output because that is all is there in the second file.

Try interchanging the files,

grep -f file2.txt file1.txt -w

Does this answer your question?

Arturo ,Mar 24, 2017 at 9:17

I understand that, but still I would like to find a way to print a match each time a pattern (even a repeated one) from 'pattern.txt' is found in 'file.txt'. Even a tool or a script rather then 'grep -f' would suffice. – Arturo Mar 24 '17 at 9:17

[Oct 23, 2018] To switch from vertical split to horizontal split fast in Vim

Nov 24, 2013 | stackoverflow.com

ДМИТРИЙ МАЛИКОВ, Nov 24, 2013 at 7:55

How can you switch your current windows from horizontal split to vertical split and vice versa in Vim?

I did that a moment ago by accident but I cannot find the key again.

Mark Rushakoff

Vim mailing list says (re-formatted for better readability):

  • To change two vertically split windows to horizontal split: Ctrl - W t Ctrl - W K
  • Horizontally to vertically: Ctrl - W t Ctrl - W H

Explanations:

  • Ctrl - W t -- makes the first (topleft) window current
  • Ctrl - W K -- moves the current window to full-width at the very top
  • Ctrl - W H -- moves the current window to full-height at far left

Note that the t is lowercase, and the K and H are uppercase.

Also, with only two windows, it seems like you can drop the Ctrl - W t part because if you're already in one of only two windows, what's the point of making it current?

Too much php Aug 13 '09 at 2:17

So if you have two windows split horizontally, and you are in the lower window, you just use ^WL

Alex Hart Dec 7 '12 at 14:10

There are a ton of interesting ^w commands (b, w, etc)

holms Feb 28 '13 at 9:07

somehow doesn't work for me.. =/ –

Lambart Mar 26 at 19:34

Just toggle your NERDTree panel closed before 'rotating' the splits, then toggle it back open. :NERDTreeToggle (I have it mapped to a function key for convenience).

xxx Feb 19 '13 at 20:26

^w followed by capital H , J , K or L will move the current window to the far left , bottom , top or right respectively like normal cursor navigation.

The lower case equivalents move focus instead of moving the window.

respectTheCode, Jul 21 '13 at 9:55

1 Wow, cool! Thanks! :-) – infous Feb 6 at 8:46 it's much better since users use hjkl to move between buffers. – Afshin Mehrabani

In VIM, take a look at the following to see different alternatives for what you might have done:

:help opening-window

For instance:

Ctrl - W s
Ctrl - W o
Ctrl - W v
Ctrl - W o
Ctrl - W s

Anon, Apr 29 at 21:45

The command ^W-o is great! I did not know it. – Masi Aug 13 '09 at 2:20 add a comment | up vote 6 down vote The following ex commands will (re-)split any number of windows:

If there are hidden buffers, issuing these commands will also make the hidden buffers visible.

Mark Oct 22 at 19:31

When you have two or more windows open horizontally or vertically and want to switch them all to the other orientation, you can use the following:

[Oct 22, 2018] move selection to a separate file

Highly recommended!
Oct 22, 2018 | superuser.com

greg0ire ,Jan 23, 2013 at 13:29

With vim, how can I move a piece of text to a new file? For the moment, I do this:

Is there a more efficient way to do this?

Before

a.txt

sometext
some other text
some other other text
end
After

a.txt

sometext
end

b.txt

some other text
some other other text

Ingo Karkat, Jan 23, 2013 at 15:20

How about these custom commands:
:command! -bang -range -nargs=1 -complete=file MoveWrite  <line1>,<line2>write<bang> <args> | <line1>,<line2>delete _
:command! -bang -range -nargs=1 -complete=file MoveAppend <line1>,<line2>write<bang> >> <args> | <line1>,<line2>delete _

greg0ire ,Jan 23, 2013 at 15:27

This is very ugly, but hey, it seems to do in one step exactly what I asked for (I tried). +1, and accepted. I was looking for a native way to do this quickly but since there does not seem to be one, yours will do just fine. Thanks! – greg0ire Jan 23 '13 at 15:27

Ingo Karkat ,Jan 23, 2013 at 16:15

Beauty is in the eye of the beholder. I find this pretty elegant; you only need to type it once (into your .vimrc). – Ingo Karkat Jan 23 '13 at 16:15

greg0ire ,Jan 23, 2013 at 16:21

You're right, "very ugly" shoud have been "very unfamiliar". Your command is very handy, and I think I definitely going to carve it in my .vimrc – greg0ire Jan 23 '13 at 16:21

embedded.kyle ,Jan 23, 2013 at 14:08

By "move a piece of text to a new file" I assume you mean cut that piece of text from the current file and create a new file containing only that text.

Various examples:

The above only copies the text and creates a new file containing that text. You will then need to delete afterward.

This can be done using the same range and the d command:

Or by using dd for the single line case.

If you instead select the text using visual mode, and then hit : while the text is selected, you will see the following on the command line:

:'<,'>

Which indicates the selected text. You can then expand the command to:

:'<,'>w >> old_file

Which will append the text to an existing file. Then delete as above.


One liner:

:2,3 d | new +put! "

The breakdown:

greg0ire, Jan 23, 2013 at 14:09

Your assumption is right. This looks good, I'm going to test. Could you explain 2. a bit more? I'm not very familiar with ranges. EDIT: If I try this on the second line, it writes the first line to the other file, not the second line. – greg0ire Jan 23 '13 at 14:09

embedded.kyle ,Jan 23, 2013 at 14:16

@greg0ire I got that a bit backward, I'll edit to better explain – embedded.kyle Jan 23 '13 at 14:16

greg0ire ,Jan 23, 2013 at 14:18

I added an example to make my question clearer. – greg0ire Jan 23 '13 at 14:18

embedded.kyle ,Jan 23, 2013 at 14:22

@greg0ire I corrected my answer. It's still two steps. The first copies and writes. The second deletes. – embedded.kyle Jan 23 '13 at 14:22

greg0ire ,Jan 23, 2013 at 14:41

Ok, if I understand well, the trick is to use ranges to select and write in the same command. That's very similar to what I did. +1 for the detailed explanation, but I don't think this is more efficient, since the trick with hitting ':' is what I do for the moment. – greg0ire Jan 23 '13 at 14:41

Xyon ,Jan 23, 2013 at 13:32

Select the text in visual mode, then press y to "yank" it into the buffer (copy) or d to "delete" it into the buffer (cut).

Then you can :split <new file name> to split your vim window up, and press p to paste in the yanked text. Write the file as normal.

To close the split again, pass the split you want to close :q .

greg0ire ,Jan 23, 2013 at 13:42

I have 4 steps for the moment: select, write, select, delete. With your method, I have 6 steps: select, delete, split, paste, write, close. I asked for something more efficient :P – greg0ire Jan 23 '13 at 13:42

Xyon ,Jan 23, 2013 at 13:44

Well, if you pass the split :x instead, you can combine writing and closing into one and make it five steps. :P – Xyon Jan 23 '13 at 13:44

greg0ire ,Jan 23, 2013 at 13:46

That's better, but 5 still > 4 :P – greg0ire Jan 23 '13 at 13:46 Based on @embedded.kyle's answer and this Q&A , I ended up with this one liner to append a selection to a file and delete from current file. After selecting some lines with Shift+V , hit : and run:
'<,'>w >> test | normal gvd

The first part appends selected lines. The second command enters normal mode and runs gvd to select the last selection and then deletes.

[Oct 22, 2018] linux - If I rm -rf a symlink will the data the link points to get erased, to

Notable quotes:
"... Put it in another words, those symlink-files will be deleted. The files they "point"/"link" to will not be touch. ..."
Oct 22, 2018 | unix.stackexchange.com

user4951 ,Jan 25, 2013 at 2:40

This is the contents of the /home3 directory on my system:
./   backup/    hearsttr@  lost+found/  randomvi@  sexsmovi@
../  freemark@  investgr@  nudenude@    romanced@  wallpape@

I want to clean this up but I am worried because of the symlinks, which point to another drive.

If I say rm -rf /home3 will it delete the other drive?

John Sui

rm -rf /home3 will delete all files and directory within home3 and home3 itself, which include symlink files, but will not "follow"(de-reference) those symlink.

Put it in another words, those symlink-files will be deleted. The files they "point"/"link" to will not be touch.

[Oct 22, 2018] Does rm -rf follow symbolic links?

Jan 25, 2012 | superuser.com
I have a directory like this:
$ ls -l
total 899166
drwxr-xr-x 12 me scicomp       324 Jan 24 13:47 data
-rw-r--r--  1 me scicomp     84188 Jan 24 13:47 lod-thin-1.000000-0.010000-0.030000.rda
drwxr-xr-x  2 me scicomp       808 Jan 24 13:47 log
lrwxrwxrwx  1 me scicomp        17 Jan 25 09:41 msg -> /home/me/msg

And I want to remove it using rm -r .

However I'm scared rm -r will follow the symlink and delete everything in that directory (which is very bad).

I can't find anything about this in the man pages. What would be the exact behavior of running rm -rf from a directory above this one?

LordDoskias Jan 25 '12 at 16:43, Jan 25, 2012 at 16:43

How hard it is to create a dummy dir with a symlink pointing to a dummy file and execute the scenario? Then you will know for sure how it works! –

hakre ,Feb 4, 2015 at 13:09

X-Ref: If I rm -rf a symlink will the data the link points to get erased, too? ; Deleting a folder that contains symlinkshakre Feb 4 '15 at 13:09

Susam Pal ,Jan 25, 2012 at 16:47

Example 1: Deleting a directory containing a soft link to another directory.
susam@nifty:~/so$ mkdir foo bar
susam@nifty:~/so$ touch bar/a.txt
susam@nifty:~/so$ ln -s /home/susam/so/bar/ foo/baz
susam@nifty:~/so$ tree
.
├── bar
│   └── a.txt
└── foo
    └── baz -> /home/susam/so/bar/

3 directories, 1 file
susam@nifty:~/so$ rm -r foo
susam@nifty:~/so$ tree
.
└── bar
    └── a.txt

1 directory, 1 file
susam@nifty:~/so$

So, we see that the target of the soft-link survives.

Example 2: Deleting a soft link to a directory

susam@nifty:~/so$ ln -s /home/susam/so/bar baz
susam@nifty:~/so$ tree
.
├── bar
│   └── a.txt
└── baz -> /home/susam/so/bar

2 directories, 1 file
susam@nifty:~/so$ rm -r baz
susam@nifty:~/so$ tree
.
└── bar
    └── a.txt

1 directory, 1 file
susam@nifty:~/so$

Only, the soft link is deleted. The target of the soft-link survives.

Example 3: Attempting to delete the target of a soft-link

susam@nifty:~/so$ ln -s /home/susam/so/bar baz
susam@nifty:~/so$ tree
.
├── bar
│   └── a.txt
└── baz -> /home/susam/so/bar

2 directories, 1 file
susam@nifty:~/so$ rm -r baz/
rm: cannot remove 'baz/': Not a directory
susam@nifty:~/so$ tree
.
├── bar
└── baz -> /home/susam/so/bar

2 directories, 0 files

The file in the target of the symbolic link does not survive.

The above experiments were done on a Debian GNU/Linux 9.0 (stretch) system.

Wyrmwood ,Oct 30, 2014 at 20:36

rm -rf baz/* will remove the contents – Wyrmwood Oct 30 '14 at 20:36

Buttle Butkus ,Jan 12, 2016 at 0:35

Yes, if you do rm -rf [symlink], then the contents of the original directory will be obliterated! Be very careful. – Buttle Butkus Jan 12 '16 at 0:35

frnknstn ,Sep 11, 2017 at 10:22

Your example 3 is incorrect! On each system I have tried, the file a.txt will be removed in that scenario. – frnknstn Sep 11 '17 at 10:22

Susam Pal ,Sep 11, 2017 at 15:20

@frnknstn You are right. I see the same behaviour you mention on my latest Debian system. I don't remember on which version of Debian I performed the earlier experiments. In my earlier experiments on an older version of Debian, either a.txt must have survived in the third example or I must have made an error in my experiment. I have updated the answer with the current behaviour I observe on Debian 9 and this behaviour is consistent with what you mention. – Susam Pal Sep 11 '17 at 15:20

Ken Simon ,Jan 25, 2012 at 16:43

Your /home/me/msg directory will be safe if you rm -rf the directory from which you ran ls. Only the symlink itself will be removed, not the directory it points to.

The only thing I would be cautious of, would be if you called something like "rm -rf msg/" (with the trailing slash.) Do not do that because it will remove the directory that msg points to, rather than the msg symlink itself.

> ,Jan 25, 2012 at 16:54

"The only thing I would be cautious of, would be if you called something like "rm -rf msg/" (with the trailing slash.) Do not do that because it will remove the directory that msg points to, rather than the msg symlink itself." - I don't find this to be true. See the third example in my response below. – Susam Pal Jan 25 '12 at 16:54

Andrew Crabb ,Nov 26, 2013 at 21:52

I get the same result as @Susam ('rm -r symlink/' does not delete the target of symlink), which I am pleased about as it would be a very easy mistake to make. – Andrew Crabb Nov 26 '13 at 21:52

,

rm should remove files and directories. If the file is symbolic link, link is removed, not the target. It will not interpret a symbolic link. For example what should be the behavior when deleting 'broken links'- rm exits with 0 not with non-zero to indicate failure

[Oct 21, 2018] Common visual block selection scenarios

Notable quotes:
"... column oriented ..."
Oct 21, 2018 | stackoverflow.com
You are talking about text selecting and copying, I think that you should give a look to the Vim Visual Mode .

In the visual mode, you are able to select text using Vim commands, then you can do whatever you want with the selection.

Consider the following common scenarios:

You need to select to the next matching parenthesis.

You could do:

You want to select text between quotes:

You want to select a curly brace block (very common on C-style languages):

You want to select the entire file:

Visual block selection is another really useful feature, it allows you to select a rectangular area of text, you just have to press Ctrl - V to start it, and then select the text block you want and perform any type of operation such as yank, delete, paste, edit, etc. It's great to edit column oriented text.

[Oct 21, 2018] Moving lines between split windows in vim

Notable quotes:
"... "send the line I am on (or the test I selected) to the other window" ..."
Oct 21, 2018 | superuser.com

brad ,Nov 24, 2015 at 12:28

I have two files, say a.txt and b.txt , in the same session of vim and I split the screen so I have file a.txt in the upper window and b.txt in the lower window.

I want to move lines here and there from a.txt to b.txt : I select a line with Shift + v , then I move to b.txt in the lower window with Ctrl + w , paste with p , get back to a.txt with Ctrl + w and I can repeat the operation when I get to another line I want to move.

My question: is there a quicker way to say vim "send the line I am on (or the test I selected) to the other window" ?

Chong ,Nov 24, 2015 at 12:33

Use q macro? q[some_letter] [whatever operations] q , then call the macro with [times to be called]@qChong Nov 24 '15 at 12:33

Anthony Geoghegan ,Nov 24, 2015 at 13:00

I presume that you're deleting the line that you've selected in a.txt . If not, you'd be pasting something else into b.txt . If so, there's no need to select the line first. – Anthony Geoghegan Nov 24 '15 at 13:00

Anthony Geoghegan ,Nov 24, 2015 at 13:17

This sounds like a good use case for a macro. Macros are commands that can be recorded and stored in a Vim register. Each register is identified by a letter from a to z. Recording

From Recording keys for repeated jobs - Vim Tips

To start recording, press q in Normal mode followed by a letter (a to z). That starts recording keystrokes to the specified register. Vim displays "recording" in the status line. Type any Normal mode commands, or enter Insert mode and type text. To stop recording, again press q while in Normal mode.

For this particular macro, I chose the m (for move) register to store it.

I pressed qm to record the following commands:

When I typed q to finish recording the macro, the contents of the m register were:

dd^Wjp^Wk
Usage

brad ,Nov 24, 2015 at 14:26

I asked to see if there is a command unknown to me that does the job: it seems there is none. In absence of such a command, this can be a good solution. – brad Nov 24 '15 at 14:26

romainl ,Nov 26, 2015 at 9:54

@brad, you can find all the commands available to you in the documentation. If it's not there it doesn't exist no need to ask random strangers. – romainl Nov 26 '15 at 9:54

brad ,Nov 26, 2015 at 10:17

@romainl, yes, I know this but vim documentation is really huge and, although it doesn't scare me, there is always the possibility to miss something. Moreover, it could also be that you can obtain the effect using the combination of 2 commands and in this case it would be hardly documented – brad Nov 26 '15 at 10:17

[Oct 21, 2018] How to move around buffers in vim?

Oct 21, 2018 | stackoverflow.com

user3721893 ,Jul 23, 2014 at 5:43

I normally work with more than 5 files at a time. I use buffers to open different files. I use commands such as :buf file1, :buf file2 etc. Is there a faster way to move to different files?

eckes ,Jul 23, 2014 at 5:49

What I use:

And have a short look on :he buffer

And the wiki entry on Easier Buffer Switching on the Vim Wiki: http://vim.wikia.com/wiki/Easier_buffer_switching

SO already has a question regarding yours: How do you prefer to switch between buffers in Vim?

romainl ,Jul 23, 2014 at 6:13

A few mappings can make your life a lot easier.

This one lists your buffers and prompts you for a number:

nnoremap gb :buffers<CR>:buffer<Space>

This one lists your buffers in the "wildmenu". Depends on the 'wildcharm' option as well as 'wildmenu' and 'wildmode' :

nnoremap <leader>b :buffer <C-z>

These ones allow you to cycle between all your buffers without much thinking:

nnoremap <PageUp>   :bprevious<CR>
nnoremap <PageDown> :bnext<CR>

Also, don't forget <C-^> which allows you to alternate between two buffers.

mikew ,Jul 23, 2014 at 6:38

Once the buffers are already open, you can just type :b partial_filename to switch

So if :ls shows that i have my ~./vimrc open, then I can just type :b vimr or :b rc to switch to that buffer

Brady Trainor ,Jul 25, 2014 at 22:13

Below I describe some excerpts from sections of my .vimrc . It includes mapping the leader key, setting wilds tab completion, and finally my buffer nav key choices (all mostly inspired by folks on the interweb, including romainl). Edit: Then I ramble on about my shortcuts for windows and tabs.
" easier default keys {{{1

let mapleader=','
nnoremap <leader>2 :@"<CR>

The leader key is a prefix key for mostly user-defined key commands (some plugins also use it). The default is \ , but many people suggest the easier to reach , .

The second line there is a command to @ execute from the " clipboard, in case you'd like to quickly try out various key bindings (without relying on :so % ). (My nmeumonic is that Shift - 2 is @ .)

" wilds {{{1

set wildmenu wildmode=list:full
set wildcharm=<C-z>
set wildignore+=*~ wildignorecase

For built-in completion, wildmenu is probably the part that shows up yellow on your Vim when using tab completion on command-line. wildmode is set to a comma-separated list, each coming up in turn on each tab completion (that is, my list is simply one element, list:full ). list shows rows and columns of candidates. full 's meaning includes maintaining existence of the wildmenu . wildcharm is the way to include Tab presses in your macros. The *~ is for my use in :edit and :find commands.

" nav keys {{{1
" windows, buffers and tabs {{{2
" buffers {{{3

nnoremap <leader>bb :b <C-z><S-Tab>
nnoremap <leader>bh :ls!<CR>:b<Space>
nnoremap <leader>bw :ls!<CR>:bw<Space>
nnoremap <leader>bt :TSelectBuffer<CR>
nnoremap <leader>be :BufExplorer<CR>
nnoremap <leader>bs :BufExplorerHorizontalSplit<CR>
nnoremap <leader>bv :BufExplorerVerticalSplit<CR>
nnoremap <leader>3 :e#<CR>
nmap <C-n> :bn<cr>
nmap <C-p> :bp<cr>

The ,3 is for switching between the "two" last buffers (Easier to reach than built-in Ctrl - 6 ). Nmeuonic is Shift - 3 is # , and # is the register symbol for last buffer. (See :marks .)

,bh is to select from hidden buffers ( ! ).

,bw is to bwipeout buffers by number or name. For instance, you can wipeout several while looking at the list, with ,bw 1 3 4 8 10 <CR> . Note that wipeout is more destructive than :bdelete . They have their pros and cons. For instance, :bdelete leaves the buffer in the hidden list, while :bwipeout removes global marks (see :help marks , and the description of uppercase marks).

I haven't settled on these keybindings, I would sort of prefer that my ,bb was simply ,b (simply defining while leaving the others defined makes Vim pause to see if you'll enter more).

Those shortcuts for :BufExplorer are actually the defaults for that plugin, but I have it written out so I can change them if I want to start using ,b without a hang.

You didn't ask for this:

If you still find Vim buffers a little awkward to use, try to combine the functionality with tabs and windows (until you get more comfortable?).

" windows {{{3

" window nav
nnoremap <leader>w <C-w>
nnoremap <M-h> <C-w>h
nnoremap <M-j> <C-w>j
nnoremap <M-k> <C-w>k
nnoremap <M-l> <C-w>l
" resize window
nnoremap <C-h> <C-w><
nnoremap <C-j> <C-w>+
nnoremap <C-k> <C-w>-
nnoremap <C-l> <C-w>>

Notice how nice ,w is for a prefix. Also, I reserve Ctrl key for resizing, because Alt ( M- ) is hard to realize in all environments, and I don't have a better way to resize. I'm fine using ,w to switch windows.

" tabs {{{3

nnoremap <leader>t :tab
nnoremap <M-n> :tabn<cr>
nnoremap <M-p> :tabp<cr>
nnoremap <C-Tab> :tabn<cr>
nnoremap <C-S-Tab> :tabp<cr>
nnoremap tn :tabe<CR>
nnoremap te :tabe<Space><C-z><S-Tab>
nnoremap tf :tabf<Space>
nnoremap tc :tabc<CR>
nnoremap to :tabo<CR>
nnoremap tm :tabm<CR>
nnoremap ts :tabs<CR>

nnoremap th :tabr<CR>
nnoremap tj :tabn<CR>
nnoremap tk :tabp<CR>
nnoremap tl :tabl<CR>

" or, it may make more sense to use
" nnoremap th :tabp<CR>
" nnoremap tj :tabl<CR>
" nnoremap tk :tabr<CR>
" nnoremap tl :tabn<CR>

In summary of my window and tabs keys, I can navigate both of them with Alt , which is actually pretty easy to reach. In other words:

" (modifier) key choice explanation {{{3
"
"       KEYS        CTRL                  ALT            
"       hjkl        resize windows        switch windows        
"       np          switch buffer         switch tab      
"
" (resize windows is hard to do otherwise, so we use ctrl which works across
" more environments. i can use ',w' for windowcmds o.w.. alt is comfortable
" enough for fast and gui nav in tabs and windows. we use np for navs that 
" are more linear, hjkl for navs that are more planar.) 
"

This way, if the Alt is working, you can actually hold it down while you find your "open" buffer pretty quickly, amongst the tabs and windows.

,

There are many ways to solve. The best is the best that WORKS for YOU. You have lots of fuzzy match plugins that help you navigate. The 2 things that impress me most are

1) CtrlP or Unite's fuzzy buffer search

2) LustyExplorer and/or LustyJuggler

And the simplest :

:map <F5> :ls<CR>:e #

Pressing F5 lists all buffer, just type number.

[Oct 21, 2018] What is vim recording and how can it be disabled?

Oct 21, 2018 | stackoverflow.com

vehomzzz, Oct 6, 2009 at 20:03

I keep seeing the recording message at the bottom of my gvim 7.2 window.

What is it and how do I turn it off?

Joey Adams, Aug 17, 2010 at 16:26

To turn off vim recording for good, add map q <Nop> to your .vimrc file. – Joey Adams Aug 17 '10 at 16:26

0xc0de, Aug 12, 2016 at 9:04

I can't believe you want to turn recording off! I would show a really annoying popup 'Are you sure?' if one asks to turn it off (or probably would like to give options like the Windows 10 update gives). – 0xc0de Aug 12 '16 at 9:04

yogsototh, Oct 6, 2009 at 20:08

You start recording by q<letter> and you can end it by typing q again.

Recording is a really useful feature of Vim.

It records everything you type. You can then replay it simply by typing @<letter> . Record search, movement, replacement...

One of the best feature of Vim IMHO.

Cascabel, Oct 6, 2009 at 20:13

As seen other places, it's q followed by a register. A really cool (and possibly non-intuitive) part of this is that these are the same registers used by things like delete, yank, and put. This means that you can yank text from the editor into a register, then execute it as a command. – Cascabel Oct 6 '09 at 20:13

Tolga E, Aug 17, 2013 at 3:07

One more thing to note is you can hit any number before the @ to replay the recording that many times like (100@<letter>) will play your actions 100 times – Tolga E Aug 17 '13 at 3:07

anisoptera, Dec 4, 2014 at 9:43

You could add it afterward, by editing the register with put/yank. But I don't know why you'd want to turn recording on or off as part of a macro. ('q' doesn't affect anything when typed in insert mode.) – anisoptera Dec 4 '14 at 9:43

L0j1k, Jul 16, 2015 at 21:08

Vim is so freakin' cool, man. – L0j1k Jul 16 '15 at 21:08

Cascabel, Jul 29, 2015 at 14:52

@Wade " - it's called the default register. – Cascabel Jul 29 '15 at 14:52

ephemient, Oct 6, 2009 at 20:17

Type :h recording to learn more.
                           *q* *recording*
q{0-9a-zA-Z"}           Record typed characters into register {0-9a-zA-Z"}
                        (uppercase to append).  The 'q' command is disabled
                        while executing a register, and it doesn't work inside
                        a mapping.  {Vi: no recording}

q                       Stops recording.  (Implementation note: The 'q' that
                        stops recording is not stored in the register, unless
                        it was the result of a mapping)  {Vi: no recording}


                                                        *@*
@{0-9a-z".=*}           Execute the contents of register {0-9a-z".=*} [count]
                        times.  Note that register '%' (name of the current
                        file) and '#' (name of the alternate file) cannot be
                        used.  For "@=" you are prompted to enter an
                        expression.  The result of the expression is then
                        executed.  See also |@:|.  {Vi: only named registers}

Tim Henigan, Oct 6, 2009 at 20:07

It sounds like you have macro recording turned on. To shut it off, press q .

Refer to " :help recording " for further information.

Related links:

mitchus, Feb 13, 2015 at 14:16

Typing q starts macro recording, and the recording stops when the user hits q again.

As Joey Adams mentioned, to disable recording, add the following line to .vimrc in your home directory:

map q <Nop>

n611x007, Oct 4, 2015 at 7:16

only answer about "how to turn off" part of the question. Well, it makes recording inaccessible, effectively turning it off - at least noone expects vi to have a separate thread for this code, I guess, including me. – n611x007 Oct 4 '15 at 7:16

JeffH, Oct 6, 2009 at 20:10

As others have said, it's macro recording, and you turn it off with q. Here's a nice article about how-to and why it's useful.

John Millikin, Oct 6, 2009 at 20:06

It means you're in "record macro" mode. This mode is entered by typing q followed by a register name, and can be exited by typing q again.

ephemient, Oct 6, 2009 at 20:08

It's actually entered by typing q followed by any register name, which is 0-9, a-z, A-Z, and ". – ephemient Oct 6 '09 at 20:08

Cascabel, Oct 6, 2009 at 20:08

Actually, it's q{0-9a-zA-Z"} - you can record a macro into any register (named by digit, letter, "). In case you actually want to use it... you execute the contents of a register with @<register>. See :help q and :help @ if you're interested in using it. – Cascabel Oct 6 '09 at 20:08

[Oct 21, 2018] vim - how to move a block or column of text

Oct 21, 2018 | stackoverflow.com

how to move a block or column of text Ask Question up vote 23 down vote favorite 5


David.Chu.ca ,Mar 6, 2009 at 20:47

I have the following text as a simple case:

...
abc xxx 123 456
wer xxx 345 678676
...

what I need to move a block of text xxx to another location:

...
abc 123 xxx 456
wer 345 xxx 678676
...

I think I use visual mode to block a column of text, what are the other commands to move the block to another location?

Paul ,Mar 6, 2009 at 20:52

You should use blockwise visual mode ( Ctrl + v ). Then d to delete block, p or P to paste block.

Klinger ,Mar 6, 2009 at 20:53

Try the link .


Marking text (visual mode)

Visual commands

Cut and Paste

Kemin Zhou ,Nov 6, 2015 at 23:59

One of the few useful command I learned at the beginning of learning VIM is :1,3 mo 5 This means move text line 1 through 3 to line 5.

Jϊda Ronιn ,Jan 18, 2017 at 21:20

And you can select the lines in visual mode, then press : to get :'<,'> (equivalent to the :1,3 part in your answer), and add mo N . If you want to move a single line, just :mo N . If you are really lazy, you can omit the space (e.g. :mo5 ). Use marks with mo '{a-zA-Z} . – Jϊda Ronιn Jan 18 '17 at 21:20

Miles ,Jun 29, 2017 at 23:44

just m also works – Miles Jun 29 '17 at 23:44

John Ellinwood ,Mar 6, 2009 at 20:52

  1. In VIM, press Ctrl + V to go in Visual Block mode
  2. Select the required columns with your arrow keys and press x to cut them in the buffer.
  3. Move cursor to row 1 column 9 and press P (thats capital P) in command mode.
  4. Press Ctrl + Shift + b to get in and out of it. ( source )

SergioAraujo ,Jan 4 at 21:49

Using an external command "awk".

%!awk '{print $1,$3,$2,$4}' test.txt

With pure vim

:%s,\v(\w+) (\w+) (\w+) (\w+),\1 \3 \2 \4,g

Another vim solution using global command

:g/./normal wdwwP

[Oct 21, 2018] Camtasia Studio 8

Notable quotes:
"... What did you use to make the gif? ..."
"... That was done with Camtasia Studio 8. Very easy actually. ..."
"... @Zoltán you can use LiceCap, which is small size – ..."
"... GifCam is a simple to use tool, too: blog.bahraniapps.com/gifcam ..."
Oct 21, 2018 | stackoverflow.com

218 down vote


Adam ,Feb 7, 2014 at 22:20

Doesn't get any simpler than this! From normal mode:

yy

then move to the line you want to paste at and

p

Zoltán ,Jul 2, 2014 at 7:42

What did you use to make the gif? Zoltán Jul 2 '14 at 7:42

Adam ,Sep 19, 2014 at 20:15

That was done with Camtasia Studio 8. Very easy actually. Adam Sep 19 '14 at 20:15

onmyway133 ,Feb 23, 2016 at 15:29

@Zoltán you can use LiceCap, which is small size – onmyway133 Feb 23 '16 at 15:29

Jared ,Jun 2, 2016 at 13:44

GifCam is a simple to use tool, too: blog.bahraniapps.com/gifcamJared Jun 2 '16 at 13:44

[Oct 21, 2018] Favorite (G)Vim plugins/scripts?

Dec 27, 2009 | stackoverflow.com
What are your favorite (G)Vim plugins/scripts?

community wiki 2 revs ,Jun 24, 2009 at 13:35

Nerdtree

The NERD tree allows you to explore your filesystem and to open files and directories. It presents the filesystem to you in the form of a tree which you manipulate with the keyboard and/or mouse. It also allows you to perform simple filesystem operations.

The tree can be toggled easily with :NERDTreeToggle which can be mapped to a more suitable key. The keyboard shortcuts in the NERD tree are also easy and intuitive.

Edit: Added synopsis

SpoonMeiser ,Sep 17, 2008 at 19:32

For those of us not wanting to follow every link to find out about each plugin, care to furnish us with a brief synopsis? – SpoonMeiser Sep 17 '08 at 19:32

AbdullahDiaa ,Sep 10, 2012 at 19:51

and NERDTree with NERDTreeTabs are awesome combination github.com/jistr/vim-nerdtree-tabs – AbdullahDiaa Sep 10 '12 at 19:51

community wiki 2 revs ,May 27, 2010 at 0:08

Tim Pope has some kickass plugins. I love his surround plugin.

Taurus Olson ,Feb 21, 2010 at 18:01

Surround is a great plugin for sure. – Taurus Olson Feb 21 '10 at 18:01

Benjamin Oakes ,May 27, 2010 at 0:11

Link to all his vim contributions: vim.org/account/profile.php?user_id=9012 – Benjamin Oakes May 27 '10 at 0:11

community wiki SergioAraujo, Mar 15, 2011 at 15:35

Pathogen plugin and more things commented by Steve Losh

Patrizio Rullo ,Sep 26, 2011 at 12:11

Pathogen is the FIRST plugin you have to install on every Vim installation! It resolves the plugin management problems every Vim developer has. – Patrizio Rullo Sep 26 '11 at 12:11

Profpatsch ,Apr 12, 2013 at 8:53

I would recommend switching to Vundle . It's better by a long shot and truly automates. You can give vim-addon-manager a try, too. – Profpatsch Apr 12 '13 at 8:53

community wiki JPaget, Sep 15, 2008 at 20:47

Taglist , a source code browser plugin for Vim, is currently the top rated plugin at the Vim website and is my favorite plugin.

mindthief ,Jun 27, 2012 at 20:53

A more recent alternative to this is Tagbar , which appears to have some improvements over Taglist. This blog post offers a comparison between the two plugins. – mindthief Jun 27 '12 at 20:53

community wiki 1passenger, Nov 17, 2009 at 9:15

I love snipMate . It's simular to snippetsEmu, but has a much better syntax to read (like Textmate).

community wiki cschol, Aug 22, 2008 at 4:19

A very nice grep replacement for GVim is Ack . A search plugin written in Perl that beats Vim's internal grep implementation and externally invoked greps, too. It also by default skips any CVS directories in the project directory, e.g. '.svn'. This blog shows a way to integrate Ack with vim.

FUD, Aug 27, 2013 at 15:50

github.com/mileszs/ack.vim – FUD Aug 27 '13 at 15:50

community wiki Dominic Dos Santos ,Sep 12, 2008 at 12:44

A.vim is a great little plugin. It allows you to quickly switch between header and source files with a single command. The default is :A , but I remapped it to F2 reduce keystrokes.

community wiki 2 revs, Aug 25, 2008 at 15:06

I really like the SuperTab plugin, it allows you to use the tab key to do all your insert completions.

community wiki Greg Hewgill, Aug 25, 2008 at 19:23

I have recently started using a plugin that highlights differences in your buffer from a previous version in your RCS system (Subversion, git, whatever). You just need to press a key to toggle the diff display on/off. You can find it here: http://github.com/ghewgill/vim-scmdiff . Patches welcome!

Nathan Fellman, Sep 15, 2008 at 18:51

Do you know if this supports bitkeeper? I looked on the website but couldn't even see whom to ask. – Nathan Fellman Sep 15 '08 at 18:51

Greg Hewgill, Sep 16, 2008 at 9:26

It doesn't explicitly support bitkeeper at the moment, but as long as bitkeeper has a "diff" command that outputs a normal patch file, it should be easy enough to add. – Greg Hewgill Sep 16 '08 at 9:26

Yogesh Arora, Mar 10, 2010 at 0:47

does it support clearcase – Yogesh Arora Mar 10 '10 at 0:47

Greg Hewgill, Mar 10, 2010 at 1:39

@Yogesh: No, it doesn't support ClearCase at this time. However, if you can add ClearCase support, a patch would certainly be accepted. – Greg Hewgill Mar 10 '10 at 1:39

Olical ,Jan 23, 2013 at 11:05

This version can be loaded via pathogen in a git submodule: github.com/tomasv/vim-scmdiff – Olical Jan 23 '13 at 11:05

community wiki 4 revs, May 23, 2017 at 11:45

  1. Elegant (mini) buffer explorer - This is the multiple file/buffer manager I use. Takes very little screen space. It looks just like most IDEs where you have a top tab-bar with the files you've opened. I've tested some other similar plugins before, and this is my pick.
  2. TagList - Small file explorer, without the "extra" stuff the other file explorers have. Just lets you browse directories and open files with the "enter" key. Note that this has already been noted by previous commenters to your questions.
  3. SuperTab - Already noted by WMR in this post, looks very promising. It's an auto-completion replacement key for Ctrl-P.
  4. Desert256 color Scheme - Readable, dark one.
  5. Moria color scheme - Another good, dark one. Note that it's gVim only.
  6. Enahcned Python syntax - If you're using Python, this is an enhanced syntax version. Works better than the original. I'm not sure, but this might be already included in the newest version. Nonetheless, it's worth adding to your syntax folder if you need it.
  7. Enhanced JavaScript syntax - Same like the above.
  8. EDIT: Comments - Great little plugin to [un]comment chunks of text. Language recognition included ("#", "/", "/* .. */", etc.) .

community wiki Konrad Rudolph, Aug 25, 2008 at 14:19

Not a plugin, but I advise any Mac user to switch to the MacVim distribution which is vastly superior to the official port.

As for plugins, I used VIM-LaTeX for my thesis and was very satisfied with the usability boost. I also like the Taglist plugin which makes use of the ctags library.

community wiki Yariv ,Nov 25, 2010 at 19:58

clang complete - the best c++ code completion I have seen so far. By using an actual compiler (that would be clang) the plugin is able to complete complex expressions including STL and smart pointers.

community wiki Greg Bowyer, Jul 30, 2009 at 19:51

No one said matchit yet ? Makes HTML / XML soup much nicer http://www.vim.org/scripts/script.php?script_id=39

community wiki 2 revs, 2 users 91% ,Nov 24, 2011 at 5:18

Tomas Restrepo posted on some great Vim scripts/plugins . He has also pointed out some nice color themes on his blog, too. Check out his Vim category .

community wiki HaskellElephant ,Mar 29, 2011 at 17:59,

With version 7.3, undo branches was added to vim. A very powerful feature, but hard to use, until Steve Losh made Gundo which makes this feature possible to use with a ascii representation of the tree and a diff of the change. A must for using undo branches.

community wiki, Auguste ,Apr 20, 2009 at 8:05

Matrix Mode .

community wiki wilhelmtell ,Dec 10, 2010 at 19:11

My latest favourite is Command-T . Granted, to install it you need to have Ruby support and you'll need to compile a C extension for Vim. But oy-yoy-yoy does this plugin make a difference in opening files in Vim!

Victor Farazdagi, Apr 19, 2011 at 19:16

Definitely! Let not the ruby + c compiling stop you, you will be amazed on how well this plugin enhances your toolset. I have been ignoring this plugin for too long, installed it today and already find myself using NERDTree lesser and lesser. – Victor Farazdagi Apr 19 '11 at 19:16

datentyp ,Jan 11, 2012 at 12:54

With ctrlp now there is something as awesome as Command-T written in pure Vimscript! It's available at github.com/kien/ctrlp.vim – datentyp Jan 11 '12 at 12:54

FUD ,Dec 26, 2012 at 4:48

just my 2 cents.. being a naive user of both plugins, with a few first characters of file name i saw a much better result with commandt plugin and a lots of false positives for ctrlp. – FUD Dec 26 '12 at 4:48

community wiki
f3lix
,Mar 15, 2011 at 12:55

Conque Shell : Run interactive commands inside a Vim buffer

Conque is a Vim plugin which allows you to run interactive programs, such as bash on linux or powershell.exe on Windows, inside a Vim buffer. In other words it is a terminal emulator which uses a Vim buffer to display the program output.

http://code.google.com/p/conque/

http://www.vim.org/scripts/script.php?script_id=2771

community wiki 2 revs ,Nov 20, 2009 at 14:51

The vcscommand plugin provides global ex commands for manipulating version-controlled source files and it supports CVS,SVN and some other repositories.

You can do almost all repository related tasks from with in vim:
* Taking the diff of current buffer with repository copy
* Adding new files
* Reverting the current buffer to the repository copy by nullifying the local changes....

community wiki Sirupsen ,Nov 20, 2009 at 15:00

Just gonna name a few I didn't see here, but which I still find extremely helpful:

community wiki thestoneage ,Dec 22, 2011 at 16:25

One Plugin that is missing in the answers is NERDCommenter , which let's you do almost anything with comments. For example {add, toggle, remove} comments. And more. See this blog entry for some examples.

community wiki james ,Feb 19, 2010 at 7:17

I like taglist and fuzzyfinder, those are very cool plugin

community wiki JAVH ,Aug 15, 2010 at 11:54

TaskList

This script is based on the eclipse Task List. It will search the file for FIXME, TODO, and XXX (or a custom list) and put them in a handy list for you to browse which at the same time will update the location in the document so you can see exactly where the tag is located. Something like an interactive 'cw'

community wiki Peter Hoffmann ,Aug 29, 2008 at 4:07

I really love the snippetsEmu Plugin. It emulates some of the behaviour of Snippets from the OS X editor TextMate, in particular the variable bouncing and replacement behaviour.

community wiki Anon ,Sep 11, 2008 at 10:20

Zenburn color scheme and good fonts - [Droid Sans Mono]( http://en.wikipedia.org/wiki/Droid_(font)) on Linux, Consolas on Windows.

Gary Willoughby ,Jul 7, 2011 at 21:21

Take a look at DejaVu Sans Mono too dejavu-fonts.org/wiki/Main_Page – Gary Willoughby Jul 7 '11 at 21:21

Santosh Kumar ,Mar 28, 2013 at 4:48

Droid Sans Mono makes capital m and 0 appear same. – Santosh Kumar Mar 28 '13 at 4:48

community wiki julienXX ,Jun 22, 2010 at 12:05

If you're on a Mac, you got to use peepopen , fuzzyfinder on steroids.

Khaja Minhajuddin ,Apr 5, 2012 at 9:24

Command+T is a free alternative to this: github.com/wincent/Command-T – Khaja Minhajuddin Apr 5 '12 at 9:24

community wiki Peter Stuifzand ,Aug 25, 2008 at 19:16

I use the following two plugins all the time:

Csaba_H ,Jun 24, 2009 at 13:47

vimoutliner is really good for managing small pieces of information (from tasks/todo-s to links) – Csaba_H Jun 24 '09 at 13:47

ThiefMaster ♦ ,Nov 25, 2010 at 20:35

Adding some links/descriptions would be nice – ThiefMaster ♦ Nov 25 '10 at 20:35

community wiki chiggsy ,Aug 26, 2009 at 18:22

For vim I like a little help with completions. Vim has tons of completion modes, but really, I just want vim to complete anything it can, whenver it can.

I hate typing ending quotes, but fortunately this plugin obviates the need for such misery.

Those two are my heavy hitters.

This one may step up to roam my code like an unquiet shade, but I've yet to try it.

community wiki Brett Stahlman, Dec 11, 2009 at 13:28

Txtfmt (The Vim Highlighter) Screenshots

The Txtfmt plugin gives you a sort of "rich text" highlighting capability, similar to what is provided by RTF editors and word processors. You can use it to add colors (foreground and background) and formatting attributes (all combinations of bold, underline, italic, etc...) to your plain text documents in Vim.

The advantage of this plugin over something like Latex is that with Txtfmt, your highlighting changes are visible "in real time", and as with a word processor, the highlighting is WYSIWYG. Txtfmt embeds special tokens directly in the file to accomplish the highlighting, so the highlighting is unaffected when you move the file around, even from one computer to another. The special tokens are hidden by the syntax; each appears as a single space. For those who have applied Vince Negri's conceal/ownsyntax patch, the tokens can even be made "zero-width".

community wiki 2 revs, Dec 10, 2010 at 4:37

tcomment

"I map the "Command + /" keys so i can just comment stuff out while in insert mode imap :i

[Oct 21, 2018] What is your most productive shortcut with Vim?

Notable quotes:
"... less productive ..."
"... column oriented ..."
Feb 17, 2013 | stackoverflow.com
I've heard a lot about Vim, both pros and cons. It really seems you should be (as a developer) faster with Vim than with any other editor. I'm using Vim to do some basic stuff and I'm at best 10 times less productive with Vim.

The only two things you should care about when you talk about speed (you may not care enough about them, but you should) are:

  1. Using alternatively left and right hands is the fastest way to use the keyboard.
  2. Never touching the mouse is the second way to be as fast as possible. It takes ages for you to move your hand, grab the mouse, move it, and bring it back to the keyboard (and you often have to look at the keyboard to be sure you returned your hand properly to the right place)

Here are two examples demonstrating why I'm far less productive with Vim.

Copy/Cut & paste. I do it all the time. With all the contemporary editors you press Shift with the left hand, and you move the cursor with your right hand to select text. Then Ctrl + C copies, you move the cursor and Ctrl + V pastes.

With Vim it's horrible:

Another example? Search & replace.

And everything with Vim is like that: it seems I don't know how to handle it the right way.

NB : I've already read the Vim cheat sheet :)

My question is: What is the way you use Vim that makes you more productive than with a contemporary editor?

community wiki 18 revs, 16 users 64%, Dec 22, 2011 at 11:43

Your problem with Vim is that you don't grok vi .

You mention cutting with yy and complain that you almost never want to cut whole lines. In fact programmers, editing source code, very often want to work on whole lines, ranges of lines and blocks of code. However, yy is only one of many way to yank text into the anonymous copy buffer (or "register" as it's called in vi ).

The "Zen" of vi is that you're speaking a language. The initial y is a verb. The statement yy is a synonym for y_ . The y is doubled up to make it easier to type, since it is such a common operation.

This can also be expressed as dd P (delete the current line and paste a copy back into place; leaving a copy in the anonymous register as a side effect). The y and d "verbs" take any movement as their "subject." Thus yW is "yank from here (the cursor) to the end of the current/next (big) word" and y'a is "yank from here to the line containing the mark named ' a '."

If you only understand basic up, down, left, and right cursor movements then vi will be no more productive than a copy of "notepad" for you. (Okay, you'll still have syntax highlighting and the ability to handle files larger than a piddling ~45KB or so; but work with me here).

vi has 26 "marks" and 26 "registers." A mark is set to any cursor location using the m command. Each mark is designated by a single lower case letter. Thus ma sets the ' a ' mark to the current location, and mz sets the ' z ' mark. You can move to the line containing a mark using the ' (single quote) command. Thus 'a moves to the beginning of the line containing the ' a ' mark. You can move to the precise location of any mark using the ` (backquote) command. Thus `z will move directly to the exact location of the ' z ' mark.

Because these are "movements" they can also be used as subjects for other "statements."

So, one way to cut an arbitrary selection of text would be to drop a mark (I usually use ' a ' as my "first" mark, ' z ' as my next mark, ' b ' as another, and ' e ' as yet another (I don't recall ever having interactively used more than four marks in 15 years of using vi ; one creates one's own conventions regarding how marks and registers are used by macros that don't disturb one's interactive context). Then we go to the other end of our desired text; we can start at either end, it doesn't matter. Then we can simply use d`a to cut or y`a to copy. Thus the whole process has a 5 keystrokes overhead (six if we started in "insert" mode and needed to Esc out command mode). Once we've cut or copied then pasting in a copy is a single keystroke: p .

I say that this is one way to cut or copy text. However, it is only one of many. Frequently we can more succinctly describe the range of text without moving our cursor around and dropping a mark. For example if I'm in a paragraph of text I can use { and } movements to the beginning or end of the paragraph respectively. So, to move a paragraph of text I cut it using { d} (3 keystrokes). (If I happen to already be on the first or last line of the paragraph I can then simply use d} or d{ respectively.

The notion of "paragraph" defaults to something which is usually intuitively reasonable. Thus it often works for code as well as prose.

Frequently we know some pattern (regular expression) that marks one end or the other of the text in which we're interested. Searching forwards or backwards are movements in vi . Thus they can also be used as "subjects" in our "statements." So I can use d/foo to cut from the current line to the next line containing the string "foo" and y?bar to copy from the current line to the most recent (previous) line containing "bar." If I don't want whole lines I can still use the search movements (as statements of their own), drop my mark(s) and use the `x commands as described previously.

In addition to "verbs" and "subjects" vi also has "objects" (in the grammatical sense of the term). So far I've only described the use of the anonymous register. However, I can use any of the 26 "named" registers by prefixing the "object" reference with " (the double quote modifier). Thus if I use "add I'm cutting the current line into the ' a ' register and if I use "by/foo then I'm yanking a copy of the text from here to the next line containing "foo" into the ' b ' register. To paste from a register I simply prefix the paste with the same modifier sequence: "ap pastes a copy of the ' a ' register's contents into the text after the cursor and "bP pastes a copy from ' b ' to before the current line.

This notion of "prefixes" also adds the analogs of grammatical "adjectives" and "adverbs' to our text manipulation "language." Most commands (verbs) and movement (verbs or objects, depending on context) can also take numeric prefixes. Thus 3J means "join the next three lines" and d5} means "delete from the current line through the end of the fifth paragraph down from here."

This is all intermediate level vi . None of it is Vim specific and there are far more advanced tricks in vi if you're ready to learn them. If you were to master just these intermediate concepts then you'd probably find that you rarely need to write any macros because the text manipulation language is sufficiently concise and expressive to do most things easily enough using the editor's "native" language.


A sampling of more advanced tricks:

There are a number of : commands, most notably the :% s/foo/bar/g global substitution technique. (That's not advanced but other : commands can be). The whole : set of commands was historically inherited by vi 's previous incarnations as the ed (line editor) and later the ex (extended line editor) utilities. In fact vi is so named because it's the visual interface to ex .

: commands normally operate over lines of text. ed and ex were written in an era when terminal screens were uncommon and many terminals were "teletype" (TTY) devices. So it was common to work from printed copies of the text, using commands through an extremely terse interface (common connection speeds were 110 baud, or, roughly, 11 characters per second -- which is slower than a fast typist; lags were common on multi-user interactive sessions; additionally there was often some motivation to conserve paper).

So the syntax of most : commands includes an address or range of addresses (line number) followed by a command. Naturally one could use literal line numbers: :127,215 s/foo/bar to change the first occurrence of "foo" into "bar" on each line between 127 and 215. One could also use some abbreviations such as . or $ for current and last lines respectively. One could also use relative prefixes + and - to refer to offsets after or before the curent line, respectively. Thus: :.,$j meaning "from the current line to the last line, join them all into one line". :% is synonymous with :1,$ (all the lines).

The :... g and :... v commands bear some explanation as they are incredibly powerful. :... g is a prefix for "globally" applying a subsequent command to all lines which match a pattern (regular expression) while :... v applies such a command to all lines which do NOT match the given pattern ("v" from "conVerse"). As with other ex commands these can be prefixed by addressing/range references. Thus :.,+21g/foo/d means "delete any lines containing the string "foo" from the current one through the next 21 lines" while :.,$v/bar/d means "from here to the end of the file, delete any lines which DON'T contain the string "bar."

It's interesting that the common Unix command grep was actually inspired by this ex command (and is named after the way in which it was documented). The ex command :g/re/p (grep) was the way they documented how to "globally" "print" lines containing a "regular expression" (re). When ed and ex were used, the :p command was one of the first that anyone learned and often the first one used when editing any file. It was how you printed the current contents (usually just one page full at a time using :.,+25p or some such).

Note that :% g/.../d or (its reVerse/conVerse counterpart: :% v/.../d are the most common usage patterns. However there are couple of other ex commands which are worth remembering:

We can use m to move lines around, and j to join lines. For example if you have a list and you want to separate all the stuff matching (or conversely NOT matching some pattern) without deleting them, then you can use something like: :% g/foo/m$ ... and all the "foo" lines will have been moved to the end of the file. (Note the other tip about using the end of your file as a scratch space). This will have preserved the relative order of all the "foo" lines while having extracted them from the rest of the list. (This would be equivalent to doing something like: 1G!GGmap!Ggrep foo<ENTER>1G:1,'a g/foo'/d (copy the file to its own tail, filter the tail through grep, and delete all the stuff from the head).

To join lines usually I can find a pattern for all the lines which need to be joined to their predecessor (all the lines which start with "^ " rather than "^ * " in some bullet list, for example). For that case I'd use: :% g/^ /-1j (for every matching line, go up one line and join them). (BTW: for bullet lists trying to search for the bullet lines and join to the next doesn't work for a couple reasons ... it can join one bullet line to another, and it won't join any bullet line to all of its continuations; it'll only work pairwise on the matches).

Almost needless to mention you can use our old friend s (substitute) with the g and v (global/converse-global) commands. Usually you don't need to do so. However, consider some case where you want to perform a substitution only on lines matching some other pattern. Often you can use a complicated pattern with captures and use back references to preserve the portions of the lines that you DON'T want to change. However, it will often be easier to separate the match from the substitution: :% g/foo/s/bar/zzz/g -- for every line containing "foo" substitute all "bar" with "zzz." (Something like :% s/\(.*foo.*\)bar\(.*\)/\1zzz\2/g would only work for the cases those instances of "bar" which were PRECEDED by "foo" on the same line; it's ungainly enough already, and would have to be mangled further to catch all the cases where "bar" preceded "foo")

The point is that there are more than just p, s, and d lines in the ex command set.

The : addresses can also refer to marks. Thus you can use: :'a,'bg/foo/j to join any line containing the string foo to its subsequent line, if it lies between the lines between the ' a ' and ' b ' marks. (Yes, all of the preceding ex command examples can be limited to subsets of the file's lines by prefixing with these sorts of addressing expressions).

That's pretty obscure (I've only used something like that a few times in the last 15 years). However, I'll freely admit that I've often done things iteratively and interactively that could probably have been done more efficiently if I'd taken the time to think out the correct incantation.

Another very useful vi or ex command is :r to read in the contents of another file. Thus: :r foo inserts the contents of the file named "foo" at the current line.

More powerful is the :r! command. This reads the results of a command. It's the same as suspending the vi session, running a command, redirecting its output to a temporary file, resuming your vi session, and reading in the contents from the temp. file.

Even more powerful are the ! (bang) and :... ! ( ex bang) commands. These also execute external commands and read the results into the current text. However, they also filter selections of our text through the command! This we can sort all the lines in our file using 1G!Gsort ( G is the vi "goto" command; it defaults to going to the last line of the file, but can be prefixed by a line number, such as 1, the first line). This is equivalent to the ex variant :1,$!sort . Writers often use ! with the Unix fmt or fold utilities for reformating or "word wrapping" selections of text. A very common macro is {!}fmt (reformat the current paragraph). Programmers sometimes use it to run their code, or just portions of it, through indent or other code reformatting tools.

Using the :r! and ! commands means that any external utility or filter can be treated as an extension of our editor. I have occasionally used these with scripts that pulled data from a database, or with wget or lynx commands that pulled data off a website, or ssh commands that pulled data from remote systems.

Another useful ex command is :so (short for :source ). This reads the contents of a file as a series of commands. When you start vi it normally, implicitly, performs a :source on ~/.exinitrc file (and Vim usually does this on ~/.vimrc, naturally enough). The use of this is that you can change your editor profile on the fly by simply sourcing in a new set of macros, abbreviations, and editor settings. If you're sneaky you can even use this as a trick for storing sequences of ex editing commands to apply to files on demand.

For example I have a seven line file (36 characters) which runs a file through wc, and inserts a C-style comment at the top of the file containing that word count data. I can apply that "macro" to a file by using a command like: vim +'so mymacro.ex' ./mytarget

(The + command line option to vi and Vim is normally used to start the editing session at a given line number. However it's a little known fact that one can follow the + by any valid ex command/expression, such as a "source" command as I've done here; for a simple example I have scripts which invoke: vi +'/foo/d|wq!' ~/.ssh/known_hosts to remove an entry from my SSH known hosts file non-interactively while I'm re-imaging a set of servers).

Usually it's far easier to write such "macros" using Perl, AWK, sed (which is, in fact, like grep a utility inspired by the ed command).

The @ command is probably the most obscure vi command. In occasionally teaching advanced systems administration courses for close to a decade I've met very few people who've ever used it. @ executes the contents of a register as if it were a vi or ex command.
Example: I often use: :r!locate ... to find some file on my system and read its name into my document. From there I delete any extraneous hits, leaving only the full path to the file I'm interested in. Rather than laboriously Tab -ing through each component of the path (or worse, if I happen to be stuck on a machine without Tab completion support in its copy of vi ) I just use:

  1. 0i:r (to turn the current line into a valid :r command),
  2. "cdd (to delete the line into the "c" register) and
  3. @c execute that command.

That's only 10 keystrokes (and the expression "cdd @c is effectively a finger macro for me, so I can type it almost as quickly as any common six letter word).


A sobering thought

I've only scratched to surface of vi 's power and none of what I've described here is even part of the "improvements" for which vim is named! All of what I've described here should work on any old copy of vi from 20 or 30 years ago.

There are people who have used considerably more of vi 's power than I ever will.

Jim Dennis, Feb 12, 2010 at 4:08

@Wahnfieden -- grok is exactly what I meant: en.wikipedia.org/wiki/Grok (It's apparently even in the OED --- the closest we anglophones have to a canonical lexicon). To "grok" an editor is to find yourself using its commands fluently ... as if they were your natural language. – Jim Dennis Feb 12 '10 at 4:08

knittl, Feb 27, 2010 at 13:15

wow, a very well written answer! i couldn't agree more, although i use the @ command a lot (in combination with q : record macro) – knittl Feb 27 '10 at 13:15

Brandon Rhodes, Mar 29, 2010 at 15:26

Superb answer that utterly redeems a really horrible question. I am going to upvote this question, that normally I would downvote, just so that this answer becomes easier to find. (And I'm an Emacs guy! But this way I'll have somewhere to point new folks who want a good explanation of what vi power users find fun about vi. Then I'll tell them about Emacs and they can decide.) – Brandon Rhodes Mar 29 '10 at 15:26

Marko, Apr 1, 2010 at 14:47

Can you make a website and put this tutorial there, so it doesn't get burried here on stackoverflow. I have yet to read better introduction to vi then this. – Marko Apr 1 '10 at 14:47

CMS, Aug 2, 2009 at 8:27

You are talking about text selecting and copying, I think that you should give a look to the Vim Visual Mode .

In the visual mode, you are able to select text using Vim commands, then you can do whatever you want with the selection.

Consider the following common scenarios:

You need to select to the next matching parenthesis.

You could do:

You want to select text between quotes:

You want to select a curly brace block (very common on C-style languages):

You want to select the entire file:

Visual block selection is another really useful feature, it allows you to select a rectangular area of text, you just have to press Ctrl - V to start it, and then select the text block you want and perform any type of operation such as yank, delete, paste, edit, etc. It's great to edit column oriented text.

finnw, Aug 2, 2009 at 8:49

Every editor has something like this, it's not specific to vim. – finnw Aug 2 '09 at 8:49

guns, Aug 2, 2009 at 9:54

Yes, but it was a specific complaint of the poster. Visual mode is Vim's best method of direct text-selection and manipulation. And since vim's buffer traversal methods are superb, I find text selection in vim fairly pleasurable. – guns Aug 2 '09 at 9:54

Hamish Downer, Mar 16, 2010 at 13:34

I think it is also worth mentioning Ctrl-V to select a block - ie an arbitrary rectangle of text. When you need it it's a lifesaver. – Hamish Downer Mar 16 '10 at 13:34

CMS, Apr 2, 2010 at 2:07

@viksit: I'm using Camtasia, but there are plenty of alternatives: codinghorror.com/blog/2006/11/screencasting-for-windows.htmlCMS Apr 2 '10 at 2:07

Nathan Long, Mar 1, 2011 at 19:05

Also, if you've got a visual selection and want to adjust it, o will hop to the other end. So you can move both the beginning and the end of the selection as much as you like. – Nathan Long Mar 1 '11 at 19:05

community wiki
12 revs, 3 users 99%
,Oct 29, 2012 at 18:51

Some productivity tips:

Smart movements

Quick editing commands

Combining commands

Most commands accept a amount and direction, for example:

Useful programmer commands

Macro recording

By using very specific commands and movements, VIM can replay those exact actions for the next lines. (e.g. A for append-to-end, b / e to move the cursor to the begin or end of a word respectively)

Example of well built settings

# reset to vim-defaults
if &compatible          # only if not set before:
  set nocompatible      # use vim-defaults instead of vi-defaults (easier, more user friendly)
endif

# display settings
set background=dark     # enable for dark terminals
set nowrap              # dont wrap lines
set scrolloff=2         # 2 lines above/below cursor when scrolling
set number              # show line numbers
set showmatch           # show matching bracket (briefly jump)
set showmode            # show mode in status bar (insert/replace/...)
set showcmd             # show typed command in status bar
set ruler               # show cursor position in status bar
set title               # show file in titlebar
set wildmenu            # completion with menu
set wildignore=*.o,*.obj,*.bak,*.exe,*.py[co],*.swp,*~,*.pyc,.svn
set laststatus=2        # use 2 lines for the status bar
set matchtime=2         # show matching bracket for 0.2 seconds
set matchpairs+=<:>     # specially for html

# editor settings
set esckeys             # map missed escape sequences (enables keypad keys)
set ignorecase          # case insensitive searching
set smartcase           # but become case sensitive if you type uppercase characters
set smartindent         # smart auto indenting
set smarttab            # smart tab handling for indenting
set magic               # change the way backslashes are used in search patterns
set bs=indent,eol,start # Allow backspacing over everything in insert mode

set tabstop=4           # number of spaces a tab counts for
set shiftwidth=4        # spaces for autoindents
#set expandtab           # turn a tabs into spaces

set fileformat=unix     # file mode is unix
#set fileformats=unix,dos    # only detect unix file format, displays that ^M with dos files

# system settings
set lazyredraw          # no redraws in macros
set confirm             # get a dialog when :q, :w, or :wq fails
set nobackup            # no backup~ files.
set viminfo='20,\"500   # remember copy registers after quitting in the .viminfo file -- 20 jump links, regs up to 500 lines'
set hidden              # remember undo after quitting
set history=50          # keep 50 lines of command history
set mouse=v             # use mouse in visual mode (not normal,insert,command,help mode


# color settings (if terminal/gui supports it)
if &t_Co > 2 || has("gui_running")
  syntax on          # enable colors
  set hlsearch       # highlight search (very useful!)
  set incsearch      # search incremently (search while typing)
endif

# paste mode toggle (needed when using autoindent/smartindent)
map <F10> :set paste<CR>
map <F11> :set nopaste<CR>
imap <F10> <C-O>:set paste<CR>
imap <F11> <nop>
set pastetoggle=<F11>

# Use of the filetype plugins, auto completion and indentation support
filetype plugin indent on

# file type specific settings
if has("autocmd")
  # For debugging
  #set verbose=9

  # if bash is sh.
  let bash_is_sh=1

  # change to directory of current file automatically
  autocmd BufEnter * lcd %:p:h

  # Put these in an autocmd group, so that we can delete them easily.
  augroup mysettings
    au FileType xslt,xml,css,html,xhtml,javascript,sh,config,c,cpp,docbook set smartindent shiftwidth=2 softtabstop=2 expandtab
    au FileType tex set wrap shiftwidth=2 softtabstop=2 expandtab

    # Confirm to PEP8
    au FileType python set tabstop=4 softtabstop=4 expandtab shiftwidth=4 cinwords=if,elif,else,for,while,try,except,finally,def,class
  augroup END

  augroup perl
    # reset (disable previous 'augroup perl' settings)
    au!  

    au BufReadPre,BufNewFile
    \ *.pl,*.pm
    \ set formatoptions=croq smartindent shiftwidth=2 softtabstop=2 cindent cinkeys='0{,0},!^F,o,O,e' " tags=./tags,tags,~/devel/tags,~/devel/C
    # formatoption:
    #   t - wrap text using textwidth
    #   c - wrap comments using textwidth (and auto insert comment leader)
    #   r - auto insert comment leader when pressing <return> in insert mode
    #   o - auto insert comment leader when pressing 'o' or 'O'.
    #   q - allow formatting of comments with "gq"
    #   a - auto formatting for paragraphs
    #   n - auto wrap numbered lists
    #   
  augroup END


  # Always jump to the last known cursor position. 
  # Don't do it when the position is invalid or when inside
  # an event handler (happens when dropping a file on gvim). 
  autocmd BufReadPost * 
    \ if line("'\"") > 0 && line("'\"") <= line("$") | 
    \   exe "normal g`\"" | 
    \ endif 

endif # has("autocmd")

The settings can be stored in ~/.vimrc, or system-wide in /etc/vimrc.local and then by read from the /etc/vimrc file using:

source /etc/vimrc.local

(you'll have to replace the # comment character with " to make it work in VIM, I wanted to give proper syntax highlighting here).

The commands I've listed here are pretty basic, and the main ones I use so far. They already make me quite more productive, without having to know all the fancy stuff.

naught101, Apr 28, 2012 at 2:09

Better than '. is g;, which jumps back through the changelist . Goes to the last edited position, instead of last edited line – naught101 Apr 28 '12 at 2:09

community wiki
5 revs, 4 users 53%
,Apr 12, 2012 at 7:46

The Control + R mechanism is very useful :-) In either insert mode or command mode (i.e. on the : line when typing commands), continue with a numbered or named register:

See :help i_CTRL-R and :help c_CTRL-R for more details, and snoop around nearby for more CTRL-R goodness.

vdboor, Jun 3, 2010 at 9:08

FYI, this refers to Ctrl+R in insert mode . In normal mode, Ctrl+R is redo. – vdboor Jun 3 '10 at 9:08

Aryeh Leib Taurog, Feb 26, 2012 at 19:06

+1 for current/alternate file name. Control-A also works in insert mode for last inserted text, and Control-@ to both insert last inserted text and immediately switch to normal mode. – Aryeh Leib Taurog Feb 26 '12 at 19:06

community wiki
Benson
, Apr 1, 2010 at 3:44

Vim Plugins

There are a lot of good answers here, and one amazing one about the zen of vi. One thing I don't see mentioned is that vim is extremely extensible via plugins. There are scripts and plugins to make it do all kinds of crazy things the original author never considered. Here are a few examples of incredibly handy vim plugins:

rails.vim

Rails.vim is a plugin written by tpope. It's an incredible tool for people doing rails development. It does magical context-sensitive things that allow you to easily jump from a method in a controller to the associated view, over to a model, and down to unit tests for that model. It has saved dozens if not hundreds of hours as a rails developer.

gist.vim

This plugin allows you to select a region of text in visual mode and type a quick command to post it to gist.github.com . This allows for easy pastebin access, which is incredibly handy if you're collaborating with someone over IRC or IM.

space.vim

This plugin provides special functionality to the spacebar. It turns the spacebar into something analogous to the period, but instead of repeating actions it repeats motions. This can be very handy for moving quickly through a file in a way you define on the fly.

surround.vim

This plugin gives you the ability to work with text that is delimited in some fashion. It gives you objects which denote things inside of parens, things inside of quotes, etc. It can come in handy for manipulating delimited text.

supertab.vim

This script brings fancy tab completion functionality to vim. The autocomplete stuff is already there in the core of vim, but this brings it to a quick tab rather than multiple different multikey shortcuts. Very handy, and incredibly fun to use. While it's not VS's intellisense, it's a great step and brings a great deal of the functionality you'd like to expect from a tab completion tool.

syntastic.vim

This tool brings external syntax checking commands into vim. I haven't used it personally, but I've heard great things about it and the concept is hard to beat. Checking syntax without having to do it manually is a great time saver and can help you catch syntactic bugs as you introduce them rather than when you finally stop to test.

fugitive.vim

Direct access to git from inside of vim. Again, I haven't used this plugin, but I can see the utility. Unfortunately I'm in a culture where svn is considered "new", so I won't likely see git at work for quite some time.

nerdtree.vim

A tree browser for vim. I started using this recently, and it's really handy. It lets you put a treeview in a vertical split and open files easily. This is great for a project with a lot of source files you frequently jump between.

FuzzyFinderTextmate.vim

This is an unmaintained plugin, but still incredibly useful. It provides the ability to open files using a "fuzzy" descriptive syntax. It means that in a sparse tree of files you need only type enough characters to disambiguate the files you're interested in from the rest of the cruft.

Conclusion

There are a lot of incredible tools available for vim. I'm sure I've only scratched the surface here, and it's well worth searching for tools applicable to your domain. The combination of traditional vi's powerful toolset, vim's improvements on it, and plugins which extend vim even further, it's one of the most powerful ways to edit text ever conceived. Vim is easily as powerful as emacs, eclipse, visual studio, and textmate.

Thanks

Thanks to duwanis for his vim configs from which I have learned much and borrowed most of the plugins listed here.

Tom Morris, Apr 1, 2010 at 8:50

The magical tests-to-class navigation in rails.vim is one of the more general things I wish Vim had that TextMate absolutely nails across all languages: if I am working on Person.scala and I do Cmd+T, usually the first thing in the list is PersonTest.scala. – Tom Morris Apr 1 '10 at 8:50

Gavin Gilmour, Jan 15, 2011 at 13:44

I think it's time FuzzyFinderTextmate started to get replaced with github.com/wincent/Command-TGavin Gilmour Jan 15 '11 at 13:44

Nathan Long, Mar 1, 2011 at 19:07

+1 for Syntastic. That, combined with JSLint, has made my Javascript much less error-prone. See superuser.com/questions/247012/ about how to set up JSLint to work with Syntastic. – Nathan Long Mar 1 '11 at 19:07

AlG, Sep 13, 2011 at 17:37

@Benson Great list! I'd toss in snipMate as well. Very helpful automation of common coding stuff. if<tab> instant if block, etc. – AlG Sep 13 '11 at 17:37

EarlOfEgo, May 12, 2012 at 15:13

I think nerdcommenter is also a good plugin: here . Like its name says, it is for commenting your code. – EarlOfEgo May 12 '12 at 15:13

community wiki
4 revs, 2 users 89%
,Mar 31, 2010 at 23:01

. Repeat last text-changing command

I save a lot of time with this one.

Visual mode was mentioned previously, but block visual mode has saved me a lot of time when editing fixed size columns in text file. (accessed with Ctrl-V).

vdboor, Apr 1, 2010 at 8:34

Additionally, if you use a concise command (e.g. A for append-at-end) to edit the text, vim can repeat that exact same action for the next line you press the . key at. – vdboor Apr 1 '10 at 8:34

community wiki
3 revs, 3 users 87%
,Dec 24, 2012 at 14:50

gi

Go to last edited location (very useful if you performed some searching and than want go back to edit)

^P and ^N

Complete previous (^P) or next (^N) text.

^O and ^I

Go to previous ( ^O - "O" for old) location or to the next ( ^I - "I" just near to "O" ). When you perform searches, edit files etc., you can navigate through these "jumps" forward and back.

R. Martinho Fernandes, Apr 1, 2010 at 3:02

Thanks for gi ! Now I don't need marks for that! – R. Martinho Fernandes Apr 1 '10 at 3:02

Kungi, Feb 10, 2011 at 16:23

I Think this can also be done with `` – Kungi Feb 10 '11 at 16:23

Grant McLean, Aug 23, 2011 at 8:21

@Kungi `. will take you to the last edit `` will take you back to the position you were in before the last 'jump' - which /might/ also be the position of the last edit. – Grant McLean Aug 23 '11 at 8:21

community wiki
Ronny Brendel
, Mar 31, 2010 at 19:37

I recently (got) discovered this site: http://vimcasts.org/

It's pretty new and really really good. The guy who is running the site switched from textmate to vim and hosts very good and concise casts on specific vim topics. Check it out!

Jeromy Anglim, Jan 13, 2011 at 6:40

If you like vim tutorials, check out Derek Wyatt's vim videos as well. They're excellent. – Jeromy Anglim Jan 13 '11 at 6:40

community wiki
2 revs, 2 users 67%
,Feb 27, 2010 at 11:20

CTRL + A increments the number you are standing on.

innaM, Aug 3, 2009 at 9:14

... and CTRL-X decrements. – innaM Aug 3 '09 at 9:14

SolutionYogi, Feb 26, 2010 at 20:43

It's a neat shortcut but so far I have NEVER found any use for it. – SolutionYogi Feb 26 '10 at 20:43

matja, Feb 27, 2010 at 14:21

if you run vim in screen and wonder why this doesn't work - ctrl+A, A – matja Feb 27 '10 at 14:21

hcs42, Feb 27, 2010 at 19:05

@SolutionYogi: Consider that you want to add line number to the beginning of each line. Solution: ggI1<space><esc>0qqyawjP0<c-a>0q9999@q – hcs42 Feb 27 '10 at 19:05

blueyed, Apr 1, 2010 at 14:47

Extremely useful with Vimperator, where it increments (or decrements, Ctrl-X) the last number in the URL. Useful for quickly surfing through image galleries etc. – blueyed Apr 1 '10 at 14:47

community wiki
3 revs
,Aug 28, 2009 at 15:23

All in Normal mode:

f<char> to move to the next instance of a particular character on the current line, and ; to repeat.

F<char> to move to the previous instance of a particular character on the current line and ; to repeat.

If used intelligently, the above two can make you killer-quick moving around in a line.

* on a word to search for the next instance.

# on a word to search for the previous instance.

Jim Dennis, Mar 14, 2010 at 6:38

Whoa, I didn't know about the * and # (search forward/back for word under cursor) binding. That's kinda cool. The f/F and t/T and ; commands are quick jumps to characters on the current line. f/F put the cursor on the indicated character while t/T puts it just up "to" the character (the character just before or after it according to the direction chosen. ; simply repeats the most recent f/F/t/T jump (in the same direction). – Jim Dennis Mar 14 '10 at 6:38

Steve K, Apr 3, 2010 at 23:50

:) The tagline at the top of the tips page at vim.org: "Can you imagine how many keystrokes could have been saved, if I only had known the "*" command in time?" - Juergen Salk, 1/19/2001" – Steve K Apr 3 '10 at 23:50

puk, Feb 24, 2012 at 6:45

As Jim mentioned, the "t/T" combo is often just as good, if not better, for example, ct( will erase the word and put you in insert mode, but keep the parantheses! – puk Feb 24 '12 at 6:45

community wiki
agfe2
, Aug 19, 2010 at 8:08

Session

a. save session

:mks sessionname

b. force save session

:mks! sessionname

c. load session

gvim or vim -S sessionname


Adding and Subtracting

a. Adding and Subtracting

CTRL-A ;Add [count] to the number or alphabetic character at or after the cursor. {not in Vi

CTRL-X ;Subtract [count] from the number or alphabetic character at or after the cursor. {not in Vi}

b. Window key unmapping

In window, Ctrl-A already mapped for whole file selection you need to unmap in rc file. mark mswin.vim CTRL-A mapping part as comment or add your rc file with unmap

c. With Macro

The CTRL-A command is very useful in a macro. Example: Use the following steps to make a numbered list.

  1. Create the first list entry, make sure it starts with a number.
  2. qa - start recording into buffer 'a'
  3. Y - yank the entry
  4. p - put a copy of the entry below the first one
  5. CTRL-A - increment the number
  6. q - stop recording
  7. @a - repeat the yank, put and increment times

Don Reba, Aug 22, 2010 at 5:22

Any idea what the shortcuts are in Windows? – Don Reba Aug 22 '10 at 5:22

community wiki
8 revs, 2 users 98%
,Aug 18, 2012 at 21:44

Last week at work our project inherited a lot of Python code from another project. Unfortunately the code did not fit into our existing architecture - it was all done with global variables and functions, which would not work in a multi-threaded environment.

We had ~80 files that needed to be reworked to be object oriented - all the functions moved into classes, parameters changed, import statements added, etc. We had a list of about 20 types of fix that needed to be done to each file. I would estimate that doing it by hand one person could do maybe 2-4 per day.

So I did the first one by hand and then wrote a vim script to automate the changes. Most of it was a list of vim commands e.g.

" delete an un-needed function "
g/someFunction(/ d

" add wibble parameter to function foo "
%s/foo(/foo( wibble,/

" convert all function calls bar(thing) into method calls thing.bar() "
g/bar(/ normal nmaf(ldi(`aPa.

The last one deserves a bit of explanation:

g/bar(/  executes the following command on every line that contains "bar("
normal   execute the following text as if it was typed in in normal mode
n        goes to the next match of "bar(" (since the :g command leaves the cursor position at the start of the line)
ma       saves the cursor position in mark a
f(       moves forward to the next opening bracket
l        moves right one character, so the cursor is now inside the brackets
di(      delete all the text inside the brackets
`a       go back to the position saved as mark a (i.e. the first character of "bar")
P        paste the deleted text before the current cursor position
a.       go into insert mode and add a "."

For a couple of more complex transformations such as generating all the import statements I embedded some python into the vim script.

After a few hours of working on it I had a script that will do at least 95% of the conversion. I just open a file in vim then run :source fixit.vim and the file is transformed in a blink of the eye.

We still have the work of changing the remaining 5% that was not worth automating and of testing the results, but by spending a day writing this script I estimate we have saved weeks of work.

Of course it would have been possible to automate this with a scripting language like Python or Ruby, but it would have taken far longer to write and would be less flexible - the last example would have been difficult since regex alone would not be able to handle nested brackets, e.g. to convert bar(foo(xxx)) to foo(xxx).bar() . Vim was perfect for the task.

Olivier Pons, Feb 28, 2010 at 14:41

Thanks a lot for sharing that's really nice to learn from "useful & not classical" macros. – Olivier Pons Feb 28 '10 at 14:41

Ipsquiggle, Mar 23, 2010 at 16:55

%s/\(bar\)(\(.\+\))/\2.\1()/ would do that too. (Escapes are compatible with :set magic .) Just for the record. :) – Ipsquiggle Mar 23 '10 at 16:55

Ipsquiggle, Mar 23, 2010 at 16:56

Of if you don't like vim-style escapes, use \v to turn on Very Magic: %s/\v(bar)\((.+)\)/\2.\1()/Ipsquiggle Mar 23 '10 at 16:56

Dave Kirby, Mar 23, 2010 at 17:16

@lpsquiggle: your suggestion would not handle complex expressions with more than one set of brackets. e.g. if bar(foo(xxx)) or wibble(xxx): becomes if foo(xxx)) or wibble(xxx.bar(): which is completely wrong. – Dave Kirby Mar 23 '10 at 17:16

community wiki
2 revs
,Aug 2, 2009 at 11:17

Use the builtin file explorer! The command is :Explore and it allows you to navigate through your source code very very fast. I have these mapping in my .vimrc :
map <silent> <F8>   :Explore<CR>
map <silent> <S-F8> :sp +Explore<CR>

The explorer allows you to make file modifications, too. I'll post some of my favorite keys, pressing <F1> will give you the full list:

Svend, Aug 2, 2009 at 8:48

I always thought the default methods for browsing kinda sucked for most stuff. It's just slow to browse, if you know where you wanna go. LustyExplorer from vim.org's script section is a much needed improvement. – Svend Aug 2 '09 at 8:48

Taurus Olson, Aug 6, 2009 at 17:37

Your second mapping could be more simple: map <silent> <S-F8> :Sexplore<CR> – Taurus Olson Aug 6 '09 at 17:37

kprobst, Apr 1, 2010 at 3:53

I recommend NERDtree instead of the built-in explorer. It has changed the way I used vim for projects and made me much more productive. Just google for it. – kprobst Apr 1 '10 at 3:53

dash-tom-bang, Aug 24, 2011 at 0:35

I never feel the need to explore the source tree, I just use :find, :tag and the various related keystrokes to jump around. (Maybe this is because the source trees I work on are big and organized differently than I would have done? :) ) – dash-tom-bang Aug 24 '11 at 0:35

community wiki
2 revs, 2 users 92%
,Jun 15, 2011 at 13:39

I am a member of the American Cryptogram Association. The bimonthly magazine includes over 100 cryptograms of various sorts. Roughly 15 of these are "cryptarithms" - various types of arithmetic problems with letters substituted for the digits. Two or three of these are sudokus, except with letters instead of numbers. When the grid is completed, the nine distinct letters will spell out a word or words, on some line, diagonal, spiral, etc., somewhere in the grid.

Rather than working with pencil, or typing the problems in by hand, I download the problems from the members area of their website.

When working with these sudokus, I use vi, simply because I'm using facilities that vi has that few other editors have. Mostly in converting the lettered grid into a numbered grid, because I find it easier to solve, and then the completed numbered grid back into the lettered grid to find the solution word or words.

The problem is formatted as nine groups of nine letters, with - s representing the blanks, written in two lines. The first step is to format these into nine lines of nine characters each. There's nothing special about this, just inserting eight linebreaks in the appropriate places.

The result will look like this:

T-O-----C
-E-----S-
--AT--N-L
---NASO--
---E-T---
--SPCL---
E-T--OS--
-A-----P-
S-----C-T

So, first step in converting this into numbers is to make a list of the distinct letters. First, I make a copy of the block. I position the cursor at the top of the block, then type :y}}p . : puts me in command mode, y yanks the next movement command. Since } is a move to the end of the next paragraph, y} yanks the paragraph. } then moves the cursor to the end of the paragraph, and p pastes what we had yanked just after the cursor. So y}}p creates a copy of the next paragraph, and ends up with the cursor between the two copies.

Next, I to turn one of those copies into a list of distinct letters. That command is a bit more complex:

:!}tr -cd A-Z | sed 's/\(.\)/\1\n/g' | sort -u | tr -d '\n'

: again puts me in command mode. ! indicates that the content of the next yank should be piped through a command line. } yanks the next paragraph, and the command line then uses the tr command to strip out everything except for upper-case letters, the sed command to print each letter on a single line, and the sort command to sort those lines, removing duplicates, and then tr strips out the newlines, leaving the nine distinct letters in a single line, replacing the nine lines that had made up the paragraph originally. In this case, the letters are: ACELNOPST .

Next step is to make another copy of the grid. And then to use the letters I've just identified to replace each of those letters with a digit from 1 to 9. That's simple: :!}tr ACELNOPST 0-9 . The result is:

8-5-----1
-2-----7-
--08--4-3
---4075--
---2-8---
--7613---
2-8--57--
-0-----6-
7-----1-8

This can then be solved in the usual way, or entered into any sudoku solver you might prefer. The completed solution can then be converted back into letters with :!}tr 1-9 ACELNOPST .

There is power in vi that is matched by very few others. The biggest problem is that only a very few of the vi tutorial books, websites, help-files, etc., do more than barely touch the surface of what is possible.

hhh, Jan 14, 2011 at 17:12

and an irritation is that some distros such as ubuntu has aliases from the word "vi" to "vim" so people won't really see vi. Excellent example, have to try... +1 – hhh Jan 14 '11 at 17:12

dash-tom-bang, Aug 24, 2011 at 0:45

Doesn't vim check the name it was started with so that it can come up in the right 'mode'? – dash-tom-bang Aug 24 '11 at 0:45

sehe, Mar 4, 2012 at 20:47

I'm baffled by this repeated error: you say you need : to go into command mode, but then invariably you specify normal mode commands (like y}}p ) which cannot possibly work from the command mode?! – sehe Mar 4 '12 at 20:47

sehe, Mar 4, 2012 at 20:56

My take on the unique chars challenge: :se tw=1 fo= (preparation) VG:s/./& /g (insert spaces), gvgq (split onto separate lines), V{:sort u (sort and remove duplicates) – sehe Mar 4 '12 at 20:56

community wiki
jqno
, Aug 2, 2009 at 8:59

Bulk text manipulations!

Either through macros:

Or through regular expressions:

(But be warned: if you do the latter, you'll have 2 problems :).)

Jim Dennis, Jan 10, 2010 at 4:03

+1 for the Jamie Zawinski reference. (No points taken back for failing to link to it, even). :) – Jim Dennis Jan 10 '10 at 4:03

jqno, Jan 10, 2010 at 10:06

@Jim I didn't even know it was a Jamie Zawinski quote :). I'll try to remember it from now on. – jqno Jan 10 '10 at 10:06

Jim Dennis, Feb 12, 2010 at 4:15

I find the following trick increasingly useful ... for cases where you want to join lines that match (or that do NOT match) some pattern to the previous line: :% g/foo/-1j or :'a,'z v/bar/-1j for example (where the former is "all lines and matching the pattern" while the latter is "lines between mark a and mark z which fail to match the pattern"). The part after the patter in a g or v ex command can be any other ex commmands, -1j is just a relative line movement and join command. – Jim Dennis Feb 12 '10 at 4:15

JustJeff, Feb 27, 2010 at 12:54

of course, if you name your macro '2', then when it comes time to use it, you don't even have to move your finger from the '@' key to the 'q' key. Probably saves 50 to 100 milliseconds every time right there. =P – JustJeff Feb 27 '10 at 12:54

Simon Steele, Apr 1, 2010 at 13:12

@JustJeff Depends entirely on your keyboard layout, my @ key is at the other side of the keyboard from my 2 key. – Simon Steele Apr 1 '10 at 13:12

community wiki
David Pope
, Apr 2, 2012 at 7:56

I recently discovered q: . It opens the "command window" and shows your most recent ex-mode (command-mode) commands. You can move as usual within the window, and pressing <CR> executes the command. You can edit, etc. too. Priceless when you're messing around with some complex command or regex and you don't want to retype the whole thing, or if the complex thing you want to do was 3 commands back. It's almost like bash's set -o vi, but for vim itself (heh!).

See :help q: for more interesting bits for going back and forth.

community wiki
2 revs, 2 users 56%
,Feb 27, 2010 at 11:29

I just discovered Vim's omnicompletion the other day, and while I'll admit I'm a bit hazy on what does which, I've had surprisingly good results just mashing either Ctrl + x Ctrl + u or Ctrl + n / Ctrl + p in insert mode. It's not quite IntelliSense, but I'm still learning it.

Try it out! :help ins-completion

community wiki
tfmoraes
, Mar 14, 2010 at 19:49

These are not shortcuts, but they are related:
  1. Make capslock an additional ESC (or Ctrl)
  2. map leader to "," (comma), with this command: let mapleader=","

They boost my productivity.

Olivier Pons, Mar 15, 2010 at 10:09

Hey nice hint about the "\"! Far better to type "," than "\". – Olivier Pons Mar 15 '10 at 10:09

R. Martinho Fernandes, Apr 1, 2010 at 3:30

To make Caps Lock an additional Esc in Windows (what's a caps lock key for? An "any key"?), try this: web.archive.org/web/20100418005858/http://webpages.charter.net/R. Martinho Fernandes Apr 1 '10 at 3:30

Tom Morris, Apr 1, 2010 at 8:45

On Mac, you need PCKeyboardHack - details at superuser.com/questions/34223/Tom Morris Apr 1 '10 at 8:45

Jeromy Anglim, Jan 10, 2011 at 4:43

On Windows I use AutoHotKey with Capslock::EscapeJeromy Anglim Jan 10 '11 at 4:43

community wiki
Costyn
, Sep 20, 2010 at 10:34

Another useful vi "shortcut" I frequently use is 'xp'. This will swap the character under the cursor with the next character.

tester, Aug 22, 2011 at 17:19

Xp to go the other way – tester Aug 22 '11 at 17:19

kguest, Aug 27, 2011 at 8:21

Around the time that Windows xp came out, I used to joke that this is the only good use for it. – kguest Aug 27 '11 at 8:21

community wiki
Peter Ellis
, Aug 2, 2009 at 9:47

<Ctrl> + W, V to split the screen vertically
<Ctrl> + W, W to shift between the windows

!python % [args] to run the script I am editing in this window

ZF in visual mode to fold arbitrary lines

Andrew Scagnelli, Apr 1, 2010 at 2:58

<Ctrl> + W and j/k will let you navigate absolutely (j up, k down, as with normal vim). This is great when you have 3+ splits. – Andrew Scagnelli Apr 1 '10 at 2:58

coder_tim, Jan 30, 2012 at 20:08

+1 for zf in visual mode, I like code folding, but did not know about that. – coder_tim Jan 30 '12 at 20:08

puk, Feb 24, 2012 at 7:00

after bashing my keyboard I have deduced that <C-w>n or <C-w>s is new horizontal window, <C-w>b is bottom right window, <C-w>c or <C-w>q is close window, <C-w>x is increase and then decrease window width (??), <C-w>p is last window, <C-w>backspace is move left(ish) window – puk Feb 24 '12 at 7:00

sjas, Jun 25, 2012 at 0:25

:help ctrl-w FTW... do yourself a favour, and force yourself to try these things for at least 15 minutes! – sjas Jun 25 '12 at 0:25

community wiki
2 revs
,Apr 1, 2010 at 17:00

Visual Mode

As several other people have said, visual mode is the answer to your copy/cut & paste problem. Vim gives you 'v', 'V', and C-v. Lower case 'v' in vim is essentially the same as the shift key in notepad. The nice thing is that you don't have to hold it down. You can use any movement technique to navigate efficiently to the starting (or ending) point of your selection. Then hit 'v', and use efficient movement techniques again to navigate to the other end of your selection. Then 'd' or 'y' allows you to cut or copy that selection.

The advantage vim's visual mode has over Jim Dennis's description of cut/copy/paste in vi is that you don't have to get the location exactly right. Sometimes it's more efficient to use a quick movement to get to the general vicinity of where you want to go and then refine that with other movements than to think up a more complex single movement command that gets you exactly where you want to go.

The downside to using visual mode extensively in this manner is that it can become a crutch that you use all the time which prevents you from learning new vi(m) commands that might allow you to do things more efficiently. However, if you are very proactive about learning new aspects of vi(m), then this probably won't affect you much.

I'll also re-emphasize that the visual line and visual block modes give you variations on this same theme that can be very powerful...especially the visual block mode.

On Efficient Use of the Keyboard

I also disagree with your assertion that alternating hands is the fastest way to use the keyboard. It has an element of truth in it. Speaking very generally, repeated use of the same thing is slow. This most significant example of this principle is that consecutive keystrokes typed with the same finger are very slow. Your assertion probably stems from the natural tendency to use the s/finger/hand/ transformation on this pattern. To some extent it's correct, but at the extremely high end of the efficiency spectrum it's incorrect.

Just ask any pianist. Ask them whether it's faster to play a succession of a few notes alternating hands or using consecutive fingers of a single hand in sequence. The fastest way to type 4 keystrokes is not to alternate hands, but to type them with 4 fingers of the same hand in either ascending or descending order (call this a "run"). This should be self-evident once you've considered this possibility.

The more difficult problem is optimizing for this. It's pretty easy to optimize for absolute distance on the keyboard. Vim does that. It's much harder to optimize at the "run" level, but vi(m) with it's modal editing gives you a better chance at being able to do it than any non-modal approach (ahem, emacs) ever could.

On Emacs

Lest the emacs zealots completely disregard my whole post on account of that last parenthetical comment, I feel I must describe the root of the difference between the emacs and vim religions. I've never spoken up in the editor wars and I probably won't do it again, but I've never heard anyone describe the differences this way, so here it goes. The difference is the following tradeoff:

Vim gives you unmatched raw text editing efficiency Emacs gives you unmatched ability to customize and program the editor

The blind vim zealots will claim that vim has a scripting language. But it's an obscure, ad-hoc language that was designed to serve the editor. Emacs has Lisp! Enough said. If you don't appreciate the significance of those last two sentences or have a desire to learn enough about functional programming and Lisp to develop that appreciation, then you should use vim.

The emacs zealots will claim that emacs has viper mode, and so it is a superset of vim. But viper mode isn't standard. My understanding is that viper mode is not used by the majority of emacs users. Since it's not the default, most emacs users probably don't develop a true appreciation for the benefits of the modal paradigm.

In my opinion these differences are orthogonal. I believe the benefits of vim and emacs as I have stated them are both valid. This means that the ultimate editor doesn't exist yet. It's probably true that emacs would be the easiest platform on which to base the ultimate editor. But modal editing is not entrenched in the emacs mindset. The emacs community could move that way in the future, but that doesn't seem very likely.

So if you want raw editing efficiency, use vim. If you want the ultimate environment for scripting and programming your editor use emacs. If you want some of both with an emphasis on programmability, use emacs with viper mode (or program your own mode). If you want the best of both worlds, you're out of luck for now.

community wiki
konryd
, Mar 31, 2010 at 22:44

Spend 30 mins doing the vim tutorial (run vimtutor instead of vim in terminal). You will learn the basic movements, and some keystrokes, this will make you at least as productive with vim as with the text editor you used before. After that, well, read Jim Dennis' answer again :)

dash-tom-bang, Aug 24, 2011 at 0:47

This is the first thing I thought of when reading the OP. It's obvious that the poster has never run this; I ran through it when first learning vim two years ago and it cemented in my mind the superiority of Vim to any of the other editors I've used (including, for me, Emacs since the key combos are annoying to use on a Mac). – dash-tom-bang Aug 24 '11 at 0:47

community wiki
Johnsyweb
, Jan 12, 2011 at 22:52

What is the way you use Vim that makes you more productive than with a contemporary editor?

Being able to execute complex, repetitive edits with very few keystrokes (often using macros ). Take a look at VimGolf to witness the power of Vim!

After over ten years of almost daily usage, it's hard to imagine using any other editor.

community wiki
2 revs, 2 users 67%
,Jun 15, 2011 at 13:42

Use \c anywhere in a search to ignore case (overriding your ignorecase or smartcase settings). E.g. /\cfoo or /foo\c will match foo, Foo, fOO, FOO, etc.

Use \C anywhere in a search to force case matching. E.g. /\Cfoo or /foo\C will only match foo.

community wiki
2 revs, 2 users 67%
,Jun 15, 2011 at 13:44

I was surprised to find no one mention the t movement. I frequently use it with parameter lists in the form of dt, or yt,

hhh, Jan 14, 2011 at 17:09

or dfx, dFx, dtx, ytx, etc where x is a char, +1 – hhh Jan 14 '11 at 17:09

dash-tom-bang, Aug 24, 2011 at 0:48

@hhh yep, T t f and F are all pretty regular keys for me to hit... – dash-tom-bang Aug 24 '11 at 0:48

markle976, Mar 30, 2012 at 13:52

Yes! And don't forget ct (change to). – markle976 Mar 30 '12 at 13:52

sjas, Jun 24, 2012 at 23:35

t for teh win!!! – sjas Jun 24 '12 at 23:35

community wiki
3 revs
,May 6, 2012 at 20:50

Odd nobody's mentioned ctags. Download "exuberant ctags" and put it ahead of the crappy preinstalled version you already have in your search path. Cd to the root of whatever you're working on; for example the Android kernel distribution. Type "ctags -R ." to build an index of source files anywhere beneath that dir in a file named "tags". This contains all tags, nomatter the language nor where in the dir, in one file, so cross-language work is easy.

Then open vim in that folder and read :help ctags for some commands. A few I use often:

community wiki
2 revs, 2 users 67%
,Feb 27, 2010 at 11:19

Automatic indentation:

gg (go to start of document)
= (indent time!)
shift-g (go to end of document)

You'll need 'filetype plugin indent on' in your .vimrc file, and probably appropriate 'shiftwidth' and 'expandtab' settings.

xcramps, Aug 28, 2009 at 17:14

Or just use the ":set ai" (auto-indent) facility, which has been in vi since the beginning. – xcramps Aug 28 '09 at 17:14

community wiki
autodidakto
, Jul 24, 2010 at 5:41

You asked about productive shortcuts, but I think your real question is: Is vim worth it? The answer to this stackoverflow question is -> "Yes"

You must have noticed two things. Vim is powerful, and vim is hard to learn. Much of it's power lies in it's expandability and endless combination of commands. Don't feel overwhelmed. Go slow. One command, one plugin at a time. Don't overdo it.

All that investment you put into vim will pay back a thousand fold. You're going to be inside a text editor for many, many hours before you die. Vim will be your companion.

community wiki
2 revs, 2 users 67%
,Feb 27, 2010 at 11:23

Multiple buffers, and in particular fast jumping between them to compare two files with :bp and :bn (properly remapped to a single Shift + p or Shift + n )

vimdiff mode (splits in two vertical buffers, with colors to show the differences)

Area-copy with Ctrl + v

And finally, tab completion of identifiers (search for "mosh_tab_or_complete"). That's a life changer.

community wiki
David Wolever
, Aug 28, 2009 at 16:07

Agreed with the top poster - the :r! command is very useful.

Most often I use it to "paste" things:

:r!cat
**Ctrl-V to paste from the OS clipboard**
^D

This way I don't have to fiddle with :set paste .

R. Martinho Fernandes, Apr 1, 2010 at 3:17

Probably better to set the clipboard option to unnamed ( set clipboard=unnamed in your .vimrc) to use the system clipboard by default. Or if you still want the system clipboard separate from the unnamed register, use the appropriately named clipboard register: "*p . – R. Martinho Fernandes Apr 1 '10 at 3:17

kevpie, Oct 12, 2010 at 22:38

Love it! After being exasperated by pasting code examples from the web and I was just starting to feel proficient in vim. That was the command I dreamed up on the spot. This was when vim totally hooked me. – kevpie Oct 12 '10 at 22:38

Ben Mordecai, Feb 6, 2013 at 19:54

If you're developing on a Mac, Command+C and Command+V copy and paste using the system clipboard, no remap required. – Ben Mordecai Feb 6 '13 at 19:54

David Wolever, Feb 6, 2013 at 20:55

Only with GVIm From the console, pasting without :set paste doesn't work so well if autoindent is enabled. – David Wolever Feb 6 '13 at 20:55

[Oct 21, 2018] What are the dark corners of Vim your mom never told you about?

Notable quotes:
"... Want to look at your :command history? q: Then browse, edit and finally to execute the command. ..."
"... from the ex editor (:), you can do CTRL-f to pop up the command history window. ..."
"... q/ and q? can be used to do a similar thing for your search patterns. ..."
"... adjacent to the one I just edit ..."
Nov 16, 2011 | stackoverflow.com

Ask Question, Nov 16, 2011 at 0:44

There are a plethora of questions where people talk about common tricks, notably " Vim+ctags tips and tricks ".

However, I don't refer to commonly used shortcuts that someone new to Vim would find cool. I am talking about a seasoned Unix user (be they a developer, administrator, both, etc.), who thinks they know something 99% of us never heard or dreamed about. Something that not only makes their work easier, but also is COOL and hackish .

After all, Vim resides in the most dark-corner-rich OS in the world, thus it should have intricacies that only a few privileged know about and want to share with us.

user3218088, Jun 16, 2014 at 9:51

:Sex -- Split window and open integrated file explorer (horizontal split) – user3218088 Jun 16 '14 at 9:51

community wiki, 2 revs, Apr 7, 2009 at 19:04

Might not be one that 99% of Vim users don't know about, but it's something I use daily and that any Linux+Vim poweruser must know.

Basic command, yet extremely useful.

:w !sudo tee %

I often forget to sudo before editing a file I don't have write permissions on. When I come to save that file and get a permission error, I just issue that vim command in order to save the file without the need to save it to a temp file and then copy it back again.

You obviously have to be on a system with sudo installed and have sudo rights.

jm666, May 12, 2011 at 6:09

cmap w!! w !sudo tee % – jm666 May 12 '11 at 6:09

Gerardo Marset, Jul 5, 2011 at 0:49

You should never run sudo vim . Instead you should export EDITOR as vim and run sudoedit . – Gerardo Marset Jul 5 '11 at 0:49

migu, Sep 2, 2013 at 20:42

@maximus: vim replaces % by the name of the current buffer/file. – migu Sep 2 '13 at 20:42

community wiki
Chad Birch
, Apr 7, 2009 at 18:09

Something I just discovered recently that I thought was very cool:
:earlier 15m

Reverts the document back to how it was 15 minutes ago. Can take various arguments for the amount of time you want to roll back, and is dependent on undolevels. Can be reversed with the opposite command :later

ephemient, Apr 8, 2009 at 16:15

@skinp: If you undo and then make further changes from the undone state, you lose that redo history. This lets you go back to a state which is no longer in the undo stack. – ephemient Apr 8 '09 at 16:15

Etienne PIERRE, Jul 21, 2009 at 13:53

Also very usefull is g+ and g- to go backward and forward in time. This is so much more powerfull than an undo/redo stack since you don't loose the history when you do something after an undo. – Etienne PIERRE Jul 21 '09 at 13:53

Ehtesh Choudhury, Nov 29, 2011 at 12:09

You don't lose the redo history if you make a change after an undo. It's just not easily accessed. There are plugins to help you visualize this, like Gundo.vim – Ehtesh Choudhury Nov 29 '11 at 12:09

Igor Popov, Dec 29, 2011 at 6:59

Wow, so now I can just do :later 8h and I'm done for today? :P – Igor Popov Dec 29 '11 at 6:59

Ring Ø, Jul 11, 2014 at 5:14

Your command assumes one will spend at least 15 minutes in vim ! – Ring Ø Jul 11 '14 at 5:14

community wiki,2 revs, 2 users 92%, ,Mar 31, 2016 at 17:54

:! [command] executes an external command while you're in Vim.

But add a dot after the colon, :.! [command], and it'll dump the output of the command into your current window. That's : . !

For example:

:.! ls

I use this a lot for things like adding the current date into a document I'm typing:

:.! date

saffsd, May 6, 2009 at 14:41

This is quite similar to :r! The only difference as far as I can tell is that :r! opens a new line, :.! overwrites the current line. – saffsd May 6 '09 at 14:41

hlovdal, Jan 25, 2010 at 21:11

An alternative to :.!date is to write "date" on a line and then run !$sh (alternatively having the command followed by a blank line and run !jsh ). This will pipe the line to the "sh" shell and substitute with the output from the command. – hlovdal Jan 25 '10 at 21:11

Nefrubyr, Mar 25, 2010 at 16:24

:.! is actually a special case of :{range}!, which filters a range of lines (the current line when the range is . ) through a command and replaces those lines with the output. I find :%! useful for filtering whole buffers. – Nefrubyr Mar 25 '10 at 16:24

jabirali, Jul 13, 2010 at 4:30

@sundar: Why pass a line to sed, when you can use the similar built-in ed / ex commands? Try running :.s/old/new/g ;-) – jabirali Jul 13 '10 at 4:30

aqn, Apr 26, 2013 at 20:52

And also note that '!' is like 'y', 'd', 'c' etc. i.e. you can do: !!, number!!, !motion (e.g. !Gshell_command<cr> replace from current line to end of file ('G') with output of shell_command). – aqn Apr 26 '13 at 20:52

community wiki 2 revs , Apr 8, 2009 at 12:17

Not exactly obscure, but there are several "delete in" commands which are extremely useful, like..

Others can be found on :help text-objects

sjh, Apr 8, 2009 at 15:33

dab "delete arounb brackets", daB for around curly brackets, t for xml type tags, combinations with normal commands are as expected cib/yaB/dit/vat etc – sjh Apr 8 '09 at 15:33

Don Reba, Apr 13, 2009 at 21:41

@Masi: yi(va(p deletes only the brackets – Don Reba Apr 13 '09 at 21:41

thomasrutter, Apr 26, 2009 at 11:11

This is possibly the biggest reason for me staying with Vim. That and its equivalent "change" commands: ciw, ci(, ci", as well as dt<space> and ct<space> – thomasrutter Apr 26 '09 at 11:11

Roger Pate Oct 12 '10 at 16:40 ,

@thomasrutter: Why not dW/cW instead of dt<space>? –

Roger Pate Oct 12 '10 at 16:43, Oct 12, 2010 at 16:43

@Masi: With the surround plugin: ds(. –

community wiki, 9 revs, 9 users 84%, ultraman, Apr 21, 2017 at 14:06

de Delete everything till the end of the word by pressing . at your heart's desire.

ci(xyz[Esc] -- This is a weird one. Here, the 'i' does not mean insert mode. Instead it means inside the parenthesis. So this sequence cuts the text inside parenthesis you're standing in and replaces it with "xyz". It also works inside square and figure brackets -- just do ci[ or ci{ correspondingly. Naturally, you can do di (if you just want to delete all text without typing anything. You can also do a instead of i if you want to delete the parentheses as well and not just text inside them.

ci" - cuts the text in current quotes

ciw - cuts the current word. This works just like the previous one except that ( is replaced with w .

C - cut the rest of the line and switch to insert mode.

ZZ -- save and close current file (WAY faster than Ctrl-F4 to close the current tab!)

ddp - move current line one row down

xp -- move current character one position to the right

U - uppercase, so viwU upercases the word

~ - switches case, so viw~ will reverse casing of entire word

Ctrl+u / Ctrl+d scroll the page half-a-screen up or down. This seems to be more useful than the usual full-screen paging as it makes it easier to see how the two screens relate. For those who still want to scroll entire screen at a time there's Ctrl+f for Forward and Ctrl+b for Backward. Ctrl+Y and Ctrl+E scroll down or up one line at a time.

Crazy but very useful command is zz -- it scrolls the screen to make this line appear in the middle. This is excellent for putting the piece of code you're working on in the center of your attention. Sibling commands -- zt and zb -- make this line the top or the bottom one on the sreen which is not quite as useful.

% finds and jumps to the matching parenthesis.

de -- delete from cursor to the end of the word (you can also do dE to delete until the next space)

bde -- delete the current word, from left to right delimiter

df[space] -- delete up until and including the next space

dt. -- delete until next dot

dd -- delete this entire line

ye (or yE) -- yanks text from here to the end of the word

ce - cuts through the end of the word

bye -- copies current word (makes me wonder what "hi" does!)

yy -- copies the current line

cc -- cuts the current line, you can also do S instead. There's also lower cap s which cuts current character and switches to insert mode.

viwy or viwc . Yank or change current word. Hit w multiple times to keep selecting each subsequent word, use b to move backwards

vi{ - select all text in figure brackets. va{ - select all text including {}s

vi(p - highlight everything inside the ()s and replace with the pasted text

b and e move the cursor word-by-word, similarly to how Ctrl+Arrows normally do . The definition of word is a little different though, as several consecutive delmiters are treated as one word. If you start at the middle of a word, pressing b will always get you to the beginning of the current word, and each consecutive b will jump to the beginning of the next word. Similarly, and easy to remember, e gets the cursor to the end of the current, and each subsequent, word.

similar to b / e, capital B and E move the cursor word-by-word using only whitespaces as delimiters.

capital D (take a deep breath) Deletes the rest of the line to the right of the cursor, same as Shift+End/Del in normal editors (notice 2 keypresses -- Shift+D -- instead of 3)

Nick Lewis, Jul 17, 2009 at 16:41

zt is quite useful if you use it at the start of a function or class definition. – Nick Lewis Jul 17 '09 at 16:41

Nathan Fellman, Sep 7, 2009 at 8:27

vity and vitc can be shortened to yit and cit respectively. – Nathan Fellman Sep 7 '09 at 8:27

Laurence Gonsalves, Feb 19, 2011 at 23:49

All the things you're calling "cut" is "change". eg: C is change until the end of the line. Vim's equivalent of "cut" is "delete", done with d/D. The main difference between change and delete is that delete leaves you in normal mode but change puts you into a sort of insert mode (though you're still in the change command which is handy as the whole change can be repeated with . ). – Laurence Gonsalves Feb 19 '11 at 23:49

Almo, May 29, 2012 at 20:09

I thought this was for a list of things that not many people know. yy is very common, I would have thought. – Almo May 29 '12 at 20:09

Andrea Francia, Jul 3, 2012 at 20:50

bye does not work when you are in the first character of the word. yiw always does. – Andrea Francia Jul 3 '12 at 20:50

community wiki 2 revs, 2 users 83%, ,Sep 17, 2010 at 16:55

One that I rarely find in most Vim tutorials, but it's INCREDIBLY useful (at least to me), is the

g; and g,

to move (forward, backward) through the changelist.

Let me show how I use it. Sometimes I need to copy and paste a piece of code or string, say a hex color code in a CSS file, so I search, jump (not caring where the match is), copy it and then jump back (g;) to where I was editing the code to finally paste it. No need to create marks. Simpler.

Just my 2cents.

aehlke, Feb 12, 2010 at 1:19

similarly, '. will go to the last edited line, And `. will go to the last edited position – aehlke Feb 12 '10 at 1:19

Kimball Robinson, Apr 16, 2010 at 0:29

Ctrl-O and Ctrl-I (tab) will work similarly, but not the same. They move backward and forward in the "jump list", which you can view by doing :jumps or :ju For more information do a :help jumplist – Kimball Robinson Apr 16 '10 at 0:29

Kimball Robinson, Apr 16, 2010 at 0:30

You can list the change list by doing :changes – Kimball Robinson Apr 16 '10 at 0:30

Wayne Werner, Jan 30, 2013 at 14:49

Hot dang that's useful. I use <C-o> / <C-i> for this all the time - or marking my place. – Wayne Werner Jan 30 '13 at 14:49

community wiki, 4 revs, 4 users 36%, ,May 5, 2014 at 13:06

:%!xxd

Make vim into a hex editor.

:%!xxd -r

Revert.

Warning: If you don't edit with binary (-b), you might damage the file. – Josh Lee in the comments.

Christian, Jul 7, 2009 at 19:11

And how do you revert it back? – Christian Jul 7 '09 at 19:11

Naga Kiran, Jul 8, 2009 at 13:46

:!xxd -r //To revert back from HEX – Naga Kiran Jul 8 '09 at 13:46

Andreas Grech, Nov 14, 2009 at 10:37

I actually think it's :%!xxd -r to revert it back – Andreas Grech Nov 14 '09 at 10:37

dotancohen, Jun 7, 2013 at 5:50

@JoshLee: If one is careful not to traverse newlines, is it safe to not use the -b option? I ask because sometimes I want to make a hex change, but I don't want to close and reopen the file to do so. – dotancohen Jun 7 '13 at 5:50

Bambu, Nov 23, 2014 at 23:58

@dotancohen: If you don't want to close/reopen the file you can do :set binary – Bambu Nov 23 '14 at 23:58

community wiki AaronS, Jan 12, 2011 at 20:03

gv

Reselects last visual selection.

community wiki 3 revs, 2 users 92%
,Jul 7, 2014 at 19:10

Sometimes a setting in your .vimrc will get overridden by a plugin or autocommand. To debug this a useful trick is to use the :verbose command in conjunction with :set. For example, to figure out where cindent got set/unset:
:verbose set cindent?

This will output something like:

cindent
    Last set from /usr/share/vim/vim71/indent/c.vim

This also works with maps and highlights. (Thanks joeytwiddle for pointing this out.) For example:

:verbose nmap U
n  U             <C-R>
        Last set from ~/.vimrc

:verbose highlight Normal
Normal         xxx guifg=#dddddd guibg=#111111 font=Inconsolata Medium 14
        Last set from ~/src/vim-holodark/colors/holodark.vim

Artem Russakovskii, Oct 23, 2009 at 22:09

Excellent tip - exactly what I was looking for today. – Artem Russakovskii Oct 23 '09 at 22:09

joeytwiddle, Jul 5, 2014 at 22:08

:verbose can also be used before nmap l or highlight Normal to find out where the l keymap or the Normal highlight were last defined. Very useful for debugging! – joeytwiddle Jul 5 '14 at 22:08

SidOfc, Sep 24, 2017 at 11:26

When you get into creating custom mappings, this will save your ass so many times, probably one of the most useful ones here (IMO)! – SidOfc Sep 24 '17 at 11:26

community wiki 3 revs, 3 users 70% ,May 31, 2015 at 19:30

Not sure if this counts as dark-corner-ish at all, but I've only just learnt it...
:g/match/y A

will yank (copy) all lines containing "match" into the "a / @a register. (The capitalization as A makes vim append yankings instead of replacing the previous register contents.) I used it a lot recently when making Internet Explorer stylesheets.

tsukimi, May 27, 2012 at 6:17

You can use :g! to find lines that don't match a pattern e.x. :g!/set/normal dd (delete all lines that don't contain set) – tsukimi May 27 '12 at 6:17

pandubear, Oct 12, 2013 at 8:39

Sometimes it's better to do what tsukimi said and just filter out lines that don't match your pattern. An abbreviated version of that command though: :v/PATTERN/d Explanation: :v is an abbreviation for :g!, and the :g command applies any ex command to lines. :y[ank] works and so does :normal, but here the most natural thing to do is just :d[elete] . – pandubear Oct 12 '13 at 8:39

Kimball Robinson, Feb 5, 2016 at 17:58

You can also do :g/match/normal "Ayy -- the normal keyword lets you tell it to run normal-mode commands (which you are probably more familiar with). – Kimball Robinson Feb 5 '16 at 17:58

community wiki 2 revs, 2 users 80% ,Apr 5, 2013 at 15:55

:%TOhtml Creates an html rendering of the current file.

kenorb, Feb 19, 2015 at 11:27

Related: How to convert a source code file into HTML? at Vim SE – kenorb Feb 19 '15 at 11:27

community wiki 2 revs, 2 users 86% ,May 11, 2011 at 19:30

Want to look at your :command history? q: Then browse, edit and finally to execute the command.

Ever make similar changes to two files and switch back and forth between them? (Say, source and header files?)

:set hidden
:map <TAB> :e#<CR>

Then tab back and forth between those files.

Josh Lee, Sep 22, 2009 at 16:58

I hit q: by accident all the time... – Josh Lee Sep 22 '09 at 16:58

Jason Down, Oct 6, 2009 at 4:14

Alternatively, from the ex editor (:), you can do CTRL-f to pop up the command history window.Jason Down Oct 6 '09 at 4:14

bradlis7, Mar 23, 2010 at 17:10

@jleedev me too. I almost hate this command, just because I use it accidentally way too much. – bradlis7 Mar 23 '10 at 17:10

bpw1621, Feb 19, 2011 at 15:01

q/ and q? can be used to do a similar thing for your search patterns. bpw1621 Feb 19 '11 at 15:01

idbrii, Feb 23, 2011 at 19:07

Hitting <C-f> after : or / (or any time you're in command mode) will bring up the same history menu. So you can remap q: if you hit it accidentally a lot and still access this awesome mode. – idbrii Feb 23 '11 at 19:07

community wiki, 2 revs, 2 users 89%, ,Jun 4, 2014 at 14:52

Vim will open a URL, for example
vim http://stackoverflow.com/

Nice when you need to pull up the source of a page for reference.

Ivan Vučica, Sep 21, 2010 at 8:07

For me it didn't open the source; instead it apparently used elinks to dump rendered page into a buffer, and then opened that. – Ivan Vučica Sep 21 '10 at 8:07

Thomas, Apr 19, 2013 at 21:00

Works better with a slash at the end. Neat trick! – Thomas Apr 19 '13 at 21:00

Isaac Remuant, Jun 3, 2013 at 15:23

@Vdt: It'd be useful if you posted your error. If it's this one: " error (netrw) neither the wget nor the fetch command is available" you obviously need to make one of those tools available from your PATH environment variable. – Isaac Remuant Jun 3 '13 at 15:23

Dettorer, Oct 29, 2014 at 13:47

I find this one particularly useful when people send links to a paste service and forgot to select a syntax highlighting, I generally just have to open the link in vim after appending "&raw". – Dettorer Oct 29 '14 at 13:47

community wiki 2 revs, 2 users 94% ,Jan 20, 2015 at 23:14

Macros can call other macros, and can also call itself.

eg:

qq0dwj@qq@q

...will delete the first word from every line until the end of the file.

This is quite a simple example but it demonstrates a very powerful feature of vim

Kimball Robinson, Apr 16, 2010 at 0:39

I didn't know macros could repeat themselves. Cool. Note: qx starts recording into register x (he uses qq for register q). 0 moves to the start of the line. dw delets a word. j moves down a line. @q will run the macro again (defining a loop). But you forgot to end the recording with a final "q", then actually run the macro by typing @q. – Kimball Robinson Apr 16 '10 at 0:39

Yktula, Apr 18, 2010 at 5:32

I think that's intentional, as a nested and recursive macro. – Yktula Apr 18 '10 at 5:32

Gerardo Marset, Jul 5, 2011 at 1:38

qqqqqifuu<Esc>h@qq@qGerardo Marset Jul 5 '11 at 1:38

Nathan Long, Aug 29, 2011 at 15:33

Another way of accomplishing this is to record a macro in register a that does some transformation to a single line, then linewise highlight a bunch of lines with V and type :normal! @a to applyyour macro to every line in your selection. – Nathan Long Aug 29 '11 at 15:33

dotancohen, May 14, 2013 at 6:00

I found this post googling recursive VIM macros. I could find no way to stop the macro other than killing the VIM process. – dotancohen May 14 '13 at 6:00

community wiki
Brian Carper
, Apr 8, 2009 at 1:15

Assuming you have Perl and/or Ruby support compiled in, :rubydo and :perldo will run a Ruby or Perl one-liner on every line in a range (defaults to entire buffer), with $_ bound to the text of the current line (minus the newline). Manipulating $_ will change the text of that line.

You can use this to do certain things that are easy to do in a scripting language but not so obvious using Vim builtins. For example to reverse the order of the words in a line:

:perldo $_ = join ' ', reverse split

To insert a random string of 8 characters (A-Z) at the end of every line:

:rubydo $_ += ' ' + (1..8).collect{('A'..'Z').to_a[rand 26]}.join

You are limited to acting on one line at a time and you can't add newlines.

Sujoy, May 6, 2009 at 18:27

what if i only want perldo to run on a specified line? or a selected few lines? – Sujoy May 6 '09 at 18:27

Brian Carper, May 6, 2009 at 18:52

You can give it a range like any other command. For example :1,5perldo will only operate on lines 1-5. – Brian Carper May 6 '09 at 18:52

Greg, Jul 2, 2009 at 16:41

Could you do $_ += '\nNEWLINE!!!' to get a newline after the current one? – Greg Jul 2 '09 at 16:41

Brian Carper, Jul 2, 2009 at 17:26

Sadly not, it just adds a funky control character to the end of the line. You could then use a Vim search/replace to change all those control characters to real newlines though. – Brian Carper Jul 2 '09 at 17:26

Derecho, Mar 14, 2014 at 8:48

Similarly, pydo and py3do work for python if you have the required support compiled in. – Derecho Mar 14 '14 at 8:48

community wiki
4 revs
,Jul 28, 2009 at 19:05

^O and ^I

Go to older/newer position. When you are moving through the file (by searching, moving commands etc.) vim rember these "jumps", so you can repeat these jumps backward (^O - O for old) and forward (^I - just next to I on keyboard). I find it very useful when writing code and performing a lot of searches.

gi

Go to position where Insert mode was stopped last. I find myself often editing and then searching for something. To return to editing place press gi.

gf

put cursor on file name (e.g. include header file), press gf and the file is opened

gF

similar to gf but recognizes format "[file name]:[line number]". Pressing gF will open [file name] and set cursor to [line number].

^P and ^N

Auto complete text while editing (^P - previous match and ^N next match)

^X^L

While editing completes to the same line (useful for programming). You write code and then you recall that you have the same code somewhere in file. Just press ^X^L and the full line completed

^X^F

Complete file names. You write "/etc/pass" Hmm. You forgot the file name. Just press ^X^F and the filename is completed

^Z or :sh

Move temporary to the shell. If you need a quick bashing:

sehe, Mar 4, 2012 at 21:50

With ^X^F my pet peeve is that filenames include = signs, making it do rotten things in many occasions (ini files, makefiles etc). I use se isfname-== to end that nuisance – sehe Mar 4 '12 at 21:50

joeytwiddle, Jul 5, 2014 at 22:10

+1 the built-in autocomplete is just sitting there waiting to be discovered. – joeytwiddle Jul 5 '14 at 22:10

community wiki
2 revs
,Apr 7, 2009 at 18:59

This is a nice trick to reopen the current file with a different encoding:
:e ++enc=cp1250 %:p

Useful when you have to work with legacy encodings. The supported encodings are listed in a table under encoding-values (see help encoding-values ). Similar thing also works for ++ff, so that you can reopen file with Windows/Unix line ends if you get it wrong for the first time (see help ff ).

>, Apr 7, 2009 at 18:43

Never had to use this sort of a thing, but we'll certainly add to my arsenal of tricks... – Sasha Apr 7 '09 at 18:43

Adriano Varoli Piazza, Apr 7, 2009 at 18:44

great tip, thanks. For bonus points, add a list of common valid encodings. – Adriano Varoli Piazza Apr 7 '09 at 18:44

Ivan Vučica, Jul 8, 2009 at 19:29

I have used this today, but I think I didn't need to specify "%:p"; just opening the file and :e ++enc=cp1250 was enough. I – Ivan Vučica Jul 8 '09 at 19:29

laz, Jul 8, 2009 at 19:32

would :set encoding=cp1250 have the same effect? – laz Jul 8 '09 at 19:32

intuited, Jun 4, 2010 at 2:51

`:e +b %' is similarly useful for reopening in binary mode (no munging of newlines) – intuited Jun 4 '10 at 2:51

community wiki
4 revs, 3 users 48%
,Nov 6, 2012 at 8:32

" insert range ip's
"
"          ( O O )
" =======oOO=(_)==OOo======

:for i in range(1,255) | .put='10.0.0.'.i | endfor

Ryan Edwards, Nov 16, 2011 at 0:42

I don't see what this is good for (besides looking like a joke answer). Can anybody else enlighten me? – Ryan Edwards Nov 16 '11 at 0:42

Codygman, Nov 6, 2012 at 8:33

open vim and then do ":for i in range(1,255) | .put='10.0.0.'.i | endfor" – Codygman Nov 6 '12 at 8:33

Ruslan, Sep 30, 2013 at 10:30

@RyanEdwards filling /etc/hosts maybe – Ruslan Sep 30 '13 at 10:30

dotancohen, Nov 30, 2014 at 14:56

This is a terrific answer. Not the bit about creating the IP addresses, but the bit that implies that VIM can use for loops in commands . – dotancohen Nov 30 '14 at 14:56

BlackCap, Aug 31, 2017 at 7:54

Without ex-mode: i10.0.0.1<Esc>Y254p$<C-v>}g<C-a>BlackCap Aug 31 '17 at 7:54

community wiki
2 revs
,Aug 6, 2010 at 0:30

Typing == will correct the indentation of the current line based on the line above.

Actually, you can do one = sign followed by any movement command. = {movement}

For example, you can use the % movement which moves between matching braces. Position the cursor on the { in the following code:

if (thisA == that) {
//not indented
if (some == other) {
x = y;
}
}

And press =% to instantly get this:

if (thisA == that) {
    //not indented
    if (some == other) {
        x = y;
    }
}

Alternately, you could do =a{ within the code block, rather than positioning yourself right on the { character.

Ehtesh Choudhury, May 2, 2011 at 0:48

Hm, I didn't know this about the indentation. – Ehtesh Choudhury May 2 '11 at 0:48

sehe, Mar 4, 2012 at 22:03

No need, usually, to be exactly on the braces. Thought frequently I'd just =} or vaBaB= because it is less dependent. Also, v}}:!astyle -bj matches my code style better, but I can get it back into your style with a simple %!astyle -ajsehe Mar 4 '12 at 22:03

kyrias, Oct 19, 2013 at 12:12

gg=G is quite neat when pasting in something. – kyrias Oct 19 '13 at 12:12

kenorb, Feb 19, 2015 at 11:30

Related: Re-indenting badly indented code at Vim SE – kenorb Feb 19 '15 at 11:30

Braden Best, Feb 4, 2016 at 16:16

@kyrias Oh, I've been doing it like ggVG= . – Braden Best Feb 4 '16 at 16:16

community wiki
Trumpi
, Apr 19, 2009 at 18:33

imap jj <esc>

hasen, Jun 12, 2009 at 6:08

how will you type jj then? :P – hasen Jun 12 '09 at 6:08

ojblass, Jul 5, 2009 at 18:29

How often to you type jj? In English at least? – ojblass Jul 5 '09 at 18:29

Alex, Oct 5, 2009 at 5:32

I remapped capslock to esc instead, as it's an otherwise useless key. My mapping was OS wide though, so it has the added benefit of never having to worry about accidentally hitting it. The only drawback IS ITS HARDER TO YELL AT PEOPLE. :) – Alex Oct 5 '09 at 5:32

intuited, Jun 4, 2010 at 4:18

@Alex: definitely, capslock is death. "wait, wtf? oh, that was ZZ?....crap." – intuited Jun 4 '10 at 4:18

brianmearns, Oct 3, 2012 at 12:45

@ojblass: Not sure how many people ever right matlab code in Vim, but ii and jj are commonly used for counter variables, because i and j are reserved for complex numbers. – brianmearns Oct 3 '12 at 12:45

community wiki
4 revs, 3 users 71%
,Feb 12, 2015 at 15:55

Let's see some pretty little IDE editor do column transposition.
:%s/\(.*\)^I\(.*\)/\2^I\1/

Explanation

\( and \) is how to remember stuff in regex-land. And \1, \2 etc is how to retrieve the remembered stuff.

>>> \(.*\)^I\(.*\)

Remember everything followed by ^I (tab) followed by everything.

>>> \2^I\1

Replace the above stuff with "2nd stuff you remembered" followed by "1st stuff you remembered" - essentially doing a transpose.

chaos, Apr 7, 2009 at 18:33

Switches a pair of tab-separated columns (separator arbitrary, it's all regex) with each other. – chaos Apr 7 '09 at 18:33

rlbond, Apr 26, 2009 at 4:11

This is just a regex; plenty of IDEs have regex search-and-replace. – rlbond Apr 26 '09 at 4:11

romandas, Jun 19, 2009 at 16:58

@rlbond - It comes down to how good is the regex engine in the IDE. Vim's regexes are pretty powerful; others.. not so much sometimes. – romandas Jun 19 '09 at 16:58

Kimball Robinson, Apr 16, 2010 at 0:32

The * will be greedy, so this regex assumes you have just two columns. If you want it to be nongreedy use {-} instead of * (see :help non-greedy for more information on the {} multiplier) – Kimball Robinson Apr 16 '10 at 0:32

mk12, Jun 22, 2012 at 17:31

This is actually a pretty simple regex, it's only escaping the group parentheses that makes it look complicated. – mk12 Jun 22 '12 at 17:31

community wiki
KKovacs
, Apr 11, 2009 at 7:14

Not exactly a dark secret, but I like to put the following mapping into my .vimrc file, so I can hit "-" (minus) anytime to open the file explorer to show files adjacent to the one I just edit . In the file explorer, I can hit another "-" to move up one directory, providing seamless browsing of a complex directory structures (like the ones used by the MVC frameworks nowadays):
map - :Explore<cr>

These may be also useful for somebody. I like to scroll the screen and advance the cursor at the same time:

map <c-j> j<c-e>
map <c-k> k<c-y>

Tab navigation - I love tabs and I need to move easily between them:

map <c-l> :tabnext<enter>
map <c-h> :tabprevious<enter>

Only on Mac OS X: Safari-like tab navigation:

map <S-D-Right> :tabnext<cr>
map <S-D-Left> :tabprevious<cr>

Roman Plášil, Oct 1, 2009 at 21:33

You can also browse files within Vim itself, using :Explore – Roman Plášil Oct 1 '09 at 21:33

KKovacs, Oct 15, 2009 at 15:20

Hi Roman, this is exactly what this mapping does, but assigns it to a "hot key". :) – KKovacs Oct 15 '09 at 15:20

community wiki
rampion
, Apr 7, 2009 at 20:11

Often, I like changing current directories while editing - so I have to specify paths less.
cd %:h

Leonard, May 8, 2009 at 1:54

What does this do? And does it work with autchdir? – Leonard May 8 '09 at 1:54

rampion, May 8, 2009 at 2:55

I suppose it would override autochdir temporarily (until you switched buffers again). Basically, it changes directory to the root directory of the current file. It gives me a bit more manual control than autochdir does. – rampion May 8 '09 at 2:55

Naga Kiran, Jul 8, 2009 at 13:44

:set autochdir //this also serves the same functionality and it changes the current directory to that of file in buffer – Naga Kiran Jul 8 '09 at 13:44

community wiki
4 revs
,Jul 21, 2009 at 1:12

I like to use 'sudo bash', and my sysadmin hates this. He locked down 'sudo' so it could only be used with a handful of commands (ls, chmod, chown, vi, etc), but I was able to use vim to get a root shell anyway:
bash$ sudo vi +'silent !bash' +q
Password: ******
root#

RJHunter, Jul 21, 2009 at 0:53

FWIW, sudoedit (or sudo -e) edits privileged files but runs your editor as your normal user. – RJHunter Jul 21 '09 at 0:53

sundar, Sep 23, 2009 at 9:41

@OP: That was cunning. :) – sundar Sep 23 '09 at 9:41

jnylen, Feb 22, 2011 at 15:58

yeah... I'd hate you too ;) you should only need a root shell VERY RARELY, unless you're already in the habit of running too many commands as root which means your permissions are all screwed up. – jnylen Feb 22 '11 at 15:58

d33tah, Mar 30, 2014 at 17:50

Why does your sysadmin even give you root? :D – d33tah Mar 30 '14 at 17:50

community wiki
Taurus Olson
, Apr 7, 2009 at 21:11

I often use many windows when I work on a project and sometimes I need to resize them. Here's what I use:
map + <C-W>+
map - <C-W>-

These mappings allow to increase and decrease the size of the current window. It's quite simple but it's fast.

Bill Lynch, Apr 8, 2009 at 2:49

There's also Ctrl-W =, which makes the windows equal width. – Bill Lynch Apr 8 '09 at 2:49

joeytwiddle, Jan 29, 2012 at 18:12

Don't forget you can prepend numbers to perform an action multiple times in Vim. So to expand the current window height by 8 lines: 8<C-W>+ – joeytwiddle Jan 29 '12 at 18:12

community wiki
Roberto Bonvallet
, May 6, 2009 at 7:38

:r! <command>

pastes the output of an external command into the buffer.

Do some math and get the result directly in the text:

:r! echo $((3 + 5 + 8))

Get the list of files to compile when writing a Makefile:

:r! ls *.c

Don't look up that fact you read on wikipedia, have it directly pasted into the document you are writing:

:r! lynx -dump http://en.wikipedia.org/wiki/Whatever

Sudhanshu, Jun 7, 2010 at 8:40

^R=3+5+8 in insert mode will let you insert the value of the expression (3+5+8) in text with fewer keystrokes. – Sudhanshu Jun 7 '10 at 8:40

dcn, Mar 27, 2011 at 10:13

How can I get the result/output to a different buffer than the current? – dcn Mar 27 '11 at 10:13

kenorb, Feb 19, 2015 at 11:31

Related: How to dump output from external command into editor? at Vim SE – kenorb Feb 19 '15 at 11:31

community wiki
jqno
, Jul 8, 2009 at 19:19

Map F5 to quickly ROT13 your buffer:
map <F5> ggg?G``

You can use it as a boss key :).

sehe, Mar 4, 2012 at 21:57

I don't know what you are writing... But surely, my boss would be more curious when he saw me write ROT13 jumble :) – sehe Mar 4 '12 at 21:57

romeovs, Jun 19, 2014 at 19:22

or to spoof your friends: nmap i ggg?G`` . Or the diabolical: nmap i ggg?G``i ! – romeovs Jun 19 '14 at 19:22

Amit Gold, Aug 7, 2016 at 10:14

@romeovs 2nd one is infinite loop, use nnoremap – Amit Gold Aug 7 '16 at 10:14

community wiki
mohi666
, Mar 4, 2011 at 2:20

Not an obscure feature, but very useful and time saving.

If you want to save a session of your open buffers, tabs, markers and other settings, you can issue the following:

mksession session.vim

You can open your session using:

vim -S session.vim

TankorSmash, Nov 3, 2012 at 13:45

You can also :so session.vim inside vim. – TankorSmash Nov 3 '12 at 13:45

community wiki
Grant Limberg
, May 11, 2009 at 21:59

I just found this one today via NSFAQ :

Comment blocks of code.

Enter Blockwise Visual mode by hitting CTRL-V.

Mark the block you wish to comment.

Hit I (capital I) and enter your comment string at the beginning of the line. (// for C++)

Hit ESC and all lines selected will have // prepended to the front of the line.

Neeraj Singh, Jun 17, 2009 at 16:56

I added # to comment out a block of code in ruby. How do I undo it. – Neeraj Singh Jun 17 '09 at 16:56

Grant Limberg, Jun 17, 2009 at 19:29

well, if you haven't done anything else to the file, you can simply type u for undo. Otherwise, I haven't figured that out yet. – Grant Limberg Jun 17 '09 at 19:29

nos, Jul 28, 2009 at 20:00

You can just hit ctrl+v again, mark the //'s and hit x to "uncomment" – nos Jul 28 '09 at 20:00

ZyX, Mar 7, 2010 at 14:18

I use NERDCommenter for this. – ZyX Mar 7 '10 at 14:18

Braden Best, Feb 4, 2016 at 16:23

Commented out code is probably one of the worst types of comment you could possibly put in your code. There are better uses for the awesome block insert. – Braden Best Feb 4 '16 at 16:23

community wiki
2 revs, 2 users 84%
Ian H
, Jul 3, 2015 at 23:44

I use vim for just about any text editing I do, so I often times use copy and paste. The problem is that vim by default will often times distort imported text via paste. The way to stop this is to use
:set paste

before pasting in your data. This will keep it from messing up.

Note that you will have to issue :set nopaste to recover auto-indentation. Alternative ways of pasting pre-formatted text are the clipboard registers ( * and + ), and :r!cat (you will have to end the pasted fragment with ^D).

It is also sometimes helpful to turn on a high contrast color scheme. This can be done with

:color blue

I've noticed that it does not work on all the versions of vim I use but it does on most.

jamessan, Dec 28, 2009 at 8:27

The "distortion" is happening because you have some form of automatic indentation enabled. Using set paste or specifying a key for the pastetoggle option is a common way to work around this, but the same effect can be achieved with set mouse=a as then Vim knows that the flood of text it sees is a paste triggered by the mouse. – jamessan Dec 28 '09 at 8:27

kyrias, Oct 19, 2013 at 12:15

If you have gvim installed you can often (though it depends on what your options your distro compiles vim with) use the X clipboard directly from vim through the * register. For example "*p to paste from the X xlipboard. (It works from terminal vim, too, it's just that you might need the gvim package if they're separate) – kyrias Oct 19 '13 at 12:15

Braden Best, Feb 4, 2016 at 16:26

@kyrias for the record, * is the PRIMARY ("middle-click") register. The clipboard is +Braden Best Feb 4 '16 at 16:26

community wiki
viraptor
, Apr 7, 2009 at 22:29

Here's something not obvious. If you have a lot of custom plugins / extensions in your $HOME and you need to work from su / sudo / ... sometimes, then this might be useful.

In your ~/.bashrc:

export VIMINIT=":so $HOME/.vimrc"

In your ~/.vimrc:

if $HOME=='/root'
        if $USER=='root'
                if isdirectory('/home/your_typical_username')
                        let rtuser = 'your_typical_username'
                elseif isdirectory('/home/your_other_username')
                        let rtuser = 'your_other_username'
                endif
        else
                let rtuser = $USER
        endif
        let &runtimepath = substitute(&runtimepath, $HOME, '/home/'.rtuser, 'g')
endif

It will allow your local plugins to load - whatever way you use to change the user.

You might also like to take the *.swp files out of your current path and into ~/vimtmp (this goes into .vimrc):

if ! isdirectory(expand('~/vimtmp'))
   call mkdir(expand('~/vimtmp'))
endif
if isdirectory(expand('~/vimtmp'))
   set directory=~/vimtmp
else
   set directory=.,/var/tmp,/tmp
endif

Also, some mappings I use to make editing easier - makes ctrl+s work like escape and ctrl+h/l switch the tabs:

inoremap <C-s> <ESC>
vnoremap <C-s> <ESC>
noremap <C-l> gt
noremap <C-h> gT

Kyle Challis, Apr 2, 2014 at 21:18

Just in case you didn't already know, ctrl+c already works like escape. – Kyle Challis Apr 2 '14 at 21:18

shalomb, Aug 24, 2015 at 8:02

I prefer never to run vim as root/under sudo - and would just run the command from vim e.g. :!sudo tee %, :!sudo mv % /etc or even launch a login shell :!sudo -ishalomb Aug 24 '15 at 8:02

community wiki
2 revs, 2 users 67%
,Nov 7, 2009 at 7:54

Ctrl-n while in insert mode will auto complete whatever word you're typing based on all the words that are in open buffers. If there is more than one match it will give you a list of possible words that you can cycle through using ctrl-n and ctrl-p.

community wiki
daltonb
, Feb 22, 2010 at 4:28

gg=G

Corrects indentation for entire file. I was missing my trusty <C-a><C-i> in Eclipse but just found out vim handles it nicely.

sjas, Jul 15, 2012 at 22:43

I find G=gg easier to type. – sjas Jul 15 '12 at 22:43

sri, May 12, 2013 at 16:12

=% should do it too. – sri May 12 '13 at 16:12

community wiki
mohi666
, Mar 24, 2011 at 22:44

Ability to run Vim on a client/server based modes.

For example, suppose you're working on a project with a lot of buffers, tabs and other info saved on a session file called session.vim.

You can open your session and create a server by issuing the following command:

vim --servername SAMPLESERVER -S session.vim

Note that you can open regular text files if you want to create a server and it doesn't have to be necessarily a session.

Now, suppose you're in another terminal and need to open another file. If you open it regularly by issuing:

vim new_file.txt

Your file would be opened in a separate Vim buffer, which is hard to do interactions with the files on your session. In order to open new_file.txt in a new tab on your server use this command:

vim --servername SAMPLESERVER --remote-tab-silent new_file.txt

If there's no server running, this file will be opened just like a regular file.

Since providing those flags every time you want to run them is very tedious, you can create a separate alias for creating client and server.

I placed the followings on my bashrc file:

alias vims='vim --servername SAMPLESERVER'
alias vimc='vim --servername SAMPLESERVER --remote-tab-silent'

You can find more information about this at: http://vimdoc.sourceforge.net/htmldoc/remote.html

community wiki
jm666
, May 11, 2011 at 19:54

Variation of sudo write:

into .vimrc

cmap w!! w !sudo tee % >/dev/null

After reload vim you can do "sudo save" as

:w!!

community wiki
3 revs, 3 users 74%
,Sep 17, 2010 at 17:06

HOWTO: Auto-complete Ctags when using Vim in Bash. For anyone else who uses Vim and Ctags, I've written a small auto-completer function for Bash. Add the following into your ~/.bash_completion file (create it if it does not exist):

Thanks go to stylishpants for his many fixes and improvements.

_vim_ctags() {
    local cur prev

    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"

    case "${prev}" in
        -t)
            # Avoid the complaint message when no tags file exists
            if [ ! -r ./tags ]
            then
                return
            fi

            # Escape slashes to avoid confusing awk
            cur=${cur////\\/}

            COMPREPLY=( $(compgen -W "`awk -vORS=" "  "/^${cur}/ { print \\$1 }" tags`" ) )
            ;;
        *)
            _filedir_xspec
            ;;
    esac
}

# Files matching this pattern are excluded
excludelist='*.@(o|O|so|SO|so.!(conf)|SO.!(CONF)|a|A|rpm|RPM|deb|DEB|gif|GIF|jp?(e)g|JP?(E)G|mp3|MP3|mp?(e)g|MP?(E)G|avi|AVI|asf|ASF|ogg|OGG|class|CLASS)'

complete -F _vim_ctags -f -X "${excludelist}" vi vim gvim rvim view rview rgvim rgview gview

Once you restart your Bash session (or create a new one) you can type:

Code:

~$ vim -t MyC<tab key>

and it will auto-complete the tag the same way it does for files and directories:

Code:

MyClass MyClassFactory
~$ vim -t MyC

I find it really useful when I'm jumping into a quick bug fix.

>, Apr 8, 2009 at 3:05

Amazing....I really needed it – Sasha Apr 8 '09 at 3:05

TREE, Apr 27, 2009 at 13:19

can you summarize? If that external page goes away, this answer is useless. :( – TREE Apr 27 '09 at 13:19

Hamish Downer, May 5, 2009 at 16:38

Summary - it allows ctags autocomplete from the bash prompt for opening files with vim. – Hamish Downer May 5 '09 at 16:38

community wiki
2 revs, 2 users 80%
,Dec 22, 2016 at 7:44

I often want to highlight a particular word/function name, but don't want to search to the next instance of it yet:
map m* *#

René Nyffenegger, Dec 3, 2009 at 7:36

I don't understand this one. – René Nyffenegger Dec 3 '09 at 7:36

Scotty Allen, Dec 3, 2009 at 19:55

Try it:) It basically highlights a given word, without moving the cursor to the next occurrance (like * would). – Scotty Allen Dec 3 '09 at 19:55

jamessan, Dec 27, 2009 at 19:10

You can do the same with "nnoremap m* :let @/ = '\<' . expand('<cword>') . '\>'<cr>" – jamessan Dec 27 '09 at 19:10

community wiki
Ben
, Apr 9, 2009 at 12:37

% is also good when you want to diff files across two different copies of a project without wearing out the pinkies (from root of project1):
:vert diffs /project2/root/%

community wiki
Naga Kiran
, Jul 8, 2009 at 19:07

:setlocal autoread

Auto reloads the current buffer..especially useful while viewing log files and it almost serves the functionality of "tail" program in unix from within vim.

Checking for compile errors from within vim. set the makeprg variable depending on the language let's say for perl

:setlocal makeprg = perl\ -c \ %

For PHP

set makeprg=php\ -l\ %
set errorformat=%m\ in\ %f\ on\ line\ %l

Issuing ":make" runs the associated makeprg and displays the compilation errors/warnings in quickfix window and can easily navigate to the corresponding line numbers.

community wiki
2 revs, 2 users 73%
,Sep 14 at 20:16

Want an IDE?

:make will run the makefile in the current directory, parse the compiler output, you can then use :cn and :cp to step through the compiler errors opening each file and seeking to the line number in question.

:syntax on turns on vim's syntax highlighting.

community wiki
Luper Rouch
, Apr 9, 2009 at 12:53

Input a character from its hexadecimal value (insert mode):
<C-Q>x[type the hexadecimal byte]

MikeyB, Sep 22, 2009 at 21:57

<C-V> is the more generic command that works in both the text-mode and gui – MikeyB Sep 22 '09 at 21:57

jamessan, Dec 27, 2009 at 19:06

It's only <C-q> if you're using the awful mswin.vim (or you mapped it yourself). – jamessan Dec 27 '09 at 19:06

community wiki
Brad Cox
, May 8, 2009 at 1:54

I was sure someone would have posted this already, but here goes.

Take any build system you please; make, mvn, ant, whatever. In the root of the project directory, create a file of the commands you use all the time, like this:

mvn install
mvn clean install
... and so forth

To do a build, put the cursor on the line and type !!sh. I.e. filter that line; write it to a shell and replace with the results.

The build log replaces the line, ready to scroll, search, whatever.

When you're done viewing the log, type u to undo and you're back to your file of commands.

ojblass, Jul 5, 2009 at 18:27

This doesn't seem to fly on my system. Can you show an example only using the ls command? – ojblass Jul 5 '09 at 18:27

Brad Cox, Jul 29, 2009 at 19:30

!!ls replaces current line with ls output (adding more lines as needed). – Brad Cox Jul 29 '09 at 19:30

jamessan, Dec 28, 2009 at 8:29

Why wouldn't you just set makeprg to the proper tool you use for your build (if it isn't set already) and then use :make ? :copen will show you the output of the build as well as allowing you to jump to any warnings/errors. – jamessan Dec 28 '09 at 8:29

community wiki
2 revs, 2 users 95%
,Dec 28, 2009 at 8:38

==========================================================
In normal mode
==========================================================
gf ................ open file under cursor in same window --> see :h path
Ctrl-w f .......... open file under cursor in new window
Ctrl-w q .......... close current window
Ctrl-w 6 .......... open alternate file --> see :h #
gi ................ init insert mode in last insertion position
'0 ................ place the cursor where it was when the file was last edited

Braden Best, Feb 4, 2016 at 16:33

I believe it's <C-w> c to close a window, actually. :h ctrl-wBraden Best Feb 4 '16 at 16:33

community wiki
2 revs, 2 users 84%
,Sep 17, 2010 at 16:53

Due to the latency and lack of colors (I love color schemes :) I don't like programming on remote machines in PuTTY . So I developed this trick to work around this problem. I use it on Windows.

You will need

Setting up remote machine

Configure rsync to make your working directory accessible. I use an SSH tunnel and only allow connections from the tunnel:

address = 127.0.0.1
hosts allow = 127.0.0.1
port = 40000
use chroot = false
[bledge_ce]
    path = /home/xplasil/divine/bledge_ce
    read only = false

Then start rsyncd: rsync --daemon --config=rsyncd.conf

Setting up local machine

Install rsync from Cygwin. Start Pageant and load your private key for the remote machine. If you're using SSH tunelling, start PuTTY to create the tunnel. Create a batch file push.bat in your working directory which will upload changed files to the remote machine using rsync:

rsync --blocking-io *.cc *.h SConstruct rsync://localhost:40001/bledge_ce

SConstruct is a build file for scons. Modify the list of files to suit your needs. Replace localhost with the name of remote machine if you don't use SSH tunelling.

Configuring Vim That is now easy. We will use the quickfix feature (:make and error list), but the compilation will run on the remote machine. So we need to set makeprg:

set makeprg=push\ &&\ plink\ -batch\ xplasil@anna.fi.muni.cz\ \"cd\ /home/xplasil/divine/bledge_ce\ &&\ scons\ -j\ 2\"

This will first start the push.bat task to upload the files and then execute the commands on remote machine using SSH ( Plink from the PuTTY suite). The command first changes directory to the working dir and then starts build (I use scons).

The results of build will show conviniently in your local gVim errors list.

matpie, Sep 17, 2010 at 23:02

A much simpler solution would be to use bcvi: sshmenu.sourceforge.net/articles/bcvimatpie Sep 17 '10 at 23:02

Uri Goren, Jul 20 at 20:21

cmder is much easier and simpler, it also comes with its own ssh client – Uri Goren Jul 20 at 20:21

community wiki
3 revs, 2 users 94%
,Jan 16, 2014 at 14:10

I use Vim for everything. When I'm editing an e-mail message, I use:

gqap (or gwap )

extensively to easily and correctly reformat on a paragraph-by-paragraph basis, even with quote leadin characters. In order to achieve this functionality, I also add:

-c 'set fo=tcrq' -c 'set tw=76'

to the command to invoke the editor externally. One noteworthy addition would be to add ' a ' to the fo (formatoptions) parameter. This will automatically reformat the paragraph as you type and navigate the content, but may interfere or cause problems with errant or odd formatting contained in the message.

Andrew Ferrier, Jul 14, 2014 at 22:22

autocmd FileType mail set tw=76 fo=tcrq in your ~/.vimrc will also work, if you can't edit the external editor command. – Andrew Ferrier Jul 14 '14 at 22:22

community wiki
2 revs, 2 users 94%
,May 6, 2009 at 12:22

Put this in your .vimrc to have a command to pretty-print xml:
function FormatXml()
    %s:\(\S\)\(<[^/]\)\|\(>\)\(</\):\1\3\r\2\4:g
    set filetype=xml
    normal gg=G
endfunction

command FormatXml :call FormatXml()

David Winslow, Nov 24, 2009 at 20:43

On linuxes (where xmllint is pretty commonly installed) I usually just do :%! xmllint - for this. – David Winslow Nov 24 '09 at 20:43

community wiki
searlea
, Aug 6, 2009 at 9:33

:sp %:h - directory listing / file-chooser using the current file's directory

(belongs as a comment under rampion's cd tip, but I don't have commenting-rights yet)

bpw1621, Feb 19, 2011 at 15:13

":e ." does the same thing for your current working directory which will be the same as your current file's directory if you set autochdir – bpw1621 Feb 19 '11 at 15:13

community wiki
2 revs
,Sep 22, 2009 at 22:23

Just before copying and pasting to stackoverflow:
:retab 1
:% s/^I/ /g
:% s/^/    /

Now copy and paste code.

As requested in the comments:

retab 1. This sets the tab size to one. But it also goes through the code and adds extra tabs and spaces so that the formatting does not move any of the actual text (ie the text looks the same after ratab).

% s/^I/ /g: Note the ^I is tthe result of hitting tab. This searches for all tabs and replaces them with a single space. Since we just did a retab this should not cause the formatting to change but since putting tabs into a website is hit and miss it is good to remove them.

% s/^/ /: Replace the beginning of the line with four spaces. Since you cant actually replace the beginning of the line with anything it inserts four spaces at the beging of the line (this is needed by SO formatting to make the code stand out).

vehomzzz, Sep 22, 2009 at 20:52

explain it please... – vehomzzz Sep 22 '09 at 20:52

cmcginty, Sep 22, 2009 at 22:31

so I guess this won't work if you use 'set expandtab' to force all tabs to spaces. – cmcginty Sep 22 '09 at 22:31

Martin York, Sep 23, 2009 at 0:07

@Casey: The first two lines will not apply. The last line will make sure you can just cut and paste into SO. – Martin York Sep 23 '09 at 0:07

Braden Best, Feb 4, 2016 at 16:40

Note that you can achieve the same thing with cat <file> | awk '{print " " $line}' . So try :w ! awk '{print " " $line}' | xclip -i . That's supposed to be four spaces between the ""Braden Best Feb 4 '16 at 16:40

community wiki
Anders Holmberg
, Dec 28, 2009 at 9:21

When working on a project where the build process is slow I always build in the background and pipe the output to a file called errors.err (something like make debug 2>&1 | tee errors.err ). This makes it possible for me to continue editing or reviewing the source code during the build process. When it is ready (using pynotify on GTK to inform me that it is complete) I can look at the result in vim using quickfix . Start by issuing :cf[ile] which reads the error file and jumps to the first error. I personally like to use cwindow to get the build result in a separate window.

community wiki
quabug
, Jul 12, 2011 at 12:21

set colorcolumn=+1 or set cc=+1 for vim 7.3

Luc M, Oct 31, 2012 at 15:12

A short explanation would be appreciated... I tried it and could be very usefull! You can even do something like set colorcolumn=+1,+10,+20 :-) – Luc M Oct 31 '12 at 15:12

DBedrenko, Oct 31, 2014 at 16:17

@LucM If you tried it why didn't you provide an explanation? – DBedrenko Oct 31 '14 at 16:17

mjturner, Aug 19, 2015 at 11:16

colorcolumn allows you to specify columns that are highlighted (it's ideal for making sure your lines aren't too long). In the original answer, set cc=+1 highlights the column after textwidth . See the documentation for more information. – mjturner Aug 19 '15 at 11:16

community wiki
mpe
, May 11, 2009 at 4:39

For making vim a little more like an IDE editor:

Rook, May 11, 2009 at 4:42

How does that make Vim more like an IDE ?? – Rook May 11 '09 at 4:42

mpe, May 12, 2009 at 12:29

I did say "a little" :) But it is something many IDEs do, and some people like it, eg: eclipse.org/screenshots/images/JavaPerspective-WinXP.pngmpe May 12 '09 at 12:29

Rook, May 12, 2009 at 21:25

Yes, but that's like saying yank/paste functions make an editor "a little" more like an IDE. Those are editor functions. Pretty much everything that goes with the editor that concerns editing text and that particular area is an editor function. IDE functions would be, for example, project/files management, connectivity with compiler&linker, error reporting, building automation tools, debugger ... i.e. the stuff that doesn't actually do nothing with editing text. Vim has some functions & plugins so he can gravitate a little more towards being an IDE, but these are not the ones in question. – Rook May 12 '09 at 21:25

Rook, May 12, 2009 at 21:26

After all, an IDE = editor + compiler + debugger + building tools + ... – Rook May 12 '09 at 21:26

Rook, May 12, 2009 at 21:31

Also, just FYI, vim has an option to set invnumber. That way you don't have to "set nu" and "set nonu", i.e. remember two functions - you can just toggle. – Rook May 12 '09 at 21:31

community wiki
2 revs, 2 users 50%
PuzzleCracker
, Sep 13, 2009 at 23:20

I love :ls command.

aehlke, Oct 28, 2009 at 3:16

Well what does it do? – aehlke Oct 28 '09 at 3:16

>, Dec 7, 2009 at 10:51

gives the current file name opened ? – user59634 Dec 7 '09 at 10:51

Nona Urbiz, Dec 20, 2010 at 8:25

:ls lists all the currently opened buffers. :be opens a file in a new buffer, :bn goes to the next buffer, :bp to the previous, :b filename opens buffer filename (it auto-completes too). buffers are distinct from tabs, which i'm told are more analogous to views. – Nona Urbiz Dec 20 '10 at 8:25

community wiki
2 revs, 2 users 80%
,Sep 17, 2010 at 16:45

A few useful ones:
:set nu # displays lines
:44     # go to line 44
'.      # go to last modification line

My favourite: Ctrl + n WORD COMPLETION!

community wiki
2 revs
,Jun 18, 2013 at 11:10

In insert mode, ctrl + x, ctrl + p will complete (with menu of possible completions if that's how you like it) the current long identifier that you are typing.
if (SomeCall(LONG_ID_ <-- type c-x c-p here
            [LONG_ID_I_CANT_POSSIBLY_REMEMBER]
             LONG_ID_BUT_I_NEW_IT_WASNT_THIS_ONE
             LONG_ID_GOSH_FORGOT_THIS
             LONG_ID_ETC
             ∶

Justin L., Jun 13, 2013 at 16:21

i type <kbd>ctrl</kbd>+<kbd>p</kbd> way too much by accident while trying to hit <kbd>ctrl</kbd>+<kbd>[</kbd> >< – Justin L. Jun 13 '13 at 16:21

community wiki
Fritz G. Mehner
, Apr 22, 2009 at 16:41

Use the right mouse key to toggle insert mode in gVim with the following settings in ~/.gvimrc :
"
"------------------------------------------------------------------
" toggle insert mode <--> 'normal mode with the <RightMouse>-key
"------------------------------------------------------------------
nnoremap  <RightMouse> <Insert>
inoremap  <RightMouse> <ESC>
"

Andreas Grech, Jun 20, 2010 at 17:22

This is stupid. Defeats the productivity gains from not using the mouse. – Andreas Grech Jun 20 '10 at 17:22

Brady Trainor, Jul 5, 2014 at 21:07

Maybe fgm has head gestures mapped to mouse clicks. – Brady Trainor Jul 5 '14 at 21:07

community wiki
AIB
, Apr 27, 2009 at 13:06

Replace all
  :%s/oldtext/newtext/igc

Give a to replace all :)

Nathan Fellman, Jan 12, 2011 at 20:58

or better yet, instead of typing a, just remove the c . c means confirm replacementNathan Fellman Jan 12 '11 at 20:58

community wiki
2 revs
,Sep 13, 2009 at 18:39

Neither of the following is really diehard, but I find it extremely useful.

Trivial bindings, but I just can't live without. It enables hjkl-style movement in insert mode (using the ctrl key). In normal mode: ctrl-k/j scrolls half a screen up/down and ctrl-l/h goes to the next/previous buffer. The µ and ù mappings are especially for an AZERTY-keyboard and go to the next/previous make error.

imap <c-j> <Down>
imap <c-k> <Up>
imap <c-h> <Left>
imap <c-l> <Right>
nmap <c-j> <c-d>
nmap <c-k> <c-u>
nmap <c-h> <c-left>
nmap <c-l> <c-right>

nmap ù :cp<RETURN>
nmap µ :cn<RETURN>

A small function I wrote to highlight functions, globals, macro's, structs and typedefs. (Might be slow on very large files). Each type gets different highlighting (see ":help group-name" to get an idea of your current colortheme's settings) Usage: save the file with ww (default "\ww"). You need ctags for this.

nmap <Leader>ww :call SaveCtagsHighlight()<CR>

"Based on: http://stackoverflow.com/questions/736701/class-function-names-highlighting-in-vim
function SaveCtagsHighlight()
    write

    let extension = expand("%:e")
    if extension!="c" && extension!="cpp" && extension!="h" && extension!="hpp"
        return
    endif

    silent !ctags --fields=+KS *
    redraw!

    let list = taglist('.*')
    for item in list
        let kind = item.kind

        if     kind == 'member'
            let kw = 'Identifier'
        elseif kind == 'function'
            let kw = 'Function'
        elseif kind == 'macro'
            let kw = 'Macro'
        elseif kind == 'struct'
            let kw = 'Structure'
        elseif kind == 'typedef'
            let kw = 'Typedef'
        else
            continue
        endif

        let name = item.name
        if name != 'operator=' && name != 'operator ='
            exec 'syntax keyword '.kw.' '.name
        endif
    endfor
    echo expand("%")." written, tags updated"
endfunction

I have the habit of writing lots of code and functions and I don't like to write prototypes for them. So I made some function to generate a list of prototypes within a C-style sourcefile. It comes in two flavors: one that removes the formal parameter's name and one that preserves it. I just refresh the entire list every time I need to update the prototypes. It avoids having out of sync prototypes and function definitions. Also needs ctags.

"Usage: in normal mode, where you want the prototypes to be pasted:
":call GenerateProptotypes()
function GeneratePrototypes()
    execute "silent !ctags --fields=+KS ".expand("%")
    redraw!
    let list = taglist('.*')
    let line = line(".")
    for item in list
        if item.kind == "function"  &&  item.name != "main"
            let name = item.name
            let retType = item.cmd
            let retType = substitute( retType, '^/\^\s*','','' )
            let retType = substitute( retType, '\s*'.name.'.*', '', '' ) 

            if has_key( item, 'signature' )
                let sig = item.signature
                let sig = substitute( sig, '\s*\w\+\s*,',        ',',   'g')
                let sig = substitute( sig, '\s*\w\+\(\s)\)', '\1', '' )
            else
                let sig = '()'
            endif
            let proto = retType . "\t" . name . sig . ';'
            call append( line, proto )
            let line = line + 1
        endif
    endfor
endfunction


function GeneratePrototypesFullSignature()
    "execute "silent !ctags --fields=+KS ".expand("%")
    let dir = expand("%:p:h");
    execute "silent !ctags --fields=+KSi --extra=+q".dir."/* "
    redraw!
    let list = taglist('.*')
    let line = line(".")
    for item in list
        if item.kind == "function"  &&  item.name != "main"
            let name = item.name
            let retType = item.cmd
            let retType = substitute( retType, '^/\^\s*','','' )
            let retType = substitute( retType, '\s*'.name.'.*', '', '' ) 

            if has_key( item, 'signature' )
                let sig = item.signature
            else
                let sig = '(void)'
            endif
            let proto = retType . "\t" . name . sig . ';'
            call append( line, proto )
            let line = line + 1
        endif
    endfor
endfunction

community wiki
Yada
, Nov 24, 2009 at 20:21

I collected these over the years.
" Pasting in normal mode should append to the right of cursor
nmap <C-V>      a<C-V><ESC>
" Saving
imap <C-S>      <C-o>:up<CR>
nmap <C-S>      :up<CR>
" Insert mode control delete
imap <C-Backspace> <C-W>
imap <C-Delete> <C-O>dw
nmap    <Leader>o       o<ESC>k
nmap    <Leader>O       O<ESC>j
" tired of my typo
nmap :W     :w

community wiki
jonyamo
, May 10, 2010 at 15:01

Create a function to execute the current buffer using it's shebang (assuming one is set) and call it with crtl-x.
map <C-X> :call CallInterpreter()<CR>

au BufEnter *
\ if match (getline(1),  '^\#!') == 0 |
\   execute("let b:interpreter = getline(1)[2:]") |
\ endif

fun! CallInterpreter()
    if exists("b:interpreter")
        exec("! ".b:interpreter." %")
    endif
endfun

community wiki
Marcus Borkenhagen
, Jan 12, 2011 at 15:22

map macros

I rather often find it useful to on-the-fly define some key mapping just like one would define a macro. The twist here is, that the mapping is recursive and is executed until it fails.

Example:

enum ProcStats
{
        ps_pid,
        ps_comm,
        ps_state,
        ps_ppid,
        ps_pgrp,
:map X /ps_<CR>3xixy<Esc>X

Gives:

enum ProcStats
{
        xypid,
        xycomm,
        xystate,
        xyppid,
        xypgrp,

Just an silly example :).

I am completely aware of all the downsides - it just so happens that I found it rather useful in some occasions. Also it can be interesting to watch it at work ;).

00dani, Aug 2, 2013 at 11:25

Macros are also allowed to be recursive and work in pretty much the same fashion when they are, so it's not particularly necessary to use a mapping for this. – 00dani Aug 2 '13 at 11:25

>,

Reuse

Motions to mix with other commands, more here .

tx
fx
Fx

Use your favorite tools in Vim.

:r !python anything you want or awk or Y something

Repeat in visual mode, powerful when combined with tips above.

;

[Oct 21, 2018] What are your suggestions for an ideal Vim configuration for Perl development?

Notable quotes:
"... The .vimrc settings should be heavily commented ..."
"... Look also at perl-support.vim (a Perl IDE for Vim/gVim). Comes with suggestions for customizing Vim (.vimrc), gVim (.gvimrc), ctags, perltidy, and Devel:SmallProf beside many other things. ..."
"... Perl Best Practices has an appendix on Editor Configurations . vim is the first editor listed. ..."
"... Andy Lester and others maintain the official Perl, Perl 6 and Pod support files for Vim on Github: https://github.com/vim-perl/vim-perl ..."
Aug 18, 2016 | stackoverflow.com
There are a lot of threads pertaining to how to configure Vim/GVim for Perl development on PerlMonks.org .

My purpose in posting this question is to try to create, as much as possible, an ideal configuration for Perl development using Vim/GVim. Please post your suggestions for .vimrc settings as well as useful plugins.

I will try to merge the recommendations into a set of .vimrc settings and to a list of recommended plugins, ftplugins and syntax files.

.vimrc settings
"Create a command :Tidy to invoke perltidy"
"By default it operates on the whole file, but you can give it a"
"range or visual range as well if you know what you're doing."
command -range=% -nargs=* Tidy <line1>,<line2>!
    \perltidy -your -preferred -default -options <args>

vmap <tab> >gv    "make tab in v mode indent code"
vmap <s-tab> <gv

nmap <tab> I<tab><esc> "make tab in normal mode indent code"
nmap <s-tab> ^i<bs><esc>

let perl_include_pod   = 1    "include pod.vim syntax file with perl.vim"
let perl_extended_vars = 1    "highlight complex expressions such as @{[$x, $y]}"
let perl_sync_dist     = 250  "use more context for highlighting"

set nocompatible "Use Vim defaults"
set backspace=2  "Allow backspacing over everything in insert mode"

set autoindent   "Always set auto-indenting on"
set expandtab    "Insert spaces instead of tabs in insert mode. Use spaces for indents"
set tabstop=4    "Number of spaces that a <Tab> in the file counts for"
set shiftwidth=4 "Number of spaces to use for each step of (auto)indent"

set showmatch    "When a bracket is inserted, briefly jump to the matching one"
syntax plugins ftplugins CPAN modules Debugging tools

I just found out about VimDebug . I have not yet been able to install it on Windows, but looks promising from the description.

innaM, Oct 15, 2009 at 19:06

The .vimrc settings should be heavily commented. E.g., what does perl_include_pod do? – innaM Oct 15 '09 at 19:06

Sinan άnόr, Oct 15, 2009 at 20:02

@Manni: You are welcome. I have been using the same .vimrc for many years and a recent bunch of vim related questions got me curious. I was too lazy to wade through everything that was posted on PerlMonks (and see what was current etc.), so I figured we could put together something here. – Sinan άnόr Oct 15 '09 at 20:02

innaM, Oct 16, 2009 at 8:22

I think that that's a great idea. Sorry that my own contribution is that lame. – innaM Oct 16 '09 at 8:22

Telemachus, Jul 8, 2010 at 0:40

Rather than closepairs, I would recommend delimitMate or one of the various autoclose plugins. (There are about three named autoclose, I think.) The closepairs plugin can't handle a single apostrophe inside a string (i.e. print "This isn't so hard, is it?" ), but delimitMate and others can. github.com/Raimondi/delimitMate – Telemachus Jul 8 '10 at 0:40

community wiki 2 revs, 2 users 74% ,Dec 21, 2009 at 20:57

From chromatic's blog (slightly adapted to be able to use the same mapping from all modes).
vmap, pt :!perltidy<CR> 
nmap, pt :%! perltidy<CR>

hit, pt in normal mode to clean up the whole file, or in visual mode to clean up the selection. You could also add:

imap, pt <ESC>:%! perltidy<CR>

But using commands from input mode is not recommended.

innaM, Oct 21, 2009 at 9:21

I seem to be missing something here: How can I type, ptv without vim running perltidy on the entire file? – innaM Oct 21 '09 at 9:21

innaM, Oct 21, 2009 at 9:23

Ovid's comment (#3) seems offer a much better solution. – innaM Oct 21 '09 at 9:23

innaM, Oct 21, 2009 at 13:22

Three hours later: turns out that the 'p' in that mapping is a really bad idea. It will bite you when vim's got something to paste. – innaM Oct 21 '09 at 13:22

Ether, Oct 21, 2009 at 15:23

@Manni: select a region first: with the mouse if using gvim, or with visual mode ( v and then use motion commands). – Ether Oct 21 '09 at 15:23

Ether, Oct 21, 2009 at 19:44

@Manni: I just gave it a try: if you type, pt, vim waits for you to type something else (e.g. <cr>) as a signal that the command is ended. Hitting, ptv will immediately format the region. So I would expect that vim recognizes that there is overlap between the mappings, and waits for disambiguation before proceeding. – Ether Oct 21 '09 at 19:44

community wiki hobbs, Oct 16, 2009 at 0:35

" Create a command :Tidy to invoke perltidy.
" By default it operates on the whole file, but you can give it a
" range or visual range as well if you know what you're doing.
command -range=% -nargs=* Tidy <line1>,<line2>!
    \perltidy -your -preferred -default -options <args>

community wiki Fritz G. Mehner, Oct 17, 2009 at 7:44

Look also at perl-support.vim (a Perl IDE for Vim/gVim). Comes with suggestions for customizing Vim (.vimrc), gVim (.gvimrc), ctags, perltidy, and Devel:SmallProf beside many other things.

innaM, Oct 19, 2009 at 12:32

I hate that one. The comments feature alone deserves a thorough 'rm -rf', IMHO. – innaM