Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Piping Vim Buffer Through Unix Filters
( ! and !! Commands)

News See Also Recommended Links Reference vimrc History Humor Etc

Everybody knows how pipes work at the command prompt. Text originates from some source, is processed via one or more filters and output goes either to the console display or is redirected to a file.

VI takes this same paradigm of pipes and filters and wraps it in a editor user interface in which the pipe is applied to editing buffer both as a source and as a destination. A VI pipe is thus can alter the buffer using standard Unix filters that instantly become a part of editor toolbox. This is an extremely elegant idea. The ability to pass nearly arbitrary chunks of text through any UNIX filter adds incredible flexibility  at no "additional cost" in size or performance of the editor. 

Pipes can be used both from command line commands:

With the vi filter command

:[address-range] ! external-command-name

you can process a range of lines with an external program. This enables you to do much more effective editing. For example, the command:

:1,$!indent

will beautify your program using  standard Unix beautifier (indent).  This is a classic example of using piping in vi. You can also create a macro using keystroke remapping. Instead of indent you can use any available batch beautifier most suitable for the language that you are using.

Without any filter, the command !, prompts for the name of a UNIX command (which should be a filter), then passes selected lines through the filter, replacing those selected line in the vi buffer with the output of the filter command.

Actually the ex  % operator is the easiest way to filter all the lines in your buffer, and this classic vi idiom should look like:

:%!indent

In vi mode, text is filtered through a UNIX command by typing an exclamation mark (!) followed by any of vi's movement keystrokes that indicate a block of text, and then by the UNIX command line to be executed. To edit the current line you need to use:

!!command

To edit the next paragraph: 

!}!command

If you use any keystrokes that move cursor you need to use them so that they move cursor more than one line ( G, { }, ( ), [[ ]], +, - ). To repeat the effect, a number may precede either the exclamation mark or the text object. (For example, both !10+ and 10!+ would indicate the next ten lines.) Objects such as w do not work unless enough of them are specified so as to exceed a single line. You can also use a slash (/) followed by a pattern  and a carriage return to specify the object. This takes the text up to the pattern as input to the command.

The entire sequence can be preceded by a number to repetitions:

20!!grep -v '^#'

or:

!20!grep -v '^#'

To move one paragraph down you can use !}. That's a very convenient, often  used idiom. 

Among Unix filters that can be used I would like to mention AWK and  Perl. Of course, any filter can be used, for example  tr can be used for case conversion of deletion of  some characters (tr '[a-z]' '[A-Z]').

To repeat the previous pipe command, you need to type:

! object !

Shell can be used as a filter, which gives you an ability to replace shell command that typed in the current line (lines) with their output. Here is a relevant quote from Vim tutorial (Vim Color Editor HOW-TO (Vi Improved with syntax color highlighting) Vi Tutorial)

Create a line in your file containing just the word who and absolutely no other text. Put the cursor on this line, and press !! This command is analogous to dd, cc, or yy, but instead of deleting, changing, or yanking the current line, it filters the current line. When you press the second !, the cursor drops down to the lower left corner of the screen and a single ! is displayed, prompting you to enter the name of a filter.

As the filter name, type sh and press the Return key. sh (the Bourne shell) is a filter! It reads standard input, does some processing of its input (that is, executes commands), and sends its output (the output of those commands) to standard output. Filtering the line containing who through sh causes the line containing who to be replaced with a list of the current users on the system - right in your file!

Try repeating this process with date. That is, create a line containing nothing but the word date, then put the cursor on the line, and press !!sh and the Return key. The line containing date is replaced with the output of the date command.

Put your cursor on the first line of the output of who. Count the number of lines. Suppose, for example, the number is six. Then select those six lines to be filtered through sort; press 6!!sort and the Return key. The six lines will be passed through sort, and sort's output replaces the original six lines.

The filter command can only be used on complete lines, not on characters or words.

Some other filter commands (here, < CR > means press Return):

You can use internal editor piping for a lot of interesting stuff. For example you can read list of files in the current directory into the buffer, convert then into some commands and then execute them in shell:

$ vim
:r! ls *.c
:%s/\(.*\).c/mv & \1.bla :w !sh :q!

or as one liner:

:r! ls *.c :%s/\(.*\).c/mv & \1.bla :w 
!sh :q!

You can format text without the fmt program using instead perl's Text::Wrap module (especially useful if you're a prisoner of Bill again):

:% ! perl -00 -MText::Wrap -ne 'BEGIN{$Text::Wrap::columns=40} 
print wrap("\t","",$_)'

Numbering items with pattern matches in vi

:! type foo.html | perl -pe"BEGIN{$i=1;} ++$i if s:<foo>:<bar$i>:;" > bar.html

Use filtering with a tool like perl to get variable interpolation into search patterns, unless you are lucky enough to have compiled-in support for perl or other tools that allow you to do this.

I want to introduce another quite useful and powerful Vim command which we're going to use later

:range g[lobal][!]/pattern/cmd
Execute the Ex command cmd (default ":p") on the lines within [range] where pattern matches. If pattern is preceded with a ! - only where match does not occur.

The global commands work by first scanning through the [range] of of the lines and marking each line where a match occurs. In a second scan the [cmd] is executed for each marked line with its line number prepended. If a line is changed or deleted its mark disappears. The default for the [range] is the whole file.

Note: Non-Ex commands (normal mode commands) can be also executed from the command line using :norm[al]non-ex command  mechanism.


Top updates

Bulletin Latest Past week Past month
Google Search


Old News ;-)

[Aug 7, 2001] Tip #95 - How do I pipe the output from ex commands into the text buffer?

This is a *request* for a tip.  I need to be able to pipe the output of a :blah ex command into the vim text buffer for editing.  I wanted to do this many times for different reasons and could never find a way! I would just love to be able to do :hi --> textBuffer and examine the output at my own leasure scrolling up and down and using vim search commands on it.  Same thing for :set all, and other things.  Considering that cut and paste is horrible in windows, I can't for example do :set guioptions? then cut and paste!  So I have to retype it, or cut and paste from the help manual.  I really want to be able to pipe the output of ex commands into the text buffer.  Can someone help me?

Questions & Answers about using tags with Vim

Yegappan, August 7, 2001 11:45

You can use the :redir command to redirect the output of an ex command to
a register and then paste the contents of the register into a Vim buffer.
For example:
:redir @a
:set all
:redir END
Now, register 'a' will have the output of the "set all" ex command.  You
can paste this into a Vim buffer.  You can also write a Vim function
to do the above. For more information, read
:help redir Anonymous, August 7, 2001 14:13  Wow!!!   That's awesome!!  Exactly what I want!
 

mark@zieg.com, July 25, 2002 11:28

This may be obvious to experts, but it took me a very long time to figure it out, because Google searches on terms like 'pipe', 'buffer', 'shell', etc never brought it to my attention. 

However, you can pipe the contents of the file currently being edited (the current buffer) to a shell command, and replace the current file/buffer with the _output_ of that command, using this: :%! [cmd] ie, if you didn't know the :retab command (as for a long time I didn't), you could expand tabs using basic unix commands like ":%! expand -t 4".  Wish I'd known this a long time ago, so I'm posting it here in the hopes that others might find it :-)

aniou@root.pl, February 18, 2004 14:01

The answer is (for ex.): :read !ls ~ and :help :read for more info :-)

Grateful, September 27, 2004 12:10

Thanks Anonymous and Yegappan, I've long wanted to do this too, but never known how.  Great initiative Anonymous!

[Chapter 30] vi Tips and Tricks -- from the old edition of Unix Power Tools

30.37 Neatening Lines

Have you made edits that left some of your lines too short or long? The fmt  utility can clean that up. Here's an example. Let's say you're editing a file (email message, whatever) in vi  and the lines aren't even. They look like this:

This file is a mess
with some short lines
and some lines that are too long - like this one, which goes on and on for quite a while and etc.
Let's see what 'fmt' does with it.

You put your cursor on the first line and type (in command mode):

5!!fmt

which means " filter 5 lines through fmt." Then the lines will look like this:

This file is a mess with some short lines and some lines that are too
long - like this one, which goes on and on for quite a while and etc.
Let's see what 'fmt' does with it.

This is handiest for formatting paragraphs. Put your cursor on the first line of the paragraph and type (in command mode):

!}fmt

If you don't have any text in your file that needs to be kept as is, you can neaten the whole file at once by typing:

:%!fmt

There are a few different versions of fmt, some fancier than others. Most of the articles in the chapter about editing-related tools can be handy too. For example,

To neaten columns, try filtering through with the setup in article 35.22. In general, if the utility will read its standard input and write converted text to its standard output, you can use the utility as a vi  filter.

- JP

Etc

Firefly Software Home -- Authors of TEXTools

TEXTools (updated May 20, 2005) is a powerful pipe-based text editing workbench for Windows that allows you to morph text into other forms quickly. Whereas traditional computer languages are designed to do everything from low-level memory access to instantiating objects, this generalization comes at a cost: they function at too low a level to effectively address many of the day-to-day text editing problems that pop up in the real world. Quick! Joe down the hall in accounting finally exported the data but he included unwanted columns that must be stripped out, the delimiters are wrong and blah, blah... To make matters worse, your boss needs the data in two hours! What are you going to do--fire up your compiler and begin writing text handling functions? Probably not. What's needed in these kinds of situations is a high-level tool that specializes in handling text.

That's the whole idea behind TEXTools. Because it functions at a higher level than general-purpose computer languages do, TEXTools enables you to solve many text-based problems without you having to address low-level issues like declaring variables or instantiating objects... All you do is simply combine filters together into little mini-programs called pipes. Each filter in the pipe applies some basic text processing function to the incoming text and then passes that processed text on to the next filter in the pipe for additional processing. A typical TEXTools solution consists of perhaps only a dozen such filters, (of course, your boss doesn't have to know that).

TEXTools' powerful core of text translation filters enables you to accomplish all sorts of challenging editing tasks. For example, you can search and replace text in web pages, convert between CSV, comma-delimited, tab-delimited and fixed-width files, convert between PC, Mac and Unix text files, edit mailing lists, extract email addresses, maintain lists of keywords, extract logfile data, manipulate XML data, format source code, convert exported text for use by other software, automate interactive processes... Individual TEXTools filters can do things like change text case, sort lines, remove duplicates, output unique lines, truncate lines, pad lines to a given width, trim spaces from lines, add or remove lines or columns, reorder columns, join or split lines, perform base conversions, total columns of numbers, perform math operations on numeric data, count lines, add line numbers, etc. You can even extend TEXTools by creating your own user-defined filters (UDFs) from 32-bit console executables, VBScripts, JScripts and by combining other TEXTools filters! TEXTools can be utilized from batch files, (using TEXTools Command Line) scripting languages such as VBScript, JScript, Perl, Python, etc. and also from programs written in languages such as Visual Basic, Delphi, C++, etc.   more...




Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2014 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting hosting of this site with different providers to distribute and speed up access. Currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: August 05, 2013