Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Vim Regular Expressions

News VIM -- VI-style Editor with folding capabilities Recommended Books Recommended Links Scriptorama .vimrc Ancors
RegEx page Perl Regular Expressions Overview of regular expressions in Perl   Regex Vimscript VIM Syntax Highliting
Line Ranges Examples Tips   Humor History Etc

Regular expressions appear to be rapidly gaining in popularity among VIM users as they discover the sheer programming power that regular expressions can provide. Historically, regular expressions have been associated with the UNIX platform and scripting languages like Perl (Practical Extraction and Report Language).

The syntax in VIM is slightly different then in Perl, but is pretty close.  This makes Perl regular expression examples relevant to VIM users.

Softpanorama RegEx page contain basic information about regular expressions. I would like to stress that Vim's regexp implementation is reasonably close to Perl's and skills are transferable. Among the differences between Perl and Vim  we can note:

Some meta characters are different (in yellow)

# Matching # Matching
. any character except new line    
\s whitespace character \S non-whitespace character
\d digit \D non-digit
\x hex digit \X non-hex digit
\o octal digit \O non-octal digit
\h head of word character (a,b,c...z,A,B,C...Z and _) \H non-head of word character
\p printable character \P like \p, but excluding digits
\w word character \W non-word character
\a alphabetic character \A non-alphabetic character
\l lowercase character \L non-lowercase character
\u uppercase character \U non-uppercase character

Many special characters need to be escaped. For example:

\+  matches 1 or more of the preceding characters...

\{n,m} matches from n to m of the preceding characters...

\= is used instead of \? (matches 0 or 1 more of the preceding characters)

Quantifier Description
* matches 0 or more of the preceding characters, ranges or metacharacters .* matches everything including empty line
\+ matches 1 or more of the preceding characters...
\= matches 0 or 1 more of the preceding characters...
\{n,m} matches from n to m of the preceding characters...
\{n} matches exactly n times of the preceding characters...
\{,m} matches at most m (from 0 to m) of the preceding characters...
\{n,} matches at least n of of the preceding characters...
 

Alternatives (OR) need to be escaped

Using "\|" you can combine several expressions into one which matches any of its components. The first one matched will be used.

\(Date:\|Subject:\|From:\)\(\s.*\)

will parse various mail headings and their contents into \1 and \2, respectively. The thing to remember about VIM alternation that it is not greedy. It won't search for the longest possible match, it will use the first that matched. That means that the order of the items in the alternation is important!

Tip 3:  Quick mapping to put \(\) in your pattern string

cmap ;\ \(\)<Left><Left>

Non-greed modifiers are different and more obscure then in Perl.  Perl allows you to convert any quantifier into a non-greedy version by adding an extra ? after it. So *? is a non-greedy version of a special character *

Quantifier

Description

\{-} matches 0 or more of the preceding atom, as few as possible
\{-n,m} matches 1 or more of the preceding characters...
\{-n,} matches at lease or more of the preceding characters...
\{-,m} matches 1 or more of the preceding characters...
 

Replacement rules are different

You can group parts of the pattern expression enclosing them with "\(" and "\)" and refer to them inside the replacement pattern by their special number \1, \2 ... \9. Typical example is swapping first two words of the line:

s:\(\w\+\)\(\s\+\)\(\w\+\):\3\2\1:

where \1 holds the first word, \2 - any number of spaces or tabs in between and \3 - the second word. How to decide what number holds what pair of \(\) ? - count opening "\(" from the left.

Replacement part of the S&R has its own special characters which we are going to use to fix grammar:

# Meaning # Meaning
& the whole matched pattern \L the following characters are made lowercase
\0 the whole matched pattern \U the following characters are made uppercase
\1 the matched pattern in the first pair of \(\) \E end of \U and \L
\2 the matched pattern in the second pair of \(\) \e end of \U and \L
... ... \r split line in two at this point
\9 the matched pattern in the ninth pair of \(\) \l next character made lowercase
~ the previous substitute string \u next character made uppercase

Now the full S&R to correct non-capital words at the beginning of the sentences looks like

s:\([.!?]\)\s\+\([a-z]\):\1  \u\2:g

We have corrected our grammar and as an extra job we replaced variable number of spaces between punctuation and the first letter of the next sentence with exactly two spaces.

The most common task is to make replacements in a text following some certain rules using VIM search and replace command (S&R) :s(substitute). For example here is how globally replace all occurrences of vi with VIM.

%s/1999/2003/g

This is a very common idiom in vi/vim.  Like in Perl you can also use several modifiers

Using Anchors

Adapted from volontir Vim regular expression tutorial

Suppose you want to replace all occurrences of vi with VIM. This can be easily done with

s/vi/VIM/g

If you've tried this example then you, no doubt, noticed that VIM replaced all occurrences of vi even if it's a part of the word (e.g. navigator). If we want to be more specific and replace only whole words vi then we need to correct our pattern. We may rewrite it by putting spaces around vi:

s: vi : VIM :g

But it will still miss vi followed by the punctuation or at the end of the line/file. The right way is to put special word boundary symbols "\<" and "\>" around vi.

s:\<vi\>:VIM:g

The beginning and the end of the line have their own special anchors - "^" and "$", respectively. So, for all vi only at the start of the line:

s:^vi\>:VIM:

To match the lines where vi is the only word:

s:^vi$:VIM:

Now suppose you want to replace not only all vi but also Vi and VI. There are several ways to do this:

 

Examples

Adapted from volontir Vim regular expression tutorial

Some examples of :g usage:

:g/^$/ d

- delete all empty lines in a file

:g/^$/,/./-j

- reduce multiple blank lines to a single blank

:10,20g/^/ mo 10

- reverse the order of the lines starting from the line 10 up to the line 20.

Here is a modified example from Walter Zintz vi tutorial:

:'a,'b g/^Error/ . w >> errors.txt

- in the text block marked by 'a and 'b find all the lines starting with Error and copy (append) them to "errors.txt" file. Note:  . (current line address) in front of the w is very important, omitting it will cause :write to write the whole file to "errors.txt" for every Error line found.

You can give multiple commands after :global using "|" as a separator. If you want to use "|' in an argument, precede it with "\'. Another example from Zintz tutorial:

:g/^Error:/ copy $ | s /Error/copy of the error/

- will copy all Error line to the end of the file and then make a substitution in the copied line. Without giving the line address :s will operate on the current line, which is the newly copied line.

:g/^Error:/ s /Error/copy of the error/ | copy $

- here the order is reversed: first modify the string then copy to the end.

 


Top updates

Bulletin Latest Past week Past month
Google Search


Old News ;-)

volontir Vim regular expression tutorial

Some examples of :g usage:

:g/^$/ d

- delete all empty lines in a file

:g/^$/,/./-j

- reduce multiple blank lines to a single blank

:10,20g/^/ mo 10

- reverse the order of the lines starting from the line 10 up to the line 20.

Here is a modified example from Walter Zintz vi tutorial:

:'a,'b g/^Error/ . w >> errors.txt

- in the text block marked by 'a and 'b find all the lines starting with Error and copy (append) them to "errors.txt" file. Note:  . (current line address) in front of the w is very important, omitting it will cause :write to write the whole file to "errors.txt" for every Error line found.

You can give multiple commands after :global using "|" as a separator. If you want to use "|' in an argument, precede it with "\'. Another example from Zintz tutorial:

:g/^Error:/ copy $ | s /Error/copy of the error/

- will copy all Error line to the end of the file and then make a substitution in the copied line. Without giving the line address :s will operate on the current line, which is the newly copied line.

:g/^Error:/ s /Error/copy of the error/ | copy $

- here the order is reversed: first modify the string then copy to the end.

Vim Regular Expressions

I started this tutorial for one simple reason - I like regular expressions. Nothing compares to the satisfaction from a well-crafted regexp which does exactly what you wanted it to do :-).

And yes, I have a life too. I hope it's passable as a foreword. Feel free to send me your comments, corrections and suggestions concerning this tutorial.

A Tao of Regular Expressions

A regular expression is a formula for matching strings that follow some pattern. Many people are afraid to use them because they can look confusing and complicated. Unfortunately, nothing in this write up can change that. However, I have found that with a bit of practice, it's pretty easy to write these complicated expressions. Plus, once you get the hang of them, you can reduce hours of laborious and error-prone text editing down to minutes or seconds. Regular expressions are supported by many text editors, class libraries such as Rogue Wave's Tools.h++, scripting tools such as awk, grep, sed, and increasingly in interactive development environments such as Microsoft's Visual C++.

Regular expressions usage is explained by examples in the sections that follow. Most examples are presented as vi  substitution commands or as grep file search commands, but they are representative examples and the concepts can be applied in the use of tools such as sed, awk, perl and other programs that support regular expressions. Have a look at Regular Expressions In Various Tools for examples of regular expression usage in other tools. A short explanation of vi's substitution command and syntax is provided at the end of this document.

Recommended Links

Books

***** volontir Vim regular expression tutorial by Oleg Raisky. A very nice tutorial with some useful examples

VimRegEx.vim - Regular Expression Developer for Vim vim online

created by Dave Silvia
script type
utility
description
Start with one of the commands

:Vimrex
  or
:VimRegEx

The first opens within the current (g)vim session, the second starts a new (g)vim

'z?' or the menu 'Usage' button opens a usage buffer.

VimRegEx.vim is a plugin that aids in development/construction/analysis of Vim regular expressions.  It is not a search utility, although it certainly does that.  It is a development/learning environment that allows writing Vim regular expressions, element-by-element analysis of these expressions, and execution of the expressions against source text with the results illustrated both verbally in text and graphically in different highlighting for different types of regular expression atoms.

It allows you to see, in verbal/graphical analytical terms, just what is being searched for and what will result with a given Vim regular expression.

There are comparable tools on the net for other environments (notably .NET, Perl, and Python), but nothing exists for Vim ... until now!

The plugin is written specifically for Vim, so you can build your regular expressions as you would in Vim.  In other tools, you'd have to make allowances for backslash escaping specific elements (like '(', ')', '+', '?', etc.) to use what you had developed in those tools.  You'd also have to make substitutions, like '\<' for '\b' for word boundary ('\b' has its own meaning in Vim), and '\(...\)\@=' for '(?=...)'.  This renders these other tools all but useless for Vim.

VimRegEx has its own builtin documentation, a rich set of variables for tuning its performance/appearance to your liking in your vimrc, an easy menu driven interface for gvim as well as easy to use keyboard mappings for the same commands in vim.  There's also file generation capability to save your windows in text or html.

If you've been struggling with a regular expression and can't seem to get it to do what you want, maybe what you think you're doing isn't what Vim sees.  See what Vim sees in VimRegEx.  In the VimRegEx enviroment, what you see is what you get!

http://www.vim.org/scripts/script.php?script_id=1056

place both in your .vim|vimfiles plugin dir




Etc

Society

Groupthink : Understanding Micromanagers and Control Freaks : Toxic Managers : BureaucraciesHarvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Two Party System as Polyarchy : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

Skeptical Finance : John Kenneth Galbraith : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Oscar Wilde : Talleyrand : Somerset Maugham : War and Peace : Marcus Aurelius : Eric Hoffer : Kurt Vonnegut : Otto Von Bismarck : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Oscar Wilde : Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 26, No.1 (January, 2013) Object-Oriented Cult : Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks: The efficient markets hypothesis : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

 

The Last but not Least


Copyright © 1996-2014 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine. This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting hosting of this site with different providers to distribute and speed up access. Currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: August 05, 2013