Regular Expressions Tips

O'Reilly Network Five Habits for Successful Regular Expressions

All five tips are valuable and can be recommended

The extended whitespace feature of most regex implementations allows programmers to extend their regular expressions over several lines, with comments at the end of each. Why do so few programmers use this feature? Perl 6 regular expressions, for example, will be in extended whitespace mode by default. Until your language makes extended whitespace the default, turn it on yourself.

Related Reading Regular Expression Pocket Reference By Tony Stubblebine Table of Contents Index

The only trick to remember with extended whitespace is that the regex engine ignores whitespace. So if you are hoping to match whitespace, you have to say so explicitly, often with \s.

In Perl, add an x to the end of the regex, so m/foo|bar/ becomes:

m/
  foo
  |
  bar
 /x

In PHP, add an x to the end of the regex, so "/foo|bar/" becomes:

"/
  foo
  |
  bar
 /x"

In Python, pass the mode modifier, re.VERBOSE, to the compile function:

pattern = r'''
 foo
 |
 bar
'''

regex = re.compile(pattern, re.VERBOSE)

The value of whitespace and comments becomes more important when working with more complex regular expressions. Consider the following regular expression to match a U.S. phone number:

\(?\d{3}\)? ?\d{3}[-.]\d{4}

This regex matches phone numbers like "(314)555-4000". Ask yourself if the regex would match "314-555-4000" or "555-4000". The answer is no in both cases. Writing this pattern on one line conceals both flaws and design decisions. The area code is required and the regex fails to account for a separator between the area code and prefix.

Spreading the pattern out over several lines makes the flaws more visible and the necessary modifications easier.

In Perl this would look like:

/  
    \(?     # optional parentheses
      \d{3} # area code required
    \)?     # optional parentheses
    [-\s.]? # separator is either a dash, a space, or a period.
      \d{3} # 3-digit prefix
    [-.]    # another separator
      \d{4} # 4-digit line number
/x

The rewritten regex now has an optional separator after the area code so that it matches "314-555-4000." The area code is still required. However, a new programmer who wants to make the area code optional can quickly see that it is not optional now, and that a small change will fix that.

2. Write Tests

There are three levels of testing, each adding a higher level of reliability to your code. First, you need to think hard about what you want to match and whether you can deal with false matches. Second, you need to test the regex on example data. Third, you need to formalize the tests into a test suite.

Deciding what to match is a trade-off between making false matches and missing valid matches. If your regex is too strict, it will miss valid matches. If it is too loose, it will generate false matches. Once the regex is released into live code, you probably will not notice either way. Consider the phone regex example above; it would match the text "800-555-4000 = -5355". False matches are hard to catch, so it's important to plan ahead and test.

Sticking with the phone number example, if you are validating a phone number on a web form, you may settle for ten digits in any format. However, if you are trying to extract phone numbers from a large amount of text, you might want to be more exact to avoid a unacceptable numbers of false matches.

When thinking about what you want to match, write down example cases. Then write some code that tests your regular expression against the example cases. Any complicated regular expression is best written in a small test program, as the examples below demonstrate:

In Perl:

#!/usr/bin/perl

my @tests = ( "314-555-4000",
              "800-555-4400",
	      "(314)555-4000",
              "314.555.4000",
              "555-4000",
              "aasdklfjklas",
              "1234-123-12345"          
            );

foreach my $test (@tests) {
    if ( $test =~ m/
                   \(?     # optional parentheses
                     \d{3} # area code required
                   \)?     # optional parentheses
                   [-\s.]? # separator is either a dash, a space, or a period.
                     \d{3} # 3-digit prefix
                   [-\s.]  # another separator
                     \d{4} # 4-digit line number
                   /x ) {
        print "Matched on $test\n";
     }
     else {
        print "Failed match on $test\n";
     }
}

In PHP:

<?php
$tests = array( "314-555-4000",
           "800-555-4400",
           "(314)555-4000",
           "314.555.4000",
           "555-4000",
           "aasdklfjklas",
           "1234-123-12345"
          );

$regex = "/
            \(?     # optional parentheses
              \d{3} # area code
            \)?     # optional parentheses
            [-\s.]? # separator is either a dash, a space, or a period.
              \d{3} # 3-digit prefix
            [-\s.]  # another separator
              \d{4} # 4-digit line number
           /x";

foreach ($tests as $test) {
    if (preg_match($regex, $test)) { 
        echo "Matched on $test<br />";
    }
    else {
        echo "Failed match on $test<br />";
     }
}
?>

In Python:

import re

tests = ["314-555-4000",
         "800-555-4400",
         "(314)555-4000",
         "314.555.4000",
         "555-4000",
         "aasdklfjklas",
         "1234-123-12345"        
        ]

pattern = r'''
\(?     # optional parentheses
              \d{3} # area code
            \)?     # optional parentheses
            [-\s.]? # separator is either a dash, a space, or a period.
              \d{3} # 3-digit prefix
            [-\s.]  # another separator
              \d{4} # 4-digit line number
           '''

regex = re.compile( pattern, re.VERBOSE )

for test in tests:
    if regex.match(test):
        print "Matched on", test, "\n"
    else:
        print "Failed match on", test, "\n"

Running the test script exposes yet another problem in the phone number regex: it matched "1234-123-12345". Include tests that you expect to fail as well as those you expect to match.

Ideally, you would incorporate these tests into the test suite for your entire program. Even if you do not have a test suite already, your regular expression tests are a good foundation for a suite, and now is the perfect opportunity to start on one. Even if now is not the right time (really, it is!), you should make a habit to run your regex tests after every modification. A little extra time here could save you many headaches.

3. Group the Alternation Operator

The alternation operator (|) has a low precedence. This means that it often alternates over more than the programmer intended. For example, a regex to extract email addresses out of a mail file might look like:

^CC:|To:(.*)

The above attempt is incorrect, but the bugs often go unnoticed. The intent of the above regex is to find lines starting with "CC:" or "To:" and then capture any email addresses on the rest of the line.

Unfortunately, the regex doesn't actually capture anything from lines starting with "CC:" and may capture random text if "To:" appears in the middle of a line. In plain English, the regular expression matches lines beginning with "CC:" and captures nothing, or matches any line containing the text "To:" and then captures the rest of the line. Usually, it will capture plenty of addresses and nobody will notice the failings.

If that were the real intent, you should add parentheses to say it explicitly, like this:

(^CC:)|(To:(.*))

However, the real intent of the regex is to match lines starting with "CC:" or "To:" and then capture the rest of the line. The following regex does that:

^(CC:|To:)(.*)

This is a common and hard-to-catch bug. If you develop the habit of wrapping your alternations in parentheses (or non-capturing parentheses -- (?:…)) you can avoid this error.

4. Use Lazy Quantifiers

Most people avoid using the lazy quantifiers *?, +?, and ??, even though they are easy to understand and make many regular expressions easier to write.

Lazy quantifiers match as little text as possible while still aiding the success of the overall match. If you write foo(.*?)bar, the quantifier will stop matching the first time it sees "bar", not the last time. This may be important if you are trying to capture "###" in the text "foo###bar+++bar". A regular quantifier would have captured "###bar+++".

Let's say you want to capture all of the phone numbers from an HTML file. You could use the phone number regular expression example we discussed earlier in this article. However, if you know that the file contains all of the phone numbers in the first column of a table, you can write a much simpler regex using lazy quantifiers:

<tr><td>(.+?)<td>

Many beginning regular expression programmers avoid lazy quantifiers with negated character classes. They write the above code as:

<tr><td>([^>]+)</td>

That works in this case, but leads to trouble if the text you are trying to capture contains common characters from your delimiter (in this case, </td>). If you use lazy quantifiers, you will spend less time kludging character classes and produce clearer regular expressions.

Lazy quantifiers are most valuable when you know the structure surrounding the text you want to capture.

5. Use Available Delimiters

Perl and PHP often use the forward slash to mark the start and end of a regular expression. Python uses a variety of quotes to mark the start and end of a string, which may then be used as a regular expression. If you stick with the slash delimiter in Perl and PHP, you will have to escape any slashes in your regex. If you use regular quotes in Python, you will have to escape all of your backslashes. Choosing different delimiters or quotes allows to avoid escaping half of your regex. This makes the regex easier to read and reduces the potential for bugs when you forget to escape something.

Perl and PHP allow you to use any non-alphanumeric or whitespace character as a delimiter. If you switch to a new delimiter, you can avoid having to escape the forward slashes when you are trying to match URLs or HTML tags such as "http://" or "<br />".

For example:

/http:\/\/(\S)*/

could be rewritten as:

#http://(\S)*#

Common delimiters are #, !, |. If you use square brackets, angle brackets, or curly braces, the opening and closing brackets must match. Here are some common uses of delimiters:

#…#	!…!	{…}
s|…|…| (Perl only)	s[…][…] (Perl only)	s<…>/…/ (Perl only)

In Python, regular expressions are treated as strings first. If you use quotes -- the regular string delimiter -- you will have to escape all of your backslashes. However, you can use raw strings, r'', to avoid this. If you use raw triple-quoted strings with the re.VERBOSE option, it allows you to include newlines.

For example:

regex = "(\\w+)(\\d+)"

could be rewritten as:

regex = r'''
           (\w+)
           (\d+)
         '''

perl.com Synopsis 5 [Jun. 26, 2002] Changes in regex engine in Perl 6. Some interesting...

Perl.com Apocalypse 5

 this paper does not only suggest a new syntax, what is more important it explains some of the futures of the old syntax that almost nobody knows ;-)

... ... ...

For prior Apocalypses, I've used the RFCs as a springboard for discussion of my thinking, but this one is special, because none of the RFCs were courageous enough (or foolhardy enough) to look at the big picture and propose radical change where it's needed. But Perl has often been tagged as a language in which it's easy to write programs that are difficult to read, and it's no secret that regular expression syntax that has been the chief culprit. Funny that other languages have been borrowing Perl's regular expressions as fast as they can...

That's primarily because we took several large steps in Perl 5 to enhance regex capabilities. We took one large step forwards with the /x option, which allowed whitespace between regex tokens. But we also took several large steps sideways with the (?...) extension syntax. I call them steps sideways, but they were simultaneously steps forward in terms of functionality and steps backwards in terms of readability. At the time, I rationalized it all in the name of backward compatibility, and perhaps that approach was correct for that time and place. It's not correct now, since the Perl 6 approach is to break everything that needs breaking all at once.

And unfortunately, there's a lot of regex culture that needs breaking.

Regex culture has gone wrong in a variety of ways, but it's not my intent to assign blame--there's plenty of blame to go around, and plenty of things that have gone wrong that are nobody's fault in particular. For example, it's nobody's fault that you can't realistically complement a character set anymore. It's just an accident of the way Unicode defines combining characters. The whole notion of character classes is mutating, and that will have some bearing on the future of regular expression syntax.

Given all this, I need to warn you that this Apocalypse is going to be somewhat radical. We'll be proposing changes to certain "sacred" features of regex culture, and this is guaranteed to result in future shock for some of our more conservative citizens. Do not be alarmed. We will provide ways for you to continue programming in old-fashioned regular expressions if you desire. But I hope that once you've thought about it a little and worked through some examples, you'll like most of the changes we're proposing here.

So although the RFCs did contribute greatly to my thinking for this Apocalypse, I'm going to present my own vision first for where regex culture should go, and then analyze the RFCs with respect to that vision.

First, let me enumerate some of the things that are wrong with current regex culture.

• Too much history
• Too compact and "cute"
• Poor Huffman coding
• Too much reliance on too few metacharacters
• Different things look too similar
• Poor end-weight design
• Too much reliance on modifiers
• Too many special rules and boobytraps
• Backreferences not useful enough
• Too hard to match a literal string
• Two-level interpretation is problematic
• Too little abstraction
• Little support for named captures
• Difficult to use nested patterns
• Little support for grammars
• Inability to define variants
• Poor integration with "real" language
• Missing backtracking controls
• Difficult to define assertions

I'm sure there are other problems, but that'll do for starters. Let's look at each of these in more detail. Larry Wall

Too much history

Most of the other problems stem from trying to deal with a rich history. Now there's nothing wrong with history per se, but those of us who are doomed to repeat it find that many parts of history are suboptimal and contradictory. Perl has always tried to err on the side of incorporating as much history as possible, and sometimes Perl has succeeded in that endeavor.

Cultural continuity has much to be said for it, but what can you do when the culture you're trying to be continuous with is itself discontinuous? As it says in Ecclesiastes, there's a time to build up, and a time to tear down. The first five versions of Perl mostly built up without tearing down, so now we're trying to redress that omission.

Too compact and "cute"

Regular expressions were invented by computational linguists who love to write examples like /aa*b*(cd)*ee/. While these are conducive to reasoning about pattern matching in the abstract, they aren't so good for pattern matching in the concrete. In real life, most atoms are longer than "a" or "b". In real life, tokens are more recognizable if they are separated by whitespace. In the abstract, /a+/ is reducible to /aa*/. In real life, nobody wants to repeat a 15 character token merely to satisfy somebody's idea of theoretical purity. So we have shortcuts like the + quantifier to say "one or more".

Now, you may rightly point out that + is something we already have, and we already introduced /x to allow whitespace, so why is this bullet point here? Well, there's a lot of inertia in culture, and the problem with /x is that it's not the default, so people don't think to turn it on when it would probably do a lot of good. The culture is biased in the wrong direction. Whitespace around tokens should be the norm, not the exception. It should be acceptable to use whitespace to separate tokens that could be confused. It should not be considered acceptable to define new constructs that contain a plethora of punctuation, but we've become accustomed to constructs like (?<=...) and (??{...}) and [\r\n\ck\p{Zl}\p{Zp}], so we don't complain. We're frogs who are getting boiled in a pot full of single-character morphemes, and we don't notice.

Poor Huffman coding

Huffman invented a method of data compaction in which common characters are represented by a small number of bits, and rarer characters are represented by more bits. The principle is more general, however, and language designers would do well to pay attention to the "other" Perl slogan: Easy things should be easy, and hard things should be possible. However, we haven't always taken our own advice. Consider those two regex constructs we just saw:

    (?<=...)
    (??{...})

Which one do you think is likely to be the most common in everyday use? Guess which one is longer...

There are many examples of poor Huffman coding in current regexes. Consider these:

    (...)
    (?:...)

Is it really the case that grouping is rarer than capturing? And by two gobbledygooky character's worth? Likewise there are many constructs that are the same length that shouldn't be:

    (?:...)
    (?#...)

Grouping is much more important than the ability to embed a comment. Yet they're the same length currently.

Too much reliance on too few metacharacters

A lot of our Huffman troubles came about because we were trying to shoehorn new capabilities into an old syntax without breaking anything. The (?...) construct succeeded at that goal, but it was new wine in old wineskins, as they say. More successful was the *? minimal matching hack, but it's still symptomatic of the problem that we only had three characters to choose from that would have worked at that point in the grammar. We've pretty nearly exhausted the available backslash sequences.

The waterbed theory of linguistic complexity says that if you push down one place, it goes up somewhere else. If you arbitrarily limit yourself to too few metacharacters, the complexity comes out somewhere else. So it seems obvious to me that the way out of this mess is to grab a few more metacharacters. And the metacharacters I want to grab are...well, we'll see in a moment.

Different things look too similar

Consider these constructs:

    (??{...})
    (?{...})
    (?#...)
    (?:...)
    (?i:...)
    (?=...)
    (?!...)
    (?<=...)
    (?<!...)
    (?>...)
    (?(...)...|...)

These all look quite similar, but some of them do radically different things. In particular, the (?<...) does not mean the opposite of the (?>...). The underlying visual problem is the overuse of parentheses, as in Lisp. Programs are more readable if different things look different.

Poor end-weight design

In linguistics, the notion of end-weight is the idea that people tend to prefer sentences where the short things come first and the long things come last. That minimizes the amount of stuff you have to remember while you're reading or listening. Perl violates this with regex modifiers. It's okay when you say something short like this:

    s/foo/bar/g

But when you say something like we find in RFC 360:

    while ($text =~ /name:\s*(.*?)\n\s*
                    children:\s*(?:(?@\S+)[, ]*)*\n\s*
                    favorite\ colors:\s*(?:(?@\S+)[, ]*)*\n/sigx) {...}

it's not until you read the /sigx at the end that you know how to read the regex. This actually causes problems for the Perl 5 parser, which has to defer parsing the regular expression till it sees the /x, because that changes how whitespace and comments work.

Too much reliance on modifiers

The /s modifier in the previous example changes the meaning of the . metacharacter. We could, in fact, do away with the /s modifier entirely if we only had two different representations for "any character", one of which matched a newline, and one which didn't. A similar argument applies to the /m modifier. The whole notion of something outside the regex changing the meaning of the regex is just a bit bogus, not because we're afraid of context sensitivity, but because we need to have better control within the regex of what we mean, and in this case the context supplied outside the regex is not precise enough. (Perl 5 has a way to control the inner contexts, but it uses the self-obfuscating (?...) notation.)

Modifiers that control how the regex is used as a whole do make some sense outside the regex. But they still have the end-weight problem.

Too many special rules and boobytraps

Without knowing the context, you cannot know what the pattern // will do. It might match a null string, or it might match the previously successful match.

The local operator behaves differently inside regular expressions than it does outside.

It's too easy to write a null pattern accidentally. For instance, the following will never match anything but the null string:

    /
    | foo
    | bar
    | baz
    /x

Even when it's intentional, it may not look intentional:

    (a|b|c|)

That's hard to read because it's difficult to make the absence of something visible.

It's too easy to confuse the multiple meanings of dot. Or the multiple meanings of ^, and $. And the opposite of \A is frequently not \Z, but \z. Tell me again, when do I say \1, and when do I say $1? Why are they different?

Backreferences not useful enough

Speaking of \1, backreferences have a number of shortcomings. The first is actually getting ahold of the right backreference. Since captures are numbered from the beginning, you have to count, and you can easily count wrong. For many purposes it would be better if you could ask for the last capture, or the one before that. Or perhaps if there were a way to restart the numbering part way through...

Another major problem with backreferences is that you can't easily modify one to search for a variant. Suppose you match an opening parenthesis, bracket, or curly. You'll like to search for everything up to the corresponding closing parenthesis, bracket, or curly, but there's no way to transmogrify the opening version to the closing version, because the backref search is hardwired independently of ordinary variable matching. And that's because Perl doesn't instantiate $1 soon enough. And that's because Perl relies on variable interpolation to get subexpressions into regexes. Which leads us to...

Too hard to match a literal string

Since regexes undergo an interpolation pass before they're compiled, anything you interpolate is forced to be treated as a regular expression. Often that's not what you want, so we have the klunky \Q$string\E mechanism to hide regex metacharacters. And that's because...

Two-level interpretation is problematic

The problem with \Q$string\E arises because of the fundamental mistake of using interpolation to build regexes instead of letting the regex control how it treats the variables it references. Regexes aren't strings, they're programs. Or, rather, they're strings only in the sense that any piece of program is a string. Just as you have to work to eval a string as a program, you should have to work to eval a string as a regular expression. Most people tend to expect a variable in a regular expression to match its contents literally. Perl violates that expectation. And because it violates that expectation, we can't make $1 synonymous with \1. And interpolated parentheses throw off the capture count, so you can't easily use interpolation to call subrules, so we invented (??{$var}) to get around that. But then you can't actually get at the parentheses captured by the subrule. The ramifications go on and on.

Too little abstraction

Historically, regular expressions were considered a very low-level language, a kind of glorified assembly language for the regex engine. When you're only dealing with ASCII, there is little need for abstraction, since the shortest way to say [a-z] is just that. With the advent of the eighth bit, we started getting into a little bit of trouble, and POSIX started thinking about names like [:alpha:] to deal with locale difficulties. But as with the problem of conciseness, the culture was still biased away from naming abstractly anything that could be expressed concretely.

However, it's almost impossible to write a parser without naming things, because you have to be able to name the separate grammar rules so that the various rules can refer to each other.

It's difficult to deal with any subset of Unicode without naming it. These days, if you see [a-z] in a program, it's probably an outright bug. It's much better to use a named character property so that your program will work right in areas that don't just use ASCII.

Even where we do allow names, it tends to be awkward because of the cultural bias against it. To call a subrule by name in Perl 5 you have to say this:

    (??{$rule})

That has 4 or 5 more characters than it ought to. Dearth of abstraction produces bad Huffman coding.

Little support for named captures

Make that "no support" in Perl, unless you include assignment to a list. This is just a part of the bias against naming things. Instead we are forced to number our capturing parens and count. That works okay for the top-level regular expression, when we can do list assignment or assign $1 to $foo. But it breaks down as soon as you start trying to use nested regexes. It also breaks down when the capturing parentheses match more than once. Perl handles this currently by returning only the last match. This is slightly better than useless, but not by much.

Difficult to use nested patterns

For many of the reasons we've mentioned, it's difficult to make regexes refer to each other, and even if you do, it's almost impossible to get the nested information back out of them. And there are entire classes of parsing problems that are not solvable without recursive definitions.

Little support for grammars

Even if it were easier for regexes to refer to other regexes, we'd still have the problem that those other regexes aren't organized in any meaningful way. They might be off in variables that come and go at the whim of the surrounding context.

When we have an organized system of parsing rules, we call it a grammar. One advantage of having a grammar is that you can optimize based on the assumption that the rules maintain their relationship to each other. For instance, if you think of grammar rules as a funny kind of subroutine, you can write an optimizer to inline some of the subrules--but only if you know the subrule is fixed in the grammar.

Without support for grammar classes, there's no decent way to think of deriving one grammar from another. And if you can't derive one grammar from another, you can't easily evolve your language to handle new kinds of problems.

Inability to define variants

If we want to have variant grammars for Perl dialects, then what about regex dialects? Can regexes be extended either at compile time or at run time? Perl 5 has some rudimentary overloading magic for rewriting regex strings, but that's got the same problems as source filters for Perl code; namely that you just get the raw regex source text and have to parse it yourself. Once again the fundamental assumption is that a regex is a funny kind of string, existing only at the behest of the surrounding program.

Do we think of regexes as a real, living language?

Poor integration with rich languages

Let's face it, in the culture of computing, regex languages are mostly considered second-class citizens, or worse. "Real" languages like C and C++ will exploit regexes, but only through a strict policy of apartheid. Regular expressions are our servants or slaves; we tell them what to do, they go and do it, and then they come back to say whether they succeeded or not.

At the other extreme, we have languages like Prolog or Snobol where the pattern matching is built into the very control structure of the language. These languages don't succeed in the long run because thinking about that kind of control structure is rather difficult in actual fact, and one gets tired of doing it constantly. The path to freedom is not to make everyone a slave.

However, I would like to think that there is some happy medium between those two extremes. Coming from a C background, Perl has historically treated regexes as servants. True, Perl has treated them as trusted servants, letting them move about in Perl society better than any other C-like language to date. Nevertheless, if we emancipate regexes to serve as co-equal control structures, and if we can rid ourselves of the regexist attitudes that many of us secretly harbor, we'll have a much more productive society than we currently do. We need to empower regexes with a sense of control (structure). It needs to be just as easy for a regex to call Perl code as it is for Perl code to call a regex.

Missing backtracking controls

Perl 5 started to give regexes more control of their own destiny with the "grab" construct, (?>...), which tells the regex engine that when it fails to match the rest of the pattern, it should not backtrack into the innards of the grab, but skip back to before it. That's a useful notion, but there are problems. First, the notation sucks, but you knew that already. Second, it doesn't go far enough. There's no way to backtrack out of just the current grouping. There's no way to backtrack out of just the current rule. Both of these are crucial for giving first-class status to the control flow of regexes.

Difficult to define assertions

Notionally, a regex is an organization of assertions that either succeed or fail. Some assertions are easily expressed in traditional regex language, while others are more easily expressed in a procedural language like Perl.

The natural (but wrong) solution is to try to reinvent Perl expressions within regex language. So, for instance, I'm rejecting those RFCs that propose special assertion syntax for numerics or booleans. The better solution is to make it easier to embed Perl assertions within regexes.

---

Brave New World

I've just made a ton of negative assertions about the current state of regex culture. Now I'd like you to perform a cool mental trick and turn all those negatives assertions into positive assertions about what I'm going to say, because I'm not intending to give the rationale again, but just present the design as it stands. Damian will discuss an extended example in his Exegesis 5, which will show the big picture of how these various features work together to produce a much more readable whole.

So anyway, here's what's new.

Metacharacter Reform

Some things stay the same: (...) captures text just as it did before, and the quantifiers *, +, and ? are also unchanged. The vertical bar | still separates alternatives. The backslash \ still protects the following character from its ordinary interpretation. The ? suffix character still does minimal matching. (Note that these are by far the most commonly used metacharacters, so many ordinary regexes will look nearly identical in Perl 5 and Perl 6.)

Since /x extended syntax is now the default, # is now always a metacharacter indicating a comment, and whitespace is now always "meta". Whitespace is now the standard way to separate regex tokens that would otherwise be confused as a single token.

Even in character classes, whitespace is not taken literally any more. Backwhack the space if you mean it literally, or use <sp>, or \040, or \x20, or \c[SPACE]. But speaking of character classes...

Perhaps the most radical change is that I've taken [...] away from character classes and made it the non-capturing grouping operator, because grouping is more fundamental than character classes, and explicit character classes are becoming less common than named character classes. (You can still do character classes, just not with bare square brackets.)

I've also stolen {...} from generalized quantifiers and made them into closure delimiters. (Use <n,m> for the generalized quantifier now.)

I've stolen three new metacharacters. The new extensible metasyntax for assertions uses angle brackets, <...>. And the colon : is now used for declaration and backtracking control. (Recall Larry's 2nd Law of Language Redesign: Larry gets the colon.) The colon always introduces a token that controls the meaning of what is around it. The nature of the token depends on what follows the colon. Both the colon syntax and angle syntax are extensible. (Backslash syntax is also extensible.)

This may sound like we're complexifying things, but we're really simplifying. We now have the following regex invariants:

    (...)       # always delimits a capturing group
    [...]       # always delimits a non-capturing group
    {...}       # always delimits a closure
    <...>       # always delimits an assertion
    :...        # always introduces a metasyntactic token

(Note that we're using "assertion" here in the broad sense of anything that either matches or fails, whether or not it has a width.)

The nature of the angle assertion is controlled by the first character inside it. If the first character is alphabetic, it's a grammatical assertion, and the entire first word controls the meaning. The word is first looked up in the current grammar, if any. If not found there, it is checked to see if it is one of the built-in grammar rules such as those defined by the Unicode property classes. If the first character is not alphabetic, there will be special rules in the current grammar or in the Perl grammar for looking up the parse rule. For instance, by default, any assertion that begins with ! is simply negated. Assertions that start with a digit are assumed to be a range assertion (<n,m>) regarding the previous atom. (Taking the last two together, you can say <!n,m> to exclude a range.) Assertions that start with $, @, %, or & are assumed to interpolate an indirect regex rule stored in a variable or returned by a subroutine. An assertion that starts with a parenthesis is a closure being used as an assertion. An assertion that starts with a square bracket or another angle bracket is a character class. An assertion that starts with a quote asserts the match of a literal string. And so on.

Some metacharacters are still used but have a slightly different meaning, in part to get rid of the /s and /m modifiers, and in part because most strings in Perl 6 will come from the filehandle pre-chomped. So anchors ^ and $ now always mean the real beginning and ending of the string. Use ^^ and $$ to match the beginnings and endings of lines within a string. (They're doubled because they're "fancier", because they can match in multiple places, and because they'll be rarer, so Huffman says they should be longer.) The ^^ and $$ also match where ^ and $ would.

The dot . now always matches any character including newline. (Use \N to match a non-newline. Or better, use an autochomping filehandle, if you're processing line-by-line.)

In a sense, the sigils $, @, %, and & are different metacharacters because they don't interpolate, but are now subject to the interpretation of the regex engine. This allows us to change the default behavior of ordinary "interpolation" to a literal match, and also lets us put in lvalue-ish constructs like:

    / $name := (\S+) /
    / @kids := [(\S+) \s+]* /
    / %pets := [(\S+) \: (\S+) \s+]* /

(Notice also the delicate interplay of quantified non-capturing brackets with capturing parens, particularly for gathering multiple values or even multiple key/value pairs.)

Here are some of the metacharacter differences in table form:

    Old                   New
    ---                   ---
    /pat pat #text        /pat pat #text
        pat/x                 pat/              # Look Ma, no /x!
    /patpat(?#text)/       /pat pat <('text')>/ # can always use whitespace

    /pat pat/             / pat\ pat /          # match whitespace literally
                          / pat \s* pat /       # or generically
                          / pat \h* pat /       # or horizontally
                          / pat <' '> pat /     # or as a literal string
                          / pat <sp> pat /      # or by explicit rule
                          /:w pat pat/          # or by implicit rule

    /^pat$/               /^pat\n?$/            # ^ and $ mean string
    /^pat$/m              /^^pat$$/             # no more /m
    /\A...(^pat$)*...\z/m /^...(^^pat$$)*...$/  # no more \A or \z
    /.*\n/                /\N*\n/               # \N is negated \n
                          /.*?\n/               # this still works
    /.*/s                 /.*/                  # . always matches "any"

    \Q$string\E           $string               # interpret literally

    (?{ code })           { code }              # call code, ignore return
                          { code or fail }      # use code as an assertion

    (??{$rule})           <$var>                # call $var as regex
                          <name>                # call rule from current grammar
                          <Other::rule>         # call rule from some Other grammar
                          <*rule>               # bypass local rule to call built-in
                          <@array>              # call array of alternate rules
                          <%hash>               # parse keyword as key to rule
                          <@array[1]>           # call a rule from an array
                          <%hash{"x"}>          # call a rule from a hash
                          <&sub(1,2,3)>         # call a rule returned by a sub
                          <{ code }>            # call return value as anonymous rule
                          <( code )>            # call code as boolean assertion

                          <name(expr)>          # call rule, passing Perl args
                          { .name(expr) }       # same thing.

                          <$var(expr)>          # call rule indirectly by name
                          { .$var(expr) }       # same thing.

                          <name pat>            # call rule, passing regex arg
                          { .name(/pat/) }      # same thing.

                          # maybe...
                          <name: text>          # call rule, passing string
                          { .name(q<text>) }    # same thing.

    [\040\t\p{Zs}]        \h                    # horizontal whitespace
    [\r\n\ck\p{Zl}\p{Zp}] \v                    # vertical whitespace
    [a-z]                 <[a-z]>               # equivalently non-international
                          <alpha>               # more international
    [[:alpha:][:digit:]   <<alpha><digit>>      # POSIX classes are built-in rules

    {n,m}                 <n,m>                 # assert repeat count
    {$n,$m}               <$n,$m>               # indirect repeat counts

    (?>.*)                [.*]:                 # don't backtrack through [.*]
                          .*:                   # brackets not necessary on atom
                          (.*):                 # same, but capture
                          <xyz>:                # don't backtrack into subrule

                          :                     # skip previous atom when backtracking
                          ::                    # fail all |'s when backtracking
                          :::                   # fail current rule when backtracking

                          :=                    # bind a name to following atom
    my ($x) = /(.*)/      my $x; / $x:=(.*) /   # may now bind it inside regex

    (?i)                  :i                    # ignore case in the following
                          :ignorecase           # same thing, self-documenting form
    (?i:...)              [:i ...]              # can limit scope without capture
                          (:i ...)              # can limit scope with capture

Declarations like :i are lexically scoped and do not pass to any subrules. Each rule maintains its own sensitivity. There is no built-in operator to turn case ignorance back off--just call a different rule and it's automatically case sensitive again. (If you want a parameterized subrule, that can be arranged. It's just a method, after all. Proof of this assertion is left to future generations of hackers.)

Backslash Reform

There are some changes to backslash sequences. Character properties \p and \P are no longer needed--predefined character classes are just considered intrinsic grammar rules. (You can negate any <...> assertion by using <!...> instead.) As mentioned in a previous Apocalypse, the \L, \U, and \Q sequences no longer use \E to terminate--they now require bracketing characters of some sort. And \Q will rarely be needed due to regex policy changes. In fact, they may all go away since it's easy to say things like:

    $(lc $foo)

For any bracketing construct, square brackets are preferred, but others are allowed:

    \x[...]     # preferred, indicates simple bracketing
    \x(...)     # okay, but doesn't capture.
    \x{...}     # okay, but isn't a closure.
    \x<...>     # okay, but isn't an assertion

The \c sequence is now a bracketing construct, having been extended from representing control characters to any named character.

Backreferences such as \1 are gone in favor of the corresponding variable $1. \A, \Z, and \z are gone with the disappearance of /s and /m. The position assertion \G is gone in favor of a :c modifier that forces continuation from where the last match left off. That's because \G was almost never used except at the front of a regex. In the unlikely event that you want to assert that you're at the old final position elsewhere in your regex, you can always test the current position (via the .pos method) with an assertion:

    $oldpos = pos $string;
    $string =~ m/... <( .pos == $oldpos )> .../;

You may be thinking of .pos as the final position of the previous match, but that's not what it is. It's the current position of the current match. It's just that, between matches, the current position of the current match happens to be the same as the final position of the current match, which happens to be the last match, which happens to be done. But as soon as you start another match, the last match is no longer the current match.

Note that the :c continuation is needed only on constructs that ordinarily force the search to start from the beginning. Subrules automatically continue at the current location, since their initial position is controlled by some other rule.

There are two new backslash sequences, \h and \v, which match horizontal and vertical whitespace respectively, including Unicode spacing characters and control codes. Note that \r is considered vertical even though it theoretically moves the carriage sideways. Finally, \n matches a logical newline, which is not necessarily a linefeed character on all architectures. After all, that's why it's an "n", not an "l". Your program should not break just because you happened to run it on a file from a partition mounted from a Windows machine. (Within an interpolated string, \n still produces whatever is the normal newline for the current architecture.)

    Old                 New
    ---                 ---
    \x0a                \x0a                    # same
    \x{263a}            \x263a                  # brackets required only if ambiguous
    \x{263a}abc         \x[263a]abc             # brackets required only if ambiguous
    \0123               \0123                   # same (no ambiguity with $123 now)
    \0123               \0[123]                 # can use brackets here too

    \p{prop}            <prop>                  # properties are just grammar rules
    \P{prop}            <!prop>

    [\040\t\p{Zs}]      \h                      # horizontal whitespace
    space               \h                      # not exact, but often more correct
    [\r\n\ck\p{Zl}\p{Zp}] \v                    # vertical whitespace

    \Qstring\E          \q[string]
                        <'string with spaces'>  # match literal string
                        <' '>                   # match literal space

    \E                  gone                    # use \Q[...] instead

    \A                  ^                       # ^ now invariant
    \a                  \c[BEL]                 # alarm (bell)

    \Z                  \n?$                    # clearer
    \z                  $                       # $ now invariant
    \G                  <( .pos == $oldpos )>   # match at particular position
                                                # typically just use m:c/pat/

    \N{CENT SIGN}       \c[CENT SIGN]           # named character
    \c[                 \e                      # escape
    \cX                 \c[^X]                  # control char
    \n                  \c[LF]                  # specifically a linefeed
    \x0a\x0d            \x[0a;0d]               # CRLF
    \x0a\x0d            \c[CR;LF]               # CRLF (conjectural)
    \C                  [:u0 .]                 # forced byte in utf8 (dangerous)
    [^\N[CENT SIGN]]    \C[CENT SIGN]           # match any char but CENT SIGN

    \Q$var\E            $var                    # always assumed literal,
    \1                  $1                      # so $1 is literal backref
    /$1/                my $old1 = $1; /$old1/  # must use temporary here

    \r?\n               \n                      # \n asserts logical newline

    [^\n]               \N                      # not a logical newline
                        \C[LF]                  # not a linefeed

    [^\t]               \T                      # not a tab (these are conjectural)
    [^\r]               \R                      # not a return
    [^\f]               \F                      # not a form feed
    [^\e]               \E                      # not an escape
    [^\x1B]             \X1B                    # not specified hex char
    [^\x{263a}]         \X[263a]                # not a Unicode SMILEY

    \X                  <.>                     # a grapheme (combining char seq)
                        [:u2 .]                 # At level 2+, dot means grapheme

Under level 2 Unicode support, a character is assumed to mean a grapheme, that is, a sequence consisting of a base character followed by 0 or more combining characters. That not only affects the meaning of the . character, but also any negated character, since a negated character is really a negative lookahead assertion followed by the traversal of a single character. For instance, \N really means:

    [<!before \n> . ]

So it doesn't really matter how many characters \n actually matches. \N always matches a single character--whatever that is...

Modifier Reform

You can't use colon for a regex delimiter any more. That's because regex modifiers may now be placed in front of a regex construct:

    s:w:i:e /foo/bar/           # :words :ignorecase :each

That can also be written:

    s/:w:i:e foo/bar/           # :words :ignorecase :each

Single character modifiers may be bundled like this:

    s:wie /foo/bar/             # :words :ignorecase :each

...but only if the sequence as a whole is not already defined as a long modifier, since ambiguity will be resolved in favor of the long modifier. Long modifiers may not be bundled with any other modifier. So this is legal:

    s:once:wie /foo/bar/

but not these (unless you've defined them):

    s:wieonce /foo/bar/
    s:oncewie /foo/bar/

Not only is colon disallowed as a regex delimiter, but you may no longer use parentheses as the delimiters either. This will allow us to parameterize modifiers:

    s:myoption($x) /foo/bar/

This rule also allows us to differentiate s/// from an s() function, tr/// from tr(), etc. If you want matching brackets for the delimiters I'd suggest that you use square brackets, since they now mean grouping without capturing.

Several modifiers, /x, /s, and /m, are no longer needed and have been retired. It's unclear whether /o is necessary any more. We will assume it's gone unless it's shown that caching can't handle the problem. Note that the regex now has more control over when to cache subrules because it is no longer subject to the vagaries of standard interpolation.

The old /c modifier is gone because regexes never reset the position on failure any more. To do that, set $string.pos = 0 explicitly. But note also that assigning to a string automatically resets its position to 0, so any string in your typical loop is going to start with its current search position already set 0. Modifying a string in place causes the position to move to the end of the replacement section by default, if the position was within the span replaced. (This is consistent with s/// semantics.)

The /e modifier is also gone, since it did reverse parsing magic, and :e will be short for :each--see below. It's still easy to substitute the value of an expression though:

    s/pat/$( code )/;

or even

    s(/pat/, { code });

There's a new modifier, :once, that causes a match to succeed only once (like the old ?...? construct). To reset it, use the .reset method on the regex object. (If you haven't named the regex object, too bad...)

Another new modifier is :w, which causes an implicit match of whitespace wherever there's literal whitespace in a pattern. In other words, it replaces every sequence of actual whitespace in the pattern with a \s+ (between two identifiers) or a \s* (between anything else). So

    m:w/ foo bar \: ( baz )*/

really means (expressed in Perl 5 form):

    m:p5/\s*foo\s+bar\s*:(\s*baz\s*)*/

You can still control the handling of whitespace under :w, since we extend the rule to say that any explicit whitespace-matching token can't match whitespace implicitly on either side. So:

    m:w/ foo\ bar \h* \: (baz)*/

really means (expressed in Perl 5 form):

    m:p5/\s*foo bar[\040\t\p{Zs}]*:\s*(baz)*/

The first space in

    /[:w foo bar]/

matches \s* before "foo". That's usually what you want, but if it's not what you want, you have a little problem. Unfortunately you can't just say:

    /[:wfoo bar]/

That won't work because it'll look for the :wfoo modifier. However, there are several ways to get the effect you want:

    /[:w()foo bar]/ 
    /[:w[]foo bar]/ 
    /[:w\bfoo bar]/ 
    /[:w::foo bar]/

That last one is just our friend the :: operator in disguise. If you backtrack into it, you're leaving the brackets anyway, so it's essentially a no-op.

The new :c/:cont modifier forces the regex to continue at the current "pos" of the string. It may only be used outside the regex. (Well, it could be used inside but it'd be redundant.) The modifier also forces the regex to match only the next available thing. That's not quite the same as the ^ anchor, though, because it not only disables the implicit scanning done by m// and s///, but it also works on more than the first iteration. It forces all matches to be contiguous, in other words. So :c is short for both "continue" and "contiguous". If you say

    $_ = "foofoofoo foofoofoo";
    s:each:cont/foo/FOO/;

you get:

    FOOFOOFOO foofoofoo

This may seem odd, but it's precisely the semantics of any embedded regex:

    $_ = "foofoofoo foofoofoo";
    $rx = rx/foo/;
    m/<$rx>*/;          # matches "foofoofoo"

A modifier that starts with a number causes the pattern to match that many times. It may only be used outside the regex. It may not be bundled, because ordinals are distinguished from cardinals. That is, how it treats those multiple matches depends on the next character. If you say

    s:3x /foo/bar/

then it changes the first 3 instances. But if you say

    s:3rd /foo/bar/

it changes only the 3rd instance. You can say

    s:1st /foo/bar/

but that's just the default, and should not be construed as equivalent to :once, which matches only once, ever. (Unless you .reset it, of course.)

You can combine cardinals and ordinals:

    s:3x:3rd /foo/bar/

That changes the 3rd, 6th, and 9th occurrences. To change every other quote character, say

    s:each:2nd /"/\&rquot;/;

:each is synonymous with :3x (for large values of 3). Note that :each does not, in fact, generate every possible match, because it disallows overlaps. To get every possible match, use the :any modifier. Saying:

    $_ = "abracadabra";
    @all = m:any /a[^a]+a/;

produces:

    abra aca ada abra

It can even match multiple times at the same spot as long as the rest of the regex progresses somehow. Saying:

    @all = m:any /a.*?a/;

produces:

    abra abraca abracada abracadabra aca acada acadabra ada adabra abra

If you say

    $sentence.m:any /^ <english> $/

you'll get every possible parsing of the sentence according to the rules of english (not to be confused with the rules of English, which are already confusing enough, except when they aren't).

To indicate varying levels of Unicode support we have these modifiers, which may be used either inside or outside a regex:

    :u0         # use bytes       (. is byte)
    :u1         # level 1 support (. is codepoint)
    :u2         # level 1 support (. is grapheme)
    :u3         # level 1 support (. is language dependent)

These modifiers say nothing about the state of the data, but in general internal Perl data will already be in Normalization Form C, so even under :u1, the precomposed characters will usually do the right thing. Note that these modifiers are for overriding the default support level, which was probably set by pragma at the top of the file.

Finally, there's the :p5 modifier, which causes the rest of the regex (or group) to be parsed as a Perl 5 regular expression, including any interpolated strings. (But it still doesn't enable Perl 5's trailing modifiers.)

    Old                 New
    ---                 ---
    ?pat?               m:once/pat/             # match once only
    /pat/i              m:i/pat/                # ignorecase
                        /:i pat/                # ignorecase
    /pat/x              /pat/                   # always extended
    /pat\s*pat/         /:w pat pat/            # match word sequence
    /(?i)$p5pat/        m:p5/(?i)$p5pat/        # use Perl 5 syntax
    $n = () = /.../g    $n = +/.../;            # count occurrences
    for $i (1..3){s///} s:3///;                 # do 3 times
    /^pat$/m            /^^pat$$/               # no more /m
    /./s                /./                     # no more /s
    /./                 /\N/                    # . now works like /s

Keyword and Context Reform

Deferred regex rules are now defined with rx// rather than qr//, because a regular expression is no longer a kind of quoted string.

Actually, just as you can define closures without an explicit sub, any // or rx// declares a deferred regex if it's not in a context that executes it immediately. A regex is executed automatically if it's in a boolean, numeric, or string context. But assignment to an untyped variable is not such a context, nor is passing the regex as an untyped parameter to a function. (Of course, an explicitly declared RULE parameter doesn't provide an evaluating context either.)

So these are equivalent:

    my $foo = /.../;            # create regex object
    my $foo = rx[...];          # create regex object
    my $foo = rule {...};       # create regex object

Likewise, these are equivalent:

    @x = split /.../;
    @x = split rx[...];
    @x = split rule {...};

The "rule" syntax is just a way of declaring a deferred regex as if it were a subroutine or method. More on that later.

To force an immediate evaluation of a deferred regex where it wouldn't ordinarily be, you can use the appropriate unary operator:

    my $foo = ?/.../;   # boolean context, return whether matched, 
    my $foo = +/.../;   # numeric context, return count of matches
    my $foo = _/.../;   # string context, return captured/matched string

The standard match and substitution forms also force immediate evaluation regardless of context:

    $result = m/.../;           # do match on topic string
    $result = s/.../.../;       # do substitution on topic string

These forms also force the regex to start matching at the beginning of the string in question and scan forward through the string for the match, as if there were an implicit .*? in front of every iteration. (Both of these behaviors are suppressed if you use the :c/:cont modifier). In contrast, the meaning of the deferred forms is dependent on context. In particular, a deferred regex naturally assumes :c when used as a subrule. That is, it continues where the last match left off, and the next thing has to match right there at the head of the string.

In any other context, including list context, a deferred regex is not immediately evaluated, but produces a reference to the regex object:

    my $rx = /.../;     # not evaluated
    my @foo = $rx;      # ERROR: type mismatch.
    my @foo = ($rx);    # One element, a regex object.
    my @foo = (/.../);  # Same thing.
    my @foo := $rx;     # Set autogrow rule for @foo.

To evaluate repeatedly in list context, treat the regex object as you would any other iterator:

    my @foo = <$rx>;

You can also use the more explicit form:

    my @foo = m/<$rx>/;

Those aren't identical, since the former assumes :c and starts up at the current position of the unmentioned topic, while the latter explicitly resets the position to the beginning before scanning. Also, since the deferred regex assumes a :c modifier, <$rx> won't scan through the string like m//. It can return multiple values to the list, but they have to be contiguous. You can get the scanning effect of m// by prepending the pattern with .*?.

But it's vitally important to understand this fundamental change, that // is no longer a short form of m//, but rather a short form of rx//. If you want to add modifiers to a //, you have to turn it into an rx//, not an m//. It's now wrong to call split like this:

    split m/.../

(That is, it's wrong unless you actually want the return value of the pattern match to be used as the literal split delimiter.)

The old ?...? syntax is gone. Indeed, it has to go for us to get the unary ? operator.

    Old                 New
    ---                 ---
    ?pat?               m:once/pat/
    qr//                rx//
                        rule { }

Null String Reform

The null pattern is now illegal. To match whatever you used to match with a null pattern, use one of these:

    Old                 New
    ---                 ---
    //                  /<prior>/       # match what prior match did
    //                  /<null>/        # match the null string between chars
    (a|b|)              (a|b|<null>)    # match a null alternative

Note that, as an assertion, <null> always succeeds. You never want to say:

    / <null> | single | double | triple | home run /

because you'll never get to first base.

Extension Syntax Reform

There are no longer any (?...) sequences, because parens now always capture. Some of the replacement sequences take their intrinsic scoping from <...>, while others are associated with other bracketing characters, or with any arbitrary atom that could be a bracketed construct. Looking at the metasyntax problem from the perspective of a Perl5-to-Perl6 translator, here's what the various Perl 5 extension constructs translate to:

    Old                 New
    ---                 ---
    (??{$rule})         <$rule>         # call regex in variable
    (?{ code })         { code }        # call Perl code, ignore result
    (?#...)             <('...')>       # in-line comment, rarely needed
    (?:...)             [...]           # non-capturing brackets
    (?=...)             <before ...>    # positive lookahead
    (?!...)             <!before ...>   # negative lookahead
    (?<=...)            <after ...>     # positive lookbehind
    (?<!...)            <!after ...>    # negative lookbehind
    (?>...)             [...]:          # grab (any atom)

    (?(cond)yes|no)     [ cond :: yes | no ]
    (?(1)yes|no)        [ <(defined $1)> :: yes | no ]

The <$rule> construct does a "delayed" call of another regular expression stored in the $rule variable. If it is a regex object, it's just called as if it were a subroutine, so there's no performance problem. If it's a string, it is compiled as a regex and executed. The compiled form is cached as a property of the string, so it doesn't have to be recompiled unless the string changes. (This implies that we can have properties that invalidate themselves when their base object is modified.) In either case, the evaluated regex is treated as a subrule, and any captures it does are invisible to the outer regex unless the outer regex takes steps to retrieve them. In any event, subrule parens never change the paren count of the outer rule.

The {code} form doesn't return anything meaningful--it is used for its side effects. Any such closure may behave as an assertion. It merely has to throw an exception in order to fail. To throw such an exception (on purpose) one may use fail:

    $_ = "666";
    / (\d+) { $1 < 582 or fail }/

As with any assertion, the failing closure starts backtracking at the location of the closure. In this case, it backtracks into the \d+ and ends up matching "66" rather than "666". If you didn't want that, use \d+: instead.

It's more succinct, however, to use the code assertion syntax. Just put angles around a parenthesized Perl expression:

    / (\d+) <( $1 < 582 )> /

I find the parens to be vaguely reminiscent of the parentheses you have to put around conditionals in C (but not Perl (anymore)). Also, the parentheses are meant to remind you that you only want to put an expression there, not a full statement.

Don't use a bare closure to try to interpolate a calculated regex, since the result will be ignored. Instead, use the <{expr}> form to do that. As with <&rule()>, the result will be interpreted as a subrule, not as if it were interpolated.

Since a string is usually true, you can just assert it to get the effect of an inline comment: <("this is a comment")>. But I've never used one except to show it as an example. Line ending comments are usually much clearer. (Just bear in mind you can't put the final regex delimiter on the same line, because it won't be seen in the comment.) You could also use the {'...'} construct for comments, but then you risk warnings about "useless use of a string in void context".

The [...] is the new non-capturing bracket notation. It seems to work very well for this purpose--I tried the other brackets and they tend to "disappear" faster than square brackets. So we reserve (...) and <...> for constructs where the visual distance between opening and closing is typically shorter than for square brackets or curlies. The square brackets also work nicely when lined up vertically with vertical bars. Here's a declaration of a named rule from the class Perl6Grammar. It parses Perl 6 statements. (Think of it as a funny looking method declaration.)

    rule state  { <label>
                    [ <control>          {.control}
                    | <sideff> <eostate> {.sideff}
                    | <@other_statements>
                    ]
                };

Huffman coding says that rarer forms should be longer, and that's the case with the lookahead and lookbehind assertions, <before ...> and <after ...>. (The negations are formed via the general <!...> rule.) Note that these prepositions are interpreted as assertions, not operations. For example, <before X> is to be read "Assert that we are before X" rather than "Look before where we are for X".

The new : operator replaces the (?>...) construct. It modifies whatever comes before it, much like * does, so it's naturally scoped if the preceding atom (or quantified atom) is a bracketed construct. Parsers can use this every time they commit to the parsing of a token or phrase to tell the regex engine that there's no point in backtracking through the atom in question, so backtracking will skip backwards over the atom and continue with some earlier branch point. The following takes a long time to fail if it has to look at every sequence of "a" to see if there is a "b" after it:

    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaac" =~ /^ a* b /

But we already know that the only possible match is the longest one. So if you put in the colon, it fails in one pass because the * grabs everything and gives nothing back on backtracking.

    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaac" =~ /^ a*: b /

You can use colon on a longer sequence too. The following might match a list of expressions separated by comma:

    / <expr> [ , <expr> ]*: /

It is an error to use : on any atom that does no backtracking. This will help to catch errors where you've forgotten to backslash a literal colon in things like:

    /^From: (.*)/

Perl 6 has no need for a special conditional construct like Perl 5's (?(cond)yes|no). That's because with a slight tweak, ordinary alternation can do the same thing. That tweak is our next backtracking modifier, the :: operator. If you backtrack across it, it fails all the way out of the current list of alternatives. Consider an ordinary list of alternatives:

    [ <A> <X> | <B> <Y> | <C> <Z> ]

The way the rules of backtracking work, if either <A> or <X> fail, it backtracks to the next alternative. Likewise for <B> and <Y>. In the case of <C> or <Z>, there is no next alternative, so it naturally fails out of the entire construct. That's not how a conditional is supposed to work, because in the conditional, only the condition determines which case is executed. Once you've committed to a particular case, it has to stand or fall as if the conditional hadn't been there. So all we need for our purposes is to have is something that separates the assertions that matter from those that don't. That's what :: does, and it reads rather well as a "then", or as a "corresponds to". If you write

    [ <A> :: <X>
    | <B> :: <Y>
    | <C> :: <Z>
    ]

then the failure of <A>, <B>, or <C> proceeds to the next case (if any), while any failure in <X>, <Y>, or <Z> is guaranteed to backtrack out of the front of the alternative list and revise a former choice (just as the success of <X>, <Y>, or <Z> is guaranteed to "forward track" out of the end of the alternative list and try to match more). It's a natural mapping to existing regex semantics. Here's a more realistic example from the Perl 6 grammar. It parses statement modifiers. (The <ws> rule parses optional whitespace.)

    rule modifier { if     <ws> :: <expr> { .new_cond(0,$expr) }
                  | unless <ws> :: <expr> { .new_cond(1,$expr) }
                  | while  <ws> :: <expr> { .new_loop(0,$expr) }
                  | until  <ws> :: <expr> { .new_loop(1,$expr) }
                  | for    <ws> :: <expr> { .new_for($expr)   }
                  | <@other_modifiers>  # user defined
                  | <null>              # no modifier
                  }

In each case, once we recognize a keyword (and its following whitespace), we need to look for an expression, and then call a closure that builds the syntax tree. If either of those fails, the entire modifier rule fails. We only get to the last two alternatives on failure of assertions before the ::.

Note that the :: only says that we can't backtrack from the "then" into the "if". It says nothing about backtracking into the alternative list as a whole. The alternatives are still choice points, so the regex engine is allowed to backtrack into the alternative list and try another alternative. (To disable that, simply put a : after the closing bracket of the alternative list.)

There is nothing in Perl 5 corresponding to the ::: operator, but it works just like ::, only more so. If you backtrack across it, it fails all the way out of the current rule definition (though not out of any rule invoking this definition). That is, it fails all the way out of the innermost lexically enclosing /.../, m/.../, s/...//, rx/.../, or rule {...}, skipping out through any enclosing nestings of <...>, [...], or (...). (A pattern nested within a closure is classified as its own rule, however, so it never gets the chance to pass out of a {...} closure.)

Since the alternatives in our last example are at the top level of the regex, we could have used the ::: operator to get the same effect as ::, because terminating the rule and terminating the alternation amount to the same thing in that case. You can think of all of these as variants on Prolog's "cut" operator.

If you backtrack over the :::: operator, it will delete your program from the disk. ;-)

Actually, the real name of the real :::: operator is <commit>. It fails the entire match if you backtrack over it, not just the current rule. That is, it fails all the way out of the outermost dynamically enclosing /.../, m/.../, s/...//, rx/.../, or rule {...} that is executing on the current string.

There is one "cut" operator that is beyond <commit>; it is appropriately named <cut>, for two reasons. First of all, it's a real cut operator in that, if you backtrack over it, the current match fails completely, just like <commit>. But that's just a side effect of the other reason, which is that <cut> cuts off the front of the string that you're currently matching on, turning the current position into the new beginning of the string. When you're matching on a potentially infinite string, it's important that you have a way of discarding that part of the match that you've already committed to. In Perl 5, the only way to do that was with a coordinated system of s/^pat// operations. With the <cut> assertion, however, you can just match normally, and cut at one spot in your top-level rule when you reach an "accept" state.

In the realm of idle speculation, we could go as far as to define a variant of <cut> that would render s/// slightly redundant:

    s/foo/bar/;
    m/foo <replace("bar")> /

Note that we don't need any special forms for controlling the scope of a "fail" in a closure. Just prefix the closure with the appropriate backtracking operator:

    / pattern ::: { code() or fail } /  # fails entire rule

Character Class Reform

As we mentioned earlier, character classes are becoming more like standard grammar rules, because the definition of "character" is getting fuzzier. This is part of the motivation for demoting enumerated character classes and stealing the square brackets for another purpose. Actually, for old times' sake you still use square brackets on enumerated character classes, but you have to put an extra set of angles around it. But this actually tends to save keystrokes when you want to use any named character classes or Unicode properties, particularly when you want to combine them:

    Old                 New
    ---                 ---
    [a-z]               <[a-z]>
    [[:alpha:]]         <alpha>
    [^[:alpha:]]        <-alpha>
    [[:alpha:][:digit]] <<alpha><digit>>

The outer <...> also naturally serves as a container for any extra syntax we decide to come up with for character set manipulation:

    <[_]+<alpha>+<digit>-<Swedish>>

State

[This section gets pretty abstruse. It's okay if your eyes glaze over.]

Every regex match maintains a state object, and any closure within the regex is actually an anonymous method of that object, which means in turn that the closure's topic is the current state object. Since a unary dot introduces a method call on the current topic, it follows that you can call any method in the state object that way:

    /(.*) { print .pos }/       # print current position

The state object may in fact be an instance of a grammar class. A grammar object has additional methods that know how to build a parse tree. Its rules also know how to refer to each other or to rules of related grammars.

Note that $_ within the closure refers to this state object, not the original search string. If you search on the state object, however, it pretends that you wanted to continue the search on the original string. If the internal search succeeds, the position of the external state is updated as well, just as if the internal search had been a rule invoked directly from the outer regex.

Because the state object is aware of how the tree is being built, when backtracking occurs the object can destroy parts of the parse tree that were conjectured in error. Because the grammar's action methods have control of the regex state, they can access named fields in the regex without having to explicitly pass them to the method call.

For instance, in our earlier example we passed $expr explicitly to build the parse tree, but the method can actually figure that out itself. So we could have just written:

    rule modifier { if     <ws> :: <expr> { .new_cond(0) }
                  | unless <ws> :: <expr> { .new_cond(1) }
                  | while  <ws> :: <expr> { .new_loop(0) }
                  | until  <ws> :: <expr> { .new_loop(1) }
                  | for    <ws> :: <expr> { .new_for }
                  | <@other_modifiers>  # user defined
                  | <null>              # no modifier
                  },

See Variable Scoping below for where @other_modifiers gets looked up.

Within a closure, $_ represents the current state of the current regex, and by extension, the current state of all the regexes participating in the current match. (The type of the state object is the current grammar class, which may be an anonymous type if the current grammar has no name. If the regex is not a member of a grammar, it's of type RULE.) Part of the state of the current regex is the current node of the parse tree that is being built. When the current regex succeeds, the state object becomes a result object, and is returned to the calling regex. The calling regex can refer to the returned object as a "hypothetical" variable, the name of which is either implicitly generated from the name of the rule, or explicitly bound using :=. Through that variable you can get at anything captured by the subrule. (That is what $expr was doing earlier.)

When the entire match succeeds, the top-level node is returned as a result object that has various values in various contexts, whether boolean, numeric, or string context. The name of the result object is $0. The result object contains all the other information, such as $1, $2, etc. Unlike $& in Perl 5, $0 is lexically scoped to the enclosing block. By extension, $1, etc. are also lexically scoped.

As a kind of iterator, a regex stored in a variable doesn't expand in list context unless you put angles around it or use it with m//:

    $rx = /(xxx)/;
    print 1,2,<$rx($_)>;

    print 1,2,</(xxx)/>;

    my &rx := /(xxx)/;
    print 1,2,<rx($_)>;

$0, $1, etc. are not set in iterated cases like this. Each list item is a result object, though, and you can still get at the internal values that way.

Hypothetical Variables, er Values

Values that are determined within a regular expression should usually be viewed as speculative, subject to cancellation if backtracking occurs. This applies not only to the values captured by (...) within the regex, but also to values determined within closures embedded in the regex. The scope of these values is rather strange, compared to ordinary variables. They are dynamically scoped, but not like temp variables. A temporary variable is restored at the end of the current block. A hypothetical variable keeps its value after the current block exits, and in fact keeps that value clear to the end of its natural lifetime if the regex succeeds (where the natural lifetime depends on where it's declared). But if failure causes backtracking over where the variable was set, then it is restored to its previous state. Perl 5 actually coerced the local operator into supporting this behavior, but that was a mistake. In Perl 6 temp will keep consistent semantics, and restore values on exit from the current block. A new word, let, will indicate the desire to set a variable to a hypothetical value. (I was tempted to use "suppose", but "let" is shorter, and tends to mean the same thing, at least to mathematicians.)

    my $x;
    / (\S*) { let $x = .pos } \s* foo /

After this pattern, $x will be set to the ending position of $1--but only if the pattern succeeds. If it fails, $x is restored to undef when the closure is backtracked. It's possible to do things in a closure that the regex engine doesn't know how to backtrack, of course, but a hypothetical value doesn't fall into that category. For things that do fall into that category, perhaps we need to define a BACK block that is like UNDO, but scoped to backtracking.

Sometimes we'll talk about declaring a hypothetical variable, but as with temp variables, we're not actually declaring the variable itself, but the dynamic scope of its new value. In Perl 6, you can in fact say:

    my $x = 0;
    ...
    {
        temp $x = 1;    # temporizes the lexical variable
        ...
    }
    # $x restored to 0

(This is primarily useful for dynamically scoping a file-scoped lexical, which is slightly safer than temporizing a package variable since nobody can see it outside the file.)

You may declare a hypothetical variable only when the topic is a regex state. This is not as much of a hardship as it might seem. Suppose your closure calls out to some other routine, and passes the regex state as an argument, $rx_state. It suffices to say:

    given $rx_state { let $x = .pos }

As it happens, $1 and friends are all simply hypothetical variables. When we say "hypothetical variable", we aren't speaking of where the variable is stored, but rather how its contents are treated dynamically. If a regex sets a hypothetical variable that was declared with either my or our beforehand, then the regex modifies that lexical or package variable, and let is purely a run-time operation.

On the other hand, if the variable is not pre-declared, it's actually stored in the regex state object. In this case, the let also serves to declare the variable as lexically scoped to the rest of the regex, in addition to its run-time action. Such a variable is not directly visible outside the regex, but you can get at it through the $0 object (always presuming the match succeeded). For a regex variable named $maybe, its external name is $0._var_{'maybe'}. The $0 object can behave as a hash, so $0{maybe} is the short way to say that.

All other variable names are stored with their sigil, so the external name for @maybe is $0{'@maybe'}, and for %maybe is $0{'%maybe'}.

$1 is a special case--it's visible outside the regex, not because it's predeclared, but because Perl already knows that the numbered variable $1 is really stored as a subarray of the $0 object: $0[1]. The numbered variables are available only through the array, not the hash.

Since $0 represents the state of the currently executing regex, you can't use it within a rule to get at the result of a completed subrule. When you successfully call a subrule named <somerule>, the regex state is automatically placed in a hypothetical variable named $somerule. (Rules accessed indirectly must be captured explicitly, or they won't have a name by which you can get to them. More on that in the next section.)

As the current recursive regex executes, it automatically builds a tree of hashes corresponding to all captured hypothetical variables. So from outside the regex, you could get at the $1 of the subrule <somerule> by saying $0{somerule}[1].

Named Captures

Suppose you want to use a hypothetical variable to bind a name to a capture:

    / (\S+) { let $x := $1 } /

A shorthand for that is:

    / $x:=(\S+) /

The parens are numbered independently of any name, so $x is an alias for $1.

You may also use arrays to capture appropriately quantified patterns:

    / @x := (\S+ \s*)* /                # including space
    / @x := [ (\S+) \s* ]* /            # excluding space
    / @x := [ (\S+) (\s*) ]* /          # each element is [word, space]

Note that in general, naming square brackets doesn't cause the square brackets to capture, but rather provides a destination for the parens within the square brackets. Only parens and rules can capture. It's illegal to name square brackets that don't capture something inside.

You can also capture to a hash:

    / %x := [ (\S+)\: \s* (.*) ]* /     # key/value pairs

After that match, $1 returns a list of keys, and $2 returns a list of values. You can capture just the keys:

    / %x := [ (\S+) \s* ]* /            # just enter keys, values are undef

You can capture a closure's return value too:

    / $x := { "I'm in scalar context" } /
    / @x := { "I", "am", "in", "list", "context" } /
    / %x := { "I" => "am in hash context" } /

Note that these do not use parens. If you say:

    / $x := ({ code }) /

it would capture whatever text was traversed by the closure, but ignore the closure's actual return value.

You can reorder paren groups by naming them with numeric variables:

    / $2:=(.*?), \h* $1:=(.*) /

If you use a numeric variable, the numeric variables will start renumbering from that point, so subsequent captures can be of a known number (which clobbers any previous association with that number). So for instance you can reset the numbers for each alternative:

    / $1 := (.*?) (\:)  (.*) { process $1, $2, $3 }
    | $1 := (.*?) (=\>) (.*) { process $1, $2, $3 }
    | $1 := (.*?) (-\>) (.*) { process $1, $2, $3 }
    /

It's also possible to refer to captures relative to the current location. $-1 refers to the immediately preceding capture (what used to be known as $+). $-2 refers to the one before that. If you use anything above $-3 we'll come and take you away to the insane asylum.

Subrules called via <rule> also capture their result in hypothetical variables. It's possible to name the results of any <...>, but grammar rules already have a name by default, so you don't have to give them names unless you call the same rule more than once. So, presuming you have grammar rules defining "key" and "value", you can say:

    / <key> \: <value> { let %hash{$key} = $value } /

Of course, in a typical grammar the typical rule may not return a string, but a reference to an anonymous object representing a node of the parse tree. But that depends on what the subrule decides to capture. If the only thing captured in the subrule is a single string, that's what you get. (If nothing is captured, you get the entire match.)

Any capture that will capture multiple items will, if asked to put it into a scalar variable, produce an anonymous list automatically. This should rarely be a surprise, since it's obvious by inspection whether you've quantified the capture or not. So if you say any of:

    / $x := <word>*/
    / $x := <word>+/
    / $x := <word><1,3>/

then you've "pluralized" the naming, and you can expect to get some number of values in $x as an anonymous list. However, the ? quantifier specifically doesn't pluralize. If you say:

    / $x := <word>?/

then $x will either be the result of the subrule or undef.

You can name the results of a zero-width assertion, but you'd typically only get the null string out of it. This can still be useful, since it contrasts with the undefined value you'd have if the assertion fails. (It is possible with an explicit capture to return a non-zero-width string from a zero-width assertion, however.)

Variable Scoping

When you refer to a variable @foo as an rvalue in a regex, it searches for an existing variable in the following places:

1. We first look to see if the variable is already declared lexically with either my @foo or our @foo. If so declared, we're done.
2. Next we look for @foo in the current regex's name table. The name of the variable is really $0{'@foo'}.
3. If the regex belongs to a grammar, we next look for @foo in the grammar object. If there, its real name is @.foo, or some such. (It might be objected that the grammar object is not yet constructed when the regex is compiled. After all, the regex is probably being passed to the grammar object's constructor. But I think if such a variable is declared as an object attribute we know that there will be such a variable/accessor later when we have finished constructing, and that seems like enough info to know how to compile the regex.)
4. Next we look for @foo as a declared core global variable @*foo.
5. Finally, if "strict vars" is not in effect, we assume that @foo is stored in the current package. Otherwise it's a stricture error.

Variable Interpretation

As we mentioned earlier, bare scalars match their contents literally. (Use <$var> instead to match a regex defined in $var.) Subscripted arrays and hashes behave just like a scalar as long as the subscripts aren't slices.

If you use a bare array (unsubscripted), it will match if any element of the array matches literally at that point. (A slice of an array or hash also behaves this way.) If you say

    @array = ("^", "$", ".");
    / @array /

it's as if you said

    / \^ | \$ | \. /

But if you you slice it like this:

    / @array[0..1] /

it won't match the dot.

If you want the array to be considered as a set of regex alternatives, enclose in angles:

    @array = ("^foo$", "^bar$", "^baz$");
    / <@array> /

Bare hashes in a regex provide a sophisticated match-via-lookup mechanism. Bare hashes are matched as follows:

1. Match a key at the current point in the string.

   1a. If the hash has its keymatch property set to some regex, use that regex to match the key.

   1b. Otherwise, use /\w+:/ to match the key.

2. If a key isn't found at the current position in the string, the match fails.
3. Otherwise, get the value in the hash corresponding to the matched key
4. If the is no entry for that key, the match fails.
5. If the hash doesn't have a valuematch property, the match succeeds immediately.
6. Otherwise use the hash's valuematch property (typically itself a regex) to extract the value at the current point in the string.
7. If no value can be extracted, matching of the hash fails.
8. If the extracted value string is eq to the key's actual value, matching of the original hash immediately succeeds.
9. Otherwise, matching of the original hash fails.

So matching a bare hash is equivalent to:

    rule {
        $key := <{ %hash.prop{keymatch} // /\w+:/ }>    # find key
        <( exists %hash{$key} )>                        # if exists
        [ <( not defined %hash.prop{valuematch} )> ::   # done?
            <null>                                      # succeed
        |                                               # else
            $val := <%hash.prop{valuematch}>            # find value
                <( $val eq %hash{$val} )>               # assert eq
        ]
    }

A typical valuematch might look like:

    rule {
        \s* =\> \s*             # match => 
        $q:=(<["']>)            # match initial quote 
        $0:=( [ \\. | . ]*? )   # return matched value
        $q                      # match trailing quote 
    }

In essence, the presence or absence of the valuematch property controls whether the hash tries to match only keys, or both keys and values.

A hash may be used inside angles as well. In that case, it finds the key by the same method (steps 1 and 2 above), but always treats the corresponding hash value as a regex (regardless of any properties the hash might have). The parse then continues according to the rule found in the hash. For example, we could parse a set of control structures with:

    rule { <%controls> }

The %controls hash can have keys like "if" and "while" in it. The corresponding entry says how to parse the rest of an if or a while statement. For example:

        %controls = ( 
            if     => / <condition>      <closure> /,
            unless => / <condition>      <closure> /,
            while  => / <condition>      <closure> /,
            until  => / <condition>      <closure> /,
            for    => / <list_expr>      <closure> /,
            loop   => / <loop_controls>? <closure> /,
        );

So saying:

    <%controls>

is really much as if we'd said:

    [ if     \b <%controls{if}>
    | unless \b <%controls{unless}> 
    | while  \b <%controls{while}>
    | until  \b <%controls{until}> 
    | for    \b <%controls{for}> 
    | loop   \b <%controls{loop}>
    ]

Only it actually works more like

    / $k=<{ %controls.prop{keymatch} // /\w+:/ }> <%controls{$k}> /

Note that in Perl 6 it's perfectly valid to use // inside an expression embedded in a regex delimited by slashes. That's because a regex is no longer considered a string, so we don't have to find the end of it before we parse it. Since we can parse it in one pass, the expression parser can handle the // when it gets to it without worrying about the outer slash, and the final slash is recognized as the terminator by the regex parser without having to worry about anything the expression parser saw.

A bare subroutine call may be used in a regex, provided it starts with & and uses parentheses around the arguments. The return value of the subroutine is matched literally. The subroutine may have side effects, and may throw an exception to fail.

Defining Your Own Rules

Suppose your name is Hugo and you don't like to use ! to negate an assertion. You can define your own assertion like this:

    my rule not (str $rx) { <!<{"<$rx>"}>> }            # define Hugo not
    / <not [a-z]> /     # same as <![a-z]>

That rule would be lexically scoped because of the my. If you think it looks like a sub declaration, you're right. In fact, it's possible you could even declare it anonymously like a closure:

    my $not = rule (str $rx) { <!<{"<$rx>"}>> };
    / <$not tonight dear> /

But maybe you don't want it lexically scoped because you're writing a grammar for general use:

    grammar HugoGrammar {
         rule not ($rx) { <!$rx> }
         rule identifier { <not before \d> \w+ }
         rule \j { \c[LF] }
         rule parse { ^ <identifier> \j $ }
    }

    HugoGrammar.parse($line);

In this case a rule is simply a method in a grammar class, and a grammar class is any class derived implicitly or explicitly from the universal RULE grammar class. The built-in regex assertions like <before \w> are really just calls to methods in the RULE class. The namespace of a grammar is simply the method namespace of the current class, which is the class's methods plus all inherited methods.

In addition to normal subrules, we allow some funny looking method names like:

    rule :a { ... }
    rule \a { ... }

Modules that mutate Perl's grammar on the fly can do so by deriving an anonymous grammar class from the default Perl6Grammar, and installing extra rules on the fly. The current regex state then continues parsing the rest of the lexical scope using some rule from the new rule set. Subsequent grammatical mutations will be derived from the current anonymous grammar unless you switch explicitly to an entirely different grammar.

Since we're writing grammar rules as if they were methods, we have access to the full syntax of method declaration, including formal parameter lists and compile-time properties. So we can easily annotate rules with pragmatic information such as operator precedence levels when you don't want to write a strictly recursive-descent parser, for instance. (And we don't want to, for Perl.)

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019