Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Introduction to Perl for Unix System Administrators

(Perl without excessive complexity)

by Dr Nikolai Bezroukov

Prev | Up | Contents | Down | Next

5.7. HTML Matching Examples

Below are several different examples of real-life pattern matchings, some very simple, some more complicated. They show the power and flexibility of regular expressions.

Example 1:

This is a simple example. Here, we simply open up the file given to us by the first argument to the script, i.e.: by typing 'perl5 script.p filename'. Then in line 4, the pattern 'm"\b(\w+)\b"g' iterates over the file, getting all of the words out of it and sticking it into the array words.

 Example 2: Words fitting a given criteria in a file

Now let's expand on this a little. Suppose we want to get all the words in a file that start with the letter 't':

Or equivalently:

In each case, the regular expression is in array context, with a 'g' modifier, and therefore it passes back a list of words to the context in which it was called. In the first case, this was the function grep, in the second case, a foreach loop.

After, this array is passed back, the m"^t" clause then pushes onto the array stack any and all words that start with 't'. This, for example could be used to match words in a list:

 Example 3: Parsing Times (10:00AM)

Suppose now that you have a file of the form:

in which the description is followed by the time at which the deed occurred and you want to turn this into a hash that tells you what you did at what time. There are three steps here.

With the use of regular expressions, step #2 becomes fairly straightforward. Our regular expression will look something like this:

Here, matching the comment is easy. We are given that every line is a comment/time pair, and consists of a comment first, and a time second. Hence, matching the comment becomes a simple matter of putting a '.*?' right after the beginning.

Furthermore, we use the 'm' modifier, so that '^' means 'match any character after either the beginning or a newline', and such that '$' means 'match any character that is a newline, or the end of the string'.

Now we need only to find what the regular expression for 'time' is. We could roughly think of it as this:

The first \d{1,2}: matches '10:', the second \d{2} matches 00-59, and [AP]M matches AM or PM. This expression also so happens to match 99:99PM, but the chances of such a string occurring are fairly slight, so we deem the risk acceptable (In more 'bulletproof' cases, we would have to consider this. See 'Mastering Regular Expressions' for more detail on how to only match 0-11, or 0-23, or 0-59, i.e. 'time numbers'.)

Anyway, we can now iterate through our file with this expression and make the hash. The code is below:

The loop in 6 through 10 does most of the work, making the hash by taking the results of the regular expression (line 8) and hashifying it (line 9). You could then do whatever you would like with the data.

Example 4: Substituting Bold Text HTML tags for Italic

You are probably aware of these. They are things like

or

In general, we cannot match all of these, with one regular expression, since they can be recursive, like:

Although we could write a recursive subroutine to do so.

No, in a simple subroutine, the best we can do is pick a list of tags to match, and then use that information to match, assuming that the tags are not recursive. This is usually a safe assumption.

Now, the general form of a tag is something like this:

or

In other words, the first tag consists of a '<', then a string, followed by either nothing, or a space and other text describing the tag, plus a '>'. The tag is closed by a '</STRING>' where STRING equals the same string before.

To match:

we could say:

Assuming, of course, that this tag isn't recursive. How do we generalize this so it matches any tag? Well, lets say we wanted to match bold or italic ('B' or 'I'). And furthermore, we don't know that these strings have text after them (assume we don't know if <B description> is possible). Well, then we could use the following pattern.

Which says to match first, either B or I, and then match (zero or 1 times) the combination of a space plus the minimum amount of characters. Why this complexity? Consider, if we say: m "<(B|I).*?>", then:

will match. Hence, it is essential that we put the facts that:

In pictures, our expression m"<(B|I)(\s.*?)? works like Figure 9.9:

If we now take (B|I) and substitute it with $pattern, where $pattern = 'B|I', we get:

To match the first tag <B>. And to match the whole enchilada:

Here, \1 is whatever pattern we had found in '$patterns'. Hence, this matches the whole tag (in this case, the 'B' tag.)

For our code example, Lets be a little bit simple, and only consider two possible strings to match: <B> and <I>. Furthermore, in this case, lets substitute bold text for italic, and vice-versa.

However, lets write our code in such a way that it is generalizable. The code follows now:

Its lines 5-9 which are the killers. Lets look at them closely. The first thing that we notice is that we have parenthesized everything. (<), ($patterns), etc. This way, we insure that everything that we save will be accounted for when we substitute back in.

Secondly, notice the 'sgx' modifiers on the end. We need to use the 's' modifier in the case that we have something like:

We need to use the 'x' or else the expression will become unreadable, and finally we need to use the 'g' so that this matches more than one time.

Finally, notice the '$1....$7' substitute variable. This is us, plunking in the data which we have matched. In particular, the $substituteHash{$2} takes whatever tag we find, and converts from, say 'B' to 'I'.

This example shows a lot about the crafting of regular expressions. The next, and final example goes even further.

 http, ftp tags

You know the ones I'm talking about:

Suppose that you've got a file of text containing them (for simplicity sake, lets suppose that they aren't split between lines) and you want to abstract them out. Well, the first thing we need to do is come up with a regular expression to match these. Lets look what we need to match:

This translates into something that looks like this:

This did take a little trial and error. We use (?:) so we don't get any backreferences, and the (?:ftp|http) is self explanatory, but the (?:\w+\.){2,5}\w* needs a little bit of explaining, as does the (/\S*){0,1}.

(?:\w+\.){2,5}\w*. Figure: Each one of the \w+\. is meant to match 'members.', or 'aol.'. However, this isn't enough to match the whole internet address. Why? Because the internet address has a trailing group of letters. (?:\w+\.){2,5} does not match 'members.aol.com' it matches 'members.aol.'. We need to add a '\w*' to match the last bit, the 'com' part.

(/\S*)? In English, this is saying "match a slash followed by as any non-spaces as you can find, and do it zero or one times." It refers to the fact that you can have a trailing path on an http or ftp address, but it is not a necessity to have this. This results in the optional question mark.

Note that this pattern is not perfect. If you have backslashed spaces in the http tag in particular, this won't work. Or, if you have a http daemon running on a different port (http://site:8080 for example) it won't work. However, it is as we say 'close enough': if you want to improve it to handle such cases, go right ahead.

Let's now make a loop which extracts all http, and ftp tags from a given file, for example a bookmarks file:

Here, we chop off the last character, since in a bookmarks file these tags are in double quoted strings. We shall approach this problem more directly next, when we consider matching a double quoted string.

Prev | Up | Contents | Down | Next