Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Introduction to Perl for Unix System Administrators

(Perl without excessive complexity)

by Dr Nikolai Bezroukov

Prev | Up | Contents | Down | Next

2.2: Overview of Perl Lexical Structure, Syntax and Operators

1. Overview of Lexical Structure

   • Whitespace

   • Comments

   • Numeric Literals

   • String literals

     • Single quoted literals

     • Double quoted literals

     • Backquoted literals

     • Dynamically delimited literals

     • Here literals

   • Identifiers

   • Keywords

2. Syntax

3. Summary

Overview of literature

Good introduction to Perl lexical strcuture can be found in InformIT Perl's Building Blocks Numbers and Strings Literals which is a reprint of a chapter from Sams Teach Yourself Perl in 24 Hours, 2nd Edition

2.2.1 Overview of Lexical Structure

The lexical structure of a programming language is the set of rules that specify lexical elements (tokens) from which any program can be constructed.  It is the lowest level rules and on the intuitive level they correspond to the first pass of the interpreter through the program.

On this level we specify such entitled (tokens) as identifiers, whitespace, statement delimiters (how one program statement is separated from the next), constants, comments and delimiters. Comments are usually discarded during lexical analysis of the program.

This short chapter deals with the lexical structure of the language.

Perl lexical structure is closer to lexical stricture of Unix shell languages than to traditional high level languages and is pretty complex. Due to shell heritage in several important way Perl lexical structure is different from lexical structure of C-style languages like Java. for example some literals (double quoted) are further preprocessed and actually behave more like built-in functions -- actually they are special built-in functions. Similarity with shells are visible in other areas too (overuse of prefixes for variables). At the same time Perl has the most powerful and flexible facilities for specifying text literals of all languages that I know. 

Perl is a case-sensitive language. This means that language words (called identifiers)  should be typed with a consistent capitalization of letters. there are several types of identifies:

• language keywords (words that have fixed meaning in the language and ofte start statement)

• variables names

• function names

The if keyword, for example, must be typed in lowercase, and not If or IF. Similarly, $i and $I are two distinct variable names (as we will see Perl uses $ as a prefix to some variables)

Whitespace

Whitespace are elements that can serve as identifiers separators, but that otherwise ignored much like comments. whitespace symbols include spaces, tabs, and newlines.

Like any normal language should Perl ignores whitespace that surrounds "lexical tokens" in programs. As we already know "lexical token" is a keyword, variable name, number, function name, etc.  For example those two statements are the same:

$i=1;

$i = 1;

$i =

1;

At the same time you cannot use whitespace inside the token, that splits the token into two. For example, If you place a space or tab or newline within a token, you will break it up into two tokens

$i=5E2; # 5E2 is a single numeric token ( 5 *100 equal to 500)

$i=5 E2; # here 5E2 was split into two tokens by a space (syntax error).

Because you can use whitespace freely in your program it is possible to indent your programs in a neat fashion (to beautify the program). That makes the code easy to read and understand. Special program called beautifiers can be used for this purpose. Programming editors often can beautify code automatically too, for example:

Perl comments

Comments are usually considered as a special case of whitespace and many languages treat internal comment as a space.  Perl does not have internal comments (comments that start and end on the same line and have language construct before and after it). It has simplified "line to the end" comments that start with "#". This type comments is well known in Unix world because they are used in shells. 

Perl does not have multi-line comments either.  There is no way to comment out a block of code in Perl other than to put # at the beginning of each line. For example

# this is a one-line comment

$i = 0; # this is an end line comment

# $i=0; # this is a statement that is commented out (note two # in this line)

Literals

Numeric Literals

Like most scripting languages Perl doesn't specify the type size and ranges of the numeric literal. And to a certain extent there is no any numeric data type in Perl. No byte, short and integer types are defined in the language (integrer was added in latest versions of Perl but we will not discuss it here). In classic Perl all arithmetic operations are performed by default on double precision (64 bit) IEEE 754 floating-point numbers that gives 14-15 significant digits in most calculations. You can change default with pragma use int but I saw very few scripts that are using this feature. 

If you need more control you probably need a different language.

Default base is ten but you can specify numbers in octal notation (start with zero or hexadecimal notation (have prefix 0x). Underscore can be used in big numbers for readability.

12  #  small integer in decimal notation

192_168_10_10 # underscore can be used in big numbers for readability

1123141.111  # all numbers are represented in floating point so  it's ok

3.14e+2 # explicit floating point

3.14E2  # yet another floating point number

Base 16 (hexadecimal) numeric literal are possible .

0x1F    # hexadecimal

0x1f    # same as above

Octal number are also supported (a leading zero should be used):

037     # octal -- not decimal

String literals

Perl have very flexible literal lexics. There are tree types of static string literals:

• Single quoted,

• Double quoted

• Back quoted (this is a special case of double quoted literal)

There are also four additional functions that can generate literals of this tree types. And there is also special multiline literal called HERE documents.

Single quoted literals

The single quote (') indicates that the text is to be use verbatim with minimum interpretation. Single quoted literals cannot span for more than one line. There are only two C-style escape sequences acceptable in single quoted strings literals:

\\  -- backslash

\'  -- single quote

Typical idioms:

print '\''; # you can't just put ' in the single quote literal

print 'C:\\WINDOWS\\COMMAND\\COMMAND.COM'; # backslashes are doubled

print '"'; # single quote-double quote-single quote

print 'this is "new" example'; # double quotes used inside

 

  Interpolation of scalars in double quoted string literals

The double quotes force macro substitution (for some reason called  interpolation in Perl) of any scalar variables -- variables that start with $ (dollar sign)

Here @mydomain will interpreted as an array with very undesirable results.  The correct form should be

`cat letter | mailx -s test myself\@mydomain.my`;

or

to_addr='myself\@mydomain.my;

`cat letter | mailx -s test $to_addr`;

Here is a relevant quote from the perlop man page:

While we usually think of quotes as literal values, in Perl they function as operators, providing various kinds of interpolating and pattern matching capabilities. Perl provides customary quote characters for these behaviors, but also provides a way for you to choose your quote character for any of them. In the following table, a {} represents any pair of delimiters you choose. Non-bracketing delimiters use the same character fore and aft, but the 4 sorts of brackets (round, angle, square, curly) will all nest.

Customary  Generic        Meaning    Interpolates
    ''       q{}          Literal        no
    ""      qq{}          Literal        yes
    ``      qx{}          Command        yes (unless '' is delimiter)
            qw{}         Word list       no
    //       m{}       Pattern match     yes (unless '' is delimiter)
            qr{}          Pattern        yes (unless '' is delimiter)

Assignment statement

Like in Unix scripting languages when assigning values to a variable -- double quote literals are a special kind of expressions in which a substitution of variables is performed. For some reason this macrosubstitution is called interpolation in Perl.

Typical examples:

$a = 'This is a string'; # a scalar assigned 'This is a string'

$b = 11.00; # simple scalar assigned 0 (non significant zeros will be dropped during conversion to double float, so it is like $b = 11;)

$c = '21.0'; # a string that is a well formed number

$d = "this item cost is  $c"; # interpolation will be performed, substituting $c for 21.0

$e = 'this is $a'; # no interpolation in single quoted literals

$f = ''; # just empty string.

Neither single and double quoted literals can span for more than one line. In this case concatenation operator("." -- dot)  should be used.

Perl operators

Perl has more or less typical for high level languages set of operators. Programmers who know C should have the less amount of difficulties. But there are a couple of idiosyncrasies.

The first significant difference with c is that Perl when needs to distinguish between operations on numbers and on text strings it introduces two sets of operations. For example there are two sets of  conditional operations -- one for numbers and the second one for strings: "==" in Perl mean numeric comparison and "eq" -- string comparison.  Only the first will work correctly on number comparisons like testing for zero

Other operators can be found on the perlop manual page.  One need to understand that double quoted literals in Perl are essentially a function that converts a scalar to the string, but at the same both type of comparison operators (for example "==" and "eq" dictate the type or left and right operator so in case left or right oprator is of inproper typer they will be converted before comparison

if (0.0 == 0) { } # true (no covertion performed)

if ("0.0" eq "0") { } # false (no convertion perfomed, strings are unequal as they have freerent length)

if ("0.0" == "0") { } # true (left and right operators will be converted to numeric value which is zero for both

Until recently Perl has had only one representation of  numbers -- double float. In most cases it works OK.  In more complex cases when you need additional precision that design decision leads to troubles.  Later pragma use integer was introduced, that can dictate interpreter to use integer arithmetic for numeric operations.

More on the string comparison operators (gt, lt, eq, cmp, ne)

It is very important to understand that in Perl the operators gt, lt, eq, cmp and ne presuppose conversion to string of both operands before evaluation

First 9.0 will be converted to number ("9") and then this number will be converted into string "9". After that the string '9.0' will be compared with the string '9' and they are not equal. Thus the message "not equal" will always get printed.

Likewise:

$a='10';

if ($a lt '2') { print "left is less" } ;

Evaluates to true since 10 is smaller than 2 when evaluated as a string. String comparison is done from left to right symbol by symbol until the first non-equal symbol is found:

If you compare numeric literal (a number) with a string using string comparison, then numeric literal will first be converted into a number (discarding all training zeros) and then this number will be converted to a string. For example:

$a='1';

if ($a eq 1.0) { print ' $a is equal to 1.0\n'; }

is true because first 1.0 will first be converted to numeric representation which will be converted back to string resulting in string "1". After than we will compare two strings that are equal.

More on the '.' operator

The '.' symbol denotes the concatenation operator in Perl. The operator takes two scalars, and combines them together in one scalar:

appends the string '.' to the end of $sentence.  Adding zeros with concatenation can be used for multiplication by ten:

More on forced conversion to number in Perl

A scalar is interpreted as a number if it is part of an array subscript, is in a numeric comparison operator, or is in an built-in function requiring a number. In case it does not represent a valid numeric string it zero will be used. No conversion error will be ever reported.

• numeric operators: +,-,/,*,**, %(mod)
• numeric comparisons >,<,<=,>=,==
• array subscripts [ ]
• built in functions like int(), localtime().

If the string represents  a "well formed" number it will be converted into numeric value without any problems. For example:

The '+' turns the string 111.00 into a number, so $sum becomes 123.

But if data cannot be converted to numeric, zero is used.

I would like to remind that Perl recognizes floating point numbers and hexadecimal numbers. and underscore can be used to make large numbers more readable, but this representation can not be used in string literals:

Note:  underscore is not accepted in literals and can be used only for numbers without quotes. For example, that means that 1_000_000 in double quotes will not be converted correctly to 100000, but to zero -- an unpleasant surprise. For example string "1_000_000" in not well formed number and during conversion to a number it will be converted to zero: 

The order of statement execution

Summary

Scalars are the most popular type of variable in Perl and one can think about them as strings with optional numeric representation when it makes sense and zero otherwise.  That means that any string in Perl can be converted as a number.  Perl is one of very few languages where operator determines the type of operands (and implicit conversion, if necessary) much like in assembler.

Please be very careful and check your program for typical errors before submitting it to complier. That significantly simplifies the life.

Be especially careful with numeric comparison that involved variable that are strings.  You need to be very careful not to shoot yourself in a foot by implicit conversion.

Strong point of Perl is a very flexible string literal mechanism. By using appropriate quotation mechanism one can avoid errors typical in other languages with less flexible string literal syntax. Perl has rich set of operators and flexible semantic of assignment.

Perl uses prefixes to identify type of variables, much like early languages (Fortran-66).

Prev | Up | Contents | Down | Next