gawk - Unix, Linux Command

NAME

gawk - pattern scanning and processing language.

SYNOPSIS

gawk [ POSIX or GNU style options ] -f program-file [ -- ] file ...
gawk [ POSIX or GNU style options ] [ -- ] program-text file ...

pgawk [ POSIX or GNU style options ] -f program-file [ -- ] file ...
pgawk [ POSIX or GNU style options ] [ -- ] program-text file ...

DESCRIPTION

Pgawk is the profiling version of gawk. It is identical in every way to gawk, except that programs run more slowly, and it automatically produces an execution profile in the file awkprof.out when done. See the --profile option, below.

The command line consists of options to gawk itself, the AWK program text (if not supplied via the -f or --file options), and values to be made available in the ARGC and ARGV pre-defined AWK variables.

OPTION FORMAT

Following the POSIX standard, gawk-specific options are supplied via arguments to the -W option. Multiple -W options may be supplied Each -W option has a corresponding long option, as detailed below. Arguments to long options are either joined with the option by an = sign, with no intervening spaces, or they may be provided in the next command line argument. Long options may be abbreviated, as long as the abbreviation remains unique.

OPTIONS

AWK PROGRAM EXECUTION

pattern { action statements }
function name(parameter list) { statements }

Gawk first reads the program source from the program-file(s) if specified, from arguments to --source, or from the first non-option argument on the command line. The -f and --source options may be used multiple times on the command line. Gawk reads the program text as if all the program-files and command line source texts had been concatenated together. This is useful for building libraries of AWK functions, without having to include them in each new AWK program that uses them. It also provides the ability to mix library functions with command line programs.

The environment variable AWKPATH specifies a search path to use when finding source files named with the -f option. If this variable does not exist, the default path is ".:/usr/local/share/awk". (The actual directory may vary, depending upon how gawk was built and installed.) If a file name given to the -f option contains a ‘‘/’’ character, no path search is performed.

Gawk executes AWK programs in the following order. First, all variable assignments specified via the -v option are performed. Next, gawk compiles the program into an internal form. Then, gawk executes the code in the BEGIN block(s) (if any), and then proceeds to read each file named in the ARGV array. If there are no files named on the command line, gawk reads the standard input.

If a filename on the command line has the form var=val it is treated as a variable assignment. The variable var will be assigned the value val. (This happens after any BEGIN block(s) have been run.) Command line variable assignment is most useful for dynamically assigning values to the variables AWK uses to control how input is broken into fields and records. It is also useful for controlling state if multiple passes are needed over a single data file.

If the value of a particular element of ARGV is empty (""), gawk skips over it.

For each record in the input, gawk tests to see if it matches any pattern in the AWK program. For each pattern that the record matches, the associated action is executed. The patterns are tested in the order they occur in the program.

Finally, after all the input is exhausted, gawk executes the code in the END block(s) (if any).

VARIABLES, RECORDS AND FIELDS

Records

Fields

As each input record is read, gawk splits the record into fields, using the value of the FS variable as the field separator. If FS is a single character, fields are separated by that character. If FS is the null string, then each individual character becomes a separate field. Otherwise, FS is expected to be a full regular expression. In the special case that FS is a single space, fields are separated by runs of spaces and/or tabs and/or newlines. (But see the discussion of --posix, below). NOTE: The value of IGNORECASE (see below) also affects how fields are split when FS is a regular expression, and how records are separated when RS is a regular expression.

If the FIELDWIDTHS variable is set to a space separated list of numbers, each field is expected to have fixed width, and gawk splits up the record using the specified widths. The value of FS is ignored. Assigning a new value to FS overrides the use of FIELDWIDTHS, and restores the default behavior.

Each field in the input record may be referenced by its position, $1, $2, and so on. $0 is the whole record. Fields need not be referenced by constants:

n = 5
print $n

prints the fifth field in the input record.

The variable NF is set to the total number of fields in the input record.

References to non-existent fields (i.e. fields after $NF) produce the null-string. However, assigning to a non-existent field (e.g., $(NF+2) = 5) increases the value of NF, creates any intervening fields with the null string as their value, and causes the value of $0 to be recomputed, with the fields being separated by the value of OFS. References to negative numbered fields cause a fatal error. Decrementing NF causes the values of fields past the new value to be lost, and the value of $0 to be recomputed, with the fields being separated by the value of OFS.

Assigning a value to an existing field causes the whole record to be rebuilt when $0 is referenced. Similarly, assigning a value to $0 causes the record to be resplit, creating new values for the fields.

Built-in Variables

Gawk’s built-in variables are:

Arrays

Arrays are subscripted with an expression between square brackets ([ and ]). If the expression is an expression list (expr, expr ...) then the array subscript is a string consisting of the concatenation of the (string) value of each expression, separated by the value of the SUBSEP variable. This facility is used to simulate multiply dimensioned arrays. For example:

i = "A"; j = "B"; k = "C"
x[i, j, k] = "hello, world\n"

assigns the string "hello, world\n" to the element of the array x which is indexed by the string "A\034B\034C". All arrays in AWK are associative, i.e. indexed by string values.

The special operator in may be used in an if or while statement to see if an array has an index consisting of a particular value.

if (val in array)
        print array[val]

If the array has multiple subscripts, use (i, j) in array.

The in construct may also be used in a for loop to iterate over all the elements of an array.

An element may be deleted from an array using the delete statement. The delete statement may also be used to delete the entire contents of an array, just by specifying the array name without a subscript.

Variable Typing And Conversion

Variables and fields may be (floating point) numbers, or strings, or both. How the value of a variable is interpreted depends upon its context. If used in a numeric expression, it will be treated as a number, if used as a string it will be treated as a string.

To force a variable to be treated as a number, add 0 to it; to force it to be treated as a string, concatenate it with the null string.

When a string must be converted to a number, the conversion is accomplished using strtod(3). A number is converted to a string by using the value of CONVFMT as a format string for sprintf(3), with the numeric value of the variable as the argument. However, even though all numbers in AWK are floating-point, integral values are always converted as integers. Thus, given

CONVFMT = "%2.2f"
a = 12
b = a ""

the variable b has a string value of "12" and not "12.00".

Gawk performs comparisons as follows: If two variables are numeric, they are compared numerically. If one value is numeric and the other has a string value that is a ‘‘numeric string,’’ then comparisons are also done numerically. Otherwise, the numeric value is converted to a string and a string comparison is performed. Two strings are compared, of course, as strings. Note that the POSIX standard applies the concept of ‘‘numeric string’’ everywhere, even to string constants. However, this is clearly incorrect, and gawk does not do this. (Fortunately, this is fixed in the next version of the standard.)

Note that string constants, such as "57", are not numeric strings, they are string constants. The idea of ‘‘numeric string’’ only applies to fields, getline input, FILENAME, ARGV elements, ENVIRON elements and the elements of an array created by split() that are numeric strings. The basic idea is that user input, and only user input, that looks numeric, should be treated that way.

Uninitialized variables have the numeric value 0 and the string value "" (the null, or empty, string).

Octal and Hexadecimal Constants

String Constants

String constants in AWK are sequences of characters enclosed between double quotes ("). Within strings, certain escape sequences are recognized, as in C. These are:

In compatibility mode, the characters represented by octal and hexadecimal escape sequences are treated literally when used in regular expression constants. Thus, /a\52b/ is equivalent to /a\*b/.

PATTERNS AND ACTIONS

{ print }

which prints the entire record.

Comments begin with the ‘‘#’’ character, and continue until the end of the line. Blank lines may be used to separate statements. Normally, a statement ends with a newline, however, this is not the case for lines ending in a ‘‘,’’, {, ?, :, &&, or ||. Lines ending in do or else also have their statements automatically continued on the following line. In other cases, a line can be continued by ending it with a ‘‘\’’, in which case the newline will be ignored.

Multiple statements may be put on one line by separating them with a ‘‘;’’. This applies to both the statements within the action part of a pattern-action pair (the usual case), and to the pattern-action statements themselves.

Patterns

BEGIN 
END 
/regular expression/ 
relational expression 
pattern && pattern 
pattern || pattern 
pattern ? pattern : pattern 
(pattern) 
! pattern 
pattern1, pattern2 

BEGIN and END are two special kinds of patterns which are not tested against the input. The action parts of all BEGIN patterns are merged as if all the statements had been written in a single BEGIN block. They are executed before any of the input is read. Similarly, all the END blocks are merged, and executed when all the input is exhausted (or when an exit statement is executed). BEGIN and END patterns cannot be combined with other patterns in pattern expressions. BEGIN and END patterns cannot have missing action parts.

For /regular expression/ patterns, the associated statement is executed for each input record that matches the regular expression. Regular expressions are the same as those in egrep(1), and are summarized below.

A relational expression may use any of the operators defined below in the section on actions. These generally test whether certain fields match certain regular expressions.

The &&, ||, and ! operators are logical AND, logical OR, and logical NOT, respectively, as in C. They do short-circuit evaluation, also as in C, and are used for combining more primitive pattern expressions. As in most languages, parentheses may be used to change the order of evaluation.

The ?: operator is like the same operator in C. If the first pattern is true then the pattern used for testing is the second pattern, otherwise it is the third. Only one of the second and third patterns is evaluated.

The pattern1, pattern2 form of an expression is called a range pattern. It matches all input records starting with a record that matches pattern1, and continuing until a record that matches pattern2, inclusive. It does not combine with any other sort of pattern expression.

Regular Expressions

Character classes are a new feature introduced in the POSIX standard. A character class is a special notation for describing lists of characters that have a specific attribute, but where the actual characters themselves can vary from country to country and/or from character set to character set. For example, the notion of what is an alphabetic character differs in the USA and in France.

A character class is only valid in a regular expression inside the brackets of a character list. Character classes consist of [:, a keyword denoting the class, and :]. The character classes defined by the POSIX standard are:

Two additional special sequences can appear in character lists. These apply to non-ASCII character sets, which can have single symbols (called collating elements) that are represented with more than one character, as well as several characters that are equivalent for collating, or sorting, purposes. (E.g., in French, a plain ‘‘e’’ and a grave-accented e` are equivalent.)

Actions

Operators

The operators in AWK, in order of decreasing precedence, are

Control Statements

The control statements are as follows:

if (condition) statement [ else statement ]
while (condition) statement 
do statement while (condition)
for (expr1; expr2; expr3) statement
for (var in array) statement
break
continue
delete array[index]
delete array
exit [ expression ]
{ statements }

I/O Statements

The input/output statements are as follows:

NOTE: If using a pipe or co-process to getline, or from print or printf within a loop, you must use close() to create new instances of the command. AWK does not automatically close pipes or co-processes when they return EOF.

The printf Statement

The AWK versions of the printf statement and sprintf() function (see below) accept the following conversion specification formats:

Optional, additional parameters may lie between the % and the control letter:

Special File Names

When doing I/O redirection from either print or printf into a file, or via getline from a file, gawk recognizes certain special filenames internally. These filenames allow access to open file descriptors inherited from gawk’s parent process (usually the shell). These file names may also be used on the command line to name data files. The filenames are:

print "You blew it!" > "/dev/stderr"

whereas you would otherwise have to use

print "You blew it!" | "cat 1>&2"

The following special filenames may be used with the |& co-process operator for creating TCP/IP network connections.

Numeric Functions

AWK has the following built-in arithmetic functions:

String Functions

Gawk has the following built-in string functions:

Time Functions

Bit Manipulations Functions

Internationalization Functions

USER-DEFINED FUNCTIONS

function name(parameter list) { statements }

Functions are executed when they are called from within expressions in either patterns or actions. Actual parameters supplied in the function call are used to instantiate the formal parameters declared in the function. Arrays are passed by reference, other variables are passed by value.

Since functions were not originally part of the AWK language, the provision for local variables is rather clumsy: They are declared as extra parameters in the parameter list. The convention is to separate local variables from real parameters by extra spaces in the parameter list. For example:

function  f(p, q,     a, b)     # a and b are local
{
        ...
}

/abc/   { ... ; f(1, 2) ; ... }

The left parenthesis in a function call is required to immediately follow the function name, without any intervening white space. This is to avoid a syntactic ambiguity with the concatenation operator. This restriction does not apply to the built-in functions listed above.

Functions may call each other and may be recursive. Function parameters used as local variables are initialized to the null string and the number zero upon function invocation.

Use return expr to return a value from a function. The return value is undefined if no value is provided, or if the function returns by ‘‘falling off’’ the end.

If --lint has been provided, gawk warns about calls to undefined functions at parse time, instead of at run time. Calling an undefined function at run time is a fatal error.

The word func may be used in place of function.

DYNAMICALLY LOADING NEW FUNCTIONS

SIGNALS

EXAMPLES

Print and sort the login names of all users:

        BEGIN   { FS = ":" }
                { print $1 | "sort" }

Count lines in a file:

                { nlines++ }
        END     { print nlines }

Precede each line by its number in the file:

        { print FNR, $0 }

Concatenate and line number (a variation on a theme):

        { print NR, $0 }
Run an external command for particular lines of data:

        tail -f access_log |
        awk ’/myhome.html/ { system("nmap " $1 ">> logdir/myhome.html") }’

INTERNATIONALIZATION

String constants are sequences of characters enclosed in double quotes. In non-English speaking environments, it is possible to mark strings in the AWK program as requiring translation to the native natural language. Such strings are marked in the AWK program with a leading underscore (‘‘_’’). For example,

gawk ’BEGIN { print "hello, world" }’

always prints hello, world. But,

gawk ’BEGIN { print _"hello, world" }’

might print bonjour, monde in France.

There are several steps involved in producing and running a localizable AWK program.

POSIX COMPATIBILITY

The book indicates that command line variable assignment happens when awk would otherwise open the argument as a file, which is after the BEGIN block is executed. However, in earlier implementations, when such an assignment appeared before any file names, the assignment would happen before the BEGIN block was run. Applications came to depend on this ‘‘feature.’’ When awk was changed to match its documentation, the -v option for assigning variables before program execution was added to accommodate applications that depended upon the old behavior. (This feature was agreed upon by both the Bell Laboratories and the GNU developers.)

The -W option for implementation specific features is from the POSIX standard.

When processing arguments, gawk uses the special option ‘‘--’’ to signal the end of arguments. In compatibility mode, it warns about but otherwise ignores undefined options. In normal operation, such arguments are passed on to the AWK program for it to process.

The AWK book does not define the return value of srand(). The POSIX standard has it return the seed it was using, to allow keeping track of random number sequences. Therefore srand() in gawk also returns its current seed.

Other new features are: The use of multiple -f options (from MKS awk); the ENVIRON array; the \a, and \v escape sequences (done originally in gawk and fed back into the Bell Laboratories version); the tolower() and toupper() built-in functions (from the Bell Laboratories version); and the ANSI C conversion specifications in printf (done first in the Bell Laboratories version).

HISTORICAL FEATURES

a = length      # Holy Algol 60, Batman!

is the same as either of

a = length()
a = length($0)

This feature is marked as ‘‘deprecated’’ in the POSIX standard, and gawk issues a warning about its use if --lint is specified on the command line.

The other feature is the use of either the continue or the break statements outside the body of a while, for, or do loop. Traditional AWK implementations have treated such usage as equivalent to the next statement. Gawk supports this usage if --traditional has been specified.

GNU EXTENSIONS

The following features of gawk are not available in POSIX awk.

When gawk is invoked with the --traditional option, if the fs argument to the -F option is ‘‘t’’, then FS is set to the tab character. Note that typing gawk -F\t ... simply causes the shell to quote the ‘‘t,’’, and does not pass ‘‘\t’’ to the -F option. Since this is a rather ugly special case, it is not the default behavior. This behavior also does not occur if --posix has been specified. To really get a tab character as the field separator, it is best to use single quotes: gawk -F’\t’ ....

If gawk is configured with the --enable-switch option to the configure command, then it accepts an additional control-flow statement:

switch (expression) {
case value|regex : statement
...
[ default: statement ]
}

ENVIRONMENT VARIABLES

If POSIXLY_CORRECT exists in the environment, then gawk behaves exactly as if --posix had been specified on the command line. If --lint has been specified, gawk issues a warning message to this effect.

EXAMPLES

Basic function of gawk is to search for pattern(s) in each line of input and perform specific task when the pattern is found. The input to gawk can be either file, standard input (keyboard) or even output of another command/program. A gawk program is a series of rules with each rule specifying a pattern to search for, followed by the action to take when the pattern is found.
Below is the basic syntax of gawk rule:

pattern { action }

Ways to run a gawk program

1. One-shot throw-away programs
This is useful when your program may have few instructions and you don't want to create a separate for this program.

Below command prints the lines from /etc/password file where the string "root" appears.

$ gawk '/root/ { print }' /etc/passwd

root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

$ gawk '{split($0,a,":"); print a[1]}' /etc/passwd

root
bin
daemon
adm
...
...

2. Without input files
One can run gawk without any input files. In such case input for program will be taken from standard input.

Below example is same as above but only difference here is that input comes from STDIN (or keyboard)

$ gawk '/root/ { print }'

root test      => This is user input
root test      => This is ouput from program as there is match for "root"
simple test    => This is user input. No match for "root"
complex test   => This is user input. No match for "root"

3. Running long programs
When the program is big and runs into several instructions, it is convenient to put it into separate file. By convention the the gawk program file can be named as ".awk" but it really does not matter whether it has any extension or not. If the program is in separate file, it has to be executed as shown below:

gawk -f program-file input-file1 input-file-2...

{
        split($0, a, ":")
        if ( a[1] == "root" )
                print
}

$ gawk -f test.awk /etc/passwd

root:x:0:0:root:/root:/bin/bash

4. Self contained gawk scripts
The script will have the gawk interpreter specified within itself and can be directly invoked without using explicit gawk on the command line. The script starts with "#!" followed by the full path of the gawk binary on the system.

Consider below program file: test.awk

#!/bin/gawk -f

{
        split($0, a, ":")
        if ( a[1] == "root" )
                print
}

$ chmod +x test.awk
$ ls -l test.awk
-rwxr-xr-x 1 root root 70 Jan 26 13:06 test.awk

$ ./test.awk /etc/passwd

root:x:0:0:root:/root:/bin/bash

Initialization and Cleanup

#!/bin/gawk -f

BEGIN {
        print "START: Initialization activity called here"
}
{
        print
}
END {
        print "END: Cleanup called here"
}

$ ./test.awk /etc/passwd

START: Initialization activity called here
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
...
...
END: Cleanup called here

User-Defined Functions

#!/bin/gawk -f

function printName(name)
{
        print "My name is "name
}

BEGIN {
        print "START"

        printName("John");
        printName("Tony");
}
END {
        print "END"
}

$ ./test.awk /etc/passwd
START
My name is John
My name is Tony
END

User defined Variables

#!/bin/gawk -f

BEGIN {
        print "START"
        $totalLines = 0;
}
{
        totlaLines++;
        # Do some processing here
}
END {
        print "Total Lines processed: "totlaLines
        print "END"
}

$ ./test.awk /etc/passwd

START
Total Lines processed: 35
END