Processing Text with awk
1. Understanding awk Syntax
| Component | Description | Example |
|---|---|---|
| awk 'pattern { action }' | Basic awk command structure | awk '/error/ { print $0 }' |
| pattern | Condition to match lines | /error/, $1 ~ /pattern/ |
| action | Command to execute on matched lines | { print $0 }, { $1 = "new" } |
| field | Field separator (default is whitespace) | $1, $2, $0 (entire line) |
| record | Line of input data | NR (number of records), FNR (file number of records) |
| variable | Custom variable declaration | count = 0, name = "John" |
Note: The
$0 represents the entire line, and field numbers start at 1.
2. Using Field Separator
| Option | Description | Example |
|---|---|---|
| -F fs | Set the field separator to fs | awk -F: '{print $1}' /etc/passwd |
| FS | Variable to set field separator in script | awk 'BEGIN {FS=":"} {print $1}' /etc/passwd |
3. Printing Specific Fields
| Field | Description | Example |
|---|---|---|
| NF | Number of fields in the current record | awk '{print NF}' file.txt |
| NR | Number of the current record (line number) | awk '{print NR, $0}' file.txt |
| FNR | Number of the current record in the current file | awk '{print FNR, $0}' file.txt |
| FS | Field separator (default is whitespace) | awk 'BEGIN {FS=":"} {print $1}' /etc/passwd |
| OFS | Output field separator (default is space) | awk 'BEGIN {OFS=","} {print $1, $2}' file.txt |
4. Using Patterns
| Pattern Type | Description | Example | Notes |
|---|---|---|---|
| Regular Expression | Matches text using regex syntax | /[0-9]+/ |
Matches one or more digits |
| String Literal | Matches exact text | "ERROR" |
Case-sensitive match |
| Range | Matches lines between two patterns | /start/, /end/ |
Includes lines matching either pattern |
| Field Reference | Matches specific field in record | $1 == "user" |
Field separator is space or tab |
| Boolean Expression | Combines patterns with logical operators | $1 ~ /user/ && $2 > 100 |
Uses awk's boolean logic |
Note: Patterns can be combined with actions to filter and process input lines based on content or structure.
5. Using Built-in Variables
| Variable | Description | Example Value |
|---|---|---|
| NR | Number of the current record (line) | 5 |
| NF | Number of fields in the current record | 3 |
| FS | Field separator (default is whitespace) | " " |
| RS | Record separator (default is newline) | "\n" |
| OFS | Output field separator (default is space) | " " |
| ORS | Output record separator (default is newline) | "\n" |
| FILENAME | Name of the input file being processed | "data.txt" |
| FNR | Number of the current record in the current file | 10 |
6. Using BEGIN and END Blocks
The BEGIN and END blocks in awk are special constructs that allow you to execute code before the first input line is read and after the last input line is processed, respectively.
| Block | Description | Usage |
|---|---|---|
| BEGIN | Executes once before any input is read. | BEGIN { action } |
| END | Executes once after all input has been processed. | END { action } |
Example: Initializing Variables with BEGIN
awk 'BEGIN { count = 0; print "Starting count: " count } { count++ } END { print "Final count: " count }' data.txt
7. Performing Calculations
| Operation | Description | Example |
|---|---|---|
| Arithmetic Operations | Basic arithmetic operations like addition, subtraction, multiplication, division, and modulo. | awk 'BEGIN { print 5 + 3 }' |
| Variable Assignment | Assigning values to variables for later use in calculations. | awk 'BEGIN { a = 10; b = 5; print a * b }' |
| Built-in Functions | Functions such as sqrt(), sin(), cos(), etc., for mathematical computations. |
awk 'BEGIN { print sqrt(16) }' |
| String to Number Conversion | Converting string values to numbers using + operator or int(). |
awk 'BEGIN { num = "42"; print num + 0 }' |
8. Using Conditional Statements
| Statement | Description | Example |
|---|---|---|
| if (condition) { action } | Executes action if condition is true | awk 'BEGIN { x = 5 } if (x > 3) { print "x is greater than 3" }' |
| if (condition) { action1 } else { action2 } | Executes action1 if condition is true, otherwise executes action2 | awk 'BEGIN { x = 2 } if (x > 3) { print "greater" } else { print "less or equal" }' |
| if (condition1) { action1 } else if (condition2) { action2 } else { action3 } | Chain of conditional checks | awk 'BEGIN { x = 5 } if (x < 3) { print "small" } else if (x < 7) { print "medium" } else { print "large" }' |
Example: Conditional Processing of Fields
echo "Alice 25 Engineer" | awk '{ if ($2 > 20) print $1 " is over 20 years old" }'9. Using awk Functions
| Function | Description | Example |
|---|---|---|
| length() | Returns the length of a string | length("hello") # Returns 5 |
| substr() | Extracts a substring from a string | substr("hello", 2, 3) # Returns "ell" |
| index() | Finds the position of a substring | index("hello", "l") # Returns 3 |
| tolower() | Converts a string to lowercase | tolower("HELLO") # Returns "hello" |
| toupper() | Converts a string to uppercase | toupper("hello") # Returns "HELLO" |
| sin(), cos(), tan() | Trigonometric functions (in radians) | sin(0) # Returns 0 |
| sqrt() | Calculates square root | sqrt(16) # Returns 4 |
| exp() | Exponential function (e^x) | exp(1) # Returns ~2.718 |
| log() | Natural logarithm | log(2.718) # Returns ~1 |
| int() | Truncates a number to integer | int(3.9) # Returns 3 |
Note: These built-in functions can be used in any awk script to process and manipulate text or numeric data.
10. Formatting Output
| Field | Description | Example |
|---|---|---|
| printf | Format and print data using format specifiers | printf "%-10s %d\n" "Name" 25 |
| format specifiers | Control how fields are displayed (width, precision, alignment) | %10s for right-aligned string, %.2f for 2 decimal places |
| escape sequences | Special characters in format strings (e.g., \n, \t) |
printf "Line 1\nLine 2\n" |
Note: The
printf function in awk behaves similarly to C's printf, allowing precise control over output formatting.