Processing Text with sed

1. Understanding sed Syntax

Component	Description	Example
sed [options] 'command' file	Basic sed command structure	`sed 's/old/new/g' file.txt`
s/old/new/g	Substitute command (s = substitute, g = global)	`sed 's/cat/dog/g' animals.txt`
/pattern/replacement/flags	Full substitute syntax	`sed 's/$cat$/\1 is a pet/g' animals.txt`
-i	Edit file in-place	`sed -i 's/old/new/g' file.txt`
-e	Append script command	`sed -e 's/old/new/g' -e 's/another/replacement/' file.txt`
-f script.sed	Read commands from file	`sed -f script.sed file.txt`

Example: Basic substitution

echo "The cat sat on the mat." | sed 's/cat/dog/g'

Example: In-place editing

sed -i 's/old/new/g' file.txt

Note: The s command is the most commonly used sed command. It allows for pattern matching and replacement within text files.

2. Substituting Text

Command	Description	Example
sed 's/pattern/replacement/' file	Replace first occurrence of pattern on each line	Example: Replace first occurrence `echo "hello world hello" \| sed "s/hello/hi/" # Output: hi world hello`
sed 's/pattern/replacement/g' file	Replace all occurrences of pattern on each line	Example: Replace all occurrences `echo "hello world hello" \| sed "s/hello/hi/g" # Output: hi world hi`
sed 's/pattern/replacement/i' file	Case-insensitive replacement	Example: Case-insensitive replace `echo "Hello World" \| sed "s/hello/hi/i" # Output: hi World`
sed 's/pattern/replacement/&/' file	Use matched text in replacement	Example: Use matched text `echo "apple" \| sed "s/a/&&/" # Output: aapple`
sed 's/pattern/replacement/\1/' file	Use captured group from regex	Example: Use captured group `echo "John Smith" \| sed "s/$[A-Z][a-z]$\s$[A-Z][a-z]$/\2, \1/" # Output: Smith, John`

Note: The s command supports regex patterns and can use capture groups with backreferences like \1, \2, etc.

3. Editing Files In-Place

Option	Values	Description
-i	`sed -i 's/old/new/g' file.txt`	Edit files in-place (modifies the original file)
-i.bak	`sed -i.bak 's/old/new/g' file.txt`	Create a backup with .bak extension before editing
-i ''	`sed -i '' 's/old/new/g' file.txt`	On macOS, use empty string to edit in-place without backup

Example: In-place replacement with backup

sed -i.bak 's/old/new/g' example.txt

Note: The -i flag modifies the file directly. Always make a backup if unsure.

4. Deleting Lines with Patterns

Command	Description	Example
d	Delete lines matching a pattern	`sed '/error/d' logfile.txt`
d	Delete lines from a range	`sed '1,5d' file.txt`
d	Delete all lines except those matching a pattern	`sed -n '/error/p' logfile.txt`

Note: The d command deletes lines immediately and does not process them further in the script.

Warning: Be cautious when using d without a pattern, as it will delete all lines in the file.

5. Inserting Lines

Command	Description	Example
sed 'addressi' file	Insert text before the specified line address	Example: Insert before line 3 `sed '3i New line content' file.txt`
sed 'addressa' file	Append text after the specified line address	Example: Append after line 2 `sed '2a New line after' file.txt`
sed '/pattern/i' file	Insert before the first line matching a pattern	Example: Insert before line containing "error" `sed '/error/i Inserted line' file.txt`
sed '/pattern/a' file	Append after the first line matching a pattern	Example: Append after line containing "warning" `sed '/warning/a Appended line' file.txt`

Note: The i and a commands do not read the input line into the pattern space, so they cannot reference it with &.

6. Using Address Ranges

Address Type	Description	Examples
Line Number	Matches a specific line number	`1,5` - lines 1 through 5
Regular Expression	Matches lines containing a regex pattern	`/start/,/end/` - from line matching "start" to line matching "end"
Pattern with Offset	Matches a line and applies an offset	`/pattern/+2` - 2 lines after the line matching "pattern"
Address Range	Defines a range between two addresses	`1,10` - lines 1 to 10

Example: Using Address Ranges to Replace Text in a Range

sed '1,5s/old/new/g' file.txt

Example: Using Address Ranges with Regular Expressions

sed '/start/,/end/s/old/new/g' file.txt

Note: Address ranges can be combined with other sed commands like s, d, and p to perform complex text processing operations.

7. Using Multiple Commands

Example: Chaining sed commands

echo "Hello World" | sed -e "s/World/Universe/" -e "s/Hello/Hi/"
# Output: Hi Universe

Command	Description	Usage
-e	Execute script command	`sed -e 's/old/new/' file`
-f	Read script from file	`sed -f script.sed file`
;	Command separator	`sed 's/a/b/; s/c/d/' file`

Workflow: Applying multiple sed commands

Define commands in sequence
Use -e flag to chain them
Apply transformations in order

Note: Commands are executed in the order they appear, and each can modify the input line before passing it to the next command.

8. Using sed Scripts

Example: Basic sed script to replace text

#!/bin/sed -f
s/old_text/new_text/g

Command	Description	Usage
s/old/new/g	Substitute old with new globally	`s/old_text/new_text/g`
d	Delete matched lines	`d`
p	Print matched lines	`p`
a	Append text after matched line	`a\New line`
i	Insert text before matched line	`i\New line`

Note: sed scripts can be saved in a file and executed with sed -f script.sed input.txt.

Example: sed script with multiple commands

#!/bin/sed -f
s/old/new/g
d
p

9. Printing Specific Lines

Command	Description	Example
sed -n '1p' file.txt	Prints the first line of the file	Example: Print First Line `sed -n '1p' file.txt`
sed -n '$p' file.txt	Prints the last line of the file	Example: Print Last Line `sed -n '$p' file.txt`
sed -n '5p' file.txt	Prints the 5th line of the file	Example: Print Line 5 `sed -n '5p' file.txt`
sed -n '1,5p' file.txt	Prints lines 1 through 5	Example: Print Lines 1-5 `sed -n '1,5p' file.txt`
sed -n '/pattern/p' file.txt	Prints all lines matching the pattern	Example: Print Lines Matching Pattern `sed -n '/error/p' log.txt`
sed -n '/pattern/!p' file.txt	Prints all lines NOT matching the pattern	Example: Print Lines Not Matching Pattern `sed -n '/error/!p' log.txt`

10. Using Regular Expressions

Pattern	Description	Example
^	Start of line	`sed '/^hello/ s/hello/world/' file.txt`
.	Any single character (except newline)	`sed 's/a.c/XXX/g' file.txt`
*	Zero or more of the preceding character	`sed 's/a*/A/g' file.txt`
+	One or more of the preceding character	`sed 's/a\+/A/g' file.txt`
?	Zero or one of the preceding character	`sed 's/a\?/A/g' file.txt`
[abc]	Any one of the characters in brackets	`sed 's/[abc]/X/g' file.txt`
[a-z]	Any lowercase letter from a to z	`sed 's/[a-z]/X/g' file.txt`
^abc	Line starts with "abc"	`sed '/^abc/ s/abc/ABC/' file.txt`
abc$	Line ends with "abc"	`sed '/abc$/ s/abc/ABC/' file.txt`
[0-9]	Any digit from 0 to 9	`sed 's/[0-9]/X/g' file.txt`

Example: Matching and Replacing

echo "The price is $123 and tax is $45" | sed 's/\$[0-9]\+/\$XXX/g'

Example: Using Groups

echo "John Smith, age 30" | sed 's/\([A-Z][a-z]*\) \([A-Z][a-z]*\), age \([0-9]\+\)/\1-\2:\3/'

Note: In sed, regular expressions are by default basic regular expressions (BRE). For extended regular expressions (ERE), use -E or --regexp-extended.