Working with Text Manipulation Tools

1. Sorting Lines

Command	Description	Examples
sort	Sorts lines of text in ascending order	`cat file.txt \| sort`
sort -r	Sorts lines in descending order	`sort -r file.txt`
sort -n	Numeric sort (treats content as numbers)	`sort -n numbers.txt`
sort -k	Sorts based on a specific field or column	`sort -k 2 file.csv`
sort -u	Removes duplicate lines after sorting	`sort -u file.txt`

Note: The sort command treats all input as text and sorts lexicographically by default.

2. Sorting by Field

Option	Values	Description
-k	`field1[,field2]`	Sort by specified field(s) using `field1` as primary and `field2` as secondary key
-t	`separator`	Specify field separator (default is whitespace)
-n		Sort numerically instead of lexicographically
-r		Reverse sort order (descending)
-f		Ignore case differences during sort

Example: Sort by field

cat employees.txt
John,IT,5000
Alice,HR,4500
Bob,IT,6000

sort -t',' -k3,3 employees.txt
Bob,IT,6000
John,IT,5000
Alice,HR,4500

Example: Sort numerically by field

sort -t',' -k3,3n employees.txt
Alice,HR,4500
John,IT,5000
Bob,IT,6000

3. Removing Duplicate Lines

Command	Description	Example
uniq	Filters adjacent duplicate lines from input	`cat file.txt \| uniq`
sort \| uniq	Removes all duplicate lines (not just adjacent)	`sort file.txt \| uniq`
uniq -d	Prints only duplicate lines (repeated more than once)	`uniq -d file.txt`
uniq -u	Prints only unique lines (not repeated)	`uniq -u file.txt`
uniq -c	Counts occurrences of each line and prefixes with count	`uniq -c file.txt`

Note: uniq only removes adjacent duplicate lines by default. To remove all duplicates, pipe through sort first.

Example: Remove duplicate lines from a log file

sort /var/log/syslog | uniq > unique_log.txt

4. Cutting Columns

Command	Description	Options	Example
cut	Cuts sections from each line of files	`-f` field list, `-d` delimiter	`cut -f 1,3 file.txt`
cut -f	Specify fields to extract	`-f 1,3` or `-f 1-3`	`cut -f 2- file.txt`
cut -d	Set delimiter character	`-d :`	`cut -d : -f 1,2 /etc/passwd`

Example: Extracting Fields from a CSV File

cut -d ',' -f 1,3 data.csv

Example: Extracting Fields from a Log File

cut -d ' ' -f 1-3 /var/log/syslog

5. Pasting Files Side by Side

Command	Description	Example
paste	Merges files side by side, separating columns with tabs	Example: Basic paste usage `paste file1.txt file2.txt`
paste -d	Allows custom delimiter between columns	Example: Using custom delimiter `paste -d '\|' file1.txt file2.txt`
paste - -	Reads from stdin when '-' is used as a file argument	Example: Combining stdin with file `echo 'line1' \| paste - file2.txt`

6. Translating Characters

Command	Description	Example
tr	Translates or deletes characters from input	`echo "hello" \| tr 'aeiou' 'AEIOU'`

Example: Translate lowercase vowels to uppercase

echo "hello world" | tr 'aeiou' 'AEIOU'
# Output: "hEllO wOrld"

Example: Delete specific characters

echo "hello world" | tr -d 'l'
# Output: "heo word"

Example: Complement translation

echo "hello world" | tr -c 'aeiou' 'X'
# Output: "XXXeXXXoXXXaXXX"

7. Joining Files

Command	Description	Example
cat	Concatenates and displays files	`cat file1.txt file2.txt > combined.txt`
paste	Merges files horizontally (side-by-side)	`paste file1.txt file2.txt > merged.txt`
join	Merges files based on a common field (typically sorted)	`join -1 2 -2 1 file1.txt file2.txt > joined.txt`

Example: Using cat to join files

cat <<EOF > output.txt
Line 1 from file A
Line 2 from file A
EOF

cat <<EOF >> output.txt
Line 1 from file B
Line 2 from file B
EOF

Example: Using paste to merge files side-by-side

echo -e "A1\nA2\nA3" > file1.txt
echo -e "B1\nB2\nB3" > file2.txt
paste file1.txt file2.txt

Example: Using join to merge files on a common field

echo -e "1 apple\n2 banana" > file1.txt
echo -e "1 red\n2 yellow" > file2.txt
join -1 1 -2 1 file1.txt file2.txt

8. Splitting Files

Command	Description	Usage Example
split	Splits files into fixed-size pieces	`split -b 10M largefile.zip split_part_`
split -l	Splits files into fixed number of lines	`split -l 1000 logfile.txt log_`
split -C	Splits files into chunks of specified size (in bytes) without breaking lines	`split -C 500k data.txt chunk_`

Example: Splitting a Large File into 10MB Chunks

split -b 10M largefile.zip split_part_

Example: Splitting a Log File into 1000-Line Segments

split -l 1000 logfile.txt log_

Note: By default, split creates files with names like xxa, xxb, etc. The prefix can be customized using the last argument.

Warning: When splitting files with -C, lines longer than the specified size will be split, potentially causing data loss or corruption if not handled carefully.

9. Reversing Lines

Command	Description	Example
tac	Prints lines in reverse order	`tac file.txt`
rev	Reverses characters in each line	`rev file.txt`

Note: tac reverses the order of lines, while rev reverses characters within each line.

10. Formatting Text

Command	Description	Example
fmt	Formats text to a specified width (default 75 columns)	`echo "This is a long line of text that needs to be formatted." \| fmt -w 40`
pr	Converts spaces to tabs and formats text into columns	`pr -4 file.txt`
expand	Converts tabs to spaces (default 8 spaces per tab)	`expand -t 4 file.txt`
unexpand	Converts spaces to tabs (if they are at the start of a line)	`unexpand -a file.txt`

Note: The fmt command is useful for wrapping paragraphs to a specific width, particularly when preparing text for printing or display in fixed-width environments.