Example

Lesson 22/61

AWK Examples

The best way to learn AWK is probably looking at some examples in a given context, let us create file `data.txt` file which has the following content:

MonthsWaterGasElecPhone
Jan647285133101200
Feb287354425501000
Mar2383212219001710
Apr4686986350002000
May12410979630013000

The csv formatted data file awk-data.csv has already been uploaded to the system, you should be good to go, just press the Run button!

AWK print function

By default Awk prints every line from the file.

#!/bin/bash

# Write the sdtin to a datafile
cp /dev/stdin  awk-data.csv

# Read a file awk-data.csv and print
awk -F, '{print;}' awk-data.csv

bash

Action print with out any argument prints the whole line by default. So it prints all the lines of the file with out fail. Note that the actions need to be enclosed with in the braces.

AWK print with specific field:

#!/bin/bash

# Write the sdtin to a datafile
cp /dev/stdin  awk-data.csv

# Read a file awk-data.csv and print the col 1-2
awk -F, '{print $1,$2;}' awk-data.csv

bash

Print with the actions specified in the BEGIN section executed before AWK starts reading the lines from the input and END actions will be performed after completing the reading and processing the lines from the input.

#!/bin/bash

# Write the sdtin to a datafile
cp /dev/stdin  awk-data.csv

# Read a file awk-data.csv and print the col 1-3 
# with BEGIN and END actions

awk -F, 'BEGIN {print "Period";} \
{print $1,"\t", $2,$3,$NF;} \
END{print "END\n--------------"; }' awk-data.csv

bash

Complete example

Let's consider we want to find a total of al bills in all months in the data. We then create the following script:

#!/bin/bash

# Write the sdtin to a datafile
cp /dev/stdin  awk-data.csv

echo -n "Field sum: Water + Gas + Electtricty + Phones = $"
awk -F "," '{
	if(FNR == 1){
		next;
	}
	
	Water=$2;
	Gas=$3;
	Electricity=$4;
	Phones=$5;

	fields_sum=Water + Gas + Electtricty + Phones;

	total +=fields_sum;

} END { print total; }' awk-data.csv

bash

Note that it's a bash script that calls awk from inside and we have used FNR to detect the first row which we want to avoid in the sum calculation.

AWK built-in variables: As mentioned earlier, the built-in variable $NF represents number of field, in this case last field (5):

#!/bin/bash

# Write the sdtin to a datafile
cp /dev/stdin  awk-data.csv

awk -F, '{print $1,$NF;}' awk-data.csv

bash

AWK fields comparison >: Let's now find the months with water bills > 500:

#!/bin/bash

# Write the sdtin to a datafile
cp /dev/stdin  awk-data.csv

awk -F, '$2 > 500' awk-data.csv

bash

Self-contained AWK scripts example

In Linux systems self-contained AWK scripts can be constructed using. For example, a script that prints the content of a given file may be built by creating a file named printfile.awk with the following content:

#!/bin/bash

# Write the sdtin to a datafile
cp /dev/stdin  awk-data.csv

awk -f /usr/local/data/printfile.awk  awk-data.csv

bash

Class Lessons

Introduction Introduction to Bash based Data a    Bash Shell Bash Tutorial Which Bash Shell? Bash variables and functions Bash meta characters Bash quotation Read and store user input with Ba    Bash conditional statements Bash looping statements Bash arithmetic operations Bash arrays Practice with interactive shell    Regular Expressions Introduction to Regular Expressio Basic Regular Expressions Extended Regular Expressions RegEx character classes RegEx atomic groups RegEx in Bash shell scripting AWK Learn AWK AWK Built-in variables and functi AWK functions Example    SED SED - Stream Editor SED substitutions SED and regular expressions Examples Grep About GREP Examples    GREP and regular expressions Grep with Find Project 1: University Ranking Data Preview Find the colleges in the ranklist Finding the percent of colleges List the Institutes from a given Count all institutes from each st Project decision: University tuit Video Demo Project 2: Facebook Datamining Facebook data Data preview Finding the number of status entr Find the most popular status entr Project decsion: Find the most po Video demo Project 3: Crime Stat Data Introduction Data preview Find the top most crime name Finding the top most crime per ci Project decision: The best with l Video demo Project 4: Text-mining Data Introduction Data preview Counting the Plays and Poems Stat each author works Project decision: Dig the frequen Big Data concepts Big Data and file formats Hadoop Distributed File System Map Reduce YARN Flume Spark Conclusions Conclusions

Quiz

GDPR

When you visit any of our websites, it may store or retrieve information on your browser, mostly in the form of cookies. This information might be about you, your preferences or your device and is mostly used to make the site work as you expect it to. The information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies. Click on the different category headings to find out more and manage your preferences. Please note, that blocking some types of cookies may impact your experience of the site and the services we are able to offer.