A brief history of Bash shell
A brief intro to Bash, which bash shell is active.
Learn about Bash variables and functions
In this lesson we learn about Bash meta charachers
Learn more Bash quotation basics. Single quotes, double quotes and ANSI-style quotes
Learn to read and store user input with Bash.
Learn about Bash conditional statements
Learn about Bash for-loop, while-loop and until.
Learn about Bash arithmetic functions
Learn about Bash arrays.
In this lesson we give you the opportunity to practice with an interactive shell
In this lesson we learn about the history of RegEx, what you can do with RegEx and the types.
Learn about the Basic Regular Expressions (BRE).
In this lesson, we learn about the Extended Regular Expressions (ERE) meta characters.
Learn about the RegEX character classes which make one small sequence of characters match a larger set of characters.
Learn about the Regular expressions atomic groups.
How to use RegEx in Bash shell scripting? Learn from examples.
Learn about the AWK scripting language.
Learn the AWK Built-in variables and functions.
Learn about the AWK functions
AWK useful examples with hands-on playgrounds
Learn about the SED - Stream Editor.
Detailed examples of SED based substitutions
Learn to use SED and regular expressions
SED delete, print and grouping
Learn about a brief history of GREP.
Let's learn grep by doing some examples.
Learn about the GREP and regular expressions, how to work together.
Learn how to use GREP and thee find commands together.
Learn how to do data Preview with head, pipe and csvlook.
Find the colleges in the ranklist with grep, pipe and wc commands.
Finding the percent of colleges in the ranklist. Learn to use wc and grep
How to list the Institutes from a given state?
Learn to use, cut and sort together.
Decision of the project
Project 1 demo
Preview data
Look into the FB data
Learn to use csvcut, sort and uniq commands.
Use awk to find the most popular FB status entry.
Use Bash functions for finding the most popular FB status entry.
Project 2 video demo.
Introduction to the AU crime stat data.
Preview the crime stat data.
Use sort, tail and awk to find the top most crime name in the AU cities.
Use csvcut, sort, tail and awk to find the top most crime per city.
Use Bash shell scripting to find the best with least crimes
Video demo of the project # 3.
Shakespearean-era Plays and poems data introduction
Data preview with csvkit
Use head, sed, grep and wc for counting the Plays and Poems.
Finding plays or poems by each author using the sed, sort, uniq, and head.
This is a complex example of awk and Bash functions
Learn about the Big data file formats
Concepts of the Hadoop Distributed File System
Concepts of the Map Reduce technology
Learn about the concepts of YARN.
Flume concepts.
Apache Spark
Final lesson and conclusions.
Bash may not be the best way to handle all kinds of data, but there often comes a time when you are provided with a pure Bash environment, such as what we get in the common Linux based supercomputers and you just want an early result or view of the data before driving into the real programming, using Python, R and SQL, SPSS, and so on. Expertise in data-intensive languages comes at the price of spending a lot of time on them. In contrast, bash scripting is simple, easy to learn and perfect for mining textual data. Therefore, learning Bash shell should be the first step if you want to say, Hello to “Big Data”!
This interactive course will demonstrate four practical flat file data mining projects involving four data projects (each with a different objective function): University ranking data, Facebook data, Australian statistics crime data and Shakespeare-era plays and poems data (Big data!)If you haven’t used Bash before, feel free to skip the projects and get to the tutorials part . Complete the tutorials first and then come back to the projects again. The tutorial section will introduce with bash scripting, regular expressions, AWK, sed, grep and so on. The course finishes with a near-complete list of references to all the relevant command line and Big data tools.Target audience and prerequisites: Almost everyone can benefit from learning to use Bash particularly in data mining: particularly students who want to learn Bash and the command line to improve their career prospects, researchers who want to add Bash and other command line tools to their bag of tricks, scientists who want to learn to explore and analyze the data that their lab generates.Requirements It is a self-learning course with all Linux environments provided.Outcome Use Bash to quickly sort, search, match, replace, clean and optimise various aspect of a data set Use bash in processing real-world data sets (included) Use Bash commands and scripting Use Regular Expressions (RegEX) in Bash Use AWK programming language commands to tweak and format data Use SED and GREP to quickly search in large-scale data sets
Please enroll to view the course forum.
Please describe about the report short and clearly.
Share course with your friends
Cookies and similar technologies are used on our sites to personalise content and ads. You can find further details and change your personal settings below. By clicking OK, or by clicking any content on our sites, you agree to the use of these cookies and similar technologies.
When you visit any of our websites, it may store or retrieve information on your browser, mostly in the form of cookies. This information might be about you, your preferences or your device and is mostly used to make the site work as you expect it to. The information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies. Click on the different category headings to find out more and manage your preferences. Please note, that blocking some types of cookies may impact your experience of the site and the services we are able to offer.