Linux Shell Scripting Cookbook - Second Edition

Shell Something Out -- Have A Good Command -- File In, File Out -- Texting And Driving -- Tangled Web? Not At All! -- The Backup Plan -- The Old-boy Network -- Put On The Monitor's Cap -- Administration Calls. Authors Names From About The Authors Page. Includes Index. Cover 1 Copyright 3 Credits 4 About the Authors 5 About the Reviewers 7 www.PacktPub.com 9 Table of Contents 14 Preface 18 Chapter 1: Shell Something Out 24 Introduction 25 Printing in the terminal 27 Playing with variables and environment variables 30 Function to prepend to environment variables 34 Math with the shell 36 Playing with file descriptors and redirection 38 Arrays and associative arrays 44 Visiting aliases 46 Grabbing information about the terminal 48 Getting and setting dates and delays 49 Debugging the script 53 Functions and arguments 54 Reading the output of a sequence of commands 57 Reading n characters without pressing the return key 60 Running a command until it succeeds 61 Field separators and iterators 62 Comparisons and tests 65 Chapter 2: Have a Good Command 70 Introduction 70 Concatenating with cat 71 Recording and playing back of terminal sessions 74 Finding files and file listing 75 Playing with xargs 85 Translating with tr 90 Checksum and verification 94 Cryptographic tools and hashes 97 Sorting unique and duplicates 100 Temporary file naming and random numbers 106 Splitting files and data 107 Slicing filenames based on extension 109 Renaming and moving files in bulk 112 Spell checking and dictionary manipulation 114 Automating interactive input 116 Making commands quicker by running parallel processes 119 Chapter 3: File In, File Out 122 Introduction 123 Generating files of any size 123 The intersection and set difference (A-B) on text files 124 Finding and deleting duplicate files 127 Working with file permissions, ownership, and the sticky bit 130 Making files immutable 135 Generating blank files in bulk 136 Finding symbolic links and their targets 137 Enumerating file type statistics 138 Using loopback files 141 Creating ISO files and hybrid ISO 144 Finding the difference between files, patching 147 Using head and tail for printing the last or first 10 lines 149 Listing only directories – alternative methods 152 Fast command-line navigation using pushd and popd 153 Counting the number of lines, words, and characters in a file 155 Printing the directory tree 156 Chapter 4: Texting and Driving 160 Introduction 160 Using regular expressions 161 Searching and mining a text inside a file with grep 164 Cutting a file column-wise with cut 171 Using sed to perform text replacement 175 Using awk for advanced text processing 179 Finding the frequency of words used in a given file 185 Compressing or decompressing JavaScript 187 Merging multiple files as columns 190 Printing the nth word or column in a file or line 191 Printing text between line numbers or patterns 192 Printing lines in the reverse order 193 Parsing e-mail addresses and URLs from a text 194 Removing a sentence in a file containing a word 195 Replacing a pattern with text in all the files in a directory 197 Text slicing and parameter operations 198 Chapter 5: Tangled Web? Not At All! 200 Introduction 201 Downloading from a web page 201 Downloading a web page as plain text 204 A primer on cURL 205 Accessing Gmail e-mails from the command line 209 Parsing data from a website 211 Image crawler and downloader 212 Web photo album generator 215 Twitter command-line client 218 Creating a "define" utility by using the Web backend 223 Finding broken links in a website 226 Tracking changes to a website 228 Posting to a web page and reading the response 231 Chapter 6: The Backup Plan 234 Introduction 234 Archiving with tar 235 Archiving with cpio 241 Compressing data with gzip 243 Archiving and compressing with zip 247 Faster archiving with pbzip2 248 Creating filesystems with compression 249 Backup snapshots with rsync 251 Version control based backup with Git 254 Creating entire disk images with fsarchiver 257 Chapter 7: The Old-boy Network 260 Introduction 260 Setting up the network 261 Let us ping! 267 Listing all the machines alive on a network 271 Running commands on a remote host with SSH 274 Transferring files through the network 278 Connecting to a wireless network 282 Password-less auto-login with SSH 284 Port forwarding using SSH 286 Mounting a remote drive at a local mount point 287 Network traffic and port analysis 288 Creating arbitrary sockets 291 Sharing an Internet connection 292 Basic firewall using iptables 293 Chapter 8: Put on the Monitor's Cap 296 Introduction 296 Monitoring disk usage 297 Calculating the execution time for a command 302 Collecting information about logged in users, boot logs, and boot failures 305 Listing the top 10 CPU consuming processes in an hour 308 Monitoring command outputs with watch 310 Logging access to files and directories 311 Logfile management with logrotate 313 Logging with syslog 314 Monitoring user logins to find intruders 316 Remote disk usage health monitor 320 Finding out active user hours on a system 322 Measuring and optimizing power usage 325 Monitoring disk activity 326 Checking disks and filesystems for errors 327 Chapter 9: Administration Calls 330 Introduction 330 Gathering information about processes 331 Killing processes and send or respond to signals 341 Sending messages to user terminals 344 Gathering system information 346 Using /proc for gathering information 347 Scheduling with cron 348 Writing and reading the MySQL database from Bash 352 User administration script 357 Bulk image resizing and format conversion 361 Taking screenshots from the terminal 364 Managing multiple terminals from one 365 Index 368 Over 110 practical recipes to solve real-world shell problems, guaranteed to make you wonder how you ever lived without them **Overview** * Master the art of crafting one-liner command sequence to perform text processing, digging data from files, backups to sysadmin tools, and a lot more * And if powerful text processing isn't enough, see how to make your scripts interact with the web-services like Twitter, Gmail * Explores the possibilities with the shell in a simple and elegant way - you will see how to effectively solve problems in your day to day life **In Detail** The shell remains one of the most powerful tools on a computer system — yet a large number of users are unaware of how much one can accomplish with it. Using a combination of simple commands, we will see how to solve complex problems in day to day computer usage. Linux Shell Scripting Cookbook, Second Edition will take you through useful real-world recipes designed to make your daily life easy when working with the shell. The book shows the reader how to effectively use the shell to accomplish complex tasks with ease. The book discusses basics of using the shell, general commands and proceeds to show the reader how to use them to perform complex tasks with ease. Starting with the basics of the shell, we will learn simple commands with their usages allowing us to perform operations on files of different kind. The book then proceeds to explain text processing, web interaction and concludes with backups, monitoring and other sysadmin tasks. Linux Shell Scripting Cookbook, Second Edition serves as an excellent guide to solving day to day problems using the shell and few powerful commands together to create solutions. **What you will learn from this book** * Explore a variety of regular usage tasks and how it can be made faster using shell command * Write shell scripts that can dig data from web and process it with few lines of code * Use different kinds of tools together to create solutions * Interact with simple web API from scripts * Perform and automate tasks such as automating backups and restore with archiving tools * Create and maintain file/folder archives, compression formats and encrypting techniques with shell * Set up Ethernet and Wireless LAN with the shell script * Monitor different activities on the network using logging techniques **Approach** This book is written in a Cookbook style and it offers learning through recipes with examples and illustrations. Each recipe contains step-by-step instructions about everything necessary to execute a particular task. The book is designed so that you can read it from start to end for beginners, or just open up any chapter and start following the recipes as a reference for advanced users. **Who this book is written for** This book is both for the casual GNU/Linux users who want to do amazing things with the shell, and for advanced users looking for ways to make their lives with the shell more productive. You can start writing scripts and one-liners by simply looking at the similar recipe and its descriptions without any working knowledge of shell scripting or Linux. Intermediate/advanced users as well as system administrators/ developers and programmers can use this book as a reference when they face problems while coding. Over 110 practical recipes to solve real-world shell problems, guaranteed to make you wonder how you ever lived without them Overview Master the art of crafting one-liner command sequence to perform text processing, digging data from files, backups to sysadmin tools, and a lot more And if powerful text processing isn't enough, see how to make your scripts interact with the web-services like Twitter, Gmail Explores the possibilities with the shell in a simple and elegant way - you will see how to effectively solve problems in your day to day life In Detail The shell remains one of the most powerful tools on a computer system — yet a large number of users are unaware of how much one can accomplish with it. Using a combination of simple commands, we will see how to solve complex problems in day to day computer usage. Linux Shell Scripting Cookbook, Second Edition will take you through useful real-world recipes designed to make your daily life easy when working with the shell. The book shows the reader how to effectively use the shell to accomplish complex tasks with ease. The book discusses basics of using the shell, general commands and proceeds to show the reader how to use them to perform complex tasks with ease. Starting with the basics of the shell, we will learn simple commands with their usages allowing us to perform operations on files of different kind. The book then proceeds to explain text processing, web interaction and concludes with backups, monitoring and other sysadmin tasks. Linux Shell Scripting Cookbook, Second Edition serves as an excellent guide to solving day to day problems using the shell and few powerful commands together to create solutions. What you will learn from this book Explore a variety of regular usage tasks and how it can be made faster using shell command Write shell scripts that can dig data from web and process it with few lines of code Use different kinds of tools together to create solutions Interact with simple web API from scripts Perform and automate tasks such as automating backups and restore with archiving tools Create and maintain file/folder archives, compression formats and encrypting techniques with shell Set up Ethernet and Wireless LAN with the shell script Monitor different activities on the network using logging techniques Approach This book is written in a Cookbook style and it offers learning through recipes with examples and illustrations. Each recipe contains step-by-step instructions about everything necessary to execute a particular task. The book is designed so that you can read it from start to end for beginners, or just open up any chapter and start following the recipes as a reference for advanced users. Who this book is written for This book is both for the casual GNU/Linux users who want to do amazing things with the shell, and for advanced users looking for ways to make their lives with the shell more productive. You can start writing scripts and one-liners by simply looking at the similar recipe and its descriptions without any working knowledge of shell scripting or Linux. Intermediate/advanced users as well as system administrators/ developers and programmers can use this book as a reference when they face problems while coding.

Don't neglect the shell -- this book will empower you to use simple commands to perform complex tasks. Whether you're a casual or advanced Linux user, the cookbook approach makes it all so brilliantly accessible and, above all, useful. Master the art of crafting one-liner command sequence to perform text processing, digging data from files, backups to sysadmin tools, and a lot more And if powerful text processing isn't enough, see how to make your scripts interact with the web-services like Twitter, Gmail Explores the possibilities with the shell in a simple and elegant way - you will see how to effectively solve problems in your day to day life In Detail The shell remains one of the most powerful tools on a computer system -- yet a large number of users are unaware of how much one can accomplish with it. Using a combination of simple commands, we will see how to solve complex problems in day to day computer usage. Linux Shell Scripting Cookbook, Second Edition will take you through useful real-world recipes designed to make your daily life easy when working with the shell. The book shows the reader how to effectively use the shell to accomplish complex tasks with ease. The book discusses basics of using the shell, general commands and proceeds to show the reader how to use them to perform complex tasks with ease. Starting with the basics of the shell, we will learn simple commands with their usages allowing us to perform operations on files of different kind. The book then proceeds to explain text processing, web interaction and concludes with backups, monitoring and other sysadmin tasks. Linux Shell Scripting Cookbook, Second Edition serves as an excellent guide to solving day to day problems using the shell and few powerful commands together to create solutions This book is written in a Cookbook style and it offers learning through recipes with examples and illustrations. Each recipe contains step-by-step instructions about everything necessary to execute a particular task. The book is designed so that you can read it from start to end for beginners, or just open up any chapter and start following the recipes as a reference for advanced users.If you are a beginner or an intermediate user who wants to master the skill of quickly writing scripts to perform various tasks without reading the entire manual, this book is for you. You can start writing scripts and one-liners by simply looking at the similar recipe and its descriptions without any working knowledge of shell scripting or Linux. Intermediate/advanced users as well as system administrators/ developers and programmers can use this book as a reference when they face problems while coding.