5 command line techniques to be more productive

Intended audience: developers

Unix-like operating systems power the majority of the Internet, and if you’re a web developer, chances are you interact with them daily. Whether you’re using the Linux or macOS command line interface, the Windows Subsystem for Linux, a Vagrant box, or connecting to a remote server via SSH, here’s a list of five command line techniques to be more productive.

Using grep to find a string in your codebase

Grep was first released in 1974 and is a simple program that is designed to globally search a regular expression and print the results. Most modern IDEs and file explorers have some version of this functionality but grep is faster and is guaranteed to be available on pretty much any shell.

The basic format that I use is:

$ grep -ril "the string you want to find" * .[^.]*

In the above, the r and i flag instructs grep to do a recursive case-insensitive search, and the l flag instructs it to display only files with matches.

To limit your search to only specific file types, just add the --include parameter.

$ grep -ril --include=*.php "the string you want to find" * .[^.]*

Sometimes, you may want to exclude certain subdirectories, such as cache or log, and you can do so with the --exclude-dir parameter.

$ grep -ril --include=*.php --exclude-dir=cache "the string i want to find" * .[^.]*

Using comm to compare two files

Sometimes you need to compare the contents of two files: maybe you need to identify entries that are in one file but not the other, or you need to see which entries are present in both. For instance, imagine that you have the two files below, each of which contains a list of names. These names are not stored in any specific order and are duplicated. How can you identify names that appear in one list and not the other, or find the names that appear in both files? Diff can help with the first task, but not the second. Comm to the rescue!

File 1:

Mo
Mo
Carlos
Mo
Ernesto
Ernesto
Carlos
Mo
Lisa
Peter
Mo
Doug

File2:

Lisa
Doug
Doug
Carlos
Lisa
Doug
Peter
Lisa
Carlos
Philipp
Carlos

To see the names that appear in one file but not the other (column 1 of the output displays entries that are in file1 but not in file2, and vice versa):

$ comm -3 <(sort -u file1) <(sort -u file2)

Ernesto
Mo
       Philipp

To see the names that appear in both files:

$ comm -12 <(sort -u file1) <(sort -u file2)

Carlos
Doug
Lisa
Peter

Using an SSH tunnel to remotely access localhost

Sometimes, you may need to access a service that is only accessible through localhost. For instance, when you’re trying to connect to MySQL -- either via the command line or phpMyAdmin -- when it is configured only to allow access through localhost. This can be a pain but the solution is trivially simple: an SSH tunnel.

For instance, say I want to access a remote server on port 8080 but I want the connection to look like it’s originating from localhost. I simply set up an SSH tunnel, which remains active until I end the SSH session:

$ ssh -L localhost:9090:localhost:8080 mo@mugo.ca

... and then connect locally through my command line or browser:

$ curl http://localhost:9090/phpmyadmin

Similarly, say I want to connect to a remote MySQL database that is configured not to allow remote connections (for instance, because I want to use a MySQL GUI tool on my desktop). I simply set up an SSH tunnel:

$ ssh -L localhost:3307:localhost:3306 mo@mugo.ca

... and then connect locally through my command line or GUI tool:

$ mysql -uroot -p -P 3307 -h localhost

Dumping MySQL databases directly into gzip

Sometimes you need to back up or restore MySQL databases, and on larger sites this can take forever. A big timesaver is to pipe the dump directly into gzip and then store that as a file:

$ mysqldump -uroot -p --single-transaction --databases database1 database2 database3 | gzip > backup.sql.dump.gz

Oh, and you can restore a gzipped dump back into MySQL directly -- without first decompressing to a file:

$ gunzip < backup.sql.dump.gz | mysql -uroot -p

Remember, however, that if you have an issue with the gzipped archive you won’t have an uncompressed sql file to use instead -- so running a gzip -t after the dump is a good idea.

Generating random words for passwords

This last one is a testament to the versatility of the command line interface and the core assumption of Unix-like environments. We’re all familiar with the xkcd on password strength, so here’s a one-liner that will give you 10 pseudo-random common English words to build your password sentence from:

$ for i in {1..10}; do awk "NR==$(jot -r 1 1 $(wc <(curl -s https://raw.githubusercontent.com/first20hours/google-10000-english/master/google-10000-english-no-swears.txt) | awk '{print $1;}'))" <(curl -s https://raw.githubusercontent.com/first20hours/google-10000-english/master/google-10000-english-no-swears.txt); done;

frozen
coffee
toshiba
personnel
thursday
loves
rehab
adam
shark
needed

Frozen coffee loves Thursday indeed! 

Obviously, limiting the words to a list of 10,000 common ones has security implications, but you can mitigate against that by using a system dictionary instead (for instance, /usr/share/dict/web2).