GNU Parallel

GNU Parallel is a free and open source GNU software project, a command-line shell script especially designed to be used when executing jobs in parallel, on one or more remote computers that run a GNU/Linux operating system.

Helps you to run small scripts or single commands

The tool can be used to run single commands or small scripts, as well as a command that reads from a pipe. The typical input is a list of hosts, a list of files, a list of users, a list of tables, or a list of URLs.

In addition, users can use GNU Parallel to split the input and pipe into commands, in parallel. The program has been designed from the offset as an alternative for the ‘cat | bash’ or ‘xargs’ commands.

Getting started with GNU Parallel

Installing GNU Parallel on your GNU/Linux computer is an easy task. You can always grab the latest version of the project from Softoware or via its official website (see link at the end of the article). Save the archive somewhere on your computer and extract its contents using an archive manager software.

Open a Terminal app, use the ‘cd’ command to navigate to the location of the extracted archive files (e.g. cd /home/softoware/parallel-20141122), then execute the ‘configure && make’ command to configure and compile the program for your operating system/hardware architecture.

Install it system wide by running the ‘sudo make install’ command as a privileged user or the ‘make install’ command if your logged in as root. Several command-line program will be generated (parralle, niceload, sem and sql), but the main one is ‘parallel’.

Various command-line options are available for each of the aforementioned programs. To view the usage message of each one, add the ‘--help’ option to the main command (e.g. parallel --help).

What is new in this release:

• --tollef has been retired.
• --compress has be redesigned due to bugs, making this release alpha quality.
• The format of --eta and --joblog has changed slightly.

What is new in version 20140122:

• GNU Parallel was cited in: On the likelihood of multiple bit upsets in logic circuits http://arxiv-web3.library.cornell.edu/pdf/1401.1003
• HaploClique uses GNU Parallel https://github.com/armintoepfer/haploclique
• Scraping NSScreencast https://blog.nicolai86.eu/posts/2014-01-12/scraping-nsscreencast/
• 30 Cool Open Source Software I Discovered in 2013 http://www.cyberciti.biz/open-source/30-cool-best-open-source-softwares-of-2013/
• [Unix] 13. The power of doing things in Parallel http://leetaey.tistory.com/384
• Parallel the execution of a job that read from stdin http://www.linuxask.com/questions/parallel-the-execution-of-a-job-that-read-from-stdin
• Mon Make a moi (6:38-11:50) http://videos.rennes.inria.fr/ReNaBI-GO2013/indexPierreLindenbaum.html
• Shell-Abarbeitung beschleunigen: Wie Sie mit parallelen Prozesse effizienter in der Shell arbeiten https://www.hosteurope.ch/blog/shell-abarbeitung-beschleunigen-wie-sie-mit-parallelen-prozesse-effizienter-in-der-shell-arbeiten/
• Summary of GNU Parallel tutorial http://hacktracking.blogspot.dk/2014/01/gnu-parallel-tutorial.html
• Bug fixes and man page updates.

What is new in version 20131222:

• Detect if the buffer dir $TMPDIR runs out of space. This changes a central piece of code, making this release alpha quality.
• --keep-order --(n)onall will sort according to the sshlogin.
• Detection number of CPUs on DEC Tru64.
• GNU sql: --list-tables for sqlite3.
• Bug fixes and man page updates.

What is new in version 20131122:

• A citation notice is printed on stderr only if stderr is a terminal, the user has not specified --no-notice, and the user has not run --bibtex once.
• --compress will compress temporary files. --compress-program controls which program to use for compressing temporary files.
• --bar shows progress as a progress bar compatible with zenity.
• --resume can now be used with --result: jobs already run will be skipped.
• --transfer and --basefile support paths relative to the --workdir by inserting /./ into the path.

What is new in version 20131022:

• Using --transfer with files containing /./ in the path will copy the files relative to the --workdir.
• The maximal command length is now cached in a file, halving the startup time.

What is new in version 20130922:

• PDF-files of documentation are now included.
• Bug fixes and man page updates have been made.

What is new in version 20130822:

• This version adds a tutorial showing much of GNU Parallel's functionality. man parallel_tutorial --line-buffer will buffer output on a line basis.
• --record-env records all variables and functions set (useful to record the normal environment before using --env _).
• --env _ exports all functions and variables to the remote server except for the ones recorded.

What is new in version 20130622:

• --xapply now recycles arguments if an input source has more arguments than others.
• The sleep time between jobs is now both increased and decreased exponentially.
• 10 seconds installation checks the signature using GnuPG if GnuPG is installed.

What is new in version 20130522:

• The software can now be installed in 10 seconds by running wget -O - pi.dk/3 | sh.
• CPU detection was added for HURD, IRIX, SCO OpenServer, and HPUX.
• --ctrlc will send SIGINT to tasks running on remote computers and thus kill them.
• --load now uses 'ps' to find the number of running processes instead of 'uptime' and is much faster.
• CentOS 3.9 and FreeBSD 7 compatibility bugs were fixed.
• Some of these required quite extensive changes, making this release beta quality.
• --retries now works with --onall.
• The new --load computation now works on FreeBSD 7.
• --nice works under tcsh