paexec

paexec is an utility that distributes given tasks (stdin) across several CPUs or machines in a network.

paexec has no limited internal buffers anymore. All they are resized automatically as it is needed. PAEXEC_BUFSIZE environment variable sets an *initial* buffer size, not *maximum* one.

README: notes about non-standard function getopt_long, and advice how to build paexec on platforms with no getopt_long support (JP-UX, Solaris etc.).

Installation:

0) BSD make is required. I name it 'bmake' but its real name may
vary. pmake or just make are possible variants.

If you need to change the default building options,
run bmake like this

env [YOUR_ASSIGNMENTS] bmake

See example section below

1) "libmaa" library is required to build paexec. It is a part of
"dict" project http://sourceforge.net/projects/dict

Quick sample of building:

gzip -dc /home/cheusov/downloads/dictd-X.Y.Z.tar.gz | tar -xf-
cd dictd-X-Y-Z/libmaa
./configure
gmake
gmake install

NOTE: gmake is GNU make here! Under most Linux distros
it is named make.

2) Uncompress paexec tarball you've downloaded like this
gzip -dc paexec-X-Y-Z.tar.gz | tar -xf-

3) cd paexec-X-Y-Z

4) bmake

5) (optional!) bmake install-dirs

6) bmake install

There are a lot of Makefile variables that can be changed during
building and installation.
paexec's own variables (All they are at the begining of Makefile):

PREFIX - where paexec is installed to
LIBMAA - linker option for linking "maa" library

BSD make's variables (most commonly used,
for all others - see make's documentation and .mk files)

BINDIR - where paexec executable itself is installed to
MANDIR - where manual pages are installed to

BINOWN - paexec executable owner
BINGRP - paexec executable group

MANOWN - man page owner
MANGRP - man page group

Examples of building and installing:

1) bmake all install-dirs install

2) env CC=icc
PREFIX=/home/cheusov/local
CPPFLAGS='-I/usr/pkg/include'
LDFLAGS='-L/usr/pkg/lib -Wl,-rpath -Wl,/usr/pkg/lib'
LDADD=-lextralib
CFLAGS='-Werror -Wall'
BINOWN=cheusov
BINGRP=users
MANOWN=cheusov
MANGRP=users
MKCATPAGES=no
bmake -s all install-dirs install

3) paexec needs the non-portable getopt_long(3) present in
*BSD and Linux OSes. If it is absent in your OS,
you can build paexec like the following.

echo '#include ' > ./config.h

env
CPPFLAGS='-DNO_PORTABHACKS_H=1 -DHAVE_CONFIG_H=1'
LDFLAGS='-lnbcompat'
bmake -s all install-dirs install

where libnbcompat is a NetBSD's portability library

Of course, you can also use any other implementation of getopt_long.

NOTE: No, I'll not use autotools

What is new in this release:

• libmaa is not needed anymore to build and run paexec.
• Presentation paexec.pdf was added.
• Environment variable PAEXEC_EOT was introduced.
• Option -y was added to paexec(1) and paexec_reorder(1) for setting an end-of-task marker to the "magic" string.
• Option -C was added to paexec(1) for setting "calculator" more easily.
• Option -x was added to paexec_reorder(1).
• Additions and improvements to regression tests and documentation were made.

What is new in version 0.18.0:

• NULL dereference (sigsegv) has been fixed in paexec(1).
• It happened when the first line given on input was empty.
• An entire command passed to ssh-like transport is now shquoted.
• In particular this fixes -x, which didn't work together with -t. "paexec -g" now accepts an empty string as a task.
• A minor fix has been made in the paexec_reorder man page.
• More regression tests have been added.

What is new in version 0.17.0:

• The option -x instructs paexec(1) to run one command per task.
• Improvements for the option -n.
• The option '-m t=' sets an alternative end of task marker.
• The option -md= sets a delimiter between tasks in graph mode (-g).
• An output line that contains failed dependants no longer ends with an unnecessary delimiter character.
• Long options were completely removed.
• "paexec_reorder -g" now correctly handles failed tasks' output.
• The option -m was added to paexec_reorder(1), which is similar to paexec's -m.
• More examples and regression tests.
• Documentation updates and improvements.

What is new in version 0.16.0:

• This version has a fix for compilation failures on ancient OpenBSD, HP-UX, and maybe others.
• New modes for reordering tasks were added; see -W0 and -W2 options.
• Long options are still supported but considered deprecated.
• "mkcmake test" now has a chance to succeed on HP-UX, Solaris, and other SVR4 derivates.

What is new in version 0.15.0:

• The build system has changed from mk-files to mk-configure.
• There is a new executable paexec_resort(1) for normalizing the sliced output of paexec(1).
• There is a new option -W for reordering tasks.
• With this option, paexec(1) is able to minimize total processing time by distributing tasks over the network or CPUs in a better way.
• There is a fix for a 1 byte buffer overflow that happened when the -d (debugging) option was applied.
• Documentation updates, minor fixes, and code clean-ups.

What is new in version 0.13.0:

• The algorithm for cycle detection (paexec -s) was completely reworked and now does't allocate tasks_count^2 integers.
• Now it works much faster.
• The -Z _timeout_ option was added.
• An attempt to rerun a command on a failed node is made every _timeout_ seconds.
• This option makes it possible to organize clusters over unreliable networks, hardware or software.

What is new in version 0.12.0:

• This release adds a -z option. If applied, read/write(2) operations from/to nodes become non-critical.
• In case paexec has lost connection to the node, it will reassign the failed task to another node and, if the -s option is applied, will output the string "fatal" to stdout.
• This makes paexec resistant to the I/O errors, and as a result you can create paexec clusters even over a network consisting of unreliable hosts (Internet?).
• Failed hosts are marked as such, and will not be used during the current run of paexec. There are minor fixes.

What is new in version 0.11.0:

• This release adds paexec -s, which checks an input task's graph for cycles before beginning actual work. If it/they are detected, paexec exits with an error.
• A minor manpage fix has been made.

What is new in version 0.10.1:

• A minor fix was made for "paexec -s" (rhomb-like dependencies).
• A minor fix was made in the man page.
• Clean-ups were done in tests/test.sh. getopt_long(3) falls back to getopt(3) on Solaris, HP-UX, Interix, etc.