TXR

TXR is an open source command-line and multi-platform command-line software written in the C and Lisp programming language, using Flex and Bison to provide developers with a new data munging programming language.

The software can also act as a text processing utility that can match text and extract pieces from it, and it’s capable of outputting the raw bindings that it collects from the text files into a template-driven report.

Features at a glance

Key features include support for pattern matching using regexp (regular expressions), garbage collection, exception handling, quasiliterals, UNIX signal handling, bignum integers, named blocks, quasiquotes, symbols, vectors, lexical closures, lists (lazy and eager), standard strings and lazy strings, non-local and dynamic control transfers, structural macros that contain destructuring argument lists, as well as modern text handling.

Additionally, the software supports lexically and dynamically scoped variables, different types of streams, UTF-8 and Unicode character encodings, lexically and global scoped symbol and function-like macros, and hash tables with sluggish definitions for values and key, or for regular expressions that contain advanced operators like difference, complement and intersection.

Another interesting feature is the ability to provide different ways for making input handling implicit, through a recursive, nested pattern matching notation that can bind variables. The software is influenced by many programming languages, including Python, Common Lisp, Ruby, POSIX Shell, Scheme, S-Lang, Awk, Prolog, M4, Arc and Clojure.

A cross-platform and portable software

TXR is a cross-platform and portable software that has been successfully tested under the GNU/Linux (Ubuntu/Debian), BSD (FreeBSD/NetBSD), Solaris, Microsoft Windows via Cygwin, MinGW, as well as Mac OS X operating systems. Both 64-bit and 32-bit instruction set architectures are supported at this time.

For your convenience, the application is available for download as a universal source archive, installable on any GNU/Linux or BSD distributions, as well as pre-built binary packages for the Debian, Ubuntu, Solaris, Microsoft Windows and Mac OS X operating systems.

What is new in this release:

• Features:
• New anaphoric macros ifa and conda.
• New function have, synonym of true. Usefully expressive with anaphoric ifa.
• Bugs:
• equal-based hashing function is improved so list and vector permutations do not collide to the same value.
• cat-str function detects overflow in the calculation of the total string length to allocate, and throws an exception.
• Fixed neglected null termination in mkstring function.
• Fixed garbage collector crash introduced in TXR 108, when traversing syntax_tree member of the parser_t structure.

What is new in version 107:

• Features:
• Change in representation of promise objects. The force function now detects recursion and throws an error.
• New mlet macro: a step beyond Scheme's letrec.
• A trivial optimization in the keywordp function more than doubles the interpretation speed of TXR Lisp.
• Bugs:
• Fixed regression in the garbage collector introduced in October 2014, TXR 100. This would cause runaway memory growth, due to the garbage collector creating unnecessary new heaps.

What is new in version 104:

• Features:
• Symbol macros are now shadowed by lexically scoped functions, when referenced from the interior of a form which uses Lisp-1 semantics (a [] bracketed form, a form based on the op family of operators, or explicit use of the dwim operator).
• New function, abort.
• TXR now doesn't terminate abnormally (abort) on unhandled exceptions, but only terminates unsuccessfully. This provides a better experience on the MinGW-based Windows target, where we get an annoying dialog box on abort.
• New *uhandled-hook* variable can be used to register a user-defined function which is called when an unhandled exception occurs.
• New arithmetic function, trunc-rem.
• When the numbered arguments of partial application under the op family of operators are interpolated into a string quasiliteral, modifiers may now be applied. For instance (op prinl `@{1 20} @{2 20}`) denotes a function which prints its two arguments in twenty-character-wide fields separated by a space.
• Improvements in Windows installer: reduced broadcast delay in registerting environment variable. Improved messages in this area.
• A txr-win.exe can be built now which avoids creating a console window. This is included in the installer.
• Bugs:
• Fixed bug in quasiliteral string evaluation in TXR Lisp, in conjunction with op arguments like @1, @2, ... When a value interpolated from one of these arguments looked like a form, it was mistakenly subject to recursive processing and variable substitution as if it were part of the quasiliteral target syntax.
• Fixed a serious bug in the op macro and all related operators like do and ap, in the handling of situations with missing argument numbers was broken: such as when @3 is referenced, but not @1 or @2 (which must generate a function of at least three arguments, which ignores its first two). This key feature was discovered to be utterly broken.
• Fix in internal representation of exceptions: (throw 'x "foo") is now the same as (throwf x "foo").

What is new in version 100:

• Features:
• New functions: chr-isblank and chr-isunisp
• New generic less and greater functions, which are also used as default functions in sort, find-max, max-pos, and others.
• New directives @(line) and @(chr) for binding or asserting the current line number or character position.
• Lisp expressions allowed on left side of @(bind) and @(rebind) now.
• New function lcm (lowest common multiple).
• The gcd function now takes zero or more arguments instead of exactly two.
• New @(include) for parse-time loading of code, useful for loading macros which are needed later in the same file.
• Beginning of library external to TXR executable: macros txr-if, txr-when and txr-bind for more convenient access back into the pattern language from TXR Lisp.
• New combinator function notf for negating a function.
• Bugs:
• Fixed December 2011 regression affecting @(freeform) directive.
• Fixed GC-safety bug in abs-path-p function.
• Fixed breakage in scanner and parser introduced in August. Caught by C++ compiler.
• Fixed inappropriate printed rendering of list objects produced by interpolated TXR Lisp expressions in @(output) blocks, and in the quasiliterals of the pattern language. (TXR Lisp quasiliterals not affected.) Users who depend on the old behavior not wanting to fix their programs can use --compat 99.
• Bugfix in the gcd and lognot functions: neglecting to normalize some bignum result to the fixnum type.
• Bugfix in @(eof) directive: not matching the end of interactive streams.
• Fixed abort due to assertion going off when GC is disabled and the array of new generation objects runs out of space. This could happen during large parses.
• Fixed parser stack overflow and inefficiencies when handling large TXR programs.
• Bugfix in match_fun causing memory accesses to automatic storage that has been released, as well as an invalid longjmp.
• Hash table reorganization is prevented during hash table traversal, so existing items are not skipped or visited twice.

What is new in version 99:

• Features:
• Variables in the pattern language can be bound to regexes, in which case they perform regex matches when matched against input.
• Representation for compiled regexes is streamlined.
• Regex objects now print in regex notation.
• Bugs:
• Fixed August 11 regression affecting 96 through 98: broken matching of unbound variables followed by bound variables.
• Fixed one more problem with andf function: (andf) not returning t.

What is new in version 97:

• A few GC-related fixes after code review.
• Fixed @(load), broken in TXR 94.
• Fixed broken @{var mod} syntax Lisp quasiliterals, broken in TXR 96.

What is new in version 80:

• Features:
• TXR now nicely handles null characters in text stream inputs. They are internally encoded to the Unicode value U+DC00, which will map back to a single null byte on output (that being existing behavior).
• TXR now has Lisp macros: the defmacro operator is implemented, with destructuring lambda lists and all.
• New operators tree-bind and tree-case for pattern binding similar to Common Lisp's destructuring. This piggybacks off the defmacro infrastructure which we now have.
• Big improvement in debuggability: the unhandled exception error message now gives source code location information.
• New functions pos, pos-if, posq, posql and posqual for finding item positions in sequences.
• Predicate function is now optional in the some, all and none functions.
• hash-uni and hash-isec functions take a join-func argument which lets you specify how elements from two tables are merged.
• new hash table functions inhash and hash-update-1.
• two hashes can now be tested for deep equality using equal.
• Bugs:
• Removed bogus optimization from hash table implementation.
• Syntactic fix: input like 1.0a no longer parses as a floating-point number followed by a symbol, but is diagnosed as an error. Syntax like 123a continues to work as before: it denotes one symbol, not 123 followed by a.
• Bugfix in type check for "cobj" class objects that would crash if given non-heaped values like fixnum integers.
• Corrected problems in the code walking and expansion of lambda and defun forms.
• Fixed failure to propagate line number info through the abstract syntax of string quasiliterals.
• Doc bugs: missing descriptions of gethash and gensym.

What is new in version 79:

• Features:
• New functions comb, perm, rcomp and rperm for calculating repeating and non-repeating combinations and permuations of sequences (lists, vectors and strings). Hashes are supported by comb function.
• Bugs:
• Hardening of code for handling excessively large vectors.
• Bugfix in quasistring handling in TXR Lisp.
• Bugfix in if function (not the if operator).

What is new in version 78:

• Improvements in the apply function, reduce-left, and reduce-right, and improved design and implementation of optional arguments.
• Lists, vectors, and strings are now generally function-callable, not only under the dwim operator. Some new functions were added.
• Bugs in the do operator were fixed along with some other problems.

What is new in version 77:

• Features:
• More streamlined parser containing fewer hacks, and fewer obscure cases that don't work.
• @'expr syntax now works as expected in directives for evaluating quoted TXR Lisp expressions.
• In nested uses of the do and op operators in TXR Lisp there is now a way to refer to the outer parameters from the inner nestings, using compounded meta syntax like (op (op @1 @@1)), where @1 is argument 1 of the function denoted by the inner op, and @@1 is argument 1 of the outer function. Each additional @ "escapes" out one level of nesting of the op syntax.
• New update and hash-update functions.
• The interfaces of reduce-left and reduce-right functions has been improved, making them easier to use, while mostly retaining backward compatibility.
• New functions remove-path and rename-path for removing and renaming filesystem objects.
• Catenated streams, previously an internal feature, are exposed now via the make-catenated-stream function.
• Scope rule change for expressions that provide default intialization for optional arguments. These eexpressions now have the parameters in scope, and so now uses like (lambda (x : (y (length x))) ...) are possible where y is initialized with (length x) if the argument is not supplied. Previously, parameter x would not have been considered to be in scope of the expression (length x).
• Bugs:
• Fixed neglected handling of result form in dohash syntax.
• In the object printer, the handling of syntax like @(sys:var x ...) was ignoring any additional ... forms, and rendering as @x.
• Fixed possible race condition in tail streams, whereby when a file rotates, the stream prematurely follows the new file, neglecting to read the last piece of material just added to the previous file.

What is new in version 76:

• Features:
• New time functions: time-fields-local and time-fields-utc for obtaining broken-down time from Epoch seconds.
• New group-by function for constructing a hash from a list when the list elements can be partitioned into equivalence classes tied to keys, rather than treated individually.
• Sweeping changes in TXR List to allow vectors and strings to be manipulated in many situations as if they were lists. Functions like car, cdr and mapcar work on strings and vectors.
• New command line options -e and -p for evaluating TXR Lisp expressions more conveniently from the command line.
• The and, or and if special operators are now also provided as functions, so they can be indirected upon functionally.
• New functions conses and conses*, useful for iterating over a list similarly to Common Lisp's maplist function.
• New do operator (unrelated to @(do) directive) similar to op, but geared toward doing imperative things involving special operators.
• Bugs:
• @(require ...) directive was not expanding its forms prior to evaluation.

What is new in version 75:

• Features:
• Two new stream functions: unget-char and unget-byte. Some streams now support ungetting a byte or character, which was a glaring omission in the API, without which some common scanning tasks are awkward.
• TXR Lisp functions with optional parameters can now specify expressions to give those arguments values when the arguments are missing.
• New operators in TXR Lisp: append-each and append-each*.
• Change in the Lisp structure printer. The special structure generated by the read syntax @sym and @(...) now prints back in the same syntax, rather than as (sys:var sym) and (sys:expr ...).
• Bugs:
• Fix in put-byte function: before invoking the underlying operation, it was testing whether the put-char operation exists for the stream, rather than the put-byte operation. This would result in a crash if the stream supports put-char but not put-byte.
• Mistake in calculating bitmasks for regex character class ranges, resulting in incorrect behavior for ranges whose upper range is a character code corresponding to the last bit of a word in the bitmask array, e.g. [A-\x7f], resulting in failures to match 32 or more characters in the upper end of the range.
• Missing documentation filled in for the functions throw, throwf and error.

What is new in version 74:

• Features:
• Maintenance: builds on Cygwin, MinGW and Mac OS X 10.7.3.
• New math functions: n-choose-k, n-perm-k, cum-norm-dist.
• lisp-parse function renamed read; old name is obsolescent.
• In the TXR pattern language, the @ escape can now evaluate a non-compound TXR Lisp expression also. Previously compounds like @(+ 1 1) were supported, but atoms like @foo were not.
• Bugs:
• Small fix in how exponent fields of printed floating point values are normalized. Across all platforms, there are now no leading zeros after the 'e'.

What is new in version 73:

• Features:
• The new lisp-parse function scans TXR Lisp expressions at run-time out of strings and streams, which means that TXR can now more easily keep persistent, complex data in text files.
• Bugs:
• Fixed signal-handling-related performance issue caused by excessive calls to the sigprocmask function.

What is new in version 72:

• Features:
• Syslog functionality: openlog, closelog, syslog, setlogmask. Plus: the *stdlog* stream for logging directly to syslog.
• Stream properties.
• logand and logior functions become variadic.
• Signal handling support. TXR Lisp code can catch POSIX signals.
• Syntax changes in the area of symbol names. Package prefixes are supported now, like foo:bar (symbol bar in package foo).
• Bugs:
• Nonsensical error diagnostics in intern and delete-package, in the case when a package doesn't exist.
• defvar is documented now and behaves more similarly to the Common Lisp defvar.
• seek-stream with a zero offset was reporting the current offset instead of seeking regardless of the value of the whence argument.

What is new in version 67:

• This minor maintenance release adds some new functions and a new directive.

What is new in version 66:

• This version adds many new features, bugfixes, and completion of the reference manual.

What is new in version 65:

• This version has been ported to OS X, FreeBSD, and NetBSD, supports a few popular regex tokens, exposes the regex compiler as a function (allowing programs to build and use regular expression syntax trees), provides new ways of iterating over hash tables with lazy lists, adds some time functions, improves seeding of PRNG, and fixes a bug related to argument processing in @(next) directive.

What is new in version 62:

• Floating point support has been introduced, allowing TXR to be more suitable for scientific data munging, making this a major milestone.
• Some new functional combinators have been added.
• There is a small refinement in URL/percent encoding/decoding.
• A showstopper bug in recently introduced quicksort routine has been repaired, affecting vector/string sorting.

What is new in version 059:

• This is a bugfix release.

What is new in version 058:

• TXR programs can now be split into modules.
• Embedded TXR Lisp can now call back into the pattern language.
• Exception handling is now exposed in TXR Lisp.
• Optional arguments are supported in TXR Lisp functions.
• The braced variable substitution syntax in output has been extended to do arbitrary expressions, and its indexing notation is now generic over vectors and strings.
• More generic sequence functions have been added to the library, including a generic sort.
• There's a new del operator for deleting sequence elements/ranges and hash elements denoted by the bracket syntax.

What is new in version 057:

• "Freeform" scanning (treating an arbitrarily sized remainder of the input as one big string) is now possible in constant memory.
• Bugs in hex and octal character constants, the format function, and printing of string objects were fixed.

What is new in version 056:

• This version fixes incorrect UTF-8 decoding of a valid character range, and closes security issues related to UTF-8 decoding and re-encoding.
• A regression that broke computed field widths in release 55 is fixed.
• Hex and octal escapes in string literals and quasiliterals are now recognized like the documentation says.
• The [...] notation now has evaluation semantics for symbolic arguments which are like Lisp-1.
• An operator called op has been introduced for writing simple lambdas with a currying syntax with numbered and implicit arguments.

What is new in version 055:

• This version introduces a new square bracket syntax that performs an intuitive operation based on the type of the first object.
• It provides Lisp-1 style calls for functions (no call operator) and array indexing and slicing for lists, vectors, strings, and hashes.
• Indexed elements and slices are assignable.
• Indexing and slicing can be used in the output template language and in string quasiliterals.

What is new in version 054:

• The main improvement in this release relates to the debugger.
• It is now possible to step into, over, or out of expressions, and see a complete environment and backtraces.
• A lazy version of "append" has been added to the embedded Lisp.

What is new in version 053:

• New functions and operators were included in the Lisp dialect for lazy list processing and parallel iteration.
• Variable substitutions in output and quasiliterals can have computed field widths.
• A showstopper bug in the evaluation of built-in global variables was fixed.

What is new in version 052:

• This version adds a new fuzz directive for imperfect matches over a range of lines, similar to the context line fuzz in the patch utility.
• The gather directive has been extended with an optional until/last clause to limit its range.
• New mod and modlast directives in repeat/rep specify alternative output when the repeat count is equal to some n modulo m.
• repeat/rep take an argument which nominates a variable which holds the repeat count.
• A gensym function has been added to TXR Lisp.
• Bugfixes have been made.

What is new in version 051:

• This version fixed several bugs, including a showstopper in bignum integers on 32-bit platforms.
• There are improvements in the areas of pseudo-random number generation and lazy list processing.

What is new in version 050:

• This version contains several critical bugfixes.
• There is some new functionality, including a new API for pseudo-random-number generation.
• More documentation has been written.

What is new in version 049:

• First cut at horizontal match functions.

What is new in version 048:

• This version fixes portability bugs in the MPI library leading to incorrect bignum results and crashes.
• It adds new functions: expt, exptmod, sqrt, numberp, evenp, oddp, abs, gcd, reduce-left, and reduce-right.

What is new in version 047:

• Bignum arithmetic is now integrated into the language.
• Overflow in word-sized (fixnum) integers transparently produces bignums.
• Functions have been added in Lisp dialect for character manipulation, vectors, and association lists.
• New special operators have been added: progn and flip.
• An implicit named block in defun bodies, and an implicit anonymous block in for loops.
• Minor bugfixes.

What is new in version 045:

• This version adds many new functions in Lisp dialect, and bugfixes.

What is new in version 042:

• TXR can now parse grammars, thanks to horizontal functions.
• A breakpoint/step debugger has been added.
• There are new ways of capturing text into variables, bugfixes (including ones affecting stability), and a Vim syntax highlighting file.

What is new in version 040:

• A small semantics change.
• An important GC bugfix affecting some platforms.
• Refactored for better performance and maintainability.
• Native (non-Cygwin) Win32 port.

What is new in version 039:

• Ported to Cygwin.
• Important bugfixes.
• A number of new features.

What is new in version 037:

• A filtering syntax for variable substitutions, user-defined filters, minor semantic tweaks, and bugfixes.

What is new in version 035:

• Fixed the gc-related bug and a regression bug in basic query logic.

What is new in version 033:

• Since version 028, there has have been a few GC fixes, a bunch of regex work. Check it out!

What is new in version 019:

• There is a new freeform directive for unstructured matching across multiple lines.
• Variables can be bound to regexes and used for matching.

What is new in version 018:

• Important bugfixes were made.
• The @(skip) directive was discovered not to be yielding the correct line past the material that was matched, allowing subsequent query forms to go over the same material.
• A regression in @(collect) was found: looping infinitely in the case that the collect body succeeds without matching anything (and yields the same match position); it now moves by one line in that situation.

What is new in version 017:

• A bug in exception subtype definition (defex) was fixed.
• A tail recursion optimization was done in the garbage collector.
• The -f option was added for specifying a query file, allowing more options to follow.
• More documentation on exceptions was written.

What is new in version 016:

• A throw directive for user exceptions and user-defined hierarchical exception types (subtype-supertype exceptions).
• Catch clauses can now have parameters, and throw passes arguments to catch.
• The -f command line option was renamed to -c.

What is new in version 014:

• Queries can now define functions.
• Functions can nest and recurse.
• Definitions have dynamic scope.
• A new "cases" directive has been added, which is like "some" but with short-circuiting semantics. This is useful for controlling function recursions.
• String literals have been added to the syntax.
• Arguments in function calls can be string literals.
• There are some small bugfixes.

What is new in version 013:

• Some minor garbage collection issues fixed.
• Infinite looping bug fixed.
• New @(trailer) directive.
• extract.y (match_files): Implemented trailer directive.
• extract.l (version): Bump to 013.
• lib.h (trailer): Declaration added.
• lib.c (trailer): External definition added.
• (obj_init): Initializer trailer with interned symbol.
• txr.1: Documented @(trailer) and bumped version to 013.