TextTest

As the name suggests, the TextTest project works via comparing plain text logged by programs with a previous 'gold standard' version of that text.

This is in contrast to most acceptance testing frameworks on offer today, which generally use some form of hand-written 'assertions' by the test writer that call into an application API.

What is new in this release:

• make killing tests on Windows when running in parallel work better

What is new in version 3.26:

• It is now possible to have separate locations for logs (shared disk) and sandbox (local disk).
• The dynamic GUI now shows previews of files that compared as equal.
• Improvements were made to the Jenkins plugin and integration with the StoryText editor for GUI testing.

What is new in version 3.24:

• This version added Jenkins integration in HTML reports, and direct linking to change and bugfix information. Alpha support was added for Condor.
• A third grid engine was provided in addition to GE and LSF (Condor works better on Windows).
• A new run_dependent_text syntax was provided for finding later matches in a file.

What is new in version 3.22:

• Search facilities have bee added to the HTML report.
• There are additional possibilities in run_dependent_text and in copying data files.
• There are six months worth of generally minor enhancements and bugfixes.

What is new in version 3.20:

• Various enhancements and bugfixes were made.
• Notably, tests can be rerun from the dynamic GUI with different settings.
• Config file settings can now be varied per-test.
• A "known bug" that will trigger a rerun of the test up to a certain number of times can be run.
• This release requires Python 2.6 and PyGTK 2.16 or later.

What is new in version 3.19:

• Various enhancements and bug fixes were made.
• GUI improvements were made with a focus on usability.
• Enhancements were made to the "known bugs" functionality.
• Sun Grid Engine is polled to make the status of tests in it more visible.
• Note this is the last release that will support Python 2.4 and 2.5: TextTest 3.20 will require Python 2.6 and probably GTK 2.18 as well.

What is new in version 3.16.1:

• Bug Fixes to 3.16:
• Sorted out radio button naming, which was strange in quite a few cases (guiplugins.py)
• No longer ignore "vanilla" flag when finding UI map files (pyusecase_interface.py)
• Fixes for older bugs:
• Don't stacktrace if dynamic GUI window is closed with tests running (controller.py)
• Slow-motion replay check box is shown after importing a new GUI-application for test (runningactions.py)
• Recording tab no longer shown after importing a non-GUI application for test (guiplugins.py)
• "Record Use-case" stays greyed out if the application under test isn't a GUI (guiplugins.py)
• Fixed bug where repeating non-exact matches for traffic replay didn't produce them in the correct order (traffic.py)
• Fixed performance issue with many doomed attempts to import non-existent GUI configuration modules (guiplugins.py)
• copy_test_path_merge will now also merge subdirectories as it should (sandbox.py)

What is new in version 3.16:

• Static GUI Enhancements:
• Can now rename files from the drop-down list in the file view (adminactions.py)
• "Refresh" replaces "Remove Tests" in the default toolbar, mostly for aesthetic reasons (default_gui-static.xml)
• Toolbar contents can be configured/put back, see "personalising the UI" on the website
• "Add Application" dialog now has a drop-down list of all possible GUI-testing options (adminactions.py)
• Allows you to get started testing a UI before understanding the config file
• Error handling on initial "Add Application" dialog improved (adminactions.py)
• Reject illegal characters in suffix and subdirectory names
• GUI-testing "UI map" files now visible from the Config tab (filetrees.py)
• Framework Enhancements:
• Now integrates with Atlassian's Jira bugtracking system as well as Bugzilla (jira.py)
• Works in broadly the same way as the bugzilla integration. See website for details.
• "collate_file" functionality for multiple files rewritten (sandbox.py)
• Patterns of the form "data*:data*.dump" should behave more intuitively, but may not be completely back-compatible.
• Read the migration notes and the website for details.
• "run_dependent_text" enhanced for filtering away sections of text (rundependent.py)
• Can now control whether start and end lines are filtered via e.g. "{[->]}" syntax
• Versioned config files can be now be placed in "extra_search_directory" locations (testmodel.py)
• Previously only imported files could be found there.
• "suppress_stderr_text" now supports the same syntax as "run_dependent_text" (default/__init__.py)
• Allows e.g. multiline filtering
• "text_diff_program_max_file_size" setting now renamed to "max_file_size" and is now a dictionary (comparefile.py)
• Keys are names of programs. Also get a warning dialog if you try to open a too-large file from the GUI.
• Now sets "USECASE_HOME" for testing GUIs with e.g. PyUseCase to "pyusecase_files" (default/__init__.py)
• Previously allowed SUT's usecase files to get mixed up with TextTest's own.
• default.CountTest script now reports a grand total at the end (default/__init__.py)
• Batch Report Enhancements:
• Now automatically generates a "dashboard" page containing links to all reports and current status (batch/__init__.py)
• Comes for free with "-coll", can also be done alone via -s batch.GenerateSummaryPage. See website.
• Can now create separate report tables containing (e.g.) performance or memory information. (testoverview.py)
• Use -coll web.performance: see website for details.
• If several versions are present on the same page, a "grand total" is shown.
• Rows or tables containing only "N/A" will no longer be shown (testoverview.py)
• If source files are empty, it points this out and suggests the disk may be full (testoverview.py)
• Cells now have tooltips indicating which test and date they are, to avoid scrolling large tables (testoverview.py)
• For the email report, SMTP authentication is now possible (batch/__init__.py)
• New config file settings "smtp_server_username" and "smtp_server_password"
• Grid Engine (LSF/SGE) enhancements:
• Grid Engine command line now appears in the "Run Info" tab of the dynamic GUI. (textinfo.py)
• Self-test/Internal changes:
• PyUseCase instrumentation removed, use PyUseCase 3.0 which doesn't need this.
• Uses UI map files (under etc) instead, and command-line PyUseCase.
• Now automatically pick up a local PyUseCase repository called "pyusecase" (texttest.py)
• Bug Fixes:
• Accelerators for "Cut Test", "Copy Test" and "Paste Test" are disabled when a text widget is focussed (adminactions.py)
• Means you can use Ctrl+X, Ctrl+V, Ctrl+C for text manipulation in TextTest windows.
• Newly created applications now work without restarting if version control is enabled (controller.py)
• Worked around a Mercurial bug renaming files with symbolic links in the name (hg.py)
• "copy_test_path:$ENV_VAR" will now do nothing if ENV_VAR is empty (sandbox.py)
• Previously set it to $TEXTTEST_SANDBOX/. which could cause problems.
• If a "collate_script" produces an empty file from non-empty input, don't write an empty file (sandbox.py)
• An "extra_version" will now read its GUI-testing config and not just assume it's the same as the parent (guiplugins.py)
• "save_filtered_file_stems" now supports file name expansions (comparefile.py)
• Scripts like default.CountTest won't send spurious mail if run with the batch flag ("-b") (default/__init__.py)
• Don't stacktrace if an environment variable in "copy_test_path" doesn't exist (sandbox.py)
• Don't stacktrace if a "collate_script" doesn't exist (sandbox.py)
• Don't stacktrace if the argument to "-cp" (or Times to Run) is invalid (default/__init__.py)
• Don't stacktrace if a relative interpreter program isn't found (default/__init__.py)
• Don't stacktrace if the same application name provided twice "-a app,app" on the command line (testmodel.py)

What is new in version 3.15:

• A fair amount has happened since 3.14 in early June.
• There is perhaps no single major improvement that stands out, but many smaller things that are likely to be useful.
• For example, it is now possible to rerun tests directly from the dynamic GUI.
• "Options" files can be used in the same way as environment files, so that they can be placed anywhere in the hierarchy and duplication of command-line options can be eliminated.

What is new in version 3.14:

• General GUI Enhancements:
• This ChangeLog now visible from the GUI (helpdialogs.py) From Help menu/Change Logs
• Locations chosen in FileChoosers now get remembered (plugins.py) So you don't need to keep searching to the same place.
• Static GUI Enhancements:
• "Create File" functionality enhanced. (default_gui.py) Will now handle directories correctly, and you can choose your source with a filechooser now.
• Can now run all the tests in a suite by selecting just that suite (default_gui.py) Also works for reconnecting and saving selections.
• Framework Enhancements:
• Can now run tests on a remote machine (UNIX only) (default.py) Even works if file system not shared. New config entries "remote_shell_program" and "remote_copy_program". See website for details.
• TextTest now collects exit codes from the system under test by default (default.py) Only writes files for non-zero status. Can be disabled via discard_file
• Can now remove certain elements from versioned config files (plugins.py) New syntax {CLEAR }
• Can now tell TextTest to merge data directories with each other (sandbox.py) Use new config setting "copy_test_path_merge". "copy_test_path" causes directories to be overwritten.
• Can now filter floating-point differences to within a certain tolerance (rundependent.py/fpdiff.py) New config file setting "floating_point_tolerance". See documentation on website. Thanks to Michael Behrisch for this. Can now collate result files from a variety of possible source patterns (sandbox.py)Essentially collate_file is now a list. May require migration in theory.
• Decoupled ignoring other errors from reporting bugs as "internal error" (knownbugs.py) Now separate check boxes for these things when doing "Enter failure information"
• Bugs marked as "internal error" get reported in preference to "known bug" (knownbugs.py) The former are usually some catastrophic general environment failure
• Config file setting "performance_use_normalised_%" renamed for generic application (default.py) Now called "use_normalised_percentage_change" to reflect possible usage for other purposes. Old name aliased.
• Traffic mechanism does not assume asynchronous file edits are possible any more (traffic.py)
• Need to explicitly list programs that might do this : key with "asynchronous" in "collect_traffic" config file setting, which is now a dictionary.
• No longer ignores missing traffic and usecase files. (default.py) "definition_file_stems" is now a dictionary, where items like these can be keyed "regenerate"
• Now set up the checkout if we can when running scripts, including website generation (default.py) Means repository location can depend on TEXTTEST_CHECKOUT.
• Picks up "properties" files in a similar way to "environment" files (testmodel.py) Used for Java applications. See website for more details.
• Default location of temporary files changed (engine.py) Now writes to ~/.texttest/tmp instead of $TEMP or ~/texttesttmp
• Version Control Browser Enhancements/bug fixes:
• Administration operations update the VCS correctly (version_control.py) Rename,Move and Remove will also do so in the VCS for tests under VCS-control
• Can now add to the VCS via TextTest (version_control.py) Separate action on appropriate menu
• Now supports Bazaar and Mercurial as well as CVS (bzr.py/hg.py/version_control.py)
• Hiding "unknown" category now works correctly (version_control.py)
• Internal changes:
• TextTest now structured into packages so lots of files have moved around
• Bug Fixes:
• Traffic mechanism does not handle directory removals by removing all files any more (traffic.py) Actually stores and replays the directory removal directly.
• No longer flag runs as missing if batch_collect_compulsory_version overlaps with versions given on command line (batch.py)
• Python 2.6 shouldn't produce annoying messages about deprecated modules any more.
• Worked around GTK 2.14 bug in file chooser handling, will no longer hang
• Race condition removed from virtual server handling on UNIX: shouldn't leak virtual servers any more (startXvfb.py)
• No longer stacktrace if "interpreter" is an invalid path (default/__init__.py)
• No longer stacktrace when repositioning tests if no write permissions to the files (default_gui.py)
• Now shows correct test names in the status bar after a rename (engine.py)
• Don't crash if copying or moving a suite and its contents (default_gui.py)
• Now works to run locally from bin directory. Thanks again to Michael Behrisch (texttest.py) "Extra versions" write to sensible batch repository locations when no version given on command line (batch.py)
• Website generation works for "Extra version" applications even if the parent is blocked or has no data (batch.py)