iRODS

iRODS is a new data grid application system developed by the San Diego Supercomputer Center (SDSC) Storage Resource Broker (SRB) team and collaborators. The system is based upon the expertise gained through the application of the SRB technology in support of data grids, digital libraries, persistent archives, and real-time data systems.

The management policies (sets of assertions that these communities make about their collections) are being characterized in terms of rules and state information. At the iRODS core, a Rule Engine interprets rules to decide how the system is to respond to various requests and conditions. iRODS is open source (under a BSD-type license).

We are supporting SDSC SRB while developing iRODS in parallel, and will do so for quite some time to come. At some point, we will provide tools to migrate from SDSC SRB to iRODS for those who wish to do so.

What is new in this release:

• Major New Features:
• New Rule Engine. The iRODS Rule Engine has been completely re-written. It supports the old rule syntax as well as a new rule language which is more expressive and easier to work with. The new rule language is parsed directly by the rule engine, eliminating the need to run the rulegen program. Features include strong typing of parameters, support for integer and double type parameters, indexing of the rules for faster execution. Examples are provided for use of each micro-service in the new rule syntax. System rules are now defined in the core.re file instead of the core.irb and can also be stored in the iCAT (see Rules in the iCAT below). See the rule engine document and the backward compatibility section of that for more information. The old rule engine can still be configured-in (enabled), if needed, via settings in config.mk.
• Using Boost - Improved Windows Support Soon. The Boost library, which is included in the release, can now be optionally utilized on Linux/Unix hosts and will be used on Windows. To build with it run 'buildboost.sh' and then update config.mk to have USE_BOOST set and run make. By using Boost, the Windows version will be immediately available, unlike past versions, where iRODS developers needed to port changes into the Windows environment and Windows releases would lag. Related features also under development at RENCI are the ability to install on Windows via irodssetup and support for an iCAT-Enabled Server on Windows. All of these Windows features are scheduled for the release following 3.0. Thanks to the iRODS@RENCI team for the primary development of these features (in collaboration with DICE).
• Building with C++ In support of the above feature, on most platforms, iRODS is now compiled with the C++ compiler, g++. This is largely invisible but did require minor source changes in most of the iRODS .c files. Most of the iRODS code has been tested, via the various configuration options, but contact the iRODS team if some aspect fails to build.
• Rules in the iCAT. Rules can now be registered into the iCAT catalog for central administration. When utilized, all changes to rules are stored as rule versions. The rules can be read from the iCAT into an in-memory rule structure. Rules can be written to a file from the in-memory rule structure. The rules can then be distributed to each local rule base to ensure a uniform rule environment. This mode is controlled by a set of micro-services as described in the Rule Base in iCAT page. By default, system rules are managed in a manner similar to previous versions, being read at start up from a file (core.re now instead of core.irb).
• OS-level authentication. The purpose of OS authentication is to allow iRODS to authorize users who have already logged into an iRODS client system using some form of OS-configured authentication and authorization scheme (e.g. logged in using PAM, or via ssh authentication). See OS authentication for more information. Thanks to Chris Smith of Distributed Bio for this extension.
• Rule debugger. The xMessage system has been substantially extended and now supports debugging of rules. It is possible to single step through each action, listing the rules and micro-services that are invoked in the distributed environment. See the 'idbug' i-command.
• Realizable Objects. Queries on information resources can be registered into iRODS collections. Clicking on the realizable object instantiates the request, which is stored as a replica of the realizable object. Realizable objects can be used as soft links that integrate remote resources into an iRODS collection. Drivers have been written for accessing Z39.50, web pages, anonymous ftp sites, public data in SRB data grids, and public data in an iRODS data grid. This makes it possible to build a collection that spans unfederated iRODS and SRB data grids as well as other data management systems. These are a module, msoDrivers, which, when enabled, creates a set of micro-services.
• Other Improvements:
• Run server as root mode. The "run server as root" operational mode allows one to run the iRODS servers with root privilege. This mode can help enable some behavior and features that are not possible when running iRODS as a normal, unprivileged user, although in most situations the non-root mode is still preferred and recommended. See Run server as root. Thanks to Chris Smith of Distributed Bio for this.
• Large file restart. The new iput and iget --lfrestart option (specifying a restart info file) allows the transfer to continue where it left off if interrupted. This can be used with -X to be part of an overall directory uploads restart. Files larger than 32 Mbytes can be restarted.
• --retries option for iput and iget to specify the number of times to retry. This can be used in conjunction with -X option to automatically restart the operation in case of failure.
• itrim --age and --dryrun. There is now an option to trim only files older than a specified age and also an option to display what would be done without actually trimming the files.
• Tracking and deleting orphan files - Orphan files created by deleting data objects when the resource of the physical files was down are now tracked and can be deleted with a new --orphan option of irmtrash.
• Auditing Extensions. SQL extensions are now available for enhanced auditing. A SQL script is included which can be configured and run by 'irodssetup' (or run manually) which adds tables, triggers, etc. See the server/icat/auditingExtensions directory for more information. Thanks to the NASA Center for Climate Simulation (NCCS) for this.
• Bug Fixes (partial list) and Additional Improvements
• Server infinite loop fixed. A problem was fixed where the child of the irodsServer could get into an infinite loop and use up large amount of CPU time due to a request queue problem. This was patch irods_2.5_patch_1.
• An irsync irods to irods fix. Fix was added to resolve problem in irsync where iRODS to iRODS (i:x i:y) sync did not work. This was patch irods_2.5_patch_2.
• Fix for iRODS FUSE (irodsFs) occasional hang - A fix was added to address a occasional hang caused by the overflow of connection requests by adding a wait queue. Also added the handling of socket timeout by reconnecting.
• Support for newer ODBC. A change was needed for newer versions of ODBC, in particular on Ubuntu 11.04 64-bit hosts, using MySQL. This was patch irods_2.5_patch_3.
• $userNameClient is now available for the acAclPolicy rule
• A rsFileRmdir problem on Solaris was fixed.
• Fixed bug 133 - "iput command crashes if progress report is requested" using fix given by Gene Soudlenkov.
• A cross-zone error message (authentication) is now properly returned to the client.
• A chmod problem (octal not integer) in the univMSSDriver.c was fixed.
• iqmod can now accept dates in the YYYY-MM-DD.hh:mm:ss format.
• Longer strings are now allowed for the 'iadmin' 'asq' (add specific query)
• A possible division by zero is avoided in irodsGuiProgressCallbak.
• Replace the transStat_t with transferStat_t to align 64 bit integer to 64 bit address boundary
• iqstat optional rule indicator is now a RuleID (as in iqdel and iqmod) instead of a rule name.
• Add a "fork and exec" mode to the irodsReServer since the Windows platform does not support "fork" only operation.
• ireg now has an option to calculate a checksum (thanks to Chris Smith for this).
• The irodsServer now checks and uses spLogLevel environment variable like the agent does.
• iscan was fixed to handle larger collections, avoiding 'too many concurrent statements' by closing additional queries.
• The irodsctl script system now sets the LD_LIBRARY_PATH in the GLOBUS_LOCATION (in irods.config).
• The irepl help now includes comments about using irsync for cross-zone operations.
• The msiExecGenQuery and msiExecStrCondQuery micro-services now allow "No Rows Found" as a valid response.
• ichmod on remote-zone users, "username#zonename", is now possible, via an extension to the msiSetAcl. Thanks to Jean-Yves Nief of IN2P3 for this.
• Changed logic to only call acPostProcForCollCreate, acPostProcForPut, and acPostProc, if the associated operation succeeded.
• The "rename" function has been added to the universal MSS driver. Thanks to Jean-Yves Nief of IN2P3 for this.
• The general-query and iquest and other clients can now handle a user-provided 'IN' or 'BETWEEN' condition on a general-query call, for example: iquest "SELECT RESC_NAME WHERE RESC_CLASS_NAME IN ('bundle','archive') and iquest "SELECT DATA_NAME WHERE DATA_ID BETWEEN '10000' '10020'".
• A timestamp has been added to the iput/iget -P option.
• iqdel now has a -a (all) and -u (user) option to allow deletion of all jobs and jobs belonging to a user.
• Memory leaks associated with applyRuleForPostProcForWrite and applyRuleForPostProcForRead have been fixed.
• Queries (iquest, etc) with mixed 'AND' and 'and' conditions are now handled correctly.
• A problem writing database object execution results objects (DBORs) into iRODS has been fixed.
• An ICAT database index on (data_type_name) was added so that 'idbo ls' will continue to perform well at scale.
• A problem was fixed that had prevented msiDataObjPutWithOptions from overwriting existing copies.
• The temporary password mechanism was modified for use by iDrop and iDrop-lite.
• irodsctl now checks that the server/log directory exists and is writable and gives specific error messages for these cases.
• Support was added for orphan files, including a irmtrash --orphan option.
• The -G option for registering replicas is now supported.
• A change was made to log the correct process-id in server log files in all cases. Previously, there were rare situations where the pid could be incorrect.
• Limits were increased so that very long strings can be piped into 'imeta'.
• Kerberos libraries will now be linked into the FUSE client if KRB_AUTH set. Thanks to Chris Smith for this.
• A fix was made to prevent a core dump in scanUtil if the local source path doesn't have '/'. Thanks to Chris Smith for this.
• A new client/server call (API), rcDataObjFsync was added. Thanks to John Knutson of the University of Texas for this.
• A bug was fixed avoiding an unneeded redundant internal query, slightly improving performance in some cases.
• A problem in moving ('imv') certain collections was fixed. If a collection contained a data-object that was not writable by the owner of the collection, the physical move would fail. Now, this is handled as a special case and allowed if the user is the owner of the collection.
• A fix to avoid getting the SYS_COPY_LEN_ERR error by iget and icp when the data content has been modified by some micro-services.
• A 'notify.pl' script was developed which can be run periodically to check specified (configured) collections and send email notifications when data-objects are added or removed. See the scripts/notify directory for more information.
• The 'irsync' command can now accept a blank i: argument. The command 'irsync fileName i:' now stores the file (if needed) into the current iRODS collection.
• Additional Items:
• The copyright and license text files have been updated to include the University of North Carolina at Chapel Hill and the Data Intensive Cyberinfrastructure Foundation in addition to the Regents of the University of California. Of course, the iRODS license continues to be a fully Open Source (BSD style).

What is new in version 2.0:

• Federation. Zone Federation has been added which provides facilities for two or more independent iRods systems to interact with each other and allow for seamless access of data and metadata across these iRods systems. These systems are called iRODS Zones, with each Zone running its own iCat and administrative domain. iRODS Federation requires less synchronization between zones than the similar SRB Federation system. See Federation and Federation Administration for more information.
• Master/Slave iCat with Oracle. An iRods installation or Zone can be configured to run with a single Master iCat plus zero or more Slave iCats. The purpose of the Slave iCat is to improve responsiveness of queries across a Wide-Area-Network. The Slave iCats are used for "read only" type queries. The following icommands have been converted to use the Slave iCat by default: icd, iget, ils, ilsresc and iqstat. This makes use of RDBMS functionality to sychronize the ICAT databases and so is available when using Oracle as the ICAT RDBMS.
• Initial SRB to iRODS Migration tool. There is now a preliminary version of a tool to help convert an SRB Instance to an iRods one. The 'm2icat.pl' script uses Spullmeta to get SRB-MCAT information and creates and executes sets of commands for iadmin, psql, and imeta to create resources, collections, dataObjects, and users in the iRods instance. The iRODS system can then access former-SRB data without moving the physical files. This is still incomplete and cannot handle many of the features of SRB, but you may wish to experiment with it. See the script for more information and contact us to help us plan additional extensions.
• Grid Security Infrastructure (GSI) Improvements. A significant problem in the iRODS interface to GSI was corrected, allowing regular iRODS users to authenctiate with GSI. Users can also now set the environment variable SERVER_DN to authenticate the server via the GSI system (perform mutual authentication).
• iRods FUSE improvements. iRods FUSE now works with the latest versions of FUSE instead of only version 2.7.0. Caching files and directories query results to improve the performance of the getattri call which is call frequently by FUSE. Small files are cached to improve the I/O performance on small files. File modes of files has been enabled so that chmod of files now works. You should see noticeable performance improvement in commands such as ls or cp of small files.
• iRODS Explorer for Windows. The new iRODS Explorer for Windows has been available since mid-September. As described on the windows page, this is an iRODS browser that runs as a native Windows binary and provides a rich Graphical User Interface and a fast navigation of the hierarchical collection-file structure inside iRODS. In addition, users can add, modify, and view metadata with long string values through a user-friendly metadata dialog.
• DataMode preserved. A "dataMode" metadata item was added so that the file mode of files uploaded to iRods and downloaded from iRods, can be preserved. "dataMode" is the Read-Write-Execute status for user, group, other, of a file.
• New bundling. A new 'ibun' command is used to handle the bundling of small files into structured files such as tar files. It can be used for the uploading, downloading, and archival of a large number of small files. For example, to upload a large number of small files, a user can use the normal UNIX tar command to tar these files into a single tar file. This single tar file can then be uploaded to iRods using the iput command. The "ibun -x" command can be used to request the iRods server to untar this file into many small files and register these small files with the iCat. Similarly the "ibun -c" command can be used to efficiently download a large number of small files.
• New 'ipasswd' Command. 'ipasswd' allows users to change their iRODS password. As with the corresponding iadmin command, the password is obfuscated for network transfer.
• Rule-oriented Data Access (RDA) ported to Oracle. RDA is now supported on Oracle as well as the previous PostgreSQL. RDA provides access to arbitrary databases through the iRODS system, somewhat like the SRB DAI (Database Access Interface) but implemented via rules and micro-services.
• Other RDA improvements. A msiRdaRollback micro-service was implemented; some memory leaks were fixed; and an obfuscated form of the RDA password can be set in the RDA configuration file.
• Rule-language 'break'. A 'break' statement is now accepted in the iRODS Rule language to break out of for, while, and foreach loops.
• Federation User and Administrative Changes. In the iadmin and imeta commands, users are now represented with an optional Zone name (user[#zone]), where the local Zone is default. iadmin has mkzone, modzone, and rmzone subcommands to manipulate remote-zone information. ilsresc, imeta, and irmtrash now have '-z zoneName' options to work with remote Zones. Internally, user authentication uses an optional Zone name.
• Federation Server Authentication. A capability was added where the iRODS Server which is authenticating a Client for a remote Zone, is itself authenticated. This is optional, but highly recommended for Federated Zones to improve security. See -- for additional information.
• Zone Renaming. The iadmin tool can now be used to rename your local Zone, handling the conversion of the Zone and user information and renaming user home collections (via a new Rule and micro-services). This may be useful with iRODS Zone Federation.
• Timed connection -T option. A new -T option was added to the iput, iget, irepl and icp commands which renews the socket connection between the client and server after 10 minutes of connection. This gets around the problem of sockets getting timed out by the firewall as reported by some users.
• New RBUDP data transfer mode. A new data transfer mode - RBUDP (Reliable Blast UDP) was added, in addition to the existing the sequential (single TCP stream) and parallel (multi TCP streams) modes currently supported by iRODS. RBUDP is developed by Eric He, Jason Leigh, Oliver Yu and Thomas Defanti of U of Ill at Chicago. http://www.evl.uic.edu/cavern/RBUDP/Reliable Blast UDP.html It uses the UDP protocol for high performance data transfer. A new option -Q has been added to the iput, iget,irepl and icp command to specify the use of the RBUDP protocol.
• HDF5/iRods Improvements. The HDF5/iRods client can now be built without linking to the HDF5 library. A JNI capability for JAVA client such a hdfView, was added. A memory leak was fixed.
• Inherited Access Permissions. An inheritance attribute can be set on a collection to cause new data-Objects and sub-collections created under it to acquire the access rights (ACLs) of the collection. See 'ichmod -h' and 'ils -A' for more information.
• ICAT Improvements. The iRODS Metadata Catalog interface software (ICAT) was improved in various ways, in addition to changes for iRODS Zone Federation. Access to the Audit tables via queries is now restricted by default. A bug was fixed dealing with recursively setting access control on replicated data-objects. Problems in getting the totalRowCount (when requested) were resolved (for both Oracle and Postgres). In the General-Query, any number of compound conditions (separated by || or &&) can now be handled (instead of just two). After various errors, the ICAT functions (when using PostgreSQL) will do an automatic rollback to allow subsequent SQL to function. Some ICAT-Oracle memory leaks were found and fixed. imeta and the ICAT AVU queries can now accept multiple conditions separated by 'and' and a single 'or'. 'isysmeta' can now set the data-type of a data-object. For improved long-term maintenance, internal changes were made in the way that the ICAT general queries are structured.
• Additional Micro-services. A number of new micro-services have been added to enable new functionality, including some of the features described in these release notes. These can also be used in your own rules and as examples for developing your own micro-services. See Released Micro Services for the current list. Special thanks to Romain Guinot of the Open Source Center - Atos Origin (http://www.portaildulibre.fr) for providing some of the new micro-services, both for the core and as a separate module ('guinot').
• Testing Improvements. As before, the IRODS development version is continually built and tested on local DICE hosts and occasionally on various platforms at the NMI Build & Test facility. ICAT tests were expanded to cover new ICAT functionality. Some heavy-load tests were developed.
• Installation/Control Improvements. A variety of changes were made to the iRODS installation and control scripts to handle specific error situations; also, finishSetup.pl will now update ~/.odbc.ini to include the [PostgreSQL] section that is also stored in the .../pgsql/etc/odbc.ini file, to avoid problems on some hosts, etc. Also, the Make scripts will now automatically re-link modules that have updated source files. And 'iinit' will create the ~/.irods directory if it is not present and prompt for and store the needed .irodsEnv items if they are missing. The install scripts now default to Postgres 8.3.5.
• New Transfer Logging. When enabled (manually), the transfer operations of get, put, replicate, and rsync are logged. See the comments in rsDataObjClose.c for more information.
• imkdir can create parent collection. A -p option was added to the imkdir command to cause it to create parent collections if they don't already exist.
• Other Bug Fixes. As always, other bugs have been fixed and many small improvements made; too numerous to describe.

What is new in version 1.1:

• Grid Security Infrastructure (GSI). GSI is now supported as an additional optional authentication method. When clients and servers are built GSI-enabled, users can choose to authenticate via their GSI X.509 certificates. GSI is enabled by answering a few questions in the irodssetup script.
• Electronic Records Archives (ERA) Module Extensions. The ERA module contains new collections management micro-services for manipulating objects, user accounts, access rights and metadata. Several micro-services have also been added to retrieve audit trail information from the iCAT. An XML module has been created and contains a micro-service that performs XSLT transformations on iRODS objects. Several miscellaneous functions have been added to the core set of micro-services (to print data structures, manipulate time and apply a Dublin Core metadata template).
• A rich web client: iRODS Browser (Beta). Introducing iRODS Browser (Beta), a user-friendly web application for iRODS users to access and manage iRODS collections stored on any iRODS server, using a standard web browser. iRODS Browser is hosted here: [1]. No installation is necessary for end-users, unless you wish to host your own iRODS Browser, which requires extensive knowledge of web server setup/configuration. You can get more information about iRODS Browser on this page: iRODS_Browser.
• Mounted Structured Files. This is similar to mounting a UNIX file directory to a collection implemented in iRods 1.0. In this case, a structure file (with internal structure containing files and subdirectories) such as a tar file is mounted instead of a file directory. Once the tar file is mounted, a user can use iCommands to access the files and subdirectories contained in the tar file. The implementation includes:
• Creating a framework for mounting multiple types of structured files. To implement a new type, 17 I/O functions (open, read, write, close, etc) specific to the structured file need to be provided to the driver.
• Currently we have implemented one type of structured file - the tar file.
• imcoll - A new command for managing (mount, unmount, sync, etc.) the structured files. In addition, the mounting and unmounting of UNIX file directories has been moved from the ireg command to the new imcoll command.
• More than 20 new APIs (client/server calls) have been created to support this implementation.
• iRods HDF5 Integration. HDF5, a general-purpose library and file format for storing scientific data, has been integrated into the iRods framework in the form of micro-services. Five HDF5 microservices: msiH5File_open, msiH5File_close, msiH5Dataset_read, msiH5Dataset_read_attribute and msiH5Group_read_attribute have been implemented on the server. HDF5 files can now be stored in iRods and users can use iRods client functions to make HDF5 specific calls to access HDF5 files stored in iRods. The client implementation also includes a JNI interface that allows the HFD5 Java browser HDF5View to access HDF5 files stored in iRods.
• Java Client API: JARGON. The Java Client API for the datagrid now officially includes iRODS. It supports most of current iRODS functionality, including iCAT queries, file I/O, metadata manipulation, and basic rule executions. A quick starter guide plus a full API documentation is hosted here: Jargon documentation. You can download Jargon from here: extrods project download page.
• Web Services Available as Micro-Services. One can now create iRODS micro-services that call Web Services by wrapping the web service interactions using a gsoap envelope and writing micro-service interfaces for input and output arguments for the web services. A description of how to write those interfaces and how to access web services as micro-services is available at: Web_Services_As_Micro_Services.
• RuleGen Rule Language. Rulegen is a parser that takes rules written in a nicer language to the cryptic one needed by irule and core.irb. The input files for the rulgen are recommended to be *.r (.r extensions) and the output created by the rulegen is in the form of *.ir (.ir extensions). The rulegen parser is found in icommands/bin with its source files in icommands/rulegen. There is a note called HELP.rulegen that explains how to make and use the parser. The note also contains the grammar of the rulegen language. HELP.rulegen can also be found here.
• PHP Client API: prods. Introducing prods, a PHP Client API for iRODS. It supports most of current iRODS functionality, including iCAT queries, file I/O, metadata manipulation, basic rule executions. A quick starter guide plus a full API documentation is hosted here: prods documentation. You can download prods from here: extrods project download page.
• Preliminary ICAT Auditing. A preliminary version of ICAT Auditing has been developed. When enabled, significant events (at the ICAT level) are recorded into an ICAT audit table. This is still under development.
• Preliminary Rule-oriented Database Access (RDA). There is now a preliminary version of Rule-oriented Database Access (RDA). This is generally similar to SRB-MCAT DAI as it provides access to arbitrary databases through the iRODS system, but it is based on the use of Rules and Microservices to achieve this. There are significant restrictions with this version, however, and RDA will be substantially improved in later releases.
• Windows i-commands. The pre-built binaries are available for download. Users can also build their own Windows i-command binaries from iRODS source distribution.