DIASER

DIASER (Distributed Internet Archive System for Educational Repositories), based on the DIAP technology, can create back-ups in multiple locations, thus improving the environment stability and security.

System Requirements

- Perl v5.10.0 enabled workstation.

- 3 x Linux storage nodes (can use VM's) with root access for initial setup and Perl v5.10.0, with services; sshd, crontab, iptables ssh port open, ntpd,  rsync (non daemon).

- LAN or WAN connection between each server, the 3 machines must be able to ping one another.

- SSH server on each node, not necessarily port 22.

- Perl modules
Workstation: Net::SSH::Perl, Net::SFTP, Getopt::Long and Term::ReadKey
Nodes: Net::SSH::Perl i.e. as root ]#yum -y install perl-Net-SSH-Perl or cpan>install Net::SSH::Perl

- Enough disk space, see docs/worksheet_3_26_may.pdf or docs/overview.png. I'll make all this much simpler to calculate when I have finished subroutine calculate_lmb.

Configuration

The file config.pl contains user settings, please edit as required.

Main settings:

number of months to years you want the system to run for
start date
time of daily operation
node IP addresses
SSH ports
dry run or not
time out and maximum transfer rate

Start Installation

Run diaser.pl -i as an ordinary user, node root passwords will be requested and make a note of your chosen diaser password.

Other options can be viewed with the diaser.pl -h switch.

Upgrades

Product upgrades will apply fixes and updates, use the -u switch.

Data Entry point

This machine contains directory ad0 and aFULL01 and is your machine A. Machine A is assumed to be your most robust and closest from a network perspective to  your main backup facility. The files are assumed to be read/write by user id  diaser for the moment. Use aFull01 for your FULL copy and ad0 for all subsequent  Differentials. The script fill_diaser.sh will automate the loading of DIASER, fill_diaser will be re-written in Perl.

Data Retrieval

Currently retrieval can be achieved by manually using scp between the nodes and using rsync logs. All this will be automated.

SE Linux

May cause installation problems, please disable while installing.

What is new in this release:

• This release contains a large number of code changes made according to Perl Best Practices and Perl::Critic advisories.
• Level 5 and 4 are complete for the cli management tool as well as most level 3 changes.
• Some development structures have been added to the SCM - these files contain development environment variables, settings and aliases used to govern and enhance the coding environment.
• Http conf.d has been modified to avoid duplicate aliases.

What is new in version 1.0.8:

• The Web GUI front end configuration tool contains all the variables a user needs to set and displays them clearly in four sections with pop-up tool tips. With this GUI, the user can select options and build a working diaser configuration file. The install script handler is in place. This front-end can be installed using "make install".

What is new in version 1.0.7:

• This development version includes more diaser-webgui tool design enhancements and feature implementation code, including action defined popup windows with a close button for all user input scenarios, a Makefile, and an apache system conf file.
• The tool diaser configuration file storage location is now pre-defined.
• The tool has been renamed to diaser-webgui in SVN and the Makefile.

What is new in version 1.0.6 Beta 2:

• The GUI web tool has been restructured and a Java script date picker
• introduced.
• Configuration file upload and save in place.

What is new in version 0.2.5 Beta 1:

• A node migrate facility is now functional, allowing users to move an entire node and its data to a different server.
• Added to this release are high level graphics and end user documentation.
• Exception handling is done when modules are loaded.
• The Perl version is displayed when using the --version option.
• An encrypted volume recipe is included in the technical documentation.
• Subroutine and its option --stats display disk usage for all nodes.
• New functionality has been documented.

What is new in version 0.2.4 Beta 1:

• There is now a tool that will allow the user to test the real, not theoretical, potential data throughput between two nodes. This will allow determination of the lowest maximum bandwidth between nodes. --bandwidth will generate a graph displaying the highest throughput possible.
• This release has an outline and partial implementation of sub migrate(), which will allow DIASER users to move a node from one machine or VM/VPS to another.
• There have been minor bugfixes and typo corrections, and a fix to the fill DIASER algorithm.
• The ROADMAP has also been adjusted.
• The documentation has been updated to include the new features.

What is new in version 0.2.3 Beta 1:

• This release contains a number of bug-fixes, refinements to the walk through --configure tool and a switch from using md5 to sha1 check-summing of volumes as they enter DIASER as well a corresponding documentation updates.

What is new in version 0.2.2 Beta 1:

• simple bandwidth table added to documentation
• retrieve subroutine documented
• simple retrieve data sub implemented -emergency lock subroutine implemented and documented
• fill_diaser.pl re-write documentation updated
• fill_diaser.pl or gen_fill re-written
• md5sum and date file created with each volume added to -node A
• fill_diaser.pl partially re-written
• collect.sh removed from SVN
• tab_a.pl adjusted and written to to accommodate new
• diaser fill mechanism
• variables below also made available via
• modify and write_config
• section 2.5 Why Linux added to manual.html
• five new variables for revised diaser_fill.pl added to diaser.pl config->define, uninitialised vars, GetOptions sub configure
• refined flow chart diaser_fill_flow dia and png
• placed volume md5 checksum operation in fill_diaser.pl algorithm

What is new in version 0.2.1 Alpha:

• adjusted sub gen_hvautoc for changes sub gen_dirs
• adjusted sub gen_fill for changes sub gen_dirs
• adjusted sub gen_dirs so simple fail-over and node role change can be implemented
• removal of a b c prefix and additional d0 and Full01 and Full02 directories
• input validation applied to modify options
• subroutine configure() implemented with input validation
• wrote input validation subs is_integer() is_string() is_directory() is_ip() is_integer_inc_zero()
• manual/INSTALL updates reflecting recent changes
• website links
• Manual changed to Documentation and re-ordered

Requirements:

• SSH server
• Net-SFTP
• NTP
• iptables
• rsync