Notes for Machine Administrators

Unix Filesystem Backups at ITC

(this page last updated 2008/05/27)

Notes for Machine Administrators

A backup client machine is one for which ITC performs automated periodic backups over the network. The machine's administrator is involved with the backups in initially requesting backup services, providing information concerning changes on the client affecting backups, requesting restores if needed, and viewing information related to backups of the client.

Backup Requests	Getting backup service started for a machine.
Client Changes	Keeping ITC informed about changes on a client.
Restores	How to request a restore.
Reports	Viewing information about existing backups.
Backup Facts	Facts and misconceptions about ITC backups.

Backup Requests

Contact ITC at systems@Virginia.EDU to establish a contract for backups of a particular client. Once a contract exists, supply client configuration information. Root access for ITC must be granted on the client. ITC will then install and configure the appropriate backup software on the client and on the central backup scheduler (administrative) machine; this is a manual process, so backups may not begin for a week or more.

Client Changes

The daily cycle of backups for a particular client can be interrupted by various changes on the client. Examples:

The network path to the client fails.
The client crashes, is halted or powered down.
A client upgrade destroys the backup software and/or configuration.
Root access to the client ceases.
A filesystem in the backup cycle disappears or is recreated.
I/O errors occur.
The system clock on the client is incorrect.

A client upgrade which destroys the backup software and/or configuration of course requires reinstallation/reconfiguration before backups can be resumed. For some upgrades this can be done from the copy of the client configuration on the administrative machine. But often an upgrade causes the recreation of filesystems or other massive rearrangements of data which invalidates the client configuration; this situation requires the client administrator to supply client configuration information much as when the client was originally introduced to the ITC backups. The time until resumption of backups can be minimized by emailing advance notification to itc-disk-backups@Virginia.EDU.

When backups stop, email is sent to the list of contacts supplied during configuration. A response to this email is vital after the reported problem has been fixed; failure to respond can cause long delays before backups are resumed.

Some changes on a client can cause a reduction in level of service, rather than stopping backups altogether. The feasibility of backing up many filesystems daily over the network and storing the backup archives in a limited resource, depends on an incremental backup scheme and appropriate retention periods for archives. Typically, a full backup of each filesystem is done every 4 months, around the end of each semester. Incremental backups (dumping only files which have changed) are done daily, with the expectation that the volume of an incremental backup is far less than the volume of a full backup. Activity within a particular filesystem violates this assumption when a large percentage of its data suddenly changes, causing the size of each subsequent incremental backup to approach the size of a full backup. The resources consumed by the large incremental backups is then not available for the backups of other machines' filesystems. To avoid a negative impact on the backups of other machines, if inappropriately large incremental backups are detected, the level of service for the filesystem may be reduced, or portions of or the entire filesystem may be excluded from backups.

Here are some examples of filesystem activity which may call for a reduced level of service or exclusions:

A 5 GB filesystem contains a 2 GB database file which changes daily.
A 2 GB swap file is created in a filesystem.
A filesystem is full of tar files which are all recreated nightly.
A directory inside a filesystem is used for large temporary files.
In order to standardize ownership or permissions of files, or to reorganize the contents of a filesystem, a chgrp -R, chown -R, chmod -R, recursive touch or mv is done which affects every file, or most or the largest files, in a filesystem. Although these operations do not affect data within files, the inode change time of every file is modified. This forces a backup of every file, an unfortunate consequence of the fact that inode data modification time (mtime) can be explicitly modified with the utime() function, so that mtime cannot be trusted for backup decision purposes. Note that an mv of a directory does not cause problems; it is only an mv of a file that triggers an otherwise unnecessary backup.
A filesystem contains log files which grow without bound. The usual solution is to periodically rotate the log files; e.g., log.1 is removed, log.0 is renamed log.1, log is renamed log.0. Rotation does limit the amount of log data that must be backed up daily, but in a perhaps unexpected way. One might expect that only the file currently named "log" would get backed up, but on all operating systems except OSX, a rename alters inode change time, so that ALL log files get backed up daily. As long as both the number of files retained and the size of each file are small, rotation is still an adequate solution. A better solution is to use a different renaming scheme: log is renamed log.0; in the next cycle log is renamed log.1, etc. The oldest (lowest-numbered) log file is removed. Log file numbers increase indefinitely. Only the current log and perhaps one other would be backed up daily. A refinement of this scheme puts the numbered log files into a directory, rotating a set of symbolic links in the main directory to present the low-numbered log names that the system administrator is accustomed to. Another scheme is to create directories named log.0, log.1, etc, up to some maximum number. When it is time to rotate, directories are renamed, and the current log is moved into log.0. A replacement for Solaris 5.8 /usr/lib/newsyslog using this directory rotation scheme may be found here.

The approach to dealing with excessive activity varies. In exceptional cases, the activity is unavoidable and must be accomodated at the expense of the backups of other filesystems. But in most cases, if daily backups require more data path bandwidth than is reasonable, the frequency of backups may be reduced from daily to every few days. If bandwidth is adequate but total volume is too large, the retention period of archives may be reduced so as to reduce the long-term total amount of storage space required; for example, end-of-semester archives may be retained for only a year, rather than two years.

Restores

Restores are performed by ITC staff members, and backups are not retained indefinitely. For these reasons, the ITC backups are not an archiving service; i.e. they are not suitable for restores of files that are purposely removed in order to make space on a local disk. For archiving services, see ITC's HSM service.

If a user file or directory restore is needed, call the ITC Help Desk at 434-924-3731. Calling is preferable to sending email, since a tracked referral is generated for a call. However, if the Help Desk is closed, you may send email to consult@Virginia.EDU. If a restore of system files (owned by non-user accounts such as root), or an emergency restore for a damaged disk, is needed, send email to systems@Virginia.EDU. Timing is important for mailbox restore requests since mailbox backups are only retained for a relatively short while (see the Email filesystem discussion below). If an mailbox restore request is delayed for days, for some reason such as email communications problems, there is a danger that the backup containing the desired mailbox may expire and be deleted before the restore can be done. If an mailbox restore request is significantly delayed, send email to systems@Virginia.EDU to expedite the restore.