History of the Unix Filesystem Backups at ITC

     In 1973, the Academic Computing Center at the University of Virginia supplied computing needs for faculty, administration, and students with a Burroughs B5500 mainframe which had only rudimentary interactive access via two 110-baud teletype terminals. Punched cards, paper tape, a limited amount of disk, and magnetic tape at 200 and 556 BPI were the input and output media. The B5500 was superseded by successively more powerful Control Data mainframes, and a Hewlett Packard minicomputer. Hardwired interactive dumb terminals and access via modems became common. 8080-based and Z80-based microcomputers running CP/M arrived and were quickly eclipsed by IBM PCs and compatibles. Prime super-minicomputers were purchased.

     Up to this point, disk backups were completely specific to each operating system. When ACC began purchasing Suns and using ethernet to link machines together, backups were done to local cartridge tape drives using software to automatically manage a pool of tapes for each machine. The software was the same on each machine and had to be initiated on each machine. ACC acquired a 6250 BPI nine-track Sun tape drive with performance superior to the cartridge drives, making network-based backups to the single tape drive desirable. A pool of reel-to-reel tapes was set aside for each Sun, and backups by inode were done via the network.

     Machines sporting other flavors of Unix started to appear. Fortunately, they all included a version of the Berkeley dump/rdump program. Since the RS6000s came with built-in 8mm tape drives of relatively enormous (2.3 Gb) capacity, software was developed to stack many network-based backups of Suns, RS6000s, NeXTs, and SGIs onto a single tape. The software also permitted simultaneous backups using 8mm tape drives on different machines.

     A few departments and other state organizations elected to have the ITC backup software installed on their machines, and have their own staff operate the software.

     Around 1993, serious software problems were noticed with the backup system, requiring its replacement. As a result, in 1996 uvadump was written to replace dump/rdump as the heart of the backups.

     By November 2001, about 275 Unix machines (around 1000 filesystems) were being backed up nightly to a set of ten 5 Gb and three 20 Gb Mammoth 8mm tape drives, under the control of the operations staff. Many of the machines belonged to other departments at UVa which had contracted with ITC (the new name for ACC) for systems administrative and backup services. The backup management model involved an administrative entity (admin) on a dedicated machine which centrally housed information about existing backups, tapes, tape drives, machines, and configuration files used to schedule backups. Supervisor software on various non-dedicated machines contacted admin for backup work to do, provided a continuous display of backup work in progress for an operator, interacted with an operator for tape mounts and reserved tape drives using uvadmg, a separate command to interlock the use of tape drives which would otherwise have been available to any user. Each supervisor also initiated uvadump runs on dump machines (clients), directing the dump output to a particular tape drive. Each supervisor could handle one run of uvadump to one tape drive at a time. Therefore, the number of dumps run in parallel, one per filesystem, was limited to the number of concurrently running supervisors (or tape drives). Each supervisor ran on a different machine, and each was started by a separate manual login. The increasing number and size of filesystems requiring backups made it necessary to run more dumps in parallel. The model became impractical. Occasional 8mm tape failures also made a more reliable tape medium desirable.

     In 2001, ITC acquired an IBM 3584 tape robot library controlled by an IBM RS6000 H70 running AIX 5.1 and Tivoli Storage Manager (TSM) 4.2. The robot managed 80 LTO1 Ultrium tapes, with room for 281 tapes, each with a native (uncompressed) capacity of 100 Gb, and four IBM 3580 tape drives. The backup management model retained admin, but tape drive handling was completely delegated to TSM and its Hierarchical Storage Manager (HSM). Supervisors became concurrent processes that ran along with admin on the same machine. Each uvadump running on a dump machine wrote to a directory in an HSM-managed IBM 2104 Model DU3 (RAID 5) disk array. The disk array was connected to the same machine running admin and the supervisor processes. Backups became fully automated: manual tape mounting/supervisor logins/monitoring of progress, were eliminated. The number of parallel dumps in the new model was theoretically unlimited.

     In 2002, a 3584 expansion frame was added to house 423 more tapes and to add two more 3580 tape drives. In 2003, another 3584 expansion frame was added to house 440 more tapes, but no additional tape drives.

     Since the addition of a single message to an email inbox required the backup of the entire inbox, and since typical inbox size was growing, a huge amount of backup activity was generated by inboxes. ITC was forbidden to archive electronic communications. Only a few copies necessary for disaster recovery were allowed, so the total storage space occupied by inbox backups was not a problem. But the nightly backup volume began to saturate available bandwidth. The use of RAID arrays on clients and an exponential size increase in new client disks put demand on both backup storage and bandwidth. In 2004, the H70 was replaced with an IBM p615 running AIX 5.2 and TSM 5.2.2. A dual-processor Dell PE2600 running Fedora Core 2 was added to handle disk-only backups for email to two RaidWeb Arena 8600 4-TB disk arrays, and one RaidKing 827R 4-TB disk array. At that time, the robot managed 860 LTO1 tapes, and about 260 Unix machines were being backed up. The backup management model was modified once again since dump output had to be split between disk arrays on two machines. A target entity was introduced to manage disk storage for backups on a given machine, and each target contained target directories. Typically a target directory represented the major portion of a RAID array filesystem. The separation of admin and target functions onto different machines allowed complete centralization of backups; previously admin itself could only be backed up by installing a different instance of admin on another machine, which meant that logs on more than one admin had to be monitored.

     By 2006, the volume of email had made it impossible to back up email folders to tape daily. Daily email inbox backups were possible only because fibre-channel-attached storage from a NetApp R200 was annexed. Two more RaidKing 827R 4-TB disk arrays were added. In 2007, all R200 storage was released, another 827R and a high-bandwidth Copan 220T virtual tape library system hosted by an IBM p5-52a were added. The 220T's raw capacity was 56 TB; the formatted capacity was 43 TB. In 2008, a RaidKing 842RFC 12-TB disk array was added. An 827R failed but the disks were successfully moved to another 827R so no data was lost. Two RaidKing 838 2.5-TB arrays were added to offset the lost capacity.

     History teaches lessons. The following experiences with ITC backups may benefit others.

     IBM hardware was expensive but usually reliable as long as the hardware only had to deal with other IBM hardware. The 3584 tape library has been a solid performer. On the other hand, the IBM 2104 disk array was difficult to work with, hard to get replacement parts for, and after years of solid service failed horribly with the loss of all data.

     IBM hardware sometimes unexpectedly failed to interoperate with third-party hardware. A RaidKing array with a SCSI host adapter could not talk to an IBM Ultra320 adapter, but could talk to an IBM Ultra160 adapter, and had no problem with an adapter on a Dell. IBM support was no help with this, stating that the RaidKing is not supported equipment (get together with RaidKing engineers to work out the problem? Not in IBM's universe). Later on a different IBM server with a different version of Ultra320 adapter, no problem talking to the array.

     IBM's interoperability trouble was not limited to SCSI. A RaidKing 842RFC array could not be made to work with an IBM FC HBA. No problem with a Qlogic or Emulex HBA on a Dell.

     On the positive side, onsite support from the local IBM hardware engineers has been consistently outstanding: prompt response, showing up on time, diagnosing problems intelligently, ordering parts without hassle, communicating appropriately, and staying with a problem until it is fixed. These engineers have become friends as well as professional help. It would be hard to imagine a better situation.

     Cables are important. The SCSI cables that came with two RaidWeb arrays worked well for about a year, after which mysterious I/O errors began to appear. These were finally resolved by replacing the cables.

     Tape quality matters. Imation LTO1 tapes gave us many hardware problems until we switched to Fuji. We then had an initial problem with a large number of Fuji tapes which was resolved by the installation of IBM-supplied metal "clips" into the 3580 tape drives. Smooth sailing ever since.

     Our best hardware experience has been with Dell computers. Our PE2600 server was not expensive, ran solidly as long as we had it, and was only replaced because we wanted a physically smaller server. A GX400 server used to monitor other hardware was a leftover from another project and yet continues to run flawlessly.

     Our worst experience was with an expensive 220T VTL from Copan Systems. To be fair, the VTL itself worked as advertised, although it was poorly documented and cantankerous to deal with. However, we were not interested in a VTL; in fact we would have preferred to not have a VTL. The part of the purchase that was overridingly important to us was a quoted free installation of the deduplication feature when it became available. Only deduplication would be an adequate solution to our exploding email backup volume, and a survey of other vendors made Copan's 220T with deduplication the only reasonable choice in mid-2006. The feature became available in 10/2007, but Copan chose not to implement it on the 220T, in spite of a year and a half's worth of post-purchase assurances to the contrary by sales staff, and work with their technical support to prepare for the installation, while our backup service for email had to be drastically reduced for lack of capacity. We were eventually left with the option for a return and partial refund, with which we hope to buy RaidKing storage to expand service again. It is unlikely that we will ever deal with Copan Systems again. Our experiences with various sales personnel have made it virtually certain that we will believe little of what they say.