We have been running backups to a Data Domain VTL on one site, replicating the virtual tapes to another site, importing the virtual tapes into the Cell Server on that site, and then using those replicated tapes to copy to physical tapes for archive storage. This has been working very well for several months, until about February of 2013 when it stopped working. No changes to the patches were made, or any other Data Protector configuration that we can think of. We are using DP 6.2 on a Windows 2008 R2 cell manager. We have the current patches installed, except for the CORE and the CS, which look like they just recently came out with some new ones. We have been working fine in this current configuration, until it just stopped working. When we run the copy of the session, we get the following error:
[Normal] From: BMA@host.domain.com"LTO_Drive_7" Time: 6/18/2013 2:15:56 PM STARTING Media Agent "LTO_Drive_7"
[Normal] From: BMA@host.domain.com"LTO_Drive_7" Time: 6/18/2013 2:16:01 PM By: UMA@host.domain.com@scsi3:0:0:0 Loading medium from slot 507 to device Tape32:0:0:26C
[Normal] From: RMA@host.domain.com"VTL_D25" Time: 6/18/2013 2:16:07 PM STARTING Media Agent "VTL_D25"
[Normal] From: RMA@host.domain.com"VTL_D25" Time: 6/18/2013 2:16:12 PM By: UMA@host.domain.com@scsi3:0:0:0 Loading medium from slot 118 to device Tape31:0:0:25C
[Normal] From: CSM@host.domain.com"Interactive" Time: 6/18/2013 2:16:26 PM STARTING copy of WinFS remote_host.domain.com:/C "remote_host.domain.com [/C]" from session 2013/03/01-2.
[Critical] From: BMA-NET@host.domain.com"LTO_Drive_7" Time: 6/18/2013 2:16:28 PM Invalid block sequence number: 320826 expected 0.
[Critical] From: CSM@host.domain.com"Interactive" Time: 6/18/2013 2:16:28 PM FAILED copy of WinFS remote_host.domain.com:/C "remote_host.domain.com [/C]" from session 2013/03/01-2.
From there, it fails that one object, and then continues to copy the next object. It goes until it reaches the end of that virtual tape, and then fails with some error about not any more media to copy or something. That part isn't what I'm concerned with as much as the invalid block sequence number issue. I've looked at several other posts on the forum which suggest things from the OB2SHMIPC=0 in the omnirc file to making sure that the source and target drives have the same block size, to updating patches, to getting SSP's from HP, etc, etc. The omnirc file already had the OB2SHMIPC=0 in it, we changed the block size to the same size (64K), and all patches (except for the most recent CORE and CS) were already installed. We can copy other VTL tapes to physical tape, but for some reason the ones that were replicated from the remote host and imported fail. Here is the wierd thing, the error complains about the block sequence on the TARGET drive, not the SOURCE drive! The target tape is a blank formatted tape with nothing on it. So how can it expect 0, but get some other value?