Just recently our DPM 2012 R2 server seems to be frequently failing to backup one of Exchange 2013 mailbox databases. The error logged against the failed job is:
Data consistency verification check failed for Staff Mailboxes.edb of Exchange Mailbox Database Staff Mailboxes on RC-EX2013-01.riddlesdown.local. (ID 30146 Details: Unknown error (0xfffffb4a) (0xFFFFFB4A))
In the DPM logs I can see the following:
100C119C
05/1012:25:18.31527BackupMachine.cs(2159)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNINGBackupMachine : FAILURE - BACKUP, errorCode=PrmVerifyDatasetFailure
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNINGTask Diagnostic Information - <?xml version="1.0" encoding="utf-16"?>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING<TaskExecutionContext>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <PrmWriterId>ddd34536-63d2-4f6c-98c1-2a4ad30d1ee4</PrmWriterId>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <PrmDatasourceId>04b6a489-e717-4c12-a535-072d2834a69b</PrmDatasourceId>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <PrmActiveNodeName>RC-EX2013-01.riddlesdown.local</PrmActiveNodeName>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <PrmLogicalReplicaId>4ad331b6-8510-43b4-93c3-f23be183b365</PrmLogicalReplicaId>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <PrmDatasetId>cfe8316d-f799-475b-b113-8aebe8a150da</PrmDatasetId>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <PrmPhysicalReplicaId>81d70066-bade-4a89-8a5c-d72133c901ad</PrmPhysicalReplicaId>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <PrmReplicaValidity>Valid</PrmReplicaValidity>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <PrmReplicaStatus>Idle</PrmReplicaStatus>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <PrmOwnerLockId>00000000-0000-0000-0000-000000000000</PrmOwnerLockId>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <TEVerb>DeltaReplicate</TEVerb>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <TEErrorState>Backup.OnAbort(1)RAForRead.RADeleteWorkItemPending</TEErrorState>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <TEErrorDetails><?xml version="1.0" encoding="utf-16"?>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING<q1:ErrorInfo ErrorCode="30146" DetailedCode="-1206" DetailedSource="2" ExceptionDetails="" xmlns:q1="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd">
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <q1:Parameter Name="protectedgroup" Value="Exchange 2013 Mailbox Databases" />
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <q1:Parameter Name="servername" Value="RC-EX2013-01.riddlesdown.local" />
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <q1:Parameter Name="agenttargetserver" Value="Backup1.riddlesdown.local" />
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <q1:Parameter Name="datasourceid" Value="04b6a489-e717-4c12-a535-072d2834a69b" />
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <q1:Parameter Name="replicaid" Value="4ad331b6-8510-43b4-93c3-f23be183b365" />
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <q1:Parameter Name="datasourcename" Value="Staff Mailboxes" />
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING <q1:Parameter Name="objectname" Value="Staff Mailboxes.edb" />
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING</q1:ErrorInfo></TEErrorDetails>
100C119C
05/1012:25:18.31501TaskInstance.cs(798)C70F292B-8751-474F-AE8F-34D49C8EA940
WARNING</TaskExecutionContext>
and
0EB82470 05/1012:25:15.34331verifydatasetsubtask.cpp(267)[0000000002C5C500]NORMALHr: = [0xfffffb4a] CVerifyDatasetSubTask::CancelFailureCleanUp for Component:Staff Mailboxes.edb, dwErrorCode:5369009400EB82470 05/1012:25:15.34331aasubtask.cpp(906)[0000000002C5C500]WARNING<?xml version="1.0"?>
0EB82470 05/1012:25:15.34331aasubtask.cpp(906)[0000000002C5C500]WARNING<Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-1206" Reason="Error" CommandID="RAVerifyDataset" CommandInstanceID="432095b9-45bc-42b3-94de-b3e225bd4657" GuidWorkItem="be20c9c4-e9e7-4caa-b8d5-794443b4156c" TETaskInstanceID="c70f292b-8751-474f-ae8f-34d49c8ea940"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-1206" DetailedSource="2"/><RAStatus><RAVerifyDataset xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"/></RAStatus></Status>
0EB82470 05/1012:25:15.34303runtime.cpp(1376)[0000000000864DF0]FATALSubtask failure, sending status response XML=[<?xml version="1.0"?>
0EB82470 05/1012:25:15.34303runtime.cpp(1376)[0000000000864DF0]FATAL<Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-1206" Reason="Error" CommandID="RAVerifyDataset" CommandInstanceID="432095b9-45bc-42b3-94de-b3e225bd4657" GuidWorkItem="be20c9c4-e9e7-4caa-b8d5-794443b4156c" TETaskInstanceID="c70f292b-8751-474f-ae8f-34d49c8ea940"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="536900940" DetailedCode="-1206" DetailedSource="2"><Parameter Name="ObjectName" Value="Staff Mailboxes.edb"/><Parameter Name="AgentTargetServer" Value="Backup1.riddlesdown.local"/></ErrorInfo><RAStatus><RAVerifyDataset xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"/></RAStatus></Status>
We have the Visual C++ 2012 x64 redistribute installed on the DPM server and the version of ese.dll and eseutil.exe are the same as on the Exchange server.
Any ideas?