NetApp 8.3.1 SVM DR failure

I was trying to setup a SVM DR relationship of a recently migrated 7-mode vfiler and it would never initialise. Needed some help troubleshooting the issue but the key command to use on the target cluster to get the failure reason (nothing seems to be in log files now):

snapmirror show -destination-vserver vnas-homes -fields last-transfer-error,unhealthy-reason -expand

This resulted in the following reason:

vnas-homes: vnas-homes-sm:   Failed to apply the source Vserver configuration. Reason: Unrecoverable error during configuration replication, requires manual intervention. Error: Schedule CronJob_vnas-homes_vol_homes_0 not found. Execute "snapmirror show -destination-vserver vnas-homes-sm -fields last-transfer-error,unhealthy-reason -expand" to check if the constituent volumes have enountered errors. Transfer failed
vnas-homes:vol_homes vnas-homes-sm:vol_homes -

So it turns out that if the source SVM uses:

  1. cluster based schedules for snapshots
  2. cluster based policies for snapshots
  3. SVM based snapshot policies that refer to cluster based schedules

Then these schedules and policies MUST either:

  1. exist on the target cluster
  2. be removed from the source cluster
  3. not referred to by the source SVM

If not the ALL SVM DR relationships between the two clusters will fail.

In this can I wanted to remove them So I need to find the Job schedules and then work out which policies referred to them and remove said policies.

Firstly in my setup I create cluster wide schedules and snapshot policies and associate SVM volume snapshot schedules to these as I don’t have lots of various unique backup requirements. When using the 7MMT to move a vfiler into a C-mode cluster it creates cluster based Job schedules and SVM based snapshot policies that refer to them.

The root cause of the failure was the schedule CronJob_vnas-homes_vol_homes_0 so if I try to delete that as I don’t need it on the target cluster I get:

netapp02::> job schedule delete -name CronJob_vnas-homes_vol_homes_0
Error: command failed: Cannot delete job schedule. Schedule referenced by 1 volume Snapshot policy entries, 0
       SnapMirror entries, 0 antivirus on-demand entries, 0 SIS policy entries, and 0 configuration backup
       settings. Remove references to schedule first.

So a single volume snapshot policy refers to that schedule. As it’s blocking the vnas-homes SVM update I know it’s there so I check the snapshot policies of the source SVM and the source cluster, just to be sure. I can then see that there is a single SVM based snapshot policy SnapShot_vnas-homes_vol_homes_0 which is using three schedules, all automatically created by the migration tool:

netapp02::> snapshot policy show -vserver vnas-homes
Vserver: vnas-homes
                         Number of Is
Policy Name              Schedules Enabled Comment
------------------------ --------- ------- ----------------------------------
SnapShot_vnas-homes_vol_homes_0  3 true    -
    Schedule               Count     Prefix                 SnapMirror Label
    ---------------------- -----     ---------------------- -------------------
    CronJob_vnas-homes_vol_homes_0
                              26     CronJob_vnas-homes_vol_homes_0
                                                            -
    CronJob_vnas-homes_vol_homes_1
                               7     CronJob_vnas-homes_vol_homes_1
                                                            -
    CronJob_vnas-homes_vol_homes_2
                               9     CronJob_vnas-homes_vol_homes_2
                                                            -

netapp02::> snapshot policy show -vserver netapp02
Vserver: netapp02
                         Number of Is
Policy Name              Schedules Enabled Comment
------------------------ --------- ------- ----------------------------------
std                              1 true    -
    Schedule               Count     Prefix                 SnapMirror Label
    ---------------------- -----     ---------------------- -------------------
    daily                      7     daily                  -

std_homes                        3 true    -
    Schedule               Count     Prefix                 SnapMirror Label
    ---------------------- -----     ---------------------- -------------------
    daily                      7     daily                  -
    weekly                    26     weekly                 -
    2hour                     24     2hour                  -

default                          3 true    Default policy with hourly, daily & weekly schedules.
    Schedule               Count     Prefix                 SnapMirror Label
    ---------------------- -----     ---------------------- -------------------
    hourly                     6     hourly                 -
    daily                      2     daily                  daily
    weekly                     2     weekly                 weekly

default-1weekly                  3 true    Default policy with 6 hourly, 2 daily & 1 weekly schedule.
    Schedule               Count     Prefix                 SnapMirror Label
    ---------------------- -----     ---------------------- -------------------
    hourly                     6     hourly                 -
    daily                      2     daily                  -
    weekly                     1     weekly                 -

none                             0 false   Policy for no automatic snapshots.
    Schedule               Count     Prefix                 SnapMirror Label
    ---------------------- -----     ---------------------- -------------------
    -                          -     -                      -

I need to verify no volumes on the SVM vnas-homes are using the SVM based policy I need to remove:

netapp02::> volume show -vserver vnas-homes -volume * -fields snapshot-policy
vserver    volume          snapshot-policy
---------- --------------- ---------------
vnas-homes vnas_homes_vol0 std
vnas-homes vol_homes       std_homes
2 entries were displayed.

As they’re not it’s safe to remove the snapshot policy:

netapp02::> snapshot policy delete -vserver vnas-homes -policy SnapShot_vnas-homes_vol_homes_0
netapp02::> snapshot policy show -vserver vnas-homes
There are no entries matching your query.

Now from the target cluster I should now be able to initialise the DR relationship:

netapp04::> snapmirror initialize vnas-homes-sm:
netapp04::> snapmirror show -vserver vnas-homes-sm
                                                                       Progress
Source            Destination Mirror  Relationship   Total             Last
Path        Type  Path        State   Status         Progress  Healthy Updated
----------- ---- ------------ ------- -------------- --------- ------- --------
vnas-homes: DP   vnas-homes-sm:
                              Uninitialized
                                      Transferring   93.33GB   false   <date/time>
netapp04::> snapmirror show -vserver vnas-homes-sm
                                                                       Progress
Source            Destination Mirror  Relationship   Total             Last
Path        Type  Path        State   Status         Progress  Healthy Updated
----------- ---- ------------ ------- -------------- --------- ------- --------
vnas-homes: DP   vnas-homes-sm:
                              Snapmirrored
                                      Idle           -         true    -

Fixed!

3 thoughts on “NetApp 8.3.1 SVM DR failure”

  1. Hi ,

    Just want to know, is there any way if we can check/verify the snapmirror is configured on SVM or volume or both.

    because when i run Snapmirror show command. it shows both relationship. the SVM and SVM:volumeName.

    regards
    VK

    1. Hi Vinay

      I’m not sure what you mean? If you want to know if the snapmirror relationship is a “DR” type or a more traditional “per volume” type you could run the following command on the target cluster:

      snapmirror show -fields identity-preserve

      If it’s “true” then it’s a “DR” type relationship and is snapmirroring the entire SVM.

      Or just looking at the outpout of the “snapmirror show” an identity preserving relationship with only show the source and target SVM’s followed with a colon, no mention of the volumes, for example:

      vnas-homes: DP vnas-homes-sm: Snapmirrored Idle – true –

      HTH

Leave a Reply

Your email address will not be published. Required fields are marked *