Playing with the new OCI Full Stack Disaster Recovery (Full Stack DR) – Part 2 – It is now time to Play

Note: You can review the environment setup part of the tutorial here.

Now that we have all DRPGs created and associated, it is time to add members to each DRPG (primary and standby). For the primary (Ashburn) DRPG we will add:

• The primary DB (ATP)
• The 2 Mushop Compute VM’s
• Two Volume Groups (Boot volumes of the MuShop Compute VM’s)

For the standby (Phoenix) DRPG we will add:

• The Standby DB (ATP)

At the primary DRPG (Ashburn), click at the members (1) option at the Resources tab, click “add member” (2), accept the warning (3), select the Autonomous Database as the resource type (4), ensure the correct compartment is in use (5) and then click add.

Now is time to add the first Compute VM as a member, as follows:

1. Click add member.
2. Make sure to check the box “I understand that all existing plans will be deleted”.
3. Select Resource Type as Compute.
4. Instances in Compartment, select mushop-xxxxx-0, and Click the checkmark in the Move instance on switchover or failover.
5. Destination compartment, select your compartment name, and ignore the destination dedicated VM host section.
6. Click Add VNIC mapping. This will pop up inputs for Add VNIC mapping.
7. Select VNIC as primaryvnic.
8. Destination subnet as mushop-main-xxxxx, and ignore the Network security groups.
9. Click Add.
10. Verify all and click Add (again).

After finishing adding the “mushop-xxxxx-0”, add the second compute instance “mushop-xxxxx-1“, following the same steps as when adding the previous compute instance.

Now is the time to add the first volume group,  “mushop-volume-group-0“. This volume group consists of the boot volume of “mushop-xxxx-0” VM and has cross-region replication configured to the phoenix region. For this, this time you will need to:

Add the second volume group, “mushop-volume-group-1” as done for the previous volume group added.

Now you can see the primary DRPG with all it members added and Active.

With the Primary DRPG complete, we will now move to the tab open at the Phoenix Region and start adding our member to the Standby (Phoenix) DRPG.

With this step we have all our DRPGs with all it members. Now it is time to create and customise a DR Switchover Plan. At the DRPG page at the standby region (Phoenix):

1. Click at Plans (Resources tab).
2. Click Create plan.
3. Enter  mushop-app-switchover as the plan name.
4. Select  Switchover (planned) as the Plan type.
5. Hit Create.

Select the mushop-app-switchover plan, and you should be able to various built-in plan groups.

The next step in the process is to gather a Load Balancer OCID’s. Click at the hamburger menu at the top left corner of the screen and select “networking” (1) and then, “load balancers” (2).

You will see that you have a load balancer with the name mushop-xxxx; choose the right compartment assigned to you. Select the three (4) dots on the right end of the load balancer details. Select Copy OCID (5) and paste the OCID details safely into any preferred notes. You should have something similar to below

ocid1.loadbalancer.oc1.iad.aaaaaaaa2t4kwwavlgwghuebrce6mgqm5ewrsj5kscw2t5ncdpxdpo6ztvaq (Don’t use this)

After this, connect to the Ashburn (Primary) region and collect the OCID as done before for the load balancer there.

The next step is to customise the Switchover plan by removing the primary load balancer backends group by using the browser tab where you have created the switchover plan in phoenix region. Create user-defined groups for mushop application switchover. We need to create four user-defined groups.

Let’s create those. You can do this by selecting Add group in the mushop-app-switchover plan at Phoenix and:

1. Add “Remove Primary Load Balancer Backends” User defined group

• Add Remove Primary Load Balancer Backends in Group name
• Add Remove Primary Backend on Node-0 in Step name
• Leave the enabled tick mark
• Select Error mode as “Stop on error.”
• Leave the default “3600” seconds in Timeout in seconds
• In the region, select “US East (Ashburn).”
• Select the “Run local script” option
• Select “mushop-xxxxx-0” instance in “Target instance in compartment”
• In script parameters, add the below script using the primary (Ashburn) OCID

/usr/bin/sudo /home/opc/fsdrsscripts/removeFromBackendset.py <REPLACE WITH YOUR OCID>

• Leave the field blank in “Run as user.”
• Verify all the details and hit add

Now we will add another step for Remove Primary Backend on Node-1 in Remove Primary Load Balancer Backends Plan group

• Select the three dots section in the right end from the Remove Primary Load Balancer Backends Plan group. Select Add Step
• Leave the default Group name
• Add Remove Primary Backend on Node-1 in Step name
• Leave the enabled tick mark
• Select Error mode as “Stop on error.”
• Leave the default “3600” seconds in Timeout in seconds
• In the region, select “US East (Ashburn).”
• Select the “Run local script” option
• Select “mushop-xxxxx-1” instance in “Target instance in compartment”
• In script parameters, add the below script

/usr/bin/sudo /home/opc/fsdrsscripts/removeFromBackendset.py <REPLACE WITH YOUR OCID>

Replace the OCID of the primary (Ashburn) load balancer as per step 2.4; make sure you replace the OCID of your load balancer without fail in the above command,remove angle brackets,note there is space after removeFromBackendset.py

• Leave the field blank in “Run as user.”
• Verify all the details and hit add

Now is time to customise the Switchover plan – Restore Database Wallet Group by creating a user-defined group for “Restore Database Wallet.” This can be done by selecting Add group in the Restore Database Wallet plan by:

• Add Restore Database Wallet in the Group name
• Add Restore Database Wallet on Node-0 in Step name
• Leave the enabled tick mark
• Select Error mode as “Stop on error.”
• Leave the default “3600” seconds in Timeout in seconds
• In the region, select “US East (Ashburn).”
• Select the “Run local script” option
• Select “mushop-xxxxx-0” instance in “Target instance in compartment”
• In script parameters, add the below script

/usr/bin/sudo /home/opc/fsdrsscripts/mushop_db_wallet_restore.sh

• Leave the field blank in “Run as user.”
• Verify all the details and hit add

Now as previously done, we need to add another step for Restore Database Wallet on Node-1 in Restore Database Wallet plan group.

• From the Restore Database Wallet Plan group, select the three dots section at the right end. Select Add Step
• Leave the default Group name
• Add Restore Database Wallet on Node-1 in Step name
• Leave the enabled tick mark
• Select Error mode as “Stop on error.”
• Leave the default “3600” seconds in Timeout in seconds
• In the region, select “US East (Ashburn).”
• Select the “Run local script” option
• Select “mushop-xxxxx-1” instance in “Target instance in compartment”
• In script parameters, add the below script

/usr/bin/sudo /home/opc/fsdrsscripts/mushop_db_wallet_restore.sh

• Leave the field blank in “Run as user.”
• Verify all the details and hit add

Now we will customise the Switchover plan – Restore the application group by creating a user-defined group for “Restore Application.” This can be done by selecting Add group in the mushop-app-switchover plan, by:

• Add Restore Application in the Group name
• Add Restore Application on Node-0 in Step name
• Leave the enabled tick mark
• Select Error mode as “Stop on error.”
• Leave the default “3600” seconds in Timeout in seconds
• In the region, select “US East (Ashburn).”
• Select the “Run local script” option
• Select “mushop-xxxxx-0” instance in “Target instance in compartment”
• In script parameters, add the below script

/usr/bin/sudo /home/opc/fsdrsscripts/mushop_reconfigure.sh

• Leave the field blank in “Run as user.”
• Verify all the details and hit add

Next we need to add another step for Restore Application on Node-1 in the Restore Application plan group.

• From the Restore Application Plan group, select the three dots section at the right end. Select Add Step
• Leave the default Group name
• Add Restore Application on Node-1 in Step name
• Leave the enabled tick mark
• Select Error mode as “Stop on error.”
• Leave the default “3600” seconds in Timeout in seconds
• In the region, select “US East (Ashburn).”
• Select the “Run local script” option
• Select “mushop-xxxxx-1” instance in “Target instance in compartment”
• In script parameters, add the below script

/usr/bin/sudo /home/opc/fsdrsscripts/mushop_reconfigure.sh

• Leave the field blank in “Run as user.”
• Verify all the details and hit add

Now we need to customise the Switchover plan – Add Standby Load Balancer Backends Group by creating a user-defined group for “Add Standby Load Balancer Backends.” This can be done by selecting Add group in the mushop-app-switchover plan by:

• Add “Add Standby Load Balancer Backends” User defined group
• Add Add Standby Load Balancer Backends in Group name
• Add Add Standby Backend on Node-0 in Step name
• Leave the enabled tick mark
• Select Error mode as “Stop on error.”
• Leave the default “3600” seconds in Timeout in seconds
• In the region, select “US East (Ashburn).”
• Select the “Run local script” option
• Select “mushop-xxxxx-0” instance in “Target instance in compartment”
• In script parameters, add the below script

/usr/bin/sudo /home/opc/fsdrsscripts/addToBackendset.py <REPLACE WITH YOUR OCID>

You will need to replace the OCID of the standby (Phoenix) load balancer as per step 2.6; make sure you replace the OCID of your load balancer without fail in the above command,remove angle brackets,note there is space after addToBackendset.py

• Leave the field blank in “Run as user.”
• Verify all the details and hit add

Now we need to add another step for Add Standby Backend on Node-1 in Add Standby Load Balancer Backends Plan group

• Select the three dots section in the right end from the Add Standby Load Balancer Backends Plan group. Select Add Step
• Leave the default Group name
• Add Add Standby Backend on Node-1 in Step name
• Leave the enabled tick mark
• Select Error mode as “Stop on error.”
• Leave the default “3600” seconds in Timeout in seconds
• In the region, select “US East (Ashburn).”
• Select the “Run local script” option
• Select “mushop-xxxxx-1” instance in “Target instance in compartment”
• In script parameters, add the below script

/usr/bin/sudo /home/opc/fsdrsscripts/addToBackendset.py <REPLACE WITH YOUR OCID>

You need to replace the OCID of the standby (Phoenix) load balancer as per step 2.6; make sure you replace the OCID of your load balancer without fail in the above command,remove angle brackets,note there is space after addToBackendset.py

• Leave the field blank in “Run as user.”
• Verify all the details and hit add

The next step would be to verify and reorder the User defined groups. We have created all the required user-defined groups in the mushop-app-switchover switchover plan as part of the Mushop application switchover.

Let’s review the mushop-app-switchover switchover plan:

• Built-in Prechecks – These are the built-in prechecks groups for all the Plan groups (Built-in and User defined)
• Based on the members we have added in both Primary DRPG and Standby DRPG, FSDRS created seven Built-in switchover plan
• We have manually created four user-defined groups per the Mushop application switchover requirement.
• In summary, the mushop-app-switchover switchover plan has created with one– Built-in precheck plan group, seven– Built-in Plan group,four– User defined Plan group

Plan groups can be reordered as per the switchover workflow requirement. As part of the Mushop Switchover plan, we would like to execute Remove Primary Load Balancer Backends plan group after the Built-In Prechecks plan group. Use the Actions after the Add group, and select Reorder groups.

Go to the Remove Primary Load Balancer Backends plan group, use the move up ^ symbol, and keep moving up the Remove Primary Load Balancer Backends plan group and place it after the Built-In Prechecks plan group. This is very important to execute the plan groups in the proper order. Verify and hit Save changes. Don’t move the other groups.

The next step would be to perform a run pre-checks for the mushop-app-switchover plan. Please continue connected to the page in the Phoenix Region (Standby) at the “mushop-app-switchover” and click “Run prechecks“.

In the Run prechecks window, provide the Plan execution name as mushop-app-switchover-prechecks and hit Run prechecks.

Navigate to Plan executions section under Resources and select the mushop-app-switchover-prechecks plan execution.Initially, it will show all the Built-in Prechecks as Queued.

Refresh the page; within a few seconds, the State will change from Queued to In Progress.

All the Built-in Prechecks steps will execute in parallel; you can monitor the various steps log. Navigate to the three dots section for the respective built-in step and click. You get the option to view the log and download the log. These logs are stored in the object storage bucket provided during the DRPG creation. You can monitor the Progress and download the log if necessary for troubleshooting.

After 2-3 mins, Built-in Prechecks will be completed successfully. You can verify the duration of each step, status, duration of entire prechecks, etc. It is essential to have successful completion of pre-check execution. Now login into OCI Console with your provided Credentials. Select region as Ashburn.

From the Hamburger menu, select Networking, then Load Balancers gather the Load Balancer public IP.

Open a tab in your browser and verify the Mushop Application using the gathered public IP address. Play around the site and verify various cat products.

Let us break the MuShop Application to create an outage. Click the mushop-xxxxx load balancer details; in the resources section, select Listeners.

Select the edit option using the three dots symbol at the right end of the port 80 listeners, mushop-xxxxx-80 and modify the listener port from 80 to 81 to break the application, and save changes. You will get work requests submitted and select close requests.

Wait for a few seconds and refresh the browser tab of the MuShop application. You should see that the MuShop site can’t be reached.

We have created an outage to a working MuShop application; in the next lab, we will use the Full Stack DR service to initiate the Switchover of the Full Stack (App VM’s/Database/Application Customization) from Ashburn to the Phoenix region.

Now we will execute a switchover for mushop-app-switchover plan.Login into OCI Console with your provided Credentials. Select region as Pheonix.Select Migration and Disaster Recovery from the Hamburger menu, then Disaster Recovery -> DR Protection Groups. Verify the region is Phoenix.

You will land on the Disaster Recovery Protection group home page; make sure you have selected the Phoenix region.Select the mushop-phoenix DRPG and select mushop-app-switchover plan.

Navigate to the Execute Plan section, which will be right below the mushop-app-switchover plan, and select it.

In the Execute Plan window

• Provide the Plan execution name as mushop-app-switchover-execute
• Uncheck the Enable prechecks ( Prechecks were executed successfully in Lab 5)
• Leave the Ignore warnings as it is
• Verify and hit Execute Plan

Navigate to Plan executions section under Resources and select the mushop-app-switchover-execute plan execution.Initially, it will show all the steps as Queued.

• All the plan groups will run serially, but steps inside each plan group will be parallel. As an example, Remove Primary Load Balancer Backends group will start first, and the steps (Remove Primary Backend on Node-0 and Remove Primary Backend on Node-1) are part of this group will run in parallel.
• Monitor the various plan group and steps which are running. Navigate to the three dots section for the respective plan group step and click. You get the option to view the log and download the log. These logs are stored in the object storage bucket provided during the DRPG creation. You can monitor the Progress and download the log if necessary for troubleshooting.
• Once each plan group is executed successfully, it will move on to the next group for execution. Here you can see  Remove Primary Load Balancer Backends completed successfully (State-Succeeded) and the next group Stop Compute Instances (Primary) started running (State: In progress).
• Keep monitoring the rest of the groups and steps; each step will complete depending on the actual task (DB Switchover, VM Stop, Script execution ), etc.). For example, ATP DB Switchover will take more time when compared to stopping VM. You can verify the start and end of each step, total duration, logs, etc.

• Wait for all the steps to complete successfully. It is important to monitor the progress of each step and take actions in case of any failures. Switchover Autonomous Databases (Standby) group will run for around 10 minutes. 

Approximately it will take 20-30 mins to successfully complete the switchover plan.

Verify the executed switchover plan

After the switchover completes, please:

1. From the plan execution detail, verify the duration of each step, status, duration of the entire switchover plan, etc. It is essential to have successful completion of all steps. In this example it took around 27 minutes to complete. These timings will vary.Use the Expand all button to expand all the steps and the Collapse all button for collapsing. Use the view or download log option to see step execution details.
2. Notice the Role of the mushop-phoenix DRPG; it has automatically changed to Primary. So, now we have our new primary region as Phoenix region.
3. At the OCI Console change the region to Ashburn.
4. Notice the Role of the mushop-ashburn DRPG; it has automatically changed to Standby. Now we have our new standby region as Ashburn region.
5. From the Hamburger menu, select Networking, then Load Balancers – also, please verify that the region in use now is Phoenix and gather the Public IP address of the Load Balancer.
6. Open a tab in your browser and verify the Mushop Application using the gathered public IP address. You should be able to see that the application is working as expected from the Phoenix region.
7. Mushop application is accessible from the new primary region (Phoenix).

Thank you so much for your participation and home you enjoyed this tutorial.

Kind Regards,

Francisco Munoz Alvarez