Playing with CrashSimulator

Hi everyone,

Today I will show you how CrashSimulator (Shell Script – current version 1.04) works in detail. When unzipping the zip file you will find the following files within it:

• CrashSimulator.sh.x – For high-resolution ssh emulator screens, it shows all 3 menu columns on one screen.
• CrashSimulator_Low.sh.x – For low-resolution ssh emulator screens, it shows CDB options, PDB and ASM options on separated screens.
• CrashSimulator_White.sh.x – Same as the first option but made to be used within White background ssh emulator screens.
• CrashSimulator_Low_White.sh.x – Same as the normal low-resolution option but for White background ssh emulator screens.
• Readme.lst – Text File with installation instructions.

You can download CrashSimulator here.

Note: Before starting with any simulation please ensure to have the proper backups on place. Without backups, it would be almost impossible to recover from some of the scenarios. I recommend you to:

• Run a full RMAN backup that includes your archive logs (Including CDB and PDBs).
• Backup your control file(s) and Parameter file (PFILE or SPFILE) – It could be automatically done by RMAN also.
• Ensure Flashback is on in your database.
• Run a full logical backup of your database with Data Pump.
• Backup your Oracle Home, oraInventory and /etc/oratab file.

Note: Always keep a copy of your backups on a separate storage or media.

Now that you are fully protected, let’s start CrashSimulator by running the command “./CrashSimulator.sh”

The first screen shows us the terms and conditions for the use of the tool, please read them and enter [y/Y] to accept them and continue.

Now we are on the main screen, where all the fun happens. The options available here are:

CDB (or for non-CDB environments) Crash Scenarios:

• [  1] – Loss of a Control File – This will move and rename one of your database control files (adding date and extension bck to it). Details here: /tmp/controlfile.tmp
• [  2] – Loss of all Control Files – This will move and rename all of your database control files (adding date and extension bck to it).Details here: /tmp/allcontrolfile.tmp
• [  3] – Loss of a Redo Log File Group Member – This will move and rename a redo log file (adding date and extension bck to it). Details here: /tmp/logfile.tmp
• [  4] – Loss of a Redo Log File Group – This will move and rename a redo log group files (adding date and extension bck to it). Details here: /tmp/log_groupfiles.tmp
• [  5] – Loss of a non-System Datafile – This will move and rename a non-system datafile (adding date and extension bck to it). Details here: /tmp/non_system_datafile.tmp
• [  6] – Loss of a Temporary Tempfile – This will move and rename tempfile (adding date and extension bck to it). Details here: /tmp/temporary_datafile.tmp
• [  7] – Loss of a System Datafile – This will move and rename a system datafile (adding date and extension bck to it). Details here: /tmp/non_system_datafile.tmp
• [  8] – Loss of an Undo Datafile – This will move and rename an Undo datafile (adding date and extension bck to it). Details here: /tmp/undo_datafile.tmp
• [  9] – Loss of a Read-Only Tablespace – This will move and rename all datafile of a Read-Only tablespace (adding date and extension bck). Details here: /tmp/readonly_tablespace.tmp
• [10] – Loss of an Index Tablespace – This will move and rename all datafile of a Index-Only tablespace (adding date and extension bck). Details here: /tmp/indexonly_tablespace.tmp
• [11] – Loss of all Non-Unique/primary key indexes in Users – This will Drop all Non Unique/primary key indexes on tablespace Users. Details here: /tmp/indexusers.tmp
• [12] – Loss of a non-System Tablespace – This will move and rename all datafiles of a non-system tablespace (adding date and extension bck). Details here: /tmp/non_system_datafiles.tmp
• [13] – Loss of a Temporary Tablespace – This will move and rename all tempfiles (adding date and extension bck to them). Details here: /tmp/temporary_datafiles.tmp
• [14] – Loss of a SYSTEM Tablespace – This will move and rename all System Tablespace datafiles (adding date and extension bck to them). Details here: /tmp/system_datafiles.tmp
• [15] – Loss of an UNDO tablespace – This will move and rename all Undo datafiles (adding date and extension bck to it). Details here: /tmp/undo_datafiles.tmp
• [16] – Loss of the Password File – This will move and rename your PFILE if in use (adding date and extension bck to it). Details here: /tmp/pfile.tmp
• [17] – Loss of all Datafiles – This will move and rename all database datafiles (adding date and extension bck to it). Details here: /tmp/all_datafile.tmp
• [18] – Loss of Redo Log Member of a Multiplexed Group – This will move and rename all Redo Log files in a Multiplexed Group (adding date and extension bck). Details here: /tmp/redo_group.tmp
• [19] – Loss of all Redo Log Members of INACTIVE groups – This will move and rename all Redo Log files in an Inactive Group (adding date and extension bck). Details here: /tmp/inactive_group.tmp
• [20] – Loss of all Redo Log Members of an ACTIVE group – This will move and rename all Redo Log files in an Active Group (adding date and extension bck ). Details here: /tmp/active_group.tmp
• [21] – Loss of all Redo Log Members of a CURRENT group – This will move and rename all Redo Log files in a Current Group (adding date and extension bck ). Details here: /tmp/current_group.tmp
• [22] – File Header Corruption – This will cause corruption of a datafile – Details here: /tmp/filehcorruption.tmp
• [23] – Control Files Corruption – This will cause corruption of a Control File – Details here: /tmp/filecorruption.tmp
• [24] – Current Log File Corruption – This will cause corruption of a Log File – Details here: /tmp/logcorruption.tmp
• [25] – Loss of all RMAN Backups – This will delete all RMAN backup files on disk – Details here: /tmp/rmanloss.tmp
• [26] – Loss of SPFILE – This will move and rename your SPFILE if in use (adding date and extension bck to it). Details here: /tmp/spfile.tmp
• [27] – Loss of TNSNAMES and LISTENER – This will move and rename your Listener.ora and Tnsnames.ora files (adding date and extension bck to it). Details here: /tmp/sqlnet.tmp
• [28] – Loss of ORACLE HOME – This will move your current $ORACLE_HOME directory to $ORACLE_BASE.location. Details here: /tmp/oraclehome.tmp
• [29] – Loss of FRA – This will move your current FRA directory to $ORACLE_BASE. Details here: /tmp/fra.tmp

PDB Crash Scenarios:

• [30] – Loss of a non-System Datafile – This will move and rename a non-system datafile (adding date and extension bck to it). Details here: /tmp/pdbnon_system_datafile.tmp
• [31] – Loss of a Temporary Tempfile – This will move and rename tempfile (adding date and extension bck to it). Details here: tmp/pdbtemporary_datafile.tmp
• [32] – Loss of a SYSTEM Datafile – This will move and rename a system datafile (adding date and extension bck to it). Details here: /tmp/pdbsystem_datafile.tmp
• [33] – Loss of an UNDO Datafile – This will move and rename an Undo datafile (adding date and extension bck to it). Details here: /tmp/pdbundo_datafile.tmp
• [34] – Loss of a Read-Only Tablespace – This will move and rename all datafile of a Read-Only tablespace (adding date and extension bck). Details here: /tmp/pdbreadonly_tablespace.tmp
• [35] – Loss of an Index Tablespace – This will move and rename all datafile of a Index-Only tablespace (adding date and extension bck). Details here: /tmp/pdbindexonly_tablespace.tmp
• [36] – Loss of all Non-unique/primary key indexes in USERS – This will Drop all Non Unique/primary key indexes on tablespace Users. Details here: /tmp/pdbindexusers.tmp
• [37] – Loss of a non-System Tablespace – This will move and rename all datafiles of a non-system tablespace (adding date and extension bck). Details here: /tmp/pdbnon_system_datafileall.tmp
• [38] – Loss of a Temporary Tablespace – This will move and rename all tempfiles (adding date and extension bck to them). Details here: /tmp/pdbtemporary_datafileall.tmp
• [39] – Loss of a SYSTEM Tablespace – This will move and rename all System Tablespace datafiles (adding date and extension bck to them). Details here: /tmp/pdbsystem_datafileall.tmp
• [40] – Loss of an UNDO Tablespace – This will move and rename all Undo datafiles (adding date and extension bck to it). Details here: /tmp/pdbundo_datafileall.tmp
• [41] – Loss of all Datafiles – This will move and rename all database datafiles (adding date and extension bck to it). Details here: /tmp/pdball_datafile.tmp
• [42] – Physical Block Corruption – This will cause corruption of a datafile – Details here: /tmp/pdbfilehcorruption.tmp
• [43] – Loss of a Table – This will drop an user table – Details here: /tmp/pdbtable.tmp
• [44] – Loss of a Schema – This will drop a Schema – Details here: /tmp/pdbschema.tmp
• [45] – Loss of a PDB – This will place a PDB offline and Drop it. Details here: /tmp/pdb.tmp

ASM Crash Scenarios (Coming Soon):

• [46] – Loss of a Data Disk Group

• [47] – Loss of OCR

• [48] – Loss of Voting Disk

• [49] – Loss of a ASM Spfile

Option [99] – This will perform a random scenario for you (Anyone from number 1 to 45).

Here is a view of how the Low-resolution screens looks like:

Please, add comments to this post with ideas of new scenarios to help me with this product road map. I am also currently working in a booklet with all recovery scenarios used by CrashSimulator.

Hope you enjoy this tool and please remember to follow me on Twitter and LinkedIn to be aware of all new versions, updates and news with regards to CrashSimulator.

Kind Regards,

Francisco Munoz Alvarez