29 August, 2021

Identifying Patches applied to an Oracle Installation and Database

"opatch" and "datapatch" are the methods to apply Oracle Patches in recent versions.

"opatch" applies updates to the library and binary files to the ORACLE_HOME at the OS level.

"datapatch" then applies the corresponding changes to the data dictionary in the database.  Note that if you have multiple databases running from the same ORACLE_HOME, you must run "datapatch" against each database.  Also, if you later create a new database [most of us use a template / script to do so], you must run "datapatch" against the new database as well.

The instructions for both "opatch" and "datapatch"  (including the pre-requisite checks) are included in the README file that accompanies each patch.   See Mike Dietrich's blog post on how to apply the July 2021 Release Update Patch


Here is a quick demo of the commands to identify patches that are applied.


oracle19c>$ORACLE_HOME/OPatch/opatch version
OPatch Version: 12.2.0.1.25

OPatch succeeded.
oracle19c>
oracle19c>$ORACLE_HOME/OPatch/opatch lspatches
32876380;OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)
32904851;Database Release Update : 19.12.0.0.210720 (32904851)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

OPatch succeeded.
oracle19c>
oracle19c>$ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 12.2.0.1.25
Copyright (c) 2021, Oracle Corporation.  All rights reserved.


Oracle Home       : /opt/oracle/product/19c/dbhome_1
Central Inventory : /opt/oracle/oraInventory
   from           : /opt/oracle/product/19c/dbhome_1/oraInst.loc
OPatch version    : 12.2.0.1.25
OUI version       : 12.2.0.7.0
Log file location : /opt/oracle/product/19c/dbhome_1/cfgtoollogs/opatch/opatch2021-08-29_21-20-33PM_1.log

Lsinventory Output file location : /opt/oracle/product/19c/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2021-08-29_21-20-33PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: ora19cs1
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Database 19c                                                  19.0.0.0.0
There are 1 products installed in this Oracle Home.


Interim patches (3) :

Patch  32876380     : applied on Mon Aug 09 22:59:39 GMT+08:00 2021
Unique Patch ID:  24269510
Patch description:  "OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)"
   Created on 5 Jul 2021, 04:53:50 hrs UTC
   Bugs fixed:
     29445548, 29254623, 29540327, 29774362, 30134746, 30160625, 30534662
     29512125, 29942275, 30855101, 31306261, 31359215, 30895577, 29224710
     26716835, 31668872, 32165759, 32069696, 32032733, 30889443, 30674373
     32167592, 32523206, 29415774, 28777073, 32124570, 31247838, 29540831
     32892883, 31776121

Patch  32904851     : applied on Mon Aug 09 21:44:24 GMT+08:00 2021
Unique Patch ID:  24343243
Patch description:  "Database Release Update : 19.12.0.0.210720 (32904851)"
   Created on 20 Jul 2021, 09:21:24 hrs UTC
   Bugs fixed:
     7391838, 8460502, 8476681, 14570574, 14735102, 15931756, 16662822
     16664572, 16750494, 17275499, 17395507, 17428816, 17468475, 17777718
     18534283, 18697534, 19080742, 19138896, 19697993, 20007421, 20083476
     20313356, 20319830, 20479545, 20867658, 20922160, 21119541, 21232786
     21245711, 21374587, 21528318, 21629064, 21639146, 21888352, 21965541
     22066547, 22252368, 22325312, 22387320, 22580355, 22725871, 22729345
     22748979, 23020668, 23094775, 23125587, 23294761, 23296836, 23311885
     23606241, 23645975, 23734075, 23763462, 24336782, 24356932, 24561942
     24596874, 24669730, 24687075, 24833686, 24957575, 24971456, 25030027
     25092651, 25093917, 25148135, 25404117, 25416731, 25560538, 25562258
     25607406, 25607716, 25756945, 25792962, 25804387, 25804908, 25806201
...
... cut short a VERY LONG LIST of Bug Numbers
...
     32677702, 32686850, 32697781, 32698569, 32700989, 32704765, 32711741
     32712220, 32716726, 32718316, 32720458, 32725484, 32728984, 32738356
     32739966, 32740503, 32754845, 32758096, 32765738, 32784393, 32784403
     32786309, 32795712, 32810668, 32811069, 32811131, 32816003, 32817950
     32818019, 32858446, 32874995, 32881853, 32889434, 32895105, 32900208
     32902635, 32936961, 32941509, 32996071, 33034103, 33048277, 33127032
     32490416

Patch  29585399     : applied on Thu Apr 18 15:21:33 GMT+08:00 2019
Unique Patch ID:  22840393
Patch description:  "OCW RELEASE UPDATE 19.3.0.0.0 (29585399)"
   Created on 9 Apr 2019, 19:12:47 hrs PST8PDT
   Bugs fixed:
     27222128, 27572040, 27604329, 27760043, 27877830, 28302580, 28470673
     28621543, 28642469, 28699321, 28710663, 28755846, 28772816, 28785321
     28800508, 28808652, 28815557, 28847541, 28847572, 28870496, 28871040
     28874416, 28877252, 28881191, 28881848, 28888083, 28911140, 28925250
     28925460, 28935956, 28940472, 3, 28942694, 28951332, 28963036, 28968779
     28980448, 28995287, 29003207, 29003617, 29016294, 29018680, 29024876
     29026154, 29027933, 29047127, 29052850, 29058476, 29111631, 29112455
     29117337, 29123444, 29125708, 29125786, 29129476, 29131772, 29132456
     29139727, 29146157, 29147849, 29149170, 29152603, 29152752, 29154631
     29154636, 29154829, 29159216, 29159661, 29160462, 29161923, 29169540
     29169739, 29170717, 29173618, 29181568, 29182920, 29183298, 29186091
     29191827, 29201143, 29201695, 29209545, 29210577, 29210610, 29210624
     29210683, 29213641, 29219627, 29224294, 29225861, 29229839, 29235934
     29242906, 29243749, 29244495, 29244766, 29244968, 29248723, 29249583
     29251564, 29255616, 29260224, 29261695, 29271019, 29273360, 29282090
     29282666, 29285453, 29285621, 29290235, 29292232, 29293806, 29294753
     29299830, 29307090, 29307109, 29311336, 29329675, 29330791, 29339299
     29357821, 29360467, 29360775, 29367971, 29368725, 29379299, 29379381
     29380527, 29381000, 29382296, 29391301, 29393649, 29402110, 29411931
     29413360, 29457319, 29465047



--------------------------------------------------------------------------------

OPatch succeeded.
oracle19c>


The first command "opatch version" shows the current version
The second command "opatch lspatches" provides a high level listing of the patches that have been applied.
The third command "opatch lsinventory" provides a detailed listing of the patches that includes the dates when they were applied by the DBA, the dates they were actually created and released by Oracle Development / Support and each individual Bug# numbers fixed in the patches.  
Release Updates in 12.2 and higher are cumulative.  Here, "19.12.0.0.210720" indicates that I have applied the 19.12 Release Update of July 2021 ("2107" from the patch identifier)


Another method, using sql is the DBMS_QOPATCH package which has been implemented and substantially improved in recent releases.  (Oracle Support Document "How to Find PSU/One-off Patches is Applied Using QOPatch(DBMS_QOPATCH) (Doc ID 2169610.1)" is a good starting point :


SQL> set pages50000
SQL> set long 100000
SQL> spool dbms_qopatch_lsinventory.TXT
SQL> select dbms_qopatch.get_opatch_lsinventory from dual;
...
...
SQL> spool off
SQL> quit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
oracle19c>ls -l dbms_qopatch_lsinventory.TXT
-rw-r--r--. 1 oracle oinstall 237880 Aug 29 21:50 dbms_qopatch_lsinventory.TXT
oracle19c>
oracle19c>wc -l dbms_qopatch_lsinventory.TXT
1792 dbms_qopatch_lsinventory.TXT

dbms_qopatch.get_opatch_lsinventory


Since my output file from the dbms_qopatch.get_opatch_lsinventory is very long (1792 lines, 237KBytes, I've presented a screenshot of some of the lines f rom the file).  In addition to showing Bug# numbers, it also shows the Bug Titles (i.e. Descriptions) which the command-line opatch lsinventory doe not provide.


DBMS_QOPATCH is very useful.  You can query to check if a certain Patch has been installed (Patch 32904851 is the July 2021 Release Update Patch 19.12 that I installed on 09-Aug-2021).


SQL> set pages5000
SQL> set pages50000
SQL> set long 1000000
SQL> set pages50000
SQL> set long 1000000
SQL> select xmltransform(dbms_qopatch.is_patch_installed('32904851'),dbms_qopatch.get_opatch_xslt) from dual;

XMLTRANSFORM(DBMS_QOPATCH.IS_PATCH_INSTALLED('32904851'),DBMS_QOPATCH.GET_OPATCH_XSLT)
------------------------------------------------------------------------------------------------------------------------------------

Patch Information:
         32904851:   applied on 2021-08-09T21:44:24+08:00


SQL>
SQL>


"opatch" only lists patches that are applied to the ORACLE_HOME.  However, you must also confirm if the database data dictionary has also been updated -- which would be so if "datapatch" has been executed.  (side note : "datapatch" also has a number of command-line arguments which you might want to explore)


This information is in the dba_registry and dba_registry_sqlpatch  views in the database


SQL> set linesize 132
SQL> select * from dba_registry_history order by action_time;

ACTION_TIME                                                                 ACTION
--------------------------------------------------------------------------- ------------------------------
NAMESPACE                      VERSION                                ID
------------------------------ ------------------------------ ----------
COMMENTS
------------------------------------------------------------------------------------------------------------------------------------
BUNDLE_SERIES
------------------------------
04-MAY-19 11.30.32.133846 PM                                                RU_APPLY
SERVER                         19.0.0.0.0
Patch applied on 19.3.0.0.0: Release_Update - 190410122720


09-AUG-21 10.13.42.105236 PM                                                RU_APPLY
SERVER                         19.0.0.0.0
Patch applied from 19.3.0.0.0 to 19.12.0.0.0: Release_Update - 210716141810


09-AUG-21 11.10.15.544636 PM                                                jvmpsu.sql
SERVER                         19.12.0.0.210720OJVMRU                  0
RAN jvmpsu.sql


09-AUG-21 11.10.15.722597 PM                                                APPLY
SERVER                         19.12.0.0.210720OJVMRU                  0
OJVM RU post-install


                                                                            BOOTSTRAP
DATAPATCH                      19
RDBMS_19.12.0.0.0DBRU_LINUX.X64_210715



SQL>
SQL> select * from dba_registry_sqlpatch order by action_time;

INSTALL_ID   PATCH_ID  PATCH_UID PATCH_TYPE ACTION          STATUS
---------- ---------- ---------- ---------- --------------- -------------------------
ACTION_TIME
---------------------------------------------------------------------------
DESCRIPTION
----------------------------------------------------------------------------------------------------
LOGFILE
------------------------------------------------------------------------------------------------------------------------------------
RU_LOGFILE
------------------------------------------------------------------------------------------------------------------------------------
FLAGS
----------
PATCH_DESCRIPTOR
------------------------------------------------------------------------------------------------------------------------------------
PATCH_DIRECTORY
------------------------------------------------------------------------------------------------------------------------------------
SOURCE_VERSION  SOURCE_BUILD_DESCRIPTION
--------------- --------------------------------------------------------------------------------
SOURCE_BUILD_TIMESTAMP                                                      TARGET_VERSION
--------------------------------------------------------------------------- ---------------
TARGET_BUILD_DESCRIPTION
--------------------------------------------------------------------------------
TARGET_BUILD_TIMESTAMP
---------------------------------------------------------------------------
         1   29517242   22862832 RU         APPLY           SUCCESS
04-MAY-19 11.31.01.355942 PM
Database Release Update : 19.3.0.0.190416 (29517242)
/opt/oracle/cfgtoollogs/sqlpatch/29517242/22862832/29517242_apply_ORCLCDB_CDBROOT_2019May04_23_23_03.log
/opt/oracle/cfgtoollogs/sqlpatch/29517242/22862832/29517242_ru_apply_ORCLCDB_CDBROOT_2019May04_23_22_59.log
N

sqlPatch ID="29517242" uniquePatchID=

19.1.0.0.0      Feature Release
                                                                            19.3.0.0.0
Release_Update
10-APR-19 12.27.20.000000 PM



         2   32904851   24343243 RU         APPLY           SUCCESS
09-AUG-21 10.35.37.696526 PM
Database Release Update : 19.12.0.0.210720 (32904851)
/opt/oracle/cfgtoollogs/sqlpatch/32904851/24343243/32904851_apply_ORCLCDB_CDBROOT_2021Aug09_22_04_05.log
/opt/oracle/cfgtoollogs/sqlpatch/32904851/24343243/32904851_ru_apply_ORCLCDB_CDBROOT_2021Aug09_22_03_59.log
N

sqlPatch ID="32904851" uniquePatchID=

19.3.0.0.0      Release_Update
10-APR-19 12.27.20.000000 PM                                                19.12.0.0.0
Release_Update
16-JUL-21 02.18.10.000000 PM



         3   32876380   24269510 INTERIM    APPLY           SUCCESS
09-AUG-21 11.12.07.815060 PM
OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)
/opt/oracle/cfgtoollogs/sqlpatch/32876380/24269510/32876380_apply_ORCLCDB_CDBROOT_2021Aug09_23_06_42.log

NJ

sqlPatch ID="32876380" uniquePatchID=
504B03041400000008007A77E552F6AD657DF40100006C0400000C00000033323837363338302E786D6CB5534D73DA3014BCFB57BCEAD41EC01F044232B6339EE076
C804C81092999E3A8A2D374A6559
19.12.0.0.0     Release_Update
16-JUL-21 02.18.10.000000 PM                                                19.12.0.0.0
Release_Update
16-JUL-21 02.18.10.000000 PM


SQL>


The first query shows that "datapatch" was executed on 09-Aug-21 and the second query shows that it executed the changes for both Patches 32904851 (Database Release Update) and 32876380 (OJVM Release Update)


14 August, 2021

Restoring Standby Database after Failover

 In the previous post, I demonstrated how the Database Incarnation changed after a Failover was issued for a Standby Database.

Let's assume that Read-Write testing of the Standby has been done and now I need to restore and revert STDBYDB to the state of being a Standby for ORCLCDB


The Primary is still at Database Incarnation#=3  and is now at Sequence#=409



SQL> show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      ORCLCDB
SQL> select incarnation#
  2  from v$database_incarnation
  3  where status = 'CURRENT'
  4  /

INCARNATION#
------------
           3

SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /opt/oracle/archivelog/ORCLCDB
Oldest online log sequence     407
Next log sequence to archive   409
Current log sequence           409
SQL>


The former Standby STDBYDB has already diverged and is at Incarnation#=4 and its current Sequence#=5


SQL> show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      STDBYDB
SQL> select incarnation#
  2  from v$database_incarnation
  3  where status = 'CURRENT'
  4  /

INCARNATION#
------------
           4

SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /opt/oracle/archivelog/STDBYDB
Oldest online log sequence     3
Next log sequence to archive   5
Current log sequence           5
SQL>


I now delete all the datafiles STDBYDB and restore the the backup that I made before I executed the Failover (see the previous blog post)


SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> quit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
oracle19c>cd
oracle19c>sh ./to_delete_STDBYDB.SH
oracle19c>
oracle19c>echo $ORACLE_SID
STDBYDB
oracle19c>rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Sat Aug 14 17:49:50 2021
Version 19.12.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database (not started)

RMAN> startup nomount;

Oracle instance started

Total System Global Area    1207955544 bytes

Fixed Size                     9134168 bytes
Variable Size                486539264 bytes
Database Buffers             704643072 bytes
Redo Buffers                   7639040 bytes

RMAN> restore controlfile from '/opt/oracle/FRA/STDBYDB/STDBYDB/autobackup/2021_08_13/o1_mf_s_1080515319_jkf2tzhp_.bkp';

Starting restore at 14-AUG-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=245 device type=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
output file name=/opt/oracle/oradata/STDBYDB/control01.ctl
output file name=/opt/oracle/oradata/STDBYDB/control02.ctl
Finished restore at 14-AUG-21

RMAN> quit


Recovery Manager complete.
oracle19c>echo $ORACLE_SID
STDBYDB
oracle19c>sqlplus '/ as sysdba'

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Aug 14 17:52:34 2021
Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> alter database mount;

Database altered.

SQL>  show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      STDBYDB
SQL> select database_role, open_mode
  2  from v$database
  3  /

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY MOUNTED

SQL> select incarnation#, resetlogs_change#, to_char(resetlogs_time,'DD-MON-RR HH24:MI') resetlogs_time,
  2  status, resetlogs_id
  3  from v$database_incarnation
  4  order by 1
  5  /

INCARNATION# RESETLOGS_CHANGE# RESETLOGS_TIME           STATUS  RESETLOGS_ID
------------ ----------------- ------------------------ ------- ------------
           1                 1 17-APR-19 00:55          PARENT    1005785759
           2           1920977 04-MAY-19 23:21          PARENT    1007421686
           3           4797184 27-MAR-20 00:00          CURRENT   1036108814

SQL>


So, this restored Controlfile confirms that I have a Standby Database that is now back at Incarnation#=3.
I can now proceed to restore the database and I assume that I can reapply all the ArchiveLogs from the Primary which is also at Incarnation#=3


SQL> quit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
oracle19c>rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Sat Aug 14 17:56:06 2021
Version 19.12.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCLCDB (DBID=2778483057, not open)

RMAN> restore database;

Starting restore at 14-AUG-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=11 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=129 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00003 to /opt/oracle/oradata/STDBYDB/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00007 to /opt/oracle/oradata/STDBYDB/users01.dbf
channel ORA_DISK_1: reading from backup piece /opt/oracle/FRA/STDBYDB/STDBYDB/backupset/2021_08_13/o1_mf_nnndf_TAG20210813T230720_jkf2ob5r_.bkp
channel ORA_DISK_2: starting datafile backup set restore
channel ORA_DISK_2: specifying datafile(s) to restore from backup set
channel ORA_DISK_2: restoring datafile 00001 to /opt/oracle/oradata/STDBYDB/system01.dbf
channel ORA_DISK_2: restoring datafile 00004 to /opt/oracle/oradata/STDBYDB/undotbs01.dbf
channel ORA_DISK_2: reading from backup piece /opt/oracle/FRA/STDBYDB/STDBYDB/backupset/2021_08_13/o1_mf_nnndf_TAG20210813T230720_jkf2o9j4_.bkp
....
....
....

channel ORA_DISK_2: restored backup piece 1
channel ORA_DISK_2: restore complete, elapsed time: 00:00:36
Finished restore at 14-AUG-21

RMAN> quit


Recovery Manager complete.
oracle19c>sqlplus '/ as sysdba'

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Aug 14 18:01:50 2021
Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      STDBYDB
SQL> select database_role, open_mode
  2  from v$database
  3  /

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY MOUNTED

SQL> select incarnation#, resetlogs_change#, to_char(resetlogs_time,'DD-MON-RR HH24:MI') resetlogs_time,
  2  status, resetlogs_id
  3  from v$database_incarnation
  4  order by 1
  5  /

INCARNATION# RESETLOGS_CHANGE# RESETLOGS_TIME           STATUS  RESETLOGS_ID
------------ ----------------- ------------------------ ------- ------------
           1                 1 17-APR-19 00:55          PARENT    1005785759
           2           1920977 04-MAY-19 23:21          PARENT    1007421686
           3           4797184 27-MAR-20 00:00          CURRENT   1036108814

SQL>


Let my now try to Recover this Database as a Standby using ArchiveLogs from the PrimaryR (after " alter system set log_archive_dest_state_2='enable';" at the Primary)


SQL> select l.group#, l.bytes/1048576, l.status, f.member
  2  from v$standby_log l, v$logfile f
  3  where l.group#=f.group#
  4  order by 1
  5  /

    GROUP# L.BYTES/1048576 STATUS
---------- --------------- ----------
MEMBER
------------------------------------------------------------------------------------------------------------------------------------
         4             200 ACTIVE
/opt/oracle/oradata/STDBYDB/stdbredo01.log

         5             200 ACTIVE
/opt/oracle/oradata/STDBYDB/stdbredo02.dbf


SQL> alter database clear logfile group 4;

Database altered.

SQL> alter database clear logfile group 5;

Database altered.

SQL>
SQL>  select l.group#, l.bytes/1048576, l.status, f.member
  2  from v$standby_log l, v$logfile f
  3  where l.group#=f.group#
  4  order by 1
  5  /

    GROUP# L.BYTES/1048576 STATUS
---------- --------------- ----------
MEMBER
------------------------------------------------------------------------------------------------------------------------------------
         4             200 UNASSIGNED
/opt/oracle/oradata/STDBYDB/stdbredo01.log

         5             200 UNASSIGNED
/opt/oracle/oradata/STDBYDB/stdbredo02.dbf


SQL>
SQL> alter database recover managed standby database using current logfile disconnect from session;

Database altered.

SQL>


STDBYDB takes a few minutes to get all the ArchiveLogs from Sequence#402 onwards and apply them


2021-08-14T18:16:46.242533+08:00
 rfs (PID:24643): Selected LNO:5 for T-1.S-416 dbid 2778483057 branch 1036108814
2021-08-14T18:16:46.279518+08:00
ARC1 (PID:15987): Archived Log entry 19 added for T-1.S-415 ID 0xa7521ccd LAD:1
2021-08-14T18:16:46.408474+08:00
PR00 (PID:23234): Media Recovery Waiting for T-1.S-416 (in transit)
2021-08-14T18:16:46.420169+08:00
Recovery of Online Redo Log: Thread 1 Group 5 Seq 416 Reading mem 0
  Mem# 0: /opt/oracle/oradata/STDBYDB/stdbredo02.dbf


The Primary also now shows :


SQL> l
  1  select dest_id, db_unique_name, status, type, archived_seq#, applied_seq#, synchronized, gap_status
  2  from v$archive_dest_status
  3  where dest_id in (1,2,3)
  4* order by 1
SQL> /

   DEST_ID DB_UNIQUE_NAME                 STATUS    TYPE             ARCHIVED_SEQ# APPLIED_SEQ# SYN GAP_STATUS
---------- ------------------------------ --------- ---------------- ------------- ------------ --- ------------------------
         1 NONE                           VALID     LOCAL                      415            0 NO
         2 STDBYDB                        VALID     PHYSICAL                   415          414 YES NO GAP
         3 STDB2                          VALID     PHYSICAL                   415          405 NO  NO GAP

SQL>

I can also see the ArchiveLogs that were generated from STDBYDB :

oracle19c>pwd
/opt/oracle/archivelog/STDBYDB
oracle19c>ls -latr
total 196820
drwxr-xr-x. 6 oracle oinstall       63 Aug 11 14:25 ..
-rw-r-----. 1 oracle oinstall   269824 Aug 13 22:47 1_394_1036108814.dbf
-rw-r-----. 1 oracle oinstall  4147200 Aug 13 23:02 1_399_1036108814.dbf
-rw-r-----. 1 oracle oinstall     3584 Aug 13 23:02 1_400_1036108814.dbf
-rw-r-----. 1 oracle oinstall    24064 Aug 13 23:02 1_401_1036108814.dbf
-rw-r-----. 1 oracle oinstall     3072 Aug 13 23:03 1_402_1036108814.dbf
-rw-r-----. 1 oracle oinstall     1024 Aug 13 23:13 1_1_1080515513.dbf
-rw-r-----. 1 oracle oinstall 21302784 Aug 14 17:13 1_2_1080515513.dbf
-rw-r-----. 1 oracle oinstall 96683520 Aug 14 17:41 1_3_1080515513.dbf
-rw-r-----. 1 oracle oinstall    14848 Aug 14 17:42 1_4_1080515513.dbf
-rw-r-----. 1 oracle oinstall    45568 Aug 14 18:04 1_405_1036108814.dbf
-rw-r-----. 1 oracle oinstall   703488 Aug 14 18:04 1_404_1036108814.dbf
-rw-r-----. 1 oracle oinstall   486400 Aug 14 18:04 1_403_1036108814.dbf
-rw-r-----. 1 oracle oinstall 20830208 Aug 14 18:04 1_406_1036108814.dbf
-rw-r-----. 1 oracle oinstall  5378048 Aug 14 18:04 1_407_1036108814.dbf
-rw-r-----. 1 oracle oinstall   538112 Aug 14 18:04 1_408_1036108814.dbf
-rw-r-----. 1 oracle oinstall  4228096 Aug 14 18:04 1_409_1036108814.dbf
-rw-r-----. 1 oracle oinstall     4096 Aug 14 18:04 1_410_1036108814.dbf
-rw-r-----. 1 oracle oinstall   197120 Aug 14 18:07 1_411_1036108814.dbf
-rw-r-----. 1 oracle oinstall   176128 Aug 14 18:16 1_414_1036108814.dbf
-rw-r-----. 1 oracle oinstall   489984 Aug 14 18:16 1_412_1036108814.dbf
-rw-r-----. 1 oracle oinstall    91136 Aug 14 18:16 1_413_1036108814.dbf
-rw-r-----. 1 oracle oinstall     4096 Aug 14 18:16 1_415_1036108814.dbf
drwxr-xr-x. 2 oracle oinstall     4096 Aug 14 18:25 .
-rw-r-----. 1 oracle oinstall 45876224 Aug 14 18:25 1_416_1036108814.dbf
oracle19c>

Note how Sequence#1 to Sequence#4 were generated in the new Incarnation with Resetlogs ID 1080515513.  This was the Resetlogs ID assigned when STDBYDB was opened with Failover, thus a New Incarnation and effectively a Resetlogs. Sequence#403 onwards are from the Primary with Resetlogs ID 1036108814
(You can check the Resetlogs IDs shown in the previous blog post)

13 August, 2021

Failover and Database Incarnation

In previous blog posts, I have demonstrated : 

1.  Using Flashback Database for [destructive] testing at a Standby

2.  Using Snapshot Standby Database for [destructive] testing at a Standby



In this blog post, I will demonstrate the Database Incarnation feature when a Failover is executed.


This is the current status at the Primary :


SQL> show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      ORCLCDB
SQL> archive log list

SQL>
SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /opt/oracle/archivelog/ORCLCDB
Oldest online log sequence     401
Next log sequence to archive   403
Current log sequence           403
SQL>

SQL> select incarnation#, resetlogs_change#, to_char(resetlogs_time,'DD-MON-RR HH24:MI') resetlogs_time,
  2  status, resetlogs_id
  3  from v$database_incarnation
  4  order by 1
  5  /

INCARNATION# RESETLOGS_CHANGE# RESETLOGS_TIME           STATUS  RESETLOGS_ID
------------ ----------------- ------------------------ ------- ------------
           1                 1 17-APR-19 00:55          PARENT    1005785759
           2           1920977 04-MAY-19 23:21          PARENT    1007421686
           3           4797184 27-MAR-20 00:00          CURRENT   1036108814

SQL>
SQL> l
  1  select dest_id, db_unique_name, status, type, archived_seq#, applied_seq#, synchronized, gap_status
  2  from v$archive_dest_status
  3  where dest_id in (1,2,3)
  4* order by 1
SQL> /

   DEST_ID DB_UNIQUE_NAME STATUS    TYPE             ARCHIVED_SEQ# APPLIED_SEQ# SYN GAP_STATUS
---------- -------------- --------- ---------------- ------------- ------------ --- ------------------------
         1 NONE           VALID     LOCAL                      402            0 NO
         2 STDBYDB        VALID     PHYSICAL                   402          401 YES NO GAP
         3 STDB2          VALID     PHYSICAL                   402          384 NO  NO GAP

SQL>


And this is the current status at the first Standby (STDBYDB)


From the alert log :

2021-08-13T23:03:00.998407+08:00
Recovery of Online Redo Log: Thread 1 Group 4 Seq 403 Reading mem 0
  Mem# 0: /opt/oracle/oradata/STDBYDB/stdbredo01.log


SQL> show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      STDBYDB
SQL> select incarnation#, resetlogs_change#, to_char(resetlogs_time,'DD-MON-RR HH24:MI') resetlogs_time,
  2  status, resetlogs_id
  3  from v$database_incarnation
  4  order by 1
  5  /

INCARNATION# RESETLOGS_CHANGE# RESETLOGS_TIME           STATUS  RESETLOGS_ID
------------ ----------------- ------------------------ ------- ------------
           1                 1 17-APR-19 00:55          PARENT    1005785759
           2           1920977 04-MAY-19 23:21          PARENT    1007421686
           3           4797184 27-MAR-20 00:00          CURRENT   1036108814

SQL>


I now create a Backup of the Standby (STDBYDB) (for this test I will not be using Flashback Database or Snapshot Standby Database {see the first two blog posts listed at the beginning for demonstrations of both methods}) :


oracle19c>rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Fri Aug 13 23:07:04 2021
Version 19.12.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCLCDB (DBID=2778483057, not open)

RMAN> backup as compressed backupset database;

Starting backup at 13-AUG-21
Starting implicit crosscheck backup at 13-AUG-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=128 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=252 device type=DISK
Crosschecked 10 objects
Crosschecked 11 objects
Finished implicit crosscheck backup at 13-AUG-21

Starting implicit crosscheck copy at 13-AUG-21
using channel ORA_DISK_1
using channel ORA_DISK_2
Finished implicit crosscheck copy at 13-AUG-21

searching for all files in the recovery area
cataloging files...
no files cataloged

using channel ORA_DISK_1
using channel ORA_DISK_2
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=/opt/oracle/oradata/STDBYDB/system01.dbf
input datafile file number=00004 name=/opt/oracle/oradata/STDBYDB/undotbs01.dbf
channel ORA_DISK_1: starting piece 1 at 13-AUG-21
channel ORA_DISK_2: starting compressed full datafile backup set
channel ORA_DISK_2: specifying datafile(s) in backup set
input datafile file number=00003 name=/opt/oracle/oradata/STDBYDB/sysaux01.dbf
input datafile file number=00007 name=/opt/oracle/oradata/STDBYDB/users01.dbf
channel ORA_DISK_2: starting piece 1 at 13-AUG-21


....
....
....


Starting Control File and SPFILE Autobackup at 13-AUG-21
piece handle=/opt/oracle/FRA/STDBYDB/STDBYDB/autobackup/2021_08_13/o1_mf_s_1080515319_jkf2tzhp_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at 13-AUG-21

RMAN>
RMAN> quit


Recovery Manager complete.
oracle19c>



I now execute a FAILOVER command at STDBYDB :


oracle19c>sqlplus '/ as sysdba'

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Aug 13 23:11:08 2021
Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      STDBYDB
SQL> alter database failover to STDBYDB;

Database altered.

SQL>
SQL> shutdown immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup open;
ORACLE instance started.

Total System Global Area 1207955544 bytes
Fixed Size                  9134168 bytes
Variable Size             436207616 bytes
Database Buffers          754974720 bytes
Redo Buffers                7639040 bytes
Database mounted.
Database opened.
SQL> show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      STDBYDB
SQL>
SQL> select incarnation#, resetlogs_change#, to_char(resetlogs_time,'DD-MON-RR HH24:MI') resetlogs_time,
  2  status, resetlogs_id
  3  from v$database_incarnation
  4  order by 1
  5  /

INCARNATION# RESETLOGS_CHANGE# RESETLOGS_TIME           STATUS  RESETLOGS_ID
------------ ----------------- ------------------------ ------- ------------
           1                 1 17-APR-19 00:55          PARENT    1005785759
           2           1920977 04-MAY-19 23:21          PARENT    1007421686
           3           4797184 27-MAR-20 00:00          PARENT    1036108814
           4          14030903 13-AUG-21 23:11          CURRENT   1080515513

SQL>

>
The STDYDB alert log also shows :


2021-08-13T23:11:18.443497+08:00
alter database recover managed standby database cancel
2021-08-13T23:11:18.475672+08:00
PR00 (PID:354): MRP0: Background Media Recovery cancelled with status 16037
2021-08-13T23:11:18.475814+08:00
Errors in file /opt/oracle/diag/rdbms/stdbydb/STDBYDB/trace/STDBYDB_pr00_354.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:354): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 14030850
Stopping change tracking
2021-08-13T23:11:18.858990+08:00
Errors in file /opt/oracle/diag/rdbms/stdbydb/STDBYDB/trace/STDBYDB_pr00_354.trc:
ORA-16037: user requested cancel of managed recovery operation
2021-08-13T23:11:19.004299+08:00
Background Media Recovery process shutdown (STDBYDB)
2021-08-13T23:11:19.477146+08:00
Managed Standby Recovery Canceled (STDBYDB)
Completed: alter database recover managed standby database cancel
2021-08-13T23:11:40.986985+08:00
alter database failover to STDBYDB

2021-08-13T23:11:40.987075+08:00
.... (PID:4479): The Time Management Interface (TMI) is being enabled for role transition
.... (PID:4479): information.  This will result in messages beingoutput to the alert log
.... (PID:4479): file with the prefix 'TMI: '.  This is being enabled to make the timing of
.... (PID:4479): the various stages of the role transition available for diagnostic purposes.
.... (PID:4479): This output will end when the role transition is complete.
TMI: dbsdrv failover to target BEGIN 2021-08-13 23:11:40.987474
Terminal Recovery requested in process 4479
TMI: adbdrv termRecovery BEGIN 2021-08-13 23:11:40.989180
2021-08-13T23:11:41.082414+08:00
Attempt to do a Terminal Recovery (STDBYDB)
TMI: adbdrv termRecovery END 2021-08-13 23:11:41.082472
2021-08-13T23:11:41.083019+08:00
Attempt to do a Terminal Recovery (STDBYDB)
TMI: adbdrv termRecovery END 2021-08-13 23:11:41.082472
2021-08-13T23:11:41.083019+08:00
Media Recovery Start: Managed Standby Recovery (STDBYDB)
 Started logmerger process
2021-08-13T23:11:41.141008+08:00
PR00 (PID:4636): Managed Standby Recovery not using Real Time Apply
2021-08-13T23:11:42.361339+08:00
max_pdb is 5
2021-08-13T23:11:43.009492+08:00
Parallel Media Recovery started with 4 slaves
2021-08-13T23:11:44.585417+08:00
Stopping change tracking
PR00 (PID:4636): Media Recovery Waiting for T-1.S-403 (in transit)
2021-08-13T23:11:45.982280+08:00
PR00 (PID:4636): Killing 3 processes (PIDS:375,391,394) (all RFS, wait for I/O) in order to disallow current and future RFS connections. Requested by OS process 4636
2021-08-13T23:11:45.982627+08:00
Process termination requested for pid 375 [source = rdbms], [info = 2] [request issued by pid: 4636, uid: 54321]
2021-08-13T23:11:45.983224+08:00
Process termination requested for pid 391 [source = rdbms], [info = 2] [request issued by pid: 4636, uid: 54321]
2021-08-13T23:11:45.983244+08:00
Process termination requested for pid 394 [source = rdbms], [info = 2] [request issued by pid: 4636, uid: 54321]
2021-08-13T23:11:47.984399+08:00
PR00 (PID:4636): Begin: SRL archival
PR00 (PID:4636): End: SRL archival
PR00 (PID:4636): Terminal Recovery timestamp is '08/13/2021 23:11:48'
2021-08-13T23:11:49.723810+08:00
PR00 (PID:4636): Terminal Recovery: applying standby redo logs.
PR00 (PID:4636): Terminal Recovery: thread 1 seq# 403 redo required
2021-08-13T23:11:49.887210+08:00
PR00 (PID:4636): Terminal Recovery:
2021-08-13T23:11:50.007728+08:00
Recovery of Online Redo Log: Thread 1 Group 4 Seq 403 Reading mem 0
  Mem# 0: /opt/oracle/oradata/STDBYDB/stdbredo01.log
2021-08-13T23:11:50.853550+08:00
Terminal Recovery finished with No-Data-Loss
2021-08-13T23:11:51.223461+08:00
Incomplete Recovery applied until change 14030902 time 08/13/2021 23:11:45
2021-08-13T23:11:51.329028+08:00
Media Recovery Complete (STDBYDB)
2021-08-13T23:11:52.416801+08:00
Terminal Recovery: successful completion

PR00 (PID:4636): Forcing ARSCN to IRSCN for TR SCN:0x0000000000d61836
PR00 (PID:4636): Attempt to set limbo arscn SCN:0x0000000000d61836 irscn SCN:0x0000000000d61836
PR00 (PID:4636): Resetting standby activation ID 2807176397 (0xa7521ccd)
Stopping change tracking
2021-08-13T23:11:52.625495+08:00
ALTER DATABASE SWITCHOVER TO PRIMARY (STDBYDB)
2021-08-13T23:11:52.626348+08:00
Maximum wait for role transition is 15 minutes.
TMI: kcv_commit_to_so_to_primary wait for MRP to finish BEGIN 2021-08-13 23:11:52.626362
TMI: kcv_commit_to_so_to_primary wait for MRP to finish END 2021-08-13 23:11:52.626450
TMI: kcv_commit_to_so_to_primary Switchover from physical BEGIN 2021-08-13 23:11:52.626957
Backup controlfile written to trace file /opt/oracle/diag/rdbms/stdbydb/STDBYDB/trace/STDBYDB_ora_4479.trc
Standby terminal recovery start SCN: 14030850
RESETLOGS after incomplete recovery UNTIL CHANGE 14030902 time 08/13/2021 23:11:45
NET  (PID:4479): ORL pre-clearing operation disabled by switchover
Online log /opt/oracle/oradata/STDBYDB/redo01.log: Thread 1 Group 1 was previously cleared
Online log /opt/oracle/oradata/STDBYDB/redo02.log: Thread 1 Group 2 was previously cleared
Online log /opt/oracle/oradata/STDBYDB/redo03.log: Thread 1 Group 3 was previously cleared
2021-08-13T23:11:53.667669+08:00
Standby became primary SCN: 14030849
2021-08-13T23:11:53.668267+08:00
Setting recovery target incarnation to 4
2021-08-13T23:11:53.715844+08:00

NET (PID:4479): RT: Role transition work is not done
NET (PID:4479): The Time Management Interface (TMI) is being enabled for role transition
NET (PID:4479): information. This will result in messages beingoutput to the alert log
NET (PID:4479): file with the prefix 'TMI: '. This is being enabled to make the timing of
NET (PID:4479): the various stages of the role transition available for diagnostic purposes.
NET (PID:4479): This output will end when the role transition is complete.
NET (PID:4479): Redo network throttle feature is disabled at mount time
2021-08-13T23:11:53.934288+08:00
NET (PID:4479): Database role cleared from PHYSICAL STANDBY [kcvs.c:1099]
Switchover: Complete - Database mounted as primary
TMI: kcv_commit_to_so_to_primary Switchover from physical END 2021-08-13 23:11:53.935283
TMI: dbsdrv failover to target END 2021-08-13 23:11:53.935372
Failover completed with No-Data-Loss.
Completed: alter database failover to STDBYDB

2021-08-13T23:12:16.059509+08:00
ARC1 (PID:32662): Becoming the 'no SRL' ARCH
2021-08-13T23:12:26.958308+08:00
Shutting down ORACLE instance (immediate) (OS id: 4479)
....
....
....
2021-08-13T23:12:45.501291+08:00
Instance shutdown complete (OS id: 4479)
....
....
....
2021-08-13T23:12:55.538734+08:00
Starting ORACLE instance (normal) (OS id: 5005)
....
....
....
2021-08-13T23:14:09.422553+08:00
CJQ0 started with pid=64, OS id=6101
Completed: ALTER DATABASE OPEN


Meanwhile, the Primary shows that it no longer communicates with STDBYDB


SQL> l
  1  select dest_id, db_unique_name, status, type, archived_seq#, applied_seq#, synchronized, gap_status
  2  from v$archive_dest_status
  3  where dest_id in (1,2,3)
  4* order by 1
SQL> /

   DEST_ID DB_UNIQUE_NAME STATUS    TYPE             ARCHIVED_SEQ# APPLIED_SEQ# SYN GAP_STATUS
---------- -------------- --------- ---------------- ------------- ------------ --- ------------------------
         1 NONE           VALID     LOCAL                      403            0 NO
         2 STDBYDB        ERROR     PHYSICAL                   401          401 NO  RESOLVABLE GAP
         3 STDB2          VALID     PHYSICAL                   403          386 NO  NO GAP

SQL>


from the alert log :
2021-08-13T23:02:58.137970+08:00
NET  (PID:32207): Archived Log entry 956 added for T-1.S-401 ID 0xa7521ccd LAD:1
2021-08-13T23:03:00.379365+08:00
ALTER SYSTEM ARCHIVE LOG
2021-08-13T23:03:01.008064+08:00
Thread 1 advanced to log sequence 403 (LGWR switch),  current SCN: 14029372
  Current log# 1 seq# 403 mem# 0: /opt/oracle/oradata/ORCLCDB/redo01.log
2021-08-13T23:03:01.143735+08:00
NET  (PID:32207): Archived Log entry 959 added for T-1.S-402 ID 0xa7521ccd LAD:1
2021-08-13T23:11:46.487983+08:00
LGWR (PID:3724): Attempting LAD:2 network reconnect (3113)
LGWR (PID:3724): LAD:2 network reconnect abandoned
2021-08-13T23:11:46.488223+08:00
Errors in file /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/trace/ORCLCDB_lgwr_3724.trc:
ORA-03113: end-of-file on communication channel
LGWR (PID:3724): Error 3113 for LNO:1 to 'STDBYDB'
2021-08-13T23:11:46.532708+08:00
LGWR (PID:3724): LAD:2 is UNSYNCHRONIZED
2021-08-13T23:11:48.149800+08:00
LGWR (PID:3724): Failed to archive LNO:1 T-1.S-403, error=3113
LGWR (PID:3724): Error 1041 disconnecting from LAD:2 standby host 'STDBYDB'
2021-08-13T23:11:48.300060+08:00
Thread 1 advanced to log sequence 404 (LGWR switch),  current SCN: 14030909
  Current log# 2 seq# 404 mem# 0: /opt/oracle/oradata/ORCLCDB/redo02.log
2021-08-13T23:11:49.005544+08:00
ARC1 (PID:3853): Archived Log entry 961 added for T-1.S-403 ID 0xa7521ccd LAD:1
2021-08-13T23:13:23.840728+08:00
 rfs (PID:5505): krsr_rfs_atc: Identified database type as 'PRIMARY': Client is Foreground (PID:5166)
 rfs (PID:5505): Database mount ID mismatch [0xa9f74b79:0xa9f7dc16] (2851556217:2851593238)
 rfs (PID:5505): Not using real application clusters


So, now the STDBYDB database has diverged from the Primary at SCN#14030903 and no longer accepts Redo from the Primary.
The STDBYDB database is now at Incarnation#4


I can stop Redo Shipping from the Primary


SQL> show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      ORCLCDB
SQL> select incarnation#, resetlogs_change#, to_char(resetlogs_time,'DD-MON-RR HH24:MI') resetlogs_time,
  2  status, resetlogs_id
  3  from v$database_incarnation
  4  order by 1
  5  /

INCARNATION# RESETLOGS_CHANGE# RESETLOGS_TIME           STATUS  RESETLOGS_ID
------------ ----------------- ------------------------ ------- ------------
           1                 1 17-APR-19 00:55          PARENT    1005785759
           2           1920977 04-MAY-19 23:21          PARENT    1007421686
           3           4797184 27-MAR-20 00:00          CURRENT   1036108814

SQL>
SQL> l
  1  select dest_id, db_unique_name, status, type, archived_seq#, applied_seq#, synchronized, gap_status
  2  from v$archive_dest_status
  3  where dest_id in (1,2,3)
  4* order by 1
SQL> /

   DEST_ID DB_UNIQUE_NAME STATUS    TYPE             ARCHIVED_SEQ# APPLIED_SEQ# SYN GAP_STATUS
---------- -------------- --------- ---------------- ------------- ------------ --- ------------------------
         1 NONE           VALID     LOCAL                      403            0 NO
         2 STDBYDB        ERROR     PHYSICAL                   401          401 NO  RESOLVABLE GAP
         3 STDB2          VALID     PHYSICAL                   403          386 NO  NO GAP

SQL> 
SQL> show parameter log_archive_dest_state_2

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_state_2             string      enable
log_archive_dest_state_20            string      enable
log_archive_dest_state_21            string      enable
log_archive_dest_state_22            string      enable
log_archive_dest_state_23            string      enable
log_archive_dest_state_24            string      enable
log_archive_dest_state_25            string      enable
log_archive_dest_state_26            string      enable
log_archive_dest_state_27            string      enable
log_archive_dest_state_28            string      enable
log_archive_dest_state_29            string      enable
SQL> alter system set log_archive_dest_state_2='DEFER';

System altered.

SQL>

from the alert log
2021-08-13T23:21:58.120088+08:00
ALTER SYSTEM SET log_archive_dest_state_2='DEFER' SCOPE=BOTH;


Notice how v$archive_dest_status at the Primary now show that Sequence#401 is the last Archived and Applied at STDBYDB. Sequence#402 and subsequent ones are no longer relevant to STDBYDB.

You might note that the STDB2 which I have configured as a Lagging Standby (DEST_ID=3) is still receiving Redo from the Primary.  So, that Standby is still active.  (See here and here as to how I configured STDB2 as a second Standby that is lagging)


In the next Blog Post, I will show what happens if I attempt to restore the backup of STDBYDB and resync with the Primary after executing multiple changes (redo) in this database want to discard it, inspite of it now running at a different Incarnation# (Incarnation=4)