03 May, 2022

PDB Flashback in 21c

 Oracle 21c allows you to Flashback a Pluggable Database to a past point-in-time / SCN.


Here is a quick demo  (also with alert log messages from the Primary and Standby) :


On the Primary :


SQL> create pluggable database pdbhkc admin user hemant identified by hemant;

Pluggable database created.

SQL> alter pluggable database pdbhkc open;

Pluggable database altered.

SQL>
SQL> select current_scn from v$database;

CURRENT_SCN
-----------
    3688309

SQL>
SQL> alter session set container=PDBHKC;

Session altered.

SQL> grant dba to hemant;  -- grant is done *after* SCN 3688309

Grant succeeded.

SQL> connect hemant/hemant@pdbhkc
Connected.
SQL>
SQL> create table obj_list as select * from user_objects;

Table created.

SQL> select count(*) from obj_list;  

  COUNT(*)
----------
         0

SQL>
SQL> select current_scn from v$database;

CURRENT_SCN
-----------
    3688878

SQL>
SQL> insert into obj_list select * from user_objects;

1 row created.

SQL> commit;

Commit complete.

SQL> insert into obj_list select * from obj_list;

1 row created.

SQL> commit;

Commit complete.

SQL> insert into obj_list select * from obj_list;

2 rows created.

SQL> commit;

Commit complete.

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
    3689045

SQL>
SQL> quit
Disconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0


Now I will attempt a FLASHBACK to SCN 3688309

[oracle@primary admin]$ rman target /

Recovery Manager: Release 21.0.0.0.0 - Production on Tue May 3 14:18:15 2022
Version 21.3.0.0.0

Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.

connected to target database: DB213H1 (DBID=3870017436)

RMAN> alter pluggable database pdbhkc close;

Statement processed

RMAN>
RMAN> flashback pluggable database pdbhkc to scn 3688309;

Starting flashback at 03-MAY-22
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=152 device type=DISK


starting media recovery
media recovery complete, elapsed time: 00:00:03

Finished flashback at 03-MAY-22

RMAN>
RMAN> alter pluggable database pdbhkc open resetlogs;

Statement processed

RMAN>
RMAN> quit


Recovery Manager complete.
[oracle@primary admin]$ sqlplus hemant/hemant@pdbhkc

SQL*Plus: Release 21.0.0.0.0 - Production on Tue May 3 14:20:01 2022
Version 21.3.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.


Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SQL> select count(*) from user_objects;

  COUNT(*)
----------
         0

SQL>
SQL> select current_scn from v$database;   -- this fails because DBA grant is not yet present
select current_scn from v$database
                        *
ERROR at line 1:
ORA-00942: table or view does not exist


SQL> connect / as sysdba
Connected.
SQL> alter session set container=PDBHKC;

Session altered.

SQL> grant dba to hemant;

Grant succeeded.

SQL> connect hemant/hemant@pdbhkc;
Connected.
SQL> select current_scn from v$database;

CURRENT_SCN
-----------
    3691628

SQL>
SQL> connect / as sysdba
Connected.
SQL> alter system archive log current;

System altered.

SQL>
SQL> col name format a8
SQL> get pdb_incarnations.sql
  1  select pi.con_id, p.name,  pi.pdb_incarnation#,
  2  pi.incarnation_scn, pi.prior_pdb_incarnation#, pi.status
  3  from v$pdb_incarnation pi, v$pdbs p
  4  where pi.con_id=p.con_id
  5* order by 1,2,3,4
SQL> /

    CON_ID NAME     PDB_INCARNATION# INCARNATION_SCN PRIOR_PDB_INCARNATION#                   STATUS
---------- -------- ---------------- --------------- ---------------------------------------- -------
         2 PDB$SEED                0               1                                          PARENT
         2 PDB$SEED                0         2601843 0                                        CURRENT
         3 PDB1                    0               1                                          PARENT
         3 PDB1                    0         2601843 0                                        CURRENT
         4 PDBHKC                  0         2601843                                          PARENT
         4 PDBHKC                  1         3688310 0                                        CURRENT

6 rows selected.

SQL>


Note how the "GRANT DBA" was after the Flashback SCN and had to be regranted. After the FLASHBACK, the custom table "obj_list" does not exist.


Primary Alert Log Entries :

Primary Instance alert log :
==============================
2022-05-03T14:18:28.808263+08:00
alter pluggable database pdbhkc close
2022-05-03T14:18:28.815690+08:00
PDBHKC(4):Pluggable database PDBHKC closing
PDBHKC(4):JIT: pid 7263 requesting stop
PDBHKC(4):Closing sequence subsystem (5915010500).
PDBHKC(4):Buffer Cache flush started: 4
PDBHKC(4):Buffer Cache flush finished: 4
Pluggable database PDBHKC closed
Completed: alter pluggable database pdbhkc close
2022-05-03T14:19:12.776065+08:00
RMAN flashback PDB to scn 3688309
Flashback Restore Start
Restore Flashback Pluggable Database PDBHKC (4) until change 3686342
Flashback Restore Complete
Flashback Media Recovery Start
2022-05-03T14:19:13.675674+08:00
Serial Media Recovery started
max_pdb is 4
2022-05-03T14:19:13.769641+08:00
Recovery of Online Redo Log: Thread 1 Group 1 Seq 28 Reading mem 0
  Mem# 0: /u02/oradata/DB213H1/redo01.log
2022-05-03T14:19:13.970865+08:00
Recovery of Online Redo Log: Thread 1 Group 2 Seq 29 Reading mem 0
  Mem# 0: /u02/oradata/DB213H1/redo02.log
2022-05-03T14:19:14.621235+08:00
Incomplete Recovery applied until change 3688310 time 05/03/2022 14:12:57
Flashback Media Recovery Complete
Flashback Pluggable Database PDBHKC (4) recovered until change 3688310
Completed: RMAN flashback PDB to scn 3688309
2022-05-03T14:19:42.382165+08:00
alter pluggable database pdbhkc open resetlogs
2022-05-03T14:19:42.461760+08:00
Online datafile 15
Online datafile 14
Online datafile 13
PDBHKC(4):Pluggable database PDBHKC pseudo opening
PDBHKC(4):SUPLOG: Initialize PDB SUPLOG SGA, old value 0x0, new value 0x18
PDBHKC(4):Autotune of undo retention is turned on.
PDBHKC(4):Endian type of dictionary set to little
PDBHKC(4):Undo initialization recovery: Parallel FPTR failed: start:5988531 end:5988533 diff:2 ms (0.0 seconds)
PDBHKC(4):Undo initialization recovery: err:0 start: 5988531 end: 5988547 diff: 16 ms (0.0 seconds)
PDBHKC(4):[7263] Successfully onlined Undo Tablespace 2.
PDBHKC(4):Undo initialization online undo segments: err:0 start: 5988547 end: 5988555 diff: 8 ms (0.0 seconds)
PDBHKC(4):Undo initialization finished serial:0 start:5988531 end:5988556 diff:25 ms (0.0 seconds)
PDBHKC(4):Database Characterset for PDBHKC is AL32UTF8
PDBHKC(4):Pluggable database PDBHKC pseudo closing
PDBHKC(4):JIT: pid 7263 requesting stop
PDBHKC(4):Closing sequence subsystem (5988605782).
PDBHKC(4):Buffer Cache flush started: 4
PDBHKC(4):Buffer Cache flush finished: 4
PDBHKC(4):Pluggable database PDBHKC opening in read write
PDBHKC(4):SUPLOG: Initialize PDB SUPLOG SGA, old value 0x0, new value 0x18
PDBHKC(4):Autotune of undo retention is turned on.
PDBHKC(4):Endian type of dictionary set to little
PDBHKC(4):Undo initialization recovery: Parallel FPTR complete: start:5989023 end:5989027 diff:4 ms (0.0 seconds)
PDBHKC(4):Undo initialization recovery: err:0 start: 5989023 end: 5989027 diff: 4 ms (0.0 seconds)
PDBHKC(4):[7263] Successfully onlined Undo Tablespace 2.
PDBHKC(4):Undo initialization online undo segments: err:0 start: 5989027 end: 5989188 diff: 161 ms (0.2 seconds)
PDBHKC(4):Undo initialization finished serial:0 start:5989023 end:5989194 diff:171 ms (0.2 seconds)
PDBHKC(4):Pluggable database PDBHKC dictionary check beginning
2022-05-03T14:19:43.379867+08:00
PDBHKC(4):Pluggable Database PDBHKC Dictionary check complete
PDBHKC(4):Database Characterset for PDBHKC is AL32UTF8
2022-05-03T14:19:43.935738+08:00
PDBHKC(4):SUPLOG: Set PDB SUPLOG SGA at PDB OPEN, old 0x18, new 0x0 (no suplog)
2022-05-03T14:19:44.385306+08:00
PDBHKC(4):Opening pdb with no Resource Manager plan active
2022-05-03T14:19:44.967205+08:00
Control autobackup written to DISK device

handle '/u01/app/oracle/DB213H1/autobackup/2022_05_03/o1_mf_s_1103725184_k71ld0sw_.bkp'

Pluggable database PDBHKC closed
Completed: alter pluggable database pdbhkc open resetlogs
2022-05-03T14:21:30.693825+08:00
2022-05-03T14:21:30.693825+08:00
PDBHKC(4):TABLE AUDSYS.AUD$UNIFIED: ADDED INTERVAL PARTITION SYS_P328 (2864) VALUES LESS THAN (TIMESTAMP' 2022-05-04 00:00:00')
2022-05-03T14:23:50.678383+08:00
ALTER SYSTEM ARCHIVE LOG
2022-05-03T14:23:50.737171+08:00
Thread 1 advanced to log sequence 30 (LGWR switch),  current SCN: 3691895
  Current log# 3 seq# 30 mem# 0: /u02/oradata/DB213H1/redo03.log
2022-05-03T14:23:50.840999+08:00
Deleted Oracle managed file /u01/app/oracle/DB213H1/archivelog/2022_04_20/o1_mf_1_9_k5yopcl1_.arc
2022-05-03T14:23:50.843496+08:00
NET  (PID:7571): Archived Log entry 29 added for B-1102494684.T-1.S-29 ID 0x7f6ee6abae9c LAD:1 [krse.c:4933]
2022-05-03T14:23:50.879882+08:00
Deleted Oracle managed file /u01/app/oracle/DB213H1/archivelog/2022_04_20/o1_mf_1_10_k5yos1cc_.arc


Note the messages 
"RMAN flashback PDB to scn 3688309" 
"Restore Flashback Pluggable Database PDBHKC (4) until change 3686342" 
"Incomplete Recovery applied until change 3688310" 
"Completed: RMAN flashback PDB to scn 3688309"



Standby Alert Log Entries :

Standby Instance alert log :
============================
2022-05-03T14:19:42.855709+08:00
PDBHKC(4):PDB Side Media Recovery started for pdbid(4)
PDBHKC(4):.... (PID:6103): Managed Recovery starting Real Time Apply [krsm.c:15865]
PDBHKC(4):max_pdb is 4
PDBHKC(4):Datafile 15 (ckpscn 3690771) is orphaned on PDB incarnation#=1
PDBHKC(4):Datafile 14 (ckpscn 3690771) is orphaned on PDB incarnation#=1
PDBHKC(4):Datafile 13 (ckpscn 3690771) is orphaned on PDB incarnation#=1
PDBHKC(4):PSR[4]: the incarnation of data files don't match, so we are trying to fetch data files again from the primary database.
2022-05-03T14:19:42.962411+08:00
PDBHKC(4):Media Recovery failed with error 65138
PDBHKC(4):ORA-65138 is signaled while preparing the media recovery.
2022-05-03T14:19:43.506242+08:00
PDBHKC(4):Applying tablespace dictionary check redo for tablespace #0
PDBHKC(4):Applying tablespace dictionary check redo for tablespace #1
PDBHKC(4):Applying tablespace dictionary check redo for tablespace #2
PDBHKC(4):Applying tablespace dictionary check redo for tablespace #3
PDBHKC(4):Applying tablespace dictionary check redo for tablespace #4
PDBHKC(4):Applying datafile dictionary check redo for datafile #13
PDBHKC(4):Applying datafile dictionary check redo for datafile #14
PDBHKC(4):Applying datafile dictionary check redo for datafile #15
2022-05-03T14:19:45.892494+08:00
PDBHKC(4):Full restore complete of datafile 15 /u02/oradata/DB213H1_STDBY/DE155F0301020A30E0538338A8C07AFA/datafile/o1_mf_undotbs1_k71kvzq4_.dbf.  Elapsed time: 0:00:00
PDBHKC(4):  checkpoint is 3691257
PDBHKC(4):  last deallocation scn is 3
PDBHKC(4):Datafile #15 has been copied to the standby.
2022-05-03T14:19:49.845284+08:00
PDBHKC(4):Full restore complete of datafile 14 /u02/oradata/DB213H1_STDBY/DE155F0301020A30E0538338A8C07AFA/datafile/o1_mf_sysaux_k71kvzq3_.dbf.  Elapsed time: 0:00:03
PDBHKC(4):  checkpoint is 3691259
PDBHKC(4):  last deallocation scn is 1071312
PDBHKC(4):Datafile #14 has been copied to the standby.
2022-05-03T14:19:53.340847+08:00
PDBHKC(4):Full restore complete of datafile 13 /u02/oradata/DB213H1_STDBY/DE155F0301020A30E0538338A8C07AFA/datafile/o1_mf_system_k71kvzpm_.dbf.  Elapsed time: 0:00:03
PDBHKC(4):  checkpoint is 3691268
PDBHKC(4):  last deallocation scn is 2645444
PDBHKC(4):Datafile #13 has been copied to the standby.
2022-05-03T14:19:53.354185+08:00
PDBHKC(4):PDB Side Media Recovery started for pdbid(4)
PDBHKC(4):.... (PID:6103): Managed Recovery starting Real Time Apply [krsm.c:15865]
PDBHKC(4):max_pdb is 4
PDBHKC(4):.... (PID:6103): Media Recovery Waiting for T-1.S-29 (in transit) [krsm.c:6185]
2022-05-03T14:19:53.419125+08:00
PDBHKC(4):Media Recovery of Online Log [Thread=1, Seq=29]
2022-05-03T14:19:53.419332+08:00
PDBHKC(4):Recovery of Online Redo Log: Thread 1 Group 5 Seq 29 Reading mem 0
PDBHKC(4):  Mem# 0: /u02/oradata/DB213H1_STDBY/onlinelog/o1_mf_5_k5yt8yjw_.log
PDBHKC(4):PSR[4]:influxScn=3691275,nxtCkptScn=3691275,minScn=3691268,krdsini=0,krsmrs=9082,nof=0,kcvrlc=1102494684,kcvrls=2601843
PDBHKC(4):The merge request has been submitted by side recovery for pdbid 4
2022-05-03T14:19:53.462873+08:00
PR00 (PID:1896): Background Media Recovery cancelled with status 16037 [krd.c:27039]
PSR: Background Media Recovery will be reactivated in an attempt to take over PDB side recovery sessions
2022-05-03T14:19:53.466115+08:00
Errors in file /u01/app/oracle/diag/rdbms/db213h1_stdby/DB213H1/trace/DB213H1_pr00_1896.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:1896): Managed Recovery not using Real Time Apply [krsm.c:15876]
Recovery interrupted!
Recovered data files to a consistent state at change 3691276
Stopping change tracking
2022-05-03T14:19:53.679424+08:00
Errors in file /u01/app/oracle/diag/rdbms/db213h1_stdby/DB213H1/trace/DB213H1_pr00_1896.trc:
ORA-16037: user requested cancel of managed recovery operation
2022-05-03T14:19:54.506794+08:00
PDBHKC(4):.... (PID:6103): Side Recovery Complete [krds.c:1584]
2022-05-03T14:20:14.680123+08:00
all data files of pdbid 4 are brought online.
 Started logmerger process
2022-05-03T14:20:14.739054+08:00

IM on ADG: Start of Empty Journal

IM on ADG: End of Empty Journal
PR00 (PID:6144): Managed Recovery starting Real Time Apply [krsm.c:15865]
max_pdb is 4
2022-05-03T14:20:14.902709+08:00
Parallel Media Recovery started with 4 slaves
2022-05-03T14:20:15.022317+08:00
Stopping change tracking
PR00 (PID:6144): Media Recovery Waiting for T-1.S-29 (in transit) [krsm.c:6185]
2022-05-03T14:20:15.207572+08:00
Recovery of Online Redo Log: Thread 1 Group 5 Seq 29 Reading mem 0
  Mem# 0: /u02/oradata/DB213H1_STDBY/onlinelog/o1_mf_5_k5yt8yjw_.log
2022-05-03T14:23:50.977661+08:00
 rfs (PID:2029): Opened LNO:4 for DBID:3870017436 B-1102494684.T-1.S-30 [krsr.c:18143]
2022-05-03T14:23:50.978472+08:00
ARC1 (PID:1786): Archived Log entry 11 added for B-1102494684.T-1.S-29 ID 0x7f21e6abae9c LAD:1 [krse.c:4933]
2022-05-03T14:23:51.046002+08:00
PR00 (PID:6144): Media Recovery Waiting for T-1.S-30 (in transit) [krsm.c:6185]
2022-05-03T14:23:51.046688+08:00
Recovery of Online Redo Log: Thread 1 Group 4 Seq 30 Reading mem 0
  Mem# 0: /u02/oradata/DB213H1_STDBY/onlinelog/o1_mf_4_k5yt8y5f_.log


Note the messages 
"PDBHKC(4):PSR[4]: the incarnation of data files don't match, so we are trying to fetch data files again from the primary database."
"PDBHKC(4):Media Recovery failed with error 65138"
"PDBHKC(4):ORA-65138 is signaled while preparing the media recovery."
"PDBHKC(4):Full restore complete of datafile ..<fileid>" for each datafile in the PDB
"PDBHKC(4):Datafile #<fileid> has been copied to the standby." for each datafile in the PDB
"PDBHKC(4):PDB Side Media Recovery started for pdbid(4)"
"PDBHKC(4):.... (PID:6103): Managed Recovery starting Real Time Apply [krsm.c:15865]"
"PDBHKC(4):PSR[4]:influxScn=3691275,nxtCkptScn=3691275,minScn=3691268,krdsini=0,krsmrs=9082,nof=0,kcvrlc=1102494684,kcvrls=2601843"
"PDBHKC(4):The merge request has been submitted by side recovery for pdbid 4"
"PR00 (PID:1896): Background Media Recovery cancelled with status 16037 [krd.c:27039]"
"PSR: Background Media Recovery will be reactivated in an attempt to take over PDB side recovery sessions"
"ORA-16037: user requested cancel of managed recovery operation"
"PR00 (PID:1896): Managed Recovery not using Real Time Apply [krsm.c:15876]"
"PDBHKC(4):.... (PID:6103): Side Recovery Complete [krds.c:1584]"
"all data files of pdbid 4 are brought online."
" Started logmerger process" 
"PR00 (PID:6144): Managed Recovery starting Real Time Apply [krsm.c:15865]"

These messages would relate to the new 12c feature "PDB Recovery Isolation" whereby Oracle DataGuard detects that one PDB has to be recovered separately -- i.e. using a separate background session.  (this seems to be called "side recovery" in the messages)

Here are the ORA error descriptions :


65138, 00000, "Data file %s of pluggable database %s belongs to an orphan PDB incarnation."
// *Cause: Either the specified data file was restored from a backup that was
//         taken during a period of time that was  discarded by
//         a RESETLOGS operation, or Oracle could not identify which pluggable
//         database incarnation the file belongs to. One would also encounter
//         this error if a pluggable database (PDB) flashback or restore was
//         performed to an orphan PDB incarnation followed by complete
//         recovery of the PDB. The alert log contains more information.
// *Action: Restore a backup of this file that belonged to either the current
//          or a prior incarnation of the pluggable database.  If you are using
//          RMAN to restore, RMAN will automatically select a correct backup.
//          If you are using Flashback, then perform flashback to a point
//          prior to the start of the orphan incarnation.


16037, 00000, "user requested cancel of managed recovery operation"
// *Cause:  The managed standby database recovery operation has been
//          canceled per user request.
// *Action: No action is required.
//

16037 is a standard message because Oracle stops media recovery for a while.

However, this behaviour indicates that Oracle actually attempts to copy the entire PDB from the Primary to the Standby.  It would be interesting to see how it works with an available backup of the PDB at the Standby -- particularly so if the PDB is very large and copy-over-the-network would take time.

No comments: