:

ORA-00600, arguments: [3020]

Platform: Windows Server 2003, SP2
Oracle: 11.0.2.0, patchset 10

The Problem

From our PROD DB servers we have two data guard DBs up and running as physical standby DBs; one very nearby and another one across the water in another country. These standby DBs were running along very happily, shipping logs from the primary DB to the standby and applying the logs as it went along.

We had to carry out some work on PROD to shrink the datafiles in one of the tablesapces (UNDO) because it had been increased massively for a large backload that we were running to populate our data warehouse. After performing the shrink operation we noticed that the data guard DBs were not applying the logs and we found ORA-00600 errors in the alert logs:

Errors in file D:\ORADATA\GENERAL\ADMIN\diag\rdbms\sid\trace\pr03_5568.trc (incident=30250):

ORA-00600: internal error code, arguments: [3020], [3], [8760], [12591672], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 3, block# 8760, file offset is 71761920 bytes)
ORA-10564: tablespace UNDOTBS1
ORA-01110: data file 3: ‘D:\ORADATA\UNDO\UNDOTBS01.DBF’
ORA-10560: block type ‘KTU SMU HEADER BLOCK’
Incident details in: D:\ORADATA\GENERAL\ADMIN\diag\rdbms\incident\incdir_30250\dw3_pr03_5568_i30250.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Slave exiting with ORA-600 exception

The Cause

This is BUG 11689702 – ORA-00600: INTERNAL ERROR CODE, ARGUMENTS: [3020], [8], [883263], [883263]

Metalink ID 1302614.1 has full description
“Rman or User Managed Restore/Recovery Fails With Ora-600 [3020] if Datafile resize Operation was Performed”

The Solution

1. Stop recovery on physical standby DB

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

2. Find out which datafiles are affected, in my case it was the UNDOTBS01.DBF datafile as it says in the alert log.

3. Backup the files on PROD and restore them to your standy DB. I tried that but it didn’t work as there was more than one datafile affected. I knew that it was only the UNDOTBS1 tablespace which was affected so I backed up and restored the whole tablespace, here are the commands i used for both the single datafile attempt first time round which was unsuccessful and the tablespace restore which worked successfully.

Attempt 1 – Single Datafile Restore

— Run on PROD RMAN
run
{
backup datafile 3 tag = ‘undobkp_03’;
}

— on Standby DB SQLPLUS
alter database recover managed standby database cancel;
shutdown immediate
startup mount

— OS level on standby system

Move original files which you are about to restore and put them somewhere safe
Copy RMAN backup files from PROD to Data Guard server

— RMAN on standby DB
catalog backuppiece ‘D:\oradata\RMAN\DB_20111109_353_766755073’;
run
{
restore database from tag = ‘undbkp_03’;
}

*** THIS DID NOT WORK FOR ME AS I NEEDED TO RESTORE ADDITIONAL DATAFILES ***

Attempt 2 – Tablespace Restore

— Run on PROD RMAN
run
{
backup tablespace ‘UNDOTBS1’ tag = ‘undotbs_bkp’;
}

— on Standby DB SQLPLUS
alter database recover managed standby database cancel;
shutdown immediate
startup mount

— OS level on standby system
Move original files which you are about to restore and put them somewhere safe
Copy RMAN backup files from PROD to Data Guard server

— RMAN on standby DB
catalog start with ‘D:\oradata\RMAN\DW3_20111109’;

— RMAN on standby DB
run
{
restore tablespace ‘UNDOTBS1’ from tag = ‘undotbs_bkp’;
}
EXIT

— on standby DB, SQLPLUS
alter database recover managed standby database disconnect from session;

Like it, share it...

Category: ORA 600


Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *