I was working on Real application testing on an Exadata box and we faced following error while running the test.The changes made to the database:
ORA-00600: internal error code, arguments: [kdBlkCheckError], [75], [409240], [18018], [], [], [], [], [], [], [], []
On examining the alert log I noticed following error.This error is pointing for corruption in tablespace ABC.There is no segment name or segment owner reported ,so my initial thought was that the corroupt block is actually a free space in this locally managed tablespace. I ran dbverify,rman validate and some other tests to identify the location of this corrupt block but all in vain .Then I looked at the trace file to find some more details of the corruption.The corruption actually happened at the time of flashback and it was in the bimap free space.The workaround was to restart the flashback and rebuild the bitmap segment.
arg[a] --- File #
arg[b] --- Block #
arg[c] --- Application
So in case you are looking for a solution of other ORA-600 issues, you should be searching with ORA 600 and arg[c].
- We flashed back the database to guaranteed restore point after doing a Database replay.
ORA-00600: internal error code, arguments: [kdBlkCheckError], [75], [409240], [18018], [], [], [], [], [], [], [], []
On examining the alert log I noticed following error.This error is pointing for corruption in tablespace ABC.There is no segment name or segment owner reported ,so my initial thought was that the corroupt block is actually a free space in this locally managed tablespace. I ran dbverify,rman validate and some other tests to identify the location of this corrupt block but all in vain .Then I looked at the trace file to find some more details of the corruption.The corruption actually happened at the time of flashback and it was in the bimap free space.The workaround was to restart the flashback and rebuild the bitmap segment.
Corrupt Block Found
TSN = 12, TSNAME = ABC
RFN = 75, BLK = 409240, RDBA = -998346247
OBJN = 2, OBJD = -1, OBJECT = , SUBOBJECT =
SEGMENT OWNER = , SEGMENT TYPE =
- The ORA -600 error number is explained as below
arg[a] --- File #
arg[b] --- Block #
arg[c] --- Application
So in case you are looking for a solution of other ORA-600 issues, you should be searching with ORA 600 and arg[c].
Workaround
1)
srvctl stop instance -d rac -i rac1
srvctl start instance -d rac -i rac1 -o MOUNT
sqlplus / as sysdba
Alter Database Flashback OFF;
Alter Database Flashback ON;
exit
srvctl stop instance -d rac -i rac1
srvctl start database -d rac1
2) EXECUTE DBMS_SPACE_ADMIN.TABLESPACE_REBUILD_BITMAPS('ABC');
5 comments:
Excellent blog we just came across this issue during an upgrade and there is no workaround defined on Oracle support apart from fixed in 11.2.0.4 which is not out for our platform yet!
Lifesaver - many thanks again
I am glad the solution was helpful for you.
just another "me too" comment here. We had a two node database that was crashing with this error. We opened a sev 1 request with Oracle support but before they could finish analyzing the ADR bundle I was able to restore service using a modified version of this procedure.
Because both instances were crashing whenever we tried to start them, I had to modify the steps, as follows:
1. tail -100f alert.log
2. srvctl start database -d rac -o mount
3. sqlplus / as sysdba
4. As soon "alter database open" shows in the alert log, issue "alter database flashback off;" (should return "database altered")
5. both instances will terminate again
6. srvctl start database -d rac (normal mode this time)
7. Again, as soon as "alter database open" shows in alert log, issue "exec dbms_space_admin.tablespace_rebuild_bitmap('ABC');"
8. Instances may or may not crash. Mine did and I had to restart them again.
9. With the database open, issue "alter database flashback on;" and it should report "database altered"
10. One of my instances crashed again, but I started it again and it joined the cluster gracefully
11. The alert log continued to report the corrupt block error. I started rman and issued "recover datafile 9 block 123456;" (using the numbers that had been reported previously in the alert log).
This resolved the errors and everything is back to normal.
Some clarification and correction on my previous comment:
step 2 should be "srvctl start database -d rac"
Step 7 should have "exec dbms_space_admin.tablespace_rebuild_bitmaps('ABC');" (was missing the 's' in bitmaps)
And be aware that this procedure might have to be repeated more than once to get it right. I don't know why.
@ajwatt..Thanks for posting the detailed steps that you used .Sometimes different solutions work at different places.
Post a Comment