Friday, March 11, 2016

How to check the flash cache hit percent


select snap_id, end_interval,
       round((mf_total/mt_total)*100,1)      as "FC Read Hit PCT",
       TO_CHAR(mt_total,  '999,999,999,999') as "Total read IO",
       TO_CHAR(mf_total,  '999,999,999,999') as "Read Cache IO",
       TO_CHAR(mwio_total,'999,999,999,999') as " Total Write IO Req",
       TO_CHAR(wb_total,  '999,999,999,999') as "WBFC Total"
from
(with snap as
    (select snap_id, end_interval, stat_name, total - lag(total,1,0) over(partition by stat_name order by snap_id) as total
       from
        (select st.snap_id, TO_CHAR((cast(sp.end_interval_time as date)),'MON-DD-YYYY hh24 mi') End_interval, st.stat_name, sum(st.value) Total
           from dba_hist_snapshot sp,dba_hist_sysstat st
          where sp.snap_id=st.snap_id and sp.instance_number=st.instance_number
            and st.stat_name in ('cell flash cache read hits'
                                ,'physical read total IO requests'
                                ,'cell writes to flash cache'
                                ,'physical write IO requests')
            and  cast(sp.end_interval_time as date) > (sysdate -&num_hrs/24 - 1/96)
          group by st.snap_id, TO_CHAR((cast(sp.end_interval_time as date)),'MON-DD-YYYY hh24 mi'),st.stat_name
          order by st.snap_id)
     )
 select * from snap
  where snap_id > (select min(snap_id) from snap)
    and total > 0
)
pivot (sum(total) as total for (stat_name) in ('physical read total IO requests' as mt
                                              ,'cell flash cache read hits'      as mf
                                              ,'physical write IO requests'      as mwio
                                              ,'cell writes to flash cache'      as wb))

 order by snap_id;

Wednesday, December 2, 2015

How to run a sql script on all the nodes of the exadata box and all running instances.


This method can be used to run a sql script in the "connect" file on all the running instances in the exadata cluster.There can be many ways of doing it .I am highlighting one of the ways of doing it as below.
The first step is to create the following two files containing our scripts.The first script command_20151130.sh contains the OS command to get the instance name and to run the second script.The second script named "connect" contains the actual sqls that we want to run on each of the running instances on the exadata machine

$cat command_20151130.sh
ps -ef |grep ora_pmon | grep -v grep | awk '{print $NF}' | cut -d '_' -f3 | while read line; do /tmp/connect_20151130.sh $line; done

$cat connect.sh
export ORACLE_SID=$1
export ORAENV_ASK=NO
. oraenv
sqlplus -s / as sysdba << EOF
select instance_name from v\$instance;
show parameter local_listener
exit;
EOF

Now since we have the OS command and the sql script prepared following steps can be used .

#Step 1. Following command can be used to copy the above 2 files to all the nodes in the /tmp or any other location of the choice.

$ for f in $( cat ~/dbs_group ) ; do scp connect.sh,command.sh oracle@$f:/tmp; done

#Step 2. Change the permission of these files on all the nodes to make them executatble

dcli -l oracle -g ~/dbs_group chmod u+x /tmp/connect.sh ; chmod u+x command_20151130.sh

#Step 3 .Run the final command  as following on all the nodes.

$dcli -l oracle -g dbs_group /tmp/command_20151130.sh

#Step 4.  After it's all done it is good idea to clean up the /tmp dir on each db node to free up the space.

$dcli -g ~/dbs_group -l oracle rm /tmp/connect_20151130.sh ; rm /tmp/command_20151130.sh


Thursday, September 10, 2015

ADD SCHEMATRANDATA failing with OGG-06522 or OGG-01783


This issue can happen when you are trying to run "ADD SCHEMATRANDATA" data from the golden gate command line interface.Most probably this is the new setup and you are trying to configure goldengate  for the first time on this server.

Following is the example of the error message.

OGG-06522  Cannot verify existence of table function that is required to ADD schema level supplemental logging, failed.

  1. In this case the first thing that you want to check is if you have all the required patches to support the SCHEMATRANDATA in oracle golden gate.If you have the MOS access following note will help you to identify all the required patches


DB Patches Needed to Support SCHEMATRANDATA in OGG (Oracle GoldenGate) (Doc ID 1426440.1)


2. If you have all the required patches and still you are getting the error while running the "ADD SCHEMATRANDATE" command then you need to check if there are any invalid objects inside the database.


You can run the $ORACLE_HOME/rdbms/admin/utlrp.sql to compile any invalid objects in the database.

3. If the issue occurs again then you may want to run sql trace on the goldengate session .

If the problem reproduces then enable the sql trace on the GG session using the following commands from sqlplus (Run ADD SCHEMATRANDATA in the gg sesssion) and analyze the trace files.

Specify OS Process ID.
SQL> oradebug setospid
SQL> oradebug unlimit
Enable SQL trace on that OS Process ID.
SQL> oradebug event 10046 trace name context forever, level 12


In the trace file you will be able to find the offending sql and you can take action based on that .You can also contact oracle support with the trace files.

If this solution fixed your issue or you need any help please leave a comment below.

Thanks !!

Sunday, September 6, 2015

Oracle Orphaned Sessions: Killed blocking session is still in the Killed state and the corresponding row is no more found in the v$process

Hello folks,

I saw this interesting issue some time back .The problem in this case was that there were many sessions waiting for the lock on an object and there was a terminal blocker blocking these sessions.

To resolve this issue DBA did what he was supposed to do .The terminal blocker was killed using the commands "alter system kill session...." and the session was marked for kill.After some time the session was showing in the killed state but the session was still lingering around and it was never actually released out of the database.

Now to kill the session old ways we tried to get the corresponding process in the v$process view but there was no entry for matching the paddr column of v$session to the addr column of the v$process.

That is no matching row for that session in the v$process.So there is no way we can kill the process using the commands kill -9 .

The following query for that session will show no rows because there is no corresponding entry in the v$process view

SELECT s.sid, s.serial#, p.spid
FROM v$session s,v$process p
WHERE s.paddr = p.addr and s.sid=
/


If you take a sytemstate dump of the entire database this session will show up under ORPHAN SESSIONS (upper case if you are on unix based platform).

For some reason pmon was not able to cleanup this session and it is still there in the database as orphaned session.

The most irritating part of this problem is that this is locking many objects seen from select count(*) from v$locked_objects where sid= .

Now how to get rid of this session whose corresponding row is not there in the v$process.


SOLUTION


There are two solutions to this problem depending on what works best and is easy in your case.

1) Bouncing the database.

Most obvious solution anyone can think of is bouncing the database.This will no doubt fix this problem but it may not be acceptable in a prod environment to bounce the database just to get rid of an orphaned process.

2) Get help from oracle Doc ID 387077.1.

This is what I ended up doing.If you are in luck and you are on 11g then you can use this MOS note to try to resolve your issue.


Basically what happens when a session gets killed then a new process id gets created and the addr row  in v$process matching the addr row for the session which was there earlier will no longer be there.Instead a new row will be there.To uniquely identify the process and to fix this issue in 11g additional columns have been added to V$SESSION :

V$SESSION
CREATOR_ADDR - state object address of creating process
CREATOR_SERIAL# - serial number of creating process
CREATOR_ADDR is the column that can be joined with the ADDR column in V$PROCESS to uniquely identify the killed process corresponding to the former session.


Using these columns we can identify the process id of the killed session.I
n order to identify the new process id you can use the following query.

select * from v$process where addr=(select creator_addr from v$session where sid=);

Once you get the process id from the above query rest of the task is simple.Now we just need to kill the corresponding process id from the OS.

-> Kill -9

Now be patient , it will take some time for the pmon process to finally do the cleanup.

If this is the issue with the stuck cleanup of the killed session then killing the OS process will help to restart the cleanup.

In some cases it may not be able to cleanup and you may need to bounce the database.

If this solution works for you its great .

If not then you know what to do next =>BOUNCE !!

Good Luck!

Thursday, September 25, 2014

Force the SQL to use a plan using SQL baseline

This is an example where we see how to force a particular execution plan.It is useful in the cases when there are multiple execution plans for the SQL and its not using optimal execution plan or we want to force a hint to be used to improve the execution plan but can't use it with the packaged application.

So here we go..Following are the steps that can be followed to test this out..



Step 1) Create the test table and load the data to create the test case.

SQL> create table shas ( id number  primary key,name varchar2(128) );

Table created.

SQL>
SQL> insert into shas select OBJECT_ID,OBJECT_NAME from dba_objects;
 72884 rows created.

SQL> commit;

Commit complete.

Step 2) For testing purpose I am going to use the following SQL which is doing an Index  unique scan and I want to force it to do a full table scan.

SQL>  select * from shas where id='75322';

        ID
----------
NAME
--------------------------------------------------------------------------------
     75322
EMP



Execution Plan
----------------------------------------------------------
Plan hash value: 1793678797

--------------------------------------------------------------------------------
------------

| Id  | Operation                   | Name         | Rows  | Bytes | Cost (%CPU)
| Time     |

--------------------------------------------------------------------------------
------------

|   0 | SELECT STATEMENT            |              |     1 |    79 |     1   (0)
| 00:00:01 |

|   1 |  TABLE ACCESS BY INDEX ROWID| SHAS         |     1 |    79 |     1   (0)
| 00:00:01 |

|*  2 |   INDEX UNIQUE SCAN         | SYS_C0012285 |     1 |       |     1   (0)
| 00:00:01 |

--------------------------------------------------------------------------------
------------


Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("ID"=75322)


Step 3) Suppose I want to force the full table scan hint(although not a very desired option in this case). I run the following hinted SQL

SQL> select /*+ FULL(shas) */ * from shas where id ='75322';

        ID
----------
NAME
--------------------------------------------------------------------------------
     75322
EMP



Execution Plan
----------------------------------------------------------
Plan hash value: 622179821

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |     1 |    79 |     2   (0)| 00:00:01 |
|*  1 |  TABLE ACCESS FULL| SHAS |     1 |    79 |     2   (0)| 00:00:01 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("ID"=75322)

Step 4) Get the SQL id from the v$sql_text.
 

SQL> select sql_text,sql_id from v$sql where sql_text like '%75322%';

SQL_TEXT
--------------------------------------------------------------------------------
SQL_ID
-------------
 select * from shas where id='75322'
5zfadn00x105u

select /*+ FULL(shas) */ * from shas where id ='75322'
d8kdxq5baunx4


Step 6) Load the baseline doing Index unique scan from the cursor cache.

var res number ;
exec :res := dbms_spm.load_plans_from_cursor_cache(sql_id => '5zfadn00x105u', plan_hash_value => '1793678797' );

SQL> select sql_text, sql_handle, plan_name, enabled, accepted from dba_sql_plan_baselines;

SQL_TEXT
--------------------------------------------------------------------------------
SQL_HANDLE                     PLAN_NAME                      ENA ACC
------------------------------ ------------------------------ --- ---
 select * from shas where id='75322'
SYS_SQL_9d09bd8936f5f4b0       SQL_PLAN_9u2dxj4vgbx5h230cac57 YES YES



SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
SQL_ID  d8kdxq5baunx4, child number 0
-------------------------------------
select /*+ FULL(shas) */ * from shas where id ='75322'

Plan hash value: 622179821

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |    95 (100)|          |
|*  1 |  TABLE ACCESS FULL| SHAS |   282 | 22278 |    95   (0)| 00:00:02 |

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("ID"=75322)


18 rows selected.

Step 7) Force the SQL id to use the PHV 622179821(Full table scan)


var res number
exec :res := dbms_spm.load_plans_from_cursor_cache( -
sql_id => 'd8kdxq5baunx4', -
plan_hash_value => '622179821', -
sql_handle => 'SYS_SQL_9d09bd8936f5f4b0');

Step 8) Drop the original baseline with the "Index unique scan plan"

SQL> exec :res :=DBMS_SPM.DROP_SQL_PLAN_BASELINE ('&original_sql_handle','&original_plan_name');
Enter value for original_sql_handle: SYS_SQL_9d09bd8936f5f4b0
Enter value for original_plan_name: SQL_PLAN_9u2dxj4vgbx5h230cac57

PL/SQL procedure successfully completed.

Step 7) Test that the baseline is being used and the SQL is doing a full table scan without a hint

SQL> set autotrace on explain
SQL> select * from shas where id ='75322';

        ID
----------
NAME
--------------------------------------------------------------------------------
     75322
EMP



Execution Plan
----------------------------------------------------------
Plan hash value: 622179821

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |   282 | 22278 |    95   (0)| 00:00:02 |
|*  1 |  TABLE ACCESS FULL| SHAS |   282 | 22278 |    95   (0)| 00:00:02 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("ID"=75322)

Note
-----
   - SQL plan baseline "SQL_PLAN_9u2dxj4vgbx5h55888a5b" used for this statement

Wednesday, August 27, 2014

Can you take the RMAN backup of catalog database using the same catalog database ??

This is very interesting question.Can you take the backup of rman catalog database by connecting to the same database.The document suggest that the catalog database is same as any other database and the restore and recovery operations are also the same .So I thought of running a test to confirm.


1) Create tablespace to keep the recovery catalog schema

SQL> create tablespace rman datafile  '/u01/app/oracle/oradata/fsdb/rman01.dbs' size 10m autoextend on ;

Tablespace created.

2) Create the recovery catalog schema

SQL> CREATE USER rman IDENTIFIED BY XXXXX  DEFAULT TABLESPACE rman TEMPORARY TABLESPACE temp  QUOTA UNLIMITED ON rman;

User created.

SQL> GRANT CONNECT, RECOVERY_CATALOG_OWNER TO rman;

Grant succeeded.


3) Create catalog using rman command line

[oracle@localhost ~]$ rman catalog rman/password@fsdb

Recovery Manager: Release 11.2.0.1.0 - Production on Wed Aug 27 15:01:01 2014

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to recovery catalog database

RMAN> create catalog;

recovery catalog created


4) Register the  same database to the catalog

[oracle@localhost ~]$ rman target / catalog rman/password@fsdb

Recovery Manager: Release 11.2.0.1.0 - Production on Wed Aug 27 15:02:59 2014

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: FSDB (DBID=1085674687)
connected to recovery catalog database

RMAN> register database;

database registered in recovery catalog
starting full resync of recovery catalog
full resync complete

5)Run rman L0 backup to check

RMAN> backup incremental level 0 database;

Starting backup at 27-AUG-14
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=55 device type=DISK
channel ORA_DISK_1: starting incremental level 0 datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
.
.
.
.
Finished backup at 27-AUG-14

Starting Control File and SPFILE Autobackup at 27-AUG-14
piece handle=/u01/app/oracle/flash_recovery_area/FSDB/autobackup/2014_08_27/o1_mf_s_856710457_9zwl4vls_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at 27-AUG-14

Seems like its working but still I feel that we should backup the  control file and spfile seperately even though the spfile and control file autobackup is on.

Tuesday, August 19, 2014

Fine grained auditing (FGA) : Use AND,OR operators in the audit condition using funtions and avoid ORA-28138



The issue here is that we want to use the "and" operator  as the auditing condition to compare two values  of the columns with fine grained auditing or FGA. But If  I do something like the following.I get error ORA-28138.

begin
 DBMS_FGA.ADD_POLICY (
       object_schema => 'hr',
       object_name => 'fg_test',
       policy_name => 'test_users_fga_policy',
       audit_column => 'C1, C2', 
       statement_types => 'SELECT,INSERT,UPDATE,DELETE',
       audit_condition => 'C1=35 AND C2= 45',
       audit_column_opts => DBMS_FGA.ANY_COLUMNS,
       audit_trail => DBMS_FGA.DB + DBMS_FGA.EXTENDED );
end;
/
 
SQL> insert into fg_test values('35','45');
insert into fg_test values('35','45')
            *
ERROR at line 1:
ORA-28138: Error in Policy Predicate

As per the oracle documentation you cannot use a complex condition in the FGA policies. See the documentation:

http://docs.oracle.com/cd/E11882_01/appdev.112/e25788/d_fga.htm#CDEIECAG

"The audit_condition must be a Boolean expression that can be evaluated using the values in the row being inserted, updated, or deleted. The expression can also use functions, such as the USER or SYS_CONTEXT functions.
The expression must not combine conditions using operators such as AND and OR. audit_condition can be NULL (or omitted), which is interpreted as TRUE, but it cannot contain the following elements:"


So to fix the problem we need to create a function that returns a Boolean value .See the following test case.
   Environment:OEL 5 64 bit,DB 11.2.0.1 
1) Create Table

SQL> create table hr.fg_test(c1 number,c2 number);

Table created.

2) Create function


create or replace function hr.cond_fg(col1 number ,col2 number ) return number is
begin
if (col1 =35 and col2=45)
then
return 1;
else
return 0;
end if;
end;
/


3) Add Policy

exec DBMS_FGA.drop_policy ( -
       object_schema => 'hr', -
       object_name => 'fg_test', -
       policy_name => 'test_users_fga_policy');

begin
 DBMS_FGA.ADD_POLICY (
       object_schema => 'hr',
       object_name => 'fg_test',
       policy_name => 'test_users_fga_policy',
       audit_column => 'C1, C2', 
       statement_types => 'SELECT,INSERT,UPDATE,DELETE',
       audit_condition => 'hr.cond_fg(C1,C2)=1',
       audit_column_opts => DBMS_FGA.ANY_COLUMNS,
       audit_trail => DBMS_FGA.DB + DBMS_FGA.EXTENDED );
end;
/


4)Enable Policy


 begin
   DBMS_FGA.ENABLE_POLICY (
   object_schema    =>  'HR',
   object_name      =>  'FG_TEST',
   policy_name      =>  'test_users_fga_policy',
   enable           =>   TRUE);
   end;
   /


SQL> select OBJECT_SCHEMA, OBJECT_NAME, POLICY_NAME, POLICY_TEXT,
POLICY_COLUMN, ENABLED, SEL, INS, UPD, DEL
from DBA_AUDIT_POLICIES ;  2    3

OBJECT_SCHEMA                  OBJECT_NAME
------------------------------ ------------------------------
POLICY_NAME
------------------------------
POLICY_TEXT
--------------------------------------------------------------------------------
POLICY_COLUMN                  ENA SEL INS UPD DEL
------------------------------ --- --- --- --- ---
HR                             FG_TEST
TEST_USERS_FGA_POLICY
hr.cond_fg(C1,C2)=1
C1                             YES YES YES YES YES

5) Run the test


As HR User
===========================================
SQL>  insert into fg_test values('&a','&b');
Enter value for a: 35
Enter value for b: 45
old   1:  insert into fg_test values('&a','&b')
new   1:  insert into fg_test values('35','45')

1 row created.

SQL> commit;

Commit complete.


As SYS User
==========================================================================================
SQL> select db_user,sql_text from dba_fga_audit_trail;

DB_USER
------------------------------
SQL_TEXT
--------------------------------------------------------------------------------
HR
 insert into fg_test values('35','45')

 



It seems to be working but as per the guidelines the function should not be too complex to avoid the performance issues.