ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"

This error can happen due to many different reasons .When it happens you may need to think of what has changed  in your environment.It was on my testing machine that I started getting this error  .I was wondering what could have gone wrong. I searched on the ineternet but I was not able to get anything which can match my  error and the environment that I have.It am using a standalone 11.2 machine on OEL5 with oracle restart feature enabled.From 11.2 onwards ASM is part of grid infrastructure and a diskgroup resource is always there under clusterware.This diskgroup houses the most important file for the ASM instance ie  SPFILE.This diskgroup resource (ora.DATA.dg) was not coming up and complaining ORA-15063 when I was trying to start it manually.



[oracle@localhost ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  OFFLINE      localhost
ora.LISTENER.lsnr
               ONLINE  ONLINE       localhost
ora.LISTENER_GRID.lsnr
               ONLINE  ONLINE       localhost
ora.asm
               ONLINE  OFFLINE      localhost                Instance Shutdown
ora.eons
               OFFLINE OFFLINE      localhost
ora.ons
               OFFLINE OFFLINE      localhost
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       localhost
ora.diskmon
      1        ONLINE  ONLINE       localhost
ora.migrate.db
      1        ONLINE  OFFLINE                               Instance Shutdown
ora.orcl.db
      1        OFFLINE OFFLINE
ora.orcl.test.svc
      1        OFFLINE OFFLINE
ora.stby.db
      1        OFFLINE OFFLINE

From the above output I can see that the resource ora.DATA.dg is offline which is the crs resource of my data diskgroup DATA.Now when I tried to start it manually I got the following error.



[oracle@localhost ~]$ crsctl start resource ora.DATA.dg
CRS-2672: Attempting to start 'ora.asm' on 'localhost'
ORA-01012: not logged on
CRS-2676: Start of 'ora.asm' on 'localhost' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'localhost'
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
CRS-2674: Start of 'ora.DATA.dg' on 'localhost' failed
CRS-2679: Attempting to clean 'ora.DATA.dg' on 'localhost'
CRS-2681: Clean of 'ora.DATA.dg' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'localhost'
CRS-2677: Stop of 'ora.asm' on 'localhost' succeeded
CRS-4000: Command Start failed, or completed with errors.

Clearly from the above output I can see that I am not able to access the diskgroup data  and this error makes me believe that there could be some issue with the disks like corruption or missing metadata but that's not true in my case .This error message is misleading.

When I try to start the ASM instance manually I am able to do that but with the following error that no spfile found.

[oracle@localhost ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on Thu Feb 7 02:05:37 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup

ORA-00099: warning: no parameter file specified for ASM instance

ASM instance started

Total System Global Area  283930624 bytes

Fixed Size                  2212656 bytes

Variable Size             256552144 bytes

ASM Cache                  25165824 bytes

ORA-15110: no diskgroups mounted

SQL> show parameter asm_

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

asm_diskgroups                       string

asm_diskstring                       string

asm_power_limit                      integer     1

asm_preferred_read_failure_groups    string

SQL>

So I needed to investigate further about my ASM resource and I looked at the ASM resource profile.Suddenly I remembered that I had removed and added ASM resource again one day before. Following is the output of the ASM resource profile and you can clearly see that that spfile and ASM_diskstring are blank.The  point is I didn't specify the ASM_DISKstring at the time of adding the ASM resource and that was my mistake.

[oracle@localhost ~]$ crsctl stat resource ora.asm -f

NAME=ora.asm

TYPE=ora.asm.type

STATE=ONLINE

TARGET=ONLINE

ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--

ACTION_FAILURE_TEMPLATE=

ACTION_SCRIPT=

AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%

ALIAS_NAME=

ASM_DISKSTRING=              <<<<<<<<<<<<<<

AUTO_START=restore

CHECK_INTERVAL=1

CHECK_TIMEOUT=600

CREATION_SEED=83

CURRENT_RCOUNT=0

DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)

DEGREE=1

DESCRIPTION=Oracle ASM resource

ENABLED=1

FAILURE_COUNT=0

FAILURE_HISTORY=

GEN_USR_ORA_INST_NAME=+ASM

ID=ora.asm

INCARNATION=0

LAST_FAULT=0

LAST_RESTART=0

LAST_SERVER=

LOAD=1

LOGGING_LEVEL=1

NLS_LANG=

NOT_RESTARTING_TEMPLATE=

OFFLINE_CHECK_INTERVAL=0

PROFILE_CHANGE_TEMPLATE=

RESTART_ATTEMPTS=5

SCRIPT_TIMEOUT=60

SPFILE=    <<<<<<<<<<<<<

START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)

START_TIMEOUT=900

STATE_CHANGE_TEMPLATE=

STATE_CHANGE_VERS=0

STATE_DETAILS=

STOP_DEPENDENCIES=hard(ora.cssd)

STOP_TIMEOUT=600

UPTIME_THRESHOLD=1d

USR_ORA_ENV=

USR_ORA_INST_NAME=+ASM

USR_ORA_OPEN_MODE=mount

USR_ORA_OPI=false

USR_ORA_STOP_MODE=immediate

VERSION=11.2.0.1.0

Solution:  

The Solution was to modeify or remove and add asm resource again with the correct asm_diskstring path.Everything came online after that.You can also modify the resource in case you don't want to remove it.

srvctl modeify asm -d /dev/oracleasm/disks

And stop and start the ASM

OR

[oracle@localhost ~]$ srvctl stop asm

[oracle@localhost disks]$ srvctl remove asm

[oracle@localhost disks]$  srvctl add asm -d /dev/oracleasm/disks

[oracle@localhost disks]$ srvctl start asm

[oracle@localhost disks]$ crsctl stat res -t

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

               ONLINE  ONLINE       localhost

ora.LISTENER.lsnr

               ONLINE  ONLINE       localhost

ora.LISTENER_GRID.lsnr

               ONLINE  ONLINE       localhost

ora.asm

               ONLINE  ONLINE       localhost                Started

ora.eons

               OFFLINE OFFLINE      localhost

ora.ons

               OFFLINE OFFLINE      localhost

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.cssd

      1        ONLINE  ONLINE       localhost

ora.diskmon

      1        ONLINE  ONLINE       localhost

ora.migrate.db

      1        ONLINE  OFFLINE                               Instance Shutdown

ora.orcl.db

      1        OFFLINE OFFLINE

ora.orcl.test.svc

      1        OFFLINE OFFLINE

ora.stby.db

      1        OFFLINE OFFLINE