Thursday, December 27, 2012

DBFS filesystem Write error

I was trying to use the DBFS file system on test environment to take rman backup(Although it is not one of the recommended filesystem for keeping the backups).DBFS is basically used to store LOB data which is referred a securefiles in 11g.The LOB data of format pre 11g version is referred as basicfiles. After configuration of DBFS I used the most basic method to mount the file system


dbfs_client dbfs_user@migrate  /mnt/dbfs.

My dbfs filesystem got mounted and I was able to create the files manually there but I had issue while invoking the RMAN backup


[oracle@localhost ~]$ df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      37295632  27205460   8165052  77% /
/dev/sda1               101086     43302     52565  46% /boot
tmpfs                   765228    158800    606428  21% /dev/shm
dbfs                     19648       184     19464   1% /mnt/dbfs


RMAN BACKUP was throwing following error

RMAN> backup as copy datafile 2 format '/mnt/dbfs/staging_area/file2.bkp';

Starting backup at 27-DEC-12
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=49 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA/migrate/datafile/sysaux.259.793193181
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 12/27/2012 08:24:22
ORA-19504: failed to create file "/mnt/dbfs/staging_area/file2.bkp"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 13: Permission denied
Additional information: 1


The cause of the problem could be following.The staging area directory is owned by root user.



[oracle@localhost dbfs]$ ls -lrt
total 0
drwxrwxrwx 3 root root 0 Dec 26 15:44 staging_area

So I mounted again the filesystem with the following option (allow_other).


 dbfs_client dbfs_user@migrate --pass-through-fuse-options -o allow_other,direct_io  /mnt/dbfs

Still it was not working.Then I got the following error.Note that it was not letting me to use allow_other and allow_root options together. 

RMAN>  backup as copy datafile 2 format '/mnt/dbfs/staging_area/file2.bkp';

Starting backup at 27-DEC-12
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=38 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA/migrate/datafile/sysaux.259.793193181
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 12/27/2012 08:26:40
ORA-19502: write error on file "/mnt/dbfs/staging_area/file2.bkp", block number 13568 (block size=8192)
ORA-27072: File I/O error
Additional information: 4
Additional information: 13568
Additional information: 917504
ORA-19502: write error on file "/mnt/dbfs/staging_area/file2.bkp", block number 13568 (block size=8192)
As per the oracle documentation we have following.
  • allow_root allows the root user to access the filesystem. This option requires setting the user_allow_other parameter in the /etc/fuse.confconfiguration file.
  • allow_other allows other users to access the filesystem. This option requires setting the user_allow_other parameter in the /etc/fuse.confconfiguration file.
I had mounted the FS again with the allow_root option and it worked !!!!!!!!
I used the following option (allow_root) this time.

[oracle@localhost staging_area]$ nohup dbfs_client dbfs_user@migrate -o allow_other,direct_io /mnt/dbfs  < .passwd_dbfs &

[oracle@localhost ~]$ rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Thu Feb 7 10:10:27 2013


Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.


connected to target database: MIGRATE (DBID=3570484994)


RMAN> backup as copy datafile 2 format '/mnt/dbfs/staging_area/file2.bkp';


Starting backup at 07-FEB-13

using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=63 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA/migrate/datafile/sysaux.259.793193181
output file name=/mnt/dbfs/staging_area/file2.bkp tag=TAG20130207T101052 RECID=17 STAMP=806753697
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:04:12
Finished backup at 07-FEB-13

Starting Control File and SPFILE Autobackup at 07-FEB-13

piece handle=/u01/app/oracle/flash_recovery_area/MIGRATE/autobackup/2013_02_07/o1_mf_s_806753705_8k7r8xln_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at 07-FEB-13



[oracle@localhost ~]$ cd /mnt/dbfs/*/

[oracle@localhost staging_area]$ ls -lrt
total 552968
-rw-r----- 1 oracle asmadmin 566239232 Feb  7 10:14 file2.bkp



My backup file was listed in the metadata of DBFS filesystem as well.

SQL> select pathname from t_staging_area;

PATHNAME
--------------------------------------------------------------------------------
/
/.sfs
/.sfs/RECYCLE
/.sfs/attributes
/.sfs/content
/.sfs/snapshots
/.sfs/tools
/file2.bkp

8 rows selected.