Scribd
Upload
86 views3 pages

Disk Space Is Not Released After Deleting Many Files

This document describes how to identify and resolve orphaned files in an OCFS2 filesystem that are preventing disk space from being released after files have been deleted. It provides instructions to run diagnostics scripts, identify processes holding open file descriptors for deleted files, and kill those processes to free up the occupied disk space.

Uploaded by

elcaso34
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
86 views3 pages

Disk Space Is Not Released After Deleting Many Files

This document describes how to identify and resolve orphaned files in an OCFS2 filesystem that are preventing disk space from being released after files have been deleted. It provides instructions to run diagnostics scripts, identify processes holding open file descriptors for deleted files, and kill those processes to free up the occupied disk space.

Uploaded by

elcaso34
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 3

Disk Space is not Released After Deleting Many Files (Doc ID 468923.

1) To Bottom

Language
Modified:06-Nov-2013 Type:PROBLEM :
English

In this Document

Symptoms
Cause
Solution
  1. Run the stat_sysdir.sh script
  2. Review stat_sysdir.sh output for orphaned files
  3. Identify cluster node and application associated with orphan_dir
  4. Releasing disk space associated with OCFS2 orphan directories
References

APPLIES TO:

Linux OS - Version 1 and later


Linux x86-64
Linux x86
***Checked for relevance on 06-Nov-2013***

SYMPTOMS

Having deleted a large number of files and/or directories from an OCFS2 filesystem,  disk utilisation tools
(df, du) still report unexpectedly high filesystem/disk space utilisation, for example:
# rm -rf /ocfs/*
# ls -l /ocfs
total 0
# df -k
Filesystem        1K-blocks     Used Available Use%  Mounted on
/dev/emcpowerd     20971520  1153520  19818000   6%  /ocfs

CAUSE

Despite deleting many files and/or directories, it is possible that 'orphaned' files may exist i.e. files may have
been deleted, but they are still being accessed by running processes on one or more OCFS2 cluster nodes.

When an object e.g. a directory is deleted, the filesystem unlinks the directory entry from the existing
directory and links it  as an entry against that cluster node's orphan directory (orphan_dir). When the object
is eventually no longer used across the cluster, the filesystem frees it's inode including all disk space
associated with it.

The contents of an OCFS2 volume's orphan_dir entries can be seen using the /sbin/debugfs.ocfs2 utility, for
example:

# debugfs.ocfs2 /dev/sda2
debugfs.ocfs2 1.2.7
debugfs: ls -l //orphan_dir:0000
24 drwxr-xr-x 2 0 0 4096 9-May-2008 2 0:29 .
18 drwxr-xr-x 6 0 0 4096 9-May-2008 2 0:29 ..
7228313 -rw-r--r-- 0 200 200 79261966336 9-May-2007 20:15 00000000006e4b99
In the example above, debugfs.ocfs2 is used to list the orphan directory contents of the first OCFS2 cluster
node (num 0) on the OCFS2 volume on partition /dev/sda2.

SOLUTION

The following procedure describes how to identify the process(es) for which OCFS2 orphan_dir entries
exist.

1. Run the stat_sysdir.sh script

Run the "stat_sysdir.sh" script available in Note 391292.1 on all OCFS2 cluster nodes, for example:

# ./stat_sysdir.sh -d emcpowerb > stat_sysdir.log

2. Review stat_sysdir.sh output for orphaned files

Check the stat_sysdir.log to see which, if any, cluster nodes contain orphaned file entries i.e. deleted files for
which the associated disk space has yet to have been released, for example:
//orphan_dir:0000
14 drwxr-xr-x 2 0 0 4096 15-Jan-2008 12:53 .
6 drwxr-xr-x 8 0 0 4096 11-Jan-2008 13:55 ..
354947 -rw-r--r-- 0 1001 3001 34131492864 15-Jan-2008 04:31 0000000000056a83
354845 -rw-r----- 0 1001 3001 722 14-Jan-2008 15:44 0000000000056a1d
331950 -rw-r----- 0 1001 3001 720 14-Jan-2008 15:44 00000000000510ae

3. Identify cluster node and application associated with orphan_dir

Run the following command(s) on each cluster node to identify which node, application or user (holders) are
associated with orphan_dir entries.
# find /proc -name fd -exec ls -l {} \; | grep deleted
 or
# lsof | grep -i deleted

Next, review the output of the above command(s) noting any that relate to the OCFS2 filesystem in
question.
At this point, you should be able to determine the holding process id (pid), for example:
# lsof | grep -i deleted | grep -i ocfs
oracle 15131 oracle 3r REG 8,1 20 33266 /ocfs/clusterware/votingtest (deleted)

In the example above, file /ocfs/clusterware/votingtest previously resided on the OCFS2 mountpoint /ocfs.
Since having been deleted at the operating system-level (rm -rf /ocfs/clusterware/votingtest), the file (or at
least it's file descriptor), remains open by oracle pid 15131.

If the stat_sysdir.sh script output fails to list any orphaned files, refer to Note.558824.1 for other reasons
why the df and du commands may display vastly different or contradictory results.

4. Releasing disk space associated with OCFS2 orphan directories


Step 3. above allows you to identify the pid associated with orphaned files.
If the holding process(es) can still be gracefully interacted with via their user interface, and you are certain
that the process is safe to stop without adverse effect upon your environment, then shutdown the process(es)
in question. Once the process(es) close their open file descriptors, orphaned files will be deleted and the
associated disk space made available.

If the process(es) in question cannot be interacted with via their user interface, or if you are certain the
processes are no longer required, then kill the associated process(es) i.e. `kill <pid>`. If any process(es) are
no longer communicable (i.e. zombie) or cannot be successfully killed, a forced unmount of the OCFS2
volume in question and/or reboot of the associated cluster node may be necessary in order to recover the
disk space associated with orphaned files.

REFERENCES

NOTE:391771.1 - OCFS2 1.2 - FREQUENTLY ASKED QUESTIONS


NOTE:391292.1 - Script to gather OCFS2 diagnostic information

Related
 
Product
s

 Oracle Linux and Virtualization > Oracle Linux > Operating System > Linux OS > OCFS version 2 component

Keywords
DEBUGFS; FILES; OCFS2; OPEN FILES; ORPHAN; VOLUME

 
Translation
s
 English Source  Japanese 日本語

You might also like