• David Howells's avatar
    CacheFiles: Fix occasional EIO on call to vfs_unlink() · c61ea31d
    David Howells authored
    Fix an occasional EIO returned by a call to vfs_unlink():
    
    	[ 4868.465413] CacheFiles: I/O Error: Unlink failed
    	[ 4868.465444] FS-Cache: Cache cachefiles stopped due to I/O error
    	[ 4947.320011] CacheFiles: File cache on md3 unregistering
    	[ 4947.320041] FS-Cache: Withdrawing cache "mycache"
    	[ 5127.348683] FS-Cache: Cache "mycache" added (type cachefiles)
    	[ 5127.348716] CacheFiles: File cache on md3 registered
    	[ 7076.871081] CacheFiles: I/O Error: Unlink failed
    	[ 7076.871130] FS-Cache: Cache cachefiles stopped due to I/O error
    	[ 7116.780891] CacheFiles: File cache on md3 unregistering
    	[ 7116.780937] FS-Cache: Withdrawing cache "mycache"
    	[ 7296.813394] FS-Cache: Cache "mycache" added (type cachefiles)
    	[ 7296.813432] CacheFiles: File cache on md3 registered
    
    What happens is this:
    
     (1) A cached NFS file is seen to have become out of date, so NFS retires the
         object and immediately acquires a new object with the same key.
    
     (2) Retirement of the old object is done asynchronously - so the lookup/create
         to generate the new object may be done first.
    
         This can be a problem as the old object and the new object must exist at
         the same point in the backing filesystem (i.e. they must have the same
         pathname).
    
     (3) The lookup for the new object sees that a backing file already exists,
         checks to see whether it is valid and sees that it isn't.  It then deletes
         that file and creates a new one on disk.
    
     (4) The retirement phase for the old file is then performed.  It tries to
         delete the dentry it has, but ext4_unlink() returns -EIO because the inode
         attached to that dentry no longer matches the inode number associated with
         the filename in the parent directory.
    
    The trace below shows this quite well.
    
    	[md5sum] ==> __fscache_relinquish_cookie(ffff88002d12fb58{NFS.fh,ffff88002ce62100},1)
    	[md5sum] ==> __fscache_acquire_cookie({NFS.server},{NFS.fh},ffff88002ce62100)
    
    NFS has retired the old cookie and asked for a new one.
    
    	[kslowd] ==> fscache_object_state_machine({OBJ52,OBJECT_ACTIVE,24})
    	[kslowd] <== fscache_object_state_machine() [->OBJECT_DYING]
    	[kslowd] ==> fscache_object_state_machine({OBJ53,OBJECT_INIT,0})
    	[kslowd] <== fscache_object_state_machine() [->OBJECT_LOOKING_UP]
    	[kslowd] ==> fscache_object_state_machine({OBJ52,OBJECT_DYING,24})
    	[kslowd] <== fscache_object_state_machine() [->OBJECT_RECYCLING]
    
    The old object (OBJ52) is going through the terminal states to get rid of it,
    whilst the new object - (OBJ53) - is coming into being.
    
    	[kslowd] ==> fscache_object_state_machine({OBJ53,OBJECT_LOOKING_UP,0})
    	[kslowd] ==> cachefiles_walk_to_object({ffff88003029d8b8},OBJ53,@68,)
    	[kslowd] lookup '@68'
    	[kslowd] next -> ffff88002ce41bd0 positive
    	[kslowd] advance
    	[kslowd] lookup 'Es0g00og0_Nd_XCYe3BOzvXrsBLMlN6aw16M1htaA'
    	[kslowd] next -> ffff8800369faac8 positive
    
    The new object has looked up the subdir in which the file would be in (getting
    dentry ffff88002ce41bd0) and then looked up the file itself (getting dentry
    ffff8800369faac8).
    
    	[kslowd] validate 'Es0g00og0_Nd_XCYe3BOzvXrsBLMlN6aw16M1htaA'
    	[kslowd] ==> cachefiles_bury_object(,'@68','Es0g00og0_Nd_XCYe3BOzvXrsBLMlN6aw16M1htaA')
    	[kslowd] remove ffff8800369faac8 from ffff88002ce41bd0
    	[kslowd] unlink stale object
    	[kslowd] <== cachefiles_bury_object() = 0
    
    It then checks the file's xattrs to see if it's valid.  NFS says that the
    auxiliary data indicate the file is out of date (obvious to us - that's why NFS
    ditched the old version and got a new one).  CacheFiles then deletes the old
    file (dentry ffff8800369faac8).
    
    	[kslowd] redo lookup
    	[kslowd] lookup 'Es0g00og0_Nd_XCYe3BOzvXrsBLMlN6aw16M1htaA'
    	[kslowd] next -> ffff88002cd94288 negative
    	[kslowd] create -> ffff88002cd94288{ffff88002cdaf238{ino=148247}}
    
    CacheFiles then redoes the lookup and gets a negative result in a new dentry
    (ffff88002cd94288) which it then creates a file for.
    
    	[kslowd] ==> cachefiles_mark_object_active(,OBJ53)
    	[kslowd] <== cachefiles_mark_object_active() = 0
    	[kslowd] === OBTAINED_OBJECT ===
    	[kslowd] <== cachefiles_walk_to_object() = 0 [148247]
    	[kslowd] <== fscache_object_state_machine() [->OBJECT_AVAILABLE]
    
    The new object is then marked active and the state machine moves to the
    available state - at which point NFS can start filling the object.
    
    	[kslowd] ==> fscache_object_state_machine({OBJ52,OBJECT_RECYCLING,20})
    	[kslowd] ==> fscache_release_object()
    	[kslowd] ==> cachefiles_drop_object({OBJ52,2})
    	[kslowd] ==> cachefiles_delete_object(,OBJ52{ffff8800369faac8})
    
    The old object, meanwhile, goes on with being retired.  If allocation occurs
    first, cachefiles_delete_object() has to wait for dir->d_inode->i_mutex to
    become available before it can continue.
    
    	[kslowd] ==> cachefiles_bury_object(,'@68','Es0g00og0_Nd_XCYe3BOzvXrsBLMlN6aw16M1htaA')
    	[kslowd] remove ffff8800369faac8 from ffff88002ce41bd0
    	[kslowd] unlink stale object
    	EXT4-fs warning (device sda6): ext4_unlink: Inode number mismatch in unlink (148247!=148193)
    	CacheFiles: I/O Error: Unlink failed
    	FS-Cache: Cache cachefiles stopped due to I/O error
    
    CacheFiles then tries to delete the file for the old object, but the dentry it
    has (ffff8800369faac8) no longer points to a valid inode for that directory
    entry, and so ext4_unlink() returns -EIO when de->inode does not match i_ino.
    
    	[kslowd] <== cachefiles_bury_object() = -5
    	[kslowd] <== cachefiles_delete_object() = -5
    	[kslowd] <== fscache_object_state_machine() [->OBJECT_DEAD]
    	[kslowd] ==> fscache_object_state_machine({OBJ53,OBJECT_AVAILABLE,0})
    	[kslowd] <== fscache_object_state_machine() [->OBJECT_ACTIVE]
    
    (Note that the above trace includes extra information beyond that produced by
    the upstream code).
    
    The fix is to note when an object that is being retired has had its object
    deleted preemptively by a replacement object that is being created, and to
    skip the second removal attempt in such a case.
    Reported-by: default avatarGreg M <gregm@servu.net.au>
    Reported-by: default avatarMark Moseley <moseleymark@gmail.com>
    Reported-by: default avatarRomain DEGEZ <romain.degez@smartjog.com>
    Signed-off-by: default avatarDavid Howells <dhowells@redhat.com>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    c61ea31d
internal.h 11 KB