Merge git://git.kernel.org/pub/scm/linux/kernel/git/aia21/ntfs

Pull NTFS update from Anton Altaparmakov:
 "Here is a small NTFS update notably implementing FIBMAP ioctl for NTFS
  by adding the bmap address space operation.  People seem to still want
  FIBMAP"

* git://git.kernel.org/pub/scm/linux/kernel/git/aia21/ntfs:
  NTFS: Bump version to 2.1.31.
  NTFS: Add bmap address space operation needed for FIBMAP ioctl.
  NTFS: Remove changelog from Documentation/filesystems/ntfs.txt.
  NTFS: Split ntfs_aops into ntfs_normal_aops and ntfs_compressed_aops in preparation for them diverging.

Documentation/filesystems/ntfs.txt

@@ -14,7 +14,6 @@ Table of contents
   - The Device-Mapper driver
   - The Software RAID / MD driver
   - Limitations when using the MD driver
-- ChangeLog
 
 
 Overview
@@ -450,270 +449,3 @@ number of sectors BEFORE attempting to use it.  You have been warned!
 
 Even better is to simply use the Device-Mapper for linear raid and then you do
 not have this problem with odd numbers of sectors.
-
-
-ChangeLog
-=========
-
-2.1.30:
-	- Fix writev() (it kept writing the first segment over and over again
-	  instead of moving onto subsequent segments).
-	- Fix crash in ntfs_mft_record_alloc() when mapping the new extent mft
-	  record failed.
-2.1.29:
-	- Fix a deadlock when mounting read-write.
-2.1.28:
-	- Fix a deadlock.
-2.1.27:
-	- Implement page migration support so the kernel can move memory used
-	  by NTFS files and directories around for management purposes.
-	- Add support for writing to sparse files created with Windows XP SP2.
-	- Many minor improvements and bug fixes.
-2.1.26:
-	- Implement support for sector sizes above 512 bytes (up to the maximum
-	  supported by NTFS which is 4096 bytes).
-	- Enhance support for NTFS volumes which were supported by Windows but
-	  not by Linux due to invalid attribute list attribute flags.
-	- A few minor updates and bug fixes.
-2.1.25:
-	- Write support is now extended with write(2) being able to both
-	  overwrite existing file data and to extend files.  Also, if a write
-	  to a sparse region occurs, write(2) will fill in the hole.  Note,
-	  mmap(2) based writes still do not support writing into holes or
-	  writing beyond the initialized size.
-	- Write support has a new feature and that is that truncate(2) and
-	  open(2) with O_TRUNC are now implemented thus files can be both made
-	  smaller and larger.
-	- Note: Both write(2) and truncate(2)/open(2) with O_TRUNC still have
-	  limitations in that they
-	  - only provide limited support for highly fragmented files.
-	  - only work on regular, i.e. uncompressed and unencrypted files.
-	  - never create sparse files although this will change once directory
-	    operations are implemented.
-	- Lots of bug fixes and enhancements across the board.
-2.1.24:
-	- Support journals ($LogFile) which have been modified by chkdsk.  This
-	  means users can boot into Windows after we marked the volume dirty.
-	  The Windows boot will run chkdsk and then reboot.  The user can then
-	  immediately boot into Linux rather than having to do a full Windows
-	  boot first before rebooting into Linux and we will recognize such a
-	  journal and empty it as it is clean by definition.
-	- Support journals ($LogFile) with only one restart page as well as
-	  journals with two different restart pages.  We sanity check both and
-	  either use the only sane one or the more recent one of the two in the
-	  case that both are valid.
-	- Lots of bug fixes and enhancements across the board.
-2.1.23:
-	- Stamp the user space journal, aka transaction log, aka $UsnJrnl, if
-	  it is present and active thus telling Windows and applications using
-	  the transaction log that changes can have happened on the volume
-	  which are not recorded in $UsnJrnl.
-	- Detect the case when Windows has been hibernated (suspended to disk)
-	  and if this is the case do not allow (re)mounting read-write to
-	  prevent data corruption when you boot back into the suspended
-	  Windows session.
-	- Implement extension of resident files using the normal file write
-	  code paths, i.e. most very small files can be extended to be a little
-	  bit bigger but not by much.
-	- Add new mount option "disable_sparse".  (See list of mount options
-	  above for details.)
-	- Improve handling of ntfs volumes with errors and strange boot sectors
-	  in particular.
-	- Fix various bugs including a nasty deadlock that appeared in recent
-	  kernels (around 2.6.11-2.6.12 timeframe).
-2.1.22:
-	- Improve handling of ntfs volumes with errors.
-	- Fix various bugs and race conditions.
-2.1.21:
-	- Fix several race conditions and various other bugs.
-	- Many internal cleanups, code reorganization, optimizations, and mft
-	  and index record writing code rewritten to fit in with the changes.
-	- Update Documentation/filesystems/ntfs.txt with instructions on how to
-	  use the Device-Mapper driver with NTFS ftdisk/LDM raid.
-2.1.20:
-	- Fix two stupid bugs introduced in 2.1.18 release.
-2.1.19:
-	- Minor bugfix in handling of the default upcase table.
-	- Many internal cleanups and improvements.  Many thanks to Linus
-	  Torvalds and Al Viro for the help and advice with the sparse
-	  annotations and cleanups.
-2.1.18:
-	- Fix scheduling latencies at mount time.  (Ingo Molnar)
-	- Fix endianness bug in a little traversed portion of the attribute
-	  lookup code.
-2.1.17:
-	- Fix bugs in mount time error code paths.
-2.1.16:
-	- Implement access time updates (including mtime and ctime).
-	- Implement fsync(2), fdatasync(2), and msync(2) system calls.
-	- Enable the readv(2) and writev(2) system calls.
-	- Enable access via the asynchronous io (aio) API by adding support for
-	  the aio_read(3) and aio_write(3) functions.
-2.1.15:
-	- Invalidate quotas when (re)mounting read-write.
-	  NOTE:  This now only leave user space journalling on the side.  (See
-	  note for version 2.1.13, below.)
-2.1.14:
-	- Fix an NFSd caused deadlock reported by several users.
-2.1.13:
-	- Implement writing of inodes (access time updates are not implemented
-	  yet so mounting with -o noatime,nodiratime is enforced).
-	- Enable writing out of resident files so you can now overwrite any
-	  uncompressed, unencrypted, nonsparse file as long as you do not
-	  change the file size.
-	- Add housekeeping of ntfs system files so that ntfsfix no longer needs
-	  to be run after writing to an NTFS volume.
-	  NOTE:  This still leaves quota tracking and user space journalling on
-	  the side but they should not cause data corruption.  In the worst
-	  case the charged quotas will be out of date ($Quota) and some
-	  userspace applications might get confused due to the out of date
-	  userspace journal ($UsnJrnl).
-2.1.12:
-	- Fix the second fix to the decompression engine from the 2.1.9 release
-	  and some further internals cleanups.
-2.1.11:
-	- Driver internal cleanups.
-2.1.10:
-	- Force read-only (re)mounting of volumes with unsupported volume
-	  flags and various cleanups.
-2.1.9:
-	- Fix two bugs in handling of corner cases in the decompression engine.
-2.1.8:
-	- Read the $MFT mirror and compare it to the $MFT and if the two do not
-	  match, force a read-only mount and do not allow read-write remounts.
-	- Read and parse the $LogFile journal and if it indicates that the
-	  volume was not shutdown cleanly, force a read-only mount and do not
-	  allow read-write remounts.  If the $LogFile indicates a clean
-	  shutdown and a read-write (re)mount is requested, empty $LogFile to
-	  ensure that Windows cannot cause data corruption by replaying a stale
-	  journal after Linux has written to the volume.
-	- Improve time handling so that the NTFS time is fully preserved when
-	  converted to kernel time and only up to 99 nano-seconds are lost when
-	  kernel time is converted to NTFS time.
-2.1.7:
-	- Enable NFS exporting of mounted NTFS volumes.
-2.1.6:
-	- Fix minor bug in handling of compressed directories that fixes the
-	  erroneous "du" and "stat" output people reported.
-2.1.5:
-	- Minor bug fix in attribute list attribute handling that fixes the
-	  I/O errors on "ls" of certain fragmented files found by at least two
-	  people running Windows XP.
-2.1.4:
-	- Minor update allowing compilation with all gcc versions (well, the
-	  ones the kernel can be compiled with anyway).
-2.1.3:
-	- Major bug fixes for reading files and volumes in corner cases which
-	  were being hit by Windows 2k/XP users.
-2.1.2:
-	- Major bug fixes alleviating the hangs in statfs experienced by some
-	  users.
-2.1.1:
-	- Update handling of compressed files so people no longer get the
-	  frequently reported warning messages about initialized_size !=
-	  data_size.
-2.1.0:
-	- Add configuration option for developmental write support.
-	- Initial implementation of file overwriting. (Writes to resident files
-	  are not written out to disk yet, so avoid writing to files smaller
-	  than about 1kiB.)
-	- Intercept/abort changes in file size as they are not implemented yet.
-2.0.25:
-	- Minor bugfixes in error code paths and small cleanups.
-2.0.24:
-	- Small internal cleanups.
-	- Support for sendfile system call. (Christoph Hellwig)
-2.0.23:
-	- Massive internal locking changes to mft record locking. Fixes
-	  various race conditions and deadlocks.
-	- Fix ntfs over loopback for compressed files by adding an
-	  optimization barrier. (gcc was screwing up otherwise ?)
-	Thanks go to Christoph Hellwig for pointing these two out:
-	- Remove now unused function fs/ntfs/malloc.h::vmalloc_nofs().
-	- Fix ntfs_free() for ia64 and parisc.
-2.0.22:
-	- Small internal cleanups.
-2.0.21:
-	These only affect 32-bit architectures:
-	- Check for, and refuse to mount too large volumes (maximum is 2TiB).
-	- Check for, and refuse to open too large files and directories
-	  (maximum is 16TiB).
-2.0.20:
-	- Support non-resident directory index bitmaps. This means we now cope
-	  with huge directories without problems.
-	- Fix a page leak that manifested itself in some cases when reading
-	  directory contents.
-	- Internal cleanups.
-2.0.19:
-	- Fix race condition and improvements in block i/o interface.
-	- Optimization when reading compressed files.
-2.0.18:
-	- Fix race condition in reading of compressed files.
-2.0.17:
-	- Cleanups and optimizations.
-2.0.16:
-	- Fix stupid bug introduced in 2.0.15 in new attribute inode API.
-	- Big internal cleanup replacing the mftbmp access hacks by using the
-	  new attribute inode API instead.
-2.0.15:
-	- Bug fix in parsing of remount options.
-	- Internal changes implementing attribute (fake) inodes allowing all
-	  attribute i/o to go via the page cache and to use all the normal
-	  vfs/mm functionality.
-2.0.14:
-	- Internal changes improving run list merging code and minor locking
-	  change to not rely on BKL in ntfs_statfs().
-2.0.13:
-	- Internal changes towards using iget5_locked() in preparation for
-	  fake inodes and small cleanups to ntfs_volume structure.
-2.0.12:
-	- Internal cleanups in address space operations made possible by the
-	  changes introduced in the previous release.
-2.0.11:
-	- Internal updates and cleanups introducing the first step towards
-	  fake inode based attribute i/o.
-2.0.10:
-	- Microsoft says that the maximum number of inodes is 2^32 - 1. Update
-	  the driver accordingly to only use 32-bits to store inode numbers on
-	  32-bit architectures. This improves the speed of the driver a little.
-2.0.9:
-	- Change decompression engine to use a single buffer. This should not
-	  affect performance except perhaps on the most heavy i/o on SMP
-	  systems when accessing multiple compressed files from multiple
-	  devices simultaneously.
-	- Minor updates and cleanups.
-2.0.8:
-	- Remove now obsolete show_inodes and posix mount option(s).
-	- Restore show_sys_files mount option.
-	- Add new mount option case_sensitive, to determine if the driver
-	  treats file names as case sensitive or not.
-	- Mostly drop support for short file names (for backwards compatibility
-	  we only support accessing files via their short file name if one
-	  exists).
-	- Fix dcache aliasing issues wrt short/long file names.
-	- Cleanups and minor fixes.
-2.0.7:
-	- Just cleanups.
-2.0.6:
-	- Major bugfix to make compatible with other kernel changes. This fixes
-	  the hangs/oopses on umount.
-	- Locking cleanup in directory operations (remove BKL usage).
-2.0.5:
-	- Major buffer overflow bug fix.
-	- Minor cleanups and updates for kernel 2.5.12.
-2.0.4:
-	- Cleanups and updates for kernel 2.5.11.
-2.0.3:
-	- Small bug fixes, cleanups, and performance improvements.
-2.0.2:
-	- Use default fmask of 0177 so that files are no executable by default.
-	  If you want owner executable files, just use fmask=0077.
-	- Update for kernel 2.5.9 but preserve backwards compatibility with
-	  kernel 2.5.7.
-	- Minor bug fixes, cleanups, and updates.
-2.0.1:
-	- Minor updates, primarily set the executable bit by default on files
-	  so they can be executed.
-2.0.0:
-	- Started ChangeLog.
-

fs/ntfs/Makefile

@@ -8,7 +8,7 @@ ntfs-y := aops.o attrib.o collate.o compress.o debug.o dir.o file.o \
 
 ntfs-$(CONFIG_NTFS_RW) += bitmap.o lcnalloc.o logfile.o quota.o usnjrnl.o
 
-ccflags-y := -DNTFS_VERSION=\"2.1.30\"
+ccflags-y := -DNTFS_VERSION=\"2.1.31\"
 ccflags-$(CONFIG_NTFS_DEBUG)	+= -DDEBUG
 ccflags-$(CONFIG_NTFS_RW)	+= -DNTFS_RW
 

fs/ntfs/aops.c

 /**
  * aops.c - NTFS kernel address space operations and page cache handling.
- *	    Part of the Linux-NTFS project.
  *
- * Copyright (c) 2001-2007 Anton Altaparmakov
+ * Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc.
  * Copyright (c) 2002 Richard Russon
  *
  * This program/include file is free software; you can redistribute it and/or
@@ -1539,16 +1538,157 @@ static int ntfs_writepage(struct page *page, struct writeback_control *wbc)
 #endif	/* NTFS_RW */
 
 /**
- * ntfs_aops - general address space operations for inodes and attributes
+ * ntfs_bmap - map logical file block to physical device block
+ * @mapping:	address space mapping to which the block to be mapped belongs
+ * @block:	logical block to map to its physical device block
+ *
+ * For regular, non-resident files (i.e. not compressed and not encrypted), map
+ * the logical @block belonging to the file described by the address space
+ * mapping @mapping to its physical device block.
+ *
+ * The size of the block is equal to the @s_blocksize field of the super block
+ * of the mounted file system which is guaranteed to be smaller than or equal
+ * to the cluster size thus the block is guaranteed to fit entirely inside the
+ * cluster which means we do not need to care how many contiguous bytes are
+ * available after the beginning of the block.
+ *
+ * Return the physical device block if the mapping succeeded or 0 if the block
+ * is sparse or there was an error.
+ *
+ * Note: This is a problem if someone tries to run bmap() on $Boot system file
+ * as that really is in block zero but there is nothing we can do.  bmap() is
+ * just broken in that respect (just like it cannot distinguish sparse from
+ * not available or error).
  */
-const struct address_space_operations ntfs_aops = {
-	.readpage	= ntfs_readpage,	/* Fill page with data. */
+static sector_t ntfs_bmap(struct address_space *mapping, sector_t block)
+{
+	s64 ofs, size;
+	loff_t i_size;
+	LCN lcn;
+	unsigned long blocksize, flags;
+	ntfs_inode *ni = NTFS_I(mapping->host);
+	ntfs_volume *vol = ni->vol;
+	unsigned delta;
+	unsigned char blocksize_bits, cluster_size_shift;
+
+	ntfs_debug("Entering for mft_no 0x%lx, logical block 0x%llx.",
+			ni->mft_no, (unsigned long long)block);
+	if (ni->type != AT_DATA || !NInoNonResident(ni) || NInoEncrypted(ni)) {
+		ntfs_error(vol->sb, "BMAP does not make sense for %s "
+				"attributes, returning 0.",
+				(ni->type != AT_DATA) ? "non-data" :
+				(!NInoNonResident(ni) ? "resident" :
+				"encrypted"));
+		return 0;
+	}
+	/* None of these can happen. */
+	BUG_ON(NInoCompressed(ni));
+	BUG_ON(NInoMstProtected(ni));
+	blocksize = vol->sb->s_blocksize;
+	blocksize_bits = vol->sb->s_blocksize_bits;
+	ofs = (s64)block << blocksize_bits;
+	read_lock_irqsave(&ni->size_lock, flags);
+	size = ni->initialized_size;
+	i_size = i_size_read(VFS_I(ni));
+	read_unlock_irqrestore(&ni->size_lock, flags);
+	/*
+	 * If the offset is outside the initialized size or the block straddles
+	 * the initialized size then pretend it is a hole unless the
+	 * initialized size equals the file size.
+	 */
+	if (unlikely(ofs >= size || (ofs + blocksize > size && size < i_size)))
+		goto hole;
+	cluster_size_shift = vol->cluster_size_bits;
+	down_read(&ni->runlist.lock);
+	lcn = ntfs_attr_vcn_to_lcn_nolock(ni, ofs >> cluster_size_shift, false);
+	up_read(&ni->runlist.lock);
+	if (unlikely(lcn < LCN_HOLE)) {
+		/*
+		 * Step down to an integer to avoid gcc doing a long long
+		 * comparision in the switch when we know @lcn is between
+		 * LCN_HOLE and LCN_EIO (i.e. -1 to -5).
+		 *
+		 * Otherwise older gcc (at least on some architectures) will
+		 * try to use __cmpdi2() which is of course not available in
+		 * the kernel.
+		 */
+		switch ((int)lcn) {
+		case LCN_ENOENT:
+			/*
+			 * If the offset is out of bounds then pretend it is a
+			 * hole.
+			 */
+			goto hole;
+		case LCN_ENOMEM:
+			ntfs_error(vol->sb, "Not enough memory to complete "
+					"mapping for inode 0x%lx.  "
+					"Returning 0.", ni->mft_no);
+			break;
+		default:
+			ntfs_error(vol->sb, "Failed to complete mapping for "
+					"inode 0x%lx.  Run chkdsk.  "
+					"Returning 0.", ni->mft_no);
+			break;
+		}
+		return 0;
+	}
+	if (lcn < 0) {
+		/* It is a hole. */
+hole:
+		ntfs_debug("Done (returning hole).");
+		return 0;
+	}
+	/*
+	 * The block is really allocated and fullfils all our criteria.
+	 * Convert the cluster to units of block size and return the result.
+	 */
+	delta = ofs & vol->cluster_size_mask;
+	if (unlikely(sizeof(block) < sizeof(lcn))) {
+		block = lcn = ((lcn << cluster_size_shift) + delta) >>
+				blocksize_bits;
+		/* If the block number was truncated return 0. */
+		if (unlikely(block != lcn)) {
+			ntfs_error(vol->sb, "Physical block 0x%llx is too "
+					"large to be returned, returning 0.",
+					(long long)lcn);
+			return 0;
+		}
+	} else
+		block = ((lcn << cluster_size_shift) + delta) >>
+				blocksize_bits;
+	ntfs_debug("Done (returning block 0x%llx).", (unsigned long long)lcn);
+	return block;
+}
+
+/**
+ * ntfs_normal_aops - address space operations for normal inodes and attributes
+ *
+ * Note these are not used for compressed or mst protected inodes and
+ * attributes.
+ */
+const struct address_space_operations ntfs_normal_aops = {
+	.readpage	= ntfs_readpage,
 #ifdef NTFS_RW
-	.writepage	= ntfs_writepage,	/* Write dirty page to disk. */
+	.writepage	= ntfs_writepage,
+	.set_page_dirty	= __set_page_dirty_buffers,
+#endif /* NTFS_RW */
+	.bmap		= ntfs_bmap,
+	.migratepage	= buffer_migrate_page,
+	.is_partially_uptodate = block_is_partially_uptodate,
+	.error_remove_page = generic_error_remove_page,
+};
+
+/**
+ * ntfs_compressed_aops - address space operations for compressed inodes
+ */
+const struct address_space_operations ntfs_compressed_aops = {
+	.readpage	= ntfs_readpage,
+#ifdef NTFS_RW
+	.writepage	= ntfs_writepage,
+	.set_page_dirty	= __set_page_dirty_buffers,
 #endif /* NTFS_RW */
-	.migratepage	= buffer_migrate_page,	/* Move a page cache page from
-						   one physical page to an
-						   other. */
+	.migratepage	= buffer_migrate_page,
+	.is_partially_uptodate = block_is_partially_uptodate,
 	.error_remove_page = generic_error_remove_page,
 };
 
@@ -1564,9 +1704,8 @@ const struct address_space_operations ntfs_mst_aops = {
 						   without touching the buffers
 						   belonging to the page. */
 #endif /* NTFS_RW */
-	.migratepage	= buffer_migrate_page,	/* Move a page cache page from
-						   one physical page to an
-						   other. */
+	.migratepage	= buffer_migrate_page,
+	.is_partially_uptodate	= block_is_partially_uptodate,
 	.error_remove_page = generic_error_remove_page,
 };
 

fs/ntfs/inode.c

 /**
- * inode.c - NTFS kernel inode handling. Part of the Linux-NTFS project.
+ * inode.c - NTFS kernel inode handling.
  *
- * Copyright (c) 2001-2007 Anton Altaparmakov
+ * Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc.
  *
  * This program/include file is free software; you can redistribute it and/or
  * modify it under the terms of the GNU General Public License as published
@@ -1012,6 +1012,7 @@ static int ntfs_read_locked_inode(struct inode *vi)
 		/* Setup the operations for this inode. */
 		vi->i_op = &ntfs_dir_inode_ops;
 		vi->i_fop = &ntfs_dir_ops;
+		vi->i_mapping->a_ops = &ntfs_mst_aops;
 	} else {
 		/* It is a file. */
 		ntfs_attr_reinit_search_ctx(ctx);
@@ -1160,11 +1161,12 @@ static int ntfs_read_locked_inode(struct inode *vi)
 		/* Setup the operations for this inode. */
 		vi->i_op = &ntfs_file_inode_ops;
 		vi->i_fop = &ntfs_file_ops;
-	}
+		vi->i_mapping->a_ops = &ntfs_normal_aops;
 		if (NInoMstProtected(ni))
 			vi->i_mapping->a_ops = &ntfs_mst_aops;
-	else
-		vi->i_mapping->a_ops = &ntfs_aops;
+		else if (NInoCompressed(ni))
+			vi->i_mapping->a_ops = &ntfs_compressed_aops;
+	}
 	/*
 	 * The number of 512-byte blocks used on disk (for stat). This is in so
 	 * far inaccurate as it doesn't account for any named streams or other
@@ -1414,10 +1416,11 @@ static int ntfs_read_locked_attr_inode(struct inode *base_vi, struct inode *vi)
 		ni->allocated_size = sle64_to_cpu(
 				a->data.non_resident.allocated_size);
 	}
+	vi->i_mapping->a_ops = &ntfs_normal_aops;
 	if (NInoMstProtected(ni))
 		vi->i_mapping->a_ops = &ntfs_mst_aops;
-	else
-		vi->i_mapping->a_ops = &ntfs_aops;
+	else if (NInoCompressed(ni))
+		vi->i_mapping->a_ops = &ntfs_compressed_aops;
 	if ((NInoCompressed(ni) || NInoSparse(ni)) && ni->type != AT_INDEX_ROOT)
 		vi->i_blocks = ni->itype.compressed.size >> 9;
 	else

fs/ntfs/ntfs.h

 /*
- * ntfs.h - Defines for NTFS Linux kernel driver. Part of the Linux-NTFS
- *	    project.
+ * ntfs.h - Defines for NTFS Linux kernel driver.
  *
- * Copyright (c) 2001-2005 Anton Altaparmakov
+ * Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc.
  * Copyright (C) 2002 Richard Russon
  *
  * This program/include file is free software; you can redistribute it and/or
@@ -57,7 +56,8 @@ extern struct kmem_cache *ntfs_attr_ctx_cache;
 extern struct kmem_cache *ntfs_index_ctx_cache;
 
 /* The various operations structs defined throughout the driver files. */
-extern const struct address_space_operations ntfs_aops;
+extern const struct address_space_operations ntfs_normal_aops;
+extern const struct address_space_operations ntfs_compressed_aops;
 extern const struct address_space_operations ntfs_mst_aops;
 
 extern const struct  file_operations ntfs_file_ops;