Commit d484864d authored by Linus Torvalds's avatar Linus Torvalds

Merge branch 'for-linus' of git://git.linaro.org/people/mszyprowski/linux-dma-mapping

Pull CMA and ARM DMA-mapping updates from Marek Szyprowski:
 "These patches contain two major updates for DMA mapping subsystem
  (mainly for ARM architecture).  First one is Contiguous Memory
  Allocator (CMA) which makes it possible for device drivers to allocate
  big contiguous chunks of memory after the system has booted.

  The main difference from the similar frameworks is the fact that CMA
  allows to transparently reuse the memory region reserved for the big
  chunk allocation as a system memory, so no memory is wasted when no
  big chunk is allocated.  Once the alloc request is issued, the
  framework migrates system pages to create space for the required big
  chunk of physically contiguous memory.

  For more information one can refer to nice LWN articles:

   - 'A reworked contiguous memory allocator':
		http://lwn.net/Articles/447405/

   - 'CMA and ARM':
		http://lwn.net/Articles/450286/

   - 'A deep dive into CMA':
		http://lwn.net/Articles/486301/

   - and the following thread with the patches and links to all previous
     versions:
		https://lkml.org/lkml/2012/4/3/204

  The main client for this new framework is ARM DMA-mapping subsystem.

  The second part provides a complete redesign in ARM DMA-mapping
  subsystem.  The core implementation has been changed to use common
  struct dma_map_ops based infrastructure with the recent updates for
  new dma attributes merged in v3.4-rc2.  This allows to use more than
  one implementation of dma-mapping calls and change/select them on the
  struct device basis.  The first client of this new infractructure is
  dmabounce implementation which has been completely cut out of the
  core, common code.

  The last patch of this redesign update introduces a new, experimental
  implementation of dma-mapping calls on top of generic IOMMU framework.
  This lets ARM sub-platform to transparently use IOMMU for DMA-mapping
  calls if one provides required IOMMU hardware.

  For more information please refer to the following thread:
		http://www.spinics.net/lists/arm-kernel/msg175729.html

  The last patch merges changes from both updates and provides a
  resolution for the conflicts which cannot be avoided when patches have
  been applied on the same files (mainly arch/arm/mm/dma-mapping.c)."

Acked by Andrew Morton <akpm@linux-foundation.org>:
 "Yup, this one please.  It's had much work, plenty of review and I
  think even Russell is happy with it."

* 'for-linus' of git://git.linaro.org/people/mszyprowski/linux-dma-mapping: (28 commits)
  ARM: dma-mapping: use PMD size for section unmap
  cma: fix migration mode
  ARM: integrate CMA with DMA-mapping subsystem
  X86: integrate CMA with DMA-mapping subsystem
  drivers: add Contiguous Memory Allocator
  mm: trigger page reclaim in alloc_contig_range() to stabilise watermarks
  mm: extract reclaim code from __alloc_pages_direct_reclaim()
  mm: Serialize access to min_free_kbytes
  mm: page_isolation: MIGRATE_CMA isolation functions added
  mm: mmzone: MIGRATE_CMA migration type added
  mm: page_alloc: change fallbacks array handling
  mm: page_alloc: introduce alloc_contig_range()
  mm: compaction: export some of the functions
  mm: compaction: introduce isolate_freepages_range()
  mm: compaction: introduce map_pages()
  mm: compaction: introduce isolate_migratepages_range()
  mm: page_alloc: remove trailing whitespace
  ARM: dma-mapping: add support for IOMMU mapper
  ARM: dma-mapping: use alloc, mmap, free from dma_ops
  ARM: dma-mapping: remove redundant code and do the cleanup
  ...

Conflicts:
	arch/x86/include/asm/dma-mapping.h
parents be87cfb4 0f51596b
...@@ -508,6 +508,11 @@ bytes respectively. Such letter suffixes can also be entirely omitted. ...@@ -508,6 +508,11 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
Also note the kernel might malfunction if you disable Also note the kernel might malfunction if you disable
some critical bits. some critical bits.
cma=nn[MG] [ARM,KNL]
Sets the size of kernel global memory area for contiguous
memory allocations. For more information, see
include/linux/dma-contiguous.h
cmo_free_hint= [PPC] Format: { yes | no } cmo_free_hint= [PPC] Format: { yes | no }
Specify whether pages are marked as being inactive Specify whether pages are marked as being inactive
when they are freed. This is used in CMO environments when they are freed. This is used in CMO environments
...@@ -515,6 +520,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted. ...@@ -515,6 +520,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
a hypervisor. a hypervisor.
Default: yes Default: yes
coherent_pool=nn[KMG] [ARM,KNL]
Sets the size of memory pool for coherent, atomic dma
allocations if Contiguous Memory Allocator (CMA) is used.
code_bytes [X86] How many bytes of object code to print code_bytes [X86] How many bytes of object code to print
in an oops report. in an oops report.
Range: 0 - 8192 Range: 0 - 8192
......
...@@ -159,6 +159,9 @@ config HAVE_ARCH_TRACEHOOK ...@@ -159,6 +159,9 @@ config HAVE_ARCH_TRACEHOOK
config HAVE_DMA_ATTRS config HAVE_DMA_ATTRS
bool bool
config HAVE_DMA_CONTIGUOUS
bool
config USE_GENERIC_SMP_HELPERS config USE_GENERIC_SMP_HELPERS
bool bool
......
...@@ -5,6 +5,9 @@ config ARM ...@@ -5,6 +5,9 @@ config ARM
select HAVE_AOUT select HAVE_AOUT
select HAVE_DMA_API_DEBUG select HAVE_DMA_API_DEBUG
select HAVE_IDE if PCI || ISA || PCMCIA select HAVE_IDE if PCI || ISA || PCMCIA
select HAVE_DMA_ATTRS
select HAVE_DMA_CONTIGUOUS if (CPU_V6 || CPU_V6K || CPU_V7)
select CMA if (CPU_V6 || CPU_V6K || CPU_V7)
select HAVE_MEMBLOCK select HAVE_MEMBLOCK
select RTC_LIB select RTC_LIB
select SYS_SUPPORTS_APM_EMULATION select SYS_SUPPORTS_APM_EMULATION
...@@ -54,6 +57,14 @@ config ARM ...@@ -54,6 +57,14 @@ config ARM
config ARM_HAS_SG_CHAIN config ARM_HAS_SG_CHAIN
bool bool
config NEED_SG_DMA_LENGTH
bool
config ARM_DMA_USE_IOMMU
select NEED_SG_DMA_LENGTH
select ARM_HAS_SG_CHAIN
bool
config HAVE_PWM config HAVE_PWM
bool bool
......
...@@ -173,7 +173,8 @@ find_safe_buffer(struct dmabounce_device_info *device_info, dma_addr_t safe_dma_ ...@@ -173,7 +173,8 @@ find_safe_buffer(struct dmabounce_device_info *device_info, dma_addr_t safe_dma_
read_lock_irqsave(&device_info->lock, flags); read_lock_irqsave(&device_info->lock, flags);
list_for_each_entry(b, &device_info->safe_buffers, node) list_for_each_entry(b, &device_info->safe_buffers, node)
if (b->safe_dma_addr == safe_dma_addr) { if (b->safe_dma_addr <= safe_dma_addr &&
b->safe_dma_addr + b->size > safe_dma_addr) {
rb = b; rb = b;
break; break;
} }
...@@ -254,7 +255,7 @@ static inline dma_addr_t map_single(struct device *dev, void *ptr, size_t size, ...@@ -254,7 +255,7 @@ static inline dma_addr_t map_single(struct device *dev, void *ptr, size_t size,
if (buf == NULL) { if (buf == NULL) {
dev_err(dev, "%s: unable to map unsafe buffer %p!\n", dev_err(dev, "%s: unable to map unsafe buffer %p!\n",
__func__, ptr); __func__, ptr);
return ~0; return DMA_ERROR_CODE;
} }
dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x) mapped to %p (dma=%#x)\n", dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x) mapped to %p (dma=%#x)\n",
...@@ -307,8 +308,9 @@ static inline void unmap_single(struct device *dev, struct safe_buffer *buf, ...@@ -307,8 +308,9 @@ static inline void unmap_single(struct device *dev, struct safe_buffer *buf,
* substitute the safe buffer for the unsafe one. * substitute the safe buffer for the unsafe one.
* (basically move the buffer from an unsafe area to a safe one) * (basically move the buffer from an unsafe area to a safe one)
*/ */
dma_addr_t __dma_map_page(struct device *dev, struct page *page, static dma_addr_t dmabounce_map_page(struct device *dev, struct page *page,
unsigned long offset, size_t size, enum dma_data_direction dir) unsigned long offset, size_t size, enum dma_data_direction dir,
struct dma_attrs *attrs)
{ {
dma_addr_t dma_addr; dma_addr_t dma_addr;
int ret; int ret;
...@@ -320,21 +322,20 @@ dma_addr_t __dma_map_page(struct device *dev, struct page *page, ...@@ -320,21 +322,20 @@ dma_addr_t __dma_map_page(struct device *dev, struct page *page,
ret = needs_bounce(dev, dma_addr, size); ret = needs_bounce(dev, dma_addr, size);
if (ret < 0) if (ret < 0)
return ~0; return DMA_ERROR_CODE;
if (ret == 0) { if (ret == 0) {
__dma_page_cpu_to_dev(page, offset, size, dir); arm_dma_ops.sync_single_for_device(dev, dma_addr, size, dir);
return dma_addr; return dma_addr;
} }
if (PageHighMem(page)) { if (PageHighMem(page)) {
dev_err(dev, "DMA buffer bouncing of HIGHMEM pages is not supported\n"); dev_err(dev, "DMA buffer bouncing of HIGHMEM pages is not supported\n");
return ~0; return DMA_ERROR_CODE;
} }
return map_single(dev, page_address(page) + offset, size, dir); return map_single(dev, page_address(page) + offset, size, dir);
} }
EXPORT_SYMBOL(__dma_map_page);
/* /*
* see if a mapped address was really a "safe" buffer and if so, copy * see if a mapped address was really a "safe" buffer and if so, copy
...@@ -342,8 +343,8 @@ EXPORT_SYMBOL(__dma_map_page); ...@@ -342,8 +343,8 @@ EXPORT_SYMBOL(__dma_map_page);
* the safe buffer. (basically return things back to the way they * the safe buffer. (basically return things back to the way they
* should be) * should be)
*/ */
void __dma_unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size, static void dmabounce_unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
enum dma_data_direction dir) enum dma_data_direction dir, struct dma_attrs *attrs)
{ {
struct safe_buffer *buf; struct safe_buffer *buf;
...@@ -352,19 +353,18 @@ void __dma_unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size, ...@@ -352,19 +353,18 @@ void __dma_unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
buf = find_safe_buffer_dev(dev, dma_addr, __func__); buf = find_safe_buffer_dev(dev, dma_addr, __func__);
if (!buf) { if (!buf) {
__dma_page_dev_to_cpu(pfn_to_page(dma_to_pfn(dev, dma_addr)), arm_dma_ops.sync_single_for_cpu(dev, dma_addr, size, dir);
dma_addr & ~PAGE_MASK, size, dir);
return; return;
} }
unmap_single(dev, buf, size, dir); unmap_single(dev, buf, size, dir);
} }
EXPORT_SYMBOL(__dma_unmap_page);
int dmabounce_sync_for_cpu(struct device *dev, dma_addr_t addr, static int __dmabounce_sync_for_cpu(struct device *dev, dma_addr_t addr,
unsigned long off, size_t sz, enum dma_data_direction dir) size_t sz, enum dma_data_direction dir)
{ {
struct safe_buffer *buf; struct safe_buffer *buf;
unsigned long off;
dev_dbg(dev, "%s(dma=%#x,off=%#lx,sz=%zx,dir=%x)\n", dev_dbg(dev, "%s(dma=%#x,off=%#lx,sz=%zx,dir=%x)\n",
__func__, addr, off, sz, dir); __func__, addr, off, sz, dir);
...@@ -373,6 +373,8 @@ int dmabounce_sync_for_cpu(struct device *dev, dma_addr_t addr, ...@@ -373,6 +373,8 @@ int dmabounce_sync_for_cpu(struct device *dev, dma_addr_t addr,
if (!buf) if (!buf)
return 1; return 1;
off = addr - buf->safe_dma_addr;
BUG_ON(buf->direction != dir); BUG_ON(buf->direction != dir);
dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x) mapped to %p (dma=%#x)\n", dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x) mapped to %p (dma=%#x)\n",
...@@ -388,12 +390,21 @@ int dmabounce_sync_for_cpu(struct device *dev, dma_addr_t addr, ...@@ -388,12 +390,21 @@ int dmabounce_sync_for_cpu(struct device *dev, dma_addr_t addr,
} }
return 0; return 0;
} }
EXPORT_SYMBOL(dmabounce_sync_for_cpu);
int dmabounce_sync_for_device(struct device *dev, dma_addr_t addr, static void dmabounce_sync_for_cpu(struct device *dev,
unsigned long off, size_t sz, enum dma_data_direction dir) dma_addr_t handle, size_t size, enum dma_data_direction dir)
{
if (!__dmabounce_sync_for_cpu(dev, handle, size, dir))
return;
arm_dma_ops.sync_single_for_cpu(dev, handle, size, dir);
}
static int __dmabounce_sync_for_device(struct device *dev, dma_addr_t addr,
size_t sz, enum dma_data_direction dir)
{ {
struct safe_buffer *buf; struct safe_buffer *buf;
unsigned long off;
dev_dbg(dev, "%s(dma=%#x,off=%#lx,sz=%zx,dir=%x)\n", dev_dbg(dev, "%s(dma=%#x,off=%#lx,sz=%zx,dir=%x)\n",
__func__, addr, off, sz, dir); __func__, addr, off, sz, dir);
...@@ -402,6 +413,8 @@ int dmabounce_sync_for_device(struct device *dev, dma_addr_t addr, ...@@ -402,6 +413,8 @@ int dmabounce_sync_for_device(struct device *dev, dma_addr_t addr,
if (!buf) if (!buf)
return 1; return 1;
off = addr - buf->safe_dma_addr;
BUG_ON(buf->direction != dir); BUG_ON(buf->direction != dir);
dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x) mapped to %p (dma=%#x)\n", dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x) mapped to %p (dma=%#x)\n",
...@@ -417,7 +430,38 @@ int dmabounce_sync_for_device(struct device *dev, dma_addr_t addr, ...@@ -417,7 +430,38 @@ int dmabounce_sync_for_device(struct device *dev, dma_addr_t addr,
} }
return 0; return 0;
} }
EXPORT_SYMBOL(dmabounce_sync_for_device);
static void dmabounce_sync_for_device(struct device *dev,
dma_addr_t handle, size_t size, enum dma_data_direction dir)
{
if (!__dmabounce_sync_for_device(dev, handle, size, dir))
return;
arm_dma_ops.sync_single_for_device(dev, handle, size, dir);
}
static int dmabounce_set_mask(struct device *dev, u64 dma_mask)
{
if (dev->archdata.dmabounce)
return 0;
return arm_dma_ops.set_dma_mask(dev, dma_mask);
}
static struct dma_map_ops dmabounce_ops = {
.alloc = arm_dma_alloc,
.free = arm_dma_free,
.mmap = arm_dma_mmap,
.map_page = dmabounce_map_page,
.unmap_page = dmabounce_unmap_page,
.sync_single_for_cpu = dmabounce_sync_for_cpu,
.sync_single_for_device = dmabounce_sync_for_device,
.map_sg = arm_dma_map_sg,
.unmap_sg = arm_dma_unmap_sg,
.sync_sg_for_cpu = arm_dma_sync_sg_for_cpu,
.sync_sg_for_device = arm_dma_sync_sg_for_device,
.set_dma_mask = dmabounce_set_mask,
};
static int dmabounce_init_pool(struct dmabounce_pool *pool, struct device *dev, static int dmabounce_init_pool(struct dmabounce_pool *pool, struct device *dev,
const char *name, unsigned long size) const char *name, unsigned long size)
...@@ -479,6 +523,7 @@ int dmabounce_register_dev(struct device *dev, unsigned long small_buffer_size, ...@@ -479,6 +523,7 @@ int dmabounce_register_dev(struct device *dev, unsigned long small_buffer_size,
#endif #endif
dev->archdata.dmabounce = device_info; dev->archdata.dmabounce = device_info;
set_dma_ops(dev, &dmabounce_ops);
dev_info(dev, "dmabounce: registered device\n"); dev_info(dev, "dmabounce: registered device\n");
...@@ -497,6 +542,7 @@ void dmabounce_unregister_dev(struct device *dev) ...@@ -497,6 +542,7 @@ void dmabounce_unregister_dev(struct device *dev)
struct dmabounce_device_info *device_info = dev->archdata.dmabounce; struct dmabounce_device_info *device_info = dev->archdata.dmabounce;
dev->archdata.dmabounce = NULL; dev->archdata.dmabounce = NULL;
set_dma_ops(dev, NULL);
if (!device_info) { if (!device_info) {
dev_warn(dev, dev_warn(dev,
......
...@@ -7,12 +7,16 @@ ...@@ -7,12 +7,16 @@
#define ASMARM_DEVICE_H #define ASMARM_DEVICE_H
struct dev_archdata { struct dev_archdata {
struct dma_map_ops *dma_ops;
#ifdef CONFIG_DMABOUNCE #ifdef CONFIG_DMABOUNCE
struct dmabounce_device_info *dmabounce; struct dmabounce_device_info *dmabounce;
#endif #endif
#ifdef CONFIG_IOMMU_API #ifdef CONFIG_IOMMU_API
void *iommu; /* private IOMMU data */ void *iommu; /* private IOMMU data */
#endif #endif
#ifdef CONFIG_ARM_DMA_USE_IOMMU
struct dma_iommu_mapping *mapping;
#endif
}; };
struct omap_device; struct omap_device;
......
#ifndef ASMARM_DMA_CONTIGUOUS_H
#define ASMARM_DMA_CONTIGUOUS_H
#ifdef __KERNEL__
#ifdef CONFIG_CMA
#include <linux/types.h>
#include <asm-generic/dma-contiguous.h>
void dma_contiguous_early_fixup(phys_addr_t base, unsigned long size);
#endif
#endif
#endif
#ifndef ASMARM_DMA_IOMMU_H
#define ASMARM_DMA_IOMMU_H
#ifdef __KERNEL__
#include <linux/mm_types.h>
#include <linux/scatterlist.h>
#include <linux/dma-debug.h>
#include <linux/kmemcheck.h>
struct dma_iommu_mapping {
/* iommu specific data */
struct iommu_domain *domain;
void *bitmap;
size_t bits;
unsigned int order;
dma_addr_t base;
spinlock_t lock;
struct kref kref;
};
struct dma_iommu_mapping *
arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, size_t size,
int order);
void arm_iommu_release_mapping(struct dma_iommu_mapping *mapping);
int arm_iommu_attach_device(struct device *dev,
struct dma_iommu_mapping *mapping);
#endif /* __KERNEL__ */
#endif
This diff is collapsed.
...@@ -30,6 +30,7 @@ struct map_desc { ...@@ -30,6 +30,7 @@ struct map_desc {
#define MT_MEMORY_DTCM 12 #define MT_MEMORY_DTCM 12
#define MT_MEMORY_ITCM 13 #define MT_MEMORY_ITCM 13
#define MT_MEMORY_SO 14 #define MT_MEMORY_SO 14
#define MT_MEMORY_DMA_READY 15
#ifdef CONFIG_MMU #ifdef CONFIG_MMU
extern void iotable_init(struct map_desc *, int); extern void iotable_init(struct map_desc *, int);
......
...@@ -81,6 +81,7 @@ __setup("fpe=", fpe_setup); ...@@ -81,6 +81,7 @@ __setup("fpe=", fpe_setup);
extern void paging_init(struct machine_desc *desc); extern void paging_init(struct machine_desc *desc);
extern void sanity_check_meminfo(void); extern void sanity_check_meminfo(void);
extern void reboot_setup(char *str); extern void reboot_setup(char *str);
extern void setup_dma_zone(struct machine_desc *desc);
unsigned int processor_id; unsigned int processor_id;
EXPORT_SYMBOL(processor_id); EXPORT_SYMBOL(processor_id);
...@@ -939,12 +940,8 @@ void __init setup_arch(char **cmdline_p) ...@@ -939,12 +940,8 @@ void __init setup_arch(char **cmdline_p)
machine_desc = mdesc; machine_desc = mdesc;
machine_name = mdesc->name; machine_name = mdesc->name;
#ifdef CONFIG_ZONE_DMA setup_dma_zone(mdesc);
if (mdesc->dma_zone_size) {
extern unsigned long arm_dma_zone_size;
arm_dma_zone_size = mdesc->dma_zone_size;
}
#endif
if (mdesc->restart_mode) if (mdesc->restart_mode)
reboot_setup(&mdesc->restart_mode); reboot_setup(&mdesc->restart_mode);
......
This diff is collapsed.
...@@ -20,6 +20,7 @@ ...@@ -20,6 +20,7 @@
#include <linux/highmem.h> #include <linux/highmem.h>
#include <linux/gfp.h> #include <linux/gfp.h>
#include <linux/memblock.h> #include <linux/memblock.h>
#include <linux/dma-contiguous.h>
#include <asm/mach-types.h> #include <asm/mach-types.h>
#include <asm/memblock.h> #include <asm/memblock.h>
...@@ -226,6 +227,17 @@ static void __init arm_adjust_dma_zone(unsigned long *size, unsigned long *hole, ...@@ -226,6 +227,17 @@ static void __init arm_adjust_dma_zone(unsigned long *size, unsigned long *hole,
} }
#endif #endif
void __init setup_dma_zone(struct machine_desc *mdesc)
{
#ifdef CONFIG_ZONE_DMA
if (mdesc->dma_zone_size) {
arm_dma_zone_size = mdesc->dma_zone_size;
arm_dma_limit = PHYS_OFFSET + arm_dma_zone_size - 1;
} else
arm_dma_limit = 0xffffffff;
#endif
}
static void __init arm_bootmem_free(unsigned long min, unsigned long max_low, static void __init arm_bootmem_free(unsigned long min, unsigned long max_low,
unsigned long max_high) unsigned long max_high)
{ {
...@@ -273,12 +285,9 @@ static void __init arm_bootmem_free(unsigned long min, unsigned long max_low, ...@@ -273,12 +285,9 @@ static void __init arm_bootmem_free(unsigned long min, unsigned long max_low,
* Adjust the sizes according to any special requirements for * Adjust the sizes according to any special requirements for
* this machine type. * this machine type.
*/ */
if (arm_dma_zone_size) { if (arm_dma_zone_size)
arm_adjust_dma_zone(zone_size, zhole_size, arm_adjust_dma_zone(zone_size, zhole_size,
arm_dma_zone_size >> PAGE_SHIFT); arm_dma_zone_size >> PAGE_SHIFT);
arm_dma_limit = PHYS_OFFSET + arm_dma_zone_size - 1;
} else
arm_dma_limit = 0xffffffff;
#endif #endif
free_area_init_node(0, zone_size, min, zhole_size); free_area_init_node(0, zone_size, min, zhole_size);
...@@ -364,6 +373,12 @@ void __init arm_memblock_init(struct meminfo *mi, struct machine_desc *mdesc) ...@@ -364,6 +373,12 @@ void __init arm_memblock_init(struct meminfo *mi, struct machine_desc *mdesc)
if (mdesc->reserve) if (mdesc->reserve)
mdesc->reserve(); mdesc->reserve();
/*
* reserve memory for DMA contigouos allocations,
* must come from DMA area inside low memory
*/
dma_contiguous_reserve(min(arm_dma_limit, arm_lowmem_limit));
arm_memblock_steal_permitted = false; arm_memblock_steal_permitted = false;
memblock_allow_resize(); memblock_allow_resize();
memblock_dump_all(); memblock_dump_all();
......
...@@ -67,5 +67,8 @@ extern u32 arm_dma_limit; ...@@ -67,5 +67,8 @@ extern u32 arm_dma_limit;
#define arm_dma_limit ((u32)~0) #define arm_dma_limit ((u32)~0)
#endif #endif
extern phys_addr_t arm_lowmem_limit;
void __init bootmem_init(void); void __init bootmem_init(void);
void arm_mm_memblock_reserve(void); void arm_mm_memblock_reserve(void);
void dma_contiguous_remap(void);
...@@ -288,6 +288,11 @@ static struct mem_type mem_types[] = { ...@@ -288,6 +288,11 @@ static struct mem_type mem_types[] = {
PMD_SECT_UNCACHED | PMD_SECT_XN, PMD_SECT_UNCACHED | PMD_SECT_XN,
.domain = DOMAIN_KERNEL, .domain = DOMAIN_KERNEL,
}, },
[MT_MEMORY_DMA_READY] = {
.prot_pte = L_PTE_PRESENT | L_PTE_YOUNG | L_PTE_DIRTY,
.prot_l1 = PMD_TYPE_TABLE,
.domain = DOMAIN_KERNEL,
},
}; };
const struct mem_type *get_mem_type(unsigned int type) const struct mem_type *get_mem_type(unsigned int type)
...@@ -429,6 +434,7 @@ static void __init build_mem_type_table(void) ...@@ -429,6 +434,7 @@ static void __init build_mem_type_table(void)
if (arch_is_coherent() && cpu_is_xsc3()) { if (arch_is_coherent() && cpu_is_xsc3()) {
mem_types[MT_MEMORY].prot_sect |= PMD_SECT_S; mem_types[MT_MEMORY].prot_sect |= PMD_SECT_S;
mem_types[MT_MEMORY].prot_pte |= L_PTE_SHARED; mem_types[MT_MEMORY].prot_pte |= L_PTE_SHARED;
mem_types[MT_MEMORY_DMA_READY].prot_pte |= L_PTE_SHARED;
mem_types[MT_MEMORY_NONCACHED].prot_sect |= PMD_SECT_S; mem_types[MT_MEMORY_NONCACHED].prot_sect |= PMD_SECT_S;
mem_types[MT_MEMORY_NONCACHED].prot_pte |= L_PTE_SHARED; mem_types[MT_MEMORY_NONCACHED].prot_pte |= L_PTE_SHARED;
} }
...@@ -460,6 +466,7 @@ static void __init build_mem_type_table(void) ...@@ -460,6 +466,7 @@ static void __init build_mem_type_table(void)
mem_types[MT_DEVICE_CACHED].prot_pte |= L_PTE_SHARED; mem_types[MT_DEVICE_CACHED].prot_pte |= L_PTE_SHARED;
mem_types[MT_MEMORY].prot_sect |= PMD_SECT_S; mem_types[MT_MEMORY].prot_sect |= PMD_SECT_S;
mem_types[MT_MEMORY].prot_pte |= L_PTE_SHARED; mem_types[MT_MEMORY].prot_pte |= L_PTE_SHARED;
mem_types[MT_MEMORY_DMA_READY].prot_pte |= L_PTE_SHARED;
mem_types[MT_MEMORY_NONCACHED].prot_sect |= PMD_SECT_S; mem_types[MT_MEMORY_NONCACHED].prot_sect |= PMD_SECT_S;
mem_types[MT_MEMORY_NONCACHED].prot_pte |= L_PTE_SHARED; mem_types[MT_MEMORY_NONCACHED].prot_pte |= L_PTE_SHARED;
} }
...@@ -512,6 +519,7 @@ static void __init build_mem_type_table(void) ...@@ -512,6 +519,7 @@ static void __init build_mem_type_table(void)
mem_types[MT_HIGH_VECTORS].prot_l1 |= ecc_mask; mem_types[MT_HIGH_VECTORS].prot_l1 |= ecc_mask;
mem_types[MT_MEMORY].prot_sect |= ecc_mask | cp->pmd; mem_types[MT_MEMORY].prot_sect |= ecc_mask | cp->pmd;
mem_types[MT_MEMORY].prot_pte |= kern_pgprot; mem_types[MT_MEMORY].prot_pte |= kern_pgprot;
mem_types[MT_MEMORY_DMA_READY].prot_pte |= kern_pgprot;
mem_types[MT_MEMORY_NONCACHED].prot_sect |= ecc_mask; mem_types[MT_MEMORY_NONCACHED].prot_sect |= ecc_mask;
mem_types[MT_ROM].prot_sect |= cp->pmd; mem_types[MT_ROM].prot_sect |= cp->pmd;
...@@ -596,7 +604,7 @@ static void __init alloc_init_section(pud_t *pud, unsigned long addr, ...@@ -596,7 +604,7 @@ static void __init alloc_init_section(pud_t *pud, unsigned long addr,
* L1 entries, whereas PGDs refer to a group of L1 entries making * L1 entries, whereas PGDs refer to a group of L1 entries making
* up one logical pointer to an L2 table. * up one logical pointer to an L2 table.
*/ */
if (((addr | end | phys) & ~SECTION_MASK) == 0) { if (type->prot_sect && ((addr | end | phys) & ~SECTION_MASK) == 0) {
pmd_t *p = pmd; pmd_t *p = pmd;
#ifndef CONFIG_ARM_LPAE #ifndef CONFIG_ARM_LPAE
...@@ -814,7 +822,7 @@ static int __init early_vmalloc(char *arg) ...@@ -814,7 +822,7 @@ static int __init early_vmalloc(char *arg)
} }
early_param("vmalloc", early_vmalloc); early_param("vmalloc", early_vmalloc);
static phys_addr_t lowmem_limit __initdata = 0; phys_addr_t arm_lowmem_limit __initdata = 0;
void __init sanity_check_meminfo(void) void __init sanity_check_meminfo(void)
{ {
...@@ -897,8 +905,8 @@ void __init sanity_check_meminfo(void) ...@@ -897,8 +905,8 @@ void __init sanity_check_meminfo(void)
bank->size = newsize; bank->size = newsize;
} }
#endif #endif
if (!bank->highmem && bank->start + bank->size > lowmem_limit) if (!bank->highmem && bank->start + bank->size > arm_lowmem_limit)
lowmem_limit = bank->start + bank->size; arm_lowmem_limit = bank->start + bank->size;
j++; j++;
} }
...@@ -923,8 +931,8 @@ void __init sanity_check_meminfo(void) ...@@ -923,8 +931,8 @@ void __init sanity_check_meminfo(void)
} }
#endif #endif
meminfo.nr_banks = j; meminfo.nr_banks = j;
high_memory = __va(lowmem_limit - 1) + 1; high_memory = __va(arm_lowmem_limit - 1) + 1;
memblock_set_current_limit(lowmem_limit); memblock_set_current_limit(arm_lowmem_limit);
} }
static inline void prepare_page_table(void) static inline void prepare_page_table(void)
...@@ -949,8 +957,8 @@ static inline void prepare_page_table(void) ...@@ -949,8 +957,8 @@ static inline void prepare_page_table(void)
* Find the end of the first block of lowmem. * Find the end of the first block of lowmem.
*/ */
end = memblock.memory.regions[0].base + memblock.memory.regions[0].size; end = memblock.memory.regions[0].base + memblock.memory.regions[0].size;
if (end >= lowmem_limit) if (end >= arm_lowmem_limit)
end = lowmem_limit; end = arm_lowmem_limit;
/* /*
* Clear out all the kernel space mappings, except for the first * Clear out all the kernel space mappings, except for the first
...@@ -1093,8 +1101,8 @@ static void __init map_lowmem(void) ...@@ -1093,8 +1101,8 @@ static void __init map_lowmem(void)
phys_addr_t end = start + reg->size; phys_addr_t end = start + reg->size;
struct map_desc map; struct map_desc map;
if (end > lowmem_limit) if (end > arm_lowmem_limit)
end = lowmem_limit; end = arm_lowmem_limit;
if (start >= end) if (start >= end)
break; break;
...@@ -1115,11 +1123,12 @@ void __init paging_init(struct machine_desc *mdesc) ...@@ -1115,11 +1123,12 @@ void __init paging_init(struct machine_desc *mdesc)
{ {
void *zero_page; void *zero_page;
memblock_set_current_limit(lowmem_limit); memblock_set_current_limit(arm_lowmem_limit);
build_mem_type_table(); build_mem_type_table();
prepare_page_table(); prepare_page_table();
map_lowmem(); map_lowmem();
dma_contiguous_remap();
devicemaps_init(mdesc); devicemaps_init(mdesc);
kmap_init(); kmap_init();
......
...@@ -17,7 +17,7 @@ struct arm_vmregion { ...@@ -17,7 +17,7 @@ struct arm_vmregion {
struct list_head vm_list; struct list_head vm_list;
unsigned long vm_start; unsigned long vm_start;
unsigned long vm_end; unsigned long vm_end;
struct page *vm_pages; void *priv;
int vm_active; int vm_active;
const void *caller; const void *caller;
}; };
......
...@@ -32,6 +32,7 @@ config X86 ...@@ -32,6 +32,7 @@ config X86
select ARCH_WANT_OPTIONAL_GPIOLIB select ARCH_WANT_OPTIONAL_GPIOLIB
select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_FRAME_POINTERS
select HAVE_DMA_ATTRS select HAVE_DMA_ATTRS
select HAVE_DMA_CONTIGUOUS if !SWIOTLB
select HAVE_KRETPROBES select HAVE_KRETPROBES
select HAVE_OPTPROBES select HAVE_OPTPROBES
select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FTRACE_MCOUNT_RECORD
......
#ifndef ASMX86_DMA_CONTIGUOUS_H
#define ASMX86_DMA_CONTIGUOUS_H
#ifdef __KERNEL__
#include <linux/types.h>
#include <asm-generic/dma-contiguous.h>
static inline void
dma_contiguous_early_fixup(phys_addr_t base, unsigned long size) { }
#endif
#endif
...@@ -13,6 +13,7 @@ ...@@ -13,6 +13,7 @@
#include <asm/io.h> #include <asm/io.h>
#include <asm/swiotlb.h> #include <asm/swiotlb.h>
#include <asm-generic/dma-coherent.h> #include <asm-generic/dma-coherent.h>
#include <linux/dma-contiguous.h>
#ifdef CONFIG_ISA #ifdef CONFIG_ISA
# define ISA_DMA_BIT_MASK DMA_BIT_MASK(24) # define ISA_DMA_BIT_MASK DMA_BIT_MASK(24)
...@@ -62,6 +63,10 @@ extern void *dma_generic_alloc_coherent(struct device *dev, size_t size, ...@@ -62,6 +63,10 @@ extern void *dma_generic_alloc_coherent(struct device *dev, size_t size,
dma_addr_t *dma_addr, gfp_t flag, dma_addr_t *dma_addr, gfp_t flag,
struct dma_attrs *attrs); struct dma_attrs *attrs);
extern void dma_generic_free_coherent(struct device *dev, size_t size,
void *vaddr, dma_addr_t dma_addr,
struct dma_attrs *attrs);
#ifdef CONFIG_X86_DMA_REMAP /* Platform code defines bridge-specific code */ #ifdef CONFIG_X86_DMA_REMAP /* Platform code defines bridge-specific code */
extern bool dma_capable(struct device *dev, dma_addr_t addr, size_t size); extern bool dma_capable(struct device *dev, dma_addr_t addr, size_t size);
extern dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr); extern dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr);
......
...@@ -100,13 +100,17 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size, ...@@ -100,13 +100,17 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size,
struct dma_attrs *attrs) struct dma_attrs *attrs)
{ {
unsigned long dma_mask; unsigned long dma_mask;
struct page *page; struct page *page = NULL;
unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
dma_addr_t addr; dma_addr_t addr;
dma_mask = dma_alloc_coherent_mask(dev, flag); dma_mask = dma_alloc_coherent_mask(dev, flag);
flag |= __GFP_ZERO; flag |= __GFP_ZERO;
again: again:
if (!(flag & GFP_ATOMIC))
page = dma_alloc_from_contiguous(dev, count, get_order(size));
if (!page)
page = alloc_pages_node(dev_to_node(dev), flag, get_order(size)); page = alloc_pages_node(dev_to_node(dev), flag, get_order(size));
if (!page) if (!page)
return NULL; return NULL;
...@@ -127,6 +131,16 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size, ...@@ -127,6 +131,16 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size,
return page_address(page); return page_address(page);
} }
void dma_generic_free_coherent(struct device *dev, size_t size, void *vaddr,
dma_addr_t dma_addr, struct dma_attrs *attrs)
{
unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
struct page *page = virt_to_page(vaddr);
if (!dma_release_from_contiguous(dev, page, count))
free_pages((unsigned long)vaddr, get_order(size));
}
/* /*
* See <Documentation/x86/x86_64/boot-options.txt> for the iommu kernel * See <Documentation/x86/x86_64/boot-options.txt> for the iommu kernel
* parameter documentation. * parameter documentation.
......
...@@ -74,12 +74,6 @@ static int nommu_map_sg(struct device *hwdev, struct scatterlist *sg, ...@@ -74,12 +74,6 @@ static int nommu_map_sg(struct device *hwdev, struct scatterlist *sg,
return nents; return nents;
} }
static void nommu_free_coherent(struct device *dev, size_t size, void *vaddr,
dma_addr_t dma_addr, struct dma_attrs *attrs)
{
free_pages((unsigned long)vaddr, get_order(size));
}
static void nommu_sync_single_for_device(struct device *dev, static void nommu_sync_single_for_device(struct device *dev,
dma_addr_t addr, size_t size, dma_addr_t addr, size_t size,
enum dma_data_direction dir) enum dma_data_direction dir)
...@@ -97,7 +91,7 @@ static void nommu_sync_sg_for_device(struct device *dev, ...@@ -97,7 +91,7 @@ static void nommu_sync_sg_for_device(struct device *dev,
struct dma_map_ops nommu_dma_ops = { struct dma_map_ops nommu_dma_ops = {
.alloc = dma_generic_alloc_coherent, .alloc = dma_generic_alloc_coherent,
.free = nommu_free_coherent, .free = dma_generic_free_coherent,
.map_sg = nommu_map_sg, .map_sg = nommu_map_sg,
.map_page = nommu_map_page, .map_page = nommu_map_page,
.sync_single_for_device = nommu_sync_single_for_device, .sync_single_for_device = nommu_sync_single_for_device,
......
...@@ -49,6 +49,7 @@ ...@@ -49,6 +49,7 @@
#include <asm/pci-direct.h> #include <asm/pci-direct.h>
#include <linux/init_ohci1394_dma.h> #include <linux/init_ohci1394_dma.h>
#include <linux/kvm_para.h> #include <linux/kvm_para.h>
#include <linux/dma-contiguous.h>
#include <linux/errno.h> #include <linux/errno.h>
#include <linux/kernel.h> #include <linux/kernel.h>
...@@ -925,6 +926,7 @@ void __init setup_arch(char **cmdline_p) ...@@ -925,6 +926,7 @@ void __init setup_arch(char **cmdline_p)
} }
#endif #endif
memblock.current_limit = get_max_mapped(); memblock.current_limit = get_max_mapped();
dma_contiguous_reserve(0);
/* /*
* NOTE: On x86-32, only from this point on, fixmaps are ready for use. * NOTE: On x86-32, only from this point on, fixmaps are ready for use.
......
...@@ -192,4 +192,93 @@ config DMA_SHARED_BUFFER ...@@ -192,4 +192,93 @@ config DMA_SHARED_BUFFER
APIs extension; the file's descriptor can then be passed on to other APIs extension; the file's descriptor can then be passed on to other
driver. driver.
config CMA
bool "Contiguous Memory Allocator (EXPERIMENTAL)"
depends on HAVE_DMA_CONTIGUOUS && HAVE_MEMBLOCK && EXPERIMENTAL
select MIGRATION
help
This enables the Contiguous Memory Allocator which allows drivers
to allocate big physically-contiguous blocks of memory for use with
hardware components that do not support I/O map nor scatter-gather.
For more information see <include/linux/dma-contiguous.h>.
If unsure, say "n".
if CMA
config CMA_DEBUG
bool "CMA debug messages (DEVELOPMENT)"
depends on DEBUG_KERNEL
help
Turns on debug messages in CMA. This produces KERN_DEBUG
messages for every CMA call as well as various messages while
processing calls such as dma_alloc_from_contiguous().
This option does not affect warning and error messages.
comment "Default contiguous memory area size:"
config CMA_SIZE_MBYTES
int "Size in Mega Bytes"
depends on !CMA_SIZE_SEL_PERCENTAGE
default 16
help
Defines the size (in MiB) of the default memory area for Contiguous
Memory Allocator.
config CMA_SIZE_PERCENTAGE
int "Percentage of total memory"
depends on !CMA_SIZE_SEL_MBYTES
default 10
help
Defines the size of the default memory area for Contiguous Memory
Allocator as a percentage of the total memory in the system.
choice
prompt "Selected region size"
default CMA_SIZE_SEL_ABSOLUTE
config CMA_SIZE_SEL_MBYTES
bool "Use mega bytes value only"
config CMA_SIZE_SEL_PERCENTAGE
bool "Use percentage value only"
config CMA_SIZE_SEL_MIN
bool "Use lower value (minimum)"
config CMA_SIZE_SEL_MAX
bool "Use higher value (maximum)"
endchoice
config CMA_ALIGNMENT
int "Maximum PAGE_SIZE order of alignment for contiguous buffers"
range 4 9
default 8
help
DMA mapping framework by default aligns all buffers to the smallest
PAGE_SIZE order which is greater than or equal to the requested buffer
size. This works well for buffers up to a few hundreds kilobytes, but
for larger buffers it just a memory waste. With this parameter you can
specify the maximum PAGE_SIZE order for contiguous buffers. Larger
buffers will be aligned only to this specified order. The order is
expressed as a power of two multiplied by the PAGE_SIZE.
For example, if your system defaults to 4KiB pages, the order value
of 8 means that the buffers will be aligned up to 1MiB only.
If unsure, leave the default value "8".
config CMA_AREAS
int "Maximum count of the CMA device-private areas"
default 7
help
CMA allows to create CMA areas for particular devices. This parameter
sets the maximum number of such device private CMA areas in the
system.
If unsure, leave the default value "7".
endif
endmenu endmenu
...@@ -6,6 +6,7 @@ obj-y := core.o bus.o dd.o syscore.o \ ...@@ -6,6 +6,7 @@ obj-y := core.o bus.o dd.o syscore.o \
attribute_container.o transport_class.o \ attribute_container.o transport_class.o \
topology.o topology.o
obj-$(CONFIG_DEVTMPFS) += devtmpfs.o obj-$(CONFIG_DEVTMPFS) += devtmpfs.o
obj-$(CONFIG_CMA) += dma-contiguous.o
obj-y += power/ obj-y += power/
obj-$(CONFIG_HAS_DMA) += dma-mapping.o obj-$(CONFIG_HAS_DMA) += dma-mapping.o
obj-$(CONFIG_HAVE_GENERIC_DMA_COHERENT) += dma-coherent.o obj-$(CONFIG_HAVE_GENERIC_DMA_COHERENT) += dma-coherent.o
......
...@@ -10,6 +10,7 @@ ...@@ -10,6 +10,7 @@
struct dma_coherent_mem { struct dma_coherent_mem {
void *virt_base; void *virt_base;
dma_addr_t device_base; dma_addr_t device_base;
phys_addr_t pfn_base;
int size; int size;
int flags; int flags;
unsigned long *bitmap; unsigned long *bitmap;
...@@ -44,6 +45,7 @@ int dma_declare_coherent_memory(struct device *dev, dma_addr_t bus_addr, ...@@ -44,6 +45,7 @@ int dma_declare_coherent_memory(struct device *dev, dma_addr_t bus_addr,
dev->dma_mem->virt_base = mem_base; dev->dma_mem->virt_base = mem_base;
dev->dma_mem->device_base = device_addr; dev->dma_mem->device_base = device_addr;
dev->dma_mem->pfn_base = PFN_DOWN(bus_addr);
dev->dma_mem->size = pages; dev->dma_mem->size = pages;
dev->dma_mem->flags = flags; dev->dma_mem->flags = flags;
...@@ -176,3 +178,43 @@ int dma_release_from_coherent(struct device *dev, int order, void *vaddr) ...@@ -176,3 +178,43 @@ int dma_release_from_coherent(struct device *dev, int order, void *vaddr)
return 0; return 0;
} }
EXPORT_SYMBOL(dma_release_from_coherent); EXPORT_SYMBOL(dma_release_from_coherent);
/**
* dma_mmap_from_coherent() - try to mmap the memory allocated from
* per-device coherent memory pool to userspace
* @dev: device from which the memory was allocated
* @vma: vm_area for the userspace memory
* @vaddr: cpu address returned by dma_alloc_from_coherent
* @size: size of the memory buffer allocated by dma_alloc_from_coherent
*
* This checks whether the memory was allocated from the per-device
* coherent memory pool and if so, maps that memory to the provided vma.
*
* Returns 1 if we correctly mapped the memory, or 0 if
* dma_release_coherent() should proceed with mapping memory from
* generic pools.
*/
int dma_mmap_from_coherent(struct device *dev, struct vm_area_struct *vma,
void *vaddr, size_t size, int *ret)
{
struct dma_coherent_mem *mem = dev ? dev->dma_mem : NULL;
if (mem && vaddr >= mem->virt_base && vaddr + size <=
(mem->virt_base + (mem->size << PAGE_SHIFT))) {
unsigned long off = vma->vm_pgoff;
int start = (vaddr - mem->virt_base) >> PAGE_SHIFT;
int user_count = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
int count = size >> PAGE_SHIFT;
*ret = -ENXIO;
if (off < count && user_count <= count - off) {
unsigned pfn = mem->pfn_base + start + off;
*ret = remap_pfn_range(vma, vma->vm_start, pfn,
user_count << PAGE_SHIFT,
vma->vm_page_prot);
}
return 1;
}
return 0;
}
EXPORT_SYMBOL(dma_mmap_from_coherent);
This diff is collapsed.
...@@ -3,13 +3,15 @@ ...@@ -3,13 +3,15 @@
#ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT #ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
/* /*
* These two functions are only for dma allocator. * These three functions are only for dma allocator.
* Don't use them in device drivers. * Don't use them in device drivers.
*/ */
int dma_alloc_from_coherent(struct device *dev, ssize_t size, int dma_alloc_from_coherent(struct device *dev, ssize_t size,
dma_addr_t *dma_handle, void **ret); dma_addr_t *dma_handle, void **ret);
int dma_release_from_coherent(struct device *dev, int order, void *vaddr); int dma_release_from_coherent(struct device *dev, int order, void *vaddr);
int dma_mmap_from_coherent(struct device *dev, struct vm_area_struct *vma,
void *cpu_addr, size_t size, int *ret);
/* /*
* Standard interface * Standard interface
*/ */
......
#ifndef ASM_DMA_CONTIGUOUS_H
#define ASM_DMA_CONTIGUOUS_H
#ifdef __KERNEL__
#ifdef CONFIG_CMA
#include <linux/device.h>
#include <linux/dma-contiguous.h>
static inline struct cma *dev_get_cma_area(struct device *dev)
{
if (dev && dev->cma_area)
return dev->cma_area;
return dma_contiguous_default_area;
}
static inline void dev_set_cma_area(struct device *dev, struct cma *cma)
{
if (dev)
dev->cma_area = cma;
if (!dev || !dma_contiguous_default_area)
dma_contiguous_default_area = cma;
}
#endif
#endif
#endif
...@@ -667,6 +667,10 @@ struct device { ...@@ -667,6 +667,10 @@ struct device {
struct dma_coherent_mem *dma_mem; /* internal for coherent mem struct dma_coherent_mem *dma_mem; /* internal for coherent mem
override */ override */
#ifdef CONFIG_CMA
struct cma *cma_area; /* contiguous memory area for dma
allocations */
#endif
/* arch specific additions */ /* arch specific additions */
struct dev_archdata archdata; struct dev_archdata archdata;
......
#ifndef __LINUX_CMA_H
#define __LINUX_CMA_H
/*
* Contiguous Memory Allocator for DMA mapping framework
* Copyright (c) 2010-2011 by Samsung Electronics.
* Written by:
* Marek Szyprowski <m.szyprowski@samsung.com>
* Michal Nazarewicz <mina86@mina86.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License as
* published by the Free Software Foundation; either version 2 of the
* License or (at your optional) any later version of the license.
*/
/*
* Contiguous Memory Allocator
*
* The Contiguous Memory Allocator (CMA) makes it possible to
* allocate big contiguous chunks of memory after the system has
* booted.
*
* Why is it needed?
*
* Various devices on embedded systems have no scatter-getter and/or
* IO map support and require contiguous blocks of memory to
* operate. They include devices such as cameras, hardware video
* coders, etc.
*
* Such devices often require big memory buffers (a full HD frame
* is, for instance, more then 2 mega pixels large, i.e. more than 6
* MB of memory), which makes mechanisms such as kmalloc() or
* alloc_page() ineffective.
*
* At the same time, a solution where a big memory region is
* reserved for a device is suboptimal since often more memory is
* reserved then strictly required and, moreover, the memory is
* inaccessible to page system even if device drivers don't use it.
*
* CMA tries to solve this issue by operating on memory regions
* where only movable pages can be allocated from. This way, kernel
* can use the memory for pagecache and when device driver requests
* it, allocated pages can be migrated.
*
* Driver usage
*
* CMA should not be used by the device drivers directly. It is
* only a helper framework for dma-mapping subsystem.
*
* For more information, see kernel-docs in drivers/base/dma-contiguous.c
*/
#ifdef __KERNEL__
struct cma;
struct page;
struct device;
#ifdef CONFIG_CMA
/*
* There is always at least global CMA area and a few optional device
* private areas configured in kernel .config.
*/
#define MAX_CMA_AREAS (1 + CONFIG_CMA_AREAS)
extern struct cma *dma_contiguous_default_area;
void dma_contiguous_reserve(phys_addr_t addr_limit);
int dma_declare_contiguous(struct device *dev, unsigned long size,
phys_addr_t base, phys_addr_t limit);
struct page *dma_alloc_from_contiguous(struct device *dev, int count,
unsigned int order);
bool dma_release_from_contiguous(struct device *dev, struct page *pages,
int count);
#else
#define MAX_CMA_AREAS (0)
static inline void dma_contiguous_reserve(phys_addr_t limit) { }
static inline
int dma_declare_contiguous(struct device *dev, unsigned long size,
phys_addr_t base, phys_addr_t limit)
{
return -ENOSYS;
}
static inline
struct page *dma_alloc_from_contiguous(struct device *dev, int count,
unsigned int order)
{
return NULL;
}
static inline
bool dma_release_from_contiguous(struct device *dev, struct page *pages,
int count)
{
return false;
}
#endif
#endif
#endif
...@@ -391,4 +391,16 @@ static inline bool pm_suspended_storage(void) ...@@ -391,4 +391,16 @@ static inline bool pm_suspended_storage(void)
} }
#endif /* CONFIG_PM_SLEEP */ #endif /* CONFIG_PM_SLEEP */
#ifdef CONFIG_CMA
/* The below functions must be run on a range from a single zone. */
extern int alloc_contig_range(unsigned long start, unsigned long end,
unsigned migratetype);
extern void free_contig_range(unsigned long pfn, unsigned nr_pages);
/* CMA stuff */
extern void init_cma_reserved_pageblock(struct page *page);
#endif
#endif /* __LINUX_GFP_H */ #endif /* __LINUX_GFP_H */
...@@ -35,13 +35,39 @@ ...@@ -35,13 +35,39 @@
*/ */
#define PAGE_ALLOC_COSTLY_ORDER 3 #define PAGE_ALLOC_COSTLY_ORDER 3
#define MIGRATE_UNMOVABLE 0 enum {
#define MIGRATE_RECLAIMABLE 1 MIGRATE_UNMOVABLE,
#define MIGRATE_MOVABLE 2 MIGRATE_RECLAIMABLE,
#define MIGRATE_PCPTYPES 3 /* the number of types on the pcp lists */ MIGRATE_MOVABLE,
#define MIGRATE_RESERVE 3 MIGRATE_PCPTYPES, /* the number of types on the pcp lists */
#define MIGRATE_ISOLATE 4 /* can't allocate from here */ MIGRATE_RESERVE = MIGRATE_PCPTYPES,
#define MIGRATE_TYPES 5 #ifdef CONFIG_CMA
/*
* MIGRATE_CMA migration type is designed to mimic the way
* ZONE_MOVABLE works. Only movable pages can be allocated
* from MIGRATE_CMA pageblocks and page allocator never
* implicitly change migration type of MIGRATE_CMA pageblock.
*
* The way to use it is to change migratetype of a range of
* pageblocks to MIGRATE_CMA which can be done by
* __free_pageblock_cma() function. What is important though
* is that a range of pageblocks must be aligned to
* MAX_ORDER_NR_PAGES should biggest page be bigger then
* a single pageblock.
*/
MIGRATE_CMA,
#endif
MIGRATE_ISOLATE, /* can't allocate from here */
MIGRATE_TYPES
};
#ifdef CONFIG_CMA
# define is_migrate_cma(migratetype) unlikely((migratetype) == MIGRATE_CMA)
# define cma_wmark_pages(zone) zone->min_cma_pages
#else
# define is_migrate_cma(migratetype) false
# define cma_wmark_pages(zone) 0
#endif
#define for_each_migratetype_order(order, type) \ #define for_each_migratetype_order(order, type) \
for (order = 0; order < MAX_ORDER; order++) \ for (order = 0; order < MAX_ORDER; order++) \
...@@ -346,6 +372,13 @@ struct zone { ...@@ -346,6 +372,13 @@ struct zone {
#ifdef CONFIG_MEMORY_HOTPLUG #ifdef CONFIG_MEMORY_HOTPLUG
/* see spanned/present_pages for more description */ /* see spanned/present_pages for more description */
seqlock_t span_seqlock; seqlock_t span_seqlock;
#endif
#ifdef CONFIG_CMA
/*
* CMA needs to increase watermark levels during the allocation
* process to make sure that the system is not starved.
*/
unsigned long min_cma_pages;
#endif #endif
struct free_area free_area[MAX_ORDER]; struct free_area free_area[MAX_ORDER];
......
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
/* /*
* Changes migrate type in [start_pfn, end_pfn) to be MIGRATE_ISOLATE. * Changes migrate type in [start_pfn, end_pfn) to be MIGRATE_ISOLATE.
* If specified range includes migrate types other than MOVABLE, * If specified range includes migrate types other than MOVABLE or CMA,
* this will fail with -EBUSY. * this will fail with -EBUSY.
* *
* For isolating all pages in the range finally, the caller have to * For isolating all pages in the range finally, the caller have to
...@@ -11,27 +11,27 @@ ...@@ -11,27 +11,27 @@
* test it. * test it.
*/ */
extern int extern int
start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn); start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
unsigned migratetype);
/* /*
* Changes MIGRATE_ISOLATE to MIGRATE_MOVABLE. * Changes MIGRATE_ISOLATE to MIGRATE_MOVABLE.
* target range is [start_pfn, end_pfn) * target range is [start_pfn, end_pfn)
*/ */
extern int extern int
undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn); undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
unsigned migratetype);
/* /*
* test all pages in [start_pfn, end_pfn)are isolated or not. * Test all pages in [start_pfn, end_pfn) are isolated or not.
*/ */
extern int int test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn);
test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn);
/* /*
* Internal funcs.Changes pageblock's migrate type. * Internal functions. Changes pageblock's migrate type.
* Please use make_pagetype_isolated()/make_pagetype_movable().
*/ */
extern int set_migratetype_isolate(struct page *page); extern int set_migratetype_isolate(struct page *page);
extern void unset_migratetype_isolate(struct page *page); extern void unset_migratetype_isolate(struct page *page, unsigned migratetype);
#endif #endif
...@@ -198,7 +198,7 @@ config COMPACTION ...@@ -198,7 +198,7 @@ config COMPACTION
config MIGRATION config MIGRATION
bool "Page migration" bool "Page migration"
def_bool y def_bool y
depends on NUMA || ARCH_ENABLE_MEMORY_HOTREMOVE || COMPACTION depends on NUMA || ARCH_ENABLE_MEMORY_HOTREMOVE || COMPACTION || CMA
help help
Allows the migration of the physical location of pages of processes Allows the migration of the physical location of pages of processes
while the virtual addresses are not changed. This is useful in while the virtual addresses are not changed. This is useful in
......
...@@ -13,7 +13,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ ...@@ -13,7 +13,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \
readahead.o swap.o truncate.o vmscan.o shmem.o \ readahead.o swap.o truncate.o vmscan.o shmem.o \
prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \ prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
page_isolation.o mm_init.o mmu_context.o percpu.o \ page_isolation.o mm_init.o mmu_context.o percpu.o \
$(mmu-y) compaction.o $(mmu-y)
obj-y += init-mm.o obj-y += init-mm.o
ifdef CONFIG_NO_BOOTMEM ifdef CONFIG_NO_BOOTMEM
...@@ -32,7 +32,6 @@ obj-$(CONFIG_NUMA) += mempolicy.o ...@@ -32,7 +32,6 @@ obj-$(CONFIG_NUMA) += mempolicy.o
obj-$(CONFIG_SPARSEMEM) += sparse.o obj-$(CONFIG_SPARSEMEM) += sparse.o
obj-$(CONFIG_SPARSEMEM_VMEMMAP) += sparse-vmemmap.o obj-$(CONFIG_SPARSEMEM_VMEMMAP) += sparse-vmemmap.o
obj-$(CONFIG_SLOB) += slob.o obj-$(CONFIG_SLOB) += slob.o
obj-$(CONFIG_COMPACTION) += compaction.o
obj-$(CONFIG_MMU_NOTIFIER) += mmu_notifier.o obj-$(CONFIG_MMU_NOTIFIER) += mmu_notifier.o
obj-$(CONFIG_KSM) += ksm.o obj-$(CONFIG_KSM) += ksm.o
obj-$(CONFIG_PAGE_POISONING) += debug-pagealloc.o obj-$(CONFIG_PAGE_POISONING) += debug-pagealloc.o
......
This diff is collapsed.
...@@ -100,6 +100,39 @@ extern void prep_compound_page(struct page *page, unsigned long order); ...@@ -100,6 +100,39 @@ extern void prep_compound_page(struct page *page, unsigned long order);
extern bool is_free_buddy_page(struct page *page); extern bool is_free_buddy_page(struct page *page);
#endif #endif
#if defined CONFIG_COMPACTION || defined CONFIG_CMA
/*
* in mm/compaction.c
*/
/*
* compact_control is used to track pages being migrated and the free pages
* they are being migrated to during memory compaction. The free_pfn starts
* at the end of a zone and migrate_pfn begins at the start. Movable pages
* are moved to the end of a zone during a compaction run and the run
* completes when free_pfn <= migrate_pfn
*/
struct compact_control {
struct list_head freepages; /* List of free pages to migrate to */
struct list_head migratepages; /* List of pages being migrated */
unsigned long nr_freepages; /* Number of isolated free pages */
unsigned long nr_migratepages; /* Number of pages to migrate */
unsigned long free_pfn; /* isolate_freepages search base */
unsigned long migrate_pfn; /* isolate_migratepages search base */
bool sync; /* Synchronous migration */
int order; /* order a direct compactor needs */
int migratetype; /* MOVABLE, RECLAIMABLE etc */
struct zone *zone;
};
unsigned long
isolate_freepages_range(unsigned long start_pfn, unsigned long end_pfn);
unsigned long
isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
unsigned long low_pfn, unsigned long end_pfn);
#endif
/* /*
* function for dealing with page's order in buddy system. * function for dealing with page's order in buddy system.
......
...@@ -1404,7 +1404,7 @@ static int get_any_page(struct page *p, unsigned long pfn, int flags) ...@@ -1404,7 +1404,7 @@ static int get_any_page(struct page *p, unsigned long pfn, int flags)
/* Not a free page */ /* Not a free page */
ret = 1; ret = 1;
} }
unset_migratetype_isolate(p); unset_migratetype_isolate(p, MIGRATE_MOVABLE);
unlock_memory_hotplug(); unlock_memory_hotplug();
return ret; return ret;
} }
......
...@@ -891,7 +891,7 @@ static int __ref offline_pages(unsigned long start_pfn, ...@@ -891,7 +891,7 @@ static int __ref offline_pages(unsigned long start_pfn,
nr_pages = end_pfn - start_pfn; nr_pages = end_pfn - start_pfn;
/* set above range as isolated */ /* set above range as isolated */
ret = start_isolate_page_range(start_pfn, end_pfn); ret = start_isolate_page_range(start_pfn, end_pfn, MIGRATE_MOVABLE);
if (ret) if (ret)
goto out; goto out;
...@@ -956,7 +956,7 @@ static int __ref offline_pages(unsigned long start_pfn, ...@@ -956,7 +956,7 @@ static int __ref offline_pages(unsigned long start_pfn,
We cannot do rollback at this point. */ We cannot do rollback at this point. */
offline_isolated_pages(start_pfn, end_pfn); offline_isolated_pages(start_pfn, end_pfn);
/* reset pagetype flags and makes migrate type to be MOVABLE */ /* reset pagetype flags and makes migrate type to be MOVABLE */
undo_isolate_page_range(start_pfn, end_pfn); undo_isolate_page_range(start_pfn, end_pfn, MIGRATE_MOVABLE);
/* removal success */ /* removal success */
zone->present_pages -= offlined_pages; zone->present_pages -= offlined_pages;
zone->zone_pgdat->node_present_pages -= offlined_pages; zone->zone_pgdat->node_present_pages -= offlined_pages;
...@@ -981,7 +981,7 @@ static int __ref offline_pages(unsigned long start_pfn, ...@@ -981,7 +981,7 @@ static int __ref offline_pages(unsigned long start_pfn,
start_pfn, end_pfn); start_pfn, end_pfn);
memory_notify(MEM_CANCEL_OFFLINE, &arg); memory_notify(MEM_CANCEL_OFFLINE, &arg);
/* pushback to free area */ /* pushback to free area */
undo_isolate_page_range(start_pfn, end_pfn); undo_isolate_page_range(start_pfn, end_pfn, MIGRATE_MOVABLE);
out: out:
unlock_memory_hotplug(); unlock_memory_hotplug();
......
This diff is collapsed.
...@@ -24,6 +24,7 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) ...@@ -24,6 +24,7 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages)
* to be MIGRATE_ISOLATE. * to be MIGRATE_ISOLATE.
* @start_pfn: The lower PFN of the range to be isolated. * @start_pfn: The lower PFN of the range to be isolated.
* @end_pfn: The upper PFN of the range to be isolated. * @end_pfn: The upper PFN of the range to be isolated.
* @migratetype: migrate type to set in error recovery.
* *
* Making page-allocation-type to be MIGRATE_ISOLATE means free pages in * Making page-allocation-type to be MIGRATE_ISOLATE means free pages in
* the range will never be allocated. Any free pages and pages freed in the * the range will never be allocated. Any free pages and pages freed in the
...@@ -32,8 +33,8 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) ...@@ -32,8 +33,8 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages)
* start_pfn/end_pfn must be aligned to pageblock_order. * start_pfn/end_pfn must be aligned to pageblock_order.
* Returns 0 on success and -EBUSY if any part of range cannot be isolated. * Returns 0 on success and -EBUSY if any part of range cannot be isolated.
*/ */
int int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn) unsigned migratetype)
{ {
unsigned long pfn; unsigned long pfn;
unsigned long undo_pfn; unsigned long undo_pfn;
...@@ -56,7 +57,7 @@ start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn) ...@@ -56,7 +57,7 @@ start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn)
for (pfn = start_pfn; for (pfn = start_pfn;
pfn < undo_pfn; pfn < undo_pfn;
pfn += pageblock_nr_pages) pfn += pageblock_nr_pages)
unset_migratetype_isolate(pfn_to_page(pfn)); unset_migratetype_isolate(pfn_to_page(pfn), migratetype);
return -EBUSY; return -EBUSY;
} }
...@@ -64,8 +65,8 @@ start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn) ...@@ -64,8 +65,8 @@ start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn)
/* /*
* Make isolated pages available again. * Make isolated pages available again.
*/ */
int int undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn) unsigned migratetype)
{ {
unsigned long pfn; unsigned long pfn;
struct page *page; struct page *page;
...@@ -77,7 +78,7 @@ undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn) ...@@ -77,7 +78,7 @@ undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn)
page = __first_valid_page(pfn, pageblock_nr_pages); page = __first_valid_page(pfn, pageblock_nr_pages);
if (!page || get_pageblock_migratetype(page) != MIGRATE_ISOLATE) if (!page || get_pageblock_migratetype(page) != MIGRATE_ISOLATE)
continue; continue;
unset_migratetype_isolate(page); unset_migratetype_isolate(page, migratetype);
} }
return 0; return 0;
} }
...@@ -86,7 +87,7 @@ undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn) ...@@ -86,7 +87,7 @@ undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn)
* all pages in [start_pfn...end_pfn) must be in the same zone. * all pages in [start_pfn...end_pfn) must be in the same zone.
* zone->lock must be held before call this. * zone->lock must be held before call this.
* *
* Returns 1 if all pages in the range is isolated. * Returns 1 if all pages in the range are isolated.
*/ */
static int static int
__test_page_isolated_in_pageblock(unsigned long pfn, unsigned long end_pfn) __test_page_isolated_in_pageblock(unsigned long pfn, unsigned long end_pfn)
......
...@@ -613,6 +613,9 @@ static char * const migratetype_names[MIGRATE_TYPES] = { ...@@ -613,6 +613,9 @@ static char * const migratetype_names[MIGRATE_TYPES] = {
"Reclaimable", "Reclaimable",
"Movable", "Movable",
"Reserve", "Reserve",
#ifdef CONFIG_CMA
"CMA",
#endif
"Isolate", "Isolate",
}; };
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment