Commit 1378a5ee authored by Matthew Wilcox (Oracle)'s avatar Matthew Wilcox (Oracle) Committed by Linus Torvalds

mm: store compound_nr as well as compound_order

Patch series "THP prep patches".

These are some generic cleanups and improvements, which I would like
merged into mmotm soon.  The first one should be a performance improvement
for all users of compound pages, and the others are aimed at getting code
to compile away when CONFIG_TRANSPARENT_HUGEPAGE is disabled (ie small
systems).  Also better documented / less confusing than the current prefix
mixture of compound, hpage and thp.

This patch (of 7):

This removes a few instructions from functions which need to know how many
pages are in a compound page.  The storage used is either page->mapping on
64-bit or page->index on 32-bit.  Both of these are fine to overlay on
tail pages.
Signed-off-by: default avatarMatthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Reviewed-by: default avatarWilliam Kucharski <william.kucharski@oracle.com>
Reviewed-by: default avatarZi Yan <ziy@nvidia.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Link: http://lkml.kernel.org/r/20200629151959.15779-1-willy@infradead.org
Link: http://lkml.kernel.org/r/20200629151959.15779-2-willy@infradead.orgSigned-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 14a36a43
...@@ -922,12 +922,15 @@ static inline int compound_pincount(struct page *page) ...@@ -922,12 +922,15 @@ static inline int compound_pincount(struct page *page)
static inline void set_compound_order(struct page *page, unsigned int order) static inline void set_compound_order(struct page *page, unsigned int order)
{ {
page[1].compound_order = order; page[1].compound_order = order;
page[1].compound_nr = 1U << order;
} }
/* Returns the number of pages in this potentially compound page. */ /* Returns the number of pages in this potentially compound page. */
static inline unsigned long compound_nr(struct page *page) static inline unsigned long compound_nr(struct page *page)
{ {
return 1UL << compound_order(page); if (!PageHead(page))
return 1;
return page[1].compound_nr;
} }
/* Returns the number of bytes in this potentially compound page. */ /* Returns the number of bytes in this potentially compound page. */
......
...@@ -134,6 +134,7 @@ struct page { ...@@ -134,6 +134,7 @@ struct page {
unsigned char compound_dtor; unsigned char compound_dtor;
unsigned char compound_order; unsigned char compound_order;
atomic_t compound_mapcount; atomic_t compound_mapcount;
unsigned int compound_nr; /* 1 << compound_order */
}; };
struct { /* Second tail page of compound page */ struct { /* Second tail page of compound page */
unsigned long _compound_pad_1; /* compound_head */ unsigned long _compound_pad_1; /* compound_head */
......
...@@ -666,8 +666,6 @@ void prep_compound_page(struct page *page, unsigned int order) ...@@ -666,8 +666,6 @@ void prep_compound_page(struct page *page, unsigned int order)
int i; int i;
int nr_pages = 1 << order; int nr_pages = 1 << order;
set_compound_page_dtor(page, COMPOUND_PAGE_DTOR);
set_compound_order(page, order);
__SetPageHead(page); __SetPageHead(page);
for (i = 1; i < nr_pages; i++) { for (i = 1; i < nr_pages; i++) {
struct page *p = page + i; struct page *p = page + i;
...@@ -675,6 +673,9 @@ void prep_compound_page(struct page *page, unsigned int order) ...@@ -675,6 +673,9 @@ void prep_compound_page(struct page *page, unsigned int order)
p->mapping = TAIL_MAPPING; p->mapping = TAIL_MAPPING;
set_compound_head(p, page); set_compound_head(p, page);
} }
set_compound_page_dtor(page, COMPOUND_PAGE_DTOR);
set_compound_order(page, order);
atomic_set(compound_mapcount_ptr(page), -1); atomic_set(compound_mapcount_ptr(page), -1);
if (hpage_pincount_available(page)) if (hpage_pincount_available(page))
atomic_set(compound_pincount_ptr(page), 0); atomic_set(compound_pincount_ptr(page), 0);
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment