Commit 7aef4172 authored by Kirill A. Shutemov's avatar Kirill A. Shutemov Committed by Linus Torvalds
Browse files

mm: handle PTE-mapped tail pages in gerneric fast gup implementaiton

With new refcounting we are going to see THP tail pages mapped with PTE.
Generic fast GUP rely on page_cache_get_speculative() to obtain
reference on page.  page_cache_get_speculative() always fails on tail
pages, because ->_count on tail pages is always zero.

Let's handle tail pages in gup_pte_range().

New split_huge_page() will rely on migration entries to freeze page's
counts.  Recheck PTE value after page_cache_get_speculative() on head
page should be enough to serialize against split.
Signed-off-by: default avatarKirill A. Shutemov <>
Tested-by: default avatarSasha Levin <>
Tested-by: default avatarAneesh Kumar K.V <>
Acked-by: default avatarJerome Marchand <>
Acked-by: default avatarVlastimil Babka <>
Cc: Andrea Arcangeli <>
Cc: Hugh Dickins <>
Cc: Dave Hansen <>
Cc: Mel Gorman <>
Cc: Rik van Riel <>
Cc: Naoya Horiguchi <>
Cc: Steve Capper <>
Cc: Johannes Weiner <>
Cc: Michal Hocko <>
Cc: Christoph Lameter <>
Cc: David Rientjes <>
Signed-off-by: default avatarAndrew Morton <>
Signed-off-by: default avatarLinus Torvalds <>
parent 6742d293
......@@ -1097,7 +1097,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
* for an example see gup_get_pte in arch/x86/mm/gup.c
pte_t pte = READ_ONCE(*ptep);
struct page *page;
struct page *head, *page;
* Similar to the PMD case below, NUMA hinting must take slow
......@@ -1109,15 +1109,17 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
page = pte_page(pte);
head = compound_head(page);
if (!page_cache_get_speculative(page))
if (!page_cache_get_speculative(head))
goto pte_unmap;
if (unlikely(pte_val(pte) != pte_val(*ptep))) {
goto pte_unmap;
VM_BUG_ON_PAGE(compound_head(page) != head, page);
pages[*nr] = page;
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment