Commit aa712399 authored by Pingfan Liu's avatar Pingfan Liu Committed by Linus Torvalds
Browse files

mm/gup: speed up check_and_migrate_cma_pages() on huge page

Both hugetlb and thp locate on the same migration type of pageblock, since
they are allocated from a free_list[].  Based on this fact, it is enough
to check on a single subpage to decide the migration type of the whole
huge page.  By this way, it saves (2M/4K - 1) times loop for pmd_huge on
x86, similar on other archs.

Furthermore, when executing isolate_huge_page(), it avoid taking global
hugetlb_lock many times, and meanless remove/add to the local link list

[ make `i' and `step' unsigned]

Signed-off-by: default avatarPingfan Liu <>
Reviewed-by: default avatarAndrew Morton <>
Reviewed-by: default avatarIra Weiny <>
Cc: Mike Rapoport <>
Cc: "Kirill A. Shutemov" <>
Cc: Thomas Gleixner <>
Cc: John Hubbard <>
Cc: "Aneesh Kumar K.V" <>
Cc: Christoph Hellwig <>
Cc: Keith Busch <>
Cc: Mike Kravetz <>
Signed-off-by: default avatarAndrew Morton <>
Signed-off-by: default avatarLinus Torvalds <>
parent 520b4a44
......@@ -1449,25 +1449,31 @@ static long check_and_migrate_cma_pages(struct task_struct *tsk,
struct vm_area_struct **vmas,
unsigned int gup_flags)
long i;
unsigned long i;
unsigned long step;
bool drain_allow = true;
bool migrate_allow = true;
for (i = 0; i < nr_pages; i++) {
for (i = 0; i < nr_pages;) {
struct page *head = compound_head(pages[i]);
* gup may start from a tail page. Advance step by the left
* part.
step = (1 << compound_order(head)) - (pages[i] - head);
* If we get a page from the CMA zone, since we are going to
* be pinning these entries, we might as well move them out
* of the CMA zone if possible.
if (is_migrate_cma_page(pages[i])) {
struct page *head = compound_head(pages[i]);
if (PageHuge(head)) {
if (is_migrate_cma_page(head)) {
if (PageHuge(head))
isolate_huge_page(head, &cma_page_list);
} else {
else {
if (!PageLRU(head) && drain_allow) {
drain_allow = false;
......@@ -1482,6 +1488,8 @@ static long check_and_migrate_cma_pages(struct task_struct *tsk,
i += step;
if (!list_empty(&cma_page_list)) {
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment