Commit 66aad4fd authored by Xunlei Pang's avatar Xunlei Pang Committed by Ingo Molnar

x86/mm: Add support for gbpages to kernel_ident_mapping_init()

Kernel identity mappings on x86-64 kernels are created in two
ways: by the early x86 boot code, or by kernel_ident_mapping_init().

Native kernels (which is the dominant usecase) use the former,
but the kexec and the hibernation code uses kernel_ident_mapping_init().

There's a subtle difference between these two ways of how identity
mappings are created, the current kernel_ident_mapping_init() code
creates identity mappings always using 2MB page(PMD level) - while
the native kernel boot path also utilizes gbpages where available.

This difference is suboptimal both for performance and for memory
usage: kernel_ident_mapping_init() needs to allocate pages for the
page tables when creating the new identity mappings.

This patch adds 1GB page(PUD level) support to kernel_ident_mapping_init()
to address these concerns.

The primary advantage would be better TLB coverage/performance,
because we'd utilize 1GB TLBs instead of 2MB ones.

It is also useful for machines with large number of memory to
save paging structure allocations(around 4MB/TB using 2MB page)
when setting identity mappings for all the memory, after using
1GB page it will consume only 8KB/TB.

( Note that this change alone does not activate gbpages in kexec,
  we are doing that in a separate patch. )
Signed-off-by: default avatarXunlei Pang <>
Cc: Andy Lutomirski <>
Cc: Borislav Petkov <>
Cc: Brian Gerst <>
Cc: Dave Young <>
Cc: Denys Vlasenko <>
Cc: Eric Biederman <>
Cc: H. Peter Anvin <>
Cc: Josh Poimboeuf <>
Cc: Linus Torvalds <>
Cc: Peter Zijlstra <>
Cc: Thomas Gleixner <>
Cc: Yinghai Lu <>
Link: default avatarIngo Molnar <>
parent 60854a12
......@@ -70,7 +70,7 @@ static unsigned long level4p;
* Due to relocation, pointers must be assigned at run time not build time.
static struct x86_mapping_info mapping_info = {
.page_flag = __PAGE_KERNEL_LARGE_EXEC,
/* Locates and clears a region for a new top level page table. */
......@@ -4,8 +4,9 @@
struct x86_mapping_info {
void *(*alloc_pgt_page)(void *); /* allocate buf for page table */
void *context; /* context for alloc_pgt_page */
unsigned long pmd_flag; /* page flag for PMD entry */
unsigned long page_flag; /* page flag for PMD or PUD entry */
unsigned long offset; /* ident mapping offset */
bool direct_gbpages; /* PUD level 1GB page support */
int kernel_ident_mapping_init(struct x86_mapping_info *info, pgd_t *pgd_page,
......@@ -113,7 +113,7 @@ static int init_pgtable(struct kimage *image, unsigned long start_pgtable)
struct x86_mapping_info info = {
.alloc_pgt_page = alloc_pgt_page,
.context = image,
.page_flag = __PAGE_KERNEL_LARGE_EXEC,
unsigned long mstart, mend;
pgd_t *level4p;
......@@ -13,7 +13,7 @@ static void ident_pmd_init(struct x86_mapping_info *info, pmd_t *pmd_page,
if (pmd_present(*pmd))
set_pmd(pmd, __pmd((addr - info->offset) | info->pmd_flag));
set_pmd(pmd, __pmd((addr - info->offset) | info->page_flag));
......@@ -30,6 +30,18 @@ static int ident_pud_init(struct x86_mapping_info *info, pud_t *pud_page,
if (next > end)
next = end;
if (info->direct_gbpages) {
pud_t pudval;
if (pud_present(*pud))
addr &= PUD_MASK;
pudval = __pud((addr - info->offset) | info->page_flag);
set_pud(pud, pudval);
if (pud_present(*pud)) {
pmd = pmd_offset(pud, 0);
ident_pmd_init(info, pmd, addr, next);
......@@ -104,7 +104,7 @@ static int set_up_temporary_mappings(void)
struct x86_mapping_info info = {
.alloc_pgt_page = alloc_pgt_page,
.page_flag = __PAGE_KERNEL_LARGE_EXEC,
.offset = __PAGE_OFFSET,
unsigned long mstart, mend;
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment