• John Hubbard's avatar
    mm/gup: page->hpage_pinned_refcount: exact pin counts for huge pages · 47e29d32
    John Hubbard authored
    For huge pages (and in fact, any compound page), the GUP_PIN_COUNTING_BIAS
    scheme tends to overflow too easily, each tail page increments the head
    page->_refcount by GUP_PIN_COUNTING_BIAS (1024).  That limits the number
    of huge pages that can be pinned.
    This patch removes that limitation, by using an exact form of pin counting
    for compound pages of order > 1.  The "order > 1" is required because this
    approach uses the 3rd struct page in the compound page, and order 1
    compound pages only have two pages, so that won't work there.
    A new struct page field, hpage_pinned_refcount, has been added, replacing
    a padding field in the union (so no new space is used).
    This enhancement also has a useful side effect: huge pages and compound
    pages (of order > 1) do not suffer from the "potential false positives"
    problem that is discussed in the page_dma_pinned() comment block.  That is
    because these compound pages have extra space for tracking things, so they
    get exact pin counts instead of overloading page->_refcount.
    Documentation/core-api/pin_user_pages.rst is updated accordingly.
    Suggested-by: default avatarJan Kara <jack@suse.cz>
    Signed-off-by: default avatarJohn Hubbard <jhubbard@nvidia.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Reviewed-by: default avatarJan Kara <jack@suse.cz>
    Acked-by: default avatarKirill A. Shutemov <kirill.shutemov@linux.intel.com>
    Cc: Ira Weiny <ira.weiny@intel.com>
    Cc: Jérôme Glisse <jglisse@redhat.com>
    Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
    Cc: Al Viro <viro@zeniv.linux.org.uk>
    Cc: Christoph Hellwig <hch@infradead.org>
    Cc: Dan Williams <dan.j.williams@intel.com>
    Cc: Dave Chinner <david@fromorbit.com>
    Cc: Jason Gunthorpe <jgg@ziepe.ca>
    Cc: Jonathan Corbet <corbet@lwn.net>
    Cc: Michal Hocko <mhocko@suse.com>
    Cc: Mike Kravetz <mike.kravetz@oracle.com>
    Cc: Shuah Khan <shuah@kernel.org>
    Cc: Vlastimil Babka <vbabka@suse.cz>
    Link: http://lkml.kernel.org/r/20200211001536.1027652-8-jhubbard@nvidia.com
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
gup.c 79.8 KB