- 23 Feb, 2017 40 commits
-
-
Andrea Arcangeli authored
This will verify -EINVAL is returned with hugetlbfs/shmem and it'll do a functional test of UFFDIO_ZEROPAGE on anonymous memory. Link: http://lkml.kernel.org/r/20161216144821.5183-42-aarcange@redhat.comSigned-off-by:
Andrea Arcangeli <aarcange@redhat.com> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com> Cc: Hillf Danton <hillf.zj@alibaba-inc.com> Cc: Michael Rapoport <RAPOPORT@il.ibm.com> Cc: Mike Kravetz <mike.kravetz@oracle.com> Cc: Mike Rapoport <rppt@linux.vnet.ibm.com> Cc: Pavel Emelyanov <xemul@parallels.com> Signed-off-by:
Andrew Morton <akpm@linux-foundation.org> Signed-off-by:
Linus Torvalds <torvalds@linux-foundation.org>
-
Mike Rapoport authored
Add test for userfaultfd events used in non-cooperative scenario when the process that monitors the userfaultfd and handles user faults is not the same process that causes the page faults. Link: http://lkml.kernel.org/r/20161216144821.5183-41-aarcange@redhat.comSigned-off-by:
Mike Rapoport <rppt@linux.vnet.ibm.com> Signed-off-by:
-
Mike Rapoport authored
With future addition of event tests, copy_page will be called with different userfault file descriptors Link: http://lkml.kernel.org/r/20161216144821.5183-40-aarcange@redhat.comSigned-off-by:
-
Mike Rapoport authored
userfaultfd_open will be needed by the non cooperative selftest. Link: http://lkml.kernel.org/r/20161216144821.5183-39-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
Userland developers asked to be notified immediately by the UFFDIO_API ioctl if shmem missing mode is supported by userfaultfd in the running kernel. This avoids the need to run UFFDIO_REGISTER on a shmem virtual memory range to find out. Link: http://lkml.kernel.org/r/20161216144821.5183-38-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
If the atomic copy_user fails because of a real dangling userland pointer, we won't go back into the shmem method, so when the method returns it must not leave anything charged up, except the page itself. Link: http://lkml.kernel.org/r/20161216144821.5183-37-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
Use the non atomic version of __SetPageUptodate while the page is still private and not visible to lookup operations. Using the non atomic version after the page is already visible to lookups is unsafe as there would be concurrent lock_page operation modifying the page->flags while it runs. This solves a lockup in find_lock_entry with the userfaultfd_shmem selftest. userfaultfd_shm D14296 691 1 0x00000004 Call Trace: schedule+0x3d/0x90 schedule_timeout+0x228/0x420 io_schedule_timeout+0xa4/0x110 __lock_page+0x12d/0x170 find_lock_entry+0xa4/0x190 shmem_getpage_gfp+0xb9/0xc30 shmem_fault+0x70/0x1c0 __do_fault+0x21/0x150 handle_mm_fault+0xec9/0x1490 __do_page_fault+0x20d/0x520 trace_do_page_fault+0x61/0x270 do_async_page_fault+0x19/0x80 async_page_fault+0x25/0x30 Link: http://lkml.kernel.org/r/20170116180408.12184-2-aarcange@redhat.comSigned-off-by:
Hillf Danton <hillf.zj@alibaba-inc.com> Signed-off-by:
-
Andrea Arcangeli authored
A VM_BUG_ON triggered on the shmem selftest. Link: http://lkml.kernel.org/r/20161216144821.5183-36-aarcange@redhat.comSigned-off-by:
-
Mike Rapoport authored
The test verifies that anonymous shared mapping can be used with userfault using the existing testing method. The shared memory area is allocated using mmap(..., MAP_SHARED | MAP_ANONYMOUS, ...) and released using madvise(MADV_REMOVE) Link: http://lkml.kernel.org/r/20161216144821.5183-35-aarcange@redhat.comSigned-off-by:
-
Mike Kravetz authored
When userfaultfd hugetlbfs support was originally added, it followed the pattern of anon mappings and did not support any vmas marked VM_SHARED. As such, support was only added for private mappings. Remove this limitation and support shared mappings. The primary functional change required is adding pages to the page cache. More subtle changes are required for huge page reservation handling in error paths. A lengthy comment in the code describes the reservation handling. [mike.kravetz@oracle.com: update] Link: http://lkml.kernel.org/r/c9c8cafe-baa7-05b4-34ea-1dfa5523a85f@oracle.com Link: http://lkml.kernel.org/r/1487195210-12839-1-git-send-email-mike.kravetz@oracle.comSigned-off-by:
Mike Kravetz <mike.kravetz@oracle.com> Reviewed-by:
-
Mike Rapoport authored
Expand the userfaultfd_register/unregister routines to allow shared memory VMAs. Currently, there is no UFFDIO_ZEROPAGE and write-protection support for shared memory VMAs, which is reflected in ioctl methods supported by uffdio_register. Link: http://lkml.kernel.org/r/20161216144821.5183-34-aarcange@redhat.comSigned-off-by:
-
Mike Rapoport authored
When processing a page fault in shared memory area for not present page, check the VMA determine if faults are to be handled by userfaultfd. If so, delegate the page fault to handle_userfault. Link: http://lkml.kernel.org/r/20161216144821.5183-33-aarcange@redhat.comSigned-off-by:
-
Mike Rapoport authored
The shmem_mcopy_atomic_pte implements low lever part of UFFDIO_COPY operation for shared memory VMAs. It's based on mcopy_atomic_pte with adjustments necessary for shared memory pages. Link: http://lkml.kernel.org/r/20161216144821.5183-32-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
It resolves this build error: All errors (new ones prefixed by >>): mm/shmem.c: In function 'shmem_mcopy_atomic_pte': >> mm/shmem.c:2228:2: error: implicit declaration of function 'update_mmu_cache' [-Werror=implicit-function-declaration] update_mmu_cache(dst_vma, dst_addr, dst_pte); microblaze may have to be also updated to define it in asm/pgtable.h like the other archs, then this header inclusion can be removed. Link: http://lkml.kernel.org/r/20161216144821.5183-31-aarcange@redhat.comSigned-off-by: -
Mike Rapoport authored
Currently userfault relies on vma_is_anonymous and vma_is_hugetlb to ensure compatibility of a VMA with userfault. Introduction of vma_is_shmem allows detection if tmpfs backed VMAs, so that they may be used with userfaultfd. Current implementation presumes usage of vma_is_shmem only by slow path routines in userfaultfd, therefore the vma_is_shmem is not made inline to leave the few remaining free bits in vm_flags. Link: http://lkml.kernel.org/r/20161216144821.5183-30-aarcange@redhat.comSigned-off-by:
-
Mike Rapoport authored
shmem_mcopy_atomic_pte is the low level routine that implements the userfaultfd UFFDIO_COPY command. It is based on the existing mcopy_atomic_pte routine with modifications for shared memory pages. Link: http://lkml.kernel.org/r/20161216144821.5183-29-aarcange@redhat.comSigned-off-by:
-
Mike Rapoport authored
Check whether a VMA can be used with userfault in more compact way Link: http://lkml.kernel.org/r/20161216144821.5183-28-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
Userland developers asked to be notified immediately by the UFFDIO_API ioctl if hugetlbfs missing mode is supported by userfaultfd in the running kernel. This avoids the need to run UFFDIO_REGISTER on a hugetlbfs virtual memory range to find out. Link: http://lkml.kernel.org/r/20161216144821.5183-27-aarcange@redhat.comSigned-off-by:
-
Mike Kravetz authored
If __mcopy_atomic_hugetlb exits with an error, put_page will be called if a huge page was allocated and needs to be freed. If a reservation was associated with the huge page, the PagePrivate flag will be set. Clear PagePrivate before calling put_page/free_huge_page so that the global reservation count is not incremented. Link: http://lkml.kernel.org/r/20161216144821.5183-26-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
Add support for VM_FAULT_RETRY to follow_hugetlb_page() so that get_user_pages_unlocked/locked and "nonblocking/FOLL_NOWAIT" features will work on hugetlbfs. This is required for fully functional userfaultfd non-present support on hugetlbfs. Link: http://lkml.kernel.org/r/20161216144821.5183-25-aarcange@redhat.comSigned-off-by:
-
Mike Kravetz authored
Add routine userfaultfd_huge_must_wait which has the same functionality as the existing userfaultfd_must_wait routine. Only difference is that new routine must handle page table structure for hugepmd vmas. Link: http://lkml.kernel.org/r/20161216144821.5183-24-aarcange@redhat.comSigned-off-by:
-
Mike Kravetz authored
Test userfaultfd hugetlb functionality by using the existing testing method (in userfaultfd.c). Instead of an anonymous memeory, a hugetlbfs file is mmap'ed private. In this way fallocate hole punch can be used to release pages. This is because madvise(MADV_DONTNEED) is not supported for huge pages. Use the same file, but create wrappers for allocating ranges and releasing pages. Compile userfaultfd.c with HUGETLB_TEST defined to produce an executable to test userfaultfd hugetlb functionality. Link: http://lkml.kernel.org/r/20161216144821.5183-23-aarcange@redhat.comSigned-off-by:
-
Mike Kravetz authored
Expand the userfaultfd_register/unregister routines to allow VM_HUGETLB vmas. huge page alignment checking is performed after a VM_HUGETLB vma is encountered. Also, since there is no UFFDIO_ZEROPAGE support for huge pages do not return that as a valid ioctl method for huge page ranges. Link: http://lkml.kernel.org/r/20161216144821.5183-22-aarcange@redhat.comSigned-off-by:
-
Mike Kravetz authored
When processing a hugetlb fault for no page present, check the vma to determine if faults are to be handled via userfaultfd. If so, drop the hugetlb_fault_mutex and call handle_userfault(). Link: http://lkml.kernel.org/r/20161216144821.5183-21-aarcange@redhat.comSigned-off-by:
-
Mike Kravetz authored
The new routine copy_huge_page_from_user() uses kmap_atomic() to map PAGE_SIZE pages. However, this prevents page faults in the subsequent call to copy_from_user(). This is OK in the case where the routine is copied with mmap_sema held. However, in another case we want to allow page faults. So, add a new argument allow_pagefault to indicate if the routine should allow page faults. [dan.carpenter@oracle.com: unmap the correct pointer] Link: http://lkml.kernel.org/r/20170113082608.GA3548@mwanda [akpm@linux-foundation.org: kunmap() takes a page*, per Hugh] Link: http://lkml.kernel.org/r/20161216144821.5183-20-aarcange@redhat.comSigned-off-by:
Dan Carpenter <dan.carpenter@oracle.com> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com> Cc: Hillf Danton <hillf.zj@alibaba-inc.com> Cc: Michael Rapoport <RAPOPORT@il.ibm.com> Cc: Mike Rapoport <rppt@linux.vnet.ibm.com> Cc: Pavel Emelyanov <xemul@parallels.com> Cc: Hugh Dickins <hughd@google.com> Cc: Hugh Dickins <hughd@google.com> Signed-off-by:
-
Mike Kravetz authored
__mcopy_atomic_hugetlb performs the UFFDIO_COPY operation for huge pages. It is based on the existing __mcopy_atomic routine for normal pages. Unlike normal pages, there is no huge page support for the UFFDIO_ZEROPAGE operation. Link: http://lkml.kernel.org/r/20161216144821.5183-19-aarcange@redhat.comSigned-off-by:
-
Mike Kravetz authored
hugetlb_mcopy_atomic_pte is the low level routine that implements the userfaultfd UFFDIO_COPY command. It is based on the existing mcopy_atomic_pte routine with modifications for huge pages. Link: http://lkml.kernel.org/r/20161216144821.5183-18-aarcange@redhat.comSigned-off-by:
-
Mike Kravetz authored
userfaultfd UFFDIO_COPY allows user level code to copy data to a page at fault time. The data is copied from user space to a newly allocated huge page. The new routine copy_huge_page_from_user performs this copy. Link: http://lkml.kernel.org/r/20161216144821.5183-17-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
Userfaults may still happen after the userfaultfd monitor thread received a UFFD_EVENT_MADVDONTNEED until UFFDIO_UNREGISTER is run. Wake any pending userfault within UFFDIO_UNREGISTER protected by the mmap_sem for writing, so they will not be reported to userland leading to UFFDIO_COPY returning -EINVAL (as the range was already unregistered) and they will not hang permanently either. Link: http://lkml.kernel.org/r/20161216144821.5183-16-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
MADV_DONTNEED must be notified to userland before the pages are zapped. This allows userland to immediately stop adding pages to the userfaultfd ranges before the pages are actually zapped or there could be non-zeropage leftovers as result of concurrent UFFDIO_COPY run in between zap_page_range and madvise_userfault_dontneed (both MADV_DONTNEED and UFFDIO_COPY runs under the mmap_sem for reading, so they can run concurrently). Link: http://lkml.kernel.org/r/20161216144821.5183-15-aarcange@redhat.comSigned-off-by:
-
Pavel Emelyanov authored
If the page is punched out of the address space the uffd reader should know this and zeromap the respective area in case of the #PF event. Link: http://lkml.kernel.org/r/20161216144821.5183-14-aarcange@redhat.comSigned-off-by:
Pavel Emelyanov <xemul@parallels.com> Signed-off-by:
-
Andrea Arcangeli authored
Optimize the mremap_userfaultfd_complete() interface to pass only the vm_userfaultfd_ctx pointer through the stack as a microoptimization. Link: http://lkml.kernel.org/r/20161216144821.5183-13-aarcange@redhat.comSigned-off-by:
-
Pavel Emelyanov authored
The event denotes that an area [start:end] moves to different location. Length change isn't reported as "new" addresses, if they appear on the uffd reader side they will not contain any data and the latter can just zeromap them. Waiting for the event ACK is also done outside of mmap sem, as for fork event. Link: http://lkml.kernel.org/r/20161216144821.5183-12-aarcange@redhat.comSigned-off-by:
-
Mike Rapoport authored
Since commit d2005e3f ("userfaultfd: don't pin the user memory in userfaultfd_file_create()") userfaultfd uses mm_count rather than mm_users to pin mm_struct. Make dup_userfaultfd consistent with this behaviour Link: http://lkml.kernel.org/r/20161216144821.5183-11-aarcange@redhat.comSigned-off-by:
-
Pavel Emelyanov authored
When the mm with uffd-ed vmas fork()-s the respective vmas notify their uffds with the event which contains a descriptor with new uffd. This new descriptor can then be used to get events from the child and populate its mm with data. Note, that there can be different uffd-s controlling different vmas within one mm, so first we should collect all those uffds (and ctx-s) in a list and then notify them all one by one but only once per fork(). The context is created at fork() time but the descriptor, file struct and anon inode object is created at event read time. So some trickery is added to the userfaultfd_ctx_read() to handle the ctx queues' locking vs file creation. Another thing worth noticing is that the task that fork()-s waits for the uffd event to get processed WITHOUT the mmap sem. [aarcange@redhat.com: build warning fix] Link: http://lkml.kernel.org/r/20161216144821.5183-10-aarcange@redhat.com Link: http://lkml.kernel.org/r/20161216144821.5183-9-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
This will allow userland to probe all features available in the kernel. It will however only enable the requested features in the open userfaultfd context. Link: http://lkml.kernel.org/r/20161216144821.5183-8-aarcange@redhat.comSigned-off-by:
-
Pavel Emelyanov authored
The custom events are queued in ctx->event_wqh not to disturb the fast-path-ed PF queue-wait-wakeup functions. The events to be generated (other than PF-s) are requested in UFFD_API ioctl with the uffd_api.features bits. Those, known by the kernel, are then turned on and reported back to the user-space. Link: http://lkml.kernel.org/r/20161216144821.5183-7-aarcange@redhat.comSigned-off-by:
-
Pavel Emelyanov authored
I will need one to lookup for userfaultfd_wait_queue-s in different wait queue Link: http://lkml.kernel.org/r/20161216144821.5183-6-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
Cleanup the vma->vm_ops usage. Side note: it would be more robust if vma_is_anonymous() would also check that vm_flags hasn't VM_PFNMAP set. Link: http://lkml.kernel.org/r/20161216144821.5183-5-aarcange@redhat.comSigned-off-by:
-
Andrea Arcangeli authored
Avoid BUG_ON()s and only WARN instead. This is just a cleanup, it can't make any runtime difference. This BUG_ON has never triggered and cannot trigger. Link: http://lkml.kernel.org/r/20161216144821.5183-4-aarcange@redhat.comSigned-off-by:
-
