mm/vma: do not try to unmap a VMA if mmap_prepare() invoked from mmap()

In the Linux kernel, the following vulnerability has been resolved:

mm/vma: do not try to unmap a VMA if mmap_prepare() invoked from mmap()

The mmap_prepare hook functionality includes the ability to invoke

mmap_prepare() from the mmap() hook of existing 'stacked' drivers, that is

ones which are capable of calling the mmap hooks of other drivers/file

systems (e.g. overlayfs, shm).

As part of the mmap_prepare action functionality, we deal with errors by

unmapping the VMA should one arise. This works in the usual mmap_prepare

case, as we invoke this action at the last moment, when the VMA is

established in the maple tree.

However, the mmap() hook passes a not-fully-established VMA pointer to the

caller (which is the motivation behind the mmap_prepare() work), which is

detached.

So attempting to unmap a VMA in this state will be problematic, with the

most obvious symptom being a warning in vma_mark_detached(), because the

VMA is already detached.

It's also unncessary - the mmap() handler will clean up the VMA on error.

So to fix this issue, this patch propagates whether or not an mmap action

is being completed via the compatibility layer or directly.

If the former, then we do not attempt VMA cleanup, if the latter, then we

do.

This patch also updates the userland VMA tests to reflect the change.