btrfs: fix transaction abort on set received ioctl due to item overflow

In the Linux kernel, the following vulnerability has been resolved:

btrfs: fix transaction abort on set received ioctl due to item overflow

If the set received ioctl fails due to an item overflow when attempting to

add the BTRFS_UUID_KEY_RECEIVED_SUBVOL we have to abort the transaction

since we did some metadata updates before.

This means that if a user calls this ioctl with the same received UUID

field for a lot of subvolumes, we will hit the overflow, trigger the

transaction abort and turn the filesystem into RO mode. A malicious user

could exploit this, and this ioctl does not even requires that a user

has admin privileges (CAP_SYS_ADMIN), only that he/she owns the subvolume.

Fix this by doing an early check for item overflow before starting a

transaction. This is also race safe because we are holding the subvol_sem

semaphore in exclusive (write) mode.

A test case for fstests will follow soon.