btrfs: reject invalid reloc tree root keys with stack dump

In the Linux kernel, the following vulnerability has been resolved:

btrfs: reject invalid reloc tree root keys with stack dump

[BUG]

Syzbot reported a crash that an ASSERT() got triggered inside

prepare_to_merge().

That ASSERT() makes sure the reloc tree is properly pointed back by its

subvolume tree.

[CAUSE]

After more debugging output, it turns out we had an invalid reloc tree:

BTRFS error (device loop1): reloc tree mismatch, root 8 has no reloc root, expect reloc root key (-8, 132, 8) gen 17

Note the above root key is (TREE_RELOC_OBJECTID, ROOT_ITEM,

QUOTA_TREE_OBJECTID), meaning it's a reloc tree for quota tree.

But reloc trees can only exist for subvolumes, as for non-subvolume

trees, we just COW the involved tree block, no need to create a reloc

tree since those tree blocks won't be shared with other trees.

Only subvolumes tree can share tree blocks with other trees (thus they

have BTRFS_ROOT_SHAREABLE flag).

Thus this new debug output proves my previous assumption that corrupted

on-disk data can trigger that ASSERT().

[FIX]

Besides the dedicated fix and the graceful exit, also let tree-checker to

check such root keys, to make sure reloc trees can only exist for subvolumes.