jfs: validate AG parameters in dbMount() to prevent crashes

In the Linux kernel, the following vulnerability has been resolved:

jfs: validate AG parameters in dbMount() to prevent crashes

Validate db_agheight, db_agwidth, and db_agstart in dbMount to catch

corrupted metadata early and avoid undefined behavior in dbAllocAG.

Limits are derived from L2LPERCTL, LPERCTL/MAXAG, and CTLTREESIZE:

- agheight: 0 to L2LPERCTL/2 (0 to 5) ensures shift

(L2LPERCTL - 2*agheight) >= 0.

- agwidth: 1 to min(LPERCTL/MAXAG, 2^(L2LPERCTL - 2*agheight))

ensures agperlev >= 1.

- Ranges: 1-8 (agheight 0-3), 1-4 (agheight 4), 1 (agheight 5).

- LPERCTL/MAXAG = 1024/128 = 8 limits leaves per AG;

2^(10 - 2*agheight) prevents division to 0.

- agstart: 0 to CTLTREESIZE-1 - agwidth*(MAXAG-1) keeps ti within

stree (size 1365).

- Ranges: 0-1237 (agwidth 1), 0-348 (agwidth 8).

UBSAN: shift-out-of-bounds in fs/jfs/jfs_dmap.c:1400:9

shift exponent -335544310 is negative

CPU: 0 UID: 0 PID: 5822 Comm: syz-executor130 Not tainted 6.14.0-rc5-syzkaller #0

Hardware name: Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2025

Call Trace:

<TASK>

__dump_stack lib/dump_stack.c:94 [inline]

dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120

ubsan_epilogue lib/ubsan.c:231 [inline]

__ubsan_handle_shift_out_of_bounds+0x3c8/0x420 lib/ubsan.c:468

dbAllocAG+0x1087/0x10b0 fs/jfs/jfs_dmap.c:1400

dbDiscardAG+0x352/0xa20 fs/jfs/jfs_dmap.c:1613

jfs_ioc_trim+0x45a/0x6b0 fs/jfs/jfs_discard.c:105

jfs_ioctl+0x2cd/0x3e0 fs/jfs/ioctl.c:131

vfs_ioctl fs/ioctl.c:51 [inline]

__do_sys_ioctl fs/ioctl.c:906 [inline]

__se_sys_ioctl+0xf5/0x170 fs/ioctl.c:892

do_syscall_x64 arch/x86/entry/common.c:52 [inline]

do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83

entry_SYSCALL_64_after_hwframe+0x77/0x7f

Found by Linux Verification Center (linuxtesting.org) with Syzkaller.