mm/page_alloc: prevent pcp corruption with SMP=n

In the Linux kernel, the following vulnerability has been resolved:

mm/page_alloc: prevent pcp corruption with SMP=n

The kernel test robot has reported:

BUG: spinlock trylock failure on UP on CPU#0, kcompactd0/28

lock: 0xffff888807e35ef0, .magic: dead4ead, .owner: kcompactd0/28, .owner_cpu: 0

CPU: 0 UID: 0 PID: 28 Comm: kcompactd0 Not tainted 6.18.0-rc5-00127-ga06157804399 #1 PREEMPT 8cc09ef94dcec767faa911515ce9e609c45db470

Call Trace:

<IRQ>

__dump_stack (lib/dump_stack.c:95)

dump_stack_lvl (lib/dump_stack.c:123)

dump_stack (lib/dump_stack.c:130)

spin_dump (kernel/locking/spinlock_debug.c:71)

do_raw_spin_trylock (kernel/locking/spinlock_debug.c:?)

_raw_spin_trylock (include/linux/spinlock_api_smp.h:89 kernel/locking/spinlock.c:138)

__free_frozen_pages (mm/page_alloc.c:2973)

___free_pages (mm/page_alloc.c:5295)

__free_pages (mm/page_alloc.c:5334)

tlb_remove_table_rcu (include/linux/mm.h:? include/linux/mm.h:3122 include/asm-generic/tlb.h:220 mm/mmu_gather.c:227 mm/mmu_gather.c:290)

? __cfi_tlb_remove_table_rcu (mm/mmu_gather.c:289)

? rcu_core (kernel/rcu/tree.c:?)

rcu_core (include/linux/rcupdate.h:341 kernel/rcu/tree.c:2607 kernel/rcu/tree.c:2861)

rcu_core_si (kernel/rcu/tree.c:2879)

handle_softirqs (arch/x86/include/asm/jump_label.h:36 include/trace/events/irq.h:142 kernel/softirq.c:623)

__irq_exit_rcu (arch/x86/include/asm/jump_label.h:36 kernel/softirq.c:725)

irq_exit_rcu (kernel/softirq.c:741)

sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1052)

</IRQ>

<TASK>

RIP: 0010:_raw_spin_unlock_irqrestore (arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:194)

free_pcppages_bulk (mm/page_alloc.c:1494)

drain_pages_zone (include/linux/spinlock.h:391 mm/page_alloc.c:2632)

__drain_all_pages (mm/page_alloc.c:2731)

drain_all_pages (mm/page_alloc.c:2747)

kcompactd (mm/compaction.c:3115)

kthread (kernel/kthread.c:465)

? __cfi_kcompactd (mm/compaction.c:3166)

? __cfi_kthread (kernel/kthread.c:412)

ret_from_fork (arch/x86/kernel/process.c:164)

? __cfi_kthread (kernel/kthread.c:412)

ret_from_fork_asm (arch/x86/entry/entry_64.S:255)

</TASK>

Matthew has analyzed the report and identified that in drain_page_zone()

we are in a section protected by spin_lock(&pcp->lock) and then get an

interrupt that attempts spin_trylock() on the same lock. The code is

designed to work this way without disabling IRQs and occasionally fail the

trylock with a fallback. However, the SMP=n spinlock implementation

assumes spin_trylock() will always succeed, and thus it's normally a

no-op. Here the enabled lock debugging catches the problem, but otherwise

it could cause a corruption of the pcp structure.

The problem has been introduced by commit 574907741599 ("mm/page_alloc:

leave IRQs enabled for per-cpu page allocations"). The pcp locking scheme

recognizes the need for disabling IRQs to prevent nesting spin_trylock()

sections on SMP=n, but the need to prevent the nesting in spin_lock() has

not been recognized. Fix it by introducing local wrappers that change the

spin_lock() to spin_lock_iqsave() with SMP=n and use them in all places

that do spin_lock(&pcp->lock).

[[email protected]: add pcp_ prefix to the spin_lock_irqsave wrappers, per Steven]