Root Cause Analysis
1. Mechanism and Invariant Violation
The functions perf_swevent_get_recursion_context() and perf_swevent_put_recursion_context() (kernel/events/core.c:11233-11242) track software event recursion by indexing into a per-task context array, current->perf_recursion[PERF_NR_CONTEXTS].
The helper get_recursion_context(u8 *recursion) in kernel/events/internal.h:214-225:
static inline int get_recursion_context(u8 *recursion)
{
unsigned char rctx = interrupt_context_level();
if (recursion[rctx])
return -1;
recursion[rctx]++;
barrier();
return rctx;
}
interrupt_context_level() (include/linux/preempt.h:118-128) computes the execution level based on preempt_count():
0 - task (normal context)
1 - softirq context
2 - hardirq context
3 - NMI context
get_recursion_context() and put_recursion_context() use plain, non-atomic reads and increments/decrements with only compiler barriers. This mechanism relies on an architectural invariant:
- Interrupts (softirq, hardirq, NMI) operate at
rctx >= 1, accessing disjoint entries of the array without colliding with recursion[0].
- For
rctx == 0 (task context), execution must be non-preemptible to guarantee that multiple task-context callers on the same task cannot execute re-entrantly or concurrently.
Standard callers adhere to this requirement:
__perf_sw_event() runs enclosed within preempt_disable_notrace() and preempt_enable_notrace().
perf_trace_##call() macros run under guard(preempt_notrace)().
perf_pending_irq() and perf_pending_disable() execute from irq_work_run() in interrupt or softirq contexts (rctx >= 1).
2. Triggering Condition in perf_pending_task()
In contrast to the other callers, perf_pending_task() (kernel/events/core.c:7767-7787) is queued via task_work_add() and executed from task_work_run() in normal, sleepable, preemptible task context (rctx == 0). Both IRQs and preemption are enabled throughout the function.
- Task
T enters perf_pending_task() and invokes perf_swevent_get_recursion_context().
- Inside
get_recursion_context(), interrupt_context_level() returns 0, and T reads current->perf_recursion[0].
- An asynchronous interrupt (e.g.,
sysvec_apic_timer_interrupt) interrupts task T.
- The interrupt executes. On return to kernel mode (
irqentry_exit() -> irqentry_exit_to_kernel_mode_preempt() -> irqentry_exit_cond_resched()), the hardirq state has already ended (!preempt_count()).
- Because rescheduling is needed (
need_resched()), preempt_schedule_irq() calls __schedule(SM_PREEMPT).
- Inside
preempt_schedule_irq(), preempt_disable() sets only PREEMPT_OFFSET. Therefore, interrupt_context_level() evaluates to 0. Prior to switching out the running process, current is still task T.
__schedule() fires the trace_sched_switch tracepoint. The attached perf handler (perf_trace_sched_switch() -> perf_trace_buf_alloc()) invokes perf_swevent_get_recursion_context().
get_recursion_context() reads and increments current->perf_recursion[0], concurrently racing with the preempted access in perf_pending_task() on task T.
Key Functions, Structs, and Files Involved