possible deadlock in perf_trace_destroy

Status: upstream: reported C repro on 2019/04/15 13:38
Reported-by: syzbot+571905a7ebe006bdeeac@syzkaller.appspotmail.com
First crash: 2656d, last: 2244d
Fix bisection the fix commit could be any of (bisect log):
  9fa690a2a016 Linux 4.14.169
  4139fb08c05f Linux 4.14.187
   
▶ ▼ Similar bugs (5)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in perf_trace_destroy (3) trace 4 12 2776d 2905d 0/29 auto-closed as invalid on 2019/06/14 09:18
upstream possible deadlock in perf_trace_destroy (2) trace 4 C 2006 3094d 3180d 5/29 fixed on 2018/04/24 21:47
android-414 possible deadlock in perf_trace_destroy 4 1 2873d 2873d 0/1 auto-closed as invalid on 2019/03/09 03:11
android-414 possible deadlock in perf_trace_destroy (2) 4 1 2663d 2657d 0/1 auto-closed as invalid on 2019/10/05 09:33
upstream possible deadlock in perf_trace_destroy trace 4 C 525 3201d 3185d 0/29 closed as invalid on 2017/11/01 19:39
▶ ▼ Last patch testing requests (10)
Created Duration User Patch Repo Result
2023/01/14 21:32 9m retest repro linux-4.14.y report log
2023/01/14 20:32 11m retest repro linux-4.14.y report log
2023/01/14 12:32 10m retest repro linux-4.14.y report log
2023/01/14 01:32 10m retest repro linux-4.14.y report log
2023/01/13 16:32 9m retest repro linux-4.14.y report log
2023/01/13 05:32 10m retest repro linux-4.14.y report log
2023/01/12 19:32 10m retest repro linux-4.14.y report log
2023/01/12 09:32 11m retest repro linux-4.14.y report log
2023/01/11 23:32 9m retest repro linux-4.14.y report log
2023/01/11 14:32 10m retest repro linux-4.14.y report log
▶ ▼ Fix bisection attempts (5)
Created Duration User Patch Repo Result
2020/06/29 17:33 31m (2) bisect fix linux-4.14.y OK (2) job log
2020/05/30 17:08 25m bisect fix linux-4.14.y OK (0) job log log
2020/04/30 16:39 28m bisect fix linux-4.14.y OK (0) job log log
2020/03/31 16:15 23m bisect fix linux-4.14.y OK (0) job log log
2020/03/01 15:49 25m bisect fix linux-4.14.y OK (0) job log log

Sample crash report:
audit: type=1400 audit(1580399989.090:39): avc:  denied  { read } for  pid=7412 comm="syz-executor235" scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=netlink_generic_socket permissive=1
======================================================
WARNING: possible circular locking dependency detected
4.14.169-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor235/7412 is trying to acquire lock:
 (event_mutex){+.+.}, at: [<ffffffff8162f908>] perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234

but task is already holding lock:
 (&event->child_mutex){+.+.}, at: [<ffffffff816d7ba7>] perf_event_release_kernel+0x207/0x880 kernel/events/core.c:4397

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #5 (&event->child_mutex){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_event_for_each_child+0x8a/0x150 kernel/events/core.c:4682
       _perf_ioctl kernel/events/core.c:4869 [inline]
       perf_ioctl+0x1d9/0xe50 kernel/events/core.c:4881
       vfs_ioctl fs/ioctl.c:46 [inline]
       file_ioctl fs/ioctl.c:500 [inline]
       do_vfs_ioctl+0x7ae/0x1060 fs/ioctl.c:684
       SYSC_ioctl fs/ioctl.c:701 [inline]
       SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #4 (&cpuctx_mutex){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_event_init_cpu+0xc2/0x170 kernel/events/core.c:11226
       perf_event_init+0x2d8/0x31a kernel/events/core.c:11273
       start_kernel+0x3b6/0x67d init/main.c:620
       x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:399
       x86_64_start_kernel+0x77/0x7b arch/x86/kernel/head64.c:380
       secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:240

-> #3 (pmus_lock){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_event_init_cpu+0x2f/0x170 kernel/events/core.c:11220
       cpuhp_invoke_callback+0x1ea/0x1ab0 kernel/cpu.c:184
       cpuhp_up_callbacks kernel/cpu.c:573 [inline]
       _cpu_up+0x228/0x530 kernel/cpu.c:1135
       do_cpu_up kernel/cpu.c:1170 [inline]
       do_cpu_up+0xa6/0x160 kernel/cpu.c:1142
       cpu_up+0x1b/0x20 kernel/cpu.c:1178
       smp_init+0x157/0x170 kernel/smp.c:578
       kernel_init_freeable+0x30b/0x532 init/main.c:1066
       kernel_init+0x12/0x162 init/main.c:998
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404

-> #2 (cpu_hotplug_lock.rw_sem){++++}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36 [inline]
       percpu_down_read include/linux/percpu-rwsem.h:59 [inline]
       cpus_read_lock+0x3d/0xc0 kernel/cpu.c:295
       static_key_slow_inc+0x13/0x30 kernel/jump_label.c:123
       tracepoint_add_func kernel/tracepoint.c:223 [inline]
       tracepoint_probe_register_prio+0x4d6/0x6d0 kernel/tracepoint.c:283
       tracepoint_probe_register+0x2b/0x40 kernel/tracepoint.c:304
       trace_event_reg+0x277/0x330 kernel/trace/trace_events.c:305
       perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
       perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
       perf_trace_init+0x449/0xaa0 kernel/trace/trace_event_perf.c:221
       perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:8109
       perf_try_init_event+0x164/0x200 kernel/events/core.c:9345
       perf_init_event kernel/events/core.c:9383 [inline]
       perf_event_alloc.part.0+0xd92/0x25c0 kernel/events/core.c:9643
       perf_event_alloc kernel/events/core.c:9996 [inline]
       SYSC_perf_event_open+0xad1/0x2690 kernel/events/core.c:10100
       SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9986
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #1 (tracepoints_mutex){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       tracepoint_probe_register_prio+0x36/0x6d0 kernel/tracepoint.c:279
       tracepoint_probe_register+0x2b/0x40 kernel/tracepoint.c:304
       trace_event_reg+0x277/0x330 kernel/trace/trace_events.c:305
       perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
       perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
       perf_trace_init+0x449/0xaa0 kernel/trace/trace_event_perf.c:221
       perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:8109
       perf_try_init_event+0x164/0x200 kernel/events/core.c:9345
       perf_init_event kernel/events/core.c:9383 [inline]
       perf_event_alloc.part.0+0xd92/0x25c0 kernel/events/core.c:9643
       perf_event_alloc kernel/events/core.c:9996 [inline]
       SYSC_perf_event_open+0xad1/0x2690 kernel/events/core.c:10100
       SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9986
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #0 (event_mutex){+.+.}:
       check_prev_add kernel/locking/lockdep.c:1901 [inline]
       check_prevs_add kernel/locking/lockdep.c:2018 [inline]
       validate_chain kernel/locking/lockdep.c:2460 [inline]
       __lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234
       tp_perf_event_destroy+0x16/0x20 kernel/events/core.c:8093
       _free_event+0x330/0xe70 kernel/events/core.c:4238
       free_event+0x38/0x50 kernel/events/core.c:4265
       perf_event_release_kernel+0x364/0x880 kernel/events/core.c:4409
       perf_release+0x37/0x50 kernel/events/core.c:4435
       __fput+0x275/0x7a0 fs/file_table.c:210
       ____fput+0x16/0x20 fs/file_table.c:244
       task_work_run+0x114/0x190 kernel/task_work.c:113
       exit_task_work include/linux/task_work.h:22 [inline]
       do_exit+0xa1a/0x2cd0 kernel/exit.c:858
       do_group_exit+0x111/0x330 kernel/exit.c:955
       SYSC_exit_group kernel/exit.c:966 [inline]
       SyS_exit_group+0x1d/0x20 kernel/exit.c:964
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

other info that might help us debug this:

Chain exists of:
  event_mutex --> &cpuctx_mutex --> &event->child_mutex

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&event->child_mutex);
                               lock(&cpuctx_mutex);
                               lock(&event->child_mutex);
  lock(event_mutex);

 *** DEADLOCK ***

2 locks held by syz-executor235/7412:
 #0:  (&ctx->mutex){+.+.}, at: [<ffffffff816d7b9d>] perf_event_release_kernel+0x1fd/0x880 kernel/events/core.c:4396
 #1:  (&event->child_mutex){+.+.}, at: [<ffffffff816d7ba7>] perf_event_release_kernel+0x207/0x880 kernel/events/core.c:4397

stack backtrace:
CPU: 0 PID: 7412 Comm: syz-executor235 Not tainted 4.14.169-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x142/0x197 lib/dump_stack.c:58
 print_circular_bug.isra.0.cold+0x1cc/0x28f kernel/locking/lockdep.c:1258
 check_prev_add kernel/locking/lockdep.c:1901 [inline]
 check_prevs_add kernel/locking/lockdep.c:2018 [inline]
 validate_chain kernel/locking/lockdep.c:2460 [inline]
 __lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
 lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
 __mutex_lock_common kernel/locking/mutex.c:756 [inline]
 __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
 perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234
 tp_perf_event_destroy+0x16/0x20 kernel/events/core.c:8093
 _free_event+0x330/0xe70 kernel/events/core.c:4238
 free_event+0x38/0x50 kernel/events/core.c:4265
 perf_event_release_kernel+0x364/0x880 kernel/events/core.c:4409
 perf_release+0x37/0x50 kernel/events/core.c:4435
 __fput+0x275/0x7a0 fs/file_table.c:210
 ____fput+0x16/0x20 fs/file_table.c:244
 task_work_run+0x114/0x190 kernel/task_work.c:113
 exit_task_work include/linux/task_work.h:22 [inline]
 do_exit+0xa1a/0x2cd0 kernel/exit.c:858
 do_group_exit+0x111/0x330 kernel/exit.c:955
 SYSC_exit_group kernel/exit.c:966 [inline]
 SyS_exit_group+0x1d/0x20 kernel/exit.c:964
 do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x44d7c8
RSP: 002b:00007ffe984bae88 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000000000044d7c8
RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
RBP: 00000000004d1450 R08: 00000000000000e7 R09: ffffffffffffffd4
R10: 00007ffe984baee0 R11: 0000000000000246 R12: 0000000000000001
R13: 00000000006e4380 R14: 0000000000000040 R15: 20c49ba5e353f7cf

Crashes (829):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/01/30 16:03 linux-4.14.y 9fa690a2a016 5ed23f9a .config console log report syz C ci2-linux-4-14
2020/01/15 10:04 linux-4.14.y c04fc6fa5c96 fa12bd3c .config console log report syz C ci2-linux-4-14
2019/12/31 06:20 linux-4.14.y e1f7d50ae3a3 7f117e28 .config console log report syz C ci2-linux-4-14
2019/11/26 09:31 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report syz C ci2-linux-4-14
2019/11/25 06:45 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report syz C ci2-linux-4-14
2019/11/24 14:58 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report syz C ci2-linux-4-14
2019/11/24 02:51 linux-4.14.y f56f3d0e65ad 598ca6c8 .config console log report syz C ci2-linux-4-14
2019/11/10 13:54 linux-4.14.y c9fda4f22428 dc438b91 .config console log report syz C ci2-linux-4-14
2019/11/02 17:20 linux-4.14.y ddef1e8e3f6e 997ccc67 .config console log report syz C ci2-linux-4-14
2019/10/30 14:50 linux-4.14.y ddef1e8e3f6e 5ea87a66 .config console log report syz C ci2-linux-4-14
2019/10/28 04:31 linux-4.14.y b98aebd29824 25bb509e .config console log report syz C ci2-linux-4-14
2019/10/24 16:33 linux-4.14.y b98aebd29824 d01bb02a .config console log report syz C ci2-linux-4-14
2019/10/23 08:08 linux-4.14.y b98aebd29824 d0686497 .config console log report syz C ci2-linux-4-14
2019/10/21 19:47 linux-4.14.y b98aebd29824 b24d2b8a .config console log report syz C ci2-linux-4-14
2019/10/20 00:31 linux-4.14.y b98aebd29824 8c88c9c1 .config console log report syz C ci2-linux-4-14
2019/10/10 00:41 linux-4.14.y 42327896f194 c4b9981b .config console log report syz C ci2-linux-4-14
2019/10/09 14:59 linux-4.14.y 42327896f194 312c6a5a .config console log report syz C ci2-linux-4-14
2019/10/08 05:43 linux-4.14.y 42327896f194 28ac6e64 .config console log report syz C ci2-linux-4-14
2019/10/04 18:25 linux-4.14.y f6e27dbb1afa c86336cf .config console log report syz C ci2-linux-4-14
2019/10/02 23:12 linux-4.14.y f6e27dbb1afa 2e29b534 .config console log report syz C ci2-linux-4-14
2019/10/02 08:38 linux-4.14.y f6e27dbb1afa b7a87a83 .config console log report syz C ci2-linux-4-14
2019/09/22 23:48 linux-4.14.y f6e27dbb1afa d96e88f3 .config console log report syz C ci2-linux-4-14
2019/09/22 00:05 linux-4.14.y f6e27dbb1afa d96e88f3 .config console log report syz C ci2-linux-4-14
2019/09/21 20:43 linux-4.14.y f6e27dbb1afa d96e88f3 .config console log report syz C ci2-linux-4-14
2019/09/16 11:40 linux-4.14.y 968722f5371a 55c50e70 .config console log report syz C ci2-linux-4-14
2019/09/08 23:36 linux-4.14.y 414510bc00a5 a60cb4cd .config console log report syz C ci2-linux-4-14
2019/09/02 02:59 linux-4.14.y 01fd1694b93c bad3cce2 .config console log report syz C ci2-linux-4-14
2019/08/27 01:03 linux-4.14.y b5260801526c d21c5d9d .config console log report syz C ci2-linux-4-14
2019/08/26 23:30 linux-4.14.y b5260801526c d21c5d9d .config console log report syz C ci2-linux-4-14
2019/08/16 11:50 linux-4.14.y 45f092f9e9cb 8fd428a1 .config console log report syz C ci2-linux-4-14
2019/08/13 14:57 linux-4.14.y 3ffe1e79c174 74d61399 .config console log report syz C ci2-linux-4-14
2019/08/12 12:14 linux-4.14.y 3ffe1e79c174 8620c2c2 .config console log report syz C ci2-linux-4-14
2019/08/11 03:12 linux-4.14.y 3ffe1e79c174 acb51638 .config console log report syz C ci2-linux-4-14
2019/08/06 16:03 linux-4.14.y 7d80e1218adf da562c0b .config console log report syz C ci2-linux-4-14
2019/08/06 12:28 linux-4.14.y 7d80e1218adf 6affd8e8 .config console log report syz C ci2-linux-4-14
2019/08/05 10:12 linux-4.14.y 7d80e1218adf 6affd8e8 .config console log report syz C ci2-linux-4-14
2019/08/05 09:16 linux-4.14.y 7d80e1218adf 6affd8e8 .config console log report syz C ci2-linux-4-14
2019/04/15 20:32 linux-4.14.y 1ec8f1f0bffe 505ab413 .config console log report syz C ci2-linux-4-14
2019/11/25 15:10 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report syz ci2-linux-4-14
2019/11/20 11:56 linux-4.14.y 775d01b65b5d 432c7650 .config console log report syz ci2-linux-4-14
2019/11/16 16:12 linux-4.14.y 775d01b65b5d cdac920b .config console log report syz ci2-linux-4-14
2019/11/16 11:15 linux-4.14.y 775d01b65b5d cdac920b .config console log report syz ci2-linux-4-14
2020/01/31 15:46 linux-4.14.y 9fa690a2a016 5ed23f9a .config console log report ci2-linux-4-14
2020/01/31 10:05 linux-4.14.y 9fa690a2a016 5ed23f9a .config console log report ci2-linux-4-14
2020/01/31 01:48 linux-4.14.y 9fa690a2a016 5ed23f9a .config console log report ci2-linux-4-14
2020/01/29 15:33 linux-4.14.y 9fa690a2a016 5ed23f9a .config console log report ci2-linux-4-14
2020/01/25 19:18 linux-4.14.y 8bac50406cca 2e95ab33 .config console log report ci2-linux-4-14
2020/01/25 12:52 linux-4.14.y 8bac50406cca 2e95ab33 .config console log report ci2-linux-4-14
2020/01/23 02:44 linux-4.14.y c1141b3aab36 3334d684 .config console log report ci2-linux-4-14
2020/01/21 12:56 linux-4.14.y c1141b3aab36 8eda0b95 .config console log report ci2-linux-4-14
2020/01/16 08:40 linux-4.14.y c04fc6fa5c96 f9b69507 .config console log report ci2-linux-4-14
2020/01/13 11:34 linux-4.14.y 6d0c334a400d 99565c1a .config console log report ci2-linux-4-14
2020/01/13 04:04 linux-4.14.y 6d0c334a400d 53faa9fe .config console log report ci2-linux-4-14
2020/01/12 13:57 linux-4.14.y 6d0c334a400d 31290a45 .config console log report ci2-linux-4-14
2020/01/11 04:01 linux-4.14.y b0cdffaa546e 4c04afaa .config console log report ci2-linux-4-14
2020/01/07 22:32 linux-4.14.y 84f5ad468100 6738e0b3 .config console log report ci2-linux-4-14
2020/01/06 20:25 linux-4.14.y 84f5ad468100 53430d97 .config console log report ci2-linux-4-14
2020/01/05 04:17 linux-4.14.y 84f5ad468100 68256974 .config console log report ci2-linux-4-14
2020/01/04 03:17 linux-4.14.y 4c5bf01e16a7 68256974 .config console log report ci2-linux-4-14
2019/12/29 02:14 linux-4.14.y e1f7d50ae3a3 af6b8ef8 .config console log report ci2-linux-4-14
2019/12/25 06:20 linux-4.14.y e1f7d50ae3a3 be5c2c81 .config console log report ci2-linux-4-14
2019/12/20 14:36 linux-4.14.y bfb9e5c03076 34011c05 .config console log report ci2-linux-4-14
2019/12/20 13:31 linux-4.14.y bfb9e5c03076 34011c05 .config console log report ci2-linux-4-14
2019/12/18 19:49 linux-4.14.y bfb9e5c03076 79b211f7 .config console log report ci2-linux-4-14
2019/12/18 09:23 linux-4.14.y bfb9e5c03076 64ca0a37 .config console log report ci2-linux-4-14
2019/12/17 04:50 linux-4.14.y a844dc4c5442 d13d7958 .config console log report ci2-linux-4-14
2019/12/13 18:56 linux-4.14.y a844dc4c5442 a5c1ab05 .config console log report ci2-linux-4-14
2019/12/06 12:02 linux-4.14.y a844dc4c5442 12c3b6cd .config console log report ci2-linux-4-14
2019/12/06 05:10 linux-4.14.y a844dc4c5442 98b4ef2d .config console log report ci2-linux-4-14
2019/12/06 01:40 linux-4.14.y a844dc4c5442 98b4ef2d .config console log report ci2-linux-4-14
2019/12/06 00:35 linux-4.14.y a844dc4c5442 98b4ef2d .config console log report ci2-linux-4-14
2019/12/05 05:40 linux-4.14.y fbc5fe7a54d0 b2088328 .config console log report ci2-linux-4-14
2019/12/04 15:53 linux-4.14.y fbc5fe7a54d0 b2088328 .config console log report ci2-linux-4-14
2019/12/02 20:09 linux-4.14.y fbc5fe7a54d0 ab342da3 .config console log report ci2-linux-4-14
2019/12/01 20:28 linux-4.14.y fbc5fe7a54d0 f879db37 .config console log report ci2-linux-4-14
2019/11/30 23:24 linux-4.14.y 43598c571e7e a76bf83f .config console log report ci2-linux-4-14
2019/11/28 22:24 linux-4.14.y 43598c571e7e 76357d6f .config console log report ci2-linux-4-14
2019/11/28 10:24 linux-4.14.y 43598c571e7e 97264cb1 .config console log report ci2-linux-4-14
2019/11/28 05:32 linux-4.14.y 43598c571e7e 0d63f89c .config console log report ci2-linux-4-14
2019/11/27 16:53 linux-4.14.y 43598c571e7e 0d63f89c .config console log report ci2-linux-4-14
2019/11/26 22:01 linux-4.14.y 43598c571e7e 1048481f .config console log report ci2-linux-4-14
2019/11/26 20:00 linux-4.14.y 43598c571e7e 1048481f .config console log report ci2-linux-4-14
2019/11/26 16:02 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report ci2-linux-4-14
2019/11/26 13:13 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report ci2-linux-4-14
2019/11/26 04:47 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report ci2-linux-4-14
2019/11/26 00:39 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report ci2-linux-4-14
2019/11/25 19:45 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report ci2-linux-4-14
2019/11/25 13:44 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report ci2-linux-4-14
2019/11/25 01:48 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report ci2-linux-4-14
2019/11/24 23:58 linux-4.14.y 43598c571e7e 598ca6c8 .config console log report ci2-linux-4-14
2019/04/15 12:37 linux-4.14.y 1ec8f1f0bffe 505ab413 .config console log report ci2-linux-4-14
* Struck through repros no longer work on HEAD.