CPU hang during compilation

The system suddenly seems stuck while compiling codes.
Only serial port log can be seen as follow:

[ 5228.269855] watchdog: BUG: soft lockup - CPU#1 stuck for 917s! [migration/1:21]
[ 5228.277167] Modules linked in: overlay xsk_diag nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr sunrpc ofpart sophgo_spifmc spi_nor mtd uio_pdrv_genirq uio ip6_tables ip_tables loop zram spi_dw_mmio spi_dw gpio_dwapb r8169 scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath
[ 5228.320824] CPU: 1 PID: 21 Comm: migration/1 Tainted: G L 6.1.55 #1
[ 5228.328473] Hardware name: Sophgo Mango (DT)
[ 5228.332737] Stopper: multi_cpu_stop+0x0/0x146 ← migrate_swap+0xae/0x112
[ 5228.339439] epc : rcu_momentary_dyntick_idle+0x3a/0x80
[ 5228.344575] ra : multi_cpu_stop+0xde/0x146
[ 5228.348754] epc : ffffffff8009003a ra : ffffffff800deeca sp : ffffffc80a553d80
[ 5228.355968] gp : ffffffff81e76ff0 tp : ffffffd8ffda0000 t0 : 0000000000000080
[ 5228.363181] t1 : 0000000001806000 t2 : 0000000000004833 s0 : ffffffc80a553e00
[ 5228.370394] s1 : ffffffc81a3c39f8 a0 : ffffffff8120e228 a1 : ffffffc81a3c3a20
[ 5228.377606] a2 : ffffffc81a3c3a20 a3 : ffffffff81efc588 a4 : 00000000750dde7c
[ 5228.384819] a5 : fffffff6df72a848 a6 : 0000000000000007 a7 : 0000000000000008
[ 5228.392032] s2 : ffffffc81a3c3a1c s3 : ffffffffffffffff s4 : ffffffff8120e228
[ 5228.399246] s5 : 0000000000000001 s6 : 0000000000000001 s7 : 0000000000000003
[ 5228.406459] s8 : 0000000000000004 s9 : 0000000000000002 s10: 0000000000000000
[ 5228.413672] s11: 0000000000000001 t3 : 0000000000000000 t4 : 000000000000778f
[ 5228.420885] t5 : ffffffe85e43d900 t6 : 0000000000000003
[ 5228.426189] status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000005
[ 5228.434096] [] rcu_momentary_dyntick_idle+0x3a/0x80
[ 5228.440533] [] cpu_stopper_thread+0x102/0x180
[ 5228.446449] [] smpboot_thread_fn+0xe6/0x11a
[ 5228.452192] [] kthread+0xbc/0xd2
[ 5228.456978] [] ret_from_exception+0x0/0x16
[ 5238.329686] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 413s!
[ 5238.337721] BUG: workqueue lockup - pool cpus=58 node=3 flags=0x0 nice=0 stuck for 979s!
[ 5238.345826] Showing busy workqueues and worker pools:
[ 5238.350890] workqueue events: flags=0x0
[ 5238.354737] pwq 42: cpus=21 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5238.354748] in-flight: 3012:netstamp_clear
[ 5238.354802] workqueue events_unbound: flags=0x2
[ 5238.370729] pwq 130: cpus=8-15,24-31 node=1 flags=0x4 nice=0 active=2/512 refcnt=4
[ 5238.370741] pending: flush_to_ldisc, flush_to_ldisc BAR(1752)
[ 5238.370785] workqueue events_power_efficient: flags=0x80
[ 5238.389951] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
[ 5238.389962] pending: neigh_periodic_work, neigh_managed_work
[ 5238.390032] workqueue mm_percpu_wq: flags=0x8
[ 5238.407165] pwq 116: cpus=58 node=3 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5238.407176] pending: vmstat_update
[ 5238.407210] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5238.407220] pending: vmstat_update
[ 5238.407229] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5238.407238] pending: vmstat_update
[ 5238.407317] workqueue writeback: flags=0x4a
[ 5238.443422] pwq 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 active=1/256 refcnt=2
[ 5238.443433] in-flight: 222:wb_workfn
[ 5238.443443] pwq 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
[ 5238.443454] in-flight: 14490:wb_workfn
[ 5238.443732] pool 42: cpus=21 node=0 flags=0x0 nice=0 hung=0s workers=2 idle: 9576
[ 5238.443770] pool 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 11998 15044 8792
[ 5238.443786] pool 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 hung=0s workers=3 manager: 11900 idle: 11976
[ 5256.269346] watchdog: BUG: soft lockup - CPU#1 stuck for 943s! [migration/1:21]
[ 5256.276657] Modules linked in: overlay xsk_diag nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr sunrpc ofpart sophgo_spifmc spi_nor mtd uio_pdrv_genirq uio ip6_tables ip_tables loop zram spi_dw_mmio spi_dw gpio_dwapb r8169 scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath
[ 5256.320312] CPU: 1 PID: 21 Comm: migration/1 Tainted: G L 6.1.55 #1
[ 5256.327961] Hardware name: Sophgo Mango (DT)
[ 5256.332225] Stopper: multi_cpu_stop+0x0/0x146 ← migrate_swap+0xae/0x112
[ 5256.338926] epc : rcu_momentary_dyntick_idle+0x3a/0x80
[ 5256.344064] ra : multi_cpu_stop+0xde/0x146
[ 5256.348243] epc : ffffffff8009003a ra : ffffffff800deeca sp : ffffffc80a553d80
[ 5256.355456] gp : ffffffff81e76ff0 tp : ffffffd8ffda0000 t0 : 0000000000000080
[ 5256.362669] t1 : 0000000001806000 t2 : 0000000000004833 s0 : ffffffc80a553e00
[ 5256.369883] s1 : ffffffc81a3c39f8 a0 : ffffffff8120e228 a1 : ffffffc81a3c3a20
[ 5256.377095] a2 : ffffffc81a3c3a20 a3 : ffffffff81efc588 a4 : 000000005355c3e4
[ 5256.384308] a5 : fffffff6df72a848 a6 : 0000000000000007 a7 : 0000000000000008
[ 5256.391522] s2 : ffffffc81a3c3a1c s3 : ffffffffffffffff s4 : ffffffff8120e228
[ 5256.398734] s5 : 0000000000000001 s6 : 0000000000000001 s7 : 0000000000000003
[ 5256.405947] s8 : 0000000000000004 s9 : 0000000000000002 s10: 0000000000000000
[ 5256.413160] s11: 0000000000000001 t3 : 0000000000000000 t4 : 000000000000778f
[ 5256.420372] t5 : ffffffe85e43d900 t6 : 0000000000000003
[ 5256.425677] status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000005
[ 5256.433584] [] rcu_momentary_dyntick_idle+0x3a/0x80
[ 5256.440020] [] cpu_stopper_thread+0x102/0x180
[ 5256.445934] [] smpboot_thread_fn+0xe6/0x11a
[ 5256.451678] [] kthread+0xbc/0xd2
[ 5256.456465] [] ret_from_exception+0x0/0x16
[ 5269.029129] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 443s!
[ 5269.037163] BUG: workqueue lockup - pool cpus=58 node=3 flags=0x0 nice=0 stuck for 1010s!
[ 5269.045355] Showing busy workqueues and worker pools:
[ 5269.050417] workqueue events: flags=0x0
[ 5269.054264] pwq 42: cpus=21 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5269.054276] in-flight: 3012:netstamp_clear
[ 5269.054330] workqueue events_unbound: flags=0x2
[ 5269.070256] pwq 130: cpus=8-15,24-31 node=1 flags=0x4 nice=0 active=2/512 refcnt=4
[ 5269.070268] pending: flush_to_ldisc, flush_to_ldisc BAR(1752)
[ 5269.070311] workqueue events_power_efficient: flags=0x80
[ 5269.089474] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
[ 5269.089485] pending: neigh_periodic_work, neigh_managed_work
[ 5269.089555] workqueue mm_percpu_wq: flags=0x8
[ 5269.106689] pwq 116: cpus=58 node=3 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5269.106700] pending: vmstat_update
[ 5269.106734] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5269.106744] pending: vmstat_update
[ 5269.106752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5269.106761] pending: vmstat_update
[ 5269.106841] workqueue writeback: flags=0x4a
[ 5269.142947] pwq 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 active=1/256 refcnt=2
[ 5269.142958] in-flight: 222:wb_workfn
[ 5269.142968] pwq 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
[ 5269.142978] in-flight: 14490:wb_workfn
[ 5269.143251] pool 42: cpus=21 node=0 flags=0x0 nice=0 hung=0s workers=2 idle: 9576
[ 5269.143289] pool 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 11998 15044 8792
[ 5269.143305] pool 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 hung=0s workers=3 manager: 11900 idle: 11976
[ 5284.268838] watchdog: BUG: soft lockup - CPU#1 stuck for 969s! [migration/1:21]
[ 5284.276151] Modules linked in: overlay xsk_diag nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr sunrpc ofpart sophgo_spifmc spi_nor mtd uio_pdrv_genirq uio ip6_tables ip_tables loop zram spi_dw_mmio spi_dw gpio_dwapb r8169 scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath
[ 5284.319809] CPU: 1 PID: 21 Comm: migration/1 Tainted: G L 6.1.55 #1
[ 5284.327458] Hardware name: Sophgo Mango (DT)
[ 5284.331722] Stopper: multi_cpu_stop+0x0/0x146 ← migrate_swap+0xae/0x112
[ 5284.338424] epc : rcu_momentary_dyntick_idle+0x3a/0x80
[ 5284.343561] ra : multi_cpu_stop+0xde/0x146
[ 5284.347740] epc : ffffffff8009003a ra : ffffffff800deeca sp : ffffffc80a553d80
[ 5284.354954] gp : ffffffff81e76ff0 tp : ffffffd8ffda0000 t0 : 0000000000000080
[ 5284.362167] t1 : 0000000001806000 t2 : 0000000000004833 s0 : ffffffc80a553e00
[ 5284.369381] s1 : ffffffc81a3c39f8 a0 : ffffffff8120e228 a1 : ffffffc81a3c3a20
[ 5284.376594] a2 : ffffffc81a3c3a20 a3 : ffffffff81efc588 a4 : 00000000319de97c
[ 5284.383807] a5 : fffffff6df72a848 a6 : 0000000000000007 a7 : 0000000000000008
[ 5284.391021] s2 : ffffffc81a3c3a1c s3 : ffffffffffffffff s4 : ffffffff8120e228
[ 5284.398235] s5 : 0000000000000001 s6 : 0000000000000001 s7 : 0000000000000003
[ 5284.405448] s8 : 0000000000000004 s9 : 0000000000000002 s10: 0000000000000000
[ 5284.412660] s11: 0000000000000001 t3 : 0000000000000000 t4 : 000000000000778f
[ 5284.419873] t5 : ffffffe85e43d900 t6 : 0000000000000003
[ 5284.425177] status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000005
[ 5284.433085] [] rcu_momentary_dyntick_idle+0x3a/0x80
[ 5284.439522] [] cpu_stopper_thread+0x102/0x180
[ 5284.445437] [] smpboot_thread_fn+0xe6/0x11a
[ 5284.451183] [] kthread+0xbc/0xd2
[ 5284.455968] [] ret_from_exception+0x0/0x16
[ 5299.748572] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 474s!
[ 5299.756610] BUG: workqueue lockup - pool cpus=58 node=3 flags=0x0 nice=0 stuck for 1041s!
[ 5299.764805] Showing busy workqueues and worker pools:
[ 5299.769869] workqueue events: flags=0x0
[ 5299.773715] pwq 42: cpus=21 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5299.773727] in-flight: 3012:netstamp_clear
[ 5299.773783] workqueue events_unbound: flags=0x2
[ 5299.789711] pwq 130: cpus=8-15,24-31 node=1 flags=0x4 nice=0 active=2/512 refcnt=4
[ 5299.789723] pending: flush_to_ldisc, flush_to_ldisc BAR(1752)
[ 5299.789767] workqueue events_power_efficient: flags=0x80
[ 5299.808928] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
[ 5299.808939] pending: neigh_periodic_work, neigh_managed_work
[ 5299.809009] workqueue mm_percpu_wq: flags=0x8
[ 5299.826143] pwq 116: cpus=58 node=3 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5299.826153] pending: vmstat_update
[ 5299.826187] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5299.826196] pending: vmstat_update
[ 5299.826206] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5299.826215] pending: vmstat_update
[ 5299.826294] workqueue writeback: flags=0x4a
[ 5299.862398] pwq 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 active=1/256 refcnt=2
[ 5299.862410] in-flight: 222:wb_workfn
[ 5299.862419] pwq 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
[ 5299.862430] in-flight: 14490:wb_workfn
[ 5299.862727] pool 42: cpus=21 node=0 flags=0x0 nice=0 hung=0s workers=2 idle: 9576
[ 5299.862765] pool 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 11998 15044 8792
[ 5299.862781] pool 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 hung=0s workers=3 manager: 11900 idle: 11976
[ 5312.268328] watchdog: BUG: soft lockup - CPU#1 stuck for 995s! [migration/1:21]
[ 5312.275639] Modules linked in: overlay xsk_diag nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr sunrpc ofpart sophgo_spifmc spi_nor mtd uio_pdrv_genirq uio ip6_tables ip_tables loop zram spi_dw_mmio spi_dw gpio_dwapb r8169 scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath
[ 5312.319295] CPU: 1 PID: 21 Comm: migration/1 Tainted: G L 6.1.55 #1
[ 5312.326945] Hardware name: Sophgo Mango (DT)
[ 5312.331208] Stopper: multi_cpu_stop+0x0/0x146 ← migrate_swap+0xae/0x112
[ 5312.337909] epc : rcu_momentary_dyntick_idle+0x3a/0x80
[ 5312.343048] ra : multi_cpu_stop+0xde/0x146
[ 5312.347228] epc : ffffffff8009003a ra : ffffffff800deeca sp : ffffffc80a553d80
[ 5312.354442] gp : ffffffff81e76ff0 tp : ffffffd8ffda0000 t0 : 0000000000000080
[ 5312.361655] t1 : 0000000001806000 t2 : 0000000000004833 s0 : ffffffc80a553e00
[ 5312.368868] s1 : ffffffc81a3c39f8 a0 : ffffffff8120e228 a1 : ffffffc81a3c3a20
[ 5312.376082] a2 : ffffffc81a3c3a20 a3 : ffffffff81efc588 a4 : 000000000fe5d88c
[ 5312.383295] a5 : fffffff6df72a848 a6 : 0000000000000007 a7 : 0000000000000008
[ 5312.390509] s2 : ffffffc81a3c3a1c s3 : ffffffffffffffff s4 : ffffffff8120e228
[ 5312.397723] s5 : 0000000000000001 s6 : 0000000000000001 s7 : 0000000000000003
[ 5312.404934] s8 : 0000000000000004 s9 : 0000000000000002 s10: 0000000000000000
[ 5312.412147] s11: 0000000000000001 t3 : 0000000000000000 t4 : 000000000000778f
[ 5312.419360] t5 : ffffffe85e43d900 t6 : 0000000000000003
[ 5312.424664] status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000005
[ 5312.432572] [] rcu_momentary_dyntick_idle+0x3a/0x80
[ 5312.439007] [] cpu_stopper_thread+0x102/0x180
[ 5312.444922] [] smpboot_thread_fn+0xe6/0x11a
[ 5312.450666] [] kthread+0xbc/0xd2
[ 5312.455452] [] ret_from_exception+0x0/0x16
[ 5330.468015] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 505s!
[ 5330.476047] BUG: workqueue lockup - pool cpus=58 node=3 flags=0x0 nice=0 stuck for 1072s!
[ 5330.484240] Showing busy workqueues and worker pools:
[ 5330.489303] workqueue events: flags=0x0
[ 5330.493150] pwq 42: cpus=21 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5330.493161] in-flight: 3012:netstamp_clear
[ 5330.493217] workqueue events_unbound: flags=0x2
[ 5330.509144] pwq 130: cpus=8-15,24-31 node=1 flags=0x4 nice=0 active=2/512 refcnt=4
[ 5330.509156] pending: flush_to_ldisc, flush_to_ldisc BAR(1752)
[ 5330.509199] workqueue events_power_efficient: flags=0x80
[ 5330.528363] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
[ 5330.528374] pending: neigh_periodic_work, neigh_managed_work
[ 5330.528444] workqueue mm_percpu_wq: flags=0x8
[ 5330.545579] pwq 116: cpus=58 node=3 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5330.545590] pending: vmstat_update
[ 5330.545624] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5330.545633] pending: vmstat_update
[ 5330.545642] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5330.545651] pending: vmstat_update
[ 5330.545730] workqueue writeback: flags=0x4a
[ 5330.581832] pwq 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 active=1/256 refcnt=2
[ 5330.581844] in-flight: 222:wb_workfn
[ 5330.581854] pwq 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
[ 5330.581864] in-flight: 14490:wb_workfn
[ 5330.582133] pool 42: cpus=21 node=0 flags=0x0 nice=0 hung=0s workers=2 idle: 9576
[ 5330.582170] pool 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 11998 15044 8792
[ 5330.582186] pool 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 hung=0s workers=3 manager: 11900 idle: 11976
[ 5340.267820] watchdog: BUG: soft lockup - CPU#1 stuck for 1021s! [migration/1:21]
[ 5340.275219] Modules linked in: overlay xsk_diag nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr sunrpc ofpart sophgo_spifmc spi_nor mtd uio_pdrv_genirq uio ip6_tables ip_tables loop zram spi_dw_mmio spi_dw gpio_dwapb r8169 scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath
[ 5340.318875] CPU: 1 PID: 21 Comm: migration/1 Tainted: G L 6.1.55 #1
[ 5340.326524] Hardware name: Sophgo Mango (DT)
[ 5340.330788] Stopper: multi_cpu_stop+0x0/0x146 ← migrate_swap+0xae/0x112
[ 5340.337490] epc : rcu_momentary_dyntick_idle+0x3a/0x80
[ 5340.342627] ra : multi_cpu_stop+0xde/0x146
[ 5340.346806] epc : ffffffff8009003a ra : ffffffff800deeca sp : ffffffc80a553d80
[ 5340.354019] gp : ffffffff81e76ff0 tp : ffffffd8ffda0000 t0 : 0000000000000080
[ 5340.361232] t1 : 0000000001806000 t2 : 0000000000004833 s0 : ffffffc80a553e00
[ 5340.368445] s1 : ffffffc81a3c39f8 a0 : ffffffff8120e228 a1 : ffffffc81a3c3a20
[ 5340.375658] a2 : ffffffc81a3c3a20 a3 : ffffffff81efc588 a4 : ffffffffee2d74f4
[ 5340.382871] a5 : fffffff6df72a848 a6 : 0000000000000007 a7 : 0000000000000008
[ 5340.390084] s2 : ffffffc81a3c3a1c s3 : ffffffffffffffff s4 : ffffffff8120e228
[ 5340.397298] s5 : 0000000000000001 s6 : 0000000000000001 s7 : 0000000000000003
[ 5340.404510] s8 : 0000000000000004 s9 : 0000000000000002 s10: 0000000000000000
[ 5340.411723] s11: 0000000000000001 t3 : 0000000000000000 t4 : 000000000000778f
[ 5340.418936] t5 : ffffffe85e43d900 t6 : 0000000000000003
[ 5340.424240] status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000005
[ 5340.432147] [] rcu_momentary_dyntick_idle+0x3a/0x80
[ 5340.438584] [] cpu_stopper_thread+0x102/0x180
[ 5340.444498] [] smpboot_thread_fn+0xe6/0x11a
[ 5340.450242] [] kthread+0xbc/0xd2
[ 5340.455029] [] ret_from_exception+0x0/0x16
[ 5361.177457] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 535s!
[ 5361.185495] BUG: workqueue lockup - pool cpus=58 node=3 flags=0x0 nice=0 stuck for 1102s!
[ 5361.193689] Showing busy workqueues and worker pools:
[ 5361.198753] workqueue events: flags=0x0
[ 5361.202600] pwq 42: cpus=21 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5361.202611] in-flight: 3012:netstamp_clear
[ 5361.202667] workqueue events_unbound: flags=0x2
[ 5361.218594] pwq 130: cpus=8-15,24-31 node=1 flags=0x4 nice=0 active=2/512 refcnt=4
[ 5361.218607] pending: flush_to_ldisc, flush_to_ldisc BAR(1752)
[ 5361.218651] workqueue events_power_efficient: flags=0x80
[ 5361.237818] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
[ 5361.237830] pending: neigh_periodic_work, neigh_managed_work
[ 5361.237899] workqueue mm_percpu_wq: flags=0x8
[ 5361.255034] pwq 116: cpus=58 node=3 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5361.255045] pending: vmstat_update
[ 5361.255079] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5361.255088] pending: vmstat_update
[ 5361.255097] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5361.255106] pending: vmstat_update
[ 5361.255185] workqueue writeback: flags=0x4a
[ 5361.291290] pwq 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 active=1/256 refcnt=2
[ 5361.291302] in-flight: 222:wb_workfn
[ 5361.291312] pwq 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
[ 5361.291323] in-flight: 14490:wb_workfn
[ 5361.291619] pool 42: cpus=21 node=0 flags=0x0 nice=0 hung=0s workers=2 idle: 9576
[ 5361.291657] pool 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 11998 15044 8792
[ 5361.291673] pool 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 hung=0s workers=3 manager: 11900 idle: 11976
[ 5368.267313] watchdog: BUG: soft lockup - CPU#1 stuck for 1047s! [migration/1:21]
[ 5368.274713] Modules linked in: overlay xsk_diag nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr sunrpc ofpart sophgo_spifmc spi_nor mtd uio_pdrv_genirq uio ip6_tables ip_tables loop zram spi_dw_mmio spi_dw gpio_dwapb r8169 scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath
[ 5368.318372] CPU: 1 PID: 21 Comm: migration/1 Tainted: G L 6.1.55 #1
[ 5368.326022] Hardware name: Sophgo Mango (DT)
[ 5368.330286] Stopper: multi_cpu_stop+0x0/0x146 ← migrate_swap+0xae/0x112
[ 5368.336987] epc : rcu_momentary_dyntick_idle+0x3a/0x80
[ 5368.342123] ra : multi_cpu_stop+0xde/0x146
[ 5368.346302] epc : ffffffff8009003a ra : ffffffff800deeca sp : ffffffc80a553d80
[ 5368.353516] gp : ffffffff81e76ff0 tp : ffffffd8ffda0000 t0 : 0000000000000080
[ 5368.360729] t1 : 0000000001806000 t2 : 0000000000004833 s0 : ffffffc80a553e00
[ 5368.367942] s1 : ffffffc81a3c39f8 a0 : ffffffff8120e228 a1 : ffffffc81a3c3a20
[ 5368.375155] a2 : ffffffc81a3c3a20 a3 : ffffffff81efc588 a4 : ffffffffcc760194
[ 5368.382367] a5 : fffffff6df72a848 a6 : 0000000000000007 a7 : 0000000000000008
[ 5368.389580] s2 : ffffffc81a3c3a1c s3 : ffffffffffffffff s4 : ffffffff8120e228
[ 5368.396793] s5 : 0000000000000001 s6 : 0000000000000001 s7 : 0000000000000003
[ 5368.404006] s8 : 0000000000000004 s9 : 0000000000000002 s10: 0000000000000000
[ 5368.411218] s11: 0000000000000001 t3 : 0000000000000000 t4 : 000000000000778f
[ 5368.418432] t5 : ffffffe85e43d900 t6 : 0000000000000003
[ 5368.423736] status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000005
[ 5368.431644] [] rcu_momentary_dyntick_idle+0x3a/0x80
[ 5368.438079] [] cpu_stopper_thread+0x102/0x180
[ 5368.443994] [] smpboot_thread_fn+0xe6/0x11a
[ 5368.449737] [] kthread+0xbc/0xd2
[ 5368.454523] [] ret_from_exception+0x0/0x16
[ 5387.246966] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 5387.252892] rcu: 9-…0: (19 ticks this GP) idle=529c/1/0x4000000000000000 softirq=49058/49060 fqs=33874
[ 5387.262456] rcu: 24-…0: (11 GPs behind) idle=853c/0/0x1 softirq=51569/51569 fqs=33874
[ 5387.270542] rcu: 25-…0: (15 ticks this GP) idle=66fc/1/0x4000000000000002 softirq=41205/41207 fqs=33874
[ 5387.280188] rcu: 28-…0: (15 ticks this GP) idle=dcec/1/0x4000000000000000 softirq=44694/44696 fqs=33875
[ 5387.289833] rcu: 33-…0: (14 ticks this GP) idle=ef4c/1/0x4000000000000000 softirq=34282/34282 fqs=33875
[ 5387.299489] (detected by 6, t=114146 jiffies, g=179393, q=38096 ncpus=64)
[ 5387.306359] Task dump for CPU 9:
[ 5387.309583] task:cc1plus state:R running task stack:0 pid:13813 ppid:13812 flags:0x00000008
[ 5387.319494] Call Trace:
[ 5387.321939] [] __schedule+0x2b2/0x8fe
[ 5387.327167] Task dump for CPU 24:
[ 5387.330476] task:swapper/24 state:R running task stack:0 pid:0 ppid:1 flags:0x00000008
[ 5387.340384] Call Trace:
[ 5387.342828] [] __schedule+0x2b2/0x8fe
[ 5387.348050] Task dump for CPU 25:
[ 5387.351361] task:cc1plus state:R running task stack:0 pid:14662 ppid:14661 flags:0x00000008
[ 5387.361268] Call Trace:
[ 5387.363711] [] __schedule+0x2b2/0x8fe
[ 5387.368931] Task dump for CPU 28:
[ 5387.372242] task:cc1plus state:R running task stack:0 pid:14971 ppid:14970 flags:0x00000008
[ 5387.382149] Call Trace:
[ 5387.384592] [] __schedule+0x2b2/0x8fe
[ 5387.389814] Task dump for CPU 33:
[ 5387.393124] task:cc1plus state:R running task stack:0 pid:14903 ppid:14902 flags:0x00000008
[ 5387.403031] Call Trace:
[ 5387.405473] [] __schedule+0x2b2/0x8fe
[ 5391.926900] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 566s!
[ 5391.934938] BUG: workqueue lockup - pool cpus=58 node=3 flags=0x0 nice=0 stuck for 1133s!
[ 5391.943133] Showing busy workqueues and worker pools:
[ 5391.948197] workqueue events: flags=0x0
[ 5391.952043] pwq 42: cpus=21 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5391.952055] in-flight: 3012:netstamp_clear
[ 5391.952110] workqueue events_unbound: flags=0x2
[ 5391.968037] pwq 130: cpus=8-15,24-31 node=1 flags=0x4 nice=0 active=2/512 refcnt=4
[ 5391.968049] pending: flush_to_ldisc, flush_to_ldisc BAR(1752)
[ 5391.968093] workqueue events_power_efficient: flags=0x80
[ 5391.987259] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
[ 5391.987271] pending: neigh_periodic_work, neigh_managed_work
[ 5391.987341] workqueue mm_percpu_wq: flags=0x8
[ 5392.004476] pwq 116: cpus=58 node=3 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5392.004487] pending: vmstat_update
[ 5392.004522] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5392.004532] pending: vmstat_update
[ 5392.004541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 5392.004550] pending: vmstat_update
[ 5392.004630] workqueue writeback: flags=0x4a
[ 5392.040737] pwq 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 active=1/256 refcnt=2
[ 5392.040749] in-flight: 222:wb_workfn
[ 5392.040759] pwq 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
[ 5392.040769] in-flight: 14490:wb_workfn
[ 5392.041065] pool 42: cpus=21 node=0 flags=0x0 nice=0 hung=0s workers=2 idle: 9576
[ 5392.041104] pool 129: cpus=0-7,16-23 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 11998 15044 8792
[ 5392.041120] pool 132: cpus=40-47,56-63 node=3 flags=0x5 nice=0 hung=0s workers=3 manager: 11900 idle: 11976
[ 5396.266803] watchdog: BUG: soft lockup - CPU#1 stuck for 1073s! [migration/1:21]
[ 5396.274203] Modules linked in: overlay xsk_diag nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr sunrpc ofpart sophgo_spifmc spi_nor mtd uio_pdrv_genirq uio ip6_tables ip_tables loop zram spi_dw_mmio spi_dw gpio_dwapb r8169 scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath
[ 5396.317859] CPU: 1 PID: 21 Comm: migration/1 Tainted: G L 6.1.55 #1
[ 5396.325509] Hardware name: Sophgo Mango (DT)
[ 5396.329772] Stopper: multi_cpu_stop+0x0/0x146 ← migrate_swap+0xae/0x112
[ 5396.336474] epc : rcu_momentary_dyntick_idle+0x3a/0x80
[ 5396.341611] ra : multi_cpu_stop+0xde/0x146
[ 5396.345790] epc : ffffffff8009003a ra : ffffffff800deeca sp : ffffffc80a553d80
[ 5396.353005] gp : ffffffff81e76ff0 tp : ffffffd8ffda0000 t0 : 0000000000000080
[ 5396.360218] t1 : 0000000001806000 t2 : 0000000000004833 s0 : ffffffc80a553e00
[ 5396.367431] s1 : ffffffc81a3c39f8 a0 : ffffffff8120e228 a1 : ffffffc81a3c3a20
[ 5396.374644] a2 : ffffffc81a3c3a20 a3 : ffffffff81efc588 a4 : ffffffffaabc6544
[ 5396.381857] a5 : fffffff6df72a848 a6 : 0000000000000007 a7 : 0000000000000008
[ 5396.389071] s2 : ffffffc81a3c3a1c s3 : ffffffffffffffff s4 : ffffffff8120e228
[ 5396.396284] s5 : 0000000000000001 s6 : 0000000000000001 s7 : 0000000000000003
[ 5396.403498] s8 : 0000000000000004 s9 : 0000000000000002 s10: 0000000000000000
[ 5396.410711] s11: 0000000000000001 t3 : 0000000000000000 t4 : 000000000000778f
[ 5396.417924] t5 : ffffffe85e43d900 t6 : 0000000000000003
[ 5396.423228] status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000005
[ 5396.431136] [] rcu_momentary_dyntick_idle+0x3a/0x80
[ 5396.437574] [] cpu_stopper_thread+0x102/0x180
[ 5396.443489] [] smpboot_thread_fn+0xe6/0x11a
[ 5396.449233] [] kthread+0xbc/0xd2
[ 5396.454018] [] ret_from_exception+0x0/0x16

Just out of curiosity, try it without any GPU installed in your system. I found that a GPU (various AMD Radeon) would occasionally cause system hang under load. Possibly bug in kernel, Radeon driver, PCIe driver issue.