Trafgen exhausted system resources (CPU, memory, and NIC?)

dmesg

[Sun Feb 20 14:50:58 2022] rcu: INFO: rcu_sched self-detected stall on CPU
[Sun Feb 20 14:50:58 2022] rcu:         0-...!: (1 GPs behind) idle=ab6/1/0x4000000000000002 softirq=71528/71530 fqs=0
[Sun Feb 20 14:50:58 2022]      (t=403679 jiffies g=325261 q=466)
[Sun Feb 20 14:50:58 2022] rcu: rcu_sched kthread starved for 403679 jiffies! g325261 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
[Sun Feb 20 14:50:58 2022] rcu:         Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
[Sun Feb 20 14:50:58 2022] rcu: RCU grace-period kthread stack dump:
[Sun Feb 20 14:50:58 2022] task:rcu_sched       state:I stack:    0 pid:   12 ppid:     2 flags:0x80004000
[Sun Feb 20 14:50:58 2022] Call Trace:
[Sun Feb 20 14:50:58 2022]  __schedule+0x2d1/0x830
[Sun Feb 20 14:50:58 2022]  schedule+0x35/0xa0
[Sun Feb 20 14:50:58 2022]  schedule_timeout+0x197/0x300
[Sun Feb 20 14:50:58 2022]  ? __next_timer_interrupt+0xf0/0xf0
[Sun Feb 20 14:50:58 2022]  ? __prepare_to_swait+0x4b/0x70
[Sun Feb 20 14:50:58 2022]  rcu_gp_kthread+0x4e5/0xab0
[Sun Feb 20 14:50:58 2022]  ? rcu_accelerate_cbs_unlocked+0x80/0x80
[Sun Feb 20 14:50:58 2022]  kthread+0x10a/0x120
[Sun Feb 20 14:50:58 2022]  ? set_kthread_struct+0x40/0x40
[Sun Feb 20 14:50:58 2022]  ret_from_fork+0x1f/0x40
[Sun Feb 20 14:50:58 2022] rcu: Stack dump where RCU GP kthread last ran:
[Sun Feb 20 14:50:58 2022] Sending NMI from CPU 0 to CPUs 1:
[Sun Feb 20 14:50:58 2022] NMI backtrace for cpu 1
[Sun Feb 20 14:50:58 2022] CPU: 1 PID: 2552 Comm: kworker/1:2 Kdump: loaded Tainted: G        W    L   ---------r-  - 4.18.0-358.el8.x86_64 #1
[Sun Feb 20 14:50:58 2022] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[Sun Feb 20 14:50:58 2022] Workqueue: events e1000_reset_task [e1000]
[Sun Feb 20 14:50:58 2022] RIP: 0010:native_io_apic_read+0x32/0x40
[Sun Feb 20 14:50:58 2022] Code: 81 c7 04 02 00 00 48 8d 04 c0 c1 e7 0c 8b 04 c5 74 9a a3 b6 48 63 ff 25 ff 0f 00 00 48 2d 00 10 80 00 48 29 f8 89 30 8b 40 10 <c3> 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 8b 15 19
[Sun Feb 20 14:50:58 2022] RSP: 0018:ffffb9d202a0fd80 EFLAGS: 00010086
[Sun Feb 20 14:50:58 2022] RAX: 0000000001000000 RBX: 000000000000e826 RCX: 0000000000000000
[Sun Feb 20 14:50:58 2022] RDX: 0000000000000001 RSI: 0000000000000031 RDI: 0000000000204000
[Sun Feb 20 14:50:58 2022] RBP: 0000000000000030 R08: ffff9dc601400b68 R09: ffff9dc601400c10
[Sun Feb 20 14:50:58 2022] R10: 0000000000000000 R11: ffffffffb61e18a8 R12: 0000000000000000
[Sun Feb 20 14:50:58 2022] R13: ffff9dc6359ffe00 R14: 0000000000000246 R15: ffff9dc60319a000
[Sun Feb 20 14:50:58 2022] FS:  0000000000000000(0000) GS:ffff9dc67fd00000(0000) knlGS:0000000000000000
[Sun Feb 20 14:50:58 2022] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Sun Feb 20 14:50:58 2022] CR2: 00007faf9c017048 CR3: 000000000d810005 CR4: 00000000000306e0
[Sun Feb 20 14:50:58 2022] Call Trace:
[Sun Feb 20 14:50:58 2022]  __ioapic_read_entry+0x32/0x50
[Sun Feb 20 14:50:58 2022]  ioapic_irq_get_chip_state+0x42/0x70
[Sun Feb 20 14:50:58 2022]  __synchronize_hardirq+0xc5/0xd0
[Sun Feb 20 14:50:58 2022]  synchronize_irq+0x35/0xa0
[Sun Feb 20 14:50:58 2022]  ? e1000_down+0x117/0x1d0 [e1000]
[Sun Feb 20 14:50:58 2022]  e1000_down+0x133/0x1d0 [e1000]
[Sun Feb 20 14:50:58 2022]  e1000_reinit_locked+0x45/0x60 [e1000]
[Sun Feb 20 14:50:58 2022]  process_one_work+0x1a7/0x360
[Sun Feb 20 14:50:58 2022]  worker_thread+0x30/0x390
[Sun Feb 20 14:50:58 2022]  ? create_worker+0x1a0/0x1a0
[Sun Feb 20 14:50:58 2022]  kthread+0x10a/0x120
[Sun Feb 20 14:50:58 2022]  ? set_kthread_struct+0x40/0x40
[Sun Feb 20 14:50:58 2022]  ret_from_fork+0x1f/0x40
[Sun Feb 20 14:50:58 2022] NMI backtrace for cpu 0
[Sun Feb 20 14:50:58 2022] CPU: 0 PID: 1 Comm: systemd Kdump: loaded Tainted: G        W    L   ---------r-  - 4.18.0-358.el8.x86_64 #1
[Sun Feb 20 14:50:58 2022] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[Sun Feb 20 14:50:58 2022] Call Trace:
[Sun Feb 20 14:50:58 2022]  <IRQ>
[Sun Feb 20 14:50:58 2022]  dump_stack+0x41/0x60
[Sun Feb 20 14:50:58 2022]  nmi_cpu_backtrace.cold.6+0x13/0x4e
[Sun Feb 20 14:50:58 2022]  ? lapic_can_unplug_cpu.cold.29+0x37/0x37
[Sun Feb 20 14:50:58 2022]  nmi_trigger_cpumask_backtrace+0xde/0xe0
[Sun Feb 20 14:50:58 2022]  rcu_dump_cpu_stacks+0xc8/0xf8
[Sun Feb 20 14:50:58 2022]  rcu_sched_clock_irq.cold.99+0x116/0x39e
[Sun Feb 20 14:50:58 2022]  ? timekeeping_advance+0x372/0x5a0
[Sun Feb 20 14:50:58 2022]  ? wake_up_q+0x70/0x70
[Sun Feb 20 14:50:58 2022]  ? tick_sched_do_timer+0x50/0x50
[Sun Feb 20 14:50:58 2022]  update_process_times+0x55/0x80
[Sun Feb 20 14:50:58 2022]  tick_sched_handle+0x22/0x60
[Sun Feb 20 14:50:58 2022]  tick_sched_timer+0x37/0x70
[Sun Feb 20 14:50:58 2022]  __hrtimer_run_queues+0x100/0x280
[Sun Feb 20 14:50:58 2022]  hrtimer_interrupt+0x100/0x220
[Sun Feb 20 14:50:58 2022]  smp_apic_timer_interrupt+0x6a/0x130
[Sun Feb 20 14:50:58 2022]  apic_timer_interrupt+0xf/0x20
[Sun Feb 20 14:50:58 2022]  </IRQ>
[Sun Feb 20 14:50:58 2022] RIP: 0010:native_apic_mem_read+0x8/0x10
[Sun Feb 20 14:50:58 2022] Code: 20 bb 6c b5 e8 b5 90 0f 00 31 c0 c3 90 90 90 90 90 90 89 ff 89 b7 00 c0 5f ff c3 0f 1f 80 00 00 00 00 89 ff 8b 87 00 c0 5f ff <c3> 0f 1f 80 00 00 00 00 0f 1f 44 00 00 b8 01 00 00 00 c3 0f 1f 44
[Sun Feb 20 14:50:58 2022] RSP: 0018:ffffb9d200327b90 EFLAGS: 00010202 ORIG_RAX: ffffffffffffff13
[Sun Feb 20 14:50:58 2022] RAX: 00000000000008fd RBX: ffffb9d200327da8 RCX: 0000000000000000
[Sun Feb 20 14:50:58 2022] RDX: 0000000000040400 RSI: 00000000000000f6 RDI: 0000000000000300
[Sun Feb 20 14:50:58 2022] RBP: ffffb9d200327d68 R08: 0000088ccfb214cc R09: 0000000000000000
[Sun Feb 20 14:50:58 2022] R10: 0000000000000000 R11: ffffffffb5e5a708 R12: ffff9dc6018f0d98
[Sun Feb 20 14:50:58 2022] R13: 0000000000000020 R14: ffffb9d200327c00 R15: ffff9dc601889680
[Sun Feb 20 14:50:58 2022]  native_apic_wait_icr_idle+0x1e/0x30
[Sun Feb 20 14:50:58 2022]  irq_work_queue+0x16/0x20
[Sun Feb 20 14:50:58 2022]  perf_output_put_handle+0x65/0x80
[Sun Feb 20 14:50:58 2022]  perf_output_end+0xa/0x10
[Sun Feb 20 14:50:58 2022]  perf_event_task_output+0x12c/0x1a0
[Sun Feb 20 14:50:58 2022]  ? __mod_memcg_lruvec_state+0x22/0x100
[Sun Feb 20 14:50:58 2022]  ? copy_page_range+0x51d/0x810
[Sun Feb 20 14:50:58 2022]  ? idr_alloc_u32+0x8b/0xc0
[Sun Feb 20 14:50:58 2022]  ? perf_event_switch_output+0x150/0x150
[Sun Feb 20 14:50:58 2022]  perf_iterate_sb+0x18e/0x1e0
[Sun Feb 20 14:50:58 2022]  perf_event_task+0x7f/0xb0
[Sun Feb 20 14:50:58 2022]  perf_event_fork+0x10/0x30
[Sun Feb 20 14:50:58 2022]  copy_process+0x1764/0x1bb0
[Sun Feb 20 14:50:58 2022]  _do_fork+0x8b/0x340
[Sun Feb 20 14:50:58 2022]  do_syscall_64+0x5b/0x1a0
[Sun Feb 20 14:50:58 2022]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[Sun Feb 20 14:50:58 2022] RIP: 0033:0x7f1396f9b71c
[Sun Feb 20 14:50:58 2022] Code: db 0f 85 17 01 00 00 64 4c 8b 0c 25 10 00 00 00 45 31 c0 4d 8d 91 d0 02 00 00 31 d2 31 f6 bf 11 00 20 01 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 b0 00 00 00 41 89 c4 85 c0 0f 85 bd 00 00
[Sun Feb 20 14:50:58 2022] RSP: 002b:00007ffc0ac42e80 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
[Sun Feb 20 14:50:58 2022] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f1396f9b71c
[Sun Feb 20 14:50:58 2022] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
[Sun Feb 20 14:50:58 2022] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007f1398c459c0
[Sun Feb 20 14:50:58 2022] R10: 00007f1398c45c90 R11: 0000000000000246 R12: 0000000000000000
[Sun Feb 20 14:50:58 2022] R13: 000055d8219cfab0 R14: 000055d821ad87a0 R15: 000055d821ad9470
[Sun Feb 20 14:50:58 2022] e1000 0000:00:08.0 enp0s8: Reset adapter
[Sun Feb 20 14:51:01 2022] e1000: enp0s8 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

trafgen-l2fwd-demo.conf

{
    # --- ethernet header ---
    0x08, 0x00, 0x27, 0x00, 0x71, 0xea,  # mac destination
    0x08, 0x00, 0x27, 0x75, 0x32, 0xcf,  # mac source
    const16(0x0800), # protocol
    # --- ip header ---
    # ipv4 version (4-bit) + ihl (4-bit), tos
    0b01000101, 0,
    # ipv4 total len
    const16(40),
    # id (note: runtime dynamic random)
    drnd(2),
    # ipv4 3-bit flags + 13-bit fragment offset
    # 001 = more fragments
    0b00100000, 0,
    64, # ttl
    17, # proto udp
    # dynamic ip checksum (note: offsets are zero indexed)
    csumip(14, 33),
    192, 168, 51, 1, # source ip
    192, 168, 51, 2, # dest ip
    # --- udp header ---
    # as this is a fragment the below stuff does not matter too much
    const16(48054), # src port
    const16(43514), # dst port
    const16(20),    # udp length
    # udp checksum can be dyn calc via csumudp(offset ip, offset tcp)
    # which is csumudp(14, 34), but for udp its allowed to be zero
    const16(0),
    # payload
    'A',  fill(0x41, 11),
}

trafgen --dev enp0s0 --conf demo-1.conf

Limit the rate:

 sudo ../trafgen --dev enp0s8 --conf l2fwd-demo-1.conf --cpus 1 -b 600pps