From nobody@FreeBSD.org Mon Jun 11 00:49:54 2007 Return-Path: Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9A78516A469 for ; Mon, 11 Jun 2007 00:49:54 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [69.147.83.33]) by mx1.freebsd.org (Postfix) with ESMTP id 8955513C468 for ; Mon, 11 Jun 2007 00:49:54 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l5B0ns0k026401 for ; Mon, 11 Jun 2007 00:49:54 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id l5B0ns1M026400; Mon, 11 Jun 2007 00:49:54 GMT (envelope-from nobody) Message-Id: <200706110049.l5B0ns1M026400@www.freebsd.org> Date: Mon, 11 Jun 2007 00:49:54 GMT From: Alexey Illarionov To: freebsd-gnats-submit@FreeBSD.org Subject: [dummynet] system hangs with dummynet queues X-Send-Pr-Version: www-3.0 >Number: 113548 >Category: kern >Synopsis: [dummynet] [patch] system hangs with dummynet queues >Confidential: no >Severity: serious >Priority: medium >Responsible: oleg >State: closed >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Jun 11 00:50:03 GMT 2007 >Closed-Date: Wed May 07 17:43:22 UTC 2008 >Last-Modified: Wed May 07 17:43:22 UTC 2008 >Originator: Alexey Illarionov >Release: 6.2-STABLE >Organization: >Environment: FreeBSD ftp.orionet.ru 6.2-STABLE FreeBSD 6.2-STABLE #5: Mon Jun 11 02:31:15 MSD 2007 littlesavage@ftp.orionet.ru:/usr/obj/usr/src/sys/FTP i386 >Description: I got some system freezes and kernel panicks when i tried to use queues in ipfw. It occurs 1-2 times a day. That is my rules: .. 02100 queue 105 ip from any to me in via vlan204 02200 queue 150 tag 1 ip from 10.100.1.6 to any not tagged 1 out via vlan204 02300 queue 140 tag 1 ip from me 8767,6667,411,4111,22 to any not tagged 1 out via vlan204 02400 queue 130 tag 1 ip from me 80 to any not tagged 1 out via vlan204 02500 queue 120 tag 1 ip from me to any not tagged 1 out via vlan204 02600 queue 110 tag 1 ip from any to any not tagged 1 out via vlan204 .. When i use rule 2100 only, the kernel panics. When i use rules rules 2200-2600, its freezes. When i use all that rules, its panics and freezes. # ipfw pipe show 00001: 32.000 Kbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 5 ip 0.0.0.0/0 10.100.39.13/0 6 288 0 0 0 00002: 32.000 Kbit/s 0 ms 50 sl. 0 queues (64 buckets) droptail 00100: 50.000 Mbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail 00105: 50.000 Mbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail q00140: weight 80 pipe 100 50 sl. 0 queues (1 buckets) droptail q00110: weight 10 pipe 100 50 sl. 0 queues (1 buckets) droptail q00130: weight 70 pipe 100 50 sl. 0 queues (1 buckets) droptail q00150: weight 100 pipe 100 50 sl. 0 queues (1 buckets) droptail q00120: weight 10 pipe 100 50 sl. 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 q00105: weight 50 pipe 105 50 sl. 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 Once I could reproduce the same kernel panic on different machine with the following configuration: dmesg: FreeBSD switch1.orionet.ru 6.2-STABLE FreeBSD 6.2-STABLE #0: Thu Jun 7 21:27:46 MSD 2007 littlesavage@switch1.orionet.ru:/usr/obj/usr/src/sys/VOIP i386 # dmesg Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-STABLE #0: Thu Jun 7 21:27:46 MSD 2007 littlesavage@switch1.orionet.ru:/usr/obj/usr/src/sys/VOIP WARNING: WITNESS option enabled, expect reduced performance. module_register: module g_concat already exists! Module g_concat failed to register: 17 module_register: module g_label already exists! Module g_label failed to register: 17 ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel Pentium III (930.96-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x686 Stepping = 6 Features=0x383fbff real memory = 536469504 (511 MB) avail memory = 515223552 (491 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 pci1: at device 0.0 (no driver attached) pcib2: at device 30.0 on pci0 pci2: on pcib2 xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xec80-0xecff mem 0xf9fffc00-0xf9fffc7f irq 16 at device 4.0 on pci2 miibus0: on xl0 xlphy0: <3c905C 10/100 internal PHY> on miibus0 xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto xl0: Ethernet address: 00:b0:d0:3e:37:0e pci2: at device 9.0 (no driver attached) pcib3: at device 14.0 on pci2 pci3: on pcib3 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 uhci0: port 0xff80-0xff9f irq 19 at device 31.2 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered pci0: at device 31.3 (no driver attached) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A ppc0: port 0x378-0x37f,0x778-0x77f irq 7 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/8 bytes threshold ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc97ff,0xc9800-0xcbfff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec ipfw2 initialized, divert enabled, rule-based forwarding enabled, default to deny, logging limited to 100 packets/entry by default The GEOM class CONCAT is already loaded. The GEOM class LABEL is already loaded. ad0: 58643MB at ata0-master UDMA66 acd0: CDROM at ata1-master PIO4 SMP: AP CPU #1 Launched! ifconfig: xl0: flags=8843 mtu 1500 options=9 inet 10.100.6.7 netmask 0xffffff00 broadcast 10.100.6.255 ether 00:b0:d0:3e:37:0e media: Ethernet autoselect (100baseTX ) status: active ipfw rules: 00010 allow ip from 10.100.1.1 to me 00011 allow ip from me to 10.100.1.1 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 00400 deny ip from any to any dst-port 135-139,445 00500 deny log logamount 100 ip from any to 0.0.0.0/8 00600 deny log logamount 100 ip from any to 169.254.0.0/16 00700 deny log logamount 100 ip from any to 192.0.2.0/24 00800 deny log logamount 100 ip from any to 198.18.0.0/15 00900 deny log logamount 100 ip from any to 224.0.0.0/4 01000 deny log logamount 100 ip from any to 240.0.0.0/4 01700 queue 105 ip from any to me in via xl0 01800 queue 150 tag 1 ip from 10.100.1.6 to any not tagged 1 out via xl0 01900 queue 140 tag 1 ip from me 8767,6667,411,4111,22 to any not tagged 1 out via xl0 02000 queue 130 tag 1 ip from me 80 to any not tagged 1 out via xl0 02100 queue 120 tag 1 ip from me to any not tagged 1 out via xl0 02200 queue 110 tag 1 ip from any to any not tagged 1 out via xl0 02300 allow ip from any to me 02400 allow ip from me to any pipes: 00100: 50.000 Mbit/s 0 ms 100 sl. 0 queues (1 buckets) GRED w_q 0.001999 min_th 10 max_th 30 max_p 0.099991 00105: 50.000 Mbit/s 0 ms 100 sl. 0 queues (1 buckets) GRED w_q 0.001999 min_th 10 max_th 30 max_p 0.099991 q00140: weight 80 pipe 100 100 sl. 1 queues (1 buckets) GRED w_q 0.001999 min_th 10 max_th 30 max_p 0.099991 mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 q00110: weight 1 pipe 100 100 sl. 0 queues (1 buckets) GRED w_q 0.001999 min_th 10 max_th 30 max_p 0.099991 q00130: weight 70 pipe 100 100 sl. 0 queues (1 buckets) GRED w_q 0.001999 min_th 10 max_th 30 max_p 0.099991 q00150: weight 100 pipe 100 100 sl. 0 queues (1 buckets) GRED w_q 0.001999 min_th 10 max_th 30 max_p 0.099991 q00120: weight 10 pipe 100 100 sl. 1 queues (1 buckets) GRED w_q 0.001999 min_th 10 max_th 30 max_p 0.099991 mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 q00105: weight 50 pipe 105 100 sl. 1 queues (1 buckets) GRED w_q 0.001999 min_th 10 max_th 30 max_p 0.099991 mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 kgdb: [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: dummynet: 1657171 q: 0 dummynet: avg: 0 dummynet: waking up pipe 105 at 1 dummynet: 1657171 q: 0 dummynet: avg: 0 dummynet: waking up pipe 105 at 1 Limiting icmp ping response from 275 to 200 packets/sec dummynet: 1657171 q: 0 dummynet: avg: 0 dummynet: waking up pipe 100 at 8 dummynet: 1657243 q: 7 dummynet: avg: 0 dummynet: 1657243 q: 0 dummynet: avg: 0 dummynet: waking up pipe 105 at 1 dummynet: 1657243 q: 8 dummynet: avg: 0 panic: packet on dummynet queue w/o dummynet tag! cpuid = 0 KDB: stack backtrace: kdb_backtrace(100,c3314600,25,0,0,...) at kdb_backtrace+0x29 panic(c078e685,de7cac90,c060463c,c3be4700,d398b380,...) at panic+0x114 dn_tag_get(c3be4700,d398b380,0,c05a3fea,c0860e78,...) at dn_tag_get+0x24 ready_event_wfq(c34fba00,de7cacac,de7cacb0) at ready_event_wfq+0x4a4 dummynet_task(0,1) at dummynet_task+0x23f taskqueue_run(c358be80) at taskqueue_run+0xc8 taskqueue_thread_loop(c08a1070,de7cad38,c08a1070,c05a0514,0,...) at taskqueue_thread_loop+0x4a fork_exit(c05a0514,c08a1070,de7cad38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xde7cad6c, ebp = 0 --- Uptime: 27m38s Dumping 511 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 511MB (130718 pages) 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); bt #0 doadump () at pcpu.h:165 #1 0xc0580738 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0580a4d in panic (fmt=0xc078e685 "packet on dummynet queue w/o dummynet tag!") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc0603ea8 in dn_tag_get (m=0x0) at /usr/src/sys/netinet/ip_dummynet.c:437 #4 0xc060463c in ready_event_wfq (p=0xc34fba00, head=0xde7cacac, tail=0xde7cacb0) at /usr/src/sys/netinet/ip_dummynet.c:703 #5 0xc06048ef in dummynet_task (context=0x0, pending=0) at /usr/src/sys/netinet/ip_dummynet.c:803 #6 0xc05a01d4 in taskqueue_run (queue=0xc358be80) at /usr/src/sys/kern/subr_taskqueue.c:257 #7 0xc05a055e in taskqueue_thread_loop (arg=0x0) at /usr/src/sys/kern/subr_taskqueue.c:376 #8 0xc056b8c4 in fork_exit (callout=0xc05a0514 , arg=0xc08a1070, frame=0xde7cad38) at /usr/src/sys/kern/kern_fork.c:821 #9 0xc072af7c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 (kgdb) up 4 #4 0xc060463c in ready_event_wfq (p=0xc34fba00, head=0xde7cacac, tail=0xde7cacb0) at /usr/src/sys/netinet/ip_dummynet.c:703 703 dn_tag_get(p->tail)->output_time += t ; (kgdb) p *p $1 = {next = {sle_next = 0x0}, pipe_nr = 100, bandwidth = 50000000, delay = 0, head = 0x0, tail = 0xc3be4700, scheduler_heap = {size = 16, elements = 1, offset = 0, p = 0xc39eca00}, not_eligible_heap = {size = 16, elements = 0, offset = 0, p = 0xc3721e00}, idle_heap = {size = 16, elements = 0, offset = 124, p = 0xc39ece00}, V = 550502, sum = 10, numbytes = -1840590592, sched_time = 1657235, if_name = '\0' , ifp = 0x0, ready = 0, fs = {next = { sle_next = 0x0}, fs_nr = 0, flags_fs = 6, pipe = 0xc34fba00, parent_nr = 0, weight = 0, qsize = 100, plr = 0, flow_mask = {dst_ip = 0, src_ip = 0, dst_port = 0, src_port = 0, proto = 0 '\0', flags = 0 '\0', addr_type = 0 '\0', dst_ip6 = {__u6_addr = {__u6_addr8 = '\0' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = { 0, 0, 0, 0}}}, src_ip6 = {__u6_addr = {__u6_addr8 = '\0' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, flow_id6 = 0, frag_id6 = 0}, rq_size = 1, rq_elements = 0, rq = 0xc3603840, last_expired = 0, backlogged = 0, w_q = 131, max_th = 1966080, min_th = 655360, max_p = 6553, c_1 = 327, c_2 = 3270, c_3 = 1966, c_4 = 52430, w_q_lookup = 0xc34da400, lookup_depth = 256, lookup_step = 1, lookup_weight = 65274, avg_pkt_size = 512, max_pkt_size = 1500}} But the main problem is the system with following configuration: # dmesg Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-STABLE #5: Mon Jun 11 02:31:15 MSD 2007 littlesavage@ftp.orionet.ru:/usr/obj/usr/src/sys/FTP WARNING: WITNESS option enabled, expect reduced performance. ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) 64 Processor 3200+ (2009.18-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x20ff0 Stepping = 0 Features=0x78bfbff Features2=0x1 AMD Features=0xe2500800 AMD Features2=0x1 real memory = 536805376 (511 MB) avail memory = 515809280 (491 MB) ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 cpu0: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff,0xcf0-0xcf3 on acpi0 pci0: on pcib0 isab0: at device 1.0 on pci0 isa0: on isab0 pci0: at device 1.1 (no driver attached) atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 8.0 on pci0 ata0: on atapci0 ata1: on atapci0 atapci1: port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xe000-0xe00f,0xe400-0xe47f irq 20 at device 10.0 on pci0 ata2: on atapci1 ata3: on atapci1 pcib1: at device 11.0 on pci0 pci1: on pcib1 pcib2: at device 14.0 on pci0 pci2: on pcib2 atapci2: port 0xa000-0xa00f,0xa400-0xa40f,0xa800-0xa80f,0xac00-0xac0f,0xb000-0xb01f,0xb400-0xb4ff irq 19 at device 7.0 on pci2 ata4: on atapci2 ata5: on atapci2 skc0: port 0xb800-0xb8ff mem 0xf1000000-0xf1003fff irq 18 at device 10.0 on pci2 skc0: DGE-530T Gigabit Ethernet Adapter rev. (0x9) sk0: on skc0 sk0: Ethernet address: 00:17:9a:06:16:43 miibus0: on sk0 e1000phy0: on miibus0 e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto acpi_tz0: on acpi0 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A pmtimer0 on isa0 orm0: at iomem 0xc8000-0xccfff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. Timecounter "TSC" frequency 2009175908 Hz quality 800 Timecounters tick every 1.000 msec ipfw2 initialized, divert enabled, rule-based forwarding enabled, default to deny, logging limited to 100 packets/entry by default ifconfig: sk0: flags=8843 mtu 1500 options=b inet 10.100.2.9 netmask 0xffffff00 broadcast 10.100.2.255 ether 00:17:9a:06:16:43 media: Ethernet autoselect (1000baseTX ) status: active vlan204: flags=8843 mtu 1500 inet 10.100.14.9 netmask 0xffffff00 broadcast 10.100.14.255 ether 00:17:9a:06:16:43 media: Ethernet autoselect (1000baseTX ) status: active vlan: 204 parent interface: sk0 kernel config: machine i386 #cpu I486_CPU #cpu I586_CPU cpu I686_CPU ident FTP makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options KDB options DDB options BREAK_TO_DEBUGGER options KDB_TRACE options KDB_UNATTENDED options DEVICE_POLLING options WITNESS #options WITNESS_SKIPSPIN options INVARIANTS options INVARIANT_SUPPORT options GEOM_LABEL options GEOM_CONCAT options SCHED_4BSD options HZ=2000 I try to use the following patch (don't know how it correct). Kernel panics end off, but freezes are not. --- ip_dummynet.c_orig Sun Jun 10 20:19:33 2007 +++ ip_dummynet.c Mon Jun 11 03:56:54 2007 @@ -490,7 +490,8 @@ * whole pipe p and hoping in the future we are more successful. */ heap_insert(&extract_heap, pkt->output_time, pipe); - } + }else + pipe->tail = NULL; } /* @@ -700,7 +701,8 @@ if (p->bandwidth > 0) t = ( p->bandwidth -1 - p->numbytes) / p->bandwidth ; - dn_tag_get(p->tail)->output_time += t ; + if (p->head != NULL) + dn_tag_get(p->tail)->output_time += t ; p->sched_time = curr_time ; heap_insert(&wfq_ready_heap, curr_time + t, (void *)p); /* XXX should check errors on heap_insert, and drain the whole When i use only pipes and not use queues, every tings work fine. >How-To-Repeat: Make 1-2 queues in 50Mbit/s pipe, load it with some traffic and wait one day. >Fix: >Release-Note: >Audit-Trail: Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Mon Jun 11 03:02:20 UTC 2007 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=113548 From: Cristian KLEIN To: bug-followup@FreeBSD.org, littlesavage@orionet.ru Cc: Subject: Re: kern/113548: [dummynet] [patch] system hangs with dummynet queues Date: Mon, 11 Jun 2007 23:35:21 +0300 I think the problem occurs because you use ipfw tags. As far as I know, ipfw tags are stored as mbuf_tags(9). Dummynet uses mbuf tags too to mark it's own packets. However, I suspect that in dn_tag_get(), dummynet incorrectly assumes it is the only one using mbuf_tags(9). Could you please apply the following patch? Also, could you test whether removing "tag 1" from ipfw rules has any impact? --- ip_dummynet.c.orig Sat Jul 29 11:24:12 2006 +++ ip_dummynet.c Mon Jun 11 23:27:34 2007 @@ -412,7 +412,7 @@ static struct dn_pkt_tag * dn_tag_get(struct mbuf *m) { - struct m_tag *mtag = m_tag_first(m); + struct m_tag *mtag = m_tag_find(m, PACKET_TAG_DUMMYNET, NULL); KASSERT(mtag != NULL && mtag->m_tag_cookie == MTAG_ABI_COMPAT && mtag->m_tag_id == PACKET_TAG_DUMMYNET, -- +-------------------------------------+ | Cristian KLEIN | | Network Engineer | | Communication Center | | Technical University of Cluj-Napoca | +-------------------------------------+ | Tel: +40-264-401247, int. 247 | | WWW: http://www.cc.utcluj.ro | +-------------------------------------+ From: Alexey Illarionov To: Cristian KLEIN Cc: bug-followup@FreeBSD.org Subject: Re: kern/113548: [dummynet] [patch] system hangs with dummynet queues Date: Fri, 15 Jun 2007 11:11:39 +0400 This is a multi-part message in MIME format. --------------040704070900010000020204 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cristian KLEIN wrote: > I think the problem occurs because you use ipfw tags. As far as I know, > ipfw tags are stored as mbuf_tags(9). Dummynet uses mbuf tags too to > mark it's own packets. However, I suspect that in dn_tag_get(), dummynet > incorrectly assumes it is the only one using mbuf_tags(9). > Could you please apply the following patch? Also, could you test whether > removing "tag 1" from ipfw rules has any impact? Thanks for a fast reply and for the patch. It seems that panics have really been caused by ipfw tags. When I apply this patch, there were no panics for several days, but I have got the following dump today: kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xec221d87 fault code = supervisor read, page not present instruction pointer = 0x20:0xc05dafc6 stack pointer = 0x28:0xde7b0c24 frame pointer = 0x28:0xde7b0c28 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 30 (dummynet) trap number = 12 panic: page fault KDB: stack backtrace: kdb_backtrace(100,c52ad480,28,de7b0be4,c,...) at kdb_backtrace+0x29 panic(c078df19,c07d4928,0,fffff,c09b,...) at panic+0xa4 trap_fatal(de7b0be4,ec221d87,c52ad480,c104b000,ec221000,...) at trap_fatal+0x2b7 trap_pfault(de7b0be4,0,ec221d87) at trap_pfault+0x16b trap(8,28,28,1,0,...) at trap+0x331 calltrap() at calltrap+0x5 --- trap 0xc, eip = 0xc05dafc6, esp = 0xde7b0c24, ebp = 0xde7b0c28 --- m_tag_locate(c55df900,0,f,0) at m_tag_locate+0x36 dn_tag_get(c55df900,2ffbd300,1,c05c3e7e,c088e858,...) at dn_tag_get+0x1d ready_event_wfq(c57b0800,de7b0cac,de7b0cb0) at ready_event_wfq+0x50b dummynet_task(0,1) at dummynet_task+0x24c taskqueue_run(c5562a00) at taskqueue_run+0xd1 taskqueue_thread_loop(c08ce950,de7b0d38,c08ce950,c05c01e0,0,...) at taskqueue_thread_loop+0x4a fork_exit(c05c01e0,c08ce950,de7b0d38) at fork_exit+0xa8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xde7b0d6c, ebp = 0 --- Uptime: 50m0s Dumping 511 MB (2 chunks) chunk 0: 1MB (156 pages) ... ok chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc059f2a6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc059f57b in panic (fmt=0xc078df19 "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc076c1f7 in trap_fatal (frame=0xde7b0be4, eva=3961658759) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc076bf0b in trap_pfault (frame=0xde7b0be4, usermode=0, eva=3961658759) at /usr/src/sys/i386/i386/trap.c:745 #5 0xc076bb71 in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 1, tf_esi = 0, tf_ebp = -562361304, tf_isp = -562361328, tf_ebx = 15, tf_edx = -333308545, tf_ecx = 0, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1067601978, tf_cs = 32, tf_eflags = 66178, tf_esp = 22, tf_ss = -562361280}) at /usr/src/sys/i386/i386/trap.c:435 #6 0xc0758bca in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc05dafc6 in m_tag_locate (m=0xec221d7f, cookie=0, type=15, t=0x0) at /usr/src/sys/kern/uipc_mbuf2.c:392 #8 0xc06279ad in dn_tag_get (m=0xec221d7f) at mbuf.h:881 #9 0xc06281fb in ready_event_wfq (p=0xc57b0800, head=0xde7b0cac, tail=0xde7b0cb0) at /usr/src/sys/netinet/ip_dummynet.c:705 #10 0xc06284cc in dummynet_task (context=0x0, pending=0) at /usr/src/sys/netinet/ip_dummynet.c:805 #11 0xc05bfe71 in taskqueue_run (queue=0xc5562a00) at /usr/src/sys/kern/subr_taskqueue.c:257 #12 0xc05c022a in taskqueue_thread_loop (arg=0x0) at /usr/src/sys/kern/subr_taskqueue.c:376 #13 0xc05897b8 in fork_exit (callout=0xc05c01e0 , arg=0xc08ce950, frame=0xde7b0d38) at /usr/src/sys/kern/kern_fork.c:821 #14 0xc0758c2c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 (kgdb) up 9 #9 0xc06281fb in ready_event_wfq (p=0xc57b0800, head=0xde7b0cac, tail=0xde7b0cb0) at /usr/src/sys/netinet/ip_dummynet.c:705 705 dn_tag_get(p->tail)->output_time += t ; (kgdb) p *p $1 = {next = {sle_next = 0xc6713600}, pipe_nr = 1700, bandwidth = 50000000, delay = 0, head = 0x0, tail = 0xc55df900, scheduler_heap = {size = 16, elements = 1, offset = 0, p = 0xc57b2800}, not_eligible_heap = {size = 16, elements = 0, offset = 0, p = 0xc57ac700}, idle_heap = {size = 16, elements = 0, offset = 124, p = 0xc56a2800}, V = 9830400, sum = 10, numbytes = -1090027776, sched_time = 2997985, if_name = '\0' , ifp = 0x0, ready = 0, fs = { next = {sle_next = 0x0}, fs_nr = 0, flags_fs = 0, pipe = 0xc57b0800, parent_nr = 0, weight = 0, qsize = 50, plr = 0, flow_mask = {dst_ip = 0, src_ip = 0, dst_port = 0, src_port = 0, proto = 0 '\0', flags = 0 '\0', addr_type = 0 '\0', dst_ip6 = {__u6_addr = {__u6_addr8 = '\0' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = { 0, 0, 0, 0}}}, src_ip6 = {__u6_addr = {__u6_addr8 = '\0' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, flow_id6 = 0, frag_id6 = 0}, rq_size = 1, rq_elements = 0, rq = 0xc55791b0, last_expired = 0, backlogged = 0, w_q = 0, max_th = 0, min_th = 0, max_p = 0, c_1 = 0, c_2 = 0, c_3 = 0, c_4 = 0, w_q_lookup = 0x0, lookup_depth = 0, lookup_step = 0, lookup_weight = 0, avg_pkt_size = 0, max_pkt_size = 0}} When I remove "tag 1" the kernel stopped panick, but deadlocks didn't pass away. When I managed to enter DDB using serial console I found dummynet_task() looped on the following code: h = heaps[i]; while (h->elements > 0 && DN_KEY_LEQ(h->p[0].key, curr_time)) { ... ready_event_wfq(p, &head, &tail); ... } It seems to me that problem is in ready_event_wfq() in the following code: if (p->bandwidth > 0) t = (p->bandwidth -1 - p->numbytes) / p->bandwidth ; Since p->bandwidth and p->numbytes are signed integers, the result can be negative (i have p->bandwidth=50000000 and p->numbytes=-2147483647) Now i test attached patch. I hope it will help. :) --------------040704070900010000020204 Content-Type: text/x-patch; name="ip_dummynet.c.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="ip_dummynet.c.patch" --- ip_dummynet.c_orig Sun Jun 10 20:19:33 2007 +++ ip_dummynet.c Fri Jun 15 07:37:46 2007 @@ -433,7 +433,7 @@ static struct dn_pkt_tag * dn_tag_get(struct mbuf *m) { - struct m_tag *mtag = m_tag_first(m); + struct m_tag *mtag = m_tag_find(m, PACKET_TAG_DUMMYNET, NULL); KASSERT(mtag != NULL && mtag->m_tag_cookie == MTAG_ABI_COMPAT && mtag->m_tag_id == PACKET_TAG_DUMMYNET, @@ -698,8 +698,10 @@ if (p->if_name[0]==0 && p->numbytes < 0) { /* this implies bandwidth >0 */ dn_key t=0 ; /* number of ticks i have to wait */ - if (p->bandwidth > 0) - t = ( p->bandwidth -1 - p->numbytes) / p->bandwidth ; + if (p->bandwidth > 0) + t = ( (u_int64_t)p->bandwidth -1 - p->numbytes) / p->bandwidth ; + + KASSERT( (curr_time + t) >= curr_time, ("wfq overflow")); dn_tag_get(p->tail)->output_time += t ; p->sched_time = curr_time ; heap_insert(&wfq_ready_heap, curr_time + t, (void *)p); --------------040704070900010000020204-- From: Cristian KLEIN To: Alexey Illarionov Cc: bug-followup@FreeBSD.org Subject: Re: kern/113548: [dummynet] [patch] system hangs with dummynet queues Date: Fri, 15 Jun 2007 10:30:43 +0300 Alexey Illarionov wrote: > Cristian KLEIN wrote: > >> I think the problem occurs because you use ipfw tags. As far as I know, >> ipfw tags are stored as mbuf_tags(9). Dummynet uses mbuf tags too to >> mark it's own packets. However, I suspect that in dn_tag_get(), dummynet >> incorrectly assumes it is the only one using mbuf_tags(9). > >> Could you please apply the following patch? Also, could you test whether >> removing "tag 1" from ipfw rules has any impact? > > Thanks for a fast reply and for the patch. It seems that panics have > really been caused by ipfw tags. When I apply this patch, there were no > panics for several days, but I have got the following dump today: > > kgdb: kvm_nlist(_stopped_cpus): > kgdb: kvm_nlist(_stoppcbs): > [GDB will not be able to debug user-mode threads: > /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-marcel-freebsd". > > Unread portion of the kernel message buffer: > > > Fatal trap 12: page fault while in kernel mode > fault virtual address = 0xec221d87 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc05dafc6 > stack pointer = 0x28:0xde7b0c24 > frame pointer = 0x28:0xde7b0c28 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 30 (dummynet) > trap number = 12 > panic: page fault > KDB: stack backtrace: > kdb_backtrace(100,c52ad480,28,de7b0be4,c,...) at kdb_backtrace+0x29 > panic(c078df19,c07d4928,0,fffff,c09b,...) at panic+0xa4 > trap_fatal(de7b0be4,ec221d87,c52ad480,c104b000,ec221000,...) at > trap_fatal+0x2b7 > trap_pfault(de7b0be4,0,ec221d87) at trap_pfault+0x16b > trap(8,28,28,1,0,...) at trap+0x331 > calltrap() at calltrap+0x5 > --- trap 0xc, eip = 0xc05dafc6, esp = 0xde7b0c24, ebp = 0xde7b0c28 --- > m_tag_locate(c55df900,0,f,0) at m_tag_locate+0x36 > dn_tag_get(c55df900,2ffbd300,1,c05c3e7e,c088e858,...) at dn_tag_get+0x1d > ready_event_wfq(c57b0800,de7b0cac,de7b0cb0) at ready_event_wfq+0x50b > dummynet_task(0,1) at dummynet_task+0x24c > taskqueue_run(c5562a00) at taskqueue_run+0xd1 > taskqueue_thread_loop(c08ce950,de7b0d38,c08ce950,c05c01e0,0,...) at > taskqueue_thread_loop+0x4a > fork_exit(c05c01e0,c08ce950,de7b0d38) at fork_exit+0xa8 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0x1, eip = 0, esp = 0xde7b0d6c, ebp = 0 --- > Uptime: 50m0s > Dumping 511 MB (2 chunks) > chunk 0: 1MB (156 pages) ... ok > chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 > 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 > 31 15 > > #0 doadump () at pcpu.h:165 > 165 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) bt > #0 doadump () at pcpu.h:165 > #1 0xc059f2a6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 > #2 0xc059f57b in panic (fmt=0xc078df19 "%s") at > /usr/src/sys/kern/kern_shutdown.c:565 > #3 0xc076c1f7 in trap_fatal (frame=0xde7b0be4, eva=3961658759) at > /usr/src/sys/i386/i386/trap.c:837 > #4 0xc076bf0b in trap_pfault (frame=0xde7b0be4, usermode=0, > eva=3961658759) at /usr/src/sys/i386/i386/trap.c:745 > #5 0xc076bb71 in trap (frame= > {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 1, tf_esi = 0, tf_ebp > = -562361304, tf_isp = -562361328, tf_ebx = 15, tf_edx = -333308545, > tf_ecx = 0, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = > -1067601978, tf_cs = 32, tf_eflags = 66178, tf_esp = 22, tf_ss = > -562361280}) at /usr/src/sys/i386/i386/trap.c:435 > #6 0xc0758bca in calltrap () at /usr/src/sys/i386/i386/exception.s:139 > #7 0xc05dafc6 in m_tag_locate (m=0xec221d7f, cookie=0, type=15, t=0x0) > at /usr/src/sys/kern/uipc_mbuf2.c:392 > #8 0xc06279ad in dn_tag_get (m=0xec221d7f) at mbuf.h:881 > #9 0xc06281fb in ready_event_wfq (p=0xc57b0800, head=0xde7b0cac, > tail=0xde7b0cb0) at /usr/src/sys/netinet/ip_dummynet.c:705 > #10 0xc06284cc in dummynet_task (context=0x0, pending=0) at > /usr/src/sys/netinet/ip_dummynet.c:805 > #11 0xc05bfe71 in taskqueue_run (queue=0xc5562a00) at > /usr/src/sys/kern/subr_taskqueue.c:257 > #12 0xc05c022a in taskqueue_thread_loop (arg=0x0) at > /usr/src/sys/kern/subr_taskqueue.c:376 > #13 0xc05897b8 in fork_exit (callout=0xc05c01e0 , > arg=0xc08ce950, frame=0xde7b0d38) > at /usr/src/sys/kern/kern_fork.c:821 > #14 0xc0758c2c in fork_trampoline () at > /usr/src/sys/i386/i386/exception.s:208 > (kgdb) up 9 > #9 0xc06281fb in ready_event_wfq (p=0xc57b0800, head=0xde7b0cac, > tail=0xde7b0cb0) at /usr/src/sys/netinet/ip_dummynet.c:705 > 705 dn_tag_get(p->tail)->output_time += t ; > (kgdb) p *p > $1 = {next = {sle_next = 0xc6713600}, pipe_nr = 1700, bandwidth = > 50000000, delay = 0, head = 0x0, tail = 0xc55df900, > scheduler_heap = {size = 16, elements = 1, offset = 0, p = > 0xc57b2800}, not_eligible_heap = {size = 16, elements = 0, > offset = 0, p = 0xc57ac700}, idle_heap = {size = 16, elements = 0, > offset = 124, p = 0xc56a2800}, V = 9830400, > sum = 10, numbytes = -1090027776, sched_time = 2997985, if_name = '\0' > , ifp = 0x0, ready = 0, fs = { > next = {sle_next = 0x0}, fs_nr = 0, flags_fs = 0, pipe = 0xc57b0800, > parent_nr = 0, weight = 0, qsize = 50, plr = 0, > flow_mask = {dst_ip = 0, src_ip = 0, dst_port = 0, src_port = 0, > proto = 0 '\0', flags = 0 '\0', addr_type = 0 '\0', > dst_ip6 = {__u6_addr = {__u6_addr8 = '\0' , > __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = { > 0, 0, 0, 0}}}, src_ip6 = {__u6_addr = {__u6_addr8 = '\0' > , __u6_addr16 = {0, 0, 0, 0, 0, 0, > 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, flow_id6 = 0, frag_id6 > = 0}, rq_size = 1, rq_elements = 0, > rq = 0xc55791b0, last_expired = 0, backlogged = 0, w_q = 0, max_th = > 0, min_th = 0, max_p = 0, c_1 = 0, c_2 = 0, > c_3 = 0, c_4 = 0, w_q_lookup = 0x0, lookup_depth = 0, lookup_step = > 0, lookup_weight = 0, avg_pkt_size = 0, > max_pkt_size = 0}} > > > When I remove "tag 1" the kernel stopped panick, but deadlocks didn't > pass away. When I managed to enter DDB using serial console I found > dummynet_task() looped on the following code: > > h = heaps[i]; > while (h->elements > 0 && DN_KEY_LEQ(h->p[0].key, curr_time)) { > ... > ready_event_wfq(p, &head, &tail); > ... > } > It seems to me that problem is in ready_event_wfq() in the following code: > if (p->bandwidth > 0) > t = (p->bandwidth -1 - p->numbytes) / p->bandwidth ; > > Since p->bandwidth and p->numbytes are signed integers, the result can > be negative (i have p->bandwidth=50000000 and p->numbytes=-2147483647) > > Now i test attached patch. I hope it will help. :) Could you please be so kind and test whether SMP has any effect on the bug. I.e. does an unpatched ip_dummynet without SMP cause panics? I ask this because I was unable to reproduce this bug on a non-SMP machine. Also, I see you have "dummynet_task" in your dumps. Are using RELENG_6 or 1.93.2.6 of ip_dummynet.c? Responsible-Changed-From-To: freebsd-net->oleg Responsible-Changed-By: oleg Responsible-Changed-When: Fri Jun 15 17:21:30 UTC 2007 Responsible-Changed-Why: take over. http://www.freebsd.org/cgi/query-pr.cgi?pr=113548 From: Oleg Bulyzhin To: Alexey Illarionov Cc: cristi@net.utcluj.ro, bug-followup@freebsd.org Subject: Re: kern/113548: [dummynet] [patch] system hangs with dummynet queues Date: Mon, 18 Jun 2007 11:54:09 +0400 --EeQfGwPcQSOJBaQU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello. As i can see cause of the problem is p->numbytes overflow. Using m_tag_find() instead of m_tag_first() should not be an issue, since dummynet's tag is always first in tag chain while packet is in dummynet. Could you please test attached patch? It is not completely correct (right way is converting p->numbytes to int64_t, but this would lead to ABI breakage and is not applicable to RELENG_6), but should cure your crashes and hangs. -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg@rinet.ru === ================================================================ --EeQfGwPcQSOJBaQU Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="ip_dummynet.c.diff" Index: sys/netinet/ip_dummynet.c =================================================================== RCS file: /home/ncvs/src/sys/netinet/ip_dummynet.c,v retrieving revision 1.107 diff -u -r1.107 ip_dummynet.c --- sys/netinet/ip_dummynet.c 17 Jun 2007 00:33:34 -0000 1.107 +++ sys/netinet/ip_dummynet.c 18 Jun 2007 07:23:51 -0000 @@ -55,6 +55,7 @@ * include files marked with XXX are probably not needed */ +#include #include #include #include @@ -604,11 +605,12 @@ int p_was_empty = (p->head == NULL) ; struct dn_heap *sch = &(p->scheduler_heap); struct dn_heap *neh = &(p->not_eligible_heap) ; + int64_t p_numbytes = p->numbytes; DUMMYNET_LOCK_ASSERT(); if (p->if_name[0] == 0) /* tx clock is simulated */ - p->numbytes += ( curr_time - p->sched_time ) * p->bandwidth; + p_numbytes += (curr_time - p->sched_time) * p->bandwidth; else { /* tx clock is for real, the ifq must be empty or this is a NOP */ if (p->ifp && p->ifp->if_snd.ifq_head != NULL) return ; @@ -622,7 +624,7 @@ * While we have backlogged traffic AND credit, we need to do * something on the queue. */ - while ( p->numbytes >=0 && (sch->elements>0 || neh->elements >0) ) { + while ( p_numbytes >=0 && (sch->elements>0 || neh->elements >0) ) { if (sch->elements > 0) { /* have some eligible pkts to send out */ struct dn_flow_queue *q = sch->p[0].object ; struct mbuf *pkt = q->head; @@ -631,7 +633,7 @@ int len_scaled = p->bandwidth ? len*8*hz : 0 ; heap_extract(sch, NULL); /* remove queue from heap */ - p->numbytes -= len_scaled ; + p_numbytes -= len_scaled ; move_pkt(pkt, q, p, len); p->V += (len<sum ; /* update V */ @@ -668,11 +670,11 @@ } if (p->if_name[0] != '\0') {/* tx clock is from a real thing */ - p->numbytes = -1 ; /* mark not ready for I/O */ + p_numbytes = -1 ; /* mark not ready for I/O */ break ; } } - if (sch->elements == 0 && neh->elements == 0 && p->numbytes >= 0 + if (sch->elements == 0 && neh->elements == 0 && p_numbytes >= 0 && p->idle_heap.elements > 0) { /* * no traffic and no events scheduled. We can get rid of idle-heap. @@ -694,11 +696,11 @@ * If we are under credit, schedule the next ready event. * Also fix the delivery time of the last packet. */ - if (p->if_name[0]==0 && p->numbytes < 0) { /* this implies bandwidth >0 */ + if (p->if_name[0]==0 && p_numbytes < 0) { /* this implies bandwidth >0 */ dn_key t=0 ; /* number of ticks i have to wait */ if (p->bandwidth > 0) - t = ( p->bandwidth -1 - p->numbytes) / p->bandwidth ; + t = (p->bandwidth - 1 - p_numbytes) / p->bandwidth; dn_tag_get(p->tail)->output_time += t ; p->sched_time = curr_time ; heap_insert(&wfq_ready_heap, curr_time + t, (void *)p); @@ -706,6 +708,14 @@ * queue on error hoping next time we are luckier. */ } + + if (p_numbytes > INT_MAX) + p->numbytes = INT_MAX; + else if (p_numbytes < INT_MIN) + p->numbytes = INT_MIN; + else + p->numbytes = p_numbytes; + /* * If the delay line was empty call transmit_event() now. * Otherwise, the scheduler will take care of it. --EeQfGwPcQSOJBaQU-- State-Changed-From-To: open->feedback State-Changed-By: oleg State-Changed-When: Mon Jun 18 08:49:05 UTC 2007 State-Changed-Why: waiting for test results. http://www.freebsd.org/cgi/query-pr.cgi?pr=113548 From: Alexey Illarionov To: Oleg Bulyzhin Cc: cristi@net.utcluj.ro, bug-followup@freebsd.org Subject: Re: kern/113548: [dummynet] [patch] system hangs with dummynet queues Date: Fri, 22 Jun 2007 09:28:10 +0400 Oleg Bulyzhin wrote: > Could you please test attached patch? It is not completely correct > (right way is converting p->numbytes to int64_t, but this would lead > to ABI breakage and is not applicable to RELENG_6), but should cure your > crashes and hangs. I have tested this patch. It has solved all my problems with dummmynet. Thank you very much! From: Sergey Bondarev To: bug-followup@FreeBSD.org, littlesavage@orionet.ru Cc: Subject: Re: kern/113548: [dummynet] [patch] system hangs with dummynet queues Date: Mon, 19 Nov 2007 16:18:22 +0300 Hello bug-followup, oh, sorry system is FreeBSD mr2 6.2-RELEASE-p7 FreeBSD 6.2-RELEASE-p7 #0: Thu Sep 27 13:51:30 MSD 2007 bond@mr2:/usr/obj/usr/src/sys/YADROP i386 С уважением, Бондарев Сергей mailto:bond@techno-r.ru From: Sergey Bondarev To: bug-followup@FreeBSD.org, littlesavage@orionet.ru Cc: Subject: Re: kern/113548: [dummynet] [patch] system hangs with dummynet queues Date: Mon, 19 Nov 2007 16:04:05 +0300 Hello bug-followup, I am getting this bug too. System freeze - ping to system is ok, but ssh,www service not responding. Also in /var/log/messages many messages: Nov 19 15:00:00 mr2 kernel: dummynet: waking up pipe 1021 at 0 Nov 19 15:00:00 mr2 kernel: dummynet: waking up pipe 1021 at 0 Nov 19 15:00:00 mr2 kernel: dummynet: waking up pipe 1020 at 28 Nov 19 15:00:00 mr2 kernel: dummynet: waking up pipe 1021 at 0 Nov 19 15:00:00 mr2 kernel: dummynet: waking up pipe 1020 at 28 Nov 19 15:00:00 mr2 kernel: dummynet: waking up pipe 1021 at 0 Nov 19 15:00:00 mr2 kernel: dummynet: waking up pipe 1020 at 11 Nov 19 15:00:00 mr2 kernel: dummynet: waking up pipe 1020 at 28 Nov 19 15:00:00 mr2 kernel: dummynet: waking up pipe 1021 at 0 Nov 19 15:00:01 mr2 kernel: dummynet: waking up pipe 1021 at 0 Nov 19 15:00:01 mr2 kernel: dummynet: waking up pipe 1020 at 28 i am reset system by power key, but flood to /var/log/messges continues... mr2# ipfw pipe show 01020: 332.000 Kbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail 01021: 332.000 Kbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail 00001: 1.024 Mbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail 00920: 256.000 Kbit/s 0 ms 50 sl. 3 queues (64 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 8 ip 0.0.0.0/0 192.168.2.88/0 100565 84924693 0 0 4 10 ip 0.0.0.0/0 192.168.2.90/0 41002 11680979 0 0 0 33 ip 0.0.0.0/0 192.168.2.49/0 221 202567 0 0 0 00002: 300.000 Kbit/s 0 ms 50 sl. 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 tcp 192.168.7.3/1620 80.82.32.21/80 1182552 549941097 50 19116 107737 00921: 256.000 Kbit/s 0 ms 50 sl. 3 queues (64 buckets) droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 10 ip 192.168.2.49/0 0.0.0.0/0 237 35167 0 0 0 24 ip 192.168.2.88/0 0.0.0.0/0 80648 4063458 0 0 0 28 ip 192.168.2.90/0 0.0.0.0/0 40374 1977138 0 0 0 01001: 64.000 Kbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail 00911: 128.000 Kbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 54 ip 192.168.2.239/0 0.0.0.0/0 8 608 0 0 0 01000: 64.000 Kbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail 00910: 128.000 Kbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 63 ip 0.0.0.0/0 192.168.2.239/0 8 608 0 0 0 00901: 64.000 Kbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 50 ip 192.168.2.13/0 0.0.0.0/0 19846 1049973 0 0 0 01010: 128.000 Kbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail 00900: 64.000 Kbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 29 ip 0.0.0.0/0 192.168.2.13/0 21861 20561061 20 28000 21 01011: 128.000 Kbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail q01020: weight 50 pipe 1020 20 sl. 3 queues (64 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 8 ip 0.0.0.0/0 192.168.2.88/0 100561 84922408 0 0 0 10 ip 0.0.0.0/0 192.168.2.90/0 41002 11680979 0 0 0 33 ip 0.0.0.0/0 192.168.2.49/0 221 202567 0 0 0 q01021: weight 50 pipe 1021 20 sl. 3 queues (64 buckets) droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 10 ip 192.168.2.49/0 0.0.0.0/0 237 35167 0 0 0 24 ip 192.168.2.88/0 0.0.0.0/0 80648 4063458 0 0 0 28 ip 192.168.2.90/0 0.0.0.0/0 40374 1977138 0 0 0 q01001: weight 50 pipe 1001 20 sl. 1 queues (64 buckets) droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 50 ip 192.168.2.13/0 0.0.0.0/0 19846 1049973 0 0 0 q01000: weight 50 pipe 1000 20 sl. 1 queues (64 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 29 ip 0.0.0.0/0 192.168.2.13/0 21820 20515438 0 0 0 q00020: weight 1 pipe 1 50 sl. 0 queues (1 buckets) droptail q00010: weight 100 pipe 1 50 sl. 0 queues (1 buckets) droptail q01010: weight 50 pipe 1010 20 sl. 1 queues (64 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 63 ip 0.0.0.0/0 192.168.2.239/0 8 608 0 0 0 q01011: weight 50 pipe 1011 20 sl. 1 queues (64 buckets) droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 54 ip 192.168.2.239/0 0.0.0.0/0 8 608 0 0 0 ipfw show 1000-2000 01001 22169 20978066 pipe 900 ip from any to 192.168.2.13 via ng0 01001 20102 1061495 pipe 901 ip from 192.168.2.13 to any via ng0 01001 22140 20950348 queue 1000 ip from any to 192.168.2.13 via ng0 01001 20102 1061495 queue 1001 ip from 192.168.2.13 to any via ng0 01002 101092 85060044 pipe 920 ip from any to 192.168.2.88 via ng1 01002 81169 4088280 pipe 921 ip from 192.168.2.88 to any via ng1 01002 101088 85057759 queue 1020 ip from any to 192.168.2.88 via ng1 01002 81169 4088280 queue 1021 ip from 192.168.2.88 to any via ng1 01004 8 608 pipe 910 ip from any to 192.168.2.239 via ng3 01004 8 608 pipe 911 ip from 192.168.2.239 to any via ng3 01004 8 608 queue 1010 ip from any to 192.168.2.239 via ng3 01004 8 608 queue 1011 ip from 192.168.2.239 to any via ng3 01007 41504 11840267 pipe 920 ip from any to 192.168.2.90 via ng6 01007 40866 2000348 pipe 921 ip from 192.168.2.90 to any via ng6 01007 41504 11840267 queue 1020 ip from any to 192.168.2.90 via ng6 01007 40865 2000308 queue 1021 ip from 192.168.2.90 to any via ng6 I am trying patch-3.diff and report result later. С уважением, Бондарев Сергей mailto:bond@techno-r.ru State-Changed-From-To: feedback->patched State-Changed-By: oleg State-Changed-When: Tue Dec 25 09:37:56 UTC 2007 State-Changed-Why: Commited to HEAD. http://www.freebsd.org/cgi/query-pr.cgi?pr=113548 From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/113548: commit references a PR Date: Tue, 25 Dec 2007 09:36:59 +0000 (UTC) oleg 2007-12-25 09:36:51 UTC FreeBSD src repository Modified files: sys/netinet ip_dummynet.c Log: Workaround p->numbytes overflow, which can result in infinite loop inside dummynet module (prerequisite is using queues with "fat" pipe). PR: kern/113548 Revision Changes Path 1.114 +22 -7 src/sys/netinet/ip_dummynet.c _______________________________________________ cvs-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/cvs-all To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org" From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/113548: commit references a PR Date: Fri, 25 Apr 2008 10:26:38 +0000 (UTC) oleg 2008-04-25 10:26:31 UTC FreeBSD src repository Modified files: (Branch: RELENG_7) sys/netinet ip_dummynet.h ip_dummynet.c ip_fw_pfil.c sys/net if_bridge.c if_ethersubr.c sbin/ipfw ipfw.8 Log: MFC: src/sys/netinet/ip_dummynet.h 1.41 src/sys/netinet/ip_dummynet.c 1.111-1.114 src/sys/netinet/ip_fw_pfil.c 1.26 src/sys/net/if_bridge.c 1.107 src/sys/net/if_ethersubr.c 1.240 src/sbin/ipfw/ipfw.8 1.206 - style(9) cleanup. - dummynet_io() declaration has changed. - Alter packet flow inside dummynet and introduce 'fast' mode of dummynet operation: allow certain packets to bypass dummynet scheduler. Benefits are: -- lower latency: if packet flow does not exceed pipe bandwidth, packets will not be (up to tick) delayed (due to dummynet's scheduler granularity). -- lower overhead: if packet avoids dummynet scheduler it shouldn't reenter ip stack later. Such packets can be fastforwarded. -- recursion (which can lead to kernel stack exhaution) eliminated. This fix long existed panic, which can be triggered this way: kldload dummynet sysctl net.inet.ip.fw.one_pass=0 ipfw pipe 1 config bw 0 for i in `jot 30`; do ipfw add 1 pipe 1 icmp from any to any; done ping -c 1 localhost - New sysctl nodes: net.inet.ip.dummynet.io_fast - enables 'fast' dummynet io net.inet.ip.dummynet.io_pkt - packets passed to dummynet net.inet.ip.dummynet.io_pkt_fast - packets avoided dummynet scheduler net.inet.ip.dummynet.io_pkt_drop - packets dropped by dummynet - Workaround p->numbytes overflow, which can result in infinite loop inside dummynet module (prerequisite is using queues with "fat" pipe). PR: kern/113548 kern/121955 Revision Changes Path 1.203.2.4 +18 -0 src/sbin/ipfw/ipfw.8 1.103.2.6 +1 -1 src/sys/net/if_bridge.c 1.236.2.2 +1 -1 src/sys/net/if_ethersubr.c 1.110.2.1 +395 -337 src/sys/netinet/ip_dummynet.c 1.40.2.1 +1 -1 src/sys/netinet/ip_dummynet.h 1.25.2.2 +28 -26 src/sys/netinet/ip_fw_pfil.c _______________________________________________ cvs-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/cvs-all To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org" From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/113548: commit references a PR Date: Fri, 25 Apr 2008 10:29:31 +0000 (UTC) oleg 2008-04-25 10:29:26 UTC FreeBSD src repository Modified files: (Branch: RELENG_6) sys/netinet ip_dummynet.h ip_dummynet.c ip_fw_pfil.c sys/net if_bridge.c if_ethersubr.c sbin/ipfw ipfw.8 Log: MFC: src/sys/netinet/ip_dummynet.h 1.41 src/sys/netinet/ip_dummynet.c 1.111-1.114 src/sys/netinet/ip_fw_pfil.c 1.26 src/sys/net/if_bridge.c 1.107 src/sys/net/if_ethersubr.c 1.240 src/sbin/ipfw/ipfw.8 1.206 - style(9) cleanup. - dummynet_io() declaration has changed. - Alter packet flow inside dummynet and introduce 'fast' mode of dummynet operation: allow certain packets to bypass dummynet scheduler. Benefits are: -- lower latency: if packet flow does not exceed pipe bandwidth, packets will not be (up to tick) delayed (due to dummynet's scheduler granularity). -- lower overhead: if packet avoids dummynet scheduler it shouldn't reenter ip stack later. Such packets can be fastforwarded. -- recursion (which can lead to kernel stack exhaution) eliminated. This fix long existed panic, which can be triggered this way: kldload dummynet sysctl net.inet.ip.fw.one_pass=0 ipfw pipe 1 config bw 0 for i in `jot 30`; do ipfw add 1 pipe 1 icmp from any to any; done ping -c 1 localhost - New sysctl nodes: net.inet.ip.dummynet.io_fast - enables 'fast' dummynet io net.inet.ip.dummynet.io_pkt - packets passed to dummynet net.inet.ip.dummynet.io_pkt_fast - packets avoided dummynet scheduler net.inet.ip.dummynet.io_pkt_drop - packets dropped by dummynet - Workaround p->numbytes overflow, which can result in infinite loop inside dummynet module (prerequisite is using queues with "fat" pipe). PR: kern/113548 kern/121955 Revision Changes Path 1.175.2.15 +18 -0 src/sbin/ipfw/ipfw.8 1.11.2.55 +1 -1 src/sys/net/if_bridge.c 1.193.2.16 +1 -1 src/sys/net/if_ethersubr.c 1.93.2.7 +395 -337 src/sys/netinet/ip_dummynet.c 1.36.2.3 +1 -1 src/sys/netinet/ip_dummynet.h 1.19.2.4 +28 -26 src/sys/netinet/ip_fw_pfil.c _______________________________________________ cvs-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/cvs-all To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org" State-Changed-From-To: patched->closed State-Changed-By: oleg State-Changed-When: Wed May 7 17:41:16 UTC 2008 State-Changed-Why: commited to RELENG_[67] http://www.freebsd.org/cgi/query-pr.cgi?pr=113548 >Unformatted: