From anders@totem.fix.no Sat Jan 21 00:00:57 2006 Return-Path: Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1702C16A41F; Sat, 21 Jan 2006 00:00:57 +0000 (GMT) (envelope-from anders@totem.fix.no) Received: from totem.fix.no (totem.fix.no [80.91.36.20]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5AE7E43D49; Sat, 21 Jan 2006 00:00:56 +0000 (GMT) (envelope-from anders@totem.fix.no) Received: by totem.fix.no (Postfix, from userid 1000) id 1BA9A8DB147; Sat, 21 Jan 2006 01:00:54 +0100 (CET) Message-Id: <20060121000054.1BA9A8DB147@totem.fix.no> Date: Sat, 21 Jan 2006 01:00:54 +0100 (CET) From: Anders Nordby Reply-To: Anders Nordby To: FreeBSD-gnats-submit@freebsd.org Cc: sam@FreeBSD.org, damien@FreeBSD.org Subject: panic using WPA on ural NIC in 6.0-RELEASE X-Send-Pr-Version: 3.113 X-GNATS-Notify: >Number: 92083 >Category: usb >Synopsis: [ural] [panic] panic using WPA on ural NIC in 6.0-RELEASE >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-usb >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Jan 21 00:10:03 GMT 2006 >Closed-Date: >Last-Modified: Sat Jan 5 16:10:01 UTC 2008 >Originator: Anders Nordby >Release: FreeBSD 6.0-RELEASE i386 >Organization: - >Environment: System: FreeBSD stream.localnet 6.0-RELEASE FreeBSD 6.0-RELEASE #0: Fri Jan 20 20:48:44 CET 2006 root@stream.localnet:/usr/obj/usr/src/sys/STREAM i386 Using D-Link DWL-G122 USB Wireless NIC. Running GENERIC kernel with IPv6 stripped and the following added to kernel config: device sound device "snd_via8233" device wlan #802.11 support device wlan_wep #802.11 WEP support device wlan_ccmp #802.11 CCMP support device wlan_tkip #802.11 TKIP support device wlan_xauth #802.11 external authenticator support device wlan_acl #802.11 MAC ACL support device acpi options KDB options DDB Using /etc/wpa_supplicant.conf like this: ctrl_interface_group=0 eapol_version=1 ap_scan=1 fast_reauth=1 network={ ssid="SOMENETWORK" scan_ssid=1 key_mgmt=WPA-PSK psk="SOMEPASSWORD" } NIC is configured through rc.conf: ifconfig_ural0="inet X.X.X.X netmask 0xYYYYYYYY WPA mode 11g" >Description: System panics: fault virtual address = 0x4 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0705b21 stack pointer = 0x28:0xcab36c00 frame pointer = 0x28:0xcab36c0c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 23 (irq12: vr0 ehci0) [thread pid 23 tid 100010 ] Stopped at ieee80211_free_node+0x9: movl 0x4(%esi),%ebx db> 1^H ^Hpanic panic: from debugger Uptime: 14m56s Dumping 223 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 223MB (57072 pages) 207 191 175 159 143 127 111 95 79 63 47 31 15 ... ok Dump complete Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... Checking where this is with gdb, I get: stream# gdb /usr/obj/usr/src/sys/STREAM/kernel.debug GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... (gdb) l *0xc0705b21 0xc0705b21 is in ieee80211_free_node (/usr/src/sys/net80211/ieee80211_node.c:154 2). 1537 ieee80211_free_node_debug(struct ieee80211_node *ni, const char *func, i nt line) 1538 #else 1539 ieee80211_free_node(struct ieee80211_node *ni) 1540 #endif 1541 { 1542 struct ieee80211_node_table *nt = ni->ni_table; 1543 1544 #ifdef IEEE80211_DEBUG_REFCNT 1545 IEEE80211_DPRINTF(ni->ni_ic, IEEE80211_MSG_NODE, 1546 "%s (%s:%u) %p<%s> refcnt %d\n", __func__, func, line, n i, (gdb) Analyzing the crashdump I get: stream# kgdb /usr/obj/usr/src/sys/STREAM/kernel.debug /var/crash/vmcore.0 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Unde fined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0x4 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0705b21 stack pointer = 0x28:0xcab36c00 frame pointer = 0x28:0xcab36c0c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 23 (irq12: vr0 ehci0) panic: from debugger Uptime: 14m56s Dumping 223 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 223MB (57072 pages) 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) where #0 doadump () at pcpu.h:165 #1 0xc067a7aa in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xc067aa70 in panic (fmt=0xc08648d1 "from debugger") at /usr/src/sys/kern/kern_shutdown.c:555 #3 0xc0487d71 in db_panic (addr=-1066378463, have_addr=0, count=-1, modif=0xcab36a2c "") at /usr/src/sys/ddb/db_command.c:438 #4 0xc0487d08 in db_command (last_cmdp=0xc0940684, cmd_table=0x0, aux_cmd_tablep=0xc08ba69c, aux_cmd_tablep_end=0xc08ba6b8) at /usr/src/sys/ddb/db_command.c:350 #5 0xc0487dd0 in db_command_loop () at /usr/src/sys/ddb/db_command.c:458 #6 0xc04899dd in db_trap (type=12, code=0) at /usr/src/sys/ddb/db_main.c:221 #7 0xc0692fa3 in kdb_trap (type=12, code=0, tf=0xcab36bc0) at /usr/src/sys/kern/subr_kdb.c:473 #8 0xc082d600 in trap_fatal (frame=0xcab36bc0, eva=4) at /usr/src/sys/i386/i386/trap.c:822 #9 0xc082d36f in trap_pfault (frame=0xcab36bc0, usermode=0, eva=4) at /usr/src/sys/i386/i386/trap.c:742 #10 0xc082cf69 in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -1055113216, tf_esi = 0, tf_e bp = -894211060, tf_isp = -894211092, tf_ebx = -1055110224, tf_edx = -1055541248 , tf_ecx = 0, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1066378463, tf_c s = 32, tf_eflags = 66118, tf_esp = -1055110224, tf_ss = -1055079424}) at /usr/src/sys/i386/i386/trap.c:432 #11 0xc081c46a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #12 0xc0705b21 in ieee80211_free_node (ni=0x0) at /usr/src/sys/net80211/ieee80211_node.c:1541 #13 0xc060453f in ural_txeof (xfer=0xc11afa00, priv=0xc11c4bb0, status=USBD_NORMAL_COMPLETION) at /usr/src/sys/dev/usb/if_ural.c:826 #14 0xc061bd80 in usb_transfer_complete (xfer=0xc11afa00) at /usr/src/sys/dev/usb/usbdi.c:851 #15 0xc05fa5c4 in ehci_idone (ex=0xc11afa00) at /usr/src/sys/dev/usb/ehci.c:867 #16 0xc05fa49f in ehci_check_intr (sc=0xc115b800, ex=0xc11afa00) at /usr/src/sys/dev/usb/ehci.c:752 #17 0xc05fa419 in ehci_softintr (v=0xc115b800) at /usr/src/sys/dev/usb/ehci.c:692 #18 0xc06190d1 in usb_schedsoftintr (bus=0x0) at /usr/src/sys/dev/usb/usb.c:871 #19 0xc05fa1fa in ehci_intr1 (sc=0xc115b800) at /usr/src/sys/dev/usb/ehci.c:592 #20 0xc05fa13a in ehci_intr (v=0xc115b800) at /usr/src/sys/dev/usb/ehci.c:551 #21 0xc0665da9 in ithread_loop (arg=0xc1082700) at /usr/src/sys/kern/kern_intr.c:547 #22 0xc0665030 in fork_exit (callout=0xc0665c50 , arg=0xc1082700, frame=0xcab36d38) at /usr/src/sys/kern/kern_fork.c:789 #23 0xc081c4cc in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 (kgdb) quit >How-To-Repeat: Run any kind of real network load (like trying to cvsup ports), and the system will panic. >Fix: N/A >Release-Note: >Audit-Trail: Responsible-Changed-From-To: freebsd-bugs->damien@FreeBSD.org Responsible-Changed-By: anders Responsible-Changed-When: Thu Mar 9 20:49:07 UTC 2006 Responsible-Changed-Why: Over to Damien, seems you are the man for this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=92083 Responsible-Changed-From-To: damien@FreeBSD.org->damien Responsible-Changed-By: linimon Responsible-Changed-When: Fri Mar 10 00:31:16 UTC 2006 Responsible-Changed-Why: Canonicalize assignment. http://www.freebsd.org/cgi/query-pr.cgi?pr=92083 Responsible-Changed-From-To: damien->freebsd-bugs Responsible-Changed-By: linimon Responsible-Changed-When: Thu Mar 30 08:34:54 UTC 2006 Responsible-Changed-Why: Assignee is no longer working on FreeBSD drivers. http://www.freebsd.org/cgi/query-pr.cgi?pr=92083 From: Jonathan Fosburgh To: bug-followup@freebsd.org, anders@freebsd.org Cc: Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.0-RELEASE Date: Sun, 12 Nov 2006 20:50:11 -0600 Has anyone been able to figure anything out with this? I hate having to use WEP... From: Anders Nordby To: Jonathan Fosburgh Cc: bug-followup@freebsd.org Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.0-RELEASE Date: Mon, 13 Nov 2006 09:27:47 +0100 Hi, I didn't get any feedback on this PR. But I haven't tested my ural NIC since 6.0, so you may or may not be lucky with a newer version. I think WEP with this NIC was more stable, but I don't remember for sure. On Sun, Nov 12, 2006 at 08:50:11PM -0600, Jonathan Fosburgh wrote: > Has anyone been able to figure anything out with this? I hate having to use > WEP... -- Anders. From: Jonathan Fosburgh To: Anders Nordby Cc: bug-followup@freebsd.org Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.0-RELEASE Date: Mon, 13 Nov 2006 07:26:40 -0600 On Monday 13 November 2006 02:27, Anders Nordby wrote: > Hi, > > I didn't get any feedback on this PR. But I haven't tested my ural NIC > since 6.0, so you may or may not be lucky with a newer version. I think > WEP with this NIC was more stable, but I don't remember for sure. > I tested last night on 6.2-PRE and it still panics the kernel. :( From: Anders Nordby To: bug-followup@FreeBSD.org, Jonathan Fosburgh , Sam Leffler Cc: Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.2-RELEASE Date: Sun, 11 Feb 2007 20:36:39 +0100 Hi, Just been trying the same thing in 6.2-RELEASE, with the same hardware. It seems the same bug still occurs, the stack trace looks very similar. Everytime I try do do a cvsup of the ports tree, the system panics. stream# uname -a FreeBSD stream.localnet 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Tue Feb 6 22:09:01 CET 2007 root@stream.localnet:/usr/obj/usr/src/sys/STREAM i386 stream# kgdb /usr/obj/usr/src/sys/STREAM/kernel.debug /var/crash/vmcore.0 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: ural0: could not transmit buffer: SHORT_XFER Fatal trap 12: page fault while in kernel mode fault virtual address = 0x4 fault code = supervisor read, page not present instruction pointer = 0x20:0xc070a305 stack pointer = 0x28:0xd5693be4 frame pointer = 0x28:0xd5693bf0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 21 (irq5: uhci0 uhci1) trap number = 12 panic: page fault KDB: stack backtrace: kdb_backtrace(100,c33a0780,28,d5693ba4,c,...) at kdb_backtrace+0x29 panic(c08cecee,c0923d2a,0,fffff,c339f49b,...) at panic+0xa8 trap_fatal(d5693ba4,4,c33a0780,0,c,...) at trap_fatal+0x2a6 trap_pfault(d5693ba4,0,4) at trap_pfault+0x1f3 trap(8,28,28,c33e2000,0,...) at trap+0x325 calltrap() at calltrap+0x5 --- trap 0xc, eip = 0xc070a305, esp = 0xd5693be4, ebp = 0xd5693bf0 --- ieee80211_free_node(0,0,c35c0700,c3563a80,0,...) at ieee80211_free_node+0x9 ural_txeof(c35c0700,c33e2cd4,0) at ural_txeof+0x83 usb_transfer_complete(c35c0700,c33a0780,c35c0754,a8,c3563a80,...) at usb_transfer_complete+0x14e uhci_idone(c35c0770,c33fb400,c35c0788,c35c0754,0,...) at uhci_idone+0x11b uhci_check_intr(c3389000,c35c0770) at uhci_check_intr+0x98 uhci_softintr(c3389000,d5693cc4,c060a82a,c3389000,c33bca80,...) at uhci_softintr+0x22 usb_schedsoftintr(c3389000,c33bca80,4,c32bfa80,d5693cd0,...) at usb_schedsoftintr+0xd uhci_intr1(c3389000,d5693cec,c0663f61,c3389000,c33ca740,...) at uhci_intr1+0x1ce uhci_intr(c3389000) at uhci_intr+0x28 ithread_execute_handlers(c339f430,c32bfa80) at ithread_execute_handlers+0x121 ithread_loop(c33bd340,d5693d38) at ithread_loop+0x54 fork_exit(c0664018,c33bd340,d5693d38) at fork_exit+0x70 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd5693d6c, ebp = 0 --- Uptime: 12m23s Dumping 511 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc0679632 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc06798f8 in panic (fmt=0xc08cecee "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc089043e in trap_fatal (frame=0xd5693ba4, eva=4) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc089016f in trap_pfault (frame=0xd5693ba4, usermode=0, eva=4) at /usr/src/sys/i386/i386/trap.c:745 #5 0xc088fd69 in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -1019338752, tf_esi = 0, tf_ebp = -714523664, tf_isp = -714523696, tf_ebx = -1019335468, tf_edx = -1019454592, tf_ecx = -1019703296, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1066360059, tf_cs = 32, tf_eflags = 590406, tf_esp = -1019335468, tf_ss = -1019205632}) at /usr/src/sys/i386/i386/trap.c:435 #6 0xc087dc8a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc070a305 in ieee80211_free_node (ni=0x0) at /usr/src/sys/net80211/ieee80211_node.c:1605 #8 0xc060016f in ural_txeof (xfer=0xc35c0700, priv=0xc33e2cd4, status=USBD_NORMAL_COMPLETION) at /usr/src/sys/dev/usb/if_ural.c:888 #9 0xc0618326 in usb_transfer_complete (xfer=0xc35c0700) at /usr/src/sys/dev/usb/usbdi.c:863 #10 0xc060aa2f in uhci_idone (ii=0x0) at /usr/src/sys/dev/usb/uhci.c:1499 #11 0xc060a90c in uhci_check_intr (sc=0xc3389000, ii=0xc35c0770) at /usr/src/sys/dev/usb/uhci.c:1374 #12 0xc060a85e in uhci_softintr (v=0xc3389000) at /usr/src/sys/dev/usb/uhci.c:1304 #13 0xc061566d in usb_schedsoftintr (bus=0x0) at /usr/src/sys/dev/usb/usb.c:871 #14 0xc060a82a in uhci_intr1 (sc=0xc3389000) at /usr/src/sys/dev/usb/uhci.c:1274 #15 0xc060a658 in uhci_intr (arg=0xc3389000) at /usr/src/sys/dev/usb/uhci.c:1189 #16 0xc0663f61 in ithread_execute_handlers (p=0xc339f430, ie=0xc32bfa80) at /usr/src/sys/kern/kern_intr.c:682 #17 0xc066406c in ithread_loop (arg=0xc33bd340) at /usr/src/sys/kern/kern_intr.c:765 #18 0xc0662ee8 in fork_exit (callout=0xc0664018 , arg=0xc33bd340, frame=0xd5693d38) at /usr/src/sys/kern/kern_fork.c:821 #19 0xc087dcec in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 (kgdb) About my ural adapter: stream# dmesg | grep ural ural0: ANI 802.11g WLAN Adapter, rev 2.00/0.01, addr 2 ural0: MAC/BBP RT2570 (rev 0x03), RF RT2526 ural0: Ethernet address: 00:11:95:da:fe:e6 ural0: if_start running deferred for Giant ural0: link state changed to UP stream# usbdevs -a 2 -f /dev/usb0 -v Controller /dev/usb0: addr 2: full speed, power 300 mA, config 1, 802.11g WLAN Adapter(0x3c00), ANI(0x2001), rev 0.01 Let me know if there is anything more I can do to help debug the problem. Regards, -- Anders. From: Jonathan Fosburgh To: Anders Nordby Cc: bug-followup@freebsd.org, Sam Leffler Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.2-RELEASE Date: Sun, 11 Feb 2007 17:36:59 -0600 On Sunday 11 February 2007 13:36, Anders Nordby wrote: > Hi, > > Just been trying the same thing in 6.2-RELEASE, with the same hardware. > > It seems the same bug still occurs, the stack trace looks very similar. > Everytime I try do do a cvsup of the ports tree, the system panics. > For other reasons I have upgraded to -CURRENT and the problem continues for me. Unfortunately I just don't have time for proper PD right now. I will be interested in seeing if the new USB stack behaves any better once it matures. From: Jonathan Fosburgh To: Anders Nordby Cc: bug-followup@freebsd.org, Sam Leffler Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.2-RELEASE Date: Sun, 11 Feb 2007 19:43:00 -0600 On Sunday 11 February 2007 13:36, Anders Nordby wrote: > Hi, > > Just been trying the same thing in 6.2-RELEASE, with the same hardware. > > It seems the same bug still occurs, the stack trace looks very similar. > Everytime I try do do a cvsup of the ports tree, the system panics. > > stream# uname -a > FreeBSD stream.localnet 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Tue Feb 6 > 22:09:01 CET 2007 root@stream.localnet:/usr/obj/usr/src/sys/STREAM > i386 Actually I recently discovered that it is not the use of WPA itself that causes the panic, it is the wpa_suplicant. Using that for WEP also caused a panic on this NIC. From: Sam Leffler To: Jonathan Fosburgh Cc: Anders Nordby , bug-followup@freebsd.org Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.2-RELEASE Date: Mon, 12 Feb 2007 09:27:39 -0800 Jonathan Fosburgh wrote: > On Sunday 11 February 2007 13:36, Anders Nordby wrote: >> Hi, >> >> Just been trying the same thing in 6.2-RELEASE, with the same hardware. >> >> It seems the same bug still occurs, the stack trace looks very similar. >> Everytime I try do do a cvsup of the ports tree, the system panics. >> >> stream# uname -a >> FreeBSD stream.localnet 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Tue Feb 6 >> 22:09:01 CET 2007 root@stream.localnet:/usr/obj/usr/src/sys/STREAM >> i386 > > Actually I recently discovered that it is not the use of WPA itself that > causes the panic, it is the wpa_suplicant. Using that for WEP also caused a > panic on this NIC. > > The last I heard about any of this stuff your problems were related to usb xfer stalls. If this no longer true then please provide me with a recipe for recreating the issue. If it's a driver/net80211 issue I will try to fix it. If it's in the usb subsystem it's unlikely I'm going to pursue it. FWIW the "new usb stack" is unlikely to help you. The author broke ural in porting it and so far as I know it's not been fixed. This is independent of whether or not any changes in his code base will resolve the original issue. Sam Responsible-Changed-From-To: freebsd-bugs->freebsd-usb Responsible-Changed-By: anders Responsible-Changed-When: Mon Feb 12 20:03:57 UTC 2007 Responsible-Changed-Why: Seems to be a USB problem. http://www.freebsd.org/cgi/query-pr.cgi?pr=92083 From: Jonathan Fosburgh To: Sam Leffler Cc: Anders Nordby , bug-followup@freebsd.org Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.2-RELEASE Date: Sun, 4 Mar 2007 13:32:01 -0600 On Monday 12 February 2007 11:27, Sam Leffler wrote: > Jonathan Fosburgh wrote: > > The last I heard about any of this stuff your problems were related to > usb xfer stalls. If this no longer true then please provide me with a > recipe for recreating the issue. If it's a driver/net80211 issue I will > try to fix it. If it's in the usb subsystem it's unlikely I'm going to > pursue it. > I'm at a loss here. As far as i can tell, my kernel is compiled with=20 debugging symbols, etc, but when I try to analyze the core dumps I get I ge= t=20 mostly unusable garbage. For instance, on a very recent panic: `--# kgdb /usr/obj/usr/src/sys/vmbsd/kernel.debug /usr/crash/vmcore.6 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:= =20 Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd". Unread portion of the kernel message buffer: =46atal trap 12: page fault while in kernel mode fault virtual address =3D 0x8 fault code =3D supervisor read data, page not present instruction pointer =3D 0x8:0xffffffff806e34be stack pointer =3D 0x10:0xffffffff92122b10 frame pointer =3D 0x10:0xffffffff92122b40 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 32 (irq21: uhci0 uhci*) trap number =3D 12 panic: page fault Uptime: 2h27m41s Physical memory: 504 MB Dumping 135 MB: 120 104 88 72 56 40 24 8 #0 doadump () at pcpu.h:172 172 pcpu.h: No such file or directory. in pcpu.h Running nm on the kernel and grepping for the instruction pointer I get=20 nothing useable, I wind up with a very large number of symbols. When I run= =20 where in the panic I get things like this: 0 doadump () at pcpu.h:172 #1 0x0000000000000004 in ?? () #2 0xffffffff80234dd9 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:411 #3 0xffffffff802353be in panic (fmt=3D0xffffff001ebf12b0 "\200s=B7\036") at /usr/src/sys/kern/kern_shutdown.c:567 #4 0xffffffff80356a82 in trap_fatal (frame=3D0xffffffff92122a60, eva=3D8) at /usr/src/sys/amd64/amd64/trap.c:696 #5 0xffffffff80356df2 in trap_pfault (frame=3D0xffffffff92122a60, usermode= =3D0) at /usr/src/sys/amd64/amd64/trap.c:614 #6 0xffffffff80357085 in trap (frame=3D0xffffffff92122a60) at /usr/src/sys/amd64/amd64/trap.c:382 #7 0xffffffff80341b3e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 #8 0xffffffff806e34be in ?? () #9 0xffffffff80a40ec0 in ?? () #10 0x0000000000000000 in ?? () #11 0xffffff000000d800 in ?? () #12 0xffffffff80a40000 in ?? () #13 0xffffffff92122b70 in ?? () =2D---------- #126 0x0000000000000000 in ?? () #127 0x0000000000000000 in ?? () #128 0x0000000000000000 in ?? () #129 0x0000000000000000 in ?? () #130 0x0000000000000000 in ?? () #131 0x0000000000000000 in ?? () #132 0x0000000000000000 in ?? () Cannot access memory at address 0xffffffff92123000 There are intermittant locations that give usable info, but not very much. = Is=20 there something else I need to do? From: Jonathan Fosburgh To: Sam Leffler Cc: Anders Nordby , bug-followup@freebsd.org Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.2-RELEASE Date: Tue, 6 Mar 2007 14:47:15 -0600 On Monday 12 February 2007 11:27, Sam Leffler wrote: > > The last I heard about any of this stuff your problems were related to > usb xfer stalls. If this no longer true then please provide me with a > recipe for recreating the issue. If it's a driver/net80211 issue I will > try to fix it. If it's in the usb subsystem it's unlikely I'm going to > pursue it. > I finally obtained what may be a useful kernel panic. I recompiled with the wlan/ural stuff in-kernel versus as modules (can someone put together, in one place, how to debug a kernel with modules? There is documentation in a few places, but it is geared to developers, and not end-users. That is fine for -CURRENT, but the issue is the same on -STABLE and the mainline releases.) So far I have captured one dump with this configuration. I will see if it crashes anymore throughout the day before I switch over to a working configuration. To reiterate: ural0: on uh ub7 ural0: MAC/BBP RT2570 (rev 0x05), RF RT2526 ural0: using obsoleted if_watchdog interface ural0: Ethernet address: 00:d0:41:a1:09:78 ural0: if_start running deferred for Giant and FreeBSD asgard.fosburgh.org 7.0-CURRENT FreeBSD 7.0-CURRENT #34: Tue Mar 6 08:07:37 CST 2007 toor@asgard.fosburgh.org:/usr/obj/usr/src/sys/vmbsd amd64 When configuring the NIC using wep in ifconfig, it is stable. When using wpa_supplicant (even in WEP-mode) the driver is unstable and panics the system. It does not appear to be under any particular load condition. I often find the system has rebooted while I have been away and there is no particular network load above background that I am aware of (emails being received, etc). Here is the panic: --# kgdb kernel.debug /usr/crash/vmcore.9 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0x8 fault code = supervisor read data, page not present instruction pointer = 0x8:0xffffffff802ee415 stack pointer = 0x10:0xffffffff9212ba70 frame pointer = 0x10:0x0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 33 (irq21: uhci0 uhci*) trap number = 12 panic: page fault Uptime: 5h13m37s Physical memory: 504 MB Dumping 78 MB: 63 47 31 15 #0 doadump () at pcpu.h:141 141 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); where: #0 doadump () at pcpu.h:141 #1 0x0000000000000004 in ?? () #2 0xffffffff80243519 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #3 0xffffffff80243afe in panic (fmt=0xffffff001ebf1560 "") at /usr/src/sys/kern/kern_shutdown.c:563 #4 0xffffffff8037e022 in trap_fatal (frame=0xffffffff9212b9c0, eva=8) at /usr/src/sys/amd64/amd64/trap.c:696 #5 0xffffffff8037e392 in trap_pfault (frame=0xffffffff9212b9c0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:614 #6 0xffffffff8037e625 in trap (frame=0xffffffff9212b9c0) at /usr/src/sys/amd64/amd64/trap.c:382 #7 0xffffffff80368fae in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 #8 0xffffffff802ee415 in ieee80211_free_node (ni=0x0) at /usr/src/sys/net80211/ieee80211_node.c:1602 #9 0xffffffff801cb131 in ural_txeof (xfer=0x0, priv=0xffffffff80a40ec0, status=USBD_NORMAL_COMPLETION) at /usr/src/sys/dev/usb/if_ural.c:890 #10 0xffffffff801e1ca3 in usb_transfer_complete (xfer=0xffffff000099d000) at /usr/src/sys/dev/usb/usbdi.c:983 #11 0xffffffff801c46db in ehci_softintr (v=0x0) at /usr/src/sys/dev/usb/ehci.c:872 #12 0xffffffff801c30d9 in ehci_intr1 (sc=0xffffff000094c000) at /usr/src/sys/dev/usb/ehci.c:591 #13 0xffffffff8022e28d in ithread_loop (arg=0xffffff0000945780) at /usr/src/sys/kern/kern_intr.c:682 #14 0xffffffff8022cd79 in fork_exit ( callout=0xffffffff8022e150 , arg=0xffffff0000945780, frame=0xffffffff9212bc90) at /usr/src/sys/kern/kern_fork.c:814 #15 0xffffffff8036931e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:397 #16 0x0000000000000000 in ?? () #17 0x0000000000000000 in ?? () #18 0x0000000000000001 in ?? () #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000800000 in ?? () #41 0xffffff001ebf1560 in ?? () #42 0x0000000000000000 in ?? () #43 0x0000000000000001 in ?? () #44 0x0000000000000000 in ?? () #45 0xffffff0016929810 in ?? () #46 0xffffffff9212bbc8 in ?? () #47 0xffffff001ebf1560 in ?? () #48 0xffffffff8025faf0 in sched_switch (td=0xffffff0000945780, newtd=0x0, flags=0) at /usr/src/sys/kern/sched_ule.c:1472 #49 0x0000000000000000 in ?? () #50 0x0000000000000000 in ?? () #51 0x0000000000000000 in ?? () #52 0x0000000000000000 in ?? () #53 0x0000000000000000 in ?? () #54 0x0000000000000000 in ?? () #55 0x0000000000000000 in ?? () #56 0x0000000000000000 in ?? () #57 0x0000000000000000 in ?? () #58 0x0000000000000000 in ?? () #59 0x0000000000000000 in ?? () #60 0x0000000000000000 in ?? () #61 0x0000000000000000 in ?? () #62 0x0000000000000000 in ?? () #63 0x0000000000000000 in ?? () #64 0x0000000000000000 in ?? () #65 0x0000000000000000 in ?? () #66 0x0000000000000000 in ?? () #67 0x0000000000000000 in ?? () #68 0x0000000000000000 in ?? () #69 0x0000000000000000 in ?? () #70 0x0000000000000000 in ?? () #71 0x0000000000000000 in ?? () #72 0x0000000000000000 in ?? () #73 0x0000000000000000 in ?? () #74 0x0000000000000000 in ?? () #75 0x0000000000000000 in ?? () #76 0x0000000000000000 in ?? () #77 0x0000000000000000 in ?? () #78 0x0000000000000000 in ?? () #79 0x0000000000000000 in ?? () #80 0x0000000000000000 in ?? () #81 0x0000000000000000 in ?? () #82 0x0000000000000000 in ?? () #83 0x0000000000000000 in ?? () #84 0x0000000000000000 in ?? () #85 0x0000000000000000 in ?? () #86 0x0000000000000000 in ?? () #87 0x0000000000000000 in ?? () #88 0x0000000000000000 in ?? () #89 0x0000000000000000 in ?? () #90 0x0000000000000000 in ?? () #91 0x0000000000000000 in ?? () #92 0x0000000000000000 in ?? () #93 0x0000000000000000 in ?? () #94 0x0000000000000000 in ?? () #95 0x0000000000000000 in ?? () #96 0x0000000000000000 in ?? () #97 0x0000000000000000 in ?? () #98 0x0000000000000000 in ?? () #99 0x0000000000000000 in ?? () #100 0x0000000000000000 in ?? () #101 0x0000000000000000 in ?? () #102 0x0000000000000000 in ?? () #103 0x0000000000000000 in ?? () #104 0x0000000000000000 in ?? () #105 0x0000000000000000 in ?? () #106 0x0000000000000000 in ?? () #107 0x0000000000000000 in ?? () #108 0x0000000000000000 in ?? () #109 0x0000000000000000 in ?? () #110 0x0000000000000000 in ?? () #111 0x0000000000000000 in ?? () #112 0x0000000000000000 in ?? () #113 0x0000000000000000 in ?? () #114 0x0000000000000000 in ?? () #115 0x0000000000000000 in ?? () #116 0x0000000000000000 in ?? () #117 0x0000000000000000 in ?? () #118 0x0000000000000000 in ?? () #119 0x0000000000000000 in ?? () #120 0x0000000000000000 in ?? () #121 0x0000000000000000 in ?? () #122 0x0000000000000000 in ?? () Cannot access memory at address 0xffffffff9212c000 The instruction pointer matches to: #8 0xffffffff802ee415 in ieee80211_free_node (ni=0x0) at /usr/src/sys/net80211/ieee80211_node.c:1602 Line 1602 is an open brace. Here is the section of the file, starting at line 1597: 1597 #ifdef IEEE80211_DEBUG_REFCNT 1598 ieee80211_free_node_debug(struct ieee80211_node *ni, const char *func, i nt line) 1599 #else 1600 ieee80211_free_node(struct ieee80211_node *ni) 1601 #endif 1602 { 1603 struct ieee80211_node_table *nt = ni->ni_table; 1604 1605 #ifdef IEEE80211_DEBUG_REFCNT 1606 IEEE80211_DPRINTF(ni->ni_ic, IEEE80211_MSG_NODE, 1607 "%s (%s:%u) %p<%s> refcnt %d\n", __func__, func, line, n i, 1608 ether_sprintf(ni->ni_macaddr), ieee80211_node_refcnt(ni )-1); 1609 #endif 1610 if (nt != NULL) { 1611 IEEE80211_NODE_LOCK(nt); 1612 if (ieee80211_node_dectestref(ni)) { 1613 /* 1614 * Last reference, reclaim state. 1615 */ 1616 _ieee80211_free_node(ni); 1617 } else if (ieee80211_node_refcnt(ni) == 1 && 1618 nt->nt_keyixmap != NULL) { 1619 ieee80211_keyix keyix; 1620 /* 1621 * Check for a last reference in the key mapping table. 1622 */ 1623 keyix = ni->ni_ucastkey.wk_rxkeyix; 1624 if (keyix < nt->nt_keyixmax && 1625 nt->nt_keyixmap[keyix] == ni) { 1626 IEEE80211_DPRINTF(ni->ni_ic, IEEE80211_M SG_NODE, 1627 "%s: %p<%s> clear key map entry", __func__, 1628 ni, ether_sprintf(ni->ni_macaddr)); 1629 nt->nt_keyixmap[keyix] = NULL; 1630 ieee80211_node_decref(ni); /* XXX needed ? */ 1631 _ieee80211_free_node(ni); 1632 } 1633 } 1634 IEEE80211_NODE_UNLOCK(nt); 1635 } else { 1636 if (ieee80211_node_dectestref(ni)) 1637 _ieee80211_free_node(ni); 1638 } 1639 } From: Jonathan Fosburgh To: Anders Nordby Cc: bug-followup@freebsd.org, Sam Leffler Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.2-RELEASE Date: Sat, 10 Mar 2007 18:54:51 -0600 On Sunday 11 February 2007 13:36:39 Anders Nordby wrote: > Hi, > > Just been trying the same thing in 6.2-RELEASE, with the same hardware. > > It seems the same bug still occurs, the stack trace looks very similar. > Everytime I try do do a cvsup of the ports tree, the system panics. > > stream# uname -a > FreeBSD stream.localnet 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Tue Feb 6 > 22:09:01 CET 2007 root@stream.localnet:/usr/obj/usr/src/sys/STREAM > i386 Check out usb/107642. That PR was created for what appears to be the same hardware you originally reported in this PR. Could it be we actually need the if_rum driver instead of ural? Does your link light work on your NIC? It does not work on mine in the FreeBSD driver (ural) but it does under Linux. Granted, maybe this driver does not light the link light, but I have no other ralink adapters with which to compare. From: Jonathan Fosburgh To: Anders Nordby Cc: bug-followup@freebsd.org, Sam Leffler Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.2-RELEASE Date: Mon, 28 May 2007 18:52:05 -0500 I'm being cautiously optimistic here. Late last week I switched over to the new USB stack amd started running my ural NIC with wpa_supplicant (still in WEP mode). So far, no new panics. I will run a few more days like this and then try switching my network to WPA. From: Jonathan Fosburgh To: Anders Nordby Cc: bug-followup@freebsd.org, Sam Leffler Subject: Re: kern/92083: [ural] [panic] panic using WPA on ural NIC in 6.2-RELEASE Date: Mon, 11 Jun 2007 01:42:49 -0500 On Sunday 11 February 2007 13:36:39 Anders Nordby wrote: > Hi, > > Just been trying the same thing in 6.2-RELEASE, with the same hardware. > > It seems the same bug still occurs, the stack trace looks very similar. > Everytime I try do do a cvsup of the ports tree, the system panics. > It does appear that ural + WPA is stable on the new USB stack. I have been running for 1-2 weeks with WPA2 security on my router and no panics due to ural+WPA. However, on the standard USB stack the panic still occurs. From: "Phillip Mumford" To: bug-followup@FreeBSD.org, anders@FreeBSD.org Cc: Subject: Re: usb/92083: [ural] [panic] panic using WPA on ural NIC in 6.0-RELEASE Date: Sun, 18 Nov 2007 14:16:35 +0000 Any fix or workaround available for this? Or any way of debugging to help get this solved? 7.0-BETA2 still has problems with ural and WPA2. I was a little disappointed when claimed that ural was high quality and WPA support was stable, but this still causes my laptop to crash and reboot regularly. From: Alexey Popov To: bug-followup@FreeBSD.org, anders@FreeBSD.org Cc: "Phillip Mumford" Subject: Re: usb/92083: [ural] [panic] panic using WPA on ural NIC in 6.0-RELEASE Date: Sat, 05 Jan 2008 19:03:43 +0300 Hi. Phillip, could you try workaround for if_ural that is similar to if_rum described at the last part of http://www.freebsd.org/cgi/query-pr.cgi?pr=117820 ? In /sys/dev/usb/if_ural.c ural_txeof() just replace the strings: ieee80211_free_node(data->ni); data->ni = NULL; with: if (data->ni != NULL) { ieee80211_free_node(data->ni); data->ni = NULL; } Then rebuild and reinstall kernel (or module, if you use it as a module). It works for me with if_rum and I think the problem here is possibly identical because the backtraces are very similar. With best regards, Alexey Popov >Unformatted: