From root@mailhub.rachie.is-a-geek.net Fri Sep 4 09:57:16 2009 Return-Path: Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67FC31065790; Fri, 4 Sep 2009 09:57:16 +0000 (UTC) (envelope-from root@mailhub.rachie.is-a-geek.net) Received: from mailhub.rachie.is-a-geek.net (rachie.is-a-geek.net [66.230.99.27]) by mx1.freebsd.org (Postfix) with ESMTP id 0DBEB8FC19; Fri, 4 Sep 2009 09:57:15 +0000 (UTC) Received: by mailhub.rachie.is-a-geek.net (Postfix, from userid 0) id B61227E854; Fri, 4 Sep 2009 01:39:07 -0800 (AKDT) Message-Id: <20090904093907.B61227E854@mailhub.rachie.is-a-geek.net> Date: Fri, 4 Sep 2009 01:39:07 -0800 (AKDT) From: Mel Flynn To: FreeBSD-gnats-submit@freebsd.org Cc: freebsd-net@FreeBSD.org Subject: [panic] Kernel corruption of pppoe lists X-Send-Pr-Version: 3.113 X-GNATS-Notify: >Number: 138525 >Category: kern >Synopsis: [ppp] [panic] Kernel corruption of pppoe lists >Confidential: no >Severity: critical >Priority: low >Responsible: freebsd-bugs >State: closed >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Sep 04 10:00:17 UTC 2009 >Closed-Date: Wed Oct 12 15:09:45 UTC 2011 >Last-Modified: Wed Oct 12 15:09:45 UTC 2011 >Originator: Mel Flynn >Release: FreeBSD 7.2-STABLE i386 >Organization: >Environment: System: FreeBSD gate.rachie.is-a-geek.net 7.2-STABLE FreeBSD 7.2-STABLE #0: Sun Jun 28 00:01:59 AKDT 2009 mdev@squish.rachie.is-a-geek.net:/data/obj/data/RELENG_7/src/sys/GATE i386 >Description: I realize the kernel is a bit old, but it also very hard to reproduce. Kernel was up 56 days and this crash happened shortly after a very long connect time, hangup by ISP and some renegotiation issues. I can provide the ppp.log of the incident if needed. What bothers me is the contents of the session list element, preceding the element cannot be accessed. Clearly, there is random kernel memory present there, judging from ether_dhost and ether_shost. # kgdb /boot/kernel/kernel /var/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0x2d465459 fault code = supervisor read, page not present instruction pointer = 0x20:0xc06cd0a0 stack pointer = 0x28:0xc3b86a98 frame pointer = 0x28:0xc3b86ac0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 35 (irq22: xl0) trap number = 12 panic: page fault Uptime: 56d6h29m38s Physical memory: 1007 MB Dumping 174 MB: 159 143 127 111 95 79 63 47 31 15 Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols from /boot/kernel/geom_journal.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_journal.ko Reading symbols from /boot/kernel/wlan_xauth.ko...Reading symbols from /boot/kernel/wlan_xauth.ko.symbols...done. done. Loaded symbols for /boot/kernel/wlan_xauth.ko Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done. done. Loaded symbols for /boot/kernel/acpi.ko Reading symbols from /boot/kernel/if_bridge.ko...Reading symbols from /boot/kernel/if_bridge.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_bridge.ko Reading symbols from /boot/kernel/bridgestp.ko...Reading symbols from /boot/kernel/bridgestp.ko.symbols...done. done. Loaded symbols for /boot/kernel/bridgestp.ko Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ether.ko Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_socket.ko Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from /boot/kernel/daemon_saver.ko.symbols...done. done. Loaded symbols for /boot/kernel/daemon_saver.ko #0 doadump () at pcpu.h:196 196 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:196 #1 0xc05ef5d3 in boot (howto=260) at /data/RELENG_7/src/sys/kern/kern_shutdown.c:418 #2 0xc05ef7de in panic (fmt=Variable "fmt" is not available. ) at /data/RELENG_7/src/sys/kern/kern_shutdown.c:574 #3 0xc085c72c in trap_fatal (frame=0xc3b86a58, eva=759583833) at /data/RELENG_7/src/sys/i386/i386/trap.c:938 #4 0xc085c9b0 in trap_pfault (frame=0xc3b86a58, usermode=0, eva=759583833) at /data/RELENG_7/src/sys/i386/i386/trap.c:851 #5 0xc085d339 in trap (frame=0xc3b86a58) at /data/RELENG_7/src/sys/i386/i386/trap.c:529 #6 0xc0844a4b in calltrap () at /data/RELENG_7/src/sys/i386/i386/exception.s:166 #7 0xc06cd0a0 in pppoe_findsession (privp=0xc4258000, wh=Variable "wh" is not available. ) at /data/RELENG_7/src/sys/netgraph/ng_pppoe.c:567 #8 0xc06ce1a0 in ng_pppoe_rcvdata_ether (hook=0xc41b6380, item=0xc4256120) at /data/RELENG_7/src/sys/netgraph/ng_pppoe.c:1612 #9 0xc06c566f in ng_apply_item (node=0xc4111e80, item=0xc4256120, rw=0) at /data/RELENG_7/src/sys/netgraph/ng_base.c:2336 #10 0xc06c47e0 in ng_snd_item (item=0xc4256120, flags=Variable "flags" is not available. ) at /data/RELENG_7/src/sys/netgraph/ng_base.c:2254 #11 0xc068de5f in ether_demux (ifp=0xc3dbb400, m=0xc8024d00) at /data/RELENG_7/src/sys/net/if_ethersubr.c:851 #12 0xc068e1b3 in ether_input (ifp=0xc3dbb400, m=0xc8024d00) at /data/RELENG_7/src/sys/net/if_ethersubr.c:692 #13 0xc07b5348 in xl_rxeof (sc=0xc3dbc000) at /data/RELENG_7/src/sys/pci/if_xl.c:2022 #14 0xc07b7834 in xl_intr (arg=0xc3dbc000) at /data/RELENG_7/src/sys/pci/if_xl.c:2257 #15 0xc05cd10b in ithread_loop (arg=0xc3dc10c0) at /data/RELENG_7/src/sys/kern/kern_intr.c:1127 #16 0xc05c9ae6 in fork_exit (callout=0xc05ccf60 , arg=0xc3dc10c0, frame=0xc3b86d38) at /data/RELENG_7/src/sys/kern/kern_fork.c:811 #17 0xc0844ac0 in fork_trampoline () at /data/RELENG_7/src/sys/i386/i386/exception.s:271 (kgdb) frame 7 #7 0xc06cd0a0 in pppoe_findsession (privp=0xc4258000, wh=Variable "wh" is not available. ) at /data/RELENG_7/src/sys/netgraph/ng_pppoe.c:567 567 if (sp->Session_ID == session && (kgdb) print sp $1 = 0x2d465455 (kgdb) print *sp Cannot access memory at address 0x2d465455 (kgdb) print *privp $2 = {node = 0xc4111e80, ethernet_hook = 0xc41b6380, debug_hook = 0x0, packets_in = 126728356, packets_out = 69301432, flags = 0, eh = {ether_dhost = "ÿÿÿÿÿÿ", ether_shost = "\000\001\002Çù6", ether_type = 25480}, listeners = {lh_first = 0x0}, sesshash = {{mtx = {lock_object = {lo_name = 0xc08b8dc2 "PPPoE hash mutex", lo_type = 0xc08b8dc2 "PPPoE hash mutex", lo_flags = 16973824, lo_witness_data = { lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 3285460096, mtx_recurse = 0}, head = {lh_first = 0xc5093780}}, {mtx = {lock_object = { lo_name = 0xc08b8dc2 "PPPoE hash mutex", lo_type = 0xc08b8dc2 "PPPoE hash mutex", lo_flags = 16973824, lo_witness_data = { lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0}, head = {lh_first = 0x0}} , {mtx = { lock_object = {lo_name = 0xc08b8dc2 "PPPoE hash mutex", lo_type = 0xc08b8dc2 "PPPoE hash mutex", lo_flags = 16973824, lo_witness_data = { lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0}, head = {lh_first = 0xc4dc5bc0}}, {mtx = {lock_object = { lo_name = 0xc08b8dc2 "PPPoE hash mutex", lo_type = 0xc08b8dc2 "PPPoE hash mutex", lo_flags = 16973824, lo_witness_data = { lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0}, head = {lh_first = 0x0}} }} (kgdb) print *privp->sesshash.head.lh_first->sessions.le_next->sessions.le_next $4 = {hook = 0x1, Session_ID = 52, state = 1920169263, creator = 1852400175, pkt_hdr = { eh = {ether_dhost = "/zcat", ether_shost = "/usr/s", ether_type = 24936}, ph = { ver = 2 '\002', type = 7 '\a', code = 101 'e', sid = 27951, length = 28257, tag = 0xc4e68524}}, neg = 0x2e6e652f, sessions = {le_next = 0x2d465455, le_prev = 0x61632f38}} >How-To-Repeat: Unknown at this time. >Fix: I don't expect this to be fixed, without a reproduction scenario, so I'm mainly reporting this to see if others have experienced a similar crash. >Release-Note: >Audit-Trail: State-Changed-From-To: open->feedback State-Changed-By: eadler State-Changed-When: Sat Sep 24 03:53:04 UTC 2011 State-Changed-Why: Is this still an issue on recent versions of FreeBSD? http://www.freebsd.org/cgi/query-pr.cgi?pr=138525 State-Changed-From-To: feedback->closed State-Changed-By: remko State-Changed-When: Wed Oct 12 15:09:45 UTC 2011 State-Changed-Why: Feedback timeout http://www.freebsd.org/cgi/query-pr.cgi?pr=138525 >Unformatted: