From kris@obsecurity.org Sat Feb 11 06:12:25 2006 Return-Path: Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 94ADD16A420 for ; Sat, 11 Feb 2006 06:12:25 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id F09BD43D78 for ; Sat, 11 Feb 2006 06:12:19 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id AC4ED1A3C1B for ; Fri, 10 Feb 2006 22:12:19 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id D34635152B; Sat, 11 Feb 2006 01:12:18 -0500 (EST) Message-Id: <20060211061218.D34635152B@obsecurity.dyndns.org> Date: Sat, 11 Feb 2006 01:12:18 -0500 (EST) From: Kris Kennaway To: FreeBSD-gnats-submit@freebsd.org Cc: Subject: Changing system date causes panic in nd6_timer X-Send-Pr-Version: 3.113 X-GNATS-Notify: >Number: 93170 >Category: kern >Synopsis: Changing system date causes panic in nd6_timer >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-bugs >State: closed >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Feb 11 06:20:07 GMT 2006 >Closed-Date: Sat Mar 28 23:44:48 UTC 2009 >Last-Modified: Sat Mar 28 23:44:48 UTC 2009 >Originator: Kris Kennaway >Release: FreeBSD 7.0-CURRENT amd64 >Organization: >Environment: FreeBSD/amd64 >Description: I ran ntpdate on an amd64 system with ipv6 enabled and a skewed clock (ntpdate stepped it back by about an hour), and immediately got a use-after-free panic in ifaddr. When I rebooted with memguard enabled on this malloc type and retried, I got this panic upon changing the date forward, then back, then forward again (also note the garbage return data from ntpdate): # date 200606011200 Thu Jun 1 12:00:00 UTC 2006 # ntpdate ntp.apple.com 16 Jan 00:40:18 ntpdate[612]: step time server 17.254.0.28 offset -~9000pm6}9426375508.195959 sec # date 200606011200 Thu Jun 1 12:00:00 UTC 2006 Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xffffffff91bd2198 fault code = supervisor write, protection violation instruction pointer = 0x8:0xffffffff80321346 stack pointer = 0x10:0xffffffffbcfa1b60 frame pointer = 0x10:0xffffffffbcfa1b90 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 14 (swi4: clock sio) [thread pid 14 tid 100010 ] Stopped at nd6_timer+0x106: movl %eax,0x198(%rbx) db> wh Tracing pid 14 tid 100010 td 0xffffff03e15d6c30 nd6_timer() at nd6_timer+0x106 softclock() at softclock+0x279 ithread_execute_handlers() at ithread_execute_handlers+0x12f ithread_loop() at ithread_loop+0x99 fork_exit() at fork_exit+0xdf fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffbcfa1d40, rbp = 0 --- Unfortunately I can't dump on this system, but: (kgdb) list *(nd6_timer+0x106) 0xffffffff80321346 is in nd6_timer (../../../netinet6/nd6.c:585). 580 goto addrloop; /* XXX: see below */ 581 } 582 if (IFA6_IS_DEPRECATED(ia6)) { 583 int oldflags = ia6->ia6_flags; 584 585 ia6->ia6_flags |= IN6_IFF_DEPRECATED; 586 587 /* 588 * If a temporary address has just become deprecated, 589 * regenerate a new one if possible. >How-To-Repeat: Run the above two commands in a loop >Fix: >Release-Note: >Audit-Trail: From: David Malone To: bug-followup@FreeBSD.org, kris@FreeBSD.org Cc: rwatson@FreeBSD.org, iedowse@maths.tcd.ie, dwmalone@maths.tcd.ie Subject: Re: kern/93170 : Changing system date causes panic in nd6_timer Date: Sat, 11 Feb 2006 16:59:38 +0000 I'm fairly sure I know what's going on here, as I've seen a few more of these panics. There seems to be a bundle of IPv6 neighbour discovery code that uses splnet() and hasn't had any locking added to it yet. One of these bits of code is walking some global lists and is obviously getting into trouble. I believe this same code is also causing dirhash panics people have reported 'cos some of the memory it is using is getting reused by dirhash. I sent the message below to Robert a while ago - since I haven't heard back from him, I'm going to have a go at adding some locking. David. Subject: IPv6 address list locking. Date: Fri, 20 Jan 2006 17:48:57 +0000 From: David Malone Message-ID: <200601201748.aa69772@salmon.maths.tcd.ie> Hi Robert, I've been trying to trace some use-after-free problems that have paniced the kernel in dirhash. A few weeks ago I got a panic that indicated that the ifnet malloc pool might be the problem. Using memguard on my desktop in Dublin paniced today and, judging by the reports of the guy who reset it for me, it paniced at nd6_timer+0xe8. This looks like a bit of nd6_timer code that accesses a global list of IPv6 addresses stored in a variable called in6_ifaddr. Rather annoyingly this variable has the same name as a structure, so it is difficult to grep for. However, I don't see any locking of accesses to this variable. Do you have a scheme for locking globals like this in the IPv6 stack? Does it look like nd6_timeout is violaing this locking? David. in6.c: for (ia = in6_ifaddr; ia; ia = ia->ia_next) { in6.c: for (ia = in6_ifaddr; ia; ia = ia->ia_next) { in6.c: for (ia = in6_ifaddr; ia; ia = ia->ia_next) { in6_src.c: for (ia = in6_ifaddr; ia; ia = ia->ia_next) { nd6.c: for (ia6 = in6_ifaddr; ia6; ia6 = nia6) { nd6.c: for (ia = in6_ifaddr; ia; ia = ia_next) { nd6_rtr.c: for (ifa = in6_ifaddr; ifa; ifa = ifa->ia_next) { nd6_rtr.c: for (ifa = in6_ifaddr; ifa; ifa = ifa->ia_next) { nd6_rtr.c: for (ifa = in6_ifaddr; ifa; ifa = ifa->ia_next) { nd6_rtr.c: for (ia = in6_ifaddr; ia; ia = ia->ia_next) { State-Changed-From-To: open->feedback State-Changed-By: kmacy State-Changed-When: Tue Nov 20 06:59:13 UTC 2007 State-Changed-Why: Does this still happen? http://www.freebsd.org/cgi/query-pr.cgi?pr=93170 State-Changed-From-To: feedback->suspended State-Changed-By: linimon State-Changed-When: Sun Mar 2 02:40:46 UTC 2008 State-Changed-Why: Feedback was not received, but it does not sound as though this problem has been fixed. Mark as suspended since no one seems to be working on it. http://www.freebsd.org/cgi/query-pr.cgi?pr=93170 State-Changed-From-To: suspended->closed State-Changed-By: brucec State-Changed-When: Sat Mar 28 23:43:52 UTC 2009 State-Changed-Why: Fixed in rev 1.63 of src/sys/netinet6/nd6.c http://www.freebsd.org/cgi/query-pr.cgi?pr=93170 >Unformatted: