From kostik@kib.kiev.ua Thu May 22 10:27:01 2003 Return-Path: Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A0B0D37B401 for ; Thu, 22 May 2003 10:27:01 -0700 (PDT) Received: from smtp3.lucky.net (smtp3.lucky.net [62.244.55.204]) by mx1.FreeBSD.org (Postfix) with ESMTP id D75C743F75 for ; Thu, 22 May 2003 10:26:58 -0700 (PDT) (envelope-from kostik@kib.kiev.ua) Received: from smtp.lucky.net (smtp1.lucky.net [193.193.193.117]) by smtp3.lucky.net (postfix-1.1.5) with ESMTP id 24BD08990A for ; Thu, 22 May 2003 20:26:56 +0300 (EEST) Received: from kozlik.carrier.kiev.ua (smmsp@kozlik.carrier.kiev.ua [193.193.193.111]) by smtp.lucky.net with ESMTP id h5MHQp2D069430 for ; Thu, 22 May 2003 20:26:54 +0300 (EEST) (envelope-from kostik@kib.kiev.ua) Received: (from uucp@localhost) by kozlik.carrier.kiev.ua with UUCP id h5MHQi04034555 for FreeBSD-gnats-submit@freebsd.org; Thu, 22 May 2003 20:26:44 +0300 (EEST) (envelope-from kostik@kib.kiev.ua) Received: from kib.UUCP (uucp@localhost) by kozlik.carrier.kiev.ua (rmail mypid=34554 childpid=34555) with UUCP; Thu, 22 May 2003 17:26:44 +0000 GMT Received: from little.home (kostik@localhost [127.0.0.1]) by little.home (8.12.8p1/8.12.8) with ESMTP id h4MHNPTR000432 for ; Thu, 22 May 2003 20:23:25 +0300 (EEST) (envelope-from kostik@little.home) Received: (from kostik@localhost) by little.home (8.12.8p1/8.12.8/Submit) id h4MHNPiP000431; Thu, 22 May 2003 20:23:25 +0300 (EEST) (envelope-from kostik) Message-Id: <200305221723.h4MHNPiP000431@little.home> Date: Thu, 22 May 2003 20:23:25 +0300 (EEST) From: "Kostik I. Belousov" Reply-To: "Kostik I. Belousov" To: FreeBSD-gnats-submit@freebsd.org Cc: Subject: Kernel panic with ipfw2 and syncookies X-Send-Pr-Version: 3.113 X-GNATS-Notify: >Number: 52585 >Category: kern >Synopsis: [netinet] [patch] Kernel panic with ipfw2 and syncookies >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-net >State: closed >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu May 22 10:30:03 PDT 2003 >Closed-Date: Sun Nov 12 09:40:16 GMT 2006 >Last-Modified: Sun Nov 12 09:40:16 GMT 2006 >Originator: Kostik I. Belousov >Release: FreeBSD 4.8-RELEASE i386 >Organization: none >Environment: System: FreeBSD little.home 4.8-RELEASE FreeBSD 4.8-RELEASE #1: Fri May 2 18:08:25 EEST 2003 root@little.home:/usr/obj/usr/src/sys/LITTLE i386 sysctl hw.machine: i386 hw.model: Pentium II/Pentium II Xeon/Celeron hw.ncpu: 2 hw.byteorder: 1234 hw.physmem: 533917696 hw.usermem: 491724800 hw.pagesize: 4096 hw.floatingpoint: 1 hw.machine_arch: i386 hw.ata.ata_dma: 1 hw.ata.wc: 1 hw.ata.tags: 0 hw.ata.atapi_dma: 0 hw.instruction_sse: 0 hw.availpages: 130185 net.inet.tcp.syncookies: 1 net.inet.tcp.syncache.bucketlimit: 30 net.inet.tcp.syncache.cachelimit: 15359 net.inet.tcp.syncache.count: 0 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.rexmtlimit: 3 Kernel compiled with ipfw2. >Description: By adding/removing aliases and manipulating ipfw rules (I caused the panic using ipfw fwd, see below), the kernel could be paniced. The trace: (kgdb) bt #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 #1 0xc0158847 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316 #2 0xc0158cb9 in panic (fmt=0xc0291b19 "%s") at /usr/src/sys/kern/kern_shutdown.c:595 #3 0xc024b459 in trap_fatal (frame=0xd3933cc0, eva=8) at /usr/src/sys/i386/i386/trap.c:974 #4 0xc024b0c5 in trap_pfault (frame=0xd3933cc0, usermode=0, eva=8) at /usr/src/sys/i386/i386/trap.c:867 #5 0xc024ac1f in trap (frame={tf_fs = 1644167192, tf_es = -1072234480, tf_ds = -745340912, tf_edi = 1644167168, tf_esi = -1054094552, tf_ebp = -745325300, tf_isp = -745325332, tf_ebx = -761024704, tf_edx = -1070824920, tf_ecx = 0, tf_eax = -1, tf_trapno = 12, tf_err = 0, tf_eip = -1071929334, tf_cs = 8, tf_eflags = 66198, tf_esp = -761024704, tf_ss = -1050054796}) at /usr/src/sys/i386/i386/trap.c:466 #6 0xc01ba80a in syncache_insert (sc=0xd2a3af40, sch=0xc12bcb28) at /usr/src/sys/netinet/tcp_syncache.c:302 #7 0xc01bb67c in syncache_add (inc=0xd3933db4, to=0xd3933e20, th=0xc1051950, sop=0xd3933db0, m=0xc1051900) at /usr/src/sys/netinet/tcp_syncache.c:1021 #8 0xc01b5809 in tcp_input (m=0xc1051900, off0=20, proto=6) at /usr/src/sys/netinet/tcp_input.c:826 #9 0xc01b026c in ip_input (m=0xc1051900) at /usr/src/sys/netinet/ip_input.c:927 #10 0xc01b02cb in ipintr () at /usr/src/sys/netinet/ip_input.c:948 #11 0xc023c051 in swi_net_next () #12 0xc017a835 in connect (p=0xd3876be0, uap=0xd3933f80) at /usr/src/sys/kern/uipc_syscalls.c:394 #13 0xc024b795 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 135174508, tf_esi = -1077953148, tf_ebp = -1077953136, tf_isp = -745324588, tf_ebx = 0, tf_edx = 134570369, tf_ecx = 135112576, tf_eax = 98, tf_trapno = 22, tf_err = 2, tf_eip = 673579312, tf_cs = 31, tf_eflags = 659, tf_esp = -1077953564, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175 #14 0xc0237f5b in Xint0x80_syscall () #15 0x805b4b5 in ?? () #16 0x8059c0e in ?? () #17 0x805985f in ?? () #18 0x806e639 in ?? () #19 0x804c03a in ?? () (kgdb) frame 6 #6 0xc01ba80a in syncache_insert (sc=0xd2a3af40, sch=0xc12bcb28) at /usr/src/sys/netinet/tcp_syncache.c:302 302 if (sc2 != NULL) (kgdb) list 297 * first non-empty timer queue with the largest 298 * timeout value. 299 */ 300 for (i = SYNCACHE_MAXREXMTS; i >= 0; i--) { 301 sc2 = TAILQ_FIRST(&tcp_syncache.timerq[i]); 302 if (sc2 != NULL) 303 break; 304 } 305 sc2->sc_tp->ts_recent = ticks; 306 syncache_drop(sc2, NULL); >How-To-Repeat: ifconfig lo0 192.168.2.1 alias ipfw 50 add fwd 192.168.2.1,23 tcp from any to 192.168.2.1 some time ... ipfw del 50 ifconfig lo0 192.168.2.1 remove some more time ... attempt to make tcp connection to the machine panics the kernel >Fix: >Release-Note: >Audit-Trail: Responsible-Changed-From-To: freebsd-bugs->luigi Responsible-Changed-By: kris Responsible-Changed-When: Mon Jul 14 03:07:48 PDT 2003 Responsible-Changed-Why: Assign to ipfw maintainer http://www.freebsd.org/cgi/query-pr.cgi?pr=52585 From: Don Bowman To: "'freebsd-gnats-submit@FreeBSD.org'" , "'tessart.kiev.ua'" Cc: Subject: Re: kern/52585: Kernel panic with ipfw2 and syncookies Date: Tue, 29 Jul 2003 20:12:00 -0400 I believe i have a fix for this. The problem is not in ipfw per se, but in syncache which doesn't expect to 're-enter' itself. when ipfw is in use, ip_output() may call ip_input(). http://www.mail-archive.com/freebsd-net@freebsd.org/msg08996.html has the trail. the below fix may not be great since it defers deletion. It also may be better to run the deferred deletion @ the beginning of the timer routine. Index: tcp_syncache.c =================================================================== RCS file: /usr/cvs/src/sys/netinet/tcp_syncache.c,v retrieving revision 1.5.2.8.1000.3 diff -U5 -r1.5.2.8.1000.3 tcp_syncache.c --- tcp_syncache.c 4 Feb 2003 01:52:03 -0000 1.5.2.8.1000.3 +++ tcp_syncache.c 1 Jul 2003 14:32:29 -0000 @@ -83,16 +83,18 @@ #endif /*IPSEC*/ #include #include +static int syncache_delete_flag; static int tcp_syncookies = 1; SYSCTL_INT(_net_inet_tcp, OID_AUTO, syncookies, CTLFLAG_RW, &tcp_syncookies, 0, "Use TCP SYN cookies if the syncache overflows"); static void syncache_drop(struct syncache *, struct syncache_head *); +static void syncache_delete(struct syncache *, struct syncache_head *); static void syncache_free(struct syncache *); static void syncache_insert(struct syncache *, struct syncache_head *); struct syncache *syncache_lookup(struct in_conninfo *, struct syncache_head **); static int syncache_respond(struct syncache *, struct mbuf *); static struct socket *syncache_socket(struct syncache *, struct socket *); @@ -125,10 +127,11 @@ u_int next_reseed; TAILQ_HEAD(, syncache) timerq[SYNCACHE_MAXREXMTS + 1]; struct callout tt_timerq[SYNCACHE_MAXREXMTS + 1]; }; static struct tcp_syncache tcp_syncache; +static TAILQ_HEAD(syncache_delete_list, syncache) sc_delete_list; SYSCTL_NODE(_net_inet_tcp, OID_AUTO, syncache, CTLFLAG_RW, 0, "TCP SYN cache"); SYSCTL_INT(_net_inet_tcp_syncache, OID_AUTO, bucketlimit, CTLFLAG_RD, &tcp_syncache.bucket_limit, 0, "Per-bucket hash limit for syncache"); @@ -202,10 +205,13 @@ rtrequest(RTM_DELETE, rt_key(rt), rt->rt_gateway, rt_mask(rt), rt->rt_flags, NULL); RTFREE(rt); } +#if defined(DIAGNOSTIC) + memset(sc, 0xee, sizeof(struct syncache)); +#endif zfree(tcp_syncache.zone, sc); } void syncache_init(void) @@ -256,10 +262,12 @@ * older one. */ tcp_syncache.cache_limit -= 1; tcp_syncache.zone = zinit("syncache", sizeof(struct syncache), tcp_syncache.cache_limit, ZONE_INTERRUPT, 0); + + TAILQ_INIT(&sc_delete_list); } static void syncache_insert(sc, sch) struct syncache *sc; @@ -312,12 +320,28 @@ static void syncache_drop(sc, sch) struct syncache *sc; struct syncache_head *sch; { + if ((sc->sc_flags & SCF_DELETE) == 0) { + sc->sc_flags |= SCF_DELETE; + syncache_delete_flag = 1; + TAILQ_INSERT_TAIL(&sc_delete_list, sc, sc_delete); + } +} + +static void +syncache_delete(sc, sch) + struct syncache *sc; + struct syncache_head *sch; +{ int s; + if ((sc->sc_flags & SCF_DELETE) == 0) { + printf("ERROR ERROR ERROR: SCF_DELETE == 0\n"); + return; + } if (sch == NULL) { #ifdef INET6 if (sc->sc_inc.inc_isipv6) { sch = &tcp_syncache.hashbase[ SYNCACHE_HASH6(&sc->sc_inc, tcp_syncache.hashmask)]; @@ -329,10 +353,12 @@ } } s = splnet(); + TAILQ_REMOVE(&sc_delete_list, sc, sc_delete); + TAILQ_REMOVE(&sch->sch_bucket, sc, sc_hash); sch->sch_length--; tcp_syncache.cache_count--; TAILQ_REMOVE(&tcp_syncache.timerq[sc->sc_rxtslot], sc, sc_timerq); @@ -357,10 +383,12 @@ int s; s = splnet(); if (callout_pending(&tcp_syncache.tt_timerq[slot]) || !callout_active(&tcp_syncache.tt_timerq[slot])) { + if (syncache_delete_flag) + goto delete_cleanup; splx(s); return; } callout_deactivate(&tcp_syncache.tt_timerq[slot]); @@ -390,10 +418,21 @@ SYNCACHE_TIMEOUT(sc, slot + 1); } if (nsc != NULL) callout_reset(&tcp_syncache.tt_timerq[slot], nsc->sc_rxttime - ticks, syncache_timer, (void *)(slot)); + +delete_cleanup: + sc = TAILQ_FIRST(&sc_delete_list); + while (sc != NULL) { + nsc = TAILQ_NEXT(sc, sc_delete); + syncache_delete(sc, NULL); + sc = nsc; + } + TAILQ_INIT(&sc_delete_list); + syncache_delete_flag = 0; + splx(s); } /* * Find an entry in the syncache. @@ -1333,10 +1372,11 @@ data = data >> SYNCOOKIE_WNDBITS; sc = zalloc(tcp_syncache.zone); if (sc == NULL) return (NULL); + bzero(sc, sizeof(*sc)); /* * Fill in the syncache values. * XXX duplicate code from syncache_add */ sc->sc_ipopts = NULL; Index: tcp_var.h =================================================================== RCS file: /usr/cvs/src/sys/netinet/tcp_var.h,v retrieving revision 1.56.2.12 diff -U5 -r1.56.2.12 tcp_var.h --- tcp_var.h 24 Aug 2002 18:40:26 -0000 1.56.2.12 +++ tcp_var.h 1 Jul 2003 02:33:57 -0000 @@ -222,12 +222,14 @@ #define SCF_WINSCALE 0x02 /* negotiated window scaling */ #define SCF_TIMESTAMP 0x04 /* negotiated timestamps */ #define SCF_CC 0x08 /* negotiated CC */ #define SCF_UNREACH 0x10 /* icmp unreachable received */ #define SCF_KEEPROUTE 0x20 /* keep cloned route */ +#define SCF_DELETE 0x40 /* I'm being deleted */ TAILQ_ENTRY(syncache) sc_hash; TAILQ_ENTRY(syncache) sc_timerq; + TAILQ_ENTRY(syncache) sc_delete; }; struct syncache_head { TAILQ_HEAD(, syncache) sch_bucket; u_int sch_length; Responsible-Changed-From-To: luigi->bms Responsible-Changed-By: bms Responsible-Changed-When: Wed Jun 23 01:49:31 GMT 2004 Responsible-Changed-Why: I'll try to look at this. I have a feeling this problem may go away with the impending ipfw/dummynet changes with regards to PFIL_HOOKS. http://www.freebsd.org/cgi/query-pr.cgi?pr=52585 From: Dmitry Pryanishnikov To: bug-followup@FreeBSD.org, kostya@tessart.kiev.ua Cc: Subject: Re: kern/52585: [netinet] [patch] Kernel panic with ipfw2 and syncookies Date: Mon, 20 Feb 2006 13:51:50 +0200 Hello! I've tried to recreate your panic case under 4.11-RELEASE-p14. I've noticed that your ipfw rule ipfw 50 add fwd 192.168.2.1,23 tcp from any to 192.168.2.1 creates cyclic redirect of TCP packets directed on this machine to 192.168.2.1. E.g., if I enable telnet in /etc/inetd.conf, add this rule and "telnet 192.168.2.1 http", rule will not only forward packets from telnet'c client to 80th port via 23th port, but will also redirect replies (directed in this case to 192.168.2.1:client_port) back to 192.168.2.1:23! This provokes system reboot (yes, w/o crash, just reset) in several seconds of this telnet session's activity. So I've changed this rule to ipfw 50 add fwd 192.168.2.1,23 tcp from any to 192.168.2.1 http and then "telnet 192.168.2.1 http" doesn't crash system anymore both under ipfw1 and ipfw2. Please reveal what activity should I do during "some time..." and "some more time..." in order to provoke panic. Sincerely, Dmitry State-Changed-From-To: open->suspended State-Changed-By: bms State-Changed-When: Fri Sep 22 17:04:22 UTC 2006 State-Changed-Why: Back to the free pool. By the looks of things, this problem may have gone away since 5.x, the reason being that the network stack was reshuffled for SMP (netisrs are now used). Responsible-Changed-From-To: bms->freebsd-net Responsible-Changed-By: bms Responsible-Changed-When: Fri Sep 22 17:04:22 UTC 2006 Responsible-Changed-Why: http://www.freebsd.org/cgi/query-pr.cgi?pr=52585 State-Changed-From-To: suspended->closed State-Changed-By: gnn State-Changed-When: Sun Nov 12 09:39:29 UTC 2006 State-Changed-Why: This can no longer be reproduced (7.0 CURRENT) http://www.freebsd.org/cgi/query-pr.cgi?pr=52585 >Unformatted: