From linimon@FreeBSD.org Mon Apr 16 19:18:08 2007 Return-Path: Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4B8F016A401 for ; Mon, 16 Apr 2007 19:18:08 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [69.147.83.40]) by mx1.freebsd.org (Postfix) with ESMTP id 39DA413C457 for ; Mon, 16 Apr 2007 19:18:08 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id l3GJI7u8084236 for ; Mon, 16 Apr 2007 19:18:07 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id l3GJI7g8084235; Mon, 16 Apr 2007 19:18:07 GMT (envelope-from linimon) Message-Id: <200704161918.l3GJI7g8084235@freefall.freebsd.org> Date: Mon, 16 Apr 2007 19:18:07 GMT From: Kai Storbeck To: FreeBSD-gnats-submit@freebsd.org Subject: FreeBSD 6.2-release (p3) panic in NFS client code/VFS subsystem >Number: 111734 >Category: kern >Synopsis: [nfs] FreeBSD 6.2-release (p3) panic in NFS client code/VFS subsystem >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: closed >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Apr 16 19:20:14 GMT 2007 >Closed-Date: Mon Jul 23 21:24:21 GMT 2007 >Last-Modified: Mon Jul 23 21:24:21 GMT 2007 >Originator: Kai Storbeck >Release: FreeBSD 6.2 stable (p3) >Organization: >Environment: FreeBSD webserver 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #2: Thu Apr 5 14:15:19 CEST 2007 kai@buildmachine:/usr/src/sys/i386/compile/SMP-DEBUG i386 >Description: We have 4 machines on freebsd 6.2 (dual 2ghz dualcore xeon). They are serving webpages (static content with a few default scripts) from an NFS server. They _all_ regularly panic since we've upgraded the machines to new hardware + freebsd 6.2 stable. This is the backtrace: [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: panic: bundirty: buffer 0xdc510d50 still on queue 1 cpuid = 0 Uptime: 8h14m42s Dumping 3326 MB (2 chunks) chunk 0: 1MB (156 pages) ... ok chunk 1: 3327MB (851552 pages) 3311 3295 3279 3263 3247 3231 3215 3199 3183 3167 3151 3135 3119 3103 3087 3071 3055 3039 3023 3007 2991 2975 2959 2943 2927 2911 2895 2879 2863 2847 2831 2815 2799 2783 2767 2751 2735 2719 2703 2687 2671 2655 2639 2623 2607 2591 2575 2559 2543 2527 2511 2495 2479 2463 2447 2431 2415 2399 2383 2367 2351 2335 2319 2303 2287 2271 2255 2239 2223 2207 2191 2175 2159 2143 2127 2111 2095 2079 2063 2047 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc06708e8 in boot (howto=260) at ../../../kern/kern_shutdown.c:409 #2 0xc0670bca in panic ( fmt=0xc08f7fec "bundirty: buffer %p still on queue %d") at ../../../kern/kern_shutdown.c:565 #3 0xc06b3409 in bundirty (bp=0xdc510d50) at ../../../kern/vfs_bio.c:1036 #4 0xc06b3d6c in brelse (bp=0xdc510d50) at ../../../kern/vfs_bio.c:1351 #5 0xc075e18e in nfs_writebp (bp=0xdc510d50, force=1, td=0xcf6db900) at ../../../nfsclient/nfs_vnops.c:2982 #6 0xc075e3f8 in nfs_bwrite (bp=0xdc510d50) at pcpu.h:162 #7 0xc06b5b7d in getblk (vp=0xca312aa0, blkno=8438, size=6535, slpflag=256, slptimeo=0, flags=0) at buf.h:412 #8 0xc0751299 in nfs_getcacheblk (vp=0xca312aa0, bn=8438, size=6535, td=0xcf6db900) at ../../../nfsclient/nfs_bio.c:1253 #9 0xc0750e62 in nfs_write (ap=0x0) at ../../../nfsclient/nfs_bio.c:1069 #10 0xc08894f2 in VOP_WRITE_APV (vop=0xc097fc00, a=0xec4babf4) at vnode_if.c:698 #11 0xc06cea9a in vn_write (fp=0xc96c7ca8, uio=0xec4bacbc, active_cred=0xc8e19380, flags=0, td=0xcf6db900) at vnode_if.h:372 #12 0xc0692867 in dofilewrite (td=0xcf6db900, fd=5, fp=0xc96c7ca8, auio=0xec4bacbc, offset=Unhandled dwarf expression opcode 0x93 ) at file.h:252 #13 0xc069270b in kern_writev (td=0xcf6db900, fd=5, auio=0xec4bacbc) at ../../../kern/sys_generic.c:402 #14 0xc0692631 in write (td=0xcf6db900, uap=0x0) at ../../../kern/sys_generic.c:326 #15 0xc087842f in syscall (frame= {tf_fs = 137101371, tf_es = 1210646587, tf_ds = -1078001605, tf_edi = 137121792, tf_esi = 1210674680, tf_ebp = -1077942760, tf_isp = -330584732, tf_ebx = 1210584448, tf_edx = 4096, tf_ecx = 0, tf_eax = 4, tf_trapno = 0, tf_err = 2, tf_eip = 1210524463, tf_cs = 51, tf_eflags = 530, tf_esp = -1077942788, tf_ss = 59}) at ../../../i386/i386/trap.c:983 #16 0xc08649ef in Xint0x80_syscall () at ../../../i386/i386/exception.s:200 #17 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) This is the dmesg: Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p3 #2: Thu Apr 5 14:15:19 CEST 2007 kai@buildmachine:/usr/src/sys/i386/compile/SMP-DEBUG WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU 5130 @ 2.00GHz (2000.08-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 Features=0xbfebfbff Features2=0x4e33d,CX16,,,> AMD Features=0x20100000 AMD Features2=0x1 Cores per package: 2 real memory = 3489005568 (3327 MB) avail memory = 3414216704 (3256 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 acpi_throttle0: on cpu0 cpu1: on acpi0 acpi_throttle1: on cpu1 acpi_throttle1: failed to attach P_CNT device_attach: acpi_throttle1 attach returned 6 cpu2: on acpi0 acpi_throttle2: on cpu2 acpi_throttle2: failed to attach P_CNT device_attach: acpi_throttle2 attach returned 6 cpu3: on acpi0 acpi_throttle3: on cpu3 acpi_throttle3: failed to attach P_CNT device_attach: acpi_throttle3 attach returned 6 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: irq 18 at device 2.0 on pci2 pci4: on pcib4 em0: port 0x2000-0x201f mem 0xdd200000-0xdd21ffff irq 18 at device 0.0 on pci4 em0: Ethernet address: 00:30:48:32:b1:ac em1: port 0x2020-0x203f mem 0xdd220000-0xdd23ffff irq 19 at device 0.1 on pci4 em1: Ethernet address: 00:30:48:32:b1:ad pcib5: at device 0.3 on pci1 pci5: on pcib5 aac0: mem 0xdc000000-0xdcffffff,0xdd000000-0xdd1fffff,0xd8000000-0xdbffffff irq 28 at device 2.0 on pci5 aac0: New comm. interface enabled aac0: Adaptec Raid Controller 2.0.0-1 aacp0: on aac0 aacp1: on aac0 pcib6: at device 4.0 on pci0 pci6: on pcib6 pcib7: at device 6.0 on pci0 pci7: on pcib7 pci0: at device 8.0 (no driver attached) uhci0: port 0x1800-0x181f irq 17 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x1820-0x183f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x1840-0x185f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] usb2: on uhci2 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered uhci3: port 0x1860-0x187f irq 16 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] usb3: on uhci3 usb3: USB revision 1.0 uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xdd700000-0xdd7003ff irq 17 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub4: 8 ports with 8 removable, self powered pcib8: at device 30.0 on pci0 pci8: on pcib8 pci8: at device 1.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1880-0x188f at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 ichsmb0: port 0x1100-0x111f irq 19 at device 31.3 on pci0 ichsmb0: [GIANT-LOCKED] smbus0: on ichsmb0 smb0: on smbus0 acpi_button0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcf7ff on isa0 ppc0: parallel port not found. sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec IPv6 packet filtering initialized, default to accept, logging limited to 100 packets/entry ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to accept, logging limited to 100 packets/entry by default acd0: DMA limited to UDMA33, controller found non-ATA66 cable acd0: DVDROM at ata0-slave UDMA33 aacd0: on aac0 aacd0: 139981MB (286681856 sectors) ses0 at aacp0 bus 0 target 6 lun 0 ses0: Fixed unknown SCSI-2 device ses0: 3.300MB/s transfers ses0: SAF-TE Compliant Device pass0 at aacp0 bus 0 target 0 lun 0 pass0: Fixed unknown SCSI-3 device pass0: 3.300MB/s transfers pass1 at aacp0 bus 0 target 1 lun 0 pass1: Fixed unknown SCSI-3 device pass1: 3.300MB/s transfers SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! Kernel config: include GENERIC # saves this config in the kernel (strings -n 3 kernel | sed -n 's/^___//p') options INCLUDE_CONFIG_FILE # higher limits options MAXDSIZ="(1024UL*1024*1024)" options MAXSSIZ="(1024UL*1024*1024)" options DFLDSIZ="(256UL*1024*1024)" # not needed options nooptions MATH_EMULATE nooptions MD_ROOT nooptions UCONSOLE options ASR_COMPAT # asr old ioctls support, needed by raidutils options SEMMNS=256 # Allow enough semaphores for pgsql options IPFIREWALL # firewall options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=100 options IPFIREWALL_FORWARD options IPFIREWALL_DEFAULT_TO_ACCEPT options IPV6FIREWALL options IPV6FIREWALL_VERBOSE options IPV6FIREWALL_VERBOSE_LIMIT=100 options IPV6FIREWALL_DEFAULT_TO_ACCEPT options TCP_DROP_SYNFIN options SMP # Symmetric MultiProcessor Kernel makeoptions DEBUG=-g options INVARIANTS options INVARIANT_SUPPORT options DIAGNOSTIC device smb device smbus device ichsmb device iic device iicbus device iicbb device iicsmb >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: From: Kai Storbeck To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/111734: [nfs] FreeBSD 6.2-release (p3) panic in NFS client code/VFS subsystem Date: Mon, 23 Jul 2007 11:56:00 +0200 This has been fixed in HEAD with version 1.164 of /usr/src/sys/nfsclient/nfs_bio.c, and has been MFC'ed to RELENG_6 in 1.152.2.5 (on July 17th 2007) Regards, Kai State-Changed-From-To: open->closed State-Changed-By: linimon State-Changed-When: Mon Jul 23 21:23:59 UTC 2007 State-Changed-Why: Fixed in current and MFCed. http://www.freebsd.org/cgi/query-pr.cgi?pr=111734 >Unformatted: