From nobody@FreeBSD.org Fri Feb 20 16:08:35 2009 Return-Path: Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 925311065670 for ; Fri, 20 Feb 2009 16:08:35 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 80A738FC08 for ; Fri, 20 Feb 2009 16:08:35 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n1KG8ZOI071401 for ; Fri, 20 Feb 2009 16:08:35 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id n1KG8YOK071388; Fri, 20 Feb 2009 16:08:34 GMT (envelope-from nobody) Message-Id: <200902201608.n1KG8YOK071388@www.freebsd.org> Date: Fri, 20 Feb 2009 16:08:34 GMT From: Arne Steinkamm To: freebsd-gnats-submit@FreeBSD.org Subject: SATA data corruption X-Send-Pr-Version: www-3.1 X-GNATS-Notify: >Number: 131906 >Category: amd64 >Synopsis: [ata] SATA data corruption with Promise PDC20378 (amd64-specific) [regression] >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-amd64 >State: closed >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Feb 20 16:10:05 UTC 2009 >Closed-Date: Mon Oct 25 18:09:48 UTC 2010 >Last-Modified: Mon Oct 25 18:09:48 UTC 2010 >Originator: Arne Steinkamm >Release: 7.1-RELEASE >Organization: >Environment: FreeBSD amd64.lab.steinkamm.com 7.1-RELEASE FreeBSD 7.1-RELEASE #0: Thu Jan 1 08:58:24 UTC 2009 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >Description: Machine has (old) Asus A8V Deluxe Motherboard with onboard Promise SATA controller. Promise is BIOS-configured to RAID mode. Machine is equipped with 4 gig mem, memory hole handling is switched off in BIOS. CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ (2202.83-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x20f32 Stepping = 2 [...] Cores per package: 2 usable memory = 3341815808 (3187 MB) avail memory = 3231543296 (3081 MB) [...] atapci0: port 0x9000-0x903f,0x8800-0x880f,0x8400-0x847f mem 0xf9700000-0xf9700fff,0xf9600000-0xf961ffff irq 18 at device 8.0 on pci0 [...] ad4: 1430799MB at ata2-master SATA150 ad6: 1430799MB at ata3-master SATA150 ar0: 1430511MB status: READY ar0: disk0 READY (master) using ad4 at ata2-master ar0: disk1 READY (mirror) using ad6 at ata3-master I have several of these boxes in daily use without any problems running FreeBSD 6.2 i386 and amd64. I installed a fresh installation of 7.1 release (GENERIC kernel from distribution) from CD and saw data corruption on the disks: $ df . Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/ar0s2f 1361274282 883732236 368640104 71% /l/ldisk1 $ ls -l Testfile* -rw-r--r-- 1 root wheel 2810811312 Dec 31 01:14 Testfile1 -rw-r--r-- 1 root wheel 2926088400 Dec 31 01:59 Testfile2 -rw-r--r-- 1 root wheel 2798053632 Dec 31 02:45 Testfile3 $ md5 Testfile* MD5 (Testfile1) = 14f9908220acd2388ec09703dc8baaad MD5 (Testfile2) = c6f2c84ec25fd97985b6374bdc89b343 MD5 (Testfile3) = 5630706a77bc7ee965076ba2a7a8dcdf $ md5 Testfile* MD5 (Testfile1) = d215c9fb3369d125d17e579f5ddf7ca8 MD5 (Testfile2) = c6f2c84ec25fd97985b6374bdc89b343 MD5 (Testfile3) = 8632aa26eec5966a49323c714f9b085e $ md5 Testfile* MD5 (Testfile1) = fb4d186d5fb900dd9271049e29eab152 MD5 (Testfile2) = 429511875320aef36a700ef67d1462c2 MD5 (Testfile3) = f7be094c9a38c198152536dffca3d733 Trying to find the problem, I booted the system from CD and accessed the subdisks ad4 and ad6 directly. Same problem with both disks so it seems not related to the ar0 device and the software raid behind it. Then I booted FreeBSD 7.1-Release i386 from CD and found no problem anymore. This indicates that the hardware is not part of the problem. Full dmesg: Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-RELEASE #0: Thu Jan 1 08:58:24 UTC 2009 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ (2202.83-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x20f32 Stepping = 2 Features=0x178bfbff Features2=0x1 AMD Features=0xe2500800 AMD Features2=0x3 Cores per package: 2 usable memory = 3341815808 (3187 MB) avail memory = 3231543296 (3081 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, c7ef0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 agp0: on hostb0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: mem 0xfb000000-0xfbffffff,0xe0000000-0xefffffff,0xfa000000-0xfaffffff irq 16 at device 0.0 on pci1 fwohci0: port 0x8000-0x807f mem 0xf9500000-0xf95007ff irq 16 at device 7.0 on pci0 fwohci0: [FILTER] fwohci0: OHCI version 1.0 (ROM=1) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:11:d8:00:00:6f:2b:1e fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0x28c4000 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:11:d8:6f:2b:1e fwe0: Ethernet address: 02:11:d8:6f:2b:1e fwip0: on firewire0 fwip0: Firewire address: 00:11:d8:00:00:6f:2b:1e @ 0xfffe00000000, S400, maxrec 2048 sbp0: on firewire0 fwohci0: Initiate bus reset fwohci0: BUS reset fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode atapci0: port 0x9000-0x903f,0x8800-0x880f,0x8400-0x847f mem 0xf9700000-0xf9700fff,0xf9600000-0xf961ffff irq 18 at device 8.0 on pci0 atapci0: [ITHREAD] atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] ata4: on atapci0 ata4: [ITHREAD] vgapci1: mem 0xf8000000-0xf8ffffff,0xd0000000-0xdfffffff,0xf7000000-0xf7ffffff irq 16 at device 9.0 on pci0 skc0: port 0x9400-0x94ff mem 0xf9a00000-0xf9a03fff irq 17 at device 10.0 on pci0 skc0: Marvell Yukon Lite Gigabit Ethernet rev. (0x9) sk0: on skc0 sk0: Ethernet address: 00:15:f2:0a:78:17 miibus0: on sk0 e1000phy0: PHY 0 on miibus0 e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto skc0: [ITHREAD] atapci1: port 0xb000-0xb007,0xa800-0xa803,0xa400-0xa407,0xa000-0xa003,0x9800-0x980f mem 0xf9c00000-0xf9c003ff irq 19 at device 14.0 on pci0 atapci1: [ITHREAD] ata5: on atapci1 ata5: [ITHREAD] ata6: on atapci1 ata6: [ITHREAD] ata7: on atapci1 ata7: [ITHREAD] ata8: on atapci1 ata8: [ITHREAD] atapci2: port 0xd000-0xd007,0xc800-0xc803,0xc400-0xc407,0xc000-0xc003,0xb800-0xb80f,0xb400-0xb4ff irq 20 at device 15.0 on pci0 atapci2: [ITHREAD] ata9: on atapci2 ata9: [ITHREAD] ata10: on atapci2 ata10: [ITHREAD] atapci3: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 15.1 on pci0 ata0: on atapci3 ata0: [ITHREAD] ata1: on atapci3 ata1: [ITHREAD] uhci0: port 0xd400-0xd41f irq 21 at device 16.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xd800-0xd81f irq 21 at device 16.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0xe000-0xe01f irq 21 at device 16.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: port 0xe400-0xe41f irq 21 at device 16.3 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb3: on uhci3 usb3: USB revision 1.0 uhub3: on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xf9e00000-0xf9e000ff irq 21 at device 16.4 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: on usb4 uhub4: 8 ports with 8 removable, self powered umass0: on uhub4 isab0: at device 17.0 on pci0 isa0: on isab0 pci0: at device 17.5 (no driver attached) acpi_button0: on acpi0 acpi_button1: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse Explorer, device ID 4 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 ppbus0: [ITHREAD] plip0: on ppbus0 plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sio0: configured irq 3 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 3 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: configured irq 4 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: configured irq 4 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 on acpi0 sio1: type 16550A sio1: [FILTER] cpu0: on acpi0 powernow0: on cpu0 device_attach: powernow0 attach returned 6 cpu1: on acpi0 powernow1: on cpu1 device_attach: powernow1 attach returned 6 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me) firewire0: bus manager 0 (me) acd0: CDRW at ata0-master UDMA33 ad4: 1430799MB at ata2-master SATA150 ad6: 1430799MB at ata3-master SATA150 ar0: 1430511MB status: READY ar0: disk0 READY (master) using ad4 at ata2-master ar0: disk1 READY (mirror) using ad6 at ata3-master SMP: AP CPU #1 Launched! da0 at umass-sim0 bus 0 target 0 lun 0 da0: Removable Direct Access SCSI-0 device da0: 40.000MB/s transfers da0: Attempt to query device size failed: NOT READY, Medium not present da1 at umass-sim0 bus 0 target 0 lun 1 da1: Removable Direct Access SCSI-0 device da1: 40.000MB/s transfers da1: Attempt to query device size failed: NOT READY, Medium not present Trying to mount root from ufs:/dev/ar0s1a pciconf -lv hostb0@pci0:0:0:0: class=0x060000 card=0x80a31043 chip=0x02821106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8T880Pro CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb1@pci0:0:0:1: class=0x060000 card=0x00000000 chip=0x12821106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8T880Pro CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb2@pci0:0:0:2: class=0x060000 card=0x00000000 chip=0x22821106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8T880Pro CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb3@pci0:0:0:3: class=0x060000 card=0x00000000 chip=0x32821106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8T880Pro CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb4@pci0:0:0:4: class=0x060000 card=0x00000000 chip=0x42821106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8T880Pro CPU to PCI Bridge' class = bridge subclass = HOST-PCI hostb5@pci0:0:0:7: class=0x060000 card=0x00000000 chip=0x72821106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'K8T880Pro CPU to PCI Bridge' class = bridge subclass = HOST-PCI pcib1@pci0:0:1:0: class=0x060400 card=0x00000000 chip=0xb1881106 rev=0x00 hdr=0x01 vendor = 'VIA Technologies Inc' device = 'VT8237 K8HTB CPU to AGP 2.0/3.0 Bridge' class = bridge subclass = PCI-PCI fwohci0@pci0:0:7:0: class=0x0c0010 card=0x808a1043 chip=0x30441106 rev=0x80 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT6306 VIA Fire II IEEE-1394 OHCI Link Layer Controller' class = serial bus subclass = FireWire atapci0@pci0:0:8:0: class=0x010400 card=0x80f51043 chip=0x3373105a rev=0x02 hdr=0x00 vendor = 'Promise Technology Inc' device = 'PDC20378 FastTrak 378/SATA 378 RAID Controller' class = mass storage subclass = RAID vgapci1@pci0:0:9:0: class=0x030000 card=0x00000000 chip=0x022110de rev=0xa1 hdr=0x00 vendor = 'Nvidia Corp' device = 'GeForce 6200' class = display subclass = VGA skc0@pci0:0:10:0: class=0x020000 card=0x811a1043 chip=0x432011ab rev=0x13 hdr=0x00 vendor = 'Marvell Semiconductor (Was: Galileo Technology Ltd)' device = 'Yukon 88E8001/8003/8010 PCI Gigabit Ethernet Controller (Copper)' class = network subclass = ethernet atapci1@pci0:0:14:0: class=0x010400 card=0x31141095 chip=0x31141095 rev=0x02 hdr=0x00 vendor = 'Silicon Image Inc (Was: CMD Technology Inc)' device = 'Sil 3114 SATALink/SATARaid Controller' class = mass storage subclass = RAID atapci2@pci0:0:15:0: class=0x010400 card=0x80ed1043 chip=0x31491106 rev=0x80 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT8237 VT6410 SATA RAID Controller' class = mass storage subclass = RAID atapci3@pci0:0:15:1: class=0x01018a card=0x80ed1043 chip=0x05711106 rev=0x06 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT82C586A/B/VT82C686/A/B/VT823x/A/C Bus Master IDE Controller' class = mass storage subclass = ATA uhci0@pci0:0:16:0: class=0x0c0300 card=0x80ed1043 chip=0x30381106 rev=0x81 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT83C572, VT6202 VIA Rev 5 or later USB Universal Host Controller' class = serial bus subclass = USB uhci1@pci0:0:16:1: class=0x0c0300 card=0x80ed1043 chip=0x30381106 rev=0x81 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT83C572, VT6202 VIA Rev 5 or later USB Universal Host Controller' class = serial bus subclass = USB uhci2@pci0:0:16:2: class=0x0c0300 card=0x80ed1043 chip=0x30381106 rev=0x81 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT83C572, VT6202 VIA Rev 5 or later USB Universal Host Controller' class = serial bus subclass = USB uhci3@pci0:0:16:3: class=0x0c0300 card=0x80ed1043 chip=0x30381106 rev=0x81 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT83C572, VT6202 VIA Rev 5 or later USB Universal Host Controller' class = serial bus subclass = USB ehci0@pci0:0:16:4: class=0x0c0320 card=0x80ed1043 chip=0x31041106 rev=0x86 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT6202/12 USB 2.0 Enhanced Host Controller' class = serial bus subclass = USB isab0@pci0:0:17:0: class=0x060100 card=0x80ed1043 chip=0x32271106 rev=0x00 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT8237 PCI-to-ISA Bridge' class = bridge subclass = PCI-ISA none0@pci0:0:17:5: class=0x040100 card=0x812a1043 chip=0x30591106 rev=0x60 hdr=0x00 vendor = 'VIA Technologies Inc' device = 'VT8237 AC97 Enhanced Audio Controller - the 8251 controller is different' class = multimedia subclass = audio hostb6@pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x11001022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron HyperTransport Technology Configuration' class = bridge subclass = HOST-PCI hostb7@pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x11011022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron Address Map' class = bridge subclass = HOST-PCI hostb8@pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x11021022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron DRAM Controller' class = bridge subclass = HOST-PCI hostb9@pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x11031022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron Miscellaneous Control' class = bridge subclass = HOST-PCI vgapci0@pci0:1:0:0: class=0x030000 card=0x080110b0 chip=0x00f510de rev=0xa2 hdr=0x00 vendor = 'Nvidia Corp' device = 'GeForce 7800 GS' class = display subclass = VGA >How-To-Repeat: Just check the integrity of large files (several gigabytes) after writing them. Files fitting in the filesystem cache will hide the sata problem. >Fix: Using FreeBSD 6.x amd64 or FreeBSD 7.1 i386 instead of FreeBSD 7.1 amd64. This is not a sollution! >Release-Note: >Audit-Trail: From: John Baldwin To: bug-followup@freebsd.org, arne@steinkamm.com Cc: Subject: Re: amd64/131906: [ata] SATA data corruption with Promise PDC20378 (amd64-specific) [regression] Date: Thu, 19 Mar 2009 16:00:04 -0400 Did you try booting amd64 with 'hw.physmem=4g' set in the loader? GENERIC on i386 doesn't include PAE so it effectively limits you to the lower 4gb. -- John Baldwin State-Changed-From-To: open->feedback State-Changed-By: jh State-Changed-When: Fri Sep 3 06:50:30 UTC 2010 State-Changed-Why: Note that submitter has been asked for feedback. http://www.freebsd.org/cgi/query-pr.cgi?pr=131906 State-Changed-From-To: feedback->closed State-Changed-By: jh State-Changed-When: Mon Oct 25 18:09:46 UTC 2010 State-Changed-Why: Feedback timeout. http://www.freebsd.org/cgi/query-pr.cgi?pr=131906 >Unformatted: