From nobody@FreeBSD.org Wed Mar 9 14:51:48 2011 Return-Path: Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A43F5106567A for ; Wed, 9 Mar 2011 14:51:48 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 7A0798FC08 for ; Wed, 9 Mar 2011 14:51:48 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p29EpmrB023626 for ; Wed, 9 Mar 2011 14:51:48 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id p29Epm3w023625; Wed, 9 Mar 2011 14:51:48 GMT (envelope-from nobody) Message-Id: <201103091451.p29Epm3w023625@red.freebsd.org> Date: Wed, 9 Mar 2011 14:51:48 GMT From: "Philip M. Gollucci" To: freebsd-gnats-submit@FreeBSD.org Subject: [regression] [8.2-release] [tmpfs]: mount: tmpfs : No space left on device X-Send-Pr-Version: www-3.1 X-GNATS-Notify: >Number: 155411 >Category: kern >Synopsis: [regression] [8.2-release] [tmpfs]: mount: tmpfs : No space left on device >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-fs >State: feedback >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Wed Mar 09 15:00:22 UTC 2011 >Closed-Date: >Last-Modified: Tue Nov 22 18:06:08 UTC 2011 >Originator: Philip M. Gollucci >Release: 8.2-RELEASE >Organization: Apache Software Foundation >Environment: FreeBSD eris.apache.org 8.2-RELEASE FreeBSD 8.2-RELEASE #0 r219046: Sat Feb 26 00:50:03 UTC 2011 root@loki.apache.org:/usr/obj/usr/src/sys/GENERIC amd64 >Description: sudo mount /tmp mount: tmpfs : No space left on device $uname -a FreeBSD eris.apache.org 8.2-RELEASE FreeBSD 8.2-RELEASE #0 r219046: Sat Feb 26 00:50:03 UTC 2011 root@loki.apache.org:/usr/obj/usr/src/sys/GENERIC amd64 from top: Mem: 592M Active, 1162M Inact, 21G Wired, 236M Cache, 2465M Buf, 758M Free Swap: 8317M Total, 59M Used, 8258M Free $cat /etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/mirror/gm0s1b none swap sw 0 0 /dev/mirror/gm0s1a / ufs rw,noatime 1 1 /dev/acd0 /cdrom cd9660 ro,noauto 0 0 tmpfs /tmp tmpfs rw 2 0 nfs:/usr/src /usr/src nfs ro,tcp,intr,rdirplus,soft,wsize=32768,rsize=32768,nolockd,noauto 0 0 nfs:/usr/obj /usr/obj nfs ro,tcp,intr,rdirplus,soft,wsize=32768,rsize=32768,nolockd,noauto 0 0 nfs:/usr/ports /usr/ports nfs ro,tcp,intr,rdirplus,soft,wsize=32768,rsize=32768,nolockd,noauto 0 0 $df -h Filesystem Size Used Avail Capacity Mounted on /dev/mirror/gm0s1a 668G 299G 316G 49% / devfs 1.0K 1.0K 0B 100% /dev tank/x1 1.0T 681G 369G 65% /x1 $zpool status pool: tank state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2 ONLINE 0 0 0 mfid2 ONLINE 0 0 0 mfid3 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 cache da1 ONLINE 0 0 0 da0 ONLINE 0 0 0 errors: No known data errors >How-To-Repeat: >Fix: more info on request >Release-Note: >Audit-Trail: From: Bruce Cran To: "Philip M. Gollucci" Cc: freebsd-gnats-submit@FreeBSD.org Subject: Re: kern/155411: [regression] [8.2-release] [tmpfs]: mount: tmpfs : No space left on device Date: Wed, 9 Mar 2011 15:43:27 +0000 On Wed, 9 Mar 2011 14:51:48 GMT "Philip M. Gollucci" wrote: > $zpool status > pool: tank > state: ONLINE ZFS and tmpfs don't work well together, because zfs takes all the memory and doesn't leave any for tmpfs. It's essentially the same problem, I think, as http://bugs.opensolaris.org/bugdatabase/view_bug.do;jsessionid=e4ae9c32983000ef651e38edbba1?bug_id=6804661 . -- Bruce Cran From: "Philip M. Gollucci" To: Bruce Cran Cc: "Philip M. Gollucci" , Subject: Re: kern/155411: [regression] [8.2-release] [tmpfs]: mount: tmpfs : No space left on device Date: Wed, 9 Mar 2011 15:51:08 +0000 --------------enigAF6CF3C18DB55C143AB9DB43 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Maybe but its never once happened since zfs hit the tree in 7.0 through 8.1. I have litterally hundreds of boxes without this issue. and at least 2 8.2+ boxes that have it. Also it was just rebooted for the upgrade so 'possibly' had lots of free 'solaris' kernel memory free. On 03/09/11 15:43, Bruce Cran wrote: > On Wed, 9 Mar 2011 14:51:48 GMT > "Philip M. Gollucci" wrote: >=20 >> $zpool status >> pool: tank >> state: ONLINE >=20 > ZFS and tmpfs don't work well together, because zfs takes all the > memory and doesn't leave any for tmpfs. It's essentially the same > problem, I think, as > http://bugs.opensolaris.org/bugdatabase/view_bug.do;jsessionid=3De4ae9c= 32983000ef651e38edbba1?bug_id=3D6804661 . >=20 --=20 ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. --------------enigAF6CF3C18DB55C143AB9DB43 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iD8DBQFNd6HudbiP+9ubjBwRAijvAKCMBMeCoWaa1z4srfAFT/A+ivJeTgCfYDht Ue40vbiXzpvDz+SD8DWO4D4= =AY0C -----END PGP SIGNATURE----- --------------enigAF6CF3C18DB55C143AB9DB43-- From: "Philip M. Gollucci" To: , , ASF Infra-P Cc: Subject: Re: kern/155411: [regression] [8.2-release] [tmpfs]: mount: tmpfs : No space left on device Date: Wed, 9 Mar 2011 18:34:00 +0000 --------------enigACB75630B1A3424BE0219CB9 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable /boot/loader.conf ipmi_load=3D"YES" kern.maxswzone=3D67108864 geom_mirror_load=3D"YES" accf_http_load=3D"YES" accf_data_load=3D"YES" --=20 ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. --------------enigACB75630B1A3424BE0219CB9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iD8DBQFNd8gadbiP+9ubjBwRAoiZAJ4orSya2l89tIGp4oCzT0aCAK+2pgCggaki C4mTg/A2EseZyfKByYqKfR8= =kGuh -----END PGP SIGNATURE----- --------------enigACB75630B1A3424BE0219CB9-- Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Wed Mar 9 19:58:54 UTC 2011 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=155411 From: Gleb Kurtsou To: bug-followup@FreeBSD.org, pgollucci@FreeBSD.org Cc: Subject: Re: kern/155411: [regression] [8.2-release] [tmpfs]: mount: tmpfs : No space left on device Date: Thu, 10 Mar 2011 14:53:54 +0200 Could you test the patch. It changes the way tmpfs grows, sets filesystem default size to half of RAM by default. Tmpfs no longer depends on inactive/wired memory stats, but checks if swap is nearly full. I've added vfs.tmpfs.swap_reserved sysctl to limit tmpfs growth. Bottom line is, that you should specify meaningful filesystem size to prevent resource exhaustion. In my tests system didn't panic nor invoked OOM killer while consuming nearly all available ram and swap. Patch: http://marc.info/?l=freebsd-fs&m=129735686129438&w=2 It also handles test case described in OpenSolaris bug report for me: http://marc.info/?l=freebsd-fs&m=129747362722933&w=2 Thanks, Gleb. From: "Philip M. Gollucci" To: Gleb Kurtsou Cc: , Subject: Re: kern/155411: [regression] [8.2-release] [tmpfs]: mount: tmpfs : No space left on device Date: Thu, 10 Mar 2011 15:28:26 +0000 --------------enig2D2A45DAF19A5900C29D42FA Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I/we would be glad to test it; however, we'll have to schedule since the computers in question are both front facing ASF services. On 03/10/11 12:53, Gleb Kurtsou wrote: > Could you test the patch. It changes the way tmpfs grows, sets > filesystem default size to half of RAM by default. Tmpfs no longer > depends on inactive/wired memory stats, but checks if swap is nearly > full. I've added vfs.tmpfs.swap_reserved sysctl to limit tmpfs growth. >=20 > Bottom line is, that you should specify meaningful filesystem size to > prevent resource exhaustion. >=20 > In my tests system didn't panic nor invoked OOM killer while consuming > nearly all available ram and swap. >=20 > Patch: > http://marc.info/?l=3Dfreebsd-fs&m=3D129735686129438&w=3D2 >=20 > It also handles test case described in OpenSolaris bug report for me: > http://marc.info/?l=3Dfreebsd-fs&m=3D129747362722933&w=3D2 >=20 > Thanks, > Gleb. >=20 --=20 ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. --------------enig2D2A45DAF19A5900C29D42FA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iD8DBQFNeO4kdbiP+9ubjBwRAvsaAJ9mCFT6ih/2g0aB2lgnTToajxB5dACfdGep 2Iw1F55WtSV162sdhrgOSYs= =hNNY -----END PGP SIGNATURE----- --------------enig2D2A45DAF19A5900C29D42FA-- State-Changed-From-To: open->feedback State-Changed-By: jh State-Changed-When: Tue Nov 22 18:04:58 UTC 2011 State-Changed-Why: Do you still see the problem after r227802? http://www.freebsd.org/cgi/query-pr.cgi?pr=155411 >Unformatted: