From nobody@FreeBSD.org Tue Sep 28 19:11:13 2004 Return-Path: Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 02E8416A4CE for ; Tue, 28 Sep 2004 19:11:12 +0000 (GMT) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id A357943D53 for ; Tue, 28 Sep 2004 19:11:12 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.12.11/8.12.11) with ESMTP id i8SJBCgW092354 for ; Tue, 28 Sep 2004 19:11:12 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.12.11/8.12.11/Submit) id i8SJBCvk092353; Tue, 28 Sep 2004 19:11:12 GMT (envelope-from nobody) Message-Id: <200409281911.i8SJBCvk092353@www.freebsd.org> Date: Tue, 28 Sep 2004 19:11:12 GMT From: Andrzej Bialecki To: freebsd-gnats-submit@FreeBSD.org Subject: JVM crash on 5.2.1-R X-Send-Pr-Version: www-2.3 >Number: 72151 >Category: java >Synopsis: JVM crash on 5.2.1-R >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-java >State: closed >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Sep 28 19:20:20 GMT 2004 >Closed-Date: Fri Jan 25 21:51:41 UTC 2008 >Last-Modified: Fri Jan 25 21:51:41 UTC 2008 >Originator: Andrzej Bialecki >Release: 5.2.1-RELEASE >Organization: GetOpt.org >Environment: FreeBSD web01.kage.se 5.2.1-RELEASE-p10 FreeBSD 5.2.1-RELEASE-p10 #2: Thu Sep 23 15:41:18 CEST 2004 root@web01.kage.se:/usr/src/sys/i386/compile/TUNE i386 >Description: Running an application for Internet search engine (www.nutch.org). JDK built from ports: # java -version java version "1.4.2-p6" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2-p6-root_21_sep_2004_22_20) Java HotSpot(TM) Client VM (build 1.4.2-p6-root_21_sep_2004_22_20, mixed mode) When executing a multi-threaded crawl (~50 threads), and running it for a couple of hours, JVM crashes with the following error log: An unexpected exception has been detected in native code outside the VM. Unexpected Signal : 11 occurred at PC=0x28151624 Function=flockfile+0x24 Library=/lib/libc.so.5 Current Java thread: at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:770) at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1059) at java.net.InetAddress.getAllByName0(InetAddress.java:1009) at java.net.InetAddress.getAllByName0(InetAddress.java:981) at java.net.InetAddress.getAllByName(InetAddress.java:975) at java.net.InetAddress.getByName(InetAddress.java:889) at java.net.InetSocketAddress.(InetSocketAddress.java:114) at net.nutch.protocol.http.HttpResponse.(HttpResponse.java:94) at net.nutch.protocol.http.HttpResponse.(HttpResponse.java:53) at net.nutch.protocol.http.RobotRulesParser.isAllowed(RobotRulesParser.java:364) at net.nutch.protocol.http.Http.getContent(Http.java:145) at net.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:94) Dynamic libraries: 0x8048000 /usr/local/jdk1.4.2/bin/java 0x28078000 /usr/lib/libc_r.so.5 0x2809c000 /lib/libc.so.5 0x28177000 /usr/local/jdk1.4.2/jre/lib/i386/client/libjvm.so 0x285b7000 /usr/lib/libstdc++.so.4 0x28673000 /lib/libm.so.2 0x2868c000 /usr/local/jdk1.4.2/jre/lib/i386/native_threads/libhpi.so 0x2869a000 /usr/local/jdk1.4.2/jre/lib/i386/libverify.so 0x286b0000 /usr/local/jdk1.4.2/jre/lib/i386/libjava.so 0x286ce000 /usr/local/jdk1.4.2/jre/lib/i386/libzip.so 0x8e94d000 /usr/local/jdk1.4.2/jre/lib/i386/libnet.so 0x2804e000 /libexec/ld-elf.so.1 Heap at VM Abort: Heap def new generation total 28800K, used 19206K [0x2c470000, 0x2e3b0000, 0x337d0000) eden space 25600K, 69% used [0x2c470000, 0x2d5b8d48, 0x2dd70000) from space 3200K, 47% used [0x2dd70000, 0x2dee8ae8, 0x2e090000) to space 3200K, 0% used [0x2e090000, 0x2e090000, 0x2e3b0000) tenured generation total 381840K, used 283107K [0x337d0000, 0x4acb4000, 0x8a070000) the space 381840K, 74% used [0x337d0000, 0x44c48c48, 0x44c48e00, 0x4acb4000) compacting perm gen total 6912K, used 6815K [0x8a070000, 0x8a730000, 0x8e070000) the space 6912K, 98% used [0x8a070000, 0x8a717c40, 0x8a717e00, 0x8a730000) Local Time = Tue Sep 28 14:28:02 2004 Elapsed Time = 38319 # # The exception above was detected in native code outside the VM # # Java VM: Java HotSpot(TM) Client VM (1.4.2-p6-root_21_sep_2004_22_20 mixed mode) # >How-To-Repeat: it happens from time to time, after running for several hours. >Fix: >Release-Note: >Audit-Trail: From: Dan Nelson To: Andrzej Bialecki Cc: freebsd-gnats-submit@FreeBSD.org Subject: Re: java/72151: JVM crash on 5.2.1-R Date: Tue, 28 Sep 2004 17:21:58 -0500 In the last episode (Sep 28), Andrzej Bialecki said: > Running an application for Internet search engine (www.nutch.org). JDK built from ports: > > # java -version > java version "1.4.2-p6" > Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2-p6-root_21_sep_2004_22_20) > Java HotSpot(TM) Client VM (build 1.4.2-p6-root_21_sep_2004_22_20, mixed mode) > > When executing a multi-threaded crawl (~50 threads), and running it for a couple of hours, JVM crashes with the following error log: > > An unexpected exception has been detected in native code outside the VM. > Unexpected Signal : 11 occurred at PC=0x28151624 > Function=flockfile+0x24 > Library=/lib/libc.so.5 > > Current Java thread: > at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) This is because java is calling gethostent() which is not thread-safe (POSIX does not require it to be, either). It uses a global "hostf" FILE* to read /etc/hosts, so multiple threads doing DNS lookups will almost certainly trash each other. My quick hack is this, which returns a DNS error if hostf ends up NULL. A better fix would be to add a mutex around the DNS routines in Inet4AddressImpl.lookupAllHostAddr. Index: gethostbyht.c =================================================================== RCS file: /home/ncvs/src/lib/libc/net/gethostbyht.c,v retrieving revision 1.16 diff -u -p -r1.16 gethostbyht.c --- gethostbyht.c 22 Mar 2002 21:52:29 -0000 1.16 +++ gethostbyht.c 3 Aug 2004 16:43:02 -0000 @@ -112,6 +112,10 @@ gethostent() return (NULL); } again: + if (!hostf) { + h_errno = NETDB_INTERNAL; + return (NULL); + } if (!(p = fgets(hostbuf, sizeof hostbuf, hostf))) { h_errno = HOST_NOT_FOUND; return (NULL); -- Dan Nelson dnelson@allantgroup.com From: Dan Nelson To: Andrzej Bialecki Cc: freebsd-gnats-submit@FreeBSD.org Subject: Re: java/72151: JVM crash on 5.2.1-R Date: Tue, 28 Sep 2004 17:39:29 -0500 In the last episode (Sep 28), Dan Nelson said: > In the last episode (Sep 28), Andrzej Bialecki said: > > An unexpected exception has been detected in native code outside the VM. > > Unexpected Signal : 11 occurred at PC=0x28151624 > > Function=flockfile+0x24 > > Library=/lib/libc.so.5 > > > > Current Java thread: > > at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) > > almost certainly trash each other. My quick hack is this, which > returns a DNS error if hostf ends up NULL. A better fix would be to > add a mutex around the DNS routines in > Inet4AddressImpl.lookupAllHostAddr. I'm wrong; the best fix would be thread-safe resolver routines in libc :) -- Dan Nelson dnelson@allantgroup.com Responsible-Changed-From-To: freebsd-java->phantom Responsible-Changed-By: phantom Responsible-Changed-When: Fri Dec 24 08:49:22 GMT 2004 Responsible-Changed-Why: I'll track this one http://www.freebsd.org/cgi/query-pr.cgi?pr=72151 State-Changed-From-To: open->feedback State-Changed-By: linimon State-Changed-When: Fri Mar 31 08:52:06 UTC 2006 State-Changed-Why: Submitter: does this still happen on 5.5-PRERELEASE? Responsible-Changed-From-To: phantom->freebsd-java Responsible-Changed-By: linimon Responsible-Changed-When: Fri Mar 31 08:52:06 UTC 2006 Responsible-Changed-Why: Reassign from phantom since he has been inactive for more than one year. Hat: gnats-admin http://www.freebsd.org/cgi/query-pr.cgi?pr=72151 From: Stefan Walter To: Andrzej Bialecki , Dan Nelson Cc: GNATS Subject: Re: java/72151: JVM crash on 5.2.1-R Date: Wed, 26 Sep 2007 19:46:15 +0200 Hi Andrzej, there hasn't been an update on the state of this for quite a while. Are you still experiencing these problems on a more recent system/with a more recent VM? Or can this PR be closed? Best regards, Stefan State-Changed-From-To: feedback->closed State-Changed-By: linimon State-Changed-When: Fri Jan 25 21:51:26 UTC 2008 State-Changed-Why: Feedback timeout (> 6 months). http://www.freebsd.org/cgi/query-pr.cgi?pr=72151 >Unformatted: