From nobody@FreeBSD.org Fri May 28 13:06:13 2010 Return-Path: Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05581106567B for ; Fri, 28 May 2010 13:06:13 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id E8FD78FC14 for ; Fri, 28 May 2010 13:06:12 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o4SD6B1t073249 for ; Fri, 28 May 2010 13:06:11 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id o4SD6BvN073233; Fri, 28 May 2010 13:06:11 GMT (envelope-from nobody) Message-Id: <201005281306.o4SD6BvN073233@www.freebsd.org> Date: Fri, 28 May 2010 13:06:11 GMT From: Tom Dewaele To: freebsd-gnats-submit@FreeBSD.org Subject: Page Fault / Kernel panic when jail starts on boot X-Send-Pr-Version: www-3.1 X-GNATS-Notify: >Number: 147162 >Category: kern >Synopsis: [jail] [panic] Page Fault / Kernel panic when jail starts on boot >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-jail >State: closed >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri May 28 13:10:00 UTC 2010 >Closed-Date: Mon Sep 06 07:18:37 UTC 2010 >Last-Modified: Mon Sep 06 07:18:37 UTC 2010 >Originator: Tom Dewaele >Release: FreeBSD 8.0p3 AMD64 >Organization: ABVV >Environment: FreeBSD T00FVS01.fed.diva.net 8.0-RELEASE-p3 FreeBSD 8.0-RELEASE-p3 #0: Tue May 25 20:54:11 UTC 2010 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 >Description: A short system explanation: Hardware: HP DL 360G6 2x E5504 / 8GB I'am running 3 jails on this server with ezjail (latest port) I've updated from 8.0p2 to 8.0p3 yesterday. After reboot the server did not come back. It showed a page fault crash on the console after ezjail started the first jail. The page fault happen with the 'ping' proces. The jail that started and crashed the system is a network monitoring server with several ping scripts. I have "security.jail.allow_raw_sockets=1" active on the system to allow this. I had to reboot in single user mode and disable the autostart of my jails. I've updated all jails to 8.0p3 (with ezjail - update binary via FTP) and rebuilt all ports to make sure that this was not the issue. That did not solve my problem. However, once the system is booted, I can start the jails via ezjail with crashing. The crash only happens when the jails start on boot. I think this is caused by the security fixes in p3 for the jail system. I've never had this problem before this. >How-To-Repeat: It happens everytime I boot with ezjail_enable="YES". After the first jail start, the system goes down with kernel panic / page fault. >Fix: I does not happen when I start the jails manually using ezjail-admin onestart [jail] >Release-Note: >Audit-Trail: Responsible-Changed-From-To: freebsd-amd64->freebsd-jail Responsible-Changed-By: linimon Responsible-Changed-When: Mon May 31 05:01:35 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=147162 From: Andriy Gapon To: bug-followup@FreeBSD.org, tom.dewaele@abvv.be Cc: Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot Date: Tue, 01 Jun 2010 18:41:26 +0300 Are you able to get a crash dump? Or at least a screen shot/capture of a panic message _with_ backtrace. Otherwise, chances of getting this resolved are lower. -- Andriy Gapon From: To: Cc: Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot Date: Thu, 3 Jun 2010 15:32:03 +0200 This is a multi-part message in MIME format. ------_=_NextPart_001_01CB0321.2763EF58 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I've obtained some more information. =20 I had a hunch this was caused by a PF rule. I disabled PF and the jails did not cause a panic. Then I disabled the suspicious rule and again no panic. Re-enabled the rule -> panic. =20 This is the rule that is either directly or indirectly involved in the kernel panic: pass out quick route-to (lagg0 10.200.0.1) from 10.200.0.0/24 to ! 10.200.0.0/24 =20 I have this rule because my jails are on another vlan so they need to use another interface for outgoing packets. The problem must be linked to this rerouting. =20 Today I've built a debugging kernel with the options you suggested. I was able to get a backtrace with DDB. =20 You can find the panic here: http://tweakers.net/ext/f/iirS2Omm8Nfmbpb0BLKSRSzY/full.jpg =20 and the backtrace I did here: http://tweakers.net/ext/f/doP5G3aux6tSZdZt6hTtaWTR/full.jpg =20 =20 The backtrace has 2 pf_route calls so that confirms again my hunch. =20 This must be caused by kernel code changes made from 8.0p2 to 8.0p3 8.0p2 did not cause the kernel panic. =20 ------_=_NextPart_001_01CB0321.2763EF58 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I've obtained some more = information.

 

I had a hunch this was caused = by a PF rule.

I disabled PF and the jails = did not cause a panic.

Then I disabled the = suspicious rule and again no panic.

Re-enabled the rule -> = panic.

 

This is the rule that is = either directly or indirectly involved in the kernel panic:

pass out quick route-to = (lagg0 10.200.0.1) from 10.200.0.0/24 to ! 10.200.0.0/24

 

I have this rule because my = jails are on another vlan so they need to use another interface for outgoing = packets.

The problem must be linked to = this rerouting.

 

Today I've built a debugging = kernel with the options you suggested.

I was able to get a backtrace = with DDB.

 

You can find the panic = here:

http://tweakers.net/ext/f/iirS2Omm8Nfmbpb0BLKSRSzY/full.jpg<= /span>

and the backtrace I did = here:

http://tweakers.net/ext/f/doP5G3aux6tSZdZt6hTtaWTR/full.jpg<= /span>

 

The backtrace has 2 pf_route = calls so that confirms again my hunch.

 

This must be caused by kernel = code changes made from 8.0p2 to 8.0p3

8.0p2 did not cause the = kernel panic.

 

------_=_NextPart_001_01CB0321.2763EF58-- From: To: , Cc: Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot Date: Thu, 3 Jun 2010 16:26:25 +0200 I've obtained some more information. I had a hunch this was caused by a PF rule. I disabled PF and the jails did not cause a panic. Then I disabled the suspicious rule and again no panic. Re-enabled the rule -> panic. This is the rule that is either directly or indirectly involved in the kernel panic: pass out quick route-to (lagg0 10.200.0.1) from 10.200.0.0/24 to ! 10.200.0.0/24 I have this rule because my jails are on another vlan so they need to use another interface for outgoing packets. The problem must be linked to this rerouting. Today I've built a debugging kernel with the options you suggested. I was able to get a backtrace with DDB. You can find the panic here: http://tweakers.net/ext/f/iirS2Omm8Nfmbpb0BLKSRSzY/full.jpg and the backtrace I did here: http://tweakers.net/ext/f/doP5G3aux6tSZdZt6hTtaWTR/full.jpg The backtrace has 2 pf_route calls so that confirms again my hunch. This must be caused by kernel code changes made from 8.0p2 to 8.0p3 8.0p2 did not cause the kernel panic. From: "Bjoern A. Zeeb" To: bug-followup@FreeBSD.org, tom.dewaele@abvv.be Cc: Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot Date: Thu, 3 Jun 2010 14:39:59 +0000 (UTC) Hi, the only kernel changes from 8.0p2 to 8.0p3 are related to NFS imho. You are not also using NFS with this machine? If you are not, this is likely due to some changed timing or similar and you are just hitting generel problem that most likely is entirely unrelated to jails and would either be a pf@ or a net@ kernel issue. /bz From: To: , Cc: Subject: RE: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot Date: Thu, 3 Jun 2010 21:08:44 +0200 Hi Bjoern, I am not using NFS. I received an security announcement concerning jails (FreeBSD-SA-10:04.jail) which was fixed in 8.0p3. That's why I assumed that there were kernel changes for the jail system. Then can be a wrongful assumption of my part. You can be entirely right that this has nothing the to do with the jail system or updating to 8.0p3. But this is the first time I ran into this kernel panic. It started after my first reboot with the 8.0p3 kernel. It panics every time my first jail boots and always due to the ping proces. I did not change my system between 8.0p2 and 8.0p3. No hardware changes/firmware updates or changes to pf.conf or rc.conf. I only kept my ports up to date. I feel it is linked to 8.0p3 but it might indeed be a net/pf issue. Kind regards, Tom From: pred@telenet.be To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot Date: Wed, 1 Sep 2010 09:55:47 +0200 (CEST) ------=_Part_235589_241905550.1283327747791 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Just a quick note that I've upgraded this server to 8.1-RELEASE and re-enab= led the PF routing rule that was causing the kernel panic.=20 The server did a clean reboot so it seems this problem is no longer present= in 8.1-RELEASE, which is ofcourse=C2=A0good news.=20 ------=_Part_235589_241905550.1283327747791-- State-Changed-From-To: open->closed State-Changed-By: linimon State-Changed-When: Mon Sep 6 07:18:03 UTC 2010 State-Changed-Why: Submitter notes that this seems to be fixed on 8.1. http://www.freebsd.org/cgi/query-pr.cgi?pr=147162 >Unformatted: