From nobody@FreeBSD.org Mon Feb 27 06:42:09 2006 Return-Path: Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8CD5216A420 for ; Mon, 27 Feb 2006 06:42:09 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id 33C1643D48 for ; Mon, 27 Feb 2006 06:42:09 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id k1R6g8IS025892 for ; Mon, 27 Feb 2006 06:42:08 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id k1R6g8VT025891; Mon, 27 Feb 2006 06:42:08 GMT (envelope-from nobody) Message-Id: <200602270642.k1R6g8VT025891@www.freebsd.org> Date: Mon, 27 Feb 2006 06:42:08 GMT From: Arthur Hartwig To: freebsd-gnats-submit@FreeBSD.org Subject: cpu_spinwait calls missing X-Send-Pr-Version: www-2.3 >Number: 93887 >Category: kern >Synopsis: [kernel] cpu_spinwait calls missing in subr_smp.c >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Feb 27 06:50:04 GMT 2006 >Closed-Date: >Last-Modified: Tue Feb 28 09:25:05 GMT 2006 >Originator: Arthur Hartwig >Release: 6.1-BETA1 >Organization: Nokia >Environment: oz-net-11# uname -a FreeBSD oz-net-11.nes.nokia.com 6.1-BETA1 FreeBSD 6.1-BETA1 #4: Tue Feb 21 14:01:31 EST 2006 root@oz-net-11.nes.nokia.com:/usr/src/sys/amd64/compile/oz-net-11 amd64 >Description: There are five busy wait loops in kern/subr_smp.c which would be more "multi-processor friendly" on i386 and amd64 architectures if they included a call to cpu_spinwait(). On these architectures the cpu_spinwait() executes a pause instruction which the Intel IA32 architecture manual says "improves the performance of spin wait loops." Loops which could include the call to cpu_spinwait() occur in stop_cpus(), stop_cpus_nmi(), restart_cpus(), and two in smp_rendezvous_action() >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: From: Kris Kennaway To: Arthur Hartwig Cc: freebsd-gnats-submit@FreeBSD.org Subject: Re: misc/93887: cpu_spinwait calls missing Date: Mon, 27 Feb 2006 18:29:44 -0500 Why would you want to do this in stop_cpus(), stop_cpus_nmi() and restart_cpus()? They're used e.g. as preparation to break into/out of the debugger, and AFAICT they're not in the critical path for anything. Not sure about smp_rendezvous_action(), maybe that one is OK. Kris From: Arthur Hartwig To: ext Kris Kennaway Cc: freebsd-gnats-submit@FreeBSD.org Subject: Re: misc/93887: cpu_spinwait calls missing Date: Tue, 28 Feb 2006 12:32:08 +1000 >Why would you want to do this in stop_cpus(), stop_cpus_nmi() and >restart_cpus()? They're used e.g. as preparation to break into/out of >the debugger, and AFAICT they're not in the critical path for >anything. > >Not sure about smp_rendezvous_action(), maybe that one is OK. > >Kris > Good question which didn't occur to me because I was coming from another context. I was looking at using these functions to support a particular piece of "hotswap" hardware. The hardware generates an interrupt to say a PCI device has been removed (or plugged in). In the removal case some work needs to be done to remap to main memory the memory address space assigned to the device being removed so that if a device driver is accessing the device's memory space it won't cause the system to die. On a UP system this is "easy" in that because the hotswap interrupt is running we know the driver for the removed device is not currently running and even if it was running before the interrupt by the time it runs again the memory remapping will have occurred and it will be safe for the driver to access the memory space previously assigned to the removed device.However on a MP system the driver could be executing concurrently on another CPU and cause the system to die by accessing memory that is no longer present. I thought of using these functions (stop_cpus() etc) to momentarily "pause" all other CPUs while adjusting the memory mapping. It would be of advantage to cause other CPUs to pause as quickly as possible to minimise the likelihood that the driver for the removed device will attempt to access memory on the removed device. I have not been able to find any data on how long I have between the interrupt to say a device is being removed and the actual removal of the device. Nor have I any data on how long a busy CPU of a hyperthreaded pair (for example) is able to "lock out" its hyperthreaded mate. The referenced functions are clearly adequate for their existing use by the debugger so I accept that I might be better off making a customised copy. I still have doubts about the effectiveness of the approach using stop_cpus() and co and I'm inclined to see if I can get away with saying "you can't pull out a board untill you have disabled all the devices on that board" but this is somewhat restricted compared with what users have become accustomed to. Arthur >Unformatted: