From cperciva@hexahedron.daemonology.net Thu Aug 5 06:19:21 2004 Return-Path: Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6A54D16A4CF for ; Thu, 5 Aug 2004 06:19:21 +0000 (GMT) Received: from hexahedron.daemonology.net (S0106006067227a4a.vc.shawcable.net [24.87.233.42]) by mx1.FreeBSD.org (Postfix) with SMTP id 4EBE143D1D for ; Thu, 5 Aug 2004 06:19:20 +0000 (GMT) (envelope-from cperciva@hexahedron.daemonology.net) Received: (qmail 29672 invoked by uid 0); 5 Aug 2004 06:18:47 -0000 Message-Id: <20040805061847.29655.qmail@hexahedron.daemonology.net> Date: 5 Aug 2004 06:18:47 -0000 From: Colin Percival Reply-To: Colin Percival To: FreeBSD-gnats-submit@freebsd.org Cc: Subject: `make readmes` is gratuitously slow X-Send-Pr-Version: 3.113 X-GNATS-Notify: >Number: 70018 >Category: ports >Synopsis: [patch] `make readmes` is gratuitously slow >Confidential: no >Severity: non-critical >Priority: low >Responsible: portmgr >State: closed >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Thu Aug 05 06:20:24 GMT 2004 >Closed-Date: Fri Nov 19 13:49:03 GMT 2004 >Last-Modified: Fri Nov 19 13:49:03 GMT 2004 >Originator: Colin Percival >Release: FreeBSD 5.2.1-SECURITY i386 >Organization: >Environment: System: FreeBSD hexahedron.daemonology.net 5.2.1-SECURITY FreeBSD 5.2.1-SECURITY #0: Wed May 26 04:19:54 GMT 2004 root@builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386 >Description: `make readmes` recursively runs `make readme` in every port directory. This involves parsing around 5000 lines of Makefile code several times per port. Piping INDEX through a perl script is much faster. >How-To-Repeat: >Fix: This patch adds a perl script "Tools/make_readmes", and modifies bsd.port.subdir.mk to avoid recursing into individual port directories; instead, it runs the perl script at the top level, after the category README.html files have been created. This could be improved further by creating the category README.html files via perl script, but this would be slightly more complicated due to the category descriptions not being pre-indexed. --- make_readmes.diff begins here --- Index: Mk/bsd.port.subdir.mk =================================================================== RCS file: /usr/cvsroot/ports/Mk/bsd.port.subdir.mk,v retrieving revision 1.55 diff -u -r1.55 bsd.port.subdir.mk --- Mk/bsd.port.subdir.mk 14 Jul 2004 08:18:16 -0000 1.55 +++ Mk/bsd.port.subdir.mk 5 Aug 2004 06:03:35 -0000 @@ -236,7 +236,13 @@ .endif .if !target(readmes) +.if defined(PORTSTOP) readmes: readme ${SUBDIR:S/^/_/:S/$/.readmes/} + @${ECHO_MSG} "===> Creating README.html for all ports" + @perl ${.CURDIR}/Tools/make_readmes < ${.CURDIR}/${INDEXFILE} +.else +readmes: readme +.endif .endif .if !target(readme) --- /dev/null Wed Aug 4 23:00:00 2004 +++ Tools/make_readmes Wed Aug 4 21:45:05 2004 @@ -0,0 +1,47 @@ +#!/usr/bin/perl + +$README=`cat Templates/README.port`; + +while(<>) { + split '\|'; + + $PKG=$_[0]; + $PORT=$_[1]; + $COMMENT=$_[3]; + $DESCR=$_[4]; + $EMAIL=$_[5]; + $BUILD_DEPENDS=$_[7]; + $RUN_DEPENDS=$_[8]; + $WEBSITE=$_[9]; + + $DESCR=~s|^\Q$PORT/\E||; + $PORT=~s|`pwd`||; + + if($WEBSITE) { + $WEBSITE=" and/or visit the web site for futher informations" + }; + if($BUILD_DEPENDS) { + $BUILD_DEPENDS="This port requires package(s) \"$BUILD_DEPENDS\" to build." + }; + if($RUN_DEPENDS) { + $RUN_DEPENDS="This port requires package(s) \"$RUN_DEPENDS\" to run." + }; + + $TOP=$PORT; + $TOP=~s|[^/]+|..|g; + + $tmp=$README; + $tmp=~s|%%PKG%%|$PKG|g; + $tmp=~s|%%PORT%%|$PORT|g; + $tmp=~s|%%COMMENT%%|$COMMENT|g; + $tmp=~s|%%DESCR%%|$DESCR|g; + $tmp=~s|%%EMAIL%%|$EMAIL|g; + $tmp=~s|%%WEBSITE%%|$WEBSITE|g; + $tmp=~s|%%BUILD_DEPENDS%%|$BUILD_DEPENDS|g; + $tmp=~s|%%RUN_DEPENDS%%|$RUN_DEPENDS|g; + $tmp=~s|%%TOP%%|$TOP|g; + + open F,">$PORT/README.html"; + print F $tmp; + close F +} --- make_readmes.diff ends here --- >Release-Note: >Audit-Trail: Responsible-Changed-From-To: freebsd-ports-bugs->portmgr Responsible-Changed-By: cperciva Responsible-Changed-When: Thu Aug 5 06:24:39 GMT 2004 Responsible-Changed-Why: This is a port infrastructure issue. http://www.freebsd.org/cgi/query-pr.cgi?pr=70018 State-Changed-From-To: open->feedback State-Changed-By: krion State-Changed-When: Fri Nov 5 12:00:08 GMT 2004 State-Changed-Why: Asked for feedback. http://www.freebsd.org/cgi/query-pr.cgi?pr=70018 From: Kirill Ponomarew To: Colin Percival Cc: FreeBSD-gnats-submit@FreeBSD.org Subject: Re: ports/70018: `make readmes` is gratuitously slow Date: Fri, 5 Nov 2004 12:59:56 +0100 --o0ZfoUVt4BxPQnbU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, On Thu, Aug 05, 2004 at 06:18:47AM -0000, Colin Percival wrote: > This patch adds a perl script "Tools/make_readmes", and modifies > bsd.port.subdir.mk to avoid recursing into individual port directories; > instead, it runs the perl script at the top level, after the category > README.html files have been created. >=20 > This could be improved further by creating the category README.html > files via perl script, but this would be slightly more complicated due > to the category descriptions not being pre-indexed. In that case this patch slightly destroys the sense of "make readmes", since it creates README only at the top level of category without going in every single port, links in these README.html files are not valid, because README.html doesn't exist for a single port. -Kirill --o0ZfoUVt4BxPQnbU Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQFBi2s8QC1G6a60JuURAgocAKDZEgkhOaXtI9scYaLKwWBREHOXtQCePibv AFdpVufVUoIRi+IbQI8mdEA= =My4U -----END PGP SIGNATURE----- --o0ZfoUVt4BxPQnbU-- State-Changed-From-To: feedback->open State-Changed-By: krion State-Changed-When: Fri Nov 5 13:18:14 GMT 2004 State-Changed-Why: Provided patch is correct. http://www.freebsd.org/cgi/query-pr.cgi?pr=70018 From: Kirill Ponomarew To: Colin Percival Cc: FreeBSD-gnats-submit@FreeBSD.org Subject: Re: ports/70018: `make readmes` is gratuitously slow Date: Fri, 5 Nov 2004 14:13:53 +0100 --rV8arf8D5Dod9UkK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, On Fri, Nov 05, 2004 at 12:00:56PM +0000, Kirill Ponomarew wrote: > In that case this patch slightly destroys the sense of "make > readmes", since it creates README only at the top level of category > without going in every single port, links in these README.html > files are not valid, because README.html doesn't exist for a single > port. Sorry, I misunderstood the patch, it works correctly after I tried "readmes" target. So it seems good, and is really faster now to build readmes files. -Kirill --rV8arf8D5Dod9UkK Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQFBi3yRQC1G6a60JuURAnCJAKDmXkC/uq7caI2z/A4Tb6TOIy25VACgkynW xZDTiO2lWBZhbpkpnbFYT+U= =qGAv -----END PGP SIGNATURE----- --rV8arf8D5Dod9UkK-- State-Changed-From-To: open->closed State-Changed-By: krion State-Changed-When: Fri Nov 19 13:48:59 GMT 2004 State-Changed-Why: Committed, thanks! http://www.freebsd.org/cgi/query-pr.cgi?pr=70018 >Unformatted: