ports rebuilt w/o good reason

Joe Marcus Clarke marcus at marcuscom.com
Sun Nov 30 13:44:46 EST 2008 

On Sun, 2008-11-30 at 15:07 +0200, Ion-Mihai Tetcu wrote:
> Hi,
> 
> 
> For a long time, and especially since running QAT, I saw lots of ports,
> dependencies of the target port, being rebuilt with no apparent reason
> (a package for the current version of those ports existing).
> 
> This happens because of the way Tindy works: for a given target port it
> will generate a Makefile in BUILD/, linked in PGKDIR/BUILD/All/ which
> contains the dependencies of the target port, and instructions on how
> to build them and the target port.
> 
> An example Makefile for devel/cons-test is bellow (this is on a 2.x
> Tindy, but the mechanism it's the same on 3.x):
> 
> root at hive/SU >-SSH-> /home/itetcu [5:27:22] 0
>  # cat /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/Makefile
> all: cons-test-2.2.0_1.tbz
> cons-test-2.2.0_1: cons-test-2.2.0_1.tbz
> cons-test-2.2.0_1.tbz: perl-5.8.8_1.tbz p5-Digest-MD5-2.38.tbz cons-2.2.0_2.tbz
>         @/usr/local/tinderbox/scripts/portbuild 6-STABLE-i386-myPT 6 myPT -nullfs "perl-5.8.8_1.tbz" "perl-5.8.8_1.tbz" "" "perl-5.8.8_1.tbz" "perl-5.8.8_1.tbz p5-Digest-MD5-2.38.tbz cons-2.2.0_2.tbz" cons-test-2.2.0_1.tbz /usr/ports/devel/cons-test
> all: perl-5.8.8_1.tbz
> perl-5.8.8_1: perl-5.8.8_1.tbz
> perl-5.8.8_1.tbz:
>         @/usr/local/tinderbox/scripts/portbuild 6-STABLE-i386-myPT 6 myPT -nullfs "" "" "" "" "" perl-5.8.8_1.tbz /usr/ports/lang/perl5.8
> all: cons-2.2.0_2.tbz
> cons-2.2.0_2: cons-2.2.0_2.tbz
> cons-2.2.0_2.tbz: perl-5.8.8_1.tbz p5-Digest-MD5-2.38.tbz
>         @/usr/local/tinderbox/scripts/portbuild 6-STABLE-i386-myPT 6 myPT -nullfs "perl-5.8.8_1.tbz" "perl-5.8.8_1.tbz" "" "perl-5.8.8_1.tbz" "perl-5.8.8_1.tbz p5-Digest-MD5-2.38.tbz" cons-2.2.0_2.tbz /usr/ports/devel/cons
> all: p5-Digest-MD5-2.38.tbz
> p5-Digest-MD5-2.38: p5-Digest-MD5-2.38.tbz
> p5-Digest-MD5-2.38.tbz: perl-5.8.8_1.tbz
>         @/usr/local/tinderbox/scripts/portbuild 6-STABLE-i386-myPT 6 myPT -nullfs "perl-5.8.8_1.tbz" "perl-5.8.8_1.tbz" "" "perl-5.8.8_1.tbz" "perl-5.8.8_1.tbz" p5-Digest-MD5-2.38.tbz /usr/ports/security/p5-Digest-MD5
> 
> 
> So:
>  cons-test-2.2.0_1 depends on lang/perl5.8, security/p5-Digest-MD5 and
>                               devel/cons
>  cons-2.2.0_2 depends on lang/perl5.8, security/p5-Digest-MD5
> 
> When building a port Tindy is doing in tinderbuild_phase() a
> 'make all' in PGKDIR/BUILD/All/ using a Makefile like the one above.
> 
> all: depends on cons-test-2.2.0_1.tbz (which doesn't exist, since it's
> the target port of this run).
> cons-test-2.2.0_1.tbz: depends on perl-5.8.8_1.tbz,
> p5-Digest-MD5-2.38.tbz and cons-2.2.0_2.tbz
> 
> We now have 2 cases:
> 
> If one of those packages doesn't exist it is built using the
> corresponding portbuild instruction -- all OK
> 
> If however one of those packages does exist we wouldn't expect it to be
> rebuilt.
> 
> Now suppose that for cons-test build above perl-5.8.8_1.tbz,
> p5-Digest-MD5-2.38.tbz and cons-2.2.0_2.tbz allready exist (tindy built
> lang/perl5.8, security/p5-Digest-MD5 and devel/cons before this, so we
> have packages for the current version of this 3 ports). No reason to
> rebuild them for the current phase, right? WRONG!
> 
> This being a Makefile, the things written on the right side of ':' are
> considered "sources". If a source is newer (creation or modified time)
> that the target (the left side of ':') then the target is considered
> being out-of-date and make will rebuild it.
> 
> 
> If, for example, p5-Digest-MD5-2.38.tbz is newer that cons-2.2.0_2.tbz,
> since it's a source for cons-2.2.0_2.tbz, cons-2.2.0_2.tbz will be
> rebuilt when building cons-test-2.2.0_1.
> 
> In other words, if A -> B -> C -> D ( where '->' means depends),
> up-to-date packages exists for B, C and D, but for one reason or an
> other you rebuild D after it was built as a dependency of C, when you
> build A: C will also be rebuilt; and since now C is newer that B, B will
> also be rebuilt.
> So rebuilding D after B and C were built is equivalent for tindy's way
> of building ports to bumping PORTREVISION for C and B.
> 
> Now think of a long dependency line (like, let's say one of the myriad
> of X ports, or perl, or ...); rebuild it and Tindy will end up
> rebuilding all ports depending on it.
> 
> 
> This leads to 2 big problems:
> 1. Ports being rebuilt when there's no reason --> lost time.
> 2. Problems being masked by this behaviour (ports being rebuilt
> silently that maybe needed PORTREVISION bump or something).
> (I do hope this is not the way pointy does it; if it is it would
> explain a lot of the strange differences between using ports and
> official packages).

This is how pointyhat works.  I'm not sure pointyhat ever had a need not
to do this.  Most times partial builds are done are when PORTREVISION
changes at the very least.  And in your example (A -> B -> C -> D), if C
had a PORTREVISION change both B AND A would need to be rebuilt to keep
the dependency map correct.

As for introducing an option to disable this behavior, the only fix I
can think of is to use a pointed touch on all packages in the All
subdirectory.  For example:

now=$(date "+%Y%m%d%H%M%S")
find ${pkgDir}/All -type f -name "*${PKGSUFFIX}" | xargs touch -f -t
${now}

If you do something like that in tinderbuild() does that work for you?

Joe

> 
> 
> A reproduction of this bug is bellow:
> 
> root at hive/SU >-SSH-> /home/itetcu [5:09:11] 0
>  # ll /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/p5-Digest-MD5-2.38.tbz
> -rw-r--r--  1 root  wheel  20775 Nov 16 15:50 /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/p5-Digest-MD5-2.38.tbz
> root at hive/SU >-SSH-> /home/itetcu [5:09:23] 0
>  # ll /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/cons-2.2.0_2.tbz
> -rw-r--r--  1 root  wheel  116769 Nov 28 16:48 /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/cons-2.2.0_2.tbz
> root at hive/SU >-SSH-> /home/itetcu [5:09:27] 0
>  # ll /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/cons-test-2.2.0_1.tbz
> -rw-r--r--  1 root  wheel  79417 Nov 30 05:09 /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/cons-test-2.2.0_1.tbz
> root at hive/SU >-SSH-> /home/itetcu [5:09:45] 0
>  # touch /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/p5-Digest-MD5-2.38.tbz
> root at hive/SU >-SSH-> /home/itetcu [5:09:51] 0
>  # ll /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/p5-Digest-MD5-2.38.tbz
> -rw-r--r--  1 root  wheel  20775 Nov 30 05:09 /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/p5-Digest-MD5-2.38.tbz
> root at hive/SU >-SSH-> /home/itetcu [5:09:56] 0
>  # ~itetcu/bin/testport.sh devel/cons-test
> mount_nullfs: /usr/local/tinderbox/portstrees/myPT/ports (/usr/local/tinderbox/portstrees/myPT/ports) and /usr/local/tinderbox/portstrees/myPT/ports are not distinct paths
> mount_nullfs: /usr/local/tinderbox/jails/6 (/usr/local/tinderbox/jails/6) and /usr/local/tinderbox/jails/6/src are not distinct paths
> mount_nullfs: /usr/local/tinderbox/portstrees/myPT/ports (/usr/local/tinderbox/portstrees/myPT/ports) and /usr/local/tinderbox/portstrees/myPT/ports are not distinct paths
> mount_nullfs: /usr/local/tinderbox/jails/6 (/usr/local/tinderbox/jails/6) and /usr/local/tinderbox/jails/6/src are not distinct paths
> INFO: Creating build directory for 6-STABLE-i386-myPT from repository...
> ================================================
> building packages (phase 1)
> ================================================
> started at Sun Nov 30 05:10:22 CST 2008
> ended at Sun Nov 30 05:13:35 CST 2008
> phase 1 took 00:03:13
> 1285 packages built
> 3.4G  of packages
> ================================================
> building packages (phase 2)
> ================================================
> started at Sun Nov 30 05:13:35 CST 2008
> ended at Sun Nov 30 05:13:35 CST 2008
> phase 2 took 00:00:00
> 1285 packages built
> 3.4G  of packages
> tinderbuild: Cleaning up after tinderbuild.  Please be patient.
> 
> root at hive/SU >-SSH-> /home/itetcu [5:13:35] 0
>  # ll /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/p5-Digest-MD5-2.38.tbz
> -rw-r--r--  1 root  wheel  20775 Nov 30 05:09 /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/p5-Digest-MD5-2.38.tbz
> root at hive/SU >-SSH-> /home/itetcu [5:14:25] 0
>  # ll /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/cons-2.2.0_2.tbz
> -rw-r--r--  1 root  wheel  116749 Nov 30 05:11 /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/cons-2.2.0_2.tbz
> root at hive/SU >-SSH-> /home/itetcu [5:14:29] 0
>  # ll /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/cons-test-2.2.0_1.tbz
> -rw-r--r--  1 root  wheel  79398 Nov 30 05:13 /usr/local/tinderbox/packages/6-STABLE-i386-myPT/All/cons-test-2.2.0_1.tbz
> root at hive/SU >-SSH-> /home/itetcu [5:14:35] 0
>  #                                                                        
>                                                                               
> 
> _______________________________________________
> tinderbox-list at marcuscom.com mailing list
> http://marcuscom.com/mailman/listinfo/tinderbox-list
> To unsubscribe, send any mail to "tinderbox-list-unsubscribe at marcuscom.com"
-- 
PGP Key : http://www.marcuscom.com/pgp.asc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: This is a digitally signed message part
URL: <http://marcuscom.com/pipermail/tinderbox-list/attachments/20081130/0ff82028/attachment.bin>

More information about the tinderbox-list mailing list