No, I'm not dead...

[Joe Marcus Clarke]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ade Lovett wrote:
> ... just stupidly busy.
> 
> But I'm getting a little bit of time in between conference calls and  
> flying to start dinking around with tinderbox again.
> 
> First off, the bad news.
> 
> As the tinderbox system is currently architected, shoehorning in  
> clustering is going to be essentially impossible.  I've studied both  
> the tinderbox code, and the pointyhat code (from the clustering  
> angle) at length, and there are a number of design decisions that  
> were made (when clustering wasn't considered to be an option) within  
> TB that essentially precludes it from being extended in this manner.
> 
> Now for the good news.
> 
> There's still a lot of places where things can be cleaned up, made to  
> run a little better, remove duplication of code etc., and this is  
> where I'm concentrating my work.
> 
> 1.  Re-introduction of tinderd to the HEAD branch given the  
> clustering issues mentioned above.  This will at least bring back all  
> of the functionality currently in 2.x into 3.x, allowing for better  
> exposure of the code.  This has, for the most part, been done - I  
> just need to run some more tests before committing back.
> 
> 2.  The way in which both src/ and ports/ trees are pulled down prior  
> to use has been completely refactored.  The update command is now one  
> of CSUP, CVSUP, USER, and NONE.
> 
> In the first two cases, not only is a supfile written, but also a  
> jails/<name>/update.sh and portstrees/<name>/update.sh script.  In  
> the case of "USER", this calls a user-supplied update.sh script  
> within the relevant directory, which can basically do anything you  
> like (I use it to keep trees up-to-date with CVS and SVN for  
> example).  The "NONE" case is, well, for doing nothing.  Useful when  
> working with -RELEASE src codebases where you can set up the relevant  
> src/ and ports/ directories from the CD/FTP/whatever, and they remain  
> static.
> 
> This has resulted in a bunch of duplicated code being removed, and  
> the update process becoming considerably simplified, with none of the  
> special casing for CVSUP that was present previously.
> 
> Whilst trees will have to be rebuilt, or subtly hacked (with  
> appropriate update.sh scripts slid into place), in terms of  
> commandline options, this is completely backwards compatible with  
> what we have now.
> 
> Again, once I'm happy with this, it'll get committed.
> 
> 
> 3. Is likely to generate a little more in the way of discussion.
> 
> I'm proposing to change the core building blocks of the tinderbox as  
> follows:
> 
> (a) a new object "Tree", which handles both ports/ and src/ trees.   
> This replaces PortsTree and detaches the srctree from the jail.  It  
> allows for a single src/ tree to be shared across multiple jails,  
> though is not limited to that.  Creation of a jail would then become:
> 
> 	createTree -n 6.x -t src ...
> 	createJail -n 6-amd64 -s 6.x ...
> 
> though it will be trivial to provide simple wrappers and enhancements  
> to have createJail automatically call createTree to preserve the  
> existing behavior of one jail, one src/
> 
> (b) renaming of the "Jail" object to "BinDist".  This serves a number  
> of purposes:
> - removal of confusion between tinderbox jails and jail(8)
> - more closely aligned with the pointyhat naming conventions
> - all for the ability to generate BinDists from snapshots provided on  
> CD/FTP/etc, without having to go through all that pesky compilation  
> part.

I'm okay with all of these points.  The terminology could probably do
with some clarification.

> 
> 
> 4.  This is a little further down the line, but there's the option to  
> remove the "Host" table, along with all the associated code hackery.   
> Current host management seems to be, at best, a little confused -  
> it's certainly obfuscating the code for what appears to me to be very  
> little gain.  More discussion on this welcome.
> 
> 
> 5.  Extension of the web interface to allow for all tinderbox  
> operations to be carried out (such as srctree, portstree and bindist  
> creation), etc.. etc..   This also ties in nicely with...

This would be nice.  I think a lot of users would like this.

> 
> 6.  Providing state information about each component.  For example,  
> right now, it's entirely possible to update a build's port and/or  
> jail whilst a long-running package building session is going on,  
> resulting in a fair amount of hilarity (particularly for ports tree  
> updates).  Extending things such that reference counts are kept of  
> the various objects, and thus preventing situations such as above,  
> are a necessity.

Nice to have, but not quickly required.  Hopefully TB administrators are
smart enough to avoid such foot shooting.

> 
> 7.  The *tough* one.  Throwing away the current Makefile-based method  
> of generating packages, and maintaining dependency and other  
> information directly within the SQL datastore.  This *may* open up a  
> form of mini-clustering, but certainly not to the scales of pointyhat.

I'm very leery about doing this.  Pointyhat still uses make, and is
clusterable.  If we move away from make, we have to be damn sure that
dependencies are still handled the same way as on pointyhat.
Additionally, this might cost us in the performance realm.

I think your order of priorities are good.

Joe


- --
PGP Key : http://www.marcuscom.com/pgp.asc
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEsqJ9b2iPiv4Uz4cRAj6/AJwKRhO35tiU1AoBff5t4TFHNHRKdACgqc7L
Dj6fUf63moluCA5JCLFvPqE=
=qoXX
-----END PGP SIGNATURE-----