feature request: record build space used

[Joe Marcus Clarke]

On Sat, 2008-07-26 at 12:37 +0300, Ion-Mihai Tetcu wrote:
> On Thu, 24 Jul 2008 22:11:06 -0400
> Joe Marcus Clarke <marcus at marcuscom.com> wrote:
> 
> > On Wed, 2008-07-23 at 21:46 +0300, Ion-Mihai Tetcu wrote:
> > > On Wed, 23 Jul 2008 12:18:16 -0400
> > > Wesley Shields <wxs at FreeBSD.org> wrote:
> > > 
> > >  On Wed, Jul 23, 2008 at 12:11:48PM -0400, Joe Marcus Clarke wrote:
> > > > > On Wed, 2008-07-23 at 13:59 +0300, Ion-Mihai Tetcu wrote:
> > > > > > Yeh, an other one ;)
> > > > > > 
> > > > > > 
> > > > > > It would be very useful to me to have, for each port, stored
> > > > > > in the database the total size of the space it occupies,
> > > > > > together with base OS and dependencies, at build time. This
> > > > > > value should be updated, if bigger that current values, even
> > > > > > if the build fails.
> > > > > 
> > > > > How would one know the size before the port builds?
> > > > > Dependencies are one thing, but we have no concept of size
> > > > > (even distfile size) before buildscript is called.
> > > > 
> > > > You can get a rough idea of it based upon the previous build?
> > > > Take the previous build and add on N% of space for growth?
> > > 
> > > Exactly. While it might fail for a few ports that suddenly grow, it
> > > will work for 99%.
> > > 
> > > From my educated guess:
> > > - for at least 50% of the ports tindy is spending more time untaring
> > > the jail that doing anything else. 
> > > - for 95% of the ports 2GB of RAM-based build space is more that
> > > enough;
> > > 
> > > What I'd like is to be able to have everything in RAM for those 95%
> > > without maintaining an exceptions list by hand. For the current
> > > hardware QA Tindy is running on I should get at least 35% "speed"
> > > improvement for a full run. (The jail tarball is cached by the fs in
> > > memory for me).
> > 
> > Any word on my follow up questions?  I am interested in implementing
> > these features, but I need more details.
> 
> Yeh, sorry, tough days.
> 
> Let me try to rephrase it:
> What I want to know is how much space is required to build a port so I
> can have all that done in memory-backed file system.
> 
> If I'm understanding the way tindy works correctly that space is:
> - the untar'ed jail
> - the untar'ed dependecies
> - the untar'ed sources of the port to be built
> - the results of the build process (.o files, binaries, docs, etc.)
> 
> The distfiles and packages are kept outside the build dir (which looks
> something like /TINDERBOX_ROOT/BUILD_NAME) so they shouldn't be counted
> in the necessary space.

But they're not.  Everything needed to install a port is copied to the
chroot ({pb}/BUILD) during pre-setup.  This way, buildscript can install
all the dependent packages, build the port, install it, then create the
package.  So the total space required will be sizeof(jail + dependency
tbzs + dependencies installed + port built + port installed + package).

> 
> The way I see it implemented:
> - record the space size for the last build if successful.
> - record the space size of the last build if unsuccessful and bigger
> that the one currently recorded.
> - add a ports_fail_pattered to look for 'out of space'
> - have the recorded space exported in prePortBuild hook so that one
> could use his favorite mount command for /TINDERBOX_ROOT/BUILD_NAME

All of this is doable, but we need to come to an understanding of what
is counted toward total space.  If you agree with what I've said above,
then I'll start work today.

> 
> 
> Of course, this implementation will be far from perfect:
> - needed space may vary because of non-default OPTIONS, NOPORT*, etc.
> - it may change because of new/changed depedns
> - etc.

Agreed.

> 
> But it would give a good indication of the magnitude of space needed.

What about the other two requests (for OS info and dependencies)?  I
asked questions about those as well.

Joe

> 
> 
-- 
PGP Key : http://www.marcuscom.com/pgp.asc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: This is a digitally signed message part
URL: <http://marcuscom.com/pipermail/tinderbox-list/attachments/20080726/8b4123a7/attachment.bin>