- developers-list - monetdb.org

[Monetdb-developers] BATpreload_ and _BATpreload
by Sjoerd Mullender 15 Feb '10

15 Feb '10

I am quite unhappy with the names BATpreload, BATpreload_, and _BATpreload. I really don't like names that differ only by an initial or final _. I would much rather see more descriptive names. BATpreload_ is now more an interface to give advice to the underlying system about how the BAT is going to be accessed in the near future, and _BATpreload is telling the underlying system that you're done with accessing the BAT according to the advice given. Names such as BATaccessadvice/BATendaccessadvice, though longer, are more descriptive. I have seen some checkins fixing the incorrect use of BATpreload_ and _BATpreload, proofing my point. Now we still have a chance to fix this. -- Sjoerd Mullender

4 7

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_posix.mx, Feb2010, 1.176.2.14, 1.176.2.15
by Stefan Manegold 15 Feb '10

15 Feb '10

On Sun, Feb 14, 2010 at 09:23:10PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv28118 > > Modified Files: > Tag: Feb2010 > gdk_posix.mx > Log Message: > Move the load advice outside the lock. FYI: This checkin also changes the semantics of MT_mmap_inform() such that, now, (1) with preload > 0, posix_madvise(base, len, advise) is also called on areas that are not maintained in MT_mmap_tab[] (2) with preload < 0, posix_madvise(base, len, BUF_NORMAL) is never called on areas that are not maintained in MT_mmap_tab[] Was this intended? Stefan > > Index: gdk_posix.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_posix.mx,v > retrieving revision 1.176.2.14 > retrieving revision 1.176.2.15 > diff -u -d -r1.176.2.14 -r1.176.2.15 > --- gdk_posix.mx 14 Feb 2010 13:17:50 -0000 1.176.2.14 > +++ gdk_posix.mx 14 Feb 2010 21:22:29 -0000 1.176.2.15 > @@ -884,7 +884,7 @@ > void > MT_mmap_inform(void *base, size_t len, int preload, int advise, int writable) > { > - int i; > + int i, unload =0; > > (void) pthread_mutex_lock(&MT_mmap_lock); > i = MT_mmap_idx(base, len); > @@ -892,12 +892,13 @@ > if (writable) MT_mmap_tab[i].writable = (writable > 0); > MT_mmap_tab[i].random += preload * (advise == BUF_WILLNEED); /* done as a counter to keep track of multiple threads */ > MT_mmap_tab[i].usecnt += preload; /* active thread count */ > - if (preload > 0) > - (void) posix_madvise(base, len, advise); > - else if (MT_mmap_tab[i].usecnt == 0) > - (void) posix_madvise(base, len, BUF_NORMAL); > + unload = MT_mmap_tab[i].usecnt == 0; > } > (void) pthread_mutex_unlock(&MT_mmap_lock); > + if (preload > 0) > + (void) posix_madvise(base, len, advise); > + else if (unload) > + (void) posix_madvise(base, len, BUF_NORMAL); > > } > > > > ------------------------------------------------------------------------------ > SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, > Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW > http://p.sf.net/sfu/solaris-dev2dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/kernel group.mx, Feb2010, 1.146.2.14, 1.146.2.15
by Martin Kersten 15 Feb '10

15 Feb '10

Stefan Manegold wrote: > when BATpreloading the sequentially accessed outer BAT, > make sure we also BATpreload the (often) randomly > accessed "inner" BAT We probably also have to inject advice for the target variable(s), because it is likely to be used in the future a BUF_WILLNEED might be appropriate to avoid early eviction. This advice should be given when we create them, e.g. in BATnew?

1 0

[Monetdb-developers] cracking pipeline
by Isidor Zeuner 13 Feb '10

13 Feb '10

Dear MonetDB developers, I'd like to write some words of gratitude for your work on the cracking code. I have read your paper describing the approach, which looks promising. And current code already seems to be somehow stable (I could get it to segfault using integer equality, and got wrong results using integer comparisons, but both were easy to work around .. nothing serious compared with the MALexception whack-a-mole I experienced with mitosis_pipe). Is there a roadmap somewhere for this part of the code, so I could see what to expect, or where to test/patch/contribute most appropriately? Best regards, Isidor Zeuner

1 0

[Monetdb-developers] working with the SQL engine's MAL explanations
by Isidor Zeuner 13 Feb '10

13 Feb '10

Dear MonetDB developers, when using the EXPLAIN keyword within "mclient --language=sql", I get output like: % .explain # table_name % mal # name % clob # type % 0 # length function user.s0_2{autoCommit=true}():void; .. .. Most notably, apart from the lines starting with %, the syntax resembles the code "mclient --language=mal" would expect. So, a naive approach to use this observation would be to filter out the % lines and feed the output together with some boilerplate code into "mclient --language=mal" (example using a bash unix shell): echo "EXPLAIN SELECT 1;"| mclient --language=sql| ( echo "sql.init();" grep -v ^% echo "user.s0_2();" )| mclient --language=mal For very simple SQL statements, this works and creates the expected output. However, as soon as tables come into play, I'm getting errors because of uninitialized variables. So there seem to be at least semantical differences between EXPLAIN's MAL output and the MAL "mclient --language=mal" expects. Is this difference documented somewhere? Is there a better way for ad-hoc translation from executable SQL to executable MAL? Best regards, Isidor Zeuner

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk.mx, Feb2010, 1.301.2.9, 1.301.2.10 gdk_bat.mx, Feb2010, 1.236.2.4, 1.236.2.5 gdk_batop.mx, Feb2010, 1.180.2.15, 1.180.2.16 gdk_bbp.mx, Feb2010, 1.269.2.5, 1.269.2.6 gdk_posix.mx, Feb2010, 1.176.2.11, 1.176.2.12 gdk_relop.mx, Feb2010, 1.179.2.13, 1.179.2.14 gdk_storage.mx, Feb2010, 1.149.2.21, 1.149.2.22 gdk_utils.mx, Feb2010, 1.262.2.10, 1.262.2.11
by Stefan Manegold 12 Feb '10

12 Feb '10

Peter, > +/* a thread informs it is goin to (preload==1) or stops (preload==-1) using a range of memory */ > void > -MT_mmap_pin(void *base, size_t len) > +MT_mmap_inform(void *base, size_t len, int preload, int advise, int writable) > { > int i; > > (void) pthread_mutex_lock(&MT_mmap_lock); > i = MT_mmap_idx(base, len); > if (i >= 0) { > - MT_mmap_tab[i].pincnt++; > + if (writable) MT_mmap_tab[i].writable = (writable > 0); > + MT_mmap_tab[i].random += (advise == BUF_WILLNEED); /* done as a counter to keep track of multiple threads */ ^^^^^^^^^^^^^^^^^^^^^^^^ shouldn't this rather be something like MT_mmap_tab[i].random += preload * (advise == BUF_WILLNEED); ? Otherwise (i.e., in the current case), both BATpreload_() & _BATpreload() only increment MT_mmap_tab[i].random, but never decrement it, right? Stefan > + MT_mmap_tab[i].usecnt += preload; /* active thread count */ > + if (preload > 0) (void) posix_madvise(base, len, advise); > } > (void) pthread_mutex_unlock(&MT_mmap_lock); > > } -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 2

Re: [Monetdb-developers] [Monetdb-checkins] clients/src/mapiclient MapiClient.mx, , 1.194, 1.195
by Stefan de Konink 12 Feb '10

12 Feb '10

Op 12-02-10 18:01, Fabian Groffen schreef: > On 12-02-2010 16:37:28 +0000, Stefan de Konink wrote: >> Update of /cvsroot/monetdb/clients/src/mapiclient >> In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv30000 >> >> Modified Files: >> MapiClient.mx >> Log Message: >> Minor issue with save_history = false. > > you forgot a changelog entry for this feature. I didn't know there was a changelog ;) But it looks that that is branch related? And I just checked it into head? Stefan

2 1

[Monetdb-developers] @= atomtostr memory usage
by Stefan de Konink 11 Feb '10

11 Feb '10

Hi, While creating some new code and seeing unexpected, but interesting, behavior I stumbled upon the @= atomtostr code, some simple questions regarding this. It seems that the memory requested in @= atommem is rather pessimistic. It seems that for a regular integer 24 characters are allocated. Now since this integer seems to go up to 2^31 (10 chars) + sign + \0 (2 chars), why is the double amount needed if a per type approach is available in MAL? The actual toStr code finishes with a strlen, now this is nice behavior, but if we now know what the length of the string actually is, why not shoot a GDKrealloc on it, and make the *len the actual size? Regarding to memory management and memory function calling, the code isn't very efficient. Effectively all the functions seem to introduce their own buffers. If we look from a perspective of for example strConcat, the value is copied several times from serveral position. We could overcome this bahavior if we would return a maximum length for a type, the concat function would add the length of the second string alloc this buffer, forwards the pointer to the typical atomToStr call, and instead of allocing a new string, using the prealloced buffer. (...and yes, doing a GDKrealloc after that function wouldn't be good.) I took the liberty to come up with a patch for the second issue; Since snprintf will return the amount of bytes written, and since we have established that there is no numeric type overflowing the string space, we can ommit the strlen, use the amount of bytes + 1 from snprintf, realloc, and return the actual amount of bytes representing the string. Catches; 1) Old implementation resulted bestcase in an off by one return vs len 2) Is there any code dependent on the *len or that length will include the \0 value? If there are no objections, I'll check in that patch later today in head. >From benchmarking point of view, it might be worthwhile to test with and without the realloc. Stefan

2 2

[Monetdb-developers] confused by Timer/QueryHistory/CallHistory
by Mo Zhou 08 Feb '10

08 Feb '10

Hi, Sorry for bothering you again... I am confused by the time shown in Timer/queryHistory/callHistory. In queryHistory, there are two time: parse time and optimization time. In callHistory, there are two important time: xtime and rtime. These times should not overlap each other. but when I sum up these times, the number is different from the one returned by Timer. I wonder which time should I use for performance test. Thank you very much. -- Thanks, Mo

1 1

[Monetdb-developers] \f none not working
by Mo Zhou 08 Feb '10

08 Feb '10

Hi, I found "\f none" not working in XQuery. here is my commands: >\f none >\f Current formatter: none >for $t in doc("...")//Product return <pid> {pf:nid($t)} <XQueryResult>.....</pid><pid>1711511</pid></XQueryResult> >\f >Current formatter: none The results should not be printed according to the manual. "none: Finally, the none mode suppresses any output. This mode is mostly useful for performance measurements, if one wants to exclude the result serialization time." Could you please let me know if there are any solutions? Thanks! -- Thanks, Mo

1 0