- developers-list - monetdb.org

[Monetdb-developers] FreeBSD port of MonetDB Feb2009-SP1 release
by George Cox 13 Apr '09

13 Apr '09

Hi, I've just finished my first attempt at creating some ports files for the installation of MonetDB on FreeBSD. I've put them up at: http://www.gjvc.com/software/freebsd/ports/monetdb/ Please have a look and let me know what you think. Things to do include (but not limited to :-)) are: * checking the packing list for each component * adding rc.d scripts * adding monetdb user best; gjvc

2 3

[Monetdb-developers] Another big group by query
by Stefan de Konink 13 Apr '09

13 Apr '09

Hi All, Might be better to go by the mailinglist instead of adding many people to the CC. Currently I am running the latest CVS code disabled mitosis and am running the following code: CREATE TABLE "sys"."way_nds" ( "way" int NOT NULL, "idx" int NOT NULL, "to_node" int, CONSTRAINT "pk_way_nds" PRIMARY KEY ("way", "idx") ); select count(way), way, to_node from way_nds group by way, to_node having count(way) > 1; The table contains about 362420451 rows, and I realised too late there could be many results comming from it. top - 03:12:15 up 1 day, 17:28, 8 users, load average: 1.00, 1.00, 1.00 Tasks: 193 total, 1 running, 192 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 94.6%id, 5.2%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 66113464k total, 39353192k used, 26760272k free, 326968k buffers Swap: 134215032k total, 94284k used, 134120748k free, 8489128k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11856 konink 20 0 51.5g 34g 5.9g S 0.3 54.6 45:34.93 mserver5 But again the load is low, the memory empty and it seems to do so few. Is there something that would be interesting to look at? Stefan

3 7

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_bbp.mx, , 1.251, 1.252 gdk_utils.mx, , 1.244, 1.245
by Sjoerd Mullender 09 Apr '09

09 Apr '09

Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv27450 > > Modified Files: > gdk_bbp.mx gdk_utils.mx > Log Message: > gdk_utils.mx: > - performance fix: do not use malloc() on Linux either > its malloc() implementation appears to give sizable private arenas to > each new thread that performs significant malloc work; this leads > to huge memory fragmentation (especially when used with mitosis) > > gdk_bbp.mx: > - BBPtrim() should not be allowed to trim logical views > this may lead to BBPdecref errors!! Is this a bug fix that should also be applied to the stable branch? > > > U gdk_bbp.mx > Index: gdk_bbp.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_bbp.mx,v > retrieving revision 1.251 > retrieving revision 1.252 > diff -u -d -r1.251 -r1.252 > --- gdk_bbp.mx 31 Mar 2009 13:42:54 -0000 1.251 > +++ gdk_bbp.mx 9 Apr 2009 15:29:18 -0000 1.252 > @@ -1600,7 +1600,7 @@ > @{ > @h > #define BBPswappable(b) ((b) && BBP_refs((b)->batCacheid) == 0) > -#define BBPtrimmable(b) (BBPswappable(b) && !isVIEW(b) && (BBP_status((b)->batCacheid)&BBPWAITING) == 0) > +#define BBPtrimmable(b) (BBPswappable(b) && b->P->lview == 0 && isVIEW(b) == 0 && (BBP_status((b)->batCacheid)&BBPWAITING) == 0) > @- > The @%BBP_ref@ contains the amount of live references to a BAT. > These might be in recursive BATs, C or MIL variables. The count is > > U gdk_utils.mx > Index: gdk_utils.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > retrieving revision 1.244 > retrieving revision 1.245 > diff -u -d -r1.244 -r1.245 > --- gdk_utils.mx 7 Apr 2009 14:22:55 -0000 1.244 > +++ gdk_utils.mx 9 Apr 2009 15:29:18 -0000 1.245 > @@ -1053,7 +1053,6 @@ > #endif > } > size = (size + 7) & ~7; /* round up to a multiple of eight */ > -#ifndef HAVE_POSIX_FADVISE > if (size > GDK_mem_bigsize) { > size_t newsize = size + sizeof(size_t) + sizeof(size_t); > size_t newmax = MAX(*maxsize, newsize); > @@ -1068,7 +1067,6 @@ > *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); > return (void *) s; > } > -#endif > CHKMEM(size, 0); > GDKmalloc_prefixsize(s, size); > if (s == NULL) { > @@ -1205,7 +1203,6 @@ > } > } > #ifndef GDK_MEM_MISALIGN > -#ifndef HAVE_POSIX_FADVISE > else if (size <= GDK_mem_bigsize) { > void *oldblk = blk; > size_t newsize = size + 8; > @@ -1239,7 +1236,6 @@ > return blk; > } > } > -#endif > /* alloc&copy due to failed realloc (may be very big heap that needs vm) */ > #else > /* alloc&copy because we cannot trust realloc due to misalignment */ > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > http://p.sf.net/sfu/www-ibm-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

2 2

[Monetdb-developers] monetdb-install.sh & MAKEOPTS=-j
by Stefan Manegold 07 Apr '09

07 Apr '09

Fabian, `make -j[N] install` is know not to work (in particular with MonetDB5) due to race conditions and/or dependency issues, and there is hardly any way to fix this. Hence, setting MAKEOPTS=-j[N] before calling monetdb-install.sh might (will) result in an error during make install. I propose the following (or similar) patch to omit -j[N] when calling make install (of course, the "simpler" but maybe less desireable option is to omit MAKEOPTS completely with make install ...): Index: monetweb/Assets/monetdb-install.sh =================================================================== RCS file: /ufs/monet/repository/monetweb/Assets/monetdb-install.sh,v retrieving revision 1.42 diff -u -r1.42 monetdb-install.sh --- monetweb/Assets/monetdb-install.sh 16 Mar 2009 20:27:42 -0000 1.42 +++ monetweb/Assets/monetdb-install.sh 7 Apr 2009 09:00:33 -0000 @@ -434,7 +434,7 @@ MAKE=make fi fi - ${MAKE} ${MAKEOPTS} "$@" > $QUIET + ${MAKE} "$@" > $QUIET return $? } @@ -502,8 +502,8 @@ echo " * Processing $p" > $QUIET echo $myconf > $QUIET eval "$myconf" > $QUIET || die "configuring $p failed" - emake || die "making $p failed" - emake install || die "installing $p failed" + emake ${MAKEOPTS} || die "making $p failed" + emake $(echo "${MAKEOPTS}" | sed 's|-j *[0-9]*||g') install || die "installing $p failed" popd > /dev/null done } @@ -533,7 +533,7 @@ ${DO_XQUERY+"--with-monetdb4=included"} \ ${DO_SQL+"--with-monetdb5=included"} \ ${DEBUG} - emake || die "making $p failed" + emake ${MAKEOPTS} || die "making $p failed" done } @@ -551,7 +551,7 @@ for p in ${A} ; do cd "${S}"/$p/$(get_cvs_module $p) || die "cannot find directory $p" - emake install || die "installing $p failed" + emake $(echo "${MAKEOPTS}" | sed 's|-j *[0-9]*||g') install || die "installing $p failed" done } Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.278, 1.279 gdk_bat.mx, , 1.213, 1.214 gdk_posix.mx, , 1.168, 1.169 gdk_relop.mx, , 1.161, 1.162 gdk_utils.mx, , 1.241, 1.242
by Martin Kersten 07 Apr '09

07 Apr '09

Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv31767/gdk > > Modified Files: > gdk.mx gdk_bat.mx gdk_posix.mx gdk_relop.mx gdk_utils.mx > Log Message: > various tweaks to the memory mangement > > - tune the GDK_mmap_minsize to vary between full RAM size and 128MB > (lowest when memory pressure is extreme) > > - in fetchjoin, when the optimization of string heap copying is applied > (instead of creating a new string heap and inserting into it), try > to share the heaps in virtual memory (rather than copying it) > This is possible with the logical view mechanism. > > now fetchjoin, batcopy and remap use this VM heap sharing. If we do a fetchjoin(L,R) to a table R that does not fit in memory, and the operand L has more oids then the number of pages in R then we should also sort L on the tail oids first This under the constraint that sorting L also cost some IO If you are tweaking in that area anyway, such a patch is appreciated. > > - disable vmalloc on Linux -- well actually on platforms > that do not have posix_fadvise. A bit of a hack. > >

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.278, 1.279 gdk_bat.mx, , 1.213, 1.214 gdk_posix.mx, , 1.168, 1.169 gdk_relop.mx, , 1.161, 1.162 gdk_utils.mx, , 1.241, 1.242
by Stefan Manegold 03 Apr '09

03 Apr '09

Peter, these changes lead to $ Mserver # MonetDB Server v4.29.0 # based on GDK v1.29.0 # Copyright (c) 1993-July 2008, CWI. All rights reserved. # Copyright (c) August 2008-2009, MonetDB B.V.. All rights reserved. # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. !ERROR: CMDfind: value not found. !ERROR: CMDfind: operation failed. !ERROR: interpret_params: int(param 1): evaluation error. MonetDB> In fact, Sjoerd fixed this yesterday in =================================================================== 2009/04/02 - sjoerd: MonetDB/src/gdk/gdk_utils.mx,1.241 Reintroduced monet_pid, gdk_mem_pagebits, and gdk_mem_bigsize into the MonetDB environment, but not through advertizing them as options. This should fix bug 2726337. =================================================================== Your checkin apparently removed this fix, again. Why? Stefan ps: page 14 of http://homepages.cwi.nl/~manegold/Presentations/MonetDBdevTTT.pdf says "Do never copy/overwrite files checked-out from CVS!" On Fri, Apr 03, 2009 at 09:14:55AM +0000, Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv31767/gdk > > Modified Files: > gdk.mx gdk_bat.mx gdk_posix.mx gdk_relop.mx gdk_utils.mx > Log Message: > various tweaks to the memory mangement > > - tune the GDK_mmap_minsize to vary between full RAM size and 128MB > (lowest when memory pressure is extreme) > > - in fetchjoin, when the optimization of string heap copying is applied > (instead of creating a new string heap and inserting into it), try > to share the heaps in virtual memory (rather than copying it) > This is possible with the logical view mechanism. > > now fetchjoin, batcopy and remap use this VM heap sharing. > > - disable vmalloc on Linux -- well actually on platforms > that do not have posix_fadvise. A bit of a hack. > > > > U gdk.mx > Index: gdk.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk.mx,v > retrieving revision 1.278 > retrieving revision 1.279 > diff -u -d -r1.278 -r1.279 > --- gdk.mx 13 Mar 2009 17:58:12 -0000 1.278 > +++ gdk.mx 3 Apr 2009 09:14:51 -0000 1.279 > @@ -1048,6 +1048,7 @@ > gdk_export int HEAPextend(Heap *h, size_t size); > gdk_export int HEAPload(Heap *h, str nme, str ext, int trunc); > gdk_export int HEAPsave(Heap *h, str nme, str ext); > +gdk_export int HEAPshare(Heap *dst, Heap *src, int copy_on_write); > gdk_export int HEAPcopy(Heap *dst, Heap *src); > gdk_export int HEAPwarm(Heap *h); > gdk_export size_t HEAPsubcopy(Heap *dst, Heap *src, size_t size, size_t offset); > > U gdk_posix.mx > Index: gdk_posix.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_posix.mx,v > retrieving revision 1.168 > retrieving revision 1.169 > diff -u -d -r1.168 -r1.169 > --- gdk_posix.mx 1 Apr 2009 12:37:27 -0000 1.168 > +++ gdk_posix.mx 3 Apr 2009 09:14:52 -0000 1.169 > @@ -824,6 +824,13 @@ > } > } > } > + /* could not save anything -- reset the save_tile to the start to try all heaps again */ > + for (i = MT_mmap_next(MT_mmap_cur); i != MT_mmap_cur; i = MT_mmap_next(i)) { > + if (MT_mmap_tab[i].fd >= 0 && > + MT_mmap_tab[i].writable && > + (MT_mmap_tab[i].pincnt == 0 || MT_mmap_tab[i].len > target)) > + MT_mmap_tab[i].save_tile = 1; > + } > MT_mmap_cur = i; > } > (void) pthread_mutex_unlock(&MT_mmap_lock); > > U gdk_relop.mx > Index: gdk_relop.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_relop.mx,v > retrieving revision 1.161 > retrieving revision 1.162 > diff -u -d -r1.161 -r1.162 > --- gdk_relop.mx 30 Mar 2009 07:30:13 -0000 1.161 > +++ gdk_relop.mx 3 Apr 2009 09:14:52 -0000 1.162 > @@ -972,7 +972,13 @@ > bn->theap->filename = (str) GDKmalloc(strlen(nme) + 12); > GDKfilepath(bn->theap->filename, NULL, nme, "theap"); > } > - if (HEAPcopy(bn->theap, r->theap) < 0) { > + if (r->batRestricted == BAT_READ && HEAPshare(bn->theap, r->theap, 0) == 0) { > + int parent = r->T->parentid; > + if (!parent) parent = -r->batCacheid; > + bn->P->lview = TRUE; > + bn->T->parentid = parent; > + BBPshare(parent); > + } else if (HEAPcopy(bn->theap, r->theap) < 0) { > BBPreclaim(bn); > goto ready; > } > > U gdk_utils.mx > Index: gdk_utils.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > retrieving revision 1.241 > retrieving revision 1.242 > diff -u -d -r1.241 -r1.242 > --- gdk_utils.mx 2 Apr 2009 08:44:03 -0000 1.241 > +++ gdk_utils.mx 3 Apr 2009 09:14:53 -0000 1.242 > @@ -1053,6 +1053,7 @@ > #endif > } > size = (size + 7) & ~7; /* round up to a multiple of eight */ > +#ifndef HAVE_POSIX_FADVISE > if (size > GDK_mem_bigsize) { > size_t newsize = size + sizeof(size_t) + sizeof(size_t); > size_t newmax = MAX(*maxsize, newsize); > @@ -1067,6 +1068,7 @@ > *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); > return (void *) s; > } > +#endif > CHKMEM(size, 0); > GDKmalloc_prefixsize(s, size); > if (s == NULL) { > @@ -1203,6 +1205,7 @@ > } > } > #ifndef GDK_MEM_MISALIGN > +#ifndef HAVE_POSIX_FADVISE > else if (size <= GDK_mem_bigsize) { > void *oldblk = blk; > size_t newsize = size + 8; > @@ -1236,6 +1239,7 @@ > return blk; > } > } > +#endif > /* alloc&copy due to failed realloc (may be very big heap that needs vm) */ > #else > /* alloc&copy because we cannot trust realloc due to misalignment */ > @@ -1446,7 +1450,9 @@ > } > } > > -#ifndef WIN32 > +#ifdef HAVE_POSIX_FADVISE > +static str highload_name[] = { "idle", "low", "medium", "high", "extreme" }; > + > static int membits = 0; > static void > GDKvmtrim(void *limit) > @@ -1458,21 +1464,25 @@ > /* compute rounded down log2 of memsize */ > size_t s = GDK_mem_maxsize_max; > while((s >>= 1) > 0) membits++; > + membits = (MAX(0,membits-27))/4; > } > - > do { > stream *fp = NULL; > - > MT_sleep_ms(highload ? 500 : 5000); > MEMDEBUG { > fp = GDKout; > - THRprintf(fp, "#GDKvmtrim(" LLFMT ")\n", (long long) *(size_t *) limit); > + THRprintf(fp, "#GDKvmtrim(load=%s, rsstarget=" SZFMT ", GDK_mmap_minsize=" SZFMT ")\n", > + highload_name[highload], *(size_t *) limit, GDK_mmap_minsize); > } > highload = MT_mmap_trim(*(size_t *) limit, fp); > if (highload) { /* memcrunch distress varies from 1 to 4 */ > /* rss is above 0.75mem_maxsize!! start using mmap-on-tempfile */ > if (!bak_mmap_minsize) bak_mmap_minsize = GDK_mmap_minsize; > - GDK_mmap_minsize = ((size_t) 1) << (membits-(highload+highload)); /* HEAPalloc/extend > [256-16]MB uses it */ > + /* minsize = memsize iff highload = 0 > + * = ..in between.. iff highload = 1-3 > + * = 128MB iff highload = 4 > + */ > + GDK_mmap_minsize = ((size_t) 1) << (27 + membits*(4-highload)); > } else if (bak_mmap_minsize) { > GDK_mmap_minsize = bak_mmap_minsize; /* revert to default setting */ > bak_mmap_minsize = 0; > @@ -1490,7 +1500,6 @@ > char *p; > opt *n; > int i, j, nlen = 0; > - char buf[16]; > > /* some sanity checks (should also find if symbols are not defined) */ > assert(sizeof(char) == SIZEOF_CHAR); > @@ -1606,20 +1615,8 @@ > } > if (GDKgetenv_isyes("gdk_embedded") || GDKgetenv_isyes("embedded")) > GDKembedded = 1; > - if (GDKgetenv("gdk_mem_pagebits") == NULL) { > - snprintf(buf, sizeof(buf), "%d", GDK_mem_pagebits); > - GDKsetenv("gdk_mem_pagebits", buf); > - } > - if (GDKgetenv("gdk_mem_bigsize") == NULL) { > - snprintf(buf, sizeof(buf), SZFMT, GDK_mem_bigsize); > - GDKsetenv("gdk_mem_bigsize", buf); > - } > - if (GDKgetenv("monet_pid") == NULL) { > - snprintf(buf, sizeof(buf), "%d", (int) getpid()); > - GDKsetenv("monet_pid", buf); > - } > > -#ifndef WIN32 > +#ifdef HAVE_POSIX_FADVISE > if (!GDKembedded && GDK_vm_trim) { > MT_Id t; > > > U gdk_bat.mx > Index: gdk_bat.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_bat.mx,v > retrieving revision 1.213 > retrieving revision 1.214 > diff -u -d -r1.213 -r1.214 > --- gdk_bat.mx 30 Mar 2009 07:30:07 -0000 1.213 > +++ gdk_bat.mx 3 Apr 2009 09:14:52 -0000 1.214 > @@ -670,11 +670,9 @@ > as rcopy(). > @{ > @c > -static int > -heapcopy(Heap *dst, Heap *src, int *remap) > -{ > - if (*remap && src->storage == STORE_MMAP) { > - /* use copy-on-write mmap for isolatable copy */ > +int HEAPshare(Heap *dst, Heap *src, int copy_on_write) { > + /* use copy-on-write mmap for isolatable copy */ > + if (src->storage == STORE_MMAP) { > *dst = *src; > dst->base = (char *) -1; > dst->filename = GDKstrdup(src->filename); > @@ -682,16 +680,25 @@ > char path[PATHLENGTH]; > > GDKfilepath(path, BATDIR, dst->filename, NULL); > - dst->base = GDKmmap(path, MMAP_READ | MMAP_SEQUENTIAL | MMAP_COPY, 0, dst->maxsize); > + dst->base = GDKmmap(path, MMAP_READ | MMAP_SEQUENTIAL | (copy_on_write?MMAP_COPY:0), 0, dst->maxsize); > if (dst->base != (char *) -1) { > dst->copied = 1; > - dst->newstorage = dst->storage = STORE_PRIV; > + dst->newstorage = dst->storage = copy_on_write?STORE_PRIV:STORE_MMAP; > return 0; > } > GDKfree(dst->filename); > dst->filename = NULL; > } > } > + return -1; > +} > + > +static int > +heapcopy(Heap *dst, Heap *src, int *remap) > +{ > + if (*remap && HEAPshare(dst, src, *remap) == 0) { > + return 0; /* use a shared memory heap */ > + } > *remap = 0; > return HEAPcopy(dst, src); > } > @@ -2766,7 +2773,7 @@ > int b0, b1, b2 = 0, b3 = 0; > > /* copy-on-write isolated bats that change mode should be made independent */ > - if (b->P->lview && batunshare(b, "BATsetaccess") < 0) > + if (b->P->lview && newmode != BAT_READ && batunshare(b, "BATsetaccess") < 0) > return NULL; > > if (b->batSharecnt && newmode != BAT_READ) { > > > ------------------------------------------------------------------------------ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal mat.mx, , 1.25, 1.26
by Stefan Manegold 02 Apr '09

02 Apr '09

On Wed, Apr 01, 2009 at 07:59:56PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv25709 > > Modified Files: > mat.mx > Log Message: > Packing of BATs should avoid falling into the BATextend trap. > Preclaim enough space. Don't bother about the properties of the > result. That should be decided later. > This checking removes the only initialization of the pointer variable int *ret; which is then later dereferenced for assginment --- this might/will result in a segfault or other non-deterministic behavior. See details below. Stefan > > U mat.mx > Index: mat.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/mat.mx,v > retrieving revision 1.25 > retrieving revision 1.26 > diff -u -d -r1.25 -r1.26 > --- mat.mx 7 Jan 2009 14:15:13 -0000 1.25 > +++ mat.mx 1 Apr 2009 19:59:50 -0000 1.26 > @@ -134,22 +134,31 @@ > The pack operation could be quite expensive, because it > may create a really large BAT. > For the experiments we materialize the BAT in the first > -component. > +component. It should be extended upfront to accomodate > +all elements. > @c > str > MATpackInternal(MalStkPtr stk, InstrPtr p,int first) > { > int i,*ret; ^^^^ > BAT *b, *bn; > + BUN cap=0; > > bn= BATdescriptor(stk->stk[getArg(p,first)].val.ival); > - if( bn == NULL) > - throw(MAL, "mat.pack","Could access BAT"); > - if (BAThrestricted(bn) || BATtrestricted(bn)) { > + if (bn && (BAThrestricted(bn) || BATtrestricted(bn)) ){ > b = BATcopy(bn, bn->htype, bn->ttype, TRUE); > BBPunfix(bn->batCacheid); > bn = b; > } > + if( bn == NULL) > + throw(MAL, "mat.pack","Could not access BAT"); > + > + for(i=first; i<p->argc; i++){ > + b= BATdescriptor(stk->stk[getArg(p,i)].val.ival); > + cap += BATcount(b); > + BBPunfix(b->batCacheid); > + } > + bn = BATextend(bn, cap); > for(i=first+1; i<p->argc; i++){ > b= BATdescriptor(stk->stk[getArg(p,i)].val.ival); > if( b == NULL) > @@ -158,12 +167,6 @@ > BATins(bn,b,FALSE); > BBPunfix(b->batCacheid); > } > - if (bn->htype == TYPE_oid && !BAThordered(bn)) { > - BAT *b = BATsort(bn); > - BBPunfix(bn->batCacheid); > - bn = b; > - } > - ret= (int*) getArgReference(stk,p,0); ^ ^^^^ > BBPkeepref(*ret = bn->batCacheid); ^^^^ > return MAL_SUCCEED; > } > > > ------------------------------------------------------------------------------ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] MAL Instructions and a New Client Development
by Mohamed Saber 30 Mar '09

30 Mar '09

Hi All, I am new to MonetDB. I want to create new MonetDB commands (like the SQL) in my research project. So, I want a good reference/tutorial on the MAL instructions. After that I will do a compiler to compile my new commands to the equivalent MAL instructions. I think that the MonetDB_Manual is insufficient. If you have any reference or recommendations to achieve what I need, please let me know. I want to master the MAL language. Any help will be appreciated. Thank you so much and have a great day, Kind Regards, Mohamed Saber _________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

1 0

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/modules/pftijah serialize_pftijah.mx, Feb2009, 1.68.2.1, 1.68.2.2
by Stefan Manegold 30 Mar '09

30 Mar '09

On Mon, Mar 30, 2009 at 04:04:10PM +0000, Wouter Alink wrote: > Update of /cvsroot/monetdb/pathfinder/modules/pftijah > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv16148 > > Modified Files: > Tag: Feb2009 > serialize_pftijah.mx > Log Message: > -- step 1 in crossing trhe oid32 boundary with PF/TIJAH: remove some casts to int. > This commit will (partially) fix bug "2722117 PF/TIJAH: dbat_finalize fails". Cetero senseo: never (down-)cast unless it is strictly required and you have strictly ensured/asserted that the actual value of the (larger) source type at all times fits into the range of the (smaller) target type. In fact, I made the experiences that avoiding casts as far as possible (e.g.) by consistently using the "proper" types, keeps the code much cleaner, more maintainable and less error prone, giving the compilers a chance to warn us about potential over/underflows in case we accidently assign/use non-matching types ... ;-) Stefan ps: Just sharing my experiences and lessons I learned, no "blame" at all! ;-) > The fix was suggested by Henning/JanF, and seems to at least partially fix the indexing of a 30GB XML document. A new bug "2722117 PF/TIJAH: dbat_finalize fails" has been filed with the next issue. Bug "2722117" can be closed. No test can be created as the data is simply too large. > > > > U serialize_pftijah.mx > Index: serialize_pftijah.mx > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/modules/pftijah/serialize_pftijah.mx,v > retrieving revision 1.68.2.1 > retrieving revision 1.68.2.2 > diff -u -d -r1.68.2.1 -r1.68.2.2 > --- serialize_pftijah.mx 25 Mar 2009 13:17:56 -0000 1.68.2.1 > +++ serialize_pftijah.mx 30 Mar 2009 16:04:08 -0000 1.68.2.2 > @@ -982,7 +982,7 @@ > --tjctx->doc_height; > oid start = tj_popTag(tjctx); /* oid of the first node of the element */ > int size = tjctx->tijahPre - start - 1; /* the Tijah document size */ > - dbat_set_int(&tjctx->dbat_collSize,(int)start,size); > + dbat_set_int(&tjctx->dbat_collSize,start,size); > #else > (void) ctx; > #endif > @@ -1136,7 +1136,7 @@ > if ( DOEMIT(tjctx) ) { > oid start = tj_popTag(tjctx); /* oid of the first node of the element */ > int size = tjctx->tijahPre - start - 1; /* the Tijah element size */ > - dbat_set_int(&tjctx->dbat_collSize,(int)start,size); > + dbat_set_int(&tjctx->dbat_collSize,start,size); > }; > #ifdef TJ_TRACE > if ( TJ_TRACE ) stream_printf(GDKout,"C[%s]:empty_endElement: \"%s\"\n", tjctx->name,""); > @@ -1167,7 +1167,7 @@ > if ( DOEMIT(tjctx) ) { > oid start = tj_popTag(tjctx); /* oid of the first node of the element */ > int size = tjctx->tijahPre - start - 1; /* the Tijah element size */ > - dbat_set_int(&tjctx->dbat_collSize,(int)start,size); > + dbat_set_int(&tjctx->dbat_collSize,start,size); > #ifdef TJ_TRACE > if ( TJ_TRACE ) stream_printf(GDKout,"C[%s]:endElement: \"%s\"\n", tjctx->name,name); > #endif > > > ------------------------------------------------------------------------------ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/common monet_options.mx, , 1.32, 1.33
by Roberto Cornacchia 30 Mar '09

30 Mar '09

Oh thanks!! That is really handy! :) On Mon, 2009-03-30 at 14:51 +0000, Sjoerd Mullender wrote: > Update of /cvsroot/monetdb/MonetDB/src/common > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv4062/src/common > > Modified Files: > monet_options.mx > Log Message: > If the logdir argument to logger_create is a relative path name, the > logdir will be located inside the dbfarm/dbname directory. If it is > an absolute path name, that path is used. > > The default log directory for both SQL and XQuery have been changed > accordingly. > > This change puts the complete database in one location. -- | M.Sc. Roberto Cornacchia | CWI (Centrum voor Wiskunde en Informatica) | Science Park 123, 1098XG Amsterdam, The Netherlands | tel: +31 20 592 4322 , http://www.cwi.nl/~roberto

1 0