- developers-list - monetdb.org

New Feb2013 stable branch created
by Sjoerd Mullender 29 Nov '12

29 Nov '12

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Feb2013. So in each of your stable checkout directories you can do hg pull hg update -rFeb2013 to update to this new branch. -- Sjoerd Mullender _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

Announcement: New Oct2012-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 21 Nov '12

21 Nov '12

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2012-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2012-SP1 bugfix release Bug Fixes * 3155: Provide an option for setting discovery to false * 3161: invalid tuple received from server, got 138 columns, expected 476, ignoring * 3182: When using aggregate SQL functions, precision & decimal metadata for numeric/decimal type returned is 0,0 _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

[Bug 3178] one scan is enough to implement ALGstdev_@1 in monetdb5/modules/kernel/algebra.mx
by Yinhe Cao 20 Nov '12

20 Nov '12

Hi Fabian, Thank you very much for your email. I had the same concern about the accumulation error of one-scan approach as you do, even though mathematically they should be exactly the same. I remember about 10 years ago, I did some experimentations on both of the approaches and I did not found any noticeable differences between the results, but the sample size I tried then is not huge. For the sample size of the order 1 trillion, I have the same concern even with the two-scan approach. For example, After 500 billion entries have been added to the sum of square, the rest may not be able to be added up anymore, since it may well with the range of accumulation errors! For a huge BAT, if the precision is top priority, we may need to use an hierarchical approach: divide the huge sample into some manageable block, compute it for each block first, and then aggregate the results. Thanks, Yinhe _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

Re: MonetDB: default - Cleanup code upon request.
by Fabian Groffen 15 Nov '12

15 Nov '12

On 15-11-2012 08:31:51 +0100, Martin Kersten wrote: > +/* There is just a single queue for the workers */ > static void > MRqueueCreate(int sz) > { > @@ -54,14 +55,21 @@ MRqueueCreate(int sz) > MT_lock_init(&mrqlock, "q_create"); > MT_lock_set(&mrqlock, "q_create"); > MT_sema_init(&mrqsema, 0, "q_create"); > - sz = ((sz << 1) >> 1); /* we want a multiple of 2 */ > + if ( mrqueue ) { > + GDKerror("One map-reduce queue allowed"); > + return; > + } > + sz *= 2; > mrqueue = (MRqueue *) GDKzalloc(sizeof(MRqueue) * sz); > - assert(mrqueue); > + if ( mrqueue == 0) { > + GDKerror("Could not create the map-reduce queue"); > + return; shouldn't you unset the mrqlock here? > + } _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

BugID=3178: A patch to compute stdev using one scan of BAT is available
by Yinhe Cao 13 Nov '12

13 Nov '12

Hi, The current implementation of ALGstdev_@1 in monetdb5/modules/kernel/algebra.mx takes two scans of the BAT. First scan to compute the sum (average) Second scan to compute stdev. By using a well-known mathematical property of the stdev, one scan is enough: during the scan, the sum of the value and the square of the value can be computed together, and the stdev can be computed then. The new implementation would roughly cut the running time by half for most of meaningful cases, especially when the bat is big. Link to the patch: http://bugs.monetdb.org/attachment.cgi?id=158&action=edit Link to the bug: http://bugs.monetdb.org/show_bug.cgi?id=3178 Any comments or suggestions will be greatly appreciated. Thanks, Yinhe _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

Clean install gcc pragma warnings
by Stefan de Konink 10 Nov '12

10 Nov '12

Hi, I just try to install MonetDB on a new system. Used hg and updated to Oct2013. y.tab.c: In function 'sqlparse': y.tab.c:6609:3: error: unknown option after '#pragma GCC diagnostic' kind [-Werror=pragmas] y.tab.c:13496:3: error: unknown option after '#pragma GCC diagnostic' kind [-Werror=pragmas] cc1: all warnings being treated as errors I'm currently running gcc (Gentoo 4.6.3 p1.7, pie-0.5.2) 4.6.3 Is this known? Stefan _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

2 2

RubyGem on Jul2012
by Charalampos Nikolaou 08 Nov '12

08 Nov '12

Hi there, it's my first time playing around with the Jul2012 version of MonetDB and during "make install" I encountered the following error: mkdir -p '/Library/Ruby/Site/1.8' gem install --local --install-dir '/Library/Ruby/Site/1.8' --force --rdoc 'activerecord-monetdb-adapter-0.1.gem' ERROR: While executing gem ... (Gem::FilePermissionError) You don't have write permissions into the /Library/Ruby/Site/1.8 directory. As you can see, I am on a Mac (10.7) and Ruby is installed in /Library/Ruby as indicated above, BUT since I work with many versions of MonetDB, I install all of them "locally", in my home directory. Thus, I would prefer not to install the MonetDB gem file in a system directory. Is that possible? By reading the help output of the configuration, I saw that this is not possible, or might be? Moreover, I understood by the commit messages that this addition (Ruby) is needed for providing another client for MonetDB server. While I do not need such a client, is there an option to avoid building/using/producing it in the compilation process? I didn't see any option in the configuration as well. Thanks a lot, Babis _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

3 5

Re: MonetDB: default - Low-level task scheduler.
by Sjoerd Mullender 07 Nov '12

07 Nov '12

There are a few things wrong with this code: - sz·=·((sz·<<·1)·>>·1); does *not* turn sz into a multiple of two (as suggested by the comment). This statement basically is a no-op. - when you create joinable threads, you should join them. - it's not a great idea to use assert to make sure that GDKmalloc succeeds. Better is to return an error. - The include of monet_options.h should be in the C file, not in the include file (which would be included elsewhere where monet_options.h will already be included). On 2012-11-07 13:03, Martin Kersten wrote: > Changeset: 5ff3c16e865f for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=5ff3c16e865f > Added Files: > gdk/gdk_mapreduce.c > gdk/gdk_mapreduce.h > Modified Files: > gdk/Makefile.ag > monetdb5/modules/mal/groups.c > Branch: default > Log Message: > > Low-level task scheduler. > This module provide a lightweight map-reduce scheduler for multicore systems. > A limited number of workers are initialized upfront, which take the tasks > from a central queue. The header of these task descriptors should comply > with the MRtask structure. > > > diffs (239 lines): > > diff --git a/gdk/Makefile.ag b/gdk/Makefile.ag > --- a/gdk/Makefile.ag > +++ b/gdk/Makefile.ag > @@ -36,7 +36,7 @@ lib_gdk = { > gdk_private.h gdk_delta.h gdk_logger.h gdk_posix.h \ > gdk_system.h gdk_tm.h gdk_storage.h \ > gdk_calc.c gdk_calc.h gdk_calc_compare.h gdk_calc_private.h \ > - gdk_aggr.c gdk_group.c \ > + gdk_aggr.c gdk_group.c gdk_mapreduce.c gdk_mapreduce.h \ > bat.feps bat1.feps bat2.feps \ > libbat.rc > LIBS = ../common/options/libmoptions \ > diff --git a/gdk/gdk_mapreduce.c b/gdk/gdk_mapreduce.c > new file mode 100644 > --- /dev/null > +++ b/gdk/gdk_mapreduce.c > @@ -0,0 +1,141 @@ > +/* > + * The contents of this file are subject to the MonetDB Public License > + * Version 1.1 (the "License"); you may not use this file except in > + * compliance with the License. You may obtain a copy of the License at > + * http://www.monetdb.org/Legal/MonetDBLicense > + * > + * Software distributed under the License is distributed on an "AS IS" > + * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the > + * License for the specific language governing rights and limitations > + * under the License. > + * > + * The Original Code is the MonetDB Database System. > + * > + * The Initial Developer of the Original Code is CWI. > + * Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > + * Copyright August 2008-2012 MonetDB B.V. > + * All Rights Reserved. > + */ > + > +/* > + * (co) Martin L. Kersten > + * This module provide a lightweight map-reduce scheduler for multicore systems. > + * A limited number of workers are initialized upfront, which take the tasks > + * from a central queue. The header of these task descriptors should comply > + * with the MRtask structure. > + * > + */ > +#include "monetdb_config.h" > +#include "gdk.h" > +#include "gdk_mapreduce.h" > + > +/* each entry in the queue contains a list of tasks */ > +typedef struct MRQUEUE { > + MRtask **tasks; > + int index; /* next available task */ > + int size; /* number of tasks */ > +} MRqueue; > + > +static MRqueue *mrqueue; > +static int mrqsize= -1; /* size of queue */ > +static int mrqlast= -1; > +static MT_Lock mrqlock; /* its a shared resource, ie we need locks */ > +static MT_Sema mrqsema; /* threads wait on empty queues */ > + > + > +static void MRworker(void *); > + > +static void > +MRqueueCreate(int sz) > +{ > + int i; > + MT_Id tid; > + > + MT_lock_init(&mrqlock, "q_create"); > + MT_lock_set(&mrqlock,"q_create"); > + MT_sema_init(&mrqsema, 0, "q_create"); > + sz = ((sz << 1) >> 1); /* we want a multiple of 2 */ > + mrqueue = (MRqueue*)GDKzalloc(sizeof(MRqueue) *sz); > + assert(mrqueue); > + mrqsize = sz; > + mrqlast = 0; > + /* create a worker thread for each core as specified as system parameter*/ > + for ( i =0; i < GDKnr_threads; i++) > + MT_create_thread(&tid, MRworker, (void *) 0, MT_THR_JOINABLE); > + MT_lock_unset(&mrqlock,"q_create"); > +} > + > +static void > +MRenqueue(int taskcnt, MRtask **tasks) > +{ > + assert(taskcnt > 0); > + MT_lock_set(&mrqlock, "mrqlock"); > + if (mrqlast == mrqsize) { > + mrqsize <<= 1; > + mrqueue = (MRqueue*) GDKrealloc(mrqueue, sizeof(MRqueue) * mrqsize); > + } > + mrqueue[mrqlast].index = 0; > + mrqueue[mrqlast].tasks = tasks; > + mrqueue[mrqlast].size = taskcnt; > + mrqlast++; > + MT_lock_unset(&mrqlock, "mrqlock"); > + /* a task list is added for consumption*/ > + while (taskcnt-- > 0) > + MT_sema_up(&mrqsema, "mrqsema"); > +} > + > +static MRtask * > +MRdequeue(void) > +{ > + MRtask *r = NULL; > + int idx; > + > + MT_sema_down(&mrqsema, "mrqsema"); > + assert(mrqlast); > + MT_lock_set(&mrqlock, "mrqlock"); > + if (mrqlast > 0) { > + idx = mrqueue[mrqlast-1].index; > + r = mrqueue[mrqlast-1].tasks[idx++]; > + if ( mrqueue[mrqlast-1].size == idx) > + mrqlast--; > + else > + mrqueue[mrqlast-1].index = idx; > + } > + MT_lock_unset(&mrqlock, "mrqlock"); > + assert(r); > + return r; > +} > + > +static void > +MRworker(void * arg) > +{ > + MRtask *task; > + (void) arg; > + do{ > + task= MRdequeue(); > + (task->cmd)(task); > + MT_sema_up(task->sema, "mrqsema"); > + } while (1); > +} > + > +/* schedule the tasks and return when all are done */ > +void > +MRschedule(int taskcnt, void **arg, void (*cmd)(void*p)) > +{ > + int i; > + MT_Sema sema; > + MRtask **task = (MRtask**) arg; > + > + if ( mrqueue == 0) > + MRqueueCreate(1024); > + > + MT_sema_init(&sema, 0, "q_create"); > + for ( i= 0; i < taskcnt; i++){ > + task[i]->sema = & sema; > + task[i]->cmd = cmd; > + } > + MRenqueue(taskcnt,task); > + /* waiting for all report result */ > + for ( i= 0; i < taskcnt; i++) > + MT_sema_down(&sema, "mrqsema"); > +} > diff --git a/gdk/gdk_mapreduce.h b/gdk/gdk_mapreduce.h > new file mode 100644 > --- /dev/null > +++ b/gdk/gdk_mapreduce.h > @@ -0,0 +1,32 @@ > +/* > + * The contents of this file are subject to the MonetDB Public License > + * Version 1.1 (the "License"); you may not use this file except in > + * compliance with the License. You may obtain a copy of the License at > + * http://www.monetdb.org/Legal/MonetDBLicense > + * > + * Software distributed under the License is distributed on an "AS IS" > + * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the > + * License for the specific language governing rights and limitations > + * under the License. > + * > + * The Original Code is the MonetDB Database System. > + * > + * The Initial Developer of the Original Code is CWI. > + * Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > + * Copyright August 2008-2012 MonetDB B.V. > + * All Rights Reserved. > + */ > + > +#ifndef _GDK_MAPREDUCE_H_ > +#define _GDK_MAPREDUCE_H_ > + > +#include <monet_options.h> > + > +typedef struct{ > + MT_Sema *sema; /* micro scheduler handle */ > + void (*cmd)(void *); /* the function to be executed */ > +}MRtask; > + > +gdk_export void MRschedule(int taskcnt, void **arg, void (*cmd)(void *p)); > + > +#endif /* _GDK_MAPREDUCE_H_ */ > diff --git a/monetdb5/modules/mal/groups.c b/monetdb5/modules/mal/groups.c > --- a/monetdb5/modules/mal/groups.c > +++ b/monetdb5/modules/mal/groups.c > @@ -66,11 +66,15 @@ GRPmulticolumngroup(Client cntxt, MalBlk > /* sort order may have influences */ > /* SF100 Q16 showed < ordering is 2 times faster as > ordering */ > for ( i = 3; i< pci->argc; i++) > - for ( j = i+1; j<pci->argc; j++) > - if ( sizes[j] < sizes[i]){ > - l = sizes[j]; sizes[j]= sizes[i]; sizes[i]= l; > - bi = bid[j]; bid[j]= bid[i]; bid[i]= bi; > - } > + for ( j = i+1; j<pci->argc; j++) > + if ( sizes[j] < sizes[i]){ > + l = sizes[j]; > + sizes[j]= sizes[i]; > + sizes[i]= l; > + bi = bid[j]; > + bid[j]= bid[i]; > + bid[i]= bi; > + } > /* for (i=2; i<pci->argc; i++) > mnstr_printf(cntxt->fdout,"# after [%d] "LLFMT"\n",i, sizes[i]); */ > > @@ -82,8 +86,6 @@ GRPmulticolumngroup(Client cntxt, MalBlk > i = 4; > if (msg == MAL_SUCCEED && pci->argc > 4 ) > do { > - if (*ext) > - BBPdecref(*ext, TRUE); > /* early break when there are as many groups as histogram entries */ > b = BATdescriptor(*hist); > if ( b ){ > @@ -91,8 +93,8 @@ GRPmulticolumngroup(Client cntxt, MalBlk > BBPreleaseref(*hist); > if ( j) break; > } > - if (*hist) > - BBPdecref(*hist, TRUE); > + BBPdecref(*ext, TRUE); > + BBPdecref(*hist, TRUE); > > /* (grp,ext,hist) := group.subgroupdone(arg,grp) */ > oldgrp= *grp; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

Where can I add my join algorithm in MonetDB?
by Xiaojun Chang 05 Nov '12

05 Nov '12

Hi, I am one Ph.D student and right now I am working on IMDB. Could anyone tell me where I can add my join algorithm in MonetDB? I cannot find such document. Thanks. Xiaojun Chang _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

3 2

Re: cloning a repository from release MonetDB-11.9.7
by Chi-Young Ku 02 Nov '12

02 Nov '12

Hi, I would like to clone a repository from MonetDB-11.9.7. What would be the branch or rev number for: hg clone http://dev.monetdb.org/hg/MonetDB -b Thanks in advance. Chi ________________________________ From: Chi-Young Ku <chi(a)querymetrica.com> To: "developers-list(a)monetdb.org" <developers-list(a)monetdb.org> Cc: Chi- Young Ku <chi(a)querymetrica.com> Sent: Wednesday, October 17, 2012 5:47 PM Subject: Generating gprof report Hi, I used the "--enable_profile" option to configure my MonetDB and ran "make clean; make; make install". I, then, started my MonetDB instance and ran the following query: select count(*) from lineitem where l_partkey > 9000; Lastly, I quit from the "mclient" session and issued "monetdbd stop ...". I observed that there are several gmon.out files in different directories in my top build directory. Therefore, I issued the following gprof command: gprof /usr/local/bin/mserver5 ./sql/backends/monet5/gmon.out ./gdk/gmon.out ./monetdb5/extras/mal_optimizer_template/gmon.out ./monetdb5/modules/atoms/gmon.out ./monetdb5/modules/kernel/gmon.out ./monetdb5/modules/mal/gmon.out ./monetdb5/scheduler/gmon.out ./monetdb5/optimizer/gmon.out ./monetdb5/mal/gmon.out However, the output of the above command did not show any calling sequence containing the execution of MAL instructions. Did I miss anything? Thanks in advance. Chi _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

3 2