July 2013 - developers-list

Re: MonetDB: Feb2013 - also batxml can be handled by mitosis/mergetable
by Sjoerd Mullender 31 Jul '13

31 Jul '13

Sorry, not allowed in the release branch: this changes the ABI. On 2013-07-31 09:47, Niels Nes wrote: > Changeset: 8300b5c5bd97 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8300b5c5bd97 > Modified Files: > monetdb5/optimizer/opt_prelude.c > monetdb5/optimizer/opt_prelude.h > monetdb5/optimizer/opt_support.c > Branch: Feb2013 > Log Message: > > also batxml can be handled by mitosis/mergetable > > > diffs (41 lines): > > diff --git a/monetdb5/optimizer/opt_prelude.c b/monetdb5/optimizer/opt_prelude.c > --- a/monetdb5/optimizer/opt_prelude.c > +++ b/monetdb5/optimizer/opt_prelude.c > @@ -42,6 +42,7 @@ str boxRef; > str batstrRef; > str batmtimeRef; > str batmmathRef; > +str batxmlRef; > str bbpRef; > str tidRef; > str deltaRef; > @@ -301,6 +302,7 @@ void optimizerInit(void){ > batstrRef = putName("batstr",6); > batmtimeRef = putName("batmtime",8); > batmmathRef = putName("batmmath",8); > + batxmlRef = putName("batxml",6); > bbpRef = putName("bbp",3); > tidRef = putName("tid",3); > deltaRef = putName("delta",5); > diff --git a/monetdb5/optimizer/opt_prelude.h b/monetdb5/optimizer/opt_prelude.h > --- a/monetdb5/optimizer/opt_prelude.h > +++ b/monetdb5/optimizer/opt_prelude.h > @@ -41,6 +41,7 @@ opt_export str boxRef; > opt_export str batstrRef; > opt_export str batmtimeRef; > opt_export str batmmathRef; > +opt_export str batxmlRef; > opt_export str bbpRef; > opt_export str tidRef; > opt_export str deltaRef; > diff --git a/monetdb5/optimizer/opt_support.c b/monetdb5/optimizer/opt_support.c > --- a/monetdb5/optimizer/opt_support.c > +++ b/monetdb5/optimizer/opt_support.c > @@ -827,6 +827,7 @@ int isMapOp(InstrPtr p){ > (getModuleId(p)== batmtimeRef) || > (getModuleId(p)== batstrRef) || > (getModuleId(p)== batmmathRef) || > + (getModuleId(p)== batxmlRef) || > (getModuleId(p)== mkeyRef); > } > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 4

Re: MonetDB: SciQL-2 - forked new SciQL-2 branch off Feb2013 branch:
by Ying Zhang 25 Jul '13

25 Jul '13

On Jul 25, 2013, at 23:12, Stefan Manegold wrote: > Changeset: b1131b028770 for MonetDB > ... > Branch: SciQL-2 > Log Message: > > forked new SciQL-2 branch off Feb2013 branch: Thanks! > > The SciQL-2 branch has been initialized by applying the differences > between tag before_new_JOIN_implementation (changeset 334140294fb2) of the default branch > and tag SciQL-2_source (changeset c05174470f48) of the sciql branch > onto the Feb2013 branch (tag SciQL-2_root / changeset a2665f0d1fc5), > i.e., the SciQL-2 branch differs from the Feb2013 branch > just like the sciql branch differs from the default banch. > > While this excercise was aimed at improving the stability of the SciQL code base > (by eliminating potentially not yet stabelized development code), > the segfaults that triggered this excercise (as reported internally) > do unfortunately still occur also with the SciQL-2 branch ... I'll look into this a.s.a.p., but it's better to work based on a stable branch anyway. Jennie

1 0

(no subject)
by Ketan Sharma 22 Jul '13

22 Jul '13

Hi all, I'm Ketan and I'm new to Monet DB. I have successfully installed MonetDB. I have a table of size ~200MB which is to be inserted into another MonetDB Table. But while inserting data using SELECT command, I am getting the following error: #================================================ GDK reported error. BATfetchjoin(tmpr_2257,tmp_ 2241) does not hit always (|bn|=0 != 10=|l|) => can't use fetchjoin. #================================================ Please do help me in getting this error solved. Thanks.

3 5

Performance fixes
by Alistair Sutherland 17 Jul '13

17 Jul '13

Hi, I've recently been conducting some performance testing of MonetDB with a variety of servers (from 4 core -8GB RAM to 20 core-64GB RAM) with various data sizes, in an attempt to gain a better understanding of how MonetDB scales. During the performance tests it became obvious that much of the processing was IO bound due to: 1) Columns being unmapped from memory overly aggressively (even when there was plenty of memory still available). 2) The constant mapping/unmapping of memory mapped bat files for intermediate results. I've attached a patch which attempts to address both issues. The first patch to (gdk_utils.c) is to update the memory limit at which the GDKvmtrim kicks in to be 80% memory usage. The second patch (gdk_heap.c) limits the number of mmap/munmap calls via the existing 'heap caching' mechanism which was not working at all! In addition to fixing up the caching code, I've also wired in the heap cache into the case where extending a malloced heap results in a swap over to memory mapped storage. After applying the patches I was seeing approximately 40% performance improvements (your mileage may vary!) If the changes are deemed to be useful, how do I go about getting them accepted into the MonetDB source repository? Thanks, Alistair Sutherland ________________________________

2 2

Re: MonetDB: DVframework - fixed bug. Three subselect instructions r...
by Ying Zhang 17 Jul '13

17 Jul '13

Hai Yagiz, Thanks for fixed bugs! :) Is this check-in supposed to fix some of the demo queries? I tried the query (on a freshly created repo.) SELECT channel, sample_time, sample_value FROM mseed.dataview WHERE station = 'ISK' AND sample_time > '2010-01-12T22:15:00.000' AND sample_time < '2010-01-12T22:18:00.000'; But it still gives the error: From the recursive call: MALException:sql.delta:Object not found Is this expected or did I forget something? Jennie On Jul 16, 2013, at 15:49, Yagiz Kargin wrote: > Changeset: 95bc831eb9fb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=95bc831eb9fb > Modified Files: > monetdb5/extras/dvf/dvf.c > monetdb5/extras/dvf/opt_dvf.c > Branch: DVframework > Log Message: > > fixed bug. Three subselect instructions resulting from selection predicate on sample_time are now separately taken care of. > > > diffs (152 lines): > > diff --git a/monetdb5/extras/dvf/dvf.c b/monetdb5/extras/dvf/dvf.c > --- a/monetdb5/extras/dvf/dvf.c > +++ b/monetdb5/extras/dvf/dvf.c > @@ -77,7 +77,7 @@ str plan_modifier(Client cntxt, MalBlkPt > BATiter fli; > > int run_dataflow_opt = 0; > - int run_recycle_opt = 1; > + int run_recycle_opt = 0; > > VarRecord low, high; > > @@ -343,8 +343,9 @@ str plan_modifier(Client cntxt, MalBlkPt > /* adjust variable lifetimes */ > malGarbageCollector(mb); > > - /*chkProgram(cntxt->fdout, cntxt->nspace, mb); > - printFunction(cntxt->fdout,mb, 0, LIST_MAL_ALL); */ > + /*chkProgram(cntxt->fdout, cntxt->nspace, mb);*/ > + /*printFunction(cntxt->fdout,mb, 0, LIST_MAL_EXPLAIN); */ > + /*printFunction(cntxt->fdout,mb, 0, LIST_MAL_ALL); */ > > /* relocate the startpc, the instruction to proceed with the execution. Because it might be changed by the optimizers. */ > for (i = 0; i < limit; i++) > diff --git a/monetdb5/extras/dvf/opt_dvf.c b/monetdb5/extras/dvf/opt_dvf.c > --- a/monetdb5/extras/dvf/opt_dvf.c > +++ b/monetdb5/extras/dvf/opt_dvf.c > @@ -119,6 +119,9 @@ OPTdvfImplementation(Client cntxt, MalBl > int i, limit, which_column, actions = 0; > int last_bind_return_var_id = -1; > int last_data_tid_return_var_id = -1; > + int last_insert_bind_return_var_id = -1; > + int last_subselect_return_var_id = -1; > + int last_update_bind_second_return_var_id = -1; > > stk = stk; //to escape 'unused' parameter error. > pci = pci; //to escape 'unused' parameter error. > @@ -452,35 +455,107 @@ OPTdvfImplementation(Client cntxt, MalBl > actions++; > } > else if((state == 2 || state == 3) && > - ((getModuleId(p) == sqlRef && > + getModuleId(p) == sqlRef && > getFunctionId(p) == bindRef && > p->argc == 7 && > p->retc == 2 && > strcmp(getVarConstant(mb, getArg(p, 3)).val.sval, getVarConstant(mb, getArg(old[i1], 2)).val.sval) == 0 && > strstr(getVarConstant(mb, getArg(p, 4)).val.sval, data_table_identifier) != NULL && > - getVarConstant(mb, getArg(p, 6)).val.ival == 2) || > - (getModuleId(p) == sqlRef && > + getVarConstant(mb, getArg(p, 6)).val.ival == 2) > + { > + last_update_bind_second_return_var_id = getArg(p, 1); > + removeInstruction(mb, p); > + i--; > + actions++; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == sqlRef && > getFunctionId(p) == bindRef && > p->argc == 6 && > p->retc == 1 && > strcmp(getVarConstant(mb, getArg(p, 2)).val.sval, getVarConstant(mb, getArg(old[i1], 2)).val.sval) == 0 && > strstr(getVarConstant(mb, getArg(p, 3)).val.sval, data_table_identifier) != NULL && > - getVarConstant(mb, getArg(p, 5)).val.ival == 1))) > + getVarConstant(mb, getArg(p, 5)).val.ival == 1) > + { > + last_insert_bind_return_var_id = getArg(p, 0); > + removeInstruction(mb, p); > + i--; > + actions++; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == algebraRef && > + getFunctionId(p) == subselectRef && > + p->argc == 8 && > + p->retc == 1 && > + getArg(p, 1) == last_bind_return_var_id) > + { > + last_subselect_return_var_id = getArg(p, 0); > + > + r = newInstruction(mb, ASSIGNsymbol); > + setModuleId(r, algebraRef); > + setFunctionId(r, subselectRef); > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, getArg(p, 1)); > + r = pushArgument(mb, r, getArg(p, 3)); > + r = pushArgument(mb, r, getArg(p, 4)); > + r = pushArgument(mb, r, getArg(p, 5)); > + r = pushArgument(mb, r, getArg(p, 6)); > + r = pushArgument(mb, r, getArg(p, 7)); > + > + insertInstruction(mb, r, i+1); > + removeInstruction(mb, p); > + > + actions += 2; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == algebraRef && > + getFunctionId(p) == subselectRef && > + (p->argc == 7 || p->argc == 8) && > + p->retc == 1 && > + (getArg(p, 1) == last_update_bind_second_return_var_id || getArg(p, 1) == last_insert_bind_return_var_id)) > { > removeInstruction(mb, p); > i--; > actions++; > } > else if((state == 3) && > - (getModuleId(p) == sqlRef && > + getModuleId(p) == sqlRef && > getFunctionId(p) == projectdeltaRef && > p->argc == 6 && > p->retc == 1 && > - last_bind_return_var_id == getArg(p, 2))) > + last_bind_return_var_id == getArg(p, 2)) > + { > + r = newInstruction(mb, ASSIGNsymbol); > + > + if(getArg(p, 1) == last_data_tid_return_var_id) > + { > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, last_bind_return_var_id); > + } > + else > + { > + setModuleId(r, algebraRef); > + setFunctionId(r, leftfetchjoinRef); > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, getArg(p, 1)); > + r = pushArgument(mb, r, last_bind_return_var_id); > + } > + > + insertInstruction(mb, r, i+1); > + removeInstruction(mb, p); > + > + actions += 2; > + } > + else if((state == 3) && > + getModuleId(p) == sqlRef && > + getFunctionId(p) == subdeltaRef && > + p->argc == 6 && > + p->retc == 1 && > + last_subselect_return_var_id == getArg(p, 1)) > { > r = newInstruction(mb, ASSIGNsymbol); > r = pushReturn(mb, r, getArg(p, 0)); > - r = pushArgument(mb, r, last_bind_return_var_id); > + r = pushArgument(mb, r, last_subselect_return_var_id); > > insertInstruction(mb, r, i+1); > removeInstruction(mb, p); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list

2 1

Re: MonetDB: default - Initialize variables with a fake value so tha...
by Stefan Manegold 11 Jul '13

11 Jul '13

Thank you very much, Sjoerd! However, this is not yet sufficient to get this compiled (with optimization enabled): .../monetdb5/optimizer/opt_json.c:54:6: error: 'j' may be used uninitialized in this function [-Werror=maybe-uninitialized] However, simply initializing variable j when it is declared might not be the correct solution since it depends on itself: j = getArg(q,j); ^ ^ [...] pushArgument(mb,q,j); ^ Martin, can you please double-check your code and fix (also) the initilization / use of variable j according to the intended functionality? Thanks! Stefan ----- Original Message ----- > Changeset: fc7c72bf8a56 for MonetDB > URL: > http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fc7c72bf8a56 > Modified Files: > monetdb5/optimizer/opt_json.c > Branch: default > Log Message: > > Initialize variables with a fake value so that this can be compiled. > > > diffs (12 lines): > > diff --git a/monetdb5/optimizer/opt_json.c > b/monetdb5/optimizer/opt_json.c > --- a/monetdb5/optimizer/opt_json.c > +++ b/monetdb5/optimizer/opt_json.c > @@ -30,7 +30,7 @@ int > OPTjsonImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, > InstrPtr pci) > { > int i, j, limit, slimit; > - int bu,br,bj; > + int bu = 0, br = 0, bj = 0; > str nme; > InstrPtr p,q; > int actions = 0; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

code pattern and code generate
by 张磊 04 Jul '13

04 Jul '13

hi all, I found that in GDK of MonetDB, many source code like .h/.c is generated from .mx file when compiling, who can tell me which module control this function? -- Best Wishes! zhanglei

2 1

abou different performaces between SQL and MAL
by Yanchen 03 Jul '13

03 Jul '13

Hi, May I have aquestion about MonetDB performance on TPC-H benchmark? I do not know why I got very different performances when I run 01.sql in TPC-H and the explained 01.mal in mal domain. The average execution time for sql against SF100 scale factor is 19.94s, and the time for mal is 34.7. I also tried to use stethoscope to see the detail, seems like mal stop for several seconds before the last line code. Please let me know if you have any thoughts about that. Thank you so much in advance. Best, Lou I am using Monetdb 11.15.3(Feb. 2013). My mserver5 information is as followed: MonetDB 5 server v11.15.3 "Feb2013-SP1" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 188.9GiB available memory, 32 available cpu cores Libraries: libpcre: 8.30 2012-02-04 (compiled with 8.30) openssl: OpenSSL 1.0.1c 10 May 2012 (compiled with OpenSSL 1.0.1c 10 May 2012) libxml2: 2.8.0 (compiled with 2.8.0) Compiled by: root@LLL-MMMDB2 (x86_64-unknown-linux-gnu) Compilation: gcc -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code Linking : /usr/bin/ld -m elf_x86_64

2 1

Problem running web app using latest jdbc driver
by Casey Gum 03 Jul '13

03 Jul '13

I have a RESTful web service that stores its information in MonetDB. I have to run in a Java 1.6 environment. The latest version of the jdbc driver says that it is for both 1.6 and 1.7. I'm getting the dependency resolved through maven using the clojars repository * <repository>* * <id>monetdb</id>* * <url>http://repository.pentaho.org/artifactory/repo</url>* * <snapshots>* * <enabled>false</enabled>* * </snapshots>* * </repository>* My dependency block looks like this: * <dependency>* * <groupId>monetdb</groupId>* * <artifactId>monetdb-jdbc</artifactId>* * <version>2.9</version>* * </dependency>* Everything worked when I was running v2.8 and getting my dependency resolved from the pentaho repository, but when I get the dependency from clojars, I get the following error: *java.lang.UnsupportedClassVersionError: nl/cwi/monetdb/jdbc/MonetDriver : Unsupported major.minor version 51.0* * * Two questions: 1. Should the 2.9 version work under Java 1.6? 2. Is there some setting I need to put in my pom.xml file so that it runs in "1.6 compatibility mode" or something? Thanks, Casey Gum

2 1