- developers-list - monetdb.org

code for primitives and operators
by Chenxi Li 14 Dec '15

14 Dec '15

Hi, Could you tell me which files have the code for primitives and operators? Thanks a lot. Best Regards, Chenxi Li

2 2

About compilation time
by Mustafa Korkmaz 10 Dec '15

10 Dec '15

Hello all, I'm conducting an experiment to show pros/cons of certain DBs and I need to investigate query compilation performance in details. For this, Is there any way to measure the elapsed time in query parsing/compilation time in MonetDB? Also is there any document out there to show the query parsing/compilation details? What I find for timing is only "-i" flag which gives an overall timing stat. I'd be glad for any help. Thank you Mustafa Korkmaz

4 10

DROP SEQUENCE not working
by Anu Raj Srivastava 09 Dec '15

09 Dec '15

Hi, Whenever I try to drop I sequence I get the following error:- DROP SEQUENCE seq_12129; DROP SEQUENCE: unable to drop sequence seq_12129 (there are database objects which depend on it) I have dropped all the tables in the current schema, but I am still getting this error. Is there a way to find out what database object depends on this sequence? Anu

1 0

DBT-3 (TPC-H derivative) test kit
by Mark Wong 08 Dec '15

08 Dec '15

Greetings everyone, I just wanted to share what I've started with a test kit for MonetDB, hoping it will be useful for development efforts: http://sourceforge.net/p/osdldbt/dbt3/ci/master/tree/ The kit also relies on some additional scripts from here: http://sourceforge.net/p/osdldbt/dbttools/ci/master/tree/ Here is an example of a test report that the kit produces: http://mw2q.github.io/dbt3-monetdb-sample/ I'm hoping that the ability to collect low level profiling (on Linux), and collecting system stats will be helpful. For example, using Linux perf to collect data per query in the Power and Throughput Tests. The kit could still use a little help, and I'm not sure if I'm doing things in the best way. If it seems useful, please let me know how the kit can be improved. Thanks! Regards, Mark -- Mark Wong http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, RemoteDBA, Training & Services

2 2

Driver JDBC 2.18 rounding decimal value
by Luciano Sasso 02 Dec '15

02 Dec '15

Driver JDBC 2.18, rounding decimal/numeric value, in the insert data on table. Its the same bub, in url https://www.monetdb.org/bugzilla/show_bug.cgi?id=3290 ? Any idea? existis patch ? Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Any idea about driver jdbc 2.18 on error rounding in insert values
by Luciano Sasso 26 Nov '15

26 Nov '15

Any idea about driver jdbc 2.18 on error rounding in insert values ? -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Driver JDBC 2.8 dont work in JUL2015SP1
by Luciano Sasso 25 Nov '15

25 Nov '15

Driver JDBC 2.8 dont work in JUL2015SP1? I use 2.8 for ETL (run work with decimal, 2.18 dont work), and 2.18 for dashboards access. Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Serious error in data recording of type numeric
by Luciano Sasso 25 Nov '15

25 Nov '15

Serious error in data recording of type numeric. Using the jdbc driver 2.8, recording both base OUT2014SP4 when JUL2015SP1, records normally. But when using any jdbc driver above 2.8, the recording is wrong, for example, the value 845 is recorded as 850. Is it a bug? Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Driver JDBC, error rounding numeric value
by Luciano Sasso 25 Nov '15

25 Nov '15

Any idea, about error driver JDBC rounding error. Driver 2.18 run error, driver 2.1 run work ok. -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Error driver JDBC
by Luciano Sasso 24 Nov '15

24 Nov '15

Hi, Iam use MonetDB JUL 2015-SPI, and driver JDBC 2.18, but error on load data, in rooud value, exampla, value 854 in source, load 850 in target. I am test with driver JDBC 2.1 and run word, value 845, load in target with 845. any idea !!! -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

MonetDB concurrent commits exception
by Vijay Krishna 13 Nov '15

13 Nov '15

Hi, I am running a setup on MonetDB where in concurrent transactions take place. Everything works very fine but for a few transactions getting aborted due to SQLExceptions. I have set *auto commit to false* for all these transactions. I am getting frequent SQLExceptions with the message as *java.sql.SQLException: COMMIT: failed* But if I retry the same transaction, this works fine. Any idea on what would be the reason for frequent commit fails? When I set *auto commit to true* for the transactions, I get similarly frequent Exceptions but the message becomes *java.sql.SQLException: COMMIT: transaction is aborted because of concurrency conflicts* Is it like both the exception thrown be for the same cause? Any help much appreciated. Thanks in Advance. Vijayakrishna.P. Mobile : (+91) 9500402305.

2 1

Import bug on bug tracker
by andy.barlow＠datum360.com 12 Nov '15

12 Nov '15

Hello there I reported what I believe to be a bug in the import routines towards the end of September: https://www.monetdb.org/bugzilla/show_bug.cgi?id=3812#c0 <https://www.monetdb.org/bugzilla/show_bug.cgi?id=3812#c0> Below is a summary of the issue I encountered, I wasn’t sure if the bug tracker is live/active so trying on this mailing list. Of course, this may be expected behaviour (though I doubt it) in which case would be useful to know. [reply <https://www.monetdb.org/bugzilla/process_bug.cgi#add_comment>] [−] <https://www.monetdb.org/bugzilla/process_bug.cgi#>Description <https://www.monetdb.org/bugzilla/show_bug.cgi?id=3812#c0>Andy Barlow <mailto:andy.barlow@datum360.com> 2015-09-25 15:19:43 CEST User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 Build Identifier: If your import file contains a number of character combinations, such as \560, \561, \700 and probably lots of others, the copy into fails with a message like: Failed to import table line 5 field 1 'varchar(20)' expected in 'FIVE\700' Reproducible: Always Steps to Reproduce: 1. create table test (test varchar(20)); 2. create a load file test.txt with the following: ONE TWO THREE FOUR FIVE\700 3. copy into test from 'test.txt' using delimiters '\t','\n'; 4. fails with message: Failed to import table line 5 field 1 'varchar(20)' expected in 'FIVE\700' Actual Results: Failed to import table line 5 field 1 'varchar(20)' expected in 'FIVE\700' Expected Results: should import OK, the character combinations are valid [reply <https://www.monetdb.org/bugzilla/process_bug.cgi#add_comment>] [−] <https://www.monetdb.org/bugzilla/process_bug.cgi#>Comment 1 <https://www.monetdb.org/bugzilla/show_bug.cgi?id=3812#c1>Andy Barlow <mailto:andy.barlow@datum360.com> 2015-09-25 16:04:00 CEST it appears that the copy command is treating the input as octal. If you convert the backslash \ into its octal equivalent, \134 then it “works”. In some cases, such as \700 an error is thrown. With an import file test.txt: ONE\101 ONE\134101 When imported into table test (test varchar(20)), you get: ONEA ONE\101 So the copy command is converting octal to UTF-8,ASCII equivalen Andy Barlow t: +44 7830 302268 www.datum360.com <http://www.datum360.com/> ...delivering a measured approach to engineering information Please be advised that this email may contain confidential information. If you are not the intended recipient, please do not read, copy or re-transmit this email. If you have received this email in error, please notify us by email by replying to the sender or by telephone and delete this message and any attachments. Thank you in advance for your cooperation and assistance. In addition, Datum360 disclaim that the content of this email constitutes an offer to enter into, or the acceptance of, any contract or agreement or any amendment thereto; provided that the foregoing disclaimer does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment to this email.

2 1

BigDecimal Bug in MonetDB JDBC Driver
by Søren Haagerup 09 Nov '15

09 Nov '15

Hello developers, I found a bug when using BigDecimals in prepared statements with MonetDB¹s JDBC driver. If I have a column of type DECIMAL(15,2) and insert a big decimal with the value 627000.00 through a PreparedStatement, it will turn out as 630000.00 in the DB, since the MonetDB JDBC driver currently mixes up "precision" and "scale", and rounds 627000.00 to a precision of 2 ciphers. I made a bugfix for it, and it can be seen here: https://github.com/shaagerup/monet-jdbc/commit/ ebc5e654dc28e3b56d4c4b610d3d339e675d6c2c I¹m sorry for not committing it directly to your repository, but since I am not a Mercurial user, I hope someone else will use this information to improve the official JDBC driver. Best regards Søren Haagerup

2 1

Announcement: New Jul2015-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 03 Nov '15

03 Nov '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP1 bugfix release Client Package * In the SQL formatter of mclient (the default) we now properly align East Asian wide characters. Bug Fixes * 3789: Query on large string table fails on HEAPextend * 3794: table function sys.rejects() and view sys.rejects() are listed are metadata objects but give an (incorrect) error when they are queried * 3797: COPY INTO with incorrect number columns * 3798: SELECT query with INTERSECT causes assertion failure * 3800: LIKE is broken for many patterns * 3802: Disk space never freed: a logical ref is keeped on a deleted BATs * 3803: SQL query parser fails to parse valid SELECT query with a CASE .. END in it. It fails with parser error: identifier 'x' ambiguous * 3804: `monetdb status` command crashes under certain conditions * 3809: Inefficient plan is generated for queries with many (>= 24) joined tables which take a long time or an HEAPalloc error. I get Error: GDK reported error. HEAPalloc: Insufficie * 3810: Fix statistics gathering * 3811: NOT LIKE not working if the operand doesn't contains wildcards. * 3813: COPY INTO fails on perfectly clean CSV file * 3814: Server crash when using bitwise NOT operation in SQL query * 3816: Server crashes when trying to convert timestamp to str with incorrect format * 3818: Crash when performing UNION/GROUP BY over tables with different columns * 3819: order of tables in FROM-clause has negative impact on generated plan (using crossproducts instead of joins) * 3820: mclient accepts table with repeated constraint which causes crash on insert * 3821: Unexpected error when using a number instead of a boolean * 3822: Yet another LIKE operator issue * 3823: JDBC Connection to a schema - setSchema() error * 3825: MonetDB not cleaning intermediate results which leads to filling up disk space and ultimately server crash * 3827: Certains comparisons between UUID produce a MAL error * 3828: Schema corruption after several ALTER TABLE statements and server restart * 3829: Certains simple WHERE clause cause MonetDB to segfault without explanation * 3830: Coalesce typing inconsistencies * 3833: NULL literals refused at many places * 3834: Date comparison returns incorrect results * 3839: msqldump generates incorrect syntax ON UPDATE (null)

1 0

Crash in table.c
by 营营 02 Nov '15

02 Nov '15

Hi MonetDB Dev, I can import the data in Jul2015, but the latest code fails to import(Win7 64Bit). It crashes in table.c (I used the old version (Jul2015) of this file, there is no crash). To reproduce the crash: 1. execute bi_create_table.sql to create the table, 2. change the path of bi_data.txt in bi_copy_from_file.sql. 3. execute bi_copy_from_file.sql to copy data from file. 4. if server doesn't crash immediately, run #3 more times. Thanks Cline 从网易163邮箱发来的云附件 bi_copy_from_file.sql (109.03K, 2015年11月16日 17:11 到期) 下载 bi_create_table.sql (1.14K, 2015年11月16日 17:11 到期) 下载 bi_data.txt (50M, 2015年11月16日 17:11 到期) 下载

1 0

monetdb SIGSEGV fault
by darkbluehacker＠sohu.com 26 Oct '15

26 Oct '15

I tryied MonetDB-11.21.5 with below sql on two table. ods_ply_base has 50,000,000 rows of record, ods_agent has 100,000 rows of recored.c_cmpny_agt_cde and c_cha_cde are text column.But mserver5 exit with SIGSEGV signal.Below is the stack show by gdb.sql>select * from ods_ply_base left join (select * from ods_agent  where business_end_date > now()) as bb on c_cmpny_agt_cde=bb.c_cha_cde;81      T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)Missing separate debuginfos, use: debuginfo-install keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.12.2-15.el7_1.x86_64 libcom_err-1.42.9-7.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libxml2-2.9.1-5.el7_1.2.x86_64 ncurses-libs-5.9-13.20130511.el7.x86_64 openssl-libs-1.0.1e-42.el7.9.x86_64 pcre-8.32-14.el7.x86_64 readline-6.2-9.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64(gdb) cContinuing.[New Thread 0x7f4603dfe700 (LWP 20663)]Program received signal SIGSEGV, Segmentation fault.[Switching to Thread 0x7f46353e9700 (LWP 20592)]incref (lock=<optimized out>, logical=<optimized out>, i=1567) at gdk_bbp.c:21562156                    hp = bs->B.H->heap.parentid;(gdb)(gdb) bt#0  incref (lock=<optimized out>, logical=<optimized out>, i=1567) at gdk_bbp.c:2156#1  BBPincref (i=i@entry=1567, logical=logical@entry=0) at gdk_bbp.c:2221#2  0x00007f46477252a8 in BATdescriptor (i=1567) at ../../../gdk/gdk.h:2599#3  do_join (r1=0x7f37bc354d30, r2=0x7f37bc354d50, rid=<optimized out>, r2id=r2id@entry=0x0, slid=0x7f37bc207490, srid=0x7f37bc207490, op=op@entry=0, c1=c1@entry=0x0, c2=c2@entry=0x0,    li=li@entry=0, hi=hi@entry=0, nil_matches=nil_matches@entry=0x7f37bc2074b0 "", estimate=estimate@entry=0x7f37bc2074d0, joinfunc=0x7f464734cb20 <BATsubjoin>,    thetafunc=thetafunc@entry=0x0, bandfunc=bandfunc@entry=0x0, rangefunc=rangefunc@entry=0x0, funcname=funcname@entry=0x7f46477db3ef "algebra.subjoin", lid=<optimized out>)    at algebra.c:614#4  0x00007f464772808c in ALGsubjoin (r1=<optimized out>, r2=<optimized out>, lid=<optimized out>, rid=<optimized out>, slid=<optimized out>, srid=<optimized out>,    nil_matches=0x7f37bc2074b0 "", estimate=0x7f37bc2074d0) at algebra.c:681#5  0x00007f46476d6b84 in malCommandCall (stk=stk@entry=0x7f37bc207020, pci=pci@entry=0x7f46315dd260) at mal_interpreter.c:119#6  0x00007f46476d815b in runMALsequence (cntxt=<optimized out>, mb=<optimized out>, startpc=<optimized out>, stoppc=<optimized out>, stk=<optimized out>, env=env@entry=0x0,    pcicaller=pcicaller@entry=0x0) at mal_interpreter.c:655#7  0x00007f46476d9d0e in DFLOWworker (T=<optimized out>) at mal_dataflow.c:376#8  0x00007f4645481df5 in start_thread (arg=0x7f46353e9700) at pthread_create.c:308#9  0x00007f46451af1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113Is this bug of monetdb?zorpior

2 1

Re: MonetDB: embedded-java - Fix miltiple column type cast
by Sjoerd Mullender 21 Oct '15

21 Oct '15

Columns do not contain values of type short, long, float or double. They might contain values of type sht, lng, flt and dbl. You should not use type long in your code unless an external API requires it. Long is not portable since it can be either 32 or 64 bits, even on a 64 bit architecture. A good rule is, if you created a BAT with a TYPE_xxx, use xxx as the name of the C type. On 21/10/15 14:55, Dimitar Nedev wrote: > Changeset: bf487cfa489a for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=bf487cfa489a > Modified Files: > java/embedded/org_monetdb_embedded_result_EmbeddedQueryResult.c > Branch: embedded-java > Log Message: > > Fix miltiple column type cast > > > diffs (90 lines): > > diff --git a/java/embedded/org_monetdb_embedded_result_EmbeddedQueryResult.c b/java/embedded/org_monetdb_embedded_result_EmbeddedQueryResult.c > --- a/java/embedded/org_monetdb_embedded_result_EmbeddedQueryResult.c > +++ b/java/embedded/org_monetdb_embedded_result_EmbeddedQueryResult.c > @@ -29,18 +29,18 @@ static jobject getBooleanColumn(JNIEnv * > jboolean nul_tmp[size]; > if (b->T->nonil && !b->T->nil) { > for (i = 0; i < size; i++) { > - val_tmp[i] = (jboolean) ((jboolean*) Tloc(b, BUNfirst(b)))[i]; > + val_tmp[i] = (bit) ((bit*) Tloc(b, BUNfirst(b)))[i]; > nul_tmp[i] = false; > } > } > else { > for (i = 0; i < size; i++) { > - int v = ((jboolean*) Tloc(b, BUNfirst(b)))[i]; > + int v = ((bit*) Tloc(b, BUNfirst(b)))[i]; > if (v == bit_nil) { > val_tmp[i] = 0; > nul_tmp[i] = true; > } else { > - val_tmp[i] = (jboolean)v; > + val_tmp[i] = (bit)v; > nul_tmp[i] = false; > } > } > @@ -78,7 +78,7 @@ static jobject getByteColumn(JNIEnv *env > } > else { > for (i = 0; i < size; i++) { > - int v = ((bte*) Tloc(b, BUNfirst(b)))[i]; > + bte v = ((bte*) Tloc(b, BUNfirst(b)))[i]; > if (v == bte_nil) { > val_tmp[i] = 0; > nul_tmp[i] = true; > @@ -121,7 +121,7 @@ static jobject getShortColumn(JNIEnv *en > } > else { > for (i = 0; i < size; i++) { > - int v = ((short*) Tloc(b, BUNfirst(b)))[i]; > + short v = ((short*) Tloc(b, BUNfirst(b)))[i]; > if (v == sht_nil) { > val_tmp[i] = 0; > nul_tmp[i] = true; > @@ -207,7 +207,7 @@ static jobject getLongColumn(JNIEnv *env > } > else { > for (i = 0; i < size; i++) { > - int v = ((long*) Tloc(b, BUNfirst(b)))[i]; > + long v = ((long*) Tloc(b, BUNfirst(b)))[i]; > if (v == lng_nil) { > val_tmp[i] = 0; > nul_tmp[i] = true; > @@ -250,7 +250,7 @@ static jobject getFloatColumn(JNIEnv *en > } > else { > for (i = 0; i < size; i++) { > - int v = ((float*) Tloc(b, BUNfirst(b)))[i]; > + float v = ((float*) Tloc(b, BUNfirst(b)))[i]; > if (v == flt_nil) { > val_tmp[i] = 0.0; > nul_tmp[i] = true; > @@ -293,7 +293,7 @@ static jobject getDoubleColumn(JNIEnv *e > } > else { > for (i = 0; i < size; i++) { > - int v = ((double*) Tloc(b, BUNfirst(b)))[i]; > + double v = ((double*) Tloc(b, BUNfirst(b)))[i]; > if (v == dbl_nil) { > val_tmp[i] = 0.0; > nul_tmp[i] = true; > @@ -346,7 +346,6 @@ static jobject getStringColumn(JNIEnv *e > j++; > } > } > - > // Create the column object > // from Java StringColumn(String[] values, int columnSize, boolean[] nullIndex) > column = (*env)->NewObject(env, columnClass, columnConstructor, values, size, nulls); > @@ -380,11 +379,9 @@ JNIEXPORT jobject JNICALL Java_org_monet > return getLongColumn(env, b); > break; > case TYPE_flt: > - printf("float1\n"); > return getFloatColumn(env, b); > break; > case TYPE_dbl: > - printf("double1\n"); > return getDoubleColumn(env, b); > break; > case TYPE_str: > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 3

Boolean columns
by Dimitar Nedev 21 Oct '15

21 Oct '15

Hey guys, A boolean type column seems to fit in a byte, I guess the extra space is to represent nulls (since it ain't a qubit). But does that mean that no column type is actually ':bit'? Given a BAT, how to distinguish between a byte (tinyint) column and a boolean one? Thanks, Dimitar

3 4

Concurrent Transaction control in MonetDB
by Vijay Krishna 19 Oct '15

19 Oct '15

Hi, I read the following lines in a thesis on MonetDB titled - "Improving Transactional Stability in MonetDB" where in they had explained the concurrency control in MonetDB as "Concurrent updates (writers) are not anticipated and in case of N con􏰃icting transactions, only one of them (the one having the earliest time-stamp) is guaranteed to eventually commit its changes. Concurrent appends (on the same table) are detected as write-to-write conflicts and only the writer carrying the earliest time-stamp will end up committing successfully, the rest will be aborted." Why is that the transaction having the earliest timestamp is allowed to commit, in case of a conflict? Shouldn't it be the latest transaction that should commit on a data, since the latest transaction would be having the latest version of data? Or, what am I getting wrong here? Any help much appreciated. Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

2 3

Re: MonetDB: default - fixed bug in grant_role code (ie properly che...
by Ying Zhang 18 Oct '15

18 Oct '15

> On Oct 18, 2015, at 16:23, Niels Nes <commits(a)monetdb.org> wrote: > > Changeset: 6f3500c5cece for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=6f3500c5cece > Modified Files: > clients/Tests/MAL-signatures.stable.out > clients/Tests/MAL-signatures.stable.out.int128 > clients/Tests/SQL-dump.stable.out.int128 > sql/backends/monet5/UDF/Tests/udf-fuse.stable.out > sql/backends/monet5/UDF/Tests/udf-reverse.stable.out > sql/backends/monet5/sql.c > sql/backends/monet5/sql.mal > sql/backends/monet5/sql_gencode.c > sql/backends/monet5/sql_gencode.h > sql/backends/monet5/sql_scenario.h > sql/benchmarks/tpch/Tests/01-explain.stable.out > sql/benchmarks/tpch/Tests/01-explain.stable.out.32bit > sql/benchmarks/tpch/Tests/01-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/04-explain.stable.out > sql/benchmarks/tpch/Tests/04-explain.stable.out.32bit > sql/benchmarks/tpch/Tests/05-explain.stable.out > sql/benchmarks/tpch/Tests/05-explain.stable.out.32bit > sql/benchmarks/tpch/Tests/05-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/06-explain.stable.out > sql/benchmarks/tpch/Tests/06-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/10-explain.stable.out > sql/benchmarks/tpch/Tests/10-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/11-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/12-explain.stable.out > sql/benchmarks/tpch/Tests/12-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/14-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/15-explain.stable.out > sql/benchmarks/tpch/Tests/15-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/17-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/19-explain.stable.out > sql/benchmarks/tpch/Tests/19-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/20-explain.stable.out > sql/benchmarks/tpch/Tests/20-explain.stable.out.32bit > sql/benchmarks/tpch/Tests/20-explain.stable.out.int128 > sql/common/sql_types.c > sql/server/rel_dump.c > sql/server/rel_dump.h > sql/server/rel_optimizer.c > sql/server/sql_privileges.c > sql/server/sql_semantic.c > sql/server/sql_semantic.h > sql/test/BugTracker-2008/Tests/auto_coersion_bug.SF-2075157.stable.out > sql/test/BugTracker-2008/Tests/auto_coersion_bug.SF-2075157.stable.out.int128 > sql/test/BugTracker-2010/Tests/LIMIT_OFFSET_big-endian.Bug-2622.stable.out > sql/test/BugTracker-2011/Tests/crash_on_alias.Bug-2798.stable.out.int128 > sql/test/BugTracker-2011/Tests/func_iter_vs_bulk.Bug-2826.stable.out > sql/test/BugTracker-2012/Tests/rewrite_like_into_likesubselect.Bug-3179.stable.out > sql/test/BugTracker-2015/Tests/ambiguous.Bug-3803.stable.out.int128 > sql/test/BugTracker-2015/Tests/backslash_predecing_separator.Bug-3775.stable.err > sql/test/BugTracker-2015/Tests/incorrect_result_type.Bug-3687.stable.out.int128 > sql/test/BugTracker-2015/Tests/sql2pcre.Bug-3800.stable.out > sql/test/BugTracker-2015/Tests/useless_casts.Bug-3756.stable.out > sql/test/Users/Tests/columnRights.stable.err > sql/test/Users/Tests/columnRights.stable.out > sql/test/Users/Tests/dropManyUsers.Bug-3764.stable.out > sql/test/Users/Tests/grantAndRevokeUserLogedIN.Bug-3476.SQL.py > sql/test/Users/Tests/grantAndRevokeUserLogedIN.Bug-3476.stable.err > sql/test/Users/Tests/grantAndRevokeUserLogedIN.Bug-3476.stable.out > sql/test/Users/Tests/grantMonetdb.SQL.py > sql/test/Users/Tests/grantMonetdbToRegularUser.Bug-3771.SQL.py > sql/test/Users/Tests/grantMonetdbToSchemaOwner.Bug-3771.SQL.py > sql/test/Users/Tests/grantMonetdbToSchemaOwner.Bug-3771.stable.err > sql/test/Users/Tests/grantPrivilegesNonDefaultRole.Bug-3365.stable.err > sql/test/Users/Tests/grantRevokeAndGrantAgain.Bug-3765.SQL.py > sql/test/Users/Tests/grantRevokeAndGrantAgain.Bug-3765.stable.err > sql/test/Users/Tests/grantRevokeAndGrantAgain.Bug-3765.stable.out > sql/test/leaks/Tests/check1.stable.out.int128 > sql/test/leaks/Tests/check2.stable.out.int128 > sql/test/leaks/Tests/check3.stable.out.int128 > sql/test/leaks/Tests/check4.stable.out.int128 > sql/test/leaks/Tests/check5.stable.out.int128 > sql/test/mapi/Tests/sql_int128.stable.out > sql/test/pg_regress/Tests/int2.stable.out > sql/test/pg_regress/Tests/int8.stable.out > sql/test/pg_regress/Tests/int8.stable.out.int128 > sql/test/pg_regress/Tests/select_views.stable.out > sql/test/pg_regress/Tests/strings.stable.err.int128 > sql/test/pg_regress/Tests/strings.stable.out > Branch: default > Log Message: > > fixed bug in grant_role code (ie properly check rights) is this the same bug you just fixed in Jul2015? Can you give me a SQL query so I can add a test for it (if you think is necessary)? Or two SQL queries, if they are different problems. Jennie > added check to not grant 'role' rights when the role is a user (ie > grant some_user to role_or_user, isn't allowed) > > fixed bug in topn rewriter (in combination with distinct) > > we now ship relational algebra instead of mal for remote execution > > more explicit typing in code generation (fixes problems with R/Python) > > > diffs (truncated from 15187 to 300 lines): > > diff --git a/clients/Tests/MAL-signatures.stable.out b/clients/Tests/MAL-signatures.stable.out > --- a/clients/Tests/MAL-signatures.stable.out > +++ b/clients/Tests/MAL-signatures.stable.out > @@ -40743,6 +40743,10 @@ pattern sql.restart{unsafe}(sname:str,se > address mvc_restart_seq; > comment restart the sequence with value start > > +pattern sql.register(mod:str,fname:str,rel_stmt:str,sig:str):int > +address RAstatement2; > +comment Compile the relational statement (rel_smt) and register it as mal function, mod.fname(signature) > + > function sql.sql_environment() (name:bat[:oid,:str],value:bat[:oid,:str]); > pattern sql.shrink(sch:str,tbl:str):void > address SQLshrink; > diff --git a/clients/Tests/MAL-signatures.stable.out.int128 b/clients/Tests/MAL-signatures.stable.out.int128 > --- a/clients/Tests/MAL-signatures.stable.out.int128 > +++ b/clients/Tests/MAL-signatures.stable.out.int128 > @@ -51682,6 +51682,10 @@ pattern sql.restart{unsafe}(sname:str,se > address mvc_restart_seq; > comment restart the sequence with value start > > +pattern sql.register(mod:str,fname:str,rel_stmt:str,sig:str):int > +address RAstatement2; > +comment Compile the relational statement (rel_smt) and register it as mal function, mod.fname(signature) > + > function sql.sql_environment() (name:bat[:oid,:str],value:bat[:oid,:str]); > pattern sql.shrink(sch:str,tbl:str):void > address SQLshrink; > diff --git a/clients/Tests/SQL-dump.stable.out.int128 b/clients/Tests/SQL-dump.stable.out.int128 > --- a/clients/Tests/SQL-dump.stable.out.int128 > +++ b/clients/Tests/SQL-dump.stable.out.int128 > @@ -1534,234 +1534,234 @@ 2149 "statement" "varchar" 2048 0 2139 N > 2151 "id" "int" 32 0 2150 NULL true 0 NULL > 2152 "name" "varchar" 1024 0 2150 NULL true 1 NULL > 2153 "nr" "int" 32 0 2150 NULL true 2 NULL > -5598 "id" "int" 32 0 5597 NULL true 0 NULL > -5599 "name" "varchar" 1024 0 5597 NULL true 1 NULL > -5600 "schema_id" "int" 32 0 5597 NULL true 2 NULL > -5601 "query" "varchar" 2048 0 5597 NULL true 3 NULL > -5602 "type" "smallint" 16 0 5597 NULL true 4 NULL > -5603 "system" "boolean" 1 0 5597 NULL true 5 NULL > -5604 "commit_action" "smallint" 16 0 5597 NULL true 6 NULL > -5605 "access" "smallint" 16 0 5597 NULL true 7 NULL > -5606 "temporary" "smallint" 16 0 5597 NULL true 8 NULL > -5608 "id" "int" 32 0 5607 NULL true 0 NULL > -5609 "name" "varchar" 1024 0 5607 NULL true 1 NULL > -5610 "type" "varchar" 1024 0 5607 NULL true 2 NULL > -5611 "type_digits" "int" 32 0 5607 NULL true 3 NULL > -5612 "type_scale" "int" 32 0 5607 NULL true 4 NULL > -5613 "table_id" "int" 32 0 5607 NULL true 5 NULL > -5614 "default" "varchar" 2048 0 5607 NULL true 6 NULL > -5615 "null" "boolean" 1 0 5607 NULL true 7 NULL > -5616 "number" "int" 32 0 5607 NULL true 8 NULL > -5617 "storage" "varchar" 2048 0 5607 NULL true 9 NULL > -5624 "name" "varchar" 1024 0 5623 NULL true 0 NULL > -5625 "fullname" "varchar" 2048 0 5623 NULL true 1 NULL > -5626 "default_schema" "int" 9 0 5623 NULL true 2 NULL > -5630 "name" "varchar" 1024 0 5629 NULL true 0 NULL > -5631 "fullname" "varchar" 2024 0 5629 NULL true 1 NULL > -5632 "default_schema" "int" 9 0 5629 NULL true 2 NULL > -5634 "login_id" "int" 32 0 5633 NULL true 0 NULL > -5635 "role_id" "int" 32 0 5633 NULL true 1 NULL > -5637 "id" "int" 32 0 5636 NULL true 0 NULL > -5638 "name" "varchar" 1024 0 5636 NULL true 1 NULL > -5639 "grantor" "int" 32 0 5636 NULL true 2 NULL > -5641 "obj_id" "int" 32 0 5640 NULL true 0 NULL > -5642 "auth_id" "int" 32 0 5640 NULL true 1 NULL > -5643 "privileges" "int" 32 0 5640 NULL true 2 NULL > -5644 "grantor" "int" 32 0 5640 NULL true 3 NULL > -5645 "grantable" "int" 32 0 5640 NULL true 4 NULL > -5854 "id" "oid" 63 0 5862 NULL true 0 NULL > -5855 "owner" "clob" 0 0 5862 NULL true 1 NULL > -5856 "defined" "timestamp" 7 0 5862 NULL true 2 NULL > -5857 "query" "clob" 0 0 5862 NULL true 3 NULL > -5858 "pipe" "clob" 0 0 5862 NULL true 4 NULL > -5859 "plan" "clob" 0 0 5862 NULL true 5 NULL > -5860 "mal" "int" 32 0 5862 NULL true 6 NULL > -5861 "optimize" "bigint" 64 0 5862 NULL true 7 NULL > -5864 "id" "oid" 63 0 5873 NULL true 0 NULL > -5865 "start" "timestamp" 7 0 5873 NULL true 1 NULL > -5866 "stop" "timestamp" 7 0 5873 NULL true 2 NULL > -5867 "arguments" "clob" 0 0 5873 NULL true 3 NULL > -5868 "tuples" "wrd" 64 0 5873 NULL true 4 NULL > -5869 "run" "bigint" 64 0 5873 NULL true 5 NULL > -5870 "ship" "bigint" 64 0 5873 NULL true 6 NULL > -5871 "cpu" "int" 32 0 5873 NULL true 7 NULL > -5872 "io" "int" 32 0 5873 NULL true 8 NULL > -5875 "id" "oid" 63 0 5891 NULL true 0 NULL > -5876 "owner" "clob" 0 0 5891 NULL true 1 NULL > -5877 "defined" "timestamp" 7 0 5891 NULL true 2 NULL > -5878 "query" "clob" 0 0 5891 NULL true 3 NULL > -5879 "pipe" "clob" 0 0 5891 NULL true 4 NULL > -5880 "plan" "clob" 0 0 5891 NULL true 5 NULL > -5881 "mal" "int" 32 0 5891 NULL true 6 NULL > -5882 "optimize" "bigint" 64 0 5891 NULL true 7 NULL > -5883 "start" "timestamp" 7 0 5891 NULL true 8 NULL > -5884 "stop" "timestamp" 7 0 5891 NULL true 9 NULL > -5885 "arguments" "clob" 0 0 5891 NULL true 10 NULL > -5886 "tuples" "wrd" 64 0 5891 NULL true 11 NULL > -5887 "run" "bigint" 64 0 5891 NULL true 12 NULL > -5888 "ship" "bigint" 64 0 5891 NULL true 13 NULL > -5889 "cpu" "int" 32 0 5891 NULL true 14 NULL > -5890 "io" "int" 32 0 5891 NULL true 15 NULL > -5917 "event" "int" 32 0 5930 NULL true 0 NULL > -5918 "clk" "varchar" 20 0 5930 NULL true 1 NULL > -5919 "pc" "varchar" 50 0 5930 NULL true 2 NULL > -5920 "thread" "int" 32 0 5930 NULL true 3 NULL > -5921 "ticks" "bigint" 64 0 5930 NULL true 4 NULL > -5922 "rrsmb" "bigint" 64 0 5930 NULL true 5 NULL > -5923 "vmmb" "bigint" 64 0 5930 NULL true 6 NULL > -5924 "reads" "bigint" 64 0 5930 NULL true 7 NULL > -5925 "writes" "bigint" 64 0 5930 NULL true 8 NULL > -5926 "minflt" "bigint" 64 0 5930 NULL true 9 NULL > -5927 "majflt" "bigint" 64 0 5930 NULL true 10 NULL > -5928 "nvcsw" "bigint" 64 0 5930 NULL true 11 NULL > -5929 "stmt" "clob" 0 0 5930 NULL true 12 NULL > -6064 "user" "clob" 0 0 6070 NULL true 0 NULL > -6065 "login" "timestamp" 7 0 6070 NULL true 1 NULL > -6066 "sessiontimeout" "bigint" 64 0 6070 NULL true 2 NULL > -6067 "lastcommand" "timestamp" 7 0 6070 NULL true 3 NULL > -6068 "querytimeout" "bigint" 64 0 6070 NULL true 4 NULL > -6069 "active" "boolean" 1 0 6070 NULL true 5 NULL > -6147 "name" "clob" 0 0 6150 NULL true 0 NULL > -6148 "def" "clob" 0 0 6150 NULL true 1 NULL > -6149 "status" "clob" 0 0 6150 NULL true 2 NULL > -6156 "name" "clob" 0 0 6158 NULL true 0 NULL > -6157 "value" "clob" 0 0 6158 NULL true 1 NULL > -6194 "qtag" "bigint" 64 0 6202 NULL true 0 NULL > -6195 "user" "clob" 0 0 6202 NULL true 1 NULL > -6196 "started" "timestamp" 7 0 6202 NULL true 2 NULL > -6197 "estimate" "timestamp" 7 0 6202 NULL true 3 NULL > -6198 "progress" "int" 32 0 6202 NULL true 4 NULL > -6199 "status" "clob" 0 0 6202 NULL true 5 NULL > -6200 "tag" "oid" 63 0 6202 NULL true 6 NULL > -6201 "query" "clob" 0 0 6202 NULL true 7 NULL > -6228 "rowid" "bigint" 64 0 6232 NULL true 0 NULL > -6229 "fldid" "int" 32 0 6232 NULL true 1 NULL > -6230 "message" "clob" 0 0 6232 NULL true 2 NULL > -6231 "input" "clob" 0 0 6232 NULL true 3 NULL > -6887 "keyword" "varchar" 40 0 6890 NULL false 0 NULL > -6892 "table_type_id" "smallint" 16 0 6898 NULL false 0 NULL > -6895 "table_type_name" "varchar" 25 0 6898 NULL false 1 NULL > -6900 "dependency_type_id" "smallint" 16 0 6906 NULL false 0 NULL > -6903 "dependency_type_name" "varchar" 15 0 6906 NULL false 1 NULL > -6921 "file_id" "int" 32 0 6923 NULL true 0 NULL > -6922 "location" "char" 256 0 6923 NULL true 1 NULL > -6925 "dim_id" "int" 32 0 6929 NULL true 0 NULL > -6926 "file_id" "int" 32 0 6929 NULL true 1 NULL > -6927 "name" "varchar" 64 0 6929 NULL true 2 NULL > -6928 "length" "int" 32 0 6929 NULL true 3 NULL > -6931 "var_id" "int" 32 0 6937 NULL true 0 NULL > -6932 "file_id" "int" 32 0 6937 NULL true 1 NULL > -6933 "name" "varchar" 64 0 6937 NULL true 2 NULL > -6934 "vartype" "varchar" 64 0 6937 NULL true 3 NULL > -6935 "ndim" "int" 32 0 6937 NULL true 4 NULL > -6936 "coord_dim_id" "int" 32 0 6937 NULL true 5 NULL > -6939 "var_id" "int" 32 0 6943 NULL true 0 NULL > -6940 "dim_id" "int" 32 0 6943 NULL true 1 NULL > -6941 "file_id" "int" 32 0 6943 NULL true 2 NULL > -6942 "dimpos" "int" 32 0 6943 NULL true 3 NULL > -6945 "obj_name" "varchar" 256 0 6951 NULL true 0 NULL > -6946 "att_name" "varchar" 256 0 6951 NULL true 1 NULL > -6947 "att_type" "varchar" 64 0 6951 NULL true 2 NULL > -6948 "value" "clob" 0 0 6951 NULL true 3 NULL > -6949 "file_id" "int" 32 0 6951 NULL true 4 NULL > -6950 "gr_name" "varchar" 256 0 6951 NULL true 5 NULL > -6976 "schema" "clob" 0 0 6990 NULL true 0 NULL > -6977 "table" "clob" 0 0 6990 NULL true 1 NULL > -6978 "column" "clob" 0 0 6990 NULL true 2 NULL > -6979 "type" "clob" 0 0 6990 NULL true 3 NULL > -6980 "mode" "clob" 0 0 6990 NULL true 4 NULL > -6981 "location" "clob" 0 0 6990 NULL true 5 NULL > -6982 "count" "bigint" 64 0 6990 NULL true 6 NULL > -6983 "typewidth" "int" 32 0 6990 NULL true 7 NULL > -6984 "columnsize" "bigint" 64 0 6990 NULL true 8 NULL > -6985 "heapsize" "bigint" 64 0 6990 NULL true 9 NULL > -6986 "hashes" "bigint" 64 0 6990 NULL true 10 NULL > -6987 "phash" "boolean" 1 0 6990 NULL true 11 NULL > -6988 "imprints" "bigint" 64 0 6990 NULL true 12 NULL > -6989 "sorted" "boolean" 1 0 6990 NULL true 13 NULL > -6992 "schema" "clob" 0 0 7002 NULL true 0 NULL > -6993 "table" "clob" 0 0 7002 NULL true 1 NULL > -6994 "column" "clob" 0 0 7002 NULL true 2 NULL > -6995 "type" "clob" 0 0 7002 NULL true 3 NULL > -6996 "typewidth" "int" 32 0 7002 NULL true 4 NULL > -6997 "count" "bigint" 64 0 7002 NULL true 5 NULL > -6998 "distinct" "bigint" 64 0 7002 NULL true 6 NULL > -6999 "atomwidth" "int" 32 0 7002 NULL true 7 NULL > -7000 "reference" "boolean" 1 0 7002 NULL true 8 NULL > -7001 "sorted" "boolean" 1 0 7002 NULL true 9 NULL > -7040 "schema" "clob" 0 0 7050 NULL true 0 NULL > -7041 "table" "clob" 0 0 7050 NULL true 1 NULL > -7042 "column" "clob" 0 0 7050 NULL true 2 NULL > -7043 "type" "clob" 0 0 7050 NULL true 3 NULL > -7044 "count" "bigint" 64 0 7050 NULL true 4 NULL > -7045 "columnsize" "bigint" 64 0 7050 NULL true 5 NULL > -7046 "heapsize" "bigint" 64 0 7050 NULL true 6 NULL > -7047 "hashes" "bigint" 64 0 7050 NULL true 7 NULL > -7048 "imprints" "bigint" 64 0 7050 NULL true 8 NULL > -7049 "sorted" "boolean" 1 0 7050 NULL true 9 NULL > -7052 "schema" "clob" 0 0 7060 NULL true 0 NULL > -7053 "table" "clob" 0 0 7060 NULL true 1 NULL > -7054 "count" "bigint" 64 0 7060 NULL true 2 NULL > -7055 "columnsize" "hugeint" 128 0 7060 NULL true 3 NULL > -7056 "heapsize" "hugeint" 128 0 7060 NULL true 4 NULL > -7057 "hashes" "hugeint" 128 0 7060 NULL true 5 NULL > -7058 "imprints" "hugeint" 128 0 7060 NULL true 6 NULL > -7059 "auxiliary" "hugeint" 128 0 7060 NULL true 7 NULL > -7062 "column_id" "int" 32 0 7073 NULL true 0 NULL > -7063 "type" "clob" 0 0 7073 NULL true 1 NULL > -7064 "width" "int" 32 0 7073 NULL true 2 NULL > -7065 "stamp" "timestamp" 7 0 7073 NULL true 3 NULL > -7066 "sample" "bigint" 64 0 7073 NULL true 4 NULL > -7067 "count" "bigint" 64 0 7073 NULL true 5 NULL > -7068 "unique" "bigint" 64 0 7073 NULL true 6 NULL > -7069 "nils" "bigint" 64 0 7073 NULL true 7 NULL > -7070 "minval" "clob" 0 0 7073 NULL true 8 NULL > -7071 "maxval" "clob" 0 0 7073 NULL true 9 NULL > -7072 "sorted" "boolean" 1 0 7073 NULL true 10 NULL > -7163 "file_id" "bigint" 64 0 7171 NULL false 0 NULL > -7164 "file_location" "clob" 0 0 7171 NULL false 1 NULL > -7165 "dbschema" "smallint" 16 0 7171 NULL false 2 NULL > -7166 "format_version" "varchar" 7 0 7171 NULL true 3 NULL > -7167 "sorting_order" "varchar" 10 0 7171 NULL true 4 NULL > -7168 "comments" "clob" 0 0 7171 NULL true 5 NULL > -7173 "sn" "clob" 0 0 7184 NULL false 0 NULL > -7174 "file_id" "bigint" 64 0 7184 NULL false 1 NULL > -7175 "ln" "int" 32 0 7184 NULL true 2 NULL > -7176 "as" "int" 32 0 7184 NULL true 3 NULL > -7177 "m5" "clob" 0 0 7184 NULL true 4 NULL > -7178 "sp" "clob" 0 0 7184 NULL true 5 NULL > -7179 "ur" "clob" 0 0 7184 NULL true 6 NULL > -7186 "id" "clob" 0 0 7203 NULL false 0 NULL > -7187 "file_id" "bigint" 64 0 7203 NULL false 1 NULL > -7188 "cn" "clob" 0 0 7203 NULL true 2 NULL > -7189 "ds" "clob" 0 0 7203 NULL true 3 NULL > -7190 "dt" "timestamp" 7 0 7203 NULL true 4 NULL > -7191 "fo" "clob" 0 0 7203 NULL true 5 NULL > -7192 "ks" "clob" 0 0 7203 NULL true 6 NULL > -7193 "lb" "clob" 0 0 7203 NULL true 7 NULL > -7194 "pg" "clob" 0 0 7203 NULL true 8 NULL > -7195 "pi" "int" 32 0 7203 NULL true 9 NULL > -7196 "pl" "clob" 0 0 7203 NULL true 10 NULL > -7197 "pu" "clob" 0 0 7203 NULL true 11 NULL > -7198 "sm" "clob" 0 0 7203 NULL true 12 NULL > -7205 "id" "clob" 0 0 7215 NULL false 0 NULL > -7206 "file_id" "bigint" 64 0 7215 NULL false 1 NULL > -7207 "pn" "clob" 0 0 7215 NULL true 2 NULL > -7208 "cl" "clob" 0 0 7215 NULL true 3 NULL > -7209 "pp" "clob" 0 0 7215 NULL true 4 NULL > -7210 "vn" "clob" 0 0 7215 NULL true 5 NULL > -7217 "qname" "clob" 0 0 7228 NULL false 0 NULL > -7218 "flag" "smallint" 16 0 7228 NULL false 1 NULL > -7219 "rname" "clob" 0 0 7228 NULL false 2 NULL > -7220 "pos" "int" 32 0 7228 NULL false 3 NULL > -7221 "mapq" "smallint" 16 0 7228 NULL false 4 NULL > -7222 "cigar" "clob" 0 0 7228 NULL false 5 NULL > -7223 "rnext" "clob" 0 0 7228 NULL false 6 NULL > -7224 "pnext" "int" 32 0 7228 NULL false 7 NULL > -7225 "tlen" "int" 32 0 7228 NULL false 8 NULL > -7226 "seq" "clob" 0 0 7228 NULL false 9 NULL > -7227 "qual" "clob" 0 0 7228 NULL false 10 NULL > -7309 "function_id" "int" 32 0 7310 NULL true 0 NULL > +5688 "id" "int" 32 0 5687 NULL true 0 NULL > +5689 "name" "varchar" 1024 0 5687 NULL true 1 NULL > +5690 "schema_id" "int" 32 0 5687 NULL true 2 NULL > +5691 "query" "varchar" 2048 0 5687 NULL true 3 NULL > +5692 "type" "smallint" 16 0 5687 NULL true 4 NULL > +5693 "system" "boolean" 1 0 5687 NULL true 5 NULL > +5694 "commit_action" "smallint" 16 0 5687 NULL true 6 NULL > +5695 "access" "smallint" 16 0 5687 NULL true 7 NULL > +5696 "temporary" "smallint" 16 0 5687 NULL true 8 NULL > +5698 "id" "int" 32 0 5697 NULL true 0 NULL > +5699 "name" "varchar" 1024 0 5697 NULL true 1 NULL > +5700 "type" "varchar" 1024 0 5697 NULL true 2 NULL > +5701 "type_digits" "int" 32 0 5697 NULL true 3 NULL > +5702 "type_scale" "int" 32 0 5697 NULL true 4 NULL > +5703 "table_id" "int" 32 0 5697 NULL true 5 NULL > +5704 "default" "varchar" 2048 0 5697 NULL true 6 NULL > +5705 "null" "boolean" 1 0 5697 NULL true 7 NULL > +5706 "number" "int" 32 0 5697 NULL true 8 NULL > +5707 "storage" "varchar" 2048 0 5697 NULL true 9 NULL > +5714 "name" "varchar" 1024 0 5713 NULL true 0 NULL > +5715 "fullname" "varchar" 2048 0 5713 NULL true 1 NULL > +5716 "default_schema" "int" 9 0 5713 NULL true 2 NULL > +5720 "name" "varchar" 1024 0 5719 NULL true 0 NULL > +5721 "fullname" "varchar" 2024 0 5719 NULL true 1 NULL > +5722 "default_schema" "int" 9 0 5719 NULL true 2 NULL > +5724 "login_id" "int" 32 0 5723 NULL true 0 NULL > +5725 "role_id" "int" 32 0 5723 NULL true 1 NULL > +5727 "id" "int" 32 0 5726 NULL true 0 NULL > +5728 "name" "varchar" 1024 0 5726 NULL true 1 NULL > +5729 "grantor" "int" 32 0 5726 NULL true 2 NULL > +5731 "obj_id" "int" 32 0 5730 NULL true 0 NULL > +5732 "auth_id" "int" 32 0 5730 NULL true 1 NULL > +5733 "privileges" "int" 32 0 5730 NULL true 2 NULL > +5734 "grantor" "int" 32 0 5730 NULL true 3 NULL > +5735 "grantable" "int" 32 0 5730 NULL true 4 NULL > +5944 "id" "oid" 63 0 5952 NULL true 0 NULL > +5945 "owner" "clob" 0 0 5952 NULL true 1 NULL > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

1 0

Re: MonetDB: Jul2015 - fixed problem with table privileges
by Ying Zhang 18 Oct '15

18 Oct '15

> On Oct 18, 2015, at 14:30, Niels Nes <commits(a)monetdb.org> wrote: > > Changeset: d1da7aeae75e for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=d1da7aeae75e > Modified Files: > sql/server/sql_privileges.c > Branch: Jul2015 > Log Message: > > fixed problem with table privileges What’s the problem? The diff is easy to read, but what’s the symptom of the problem at, say, SQL level? Thanks! Jennie > > > diffs (12 lines): > > diff --git a/sql/server/sql_privileges.c b/sql/server/sql_privileges.c > --- a/sql/server/sql_privileges.c > +++ b/sql/server/sql_privileges.c > @@ -467,7 +467,7 @@ sql_grantable_(mvc *m, int grantorid, in > int priv; > > (void) sub; > - for (priv = 1; priv < privs; priv <<= 1) { > + for (priv = 1; priv <= privs; priv <<= 1) { > if (!(priv & privs)) > continue; > rid = table_funcs.column_find_row(m->session->tr, priv_obj, &obj_id, priv_auth, &grantorid, priv_priv, &priv, NULL); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 1

Grails Integration
by Federico G Agripa 09 Oct '15

09 Oct '15

Hello, i'm working on a grails plugin to easily integrate with monetDB. So I would like to here what functionalities will be useful to you guys because I want it to be useful to all of you to. I'm implementing it from the monet jar 2.17 and I hope to have it up in grails plugins as soon as possible. Btw any collaboration will be really welcome Regards

1 0

MonetDB via ODBC questions
by d.tikhonov＠revotechs.com 08 Oct '15

08 Oct '15

3 5

Re: MonetDB: default - Addition of the candidatelist property
by Sjoerd Mullender 07 Oct '15

07 Oct '15

opt_candidates.c and opt_candidates.h are missing. On 07/10/15 14:33, Martin Kersten wrote: > Changeset: c6e5763417d9 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=c6e5763417d9 > Modified Files: > clients/Tests/exports.stable.out > monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.err > monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out.single > monetdb5/mal/mal.h > monetdb5/mal/mal_instruction.c > monetdb5/mal/mal_instruction.h > monetdb5/mal/mal_parser.c > monetdb5/mal/mal_type.h > monetdb5/optimizer/Makefile.ag > monetdb5/optimizer/opt_pipes.c > monetdb5/optimizer/opt_prelude.c > monetdb5/optimizer/opt_prelude.h > monetdb5/optimizer/opt_pushselect.c > monetdb5/optimizer/opt_remoteQueries.c > monetdb5/optimizer/opt_support.h > monetdb5/optimizer/opt_wrapper.c > monetdb5/optimizer/optimizer.mal > sql/backends/monet5/UDF/Tests/udf-fuse.stable.out > sql/backends/monet5/UDF/Tests/udf-reverse.stable.out > sql/backends/monet5/sql.mal > sql/backends/monet5/sql_gencode.c > sql/benchmarks/ssbm/Tests/01-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/02-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/03-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/04-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/05-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/06-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/07-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/08-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/09-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/10-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/11-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/12-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/13-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/01-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/02-explain.stable.out > sql/benchmarks/tpch/Tests/03-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/04-explain.stable.out > sql/benchmarks/tpch/Tests/05-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/06-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/07-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/08-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/09-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/10-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/11-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/12-explain.stable.out > sql/benchmarks/tpch/Tests/12-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/13-explain.stable.out > sql/benchmarks/tpch/Tests/13-explain.stable.out.32bit > sql/benchmarks/tpch/Tests/14-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/15-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/16-explain.stable.out > sql/benchmarks/tpch/Tests/16-explain.stable.out.32bit > sql/benchmarks/tpch/Tests/17-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/18-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/19-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/20-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/21-explain.stable.out > sql/benchmarks/tpch/Tests/21-explain.stable.out.32bit > sql/benchmarks/tpch/Tests/22-explain.stable.out.int128 > sql/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.stable.out > sql/test/BugTracker-2010/Tests/LIMIT_OFFSET_big-endian.Bug-2622.stable.out > sql/test/BugTracker-2010/Tests/group-by_ordered_column.Bug-2564.stable.out > sql/test/BugTracker-2010/Tests/group-by_ordered_column.Bug-2564.stable.out.32bit > sql/test/BugTracker-2011/Tests/crash_on_alias.Bug-2798.stable.out.int128 > sql/test/BugTracker-2011/Tests/func_iter_vs_bulk.Bug-2826.stable.out > sql/test/BugTracker-2012/Tests/rewrite_like_into_likesubselect.Bug-3179.stable.out > sql/test/BugTracker-2013/Tests/between.Bug-3259.stable.out.int128 > sql/test/BugTracker-2014/Tests/manifold.Bug-3556.stable.out > sql/test/BugTracker/Tests/explain.SF-1739353.stable.out > sql/test/BugTracker/Tests/jdbc_no_debug.SF-1739356.stable.out > sql/test/Tests/setoptimizer.stable.err > sql/test/Tests/setoptimizer.stable.out > sql/test/Tests/setoptimizer.stable.out.Windows > sql/test/mergetables/Tests/mergequery.stable.out > Branch: default > Log Message: > > Addition of the candidatelist property > MAL variables can have a candidate list flag, which is set by > a cheap optimizer at the start of the pipe. > This property leads to different rendering of the variables. > > > diffs (truncated from 5982 to 300 lines): > > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -1559,6 +1559,7 @@ str MTIMEtzone_tostr(str *s, const tzone > str MTIMEunix_epoch(timestamp *ret); > void OPTaliasRemap(InstrPtr p, int *alias); > int OPTaliasesImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p); > +int OPTcandidatesImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p); > int OPTcoercionImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > int OPTcommonTermsImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > int OPTconstantsImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p); > @@ -2186,6 +2187,7 @@ str ifthenelseRef; > str ilikeRef; > str ilikeselectRef; > str ilikesubselectRef; > +str ilikethetasubselectRef; > str ilikeuselectRef; > void initHeartbeat(void); > void initLibraries(void); > @@ -2257,6 +2259,7 @@ str levenshteinbasic_impl(int *result, s > str likeRef; > str likeselectRef; > str likesubselectRef; > +str likethetasubselectRef; > str likeuselectRef; > str listRef; > str loadLibrary(str modulename, int flag); > @@ -2587,6 +2590,7 @@ str subsliceRef; > str subsortRef; > str subsumRef; > str subthetajoinRef; > +str subuniqueRef; > str sumRef; > int tableProp; > str takeOid(oid id, str *val); > diff --git a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.err b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.err > --- a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.err > +++ b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.err > @@ -30,16 +30,16 @@ stderr of test 'opt_sql_append` in direc > # 22:58:57 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" "--host=rome" "--port=36275" > # 22:58:57 > > > -MAPI = (monetdb) /var/tmp/mtest-28239/.s.monetdb.30808 > +MAPI = (monetdb) /var/tmp/mtest-21015/.s.monetdb.39072 > QUERY = explain copy into ttt from '\tmp/xyz'; > ERROR = !COPY INTO: filename must have absolute path: \tmp/xyz > -MAPI = (monetdb) /var/tmp/mtest-28239/.s.monetdb.30808 > +MAPI = (monetdb) /var/tmp/mtest-21015/.s.monetdb.39072 > QUERY = explain copy into ttt from 'a:\tmp/xyz'; > ERROR = !COPY INTO: filename must have absolute path: a:\tmp/xyz > -MAPI = (monetdb) /var/tmp/mtest-28239/.s.monetdb.30808 > +MAPI = (monetdb) /var/tmp/mtest-21015/.s.monetdb.39072 > QUERY = explain copy into ttt from '\tmp/xyz'; > ERROR = !COPY INTO: filename must have absolute path: \tmp/xyz > -MAPI = (monetdb) /var/tmp/mtest-28239/.s.monetdb.30808 > +MAPI = (monetdb) /var/tmp/mtest-21015/.s.monetdb.39072 > QUERY = explain copy into ttt from 'Z:/tmp/xyz'; > ERROR = !COPY INTO: filename must have absolute path: Z:/tmp/xyz > > diff --git a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > --- a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > +++ b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > @@ -55,8 +55,8 @@ Ready. > % .L # table_name > % def # name > % clob # type > -% 491 # length > -[ "optimizer.inline();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.garbageCollector();" ] > +% 514 # length > +[ "optimizer.inline();optimizer.candidates();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.garbageCollector();" ] > #explain copy into ttt from '/tmp/xyz'; > % .explain # table_name > % mal # name > @@ -89,8 +89,8 @@ end user.s4_1; > % .L # table_name > % def # name > % clob # type > -% 514 # length > -[ "optimizer.inline();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.sql_append();optimizer.garbageCollector();" ] > +% 537 # length > +[ "optimizer.inline();optimizer.candidates();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.sql_append();optimizer.garbageCollector();" ] > #explain copy into ttt from '/tmp/xyz'; > % .explain # table_name > % mal # name > diff --git a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out.single b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out.single > --- a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out.single > +++ b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out.single > @@ -55,8 +55,8 @@ Ready. > % .L # table_name > % def # name > % clob # type > -% 491 # length > -[ "optimizer.inline();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.garbageCollector();" ] > +% 514 # length > +[ "optimizer.inline();optimizer.candidates();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.garbageCollector();" ] > #explain copy into ttt from '/tmp/xyz'; > % .explain # table_name > % mal # name > @@ -87,8 +87,8 @@ end user.s4_1; > % .L # table_name > % def # name > % clob # type > -% 514 # length > -[ "optimizer.inline();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.sql_append();optimizer.garbageCollector();" ] > +% 537 # length > +[ "optimizer.inline();optimizer.candidates();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.sql_append();optimizer.garbageCollector();" ] > #explain copy into ttt from '/tmp/xyz'; > % .explain # table_name > % mal # name > diff --git a/monetdb5/mal/mal.h b/monetdb5/mal/mal.h > --- a/monetdb5/mal/mal.h > +++ b/monetdb5/mal/mal.h > @@ -120,7 +120,8 @@ mal_export void mal_exit(void); > #define VAR_CLEANUP 16 > #define VAR_INIT 32 > #define VAR_USED 64 > -#define VAR_DISABLED 128 /* used for comments and scheduler */ > +#define VAR_CLIST 128 /* Candidate list variable */ > +#define VAR_DISABLED 256 /* used for comments and scheduler */ > > /* type check status is kept around to improve type checking efficiency */ > #define TYPE_ERROR -1 > diff --git a/monetdb5/mal/mal_instruction.c b/monetdb5/mal/mal_instruction.c > --- a/monetdb5/mal/mal_instruction.c > +++ b/monetdb5/mal/mal_instruction.c > @@ -653,7 +653,7 @@ getVarName(MalBlkPtr mb, int i) > nme = mb->var[i]->name; > > if (nme == 0 || *nme =='_') { > - snprintf(buf, PATHLENGTH, "%c_%d", REFMARKER, mb->var[i]->tmpindex); > + snprintf(buf, PATHLENGTH, "%c_%d", refMarker(mb,i), mb->var[i]->tmpindex); > nme = mb->var[i]->name = GDKstrdup(buf); > } > return nme; > diff --git a/monetdb5/mal/mal_instruction.h b/monetdb5/mal/mal_instruction.h > --- a/monetdb5/mal/mal_instruction.h > +++ b/monetdb5/mal/mal_instruction.h > @@ -86,6 +86,10 @@ > #define setVarConstant(M,I) ((M)->var[I]->flags |= VAR_CONSTANT) > #define isVarConstant(M,I) ((M)->var[I]->flags & VAR_CONSTANT) > > +#define clrVarCList(M,I) ((M)->var[I]->flags &= ~VAR_CLIST) > +#define setVarCList(M,I) ((M)->var[I]->flags |= VAR_CLIST) > +#define isVarCList(M,I) ((M)->var[I]->flags & VAR_CLIST) > + > #define getVarConstant(M,I) ((M)->var[I]->value) > #define getVarValue(M,I) VALget(&(M)->var[I]->value) > > diff --git a/monetdb5/mal/mal_parser.c b/monetdb5/mal/mal_parser.c > --- a/monetdb5/mal/mal_parser.c > +++ b/monetdb5/mal/mal_parser.c > @@ -116,7 +116,9 @@ initParser(void) > static int > idLength(Client cntxt) > { > - str s, t; > + str s,t; > + int len = 0; > + > skipSpace(cntxt); > s = CURRENT(cntxt); > t = s; > @@ -128,9 +130,15 @@ idLength(Client cntxt) > s[0] = REFMARKER; > /* prepare escape of temporary names */ > s++; > - while (idCharacter2[(int) (*s)]) > + while (len < IDLENGTH && idCharacter2[(int) (*s)]){ > s++; > - return (int) (s - t); > + len++; > + } > + if( len == IDLENGTH) > + // skip remainder > + while (idCharacter2[(int) (*s)]) > + s++; > + return (int) (s-t);; > } > > /* Simple type identifiers can not be marked with a type variable. */ > diff --git a/monetdb5/mal/mal_type.h b/monetdb5/mal/mal_type.h > --- a/monetdb5/mal/mal_type.h > +++ b/monetdb5/mal/mal_type.h > @@ -15,6 +15,8 @@ > #define malVARG " malVARG" > #define TMPMARKER '_' > #define REFMARKER 'X' > +#define REFMARKERC 'C' > +#define refMarker(M,I) (isVarCList(M,I)? 'C':'X') > > #define newBatType(H,T) (1<<16 | (((TYPE_oid & 0377) <<8) | (T & 0377) )) > #define isaBatType(X) ((1<<16) & (X) && (X)!= TYPE_any) > diff --git a/monetdb5/optimizer/Makefile.ag b/monetdb5/optimizer/Makefile.ag > --- a/monetdb5/optimizer/Makefile.ag > +++ b/monetdb5/optimizer/Makefile.ag > @@ -21,6 +21,7 @@ lib_optimizer = { > opt_aliases.c opt_aliases.h \ > opt_coercion.c opt_coercion.h \ > opt_commonTerms.c opt_commonTerms.h \ > + opt_candidates.c opt_candidates.h \ > opt_constants.c opt_constants.h \ > opt_costModel.c opt_costModel.h \ > opt_dataflow.c opt_dataflow.h \ > diff --git a/monetdb5/optimizer/opt_pipes.c b/monetdb5/optimizer/opt_pipes.c > --- a/monetdb5/optimizer/opt_pipes.c > +++ b/monetdb5/optimizer/opt_pipes.c > @@ -46,6 +46,7 @@ static struct PIPELINES { > */ > {"minimal_pipe", > "optimizer.inline();" > + "optimizer.candidates();" > "optimizer.remap();" > "optimizer.deadcode();" > "optimizer.multiplex();" > @@ -63,6 +64,7 @@ static struct PIPELINES { > */ > {"default_pipe", > "optimizer.inline();" > + "optimizer.candidates();" > "optimizer.remap();" > "optimizer.costModel();" > "optimizer.coercions();" > @@ -99,6 +101,7 @@ static struct PIPELINES { > {"no_mitosis_pipe", > "optimizer.inline();" > "optimizer.remap();" > + "optimizer.candidates();" > "optimizer.costModel();" > "optimizer.coercions();" > "optimizer.evaluate();" > @@ -132,6 +135,7 @@ static struct PIPELINES { > */ > {"sequential_pipe", > "optimizer.inline();" > + "optimizer.candidates();" > "optimizer.remap();" > "optimizer.costModel();" > "optimizer.coercions();" > diff --git a/monetdb5/optimizer/opt_prelude.c b/monetdb5/optimizer/opt_prelude.c > --- a/monetdb5/optimizer/opt_prelude.c > +++ b/monetdb5/optimizer/opt_prelude.c > @@ -114,10 +114,10 @@ str kdifferenceRef; > str languageRef; > str leftfetchjoinRef; > str leftfetchjoinPathRef; > -str likeselectRef; > -str ilikeselectRef; > -str likeuselectRef; > -str ilikeuselectRef; > +str likesubselectRef; > +str likethetasubselectRef; > +str ilikesubselectRef; > +str ilikethetasubselectRef; > str likeRef; > str ilikeRef; > str not_likeRef; > @@ -211,6 +211,7 @@ str subavgRef; > str subsortRef; > str takeRef; > str not_uniqueRef; > +str subuniqueRef; > str unlockRef; > str unpackRef; > str unpinRef; > @@ -357,10 +358,8 @@ void optimizerInit(void) > languageRef= putName("language",8); > leftfetchjoinRef = putName("leftfetchjoin",13); > leftfetchjoinPathRef = putName("leftfetchjoinPath",17); > - likeselectRef = putName("like_select",11); > - ilikeselectRef = putName("ilike_select",12); > - likeuselectRef = putName("like_uselect",12); > - ilikeuselectRef = putName("ilike_uselect",13); > + likesubselectRef = putName("likesubselect",13); > + ilikesubselectRef = putName("ilikesubselect",14); > listRef = putName("list",4); > likeRef = putName("like",4); > ilikeRef = putName("ilike",5); > @@ -455,6 +454,7 @@ void optimizerInit(void) > takeRef= putName("take",5); > timestampRef = putName("timestamp", 9); > not_uniqueRef= putName("not_unique",10); > + subuniqueRef= putName("subunique",9); > unlockRef= putName("unlock",6); > unpackRef = putName("unpack",6); > unpinRef = putName("unpin",5); > @@ -462,7 +462,9 @@ void optimizerInit(void) > subselectRef = putName("subselect",9); > thetasubselectRef = putName("thetasubselect",14); > likesubselectRef = putName("likesubselect",13); > + likethetasubselectRef = putName("likethetasubselect",18); > ilikesubselectRef = putName("ilikesubselect",14); > + ilikethetasubselectRef = putName("ilikethetasubselect",19); > vectorRef = putName("vector",6); > zero_or_oneRef = putName("zero_or_one",11); > userRef = putName("user",4); > diff --git a/monetdb5/optimizer/opt_prelude.h b/monetdb5/optimizer/opt_prelude.h > --- a/monetdb5/optimizer/opt_prelude.h > +++ b/monetdb5/optimizer/opt_prelude.h > @@ -117,10 +117,6 @@ opt_export str ilikeselectRef; > opt_export str likeuselectRef; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

MonetDB type system documentation
by Stefan Manegold 07 Oct '15

07 Oct '15

Dear all, since there is ever so often confusion about the (admittedly not entirely trivial) type system of MonetDB, and about which type to use for what to ensure correct semantics and portability, I felt free to collect some of the crucial information on the MonetDB wiki. Under https://www.monetdb.org/wiki/Main_Page following links "3.1 Software development" and then "MonetDB type system" gets you to https://www.monetdb.org/wiki/MonetDB_type_system I hope this helps to clarify some details. In particular mind the "Type Rules" given there (with any of your MonetDB-related code that you eventually want to see in some future MonetDB release). Your cooperation is highly appreciated! In case you have any questions about correct and proper type usage in MonetDB, please do not hesitate to ask Sjoerd or myself (preferably via the MonetDB developers mailing list, if not in person). Enjoy! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Getting a column type for result column
by Dimitar Nedev 03 Oct '15

03 Oct '15

Dear co-developers, How does one get the type of a column given a 'res_col'. I mean a humanly readable one, in other words get the value that determines the type and how they map to data types. Cheers, Dimitar

2 2

Server crash when trying convert timestamp to string with incorrect format
by Sherzod Mutalov 01 Oct '15

01 Oct '15

Hello Developers, MonetDB server crashes when trying to execute timestamp_to_str function with incorrect format string. SQL: SELECT timestamp_to_str(current_timestamp, '%Q'); I send bug report, https://www.monetdb.org/bugzilla/show_bug.cgi?id=3816 But, also I need QUARTER extraction from date values by SQL server, like "extract(quarter from date '2015-10-01')" = 4 Now, I am using this "((extract(month from {0}) - 1) / 3 + 1)" -- Best regards, Sherzod mailto:shmutalov@gmail.com

1 0

Some annoying bugs with date and number operations
by Sherzod Mutalov 01 Oct '15

01 Oct '15

Hello Developers, Firstly, I must say thank you for releasing MonetDb Jul2015. Secondly, Problem 1: Following query crashes the mserver5: SELECT ~0; -- here can be any number or numeric column There is NULL dereference problem in rel_select.c file at line 3915 (rel_binop function). dl->next->next is null, but rel_value_exp receives dl->next->next->data.sym as argument. How correctly use bitwise NOT operator with SQL? Problem 2: Following queries produces incorrect results: SELECT ( (current_timestamp - str_to_timestamp('2015-09-30', '%Y-%m-%d'))/86400 , BIGINT); SELECT CONVERT( CONVERT( (current_timestamp - str_to_timestamp('2015-09-30', '%Y-%m-%d')) , BIGINT) /86400 , BIGINT); SELECT (current_timestamp - str_to_timestamp('2015-09-30', '%Y-%m-%d'))/86400.0; Server info: # MonetDB 5 server v11.21.5 "Jul2015" # Serving database 'demo', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 15.916 GiB available main-memory. -- Best regards, Sherzod mailto:shmutalov@gmail.com

2 2

Inserting in 'raw' mode??
by Vijay Krishna 15 Sep '15

15 Sep '15

Hi, I was analysing the Insert performance of MonetDB... I am trying to optimize a process where in a single tuple has to be inserted into a table of 1200 columns, which takes about 7.5 seconds presently... The query has values for only 5 columns out of 1200, yet the speed is still the same.. Is it possible to reduce this time even?? While digging on this, I came across this thread.. https://www.monetdb.org/pipermail/users-list/2005-August/000139.html *What is the insert speed relatively to other DBs? > * > Using SQL, with the current version we won't compete very well. We get > much more acceptable speeds in 'raw' mode, where lots of the parsing > overhead is removed. > > Can anyone please help me understand what is the 'raw' mode that is explained in this thread?? I am using Oct-2014 version on a CentOS machine with 128 GB RAM... Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

2 1

Why doesn't MonetDB exploit SIMD feature?
by Xu,Wenjian 15 Sep '15

15 Sep '15

Hi, We know that MonetDB excels in application where the database hot-set can be largely held in main-memory, since the physical operators (e.g., hashjoin, sort) are highly optimized for main-memory. But it is strange to me that these operators do not utilize SIMD which a performance-critical feature of modern CPUs. For example, MonetDB use *timsort* as its underlying stable sort algorithm (do_ssort() in gdk/gdk_ssort_impl.h), but I cannot find any SIMD instructions there. Why doesn't MonetDB exploit SIMD feature? or did I miss something? In paper 'Vectorwise: Beyond Column Stores', the authors claim that X100/Vectorwise use SIMD instructions. So my follow-up question is that whether any of X100/Vectorwise's technologies/features has been integrated into MonetDB's code base. Thank you. Best regards, Xu, Wenjian

1 0

DbVisualizer support for MonetDB
by Robert Koch 14 Sep '15

14 Sep '15

Hi Folks, DbVisualizer is my preferred GUI to MonetDB. DbVis offers capability to write own database profiles for specific DB's. Today I committed a MonetDB database profile for DbVisualizer. https://github.com/dbvisualizer/dbvis-database-profiles/tree/master/monetdb Anyone familiar to DbVis can import and/or improve this profile. There's a free DbVis version, but the Pro offers much more features, which make life easier. BTW, I'm not working for them. :-) Please feel free to write me your desired features by mail or directly into the MD-file inside the monetdb-directory. If possible, I will implement it inte the profile. Best regards Mario

1 0

Announcement: New Jul2015 Feature release of MonetDB suite
by Sjoerd Mullender 09 Sep '15

09 Sep '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015 feature release MonetDB5 Server * Implemented batcalc.min and batcalc.max. Made calc.min and calc.max generic so that no other implementations are needed. * Removed function batcalc.ifthen. Build Environment * Upgraded the license to the Mozilla Public License Version 2.0. Client Package * Added a new output format to mclient: --format=expanded (or -fx). In this format, column values are shown in full and below each other. MonetDB Common * BUNtvar and BUNhvar macros no longer work for TYPE_void columns. * Changed interfaces of a lot of GDK-level functions. When they modify a BAT, don't return the same BAT or NULL, but instead return GDK_SUCCEED or GDK_FAIL. * Changed a bunch of hash-related functions to work on the tail column. The functions that have been changed to work on the tail column are: BAThash, BATprepareHash, HASHgonebad, HASHins, and HASHremove. Java Module * Improved JDBC driver to not throw NullPointerException anymore when calling isNullable() or getPrecision() or getScale() or getColumnDisplaySize() or getSchemaName() or getTableName() or getColumnClassName() on a ResultSetMetaData object. * We now compile the Java classes using the latest Java 1.8 version, and we tell it to compile for Java 1.7. SQL * Differences between time, timestamp, and date values now return properly typed interval types (second or month intervals) instead of integers. * Added support for 128-bit integers (called HUGEINT) on platforms that support this. * Removed support for the mseed library. * Removed support for RDF. * Removed DataCell. It was experimental code that was never enabled. Bug Fixes * 2618: Implement master slave scheme * 3361: constants as MAL function parameters prevent intermediate reuse * 3364: Cannot set role back to a user's default role * 3365: Unable to grant object privileges while having a non-default current_role * 3440: Sequence type errors * 3449: mserver crash on start - Freebsd 10 amd64 * 3476: Cannot revoke object access * 3496: autocompletion table names does not work correctly * 3556: when 2 multiplexed functions in MAL plan, only one is mapped correctly to bat<mod>.function primitive * 3564: Request: add support for postgresql specific scalar function: split_part(string text, delimiter text, field int) * 3625: SIGSEGV because mat array can overrun in opt_mergetable.c * 3627: SQRT in CASE does not work as of Oct2014 * 3654: configure --enable-fits requires extra commands after creating a database instance * 3673: mclient 'expanded' row formatter * 3674: Obfuscate event tracing * 3679: No error is given when incorrect timezone value is specified for a timetz column * 3686: Wrong associativity of multiply/divide * 3702: Filter function not found if created in a user schema * 3708: wrong scoping for cross-schema view references * 3716: alter table my_merge_table drop table t1; crashes mserver5 with Segmentation fault * 3724: Wrong size calculation in BATsubjoin * 3730: SAMPLE function not sampling randomly * 3732: memory leak (of InstrRecord) in opt_mergetable * 3733: "(TRUE OR <Exp>) AND <Exp>" is evaluated incorrectly * 3735: python connection with unix_socket * 3736: crash if mclient disconnects abruptly during a query * 3738: Database inconsistency when using savepoint * 3739: CASE statements do not handle NULLs in the IN () operator properly * 3740: select epoch(now()); types timestamptz(7,0) and bigint(64,0) are not equal * 3742: Division By Zero * 3744: cast to int gives different results for decimal than double * 3747: joins fail in the presence of nulls * 3748: Missing META-INF/services/java.sql.Driver in JDBC package * 3753: Hang on json field parsing * 3754: select from a REMOTE TABLE referring local table crashes mserver5 * 3756: column type conversion sticks to subsequent queries * 3758: "COPY INTO ..." doesn't work, if executing from 2 processes concurrently. * 3759: select data from "sys"."rejects" returns unexpected error and when next select data from "sys"."sessions" causes an assertion failure in mal_interpreter.c:646. * 3760: SQL parser has problem with (position of) a scalar subquery in a SELECT-list * 3761: SQL executor has problem with (position of) a subquery in a SELECT-list. Inconsistent behavior. * 3763: JDBC PreparedStatement for a table with 14 Foreign Keys crashing the Database * 3764: DROPping multiple users causes a crash * 3765: Re-granting a revoked privilege does not work * 3766: VIEW not visible if created under a different schema * 3767: CREATE TEMP TABLE using "LIKE" incorrectly handled * 3769: SIGSEGV when combining a cast/column alias with a UNION ALL view * 3770: combined conditions on declared table in User Defined Function definition crashes monetdb * 3771: Owner of the schema loses rights if assumes the monetdb role. * 3772: Any user can grant a role. * 3773: quantile(col, 0) and quantile(col, 1) fail * 3774: mclient is unaware of merge tables and remote tables * 3775: COPY INTO: Backslash preceding field separator kills import * 3778: Crash on remote table schema mismatch * 3779: server crashes on MAX() on SELECT DISTINCT something combo * 3783: Behavioural change in Jul2015 for 'timestamp minus timestamp' * 3784: Assertion failed: (bn->batCapacity >= cnt), function BAT_scanselect, file gdk_select.c, line 1008. * 3785: sum(interval) causes overflow in conversion to bte * 3786: ResultSet.close() never sends Xclose to free resources * 3787: "b and g must be aligned" from complex group/union query * 3791: HEAPextend: failed to extend to 2420077101056

1 1

Install Monetdb
by HENGKI EFFENDI 01 Sep '15

01 Sep '15

Hi, I try to install Monetdb in Linux RH, when configure there's error message : configure: error: MonetDB/SQL requires bison Please help me, what should i do. Regards Hengki

2 1

How to get scalar value of "floatlike" decimal?
by Sherzod Mutalov 19 Aug '15

19 Aug '15

Hello Developers-list, I am writing some C function (something like "cast"), and I have problem with proper handling of value type "any". Say, I have C function: str testFn(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) { str* ret = getArgReference_str(stk, pci, 0); ValPtr sourceValue = &stk->stk[pci->argv[1]]; char tmp[256] = {0}; sprintf(&tmp[0], "Source value type is \"%s\" (%d)\n", getTypeName(sourceValue->vtype), sourceValue->vtype); *ret = GDKstrdup(&tmp[0}); return MAL_SUCCEED; } MAL function: pattern testFn(value:any_1):str address testFn comment "Test function"; Then I added follow code to sql_types.c, where all built-in types and functions gets initialized: sql_create_func(sa, "testFn", "somemodule", "testFn", ANY, NULL, STR, SCALE_NONE); When I want to pass "float" number, I get only integer, for example: SELECT testFn(123.12); I will get "int" type of ValRecord. And value of passed argument will be "12312" (without dot-delimiter). But, I expecting double or float value. Please, help to solve this. -- Best regards, Sherzod mailto:shmutalov@gmail.com

2 1

Re: Request Buffer pools
by Vijay Krishna 13 Aug '15

13 Aug '15

Thanks for the links Brian.. Dear all, >From this, I see that a database can be set to any number of clients, which gives me a question on number of queries processed at a time. If I am using an 8 core machine and connecting 50 client threads (each executing many queries one by one) at a time to a database, will the mserver be processing queries from ALL the 50 clients at any given instance?? Or will that be restricted by the number of cores?? If this is restricted, then is there a default queue maintained by the server?? Put in simpler words, is the maximum number of queries processed by MonetDB at a given time equal to the number of client connections?? If not, then how is this handled (with a queue) ?? Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305. On Wed, Aug 12, 2015 at 6:11 PM, Brian Hood <brianh6854(a)googlemail.com> wrote: > Hi Vijay, > > I have found some interesting info in the monetdb man page actually. > > --- This is interests me as it alllows you to limit the amount of clients > that can connect to a database not just an mserver > > *nclients=**number* > > Sets the maximum amount of clients that can connect to this database at > the same time. Setting this to a high value is discouraged. A > multiplex-funnel may be more performant, see *MULTIPLEX-FUNNEL* below. > > Regards, > > Brian Hood > > On Wed, Aug 12, 2015 at 7:48 AM, Vijay Krishna <vijayakrishna55(a)gmail.com> > wrote: > >> Thanks Stefan and Brian. >> >> When I went through monetdb man page, the usage of nthreads (set as >> gdk_nr_threads) environmental variable was defined as >> >> >> Defines how many worker threads the server should use to perform main >> processing. Normally, this number equals the number of available CPU cores >> in the system. Reducing this number forces the server to use less >> parallelism when executing queries, or none at all if set to *1*. >> I am a bit confused in understanding this statement. The variable >> "nthreads" controls >> >> 1. the number of threads processing a single query using multiple >> cores. >> 2. the number of queries currently being executed over all databases >> by the mserver. >> >> Please help me understand which of these two is correct.. >> This doubt arises because I have read from MonetDB home page that "MonetDB >> is designed for multi-core parallel execution on desktops to reduce >> response time for complex query processing". >> >> If the variable "nthreads" controls the latter, then does monetdb have an >> in built queue to store the queries from remaining connections (controlled >> by the variable nclients) to the mserver?? I have read about multiplex >> funnel and a bit afraid over the transactional disadvantages mentioned in >> there.. >> Is there any other solution? >> >> >> >> >> Thanks & Regards, >> >> Vijayakrishna.P. >> Mobile : (+91) 9500402305. >> >> On Tue, Aug 11, 2015 at 6:56 PM, Stefan Manegold <Stefan.Manegold(a)cwi.nl> >> wrote: >> >>> Hi, >>> >>> indeed; >>> >>> see also >>> `man mserver5` and `man monetdb` >>> or >>> https://www.monetdb.org/Documentation/mserver5-man-page >>> and >>> https://www.monetdb.org/Documentation/monetdb-man-page >>> >>> Stefan >>> >>> ----- On Aug 11, 2015, at 3:11 PM, Brian Hood brianh6854(a)googlemail.com >>> wrote: >>> >>> > Hi Vijay, >>> > >>> > Well if you take a look at your merovingian.log you can see the >>> command that >>> > starts the database. >>> > >>> > /usr/bin/mserver5 --dbpath=/data2/threatmonitor/threatmonitor --set >>> > merovingian_uri=mapi:monetdb://orville.lan:50000/threatmonitor --set >>> > mapi_open=false --set mapi_port=0 --set >>> > mapi_usock=/data2/threatmonitor/threatmonitor/.mapi.sock --set >>> > monet_vault_key=/data2/threatmonitor/threatmonitor/.vaultkey --set >>> > gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe >>> --set >>> > monet_daemon=yes >>> > >>> > >>> > These 2 i think are the ones your looking for. >>> > >>> > --set gdk_nr_threads=2 --set max_clients=64 >>> > >>> > Regards, >>> > >>> > Brian Hood >>> > >>> > >>> > On Tue, Aug 11, 2015 at 7:14 AM, Vijay Krishna < >>> vijayakrishna55(a)gmail.com > >>> > wrote: >>> > >>> > >>> > >>> > Oops.. My bad.. Thanks Brian.. >>> > >>> > Yes I was indeed looking for connection pools... Something like >>> maximum thread >>> > concurrency option in innodb... >>> > >>> > Is there any such available with MonetDB? >>> > >>> > I had gone through this option... mserver5 --set max_clients=<num> >>> > Is this the only way to control parallel threads?? >>> > >>> > >>> > >>> > >>> > >>> > Thanks & Regards, >>> > >>> > Vijayakrishna.P. >>> > Mobile : (+91) 9500402305 . >>> > >>> > On Mon, Aug 10, 2015 at 7:48 PM, Brian Hood < >>> brianh6854(a)googlemail.com > wrote: >>> > >>> > >>> > >>> > Hi Vijay, >>> > >>> > I think the words your looking for a connection pooling. >>> > >>> > InnoDB Buffer pool in MySQL is the amount of data stored in memory >>> which you set >>> > to a certain size and has nothing to do with network traffic. >>> > >>> > Regards, >>> > >>> > Brian Hood >>> > >>> > On Mon, Aug 10, 2015 at 8:45 AM, Vijay Krishna < >>> vijayakrishna55(a)gmail.com > >>> > wrote: >>> > >>> > >>> > >>> > Hi, >>> > >>> > Does MonetDB have any built in buffer pools to handle parallel >>> requests? >>> > >>> > I see the default number of parallel connections accepted and >>> processed at a >>> > time by MonetDB is 64.. >>> > While I see that there are ways to increase it, is there anyway built >>> in to pool >>> > incoming requests until the live 64 requests are processed? >>> > >>> > For example, our application would receive 100 requests at a time, >>> while we wish >>> > to limit the number of queries handled at a time by MonetDB to 50. Is >>> there any >>> > way to automatically handle the remaining 50 (something like innodb >>> buffer >>> > pool) ?? >>> > >>> > Thanks & Regards, >>> > >>> > Vijayakrishna.P. >>> > Mobile : (+91) 9500402305 . >>> > >>> > _______________________________________________ >>> > users-list mailing list >>> > users-list(a)monetdb.org >>> > https://www.monetdb.org/mailman/listinfo/users-list >>> > >>> > >>> > >>> > _______________________________________________ >>> > users-list mailing list >>> > users-list(a)monetdb.org >>> > https://www.monetdb.org/mailman/listinfo/users-list >>> > >>> > >>> > >>> > _______________________________________________ >>> > users-list mailing list >>> > users-list(a)monetdb.org >>> > https://www.monetdb.org/mailman/listinfo/users-list >>> > >>> > >>> > >>> > _______________________________________________ >>> > users-list mailing list >>> > users-list(a)monetdb.org >>> > https://www.monetdb.org/mailman/listinfo/users-list >>> >>> -- >>> | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | >>> | www.CWI.nl/~manegold/ | Science Park 123 (L321) | >>> | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | >>> _______________________________________________ >>> users-list mailing list >>> users-list(a)monetdb.org >>> https://www.monetdb.org/mailman/listinfo/users-list >>> >> >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list >> >> > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > >

1 0

Re: MonetDB causing disk space to fill up
by Anu Raj Srivastava 12 Aug '15

12 Aug '15

I am sorry didn¹t realize mailing list is not their in my reply. On 7/20/15, 14:24, "Ying Zhang" <Y.Zhang(a)cwi.nl> wrote: >Hello Anu, > >Would you please post this to the MonetDB users mailing list? Thanks! > >Regards, > >Jennie > >> On Jul 20, 2015, at 19:44, Anu Raj Srivastava <Anu.Srivastava(a)rms.com> >>wrote: >> >> Hi Hannes, >> >> I am using apache dbcp2 connection Pool. I am running queries in a >>constant loop with 8 concurrent invocations. And each 8 threads are >>running 9 queries each again and again. >> >> Java code is quite straight forward:- >> >> private ConnectionPoolPG() { >> try { >> final Properties properties = new Properties(); >> >>properties.load(ConnectionPoolPG.class.getClassLoader().getResourceAsStre >>am("monetDB.properties")); >> dataSource = >>BasicDataSourceFactory.createDataSource(properties); >> schemaName = properties.getProperty("schemaname"); >> } catch (Exception e) { >> throw new ExposureStoreException("Unable to instantiate >>connection pool: " + e.getMessage(), >> EAPAException.ERR_STORE_CONNECTION_CONFIG_ERROR); >> } >> } >> >> public void AllQueryPerformance() throws Exception { >> while (true) { >> if (queries != null) { >> for (String query : queries) { >> try (Connection connection = >>ConnectionPoolPG.getInstance().getConnection()) { >> try (Statement statement = >>connection.createStatement()) { >> statement.execute(query); >> } >> } >> } >> } >> } >> } >> >> -----Original Message----- >> From: Hannes Mühleisen [mailto:Hannes.Muehleisen@cwi.nl] >> Sent: Sunday, July 19, 2015 4:47 AM >> To: Communication channel for developers of the MonetDB suite. >> Cc: Ying Zhang; Anu Raj Srivastava >> Subject: Re: MonetDB causing disk space to fill up >> >> >>> On 19 Jul 2015, at 13:23, Hannes Mühleisen <Hannes.Muehleisen(a)cwi.nl> >>>wrote: >>> >>> Hi, >>> >>>> On 16 Jul 2015, at 23:11, Ying Zhang <Y.Zhang(a)cwi.nl> wrote: >>>> >>>> On Jul 13, 2015, at 22:32, Anu Raj Srivastava >>>><Anu.Srivastava(a)rms.com> wrote: >>>>> >>>>> Hey Jennie, >>>>> >>>>> I did upgrade to latest version but I am facing the same issue. >>>>>Further debugging into the issue it seems like it is ConnectionPool >>>>>issue. When I use new connection for each query MonetDB seems to be >>>>>able to clean up the resources >>>> >>>> Ok, MonetDB is doing what it¹s supposed to do. >>>> >>>>> but if I use ConnectionPool for some reason DiskSpace keep filling >>>>>up. >>>> >>>> If a connection is kept open, I¹d expect MonetDB to be less eager to >>>>clean up resources. >>>> However, we generally recommend using a ConnectionPool, to avoid too >>>>much overhead caused by making a new connection for every query. So >>>>my suggestion for now would be to keep using the ConnectionPool, but >>>>try to find the reason why your disk is filled up. >>>> >>>>> Do you have any idea why that could happen? >>>> >>>> Hmmm, no. We¹ve fixed quite some leaks in Jul2015... >>>> Some more thinking: >>>> - After how many iterations of repeatedly executing the queries is >>>>your disk filled up? >>>> - Can you please try to run your queries with mclient to observe the >>>>behaviour of MonetDB (and disk usage)? >>>> >>>> @Hannes: sorry to disturb you during holiday. But if possible, would >>>>you please have a look at this thread. Does anything ring a bell? >>>>Thanks! >>> >>> I tested this with ac3p0 connection pool (limited to 4 concurrent >>>connections) and repeating queries from 10 threads (SELECT >>>MIN(something) FROM Š). I was not able to observe any (!) increase in >>>storage footprint. So without a minimal reproducible example of Java >>>code + DB schema + sample data, it is hard to tell what is going on. Is >>>this running in auto commit? Using temporary tables? >> >> Small follow-up: When forcing the allocation of a large intermediate >>(SELECT MIN(something+1) FROM Š), additional storage space is indeed >>claimed. The attached plot shows dbfarm size over time while 10 threads >>are running 1000 queries each using a 5-connection pool. You can see how >>the amount of additional storage drops back to exactly where it was once >>the pool shuts down at around 350s, and that there is no accumulation of >>additional storage over the course of the pool being active. >> >> Hannes >> >

2 12

TRACE option / runtime feature gone in Jul2015 ?
by Stefan Manegold 11 Aug '15

11 Aug '15

Hi, I might have missed that, but the TRACE option / runtime feature to profile SQL queries (cf., https://www.monetdb.org/Documentation/Manuals/SQLreference/Runtime) appears to be gone in Jul2015. Bug or feature? Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

3 3

*Sub* prefix for operators in MonetDB
by Xu,Wenjian 06 Aug '15

06 Aug '15

Hi, When browsing the source code of MonetDB, I found that functions for operators all have a *sub* prefix, e.g., BATsubsort, BATsubjoin, etc. Any information about how to understand this *sub* functionality? Thanks. Best regards, Wenjian

2 1

Inserting record between table rows
by Sherzod Mutalov 31 Jul '15

31 Jul '15

Hello Developers, How can I insert record(s) between two rows of SQL table? For example, I have table with three column and there are some records in it: | column1 | column2 | column3 | ------------------------------- | 123 | asffasf | true | ------------------------------- | 432 | lkjlkhk | false | ------------------------------- | 555 | sadda | true | ------------------------------- | ... | ... | ... | ------------------------------- I need to add a new record (555, "asdas", false) between first and second record of the existing table. Result what I expect: | column1 | column2 | column3 | ------------------------------- | 123 | asffasf | true | ------------------------------- | 555 | asdas | false | ------------------------------- | 432 | lkjlkhk | false | ------------------------------- | 555 | sadda | true | ------------------------------- | ... | ... | ... | ------------------------------- -- Best regards, Sherzod mailto:shmutalov@gmail.com

2 1

Examples for changing table scheme from MAL/C
by Sherzod Mutalov 27 Jul '15

27 Jul '15

Hello Developers, I need some help to understand monetdb kernel/sql structure. Please, provide example code/paper or link for follow situation: I have a table (SQL), and I want to change structure of it (add/drop/rename/order column). I know, I can do it with SQL, but I need to view examples in MAL or C. I want to write function like this: int add_column(str schemaName, str tableName, str newColumnName) { // return 1 if column successfully created } Before posting this message I saw the source code of SQL kernel and found some functions like sys_drop_column, mvc_create_column and others. But, I don't know how to execute these functions from my own user defined C/MAL function. -- Best regards, Sherzod mailto:shmutalov@gmail.com

1 1

Re: MonetDB: Jul2015 - Try not to overflow snprintf buffers.
by Ying Zhang 24 Jul '15

24 Jul '15

> On Jul 24, 2015, at 16:36, Sjoerd Mullender <commits(a)monetdb.org> wrote: > > Changeset: c84b68960bca for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=c84b68960bca > Modified Files: > clients/mapiclient/ReadlineTools.c > clients/mapiclient/dump.c > clients/mapilib/mapi.c > clients/odbc/driver/SQLExecute.c > common/options/monet_options.c > common/stream/stream.c > common/utils/mutils.c > gdk/gdk_bbp.c > Branch: Jul2015 > Log Message: > > Try not to overflow snprintf buffers. > > > diffs (truncated from 373 to 300 lines): > > diff --git a/clients/mapiclient/ReadlineTools.c b/clients/mapiclient/ReadlineTools.c > --- a/clients/mapiclient/ReadlineTools.c > +++ b/clients/mapiclient/ReadlineTools.c > @@ -61,12 +61,16 @@ sql_tablename_generator(const char *text > static MapiHdl table_hdl; > > if (!state) { > - char query[512]; > + char *query; > > seekpos = 0; > len = strlen(text); > - snprintf(query, sizeof(query), "SELECT t.\"name\", s.\"name\" FROM \"sys\".\"tables\" t, \"sys\".\"schemas\" s where t.schema_id = s.id AND t.\"name\" like '%s%%'", text); > - if ((table_hdl = mapi_query(_mid, query)) == NULL || mapi_error(_mid)) { > + if ((query = malloc(len + 128)) == NULL) > + return NULL; > + snprintf(query, len + 128, "SELECT t.\"name\", s.\"name\" FROM \"sys\".\"tables\" t, \"sys\".\"schemas\" s where t.schema_id = s.id AND t.\"name\" like '%s%%'", text); > + table_hdl = mapi_query(_mid, query); > + free(query); > + if (table_hdl == NULL || mapi_error(_mid)) { > if (table_hdl) { > mapi_explain_query(table_hdl, stderr); > mapi_close_handle(table_hdl); > @@ -176,7 +180,7 @@ static char *mal_commands[] = { > static int > mal_help(int cnt, int key) > { > - char *name, *c, buf[512]; > + char *name, *c, *buf; > int seekpos = 0, rowcount; > MapiHdl table_hdl; > > @@ -188,8 +192,12 @@ mal_help(int cnt, int key) > c--; > while (c > rl_line_buffer && !isspace(*c)) > c--; > - snprintf(buf, sizeof(buf), "manual.help(\"%s\");", c); > - if ((table_hdl = mapi_query(_mid, buf)) == NULL || mapi_error(_mid)) { > + if ((buf = malloc(strlen(c) + 20)) == NULL) > + return 0; > + snprintf(buf, strlen(c) + 20, "manual.help(\"%s\");", c); > + table_hdl = mapi_query(_mid, buf); > + free(buf); > + if (table_hdl == NULL || mapi_error(_mid)) { > if (table_hdl) { > mapi_explain_query(table_hdl, stderr); > mapi_close_handle(table_hdl); > @@ -220,7 +228,7 @@ mal_command_generator(const char *text, > static int idx; > static int seekpos, len, rowcount; > static MapiHdl table_hdl; > - char *name, buf[512]; > + char *name, *buf; > > /* we pick our own portion of the linebuffer */ > text = rl_line_buffer + strlen(rl_line_buffer) - 1; > @@ -250,14 +258,18 @@ mal_command_generator(const char *text, > text = c + 2; > while (isspace((int) *text)) > text++; > + if ((buf = malloc(strlen(text) + 32)) == NULL) > + return NULL; > if (strchr(text, '.') == NULL) > - snprintf(buf, sizeof(buf), > + snprintf(buf, strlen(text) + 32, > "manual.completion(\"%s.*(\");", text); > else > - snprintf(buf, sizeof(buf), > + snprintf(buf, strlen(text) + 32, > "manual.completion(\"%s(\");", text); > seekpos = 0; > - if ((table_hdl = mapi_query(_mid, buf)) == NULL || mapi_error(_mid)) { > + table_hdl = mapi_query(_mid, buf); > + free(buf); > + if (table_hdl == NULL || mapi_error(_mid)) { > if (table_hdl) { > mapi_explain_query(table_hdl, stderr); > mapi_close_handle(table_hdl); > diff --git a/clients/mapiclient/dump.c b/clients/mapiclient/dump.c > --- a/clients/mapiclient/dump.c > +++ b/clients/mapiclient/dump.c > @@ -212,7 +212,7 @@ dump_foreign_keys(Mapi mid, const char * > "fkt.name = '%s' " > "ORDER BY fkk.name, nr", schema, tname); > } else if (tid != NULL) { > - maxquerylen = 1024; > + maxquerylen = 1024 + strlen(tid); > query = malloc(maxquerylen); > snprintf(query, maxquerylen, > "SELECT ps.name, " /* 0 */ > @@ -529,6 +529,8 @@ dump_column_definition(Mapi mid, stream > maxquerylen = 1024; > if (tid == NULL) > maxquerylen += strlen(tname) + strlen(schema); > + else > + maxquerylen += strlen(tid); > if ((query = malloc(maxquerylen)) == NULL) > goto bailout; > > diff --git a/clients/mapilib/mapi.c b/clients/mapilib/mapi.c > --- a/clients/mapilib/mapi.c > +++ b/clients/mapilib/mapi.c > @@ -1918,7 +1918,7 @@ mapi_new(void) > Mapi > mapi_mapiuri(const char *url, const char *user, const char *pass, const char *lang) > { > - char uri[8096]; > + char *uri; > char *host; > int port; > char *dbname; > @@ -1961,12 +1961,13 @@ mapi_mapiuri(const char *url, const char > } > > /* copy to a writable working buffer */ > - snprintf(uri, 8096, "%s", url + sizeof("mapi:monetdb://") - 1); > + uri = strdup(url + sizeof("mapi:monetdb://") - 1); > > if (uri[0] != '/') { > if ((p = strchr(uri, ':')) == NULL) { > mapi_setError(mid, "URI must contain a port number after " > "the hostname", "mapi_mapiuri", MERROR); > + free(uri); > return mid; > } > *p++ = '\0'; > @@ -1990,6 +1991,7 @@ mapi_mapiuri(const char *url, const char > if (port <= 0) { > mapi_setError(mid, "URI contains invalid port", > "mapi_mapiuri", MERROR); > + free(uri); > return mid; > } > } else { > @@ -2029,6 +2031,7 @@ mapi_mapiuri(const char *url, const char > mid->database = strdup(dbname); > > set_uri(mid); > + free(uri); > > return mid; > } > @@ -2128,7 +2131,8 @@ mapi_mapi(const char *host, int port, co > while ((e = readdir(d)) != NULL) { > if (strncmp(e->d_name, ".s.monetdb.", 11) != 0) > continue; > - snprintf(buf, sizeof(buf), "/tmp/%s", e->d_name); > + if (snprintf(buf, sizeof(buf), "/tmp/%s", e->d_name) >= sizeof(buf)) Hai Sjoerd, This line triggers the compiler error: /.../src/MonetDB-11.21/clients/mapilib/mapi.c:2134:59: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare] if (snprintf(buf, sizeof(buf), "/tmp/%s", e->d_name) >= sizeof(buf)) Jennie > + continue; /* ignore long name */ > if (stat(buf, &st) != -1 && S_ISSOCK(st.st_mode)) > socks[i++] = atoi(e->d_name + 11); > if (i == sizeof(socks)) > @@ -2320,7 +2324,8 @@ parse_uri_query(Mapi mid, char *uri) > static void > set_uri(Mapi mid) > { > - char uri[1024]; > + size_t urilen = strlen(mid->hostname) + (mid->database ? strlen(mid->database) : 0) + 32; > + char *uri = malloc(urilen); > > /* uri looks as follows: > * mapi:monetdb://host:port/database > @@ -2330,25 +2335,25 @@ set_uri(Mapi mid) > > if (mid->database != NULL) { > if (mid->hostname[0] == '/') { > - snprintf(uri, sizeof(uri), "mapi:monetdb://%s?database=%s", > - mid->hostname, mid->database); > + snprintf(uri, urilen, "mapi:monetdb://%s?database=%s", > + mid->hostname, mid->database); > } else { > - snprintf(uri, sizeof(uri), "mapi:monetdb://%s:%d/%s", > - mid->hostname, mid->port, mid->database); > + snprintf(uri, urilen, "mapi:monetdb://%s:%d/%s", > + mid->hostname, mid->port, mid->database); > } > } else { > if (mid->hostname[0] == '/') { > - snprintf(uri, sizeof(uri), "mapi:monetdb://%s", > - mid->hostname); > + snprintf(uri, urilen, "mapi:monetdb://%s", > + mid->hostname); > } else { > - snprintf(uri, sizeof(uri), "mapi:monetdb://%s:%d", > - mid->hostname, mid->port); > + snprintf(uri, urilen, "mapi:monetdb://%s:%d", > + mid->hostname, mid->port); > } > } > > if (mid->uri != NULL) > free(mid->uri); > - mid->uri = strdup(uri); > + mid->uri = uri; > } > > /* (Re-)establish a connection with the server. */ > @@ -2657,7 +2662,7 @@ mapi_start_talking(Mapi mid) > pwdhash = mcrypt_MD5Sum(mid->password, > strlen(mid->password)); > } else { > - snprintf(buf, BLOCK, "server requires unknown hash '%s'", > + snprintf(buf, BLOCK, "server requires unknown hash '%.100s'", > serverhash); > close_connection(mid); > return mapi_setError(mid, buf, "mapi_start_talking", MERROR); > @@ -2688,7 +2693,7 @@ mapi_start_talking(Mapi mid) > } > if (hash == NULL) { > /* the server doesn't support what we can */ > - snprintf(buf, BLOCK, "unsupported hash algorithms: %s", hashes); > + snprintf(buf, BLOCK, "unsupported hash algorithms: %.100s", hashes); > close_connection(mid); > return mapi_setError(mid, buf, "mapi_start_talking", MERROR); > } > @@ -2697,14 +2702,18 @@ mapi_start_talking(Mapi mid) > > /* note: if we make the database field an empty string, it > * means we want the default. However, it *should* be there. */ > - snprintf(buf, BLOCK, "%s:%s:%s:%s:%s:\n", > + if (snprintf(buf, BLOCK, "%s:%s:%s:%s:%s:\n", > #ifdef WORDS_BIGENDIAN > - "BIG", > + "BIG", > #else > - "LIT", > + "LIT", > #endif > - mid->username, hash, mid->language, > - mid->database == NULL ? "" : mid->database); > + mid->username, hash, mid->language, > + mid->database == NULL ? "" : mid->database) >= BLOCK) {; > + mapi_setError(mid, "combination of database name and user name too long", "mapi_start_talking", MERROR); > + free(hash); > + return mid->error; > + } > > free(hash); > } else { > @@ -2873,7 +2882,7 @@ mapi_start_talking(Mapi mid) > } else { > char re[BUFSIZ]; > snprintf(re, sizeof(re), > - "error while parsing redirect: %s\n", red); > + "error while parsing redirect: %.100s\n", red); > mapi_close_handle(hdl); > mapi_setError(mid, re, "mapi_start_talking", MERROR); > return mid->error; > diff --git a/clients/odbc/driver/SQLExecute.c b/clients/odbc/driver/SQLExecute.c > --- a/clients/odbc/driver/SQLExecute.c > +++ b/clients/odbc/driver/SQLExecute.c > @@ -443,8 +443,7 @@ MNDBExecute(ODBCStmt *stmt) > addStmtError(stmt, "HY001", NULL, 0); > return SQL_ERROR; > } > - snprintf(query, querylen, "execute %d (", stmt->queryid); > - querypos = strlen(query); > + querypos = snprintf(query, querylen, "execute %d (", stmt->queryid); > /* XXX fill in parameter values */ > if (desc->sql_desc_bind_offset_ptr) > offset = *desc->sql_desc_bind_offset_ptr; > diff --git a/common/options/monet_options.c b/common/options/monet_options.c > --- a/common/options/monet_options.c > +++ b/common/options/monet_options.c > @@ -216,7 +216,6 @@ mo_builtin_settings(opt **Set) > { > int i = 0; > opt *set; > - char buf[BUFSIZ]; > > if (Set == NULL) > return 0; > @@ -228,10 +227,8 @@ mo_builtin_settings(opt **Set) > > set[i].kind = opt_builtin; > set[i].name = strdup("gdk_dbpath"); > - snprintf(buf, BUFSIZ, "%s%c%s%c%s%c%s", > - LOCALSTATEDIR, DIR_SEP, "monetdb5", DIR_SEP, "dbfarm", > - DIR_SEP, "demo"); > - set[i].value = strdup(buf); > + set[i].value = strdup(LOCALSTATEDIR DIR_SEP_STR "monetdb5" DIR_SEP_STR > + "dbfarm" DIR_SEP_STR "demo"); > i++; > set[i].kind = opt_builtin; > set[i].name = strdup("gdk_debug"); > diff --git a/common/stream/stream.c b/common/stream/stream.c > --- a/common/stream/stream.c > +++ b/common/stream/stream.c > @@ -510,16 +510,19 @@ error(stream *s) > > switch (s->errnr) { > case MNSTR_OPEN_ERROR: > - snprintf(buf, sizeof(buf), "error could not open file %s\n", s->name); > + snprintf(buf, sizeof(buf), "error could not open file %.100s\n", > + s->name); > return strdup(buf); > case MNSTR_READ_ERROR: > - snprintf(buf, sizeof(buf), "error reading file %s\n", s->name); > + snprintf(buf, sizeof(buf), "error reading file %.100s\n", > + s->name); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

1 0

MonetDB causing disk space to fill up
by Anu Raj Srivastava 19 Jul '15

19 Jul '15

Hi, I have a bunch of test queries which I am testing against MonetDB. Queries all by itself run fine but if you run the same set of queries in a loop disk runs out of space. I am not sure why would memory map files won't clean up automatically and why do we have to start the server again to clean up these files? Is there a workaround to this problem or is this a known issue in MonetDB? Thanks, Anu

3 7

Re: Re: Re: mserver5 stability problems in acting with multithreaded clients
by Sherzod Mutalov 18 Jul '15

18 Jul '15

To: sjoerd(a)monetdb.org I build Jul2015 branch and tested with my test app and mserver5 errors disappeared :) Nice But, still have problem with Mapi library, where first 1-4 connections get lost, because WSAStartup error. Then I added changes to my test program, and when program starts it preconfigures Windows Sockets, then starts client thread. Now it works: https://gist.github.com/shmutalov/3894fdd22ec62334fe17 Regards, Sherzod Mutalov

2 1

Re: Re: mserver5 stability problems in acting with multithreaded clients
by Sherzod Mutalov 17 Jul '15

17 Jul '15

Thank you for answer, <b>Sjoerd Mullender! You said, > </b>In all, I think the situation is better in Jul2015 Does these problems have been fixed in nightly build? Regards, Sherzod Mutalov

2 1

Re: Re: mserver5 stability problems in acting with multithreaded clients
by Sherzod Mutalov 17 Jul '15

17 Jul '15

You can get MapiDemoCpp source and test it:     https://gist.github.com/shmutalov/3894fdd22ec62334fe17 Regards, Sherzod Mutalov

2 1

mserver5 stability problems in acting with multithreaded clients
by Sherzod Mutalov 17 Jul '15

17 Jul '15

Good day all, I am testing mserver5 with multithreaded client and have stability problems. I wrote a simple app in C#, which uses wrapper around Mapi (pure calls to libmapi.dll functions). The app creates 1000 tasks (threads), each thread opens connection to mserver5, then sends a simple query: "SELECT {0}, NOW();", where "{0}" is thread number, and fetchs the result. Problems are: 1. When my application starts to execute threads, always 1st, 2nd or 3rd connection will be lost, and I can't reconnect from that thread. 2. Sometimes there is error on mserver5, which causes connection lost on the client:     "!SABAOTHgetMyStatus: MALException:sabaoth.getmystatus:could not find my own database?!?" mserver5 is running on the local machine, same as client. I tryied to set a gdk_nr_threads option to 1, 8, 16, and get the same problems. I am running on Windows 7 x64, 16GB RAM, Intel i7-5500 CPU Regards, Sherzod Mutalov

1 0

Job openings at MonetDB Solutions
by Ying Zhang 14 Jul '15

14 Jul '15

Dear all, MonetDB Solutions, the supporting company of MonetDB, is looking for core database developers. If you’re interested, please see the job description here: https://www.monetdbsolutions.com/company/careers Regards, Jennie

1 0

Re: Re: mserver5 crash while login with newly created user
by Sherzod Mutalov 14 Jul '15

14 Jul '15

> > > Adding users on the mal level isn't supported on the sql level. > > Ok, how can I create sql user on server side? Is there a way to obtain this?

2 1

mserver5 crash while login with newly created user
by Sherzod Mutalov 14 Jul '15

14 Jul '15

I want to change default monetdb user password to other and create new database user when starting new database instance, before any client connects to newly created db. I starting server with follow command: mserver5.exe --dbinit="oldpwd := clients.backendsum(\"monetdb\"); newpwd := clients.backendsum(\"secret\"); clients.changePassword(oldpwd, newpwd); usrpwd:=clients.backendsum(\"secret2\");clients.addUser(\"newuser\", usrpwd);" --dbpath="D:\Projects\git\monetdb-latest-dev\NT\build\bin\dbfarm\demo" There are no errors. Then I trying to connect to this server: mclient.exe -u monetdb -P secret No errors, connected. Then, I am trying to open new session: mclient.exe -u newuser -P secret2 Ooops! Client exits with error: Connection terminated Server unexpectedly dies. I restarted server, then try connect one more time, same behaviour: "Connection terminated". Ok. Then I connected to server with default "monetdb" user and checked "users" table: mclient.exe -u monetdb -P secret sql>select * from users; +---------+---------------+----------------+ | name | fullname | default_schema | +=========+===============+================+ | monetdb | MonetDB Admin | 2000 | | newuser | null | null | +---------+---------------+----------------+ 2 tuples (5.332ms) Look, fullname and default_schema for "newuser" is null. I think "null pointer dereference" somewhere in code. If I creating user with SQL (CREATE USER ...) everything works fine, I can connect.

2 1

mserver5 crash while login with newly created user
by Sherzod Mutalov 14 Jul '15

14 Jul '15

I tested with monetdb v11.19.13 and v11.19.15 (Windows 7 x64, MSVC2013)

1 0

Hash join implementation
by Xu,Wenjian 11 Jul '15

11 Jul '15

Hi Stefan, Thank you very much for your suggestions. They are very helpful. I am also trying to understand the hashjoin() implementation in gdk/gdk_join.c. I am aware of the classical *multi-pass partitioned hash join* as described in the literature: [1] Main memory hash joins on multi-core CPUs: tuning to the underlying hardware. Cagri Balkesen et al. ICDE13 [2] Optimizing main-memory join on modern hardware. Stefan Manegold et al. TKDE (2002) I was trying to follow the idea in these papers to understand the implementation of hash join in MonetDB but failed. Do you have any suggestions? Are there any other papers/documents I can refer to? Thanks. Best regards, XU Wenjian >Hi Wenjian, >you might want to consider studying the logical (relational) >and physical (MAL) plans, or debug your SQL queries; for >details see >https://www.monetdb.org/Documentation/Manuals/SQLreference/Runtime >You might also want to consider using the MAL debugger >directly; cf., >https://www.monetdb.org/Documentation/Manuals/MonetDB/debugger >Also, a combination of mserver5's --trace and ->gdk_debug=2097152 ("ALGOMASK") >might be useful. >Best, >Stefan ----- On Jul 8, 2015, at 11:42 AM, Xu,Wenjian zeroxwj at gmail.com <https://www.monetdb.org/mailman/listinfo/developers-list> wrote: >>* Hi, *>> >>* My objective is to investigate that, given an SPJ query, how are BATs *>>* *physically* processed (changed) among relational algebra operators (e.g., *>>* select, join). I do not care about how query is translated into execution plan *>>* in MAL, what I care is the interaction of operators in the kernel (aka., gdk) . *>> >>* My current plan is to add some debug information in those operators (e.g., *>>* functions in src/gdk_join.c, src/gdk_select.c) and re-compile the system. Then, *>>* I will pose an SPJ query (through console) and check the system log to see how *>>* those operators are invoked in sequence. *>> >>* But this method seems inefficient and I would like to ask if there are any *>>* alternatives, for example, is it possible to debug the system *step-by-step* *>>* given an input query (or its logical query plan). *>> >>* Thank you very much for any kind help. *>> >>* Best regards, *>>* Wenjian *>> >> >*>*

1 0

Interaction of operators in the kernel.
by Xu,Wenjian 08 Jul '15

08 Jul '15

Hi, My objective is to investigate that, given an SPJ query, how are BATs *physically* processed (changed) among relational algebra operators (e.g., select, join). I do not care about how query is translated into execution plan in MAL, what I care is the interaction of operators in the kernel (aka., gdk) . My current plan is to add some debug information in those operators (e.g., functions in src/gdk_join.c, src/gdk_select.c) and re-compile the system. Then, I will pose an SPJ query (through console) and check the system log to see how those operators are invoked in sequence. But this method seems inefficient and I would like to ask if there are any alternatives, for example, is it possible to debug the system *step-by-step* given an input query (or its logical query plan). Thank you very much for any kind help. Best regards, Wenjian

2 1

Transaction-replication, monetdbd database properties and other changes on default
by Dimitar Nedev 23 Jun '15

23 Jun '15

Hello everyone, We've recently merged the transaction-replicaiton branch into the default. That means that you can try out the new, still experimental, transaction-replication if you build the MonetDB default branch. Since there is already a release branch for the next feature release (July2015), the transaction-replicaiton support will not be part of the next major release. In general, most changes affect a database instance if it runs as a slave, master instances should function just as before. So unless you flip the right switches (and it cannot happen accidentally*) you shouldn't notice any differences. Most code changes are in gdk_logger, where a few large functions (including logger_new) were split into smaller one to accommodate for the need of new read-only logger. That is, a logger that only reads transactions and persists them in the data store. In additional, you can now set any property to a database (mserver5) instance running under the MonetDB-Daemon (monetdbd). Previously only a few preset properties could be set via the monetdbd. The current preset properties will still function as before, both in terms of mapping monetdb-to-mserver5 names and allowed values. * I will provide more information how to turn on and try out the transaction-replication in a blog post on the website very soon. Best regards, Dimitar

1 0

New Jul2015 stable branch created
by Sjoerd Mullender 11 Jun '15

11 Jun '15

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jul2015. So in each of your stable checkout directories you can do hg pull hg update -rJul2015 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Oct2014-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 09 Jun '15

09 Jun '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2014-SP4 bugfix release Bug Fixes * 3705: Assertion failure in rel_bin.c:2274: rel2bin_project: Assertion `0' failed. * 3707: var() possibly not working in debug builds * 3712: Concurrency issue on querying the SQL catalog * 3713: Long startup cost for simple session * 3715: Crash with two ALTER TABLE statements in a transaction * 3718: Adding and dropping a non existing tablename to/from a merge table is accepted without an error * 3719: Assertion failure in /MonetDB-11.19.11/gdk/gdk_bat.c:2841: BATassertHeadProps: Assertion `!b->H->key || cmp != 0' failed. * 3720: Incorrect results on joining with same table * 3723: Assertion failure in rel_bin.c:2548: rel2bin_groupby: Assertion `0' failed. * 3725: LEFT JOIN bug with CONST value * 3731: left shift for IP addresses not available to non-system users

1 0

Hashjoin performance with large vs small tables
by Roberto Cornacchia 12 May '15

12 May '15

Hi there, Triggered by a suspiciously slow join, I had a look at BATsubjoin, in particular at how it is decided which algorithm to use. I looked at a couple of cases with a join on strings. This is one (pseudo syntax): l.count ~16M l.T.sorted = 0 l.T.revsorted = 0 r.count = 1 r.T.sorted = 1 r.T.revsorted = 1 In this case, a hash table on r is created (because it's smaller). This join takes *430ms*. I forced swapping l and r, thus built the hash table on the larger bat, and then it takes *0.8ms*. Notice that though r.count = 1 is a bit of an extreme case, it does happen often in practice. And the case with a few tuples would not be very different. The question I wanted to ask is: is it always right to build the hash table on the small table? Perhaps some more heuristics can help? Like: if the smaller is very small, and the larger is not extremely large, then hash on the larger one? Roberto

2 12

Facing issues in implementing Embedded R in MonetDB-11.19.11
by Vishal Virmani 12 May '15

12 May '15

Hi I am implementing Embedded R in MonetDB version 11.19.11. My environment is VM with Ubuntu 14.04 Server. I have build MonetDB from source and R as well as from source. Running the command sudo mserver5 --version shows me the desired output. When i try to invoke a database using sudo monetdb start testdb1 command, where testdb1 is a database i successfully created, it repeatedly throws up an error...database 'testdb1' started up, but failed to open up a communication channel. Upon checking the associated merovingian log file i see further messages saying the object code to rapi.prelude missing. Any way to resolve this...? Regards Vishal Virmani @poisoncreed

1 0

Suggestion: Windows builds ...
by g4＠novadsp.com 11 May '15

11 May '15

Hello Might it be possible to include copies of the relevant header files for the support libraries (iconv through to zdll) in the GIT repo and/or tarballs? There is a considerable amount of pain involved in acquiring/building/installing matching versions of the lib/dll files on Windows (esp. on Windows!). Also libstream.dll does not advertise any version information which makes matching just that bit harder :) Thanks. Jerry.

1 0

Announcement: New Oct2014-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 07 May '15

07 May '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2014-SP3 bugfix release Build Environment * We now also create debug packages for Debian and Ubuntu. MonetDB Common * Replaced the rangejoin implementation with one that uses imprints if it can. Bug Fixes * 3466: UPDATE statements fails with "GDKerror: MT_mremap() failed" * 3602: Surprising overload resolution of generate_series * 3613: SQL data dictionary contains columns names which are also special keywords. This causes unexpected/unneeded SQL query errors * 3645: Network address operators such as << and <<= do not work * 3647: missing BAT for a column leads to crash in gtr_update_delta * 3648: memory corruption on unclean connection shutdown with local temporary tables * 3650: Naming of persistent BATs is fragile * 3653: PREPARE crashes mserver if unbound variable is function parameter * 3655: SQL WHERE -1 in (-1) issue? * 3656: error message after calling fitsload() * 3660: Incorrect Results for Comparison Operators on inet Datatype * 3661: Ship debug symbols for pre-built binaries * 3662: UPDATE row with row value constructor crashes monetdb server * 3663: Incorrect result ROW_NUMBER in subquery * 3664: SQLstatementIntern missing parameter when using jsonstore * 3665: inter-session starvation issue, particularly affects sys.queue * 3666: casting text column to inet truncating text column and resulting inet for first occurrence only * 3667: insert of negative value for oid column aborts mserver5 process with assertion failure * 3669: ALTER TABLE <tbl_nm> ADD CONSTRAINT <tbl_uc1> UNIQUE (col1, col2, col3) causes Assertion failure and abort * 3671: ODBC-Access on Windows 2012 does not work - E_FAIL * 3672: libbat_la-gdk_utils.o: relocation R_X86_64_PC32 against `MT_global_exit' can not be used when making a shared object * 3676: merovingian hangs trying to exit * 3677: Crash in BATgroup_internal (caused by 87379087770d?) * 3678: Ruby driver installation ignores prefix * 3680: Prepared statements fail on execution with message 'Symbol type not found' * 3684: Wrong query result set WHERE "IS NULL" or "NOT IN" clauses uses in combination with ORDER, LIMIT and OFFSET * 3687: 'bat.insert' undefined * 3688: Crash at exit (overrun THRerrorcount?) * 3689: No more connections accepted if a single client misbehaves * 3690: find_fk: Assertion `t && i' failed. * 3691: conversion of whitespaces string to double or float is accepted without an error during insert * 3693: algebra.join undefined (caused by non-existing variables in the plan) * 3696: Inconsistent behavior between dbl (SQL double) and flt (SQL real) data types and across platforms * 3697: mserver5[26946]: segfault at 0 ip 00007f3d0e1ab808 sp 00007f3cefbfcad0 error 4 in lib_sql.so[7f3d0e180000+16c000] * 3699: segfault again! (during last week I found 3 segfault bugs already) * 3703: INSERT INTO a MERGE TABLE crashes mserver5 * 3704: Unknown identifier from subquery * 3705: Assertion failure in rel_bin.c:2274: rel2bin_project: Assertion `0' failed. * 3706: Assertion failure in gdk_bat.c: BATassertHeadProps: Assertion `!b->H->sorted || cmp <= 0' failed. * 3709: "BATproject: does not match always" on abusive use of ALTER TABLE SET READ ONLY

1 0

C++ compiling Fixes
by Bastian Köcher 27 Apr '15

27 Apr '15

Hi, I want to write an optimizer for MonetDB in C++. So I included the necessary headers into my project, but I ran into some compile errors and warnings. I created a patch which solves some of these problems. Regards, Bastian

1 0

Re: MonetDB: default - remove jsonstore code we do not need for the ...
by Stefan Manegold 21 Apr '15

21 Apr '15

Arjen, are these left-overs occurrences of "jsonstore" in the default branch still used / required, or can they also be removed? ======== .../default/source/MonetDB $ hg locate | grep --color -i jsonstore sql/scripts/41_jsonstore.sql .../default/source/MonetDB $ hg locate -X \**/Tests/*.stable.* | xargs grep -l -i jsonstore MonetDB.spec NT/installer32/MonetDB5-SQL-Installer.vdproj NT/installer64/MonetDB5-SQL-Installer.vdproj NT/monetdb_config.h.in NT/rules.msc clients/Tests/All monetdb5/mal/mal.c sql/backends/monet5/sql_upgrades.c sql/scripts/41_jsonstore.sql sql/scripts/Makefile.ag testing/Mtest.py.in ======== Thanks! Stefan ----- On Apr 21, 2015, at 12:37 PM, Sjoerd Mullender commits(a)monetdb.org wrote: > Changeset: e6f13746b75b for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e6f13746b75b > Removed Files: > sql/backends/monet5/rest/73_jsonstore.mal > sql/backends/monet5/rest/Makefile.ag > sql/backends/monet5/rest/Tests/All > sql/backends/monet5/rest/Tests/jsonstore00.sql > sql/backends/monet5/rest/Tests/jsonstore00.stable.err > sql/backends/monet5/rest/Tests/jsonstore00.stable.out > sql/backends/monet5/rest/Tests/jsonstore01.sql > sql/backends/monet5/rest/Tests/jsonstore01.stable.err > sql/backends/monet5/rest/Tests/jsonstore01.stable.out > sql/backends/monet5/rest/jsonstore.mal > sql/backends/monet5/rest/rest_jsonstore.c > sql/backends/monet5/rest/rest_jsonstore.h > sql/backends/monet5/rest/rest_jsonstore_handle_get.c > sql/backends/monet5/rest/rest_jsonstore_handle_get.h > Modified Files: > configure.ag > monetdb5/mal/Makefile.ag > monetdb5/mal/mal_http_daemon.c > sql/backends/monet5/Makefile.ag > Branch: default > Log Message: > > remove jsonstore code we do not need for the next feature release > > > diffs (truncated from 2238 to 300 lines): > > diff --git a/configure.ag b/configure.ag > --- a/configure.ag > +++ b/configure.ag > @@ -188,30 +188,13 @@ no-auto) > ;; > esac > > -dft_jsonstore=no > -AC_ARG_ENABLE(jsonstore, > - AS_HELP_STRING([--enable-jsonstore], > - [enable support for MonetDB/jsonstore (default=no)]), > - enable_jsonstore=$enableval, > - enable_jsonstore=$dft_jsonstore) > - > -dft_microhttpd=$enable_jsonstore > +dft_microhttpd=no > AC_ARG_ENABLE(microhttpd, > AS_HELP_STRING([--enable-microhttpd], > - [enable support for the builtin MircoHTTPD daemon (default: implied by > --enable-jsonstore)]), > + [enable support for the builtin MircoHTTPD daemon]), > enable_microhttpd=$enableval, > enable_microhttpd=$dft_microhttpd) > > -case "$enable_jsonstore-$enable_microhttpd" in > -yes-no) > - AC_MSG_ERROR([jsonstore require microhttpd]) > - ;; > -auto-no) > - enable_jsonstore=no > - disable_jsonstore="(jsonstore requires microhttpd)" > - ;; > -esac > - > dft_gsl=auto > AC_ARG_ENABLE(gsl, > AS_HELP_STRING([--enable-gsl], > @@ -2508,7 +2491,7 @@ if test x"$have_atomic_ops" = x"yes" ; t > fi > > dnl MicroHTTPD library > -dnl required for MonetDB5 SQL jsonstore, optional otherwise > +dnl optional > org_have_libmicrohttpd=auto > have_libmicrohttpd=$org_have_libmicrohttpd > if test "x$enable_microhttpd" != xno; then > @@ -2526,24 +2509,6 @@ if test x$enable_microhttpd != xno; then > AC_DEFINE(HAVE_MICROHTTPD, 1, [Define if the microhttpd server is to be > enabled]) > fi > > -org_have_liburiparser=auto > -have_liburiparser=$org_have_liburiparser > -if test "x$enable_jsonstore" != xno; then > - dnl Uriparser library > - dnl required for MonetDB5 SQL jsonstore, optional otherwise > - PKG_CHECK_MODULES([liburiparser], [liburiparser], > - [have_liburiparser=yes], > - [have_liburiparser=no; disable_jsonstore="(liburiparser not found)"; if test > "x$enable_jsonstore" = xyes; then AC_MSG_ERROR([LibUriParser library not found > but required for jsonstore]); fi]) > - > - if test "$have_libmicrohttpd-$have_liburiparser" = yes-yes; then > - AC_DEFINE(HAVE_JSONSTORE, 1, [Define that you want to build the jsonstore > library]) > - elif test "x$enable_jsonstore" = xyes; then > - AC_MSG_ERROR([cannot enable jsonstore > $why_have_libmicrohttpd$why_have_liburiparser]) > - else > - enable_jsonstore=no > - fi > -fi > - > dnl Check for gsl library for statistical functions > org_have_gsl=no > have_gsl=$org_have_gsl > @@ -2920,7 +2885,6 @@ AM_CONDITIONAL(HAVE_GDK, test "x$enable_ > AM_CONDITIONAL(HAVE_MONETDB5, test "x$enable_monetdb5" = xyes -o > "x$enable_monetdb5" = xauto) > AM_CONDITIONAL(HAVE_SQL, test "x$enable_sql" = xyes -o "x$enable_sql" = xauto) > AM_CONDITIONAL(HAVE_GEOM, test "x$enable_geom" = xyes -o "x$enable_geom" = > xauto) > -AM_CONDITIONAL(HAVE_JSONSTORE, test "x$enable_jsonstore" = xyes -o > "x$enable_jsonstore" = xauto) > AM_CONDITIONAL(HAVE_TESTING, test "x$enable_testing" = xyes -o > "x$enable_testing" = xauto) > AM_CONDITIONAL(HAVE_DEVELOPER, test "x$enable_developer" = xyes) > > @@ -3277,7 +3241,6 @@ for comp in \ > 'console ' \ > 'gsl ' \ > 'fits ' \ > - 'jsonstore ' \ > 'microhttpd ' \ > 'rintegration' \ > 'odbc ' \ > diff --git a/monetdb5/mal/Makefile.ag b/monetdb5/mal/Makefile.ag > --- a/monetdb5/mal/Makefile.ag > +++ b/monetdb5/mal/Makefile.ag > @@ -9,7 +9,6 @@ INCLUDES = ../../common/options \ > ../../common/utils \ > ../../clients/mapilib \ > ../../gdk \ > - #../../sql/backends/monet5/rest \ > $(READLINE_INCS) \ > $(libmicrohttpd_CFLAGS) > MTSAFE > diff --git a/monetdb5/mal/mal_http_daemon.c b/monetdb5/mal/mal_http_daemon.c > --- a/monetdb5/mal/mal_http_daemon.c > +++ b/monetdb5/mal/mal_http_daemon.c > @@ -12,262 +12,6 @@ > > #include "monetdb_config.h" > > -#ifdef HAVE_MICROHTTPD > -#include <sys/types.h> > -#include <sys/select.h> > -#include <sys/socket.h> > -#include <stdint.h> > -#include <stdarg.h> > -#include <stdio.h> > - > -#include "mal_client.h" > -#include "mal_http_daemon.h" > -#include "mal_private.h" > -#include <microhttpd.h> > - > -static MT_Id hdthread; > -static int volatile hdrunning; > -struct MHD_Daemon *http_daemon; > -#define PORT 8998 > -#define POSTBUFFERSIZE 512 > -#define MAXNAMESIZE 512 > -#define MAXANSWERSIZE 512 > - > -#define GET 0 > -#define POST 1 > - > -static > -http_request_handler http_handler; > - > -void register_http_handler(http_request_handler handler) { > - http_handler = handler; > -} > - > -struct connection_info_struct > -{ > - int connectiontype; > - char *answerstring; > - struct MHD_PostProcessor *postprocessor; > -}; > - > -static int > -send_page (struct MHD_Connection *connection, const char * url, > - const char * method, char *page, char * postdata) > -{ > - int ret; > - int rest; > - struct MHD_Response *response; > - > - rest = (*http_handler)(url, method, &page, postdata); > - (void)rest; > - response = > - MHD_create_response_from_buffer (strlen (page), > - (void *) page, > - MHD_RESPMEM_MUST_COPY); > - if (!response) > - return MHD_NO; > - > - ret = MHD_queue_response (connection, MHD_HTTP_OK, response); > - MHD_destroy_response (response); > - > - return ret; > -} > - > -static int > -iterate_post (void *coninfo_cls, enum MHD_ValueKind kind, const char *key, > - const char *filename, const char *content_type, > - const char *transfer_encoding, const char *data, uint64_t off, > - size_t size) > -{ > - struct connection_info_struct *con_info = coninfo_cls; > - char *answerstring; > - > - (void)key; > - (void)kind; > - (void)filename; > - (void)content_type; > - (void)transfer_encoding; > - (void)off; > - > - if (strcmp (key, "json") == 0) > - { > - if ((size > 0) && (size <= MAXNAMESIZE)) > - { > - answerstring = malloc (MAXANSWERSIZE); > - if (!answerstring) > - return MHD_NO; > - > - snprintf (answerstring, MAXANSWERSIZE, "%s", data); > - con_info->answerstring = answerstring; > - } else > - con_info->answerstring = NULL; > - > - return MHD_NO; > - } > - > - if (strcmp (key, "file") == 0) > - { > - if ((size > 0) && (size <= MAXNAMESIZE)) > - { > - answerstring = malloc (MAXANSWERSIZE); > - if (!answerstring) > - return MHD_NO; > - > - snprintf (answerstring, MAXANSWERSIZE, "%s", data); > - con_info->answerstring = answerstring; > - } else > - con_info->answerstring = NULL; > - > - return MHD_NO; > - } > - > - return MHD_YES; > -} > - > -static void > -request_completed (void *cls, struct MHD_Connection *connection, > - void **con_cls, enum MHD_RequestTerminationCode toe) > -{ > - struct connection_info_struct *con_info = *con_cls; > - > - (void)cls; > - (void)connection; > - (void)toe; > - > - if (con_info == NULL) > - return; > - > - if (con_info->connectiontype == POST) > - { > - MHD_destroy_post_processor (con_info->postprocessor); > - if (con_info->answerstring) > - free (con_info->answerstring); > - } > - > - free (con_info); > - *con_cls = NULL; > -} > - > -static int > -answer_to_connection (void *cls, struct MHD_Connection *connection, > - const char *url, const char *method, > - const char *version, const char *upload_data, > - size_t *upload_data_size, void **con_cls) > -{ > - char * page = NULL; > - struct connection_info_struct *con_info = NULL; > - int *done = cls; > - char *answerstring = NULL; > - char *errorpage = > - "<html><body>" > - "Failed to handle error in http request." > - "</body></html>"; > - > - (void)version; > - > - if (*con_cls == NULL) { > - con_info = malloc (sizeof (struct connection_info_struct)); > - if (con_info == NULL) > - return MHD_NO; > - con_info->answerstring = NULL; > - > - if (strcmp (method, "POST") == 0) { > - con_info->postprocessor = > - MHD_create_post_processor (connection, > - POSTBUFFERSIZE, > - iterate_post, > - (void *) con_info); > - > - if (con_info->postprocessor == NULL) { > - free (con_info); > - return MHD_NO; > - } > - con_info->connectiontype = POST; > - } else { > - con_info->connectiontype = GET; > - } > - *con_cls = (void *) con_info; > - return MHD_YES; > - } > - > - if (strcmp (method, "POST") == 0) { > - con_info = *con_cls; > - if (*upload_data_size != 0) { > - MHD_post_process (con_info->postprocessor, upload_data, > - *upload_data_size); > - *upload_data_size = 0; > - return MHD_YES; > - } else { > - if (con_info->answerstring != NULL) { > - return send_page(connection, url, method, page, > - con_info->answerstring); > - } > - } > - } > - > - if (strcmp (method, "PUT") == 0) { > - if (*upload_data_size != 0) { > - // TODO: check free answerstring > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

Re: developers-list Digest, Raúl
by José Raúl Pérez Rodríguez 13 Apr '15

13 Apr '15

Thank you very much Hannes, Then I can assume that queries to a distributed table can' t be sorted. It is a serious limitation in MonetDb scaling, I would like to work to help in adding this capability. Raúl > Today's Topics: > > 1. Re: Querying a monetdb cluster (Hannes M?hleisen) > > > Hello Jos? Ra?l, > >> On 09.04.2015, at 12:45, Jos? Ra?l P?rez Rodr?guez >> <joseraul.perezrodriguez(a)gmail.com> wrote: >> >> I maked a test inserting data into each particular node of a two node >> cluster, but I don't see the way of making queries to the cluster, I >> known that multiplex funnel can return data from several nodes, >> joining data obtained from each node, but with multiplex funnel, I >> can't sort the data. > This is not supported yet. > > Best, > > Hannes

2 1

Querying a monetdb cluster
by José Raúl Pérez Rodríguez 10 Apr '15

10 Apr '15

Hi, I am new in monetdb. How do I make a query to a cluster? I maked a test inserting data into each particular node of a two node cluster, but I don't see the way of making queries to the cluster, I known that multiplex funnel can return data from several nodes, joining data obtained from each node, but with multiplex funnel, I can't sort the data. There is not doc that explains this problem?

2 1

MonetDB, data loss in restart server
by Luciano Sasso 02 Apr '15

02 Apr '15

Hello, I have used the MonetDB in production for some time, and I have nothing to complain about, are tables with more than 30 columns and 30 million records, do analysis through cubes in Pentaho / Saiku. Have restarted the server several times, and never had problems with loss of data.But today, after performing a query that was consuming a lot of machine, stopped the database, and when I started, data from some columns were blank. How can it, will the data were still uncommitted? when performing a query the MonetDB changes the data or reorganized? which may have occurred, I am concerned. -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

2 1

Re: MonetDB: default - Defense against non-existing MAL primitives
by Stefan Manegold 20 Mar '15

20 Mar '15

FYI, test clients/R/Tests/survey.R still fails with "Server says '!TypeException:user.s94_2[40]:'batcalc.*' undefined in: X_168:bat[:oid,:hge] := batcalc.*(X_146,X_166);'." i.e., the check whether the requested new signature exists does not seem to work, yet. cf., http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=55075:ca15cab457b8&or… Stefan ----- Original Message ----- > Changeset: 4165dc34b2bb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4165dc34b2bb > Modified Files: > monetdb5/mal/mal_resolve.c > monetdb5/mal/mal_resolve.h > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > Defense against non-existing MAL primitives > and further limit the rewrite to the simpliest > and most expensive case encountered. > > > diffs (244 lines): > > diff --git a/monetdb5/mal/mal_resolve.c b/monetdb5/mal/mal_resolve.c > --- a/monetdb5/mal/mal_resolve.c > +++ b/monetdb5/mal/mal_resolve.c > @@ -115,7 +115,6 @@ findFunctionType(stream *out, Module sco > Symbol s; > InstrPtr sig; > int i, k, unmatched = 0, s1; > - /* int foundbutwrong=0; */ > int polytype[MAXTYPEVAR]; > int returns[256]; > int *returntype = NULL; > @@ -396,7 +395,6 @@ findFunctionType(stream *out, Module sco > } > } > if (s1 < 0) { > - /* if(getSignature(s)->token !=PATTERNsymbol) foundbutwrong++; */ > s = s->peer; > continue; > } > @@ -445,7 +443,6 @@ findFunctionType(stream *out, Module sco > * typed. This should be reflected in the symbol table. > */ > s1 = returntype[0]; /* for those interested */ > - /* foundbutwrong = 0; */ > /* > * If the call refers to a polymorphic function, we clone it > * to arrive at a bounded instance. Polymorphic patterns and > @@ -485,12 +482,6 @@ findFunctionType(stream *out, Module sco > * arguments, but that clashes with one of the target variables. > */ > wrapup: > - /* foundbutwrong has not been changed, commented out code above > - if (foundbutwrong && !silent) { > - showScriptException(out, mb, getPC(mb, p), TYPE, > - "type conflict in assignment"); > - } > - */ > if (returntype && returntype != returns) > GDKfree(returntype); > return -3; > @@ -819,13 +810,17 @@ chkTypes(stream *out, Module s, MalBlkPt > > /* > * Type checking an individual instruction is dangerous, > - * because it ignores data flow and declarations issues. > - * It is only to be used in isolated cases. > + * because it ignores data flow and variable declarations. > */ > -void > +int > chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr p) > { > - typeChecker(out, s, mb, p, FALSE); > + int olderrors= mb->errors; > + int error; > + typeChecker(out, s, mb, p, TRUE); > + error = mb->errors; > + mb->errors = olderrors; > + return error; > } > > void > diff --git a/monetdb5/mal/mal_resolve.h b/monetdb5/mal/mal_resolve.h > --- a/monetdb5/mal/mal_resolve.h > +++ b/monetdb5/mal/mal_resolve.h > @@ -20,7 +20,7 @@ > #define MAXTYPEVAR 10 > > mal_export void chkProgram(stream *out, Module s, MalBlkPtr mb); > -mal_export void chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr > p); > +mal_export int chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr > p); > mal_export void chkTypes(stream *out, Module s, MalBlkPtr mb, int silent); > mal_export void typeChecker(stream *out, Module scope, MalBlkPtr mb, > InstrPtr p, int silent); > mal_export int fcnBinder(stream *out, Module scope, MalBlkPtr mb, InstrPtr > p); > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -18,8 +18,9 @@ typedef struct{ > int fromtype; > int totype; > int src; > -/* not used, yet !?? > +/* not used, yet !?? Indeed > int digits; > + int fromscale; > int scale; > */ > } Coercion; > @@ -47,58 +48,60 @@ coercionOptimizerStep(MalBlkPtr mb, int > return 0; > } > > -/* Check coercions for numeric types that can be handled with smaller ones. > +/* Check coercions for numeric types towards :hge that can be handled with > smaller ones. > * For now, limit to +,-,/,*,% hge expressions > - * To be extended to deal with math calls as well. > + * Not every combination may be available in the MAL layer, which calls > + * for a separate type check before fixing it. > + * Superflous coercion statements will be garbagecollected later on in the > pipeline > */ > static void > -coercionOptimizerCalcStep(MalBlkPtr mb, int i, Coercion *coerce) > +coercionOptimizerCalcStep(Client cntxt, MalBlkPtr mb, int i, Coercion > *coerce) > { > InstrPtr p = getInstrPtr(mb,i); > - int r, a, b; > + int r, a, b, varid; > > + r = getColumnType(getVarType(mb, getArg(p,0))); > +#ifdef HAVE_HGE > + if ( r != TYPE_hge) > + return; > +#endif > if( getModuleId(p) != batcalcRef || getFunctionId(p) == 0) return; > if( ! (getFunctionId(p) == plusRef || getFunctionId(p) == minusRef || > getFunctionId(p) == mulRef || getFunctionId(p) == divRef || > *getFunctionId(p) =='%') || p->argc !=3) > return; > > - r = getColumnType(getVarType(mb, getArg(p,0))); > - switch(r){ > - case TYPE_bte: > - case TYPE_sht: > - case TYPE_int: > - case TYPE_lng: > -#ifdef HAVE_HGE > - case TYPE_hge: > -#endif > - break; > - case TYPE_dbl: > - case TYPE_flt: > - /* to be determined */ > - default: > - return; > - } > a = getColumnType(getVarType(mb, getArg(p,1))); > b = getColumnType(getVarType(mb, getArg(p,2))); > - if ( a == r && coerce[getArg(p,1)].src && coerce[getArg(p,1)].fromtype < r > ) /*digit/scale test as well*/ > + if ( a == r && coerce[getArg(p,1)].src && coerce[getArg(p,1)].fromtype < r > ) > + { > + varid = getArg(p,1); > getArg(p,1) = coerce[getArg(p,1)].src; > - if ( b == r && coerce[getArg(p,2)].src && coerce[getArg(p,2)].fromtype < r > ) /*digit/scale test as well*/ > + if ( chkInstruction(NULL, cntxt->nspace, mb, p)) > + p->argv[1] = varid; > + } > + if ( b == r && coerce[getArg(p,2)].src && coerce[getArg(p,2)].fromtype < r > ) > + { > + varid = getArg(p,2); > getArg(p,2) = coerce[getArg(p,2)].src; > + if ( chkInstruction(NULL, cntxt->nspace, mb, p)) > + p->argv[2] = varid; > + } > return; > } > > static void > -coercionOptimizerAggrStep(MalBlkPtr mb, int i, Coercion *coerce) > +coercionOptimizerAggrStep(Client cntxt, MalBlkPtr mb, int i, Coercion > *coerce) > { > InstrPtr p = getInstrPtr(mb,i); > int r, k; > > + (void) cntxt; > + > if( getModuleId(p) != aggrRef || getFunctionId(p) == 0) return; > if( ! (getFunctionId(p) == subavgRef ) || p->argc !=6) > return; > > r = getColumnType(getVarType(mb, getArg(p,0))); > k = getArg(p,1); > - // check the digits/scale > if( r == TYPE_dbl && coerce[k].src ) > getArg(p,1) = coerce[getArg(p,1)].src; > return; > @@ -124,35 +127,33 @@ OPTcoercionImplementation(Client cntxt,M > if (getModuleId(p) == NULL) > continue; > /* Downscale the type, avoiding hge storage when lng would be sufficient. > - * The code template can be extended to handle other downscale options as > well > */ > #ifdef HAVE_HGE > if ( getModuleId(p) == batcalcRef > && getFunctionId(p) == hgeRef > && p->retc == 1 > - && ( p->argc == 2 > - || ( p->argc == 5 > - && isVarConstant(mb,getArg(p,1)) > - && getArgType(mb,p,1) == TYPE_int > - && isVarConstant(mb,getArg(p,3)) > - && getArgType(mb,p,3) == TYPE_int > - && isVarConstant(mb,getArg(p,4)) > - && getArgType(mb,p,4) == TYPE_int > - /* from-scale == to-scale, i.e., no scale change */ > - && *(int*) getVarValue(mb, getArg(p,1)) == *(int*) > getVarValue(mb, getArg(p,4)) ) ) ){ > + && ( p->argc == 5 > + && isVarConstant(mb,getArg(p,1)) > + && getArgType(mb,p,1) == TYPE_int > + && isVarConstant(mb,getArg(p,3)) > + && getArgType(mb,p,3) == TYPE_int > + && isVarConstant(mb,getArg(p,4)) > + && getArgType(mb,p,4) == TYPE_int > + /* from-scale == to-scale, i.e., no scale change */ > + && *(int*) getVarValue(mb, getArg(p,1)) == *(int*) getVarValue(mb, > getArg(p,4)) ) ){ > k = getArg(p,0); > coerce[k].pc= i; > coerce[k].totype= TYPE_hge; > coerce[k].src= getArg(p,2); > coerce[k].fromtype= getColumnType(getArgType(mb,p,2)); > -/* not used, yet !?? > - if (p->argc == 5) { > - coerce[k].digits= getVarConstant(mb,getArg(p,3)).val.ival; > - coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > - } > +/* not used, yet !?? indeed > + coerce[k].fromscale= getVarConstant(mb,getArg(p,1)).val.ival; > + coerce[k].digits= getVarConstant(mb,getArg(p,3)).val.ival; > + coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > */ > } > #endif > +/* > if ( getModuleId(p) == batcalcRef > && getFunctionId(p) == dblRef > && p->retc == 1 > @@ -160,7 +161,7 @@ OPTcoercionImplementation(Client cntxt,M > || ( p->argc == 3 > && isVarConstant(mb,getArg(p,1)) > && getArgType(mb,p,1) == TYPE_int > - /* to-scale == 0, i.e., no scale change */ > + //to-scale == 0, i.e., no scale change > && *(int*) getVarValue(mb, getArg(p,1)) == 0 ) ) ) { > k = getArg(p,0); > coerce[k].pc= i; > @@ -168,8 +169,9 @@ OPTcoercionImplementation(Client cntxt,M > coerce[k].src= getArg(p,1 + (p->argc ==3)); > coerce[k].fromtype= getColumnType(getArgType(mb,p,1 + (p->argc ==3))); > } > - coercionOptimizerAggrStep(mb, i, coerce); > - coercionOptimizerCalcStep(mb, i, coerce); > +*/ > + coercionOptimizerAggrStep(cntxt,mb, i, coerce); > + coercionOptimizerCalcStep(cntxt,mb, i, coerce); > if (getModuleId(p)==calcRef && p->argc == 2) { > k= coercionOptimizerStep(mb, i, p); > actions += k; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 6

Re: MonetDB: default - Minor textual changes to remote tests
by Stefan Manegold 20 Mar '15

20 Mar '15

> Changeset: ca15cab457b8 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ca15cab457b8 > Modified Files: > monetdb5/modules/mal/Tests/All > monetdb5/modules/mal/Tests/remote07.mal > monetdb5/modules/mal/Tests/remote88.mal > Branch: default > Log Message: > > Minor textual changes to remote tests > > > diffs (104 lines): > > diff --git a/monetdb5/modules/mal/Tests/All b/monetdb5/modules/mal/Tests/All > --- a/monetdb5/modules/mal/Tests/All > +++ b/monetdb5/modules/mal/Tests/All > @@ -36,6 +36,7 @@ remote02 > HAVE_GEOM?remote03 > remote04 > remote06 > +# should be run as 'mclient -d demo -l mal remote07.mal For what it's worth: MAL tests with extension ".malC" (instead of ".mal") are automatically run trough "mclient -lmal ..." (instead of "mserver5 ...") by Mtest.py; cf., dev.monetdb.org/hg/MonetDB/file/tip/testing/README i.e., `hg rename monetdb5/modules/mal/Tests/remote07.mal monetdb5/modules/mal/Tests/remote07.malC` might be considered ... Stefan > remote08 > remote09 > remote11 [...] -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - Avoid coercion in aggr.subavg
by Stefan Manegold 16 Mar '15

16 Mar '15

According to Mtest, this makes TPCh Q1 return a wrong result. I suspect because this now indeed removes a scale-changing cast. Stefan ----- Original Message ----- > Changeset: 9779801a1482 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9779801a1482 > Modified Files: > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > Avoid coercion in aggr.subavg > Another example where we can avoid coercions in most cases > > > diffs (43 lines): > > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -84,6 +84,24 @@ coercionOptimizerCalcStep(MalBlkPtr mb, > return; > } > > +static void > +coercionOptimizerAggrStep(MalBlkPtr mb, int i, Coercion *coerce) > +{ > + InstrPtr p = getInstrPtr(mb,i); > + int r, k; > + > + if( getModuleId(p) != aggrRef || getFunctionId(p) == 0) return; > + if( ! (getFunctionId(p) == subavgRef ) || p->argc !=6) > + return; > + > + r = getColumnType(getVarType(mb, getArg(p,0))); > + k = getArg(p,1); > + // check the digits/scale > + if( r == TYPE_dbl && coerce[k].src ) > + getArg(p,1) = coerce[getArg(p,1)].src; > + return; > +} > + > int > OPTcoercionImplementation(Client cntxt,MalBlkPtr mb, MalStkPtr stk, InstrPtr > pci) > { > @@ -117,6 +135,14 @@ OPTcoercionImplementation(Client cntxt,M > coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > } > #endif > + if ( getModuleId(p) == batcalcRef && getFunctionId(p) == dblRef && p->retc > == 1 && ((p->argc == 3 && isVarConstant(mb,getArg(p,1))) || p->argc ==2) ){ > + k = getArg(p,0); > + coerce[k].pc= i; > + coerce[k].totype= TYPE_dbl; > + coerce[k].src= getArg(p,1 + (p->argc ==3)); > + coerce[k].fromtype= getColumnType(getArgType(mb,p,1 + (p->argc ==3))); > + } > + coercionOptimizerAggrStep(mb, i, coerce); > coercionOptimizerCalcStep(mb, i, coerce); > if (getModuleId(p)==calcRef && p->argc == 2) { > k= coercionOptimizerStep(mb, i, p); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - coercionOptimizerCalcStep(): fixing changeset...
by Stefan Manegold 16 Mar '15

16 Mar '15

IMHO this new coersion optimization as introduced in changeset 5786f1be12bb only works correctly for decimal casts (coersions) that do not change the scale (number of decimals), i.e., when the first and last argument (i.e., input and output number of decimals, respectively) are identical. This sanity check is currently missing in the code. Stefan ----- Original Message ----- > Changeset: 289a5a038774 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=289a5a038774 > Modified Files: > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > coercionOptimizerCalcStep(): fixing changeset 71bfaf7e9841: also handle > TYPE_hge > to re-activate the intended behavior of changeset 5786f1be12bb > > OPEN ISSUE / QUESTION: > > This optimizer now (only) handles (i.e., removes) decimal (up-)casts, > but no plain/pure integer casts at all. > > (a) Shouldn't we also handle (remove) plain/pure integer casts? > > (b) Can we always safely remove decimal (up-)casts without altering > semantics? > > > diffs (13 lines): > > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -65,6 +65,9 @@ coercionOptimizerCalcStep(MalBlkPtr mb, > case TYPE_sht: > case TYPE_int: > case TYPE_lng: > +#ifdef HAVE_HGE > + case TYPE_hge: > +#endif > break; > case TYPE_dbl: > case TYPE_flt: > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

About the Join Algorithm in MonetDB
by Zhiwei Fan 13 Mar '15

13 Mar '15

Dear Developers of MonetDB, ?I really appreciate your excellent work on MonetDB. I am a student in University of Wisconsin Madison and now I am doing research on processing machine learning in the column-oriented Database since we have found a way that could possibly improve the performance of learning process if we use data stored in column rather than data stored in row. For the research purpose, we need to know the detail of the join algorithm in column-store database in order to analyze the cost of the process. And monetDB, as we know, has done a very good job in this part, so I hope you could tell us about the details of the join algorithms implemented in the monetDB, that would be really helpful and we would really appreciate it! Thanks. Best Regards Zhiwei Fan ?

1 0

MAL property for thread-unsafe function
by Roberto Cornacchia 05 Mar '15

05 Mar '15

Hello, I have a function definition command batstemmer.stem(terms:bat[:oid,:str], stemmer_name:str):bat[:oid,:str] address CMDbatstem comment "Wrapper for snowball stemmer"; which internally uses the snowball stemmer (http://snowball.tartarus.org/) When the bat to be stemmed is large enough, mitosis will split it into chunks and call the function "stem" on each chunk, possibly in parallel. Problem is, the snowball stemmer implementation appears to be thread-unsafe, which causes a SIGSEGV. Indeed, using the no_mitosis_pipe solves the issue. However, this solution is suboptimal. Another solution I found is to mark the mal signature as {unsafe}. This works, although it does something a bit silly: it splits the table into chunks, then repacks everything, and finally runs my function on the re-packed bat (basically wasting effort on a useless split + repack). Now, my question is: is there a more focussed property to use? {unsafe} implies thread-unsafe, but it is actually stronger than that. For example, it also implies that there might be side-effects. Therefore, the result cannot be recycled. In my case, instead, the result is perfectly safe to be reused. Thanks for any tip. Roberto

2 2

Re: MonetDB: default - Progress bar: Ignore comments in trace output
by Ying Zhang 24 Feb '15

24 Feb '15

Hai Hannes, Compiling default branch complains about "‘profiler_querystart’ may be used uninitialized": libtool: compile: gcc -DHAVE_CONFIG_H -I. -I/ufs/zhang/monet/default/clients/mapiclient -I../.. -I/ufs/zhang/monet/default/clients/mapiclient -I../mapilib -I/ufs/zhang/monet/default/clients/mapiclient/../mapilib -I../../common/options -I/ufs/zhang/monet/default/clients/mapiclient/../../common/options -I../../common/stream -I/ufs/zhang/monet/default/clients/mapiclient/../../common/stream -I../R/MonetDB.R/src -I/ufs/zhang/monet/default/clients/mapiclient/../R/MonetDB.R/src -DLIBMCUTIL -O3 -pipe -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code -D_FORTIFY_SOURCE=2 -D_REENTRANT -c /ufs/zhang/monet/default/clients/mapiclient/prompt.c -fPIC -DPIC -o .libs/libmcutil_la-prompt.o In file included from /ufs/zhang/monet/default/clients/mapiclient/profiler-dummy.c:5:0: /ufs/zhang/monet/default/clients/mapiclient/../R/MonetDB.R/src/profiler.c: In function ‘profiler_thread’: /ufs/zhang/monet/default/clients/mapiclient/../R/MonetDB.R/src/profiler.c:219:59 error: ‘profiler_querystart’ may be used uninitialized in this function [-Werror=maybe-uninitialized] if (profiler_msgs_expect > 0 && (profiler_tsms() - profiler_querystart) > 500) { ^ This seems to only happen if I use the --enable-optimize --disable-debug configure options. Can I just initialise profiler_querystart with 0? Thanks! Jennie > On Feb 20, 2015, at 18:39, Hannes Muehleisen <commits(a)monetdb.org> wrote: > > Changeset: cde6ad120a72 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=cde6ad120a72 > Modified Files: > clients/R/MonetDB.R/src/profiler.c > Branch: default > Log Message: > > Progress bar: Ignore comments in trace output > > > diffs (13 lines): > > diff --git a/clients/R/MonetDB.R/src/profiler.c b/clients/R/MonetDB.R/src/profiler.c > --- a/clients/R/MonetDB.R/src/profiler.c > +++ b/clients/R/MonetDB.R/src/profiler.c > @@ -177,6 +177,9 @@ static void* profiler_thread(void* param > recvd = read(profiler_socket, buf, sizeof(buf)); > if (recvd > 0) { > buf[recvd] = 0; > + if (buf[0]== '#') { > + continue; > + } > mapi_line_split(buf, elems, TRACE_NCOLS); > if (strncmp(elems[TRACE_COL_STATEFL], "done", 4) != 0) { > continue; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 2

Re: MonetDB: default - mclient progress meter part one, mal statemen...
by Stefan Manegold 24 Feb '15

24 Feb '15

This fails to compile, at least when BUILDDIR != SOURCEDIR: ======== make[5]: Entering directory `/net/rome.ins.cwi.nl/export/scratch1/manegold/Monet/HG/default/builds/_/clients/mapiclient' /bin/sh ../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient -I../.. -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient -I../mapilib -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../mapilib -I../../common/options -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../../common/options -I../../common/stream -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../../common/stream -I../R/MonetDB.R/src/ -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../R/MonetDB.R/src/ -DLIBMCUTIL -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code -D_REENTRANT -c -o libmcutil_la-dump.lo `test -f 'dump.c' || echo '/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/'`dump.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient -I../.. -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient -I../mapilib -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../mapilib -I../../common/options -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../../common/options -I../../common/stream -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../../common/stream -I../R/MonetDB.R/src/ -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../R/MonetDB.R/src/ -DLIBMCUTIL -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code -D_REENTRANT -c /ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/dump.c -fPIC -DPIC -o .libs/libmcutil_la-dump.o cc1: error: ../R/MonetDB.R/src/: No such file or directory [-Werror] cc1: all warnings being treated as errors make[5]: *** [libmcutil_la-dump.lo] Error 1 make[5]: Leaving directory `/net/rome.ins.cwi.nl/export/scratch1/manegold/Monet/HG/default/builds/_/clients/mapiclient' ======== Stefan ----- Original Message ----- > Changeset: 18471e6cd453 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=18471e6cd453 > Added Files: > clients/mapiclient/mapisplit-dummy.c > clients/mapiclient/profiler-dummy.c > Modified Files: > clients/mapiclient/Makefile.ag > clients/mapiclient/mclient.c > Branch: default > Log Message: > > mclient progress meter part one, mal statement progress > > > diffs (148 lines): > > diff --git a/clients/mapiclient/Makefile.ag b/clients/mapiclient/Makefile.ag > --- a/clients/mapiclient/Makefile.ag > +++ b/clients/mapiclient/Makefile.ag > @@ -16,7 +16,7 @@ > # All Rights Reserved. > > MTSAFE > -INCLUDES = ../mapilib ../../common/options ../../common/stream > $(READLINE_INCS) > +INCLUDES = ../mapilib ../../common/options ../../common/stream > ../R/MonetDB.R/src/ $(READLINE_INCS) > > lib_mcutil = { > NOINST > @@ -24,7 +24,7 @@ lib_mcutil = { > } > > bin_mclient = { > - SOURCES = mclient.c ReadlineTools.c ReadlineTools.h > + SOURCES = mclient.c ReadlineTools.c ReadlineTools.h mapisplit-dummy.c > profiler-dummy.c > LIBS = libmcutil ../mapilib/libmapi \ > ../../common/stream/libstream \ > $(READLINE_LIBS) \ > diff --git a/clients/mapiclient/mapisplit-dummy.c > b/clients/mapiclient/mapisplit-dummy.c > new file mode 100644 > --- /dev/null > +++ b/clients/mapiclient/mapisplit-dummy.c > @@ -0,0 +1,5 @@ > +/* > +This refers to ../R/MonetDB.R/src/mapisplit.c . We cannot have this file > here, > +as the R connector needs to be able to compile stand-alone. > +*/ > +#include "mapisplit.c" > \ No newline at end of file > diff --git a/clients/mapiclient/mclient.c b/clients/mapiclient/mclient.c > --- a/clients/mapiclient/mclient.c > +++ b/clients/mapiclient/mclient.c > @@ -78,6 +78,8 @@ > #define S_ISREG(m) (((m) & S_IFMT) == S_IFREG) > #endif > > +#include "profiler.h" > + > enum modes { > MAL, > SQL, > @@ -1934,14 +1936,13 @@ doFileBulk(Mapi mid, FILE *fp) > } > } > timerResume(); > - > if (hdl == NULL) { > hdl = mapi_query_prep(mid); > CHECK_RESULT(mid, hdl, buf, continue, buf); > } > > assert(hdl != NULL); > - > + profiler_arm(); > mapi_query_part(hdl, buf, length); > CHECK_RESULT(mid, hdl, buf, continue, buf); > > @@ -2722,6 +2723,8 @@ doFile(Mapi mid, const char *file, int u > > if (hdl == NULL) { > timerStart(); > + profiler_arm(); > + > hdl = mapi_query_prep(mid); > CHECK_RESULT(mid, hdl, buf, continue, buf); > } else > @@ -2861,6 +2864,7 @@ usage(const char *prog, int xit) > fprintf(stderr, " -w nr | --width=nr for pagination\n"); > fprintf(stderr, " -D | --dump create an SQL dump\n"); > fprintf(stderr, " -N | --inserts use INSERT INTO > statements when dumping\n"); > + fprintf(stderr, " -P | --progress show progress bar\n"); > fprintf(stderr, "The file argument can be - for stdin\n"); > exit(xit); > } > @@ -2887,6 +2891,7 @@ main(int argc, char **argv) > int interactive = 0; > int has_fileargs = 0; > int option_index = 0; > + int progress = 0; > int settz = 1; > int autocommit = 1; /* autocommit mode default on */ > struct stat statb; > @@ -2912,6 +2917,7 @@ main(int argc, char **argv) > {"pager", 1, 0, '|'}, > #endif > {"port", 1, 0, 'p'}, > + {"progress", 0, 0, 'P'}, > {"rows", 1, 0, 'r'}, > {"statement", 1, 0, 's'}, > #if 0 > @@ -2970,7 +2976,7 @@ main(int argc, char **argv) > #if 0 > "t" > #endif > - "w:r:p:s:Xu:vzH?", > + "w:r:p:s:Xu:vzHP?", > long_options, &option_index)) != -1) { > switch (c) { > case 0: > @@ -3114,6 +3120,9 @@ main(int argc, char **argv) > case 'z': > settz = 0; > break; > + case 'P': > + progress = 1; > + break; > case '?': > /* a bit of a hack: look at the option that the > * current `c' is based on and see if we recognize > @@ -3226,6 +3235,16 @@ main(int argc, char **argv) > } > } > > + // switch on progress bars > + if (mode == SQL && progress) { > + char* buf = malloc(100); > + int port = profiler_start(); > + sprintf(buf, "CALL profiler_openstream('127.0.0.1', %d)", port); > + mapi_query(mid, buf); > + sprintf(buf, "CALL profiler_stethoscope(0)"); > + mapi_query(mid, buf); > + } > + > /* give the user a welcome message with some general info */ > if (!has_fileargs && command == NULL && isatty(fileno(stdin))) { > char *lang; > @@ -3296,6 +3315,7 @@ main(int argc, char **argv) > /* execute from command-line, need interactive to know whether > * to keep the mapi handle open */ > timerStart(); > + profiler_arm(); > c = doRequest(mid, command); > timerEnd(); > } > diff --git a/clients/mapiclient/profiler-dummy.c > b/clients/mapiclient/profiler-dummy.c > new file mode 100644 > --- /dev/null > +++ b/clients/mapiclient/profiler-dummy.c > @@ -0,0 +1,5 @@ > +/* > +This refers to ../R/MonetDB.R/src/profiler.c . We cannot have this file > here, > +as the R connector needs to be able to compile stand-alone. > +*/ > +#include "profiler.c" > \ No newline at end of file > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 2

About merovingian and the MySQL protocol
by Fabian Groffen 23 Feb '15

23 Feb '15

Hiya all, A long time ago, I examined the MySQL client-server protocol, to understand why it is performing so well. This resulted in the idea (back then) to create a MySQL server implementation in merovingian, that proxied commands to mserver/mapi calls. This to possibly gain from performance benefits, but obviously more to open up the world of MySQL clients that exist to MonetDB. A minimalistic start was added to the code [1], but nothing more than that ever emerged. Years later, I ended up at a similar quest, one where I had to implement the MySQL server protocol and perform the rituals to make a MySQL client believe it is dealing with a proper server [2]. Objective is different here (some will recognise merovingian's multiplex-funnel feature), but the codepath is largely re-usable for the purpose of the above. Being rusty with the MonetDB code-base, not having followed any of its developments lately, whilst on this island in the sun, I'm wondering if it makes sense to implement the first at all, or whether it is better to simply remove the if-deffed code, for it is incomplete. Opinions? Best, Fabian [1] http://dev.monetdb.org/hg/MonetDB/file/fb8e50da2917/tools/merovingian/daemo… [2] https://github.com/grobian/mysqlpq

1 0

MonetDB in Fedora 21 and 22, and EPEL 7
by Jan Holčapek 10 Feb '15

10 Feb '15

Hello, I am aware that MonetDB is available as native Fedora 18-20 [1] and EPEL 6 [2] packages. However, I am missing native Fedora 21-22, as well as EPEL 7 packages. Would you find it beneficial if I try to push MonetDB directly to Fedora/EPEL packages, ie. it would have a record in Fedora Package Database [3] and be built on Fedora infrastructure? That said, I would become a mere package maintainer for the two distributions. Cheers Jan [1] https://www.monetdb.org/Documentation/UserGuide/Downloads/Fedora [2] https://www.monetdb.org/downloads/epel/ [3] https://admin.fedoraproject.org/pkgdb/

2 1

Adapting code Feb2013 -> Oct2014
by Roberto Cornacchia 09 Feb '15

09 Feb '15

Hi there, I have some custom code running on Feb2013 release. As I'm now trying to move to Oct2014, I need to update this custom code. MonetDB API has changed a bit in the meantime, for example if I just try to compile my code within Oct2014, I get error: implicit declaration of function 'ALGfetch' Indeed, looking at the previous commits, ALGfetch is one of the interfaces that have been removed. Question: Is there any documentation that can help me see more clearly what changed and how to adapt the code? Or is browsing the commit history my only resource? Thanks, Roberto

2 3

Announcement: New Oct2014-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 29 Jan '15

29 Jan '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2014-SP2 bugfix release Client Package * Changes to the Perl interface, thanks to Stefan O'Rear: 1. removes "use sigtrap", because this has global effects and should not be used by modules, only by the application. 2. allows Perl 5.8.1+ Unicode strings to be passed to quote() and included in statements (UTF-8 encoded, as expected by Monet's str module) 3. quote and unquote now use the same quoting rules as the MonetDB server, allowing for all characters except NUL to be round-tripped 4. several character loops have been reimplemented in regex for much greater performance 5. micro-optimizations to the result fetch loop 6. block boundaries are preserved in piggyback data so that Xclose is not appended or prepended to a SQL command 7. diagnostic messages #foo before a result header are ignored, this is necessary to use recycler_pipe 8. fail quickly and loudly if we receive a continuation prompt (or any other response that starts with a non-ASCII character) 9. header lines must start with %, not merely contain %, fixing a bug when querying a table where string values contain % 10. after closing a large resultset, account for the fact that a reply will come and do not lose sync 11. allow a MAPI_TRACE environment variable to dump wire protocol frames to standard output 12. fixes maximum MAPI block size to match the server limit of 16k. previously would crash on blocks larger than 16k SQL * Fixed a typo in a column name of the sys.tablestoragemodel view (auxillary changed to auxiliary). Bug Fixes * 3467: Field aliases with '#' character excise field names in result set. * 3605: relational query without result * 3619: Missing dll on MonetDB Start * 3622: Type resolution error * 3624: insert of incomplete or invalid ip address values in inet column is silently accepted but the values are not stored (they become/show nil) * 3626: casting a type without alias results in program contains errors * 3628: mclient and ODBC driver report 'type mismatch' when stddev_pop used in a select which returns 0 rows * 3629: IF THEN ELSEIF always evaluates the first test as true * 3630: segv on rel_order_by_column_exp * 3632: running make clean twice gives an error in clients/ruby/adapter * 3633: Wrong result for HAVING with floating-point constant * 3640: Missing implementation of scalar function: sql_sub(<date>, <month interval>) * 3641: SQL lexer fails to detect string end if it the last character is U+FEFF ZERO WIDTH NO-BREAK SPACE * 3642: Combined WHERE conditions less-than plus equals-to produce incorrect results * 3643: Missing implementations of scalar function: sql_sub(<timetz>, arg2) * 3644: COPY INTO fails to import "inet" data type when value has prefix length in CIDR notation * 3646: ORDER BY clause does not produce proper results on 'inet' datatype * 3649: recycler crashes with concurrent transactions

1 0

Re: MonetDB: Oct2014 - Reimplemented rangejoin, now using imprints o...
by Roberto Cornacchia 28 Jan '15

28 Jan '15

Hi Sjoerd, This is very welcome! If I read well, this will use, in order of preference: 1. binary search (if l is sorted) 2. imprints if available 3. nested loop otherwise We use rangejoin extensively within Spinque and the previous one (just nested loop) has never an option. So far we have been using our own version which is perhaps naive but proved to be effective: - sort all inputs if not sorted already - perform a mergejon-like fast scan Though simple, I have not found so far a case where this strategy would not outperform nested loops by far. The cost of sorting was always far less than the cost of a full nested loop. I was wondering what your thoughts about this would be. Could this strategy replace the number 3 above? Does it make sense to keep the vanilla nested loop? Best, Roberto On 27 January 2015 at 14:09, Sjoerd Mullender <commits(a)monetdb.org> wrote: > Changeset: 5147add3bb38 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=5147add3bb38 > Modified Files: > gdk/ChangeLog.Oct2014 > gdk/gdk_join.c > gdk/gdk_private.h > gdk/gdk_select.c > Branch: Oct2014 > Log Message: > > Reimplemented rangejoin, now using imprints or binary search if possible. > >

2 2

Re: MonetDB: Oct2014 - Use imprints to calculate min/max of bat.
by Lefteris 22 Jan '15

22 Jan '15

Imprints code contains such a search for min/max, maybe we could expose it as a function and thus avoid having the same code twice (for the usual reasons of easy maintenance and bugfix) On Thu, Jan 22, 2015 at 2:15 PM, Sjoerd Mullender <commits(a)monetdb.org> wrote: > Changeset: 69a4f7d2fd58 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=69a4f7d2fd58 > Modified Files: > gdk/gdk_aggr.c > Branch: Oct2014 > Log Message: > > Use imprints to calculate min/max of bat. > But with limitations: no grouped min/max, and only on the complete bat > (i.e. if view, it's a view on the complete bat). > > > diffs (37 lines): > > diff --git a/gdk/gdk_aggr.c b/gdk/gdk_aggr.c > --- a/gdk/gdk_aggr.c > +++ b/gdk/gdk_aggr.c > @@ -2207,8 +2207,31 @@ BATminmax(BAT *b, void *aggr, > return NULL; > needdecref = 1; > } > - (void) (*minmax)(&pos, b, NULL, 1, 0, 0, 0, BATcount(b), > - NULL, NULL, BATcount(b), 1, 0); > + if (b->T->imprints && > + (VIEWtparent(b) == 0 || > + BATcount(b) == BATcount(BBPdescriptor(VIEWtparent(b))))) { > + pos = oid_nil; > + if (minmax == do_groupmin) { > + /* find first non-empty bin */ > + for (s = 0; s < b->T->imprints->bits; s++) { > + if (b->T->imprints->stats[s + 128]) { > + pos = b->T->imprints->stats[s] + > b->hseqbase; > + break; > + } > + } > + } else { > + /* find last non-empty bin */ > + for (s = b->T->imprints->bits - 1; s >= 0; s--) { > + if (b->T->imprints->stats[s + 128]) { > + pos = b->T->imprints->stats[s + > 64] + b->hseqbase; > + break; > + } > + } > + } > + } else { > + (void) (*minmax)(&pos, b, NULL, 1, 0, 0, 0, BATcount(b), > + NULL, NULL, BATcount(b), 1, 0); > + } > if (pos == oid_nil) { > res = ATOMnilptr(b->ttype); > } else { > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

1 1

Perl driver improvements
by Stefan O'Rear 13 Jan '15

13 Jan '15

I have been integrating MonetDB into a Perl system for the last few months, and have made a few changes to the Perl driver to improve performance and reliability. I hereby release these changes under the same license as MonetDB itself. I have permission from my employer to do this. Is this to correct place to send and/or discuss patches? In order of patch hunks; more details will be provided for any part if requested: 1. removes "use sigtrap", because this has global effects and should not be used by modules, only by the application. 2. allows Perl 5.8.1+ Unicode strings to be passed to quote() and included in statements (UTF-8 encoded, as expected by Monet's str module) 3. quote and unquote now use the same quoting rules as the MonetDB server, allowing for all characters except NUL to be round-tripped 4. several character loops have been reimplemented in regex for much greater performance 5. micro-optimizations to the result fetch loop 6. block boundaries are preserved in piggyback data so that Xclose is not appended or prepended to a SQL command 7. diagnostic messages #foo before a result header are ignored, this is necessary to use recycler_pipe 8. fail quickly and loudly if we receive a continuation prompt (or any other response that starts with a non-ASCII character) 9. header lines must start with %, not merely contain %, fixing a bug when querying a table where string values contain % 10. after closing a large resultset, account for the fact that a reply will come and do not lose sync 11. allow a MAPI_TRACE environment variable to dump wire protocol frames to standard output 12. fixes maximum MAPI block size to match the server limit of 16k. previously would crash on blocks larger than 16k

2 1

Bug in Query Plan??
by Vijay Krishna 09 Jan '15

09 Jan '15

Hi, I have been studying on multiple join performances in MonetDB. I am running the latest version of MonetDB Oct2014 - SP1 (MonetDB-11.19.3) on a CentOS machine with 128 GB RAM. All my tables have 100 to 300 columns and around 35 lakh rows. I have the following query with 5 joins and a few criteria. SELECT * FROM table1 INNER JOIN table2 on table1.col1=table2.col1 LEFT JOIN table3 ON table1.col2=table3.col1 LEFT JOIN table3 ON table1.col3=table3.col1 LEFT JOIN table4 ON table2.col2=table4.col1 LEFT JOIN table5 ON table2.col3=table5.col1 WHERE (((table2.col10 like '%a%' OR table5.col11 like '%s%') and ((table2.col12 like '%w%' *and table4.col13 like '%k'*) OR table2.col14 like '%h%') and (table1.col21 = 1) AND (table1.col22 IN (1,2,3))) AND (((table1.col23 >= 15842000000000000) AND (table1.col23 <= 15842999999999999)) OR ((table1.col23 >= 0) AND (table1.col23 <= 999999999999)))) ORDER BY 1 DESC LIMIT 10 OFFSET 0; This query has only 2000 matching rows and it returns the result in *2.9 seconds*. In the same query, if I remove one criteria alone (the bolded part), the result would still match only 2000 rows, but the result would come in *300 milliseconds* even. I used the PLAN statement and found that the query plan that the executor takes for the first query is a much complex path and *it performs 3 joins for all the 35 lakh rows* (i.e. before executing the criteria even) and hence the time has spiked to 3 seconds. May be the hot data of 35 lakh tuples would have not fit into the memory?? Not sure. But if I remove the highlighted criteria, the executor evaluates the criteria first and *it performs the joins for only 7000 rows* and hence it is faster 1000 times. Is this a bug? Or is there any way to optimise my query to match the executor to choose the right plan? *I am looking on to install MonetDB in our production setup, but finding this as a showstopper. * Any help much appreciated. I could even demo this. Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

1 0

Some fun for Christmas
by Ying Zhang 25 Dec '14

25 Dec '14

Dear all, Just to add some fun for the Christmas, I have two new videos for you (starring Thibault, Hannes ;-)): /ufs/zhang/Blaeu_demo_screencast.mp4 /ufs/zhang/monetdb-r-screencast.mp4 They are not published yet. If you want to share them externally, please check with Hannes and/or Thibault. Feedback is highly appreciated. However, please keep in mind that these videos are primarily meant to be informative. They were in a hurry without prepared scripts :P Anyways, enjoy. Wish you a nice Christmas and New Year holiday. With kind regards, Jennie

1 0

Re: MonetDB: mosaic - Fix merge conflict.
by Martin Kersten 12 Dec '14

12 Dec '14

Thank you for taking the initiative. On 12/12/14 11:03, Sjoerd Mullender wrote: > Changeset: 3e1d5cbd649d for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=3e1d5cbd649d > Modified Files: > sql/server/rel_optimizer.c > Branch: mosaic > Log Message: > > Fix merge conflict. > > > diffs (21 lines): > > diff --git a/sql/server/rel_optimizer.c b/sql/server/rel_optimizer.c > --- a/sql/server/rel_optimizer.c > +++ b/sql/server/rel_optimizer.c > @@ -6426,7 +6426,7 @@ rel_merge_table_rewrite(int *changes, mv > > /* do not include empty partitions */ > if ((nrel || nt->next) && > - pt && isTable(pt) && pt->readonly && !store_funcs.count_col(sql->session->tr, pt->columns.set->h->data, 1)){ > + pt && isTable(pt) && pt->access == TABLE_READONLY && !store_funcs.count_col(sql->session->tr, pt->columns.set->h->data, 1)){ > continue; > } > > @@ -6439,7 +6439,7 @@ rel_merge_table_rewrite(int *changes, mv > sql_exp *ne = m->data; > int i; > > - if (pt && isTable(pt) && pt->readonly && sel && (nrel || nt->next) && (i=find_col_exp(cols, e)) != -1) { > + if (pt && isTable(pt) && pt->access == TABLE_READONLY && sel && (nrel || nt->next) && (i=find_col_exp(cols, e)) != -1) { > /* check if incase of an expression if the part falls within the bounds else skip this (keep at least on part-table) */ > void *min, *max; > sql_column *col = NULL; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

1 0

Win32 build question
by g4＠novadsp.com 29 Nov '14

29 Nov '14

Hello and thanks for MonetDB. Splendid stuff. Brilliant. I am trying to build a 32 bit version of 11.19.3 on Windows from the latest tarball. Root folder is R:\src\libraries\MonetDB-11.19.3. Building in R:\src\libraries\MonetDB-11.19.3\bin The build is failing at the point shown below. What have I missed here? The build folder is a sibling of /NT as indicated in the windowsbuild.rst ...? Many thanks for any pointers. Jerry. -------------------------snip-------------------------- copy ".\..\..\..\monetdb5\scheduler\run_isolate.mal" "run_isolate.mal" 1 file(s) copied. if not exist "tools" mkdir "tools" copy ".\..\..\monetdb5\tools\Makefile.msc" "tools\Makefile" 1 file(s) copied. cd "tools" && U:\vs2013\VC\BIN\nmake.exe /nologo DEBUG=1 "prefix=R:\src\ libraries\MonetDB-11.19.3\bin" "bits=32" all NMAKE : fatal error U1073: don't know how to make '..\..\gdk\libbat.lib' Stop. NMAKE : fatal error U1077: 'cd' : return code '0x2' Stop. NMAKE : fatal error U1077: 'cd' : return code '0x2' Stop. NMAKE : fatal error U1077: 'U:\vs2013\VC\BIN\nmake.exe' : return code '0x2' Stop. R:\src\libraries\MonetDB-11.19.3\bin> -------------------------snip-------------------------- Platform details: Windows 7x64 using VS2013 pro x86 command shell. I've got a very simple script which sets up the correct include/lib paths and then invokes nmake thus: SET LIBOPENSSL=U:\pkg\OpenSSL\Win32 SET LIBICONV=U:\pkg\monet\iconv-1.9.2.win32 SET LIBZLIB=U:\pkg\monet\zlib-1.2.5 SET LIBXML2=U:\pkg\monet\libxml2-2.7.8.win32 SET LIBPCRE=U:\pkg\monet\pcre-7.0-bin SET INCLUDE=%INCLUDE%;U:\pkg\monet\iconv-1.9.2.win32\include;U:\pkg\monet\pcre-7 .0-bin\include nmake /D -f ..\NT\makefile DEBUG=1 HAVE_OPENSSL=1 HAVE_SQL=1 HAVE_MONETDB5=1 all

1 0

Re: MonetDB: default - use column_id instead of schema,table,column ...
by Niels Nes 29 Nov '14

29 Nov '14

On Sat, Nov 29, 2014 at 10:37:24AM +0100, Niels Nes wrote: > Changeset: f49a132fc195 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=f49a132fc195 > Modified Files: > sql/backends/monet5/sql_statistics.c > sql/scripts/80_statistics.sql > sql/storage/store.c > Branch: default > Log Message: > > use column_id instead of schema,table,column names to speed up insert and > lookup's beware the schema for the statistics table changed. Do a drop table statistics and recreate using the create statement in the 80_statistics.sql script. Niels > > > diffs (69 lines): > > diff --git a/sql/backends/monet5/sql_statistics.c b/sql/backends/monet5/sql_statistics.c > --- a/sql/backends/monet5/sql_statistics.c > +++ b/sql/backends/monet5/sql_statistics.c > @@ -103,7 +103,7 @@ sql_analyze(Client cntxt, MalBlkPtr mb, > > if (col && strcmp(bc->name, col)) > continue; > - snprintf(dquery, 8192, "delete from sys.statistics where \"schema\" ='%s' and \"table\"='%s' and \"column\"='%s';", b->name, bt->name, bc->name); > + snprintf(dquery, 8192, "delete from sys.statistics where \"column_id\" = %d;", c->base.id); > if (samplesize > 0) { > bsample = BATsample(bn, (BUN) 25000); > } else > @@ -147,8 +147,7 @@ sql_analyze(Client cntxt, MalBlkPtr mb, > snprintf(maxval, 4, "nil"); > snprintf(minval, 4, "nil"); > } > - snprintf(query, 8192, "insert into sys.statistics values('%s','%s','%s','%s',%d,now()," LLFMT "," LLFMT "," LLFMT "," LLFMT ",'%s','%s',%s);", b->name, bt->name, bc->name, c->type.type->sqlname, width, > - (samplesize ? samplesize : sz), sz, uniq, nils, minval, maxval, sorted ? "true" : "false"); > + snprintf(query, 8192, "insert into sys.statistics values(%d,'%s',%d,now()," LLFMT "," LLFMT "," LLFMT "," LLFMT ",'%s','%s',%s);", c->base.id, c->type.type->sqlname, width, (samplesize ? samplesize : sz), sz, uniq, nils, minval, maxval, sorted ? "true" : "false"); > #ifdef DEBUG_SQL_STATISTICS > mnstr_printf(cntxt->fdout, "%s\n", dquery); > mnstr_printf(cntxt->fdout, "%s\n", query); > diff --git a/sql/scripts/80_statistics.sql b/sql/scripts/80_statistics.sql > --- a/sql/scripts/80_statistics.sql > +++ b/sql/scripts/80_statistics.sql > @@ -21,9 +21,7 @@ > > > CREATE TABLE sys.statistics( > - "schema" string, > - "table" string, > - "column" string, > + "column_id" integer, > "type" string, > width integer, > stamp timestamp, > diff --git a/sql/storage/store.c b/sql/storage/store.c > --- a/sql/storage/store.c > +++ b/sql/storage/store.c > @@ -4317,12 +4317,8 @@ sql_trans_dist_count( sql_trans *tr, sql > sql_schema *sys = find_sql_schema(tr, "sys"); > sql_table *stats = find_sql_table(sys, "statistics"); > if (stats) { > - sql_column *stats_schema = find_sql_column(stats, "schema"); > - sql_column *stats_table = find_sql_column(stats, "table"); > - sql_column *stats_column = find_sql_column(stats, "column"); > - oid rid = table_funcs.column_find_row(tr, stats_column, col->base.name, NULL, > - stats_table, col->t->base.name, NULL, > - stats_schema, col->t->s->base.name, NULL, NULL); > + sql_column *stats_column_id = find_sql_column(stats, "column_id"); > + oid rid = table_funcs.column_find_row(tr, stats_column_id, &col->base.id, NULL); > if (rid != oid_nil) { > sql_column *stats_unique = find_sql_column(stats, "unique"); > void *v = table_funcs.column_find_value(tr, stats_unique, rid); > @@ -4344,12 +4340,8 @@ sql_trans_ranges( sql_trans *tr, sql_col > sql_schema *sys = find_sql_schema(tr, "sys"); > sql_table *stats = find_sql_table(sys, "statistics"); > if (stats) { > - sql_column *stats_schema = find_sql_column(stats, "schema"); > - sql_column *stats_table = find_sql_column(stats, "table"); > - sql_column *stats_column = find_sql_column(stats, "column"); > - oid rid = table_funcs.column_find_row(tr, stats_column, col->base.name, NULL, > - stats_table, col->t->base.name, NULL, > - stats_schema, col->t->s->base.name, NULL, NULL); > + sql_column *stats_column_id = find_sql_column(stats, "column_id"); > + oid rid = table_funcs.column_find_row(tr, stats_column_id, &col->base.id, NULL); > if (rid != oid_nil) { > sql_column *stats_min = find_sql_column(stats, "minval"); > sql_column *stats_max = find_sql_column(stats, "maxval"); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room L3.14, phone ++31 20 592-4098 sip:4098@sip.cwi.nl url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl

1 0

Announcement: New Oct2014-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 27 Nov '14

27 Nov '14

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2014-SP1 bugfix release MonetDB5 Server * Fixed adding of 0 intervals to dates. Build Environment * The libraries included in the Windows installers have been upgraded. We now use libxml2-2.9.2, openssl-1.0.1j, pcre-8.36, and zlib-1.2.8. MonetDB Common * Implemented a change to the way in which string bats are appended. We now try harder to limit the growth of the string heap. * Fixed some problems with BATsample. It was possible for BATsample to return a value that was just beyond the end of the sampled BAT. Also, on some systems the range of the rand() function is rather limited (0..32767) and trying to get a sample larger than this range would result in an infinite loop. SQL * Fixed sys.queue() implementation to report on other queries being executed. * A number of bugs were fixed in the code to upgrade a database from previous releases. This version should fix the upgrade of a database that had been upgraded to the Oct2014 release, but also properly upgrade directly from Jan2014 and Feb2013 releases. Bug Fixes * 3580: cosmetic change (append newline) * 3609: Incorrect use of generate_series * 3611: quantile() and median() commands crash when used 2x on the same variable on a null table * 3612: assertion failure when deleting rows from table to which a FK constraint is defined * 3620: ORDER BY broken when using UNION ALL * 3621: Hexadecimal literal vs decimal literal

1 0

Database can't started.
by Mhd Sulhan 24 Nov '14

24 Nov '14

I have tried to run monetdb version 11.19.3 and monetdb-e46b810ccbd0 and both can't start database. My machine is Linux x86, Linux bubu 3.17.3-1-ARCH #1 SMP PREEMPT Fri Nov 14 22:56:01 CET 2014 i686 GNU/Linux with glibc 2.20. I compile monetdb with this options, ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --disable-assert \ --without-rubygem --disable-testing --enable-optimize --enable-rintegration I can run monetdbd, but fail when running monetdb start <db> or mclient -u <user> -d <db>. Here is some log when I'm running monetdb start, $ mclient -u monetdb -d cdr password: monetdbd: internal error while starting mserver, please refer to the logs With this email, I also attach merovingian.log. -- Salam hormat, Mhd Sulhan { phone: 08567826625, gtalk: m.shulhan, github: shuLhan }

2 2

Re: Resource on the PHP API for monetdb
by Pierre-Adrien Coustillas 24 Nov '14

24 Nov '14

Hi, There is a package pour php5 monetdb. For debian : https://www.monetdb.org/downloads/deb/dists/wheezy/monetdb/binary-amd64/ package : php5-monetdb-client_11.19.3-20141028_all.deb There is two file : php_mapi.inc and php_monetdb.php Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35 ----- Mail original ----- De: "Leon-Constantin Fopa" <leon-constantin.fopa(a)imag.fr> À: "Communication monetdb" <users-list(a)monetdb.org> Cc: "developers monetdb" <developers-list(a)monetdb.org> Envoyé: Lundi 24 Novembre 2014 09:04:51 Objet: Resource on the PHP API for monetdb Hi every one, I'm interested in accessing monetdb from a PHP script. Can someone provide me a link on resource with introduce that. The research of the keyword "monetdb" from the the http://php.net website search engine give no related result. thanks in advance, Best regards, Fopa Léon Constantin _______________________________________________ users-list mailing list users-list(a)monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

1 0

Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
by Ashish Kumar Singh 21 Nov '14

21 Nov '14

Hi, Below are the steps. Unzip archive in to /opt/monetdb_test/ directroy to minimise changes in the script datbase can live anywhere in the file system. Single Big File Import: 1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql. Cat all files present inside testdata directory in to on single file by using following command: cat * > ../merged_data Load this merged file in to database using script single_import.sql from mclient Note down the timing, usually 3-5 minutes on 4 core 17 GB machine on amazon. Multiple small File Import: 1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql. 3. Load files in to database using multiple_import.sql using mclient Regards, Ashish From: Anthony Damico <ajdamico(a)gmail.com<mailto:ajdamico@gmail.com>> Reply-To: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Date: Monday, 17 November 2014 3:51 pm To: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Cc: "developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>" <developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release please provide a list of every command that you executed to reproduce the problem. you need to make the list of commands as short as possible, so people trying to help you are not wasting time on irrelevant code.. thanks!! On Mon, Nov 17, 2014 at 4:22 AM, Ashish Kumar Singh <ashishkumar.singh(a)altair.com<mailto:ashishkumar.singh@altair.com>> wrote: Thanks for looking in to it, I have uploaded required test files to you at below link: https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu<https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu> Regards, Ashish From: Anthony Damico <ajdamico(a)gmail.com<mailto:ajdamico@gmail.com>> Reply-To: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Date: Saturday, 15 November 2014 2:40 pm To: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Cc: "developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>" <developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release could you provide a minimal reproducible example? the less code that re-creates the problem, the better. thanks http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducib… On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh <ashishkumar.singh(a)altair.com<mailto:ashishkumar.singh@altair.com>> wrote: Additional information here, If I merge all these files in to one file and then import to monetdb then time taken is 3-5 minutes as compared to 10-15 hours with October 2014 branch. Hopefully some one from moentdb team will look in to this. I am glad to see that inspite of this slowness ( due to which this build is unusable) the segfaults are gone! Regards, Ashish From: Ashish Singh <ashishkumar.singh(a)altair.com<mailto:ashishkumar.singh@altair.com>> Date: Friday, 14 November 2014 7:41 pm To: "developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>" <developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>> Cc: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release Is any body else seeing this problem with October,2014 release? I wish developer team can provide some insights in this area. Regards, Ashish From: Ashish Singh <ashishkumar.singh(a)altair.com<mailto:ashishkumar.singh@altair.com>> Date: Thursday, 13 November 2014 11:22 pm To: "developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>" <developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>> Cc: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release All, With October,2014 release we are stucked in to very strange performance issue while importing data on virtualised environments. On Virtual machines in our local VMware servers and amazon machine we are seeing import performance has downgraded to close to 15 to 20 while importing ~ 2000 files as compared to 1 hour time being taken with old release. This we are seeing consistently with CentOS, SLES vm and configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm just to make sure its not an issue with a particular instance setup. On a physical machine of 4 core 24 GB we are able to match speed of import with old monetdb. We are issuing copy into command through JDBC connection. Can you guys help me with pointers on this issue, I will be happy to share any debug info. Regards, Ashish Kumar Singh Mobile: +91-9972045095<tel:%2B91-9972045095> Skype: toashishk(a)gmail.com<mailto:toashishk@gmail.com> _______________________________________________ users-list mailing list users-list(a)monetdb.org<mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org<mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list

2 4

python-monetdb
by Gijs Molenaar 17 Nov '14

17 Nov '14

Hi MonetDB developers, I noticed the python3 package was broken, and pypi was outdated. I lost the key for committing to the tree, so I did the following: * I created a temporary fork on github: https://github.com/gijzelaerr/python-monetdb * I've merge the python2 and python3 code using six. This is a reverse of the split I previously did, since it is just much easier to manage one single code base. People make mistakes. Also this way the package on pypi works with all pythons at the same time (most important reason). * cleanup the test suite * i'm using tox myself now for running the test suite with all the various pythons * I've uploaded a new version to pypi, with version ID 11.19.3.1 * This version is tested with python 2.6, 2.7 3.3, 3.4 and pypy I propose you just take my changes and merge them back in your tree, then I remove the fork. Also probably it is wise you incorporate the upload to pypi in your release process. Most people probably use pypi for installing their python requirements. For this you need to create a pypi account and let me know the username so I can give this account authorisation to upload the package. You can upload to pypi from the source folder using: $ python setup.py sdist upload Sorry for any inconvenience but I just needed to get python3 support quickly. -- Gijs Molenaar http://pythonic.nl <http://gijs.pythonic.nl>

1 0

Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
by Ashish Kumar Singh 17 Nov '14

17 Nov '14

Thanks for looking in to it, I have uploaded required test files to you at below link: https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu<https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu> Regards, Ashish From: Anthony Damico <ajdamico(a)gmail.com<mailto:ajdamico@gmail.com>> Reply-To: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Date: Saturday, 15 November 2014 2:40 pm To: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Cc: "developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>" <developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release could you provide a minimal reproducible example? the less code that re-creates the problem, the better. thanks http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducib… On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh <ashishkumar.singh(a)altair.com<mailto:ashishkumar.singh@altair.com>> wrote: Additional information here, If I merge all these files in to one file and then import to monetdb then time taken is 3-5 minutes as compared to 10-15 hours with October 2014 branch. Hopefully some one from moentdb team will look in to this. I am glad to see that inspite of this slowness ( due to which this build is unusable) the segfaults are gone! Regards, Ashish From: Ashish Singh <ashishkumar.singh(a)altair.com<mailto:ashishkumar.singh@altair.com>> Date: Friday, 14 November 2014 7:41 pm To: "developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>" <developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>> Cc: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release Is any body else seeing this problem with October,2014 release? I wish developer team can provide some insights in this area. Regards, Ashish From: Ashish Singh <ashishkumar.singh(a)altair.com<mailto:ashishkumar.singh@altair.com>> Date: Thursday, 13 November 2014 11:22 pm To: "developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>" <developers-list(a)monetdb.org<mailto:developers-list@monetdb.org>> Cc: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release All, With October,2014 release we are stucked in to very strange performance issue while importing data on virtualised environments. On Virtual machines in our local VMware servers and amazon machine we are seeing import performance has downgraded to close to 15 to 20 while importing ~ 2000 files as compared to 1 hour time being taken with old release. This we are seeing consistently with CentOS, SLES vm and configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm just to make sure its not an issue with a particular instance setup. On a physical machine of 4 core 24 GB we are able to match speed of import with old monetdb. We are issuing copy into command through JDBC connection. Can you guys help me with pointers on this issue, I will be happy to share any debug info. Regards, Ashish Kumar Singh Mobile: +91-9972045095<tel:%2B91-9972045095> Skype: toashishk(a)gmail.com<mailto:toashishk@gmail.com> _______________________________________________ users-list mailing list users-list(a)monetdb.org<mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list

1 0

Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
by Ashish Kumar Singh 15 Nov '14

15 Nov '14

All, With October,2014 release we are stucked in to very strange performance issue while importing data on virtualised environments. On Virtual machines in our local VMware servers and amazon machine we are seeing import performance has downgraded to close to 15 to 20 while importing ~ 2000 files as compared to 1 hour time being taken with old release. This we are seeing consistently with CentOS, SLES vm and configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm just to make sure its not an issue with a particular instance setup. On a physical machine of 4 core 24 GB we are able to match speed of import with old monetdb. We are issuing copy into command through JDBC connection. Can you guys help me with pointers on this issue, I will be happy to share any debug info. Regards, Ashish Kumar Singh Mobile: +91-9972045095 Skype: toashishk(a)gmail.com

1 2

Timestamp accuracy: milli-seconds or micro-seconds
by Stefan Manegold 07 Nov '14

07 Nov '14

Hi, with the Oct2014 release of MonetDB (I did no (yet) check any other version), I just came across the following: It appears that while timestamps are rendered with 6 decimal digits, i.e., suggesting micro-seconds accuracy/resolution, only milli-seconds are considered: sql>select cast('1312-11-10 12:11:10.123456' as timestamp) , cast('1312-11-10 12:11:10' as timestamp) + 0.123456 , cast('1312-11-10 12:11:10' as timestamp) + interval '0.123456' second; +----------------------------+----------------------------+----------------------------+ | L1 | L2 | sql_add_single_value | +============================+============================+============================+ | 1312-11-10 12:11:10.123000 | 1312-11-10 12:11:10.123000 | 1312-11-10 12:11:10.123000 | +----------------------------+----------------------------+----------------------------+ 1 tuple (5.459ms) Is this a bug or a "feature"? Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

Announcement: New Oct2014 Feature release of MonetDB suite
by Sjoerd Mullender 06 Nov '14

06 Nov '14

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Important consideration when upgrading to this release. In this release we have fixed bug 3474. This bug made it possible that there were duplicates despite there being a unique or primary key constraint present. This could only happen for multi-column constraints. MonetDB maintains an internal column containing a hash for multi-column constraints, and there was a bug in the maintenance code of this column. During an upgrade, this column needs to be recreated. This is done by dropping and adding back the constraints. If there are no duplicates, adding back the constrains should work without problem, but if there are duplicates, adding the constraints will fail. There is unfortunately no easy way to tell the user, so after having made the first connection to the server (when the upgrade happens), check the merovingian log (if you use merovingian -- also known as monetdbd) or the server output window (if you don't use merovingian). Jaql support is removed. The experimental code for Jaql support has been removed. If you have RPM or DEB packages for Jaql installed from an earlier version (MonetDB-jaql and monetdb5_jaql respectively), you will need to uninstall them before the upgrade. Known issue when upgrading old versions. We have noticed that upgrading directly from a version older than the Jan2014 release gave an error. Upgrading to Jan2014 first and then on to Oct2014 should work. Backup. As always, make a backup of your databases before upgrading. Oct 2014 feature release MonetDB5 Server * Removed algebra.materialize. * Removed algebra.kunique and algebra.tunique. They were subsumed by algebra.subunique. * Remove algebra.antiuselect and algebra.thetauselect. They were subsumed by algebra.subselect. * Removed algebra.topN and its imlementation BATtopN. The function was not used. * Removed aggr.histogram and its implementation BAThistogram. The function was not used, and did not produce output in the "headless" (i.e. dense-headed) format. Histograms can be created as a by-product of group.subgroup. Build Environment * Jacqueline, the MonetDB/JAQL frontend, has been removed. The frontend never grew beyond being experimental, and there is no interest anymore to maintain the code. Merovingian * monetdb create: add -p flag to set monetdb user password on creation, and therefore allow creating the database in unlocked state mapilib * Changed mapi_timeout argument from seconds to milliseconds. stream * Changed mnstr_settimeout function so that the specified timeout is now in milliseconds (used to be seconds), and that it also needs an extra argument specifying a callback function (no arguments, int result) that should return TRUE if the timeout should cause the function to abort or continue what it was doing. MonetDB Common * Added "multifarm" capability. It is now possible to separate persistent and transient BATs into different directories (presumably on different disks). This can be done by using the --dbextra option of mserver5 (see the man page). SQL * Added PostgreSQL compatible string TRIM, LTRIM, RTRIM, LPAD and RPAD functions * Stop support for upgrading directly from a database created with a server from the Oct2012 release or older. You can upgrade via the Feb2013 or Jan2014 release. Bug Fixes * 2945: evaluation of SQL "between SYMMETRIC" requires MAL iterator because there is no (bulk) MIN/MAX(a,b) * 3204: monetdb create: allow setting admin password during creation * 3390: Missing definition for pushSht in monetdb5/mal/mal_builder.h * 3402: We should have a C implementation of mal.multiplex. * 3422: Segmentation fault after large table insert * 3459: incomplete implementation of JDBC driver getNumericFunctions(), getStringFunctions(), getSystemFunctions(), getTimeDateFunctions(), getSQLKeywords() methods in MonetDatabaseM * 3471: JDBC driver: incorrect output result of SQL query: SELECT 1 ; * 3474: bulk and scalar versions of mkey.rotate_xor_hash differ * 3484: COPY INTO on a file works fine on Linux/OSX, but not on Windows * 3488: Slow SQL execution for correlated scalar subquery * 3489: SQL query with ORDER BY does not order its result as requested * 3490: SQL query kills the mserver5 (Segmentation fault) * 3491: SQL query kills the mserver5 (Segmentation fault) * 3493: Test monetdb5/modules/mal/Tests/pqueue.mal fails since recent checkins * 3494: Tests monetdb5/modules/mal/Tests/pqueue[23].mal lack correct/expected/intended output * 3495: Test sql/test/centipede/Tests/olap.sql lacks correct/expected/intended output * 3497: monetdb start reports crash in merovingian.log * 3498: SQL throws TypeException if aggregations and limit statements are both present * 3502: Database was killed by signal SIGBUS * 3504: COPY INTO does not allow OFFSET without specifying amount of records * 3505: expression with <boolean> = NOT <boolean> returns a syntax error but NOT <boolean> = <boolean> not * 3506: conversion to varchar terminates mserver * 3508: conversion of string '0 ' to type smallint or integer fails * 3510: timestamp + month interval generates bogus MAL? * 3511: When having multiple selections combined with aliases not all of them seem to be evalauted. * 3512: auto-conversion of string to `sht` type no longer works * 3513: COPY BINARY INTO fails on 6gb file; works fine on 3gb * 3516: inserting '0' into a column of datatype numeric fails * 3518: UNION with subqueries * 3521: large results of function exp() are not automatically returned as double * 3522: SQL catalog table sys.columns lists columns for table ids which do not exist in sys.tables * 3523: Window function over union gives no result * 3524: wrong error on missing aggregation column * 3527: select distinct - order by - limit 2 results in one single result * 3528: segfault at mal_session.c:521 * 3532: several geom tests crash after manifold changes * 3534: missing table name with invalid column in join using (and problems after resolving it) * 3536: program contains error with join using integer and smallint * 3542: gdk/gdk_bat.c:2904: BATassertHeadProps: Assertion `!b->H->revsorted || cmp >= 0' failed. * 3543: invalid behavior and incorrect data results for SQL data type: numeric(4,4) * 3544: sys.reuse() corrupts data * 3546: Division by zero in CASE statement that should avoid it * 3547: Empty query when selecting a field from a view made of UNION ALL * 3551: Wrong ticks in TRACE * 3552: incorrect data results for "WHERE int_col <> 0" * 3554: Issue with subselect and ORDER BY * 3555: Order of evaluation inside CASE WHEN * 3558: numeric values (as strings) are incorrectly parsed/converted and invalid strings are accepted without error * 3560: Error "BATproject: does not match always" with subselect/groupby/having * 3562: mserver5: gdk_bat.c:2855: BATassertHeadProps: Assertion `!b->H->revsorted || cmp >= 0' failed. * 3563: incorrect results for scalar function locate(in_str, search_str, occurrence) * 3565: Wrong/confusing error message when trying to add a FK to a TEMP TABLE * 3572: Table names with escaped double quotes are rejected * 3573: alter table alter_not_null_test alter test set NOT NULL; is accepted when test contains null. This used to be restricted but isn't anymore * 3575: segmentation fault in mserver5 process * 3576: Dropping default value definitions from a table does not work as expected * 3577: SIGSEGV in BATins_kdiff * 3579: segmentation fault in mserver5 process * 3581: mserver5: rel_bin.c:2504: rel2bin_groupby: Assertion `0' failed. * 3582: mserver5: sql_mem.c:48: sql_ref_dec: Assertion `r->refcnt > 0' failed. * 3583: Possible buffer overflow in max(varchar) * 3585: Incorrect query terminates connection * 3586: mserver5: sql/storage/store.c:3610: sys_drop_func: Assertion `rid_func != oid_nil' failed. * 3592: SIGSEGV in MANIFOLDjob * 3593: delta_append_val: Assertion `!c || ((c)->S->count) == bat->ibase' failed. * 3594: gdk/gdk_bat.c:2855: BATassertHeadProps: Assertion `!b->H->revsorted || cmp >= 0' failed. * 3595: Race/heap corruption on thread exit * 3596: gdk_bat.c:2861: BATassertHeadProps: Assertion `!b->H->nonil || cmp != 0' failed. * 3597: SQL to MAL listing looses types * 3598: SQL bulk load should ignore leading/trailing spaces also with type decimal (as with integers & real/double) * 3599: Double-free of imprints * 3601: Trivial typo in debian/monetdb5-sql.init.d * 3603: "monetdb create -p" hangs monetdbd * 3604: Sys.queue ignored upon errors

4 3

Did not initialise the sql scenario
by Nitul Halder 04 Nov '14

04 Nov '14

I am getting the error (highlighted)when try to start the database demo. I am puzzled . What I can do..Please suggest anyone. Thank you all Nitul 2014-11-04 09:28:26 MSG demo[26510]: arguments: /usr/local/bin/mserver5 --dbpath=/RAID/main_dbfarm/demo --set merovingian_uri=mapi:monetdb://bdserver:50000/demo --set mapi_open=false --set mapi_port=0 --set mapi_usock=/RAID/main_dbfarm/demo/.mapi.sock --set monet_vault_key=/RAID/main_dbfarm/demo/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2014-11-04 09:28:26 MSG demo[26510]: # MonetDB 5 server v11.15.3 "Feb2013-SP1" 2014-11-04 09:28:26 MSG demo[26510]: # Serving database 'demo', using 4 threads 2014-11-04 09:28:26 MSG demo[26510]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2014-11-04 09:28:26 MSG demo[26510]: # Found 3.496 GiB available main-memory. 2014-11-04 09:28:26 MSG demo[26510]: # Copyright (c) 1993-July 2008 CWI. 2014-11-04 09:28:26 MSG demo[26510]: # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved 2014-11-04 09:28:26 MSG demo[26510]: # Visit http://www.monetdb.org/ for further information 2014-11-04 09:28:26 MSG demo[26510]: # Listening for UNIX domain connection requests on mapi:monetdb:///RAID/main_dbfarm/demo/.mapi.sock 2014-11-04 09:28:26 MSG merovingian[26485]: database 'demo' (26510) has exited with exit status 0 2014-11-04 09:28:26 MSG merovingian[26485]: database 'demo' has shut down *2014-11-04 09:28:26 ERR control[26485]: (local): failed to fork mserver: database 'demo' did not initialise the sql scenario*

1 0

Problems with comments in SQL code
by Robert Koch 04 Nov '14

04 Nov '14

SELECT * FROM sys.storage WHERE 1=1 AND "schema" = 'sapsr3' AND "table" = 'dfkkop' ; works in mclient and DBVisualizer (via JDBC) SELECT * FROM sys.storage WHERE 1=1 AND "schema" = 'sapsr3' --AND "table" = 'dfkkop' ; works in mclient, but does not work in DBVisualizer. error message: 11:23:37 [SELECT - 0 row(s), 0.000 secs] 1) [Error Code: 0, SQL State: 42000] syntax error, unexpected $end, expecting SCOLON in: "select. 2) [Error Code: 0, SQL State: 22000] *. 3) [Error Code: 0, SQL State: 22000] from. 4) [Error Code: 0, SQL State: 22000] sys.storage. 5) [Error Code: 0, SQL State: 22000] where 1=1. 6) [Error Code: 0, SQL State: 22000] and "schema" = 'sapsr3'. 7) [Error Code: 0, SQL State: 22000] --AND "table" = 'dfkkop';" ... 1 statement(s) executed, 0 row(s) affected, exec/fetch time: 0.000/0.000 sec [0 successful, 0 warnings, 1 errors] I also noticed this problem (handling comments) in other statements, especially when comments appear before the semicolon. I am not sure, if this is a problem in JDBC driver or DBVisualizer. To see, what is received by monetdb server, I would like to see log files of SQL queries. I found a directory named "/sql_logs/sql", but it consits of 2 files nearly empty. Where can I enable logging?

2 2

Re: MonetDB error while restarting the database
by Ashish Kumar Singh 31 Oct '14

31 Oct '14

Thanks Dimitar, Yes we are restarting the data base and as part of that we are seeing this error. I am more worried about the error below: 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread On java client side it says monetdbd: internal error while starting mserver, please refer to the logs Regards, Ashish -----Original Message----- From: Dimitar Nedev <D.G.Nedev(a)cwi.nl> Reply-To: "users-list(a)monetdb.org" <users-list(a)monetdb.org> Date: Monday, 27 October 2014 8:15 pm To: "users-list(a)monetdb.org" <users-list(a)monetdb.org> Subject: Re: MonetDB error while restarting the database >Hi Ashish, > >The exit timeout only tells monetdbd how long should it wait when trying >to gracefully shut down a mserver process. According to the log, the >mserver process was shut down within the limit, since there are no >messages like "timeout of <exittimeout> seconds expired, sending process ><PID> the KILL signal". > >Now, the interesting entry in the log is the control process message: >2014-10-27 09:01:26 MSG control[23583]: (local): stopped database >'pbsworksdb' > >This one would indicate that someone (or something) gracefully stopped >the database using the monetdbd client. Probably with the following >command line call like: 'monetdb stop pbsworksdb'. >Do not be confused by the order of logged events - monetdbd will first >stop the process and later log that the call to stop the a database has >been successful. > >Best regards, >Dimitar > > >> On 2014-Oct-27, at 14:49 , Ashish Kumar Singh >><ashishkumar.singh(a)altair.com> wrote: >> >> Thanks! >> >> >> Appreciate your response so there are few more people getting the same >>error :). >> Any suggestions from dev team on this issue? Can monetdbd exittimeout >>be also in picture for this? >> >> >> Regards, >> Ashish >> >> >> >> From: Pierre-Adrien Coustillas <pcoustillas(a)1g6.biz> >> Date: Monday, 27 October 2014 6:51 pm >> To: "users-list(a)monetdb.org" <users-list(a)monetdb.org> >> Cc: "developers-list(a)monetdb.org" <developers-list(a)monetdb.org>, Ashish >>Singh <ashishkumar.singh(a)altair.com> >> Subject: Re: MonetDB error while restarting the database >> >>> Hello >>> >>> (google translate) >>> >>> I receive this error when there is no specific spacedisk. >>> Monetdb sometimes uses a lot of temporary diskspace (500GBisnormalfor >>>1TBof data), which arereleasedat the end oftreatment or after acrash >>> >>> Error last week : >>> 2014-10-22 22:51:46 MSG merovingian[18865]: sending process 30608 >>>(database 'lemonde') the TERM signal >>> 2014-10-22 22:51:46 ERR merovingian[18865]: unknown state: 42014-10-22 >>>22:51:46 ERR lemonde[30608]: #GDKexit: killing thread >>> >>> Pierre >>> >>> -- >>> 1G6 >>> 52 route de bischwiller >>> 67300 Schiltigheim >>> Société de Services et de Formations en Logiciels Libres >>> http://1g6.biz >>> Tél : 06 64 63 70 35 >>> >>> De: "Ashish Kumar Singh" <ashishkumar.singh(a)altair.com> >>> À: "Ashish Kumar Singh" <ashishkumar.singh(a)altair.com>, >>>users-list(a)monetdb.org >>> Cc: developers-list(a)monetdb.org >>> Envoyé: Lundi 27 Octobre 2014 13:49:19 >>> Objet: Re: MonetDB error while restarting the database >>> >>> Guys, >>> >>> Any help with this will be really helpful for us? >>> >>> >>> >>> Regards, >>> Ashish >>> >>> From: Ashish Singh <ashishkumar.singh(a)altair.com> >>> Date: Monday, 27 October 2014 4:06 pm >>> To: "users-list(a)monetdb.org" <users-list(a)monetdb.org> >>> Subject: MonetDB error while restarting the database >>> >>>> >>>> All, >>>> >>>> >>>> >>>> We are facing a new issue with monetdb where monetdb connections and >>>>process is being killed very frequently. Is this a known issue or any >>>>suggestion in getting more details will be helpful. >>>> >>>> >>>> >>>> Regards, >>>> Ashish >>>> >>>> Mervogian.log: >>>> >>>> >>>> >>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: arguments: >>>>/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5 >>>>--dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbswor >>>>ksdata/pbsworksdb --set >>>>merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set >>>>mapi_open=false --set mapi_port=0 --set >>>>mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsw >>>>orksdata/pbsworksdb/.mapi.sock --set >>>>monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb >>>>/pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set >>>>max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes >>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB 5 server >>>>v11.17.17 "Jan2014-SP2" >>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Serving database >>>>'pbsworksdb', using 4 threads >>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Compiled for >>>>x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked >>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Found 15.675 GiB >>>>available main-memory. >>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) 1993-July >>>>2008 CWI. >>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) August >>>>2008-2014 MonetDB B.V., all rights reserved >>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Visit >>>>http://www.monetdb.org/ for further information >>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Listening for UNIX domain >>>>connection requests on >>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ >>>>pbsworksdata/pbsworksdb/.mapi.sock >>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB/SQL module loaded >>>> 2014-10-27 09:01:00 MSG control[23583]: (local): started 'pbsworksdb' >>>> 2014-10-27 09:01:00 MSG merovingian[23583]: proxying client >>>>localhost.localdomain:53844 for database 'pbsworksdb' to >>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ >>>>pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb >>>> 2014-10-27 09:01:00 MSG merovingian[23583]: target connection is on >>>>local UNIX domain socket, passing on filedescriptor instead of proxying >>>> 2014-10-27 09:01:26 MSG control[23583]: (local): served status list >>>> 2014-10-27 09:01:26 MSG merovingian[23583]: sending process 6351 >>>>(database 'pbsworksdb') the TERM signal >>>> 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: >>>>42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread >>>> 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' >>>>(6351) has exited with exit status 0 >>>> 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' has >>>>shut down >>>> 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database >>>>'pbsworksdb' >>>> 2014-10-27 09:01:26 MSG control[23583]: (local): served status list >>>> 2014-10-27 09:01:26 MSG merovingian[23583]: starting database >>>>'pbsworksdb', up min/avg/max: 5s/14m/1h, crash average: 0.00 0.00 0.00 >>>>(648-648=0) >>>> 2014-10-27 09:01:27 MSG pbsworksdb[6575]: arguments: >>>>/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5 >>>>--dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbswor >>>>ksdata/pbsworksdb --set >>>>merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set >>>>mapi_open=false --set mapi_port=0 --set >>>>mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsw >>>>orksdata/pbsworksdb/.mapi.sock --set >>>>monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb >>>>/pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set >>>>max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes >>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB 5 server >>>>v11.17.17 "Jan2014-SP2" >>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Serving database >>>>'pbsworksdb', using 4 threads >>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Compiled for >>>>x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked >>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Found 15.675 GiB >>>>available main-memory. >>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) 1993-July >>>>2008 CWI. >>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) August >>>>2008-2014 MonetDB B.V., all rights reserved >>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Visit >>>>http://www.monetdb.org/ for further information >>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Listening for UNIX domain >>>>connection requests on >>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ >>>>pbsworksdata/pbsworksdb/.mapi.sock >>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB/SQL module loaded >>>> 2014-10-27 09:01:32 MSG merovingian[23583]: proxying client >>>>localhost.localdomain:53853 for database 'pbsworksdb' to >>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ >>>>pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb >>>> 2014-10-27 09:01:32 MSG merovingian[23583]: target connection is on >>>>local UNIX domain socket, passing on filedescriptor instead of proxying >>>> 2014-10-27 09:01:32 MSG c >>>> >>>> >>>> >>>> >>>> >>> >>> _______________________________________________ >>> users-list mailing list >>> users-list(a)monetdb.org >>> https://www.monetdb.org/mailman/listinfo/users-list >>> >> _______________________________________________ >> developers-list mailing list >> developers-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/developers-list > >_______________________________________________ >users-list mailing list >users-list(a)monetdb.org >https://www.monetdb.org/mailman/listinfo/users-list

3 4

Re: MonetDB error while restarting the database
by Ashish Kumar Singh 27 Oct '14

27 Oct '14

Guys, Any help with this will be really helpful for us? Regards, Ashish From: Ashish Singh <ashishkumar.singh(a)altair.com<mailto:ashishkumar.singh@altair.com>> Date: Monday, 27 October 2014 4:06 pm To: "users-list(a)monetdb.org<mailto:users-list@monetdb.org>" <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Subject: MonetDB error while restarting the database All, We are facing a new issue with monetdb where monetdb connections and process is being killed very frequently. Is this a known issue or any suggestion in getting more details will be helpful. Regards, Ashish Mervogian.log: 2014-10-27 09:01:00 MSG pbsworksdb[6351]: arguments: /opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5 --dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb --set merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock --set monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB 5 server v11.17.17 "Jan2014-SP2" 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Serving database 'pbsworksdb', using 4 threads 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Found 15.675 GiB available main-memory. 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) 1993-July 2008 CWI. 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Visit http://www.monetdb.org/ for further information 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB/SQL module loaded 2014-10-27 09:01:00 MSG control[23583]: (local): started 'pbsworksdb' 2014-10-27 09:01:00 MSG merovingian[23583]: proxying client localhost.localdomain:53844 for database 'pbsworksdb' to mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb 2014-10-27 09:01:00 MSG merovingian[23583]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2014-10-27 09:01:26 MSG control[23583]: (local): served status list 2014-10-27 09:01:26 MSG merovingian[23583]: sending process 6351 (database 'pbsworksdb') the TERM signal 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' (6351) has exited with exit status 0 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' has shut down 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database 'pbsworksdb' 2014-10-27 09:01:26 MSG control[23583]: (local): served status list 2014-10-27 09:01:26 MSG merovingian[23583]: starting database 'pbsworksdb', up min/avg/max: 5s/14m/1h, crash average: 0.00 0.00 0.00 (648-648=0) 2014-10-27 09:01:27 MSG pbsworksdb[6575]: arguments: /opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5 --dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb --set merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock --set monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB 5 server v11.17.17 "Jan2014-SP2" 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Serving database 'pbsworksdb', using 4 threads 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Found 15.675 GiB available main-memory. 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) 1993-July 2008 CWI. 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Visit http://www.monetdb.org/ for further information 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB/SQL module loaded 2014-10-27 09:01:32 MSG merovingian[23583]: proxying client localhost.localdomain:53853 for database 'pbsworksdb' to mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb 2014-10-27 09:01:32 MSG merovingian[23583]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2014-10-27 09:01:32 MSG c

2 2

Re: MonetDB: default - Merge with Oct2014 branch.
by Lefteris 27 Oct '14

27 Oct '14

tab space = 8 for gdk? On Mon, Oct 27, 2014 at 11:02 AM, Sjoerd Mullender <commits(a)monetdb.org> wrote: > Changeset: dfe9b28b23db for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=dfe9b28b23db > Modified Files: > gdk/gdk_imprints.c > gdk/gdk_select.c > Branch: default > Log Message: > > Merge with Oct2014 branch. > > > diffs (108 lines): > > diff --git a/gdk/gdk_imprints.c b/gdk/gdk_imprints.c > --- a/gdk/gdk_imprints.c > +++ b/gdk/gdk_imprints.c > @@ -453,12 +453,12 @@ do { > \ > } \ > GETBIN##B(bin,col[i]); \ > mask = IMPSsetBit(B,mask,bin); \ > - if (col[i] != nil) { /* do not count nils */ > \ > - if (!cnt_bins[bin]++) { > \ > - min_bins[bin] = max_bins[bin] = i; > \ > - } else { > \ > - if (col[i] < col[min_bins[bin]]) > min_bins[bin] = i; \ > - if (col[i] > col[max_bins[bin]]) > max_bins[bin] = i; \ > + if (col[i] != nil) { /* do not count nils */ \ > + if (!cnt_bins[bin]++) { \ > + min_bins[bin] = max_bins[bin] = i; \ > + } else { \ > + if (col[i] < col[min_bins[bin]]) > min_bins[bin] = i; \ > + if (col[i] > col[max_bins[bin]]) > max_bins[bin] = i; \ > } \ > } \ > } \ > diff --git a/gdk/gdk_select.c b/gdk/gdk_select.c > --- a/gdk/gdk_select.c > +++ b/gdk/gdk_select.c > @@ -341,27 +341,27 @@ do { > \ > } \ > } while (0) > > -#define checkMINMAX(B) > \ > -do { > \ > - int ii; > \ > - BUN *imp_cnt = (BUN *) (((oid *)imprints->stats)+128); > \ > - imp_min = imp_max = nil; > \ > - for (ii = 0; ii < B; ii++) { > \ > - if ((imp_min == nil) && (imp_cnt[ii])) { > \ > - imp_min = src[((oid *)imprints->stats)[ii]]; > \ > - } > \ > - if ((imp_max == nil) && (imp_cnt[B-1-ii])) { > \ > - imp_max = src[(((oid > *)imprints->stats)+64)[B-1-ii]]; \ > - } > \ > - } > \ > - assert((imp_min != nil) && (imp_max != nil)); > \ > - if (!s && !VIEWtparent(b)) { /* no candidate list and no views*/ > \ > - if ((vl > imp_max || vh < imp_min) || > \ > - (anti && (vl < imp_min && vh > imp_max))) { > \ > - return 0; > \ > - } > \ > - } > \ > -} while (0) > \ > +#define checkMINMAX(B) \ > +do { \ > + int ii; \ > + BUN *imp_cnt = (BUN *) (((oid *)imprints->stats)+128); \ > + imp_min = imp_max = nil; \ > + for (ii = 0; ii < B; ii++) { \ > + if ((imp_min == nil) && (imp_cnt[ii])) { \ > + imp_min = src[((oid *)imprints->stats)[ii]]; \ > + } \ > + if ((imp_max == nil) && (imp_cnt[B-1-ii])) { \ > + imp_max = src[(((oid > *)imprints->stats)+64)[B-1-ii]]; \ > + } \ > + } \ > + assert((imp_min != nil) && (imp_max != nil)); \ > + if (!s && !VIEWtparent(b)) { /* no candidate list and no views*/ \ > + if ((vl > imp_max || vh < imp_min) || \ > + (anti && (vl < imp_min && vh > imp_max))) { \ > + return 0; \ > + } \ > + } \ > +} while (0) > > /* choose number of bits */ > #define bitswitch(CAND,TEST) \ > @@ -374,10 +374,10 @@ do { > \ > s && BATtdense(s) ? "(dense)" : "", \ > anti, #TEST); \ > switch (imprints->bits) { \ > - case 8: checkMINMAX(8); impsmask(CAND,TEST,8); break; > \ > - case 16: checkMINMAX(16); impsmask(CAND,TEST,16); break; > \ > - case 32: checkMINMAX(32); impsmask(CAND,TEST,32); break; > \ > - case 64: checkMINMAX(64); impsmask(CAND,TEST,64); break; > \ > + case 8: checkMINMAX(8); impsmask(CAND,TEST,8); break; \ > + case 16: checkMINMAX(16); impsmask(CAND,TEST,16); break; \ > + case 32: checkMINMAX(32); impsmask(CAND,TEST,32); break; \ > + case 64: checkMINMAX(64); impsmask(CAND,TEST,64); break; \ > default: assert(0); break; \ > } \ > } while (0) > @@ -484,13 +484,13 @@ NAME##_##TYPE (BAT *b, BAT *s, BAT *bn, > { \ > TYPE vl = *(const TYPE *) tl; \ > TYPE vh = *(const TYPE *) th; \ > - TYPE imp_min; > \ > - TYPE imp_max; > \ > - TYPE v; > \ > - TYPE nil = TYPE##_nil; > \ > + TYPE imp_min; \ > + TYPE imp_max; \ > + TYPE v; \ > + TYPE nil = TYPE##_nil; \ > TYPE minval = MINVALUE##TYPE; \ > TYPE maxval = MAXVALUE##TYPE; \ > - const TYPE *src = (const TYPE *) Tloc(b, 0); \ > + const TYPE *src = (const TYPE *) Tloc(b, 0); \ > oid o; \ > BUN w, p = r; \ > BUN pr_off = 0; \ > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

2 1

useless WHERE conditions
by gms_xy＠gmx.de 17 Oct '14

17 Oct '14

To have a human-readable SQL code, I am used in following code formatting: SELECT a FROM tbl WHERE 1=1 --<<-- absolutely useless, but nice for formatting AND col1 = 1 AND col2 = 2 AND col3 = 3 Instead of "1=1", some other databases accept "TRUE". When I have multiple OR-conditions, I start with "1=0" or "FALSE". Writing statements this way I noticed long execution times in MonetDB. Omitting the logical where-condition "1=1" results in very fast execution time. So I guess, MonetDB is really checking on every row, if 1=1.

2 1

Partitioning
by Robert Koch 17 Oct '14

17 Oct '14

Are there plans to implement a (manual) SQL-based (multilevel) partitioning? I saw great accelerations with partitions in other databases. Is there any internal/automatic partitioning? If so, can I check it?

2 1

Abort multiline input in mclient
by Robert Koch 17 Oct '14

17 Oct '14

Is there any way to abort multiline input in mclient without leaving mclient? If I press Ctrl-C, mclient exits.

2 1

CASE/WHEN syntax in UPDATE/SET clauses
by Robert Koch 17 Oct '14

17 Oct '14

CASE/WHEN syntax would be useful for UPDATE/SET clauses. Example: UPDATE tbl1 SET col1 = CASE WHEN ... THEN ... ELSE ... END ; I am not sure if this is conform to ANSI SQL. I would find it useful. Today I have to write multiple UPDATE queries with different WHERE conditions.

1 0

Large intermediate results dureing columns update
by Robert Koch 17 Oct '14

17 Oct '14

Given a column defined as bigint (int64) with 1 billion rows. Storage size of this column is 8 GB. Values range in space of int (int32). UPDATE sapsr3.dfkkop SET vkont_int32 = vkont_int64 ; This query takes very long (minutes). During execution the data directory grows by 40 GB (!). I would expect the data cells do be converted an written into the new column directly. Even if MonetDB would materialize the whole source column as "intermediate result", I would expect not to exceed 8 GB for "intermediates" and 4 GB for the target column.

1 0

Internal type conversion
by Robert Koch 17 Oct '14

17 Oct '14

SELECT count(*) FROM sapsr3.dfkkop WHERE vkont >= '000802037649' and vkont < '000802038649' ; +-------+ | L1 | +=======+ | 81196 | +-------+ 1 tuple (1.8s) SELECT count(*) FROM sapsr3.dfkkop WHERE vkont >= 000802037649 and vkont < '000802038649' ; +-------+ | L1 | +=======+ | 81196 | +-------+ 1 tuple (1.5s) SELECT count(*) FROM sapsr3.dfkkop WHERE vkont >= 000802037649 and vkont < 000802038649 ; conversion of string 'VI0802037717' to type int failed. Column "vkont" is defined as varchar(12), values are going from '000218123761' to 'ZZ0876213465'. 2nd statement is obviously doing some internal type conversion. 3rd statement fails. If there is done any kind of internal type conversion, I would expect that the outcome of this opereration must not depend on column's values.

1 0

Problems with GLOBAL TEMPORARY TABLES (GTT)
by Robert Koch 17 Oct '14

17 Oct '14

GLOBAL TEMPORARY TABLES do not work consistently. I am not sure, if all or some of the following problems described are intended. (is it bug or feature?) Out of 2 different sessions I am able to create GTTs with same name. Out of a third session I am able to see both tables (with same name), but not beeing able to access one of these tables. I am able to delete both tables, but I am not sure, which one is deleted first. Restarting MonetDB results in a situation where temp tables are still existent, but it is not possible to drop these tables out of any session. Is it a matter of privileges?

1 0

cost-based execution plan - incorporating cardinality?
by Robert Koch 17 Oct '14

17 Oct '14

SELECT opbel FROM sapsr3.dfkkop WHERE --mandt='200' and vkont='000802037649' ; runs in 5 ms. SELECT opbel FROM sapsr3.dfkkop WHERE mandt='200' and vkont='000802037649' ; runs in 15 s. (Factor 3.000) Column "mandt" has a cardinality of 2/1.000.000.000 (2 distinct values at 1 billion rows). Column vkont has a cardinality of 12.960.000/1.000.000.000. So I would expect to do the lookup on column vkont. Output of EXPLAIN: EXPLAIN SELECT opbel FROM sapsr3.dfkkop WHERE --mandt='200' and vkont='000802037649' ; mal ----------------------------------------------------------------------------------------------------------------------------------------------- function user.s11_3{autoCommit=true}(A0:str):void; X_3 := sql.mvc(); X_7 := sql.bind(X_3,"sapsr3","dfkkop","vkont",0); X_4:bat[:oid,:oid] := sql.tid(X_3,"sapsr3","dfkkop"); X_45 := algebra.subselect(X_7,X_4,A0,A0,true,true,false); (X_10,r1_10) := sql.bind(X_3,"sapsr3","dfkkop","vkont",2); X_46 := algebra.subselect(r1_10,A0,A0,true,true,false); X_13 := sql.bind(X_3,"sapsr3","dfkkop","vkont",1); X_47 := algebra.subselect(X_13,X_4,A0,A0,true,true,false); X_15 := sql.subdelta(X_45,X_4,X_10,X_46,X_47); X_17 := sql.bind(X_3,"sapsr3","dfkkop","opbel",0); (X_19,r1_22) := sql.bind(X_3,"sapsr3","dfkkop","opbel",2); X_21 := sql.bind(X_3,"sapsr3","dfkkop","opbel",1); X_22 := sql.projectdelta(X_15,X_17,X_19,r1_22,X_21); X_23 := sql.resultSet(1,1,X_22); sql.rsColumn(X_23,"sapsr3.dfkkop","opbel","varchar",12,0,X_22); X_28 := io.stdout(); sql.exportResult(X_28,X_23); end s11_3; # querylog.define("select\n opbel\nfrom\n sapsr3.dfkkop\nwhere\n --mandt=\\'200\\' and\n vkont=\\'000802037649\\';","default_pipe") # optimizer.mitosis() # optimizer.dataflow() EXPLAIN SELECT opbel FROM sapsr3.dfkkop WHERE mandt='200' and vkont='000802037649' ; mal ------------------------------------------------------------------------------------------------------------------------------------------------------ function user.s12_3{autoCommit=true}(A0:str,A1:str):void; X_4 := sql.mvc(); X_8 := sql.bind(X_4,"sapsr3","dfkkop","vkont",0); X_16 := sql.bind(X_4,"sapsr3","dfkkop","mandt",0); X_5:bat[:oid,:oid] := sql.tid(X_4,"sapsr3","dfkkop"); X_54 := algebra.subselect(X_16,X_5,A0,A0,true,true,false); (X_18,r1_21) := sql.bind(X_4,"sapsr3","dfkkop","mandt",2); X_55 := algebra.subselect(r1_21,A0,A0,true,true,false); X_20 := sql.bind(X_4,"sapsr3","dfkkop","mandt",1); X_56 := algebra.subselect(X_20,X_5,A0,A0,true,true,false); X_21 := sql.subdelta(X_54,X_5,X_18,X_55,X_56); X_57 := algebra.subselect(X_8,X_21,A1,A1,true,true,false); (X_11,r1_11) := sql.bind(X_4,"sapsr3","dfkkop","vkont",2); X_58 := algebra.subselect(r1_11,A1,A1,true,true,false); X_14 := sql.bind(X_4,"sapsr3","dfkkop","vkont",1); X_59 := algebra.subselect(X_14,X_21,A1,A1,true,true,false); X_23 := sql.subdelta(X_57,X_21,X_11,X_58,X_59); X_24 := sql.bind(X_4,"sapsr3","dfkkop","opbel",0); (X_26,r1_32) := sql.bind(X_4,"sapsr3","dfkkop","opbel",2); X_28 := sql.bind(X_4,"sapsr3","dfkkop","opbel",1); X_29 := sql.projectdelta(X_23,X_24,X_26,r1_32,X_28); X_30 := sql.resultSet(1,1,X_29); sql.rsColumn(X_30,"sapsr3.dfkkop","opbel","varchar",12,0,X_29); X_36 := io.stdout(); sql.exportResult(X_36,X_30); end s12_3; # querylog.define("explain\nselect\n opbel\nfrom\n sapsr3.dfkkop\nwhere\n mandt=\\'200\\' and\n vkont=\\'000802037649\\';","default_pipe") # optimizer.mitosis() # optimizer.dataflow()

1 0

bad performance of LIMIT/OFFSET
by Robert Koch 17 Oct '14

17 Oct '14

SELECT col1 FROM tbl WHERE col1 = 1 ; is executed in 5 ms. Table has 1 billion rows, resultset is about 350 rows. Same statement, but with LIMIT/OFFSET SELECT col1 FROM tbl WHERE col1 = 1 LIMIT 1 OFFSET 0 ; is executed in 1,03 s. Can't believe that subsetting from 350 to 1 takes 1 second.

2 1

Introducing testing scenario on big data
by Robert Koch 17 Oct '14

17 Oct '14

Hello list (and people behind), you are doing a great job on MonetDB and I have great respect to your work. During the last time I made some testing with MonetDB. I like the concept of memory mapped files, since files are "in-memory", even if MonetDB is restarted. Please understand my following mails to this list as contribution to your project. I am not very familiar to the BAT algebra or MAL language, moreover looking for an open-source alternative to commercial in-memory databases, able to run on commodity hardware, not beeing bounded to a specific operating system. For my use cases it is important to have a rich implementation of SQL and - even more important - to have a good SQL optimizer, since I have complex SQL statements with some joins over huge tables. From point of my observations, an improved optimizer would be the key for a wider spread of MonetDBs user community. On simple statements I see a performance comparable to commercial vedors/products. If my scenarios are interesting for you (developers), please feel free to contact me to discuss internals or to look inside the servers behavior yourself. I am working on 6 tables, each up to 1 billion rows and between 4 an 12 columns. Server with 300 GB RAM, fast SSDs on RAID 10 and 2 CPUs with 6 cores each. Testing was done both in DBVisualizer and in mclient (to avoid JDBC issues and fetch times). Running MonetDB v11.17.21 (Jan2014-SP3) on a fresh and clean Debian Wheezy. I will go on to describe my dev-related observations in separate mails in order to enable a separate discussion an each issue. Robert

1 0

msqldump - option for exporting all tables of a given schema
by gms_xy＠gmx.de 09 Oct '14

09 Oct '14

Is there an option for exporting all tables of a given schema in msqldump? If not -> request for feature!

1 0

mserver5 fails in running MAL plan
by Qing Guo 09 Oct '14

09 Oct '14

Hi, I run mserver5 in standalone mode. The SQL input works fine but the corresponding MAL input fails. See below for the details. - MonetDB version 11.17.21 - Command line mserver5 --dbpath=${DBPATH} --set mapi_open=false --set mapi_port=0 --set single-user=yes --set gdk_nr_threads=1--set max_clients=1 --set sql_optimizer=default_pipe --set monet_daemon=no < ${INPUT} - Database TPC-H (scale factor: 1) - The SQL input: sql.start(); select count(*) from supplier, nation, region where r_name = 'EUROPE' and r_regionkey = n_regionkey and s_nationkey = n_nationkey; - The MAL input: (I use "explain" to get the MAL plan) sql.init(); function user.s0_0{autoCommit=true}(A0:str):void; X_3 := sql.mvc(); X_4:bat[:oid,:oid] := sql.tid(X_3,"sys","nation"); X_7 := sql.bind_idxbat(X_3,"sys","nation","nation_n_regionkey_fkey",0); (X_10,r1_10) := sql.bind_idxbat(X_3,"sys","nation","nation_n_regionkey_fkey",2); X_13 := sql.bind_idxbat(X_3,"sys","nation","nation_n_regionkey_fkey",1); X_15 := sql.projectdelta(X_4,X_7,X_10,r1_10,X_13); X_18 := sql.bind(X_3,"sys","region","r_name",0); X_16:bat[:oid,:oid] := sql.tid(X_3,"sys","region"); X_72 := algebra.subselect(X_18,X_16,A0,A0,true,true,false); (X_20,r1_21) := sql.bind(X_3,"sys","region","r_name",2); X_73 := algebra.subselect(r1_21,A0,A0,true,true,false); X_22 := sql.bind(X_3,"sys","region","r_name",1); X_74 := algebra.subselect(X_22,X_16,A0,A0,true,true,false); X_23 := sql.subdelta(X_72,X_16,X_20,X_73,X_74); X_25 := X_23; (X_26,r1_30) := algebra.join(X_15,X_25); X_28 := algebra.leftfetchjoin(X_26,X_4); X_29:bat[:oid,:oid] := sql.tid(X_3,"sys","supplier"); X_31 := sql.bind_idxbat(X_3,"sys","supplier","supplier_s_nationkey_fkey",0); (X_33,r1_37) := sql.bind_idxbat(X_3,"sys","supplier","supplier_s_nationkey_fkey",2); X_36 := sql.bind_idxbat(X_3,"sys","supplier","supplier_s_nationkey_fkey",1); X_37 := sql.projectdelta(X_29,X_31,X_33,r1_37,X_36); (X_38,r1_43) := algebra.join(X_28,X_37); X_40 := algebra.leftfetchjoin(X_38,X_28); X_41 := aggr.count(X_40); sql.exportValue(1,".L1","L1","wrd",64,0,6,X_41,""); end s0_0; user.s0_0("EUROPE"); mserver5 reports the following error message when running the MAL input: !TypeException:user.s0_0[23]:'algebra.join' undefined in: (X_38:any, r1_43:any) := algebra.join(X_28:bat[:oid,:oid], X_37:bat[:oid,:any]) It seems to me that X_28 is not in the right format. But I have no idea how it could happen and how to make it correct. Any thoughts on this will be much appreciated. Thanks! -Qing

1 0

Left/Right joins performance - Need help!!
by Vijay Krishna 08 Oct '14

08 Oct '14

Hi all, I have been comparing MySQL with MonetDB. Obviously, queries that took minutes in MySQL got executed in a matter of few seconds in Monet. However, I found a real blockade with joins. I have 2 tables - each with 150 columns. Among these (150+150) columns, around 60 are CHARACTER LARGE OBJECT type. Both the tables are populated with around 50,000 rows - with data in all the 150 columns. The average length of data in a CLOB type column is 9,000 (varying from 2 characters to 20,000 characters). The primary key of both the tables have same values and the join is always based on the primary key. The rows are by default inserted in ascending order on the primary key. When I ran *an inner join query on these two tables with about 5 criteria and with limit 1000, Monet processed this query in 5 seconds*, which is completely impressive compared to MySQL's (19 seconds). But when I ran *the same query with same criteria and limits using a left or right joins, Monet took around 5 minutes, which is clearly way behind MySQL's (just 22 seconds)*. Having read a lot about Monet's blinding speed compared to traditional relation row based DBs, I could feel that I am missing something, but couldn't figure out what. Can anyone please tell me why there is such a huge difference in such query execution time and how I can prevent it? Much grateful to have any help. Thanks a lot in advance. P.S.: I am running Monet on Macbook Pro - 2.3 GHz Core i7 - Quad core processor with 8 GB RAM. With Regards, Vijayakrishna.P. Mobile : +91-9500402305.

1 0

Re: MonetDB: default - Disable -pthread for Clang 6.*
by Sjoerd Mullender 19 Sep '14

19 Sep '14

Shouldn't this go to the Oct2014 branch? On 19/09/14 11:45, Dimitar Nedev wrote: > Changeset: 10ae6a3308bf for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=10ae6a3308bf > Modified Files: > configure.ag > Branch: default > Log Message: > > Disable -pthread for Clang 6.* > This is in line OS X Yosemite and more specifically Xcode 6.0.1 shipping with Clang verion 6 > > > diffs (19 lines): > > diff --git a/configure.ag b/configure.ag > --- a/configure.ag > +++ b/configure.ag > @@ -1885,11 +1885,11 @@ if test "x$have_pthread" != xno; then > save_LIBS="$LIBS" > save_CPPFLAGS="$CPPFLAGS" > case $GCC-$have_pthread-$CC_ver in > - yes-auto-clang-5.*|yes-yes-clang-5.*) > - # clang 5.* (at least on Mac OSX Mavericks) does not > + yes-auto-clang-5.*|yes-yes-clang-5.*|yes-auto-clang-6.*|yes-yes-clang-6.*) > + # clang 5.*/6.* (Xcode 6.0) does not > # seem to have / require -pthread as compiler > - # option; on Mac OSX Mavericks, "Apple LLVM version > - # 5.0 (clang-500.2.79) (based on LLVM 3.3svn)" > + # option; on Mac OS X Yosamite, "Apple LLVM > + # version 6.0 (clang-600.0.51) (based on LLVM 3.5svn)" > # complains about "error: argument unused during > # compilation: '-pthread'". > # However, disabling -pthread completely might be > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 2

How varchar/text column works ?
by Mike 16 Sep '14

16 Sep '14

Hi, i have been working on monetdb for quite some time and am trying to understand how string column works so as i can (try to ) make it import string columns as fast as possible.The tool is external and generates monetdb binary compatible columns( and may be used to optimize tables in offline mode). As far as reading code and understanding it. here is my part String columns is kept in 2 parts, x.tail and x.theap ( where x is number ) x.tail contains pointers into x.theap x.theap has 2 regions/parts first part contains a 8KB hash table, to detect collisions and second part contains 8 zeroes,8 Byte hash, variable length String which is null terminated and aligned on 8 byte boundary (i am talking about x64 bit arch) and next string follows and so on However, the problem i am facing is , the tools works and generates columns till x.tail contains pointers addressing x.theap positions less than 255. The moment pointers are 2 byte tail entry, monetdb is confused. The reason is tail pointers are varsized for efficiency, and if there are few small string, pointers will fit in small memory and not waste 8 bytes per entry. However, i am unable to understand how monetdb figures out, how big entry is currently being used ? in case of small strings, it's 1 byte (tested) in case of some bigger string, it's 2 bytes (tested) and even in large string tables, it 4 bytes (tested) and large (>4 GB) sting tables, it can be 8 bytes ??? ( this case has not been tested) i have understood, it has something to with bat->T->shift , bat->T->width but still it doesn't work.So, i am definitely missing something. So, my questions are 1) is it an efficient method ( not talking about space efficiency) to have var-sized tail pointers. because it might involve lots of casting and so on during calculations.. 2) how to disable var-sized tail pointers (if it's possible) so as all pointers are 4byte or 8 byte long ( any compile time option or runtime option ) 3) if 2) not possible , then how does monetdb calculate tail pointer width. for example, i am using following code, to initialize new bat with my tail and theap pointers cap=num_of_tail_entries; bs = BATcreatedesc(TYPE_void, TYPE_str, 1); bn = &bs->B; BATsetdims(bn); BATkey(bn, TRUE); BATsetcapacity(bn, cap); BATsetcount(bn, cap); bn->tsorted = 0; bn->trevsorted = 0; bn->tdense = 0; bn->tkey = 0; bn->tvarsized=1; // try to set shift to 1 and thus width to 2 bytes //bn->T->shift=1; // this property seems to be used to calculate tail ptr size //bn->T->width=2; // this property seems to be used to calculate tail ptr size bn->T->varsized=1; then i load both heap and theap. The above code works and i am able to query the table as long as tail pointers are 1 byte long. However, as soon as pointers size is 2 bytes,it fails, so i must set something in BAT init to tell tail ptr size. Any clues are welcome... ( if more information is required, please ask.) Thanks for any help . Rgds mike

1 1

Re: MonetDB: Oct2014 - split update bat within the catalog interface...
by Sjoerd Mullender 09 Sep '14

09 Sep '14

This causes an assertion failed during upgrade from a Jan2014 database. On 2014-09-09 08:24, Niels Nes wrote: > Changeset: 1c3d356ffdc2 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=1c3d356ffdc2 > Modified Files: > sql/backends/monet5/rel_bin.c > sql/backends/monet5/sql.c > sql/backends/monet5/sql_gencode.c > sql/backends/monet5/sql_optimizer.c > sql/backends/monet5/sql_statistics.c > sql/include/sql_catalog.h > sql/storage/bat/bat_storage.c > Branch: Oct2014 > Log Message: > > split update bat within the catalog interface to remove dependency on > BATselect. > > > diffs (truncated from 395 to 300 lines): > > diff --git a/sql/backends/monet5/rel_bin.c b/sql/backends/monet5/rel_bin.c > --- a/sql/backends/monet5/rel_bin.c > +++ b/sql/backends/monet5/rel_bin.c > @@ -733,7 +733,7 @@ stmt_col( mvc *sql, sql_column *c, stmt > (c->base.flag != TR_NEW || c->t->base.flag != TR_NEW /* alter */) && > (c->t->persistence == SQL_PERSIST || c->t->persistence == SQL_DECLARED_TABLE) && !c->t->commit_action) { > stmt *i = stmt_bat(sql->sa, c, RD_INS); > - stmt *u = stmt_bat(sql->sa, c, RD_UPD); > + stmt *u = stmt_bat(sql->sa, c, RD_UPD_ID); > sc = stmt_project_delta(sql->sa, sc, u, i); > sc = stmt_project(sql->sa, del, sc); > } else if (del) { /* always handle the deletes */ > @@ -751,7 +751,7 @@ stmt_idx( mvc *sql, sql_idx *i, stmt *de > (i->base.flag != TR_NEW || i->t->base.flag != TR_NEW /* alter */) && > (i->t->persistence == SQL_PERSIST || i->t->persistence == SQL_DECLARED_TABLE) && !i->t->commit_action) { > stmt *ic = stmt_idxbat(sql->sa, i, RD_INS); > - stmt *u = stmt_idxbat(sql->sa, i, RD_UPD); > + stmt *u = stmt_idxbat(sql->sa, i, RD_UPD_ID); > sc = stmt_project_delta(sql->sa, sc, u, ic); > sc = stmt_project(sql->sa, del, sc); > } else if (del) { /* always handle the deletes */ > diff --git a/sql/backends/monet5/sql.c b/sql/backends/monet5/sql.c > --- a/sql/backends/monet5/sql.c > +++ b/sql/backends/monet5/sql.c > @@ -489,7 +489,7 @@ table_has_updates(sql_trans *tr, sql_tab > > for ( n = t->columns.set->h; !cnt && n; n = n->next) { > sql_column *c = n->data; > - BAT *b = store_funcs.bind_col(tr, c, RD_UPD); > + BAT *b = store_funcs.bind_col(tr, c, RD_UPD_ID); > cnt |= BATcount(b) > 0; > BBPunfix(b->batCacheid); > } > @@ -1644,6 +1644,7 @@ mvc_bind_wrap(Client cntxt, MalBlkPtr mb > bn = BATslice(b, part_nr * psz, (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz)); > BATseqbase(bn, part_nr * psz); > } else { > + /* BAT b holds the UPD_ID bat */ > oid l, h; > BAT *c = mvc_bind(m, *sname, *tname, *cname, 0); > cnt = BATcount(c); > @@ -1651,23 +1652,39 @@ mvc_bind_wrap(Client cntxt, MalBlkPtr mb > l = part_nr * psz; > h = (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz); > h--; > - bn = BATmirror(BATselect(BATmirror(b), &l, &h)); > + bn = BATsubselect(b, NULL, &l, &h, 1, 0, 0); > BBPreleaseref(c->batCacheid); > } > BBPreleaseref(b->batCacheid); > b = bn; > + } else if (upd) { > + BAT *uv = mvc_bind(m, *sname, *tname, *cname, RD_UPD_VAL); > + int *uvl = (int *) getArgReference(stk, pci, 1); > + > + BBPkeepref(*bid = b->batCacheid); > + BBPkeepref(*uvl = uv->batCacheid); > + return MAL_SUCCEED; > } > if (upd) { > int *uvl = (int *) getArgReference(stk, pci, 1); > > if (BATcount(b)) { > - BAT *id = BATmirror(BATmark(b, 0)); > - BAT *vl = BATmirror(BATmark(BATmirror(b), 0)); > + BAT *uv = mvc_bind(m, *sname, *tname, *cname, RD_UPD_VAL); > + BAT *ui = mvc_bind(m, *sname, *tname, *cname, RD_UPD_ID); > + BAT *id = BATproject(b, ui); > + BAT *vl = BATproject(b, uv); > + bat_destroy(b); > + bat_destroy(ui); > + bat_destroy(uv); > BBPkeepref(*bid = id->batCacheid); > BBPkeepref(*uvl = vl->batCacheid); > } else { > + sql_schema *s = mvc_bind_schema(m, *sname); > + sql_table *t = mvc_bind_table(m, s, *tname); > + sql_column *c = mvc_bind_column(m, t, *cname); > + > *bid = e_bat(TYPE_oid); > - *uvl = e_bat(b->T->type); > + *uvl = e_bat(c->type.type->localtype); > } > BBPreleaseref(b->batCacheid); > } else { > @@ -1711,6 +1728,7 @@ mvc_bind_idxbat_wrap(Client cntxt, MalBl > bn = BATslice(b, part_nr * psz, (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz)); > BATseqbase(bn, part_nr * psz); > } else { > + /* BAT b holds the UPD_ID bat */ > oid l, h; > BAT *c = mvc_bind_idxbat(m, *sname, *tname, *iname, 0); > cnt = BATcount(c); > @@ -1718,23 +1736,37 @@ mvc_bind_idxbat_wrap(Client cntxt, MalBl > l = part_nr * psz; > h = (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz); > h--; > - bn = BATmirror(BATselect(BATmirror(b), &l, &h)); > + bn = BATsubselect(b, NULL, &l, &h, 1, 0, 0); > BBPreleaseref(c->batCacheid); > } > BBPreleaseref(b->batCacheid); > b = bn; > + } else if (upd) { > + BAT *uv = mvc_bind_idxbat(m, *sname, *tname, *iname, RD_UPD_VAL); > + int *uvl = (int *) getArgReference(stk, pci, 1); > + BBPkeepref(*bid = b->batCacheid); > + BBPkeepref(*uvl = uv->batCacheid); > + return MAL_SUCCEED; > } > if (upd) { > int *uvl = (int *) getArgReference(stk, pci, 1); > > if (BATcount(b)) { > - BAT *id = BATmirror(BATmark(b, 0)); > - BAT *vl = BATmirror(BATmark(BATmirror(b), 0)); > + BAT *uv = mvc_bind_idxbat(m, *sname, *tname, *iname, RD_UPD_VAL); > + BAT *ui = mvc_bind_idxbat(m, *sname, *tname, *iname, RD_UPD_ID); > + BAT *id = BATproject(b, ui); > + BAT *vl = BATproject(b, uv); > + bat_destroy(b); > + bat_destroy(ui); > + bat_destroy(uv); > BBPkeepref(*bid = id->batCacheid); > BBPkeepref(*uvl = vl->batCacheid); > } else { > + sql_schema *s = mvc_bind_schema(m, *sname); > + sql_idx *i = mvc_bind_idx(m, s, *iname); > + > *bid = e_bat(TYPE_oid); > - *uvl = e_bat(b->T->type); > + *uvl = e_bat((i->type==join_idx)?TYPE_oid:TYPE_wrd); > } > BBPreleaseref(b->batCacheid); > } else { > @@ -3959,7 +3991,7 @@ SQLcluster1(Client cntxt, MalBlkPtr mb, > c = o->data; > if (first) { > first = 0; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > msg = CLUSTER_key(&hid, &b->batCacheid); > BBPreleaseref(b->batCacheid); > if (msg) > @@ -3973,7 +4005,7 @@ SQLcluster1(Client cntxt, MalBlkPtr mb, > throw(SQL, "sql.cluster", "Can not access descriptor"); > } > > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL) > throw(SQL, "sql.cluster", "Can not access descriptor"); > msg = CLUSTER_apply(&bid, b, map); > @@ -4039,7 +4071,7 @@ SQLcluster2(Client cntxt, MalBlkPtr mb, > bat psum; > unsigned int bits = 10, off = 0; > first = 0; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > msg = MKEYbathash(&hid, &b->batCacheid); > BBPreleaseref(b->batCacheid); > if (msg) > @@ -4051,7 +4083,7 @@ SQLcluster2(Client cntxt, MalBlkPtr mb, > return msg; > } > > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL) > throw(SQL, "sql.cluster", "Can not access descriptor"); > msg = CLS_map(&bid, &mid, &b->batCacheid); > @@ -4132,7 +4164,7 @@ vacuum(Client cntxt, MalBlkPtr mb, MalSt > bids[i] = 0; > for (o = t->columns.set->h; o; o = o->next, i++) { > c = o->data; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL || (msg = (*func) (&bid, &(b->batCacheid), &(del->batCacheid))) != NULL) { > for (i--; i >= 0; i--) > BBPdecref(bids[i], TRUE); > @@ -4226,7 +4258,7 @@ SQLvacuum(Client cntxt, MalBlkPtr mb, Ma > > for (o = t->columns.set->h; o && ordered == 0; o = o->next) { > c = o->data; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL) > throw(SQL, "sql.vacuum", "Can not access descriptor"); > ordered |= BATtordered(b); > @@ -4409,7 +4441,7 @@ sql_storage(Client cntxt, MalBlkPtr mb, > for (ncol = (t)->columns.set->h; ncol; ncol = ncol->next) { > sql_base *bc = ncol->data; > sql_column *c = (sql_column *) ncol->data; > - BAT *bn = store_funcs.bind_col(tr, c, 0); > + BAT *bn = store_funcs.bind_col(tr, c, RDONLY); > lng sz; > > /*printf("schema %s.%s.%s" , b->name, bt->name, bc->name); */ > @@ -4474,7 +4506,7 @@ sql_storage(Client cntxt, MalBlkPtr mb, > sql_base *bc = ncol->data; > sql_idx *c = (sql_idx *) ncol->data; > if (c->type != no_idx) { > - BAT *bn = store_funcs.bind_idx(tr, c, 0); > + BAT *bn = store_funcs.bind_idx(tr, c, RDONLY); > lng sz; > > /*printf("schema %s.%s.%s" , b->name, bt->name, bc->name); */ > diff --git a/sql/backends/monet5/sql_gencode.c b/sql/backends/monet5/sql_gencode.c > --- a/sql/backends/monet5/sql_gencode.c > +++ b/sql/backends/monet5/sql_gencode.c > @@ -897,7 +897,7 @@ static int > q = newStmt2(mb, sqlRef, bindRef); > if (q == NULL) > return -1; > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > q = pushReturn(mb, q, newTmpVariable(mb, newBatType(ht, tt))); > } else > setVarType(mb, getArg(q, 0), newBatType(ht, tt)); > @@ -910,7 +910,7 @@ static int > return -1; > s->nr = getDestVar(q); > > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > /* rename second result */ > renameVariable(mb, getArg(q, 1), "r1_%d", s->nr); > } > @@ -924,7 +924,7 @@ static int > q = newStmt2(mb, sqlRef, bindidxRef); > if (q == NULL) > return -1; > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > q = pushReturn(mb, q, newTmpVariable(mb, newBatType(ht, tt))); > } else > setVarType(mb, getArg(q, 0), newBatType(ht, tt)); > @@ -937,7 +937,7 @@ static int > return -1; > s->nr = getDestVar(q); > > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > /* rename second result */ > renameVariable(mb, getArg(q, 1), "r1_%d", s->nr); > } > diff --git a/sql/backends/monet5/sql_optimizer.c b/sql/backends/monet5/sql_optimizer.c > --- a/sql/backends/monet5/sql_optimizer.c > +++ b/sql/backends/monet5/sql_optimizer.c > @@ -138,7 +138,7 @@ SQLgetStatistics(Client cntxt, mvc *m, M > if (i && (!isRemote(i->t) && !isMergeTable(i->t))) { > cnt = store_funcs.count_idx(tr, i, 1); > assert(cnt <= (size_t) GDK_oid_max); > - b = store_funcs.bind_idx(m->session->tr, i, 0); > + b = store_funcs.bind_idx(m->session->tr, i, RDONLY); > if (b) { > str loc; > if (b->batPersistence == PERSISTENT && BATlocation(&loc, &b->batCacheid) && loc) > @@ -158,7 +158,7 @@ SQLgetStatistics(Client cntxt, mvc *m, M > > cnt = store_funcs.count_col(tr, c, 1); > assert(cnt <= (size_t) GDK_oid_max); > - b = store_funcs.bind_col(m->session->tr, c, 0); > + b = store_funcs.bind_col(m->session->tr, c, RDONLY); > if (b) { > str loc; > if (b->batPersistence == PERSISTENT && BATlocation(&loc, &b->batCacheid) && loc) > diff --git a/sql/backends/monet5/sql_statistics.c b/sql/backends/monet5/sql_statistics.c > --- a/sql/backends/monet5/sql_statistics.c > +++ b/sql/backends/monet5/sql_statistics.c > @@ -98,7 +98,7 @@ sql_analyze(Client cntxt, MalBlkPtr mb, > for (ncol = (t)->columns.set->h; ncol; ncol = ncol->next) { > sql_base *bc = ncol->data; > sql_column *c = (sql_column *) ncol->data; > - BAT *bn = store_funcs.bind_col(tr, c, 0), *br; > + BAT *bn = store_funcs.bind_col(tr, c, RDONLY), *br; > BAT *bsample; > lng sz = BATcount(bn); > > diff --git a/sql/include/sql_catalog.h b/sql/include/sql_catalog.h > --- a/sql/include/sql_catalog.h > +++ b/sql/include/sql_catalog.h > @@ -92,8 +92,9 @@ > > #define RDONLY 0 > #define RD_INS 1 > -#define RD_UPD 2 > -#define QUICK 3 > +#define RD_UPD_ID 2 > +#define RD_UPD_VAL 3 > +#define QUICK 4 > > #define FRAME_ROWS 0 > #define FRAME_RANGE 1 > diff --git a/sql/storage/bat/bat_storage.c b/sql/storage/bat/bat_storage.c > --- a/sql/storage/bat/bat_storage.c > +++ b/sql/storage/bat/bat_storage.c > @@ -52,7 +52,7 @@ delta_bind_del(sql_dbat *bat, int access > (void) access; /* satisfy compiler */ > #endif > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

AND / OR evaluation inconsistency
by Scott Mathieson 03 Sep '14

03 Sep '14

Hi, I've come across something that is either a bug or is just my misunderstanding of how MonetDB deals with NULL results from boolean expressions. The issue I'm having is the OR and AND expressions behave differently in when evaluating a bollean expression where one of the variable is NULL (This is using version 11.15.19) If I run a query: SELECT ([some expression that evaluates to NULL] OR [some expression that evaluates to True] ) as "Result" - the returned value is <null> If I try SELECT ([some expression that evaluates to NULL] AND [some expression that evaluates to True] ) as "Result" - the returned value is True This would seem to suggest the AND evaluation is ignoring NULL or Treating NULL as meaning True whereas OR is propagating it. Which is correct and is this a bug? Thanks, Scott Mathieson ________________________________

3 2

Re: MonetDB: Oct2014 - fixed reuse
by Stefan Manegold 26 Aug '14

26 Aug '14

Niels, this checkin adds "div_by_zero_in_return.Bug-3555" (in addition to "orderby_on_constant.Bug-3554") to sql/test/BugTracker-2014/Tests/All but does not add the respective files (div_by_zero_in_return.Bug-3555.*). Do you have these files ready, or should we remove "div_by_zero_in_return.Bug-3555" from sql/test/BugTracker-2014/Tests/All ? The respective bug report is still open: https://www.monetdb.org/bugzilla/show_bug.cgi?id=3555 Thanks! Stefan ----- Original Message ----- > Changeset: 6c4080d30b3e for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=6c4080d30b3e > Added Files: > sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.sql > sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.err > sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.out > Modified Files: > monetdb5/modules/kernel/bat5.c > sql/backends/monet5/rel_bin.c > sql/backends/monet5/sql.c > sql/backends/monet5/sql_statement.c > sql/test/BugTracker-2014/Tests/All > Branch: Oct2014 > Log Message: > > fixed reuse > > fixed bug 3554, ie make sure single value order by /limit work > > > diffs (254 lines): > > diff --git a/monetdb5/modules/kernel/bat5.c b/monetdb5/modules/kernel/bat5.c > --- a/monetdb5/modules/kernel/bat5.c > +++ b/monetdb5/modules/kernel/bat5.c > @@ -2198,9 +2198,9 @@ BKCshrinkBATmap(int *ret, int *bid, int > Type *r = (Type*)Tloc(bn, BUNfirst(bn)); \ > for (;p<q; oidx++, p++) { \ > if ( *o == oidx ){ \ > - while ( *ol == bidx && ol>o) { \ > + while ( ol>o && *--ol == bidx) { \ > bidx--; \ > - ol--;q--; \ > + q--; \ > } \ > *r++ = *(--q); \ > o += (o < ol); \ > @@ -2241,9 +2241,10 @@ BKCreuseBAT(int *ret, int *bid, int *did > throw(MAL, "bat.reuse", MAL_MALLOC_FAIL ); > } > > - bidx= BUNlast(b)-1; > + oidx = b->hseqbase; > + bidx = oidx + BUNlast(b)-1; > o = (oid*)Tloc(bs, BUNfirst(bs)); > - ol= (oid*)Tloc(bs, BUNlast(bs))-1; > + ol= (oid*)Tloc(bs, BUNlast(bs)); > > switch(ATOMstorage(b->ttype) ){ > case TYPE_bte: reuseloop(bte); break; > @@ -2262,15 +2263,16 @@ BKCreuseBAT(int *ret, int *bid, int *did > > for (;p<q; oidx++, p++) { > if ( *o == oidx ){ > - while ( *ol == bidx && ol>o) { > + while ( ol > o && *--ol == bidx) { > bidx--; > - ol--;q--; > + q--; > } > BUNappend(bn, BUNtail(bi, --q), FALSE); > o += (o < ol); > bidx--; > - } else > + } else { > BUNappend(bn, BUNtail(bi, p), FALSE); > + } > } > } else { > switch( b->T->width){ > @@ -2298,6 +2300,7 @@ BKCreuseBAT(int *ret, int *bid, int *did > BBPkeepref(*ret= bn->batCacheid); > return MAL_SUCCEED; > } > + > str > BKCreuseBATmap(int *ret, int *bid, int *did) > { > @@ -2330,10 +2333,11 @@ BKCreuseBATmap(int *ret, int *bid, int * > throw(MAL, "bat.shrinkMap", MAL_MALLOC_FAIL ); > } > > - bidx= BUNlast(b)-1; > - o = (oid*)Tloc(bs, BUNfirst(bs)); > - ol= (oid*)Tloc(bs, BUNlast(bs)); > - r = (oid*)Tloc(bn, BUNfirst(bn)); > + oidx = b->hseqbase; > + bidx = oidx + BUNlast(b)-1; > + o = (oid*)Tloc(bs, BUNfirst(bs)); > + ol = (oid*)Tloc(bs, BUNlast(bs)); > + r = (oid*)Tloc(bn, BUNfirst(bn)); > > for (;oidx<bidx; oidx++) { > if ( *o == oidx ){ > @@ -2343,8 +2347,9 @@ BKCreuseBATmap(int *ret, int *bid, int * > *r++ = bidx; > o += (o < ol); > bidx--; > - } else > + } else { > *r++ = oidx; > + } > } > > BATsetcount(bn, BATcount(b)-BATcount(bs)); > diff --git a/sql/backends/monet5/rel_bin.c b/sql/backends/monet5/rel_bin.c > --- a/sql/backends/monet5/rel_bin.c > +++ b/sql/backends/monet5/rel_bin.c > @@ -2306,6 +2306,8 @@ rel2bin_project( mvc *sql, sql_rel *rel, > if (!orderbycolstmt) > return NULL; > > + /* handle constants */ > + orderbycolstmt = column(sql->sa, orderbycolstmt); > if (!limit) { /* topn based on a single column */ > limit = stmt_limit(sql->sa, orderbycolstmt, stmt_atom_wrd(sql->sa, 0), > l, LIMIT_FLAG(distinct, is_ascending(orderbycole), last, 1)); > } else { /* topn based on 2 columns */ > diff --git a/sql/backends/monet5/sql.c b/sql/backends/monet5/sql.c > --- a/sql/backends/monet5/sql.c > +++ b/sql/backends/monet5/sql.c > @@ -4136,6 +4136,11 @@ vacuum(Client cntxt, MalBlkPtr mb, MalSt > > /* get the deletions BAT */ > del = mvc_bind_dbat(m, *sch, *tbl, RD_INS); > + if (BATcount(del) == 0) { > + BBPreleaseref(del->batCacheid); > + return MAL_SUCCEED; > + } > + > > i = 0; > bids[i] = 0; > @@ -4163,6 +4168,7 @@ vacuum(Client cntxt, MalBlkPtr mb, MalSt > BBPdecref(bids[i], TRUE); > throw(SQL, name, "Too many columns to handle, use copy instead"); > } > + BBPreleaseref(del->batCacheid); > > mvc_clear_table(m, t); > for (o = t->columns.set->h, i = 0; o; o = o->next, i++) { > @@ -4245,13 +4251,15 @@ SQLvacuum(Client cntxt, MalBlkPtr mb, Ma > /* get the deletions BAT */ > del = mvc_bind_dbat(m, *sch, *tbl, RD_INS); > > - /* now decide on the algorithm */ > - if (ordered) { > - if (BATcount(del) > cnt / 20) > - SQLshrink(cntxt, mb, stk, pci); > - } else > - SQLreuse(cntxt, mb, stk, pci); > - > + if (BATcount(del) > 0) { > + /* now decide on the algorithm */ > + if (ordered) { > + if (BATcount(del) > cnt / 20) > + SQLshrink(cntxt, mb, stk, pci); > + } else { > + SQLreuse(cntxt, mb, stk, pci); > + } > + } > BBPreleaseref(del->batCacheid); > return MAL_SUCCEED; > } > diff --git a/sql/backends/monet5/sql_statement.c > b/sql/backends/monet5/sql_statement.c > --- a/sql/backends/monet5/sql_statement.c > +++ b/sql/backends/monet5/sql_statement.c > @@ -1719,6 +1719,9 @@ const_column(sql_allocator *sa, stmt *va > s->op1 = val; > s->op4.typeval = *ct; > s->nrcols = 1; > + > + s->tname = val->tname; > + s->cname = val->cname; > return s; > } > > diff --git a/sql/test/BugTracker-2014/Tests/All > b/sql/test/BugTracker-2014/Tests/All > --- a/sql/test/BugTracker-2014/Tests/All > +++ b/sql/test/BugTracker-2014/Tests/All > @@ -36,3 +36,5 @@ round.Bug-3542 > number_4_4.Bug-3543 > ifthenelse.Bug-3546 > subselect-reverse.Bug-3552 > +orderby_on_constant.Bug-3554 > +div_by_zero_in_return.Bug-3555 > diff --git a/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.sql > b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.sql > new file mode 100644 > --- /dev/null > +++ b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.sql > @@ -0,0 +1,1 @@ > +select a+b from (select 1 as a,1 as b) as q order by a limit 1; > diff --git > a/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.err > b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.err > new file mode 100644 > --- /dev/null > +++ b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.err > @@ -0,0 +1,37 @@ > +stderr of test 'orderby_on_constant.Bug-3554` in directory > 'sql/test/BugTracker-2014` itself: > + > + > +# 19:40:06 > > +# 19:40:06 > "mserver5" "--debug=10" "--set" "gdk_nr_threads=0" "--set" > "mapi_open=true" "--set" "mapi_port=31986" "--set" > "mapi_usock=/var/tmp/mtest-21225/.s.monetdb.31986" "--set" "monet_prompt=" > "--forcemito" "--set" "mal_listing=2" > "--dbpath=/home/niels/scratch/rc-monetdb/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2014" > "--set" "mal_listing=0" "--set" "embedded_r=yes" > +# 19:40:06 > > + > +# builtin opt gdk_dbpath = > /home/niels/scratch/rc-monetdb/Linux-x86_64/var/monetdb5/dbfarm/demo > +# builtin opt gdk_debug = 0 > +# builtin opt gdk_vmtrim = no > +# builtin opt monet_prompt = > > +# builtin opt monet_daemon = no > +# builtin opt mapi_port = 50000 > +# builtin opt mapi_open = false > +# builtin opt mapi_autosense = false > +# builtin opt sql_optimizer = default_pipe > +# builtin opt sql_debug = 0 > +# cmdline opt gdk_nr_threads = 0 > +# cmdline opt mapi_open = true > +# cmdline opt mapi_port = 31986 > +# cmdline opt mapi_usock = /var/tmp/mtest-21225/.s.monetdb.31986 > +# cmdline opt monet_prompt = > +# cmdline opt mal_listing = 2 > +# cmdline opt gdk_dbpath = > /home/niels/scratch/rc-monetdb/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2014 > +# cmdline opt mal_listing = 0 > +# cmdline opt embedded_r = yes > +# cmdline opt gdk_debug = 536870922 > + > +# 19:40:07 > > +# 19:40:07 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" > "--host=/var/tmp/mtest-21225" "--port=31986" > +# 19:40:07 > > + > + > +# 19:40:07 > > +# 19:40:07 > "Done." > +# 19:40:07 > > + > diff --git > a/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.out > b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.out > new file mode 100644 > --- /dev/null > +++ b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.out > @@ -0,0 +1,38 @@ > +stdout of test 'orderby_on_constant.Bug-3554` in directory > 'sql/test/BugTracker-2014` itself: > + > + > +# 19:40:06 > > +# 19:40:06 > "mserver5" "--debug=10" "--set" "gdk_nr_threads=0" "--set" > "mapi_open=true" "--set" "mapi_port=31986" "--set" > "mapi_usock=/var/tmp/mtest-21225/.s.monetdb.31986" "--set" "monet_prompt=" > "--forcemito" "--set" "mal_listing=2" > "--dbpath=/home/niels/scratch/rc-monetdb/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2014" > "--set" "mal_listing=0" "--set" "embedded_r=yes" > +# 19:40:06 > > + > +# MonetDB 5 server v11.19.0 > +# This is an unreleased version > +# Serving database 'mTests_sql_test_BugTracker-2014', using 4 threads > +# Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically > linked > +# Found 7.334 GiB available main-memory. > +# Copyright (c) 1993-July 2008 CWI. > +# Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved > +# Visit http://www.monetdb.org/ for further information > +# Listening for connection requests on > mapi:monetdb://localhost.nes.nl:31986/ > +# Listening for UNIX domain connection requests on > mapi:monetdb:///var/tmp/mtest-21225/.s.monetdb.31986 > +# MonetDB/GIS module loaded > +# MonetDB/SQL module loaded > +# MonetDB/R module loaded > + > +Ready. > + > +# 19:40:07 > > +# 19:40:07 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" > "--host=/var/tmp/mtest-21225" "--port=31986" > +# 19:40:07 > > + > +#select a+b from (select 1 as a,1 as b) as q order by a limit 1; > +% .L1 # table_name > +% L1 # name > +% smallint # type > +% 1 # length > +[ 2 ] > + > +# 19:40:07 > > +# 19:40:07 > "Done." > +# 19:40:07 > > + > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: Protobuf - Protocol buffer prototype for the record
by Stefan Manegold 26 Aug '14

26 Aug '14

Thanks, Hannes! Just for the records, new (and required) files pbstream.c pbstream.h mapi_pb-c.c mapi_pb-c.h are missing. Stefan ----- Original Message ----- > Changeset: af2757009ce3 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=af2757009ce3 > Modified Files: > clients/mapiclient/Makefile.ag > clients/mapiclient/mclient.c > clients/mapilib/Makefile.ag > clients/mapilib/mapi.c > clients/mapilib/mapi.h > common/stream/Makefile.ag > common/stream/stream.c > common/utils/mcrypt.c > common/utils/mcrypt.h > configure.ag > monetdb5/extras/jaql/Makefile.ag > monetdb5/extras/mal_optimizer_template/Makefile.ag > monetdb5/mal/Makefile.ag > monetdb5/mal/mal_authorize.c > monetdb5/mal/mal_client.c > monetdb5/mal/mal_client.h > monetdb5/mal/mal_debugger.c > monetdb5/mal/mal_readline.c > monetdb5/mal/mal_scenario.c > monetdb5/mal/mal_session.c > monetdb5/mal/mal_session.h > monetdb5/modules/atoms/Makefile.ag > monetdb5/modules/kernel/Makefile.ag > monetdb5/modules/mal/Makefile.ag > monetdb5/modules/mal/mal_mapi.c > monetdb5/modules/mal/mal_mapi.h > monetdb5/optimizer/Makefile.ag > monetdb5/scheduler/Makefile.ag > sql/backends/monet5/LSST/Makefile.ag > sql/backends/monet5/Makefile.ag > sql/backends/monet5/UDF/Makefile.ag > sql/backends/monet5/sql_result.c > sql/backends/monet5/sql_scenario.c > sql/backends/monet5/sql_scenario.h > sql/backends/monet5/vaults/Makefile.ag > sql/storage/bat/Makefile.ag > tools/merovingian/daemon/client.c > tools/mserver/Makefile.ag > tools/mserver/mserver5.c > Branch: Protobuf > Log Message: > > Protocol buffer prototype for the record > > > diffs (truncated from 2665 to 300 lines): > > diff --git a/clients/mapiclient/Makefile.ag b/clients/mapiclient/Makefile.ag > --- a/clients/mapiclient/Makefile.ag > +++ b/clients/mapiclient/Makefile.ag > @@ -16,7 +16,7 @@ > # All Rights Reserved. > > MTSAFE > -INCLUDES = ../mapilib ../../common/options ../../common/stream > $(READLINE_INCS) > +INCLUDES = ../mapilib ../../common/options ../../common/stream > $(READLINE_INCS) $(PROTOBUF_C_CFLAGS) > > lib_mcutil = { > NOINST > @@ -35,8 +35,8 @@ bin_msqldump = { > SOURCES = msqldump.c > LIBS = libmcutil ../mapilib/libmapi \ > ../../common/stream/libstream \ > - $(curl_LIBS) > -} > + $(curl_LIBS) > + } > > bin_stethoscope = { > SOURCES = stethoscope.c > diff --git a/clients/mapiclient/mclient.c b/clients/mapiclient/mclient.c > --- a/clients/mapiclient/mclient.c > +++ b/clients/mapiclient/mclient.c > @@ -347,13 +347,9 @@ fetch_line(MapiHdl hdl) > static int > fetch_row(MapiHdl hdl) > { > - char *reply; > > - do { > - if ((reply = fetch_line(hdl)) == NULL) > - return 0; > - } while (*reply != '[' && *reply != '='); > - return mapi_split_line(hdl); > + // TODO: this is almost a NOP, because we already have everything from the > PB. fetch_line() just has to pay attention to the export size in the future. > + return mapi_fetch_row(hdl); > } > > static void > @@ -668,11 +664,14 @@ CSVrenderer(MapiHdl hdl) > s = mapi_get_name(hdl, i); > if (s == NULL) > s = ""; > - mnstr_printf(toConsole, "%s%s", > + printf( "%s%s", > i == 0 ? "" : sep, s ? s : ""); > } > - mnstr_printf(toConsole, "\n"); > + printf( "\n"); > } > + > + // TODO: switch on type! > + > while (!mnstr_errnr(toConsole) && (fields = fetch_row(hdl)) != 0) { > for (i = 0; i < fields; i++) { > s = mapi_fetch_field(hdl, i); > @@ -681,37 +680,38 @@ CSVrenderer(MapiHdl hdl) > if (strchr(s, *sep) != NULL || > strchr(s, '\n') != NULL || > strchr(s, '"') != NULL) { > - mnstr_printf(toConsole, "%s\"", > + printf( "%s\"", > i == 0 ? "" : sep); > + printf("%s",s); > while (*s) { > switch (*s) { > case '\n': > - mnstr_write(toConsole, "\\n", 1, 2); > + printf( "\\n"); > break; > case '\t': > - mnstr_write(toConsole, "\\t", 1, 2); > + printf( "\\t"); > break; > case '\r': > - mnstr_write(toConsole, "\\r", 1, 2); > + printf( "\\r"); > break; > case '\\': > - mnstr_write(toConsole, "\\\\", 1, 2); > + printf( "\\\\"); > break; > case '"': > - mnstr_write(toConsole, "\"\"", 1, 2); > + printf( "\"\""); > break; > default: > - mnstr_write(toConsole, s, 1, 1); > + > break; > } > s++; > } > - mnstr_write(toConsole, "\"", 1, 1); > + printf("\""); > } else > - mnstr_printf(toConsole, "%s%s", > + printf( "%s%s", > i == 0 ? "" : sep, s ? s : ""); > } > - mnstr_printf(toConsole, "\n"); > + printf( "\n"); > } > } > > diff --git a/clients/mapilib/Makefile.ag b/clients/mapilib/Makefile.ag > --- a/clients/mapilib/Makefile.ag > +++ b/clients/mapilib/Makefile.ag > @@ -18,11 +18,11 @@ > MTSAFE > > INCLUDES = ../../common/options ../../common/stream ../../common/utils \ > - $(MSGCONTROL_FLAGS) > + $(MSGCONTROL_FLAGS) $(PROTOBUF_C_CFLAGS) > > lib_mapi = { > VERSION = $(MAPI_VERSION) > - SOURCES = mapi.c mapi.rc > + SOURCES = mapi.c mapi.rc > LIBS = $(SOCKET_LIBS) ../../common/stream/libstream \ > ../../common/options/libmoptions \ > ../../common/utils/libmcrypt $(openssl_LIBS) > diff --git a/clients/mapilib/mapi.c b/clients/mapilib/mapi.c > --- a/clients/mapilib/mapi.c > +++ b/clients/mapilib/mapi.c > @@ -772,10 +772,13 @@ > > #include "monetdb_config.h" > #include <stream.h> /* include before mapi.h */ > + > #include <stream_socket.h> > #include "mapi.h" > #include "mcrypt.h" > > +#include "pbstream.h" > + > #ifdef HAVE_UNISTD_H > # include <unistd.h> > #endif > @@ -946,6 +949,7 @@ struct MapiResultSet { > struct MapiColumn *fields; > struct MapiRowBuf cache; > int commentonly; /* only comments seen so far */ > + MapiReply__QueryResult *pb; > }; > > struct MapiStatement { > @@ -1429,7 +1433,6 @@ close_result(MapiHdl hdl) > { > struct MapiResultSet *result; > Mapi mid; > - int i; > > result = hdl->result; > if (result == NULL) > @@ -1438,125 +1441,13 @@ close_result(MapiHdl hdl) > assert(mid != NULL); > if (mid->trace == MAPI_TRACE) > printf("closing result set\n"); > - if (result->tableid >= 0 && result->querytype != Q_PREPARE) { > - if (mid->active && > - result->next == NULL && > - !mid->active->needmore && > - read_into_cache(mid->active, -1) != MOK) > - return MERROR; > - assert(hdl->npending_close == 0 || > - (hdl->npending_close > 0 && hdl->pending_close != NULL)); > - if (mid->active && > - (mid->active->active != result || > - result->cache.tuplecount < result->row_count)) > - { > - /* results for which we got all tuples at the initial > - * response, need not to be closed as the server already > - * did that immediately */ > - if (result->row_count > result->tuple_count) { > - /* can't write "X" commands now, so save for later */ > - REALLOC(hdl->pending_close, hdl->npending_close + 1); > - hdl->pending_close[hdl->npending_close] = result->tableid; > - hdl->npending_close++; > - } > - } else if (mid->to != NULL) { > - /* first close saved up to-be-closed tables */ > - for (i = 0; i < hdl->npending_close; i++) { > - char msg[256]; > - > - snprintf(msg, sizeof(msg), "Xclose %d\n", hdl->pending_close[i]); > - mapi_log_record(mid, msg); > - mid->active = hdl; > - if (mnstr_printf(mid->to, "%s", msg) < 0 || > - mnstr_flush(mid->to)) { > - close_connection(mid); > - mapi_setError(mid, mnstr_error(mid->to), "mapi_close_handle", > MTIMEOUT); > - break; > - } > - read_into_cache(hdl, 0); > - } > - hdl->npending_close = 0; > - if (hdl->pending_close) > - free(hdl->pending_close); > - hdl->pending_close = NULL; > - if (mid->to != NULL && result->tuple_count < result->row_count) { > - char msg[256]; > - > - snprintf(msg, sizeof(msg), "Xclose %d\n", result->tableid); > - mapi_log_record(mid, msg); > - mid->active = hdl; > - if (mnstr_printf(mid->to, "%s", msg) < 0 || > - mnstr_flush(mid->to)) { > - close_connection(mid); > - mapi_setError(mid, mnstr_error(mid->to), "mapi_close_handle", > MTIMEOUT); > - } else > - read_into_cache(hdl, 0); > - } > - } > - result->tableid = -1; > + if (hdl->active != NULL) { > + // TODO: cleanup > } > - if (mid->active == hdl && > - hdl->active == result && > - read_into_cache(hdl, -1) != MOK) > - return MERROR; > - assert(hdl->active != result); > - if (result->fields) { > - for (i = 0; i < result->maxfields; i++) { > - if (result->fields[i].tablename) > - free(result->fields[i].tablename); > - if (result->fields[i].columnname) > - free(result->fields[i].columnname); > - if (result->fields[i].columntype) > - free(result->fields[i].columntype); > - } > - free(result->fields); > - } > - result->fields = NULL; > - result->maxfields = result->fieldcnt = 0; > - if (result->cache.line) { > - for (i = 0; i < result->cache.writer; i++) { > - if (result->cache.line[i].rows) > - free(result->cache.line[i].rows); > - if (result->cache.line[i].anchors) { > - int j; > - > - for (j = 0; j < result->cache.line[i].fldcnt; j++) > - if (result->cache.line[i].anchors[j]) { > - free(result->cache.line[i].anchors[j]); > - result->cache.line[i].anchors[j] = NULL; > - } > - free(result->cache.line[i].anchors); > - } > - if (result->cache.line[i].lens) > - free(result->cache.line[i].lens); > - } > - free(result->cache.line); > - result->cache.line = NULL; > - result->cache.tuplecount = 0; > - } > - if (result->errorstr) > - free(result->errorstr); > - result->errorstr = NULL; > - result->hdl = NULL; > - hdl->result = result->next; > - if (hdl->result == NULL) > - hdl->lastresult = NULL; > - result->next = NULL; > - free(result); > + > return MOK; > } > > -static void > -add_error(struct MapiResultSet *result, char *error) > -{ > - /* concatenate the error messages */ > - size_t size = result->errorstr ? strlen(result->errorstr) : 0; > - > - REALLOC(result->errorstr, size + strlen(error) + 2); > - strcpy(result->errorstr + size, error); > - strcat(result->errorstr + size, "\n"); > -} > - > char * > mapi_result_error(MapiHdl hdl) > { > @@ -1571,17 +1462,11 @@ MapiMsg > mapi_next_result(MapiHdl hdl) > { > mapi_hdl_check(hdl, "mapi_next_result"); > - > - while (hdl->result != NULL) { > +// TODO: we only like to export more > + while (hdl->active != NULL) { > if (close_result(hdl) != MOK) > return MERROR; > - if (hdl->result && > - (hdl->result->querytype == -1 || > - /* basically exclude Q_PARSE and Q_BLOCK */ > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: Contents of developers-list Digest, Vol 24, Issue 3
by Foteini Alvanaki 14 Aug '14

14 Aug '14

The changes to the geos are neither documented nor send somewhere for future use. The implemented functions force users to create polygons without holes (there is no argument for the holes). In case the default geos library is used, an error informs the user that the geometry type she is using is not the correct one. My change in geos added this geometry type. Best, Foteini On 08/14/2014 12:00 PM, developers-list-request(a)monetdb.org wrote: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. Re: MonetDB: geo - MakePolygon/Polygon implmentation. Works > only for... (Stefan Manegold) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 13 Aug 2014 14:00:47 +0200 (CEST) > From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> > To: developers-list(a)monetdb.org > Subject: Re: MonetDB: geo - MakePolygon/Polygon implmentation. Works > only for... > Message-ID: <1961125276.5600498.1407931247727.JavaMail.zimbra(a)cwi.nl> > Content-Type: text/plain; charset=utf-8 > > > Thanks! > > purely out of curiosity: > > - are there checks & warnings/ error messages in place > for not (trying) to use it for polygons with holes, > as well as a (configure) check whether the geos library > provides the required functionality? > > - are the changes to the geos module documented (at least > as "diff" with reference to which code version it applies), > or have these changes even been send "up-stream" to have them > by default included in a future version of the library? > > Thanks! > Stefan > > ----- Original Message ----- >> Changeset: 4d4c6ce1a817 for MonetDB >> URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4d4c6ce1a817 >> Modified Files: >> geom/monetdb5/geom.c >> geom/monetdb5/geom.mal >> geom/sql/40_geom.sql >> Branch: geo >> Log Message: >> >> MakePolygon/Polygon implmentation. Works only for polygons without holes. >> Changes to geos library were necessary so, it will not work with standard >> geos library. >> >> >> diffs (80 lines): >> >> diff --git a/geom/monetdb5/geom.c b/geom/monetdb5/geom.c >> --- a/geom/monetdb5/geom.c >> +++ b/geom/monetdb5/geom.c >> @@ -600,7 +600,7 @@ str geom_2_geom(wkb** resWKB, wkb **valu >> int valueType = 0; >> >> int valueSRID = (*valueWKB)->srid; >> -//fprintf(stderr, "in geom_2_geom\n"); >> + >> /* get the geosGeometry from the wkb */ >> geosGeometry = wkb2geos(*valueWKB); >> /* get the number of coordinates the geometry has */ >> @@ -784,7 +784,7 @@ wkb* geos2wkb(const GEOSGeometry* geosGe >> geomWKB->srid = GEOSGetSRID(geosGeometry); >> memcpy(&geomWKB->data, w, wkbLen); >> GEOSFree(w); >> - >> + >> return geomWKB; >> } >> >> @@ -1944,13 +1944,13 @@ str wkbMakePolygon(wkb** out, wkb** exte >> } >> >> externalGeometry = wkb2geos(*external); >> - if ((GEOSGeomTypeId(externalGeometry)+1) != wkbLineString) { >> + if ((GEOSGeomTypeId(externalGeometry)+1) != wkbLinearRing) { >> *out = wkb_nil; >> GEOSGeom_destroy(externalGeometry); >> - throw(MAL, "geom.Polygon", "Geometries should be LineStrings"); >> + throw(MAL, "geom.Polygon", "Geometries should be LinearRings"); >> } >> >> - if(internalBAT_id == NULL) { >> + if(*internalBAT_id == 0) { >> geosGeometry = GEOSGeom_createPolygon(externalGeometry, NULL, 0); >> if(geosGeometry == NULL) { >> GEOSGeom_destroy(externalGeometry); >> diff --git a/geom/monetdb5/geom.mal b/geom/monetdb5/geom.mal >> --- a/geom/monetdb5/geom.mal >> +++ b/geom/monetdb5/geom.mal >> @@ -289,14 +289,14 @@ function MakeEnvelope(xmin:dbl, ymin:dbl >> return x; >> end MakeEnvelope; >> >> -#function MakePolygon(external:wkb) :wkb; >> -# x := Polygon(external, NULL, 0); >> -# return x; >> -#end MakePolygon; >> -#function MakePolygon(external:wkb, srid:int) :wkb; >> -# x := Polygon(external, NULL, srid); >> -# return x; >> -#end MakePolygon; >> +function MakePolygon(external:wkb) :wkb; >> + x := Polygon(external, nil:bat, 0); >> + return x; >> +end MakePolygon; >> +function MakePolygon(external:wkb, srid:int) :wkb; >> + x := Polygon(external, nil:bat, srid); >> + return x; >> +end MakePolygon; >> #function MakePolygon(external:wkb, internal:bat[:oid,:wkb]) :wkb; >> # x := Polygon(external, internal, 0); >> # return x; >> diff --git a/geom/sql/40_geom.sql b/geom/sql/40_geom.sql >> --- a/geom/sql/40_geom.sql >> +++ b/geom/sql/40_geom.sql >> @@ -4070,9 +4070,9 @@ CREATE FUNCTION ST_MakePointM(x double, >> --CREATE FUNCTION ST_LineFromMultiPoint(pointGeom Geometry) RETURNS Geometry >> external name geom."LineFromMultiPoint"; --gets mutlipoint returns >> linestring >> CREATE FUNCTION ST_MakeEnvelope(xmin double, ymin double, xmax double, ymax >> double, srid integer) RETURNS Geometry external name geom."MakeEnvelope"; >> CREATE FUNCTION ST_MakeEnvelope(xmin double, ymin double, xmax double, ymax >> double) RETURNS Geometry external name geom."MakeEnvelope"; >> ---CREATE FUNCTION ST_MakePolygon(geom Geometry) RETURNS Geometry external >> name geom."MakePolygon"; --gets linestring >> ---CREATE FUNCTION ST_MakePolygon(outerGeom Geometry, interiorGeoms >> Geometry[]) RETURNS Geometry external name geom."MakePolygon"; --gets >> linestrings >> ---CREATE FUNCTION ST_Polygon(geom Geometry, srid integer) RETURNS Geometry >> external name geom."MakePolygon"; --gets linestring >> +CREATE FUNCTION ST_MakePolygon(geom Geometry) RETURNS Geometry external name >> geom."MakePolygon"; --gets linestring >> +--CREATE FUNCTION ST_MakePolygon(outerGeom Geometry, interiorGeoms table(g >> Geometry)) RETURNS Geometry external name geom."MakePolygon"; --gets >> linestrings >> +CREATE FUNCTION ST_Polygon(geom Geometry, srid integer) RETURNS Geometry >> external name geom."MakePolygon"; --gets linestring >> CREATE FUNCTION ST_MakeBox2D(lowLeftPointGeom Geometry, upRightPointGeom >> Geometry) RETURNS mbr external name geom."MakeBox2D"; --gets 2d points >> --CREATE FUNCTION ST_3DMakeBox(lowLeftPointGeom Geometry, upRightPointGeom >> Geometry) RETURNS mbr external name geom."MakeBox3D"; --gets 3d points >> >> _______________________________________________ >> checkin-list mailing list >> checkin-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/checkin-list >>

1 0

Re: MonetDB: geo - MakePolygon/Polygon implmentation. Works only for...
by Stefan Manegold 13 Aug '14

13 Aug '14

Thanks! purely out of curiosity: - are there checks & warnings/ error messages in place for not (trying) to use it for polygons with holes, as well as a (configure) check whether the geos library provides the required functionality? - are the changes to the geos module documented (at least as "diff" with reference to which code version it applies), or have these changes even been send "up-stream" to have them by default included in a future version of the library? Thanks! Stefan ----- Original Message ----- > Changeset: 4d4c6ce1a817 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4d4c6ce1a817 > Modified Files: > geom/monetdb5/geom.c > geom/monetdb5/geom.mal > geom/sql/40_geom.sql > Branch: geo > Log Message: > > MakePolygon/Polygon implmentation. Works only for polygons without holes. > Changes to geos library were necessary so, it will not work with standard > geos library. > > > diffs (80 lines): > > diff --git a/geom/monetdb5/geom.c b/geom/monetdb5/geom.c > --- a/geom/monetdb5/geom.c > +++ b/geom/monetdb5/geom.c > @@ -600,7 +600,7 @@ str geom_2_geom(wkb** resWKB, wkb **valu > int valueType = 0; > > int valueSRID = (*valueWKB)->srid; > -//fprintf(stderr, "in geom_2_geom\n"); > + > /* get the geosGeometry from the wkb */ > geosGeometry = wkb2geos(*valueWKB); > /* get the number of coordinates the geometry has */ > @@ -784,7 +784,7 @@ wkb* geos2wkb(const GEOSGeometry* geosGe > geomWKB->srid = GEOSGetSRID(geosGeometry); > memcpy(&geomWKB->data, w, wkbLen); > GEOSFree(w); > - > + > return geomWKB; > } > > @@ -1944,13 +1944,13 @@ str wkbMakePolygon(wkb** out, wkb** exte > } > > externalGeometry = wkb2geos(*external); > - if ((GEOSGeomTypeId(externalGeometry)+1) != wkbLineString) { > + if ((GEOSGeomTypeId(externalGeometry)+1) != wkbLinearRing) { > *out = wkb_nil; > GEOSGeom_destroy(externalGeometry); > - throw(MAL, "geom.Polygon", "Geometries should be LineStrings"); > + throw(MAL, "geom.Polygon", "Geometries should be LinearRings"); > } > > - if(internalBAT_id == NULL) { > + if(*internalBAT_id == 0) { > geosGeometry = GEOSGeom_createPolygon(externalGeometry, NULL, 0); > if(geosGeometry == NULL) { > GEOSGeom_destroy(externalGeometry); > diff --git a/geom/monetdb5/geom.mal b/geom/monetdb5/geom.mal > --- a/geom/monetdb5/geom.mal > +++ b/geom/monetdb5/geom.mal > @@ -289,14 +289,14 @@ function MakeEnvelope(xmin:dbl, ymin:dbl > return x; > end MakeEnvelope; > > -#function MakePolygon(external:wkb) :wkb; > -# x := Polygon(external, NULL, 0); > -# return x; > -#end MakePolygon; > -#function MakePolygon(external:wkb, srid:int) :wkb; > -# x := Polygon(external, NULL, srid); > -# return x; > -#end MakePolygon; > +function MakePolygon(external:wkb) :wkb; > + x := Polygon(external, nil:bat, 0); > + return x; > +end MakePolygon; > +function MakePolygon(external:wkb, srid:int) :wkb; > + x := Polygon(external, nil:bat, srid); > + return x; > +end MakePolygon; > #function MakePolygon(external:wkb, internal:bat[:oid,:wkb]) :wkb; > # x := Polygon(external, internal, 0); > # return x; > diff --git a/geom/sql/40_geom.sql b/geom/sql/40_geom.sql > --- a/geom/sql/40_geom.sql > +++ b/geom/sql/40_geom.sql > @@ -4070,9 +4070,9 @@ CREATE FUNCTION ST_MakePointM(x double, > --CREATE FUNCTION ST_LineFromMultiPoint(pointGeom Geometry) RETURNS Geometry > external name geom."LineFromMultiPoint"; --gets mutlipoint returns > linestring > CREATE FUNCTION ST_MakeEnvelope(xmin double, ymin double, xmax double, ymax > double, srid integer) RETURNS Geometry external name geom."MakeEnvelope"; > CREATE FUNCTION ST_MakeEnvelope(xmin double, ymin double, xmax double, ymax > double) RETURNS Geometry external name geom."MakeEnvelope"; > ---CREATE FUNCTION ST_MakePolygon(geom Geometry) RETURNS Geometry external > name geom."MakePolygon"; --gets linestring > ---CREATE FUNCTION ST_MakePolygon(outerGeom Geometry, interiorGeoms > Geometry[]) RETURNS Geometry external name geom."MakePolygon"; --gets > linestrings > ---CREATE FUNCTION ST_Polygon(geom Geometry, srid integer) RETURNS Geometry > external name geom."MakePolygon"; --gets linestring > +CREATE FUNCTION ST_MakePolygon(geom Geometry) RETURNS Geometry external name > geom."MakePolygon"; --gets linestring > +--CREATE FUNCTION ST_MakePolygon(outerGeom Geometry, interiorGeoms table(g > Geometry)) RETURNS Geometry external name geom."MakePolygon"; --gets > linestrings > +CREATE FUNCTION ST_Polygon(geom Geometry, srid integer) RETURNS Geometry > external name geom."MakePolygon"; --gets linestring > CREATE FUNCTION ST_MakeBox2D(lowLeftPointGeom Geometry, upRightPointGeom > Geometry) RETURNS mbr external name geom."MakeBox2D"; --gets 2d points > --CREATE FUNCTION ST_3DMakeBox(lowLeftPointGeom Geometry, upRightPointGeom > Geometry) RETURNS mbr external name geom."MakeBox3D"; --gets 3d points > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

editing monetDB source code
by Hina Khan 12 Aug '14

12 Aug '14

Hi All, I am a PHD student and my research is related to data exploration. In my recent work I have proposed a new query processing technique using columnar data storage layer. For experiments I need to implement a new operator that works in between two "Select" operators. I wanted to know if there is any way that I can implement this new operator and add it to the source code of MonetDB and change its "Select" operator so that it calls the new operator I have inserted. So in short my question is " Is there any way to edit the source code of MonetDB". I would highly appreciate your help in this regard. Cheers, Hina

2 1

New Oct2014 stable branch created
by Sjoerd Mullender 11 Aug '14

11 Aug '14

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Oct2014. So in each of your stable checkout directories you can do hg pull hg update -rOct2014 to update to this new branch. -- Sjoerd Mullender

1 0

Re: MonetDB: default - Removed Alex's array implementation
by Ying Zhang 05 Aug '14

05 Aug '14

Dear all, I probably have broken Mtest.py with this changeset, since Mtest.py doesn't work for me any more just after this changeset. But I don't see what I did wrong. Does anyone have an idea? Thanks! Jennie On Aug 05, 2014, at 12:35, Jennie Zhang <commits(a)monetdb.org> wrote: > Changeset: b62643655146 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=b62643655146 > Removed Files: > monetdb5/mal/Tests/tst267.mal > monetdb5/mal/Tests/tst267.stable.err > monetdb5/mal/Tests/tst267.stable.out > monetdb5/modules/kernel/array.c > monetdb5/modules/kernel/array.h > monetdb5/modules/kernel/array.mal > monetdb5/optimizer/Tests/AEexample.mal > monetdb5/optimizer/Tests/AEexample.stable.err > monetdb5/optimizer/Tests/AEexample.stable.out > monetdb5/optimizer/Tests/GCexample.mal > monetdb5/optimizer/Tests/GCexample.stable.err > monetdb5/optimizer/Tests/GCexample.stable.out > monetdb5/optimizer/Tests/tst4800.mal > monetdb5/optimizer/Tests/tst4800.stable.err > monetdb5/optimizer/Tests/tst4800.stable.out > monetdb5/optimizer/Tests/tst4801.mal > monetdb5/optimizer/Tests/tst4801.stable.err > monetdb5/optimizer/Tests/tst4801.stable.out > monetdb5/optimizer/Tests/tst4810.mal > monetdb5/optimizer/Tests/tst4810.stable.err > monetdb5/optimizer/Tests/tst4810.stable.out > monetdb5/optimizer/Tests/tst4900.mal > monetdb5/optimizer/Tests/tst4900.stable.err > monetdb5/optimizer/Tests/tst4900.stable.out > Modified Files: > clients/Tests/MAL-signatures.stable.out > clients/Tests/MAL-signatures_gsl.stable.out > clients/Tests/MAL-signatures_nocfitsio.stable.out > clients/Tests/MAL-signatures_sphinxclient.stable.out > clients/Tests/exports.stable.out > monetdb5/modules/kernel/Makefile.ag > Branch: default > Log Message: > > Removed Alex's array implementation > > > diffs (truncated from 2760 to 300 lines): > > diff --git a/clients/Tests/MAL-signatures.stable.out b/clients/Tests/MAL-signatures.stable.out > --- a/clients/Tests/MAL-signatures.stable.out > +++ b/clients/Tests/MAL-signatures.stable.out > @@ -2849,46 +2849,6 @@ command alarm.usec{unsafe}():lng > address ALARMusec; > comment return cpu microseconds info > > -command array.grid(b:bat[:oid,:lng],i:lng,j:lng,k:lng,l:lng,s:lng):bat[:oid,:lng] > -address ARRAYgridBATshift_lng; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elemenets with a factor s > - > -command array.grid(b:bat[:oid,:lng],i:lng,j:lng,k:lng,l:lng):bat[:oid,:lng] > -address ARRAYgridBAT_lng; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(i:lng,j:lng,k:lng,l:lng,s:lng):bat[:oid,:lng] > -address ARRAYgridShift_lng; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elements with a factor s > - > -command array.grid(i:lng,j:lng,k:lng,l:lng):bat[:oid,:lng] > -address ARRAYgrid_lng; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(b:bat[:oid,:int],i:int,j:int,k:int,l:int,s:int):bat[:oid,:int] > -address ARRAYgridBATshift_int; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elemenets with a factor s > - > -command array.grid(b:bat[:oid,:int],i:int,j:int,k:int,l:int):bat[:oid,:int] > -address ARRAYgridBAT_int; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(i:int,j:int,k:int,l:int,s:int):bat[:oid,:int] > -address ARRAYgridShift_int; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elements with a factor s > - > -command array.grid(i:int,j:int,k:int,l:int):bat[:oid,:int] > -address ARRAYgrid_int; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.project(b:bat[:oid,:any_1],i:int):bat[:oid,:int] > -address ARRAYproject; > -comment Fill an array representation with constants > - > -command array.product(b:bat[:oid,:any_1],c:bat[:oid,:any_2]) (X_3:bat[:oid,:any_1],c:bat[:oid,:any_2]) > -address ARRAYproduct; > -comment Produce an array product > - > command batudf.fuse(one:bat[:oid,:int],two:bat[:oid,:int]):bat[:oid,:lng] > address UDFBATfuse; > comment fuse two (4-byte) int values into one (8-byte) lng value > diff --git a/clients/Tests/MAL-signatures_gsl.stable.out b/clients/Tests/MAL-signatures_gsl.stable.out > --- a/clients/Tests/MAL-signatures_gsl.stable.out > +++ b/clients/Tests/MAL-signatures_gsl.stable.out > @@ -2849,46 +2849,6 @@ command alarm.usec{unsafe}():lng > address ALARMusec; > comment return cpu microseconds info > > -command array.grid(b:bat[:oid,:lng],i:lng,j:lng,k:lng,l:lng,s:lng):bat[:oid,:lng] > -address ARRAYgridBATshift_lng; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elemenets with a factor s > - > -command array.grid(b:bat[:oid,:lng],i:lng,j:lng,k:lng,l:lng):bat[:oid,:lng] > -address ARRAYgridBAT_lng; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(i:lng,j:lng,k:lng,l:lng,s:lng):bat[:oid,:lng] > -address ARRAYgridShift_lng; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elements with a factor s > - > -command array.grid(i:lng,j:lng,k:lng,l:lng):bat[:oid,:lng] > -address ARRAYgrid_lng; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(b:bat[:oid,:int],i:int,j:int,k:int,l:int,s:int):bat[:oid,:int] > -address ARRAYgridBATshift_int; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elemenets with a factor s > - > -command array.grid(b:bat[:oid,:int],i:int,j:int,k:int,l:int):bat[:oid,:int] > -address ARRAYgridBAT_int; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(i:int,j:int,k:int,l:int,s:int):bat[:oid,:int] > -address ARRAYgridShift_int; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elements with a factor s > - > -command array.grid(i:int,j:int,k:int,l:int):bat[:oid,:int] > -address ARRAYgrid_int; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.project(b:bat[:oid,:any_1],i:int):bat[:oid,:int] > -address ARRAYproject; > -comment Fill an array representation with constants > - > -command array.product(b:bat[:oid,:any_1],c:bat[:oid,:any_2]) (X_3:bat[:oid,:any_1],c:bat[:oid,:any_2]) > -address ARRAYproduct; > -comment Produce an array product > - > command batudf.fuse(one:bat[:oid,:int],two:bat[:oid,:int]):bat[:oid,:lng] > address UDFBATfuse; > comment fuse two (4-byte) int values into one (8-byte) lng value > diff --git a/clients/Tests/MAL-signatures_nocfitsio.stable.out b/clients/Tests/MAL-signatures_nocfitsio.stable.out > --- a/clients/Tests/MAL-signatures_nocfitsio.stable.out > +++ b/clients/Tests/MAL-signatures_nocfitsio.stable.out > @@ -2849,46 +2849,6 @@ command alarm.usec{unsafe}():lng > address ALARMusec; > comment return cpu microseconds info > > -command array.grid(b:bat[:oid,:lng],i:lng,j:lng,k:lng,l:lng,s:lng):bat[:oid,:lng] > -address ARRAYgridBATshift_lng; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elemenets with a factor s > - > -command array.grid(b:bat[:oid,:lng],i:lng,j:lng,k:lng,l:lng):bat[:oid,:lng] > -address ARRAYgridBAT_lng; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(i:lng,j:lng,k:lng,l:lng,s:lng):bat[:oid,:lng] > -address ARRAYgridShift_lng; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elements with a factor s > - > -command array.grid(i:lng,j:lng,k:lng,l:lng):bat[:oid,:lng] > -address ARRAYgrid_lng; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(b:bat[:oid,:int],i:int,j:int,k:int,l:int,s:int):bat[:oid,:int] > -address ARRAYgridBATshift_int; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elemenets with a factor s > - > -command array.grid(b:bat[:oid,:int],i:int,j:int,k:int,l:int):bat[:oid,:int] > -address ARRAYgridBAT_int; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(i:int,j:int,k:int,l:int,s:int):bat[:oid,:int] > -address ARRAYgridShift_int; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elements with a factor s > - > -command array.grid(i:int,j:int,k:int,l:int):bat[:oid,:int] > -address ARRAYgrid_int; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.project(b:bat[:oid,:any_1],i:int):bat[:oid,:int] > -address ARRAYproject; > -comment Fill an array representation with constants > - > -command array.product(b:bat[:oid,:any_1],c:bat[:oid,:any_2]) (X_3:bat[:oid,:any_1],c:bat[:oid,:any_2]) > -address ARRAYproduct; > -comment Produce an array product > - > command batudf.fuse(one:bat[:oid,:int],two:bat[:oid,:int]):bat[:oid,:lng] > address UDFBATfuse; > comment fuse two (4-byte) int values into one (8-byte) lng value > diff --git a/clients/Tests/MAL-signatures_sphinxclient.stable.out b/clients/Tests/MAL-signatures_sphinxclient.stable.out > --- a/clients/Tests/MAL-signatures_sphinxclient.stable.out > +++ b/clients/Tests/MAL-signatures_sphinxclient.stable.out > @@ -2849,46 +2849,6 @@ command alarm.usec{unsafe}():lng > address ALARMusec; > comment return cpu microseconds info > > -command array.grid(b:bat[:oid,:lng],i:lng,j:lng,k:lng,l:lng,s:lng):bat[:oid,:lng] > -address ARRAYgridBATshift_lng; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elemenets with a factor s > - > -command array.grid(b:bat[:oid,:lng],i:lng,j:lng,k:lng,l:lng):bat[:oid,:lng] > -address ARRAYgridBAT_lng; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(i:lng,j:lng,k:lng,l:lng,s:lng):bat[:oid,:lng] > -address ARRAYgridShift_lng; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elements with a factor s > - > -command array.grid(i:lng,j:lng,k:lng,l:lng):bat[:oid,:lng] > -address ARRAYgrid_lng; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(b:bat[:oid,:int],i:int,j:int,k:int,l:int,s:int):bat[:oid,:int] > -address ARRAYgridBATshift_int; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elemenets with a factor s > - > -command array.grid(b:bat[:oid,:int],i:int,j:int,k:int,l:int):bat[:oid,:int] > -address ARRAYgridBAT_int; > -comment Fills an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.grid(i:int,j:int,k:int,l:int,s:int):bat[:oid,:int] > -address ARRAYgridShift_int; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset)and shift all elements with a factor s > - > -command array.grid(i:int,j:int,k:int,l:int):bat[:oid,:int] > -address ARRAYgrid_int; > -comment Generates an index BAT, (grpcount,grpsize,clustersize,offset) > - > -command array.project(b:bat[:oid,:any_1],i:int):bat[:oid,:int] > -address ARRAYproject; > -comment Fill an array representation with constants > - > -command array.product(b:bat[:oid,:any_1],c:bat[:oid,:any_2]) (X_3:bat[:oid,:any_1],c:bat[:oid,:any_2]) > -address ARRAYproduct; > -comment Produce an array product > - > command batudf.fuse(one:bat[:oid,:int],two:bat[:oid,:int]):bat[:oid,:lng] > address UDFBATfuse; > comment fuse two (4-byte) int values into one (8-byte) lng value > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -838,21 +838,6 @@ str ALGuselect1(int *result, int *bid, p > str ALGuselectInclusive(int *result, int *bid, ptr low, ptr high, bit *lin, bit *rin); > str ALGvariance(dbl *res, int *bid); > str ALGvariancep(dbl *res, int *bid); > -str ARRAYgridBAT_int(int *ret, int *bid, int *groups, int *groupsize, int *clustersize, int *offset); > -str ARRAYgridBAT_lng(lng *ret, lng *bid, lng *groups, lng *groupsize, lng *clustersize, lng *offset); > -str ARRAYgridBATshift_int(int *ret, int *bid, int *groups, int *groupsize, int *clustersize, int *offset, int *shift); > -str ARRAYgridBATshift_lng(lng *ret, lng *bid, lng *groups, lng *groupsize, lng *clustersize, lng *offset, lng *shift); > -str ARRAYgridShift_int(int *ret, int *groups, int *groupsize, int *clustersize, int *offset, int *shift); > -str ARRAYgridShift_lng(lng *ret, lng *groups, lng *groupsize, lng *clustersize, lng *offset, lng *shift); > -str ARRAYgrid_int(int *ret, int *groups, int *groupsize, int *clustersize, int *offset); > -str ARRAYgrid_lng(lng *ret, lng *groups, lng *groupsize, lng *clustersize, lng *offset); > -str ARRAYmultiply_int_int(int *ret, int *bid, int *rid); > -str ARRAYmultiply_int_lng(int *ret, int *bid, int *rid); > -str ARRAYmultiply_lng_lng(int *ret, int *bid, int *rid); > -str ARRAYmultiply_sht_int(int *ret, int *bid, int *rid); > -str ARRAYmultiply_sht_lng(int *ret, int *bid, int *rid); > -str ARRAYproduct(int *ret, int *ret2, int *bid, int *rid); > -str ARRAYproject(int *ret, int *bid, int *cst); > str AUTHaddUser(oid *ret, Client *c, str *user, str *pass); > str AUTHchangePassword(Client *c, str *oldpass, str *passwd); > str AUTHchangeUsername(Client *c, str *olduser, str *newuser); > diff --git a/monetdb5/mal/Tests/tst267.mal b/monetdb5/mal/Tests/tst267.mal > deleted file mode 100644 > --- a/monetdb5/mal/Tests/tst267.mal > +++ /dev/null > @@ -1,14 +0,0 @@ > -# Grouped sums > -a:= bat.new(:oid,:int); > -b:= bat.new(:oid,:int); > -bat.append(a,1); > -bat.append(a,2); > -bat.append(b,5); > -bat.append(b,6); > -bat.append(b,7); > -bat.append(b,8); > -io.print(a); > -io.print(b); > - > -(g,h):= array.product(a,b); > -io.print(g,h); > diff --git a/monetdb5/mal/Tests/tst267.stable.err b/monetdb5/mal/Tests/tst267.stable.err > deleted file mode 100644 > --- a/monetdb5/mal/Tests/tst267.stable.err > +++ /dev/null > @@ -1,12 +0,0 @@ > -stderr of test 'tst267` in directory 'monetdb5/mal` itself: > - > - > -# 20:55:59 > > -# 20:55:59 > Mtimeout -timeout 60 Mserver "--config=/ufs/mk/monet5/src/mal/Tests/All.conf" --debug=10 --set "monet_mod_path=/ufs/mk/monet5/Linux/lib/MonetDB5:/ufs/mk/opensource/MonetDB/Linux/lib/MonetDB" --set "gdk_dbfarm=/ufs/mk/opensource/MonetDB/Linux/var/MonetDB5/dbfarm" --set "sql_logdir=/ufs/mk/opensource/MonetDB/Linux/var/MonetDB5/log" --set mapi_port=40036 --set sql_port=52173 --set monet_prompt= --trace "--config=/ufs/mk/monet5/src/mal/Tests/All.conf" --dbname=mTests_src_mal tst267.mal </dev/null > -# 20:55:59 > > - > - > -# 20:55:59 > > -# 20:55:59 > Done. > -# 20:55:59 > > - > diff --git a/monetdb5/mal/Tests/tst267.stable.out b/monetdb5/mal/Tests/tst267.stable.out > deleted file mode 100644 > --- a/monetdb5/mal/Tests/tst267.stable.out > +++ /dev/null > @@ -1,72 +0,0 @@ > -stdout of test 'tst267` in directory 'monetdb5/mal` itself: > - > - > -# 20:55:59 > > -# 20:55:59 > Mtimeout -timeout 60 Mserver "--config=/ufs/mk/monet5/src/mal/Tests/All.conf" --debug=10 --set "monet_mod_path=/ufs/mk/monet5/Linux/lib/MonetDB5:/ufs/mk/opensource/MonetDB/Linux/lib/MonetDB" --set "gdk_dbfarm=/ufs/mk/opensource/MonetDB/Linux/var/MonetDB5/dbfarm" --set "sql_logdir=/ufs/mk/opensource/MonetDB/Linux/var/MonetDB5/log" --set mapi_port=40036 --set sql_port=52173 --set monet_prompt= --trace "--config=/ufs/mk/monet5/src/mal/Tests/All.conf" --dbname=mTests_src_mal tst267.mal </dev/null > -# 20:55:59 > > - > -# Monet Database Server V4.7.0 > -# Copyright (c) 1993-2005, CWI. All rights reserved. > -# Compiled for i686-redhat-linux-gnu/32bit; dynamically linked. > -# config:/ufs/mk/monet5/src/mal/Tests/All.conf > -# dbfarm:/ufs/mk/opensource/MonetDB/Linux/var/MonetDB5/dbfarm > -# dbname:mTests_src_mal > -# Visit http://monetdb.cwi.nl/ for further information. > -## Grouped sums > -#a:= new(:void,:int); > -#setSequenceBase(a,0:oid); > -#b:= new(:void,:int); > -#setSequenceBase(b,0:oid); > -#insert(a,1); > -#insert(a,2); > -#insert(b,3); > -#insert(b,4); > -#insert(b,5); > -#print(a); > -#print(b); > -# > -#g:= array.*(a,b); > -#print(g); > -# > -function user.main():void; > -# Grouped sums > - a := bat.new(:oid,:int); > - b := bat.new(:oid,:int); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 1

Announcement: New Jan2014-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 31 Jul '14

31 Jul '14

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jan2014-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jan 2014-SP3 bugfix release Build Environment * Fix configure to continue without Python if the python binary is too old. This instead of always aborting configure if python happens to be too old. Bug Fixes * 3487: dead link to "Professional services" * 3500: MonetDB driver wants an empty string for SQLTables and SQLColumns API calls, where other drivers expect NULL * 3514: mserver5 crash due (assertion failure in gdk_select.c) * 3515: mserver5 crash due (assertion failure in gdk_bat.c) * 3519: Uppercase TRUE/FALSE strings cannot be converted to boolean values

1 0

Re: MonetDB: transaction-replication - Silence compiler
by Sjoerd Mullender 21 Jul '14

21 Jul '14

Not the right fix. __int64_t is not portable. In this case, since strings are never so long, a cast to (int) would have been good enough. You could also use ptrdiff_t which is the actual (standard C) type for the difference of pointers. On 2014-07-20 21:52, Dimitar Nedev wrote: > Changeset: 697c8e1dfbac for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=697c8e1dfbac > Modified Files: > gdk/gdk_storage.c > Branch: transaction-replication > Log Message: > > Silence compiler > Avoid conversion from int64 to int > > > diffs (21 lines): > > diff --git a/gdk/gdk_storage.c b/gdk/gdk_storage.c > --- a/gdk/gdk_storage.c > +++ b/gdk/gdk_storage.c > @@ -100,7 +100,7 @@ GDKfilepath(int farmid, const char *dir, > char * > GDKfilepath_long(int farmid, const char *dir, const char *ext) { > char *last_dir_with_sep; > - int last_dirsep_index; > + __int64_t last_dirsep_index; > char *last_dir; > char *last_dir_parent; > > @@ -258,7 +258,7 @@ GDKfileopen(int farmid, const char * dir > /* if name is null, try to get one from dir (in case it was a path) */ > if ((name == NULL) || (*name == 0)) { > char *last_dir_with_sep; > - int last_dirsep_index; > + __int64_t last_dirsep_index; > char *last_dir; > char *last_dir_parent; > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 1

Announcement: New Jan2014-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 02 Jun '14

02 Jun '14

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jan2014-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jan 2014-SP2 bugfix release Build Environment * Lots of minor fixes were made for potential defects found by Coverity Scan. * On Windows we now build the geom module against version 3.4.2 of the geos library. Client Package * ODBC: Implemented {call procedure-name(...)} escape. The version {?=call ...} is not implemented. Bug Fixes * 3424: numeric values at the front of strings determines whether CAST works successfully * 3439: Python driver drops milliseconds from timestamps * 3446: SET READ ONLY forgets previous changes * 3455: String columns unusable from 64-bit .NET via ODBC * 3456: Insert fails * 3457: When kernel of remote client crashes, the connection remains established on server side * 3458: mserver5 crash on SQL: SELECT COUNT(*) FROM SYS.TABLES HAVING COUNT(*) > 0 * 3461: mserver5 crash on SQL: SELECT * FROM SYS.ARGS WHERE FUNC_ID NOT IN (SELECT ID FROM SYS.FUNCTIONS) OR FUNC_ID NOT IN (SELECT * FROM SYS.FUNCTIONS) * 3462: Invalid SQL (IN with subquery which returns multiple columns) is accepted * 3463: Crash on SELECT with SERIAL aggregation and GROUP BY column alias's * 3468: Local temporary table persists across sessions * 3469: Absolute network paths considered invalid for COPY INTO ... FROM statement. * 3473: Various memory leaks in SQL compilation * 3477: ODBC driver raises "unexpected end of input" for prepared string parameter from .NET application * 3481: Cannot run multiple COPY INTO statements in one 's'-command * 3482: Crossproduct error

2 1

TPC-H in MALs
by Qing Guo 21 Apr '14

21 Apr '14

Hi, I was trying to run “mserver” (standalone) in a simulator to do some performance study. As mserver only reads MALs, are there any ready-to-use MAL scripts for TPC-H? I tried to use “explain” in “mclient”, but the translated MALs seem to be too detailed and cannot simply copy and run. Any idea on this? Thank you! -Kevin

1 0

headless documentation
by Fabian Groffen 14 Mar '14

14 Mar '14

Hi devs, Is there any documentation for headless? Not that it seems completely pointless, but how exactly is it supposed to work? I'm looking for documentation that explains why, but preferably also old situation and how it should be converted into new situation kind of examples. Things like: can I simply insert into a void-headed BAT to automatically create a BAT with gaps if I e.g. insert 1, 5 and 7. Can I do a selection on the head (which should be a simple offset jump) to easily retrieve its tail, and how does the subselect interface exactly work. There appears to be not even comments on this piece of code. Regards from Fuerteventura, Fabian

2 1

Re: MonetDB: Jan2014 - configure.ag: revised way to determine compil...
by Fabian Groffen 11 Mar '14

11 Mar '14

On 05-03-2014 08:25:38 +0100, Stefan Manegold wrote: > Changeset: 36afc568f849 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=36afc568f849 > Modified Files: > configure.ag > Branch: Jan2014 > Log Message: > > configure.ag: revised way to determine compiler type (gcc vs. clang vs. icc): > in particular do not (only) rely on the result of AC_PROG_CC() (i.e., $GCC) > or the name of the call; > background is that on Mac OSX Mavericks, Apple prestends that their clang > were gcc ... FYI: they were pretending they had a GCC since OSX Lion. Fabian

4 14

Announcement: New Jan2014-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 10 Mar '14

10 Mar '14

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jan2014-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jan 2014-SP1 bugfix release Build Environment * Configure now enables the SQL front end by default. Merovingian * monetdb destroy -f now also works on running databases Bug Fixes * 3442: COPY INTO ... LOCKED reports incorrect count * 3443: DROP INDEX crashes server with BATsubselect: invalid argument: b must have a dense head * 3444: AND after ON () of LEFT OUTER JOIN with certain expressions will cause crash * 3452: ODBC driver build fails on Mac OS X due to a conflicting types for the SQLColAttribute with the unixODBC library

1 0

Re: MonetDB: Jan2014 - configure: don't choke on deprecation messages
by Stefan Manegold 06 Mar '14

06 Mar '14

Hi Fabian, thank you very much for your fix! Do I understand correctly that the problem is in fact related to (recent versions of) MacOSX, and the packages / libraries that come with it, rather than the compiler itself? If so, we might want to consider adding option "-Wno-deprecated-declarations" only on (the respective versions of) MacOSX, --- or only when there is Apple's CommonCrypto API rather than OpenSSL, provided we can easily detect that --- rather than whenever using any version of clang on any system (and with old versions of gcc on any system). Yes, I still have the ambition to disable strict compilation checks only as selectively as possible, even on "odd" platforms ... ;-) Thanks! Stefan ----- Original Message ----- > Changeset: b4d844547d39 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=b4d844547d39 > Modified Files: > configure.ag > Branch: Jan2014 > Log Message: > > configure: don't choke on deprecation messages > > Apple "deprecated" OpenSSL since Lion, don't complain about that. > > > diffs (15 lines): > > diff --git a/configure.ag b/configure.ag > --- a/configure.ag > +++ b/configure.ag > @@ -598,8 +598,11 @@ yes-*-*) > dnl not all) false positives, though, as well as in > dnl YACC/BISON-generated code; thus, we do not use > dnl "-Wunreachable-code" with clang and gcc < 4.5 . > + dnl At the same time, Apple deprecated OpenSSL in favour of > + dnl their own CommonCrypto API. We don't mind about that. > case "$CC_ver" in > clang-*|gcc-[[0-3]].*|gcc-4.[[0-4]].*) > + MCHECK_ADD_FLAG([-Wno-deprecated-declarations]) > ;; > *) > MCHECK_ADD_FLAG([-Wunreachable-code]) > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

Re: MonetDB: Jan2014 - configure.ag: with clang, disable -pthread as...
by Fabian Groffen 05 Mar '14

05 Mar '14

On 05-03-2014 08:25:38 +0100, Stefan Manegold wrote: > Changeset: a06afa30eeea for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a06afa30eeea > Modified Files: > configure.ag > Branch: Jan2014 > Log Message: > > configure.ag: with clang, disable -pthread as compiler option > (but keep it as linker option) > > Also clang 3.3 on fedora 20 appears to work fine without > -pthread as compiler option (but requires it as linker option). from man gcc: -pthread Adds support for multithreading with the pthreads library. This option sets flags for both the preprocessor and linker. This is typically something like _REENTRANT. On Mountain Lion, gcc/clang would set _REENTRANT. Same for CentOS 6. % /usr/bin/gcc -dumpspecs | grep pthread %{static:%{!dynamic:-D__STATIC__}}%{!static:-D__DYNAMIC__} %%{pthread:-D_REENTRANT} What I can imagine here, is that because clang on Mavericks now defaults to C99 mode (instead of gnu89 mode gcc defaults to) you get different defines and behaviour, possibly including _REENTRANT. A preprocessor check would have to give some facts around that. Fabian

1 0

status of default branch
by Fabian Groffen 05 Mar '14

05 Mar '14

Hi all, Since I'm no longer really in the loop with MonetDB, I was wondering what the state of the default branch is supposed to be. I just fixed a configure error and now hit a redeclaration problem regarding gdk.h and gdk_private.h. gdk_batop.c includes both headers so it seems like a valid complaint to me. Is OSX still a supported platform, these days? I guess there is little to no point in supporting it. Fabian

5 20

Building the ODBC driver on OS X 10.9.2
by Dimitar Nedev 04 Mar '14

04 Mar '14

Hello everybody, I managed to build the latest MonetDB version on my Mac with all default modules, excluding the DataCell and RDF, but the ODBC build fails. Trying to fix it I encountered several issues: A few libraries could not be found with the default build settings: hwcounters = no (no supported harwdware counters interface found) mseed = no (libmseed.h header not found) raptor = no (by default) readline = no (readline/readline.h does not contain rl_completion_func_t, is it GNU readline?) setsockopt = no (by default) sphinxclient = no (sphinxclient.h header not found) unixodbc = no (odbcinst.h not found) valgrind = no (by default) So far it makes sense, since unixODBC cannot be found, the ODBC driver cannot be built. Just that I've actually installed the library. Adding the following line: --with-unixodbc=/opt/local --with-sphinxclient=/opt/local --with-readline=/opt/local to point to some of the libs location appeared to fix the configuration: checking odbcinst.h usability... yes checking odbcinst.h presence... yes checking for odbcinst.h... yes checking for SQLGetPrivateProfileString in -lodbcinst... yes checking for SQLGetDiagRec in -lodbc... yes checking checking whether SQLColAttribute uses SQLLEN * or SQLPOINTER... using "SQLPOINTER" hwcounters = no (no supported harwdware counters interface found) mseed = no (libmseed.h header not found) raptor = no (by default) setsockopt = no (by default) valgrind = no (by default) Now, trying to build that configuration yields the following error: libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../../clients/odbc/driver -I../../.. -I../../../../clients/odbc/driver -I../../mapilib -I../../../../clients/odbc/driver/../../mapilib -I../../../common/options -I../../../../clients/odbc/driver/../../../common/options -I/opt/local/include -DLIBMONETODBC -g -O2 -I/opt/local/include -c ../../../../clients/odbc/driver/SQLColAttributes.c -fno-common -DPIC -o .libs/libMonetODBC_la-SQLColAttributes.o ../../../../clients/odbc/driver/SQLColAttribute.c:243:1: error: conflicting types for 'SQLColAttribute' SQLColAttribute(SQLHSTMT StatementHandle, ^ /opt/local/include/sql.h:609:24: note: previous declaration is here SQLRETURN SQL_API SQLColAttribute(SQLHSTMT StatementHandle, ^ ../../../../clients/odbc/driver/SQLColAttribute.c:272:1: error: conflicting types for 'SQLColAttributeA' SQLColAttributeA(SQLHSTMT StatementHandle, ^ /opt/local/include/sqlucode.h:377:19: note: previous declaration is here SQLRETURN SQL_API SQLColAttributeA( ^ ../../../../clients/odbc/driver/SQLColAttribute.c:290:1: error: conflicting types for 'SQLColAttributeW' SQLColAttributeW(SQLHSTMT StatementHandle, ^ /opt/local/include/sqlucode.h:29:19: note: previous declaration is here SQLRETURN SQL_API SQLColAttributeW( ^ 3 errors generated. make[7]: *** [libMonetODBC_la-SQLColAttribute.lo] Error 1 I am running: OS X 10.9.2 / Darwin Kernel Version 13.1.0 x86_64 Xcode 5.0.2 and gcc -v reports: Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) unixODBC 2.3.1_0 installed from MacPorts 2.2.1 Any ideas? Thanks, Dimitar

2 2

Re: MonetDB: rdf - Generate sample data for all tables
by Lefteris 26 Feb '14

26 Feb '14

Hi Duc, I did not read in detail your function, but you can also use BATsample_ (which is the void headed version of BATsample) lefteris On Wed, Feb 26, 2014 at 2:10 PM, Minh-Duc Pham <commits(a)monetdb.org> wrote: > Changeset: e7109fc24610 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e7109fc24610 > Modified Files: > monetdb5/extras/rdf/rdfschema.c > monetdb5/extras/rdf/rdfschema.h > Branch: rdf > Log Message: > > Generate sample data for all tables > > > diffs (truncated from 721 to 300 lines): > > diff --git a/monetdb5/extras/rdf/rdfschema.c b/monetdb5/extras/rdf/rdfschema.c > --- a/monetdb5/extras/rdf/rdfschema.c > +++ b/monetdb5/extras/rdf/rdfschema.c > @@ -695,6 +695,11 @@ char isInfrequentSampleProp(CS freqCS, i > if (freqCS.lstPropSupport[propIdx] * 100 < freqCS.support * SAMPLE_FILTER_THRESHOLD) return 1; > else return 0; > } > +static > +char isInfrequentSampleCol(CS freqCS, PropTypes pt){ > + if (pt.propFreq * 100 < freqCS.support * SAMPLE_FILTER_THRESHOLD) return 1; > + else return 0; > +} > > static > void genCSPropTypesColIdx(CSPropTypes* csPropTypes, int numMergedCS, CSset* freqCSset){ > @@ -4830,6 +4835,211 @@ void initSampleData(CSSample *csSample,B > > } > } > + > + > +static > +void getSubjIdFromTablePosition(int tblIdx, int pos, oid *sOid){ > + oid id; > + id = pos; > + id |= (BUN)(tblIdx + 1) << (sizeof(BUN)*8 - NBITS_FOR_CSID); > + *sOid = id; > +} > + > +static > +str getOrigSbt(oid *sbt, oid *origSbt, BAT *lmap, BAT *rmap){ > + BUN pos; > + oid *tmp; > + pos = BUNfnd(BATmirror(rmap),sbt); > + if (pos == BUN_NONE){ > + throw(RDF, "rdf.RDFdistTriplesToCSs", "This encoded subject must be in rmap"); > + } > + tmp = (oid *) Tloc(lmap, pos); > + if (*tmp == BUN_NONE){ > + throw(RDF, "rdf.RDFdistTriplesToCSs", "The encoded subject must be in lmap"); > + } > + > + *origSbt = *tmp; > + > + return MAL_SUCCEED; > +} > + > +static > +str getOrigObt(oid *obt, oid *origObt, BAT *lmap, BAT *rmap){ > + BUN pos; > + oid *tmp; > + oid tmporigOid = BUN_NONE; > + char objType; > + BUN maxObjectURIOid = ((oid)1 << (sizeof(BUN)*8 - NBITS_FOR_CSID - 1)) - 1; //Base on getTblIdxFromS > + > + objType = getObjType(*obt); > + > + if (objType == URI || objType == BLANKNODE){ > + tmporigOid = (*obt) - ((oid)objType << (sizeof(BUN)*8 - 4)); > + } > + > + if (tmporigOid > maxObjectURIOid){ > + pos = BUNfnd(BATmirror(rmap),&tmporigOid); > + if (pos == BUN_NONE){ > + throw(RDF, "rdf.RDFdistTriplesToCSs", "This encoded object must be in rmap"); > + } > + tmp = (oid *) Tloc(lmap, pos); > + if (*tmp == BUN_NONE){ > + throw(RDF, "rdf.RDFdistTriplesToCSs", "The encoded object must be in lmap"); > + } > + > + *origObt = *tmp; > + } > + else{ > + *origObt = tmporigOid; > + } > + > + return MAL_SUCCEED; > +} > + > +static > +str initFullSampleData(CSSampleExtend *csSampleEx, int *mTblIdxFreqIdxMapping, CSlabel *label, CStableStat* cstablestat, CSPropTypes *csPropTypes, CSset *freqCSset, int numTables, bat *lmapbatid, bat *rmapbatid){ > + int i, j, k; > + int freqId; > + int tmpNumcand; > + oid tmpCandidate; > + int randValue = 0; > + int ranPosition = 0; //random position of the instance in a table > + int tmpNumCols; > + int colIdx; > + BAT *tmpbat = NULL; > + BATiter tmpi; > + BAT *cursamplebat = NULL; > + int tmpNumRows = 0; > + oid tmpSoid = BUN_NONE, origSoid = BUN_NONE; > + oid origOid = BUN_NONE; > + BAT *lmap = NULL, *rmap = NULL; > + > + if ((lmap = BATdescriptor(*lmapbatid)) == NULL) { > + throw(MAL, "rdf.RDFdistTriplesToCSs", RUNTIME_OBJECT_MISSING); > + } > + > + if ((rmap = BATdescriptor(*rmapbatid)) == NULL) { > + BBPreleaseref(lmap->batCacheid); > + throw(MAL, "rdf.RDFdistTriplesToCSs", RUNTIME_OBJECT_MISSING); > + } > + srand(123456); > + for (i = 0; i < numTables; i++){ > + freqId = mTblIdxFreqIdxMapping[i]; > + csSampleEx[i].freqIdx = freqId; > + tmpNumcand = (NUM_SAMPLE_CANDIDATE > label[freqId].candidatesCount)?label[freqId].candidatesCount:NUM_SAMPLE_CANDIDATE; > + csSampleEx[i].name = cstablestat->lstcstable[i].tblname; > + csSampleEx[i].candidateCount = tmpNumcand; > + csSampleEx[i].candidates = (oid*)malloc(sizeof(oid) * tmpNumcand); > + for (k = 0; k < tmpNumcand; k++){ > + csSampleEx[i].candidates[k] = label[freqId].candidates[k]; > + } > + //Randomly exchange the value, change the position k with a random pos > + for (k = 0; k < tmpNumcand; k++){ > + randValue = rand() % tmpNumcand; > + tmpCandidate = csSampleEx[i].candidates[k]; > + csSampleEx[i].candidates[k] = csSampleEx[i].candidates[randValue]; > + csSampleEx[i].candidates[randValue] = tmpCandidate; > + } > + > + csSampleEx[i].lstSubjOid = (oid*)malloc(sizeof(oid) * NUM_SAMPLE_INSTANCE); > + for (k = 0; k < NUM_SAMPLE_INSTANCE; k++) > + csSampleEx[i].lstSubjOid[k] = BUN_NONE; > + > + tmpNumCols = csPropTypes[i].numProp - csPropTypes[i].numInfreqProp; //already remove infrequent column; > + csSampleEx[i].numProp = tmpNumCols; > + > + assert(tmpNumCols > 0); > + > + csSampleEx[i].lstProp = (oid*)malloc(sizeof(oid) * tmpNumCols); > + csSampleEx[i].lstIsInfrequentProp = (char*)malloc(sizeof(char) * tmpNumCols); > + csSampleEx[i].lstIsMVCol = (char*)malloc(sizeof(char) * tmpNumCols); > + csSampleEx[i].colBats = (BAT**)malloc(sizeof(BAT*) * tmpNumCols); > + colIdx = -1; > + csSampleEx[i].numInstances = 0; > + for(j = 0; j < csPropTypes[i].numProp; j++){ > + #if REMOVE_INFREQ_PROP > + if (csPropTypes[i].lstPropTypes[j].defColIdx == -1) continue; //Infrequent prop > + #endif > + colIdx++; > + csSampleEx[i].lstProp[colIdx] = csPropTypes[i].lstPropTypes[j].prop; > + > + csSampleEx[i].colBats[colIdx] = BATnew(TYPE_void, cstablestat->lstcstable[i].colBats[colIdx]->ttype , NUM_SAMPLE_INSTANCE + 1); > + > + //Mark whether this col is infrequent sample cols > + if ( isInfrequentSampleCol(freqCSset->items[freqId], csPropTypes[i].lstPropTypes[j])){ > + csSampleEx[i].lstIsInfrequentProp[colIdx] = 1; > + } > + else > + csSampleEx[i].lstIsInfrequentProp[colIdx] = 0; > + > + //Mark whther this col is a MV col > + csSampleEx[i].lstIsMVCol[colIdx] = csPropTypes[i].lstPropTypes[j].isMVProp; > + > + //if this is a multivalue column, get the data type of the first column > + > + } > + assert(colIdx == (tmpNumCols - 1)); > + > + > + // Inserting instances to csSampleEx > + > + tmpNumRows = BATcount(cstablestat->lstcstable[i].colBats[0]); > + > + for (k = 0; k < NUM_SAMPLE_INSTANCE; k++){ > + ranPosition = rand() % tmpNumRows; > + > + getSubjIdFromTablePosition(i, ranPosition, &tmpSoid); > + > + if (getOrigSbt(&tmpSoid, &origSoid, lmap, rmap) != MAL_SUCCEED){ > + throw(RDF, "rdf.RDFdistTriplesToCSs","Problem in getting the orignal sbt "); > + } > + > + csSampleEx[i].lstSubjOid[k] = origSoid; > + > + for (j = 0; j < tmpNumCols; j++){ > + cursamplebat = csSampleEx[i].colBats[j]; > + > + tmpbat = cstablestat->lstcstable[i].colBats[j]; > + tmpi = bat_iterator(tmpbat); > + > + if (tmpbat->ttype == TYPE_oid && csSampleEx[i].lstIsMVCol[j] == 0){ > + //Get the original object oid > + oid *tmpOid = (oid *) BUNtail(tmpi, ranPosition); > + if(*tmpOid != oid_nil){ > + if (getOrigObt(tmpOid, &origOid, lmap, rmap) != MAL_SUCCEED){ > + throw(RDF, "rdf.RDFdistTriplesToCSs","Problem in getting the orignal obt "); > + } > + BUNappend(cursamplebat, &origOid, TRUE); > + } > + else{ > + BUNappend(cursamplebat, ATOMnilptr(TYPE_oid), TRUE); > + } > + > + } > + else > + BUNappend(cursamplebat, BUNtail(tmpi, ranPosition), TRUE); > + > + > + > + } > + csSampleEx[i].numInstances++; > + } > + > + if (i == 0) > + for (j = 0; j < tmpNumCols; j++){ > + //BATprint(cstablestat->lstcstable[i].colBats[j]); > + BATprint(csSampleEx[i].colBats[j]); > + } > + > + } > + > + BBPunfix(lmap->batCacheid); > + BBPunfix(rmap->batCacheid); > + > + return MAL_SUCCEED; > + > +} > + > static > void freeSampleData(CSSample *csSample, int numCand){ > int i, j; > @@ -4846,6 +5056,25 @@ void freeSampleData(CSSample *csSample, > free(csSample); > } > > + > +static > +void freeSampleExData(CSSampleExtend *csSampleEx, int numCand){ > + int i, j; > + for (i = 0; i < numCand; i++){ > + free(csSampleEx[i].lstProp); > + free(csSampleEx[i].lstIsInfrequentProp); > + free(csSampleEx[i].lstIsMVCol); > + free(csSampleEx[i].candidates); > + free(csSampleEx[i].lstSubjOid); > + for (j = 0; j < csSampleEx[i].numProp; j++){ > + BBPunfix(csSampleEx[i].colBats[j]->batCacheid); > + } > + free(csSampleEx[i].colBats); > + } > + > + free(csSampleEx); > +} > + > static > void addSampleInstance(oid subj, oid *buffO, oid* buffP, int numP, int sampleIdx, CSSample *csSample){ > int i,j; > @@ -5217,6 +5446,295 @@ str printSampleData(CSSample *csSample, > return MAL_SUCCEED; > } > > +#if 0 > +static > +str printFullSampleData(CSSampleExtend *csSampleEx, CSset *freqCSset, BAT *mbat, int num, int sampleVersion){ > + > + int i,j, k; > + FILE *fout, *fouttb, *foutis; > + char filename[100], filename2[100], filename3[100]; > + char tmpStr[20], tmpStr2[20], tmpStr3[20]; > + int ret; > + > + str propStr; > + str subjStr; > + char* schema = "rdf"; > + CSSample sample; > + CS freqCS; > + char objType = 0; > + str objStr; > + oid objOid = BUN_NONE; > + BATiter mapi; > + str canStr; > + char isTitle = 0; > + char isUrl = 0; > + char isType = 0; > + char isDescription = 0; > + char isImage = 0; > + char isSite = 0; > + char isEmail = 0; > + char isCountry = 0; > + char isLocality = 0; > + BAT *lmap = NULL, *rmap = NULL > +#if USE_SHORT_NAMES > + str propStrShort = NULL; > + char *pch; > +#endif > + > + > + > + mapi = bat_iterator(mbat); > + > + if (TKNZRopen (NULL, &schema) != MAL_SUCCEED) { > + throw(RDF, "rdf.rdfschema", > + "could not open the tokenizer\n"); > + } > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 1

Re: MonetDB: default - Move tokenizer to the attic
by Stefan Manegold 24 Feb '14

24 Feb '14

Martin, et al., please be aware that this means that also monetdb5/extras/rdf (aka. --enable-rdf) does no longer compile, as it depends on the (now gone) tokenizer. Best, Stefan ----- Original Message ----- > Changeset: 155c3a3fcfdb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=155c3a3fcfdb > Removed Files: > monetdb5/modules/mal/tokenizer.c > monetdb5/modules/mal/tokenizer.h > monetdb5/modules/mal/tokenizer.mal > Modified Files: > monetdb5/modules/mal/Makefile.ag > Branch: default > Log Message: > > Move tokenizer to the attic > The experiment to organise urls using a variation of graph-based > tokenization requires a major overhaul to support the void-headed approach. > > > diffs (truncated from 737 to 300 lines): > > diff --git a/monetdb5/modules/mal/Makefile.ag > b/monetdb5/modules/mal/Makefile.ag > --- a/monetdb5/modules/mal/Makefile.ag > +++ b/monetdb5/modules/mal/Makefile.ag > @@ -54,7 +54,6 @@ lib_mal = { > sabaoth.c sabaoth.h \ > sysmon.c sysmon.h \ > tablet.c tablet.h \ > - tokenizer.c tokenizer.h \ > trader.c trader.h \ > transaction.c \ > txtsim.c txtsim.h \ > @@ -76,7 +75,7 @@ headers_mal = { > mal_mapi.mal sabaoth.mal remote.mal \ > txtsim.mal recycle.mal \ > cluster.mal trader.mal \ > - tokenizer.mal zorder.mal sample.mal json_util.mal \ > + zorder.mal sample.mal json_util.mal \ > calc.mal batcalc.mal batmtime.mal querylog.mal sysmon.mal > } > > @@ -84,7 +83,7 @@ EXTRA_DIST = batExtensions.mal iterator. > groupby.mal mal_init.mal manual.mal mkey.mal manifold.mal pcre.mal \ > profiler.mal recycle.mal remote.mal sabaoth.mal trader.mal \ > transaction.mal txtsim.mal tablet.mal tablet.h sample.mal json_util.mal \ > - mal_mapi.mal mat.mal tokenizer.mal pqueue.mal calc.mal \ > + mal_mapi.mal mat.mal pqueue.mal calc.mal \ > batcalc.mal batmtime.mal querylog.mal sysmon.mal > > EXTRA_DIST_DIR = Tests > diff --git a/monetdb5/modules/mal/tokenizer.c > b/monetdb5/modules/mal/tokenizer.c > deleted file mode 100644 > --- a/monetdb5/modules/mal/tokenizer.c > +++ /dev/null > @@ -1,585 +0,0 @@ > -/* > - * The contents of this file are subject to the MonetDB Public License > - * Version 1.1 (the "License"); you may not use this file except in > - * compliance with the License. You may obtain a copy of the License at > - * http://www.monetdb.org/Legal/MonetDBLicense > - * > - * Software distributed under the License is distributed on an "AS IS" > - * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the > - * License for the specific language governing rights and limitations > - * under the License. > - * > - * The Original Code is the MonetDB Database System. > - * > - * The Initial Developer of the Original Code is CWI. > - * Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > - * Copyright August 2008-2014 MonetDB B.V. > - * All Rights Reserved. > - */ > - > -/* > - * author Lefteris Sidirourgos > - * Tokenizer > - * This module implements a vertical fragmented tokenizer for strings. > - * It is based on the ideas of the urlbox module by mk. > - * > - * The input string is tokenized according to a separator character. > - * Each token is inserted to the next BAT with the same order of > - * appearance in the string. We currently support 255 tokens in each > - * string as this module is intended for use with short and similar > - * strings such as URLs. In addition we maintain a 2-dimensional index > - * that points to the depth and height of the last token of each string. > - * The 2-dimensional index is combined to one BAT where the 8 least > - * significant bits represent the depth, and the rest bits the height. > - * > - * The tokenizer can be accessed in two ways. Given the oid retrieve the > - * re-constructed string, or given a string return its oid if present, > - * otherwise nil. > - * > - * Strings can be added either in batch (from a file or a bat of > - * strings) and by appending a single string. Duplicate elimination is > - * always performed. > - * > - * There can be only one tokenizer open at the same time. This is > - * achieved by setting a TRANSaction bat. This might change in the > - * future. However there can be more than one tokenizers stored in the > - * disk, each of which is identified by its name (usually the name of > - * the active schema of the db). These administrative issues and > - * security aspects (e.g., opening a tokenizer of a different schema) > - * should be addressed more thoroughly. > - */ > -#include "monetdb_config.h" > -#include "bat5.h" > -#include "tokenizer.h" > -#include "mal_linker.h" > - > -#define MAX_TKNZR_DEPTH 256 > -#define INDEX MAX_TKNZR_DEPTH > -static int tokenDepth = 0; > -static BAT *tokenBAT[MAX_TKNZR_DEPTH + 1]; > -static BAT *TRANS = NULL; /* the catalog of tokenizers */ > -static char name[128]; > - > -#if SIZEOF_OID == 4 /* 32-bit oid */ > -#define MAX_h ((((oid) 1) << 23) - 1) > -#else /* 64-bit oid */ > -#define MAX_h ((((oid) 1) << 55) - 1) > -#endif > - > -#define COMP(h, d) ((h << 8) | (d & 255)) > -#define GET_d(x) ((sht) ((x) & 255)) > -#define GET_h(x) ((x) >> 8) > - > -static int prvlocate(BAT* b, oid *prv, str part) > -{ > - BAT *m = BATmirror(b); > - BATiter mi = bat_iterator(m); > - BUN p; > - if (m->H->hash == NULL) > - BAThash(m, 2 * BATcount(m)); > - HASHloop_str(mi, m->H->hash, p, part) > - { > - if (*((oid *) BUNtail(mi, p)) == *prv) { > - *prv = (oid) p; > - return TRUE; > - } > - } > - return FALSE; > -} > - > -str > -TKNZRopen(int *ret, str *in) > -{ > - int depth, r; > - bat idx; > - str batname = NULL; > - BAT *b; > - > - (void) ret; > - if (strlen(*in) > 127) > - throw(MAL, "tokenizer.open", > - ILLEGAL_ARGUMENT " tokenizer name too long"); > - > - MT_lock_set(&mal_contextLock, "tokenizer"); > - if (TRANS != NULL) { > - MT_lock_unset(&mal_contextLock, "tokenizer"); > - throw(MAL, "tokenizer.open", "Another tokenizer is already open"); > - } > - > - for (depth = 0; depth < MAX_TKNZR_DEPTH; depth++) { > - tokenBAT[depth] = 0; > - } > - tokenDepth = 0; > - > - TRANS = BATnew(TYPE_void, TYPE_str, MAX_TKNZR_DEPTH + 1); > - if (TRANS == NULL) { > - MT_lock_unset(&mal_contextLock, "tokenizer"); > - throw(MAL, "tokenizer.open", MAL_MALLOC_FAIL); > - } > - /* now we are sure that none overwrites the tokenizer table*/ > - MT_lock_unset(&mal_contextLock, "tokenizer"); > - BATseqbase(TRANS, 0); > - > - snprintf(name, 128, "%s", *in); > - batname = (str) GDKmalloc(134 * sizeof(char)); > - if( batname == NULL) > - throw(MAL, "tokenizer.open", MAL_MALLOC_FAIL); > - > - snprintf(batname, 134, "%s_index", name); > - idx = BBPindex(batname); > - > - if (idx == 0) { /* new tokenizer */ > - b = BATnew(TYPE_void, TYPE_oid, 1024); > - if (b == NULL) > - throw(MAL, "tokenizer.open", MAL_MALLOC_FAIL); > - BATkey(b, FALSE); > - BATseqbase(b, 0); > - tokenBAT[INDEX] = b; > - if (BKCsetName(&r, (int *) &(b->batCacheid), (str *) &batname) != > MAL_SUCCEED) > - throw(MAL, "tokenizer.open", OPERATION_FAILED); > - if (BKCsetPersistent(&r, (int *) &(b->batCacheid)) != MAL_SUCCEED) > - throw(MAL, "tokenizer.open", OPERATION_FAILED); > - BUNappend(TRANS, batname, FALSE); > - } else { /* existing tokenizer */ > - tokenBAT[INDEX] = BATdescriptor(idx); > - BUNappend(TRANS, batname, FALSE); > - > - for (depth = 0; depth < MAX_TKNZR_DEPTH; depth++) { > - snprintf(batname, 128, "%s_%d", name, depth); > - idx = BBPindex(batname); > - if (idx == 0) > - break; > - tokenBAT[depth] = BATdescriptor(idx); > - BUNappend(TRANS, batname, FALSE); > - } > - tokenDepth = depth; > - } > - > - GDKfree(batname); > - return MAL_SUCCEED; > -} > - > -str > -TKNZRclose(int *r) > -{ > - int i; > - (void) r; > - > - if (TRANS == NULL) > - throw(MAL, "tokenizer", "no tokenizer store open"); > - > - TMsubcommit(TRANS); > - > - for (i = 0; i < tokenDepth; i++) { > - BBPunfix(tokenBAT[i]->batCacheid); > - } > - BBPunfix(tokenBAT[INDEX]->batCacheid); > - > - tokenDepth = 0; > - > - BBPreclaim(TRANS); > - TRANS = NULL; > - return MAL_SUCCEED; > -} > - > -/* > - * Tokenize operations > - * The tokenizer operation assumes a private copy to mark the end of the > - * token separators with a zero byte. Tokens are separated by a single > - * character for simplicity. Might be a good scheme to assume that > - * strings to be broken are properly ended with either 0 or nl, not > - * both. It seems 0 can be assumed. > - */ > -static int > -TKNZRtokenize(str in, str *parts, char tkn) > -{ > - char *s, *t; > - int depth = 0; > - > - s = in; > - while (*s && *s != '\n') { > - t = s; > - while (*t != tkn && *t != '\n' && *t) > - t++; > - parts[depth++] = s; > - s = t + (*t != 0); > - *t = 0; > - if (depth > MAX_TKNZR_DEPTH) > - break; > - } > - return depth; > -} > - > -str > -TKNZRappend(oid *pos, str *s) > -{ > - str url; > - str batname; > - str parts[MAX_TKNZR_DEPTH]; > - int i, new, r, depth; > - BAT *b; > - BUN p; > - BUN idx = 0; > - oid prv = 0; > - oid comp; > - > - if (TRANS == NULL) > - throw(MAL, "tokenizer", "no tokenizer store open"); > - > - if ((url = GDKstrdup(*s)) == NULL) { > - throw(MAL, "tokenizer.append", OPERATION_FAILED MAL_MALLOC_FAIL); > - } > - > - depth = TKNZRtokenize(url, parts, '/'); > - new = depth; > - > - if (depth == 0) { > - GDKfree(url); > - return MAL_SUCCEED; > - } > - if (depth > MAX_TKNZR_DEPTH) { > - GDKfree(url); > - throw(MAL, "tokenizer", > - ILLEGAL_ARGUMENT "input string breaks to too many parts"); > - } > - if (depth > tokenDepth || tokenBAT[0] == NULL) { > - new = tokenDepth; > - for (i = tokenDepth; i < depth; i++) { > - /* make new bat */ > - batname = (str) GDKmalloc(128 * sizeof(char)); > - snprintf(batname, 128, "%s_%d", name, i); > - b = BATnew(TYPE_oid, TYPE_str, 1024); > - if (b == NULL) { > - GDKfree(batname); > - GDKfree(url); > - throw(MAL, "tokenizer.append", MAL_MALLOC_FAIL); > - } > - BATkey(b, FALSE); > - tokenBAT[i] = b; > - > - if (BKCsetName(&r, (int *) &(b->batCacheid), (str *) &batname) > - != MAL_SUCCEED) { > - GDKfree(batname); > - GDKfree(url); > - throw(MAL, "tokenizer.open", OPERATION_FAILED); > - } > - if (BKCsetPersistent(&r, (int *) &(b->batCacheid)) > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Announcement: New Jan2014 Feature release of MonetDB suite
by Sjoerd Mullender 20 Feb '14

20 Feb '14

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jan2014 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jan 2014 feature release Build Environment * Created development packages for RPM based systems and Debian/Ubuntu containing the files needed to create extensions to the SQL front end. * Removed Mx, the literate programming tool. All code for the server is now pure C. * Created packages for RPM based systems and Debian/Ubunty containing the MonetDB interface to the GNU Scientific Library (gsl). * We no longer install the .la files in our Fedora/Debian/Ubuntu packages. Client Package * ODBC: Implemented {fn scalar()} and {interval ...} escapes. python2 * Changed defaults for connecting (defaults to unix socket now). * Unix sockets partially working for control protocol. * Add support for unix socket. python3 * Changed defaults for connecting (defaults to unix socket now). * Unix sockets partially working for control protocol. * Add support for unix socket. R * The R connector is now distributed in the source code packages. MonetDB Common * The join code has been completely rewritten. It now optionally uses candidate lists, like the select code that was introduced in the previous release. * A new indexing structure for range selections on unsorted data has been added. * The vmtrim thread is no longer started by default on 64 bit architectures. The vmtrim thread monitors memory usage and drops BATs from memory when memory gets tight. However, in the age of large address spaces and virtual memory, the kernel does a good enough job. And in addition to dropping BATs, the thread also destroyed indexing structures which would have to be recreated the next time they were needed. * Cleaned up some of the parameters dealing with memory usage. * If available on the system, we now use atomic instructions for latching. * Removed some unused fields in the atomDesc structure. This change requires a complete recompilation of the whole suite. * Replaced the mutex implementation for both GNU C and Visual Studio with a home-grown implementation that uses atomic instructions (__sync_*() in gcc, _Interlocked*() in VS). SQL * Added support for quantiles (generalization of median). Usage: SELECT quantile(column_name,0.25) FROM table_name; The value should be in the range 0..1. Bug Fixes * 3040: Wrong NULL behavior in EXCEPT and INTERSECT * 3092: ODBC client doesn't support scalar function escape * 3198: SIGSEGV insert_string_bat (b=0x7fffe419d0a0, n=0x7fffc4006010, append=0) at gdk_batop.c:196 * 3210: Unexpected concurrency conflict when inserting to 2 tables simultaneously and querying one of them * 3273: Add support to Python DBAPI package for timetz, inet and url types * 3285: no such table 'queryHistory' * 3298: GDKmmap messages and monetdb start db takes very long * 3354: Introduce query time-out * 3371: (i)like generates batloop instead of algebra.likesubselect * 3372: Large group by queries never complete - server at 100% cpu(all cores) until MonetDB stopped * 3383: Bad performance with DISTINCT GROUP BY * 3391: Bad performance with GROUP BY and FK with out aggregate function * 3393: "COPY .. INTO ..." - escape of string quotes * 3399: server crashed on simple (malformed) query * 3401: inconsistent/strange handling of invalid dates (e.g. 2013-02-29) in where clause * 3403: NOT NULL constraint can't be applied after deleting rows with null values * 3404: Assertion `h->storage == STORE_MMAP' failed. * 3408: nested concat query crashed server * 3411: (disguised) BETWEEN clause not recognised. Hence no rangejoin. * 3412: Boolean expressions in WHERE clause, result in incorrect resulsts * 3417: Nested Common Table Expressions Crash * 3418: Segmentation fault on a query from table expression * 3419: Database does not start after upgrade * 3420: Database does not start after upgrade * 3423: Group by alias with distinct count doesn't work * 3425: Temporal extraction glitches * 3427: Consistent use of current_timestamp and now() * 3428: Aggregation over two columns is broken * 3429: SAMPLE on JOIN result crashes server * 3430: Wrong temporary handling * 3431: SQLGetInfo returns incorrect value for SQL_FN_NUM_TRUNCATE * 3432: MonetDB SQL syntax incompatible with SQL-92 <delimited identifier> syntax * 3435: INDEX prevents JOIN from discovering matches * 3436: COPY INTO from file containing leading Byte Order Mark (BOM) causes corruption

1 0

Re: MonetDB: Jan2014 - monetdb: allow databases to be created with a...
by Stefan Manegold 17 Feb '14

17 Feb '14

Fabian, when configured with --enable-strict, this fail to compile on my Fedora 20 desktop: .../MonetDB/tools/merovingian/daemon/controlrunner.c: In function ‘ctl_handle_client’: .../MonetDB/tools/merovingian/daemon/controlrunner.c:387:13: error: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Werror=unused-result] chdir(q); ^ cc1: all warnings being treated as errors I could not fix it quickly (i.e., checking and handling the return value of chdir()), as I could not quickly grasp how to do proper error/exception handling in this case ... Would be great, if you could have a look at it. Thanks! Stefan ----- Original Message ----- > Changeset: 6177dd32ac5d for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=6177dd32ac5d > Modified Files: > tools/merovingian/ChangeLog.Jan2014 > tools/merovingian/client/monetdb.1 > tools/merovingian/client/monetdb.c > tools/merovingian/daemon/Makefile.ag > tools/merovingian/daemon/controlrunner.c > Branch: Jan2014 > Log Message: > > monetdb: allow databases to be created with a password for monetdb user > > Creating a database with a custom password for the monetdb user makes it > safer than when created with default/well-known credentials, hence it is > not necessary to lock the database after creation. > > > diffs (190 lines): > > diff --git a/tools/merovingian/ChangeLog.Jan2014 > b/tools/merovingian/ChangeLog.Jan2014 > --- a/tools/merovingian/ChangeLog.Jan2014 > +++ b/tools/merovingian/ChangeLog.Jan2014 > @@ -2,5 +2,7 @@ > # This file is updated with mchangelog > > * Sun Feb 16 2014 Fabian Groffen <fabian(a)monetdb.org> > +- monetdb create: add -p flag to set monetdb user password on creation, > + and therefore allow creating the database in unlocked state > - monetdb destroy -f now also works on running databases > > diff --git a/tools/merovingian/client/monetdb.1 > b/tools/merovingian/client/monetdb.1 > --- a/tools/merovingian/client/monetdb.1 > +++ b/tools/merovingian/client/monetdb.1 > @@ -52,12 +52,14 @@ stopping a database inside the MonetDB D > For all commands, database arguments can be glob-like expressions. > This allows to do wildcard matches. For details on the syntax, see > .IR EXPRESSIONS . > -.IP "create [\-m pattern] database [database ...]" > +.IP "create [\-m pattern] [\-p password] database [database ...]" > Initialises a new database in the MonetDB Database Server. A database > created with this command makes it available under its database name, > but not yet for use by clients, as the database is put into maintenance > mode. This allows the database administrator to perform initialisation > -steps before releasing it to users. See also > +steps before releasing it to users, unless the > +.I \-p > +argument is supplied. See also > .BR "monetdb lock" . > The name of the database must match the expression [A\-Za\-z0\-9\-_]+. > .IP " \-m pattern" > @@ -74,6 +76,14 @@ or discovery. Each parallel target for > sequence, separated by commas. Here the pattern is an ordinary pattern > as would be used for connecting to a database, and can hence also be > just the name of a database. > +.IP " \-p password" > +The > +.I \-p > +flag allows to create a database with the given password for the monetdb > +user. Since this protects the database from being accessed via > +well-known credentials, the created database is not locked after > +creation. This way, a new database can be created and used right away > +using the password supplied. > .IP "destroy [\-f] database [database ...]" > Removes the given database, including all its data and logfiles. Once > destroy has completed, all data is lost. Be careful when using this > diff --git a/tools/merovingian/client/monetdb.c > b/tools/merovingian/client/monetdb.c > --- a/tools/merovingian/client/monetdb.c > +++ b/tools/merovingian/client/monetdb.c > @@ -1435,6 +1435,7 @@ command_create(int argc, char *argv[]) > { > int i; > char *mfunnel = NULL; > + char *password = NULL; > sabdb *orig = NULL; > sabdb *stats = NULL; > > @@ -1464,6 +1465,19 @@ command_create(int argc, char *argv[]) > command_help(2, &argv[-1]); > exit(1); > } > + } else if (argv[i][1] == 'p') { > + if (argv[i][2] != '\0') { > + password = &argv[i][2]; > + argv[i] = NULL; > + } else if (i + 1 < argc && argv[i + 1][0] != '-') { > + argv[i] = NULL; > + password = argv[++i]; > + argv[i] = NULL; > + } else { > + fprintf(stderr, "create: -p needs an argument\n"); > + command_help(2, &argv[-1]); > + exit(1); > + } > } else { > fprintf(stderr, "create: unknown option: %s\n", argv[i]); > command_help(argc + 1, &argv[-1]); > @@ -1492,6 +1506,13 @@ command_create(int argc, char *argv[]) > simple_argv_cmd(argv[0], orig, cmd, > "created multiplex-funnel in maintenance mode", NULL); > free(cmd); > + } else if (password != NULL) { > + size_t len = strlen("create password=") + strlen(password) + 1; > + char *cmd = malloc(len); > + snprintf(cmd, len, "create password=%s", password); > + simple_argv_cmd(argv[0], orig, cmd, > + "created database with password for monetdb user", NULL); > + free(cmd); > } else { > simple_argv_cmd(argv[0], orig, "create", > "created database in maintenance mode", NULL); > diff --git a/tools/merovingian/daemon/Makefile.ag > b/tools/merovingian/daemon/Makefile.ag > --- a/tools/merovingian/daemon/Makefile.ag > +++ b/tools/merovingian/daemon/Makefile.ag > @@ -27,7 +27,10 @@ monetdb_var_dbfarm_DATA = .merovingian_p > INCLUDES = .. \ > ../../../common/stream \ > ../../../common/utils \ > + ../../../common/options \ > ../../../clients/mapilib \ > + ../../../gdk \ > + ../../../monetdb5/mal \ > $(MSGCONTROL_FLAGS) > > EXTRA_DIST = $(man_MANS) .merovingian_properties > @@ -54,6 +57,8 @@ bin_monetdbd = { > ../../../common/utils/libmsabaoth \ > ../../../common/utils/libmuuid \ > ../../../common/utils/libmutils \ > + ../../../monetdb5/tools/libmonetdb5 \ > + ../../../gdk/libbat \ > $(UUID_LIBS) \ > $(curl_LIBS) \ > $(SOCKET_LIBS) \ > diff --git a/tools/merovingian/daemon/controlrunner.c > b/tools/merovingian/daemon/controlrunner.c > --- a/tools/merovingian/daemon/controlrunner.c > +++ b/tools/merovingian/daemon/controlrunner.c > @@ -40,6 +40,10 @@ > #include <utils/database.h> > #include <utils/control.h> > > +#include "gdk.h" /* these three for creation of dbs with password */ > +#include "gdk_private.h" > +#include "mal_authorize.h" > + > #include "merovingian.h" > #include "discoveryrunner.h" /* broadcast, remotedb */ > #include "forkmserver.h" > @@ -353,8 +357,15 @@ static void ctl_handle_client( > "database is not running: %s\n", q); > send_client("!"); > } > - } else if (strcmp(p, "create") == 0) { > - err e = db_create(q); > + } else if (strcmp(p, "create") == 0 || > + strncmp(p, "create password=", strlen("create password=")) == 0) { > + err e; > + > + p += strlen("create"); > + if (*p == ' ') > + p += strlen(" password="); > + > + e = db_create(q); > if (e != NO_ERR) { > Mfprintf(_mero_ctlerr, "%s: failed to create " > "database '%s': %s\n", origin, q, getErrMsg(e)); > @@ -363,6 +374,43 @@ static void ctl_handle_client( > send_client("!"); > free(e); > } else { > + if (*p != '\0') { > + pid_t child; > + if ((child = fork()) == 0) { > + FILE *secretf; > + size_t len; > + char *err; > + char *vaultkey; > + > + /* the child, pollute scope by loading BBP */ > + chdir(q); > + > + buf2[0] = '\0'; > + if ((secretf = fopen(".vaultkey", "r")) != NULL) { > + len = fread(buf2, 1, sizeof(buf2), secretf); > + buf2[len] = '\0'; > + len = strlen(buf2); /* secret can contain null-bytes */ > + fclose(secretf); > + } > + BBPinit(); > + vaultkey = buf2; > + AUTHunlockVault(&vaultkey); > + err = AUTHinitTables(&p); > + if (err != NULL) { > + Mfprintf(_mero_ctlerr, "%s: could not setup " > + "database '%s': %s\n", origin, q, err); > + } else { > + /* don't start locked */ > + unlink(".maintenance"); > + } > + > + exit(0); /* return to the parent */ > + } else { > + /* wait for the child to finish */ > + waitpid(child, NULL, 0); > + } > + } > + > Mfprintf(_mero_ctlout, "%s: created database '%s'\n", > origin, q); > len = snprintf(buf2, sizeof(buf2), "OK\n"); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

Re: MonetDB: Jan2014 - AUTHinitTables: allow passing a password for ...
by Sjoerd Mullender 17 Feb '14

17 Feb '14

Fabian, this reeks of a forbidden ABI change. On 2014-02-16 22:36, Fabian Groffen wrote: > Changeset: 8f6aa798f4c5 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8f6aa798f4c5 > Modified Files: > clients/Tests/exports.stable.out > monetdb5/mal/mal_authorize.c > monetdb5/mal/mal_authorize.h > Branch: Jan2014 > Log Message: > > AUTHinitTables: allow passing a password for monetdb user > > This allows to securely setup a new database. > > > diffs (62 lines): > > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -881,7 +881,7 @@ str AUTHcheckCredentials(oid *ret, Clien > str AUTHgetPasswordHash(str *ret, Client *c, str *username); > str AUTHgetUsername(str *ret, Client *c); > str AUTHgetUsers(BAT **ret, Client *c); > -str AUTHinitTables(void); > +str AUTHinitTables(str *passwd); > str AUTHremoveUser(Client *c, str *username); > str AUTHrequireAdmin(Client *c); > str AUTHresolveUser(str *ret, oid *uid); > diff --git a/monetdb5/mal/mal_authorize.c b/monetdb5/mal/mal_authorize.c > --- a/monetdb5/mal/mal_authorize.c > +++ b/monetdb5/mal/mal_authorize.c > @@ -112,12 +112,13 @@ AUTHcommit(void) > * Localize the authorization tables in the database. The authorization > * tables are a set of aligned BATs that store username, password (hashed) > * and scenario permissions. > - * If the BATs do not exist, they are created, and the monetdb/monetdb > - * administrator account is added. Initialising the authorization tables > - * can only be done after the GDK kernel has been initialized. > + * If the BATs do not exist, they are created, and the monetdb > + * administrator account is added with the given password (or 'monetdb' > + * if NULL). Initialising the authorization tables can only be done > + * after the GDK kernel has been initialized. > */ > str > -AUTHinitTables(void) { > +AUTHinitTables(str *passwd) { > bat bid; > BAT *b; > int isNew = 1; > @@ -168,12 +169,13 @@ AUTHinitTables(void) { > /* insert the monetdb/monetdb administrator account on a > * complete fresh and new auth tables system */ > str user = "monetdb"; > - str pw; /* will become the right hash for "monetdb" */ > - int len = (int) strlen(user); > + str pw = "monetdb"; > oid uid; > Client c = &mal_clients[0]; > > - pw = mcrypt_BackendSum(user /* because user == pass */, len); > + if (passwd != NULL && *passwd != NULL) > + pw = *passwd; > + pw = mcrypt_BackendSum(pw, strlen(pw)); > msg = AUTHaddUser(&uid, &c, &user, &pw); > free(pw); > if (msg) > diff --git a/monetdb5/mal/mal_authorize.h b/monetdb5/mal/mal_authorize.h > --- a/monetdb5/mal/mal_authorize.h > +++ b/monetdb5/mal/mal_authorize.h > @@ -38,7 +38,7 @@ mal_export str AUTHgetUsers(BAT **ret, C > mal_export str AUTHgetPasswordHash(str *ret, Client *c, str *username); > > mal_export str AUTHrequireAdmin(Client *c); > -mal_export str AUTHinitTables(void); > +mal_export str AUTHinitTables(str *passwd); > > > /* > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 1

Announcement: (Hopefully) Final Release Candidate for Jan2014 Release
by Sjoerd Mullender 10 Feb '14

10 Feb '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The MonetDB team at CWI/MonetDB BV is pleased to announce a new release candidate for the upcoming Jan2014 feature release. Many things have changed in this release compared to the previous (Feb2013) release (that's why it took so long). For a smooth upgrade I would urge you to try out this release candidate and to provide feedback on our bug tracker (http://bugs.monetdb.org) A number of bugs in the previous release candidate have been fixed. We hope that this release candidate is good enough to be the new release, but we will give it a few more days. The release candidate can be found at http://dev.monetdb.org/downloads/testing. - -- Sjoerd Mullender -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUviUtz7g04AjvIQpAQJy5wQArulRMgV7YWX1NoOJMoSSKINjEQLeneZC vj4NqMJ0igxYMRxqBLd7EcSfLKvgDR3ElmP7HhDkXbmCDk6POD4PjxPRmh66AEwY uYBhT3iIEe01PFzqjMhWm2HnDnfDOTzvn8Ai4n1/qCIVmW0p14kIYOkTRXWiNX5o tcGfy384cx8= =2un1 -----END PGP SIGNATURE-----

1 0

Re: MonetDB: Jan2014 - make sure we don't call BATappend(a,b) where ...
by Sjoerd Mullender 05 Feb '14

05 Feb '14

What happens to i after this? It still has a heap base pointer that is pointing into freed memory. On 2014-02-05 20:51, Niels Nes wrote: > Changeset: 262ad3ce2b2e for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=262ad3ce2b2e > Modified Files: > sql/storage/bat/bat_storage.c > sql/test/BugTracker-2014/Tests/sample-crash.Bug-3429.stable.err > Branch: Jan2014 > Log Message: > > make sure we don't call BATappend(a,b) where b is a VIEW on a. > > > diffs (29 lines): > > diff --git a/sql/storage/bat/bat_storage.c b/sql/storage/bat/bat_storage.c > --- a/sql/storage/bat/bat_storage.c > +++ b/sql/storage/bat/bat_storage.c > @@ -448,7 +448,12 @@ delta_append_bat( sql_delta *bat, BAT *i > bat_destroy(b); > b = temp_descriptor(bat->ibid); > } > - BATappend(b, i, TRUE); > + if (isVIEW(i) && b->batCacheid == ABS(VIEWtparent(i))) { > + BAT *ic = BATcopy(i, TYPE_void, i->ttype, TRUE); > + BATappend(b, ic, TRUE); > + bat_destroy(ic); > + } else > + BATappend(b, i, TRUE); > assert(BUNlast(b) > b->batInserted); > bat_destroy(b); > } > diff --git a/sql/test/BugTracker-2014/Tests/sample-crash.Bug-3429.stable.err b/sql/test/BugTracker-2014/Tests/sample-crash.Bug-3429.stable.err > --- a/sql/test/BugTracker-2014/Tests/sample-crash.Bug-3429.stable.err > +++ b/sql/test/BugTracker-2014/Tests/sample-crash.Bug-3429.stable.err > @@ -28,8 +28,6 @@ stderr of test 'sample-crash.Bug-3429` i > # 20:54:49 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" "--host=/var/tmp/mtest-31548" "--port=33290" > # 20:54:49 > > > -MAPI = (monetdb) /var/tmp/mtest-31984/.s.monetdb.34383 > -QUERY = rollback; > > > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

Announcement: Second Release Candidate for Jan2014 Release
by Sjoerd Mullender 25 Jan '14

25 Jan '14

The MonetDB team at CWI/MonetDB BV is pleased to announce a second release candidate for the upcoming Jan2014 feature release. Many things have changed in this release compared to the previous (Feb2013) release (that's why it took so long). For a smooth upgrade I would urge you to try out this release candidate and to provide feedback on our bug tracker (http://bugs.monetdb.org) A number of bugs in the previous release candidate have been fixed. Among others, packaging errors on Debian and Ubuntu have been fixed, and a problem with slow startup has been fixed. The release candidate can be found at http://dev.monetdb.org/downloads/testing. -- Sjoerd Mullender

1 0

Announcement: Release Candidate for Jan2014 Release
by Sjoerd Mullender 15 Jan '14

15 Jan '14

The MonetDB team at CWI/MonetDB BV is pleased to announce a release candidate for the upcoming Jan2014 feature release. Many things have changed in this release compared to the previous (Feb2013) release (that's why it took so long). For a smooth upgrade I would urge you to try out this release candidate and to provide feedback on our bug tracker (http://bugs.monetdb.org) The release candidate can be found at http://dev.monetdb.org/downloads/testing. -- Sjoerd Mullender

1 0

Segmentation fault on a query from table expression
by Alexander Shestakov 14 Jan '14

14 Jan '14

Hi, I am running MonetDB Database Server Toolkit v1.1 (Feb2013-SP6) , I have a table DailyData defined as follows. create table DailyData ( sym varchar(32), businessdate date, closePrice float, adjustedClosePrice float, openingPrice float, accVol float, unique(businessdate, sym) ); I make the following query. with dd as (select * from DailyData where businessdate > '2013-06-01'), r as (select t.businessdate, t.sym, 1 - t.adjustedcloseprice / y.adjustedcloseprice as ret from dd t left outer join dd y on t.businessdate = y.businessdate + (interval '1' day) and t.sym = y.sym), v as (select sym, stddev_samp(ret) as volatility from r group by sym) select * from v where v.sym like '%.L' limit 100; It results in a segmentation fault: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f5f17023700 (LWP 25178)] 0x0000003ee9327a9a in __strcmp_sse42 () from /lib64/libc.so.6 (gdb) where #0 0x0000003ee9327a9a in __strcmp_sse42 () from /lib64/libc.so.6 #1 0x00007f6f2d4ef6f8 in rel_use_index (changes=<value optimized out>, sql=0x7f5f08005470, rel=0x7f5f0850aa70) at rel_optimizer.c:4829 #2 0x00007f6f2d4ea225 in rewrite (sql=0x7f5f08005470, rel=0x7f5f0850aa70, rewriter=0x7f6f2d4ef610 <rel_use_index>, has_changes=0x7f5f17022cac) at rel_optimizer.c:6106 #3 0x00007f6f2d4ea2bc in rewrite (sql=0x7f5f08005470, rel=0x7f5f0850cb40, rewriter=0x7f6f2d4ef610 <rel_use_index>, has_changes=0x7f5f17022cac) at rel_optimizer.c:6093 #4 0x00007f6f2d4ea2bc in rewrite (sql=0x7f5f08005470, rel=0x7f5f0850ec30, rewriter=0x7f6f2d4ef610 <rel_use_index>, has_changes=0x7f5f17022cac) at rel_optimizer.c:6093 #5 0x00007f6f2d4ea2bc in rewrite (sql=0x7f5f08005470, rel=0x7f5f0850ef50, rewriter=0x7f6f2d4ef610 <rel_use_index>, has_changes=0x7f5f17022cac) at rel_optimizer.c:6093 #6 0x00007f6f2d4ea2bc in rewrite (sql=0x7f5f08005470, rel=0x7f5f08512a90, rewriter=0x7f6f2d4ef610 <rel_use_index>, has_changes=0x7f5f17022cac) at rel_optimizer.c:6093 #7 0x00007f6f2d4ea2bc in rewrite (sql=0x7f5f08005470, rel=0x7f5f0850feb0, rewriter=0x7f6f2d4ef610 <rel_use_index>, has_changes=0x7f5f17022cac) at rel_optimizer.c:6093 #8 0x00007f6f2d4ea2bc in rewrite (sql=0x7f5f08005470, rel=0x7f5f085101c0, rewriter=0x7f6f2d4ef610 <rel_use_index>, has_changes=0x7f5f17022cac) at rel_optimizer.c:6093 #9 0x00007f6f2d4ecf58 in _rel_optimizer (sql=0x7f5f08005470, rel=<value optimized out>, level=0) at rel_optimizer.c:6278 #10 0x00007f6f2d484369 in sql_symbol2relation (c=0x7f5f08005470, sym=<value optimized out>) at sql.mx:1755 #11 0x00007f6f2d454b2f in SQLparser (c=0x7f6f2df49308) at sql_scenario.c:1708 #12 0x00007f6f352ecdd3 in runPhase (c=0x7f6f2df49308) at mal_scenario.c:522 #13 runScenarioBody (c=0x7f6f2df49308) at mal_scenario.c:558 #14 0x00007f6f352ecf1f in runScenario (c=0x7f6f2df49308) at mal_scenario.c:586 #15 0x00007f6f352ed778 in MSserveClient (dummy=0x7f6f2df49308) at mal_session.c:446 #16 0x0000003ee9a07851 in start_thread () from /lib64/libpthread.so.0 #17 0x0000003ee92e890d in clone () from /lib64/libc.so.6 Apparently at rel_optimizer.c:4829 strcmp(exp_relname(nre) returns NULL which is not checked. If I rewrite the query in the following way it works if not very fast. with r as (select t.businessdate, t.sym, 1 - t.adjustedcloseprice / y.adjustedcloseprice as ret from DailyData t left outer join DailyData y on t.businessdate = y.businessdate + (interval '1' day) and t.sym = y.sym where t.businessdate > '2013-06-01'), v as (select sym, stddev_samp(ret) as volatility from r group by sym) select * from v where v.sym like '%.L' limit 100; Regards, Alex Shestakov.

2 2

user defnined function problem
by Niall Rooney 07 Jan '14

07 Jan '14

Hello, I am making the following adjustment to the user defined function template files . I am using MonetDB-11.15.15 built from source on Ubuntu 12.04. I have added to the following files in the ./sql/backends/monet5/UDF directory as shown: 80_udf.sql create function sumProb(prob1 double, prob2 double) returns double external name udf.sumProb; udf.mal module udf; command sumProb(prob1:dbl,prob2:dbl):dbl address UDFSumProb comment "sum two probabilities"; udf.c str UDFSumProb(dbl * res, dbl * prob1, dbl * prob2) { *res = *prob1 + *prob2; return MAL_SUCCEED; } udf.h udf_export str UDFSumProb(dbl * res, dbl * prob1, dbl * prob2); I did a configure/make/make install and tried the following call in mclient: declare a, b,c double; set a= 0.1; set b= 0.2; select c=jsdistance(a,b); and get the error: TypeException:user.s2_1[8]:'udf.sumprob' undefined in: _12:any := udf.sumprob(_8:dbl, _11:dbl) program contains errors

3 6

Create a table with FLOAT type
by Adnan Agbaria 05 Jan '14

05 Jan '14

Hello, I use MonetDB in Ubuntu. When I create a table with FLOAT type, MonetDB created it as DOUBLE type. Why does MonetDB decide to change FLOAT to DOUBLE? How can I fix this?

2 1

Segmentation faultin sql_optimizer
by Shestakov, Alex 02 Jan '14

02 Jan '14

Hi, I have a segmentation fault when running an SQL query against a specific table. I can execute queries like "select count(*) from <TABLE>" but the server crashes when I do "select * from <TABLE> where businessdate = '2013-12-18' limit 100". MonetDB version: MonetDB Database Server Toolkit v1.1 (Feb2013-SP5) Stack trace: #0 0x00007f7cf2987eb1 in GDKfilepath (path=0x7f6e44f0db5d "", dir=0x7f7ceb4c51d4 "bat", name=0x0, ext=0x0) at gdk_storage.c:51 #1 0x00007f7ceb3fbbd3 in BATlocation (fnme=0x7f6e44f0fbe0, bid=<value optimized out>) at sql_optimizer.c:77 #2 0x00007f7ceb3fc59a in SQLgetStatistics (c=0x7f7cebed6308) at sql_optimizer.c:160 #3 addQueryToCache (c=0x7f7cebed6308) at sql_optimizer.c:295 #4 0x00007f7ceb3fb7ed in backend_dumpproc (be=0x7f6e3809d5f0, c=0x7f7cebed6308, cq=0x7f6e383d4090, s=0x7f6e3837bd20) at sql_gencode.c:2431 #5 0x00007f7ceb3e2067 in SQLparser (c=0x7f7cebed6308) at sql_scenario.c:1755 #6 0x00007f7cf2f35b63 in runPhase (c=0x7f7cebed6308) at mal_scenario.c:522 #7 runScenarioBody (c=0x7f7cebed6308) at mal_scenario.c:558 #8 0x00007f7cf2f35caf in runScenario (c=0x7f7cebed6308) at mal_scenario.c:586 #9 0x00007f7cf2f36508 in MSserveClient (dummy=0x7f7cebed6308) at mal_session.c:446 #10 0x0000003ee9a07851 in start_thread () from /lib64/libpthread.so.0 #11 0x0000003ee92e890d in clone () from /lib64/libc.so.6 The problem is that it tries to de-reference null pointer 'name'. The pointer is calculated in BATLocation as (b->T->heap.filename ? b->T->heap.filename: b->H->heap.filename). The column in question has type real so as far as I understand there is no heap file for it. I don't know if heap.filename should contain NULL or something else. Regards, Alex Shestakov. ________________________________ This email and any files transmitted with it contain confidential and proprietary information and is solely for the use of the intended recipient. If you are not the intended recipient please return the email to the sender and delete it from your computer and you must not use, disclose, distribute, copy, print or rely on this email or its contents. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. Any comments or statements made herein do not necessarily reflect those of GSA Capital. GSA Capital Partners LLP is authorised and regulated by the Financial Conduct Authority and is registered in England and Wales at Stratton House, 5 Stratton Street, London W1J 8LA, number OC309261. GSA Capital Services Limited is registered in England and Wales at the same address, number 5320529.

1 1

about the monetdb jdbc problem
by 少湖 30 Dec '13

30 Dec '13

An exception(see the bottom stack) was thrown when I use the JDBC driver(monetdb-jdbc-2.10.jar, MonetDB Database Server v1.7 (Feb2013-SP4) ) on linux(Linux *** 2.6.18-164.11.1.el5 #1 SMP Wed Jan 6 13:26:04 EST 2010 x86_64 x86_64 x86_64 GNU/Linux), but with the same code no exceptions are thrown on windows7 When DML SQL is executed by preparestatement(or Statement) I Just got the exception on linux, can anyone tell me what happened. Exception in thread "main" java.lang.IndexOutOfBoundsException at java.nio.StringCharBuffer.subSequence(StringCharBuffer.java:92) at nl.cwi.monetdb.mcl.parser.StartOfHeaderParser.getNextAsString(StartOfHeaderP arser.java:146) at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnectio n.java:2394) at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(MonetConnectio n.java:2284) at nl.cwi.monetdb.jdbc.MonetStatement.internalExecute(MonetStatement.java:508) at nl.cwi.monetdb.jdbc.MonetStatement.internalBatch(MonetStatement.java:261) at nl.cwi.monetdb.jdbc.MonetStatement.executeBatch(MonetStatement.java:230) at MonetdbBenchmark.insert(MonetdbBenchmark.java:50) at MonetdbBenchmark.main(MonetdbBenchmark.java:19)

2 2

Skyline Queries and MonetDB
by Stephen P. Morgan 17 Dec '13

17 Dec '13

Hello, Has anyone out there done any work on extending MonetDB to support skyline queries natively? I may be interested in work along these lines but I'd rather not reinvent the wheel. If anyone has preceded me, I'd love to hear about your work. Thanks, Steve Morgan

1 0

Query Time
by Sebastian Kielmann 12 Dec '13

12 Dec '13

HI, We are using monetdb for some statistical analysis. The queries are very fast (0.2 sec), but some times (not regularly) the queries get peeks of up to 32 sec for the same query. CPU and RAM don't change and are way higher than actually needed. Does any one have this same situation? Any idea what could be doing this? Best regards and thanks to any help. sebastian

3 5

Re: MonetDB: Jan2014 - Merge with Jan2014 branch.
by Stefan Manegold 10 Dec '13

10 Dec '13

> Changeset: 1a2184860d3e for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=1a2184860d3e [...] > Branch: Jan2014 > Log Message: > > Merge with Jan2014 branch. ^^^^^^^ SORRY --- shoudl of course read "Merge with Feb2013 branch." S. -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Announcement: New Feb2013-SP6 Bugfix release of MonetDB suite
by Sjoerd Mullender 03 Dec '13

03 Dec '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013-SP6 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Feb 2013-SP6 bugfix release MonetDB5 Server * Fixed a possible buffer overflow in the COPY INTO code. * Fixed a problem that when the server is using all available threads for a query, it was not possible to attach another client and have it execute even the smallest query. This is fixed by creating extra threads for each client entering the fray at the cost of having more threads that execute queries. But at least there is guaranteed progress for all clients (modulo the operating system scheduler). See bug #3258. Merovingian * monetdbd(1) now refuses to startup if it cannot read the properties from the dbfarm, bug #3398 Client Package * mclient: Fixed a bug where the -H option only worked if the readline history file already existed. Now we properly create and use the history file. * ODBC: Fixed interpretation SQL_C_SLONG/SQL_C_ULONG/SQL_C_LONG to refer to a 32 bit integer always (i.e. "int" on 64 bit architectures despite the name and the Microsoft documentation). This seems to be the consensus. * ODBC: Fixed transaction level: MonetDB only supports the highest level (SQL_TXN_SERIALIZABLE), so setting the transaction level can be accepted and ignored. MonetDB Common * Stopped using the deprecated sbrk() system call. * Fixed a problem when reverse sorting a sorted column. * Fixed bugs that deal with problems that could possibly occur when transactions are aborted and the server is restarted. See bug #3243. * A bug was fixed in the handling of grouped aggregates when all values in a group are null. See bug #3388. SQL * Fixed a bug where the server at some point stopped saving compiled queries in the SQL query cache. Bug Fixes * 3243: Segmentation fault (possible data corruption) after clean shutdown * 3258: Scheduling issues * 3368: BAT sortedness info ignored on ORDER BY and TOPN * 3374: UNIQUE constraint does not set tkey property on the corresponding BAT * 3382: Response to PREPARE emtpy if query contains a LIMIT * 3385: Simple query fails with 'identifier not found' * 3387: mclient does not properly double quote schema names when using autofill tab * 3388: case statement in "order by" clause doesn't work when used together with "group by" * 3389: median function with "group by" - SIGSEGV * 3392: ODBC datatype conversion for INTEGER not working properly * 3394: "Cannot find column type" error in temporary tables in functions * 3395: error occurred during a query: "'CASE WHEN" sentence * 3396: Improper UDF expansion * 3397: Error in ODBC-Driver when using Prepared Statements * 3398: Cannot stop monetdbd after erroneously starting an uninitialized dbfarm

1 0

New Jan2014 stable branch created
by Sjoerd Mullender 27 Nov '13

27 Nov '13

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jan2014. So in each of your stable checkout directories you can do hg pull hg update -rJan2014 to update to this new branch. -- Sjoerd Mullender

1 0

MonetDB with OpenCL extension (Ocelot) query fails trying to cache empty BAT
by Edward Ching 25 Nov '13

25 Nov '13

I'm running into a problme with Ocelot, the MonetDB with OpenCL extension.Not sure if this should go to the developers' or users' list. I apologize if this is not the right list to post to. I'm seeing a query failure which was caused by MoneDB-OpenCL trying to cache an empty BAT and ended up calling clCreateBuffer with a 0 size buffer and failed. Any select with a where clause would trigger it. Has anyone seen this too? It fails whether I use 32 bit or 64 bit oid, Nvidia or AMD. I'm trying to figure if it's a real issue or if I'm missing anything. Any input greatly appreciated. The database was generated by TPC-H dbgen (details below). The query I used was "select c_custkey, c_name from customer where c_custkey < 2", and here's a trace of the sequence of calls leading to the failure starting from the MAL statement: 1) X_52 := ocelot.thetasubselect(r1_10,A0,"<",0);(see log below) 2) ocl_select.c::ocl_subselect() line 142: ocl_cacheBAT 3) ocl_memmanager.c::ocl_cacheBAT() line 771: allocateBuffer 4) ocl_memmanager.c::allocateBuffer() line 432: clCreateBuffer ===== Source version and environment ===== 1) Ocelot src installed from zip downloaded from monetdb-opencl: msaecker-monetdb-opencl-b513b35eb84a.zip 2) Ocelot configure options: ./configure --prefix=/usr/local --enable-oid32 --enable-optimize --disable-test ing --disable-developer --disable-debug --disable-assert --enable-sql --enable-m onetdb5 --with-opencl=/usr 3) Operating System: Fedora 17 kernel 3.9.10-100 4) Hardware: NVIDIA GTX570 on Intel i7-3820 w/ 16GB mem 5) OpenCL: Nvidia OpenCL that came with CUDA 5.5 SDK 6) Database: generated w/ TPC-H dbgen using size factor s=0.01 ===== operation log follows ===== [root@gen3-2 ocelot]# mclient -ddb001 -lmal mal>ocelot.init(); mal>ocelot.listDevices(); # Device 0: NVIDIA Corporation - GeForce GTX 570 (default device) mal>\q [root@gen3-2 ocelot]# mclient -ddb001 -lsql sql>\d TABLE sys.customer TABLE sys.lineitem TABLE sys.nation TABLE sys.orders TABLE sys.part TABLE sys.partsupp TABLE sys.region TABLE sys.supplier sql>\d customer CREATE TABLE "sys"."customer" ( "c_custkey" INTEGER NOT NULL, "c_name" VARCHAR(25) NOT NULL, "c_address" VARCHAR(40) NOT NULL, "c_nationkey" INTEGER NOT NULL, "c_phone" VARCHAR(15) NOT NULL, "c_acctbal" DECIMAL(15,2) NOT NULL, "c_mktsegment" VARCHAR(10) NOT NULL, "c_comment" VARCHAR(117) NOT NULL, CONSTRAINT "customer_c_custkey_pkey" PRIMARY KEY ("c_custkey"), CONSTRAINT "customer_c_nationkey_fkey" FOREIGN KEY ("c_nationkey") REFERENCES "sys"."nation" ("n_nationkey") ); sql>select count(*) from customer; +------+ | L1 | +======+ | 1500 | +------+ 1 tuple (1.311ms) sql>select c_custkey, c_name from customer where c_custkey < 2; +-----------+--------------------+ | c_custkey | c_name | +===========+====================+ | 1 | Customer#000000001 | +-----------+--------------------+ 1 tuple (1.776ms) sql>set optimizer='cl_pipe'; operation successful (0.981ms) sql>select c_custkey, c_name from customer where c_custkey < 2; invalid/unknown response from server, ignoring output ? #Could not allocate OpenCL buffer of size: 0. Error caching input BAT. sql> sql>explain select c_custkey, c_name from customer where c_custkey < 2; +------------------------------------------------------------------------------+ | mal +==============================================================================+ | function user.s9_1{autoCommit=true}(A0:int):void; | | ocelot.beginQuery(); | | X_3 := sql.mvc(); | | X_7 := sql.bind(X_3,"sys","customer","c_custkey",0); | | X_4:bat[:oid,:oid] := sql.tid(X_3,"sys","customer"); | | X_51 := ocelot.thetasubselect(X_7,X_4,A0,"<",0); | | (X_10,r1_10) := sql.bind(X_3,"sys","customer","c_custkey",2); | | X_52 := ocelot.thetasubselect(r1_10,A0,"<",0); | | X_13 := sql.bind(X_3,"sys","customer","c_custkey",1); | | X_53 := ocelot.thetasubselect(X_13,X_4,A0,"<",0); | | ocelot.sync(X_51); | | ocelot.sync(X_52); | | ocelot.sync(X_53); | | X_15 := sql.subdelta(X_51,X_4,X_10,X_52,X_53); | | X_17 := sql.projectdelta(X_15,X_7,X_10,r1_10,X_13); | | X_18 := sql.bind(X_3,"sys","customer","c_name",0); | | (X_20,r1_23) := sql.bind(X_3,"sys","customer","c_name",2); | | X_22 := sql.bind(X_3,"sys","customer","c_name",1); | | X_23 := sql.projectdelta(X_15,X_18,X_20,r1_23,X_22); | | X_24 := sql.resultSet(2,1,X_17); | | sql.rsColumn(X_24,"sys.customer","c_custkey","int",32,0,X_17); | | sql.rsColumn(X_24,"sys.customer","c_name","varchar",25,0,X_23); | | X_32 := io.stdout(); | | sql.exportResult(X_32,X_24); | | ocelot.endQuery(); | | end s9_1; | | # querylog.define("explain select c_custkey, c_name from customer where c_cu | : stkey < 2;","cl_pipe") : +------------------------------------------------------------------------------+ 27 tuples (1.408ms) sql> =======================================

1 0

Re: MonetDB: default - Get rid of Mx.
by Fabian Groffen 21 Nov '13

21 Nov '13

On 21-11-2013 12:54:34 +0100, Sjoerd Mullender wrote: > Changeset: f6606676fe5e for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=f6606676fe5e > Branch: default > Log Message: > > Get rid of Mx. Hooray! Congratulations :) Fabian

1 0

Re: MonetDB: default - Prepare split of sql.mx
by Stefan Manegold 20 Nov '13

20 Nov '13

Martin, what's the (intended) difference between the newly added command mmath.rand{unsafe} (v:int) :int (see below) and the existing function mmath.rand{inline}( v:int ) :int; (see sql/backends/monet5/sql.mx line 1123) ? In particular, which of the two seemingly identical signatures will the MAL interpreter choose / use to evaluate, say, a:int:=mmath.rand(0:int); ? If they are identical / indistinguishable, one should be removed (again). Thanks, Stefan ps: Found by test clients/Tests/MAL-signatures ;-) ----- Original Message ----- > Changeset: 026d2c8bc8ee for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=026d2c8bc8ee > Added Files: > sql/backends/monet5/sql.mal > sql/backends/monet5/sql_aggr_bte.mal > sql/backends/monet5/sql_aggr_flt.mal > sql/backends/monet5/sql_aggr_int.mal > sql/backends/monet5/sql_aggr_lng.mal > sql/backends/monet5/sql_aggr_sht.mal > sql/backends/monet5/sql_aggr_wrd.mal > sql/backends/monet5/sql_cast.mal > sql/backends/monet5/sql_inspect.mal > sql/backends/monet5/sql_octopus.mal > sql/backends/monet5/sql_rank.mal > sql/backends/monet5/sql_rdf.mal > sql/backends/monet5/sql_sort.mal > Modified Files: > monetdb5/modules/kernel/mmath.mal > sql/backends/monet5/Makefile.ag > Branch: default > Log Message: > > Prepare split of sql.mx > The first step in a dissection of the monster. > Splitting sql.mx into *mal and the rest > > times() has been removed. Also script 11_times > should be removed. > > sql_cast.mal calls for a shell script > whereafter we can finish the first step in > the dissection. > > > diffs (truncated from 1794 to 300 lines): > > diff --git a/monetdb5/modules/kernel/mmath.mal > b/monetdb5/modules/kernel/mmath.mal > --- a/monetdb5/modules/kernel/mmath.mal > +++ b/monetdb5/modules/kernel/mmath.mal > @@ -182,6 +182,10 @@ command rand{unsafe} () :int > address MATHrandint > comment "return a random number"; > > +command rand{unsafe} (v:int) :int > +address MATHrandint > +comment "return a random number"; > + > command srand(seed:int) :void > address MATHsrandint > comment "initialize the rand() function with a seed"; > diff --git a/sql/backends/monet5/Makefile.ag > b/sql/backends/monet5/Makefile.ag > --- a/sql/backends/monet5/Makefile.ag > +++ b/sql/backends/monet5/Makefile.ag > @@ -96,7 +96,7 @@ lib__sql = { > headers_mal = { > HEADERS = mal > DIR = libdir/monetdb5 > - SOURCES = sql.mx > + SOURCES = sql.mx sql_aggr_bte.mal sql_aggr_flt.mal sql_aggr_int.mal > sql_aggr_lng.mal sql_aggr_sht.mal sql_aggr_wrd.mal sql_cast.mal > sql_inspect.mal sql.mal sql_octopus.mal sql_rank.mal sql_rdf.mal > sql_sort.mal > } > > headers_autoload = { > diff --git a/sql/backends/monet5/sql.mal b/sql/backends/monet5/sql.mal > new file mode 100644 > --- /dev/null > +++ b/sql/backends/monet5/sql.mal > @@ -0,0 +1,1055 @@ > +# The contents of this file are subject to the MonetDB Public License > +# Version 1.1 (the "License"); you may not use this file except in > +# compliance with the License. You may obtain a copy of the License at > +# http://www.monetdb.org/Legal/MonetDBLicense > +# > +# Software distributed under the License is distributed on an "AS IS" > +# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the > +# License for the specific language governing rights and limitations > +# under the License. > +# > +# The Original Code is the MonetDB Database System. > +# > +# The Initial Developer of the Original Code is CWI. > +# Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > +# Copyright August 2008-2013 MonetDB B.V. > +# All Rights Reserved. > + > + > +module batsql; > +module sql; > + > +pattern start():void > +address SQLsession > +comment "Switch to processing SQL statements"; > + > +pattern start2():void > +address SQLsession2 > +comment "Switch to processing precompiled sql statements"; > + > +pattern init():void > +address SQLinitEnvironment > +comment "Initialize the environment for MAL"; > + > +pattern shutdown(delay:bte, force:bit):str > +address SQLshutdown_wrap; > +pattern shutdown(delay:sht, force:bit):str > +address SQLshutdown_wrap; > +pattern shutdown(delay:int, force:bit):str > +address SQLshutdown_wrap; > + > +pattern shutdown(delay:bte):str > +address SQLshutdown_wrap; > +pattern shutdown(delay:sht):str > +address SQLshutdown_wrap; > +pattern shutdown(delay:int):str > +address SQLshutdown_wrap; > + > +pattern mvc():int > +address SQLmvc > +comment "Get the multiversion catalog context. > +Needed for correct statement dependencies > +(ie sql.update, should be after sql.bind in concurrent execution)"; > + > +pattern trans(type:int,chain:int,name:str):void > +address SQLtransaction > +comment "A transaction statement (type can be commit,release,rollback or > start)"; > + > +pattern transaction{unsafe}() > +address SQLtransaction2 > +comment "Start an autocommit transaction"; > + > +pattern commit() > +address SQLcommit > +comment "Trigger the commit operation for a MAL block"; > + > +pattern abort() > +address SQLabort > +comment "Trigger the abort operation for a MAL block"; > + > +pattern catalog(type:int,sname:str,name:str,action:int):void > +address SQLcatalog; > +pattern > catalog(type:int,sname:str,name:str,funcid:int,ft:int,action:int):void > +address SQLcatalog; > +pattern > catalog(type:int,sname:str,name:str,tname:str,time:int,o:int,event:int,oname:str,nname:str,condition:str,query:str):void > +address SQLcatalog; > +pattern catalog(type:int,sname:str,t:ptr):void > +address SQLcatalog; > +pattern catalog(type:int,sname:str,t:ptr,temp:int):void > +address SQLcatalog; > +pattern catalog(type:int,sname:str,t:ptr,restart:lng):void > +address SQLcatalog > +comment "a catalog statement"; > + > +pattern catalog(type:int,grantee:str,role:str):void > +address SQLcatalog > +comment "a grant/revoke role statement"; > + > +pattern > catalog(type:int,user:str,passwd:str,enc:int,schema:str,fullname:str):void > +address SQLcatalog > +comment "a user catalog statement"; > + > +pattern > catalog(type:int,sname:str,tname:str,grantee:str,privs:int,cname:str,grant:int,grantor:int):void > +address SQLcatalog > +comment "a grant/revoke privileges statement"; > + > +pattern catalog(type:int,iname:str,itype:int,sname:str,tname:str...):void > +address SQLcatalog > +comment "a create index catalog statement"; > + > +pattern eval(cmd:str):void > +address SQLstatement; > +pattern eval(cmd:str, output:bit):void > +address SQLstatement > +comment "Compile and execute a single sql statement (and optionaly send > output on the output stream)"; > + > +pattern include(fname:str):void > +address SQLinclude > +comment "Compile and execute a sql statements on the file"; > + > +pattern evalAlgebra(cmd:str, optimize:bit):void > +address RAstatement > +comment "Compile and execute a single 'relational algebra' statement"; > + > +pattern assert(b:bit,msg:str):void > +address SQLassert; > +pattern assert(b:int,msg:str):void > +address SQLassertInt; > +pattern assert(b:wrd,msg:str):void > +address SQLassertWrd; > +pattern assert(b:lng,msg:str):void > +address SQLassertLng > +comment "Generate an exception when b!=0"; > + > +pattern setVariable(mvc:int, varname:str, value:any_1 ):int > +address setVariable > +comment "Set the value of a session variable"; > + > +pattern getVariable(mvc:int, varname:str ):any_1 > +address getVariable > +comment "Get the value of a session variable"; > + > +pattern logfile{unsafe}(filename:str):void > +address mvc_logfile > +comment "Enable/disable saving the sql statement traces"; > + > +pattern next_value( sname:str, sequence:str ):lng > +address mvc_next_value > +comment "return the next value of the sequence"; > + > +pattern batsql.next_value( sname:bat[:oid,:str], sequence:str ) > :bat[:oid,:lng] > +address mvc_bat_next_value > +comment "return the next value of the sequence"; > + > +pattern get_value( sname:str, sequence:str ):lng > +address mvc_get_value > +comment "return the current value of the sequence"; > + > +pattern restart{unsafe}( sname:str, sequence:str, start:lng ):lng > +address mvc_restart_seq > +comment "restart the sequence with value start"; > + > +pattern bind_idxbat(mvc:int, schema:str, table:str, index:str, > access:int):bat[:oid,:any_1] > +address mvc_bind_idxbat_wrap > +comment "Bind the 'schema.table.index' BAT with access kind: > + 0 - base table > + 1 - inserts > + 2 - updates"; > + > +pattern bind_idxbat(mvc:int, schema:str, table:str, index:str, > access:int)(uid:bat[:oid,:oid],uval:bat[:oid,:any_1]) > +address mvc_bind_idxbat_wrap > +comment "Bind the 'schema.table.index' BAT with access kind: > + 0 - base table > + 1 - inserts > + 2 - updates"; > + > +pattern bind_idxbat(mvc:int, schema:str, table:str, index:str, access:int, > part_nr:int, nr_parts:int):bat[:oid,:any_1] > +address mvc_bind_idxbat_wrap > +comment "Bind the 'schema.table.index' BAT with access kind: > + 0 - base table > + 1 - inserts > + 2 - updates"; > + > +pattern bind_idxbat(mvc:int, schema:str, table:str, index:str, access:int, > part_nr:int, nr_parts:int)(uid:bat[:oid,:oid],uval:bat[:oid,:any_1]) > +address mvc_bind_idxbat_wrap > +comment "Bind the 'schema.table.index' BAT with access kind: > + 0 - base table > + 1 - inserts > + 2 - updates"; > + > +pattern bind(mvc:int, schema:str, table:str, column:str, access:int > ):bat[:oid,:any_1] > +address mvc_bind_wrap > +comment "Bind the 'schema.table.column' BAT with access kind: > + 0 - base table > + 1 - inserts > + 2 - updates"; > + > +pattern bind(mvc:int, schema:str, table:str, column:str, access:int > )(uid:bat[:oid,:oid],uval:bat[:oid,:any_1]) > +address mvc_bind_wrap > +comment "Bind the 'schema.table.column' BAT with access kind: > + 0 - base table > + 1 - inserts > + 2 - updates"; > + > +pattern bind(mvc:int, schema:str, table:str, column:str, access:int, > part_nr:int, nr_parts:int ):bat[:oid,:any_1] > +address mvc_bind_wrap > +comment "Bind the 'schema.table.column' BAT partition with access kind: > + 0 - base table > + 1 - inserts > + 2 - updates"; > + > +pattern bind(mvc:int, schema:str, table:str, column:str, access:int, > part_nr:int, nr_parts:int )(uid:bat[:oid,:oid],uval:bat[:oid,:any_1]) > +address mvc_bind_wrap > +comment "Bind the 'schema.table.column' BAT with access kind: > + 0 - base table > + 1 - inserts > + 2 - updates"; > + > +command delta ( col:bat[:oid,:any_3], uid:bat[:oid,:oid], > uval:bat[:oid,:any_3], ins:bat[:oid,:any_3] ) > + :bat[:oid,:any_3] > +address DELTAbat > +comment "Return column bat with delta's applied."; > + > +command projectdelta( subselect:bat[:oid,:oid], col:bat[:oid,:any_3], > uid:bat[:oid,:oid], uval:bat[:oid,:any_3], ins:bat[:oid,:any_3] ) > :bat[:oid,:any_3] > +address DELTAproject > +comment "Return column bat with delta's applied."; > + > +command subdelta ( col:bat[:oid,:oid], cand:bat[:oid,:oid], > uid:bat[:oid,:oid], uval:bat[:oid,:oid], ins:bat[:oid,:oid] ) > :bat[:oid,:oid] > +address DELTAsub > +comment "Return a single bat of subselected delta."; > + > +command delta ( col:bat[:oid,:any_3], uid:bat[:oid,:oid], > uval:bat[:oid,:any_3]) :bat[:oid,:any_3] > +address DELTAbat2 > +comment "Return column bat with delta's applied."; > + > +command projectdelta( subselect:bat[:oid,:oid], col:bat[:oid,:any_3], > uid:bat[:oid,:oid], uval:bat[:oid,:any_3]) :bat[:oid,:any_3] > +address DELTAproject2 > +comment "Return column bat with delta's applied."; > + > +command subdelta ( col:bat[:oid,:oid], cand:bat[:oid,:oid], > uid:bat[:oid,:oid], uval:bat[:oid,:oid]) :bat[:oid,:oid] > +address DELTAsub2 > +comment "Return a single bat of subselected delta."; > + > +command getVersion(clientid:int):lng > +address mvc_getVersion > +comment "Return the database version identifier for a client"; > + > +pattern append(mvc:int, sname:str, tname:str, cname:str, ins:any):int > +address mvc_append_wrap > +comment "Append b to the column tname.cname (possibly optimized to replace > the insert bat of tname.cname (returns sequence number for order > dependence)"; > + > +pattern update(mvc:int, sname:str, tname:str, cname:str, rids:any, > upd:any):int > +address mvc_update_wrap > +comment "Update the values of the column tname.cname"; > + > +pattern clear_table{unsafe}(sname:str, tname:str) :wrd > +address mvc_clear_table_wrap > +comment "Clear table"; > + > +pattern tid( mvc:int, sname:str, tname:str):bat[:oid,:any_3] > +address SQLtid; > +pattern tid( mvc:int, sname:str, tname:str, part_nr:int, nr_parts:int > ):bat[:oid,:any_3] > +address SQLtid > +comment "Return the tables tid column."; > + > +pattern delete{unsafe}(mvc:int, sname:str, tname:str, b:any):int > +address mvc_delete_wrap > +comment "delete from table"; > + > +pattern resultSet{unsafe}( nr_cols:int, sep:str, rsep:str, ssep:str, ns:str, > order:any_1 ) :int > +address mvc_result_file_wrap; > +pattern resultSet{unsafe}( nr_cols:int, sep:str, rsep:str, ssep:str, ns:str, > order:bat[:oid,:any_1] ) :int > +address mvc_result_file_wrap; > +pattern resultSet{unsafe}( nr_cols:int, qtype:int, order:any_1 ) :int > +address mvc_result_row_wrap; > +pattern resultSet{unsafe}( nr_cols:int, qtype:int, order:bat[:oid,:any_1] ) > :int > +address mvc_result_table_wrap; > +pattern rsColumn{unsafe}(rs:int, tname:str, name:str, typename:str, > digits:int, scale:int, val:any_1 ) :void > +address mvc_result_value_wrap > +comment "Add the value to the row query result"; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Default branch does not compile for builds with --enable_oid32
by Max Heimel 17 Nov '13

17 Nov '13

Hi, I stumbled upon a build error in the latest source from the default branch when I configure with --enable_oid32. The fix is a missing curly brace in the HASHputall macro at gdk/gdk_search.h:110.

2 1

Re: MonetDB: holindex - Comment lines that use papi in order to meas...
by Stefan Manegold 13 Nov '13

13 Nov '13

Eleni, this calls for a proper configure check and conditionals to reduce/eliminate (extra) code maintenance overhead ;-) Feel free to ask in case you need more detailed pointers ... Stefan ----- Original Message ----- > Changeset: da0067120542 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=da0067120542 > Modified Files: > monetdb5/extras/crackers/crackers_selectholpl_ops.mx > Branch: holindex > Log Message: > > Comment lines that use papi in order to measure hardware events. > > > diffs (59 lines): > > diff --git a/monetdb5/extras/crackers/crackers_selectholpl_ops.mx > b/monetdb5/extras/crackers/crackers_selectholpl_ops.mx > --- a/monetdb5/extras/crackers/crackers_selectholpl_ops.mx > +++ b/monetdb5/extras/crackers/crackers_selectholpl_ops.mx > @@ -77,6 +77,7 @@ crackers_export str CRKrandomholpl_@1(in > */ > #include "monetdb_config.h" > #include "crackers.h" > +//#include <papi.h> > > /* Local support functions and macros */ > @:TypeSwitch(crackOperations)@ > @@ -384,7 +385,7 @@ CRKrandomholpl_@1(int *bid, bit *inclusi > FN->c = FN->c + incr_pieces; > if(FN->weight > 0) > FN->weight = changeWeight(FN,countBatElements,L1); > - fprintf(ofp,"Idle_Query \t 0 \t Bat_id= \t %d \t Pieces= \t %d \t > Wait_time\t %9.6lf \t Selection_time\t %9.6lf \t Low= \t %d \t Thread_id= > \t" SZFMT "\n",FN->bid,FN->c,wt,ct,(int)*low,MT_getpid()); > + fprintf(ofp,"Idle_Query \t 0 \t Bat_id= \t %d \t Pieces= \t %d \t > Wait_time\t %9.6lf \t Selection_time\t %9.6lf \t Thread_id= \t" SZFMT > "\n",FN->bid,FN->c,wt,ct,MT_getpid()); > fclose(ofp); > MT_lock_unset(&frequencylock, "getFrequencyStruct"); > > @@ -1371,6 +1372,12 @@ CRKRangeLeftNilTree_@1(int *vid, int *bi > FrequencyNode *FrequencyStructA = getFrequencyStruct('A'); > FILE *ofp; > char *outputFilename1; > + //long long counters[2]; > + //int PAPI_events[] = { > + // PAPI_L2_DCA, > + // PAPI_L2_DCM}; > + > + //PAPI_start_counters(PAPI_events, 2); > > gettimeofday(&tv0,0); > > @@ -1444,8 +1451,10 @@ CRKRangeLeftNilTree_@1(int *vid, int *bi > FN->c = FN->c + incr_pieces; > FN->weight = changeWeight(FN,countBatElements,L1); > gettimeofday(&tv1,0); ct+=dt(tv0,tv1); tv0=tv1; > - fprintf(ofp,"User_Query \t 0 \t Bat_id= \t %d \t Pieces= \t %d \t > Wait_time\t %9.6lf \t Selection_time\t %9.6lf \t Hgh= \t %d \t Thread_id= > \t" SZFMT "\n",FN->bid,FN->c,wt,ct,(int)*hgh,MT_getpid()); > -+ fclose(ofp); > + //PAPI_read_counters(counters, 2); > + //fprintf(ofp,"User_Query \t 0 \t Bat_id= \t %d \t Pieces= \t %d \t > Wait_time\t %9.6lf \t Selection_time\t %9.6lf \t Thread_id= \t" SZFMT "\t > L2_DCA= \t %lld \t L2_DCM \t > %lld\n",FN->bid,FN->c,wt,ct,MT_getpid(),counters[0],counters[1]); > + fprintf(ofp,"User_Query \t 0 \t Bat_id= \t %d \t Pieces= \t %d \t > Wait_time\t %9.6lf \t Selection_time\t %9.6lf \t Thread_id= \t" SZFMT > "\n",FN->bid,FN->c,wt,ct,MT_getpid()); > + fclose(ofp); > MT_lock_unset(&frequencylock, "getFrequencyStruct"); > > BBPincref(b->batCacheid,TRUE); > @@ -1552,8 +1561,10 @@ CRKRangeLeftNilTree_@1(int *vid, int *bi > if(FN->weight > 0) > FN->weight = changeWeight(FN,countBatElements,L1); > gettimeofday(&tv1,0); ct+=dt(tv0,tv1); tv0=tv1; > - fprintf(ofp,"User_Query \t 0 \t Bat_id= \t %d \t Pieces= \t %d \t > Wait_time\t %9.6lf \t Selection_time\t %9.6lf \t Hgh= \t %d \t Thread_id= > \t" SZFMT "\n",FN->bid,FN->c,wt,ct,(int)*hgh,MT_getpid()); > -+ fclose(ofp); > + //PAPI_read_counters(counters, 2); > + //fprintf(ofp,"User_Query \t 0 \t Bat_id= \t %d \t Pieces= \t %d \t > Wait_time\t %9.6lf \t Selection_time\t %9.6lf \t Thread_id= \t" SZFMT "\t > L2_DCA \t %lld \t L2_DCM= \t %lld > \n",FN->bid,FN->c,wt,ct,MT_getpid(),counters[0],counters[1]); > + fprintf(ofp,"User_Query \t 0 \t Bat_id= \t %d \t Pieces= \t %d \t > Wait_time\t %9.6lf \t Selection_time\t %9.6lf \t Thread_id= \t" SZFMT > "\n",FN->bid,FN->c,wt,ct,MT_getpid()); > + fclose(ofp); > MT_lock_unset(&frequencylock, "getFrequencyStruct"); > > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

Re: MonetDB: default - handle added types also in declared tables
by Sjoerd Mullender 11 Nov '13

11 Nov '13

Is this the fix for bug 3394 that you just closed? If so, was this really intended for the default branch (in which case the bug should be NEXTFEATURERELEASE) or should this have gone to the Feb2013 branch? On 2013-11-11 22:14, Niels Nes wrote: > Changeset: 63c54046f8c4 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=63c54046f8c4 > Modified Files: > sql/backends/monet5/sql.mx > sql/test/BugTracker-2013/Tests/Cannot_Find_Column_Type.Bug-3394.sql > Branch: default > Log Message: > > handle added types also in declared tables > > > diffs (43 lines): > > diff --git a/sql/backends/monet5/sql.mx b/sql/backends/monet5/sql.mx > --- a/sql/backends/monet5/sql.mx > +++ b/sql/backends/monet5/sql.mx > @@ -3993,12 +3993,13 @@ mvc_declared_table_column_wrap(Client cn > str msg = getSQLContext(cntxt,mb, &m, NULL); > sql_schema *s = NULL; > sql_table *t = NULL; > + sql_type *type = NULL; > sql_subtype tpe; > int *ret = (int *) getArgReference(stk, pci, 0); > int *rs = (int *) getArgReference(stk, pci, 1); > str *tname = (str *) getArgReference(stk, pci, 2); > str *name = (str *) getArgReference(stk, pci, 3); > - str *type = (str *) getArgReference(stk, pci, 4); > + str *typename = (str *) getArgReference(stk, pci, 4); > int *digits = (int *) getArgReference(stk, pci, 5); > int *scale = (int *) getArgReference(stk, pci, 6); > > @@ -4006,8 +4007,11 @@ mvc_declared_table_column_wrap(Client cn > return msg; > if (*rs != 0) > throw(SQL, "sql.dtColumn", "Cannot access declared table"); > - if (!sql_find_subtype(&tpe, *type, *digits, *scale)) > + if (!sql_find_subtype(&tpe, *typename, *digits, *scale) && > + (type = mvc_bind_type(m, *typename)) == NULL) > throw(SQL, "sql.dtColumn", "Cannot find column type"); > + if (type) > + sql_init_subtype(&tpe, type, 0, 0); > s = mvc_bind_schema(m, dt_schema); > if ( s == NULL) > throw(SQL,"sql.declared_table_column","3F000!Schema missing"); > diff --git a/sql/test/BugTracker-2013/Tests/Cannot_Find_Column_Type.Bug-3394.sql b/sql/test/BugTracker-2013/Tests/Cannot_Find_Column_Type.Bug-3394.sql > --- a/sql/test/BugTracker-2013/Tests/Cannot_Find_Column_Type.Bug-3394.sql > +++ b/sql/test/BugTracker-2013/Tests/Cannot_Find_Column_Type.Bug-3394.sql > @@ -45,3 +45,8 @@ begin > end; > > select tmp4(); > + > +drop function tmp4; > +drop function tmp3; > +drop function tmp2; > +drop function tmp1; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

How do I get SQL decimal scale in MAL?
by Dru Nelson 11 Nov '13

11 Nov '13

HI, I"m working on writing an aggregate in in C that is a MAL function. I've run into a problem with decimal types. I've noticed that when I inspect the BAT inputs for various types, they are coerced to simpler types. (flt,dbl -> dbl). (int, decimal -> int). There is also no other information in the BAT structure to see the type. I notice that a SQL select gets the type info passed in as parameters for the 'exportValue' output. For example: >* sql.exportValue(1,".","second_single_value","decimal",9,3,8,_4,""); * How can I get this type information ('decimal', etc.) passed into my function? Dru

3 9

about "MT" in gdk module
by 张磊 06 Nov '13

06 Nov '13

hi all, I found that in gdk module of monetdb, the file gdk_posix.c has many functions which begin with “MT”, and I want to ask that what's "MT", is that "MoneT", "Multiple Thread" or others? what's the meaning of "MT"? :) -- Best Wishes! zhanglei

2 1

Monetdb ODBC datatype conversion problem
by baraa Mohamad 02 Nov '13

02 Nov '13

Dear MonetDB users, I have created a database link from oracle to monetdb using Monetdb ODBC driver, the link works fine but i have a strange results 😓 I have the following configurations: host1: Oracle (ubuntu 64bit)host2: MonetDB (ubntu 64bit)ODBC driver: MonetDB odbc (64bit) using unixODBC driver manager on monetdb i have the following table: emp (id integer , name varchar); test query: select id from emp where id<20; when I execute this query directly on montdb I get the following results (IDs from 1 to 19)ID123..19 BUT when I execute this query from oracle using dblink select "id" from "emp"@monetdb where "id"<20 ; I GET THE FOLLOWING RESULTS: ID-------- 18589934592 21.2885E+10 31.7180E+10 42.1475E+10 52.5770E+10 63.0065E+10 73.4360E+10 83.8655E+10 94.2950E+10 10 I cannot figure out the cause of the problem and how I can solve it (it seems a problem related to datatype conversion using ODBC) Could you please tell me what I should do.... Best regards,Baraa

1 0

Re: MonetDB: newstorage - pushing changes from laptop
by Lefteris 30 Oct '13

30 Oct '13

what is interesting is not from where you are pushing the changes but what the changes do:)) On Wed, Oct 30, 2013 at 10:30 AM, Rene Tassy <commits(a)monetdb.org> wrote: > Changeset: fb471128db99 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fb471128db99 > Modified Files: > sql/storage/bat/bat_storage.c > Branch: newstorage > Log Message: > > pushing changes from laptop > > > diffs (144 lines): > > diff --git a/sql/storage/bat/bat_storage.c b/sql/storage/bat/bat_storage.c > --- a/sql/storage/bat/bat_storage.c > +++ b/sql/storage/bat/bat_storage.c > @@ -41,6 +41,14 @@ timestamp_dbat( sql_dbat *d, int ts) > return d; > } > > +/**static sql_timestamps * > +timestamp_tsbats(sql_timestamps *ts, int timestamp) { > + > + while (ts->next && ts->wtime > timestamp) > + ts = ts->next; > + return ts; > +}*/ > + > > static BAT * > delta_bind_del(sql_dbat *bat, int access) > @@ -1222,16 +1230,6 @@ load_timestamps(sql_timestamps *ts, int > } > > static int > -load_dbat(sql_dbat *bat, int bid) { > - > - BAT *b = quick_descriptor(bid); > - > - bat->dbid = temp_create(b); > - bat->cnt = BATcount(b); > - return LOG_OK; > -} > - > -static int > create_timestamps(sql_trans *tr, sql_table *t) { > > int ok = LOG_OK; > @@ -1275,6 +1273,16 @@ create_timestamps(sql_trans *tr, sql_tab > } > > static int > +load_dbat(sql_dbat *bat, int bid) { > + > + BAT *b = quick_descriptor(bid); > + > + bat->dbid = temp_create(b); > + bat->cnt = BATcount(b); > + return LOG_OK; > +} > + > +static int > create_del(sql_trans *tr, sql_table *t) > { > int ok = LOG_OK; > @@ -1369,8 +1377,7 @@ static int > snapshot_create_del(sql_trans *tr, sql_table *t) > { > sql_dbat *bat = t->data; > - BAT *b = temp_descriptor(bat->dbid); > - //BAT *trdb = temp_descriptor(bat->dtrbid); > + BAT *b = temp_descriptor(bat->dbid);; > > printf("in snapshot_create_del\n"); > > @@ -1383,13 +1390,6 @@ snapshot_create_del(sql_trans *tr, sql_t > BATmmap(b, STORE_MMAP, STORE_MMAP, STORE_MMAP, STORE_MMAP, 0); > bat_destroy(b); > > - /*bat_set_access(trdb, BAT_READ); > - if (BATcount(trdb) > SNAPSHOT_MINSIZE) > - BATmode(trdb, PERSISTENT); > - if (BATcount(trdb) > (BUN) REMAP_PAGE_MAXSIZE) > - BATmmap(trdb, STORE_MMAP, STORE_MMAP, STORE_MMAP, STORE_MMAP, 0); > - bat_destroy(trdb);*/ > - > return LOG_OK; > } > > @@ -1539,16 +1539,12 @@ destroy_dbat(sql_trans *tr, sql_dbat *ba > > if (bat->dname) > _DELETE(bat->dname); > - //if (bat->dtrname) > - //_DELETE(bat->dtrname); > + > if (bat->dbid) > temp_destroy(bat->dbid); > - //if (bat->dtrbid) > - //temp_destroy(bat->dtrbid); > + > bat->dbid = 0; > bat->dname = NULL; > - //bat->dtrbid = 0; > - //bat->dtrname = NULL; > _DELETE(bat); > if (n) > return destroy_dbat(tr, n); > @@ -1798,20 +1794,39 @@ clear_dbat(sql_trans *tr, sql_dbat *bat) > BATcommit(b); > bat_destroy(b); > } > - /*if(bat->dtrbid) { > - BAT *b = temp_descriptor(bat->dtrbid); > - > - sz+= BATcount(b); > - bat_clear(b); > - BATcommit(b); > - bat_destroy(b); > - }*/ > + > bat->cnt = 0; > bat->wtime = tr->wstime; > > return sz; > } > - > +/** > +static BUN > +clear_tsbats(sql_trans *tr, sql_timestamps *ts) { > + > + BUN sz =0; > + > + (void)tr; > + if(ts->insbid) { > + BAT *b = temp_descriptor(ts->insbid); > + > + sz += BATcount(b) > + } > +} > + > +static BUN > +clear_timestamps(sql_trans *tr, sql_table *t) { > + > + if(!t->timestamps || !t->base.allocated) { > + sql_table *ot = tr_find_table(tr->parent, t); > + sql_timestamps *ts = t->timestamps = ZNEW(sql_timestamps), *ots = timestamp_tsbats(ots->timestamps, tr->stime); > + dup_tsbats(tr, ts, ots, isNew(ot), isTempTable(t)); > + t->base.allocated = 1; > + } > + > + return clear_tsbats(tr, t->timestamps); > +} > +*/ > static BUN > clear_del(sql_trans *tr, sql_table *t) > { > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

1 0

Re: MonetDB: default - (try to) run GSL tests only if we do have GSL...
by Arjen de Rijke 29 Oct '13

29 Oct '13

Hoi Stefan, Dit zou toch niet nodig hoeven zijn. Dit wordt al geregeld via de conditional in de makefile van /sql/backends/monet5/, of vergis ik mij? De andere checkin zal ik even onthouden voor een volgende keer. Arjen ----- Original Message ----- > From: "Stefan Manegold" <commits(a)monetdb.org> > To: checkin-list(a)monetdb.org > Sent: Tuesday, October 29, 2013 9:44:26 AM > Subject: MonetDB: default - (try to) run GSL tests only if we do have GSL... > > Changeset: bb53a64a83fe for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=bb53a64a83fe > Modified Files: > sql/backends/monet5/gsl/Tests/All > Branch: default > Log Message: > > (try to) run GSL tests only if we do have GSL support > > > diffs (8 lines): > > diff --git a/sql/backends/monet5/gsl/Tests/All > b/sql/backends/monet5/gsl/Tests/All > --- a/sql/backends/monet5/gsl/Tests/All > +++ b/sql/backends/monet5/gsl/Tests/All > @@ -1,2 +1,2 @@ > -gsl00 > -gsl01 > +HAVE_GSL?gsl00 > +HAVE_GSL?gsl01 > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

2 2

Can an user function return more than one BAT?
by Chi-Young Ku 23 Oct '13

23 Oct '13

Hi, I am looking for a way for a MAL user function to return more than one BAT without success. Could you give me some help? I tried the following way: user.test(X_1:bat[:oid,:oid], X_2:bat[:oid,:int]):void; sql.init(); X_0 := sql.mvc(); X_1 := sql.bind(...); X_2 := sql.bind(...); end test; X1:bat[:oid,:oid] := bat.new(:oid,:oid); X2:bat[:oid,:int] := bat.new(:oid,:int); X_1:str := remote.put(url, X1); X_2:str := remote.put(url, X2); X_3:str := remote.exec(url, "user", "test", X_1, X_2); X3:bat[:oid,:oid] := remote.get(url, X_1); X4:bat[:oid,:int] := remote.get(url, X_2); io.print(X3); io.print(X4);

1 0

Monetdb python error in connection
by Rose Beck 20 Oct '13

20 Oct '13

I am using python-monetdb 11.16.0.7. I created my database farm and database according to instructions given below (source: http://www.monetdb.org/Documentation/monetdbd) % monetdbd start /home/my-dbfarm % monetdb create my-first-db Then I tried to connect to the database using the below mentioned command in python(https://pypi.python.org/pypi/python-monetdb/). Upon doing so I am getting the below mentioned error: >import monetdb.sql >connection=monetdb.sql.connect(username="monetdb",password="monetdb",hostname="localhost",database="my-first-db"); File "/usr/local/lib/python2.7/dist-packages/monetdb/sql/__init__.py", line 28, in connect return Connection(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/monetdb/sql/connections.py", line 58, in __init__ unix_socket=unix_socket) File "/usr/local/lib/python2.7/dist-packages/monetdb/mapi.py", line 93, in connect self.socket.connect((hostname, port)) File "/usr/lib/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 111] Connection refused Can someone please help me with this? Also my my-first-db resides inside my db-farm? I am not getting as to what should I do?

3 2

Re: MonetDB: newstorage - creating delete-transaction bat
by Martin Kersten 19 Oct '13

19 Oct '13

If intended for debugging only, wrap them in #ifdef DEBUG_STORE ... #endif On 10/19/13 6:26 PM, Rene Tassy wrote: > Changeset: d5436b5b660d for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=d5436b5b660d > Modified Files: > sql/backends/monet5/sql_scenario.c > sql/server/sql_mvc.c > sql/storage/bat/bat_storage.c > sql/storage/bat/bat_storage.h > sql/storage/bat/bat_table.c > sql/storage/store.c > Branch: newstorage > Log Message: > > creating delete-transaction bat > > > diffs (truncated from 1821 to 300 lines): > > diff --git a/sql/backends/monet5/sql_scenario.c b/sql/backends/monet5/sql_scenario.c > --- a/sql/backends/monet5/sql_scenario.c > +++ b/sql/backends/monet5/sql_scenario.c > @@ -1932,6 +1932,9 @@ SQLparser(Client c) > } > } > finalize: > + > + printf("-----finalizing\n"); This printf is not correct, use mstr_printf(c->fdout,.... > + > if (msg) > sqlcleanup(m, 0); > return msg; > diff --git a/sql/server/sql_mvc.c b/sql/server/sql_mvc.c > --- a/sql/server/sql_mvc.c > +++ b/sql/server/sql_mvc.c > @@ -182,6 +182,8 @@ mvc_trans(mvc *m) > int schema_changed = 0, err = m->session->status; > assert(!m->session->active); /* can only start a new transaction */ > > + printf("in mvc_trans\n"); > + > store_lock(); > schema_changed = sql_trans_begin(m->session); > if (m->qc && (schema_changed || m->qc->nr > m->cache || err)){ > @@ -206,6 +208,8 @@ mvc_commit(mvc *m, int chain, char *name > assert(tr); > assert(m->session->active); /* only commit an active transaction */ > > + printf("in mvc commit\n"); > + > if (mvc_debug) > fprintf(stderr, "#mvc_commit %s\n", (name) ? name : ""); > > @@ -288,6 +292,7 @@ build up the hash (not copyied in the tr > mvc_rollback(m, chain, name); > return -1; > } > + > sql_trans_end(m->session); > if (chain) > sql_trans_begin(m->session); > @@ -304,6 +309,8 @@ mvc_rollback(mvc *m, int chain, char *na > int res = 0; > sql_trans *tr = m->session->tr; > > + printf("in mvc_rollback"); > + > if (mvc_debug) > fprintf(stderr, "#mvc_rollback %s\n", (name) ? name : ""); > > @@ -340,6 +347,7 @@ mvc_rollback(mvc *m, int chain, char *na > tr = sql_trans_destroy(tr); > } > m->session-> tr = tr; > + > /* make sure we do not reuse changed data */ > if (tr->wtime) > tr->status = 1; > diff --git a/sql/storage/bat/bat_storage.c b/sql/storage/bat/bat_storage.c > --- a/sql/storage/bat/bat_storage.c > +++ b/sql/storage/bat/bat_storage.c > @@ -12,7 +12,7 @@ > * The Original Code is the MonetDB Database System. > * > * The Initial Developer of the Original Code is CWI. > - * Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > + Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > * Copyright August 2008-2013 MonetDB B.V. > * All Rights Reserved. > */ > @@ -50,6 +50,9 @@ delta_bind_del(sql_dbat *bat, int access > #ifdef NDEBUG > (void) access; /* satisfy compiler */ > #endif > + > + //printf("in delta_bind_del\n"); > + > assert(access == RDONLY || access == RD_INS); > assert(access!=RD_UPD); > > @@ -61,6 +64,9 @@ delta_bind_del(sql_dbat *bat, int access > static BAT * > bind_del(sql_trans *tr, sql_table *t, int access) > { > + > + //printf("in bind_del\n"); > + > if (!t->data) { > sql_table *ot = tr_find_table(tr->parent, t); > t->data = timestamp_dbat(ot->data, tr->stime); > @@ -77,6 +83,8 @@ delta_bind_ubat(sql_delta *bat, int acce > #ifdef NDEBUG > (void) access; /* satisfy compiler */ > #endif > + printf("in delta_bind_ubat\n"); > + > assert(access == RD_UPD); > if (bat->ubid) > b = temp_descriptor(bat->ubid); > @@ -91,6 +99,8 @@ bind_ucol(sql_trans *tr, sql_column *c, > { > BAT *u = NULL, *d, *r; > > + printf("in bind_ucol\n"); > + > if (!c->data) { > sql_column *oc = tr_find_column(tr->parent, c); > c->data = timestamp_delta(oc->data, tr->stime); > @@ -115,6 +125,8 @@ bind_uidx(sql_trans *tr, sql_idx * i, in > { > BAT *u = NULL, *d, *r; > > + printf("in bind_uidx\n"); > + > if (!i->data) { > sql_idx *oi = tr_find_idx(tr->parent, i); > i->data = timestamp_delta(oi->data, tr->stime); > @@ -139,6 +151,8 @@ delta_bind_bat( sql_delta *bat, int acce > { > BAT *b; > > + printf("in delta_bind_bat\n"); > + > assert(access == RDONLY || access == RD_INS); > assert(bat != NULL); > if (temp || access == RD_INS) { > @@ -169,6 +183,8 @@ delta_bind_bat( sql_delta *bat, int acce > static BAT * > bind_col(sql_trans *tr, sql_column *c, int access) > { > + printf("in bind_col\n"); > + > if (!c->data) { > sql_column *oc = tr_find_column(tr->parent, c); > c->data = timestamp_delta(oc->data, tr->stime); > @@ -183,6 +199,9 @@ bind_col(sql_trans *tr, sql_column *c, i > static BAT * > bind_idx(sql_trans *tr, sql_idx * i, int access) > { > + > + printf("in bind_idx\n"); > + > if (!i->data) { > sql_idx *oi = tr_find_idx(tr->parent, i); > i->data = timestamp_delta(oi->data, tr->stime); > @@ -200,6 +219,8 @@ delta_update_bat( sql_delta *bat, BAT *t > BAT *b; > BAT *upd = BATleftfetchjoin(BATmirror(tids), updates, BATcount(tids)); > > + printf("in delta_update_bat\n"); > + > if (bat->cached) { > bat_destroy(bat->cached); > bat->cached = NULL; > @@ -259,6 +280,8 @@ delta_update_val( sql_delta *bat, oid ri > { > BAT *b = NULL; > > + printf("in delta_update_val\n"); > + > assert(rid != oid_nil); > > if (bat->cached) { > @@ -297,6 +320,9 @@ delta_update_val( sql_delta *bat, oid ri > static int > dup_delta(sql_trans *tr, sql_delta *obat, sql_delta *bat, int type, int oc_isnew, int c_isnew, int temp, int sz) > { > + > + printf("in dup_delta\n"); > + > if (!obat) > return LOG_OK; > bat->ibid = obat->ibid; > @@ -359,6 +385,8 @@ dup_delta(sql_trans *tr, sql_delta *obat > int > dup_bat(sql_trans *tr, sql_table *t, sql_delta *obat, sql_delta *bat, int type, int oc_isnew, int c_isnew) > { > + printf("in dup_bat\n"); > + > return dup_delta( tr, obat, bat, type, oc_isnew, c_isnew, isTempTable(t), t->sz); > } > > @@ -368,6 +396,8 @@ update_col(sql_trans *tr, sql_column *c, > BAT *b = tids; > sql_delta *bat; > > + printf("in update_col\n"); > + > if (tpe == TYPE_bat && !BATcount(b)) > return; > > @@ -393,6 +423,8 @@ update_idx(sql_trans *tr, sql_idx * i, v > BAT *b = tids; > sql_delta *bat; > > + printf("in update_idx\n"); > + > if (tpe == TYPE_bat && !BATcount(b)) > return; > > @@ -422,6 +454,8 @@ delta_append_bat( sql_delta *bat, BAT *i > assert(!c || c->htype == TYPE_void); > #endif > > + printf("in delta_append_bat\n"); > + > if (!BATcount(i)) > return ; > b = temp_descriptor(bat->ibid); > @@ -460,12 +494,15 @@ delta_append_bat( sql_delta *bat, BAT *i > static void > delta_append_val( sql_delta *bat, void *i ) > { > + > BAT *b = temp_descriptor(bat->ibid); > #ifndef NDEBUG > BAT *c = BBPquickdesc(bat->bid, 0); > assert(!c || c->htype == TYPE_void); > #endif > > + printf("in delta append val\n"); > + > assert(b->htype == TYPE_void); > if (bat->cached) { > bat_destroy(bat->cached); > @@ -503,6 +540,8 @@ dup_idx(sql_trans *tr, sql_idx *i, sql_i > { > int ok = LOG_OK; > > + printf("in dup_idx\n"); > + > if (i->data) { > int type = (ni->type==join_idx)?TYPE_oid:TYPE_wrd; > sql_delta *bat = ni->data = ZNEW(sql_delta), *obat = i->data; > @@ -515,15 +554,27 @@ dup_idx(sql_trans *tr, sql_idx *i, sql_i > static int > dup_dbat( sql_trans *tr, sql_dbat *obat, sql_dbat *bat, int is_new, int temp) > { > + > + printf("in dup_dbat\n"); > + > bat->dbid = obat->dbid; > + > + bat->dtrbid = obat->dtrbid; > + > bat->cnt = obat->cnt; > + > bat->dname = _STRDUP(obat->dname); > + > + bat->dtrname = _STRDUP(obat->dtrname); > + > bat->wtime = obat->wtime; > if (bat->dbid) { > if (is_new) { > obat->dbid = temp_copy(bat->dbid, temp); > + obat->dtrbid = temp_copy(bat->dtrbid, temp); > } else { > bat->dbid = ebat_copy(bat->dbid, 0, temp); > + bat->dtrbid = ebat_copy(bat->dtrbid, 0, temp); > } > } > (void)tr; > @@ -546,6 +597,8 @@ append_col(sql_trans *tr, sql_column *c, > BAT *b = i; > sql_delta *bat; > > + printf("in append_col\n"); > + > if (tpe == TYPE_bat && !BATcount(b)) > return; > > @@ -574,6 +627,8 @@ append_idx(sql_trans *tr, sql_idx * i, v > BAT *b = ib; > sql_delta *bat; > > + printf("in append_idx\n"); > + > if (tpe == TYPE_bat && !BATcount(b)) > return; > > @@ -598,6 +653,8 @@ delta_delete_bat( sql_dbat *bat, BAT *i > { > BAT *b = temp_descriptor(bat->dbid); > > + printf("in delta_delete_bat\n"); > + > if (isEbat(b)) { > temp_destroy(bat->dbid); > bat->dbid = temp_copy(b->batCacheid, FALSE); > @@ -616,6 +673,8 @@ delta_delete_val( sql_dbat *bat, oid rid > { > BAT *b = temp_descriptor(bat->dbid); > > + printf("in delta_delete_val\n"); > + > if (isEbat(b)) { > temp_destroy(bat->dbid); > bat->dbid = temp_copy(b->batCacheid, FALSE); > @@ -636,6 +695,8 @@ delete_tab(sql_trans *tr, sql_table * t, > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

1 0

Announcement: New Feb2013-SP5 Bugfix release of MonetDB suite
by Hannes Muehleisen 16 Oct '13

16 Oct '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013-SP5 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Feb 2013-SP5 bugfix release Client Package * ODBC: Implemented retrieving variable-length data in parts with SQLGetData. See bug 3381. MonetDB Common * Removed the heap cache. Since the fix for bug 3323 which made that the cache was actually getting used, bug 3376 made clear that it didn't work very well. In addition, on Linux at least, the heap cache slows things down. Java Module * Rearranged order of returned columns of certain metadata functions to comply with the JDBC documentation. See bug 3356. Bug Fixes * 3323: Heapcache bugs/performance issues * 3331: SAMPLE will return same result every time. * 3356: DatabaseMetaData.getColumns() doesn't work correctly when using index-based getters * 3367: Fully qualified order by column gives "relational query without result" * 3368: BAT sortedness info ignored on ORDER BY and TOPN * 3370: SQL environment settings are updated even when the set statement fails * 3373: Setting table to read-only (Bug 3362) fails on big-endian * 3375: LIKE join: BATfetchjoin does not hit always * 3376: COPY INTO fails with HEAPextend: failed to extend: MT_mremap() failed * 3377: Query interfering with next query in same transaction, after SP4 * 3380: Python DBAPI driver throws exception when fetching EXPLAIN results * 3381: Windows ODBC driver hangs or crashes on simple queries

1 0

How to transfer monetdb database to another machine
by Rose Beck 13 Oct '13

13 Oct '13

I created monetdb database using the python library https://pypi.python.org/pypi/python-monetdb/11.16.0.7 on my ubuntu 12.04 lts server. Now I want to transfer this database to another ubuntu 12.04 lts server. However, I am unable to understand as to where(directory name) does monetdb create its database (within ubuntu 12.04) and what is the procedure which I must follow to transfer this database to another machine (so that i may query the database on another machine)?

1 0

Re: MonetDB Feb2013-SP5 Release Candidate available for testing
by Hannes Mühleisen 09 Oct '13

09 Oct '13

Hi Pete and List, On 10/09/2013 09:50 AM, Pete Hollobon wrote: > Thank you for this, we'll give it a try. Great! > Can this version be used with an existing dbfarm, or is a dump / load > necessary? This should work fine with other Feb2013 dbfarms. best, Hannes

1 0

SQLAlchemy-monetdb and change to Python API
by Gijs Molenaar 01 Oct '13

01 Oct '13

Hi monetDB people, The sqlalchemy dialect for monetdb is in quite good shape now, so I made a 0.1 release. It is also available throught Pypi. anybody who wants to give it a test and has some feedback, check it out: https://github.com/gijzelaerr/sqlalchemy-monetdb To add support for the upcomming 0.9 release I had to modify the MonetDB Python API a bit, so it support subtypes of python build-in types. I've added some unittests so the test output will change. Can one of you update the test output, I don't have a the monetdb testing output update script working on this machine. greetings, -- Gijs Molenaar http://www.astro.uva.nl/people/gijs-molenaar/

1 0

TODO statement
by Gijs Molenaar 01 Oct '13

01 Oct '13

Hi MonetDB people, I encountered this TODO message while running a query: sql>SELECT objects.name more> FROM sys.keys, sys.objects, sys.tables, sys.schemas more> WHERE sys.keys.id = objects.id more> AND keys.table_id = tables.id more> AND tables.schema_id = schemas.id more> AND keys.type = 0 more> AND tables.id = 11323; TODO: column names of level >= 3 -- Gijs Molenaar http://www.astro.uva.nl/people/gijs-molenaar/

3 3

timestamp microsecond precision
by Gijs Molenaar 30 Sep '13

30 Sep '13

Hi MonetDB people, Do the TIME(STAMP) types support microsecond precision? They accept a 'postint' argument, but no matter what I set it to it looks like the timestamp is rounded to milliseconds. greetings, -- Gijs Molenaar http://www.astro.uva.nl/people/gijs-molenaar/

2 1

Announcement: New Feb2013-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 27 Sep '13

27 Sep '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Feb 2013-SP4 bugfix release Merovingian * monetdb now no longer compresses output when not connected to a terminal, bug #3292 Bug Fixes * 2781: SQL localtime() returns GMT time * 2962: client timezone information needs to be handled also for modifications to timestamp values * 3232: The function weekofyear returns results not according to the iso 8601 standard * 3255: M5 tests inlineFunction & ifthencst fail since changeset 207ed7d7e03a * 3292: monetdb status should print full line * 3307: distinct multicolumn performance and implementation suggestion for count distinct * 3310: Cast value to integer gives wrong result * 3311: strLength assertion on join and concat * 3313: Assertion in rel2bin_project * 3315: Incorrect behavior with subqueries in DELETE * 3317: correlated subselect does not work * 3318: UPDATE does not work with DECLEAREd TABLE in FUNCTION * 3319: UNIQUE column in DECLAREd TABLE in FUNCTION results in segfault * 3329: Function AVG fails during runtime in a procedure * 3330: Corrupt mitosis plan with group-by query * 3332: MEDIAN: no such operator 'median(decimal)' * 3336: DB files not removed if all rows are deleted, even after restart * 3337: Monet ODBC gives wrong schema information * 3339: Segvault in complex function call * 3340: crash on seeming incorrect SQL query * 3341: segfault caused by select rank() on a newly-loaded table * 3342: incorrect automatic cast from decimal to tinyint * 3345: copy binary into fails with the wrong error message * 3346: Deadlock in mal_dataflow's q_dequeue() (SciQL-2 branch) * 3348: SQL: LIKE incorrect with _ * 3349: When selecting everything from a joined result, column contents are incorrect in one of my use cases. * 3351: gzip'ed files cannot be bulkloaded into DB * 3352: Median function with decimal type doesn't work * 3353: Readonly tables call for UNSET * 3354: Introduce query time-out * 3358: Remote client connection stays in CLOSE_WAIT state * 3360: uninitialized memory causing bad output in query plans * 3362: ALTER TABLE operations silently remove previously set READ ONLY property * 3363: a join causes SIGSEGV after creation of Unique Key * 3367: Fully qualified order by column gives "relational query without result" * 3369: \0 Byte appears in MAPI

1 0

Whats coming in upcoming build monetdb release
by Ashish Kumar Singh 21 Sep '13

21 Sep '13

Hi , During my periodic search for "Is a new monetdb build is going to be released soon" , I found one is placed under testing recently. Is there a place where it can be seen and reviewed what kind of features/fixes are expected in upcoming release? That will be a great help for some of the folks like me to plan ahead! Regards, Ashish

1 0

Disk space to store a table?
by baraa Mohamad 10 Sep '13

10 Sep '13

Dear MonetDB Developers, I'm doing some research about colmn oriented databases and I began with MonetDB. In fact I would like to know how can I get the real disk space used to store a given tabel?? After some research on the web I found that I can use the storage table as follow: select "schema", "table", sum("columnsize") as size from storage() group by "schema","table" having "schema" <> 'tmp'; But it seems to me that this is something like a "logical" storage space without any compression (all columns with type "int" have the same size), is there a way to know the actual disk space used by MonetDB to store a table?? Thank you in advance for your help, RegardsBaraa

1 0

Re: MonetDB: int128 - backing-out changeset 77c9c3da562c:
by Martin Kersten 08 Sep '13

08 Sep '13

On 9/8/13 8:45 PM, Stefan Manegold wrote: > Changeset: a331af9e20ea for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a331af9e20ea > Modified Files: > monetdb5/modules/mal/batcalc.mal > monetdb5/modules/mal/batcalc.mal.sh > Branch: int128 > Log Message: > > backing-out changeset 77c9c3da562c: > > Unfortunately, MonetDB / mserver5 / MAL parser / MAL interpreter > is/are unable to handle the new monetdb5/modules/mal/batcalc.mal > correctly. > > When starting the server, it complains about > !TypeException:jaqlfunc.count[7]:'batcalc.lng' undefined in: r:any := batcalc.lng(x:bat[:oid,:wrd]) > although the respective batcalc.lng(x:bat[:oid,:wrd]) > is in monetdb5/modules/mal/batcalc.mal . > Some tests indicate that the systems (MAL parser ?) > has some problems handling MAL files >= ~1 MB > correctly, but I did not manage to locate the > actual nature or origin of the problem. > > Any help to get this working is highly appreciated. Most likely it is in mal_import where the complete mal file is loaded first

1 0

Auto-vectorization
by ajonwa＠gmx.com 07 Sep '13

07 Sep '13

Hello All, Has anyone managed to auto-vectorize (automatic SIMDfication) any of the loops in the kernel? If yes, I would like to know the complier and the complier flags used. Thanks, Geoffrey

1 0

Re: MonetDB: default - add support for unix sockets
by Stefan Manegold 06 Sep '13

06 Sep '13

Hi Gijs, in case you did not notice this, yet: This checkin breaks several tests that do test or use the Python interface, resulting in errors like " Traceback (most recent call last): File "delete_all.SQL.py", line 9, in <module> dbh = monetdb.sql.Connection(port=port,database=db,autocommit=True) File "PREFIX/lib/python2.7/site-packages/monetdb/sql/connections.py", line 49, in __init__ unix_socket=unix_socket) File "PREFIX/lib/python2.7/site-packages/monetdb/mapi.py", line 96, in connect self.socket.connect(unix_socket) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 2] No such file or directory " or " Traceback (most recent call last): File "SOURCE/clients/examples/python/sqlsample.py", line 23, in <module> dbh = monetdb.sql.Connection(port=int(sys.argv[1]),database=sys.argv[2],autocommit=True) File "PREFIX/lib/python3.3/site-packages/monetdb/sql/connections.py", line 49, in __init__ unix_socket=unix_socket) File "PREFIX/lib/python3.3/site-packages/monetdb/mapi.py", line 97, in connect self.socket.connect(unix_socket) FileNotFoundError: [Errno 2] No such file or directory " See also, e.g., `Mtest.py sql/benchmarks/tpch/fileleak sql/test/concurrent sql/test/mapi` or our nightly TestWeb @ http://monetdb.cwi.nl/testweb/web/status.php?branch=default as of tomorrow morning. Best, Stefan On Thu, Sep 05, 2013 at 02:19:01PM +0200, Gijs Molenaar wrote: > Changeset: 0eaa07b061be for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=0eaa07b061be > Modified Files: > clients/python2/monetdb/control.py > clients/python2/monetdb/mapi.py > clients/python2/monetdb/sql/connections.py > clients/python2/test/runtests.py > clients/python2/test/test_control.py > clients/python3/monetdb/control.py > clients/python3/monetdb/mapi.py > clients/python3/monetdb/sql/connections.py > clients/python3/test/runtests.py > clients/python3/test/test_control.py > Branch: default > Log Message: > > add support for unix sockets > > > diffs (truncated from 649 to 300 lines): > > diff --git a/clients/python2/monetdb/control.py b/clients/python2/monetdb/control.py > --- a/clients/python2/monetdb/control.py > +++ b/clients/python2/monetdb/control.py > @@ -8,10 +8,12 @@ def parse_statusline(line): > parses a sabdb format status line. Support v1 and v2. > > """ > - if not line.startswith('=sabdb:'): > + if line.startswith("="): > + line = line[1:] > + if not line.startswith('sabdb:'): > raise OperationalError('wrong result recieved') > > - prot_version, rest = line.split(":", 2)[1:] > + code, prot_version, rest = line.split(":", 2) > > if prot_version not in ["1", "2"]: > raise InterfaceError("unsupported sabdb protocol") > @@ -60,20 +62,26 @@ class Control: > Use this module to manage your MonetDB databases. You can create, start, > stop, lock, unlock, destroy your databases and request status information. > """ > - def __init__(self, hostname, port, passphrase): > + def __init__(self, hostname=None, port=None, passphrase=None, > + unix_socket="/tmp/.s.merovingian.50000"): > self.server = mapi.Connection() > self.hostname = hostname > self.port = port > self.passphrase = passphrase > + self.unix_socket= unix_socket > > # check connection > - self.server.connect(hostname, port, 'monetdb', passphrase, > - 'merovingian', 'control') > + self.server.connect(hostname=hostname, port=port, username='monetdb', > + password=passphrase, > + database='merovingian', language='control', > + unix_socket=unix_socket) > self.server.disconnect() > > def _send_command(self, database_name, command): > - self.server.connect(self.hostname, self.port, 'monetdb', > - self.passphrase, 'merovingian', 'control') > + self.server.connect(hostname=self.hostname, port=self.port, > + username='monetdb', password=self.passphrase, > + database='merovingian', language='control', > + unix_socket=self.unix_socket) > try: > return self.server.cmd("%s %s\n" % (database_name, command)) > finally: > @@ -165,8 +173,9 @@ class Control: > """ > properties = self._send_command(database_name, "get") > values = {} > - for dirty_line in properties.split("\n"): > - line = dirty_line[1:] > + for line in properties.split("\n"): > + if line.startswith("="): > + line = line[1:] > if not line.startswith("#"): > if "=" in line: > split = line.split("=") > diff --git a/clients/python2/monetdb/mapi.py b/clients/python2/monetdb/mapi.py > --- a/clients/python2/monetdb/mapi.py > +++ b/clients/python2/monetdb/mapi.py > @@ -70,8 +70,12 @@ class Connection(object): > self.database = "" > self.language = "" > > - def connect(self, hostname, port, username, password, database, language): > - """ setup connection to MAPI server""" > + def connect(self, database, username, password, language, hostname=None, > + port=None, unix_socket=None): > + """ setup connection to MAPI server > + > + unix_socket is used if hostname is not defined. > + """ > > self.hostname = hostname > self.port = port > @@ -79,24 +83,34 @@ class Connection(object): > self.password = password > self.database = database > self.language = language > + self.unix_socket = unix_socket > > - self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > + if hostname: > + self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > + # For performance, mirror MonetDB/src/common/stream.c socket settings. > + self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 0) > + self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) > + self.socket.connect((hostname, port)) > + else: > + self.socket = socket.socket(socket.AF_UNIX) > + self.socket.connect(unix_socket) > + if self.language != 'control': > + self.socket.send('0') # don't know why, but we need to do this > > - # For performance, mirror MonetDB/src/common/stream.c socket settings. > - self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 0) > - self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) > + if not (self.language == 'control' and not self.hostname): > + # control doesn't require authentication over socket > + self._login() > > - self.socket.connect((hostname, port)) > - self.__login() > + self.state = STATE_READY > > - def __login(self, iteration=0): > + def _login(self, iteration=0): > """ Reads challenge from line, generate response and check if > everything is okay """ > > - challenge = self.__getblock() > - response = self.__challenge_response(challenge) > - self.__putblock(response) > - prompt = self.__getblock().strip() > + challenge = self._getblock() > + response = self._challenge_response(challenge) > + self._putblock(response) > + prompt = self._getblock().strip() > > if len(prompt) == 0: > # Empty response, server is happy > @@ -117,7 +131,7 @@ class Connection(object): > if redirect[1] == "merovingian": > logger.debug("restarting authentication") > if iteration <= 10: > - self.__login(iteration=iteration + 1) > + self._login(iteration=iteration + 1) > else: > raise OperationalError("maximal number of redirects " > "reached (10)") > @@ -138,9 +152,6 @@ class Connection(object): > else: > raise ProgrammingError("unknown state: %s" % prompt) > > - self.state = STATE_READY > - return True > - > def disconnect(self): > """ disconnect from the monetdb server """ > self.state = STATE_INIT > @@ -153,8 +164,8 @@ class Connection(object): > if self.state != STATE_READY: > raise(ProgrammingError, "Not connected") > > - self.__putblock(operation) > - response = self.__getblock() > + self._putblock(operation) > + response = self._getblock() > if not len(response): > return "" > elif response.startswith(MSG_OK): > @@ -166,10 +177,16 @@ class Connection(object): > return response > elif response[0] == MSG_ERROR: > raise OperationalError(response[1:]) > + elif (self.language == 'control' and not self.hostname): > + if response.startswith("OK"): > + return response[2:].strip() or "" > + else: > + return response > else: > raise ProgrammingError("unknown state: %s" % response) > > - def __challenge_response(self, challenge): > + > + def _challenge_response(self, challenge): > """ generate a response to a mapi login challenge """ > challenges = challenge.split(':') > salt, identity, protocol, hashes, endian = challenges[:5] > @@ -204,19 +221,36 @@ class Connection(object): > return ":".join(["BIG", self.username, pwhash, self.language, > self.database]) + ":" > > - def __getblock(self): > + def _getblock(self): > """ read one mapi encoded block """ > + if (self.language == 'control' and not self.hostname): > + return self._getblock_socket() # control doesn't do block > + # splitting when using a socket > + else: > + return self._getblock_inet() > + > + def _getblock_inet(self): > result = StringIO() > last = 0 > while not last: > - flag = self.__getbytes(2) > + flag = self._getbytes(2) > unpacked = struct.unpack('<H', flag)[0] # little endian short > length = unpacked >> 1 > last = unpacked & 1 > - result.write(self.__getbytes(length)) > + result.write(self._getbytes(length)) > return result.getvalue() > > - def __getbytes(self, bytes_): > + def _getblock_socket(self): > + buffer = StringIO() > + while True: > + x = self.socket.recv(1) > + if len(x): > + buffer.write(x) > + else: > + break > + return buffer.getvalue().strip() > + > + def _getbytes(self, bytes_): > """Read an amount of bytes from the socket""" > result = StringIO() > count = bytes_ > @@ -228,8 +262,15 @@ class Connection(object): > result.write(recv) > return result.getvalue() > > - def __putblock(self, block): > + def _putblock(self, block): > """ wrap the line in mapi format and put it into the socket """ > + if (self.language == 'control' and not self.hostname): > + return self.socket.send(block) # control doesn't do block > + # splitting when using a socket > + else: > + self._putblock_inet(block) > + > + def _putblock_inet(self, block): > pos = 0 > last = 0 > while not last: > diff --git a/clients/python2/monetdb/sql/connections.py b/clients/python2/monetdb/sql/connections.py > --- a/clients/python2/monetdb/sql/connections.py > +++ b/clients/python2/monetdb/sql/connections.py > @@ -28,25 +28,25 @@ class Connection(object): > """A MonetDB SQL database connection""" > default_cursor = cursors.Cursor > > - def __init__(self, username="monetdb", password="monetdb", > - hostname="localhost", port=50000, database="demo", > - autocommit=False, user=None, host=None): > + def __init__(self, database, hostname=None, port=50000, username="monetdb", > + password="monetdb", unix_socket="/tmp/.s.monetdb.50000", > + autocommit=False): > """ Set up a connection to a MonetDB SQL database. > > - username -- username for connection (default: monetdb) > - password -- password for connection (default: monetdb) > - hostname -- hostname to connect to (default: localhost) > - port -- port to connect to (default: 50000) > - database -- name of the database (default: demo) > - autocommit -- enable/disable auto commit (default: False) > + database -- name of the database > + hostname -- Hostname where monetDB is running > + port -- port to connect to (default: 50000) > + username -- username for connection (default: "monetdb") > + password -- password for connection (default: "monetdb") > + unix_socket -- socket to connect to. used when hostname not set > + (default: "/tmp/.s.monetdb.50000") > + autocommit -- enable/disable auto commit (default: False) > + > """ > - if user is not None: > - username = user > - if host is not None: > - hostname = host > self.mapi = mapi.Connection() > self.mapi.connect(hostname=hostname, port=int(port), username=username, > - password=password, database=database, language="sql") > + password=password, database=database, language="sql", > + unix_socket=unix_socket) > self.set_autocommit(autocommit) > self.set_sizeheader(True) > self.set_replysize(100) > diff --git a/clients/python2/test/runtests.py b/clients/python2/test/runtests.py > --- a/clients/python2/test/runtests.py > +++ b/clients/python2/test/runtests.py > @@ -45,6 +45,8 @@ TSTDB = os.environ.get('TSTDB', 'demo') > TSTHOSTNAME = os.environ.get('TSTHOSTNAME', 'localhost') > TSTUSERNAME = os.environ.get('TSTUSERNAME', 'monetdb') > TSTPASSWORD = os.environ.get('TSTPASSWORD', 'monetdb') > +#TSTHOSTNAME = None # set to none if test a socket > + > > if os.environ.get("TSTDEBUG", "no") == "yes": > logging.basicConfig(level=logging.DEBUG) > diff --git a/clients/python2/test/test_control.py b/clients/python2/test/test_control.py > --- a/clients/python2/test/test_control.py > +++ b/clients/python2/test/test_control.py > @@ -15,6 +15,7 @@ > # Copyright August 2008-2013 MonetDB B.V. > # All Rights Reserved. > > +import os > import unittest > import logging > > @@ -33,6 +34,10 @@ from monetdb.exceptions import Operation > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

3 10

Re: MonetDB: Feb2013 - fixed bug 3232, use dayofweek, starting at Mo...
by Sjoerd Mullender 27 Aug '13

27 Aug '13

This checkin is not allowed on the release branch: it changes the ABI by introducing a function MTIMEdate_extract_dayofworkweek. What's wrong with changing the implementation of MTIMEdate_extract_dayofweek to do what MTIMEdate_extract_dayofworkweek does now (and dropping MTIMEdate_extract_dayofworkweek)? On 2013-08-25 16:23, Niels Nes wrote: > Changeset: 0620e9df2003 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=0620e9df2003 > Added Files: > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.sql > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.stable.err > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.stable.out > Modified Files: > clients/mapiclient/mclient.c > monetdb5/modules/atoms/mtime.c > monetdb5/modules/atoms/mtime.h > monetdb5/modules/atoms/mtime.mal > sql/backends/monet5/sql.mx > sql/common/sql_types.c > sql/scripts/mtime.sql > sql/server/rel_select.c > sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > sql/test/BugTracker-2013/Tests/All > sql/test/leaks/Tests/check1.stable.out > sql/test/leaks/Tests/check2.stable.out > sql/test/leaks/Tests/check3.stable.out > sql/test/leaks/Tests/check4.stable.out > sql/test/leaks/Tests/check5.stable.out > sql/test/mapi/Tests/php_monetdb.stable.out > Branch: Feb2013 > Log Message: > > fixed bug 3232, use dayofweek, starting at Monday > fixed bug 2962, convert timestamp with time zone to local time zone, when used as input to the extract functions. > fixed bug 2781, use correct time zone in mclient > > > diffs (truncated from 761 to 300 lines): > > diff --git a/clients/mapiclient/mclient.c b/clients/mapiclient/mclient.c > --- a/clients/mapiclient/mclient.c > +++ b/clients/mapiclient/mclient.c > @@ -2559,6 +2559,7 @@ set_timezone(Mapi mid) > tmp = gmtime(&t); > gt = mktime(tmp); > tmp = localtime(&t); > + tmp->tm_isdst=0; /* We need the difference without dst */ > lt = mktime(tmp); > assert((lng) gt - (lng) lt >= (lng) INT_MIN && (lng) gt - (lng) lt <= (lng) INT_MAX); > tzone = (int) (gt - lt); > diff --git a/monetdb5/modules/atoms/mtime.c b/monetdb5/modules/atoms/mtime.c > --- a/monetdb5/modules/atoms/mtime.c > +++ b/monetdb5/modules/atoms/mtime.c > @@ -427,6 +427,7 @@ parse_substr(int *ret, str s, int min, s > return j; > } > > +/* Sunday = 1, Saturday = 7 */ > static int > date_dayofweek(date v) > { > @@ -434,6 +435,17 @@ date_dayofweek(date v) > return v <= 0 ? v + 7 : v; > } > > +/* Monday = 1, Sunday = 7 */ > +static int > +date_dayofworkweek(date v) > +{ > + int res = date_dayofweek(v); > + > + if (res >= 2) > + return res -1; > + return 7; > +} > + > #define SKIP_DAYS(d,w,i) d += i; w = (w + i)%7; if (w <= 0) w += 7; > > static date > @@ -1310,10 +1322,10 @@ date_extract_weekofyear(int *ret, date * > > fromdate((int) *v, &dummy, &dummy, &year); > year_jan_1 = todate(1, 1, year); > - dayofweek = date_dayofweek(year_jan_1); > + dayofweek = date_dayofworkweek(year_jan_1); > > if (dayofweek <= 4) { > - /* week of jan 1 belongs to this year */ > + /* 4 or more days in first week, ie week of jan 1 belongs to this year */ > *ret = (int) (1 + (*v - year_jan_1 + dayofweek - 1) / 7); > } else if (*v - year_jan_1 > 7 - dayofweek) { > /* week of jan 1 belongs to last year; but this is a later week */ > @@ -1340,6 +1352,18 @@ date_extract_dayofweek(int *ret, date *v > return MAL_SUCCEED; > } > > +/* Returns the current day of the week where 1=monday, .., 7=sunday */ > +static str > +date_extract_dayofworkweek(int *ret, date *v) > +{ > + if (*v == date_nil) { > + *ret = int_nil; > + } else { > + *ret = date_dayofworkweek(*v); > + } > + return MAL_SUCCEED; > +} > + > /* extracts hour from daytime (value between 0 and 23) */ > static str > daytime_extract_hours(int *ret, daytime *v) > @@ -1959,6 +1983,12 @@ MTIMEdate_extract_dayofweek(int *ret, da > } > > str > +MTIMEdate_extract_dayofworkweek(int *ret, date *v) > +{ > + return date_extract_dayofworkweek(ret, v); > +} > + > +str > MTIMEdaytime_extract_hours(int *ret, daytime *v) > { > return daytime_extract_hours(ret, v); > diff --git a/monetdb5/modules/atoms/mtime.h b/monetdb5/modules/atoms/mtime.h > --- a/monetdb5/modules/atoms/mtime.h > +++ b/monetdb5/modules/atoms/mtime.h > @@ -163,6 +163,7 @@ mtime_export str MTIMEdate_extract_day(i > mtime_export str MTIMEdate_extract_dayofyear(int *ret, date *v); > mtime_export str MTIMEdate_extract_weekofyear(int *ret, date *v); > mtime_export str MTIMEdate_extract_dayofweek(int *ret, date *v); > +mtime_export str MTIMEdate_extract_dayofworkweek(int *ret, date *v); > mtime_export str MTIMEdaytime_extract_hours(int *ret, daytime *v); > mtime_export str MTIMEdaytime_extract_minutes(int *ret, daytime *v); > mtime_export str MTIMEdaytime_extract_seconds(int *ret, daytime *v); > diff --git a/monetdb5/modules/atoms/mtime.mal b/monetdb5/modules/atoms/mtime.mal > --- a/monetdb5/modules/atoms/mtime.mal > +++ b/monetdb5/modules/atoms/mtime.mal > @@ -633,6 +633,10 @@ command dayofweek(d:date) :int > address MTIMEdate_extract_dayofweek > comment "Returns the current day of the week > where 1=sunday, .., 7=saturday"; > +command dayofworkweek(d:date) :int > +address MTIMEdate_extract_dayofworkweek > +comment "Returns the current day of the week > + where 1=monday, .., 7=sunday"; > > command add(v:timestamp, msecs:lng) :timestamp > address MTIMEtimestamp_add > diff --git a/sql/backends/monet5/sql.mx b/sql/backends/monet5/sql.mx > --- a/sql/backends/monet5/sql.mx > +++ b/sql/backends/monet5/sql.mx > @@ -714,6 +714,14 @@ comment "cast @1 to @1 and check for ove > @:mal_casttime(timestamp)@ > @:mal_casttime(daytime)@ > > +pattern current_time() :daytime > +address SQLcurrent_daytime > +comment "Get the clients current daytime"; > + > +pattern current_timestamp() :timestamp > +address SQLcurrent_timestamp > +comment "Get the clients current timestamp"; > + > command calc.date( v:void ) :date > address nil_2_date > comment "cast to date"; > @@ -6381,6 +6389,8 @@ sql5_export str second_interval_daytime( > sql5_export str second_interval_2_daytime( daytime *res, lng *s, int *d); > sql5_export str timestamp_2_daytime( daytime *res, timestamp *v, int *d); > sql5_export str date_2_timestamp( timestamp *res, date *v, int *d); > +str SQLcurrent_daytime(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > +str SQLcurrent_timestamp(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > @c > str > second_interval_daytime( lng *res, daytime *s, int *d, int *sk ) > @@ -6415,6 +6425,36 @@ second_interval_daytime( lng *res, dayti > return MAL_SUCCEED; > } > > +str SQLcurrent_daytime(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > +{ > + mvc *m = NULL; > + str msg = getSQLContext(cntxt, mb, &m, NULL); > + daytime t, *res = (daytime *) getArgReference(stk, pci, 0); > + > + if (msg) > + return msg; > + > + if ((msg = MTIMEcurrent_time(&t)) == MAL_SUCCEED) > + *res = t+m->timezone; > + return msg; > +} > + > +str SQLcurrent_timestamp(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > +{ > + mvc *m = NULL; > + str msg = getSQLContext(cntxt, mb, &m, NULL); > + timestamp t, *res = (timestamp *) getArgReference(stk, pci, 0); > + > + if (msg) > + return msg; > + > + if ((msg = MTIMEcurrent_timestamp(&t)) == MAL_SUCCEED) { > + lng offset = m->timezone; > + return MTIMEtimestamp_add(res, &t, &offset); > + } > + return msg; > +} > + > /* str dump_cache(int *r); */ > str > dump_cache(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > diff --git a/sql/common/sql_types.c b/sql/common/sql_types.c > --- a/sql/common/sql_types.c > +++ b/sql/common/sql_types.c > @@ -1457,8 +1457,8 @@ sqltypeinit( sql_allocator *sa) > sql_create_func(sa, "curtime", "mtime", "current_time", NULL, NULL, TMETZ, SCALE_NONE); > sql_create_func(sa, "current_time", "mtime", "current_time", NULL, NULL, TMETZ, SCALE_NONE); > sql_create_func(sa, "current_timestamp", "mtime", "current_timestamp", NULL, NULL, TMESTAMPTZ, SCALE_NONE); > - sql_create_func(sa, "localtime", "mtime", "current_time", NULL, NULL, TME, SCALE_NONE); > - sql_create_func(sa, "localtimestamp", "mtime", "current_timestamp", NULL, NULL, TMESTAMP, SCALE_NONE); > + sql_create_func(sa, "localtime", "sql", "current_time", NULL, NULL, TME, SCALE_NONE); > + sql_create_func(sa, "localtimestamp", "sql", "current_timestamp", NULL, NULL, TMESTAMP, SCALE_NONE); > > sql_create_func(sa, "sql_sub", "mtime", "date_sub_msec_interval", DTE, SECINT, DTE, SCALE_FIX); > sql_create_func(sa, "sql_sub", "mtime", "date_sub_month_interval", DTE, MONINT, DTE, SCALE_FIX); > @@ -1470,6 +1470,7 @@ sqltypeinit( sql_allocator *sa) > > sql_create_func(sa, "sql_sub", "mtime", "diff", DTE, DTE, INT, SCALE_FIX); > sql_create_func(sa, "sql_sub", "mtime", "diff", TMESTAMP, TMESTAMP, LNG, SCALE_FIX); > + sql_create_func(sa, "sql_sub", "mtime", "diff", TMESTAMPTZ, TMESTAMPTZ, LNG, SCALE_NONE); > > sql_create_func(sa, "sql_add", "mtime", "date_add_msec_interval", DTE, SECINT, DTE, SCALE_NONE); > sql_create_func(sa, "sql_add", "mtime", "addmonths", DTE, MONINT, DTE, SCALE_NONE); > @@ -1498,6 +1499,15 @@ sqltypeinit( sql_allocator *sa) > /* fix result type */ > f->res.scale = 3; > > + sql_create_func(sa, "year", "mtime", "year", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "month", "mtime", "month", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "day", "mtime", "day", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "hour", "mtime", "hours", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "minute", "mtime", "minutes", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + f = sql_create_func(sa, "second", "mtime", "sql_seconds", TMESTAMPTZ, NULL, DEC, SCALE_NONE); > + /* fix result type */ > + f->res.scale = 3; > + > sql_create_func(sa, "year", "mtime", "year", MONINT, NULL, INT, SCALE_NONE); > sql_create_func(sa, "month", "mtime", "month", MONINT, NULL, INT, SCALE_NONE); > sql_create_func(sa, "day", "mtime", "day", SECINT, NULL, LNG, SCALE_NONE); > @@ -1507,7 +1517,7 @@ sqltypeinit( sql_allocator *sa) > > sql_create_func(sa, "dayofyear", "mtime", "dayofyear", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "weekofyear", "mtime", "weekofyear", DTE, NULL, INT, SCALE_FIX); > - sql_create_func(sa, "dayofweek", "mtime", "dayofweek", DTE, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "dayofweek", "mtime", "dayofworkweek", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "dayofmonth", "mtime", "day", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "week", "mtime", "weekofyear", DTE, NULL, INT, SCALE_FIX); > > diff --git a/sql/scripts/mtime.sql b/sql/scripts/mtime.sql > --- a/sql/scripts/mtime.sql > +++ b/sql/scripts/mtime.sql > @@ -29,16 +29,16 @@ create function "current_timestamp"( ) r > external name mtime."current_timestamp"; > > create function "localtime"( ) returns TIME > - external name mtime."current_time"; > + external name sql."current_time"; > + > create function "localtimestamp"( ) returns TIMESTAMP > - external name mtime."current_timestamp"; > + external name sql."current_timestamp"; > > create type second_interval as interval day to sec; > > create function local_timezone( ) returns second_interval > external name mtime.local_timezone; > > - > create function sql_sub( date, second_interval ) returns date > external name mtime.date_sub_sec_interval; > create function sql_sub( date, month_interval ) returns date > diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c > --- a/sql/server/rel_select.c > +++ b/sql/server/rel_select.c > @@ -3290,6 +3290,17 @@ rel_unop(mvc *sql, sql_rel **rel, symbol > f = sql_bind_func(sql->sa, s, fname, t, NULL, F_AGGR); > if (f && IS_AGGR(f->func)) > return _rel_aggr(sql, rel, 0, s, fname, l->next, fs); > + > + if (f && type_has_tz(t) && f->func->fix_scale == SCALE_FIX) { > + /* set timezone (using msec) */ > + sql_subtype *intsec = sql_bind_subtype(sql->sa, "sec_interval", 10 /*hour to second */, 0); > + sql_exp *tz = exp_atom_lng(sql->sa, sql->timezone); > + > + tz = exp_convert(sql->sa, tz, exp_subtype(tz), intsec); > + e = rel_binop_(sql, e, tz, NULL, "sql_add", ek.card); > + if (!e) > + return NULL; > + } > return rel_unop_(sql, e, s, fname, ek.card); > } > > diff --git a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > --- a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > +++ b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > @@ -30,9 +30,9 @@ stderr of test 'predicate_select.Bug-309 > # 18:53:13 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" "--host=niels" "--port=32323" > # 18:53:13 > > > -MAPI = (monetdb) /var/tmp/mtest-16462/.s.monetdb.36324 > +MAPI = (monetdb) /var/tmp/mtest-4932/.s.monetdb.34449 > QUERY = select * from tables where name; > -ERROR = !conversion of string 'r' to type bit failed. > +ERROR = !conversion of string 'storagemodelinput' to type bit failed. > > # 18:53:13 > > # 18:53:13 > "Done." > diff --git a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > --- a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > +++ b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > @@ -47,19 +47,19 @@ Ready. > [ 2130, "idxs", 2102, NULL, 0, true, 2, false, 0 ] > [ 2135, "triggers", 2102, NULL, 0, true, 2, false, 0 ] > [ 2146, "objects", 2102, NULL, 0, true, 2, false, 0 ] > -[ 5060, "tables", 2000, "SELECT * FROM (SELECT p.*, 0 AS \"temporary\" FROM \"sys\".\"_tables\" AS p UNION ALL SELECT t.*, 1 AS \"temporary\" FROM \"tmp\".\"_tables\" AS t) AS tables where tables.type <> 2;", 1, true, 0, false, 0 ] > -[ 5070, "columns", 2000, "SELECT * FROM (SELECT p.* FROM \"sys\".\"_columns\" AS p UNION ALL SELECT t.* FROM \"tmp\".\"_columns\" AS t) AS columns;", 1, true, 0, false, 0 ] > -[ 5090, "db_user_info", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5098, "users", 2000, "SELECT u.\"name\" AS \"name\", ui.\"fullname\", ui.\"default_schema\" FROM db_users() AS u LEFT JOIN \"sys\".\"db_user_info\" AS ui ON u.\"name\" = ui.\"name\" ;", 1, true, 0, false, 0 ] > -[ 5102, "user_role", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5105, "auths", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5109, "privileges", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5262, "queryhistory", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5276, "callhistory", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5293, "querylog", 2000, "create view querylog as\nselect qd.*, ql.ctime, ql.arguments, ql.exec, ql.result, ql.foot, ql.memory, ql.tuples, ql.inblock, ql.oublock from queryhistory qd, callhistory ql\nwhere qd.id = ql.id;", 1, true, 0, false, 0 ] > -[ 6059, "storagemodelinput", 2000, NULL, 0, true, 0, false, 0 ] > -[ 6100, "tablestoragemodel", 2000, "-- A summary of the table storage requirement is is available as a table view.\n-- The auxillary column denotes the maximum space if all non-sorted columns\n-- would be augmented with a hash (rare situation)\ncreate view sys.tablestoragemodel\nas select \"schema\",\"table\",max(count) as \"count\",\n\tsum(columnsize) as columnsize,\n\tsum(heapsize) as heapsize,\n\tsum(indices) as indices,\n\tsum(case when sorted = false then 8 * count else 0 end) as auxillary\nfrom sys.storagemodel() group by \"schema\",\"table\";", 1, true, 0, false, 0 ] > -[ 6122, "systemfunctions", 2000, NULL, 0, true, 0, false, 0 ] > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

Re: MonetDB: Feb2013 - Organize parallelism per worker pool
by Stefan Manegold 21 Aug '13

21 Aug '13

Martin, could you please also add sql/test/BugTracker-2013/Tests/nestedcalls.sql ? And could you plase also add the test scripts and stable output for tests sql/test/BugTracker-2013/Tests/oid_handling sql/test/BugTracker-2013/Tests/constraint_checking.Bug_3335 sql/test/BugTracker-2013/Tests/pivot.Bug-3339 sql/test/BugTracker-2013/Tests/recursion or alternatively remove them from sql/test/BugTracker-2013/Tests/All Thanks! In any case, propagation of this will checking will result in (expected/unavoidable) conflicts in monetdb5/mal/mal_dataflow.[ch] and (avoidable) conflicts in sql/test/BugTracker-2013/Tests All of these will need to be solved by hand ... Stefan ----- Original Message ----- > Changeset: 489815265a61 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=489815265a61 > Added Files: > sql/test/BugTracker-2013/Tests/nestedcalls.stable.err > sql/test/BugTracker-2013/Tests/nestedcalls.stable.out > Modified Files: > monetdb5/mal/mal_dataflow.c > monetdb5/mal/mal_dataflow.h > monetdb5/modules/mal/language.c > monetdb5/modules/mal/language.h > sql/test/BugTracker-2013/Tests/All > Branch: Feb2013 > Log Message: > > Organize parallelism per worker pool > Each (recursive) dataflow block is handed a worker pool now. > Once we run out of worker pools or fail to create a worker, > we continue in sequential mode. > > Cleaning up the threads is implicit, like all other non-pool > interpreters that might be active at system shutdown. > > > diffs (truncated from 762 to 300 lines): > > diff --git a/monetdb5/mal/mal_dataflow.c b/monetdb5/mal/mal_dataflow.c > --- a/monetdb5/mal/mal_dataflow.c > +++ b/monetdb5/mal/mal_dataflow.c > @@ -35,6 +35,7 @@ > * The flow graphs should be organized such that parallel threads can > * access it mostly without expensive locking. > */ > +#include "monetdb_config.h" > #include "mal_dataflow.h" > #include "mal_client.h" > > @@ -82,10 +83,10 @@ typedef struct DATAFLOW { > Queue *done; /* instructions handled */ > } *DataFlow, DataFlowRec; > > -#define MAXQ 1024 > -static MT_Id workers[THREADS] = {0}; > -static int workerqueue[THREADS] = {0}; /* maps workers towards the todo > queues */ > -static Queue *todo[MAXQ] = {0}; /* pending instructions organized by user > MAXTODO > #users */ > +#define MAXQ 256 > +static Queue *todos[MAXQ] = {0}; /* pending instructions organized by > dataflow block */ > +static bit occupied[MAXQ]={0}; /* worker pool is in use? */ > +static int volatile exiting = 0; > > /* > * Calculate the size of the dataflow dependency graph. > @@ -108,9 +109,8 @@ DFLOWgraphSize(MalBlkPtr mb, int start, > */ > > static Queue* > -q_create(int sz) > +q_create(int sz, const char *name) > { > - const char* name = "q_create"; > Queue *q = (Queue*)GDKmalloc(sizeof(Queue)); > > if (q == NULL) > @@ -208,6 +208,8 @@ q_dequeue(Queue *q) > > assert(q); > MT_sema_down(&q->s, "q_dequeue"); > + if (exiting) > + return NULL; > MT_lock_set(&q->l, "q_dequeue"); > assert(q->last > 0); > if (q->last > 0) { > @@ -255,25 +257,23 @@ DFLOWworker(void *t) > { > DataFlow flow; > FlowEvent fe = 0, fnxt = 0; > - int id = (int) ((MT_Id *) t - workers), last = 0; > - int wq; > Thread thr; > str error = 0; > - > - int i; > - lng usec = 0; > + Queue *todo = *(Queue **) t; > + int i,last; > > thr = THRnew("DFLOWworker"); > > GDKsetbuf(GDKmalloc(GDKMAXERRLEN)); /* where to leave errors */ > GDKerrbuf[0] = 0; > while (1) { > - assert(workerqueue[id] > 0); > - wq = workerqueue[id] - 1; > if (fnxt == 0) > - fe = q_dequeue(todo[wq]); > + fe = q_dequeue(todo); > else > fe = fnxt; > + if (exiting) { > + break; > + } > fnxt = 0; > assert(fe); > flow = fe->flow; > @@ -285,22 +285,20 @@ DFLOWworker(void *t) > continue; > } > > - usec = GDKusec(); > /* skip all instructions when we have encontered an error */ > if (flow->error == 0) { > #ifdef USE_MAL_ADMISSION > if (MALadmission(fe->argclaim, fe->hotclaim)) { > fe->hotclaim = 0; /* don't assume priority anymore */ > - assert(todo[wq]); > - if (todo[wq]->last == 0) > + if (todo->last == 0) > MT_sleep_ms(DELAYUNIT); > - q_requeue(todo[wq], fe); > + q_requeue(todo, fe); > continue; > } > #endif > error = runMALsequence(flow->cntxt, flow->mb, fe->pc, fe->pc + 1, > flow->stk, 0, 0); > PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= %d claim= " LLFMT > "," LLFMT " %s\n", > - fe->pc, id, fe->argclaim, fe->hotclaim, error ? error : ""); > + fe->pc, (int)((Queue **)t - todos), fe->argclaim, fe->hotclaim, > error ? error : ""); > #ifdef USE_MAL_ADMISSION > /* release the memory claim */ > MALadmission(-fe->argclaim, -fe->hotclaim); > @@ -331,8 +329,8 @@ DFLOWworker(void *t) > InstrPtr p = getInstrPtr(flow->mb, fe->pc); > assert(p); > fe->hotclaim = 0; > - for (i = 0; i < p->retc; i++) > - fe->hotclaim += getMemoryClaim(flow->mb, flow->stk, fe->pc, i, FALSE); > + //for (i = 0; i < p->retc; i++) > + //fe->hotclaim += getMemoryClaim(flow->mb, flow->stk, p, i, FALSE); > } > #endif > MT_lock_set(&flow->flowlock, "MALworker"); > @@ -351,56 +349,64 @@ DFLOWworker(void *t) > > q_enqueue(flow->done, fe); > if ( fnxt == 0) { > - assert(todo[wq]); > - if (todo[wq]->last == 0) > + if (todo->last == 0) > profilerHeartbeatEvent("wait"); > - else > - MALresourceFairness(NULL, NULL, usec); > } > } > GDKfree(GDKerrbuf); > GDKsetbuf(0); > - workerqueue[wq] = 0; > - workers[wq] = 0; > THRdel(thr); > } > > /* > - * Create a set of DFLOW interpreters. > + * Create an interpreter pool. > * One worker will adaptively be available for each client. > * The remainder are taken from the GDKnr_threads argument and > - * typically is equal to the number of cores > + * typically is equal to the number of cores. > + * A recursive MAL function call would make for one worker less, > + * which limits the number of cores for parallel processing. > * The workers are assembled in a local table to enable debugging. > + * > + * BEWARE, failure to create a new worker thread is not an error > + * but would lead to serial execution. > */ > -static void > -DFLOWinitialize(int index) > +static int > +DFLOWinitialize(void) > { > - int i, worker, limit; > + int i, threads, grp; > + MT_Id worker; > > - assert(index >= 0); > - assert(index < THREADS); > + threads = GDKnr_threads ? GDKnr_threads : 1; > MT_lock_set(&mal_contextLock, "DFLOWinitialize"); > - if (todo[index]) { > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > - return; > + for(grp = 0; grp< MAXQ; grp++) > + if ( occupied[grp] == FALSE){ > + occupied[grp] = TRUE; > + break; > + } > + MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > + if (grp > THREADS) { > + // continue non-parallel > + return -1; > } > - todo[index] = q_create(2048); > - assert(todo[index]); > - limit = GDKnr_threads ? GDKnr_threads : 1; > - assert(limit <= THREADS); > - for (worker = 0, i = 0; i < limit; i++){ > - for (; worker < THREADS; worker++) > - if( workers[worker] == 0) > - break; > - assert(worker < THREADS); > - if (worker < THREADS) { > - assert(workers[worker] == 0); > - MT_create_thread(&workers[worker], DFLOWworker, (void *) > &workers[worker], MT_THR_JOINABLE); > - assert(workers[worker] > 0); > - workerqueue[worker] = index + 1; > + if ( todos[grp] ) > + return grp; > + > + todos[grp] = q_create(2048, "todo"); > + if (todos[grp] == NULL) > + return -1; > + > + // associate a set of workers with the pool > + for (i = 0; grp>= 0 && i < threads; i++){ > + if (MT_create_thread(&worker, DFLOWworker, (void *) &todos[grp], > MT_THR_JOINABLE) < 0) { > + //Can not create interpreter thread > + grp = -1; > + } > + if (worker == 0) { > + //Failed to create interpreter thread > + grp = -1; > } > } > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > + return grp; > } > > /* > @@ -409,18 +415,28 @@ DFLOWinitialize(int index) > * For each instruction we keep a list of instructions whose > * blocking counter should be decremented upon finishing it. > */ > -static void > +static str > DFLOWinitBlk(DataFlow flow, MalBlkPtr mb, int size) > { > int pc, i, j, k, l, n, etop = 0; > int *assign; > InstrPtr p; > > + if (flow == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Called with flow == NULL"); > + if (mb == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Called with mb == NULL"); > PARDEBUG printf("Initialize dflow block\n"); > assign = (int *) GDKzalloc(mb->vtop * sizeof(int)); > + if (assign == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Failed to allocate assign"); > etop = flow->stop - flow->start; > for (n = 0, pc = flow->start; pc < flow->stop; pc++, n++) { > p = getInstrPtr(mb, pc); > + if (p == NULL) { > + GDKfree(assign); > + throw(MAL, "dataflow", "DFLOWinitBlk(): getInstrPtr() returned NULL"); > + } > > /* initial state, ie everything can run */ > flow->status[n].flow = flow; > @@ -501,6 +517,7 @@ DFLOWinitBlk(DataFlow flow, MalBlkPtr mb > #ifdef USE_MAL_ADMISSION > memorypool = memoryclaims = 0; > #endif > + return MAL_SUCCEED; > } > > /* > @@ -528,18 +545,17 @@ static void showFlowEvent(DataFlow flow, > */ > > static str > -DFLOWscheduler(DataFlow flow) > +DFLOWscheduler(DataFlow flow, Queue *todo) > { > int last; > int i; > #ifdef USE_MAL_ADMISSION > - int j; > + //int j; > InstrPtr p; > #endif > int tasks=0, actions; > str ret = MAL_SUCCEED; > FlowEvent fe, f = 0; > - int wq; > > if (flow == NULL) > throw(MAL, "dataflow", "DFLOWscheduler(): Called with flow == NULL"); > @@ -549,19 +565,19 @@ DFLOWscheduler(DataFlow flow) > /* initialize the eligible statements */ > fe = flow->status; > > - if (fe[0].flow->cntxt->flags & timerFlag) > - fe[0].flow->cntxt->timer = GDKusec(); > - > MT_lock_set(&flow->flowlock, "MALworker"); > - wq = flow->cntxt->idx; > for (i = 0; i < actions; i++) > if (fe[i].blocks == 0) { > #ifdef USE_MAL_ADMISSION > p = getInstrPtr(flow->mb,fe[i].pc); > - for (j = p->retc; j < p->argc; j++) > - fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, > fe[i].pc, j, FALSE); > + if (p == NULL) { > + MT_lock_unset(&flow->flowlock, "MALworker"); > + throw(MAL, "dataflow", "DFLOWscheduler(): getInstrPtr(flow->mb,fe[i].pc) > returned NULL"); > + } > + //for (j = p->retc; j < p->argc; j++) > + //fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, p, j, > FALSE); > #endif > - q_enqueue(todo[wq], flow->status + i); > + q_enqueue(todo, flow->status + i); > flow->status[i].state = DFLOWrunning; > PARDEBUG mnstr_printf(GDKstdout, "#enqueue pc=%d claim=" LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } > @@ -571,6 +587,10 @@ DFLOWscheduler(DataFlow flow) > > while (actions != tasks ) { > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 2

default vs. Feb2013: opt_evaluate.c & mal_dataflow.c
by Stefan Manegold 19 Aug '13

19 Aug '13

Martin, I just merged recent changes from Feb2013 into default, and while doing so resolved --- hopefully correctly; see below --- some conflicts that occurred in opt_evaluate.c & mal_dataflow.c . In fact, (1) I first merged up until before your changeset that introduced the new multiple worker pools, cf., http://dev.monetdb.org/hg/MonetDB/rev/10125a9c3976 (2) then merged only that very changeset, cf., http://dev.monetdb.org/hg/MonetDB/rev/1eea7fa4cc7c (3) and finally merge the rest, cf., http://dev.monetdb.org/hg/MonetDB/rev/3e3d7d5f8dc9 During (1), I resolved some conflicts in opt_evaluate.c due to http://dev.monetdb.org/hg/MonetDB/rev/62ed11377998 I hope, I did this correctly. However, there are still some differences in opt_evaluate.c between Feb2013 & default; cf., hg diff -p -r Feb2013 -r default monetdb5/optimizer/opt_evaluate.c Could you please check, whether these are OK? During (2), I resolved some conflicts in mal_dataflow.c due to http://dev.monetdb.org/hg/MonetDB/rev/4c47797192f4 I hope, I did this correctly. However, there are still some differences in mal_dataflow.c between Feb2013 & default; cf., hg diff -p -r Feb2013 -r default monetdb5/mal/mal_dataflow.c Could you please check, whether these are OK? It appears, that after (2), some tests in sql/jdbc/tests/Tests/ run into a timeout (only when run multi-threaded, i.e., not when run single-threaded) --- I have not yet been able to find out why ... Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - Report correct 100% sample size
by Niels Nes 17 Aug '13

17 Aug '13

On Fri, Aug 16, 2013 at 09:56:49PM +0200, Martin Kersten wrote: > Changeset: afa6da5dd7bd for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=afa6da5dd7bd > Modified Files: > sql/backends/monet5/sql_statistics.c > Branch: default > Log Message: > > Report correct 100% sample size > > > diffs (12 lines): > > diff --git a/sql/backends/monet5/sql_statistics.c b/sql/backends/monet5/sql_statistics.c > --- a/sql/backends/monet5/sql_statistics.c > +++ b/sql/backends/monet5/sql_statistics.c > @@ -169,7 +169,7 @@ sql_analyze(Client cntxt, MalBlkPtr mb, > } > snprintf(query,8192, > "insert into sys.statistics values('%s','%s','%s','%s',%d,now(),"LLFMT","LLFMT","LLFMT","LLFMT",'%s','%s',%s);", > - b->name, bt->name, bc->name, c->type.type->sqlname,width, samplesize, sz, uniq, nils, minval,maxval, sorted?"true":"false"); > + b->name, bt->name, bc->name, c->type.type->sqlname,width, (samplesize?samplesize:sz), sz, uniq, nils, minval,maxval, sorted?"true":"false"); > #ifdef DEBUG_SQL_STATISTICS > mnstr_printf(cntxt->fdout,"%s\n",dquery); > mnstr_printf(cntxt->fdout,"%s\n",query); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list Martin Volgens mij wil je de volgende verzameling van analyze calls: analyze([opt_sample_size]) -- full db analyze(schema, [opt_sample_size]) -- full schema analyze(schema, table, [opt_sample_size]) -- full table analyze(schema, table, column, [opt_sample_size]) -- full table ie dus niet analyze(table, ...) voor die is het onduidelijk welk schema. Niels -- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room L3.14, phone ++31 20 592-4098 sip:4098@sip.cwi.nl url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl

1 0

Re: MonetDB: Feb2013 - New thread group per dataflow block
by Stefan Manegold 16 Aug '13

16 Aug '13

Martin, this checkins appears to make several tests run into timeouts; I have not yet been able to analyze why ... A complete Mtest.py run results in * 142 out of 1863 tests could not be executed 1 out of 1863 tests produced slightly different output sql/test/BugTracker-2012/predicate_select.Bug-3090 * 6 out of 1863 tests ran into timeout * sql/benchmarks/ATIS/load * sql/benchmarks/arno/insert_ATOM * sql/benchmarks/arno/insert_BOND * sql/benchmarks/arno_flt/init * sql/benchmarks/moa/load * sql/benchmarks/wisconsin/load 1 out of 1863 tests caused an abort (assertion failure) monetdb5/tests/BugTracker/kunion-and-nil.Bug-1667 7 out of 1863 tests produced SIGNIFICANTLY different output monetdb5/optimizer/inlineFunction monetdb5/optimizer/ifthencst sql/test/BugTracker-2012/conditions_when_for_triggers_do_not_work.Bug-2073 sql/test/BugTracker-2012/create_function.Bug-3172 sql/test/BugTracker-2012/currenttime.Bug-2781 sql/test/BugTracker-2012/null_except_null.Bug-3040 sql/test/BugTracker-2012/day-of-month-localization.Bug-2962 while with the version before this checkin, it results in 41 out of 1863 tests could not be executed 1 out of 1863 tests produced slightly different output sql/test/BugTracker-2012/predicate_select.Bug-3090 1 out of 1863 tests caused an abort (assertion failure) monetdb5/tests/BugTracker/kunion-and-nil.Bug-1667 7 out of 1863 tests produced SIGNIFICANTLY different output monetdb5/optimizer/inlineFunction monetdb5/optimizer/ifthencst sql/test/BugTracker-2012/conditions_when_for_triggers_do_not_work.Bug-2073 sql/test/BugTracker-2012/create_function.Bug-3172 sql/test/BugTracker-2012/currenttime.Bug-2781 sql/test/BugTracker-2012/null_except_null.Bug-3040 sql/test/BugTracker-2012/day-of-month-localization.Bug-2962 Best, Stefan ----- Original Message ----- > Changeset: 3669ddd28bf0 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=3669ddd28bf0 > Modified Files: > monetdb5/mal/mal_dataflow.c > Branch: Feb2013 > Log Message: > > New thread group per dataflow block > The centralized worker thread group could lead to an unacceptable > situation. If a user is heavily processing complex queries, then > no other user could even log into the system, for its MAL statements > ended up at the end of the shared queues. > > The problem has been resolved by introducing a thread group per dataflow > block. This may lead to a large number of processes, whose resources > are managed by the OS. > > It solves bug 3258 > > > diffs (235 lines): > > diff --git a/monetdb5/mal/mal_dataflow.c b/monetdb5/mal/mal_dataflow.c > --- a/monetdb5/mal/mal_dataflow.c > +++ b/monetdb5/mal/mal_dataflow.c > @@ -78,12 +78,12 @@ typedef struct DATAFLOW { > int *nodes; /* dependency graph nodes */ > int *edges; /* dependency graph */ > MT_Lock flowlock; /* lock to protect the above */ > + queue *todo; /* pending instructions */ > queue *done; /* instructions handled */ > + int threads; /* worker threads active */ > + MT_Id workers[THREADS]; > } *DataFlow, DataFlowRec; > > -static MT_Id workers[THREADS]; > -static queue *todo = 0; /* pending instructions */ > - > /* > * Calculate the size of the dataflow dependency graph. > */ > @@ -138,7 +138,6 @@ q_destroy(queue *q) > static void > q_enqueue_(queue *q, FlowEvent d) > { > - assert(d); > if (q->last == q->size) { > q->size <<= 1; > q->data = GDKrealloc(q->data, sizeof(FlowEvent) * q->size); > @@ -214,7 +213,6 @@ q_dequeue(queue *q) > */ > > MT_lock_unset(&q->l, "q_dequeue"); > - assert(r); > return r; > } > > @@ -239,14 +237,15 @@ q_dequeue(queue *q) > static void > DFLOWworker(void *t) > { > - DataFlow flow; > + DataFlow flow = (DataFlow) t; > FlowEvent fe = 0, fnxt = 0; > - int id = (int) ((MT_Id *) t - workers), last = 0; > + MT_Id id = MT_getpid(); > + int last = 0; > Thread thr; > str error = 0; > > int i; > - lng usec = 0; > + //lng usec = 0; > > thr = THRnew("DFLOWworker"); > > @@ -254,8 +253,10 @@ DFLOWworker(void *t) > GDKerrbuf[0] = 0; > while (1) { > if (fnxt == 0) > - fe = q_dequeue(todo); > + fe = q_dequeue(flow->todo); > else fe = fnxt; > + if ( fe == 0) > + break; > fnxt = 0; > assert(fe); > flow = fe->flow; > @@ -266,20 +267,20 @@ DFLOWworker(void *t) > continue; > } > > - usec = GDKusec(); > + //usec = GDKusec(); > /* skip all instructions when we have encontered an error */ > if (flow->error == 0) { > #ifdef USE_MAL_ADMISSION > if (MALadmission(fe->argclaim, fe->hotclaim)) { > fe->hotclaim = 0; /* don't assume priority anymore */ > - if (todo->last == 0) > + if (flow->todo->last == 0) > MT_sleep_ms(DELAYUNIT); > - q_requeue(todo, fe); > + q_requeue(flow->todo, fe); > continue; > } > #endif > error = runMALsequence(flow->cntxt, flow->mb, fe->pc, fe->pc + 1, > flow->stk, 0, 0); > - PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= %d claim= " LLFMT > "," LLFMT " %s\n", > + PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= "SZFMT" claim= " > LLFMT "," LLFMT " %s\n", > fe->pc, id, fe->argclaim, fe->hotclaim, error ? error : ""); > #ifdef USE_MAL_ADMISSION > /* release the memory claim */ > @@ -330,12 +331,15 @@ DFLOWworker(void *t) > > q_enqueue(flow->done, fe); > if ( fnxt == 0) { > - if (todo->last == 0) > + if (flow->todo->last == 0) > profilerHeartbeatEvent("wait"); > - else > - MALresourceFairness(NULL, NULL, usec); > + //else > + //MALresourceFairness(NULL, NULL, usec); > } > } > + for( i = 0; i< flow->threads; i++) > + if ( flow->workers[i] == id) > + flow->workers[i] = 0; > GDKfree(GDKerrbuf); > GDKsetbuf(0); > THRdel(thr); > @@ -349,22 +353,51 @@ DFLOWworker(void *t) > * The workers are assembled in a local table to enable debugging. > */ > static void > -DFLOWinitialize(void) > +DFLOWinitialize(DataFlow flow, int size) > { > - int i, limit; > + int i; > > - MT_lock_set(&mal_contextLock, "DFLOWinitialize"); > - if (todo) { > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > - return; > + MT_lock_init(&flow->flowlock, "DFLOWworker"); > + flow->todo = q_create(size); > + flow->done = q_create(size); > + flow->threads = GDKnr_threads ? GDKnr_threads :1; > + for (i = 0; i < flow->threads; i++){ > + MT_create_thread(&flow->workers[i], DFLOWworker, (void *) flow, > MT_THR_JOINABLE); > + /* upon failure of starting threads we reduce the count */ > + if ( flow->workers[i]== 0){ > + flow->threads --; > + i--; > + } > } > - todo = q_create(2048); > - limit = GDKnr_threads ? GDKnr_threads : 1; > - for (i = 0; i < limit; i++) > - MT_create_thread(&workers[i], DFLOWworker, (void *) &workers[i], > MT_THR_JOINABLE); > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > } > > +static str > +DFLOWfinalize(DataFlow flow) > +{ > + int i, cnt= flow->threads, runs =0; > + > + for( i = 0; i< cnt; i++) > + q_enqueue(flow->todo, 0); > + /* time out when threads are already killed */ > + do{ > + runs++; > + cnt = 0; > + MT_sleep_ms(1); > + for( i = 0; i < flow->threads; i++) > + cnt += flow->workers[i] ==0; > + } while( cnt != flow->threads && runs <5000); > + > + if ( runs == 5000) > + throw(MAL,"dataflow","Timeout on thread termination"); > + GDKfree(flow->status); > + GDKfree(flow->edges); > + GDKfree(flow->nodes); > + q_destroy(flow->done); > + q_destroy(flow->todo); > + MT_lock_destroy(&flow->flowlock); > + GDKfree(flow); > + return MAL_SUCCEED; > +} > /* > * The dataflow administration is based on administration of > * how many variables are still missing before it can be executed. > @@ -518,7 +551,7 @@ DFLOWscheduler(DataFlow flow) > for (j = p->retc; j < p->argc; j++) > fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, > fe[i].pc, j, FALSE); > #endif > - q_enqueue(todo, flow->status + i); > + q_enqueue(flow->todo, flow->status + i); > flow->status[i].state = DFLOWrunning; > PARDEBUG mnstr_printf(GDKstdout, "#enqueue pc=%d claim=" LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } > @@ -543,7 +576,7 @@ DFLOWscheduler(DataFlow flow) > if (flow->status[i].blocks == 1 ) { > flow->status[i].state = DFLOWrunning; > flow->status[i].blocks--; > - q_enqueue(todo, flow->status + i); > + q_enqueue(flow->todo, flow->status + i); > PARDEBUG > mnstr_printf(GDKstdout, "#enqueue pc=%d claim= " LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } else { > @@ -579,13 +612,11 @@ runMALdataflow(Client cntxt, MalBlkPtr m > > assert(stoppc > startpc); > > - /* check existence of workers */ > - if (workers[0] == 0) > - DFLOWinitialize(); > - assert(workers[0]); > - assert(todo); > - > flow = (DataFlow)GDKzalloc(sizeof(DataFlowRec)); > + > + DFLOWinitialize(flow, stoppc- startpc +1); > + assert(flow->todo); > + assert(flow->done); > > flow->cntxt = cntxt; > flow->mb = mb; > @@ -596,9 +627,6 @@ runMALdataflow(Client cntxt, MalBlkPtr m > flow->start = startpc + 1; > flow->stop = stoppc; > > - MT_lock_init(&flow->flowlock, "DFLOWworker"); > - flow->done = q_create(stoppc- startpc+1); > - > flow->status = (FlowEvent)GDKzalloc((stoppc - startpc + 1) * > sizeof(FlowEventRec)); > size = DFLOWgraphSize(mb, startpc, stoppc); > size += stoppc - startpc; > @@ -608,11 +636,9 @@ runMALdataflow(Client cntxt, MalBlkPtr m > > ret = DFLOWscheduler(flow); > > - GDKfree(flow->status); > - GDKfree(flow->edges); > - GDKfree(flow->nodes); > - q_destroy(flow->done); > - MT_lock_destroy(&flow->flowlock); > - GDKfree(flow); > + if( ret == MAL_SUCCEED) > + ret = DFLOWfinalize(flow); > + else (void) > + DFLOWfinalize(flow); > return ret; > } > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Kernel Function-level Profiling
by ajonwa ajonwa 15 Aug '13

15 Aug '13

Hello All, I am new to MonetDB and would like to profile the MonetDB kernel to find out the functions that are most frequently executed while it runs TPCH. So far, I have tried compiling with "--enable-profile" but it seems that gmon.out files are not generated. Is there a recommend way of doing function level profiling in MonetDB? Thanks, Geoffrey

2 1

Re: MonetDB: SciQL-2 - handle non-rectangular array tiles
by Stefan Manegold 15 Aug '13

15 Aug '13

Hoi Jennie, > Changeset: 58e31783ae0f for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=58e31783ae0f > Modified Files: > monetdb5/modules/mal/array.mx > sql/server/rel_dump.c > sql/server/rel_exp.c > sql/server/rel_select.c > sql/test/sciql/Tests/teleios_noa_bsm.sql > Branch: SciQL-2 > Log Message: > > handle non-rectangular array tiles > > - accept either interval tile range, or point tile ragnes, no mix of them > - multiple tile ranges are appended to e_column->f, thus e_column->f list can > have more than 3 items. > - to distinguish a point range from an interval range: a point range only has > one expr., while an interval range has three Thanks! > array.mx: > - save the values for the error message, since the lists are freed in > AGGR_CLEANUP Nice catch ;-) --- Thanks for fixing! ... and sorry for overlooking it myself ;-) > - disable the check on (max - min + 1 = size), since it only holds for > stepsize = 1. > maybe we should just remove this check, since we don't have step size in > the > tiling aggr impl. functions, do we? IMHO, our current tiled aggregation implementation inherently assumes (and thus requires) that all dimensions of the given array have step-size == 1. Since ARRAYtiles*() does not get the step sizes of the given array as parameters, the check (max - min + 1 = size) was/is IMHO the only way to check/ensure that we are indeed working on an array where all dimensions have step-size == 1. > teleios_noa_bsm.sql: > now we can get rid of the workarounds > > more tests needed... Indeed --- I'll resume me test tonight or tomorrow ... Best, Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

UTF-8 support for column names?
by Satoshi Nagayasu 12 Aug '13

12 Aug '13

Hi, Is there any plan to support UTF-8 charactors for column names (or object identifiers) in MonetDB? I think it's not supported so far. Right? I'm trying to port my database from a row-store one to MonetDB. Now, we are using UTF-8 charactors (Japanese) in column names, so we need to convert them manually. Obviously, it is possible, but if MonetDB can support UTF-8 charactors in column name, it would be easier to get porting our database to MonetDB. And I found the related thread and feature request on this. [MonetDB-users] 答复: Does Monetdb supports UTF-8 characters when I use 'as' clause? http://mail.monetdb.org/pipermail/users-list/2012-February/005397.html Bug 3010 - Does not currently support non-ASCII names in identifiers http://bugs.monetdb.org/show_bug.cgi?id=3010#c2 Is there any particluar plan? Is it difficult to implement? Regards, -- Satoshi Nagayasu <snaga(a)uptime.jp> Uptime Technologies, LLC. http://www.uptime.jp

1 0

Re: MonetDB repository problem!
by Stefan Manegold 07 Aug '13

07 Aug '13

In sort, there is a slight problem with a server side check to prevent users from pushing in unresolved merge conflicts. The problem materializes such that one can checkin such conflicts, but then not (easily) remove them, again. Having you .hgrc up to date as "requested", in particular the whitespace hook [hooks] pretxncommit.whitespace = python:<path-to-check_whitespace-directory>/check_whitespace.py:hook prevent the checkin of unresolved unresolved merge conflicts (already) on the client side. (We'll fully fix the server side once Sjoerd is back from vacation.) Stefan ----- Original Message ----- > Hi stefan, > > can you please share some more information about the problem? how can > configurations of local files on the client side affect the server/main > repository? I am asking for purely academic reasons. Is there some kind of > security problem in hg? > > ----- Original Message ----- > > Dear all, > > > > the MonetDB Mercurial (HG) repository should be working, again. > > > > IMPORTANT NOTE: > > > > To prevent further problems, it is mandatory that you do consciously read > > http://www.monetdb.org/Assets/MonetDB-Mercurial.html > > and follow all instructions carefully. > > > > In particular, your .hgrc must contain all of the "Important Configuration" > > mention in above document. > > > > You must double-check your .hgrc before your next hg pull, hg commit or hg > > push, > > which ever comes first. > > > > Thanks! > > Stefan > > > > ----- Original Message ----- > > > Dear all, > > > > > > there seems to be a problem with the MonetDB HG (Mercurial) repository. > > > Please refrain from pushing to it until we have sorted-out the problem. > > > > > > Thanks! > > > Stefan > > > > -- > > | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | > > | www.CWI.nl/~manegold/ | Science Park 123 (L321) | > > | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | > > > > > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

MonetDB repository problem!
by Stefan Manegold 06 Aug '13

06 Aug '13

Dear all, there seems to be a problem with the MonetDB HG (Mercurial) repository. Please refrain from pushing to it until we have sorted-out the problem. Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 1

Re: MonetDB: Feb2013 - also batxml can be handled by mitosis/mergetable
by Sjoerd Mullender 31 Jul '13

31 Jul '13

Sorry, not allowed in the release branch: this changes the ABI. On 2013-07-31 09:47, Niels Nes wrote: > Changeset: 8300b5c5bd97 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8300b5c5bd97 > Modified Files: > monetdb5/optimizer/opt_prelude.c > monetdb5/optimizer/opt_prelude.h > monetdb5/optimizer/opt_support.c > Branch: Feb2013 > Log Message: > > also batxml can be handled by mitosis/mergetable > > > diffs (41 lines): > > diff --git a/monetdb5/optimizer/opt_prelude.c b/monetdb5/optimizer/opt_prelude.c > --- a/monetdb5/optimizer/opt_prelude.c > +++ b/monetdb5/optimizer/opt_prelude.c > @@ -42,6 +42,7 @@ str boxRef; > str batstrRef; > str batmtimeRef; > str batmmathRef; > +str batxmlRef; > str bbpRef; > str tidRef; > str deltaRef; > @@ -301,6 +302,7 @@ void optimizerInit(void){ > batstrRef = putName("batstr",6); > batmtimeRef = putName("batmtime",8); > batmmathRef = putName("batmmath",8); > + batxmlRef = putName("batxml",6); > bbpRef = putName("bbp",3); > tidRef = putName("tid",3); > deltaRef = putName("delta",5); > diff --git a/monetdb5/optimizer/opt_prelude.h b/monetdb5/optimizer/opt_prelude.h > --- a/monetdb5/optimizer/opt_prelude.h > +++ b/monetdb5/optimizer/opt_prelude.h > @@ -41,6 +41,7 @@ opt_export str boxRef; > opt_export str batstrRef; > opt_export str batmtimeRef; > opt_export str batmmathRef; > +opt_export str batxmlRef; > opt_export str bbpRef; > opt_export str tidRef; > opt_export str deltaRef; > diff --git a/monetdb5/optimizer/opt_support.c b/monetdb5/optimizer/opt_support.c > --- a/monetdb5/optimizer/opt_support.c > +++ b/monetdb5/optimizer/opt_support.c > @@ -827,6 +827,7 @@ int isMapOp(InstrPtr p){ > (getModuleId(p)== batmtimeRef) || > (getModuleId(p)== batstrRef) || > (getModuleId(p)== batmmathRef) || > + (getModuleId(p)== batxmlRef) || > (getModuleId(p)== mkeyRef); > } > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 4

Re: MonetDB: SciQL-2 - forked new SciQL-2 branch off Feb2013 branch:
by Ying Zhang 25 Jul '13

25 Jul '13

On Jul 25, 2013, at 23:12, Stefan Manegold wrote: > Changeset: b1131b028770 for MonetDB > ... > Branch: SciQL-2 > Log Message: > > forked new SciQL-2 branch off Feb2013 branch: Thanks! > > The SciQL-2 branch has been initialized by applying the differences > between tag before_new_JOIN_implementation (changeset 334140294fb2) of the default branch > and tag SciQL-2_source (changeset c05174470f48) of the sciql branch > onto the Feb2013 branch (tag SciQL-2_root / changeset a2665f0d1fc5), > i.e., the SciQL-2 branch differs from the Feb2013 branch > just like the sciql branch differs from the default banch. > > While this excercise was aimed at improving the stability of the SciQL code base > (by eliminating potentially not yet stabelized development code), > the segfaults that triggered this excercise (as reported internally) > do unfortunately still occur also with the SciQL-2 branch ... I'll look into this a.s.a.p., but it's better to work based on a stable branch anyway. Jennie

1 0

(no subject)
by Ketan Sharma 22 Jul '13

22 Jul '13

Hi all, I'm Ketan and I'm new to Monet DB. I have successfully installed MonetDB. I have a table of size ~200MB which is to be inserted into another MonetDB Table. But while inserting data using SELECT command, I am getting the following error: #================================================ GDK reported error. BATfetchjoin(tmpr_2257,tmp_ 2241) does not hit always (|bn|=0 != 10=|l|) => can't use fetchjoin. #================================================ Please do help me in getting this error solved. Thanks.

3 5

Performance fixes
by Alistair Sutherland 17 Jul '13

17 Jul '13

Hi, I've recently been conducting some performance testing of MonetDB with a variety of servers (from 4 core -8GB RAM to 20 core-64GB RAM) with various data sizes, in an attempt to gain a better understanding of how MonetDB scales. During the performance tests it became obvious that much of the processing was IO bound due to: 1) Columns being unmapped from memory overly aggressively (even when there was plenty of memory still available). 2) The constant mapping/unmapping of memory mapped bat files for intermediate results. I've attached a patch which attempts to address both issues. The first patch to (gdk_utils.c) is to update the memory limit at which the GDKvmtrim kicks in to be 80% memory usage. The second patch (gdk_heap.c) limits the number of mmap/munmap calls via the existing 'heap caching' mechanism which was not working at all! In addition to fixing up the caching code, I've also wired in the heap cache into the case where extending a malloced heap results in a swap over to memory mapped storage. After applying the patches I was seeing approximately 40% performance improvements (your mileage may vary!) If the changes are deemed to be useful, how do I go about getting them accepted into the MonetDB source repository? Thanks, Alistair Sutherland ________________________________

2 2

Re: MonetDB: DVframework - fixed bug. Three subselect instructions r...
by Ying Zhang 17 Jul '13

17 Jul '13

Hai Yagiz, Thanks for fixed bugs! :) Is this check-in supposed to fix some of the demo queries? I tried the query (on a freshly created repo.) SELECT channel, sample_time, sample_value FROM mseed.dataview WHERE station = 'ISK' AND sample_time > '2010-01-12T22:15:00.000' AND sample_time < '2010-01-12T22:18:00.000'; But it still gives the error: From the recursive call: MALException:sql.delta:Object not found Is this expected or did I forget something? Jennie On Jul 16, 2013, at 15:49, Yagiz Kargin wrote: > Changeset: 95bc831eb9fb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=95bc831eb9fb > Modified Files: > monetdb5/extras/dvf/dvf.c > monetdb5/extras/dvf/opt_dvf.c > Branch: DVframework > Log Message: > > fixed bug. Three subselect instructions resulting from selection predicate on sample_time are now separately taken care of. > > > diffs (152 lines): > > diff --git a/monetdb5/extras/dvf/dvf.c b/monetdb5/extras/dvf/dvf.c > --- a/monetdb5/extras/dvf/dvf.c > +++ b/monetdb5/extras/dvf/dvf.c > @@ -77,7 +77,7 @@ str plan_modifier(Client cntxt, MalBlkPt > BATiter fli; > > int run_dataflow_opt = 0; > - int run_recycle_opt = 1; > + int run_recycle_opt = 0; > > VarRecord low, high; > > @@ -343,8 +343,9 @@ str plan_modifier(Client cntxt, MalBlkPt > /* adjust variable lifetimes */ > malGarbageCollector(mb); > > - /*chkProgram(cntxt->fdout, cntxt->nspace, mb); > - printFunction(cntxt->fdout,mb, 0, LIST_MAL_ALL); */ > + /*chkProgram(cntxt->fdout, cntxt->nspace, mb);*/ > + /*printFunction(cntxt->fdout,mb, 0, LIST_MAL_EXPLAIN); */ > + /*printFunction(cntxt->fdout,mb, 0, LIST_MAL_ALL); */ > > /* relocate the startpc, the instruction to proceed with the execution. Because it might be changed by the optimizers. */ > for (i = 0; i < limit; i++) > diff --git a/monetdb5/extras/dvf/opt_dvf.c b/monetdb5/extras/dvf/opt_dvf.c > --- a/monetdb5/extras/dvf/opt_dvf.c > +++ b/monetdb5/extras/dvf/opt_dvf.c > @@ -119,6 +119,9 @@ OPTdvfImplementation(Client cntxt, MalBl > int i, limit, which_column, actions = 0; > int last_bind_return_var_id = -1; > int last_data_tid_return_var_id = -1; > + int last_insert_bind_return_var_id = -1; > + int last_subselect_return_var_id = -1; > + int last_update_bind_second_return_var_id = -1; > > stk = stk; //to escape 'unused' parameter error. > pci = pci; //to escape 'unused' parameter error. > @@ -452,35 +455,107 @@ OPTdvfImplementation(Client cntxt, MalBl > actions++; > } > else if((state == 2 || state == 3) && > - ((getModuleId(p) == sqlRef && > + getModuleId(p) == sqlRef && > getFunctionId(p) == bindRef && > p->argc == 7 && > p->retc == 2 && > strcmp(getVarConstant(mb, getArg(p, 3)).val.sval, getVarConstant(mb, getArg(old[i1], 2)).val.sval) == 0 && > strstr(getVarConstant(mb, getArg(p, 4)).val.sval, data_table_identifier) != NULL && > - getVarConstant(mb, getArg(p, 6)).val.ival == 2) || > - (getModuleId(p) == sqlRef && > + getVarConstant(mb, getArg(p, 6)).val.ival == 2) > + { > + last_update_bind_second_return_var_id = getArg(p, 1); > + removeInstruction(mb, p); > + i--; > + actions++; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == sqlRef && > getFunctionId(p) == bindRef && > p->argc == 6 && > p->retc == 1 && > strcmp(getVarConstant(mb, getArg(p, 2)).val.sval, getVarConstant(mb, getArg(old[i1], 2)).val.sval) == 0 && > strstr(getVarConstant(mb, getArg(p, 3)).val.sval, data_table_identifier) != NULL && > - getVarConstant(mb, getArg(p, 5)).val.ival == 1))) > + getVarConstant(mb, getArg(p, 5)).val.ival == 1) > + { > + last_insert_bind_return_var_id = getArg(p, 0); > + removeInstruction(mb, p); > + i--; > + actions++; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == algebraRef && > + getFunctionId(p) == subselectRef && > + p->argc == 8 && > + p->retc == 1 && > + getArg(p, 1) == last_bind_return_var_id) > + { > + last_subselect_return_var_id = getArg(p, 0); > + > + r = newInstruction(mb, ASSIGNsymbol); > + setModuleId(r, algebraRef); > + setFunctionId(r, subselectRef); > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, getArg(p, 1)); > + r = pushArgument(mb, r, getArg(p, 3)); > + r = pushArgument(mb, r, getArg(p, 4)); > + r = pushArgument(mb, r, getArg(p, 5)); > + r = pushArgument(mb, r, getArg(p, 6)); > + r = pushArgument(mb, r, getArg(p, 7)); > + > + insertInstruction(mb, r, i+1); > + removeInstruction(mb, p); > + > + actions += 2; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == algebraRef && > + getFunctionId(p) == subselectRef && > + (p->argc == 7 || p->argc == 8) && > + p->retc == 1 && > + (getArg(p, 1) == last_update_bind_second_return_var_id || getArg(p, 1) == last_insert_bind_return_var_id)) > { > removeInstruction(mb, p); > i--; > actions++; > } > else if((state == 3) && > - (getModuleId(p) == sqlRef && > + getModuleId(p) == sqlRef && > getFunctionId(p) == projectdeltaRef && > p->argc == 6 && > p->retc == 1 && > - last_bind_return_var_id == getArg(p, 2))) > + last_bind_return_var_id == getArg(p, 2)) > + { > + r = newInstruction(mb, ASSIGNsymbol); > + > + if(getArg(p, 1) == last_data_tid_return_var_id) > + { > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, last_bind_return_var_id); > + } > + else > + { > + setModuleId(r, algebraRef); > + setFunctionId(r, leftfetchjoinRef); > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, getArg(p, 1)); > + r = pushArgument(mb, r, last_bind_return_var_id); > + } > + > + insertInstruction(mb, r, i+1); > + removeInstruction(mb, p); > + > + actions += 2; > + } > + else if((state == 3) && > + getModuleId(p) == sqlRef && > + getFunctionId(p) == subdeltaRef && > + p->argc == 6 && > + p->retc == 1 && > + last_subselect_return_var_id == getArg(p, 1)) > { > r = newInstruction(mb, ASSIGNsymbol); > r = pushReturn(mb, r, getArg(p, 0)); > - r = pushArgument(mb, r, last_bind_return_var_id); > + r = pushArgument(mb, r, last_subselect_return_var_id); > > insertInstruction(mb, r, i+1); > removeInstruction(mb, p); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list

2 1

Re: MonetDB: default - Initialize variables with a fake value so tha...
by Stefan Manegold 11 Jul '13

11 Jul '13

Thank you very much, Sjoerd! However, this is not yet sufficient to get this compiled (with optimization enabled): .../monetdb5/optimizer/opt_json.c:54:6: error: 'j' may be used uninitialized in this function [-Werror=maybe-uninitialized] However, simply initializing variable j when it is declared might not be the correct solution since it depends on itself: j = getArg(q,j); ^ ^ [...] pushArgument(mb,q,j); ^ Martin, can you please double-check your code and fix (also) the initilization / use of variable j according to the intended functionality? Thanks! Stefan ----- Original Message ----- > Changeset: fc7c72bf8a56 for MonetDB > URL: > http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fc7c72bf8a56 > Modified Files: > monetdb5/optimizer/opt_json.c > Branch: default > Log Message: > > Initialize variables with a fake value so that this can be compiled. > > > diffs (12 lines): > > diff --git a/monetdb5/optimizer/opt_json.c > b/monetdb5/optimizer/opt_json.c > --- a/monetdb5/optimizer/opt_json.c > +++ b/monetdb5/optimizer/opt_json.c > @@ -30,7 +30,7 @@ int > OPTjsonImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, > InstrPtr pci) > { > int i, j, limit, slimit; > - int bu,br,bj; > + int bu = 0, br = 0, bj = 0; > str nme; > InstrPtr p,q; > int actions = 0; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

code pattern and code generate
by 张磊 04 Jul '13

04 Jul '13

hi all, I found that in GDK of MonetDB, many source code like .h/.c is generated from .mx file when compiling, who can tell me which module control this function? -- Best Wishes! zhanglei

2 1

abou different performaces between SQL and MAL
by Yanchen 03 Jul '13

03 Jul '13

Hi, May I have aquestion about MonetDB performance on TPC-H benchmark? I do not know why I got very different performances when I run 01.sql in TPC-H and the explained 01.mal in mal domain. The average execution time for sql against SF100 scale factor is 19.94s, and the time for mal is 34.7. I also tried to use stethoscope to see the detail, seems like mal stop for several seconds before the last line code. Please let me know if you have any thoughts about that. Thank you so much in advance. Best, Lou I am using Monetdb 11.15.3(Feb. 2013). My mserver5 information is as followed: MonetDB 5 server v11.15.3 "Feb2013-SP1" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 188.9GiB available memory, 32 available cpu cores Libraries: libpcre: 8.30 2012-02-04 (compiled with 8.30) openssl: OpenSSL 1.0.1c 10 May 2012 (compiled with OpenSSL 1.0.1c 10 May 2012) libxml2: 2.8.0 (compiled with 2.8.0) Compiled by: root@LLL-MMMDB2 (x86_64-unknown-linux-gnu) Compilation: gcc -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code Linking : /usr/bin/ld -m elf_x86_64

2 1

Problem running web app using latest jdbc driver
by Casey Gum 03 Jul '13

03 Jul '13

I have a RESTful web service that stores its information in MonetDB. I have to run in a Java 1.6 environment. The latest version of the jdbc driver says that it is for both 1.6 and 1.7. I'm getting the dependency resolved through maven using the clojars repository * <repository>* * <id>monetdb</id>* * <url>http://repository.pentaho.org/artifactory/repo</url>* * <snapshots>* * <enabled>false</enabled>* * </snapshots>* * </repository>* My dependency block looks like this: * <dependency>* * <groupId>monetdb</groupId>* * <artifactId>monetdb-jdbc</artifactId>* * <version>2.9</version>* * </dependency>* Everything worked when I was running v2.8 and getting my dependency resolved from the pentaho repository, but when I get the dependency from clojars, I get the following error: *java.lang.UnsupportedClassVersionError: nl/cwi/monetdb/jdbc/MonetDriver : Unsupported major.minor version 51.0* * * Two questions: 1. Should the 2.9 version work under Java 1.6? 2. Is there some setting I need to put in my pom.xml file so that it runs in "1.6 compatibility mode" or something? Thanks, Casey Gum

2 1

MonetDB configured with "--program-suffix" does not start DBs
by Benjamin Reilly 22 Jun '13

22 Jun '13

Hi all, I downloaded a copy of the 11.15.7 (Feb2013 SP2) source and compiled and installed it. I also have a copy of the same version of MonetDB installed from elsewhere (I'm not sure where now--sorry). Now, when I run the "old" version of Monet (the one I didn't just compile) and create a DB (`monetdb create test`), release it, and start it, everything works fine. The log looks likes this: ... MSG merovingian[32695]: starting database 'test', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) MSG control[32695]: (local): started 'test' MSG test[32705]: arguments: /usr/local/bin/mserver5 ...<snip>... ... However, when I run with my compiled version (using `configure --program-suffix=-vanilla`), starting a created DB does not call mserver5-vanilla, but rather monetdbd-vanilla, which is the incorrect command, and thus the database cannot start. (If I manually run mserver5-vanilla, it works fine.) The log looks like this: MSG merovingian[2616]: starting database 'test', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) MSG test[2625]: arguments: /usr/local/bin/monetdbd-vanilla ...<snip>... MSG test[2625]: usage: monetdbd [ command [ command-options ] ] <dbfarm> MSG test[2625]: where command is one of: MSG test[2625]: create, start, stop, get, set, version or help MSG test[2625]: use the help command to get help for a particular command MSG test[2625]: The dbfarm to operate on must always be given to MSG test[2625]: monetdbd explicitly. ERR test[2625]: monetdbd: unknown command: --dbpath=/home/breilly/db/farm-vanilla/test MSG merovingian[2616]: database 'test' (2625) has exited with exit status 1 ERR control[2616]: (local): failed to fork mserver: database 'test' appears to shut itself down after starting, check monetdbd's logfile for possible hints I tried just now to compile without the suffix, but in a different directory (i.e. `configure --prefix=$HOME/apps`), and it worked fine, as with the first version above. I'm not sure why the one with the suffix fails in the way it does: it's not like it tries to find "mserver5" (without the suffix) and fails. Instead, it successfully finds a real command with the suffix, but it's the wrong one. I have also tried simply symlinking to my suffixed program with non-suffixed name (i.e. `ln -s $HOME/apps/bin/monetdbd-vanilla $HOME/apps/bin/monetdb`, for each of monetdbd, monetdb, mclient, and mserver5), and this also works. However, it does not run the correct version of mserver5: it runs /usr/local/bin/mserver5, rather than my local copy at $HOME/apps/bin/mserver5. Is there some environment variable that I'm not setting correctly? ($HOME/apps/bin is in my PATH). Please let me know if there is any more information that I can provide for you. Thanks in advance for your help. Regards, Ben Reilly

2 1

Re: Monetdb - production users
by Baskar Duraikannu 20 Jun '13

20 Jun '13

Thanks. May I know size of your database? --- Original Message --- From: "Brandon Jackson" <usbrandon(a)gmail.com> Sent: June 20, 2013 12:09 PM To: "Communication channel for developers of the MonetDB suite." <developers-list(a)monetdb.org> Subject: Re: Monetdb - production users We are running MonetDB in our enterprise in a production environment. It works well. There are nuances. We do a lot of bulk loading, so we picked a time during the day to shut MonetDB down and restart it. The reason, memory fragmentation, does occur and in our experience would cause the database to fail and lock up. It will stop accepting connections. Our database health gradually and constantly decreases. Although I can say we have never received wrong answers from the database or had any obvious effects from poor database health. We periodically recreate the entire database to get the health of the database back up to 100%. Essentially we use MonetDB as a fast query database and not the primary database for maintaining the integrity of our data. There is no repair utility for the database. We're still learning how and if we should report issues like this because it happens gradually and the log has some splat marks " *** glibc detected *** /usr/bin/mserver5: corrupted doubly-linked list: 0x00007fa12411f4d0 *** and *** glibc detected *** /user/bin/mserver5: double free or corruption (!prev): 0x00007fa12411f4d0 *** in it. Like anything else, I think you learn to live with whatever tools you are using and play into the strengths and avoid the weaknesses, reporting them and hoping they can get patched up eventually. Brandon On Thu, Jun 20, 2013 at 9:29 AM, Baskar Duraikannu < baskar.duraikannu(a)outlook.com> wrote: > I would like to know whether any enterprise is running MonetDB outside > research community. Please help. > > _______________________________________________ > developers-list mailing list > developers-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/developers-list > > _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

Monetdb - production users
by Baskar Duraikannu 20 Jun '13

20 Jun '13

I would like to know whether any enterprise is running MonetDB outside research community. Please help.

2 1