- users-list - monetdb.org

Node.js and MonetDB...
by Kevork Vartanian 03 May '17

03 May '17

Hi, For one of the projects I'm working on, I'm planning to use Node.js (and probably Angular 2) and I was wondering if any of you know of any good resources and documentation for using Node.js with MonetDB, or if anyone has hands on experience and can share that with the users list? I have seen some npm documentation but any advice from you is greatly appreciated. Thank you in advance. Regards, Kevork Vartanian Technical Solutions Consultant kev(a)innotta.com.au<mailto:kev@innotta.com.au> | +61 406 776 789 | www.innotta.com.au<http://www.innotta.com.au/> [innotta logo 673] This email correspondence may contain privileged information and it is intended to be transmitted to the addressee abovementioned. If you are not the intended recipient of this email correspondence, you must not distribute, copy, forward or rely upon its contents. We ask that you notify the sender of this error and immediately delete this email correspondence.

2 1

Error while installing Monetdb
by Amit Ambekar 03 May '17

03 May '17

Hi, I am trying to install monetdb but some error occurs on my ubuntu OS. Invalid 'Date' entry in Release file /var/lib/apt/lists/partial/dev.monetdb.org_downloads_deb_dists_xenial_Release E: Failed to fetch http://dev.monetdb.org/downloads/deb/dists/xenial/Release Invalid 'Valid-Until' entry in Release file /var/lib/apt/lists/partial/dev.monetdb.org_downloads_deb_dists_xenial_Release

1 0

Announcement: New Dec2016-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 01 May '17

01 May '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP4 bugfix release MonetDB5 Server * Fixed a bug causing a crash during cleanup when mserver5 is stopped with monetdb stop database. * Some memory leaks were plugged. MonetDB Common * A potential deadlock was fixed in order index creation. * A bug that could happen during recovery of the write-ahead log (WAL) was fixed. See changeset 98ad79c555cc for details. SQL * Some memory leaks were plugged. * Upgrade code was added for an old change in the sys.settimeout function. * A bug was fixed with the automatic "vacuum" operation on system tables. Bug Fixes * 6110: cast of a SQL boolean value to a string or clob or (var)char is wrong * 6254: Crash (and assertion failure) after querying a view which uses a correlated subquery in the select-list * 6256: Assertion Trigger on FULL OUTER JOIN with more than two BETWEEN clauses * 6257: wrong count values (1 instead of 0) for correlated aggregation queries * 6258: Vulnerability in FITS and NETCDF data vaults * 6259: crash on select query from sqlitelogictests * 6260: Sqlitelogictest crash * 6288: Function cannot find column in merge table * 6295: msqldump writes unescaped timestamp values when using inserts

1 0

Excution Model of MonetDB (column-at-a-time...or?)
by Mahmoud Mohsen 15 Apr '17

15 Apr '17

Greetings Eyal, /Thank you/ a lot for your quick /response/, >Are you interested in that for purposes of academic research? Yes, If you could suggest me similar work, that is opensource, I would very much appreciate it. Best Regards, Mahmoud Mohsen >1. Column-at-a-time. >2. I'm almost certain the answer is "No", but even if it had been "Yes", >the changes in MonetDB since then would make it unrealistic to use it >with today's code without rewriting a lot of it if not all of it. And >the changes since those days in the closed-source Vectorwise/Actian >Vector have also been significant >Are you interested in that for purposes of academic research? >Eyal On 04/14/2017 07:39 PM, Mahmoud Mohsen wrote: >/Greetings, />//>/The current query engine of the MonetDB...what model of processing does />/it follow....(column-at-a-time, tuple-at-a-time, vector-at-a-time)? />//>/one last question.... />//>/Has the X100 query engine ever been published as opensource in the older />/releases of MonetDB? />//>/Thanks a lot in advance, />//>/Best Regards, />//>/Mahmoud Mohsen /

1 0

Excution Model of MonetDB (column-at-a-time...or?)
by Mahmoud Mohsen 14 Apr '17

14 Apr '17

Greetings, The current query engine of the MonetDB...what model of processing does it follow....(column-at-a-time, tuple-at-a-time, vector-at-a-time)? one last question.... Has the X100 query engine ever been published as opensource in the older releases of MonetDB? Thanks a lot in advance, Best Regards, Mahmoud Mohsen

2 1

remote tables and OCC
by Roberto Cornacchia 12 Apr '17

12 Apr '17

I'm curious to know whether optimistic concurrency control applies across remote tables. For example: db1 contains tableA db2 defines remote tableA-> db1 say that db1 modifies tableA, while db2 already had an open transaction on the same table. Will the transaction on db2 fail? Roberto

1 0

COLcopy
by Roberto Cornacchia 04 Apr '17

04 Apr '17

I'm seeing a strange behaviour with COLcopy(). (Dec2016, optimized, non-devel compilation) In short, it seems to take almost half second to copy a 1-tuple string (view) bat. Inspected with gdb, I see that the copy falls in "(3) we can copy the heaps (memcopy, or even VM page sharing)", with the following values: cnt = 1 bunstocopy = BUN_NONE isVIEW(b) = TRUE VIEWtparent(b) = 0 b->T.heap.size = 1024 b->T.vheap.size = 1094320128 The actual tail and heap copy then takes place: heapcopy(bn, "tail", &bthp, &b->theap) heapcopy(bn, "theap", &thp, b->tvheap) Does this mean that a heap of almost 1GB has been copied for a 1-tuple view? Roberto

2 3

Wrong n. of rows in MAL plan?
by Roberto Cornacchia 04 Apr '17

04 Apr '17

Could you give me a feedback on whether this is a possible bug or not? In the following excerpt: X_377=<tmp_2516>[9]:bat[:str] := mat.packIncrement(X_376=<tmp_2516>[9]:bat[:str],X_335=<tmp_20775>[0]:bat[:str]);"] ... ... X_100=<tmp_2516>[10]:bat[:str] := mat.packIncrement(X_377=<tmp_2516>[10]:bat[:str],X_336=<tmp_24175>[1]:bat[:str]);"] X_377 has 9 rows. But when it's used as a parameter for X_100, it is reported to have 10 rows. In between the two instructions above, X_377 is never used. Roberto

1 0

Announcement: New Dec2016-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 31 Mar '17

31 Mar '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP3 bugfix release MonetDB Common * Fixed a bug when appending string bats that are fully duplicate eliminated. It could happend that the to-be-appended bat had an empty string at an offset and at that same offset in the to-be-appended-to bat there happened to be a (sequence of) NULL(s). Then this offset would be used, even though it might nog be the right offset for the empty string in the to-be-appended-to bat. This would result in multiple offsets for the empty string, breaking the promise of being duplicate eliminated. Bug Fixes * 6138: Weak duplicate elimination in string heaps > 64KB * 6183: ResultSet returns double quoted column name if name contains space characters * 6216: Assertion raised (sqlsmith) * 6219: Crash in rel_optimizer (sqlsmith) * 6227: Monetdb fails on remote tables * 6228: mclient crashes if real column is multiplied by it itself * 6229: ANALYZE, unexpected end of input * 6230: ANALYZE, syntax error * 6237: semijoin with empty right bat does not return immediately * 6242: Crash on rel_reduce_groupby_exps (sqlsmith) * 6243: Static optimization gives wrong result (1 + NULL = -127) * 6245: Nested query crashes all versions of MonetDB or gives wrong result starting from Dec2016-SP2 * 6246: update statements: references to a table do not bind to its alias * 6247: Type analysis issue (sqlsmith) * 6248: update statements: the semantic stage does not resolve the relation in the from clause * 6250: Assertion failure when querying a Blob column with order by DESC * 6251: Crash after adding an ordered index on sys.statistics column and querying sys.statistics * 6253: FITS Data Vaults does not work when using user/pw and other than sys schema name

1 0

Remote Table IS NULL
by Siklós Gábor 27 Mar '17

27 Mar '17

Hi there, We have an issue about the remote table on monetdb version of 2016 (sp1 and sp2) on Ubuntu 14.04 and on Windows Server2012 R2 The scenario for reproducing the issue with two nodes on node1 Create a table on remote node and fill it: CREATE TABLE test(id int not null, name text, valid boolean); INSERT INTO test (id, name) VALUES (1, '1'); INSERT INTO test (id, name) VALUES (2, '2'); INSERT INTO test (id) VALUES (3); on node2 CREATE REMOTE TABLE test(id int not null, name text, valid boolean) ON 'mapi:monetdb://node1:50000/dbfarm'; Then on node2: select * from test; +------+------+-------+ | id | name | valid | +======+======+=======+ | 1 | 1 | null | | 2 | 2 | null | | 3 | null | null | +------+------+-------+ It works fine, but: select * from test where name is null; +----+------+-------+ | id | name | valid | +====+======+=======+ +----+------+-------+ id 3 should appear here. Furthermore: select * from test where name is not null; (mapi:monetdb://monetdb@192.168.254.31/reports2) Cannot register project ( select ( table(sys.test) [ test.id NOT NULL, test.name, test.valid ] COUNT ) [ clob "NULL" ! <= test.name ! <= clob "NULL" ] ) [ test.id NOT NULL, test.name, test.valid ] REMOTE mapi:monetdb://.../... select * from test where valid is null; illegal input, not a JSON header (got '') and node1 is crashed (need : monetdb start farm). After downgrading on ubuntu 14.04 to version of 2015 (SP4) this scenraio works fine. Thanks, SG

2 1

Table size does not decrease despite records deletion
by Manish gupta 26 Mar '17

26 Mar '17

Dear All, Is there a way to truncate the table size when deleting some of the records? In my application, I delete and insert ~1 million records daily. But effectively, it should result in few hundreds additional records. But, I see the size of the table increases by a millions ( that can be seen in storage(), as well as hard disk size ). While the "select count(*) from table" reflects correct number. Is there a way, I can stop increase in table size in my scheme? Regards, Manish

3 10

Memory profiling of monet database
by Manish gupta 24 Mar '17

24 Mar '17

Dear all, I am looking at some memory profiling of monet database. Basically, the database size at disk is ~160 GB ( Although I am not very convinced with this big data size, there are 200M records with ~70 columns distributed among two big tables and several smaller ( relatively ) tables) . Right now, I have 64GB physical RAM dedicated machine for this database, but soon after firing queries on these tables ( with all sort of permutations on columns, but no joins between tables ), the memory is almost fully occupied, and resource crunch kills the performance. Is there any utility, which shows which all columns are in-memory and what is the size? And better some setting through which I can guide which columns should remain in memory and which should immediately trimmed after query returns? I am using windows 16 core machine on 64 bit architecture. Regards, Manish

2 4

Merge table performance on single machine with 4 cores
by Manish gupta 23 Mar '17

23 Mar '17

Dear all, I am checking the performance of merge table vs single table on single machine with 4 cores. The fact table contains ~10million records. I am using # MonetDB 5 server v11.25.9 "Dec2016-SP2" on windows machine ( core i7 16GB physical ram ). I have created a merge table which contains fact table and 3 blank copies of fact table. I am seeing surprisingly slow performance of merge table queries compared to base table queries. Base table query ================= sql>select sum(f_1_1_1), count(*), f_1_1_7,f_1_1_5 from t3760_279_1_1 group by f_1_1_7,f_1_1_5 order by sum(f_1_1_1) desc limit 10; +-----------+------+---------+---------+ | L2 | L5 | f_1_1_7 | f_1_1_5 | +===========+======+=========+=========+ | 125885351 | 14 | SALE | 8084869 | | 125629268 | 15 | SALE | 6852518 | | 121733416 | 16 | SALE | 5440605 | | 110328868 | 13 | SALE | 4842411 | | 100848236 | 11 | SALE | 8068509 | | 100719291 | 16 | SALE | 5822736 | | 93127875 | 11 | SALE | 6529612 | | 91034094 | 13 | SALE | 3401293 | | 90766181 | 10 | SALE | 8084900 | | 88848574 | 10 | SALE | 2660811 | +-----------+------+---------+---------+ 10 tuples (44.2s) Merge Table Query ================= sql>select sum(f_1_1_1), count(*), f_1_1_7,f_1_1_5 from mt279_1_1 group by f_1_1_7,f_1_1_5 order by sum(f_1_1_1) desc limit 10; +-----------+------+---------+---------+ | L2 | L5 | f_1_1_7 | f_1_1_5 | +===========+======+=========+=========+ | 125885351 | 14 | SALE | 8084869 | | 125629268 | 15 | SALE | 6852518 | | 121733416 | 16 | SALE | 5440605 | | 110328868 | 13 | SALE | 4842411 | | 100848236 | 11 | SALE | 8068509 | | 100719291 | 16 | SALE | 5822736 | | 93127875 | 11 | SALE | 6529612 | | 91034094 | 13 | SALE | 3401293 | | 90766181 | 10 | SALE | 8084900 | | 88848574 | 10 | SALE | 2660811 | +-----------+------+---------+---------+ 10 tuples (1m 51s) For the same results ( because merge table is containing the base table and 3 blank copies of base table ), merge table query is taking more than 2.5 times the base table runtime. Why is it so? Thanks, Mohit

5 11

How to execute a query string in MonetDB
by Luis Larios 20 Mar '17

20 Mar '17

Hi, I'm attempting to declare a query string to execute in MonetDB. The documentation is not clear if this is possible, but other engines allow this functionality (such as MySQL and MS SQL Sever). Trying the following execute 'select * from tables'; fails with the message Error: syntax error, unexpected STRING, expecting IDENT or sqlINT The reason behind this is that I need to declare a variable *schema name*, such that I can execute a query in the following way: declare s varchar(32); set s = 'the_schema'; execute 'select * from ' || the_schema || '.the_table'; Please let me know if this is even possible in MonetDB or if you have some tips about that. I tried with a *prepared statement*, as given in MonetDB's documentation -- but the following code cannot execute prepare 'select * from ' || the_schema || '.the_table'; because prepare expects an actual query, not a string. I'd like to achieve this with only SQL stored functions. The client I have has to execute the SQL functions directly, and there's no intermediate Java/PHP/etc scripts to build the SQL on the fly before they're being sent to MonetDB server. Thus, I should create a function like so: create function getData( dataSchema varchar(32) ) returns bigint begin declare query varchar(128); set query = 'select count(*) from ' || dataSchema || '.the_table'; return( execute query ); end; My function is more complicated than this, with other tables and even functions. The issue is that functions belong to one schema, and data tables belong to another schema, which is not known in 'compile' time. Thanks for the help! Luis Ángel The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.

3 5

weird mserver startup failure on windows
by Anthony Damico 16 Mar '17

16 Mar '17

hi, i am trying to test something with this version of mserver on windows: http://monetdb.cwi.nl/testweb/web/63169:4438fea68169/MonetDB5-SQL-Installer… it installed fine, but when i load up an mserver session, it says: # MonetDB 5 server v11.26.0 "default-4438fea68169" # Serving database 'demo', using 8 threads # Compiled for x86_64-pc-winnt/64bit # Found 15.914 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2017 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information !SyntaxException:parseError: !LoaderException:loadLibrary:Loading error failed to open library pyapi (from within file 'C:\Program Files\MonetDB\MonetDB5\lib\monetdb5\lib_pyapi.dll'): The specified module could not be found. !SyntaxException:parseError: !<identifier> expected > and then if i click on the window, it disappears. if i try to connect to an mserver session through MonetDB.R, the mserver window repeats these two lines forever: # SQL catalog created, loading sql scripts once # loading sql script: 09_like.sql here's a screenshot: http://tinypic.com/r/s6rtjd/9 any ideas about what i should try? maybe installing python? but i've never had to do so before, so maybe i'm doing something else dumb.. thanks!

2 3

Checking for a table's existence from a Python connection. (mapi ? sql?)
by Omri Schwarz 13 Mar '17

13 Mar '17

Hi, all. I’m trying to add a MonetDB interface to VoltTron, a Python based server for building and energy management systems. To that end I have to be able to use some of the mclient commands from it, for example, to check if a table exists (\d ), from Python. IIUC, monetdb.mapi.Connection objects should enable me to do that, but I have no idea how. A second question is about performance. The existing databse schema for this product looks like this in MonetDB: CREATE TABLE "voc"."data_test" ( "ts" TIMESTAMP(3) NOT NULL, "topic_id" INTEGER NOT NULL, "value_string" CHARACTER LARGE OBJECT NOT NULL, CONSTRAINT "data_test_topic_id_ts_unique" UNIQUE ("topic_id", "ts") ); There’s a time stamp and topic ID, and SELECT queries are almost always for a narrow range in time, and a small set of topic IDs (usually a set of one). The “value_string” is there because values in the topics can range in type, but are almost always floats. Can I expect MonetDB’s performance to be good For this use case? Or should I force the value_string to a float?

2 1

monetdb to remote monetdbd makes fun of me
by Roberto Cornacchia 09 Mar '17

09 Mar '17

I'm running a dbfarm on a remote server serverX The dbfarm has been created this way (passphrase changed): monetdbd create /var/lib/monetdb/dbfarm monetdbd set control=yes /var/lib/monetdb/dbfarm monetdbd set port=50000 /var/lib/monetdb/dbfarm monetdbd set listenaddr=0.0.0.0 /var/lib/monetdb/dbfarm monetdbd set passphrase=roberto /var/lib/monetdb/dbfarm Port 50000 is open. >From another machine on the same subnet, I request a status: $ monetdb -hserverX -p50000 -Proberto status status: =OK It does connect (using a different port gives a connection refused). However, it just acknowledges the command, without performing it. Whatever command I try, it behaves the same: $ monetdb -hserverX -p50000 -Proberto create x create: =OK $ monetdb -hserverX -p50000 -Proberto create x create: =OK Notice that the second one should have failed. This confirms that the commands are not executed at all. I have had this issue for a long time, it's nothing new. I just never asked before, because I normally solve it with ssh tunneling a port and using it locally. In this case, I cannot use this. Am I missing something, or is it a bug? Roberto

2 2

on windows, is it possible to limit the amount of RAM available to mserver.exe?
by Anthony Damico 04 Mar '17

04 Mar '17

https://www.monetdb.org/Documentation/mserver5-man-page makes it look like i should be able to do this, but i have tried a few different things and none of them have worked. in the mserver window, i just want this line to be reduced to 50MB or something very small. # Found 15.913 GiB available main-memory. mserver window: # MonetDB 5 server v11.25.5 "Dec2016-SP1" # Serving database 'mydb', using 8 threads # Compiled for x86_64-pc-winnt/64bit # Found 15.913 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2017 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/ # MonetDB/SQL module loaded here's what my database.bat file looks like: @echo off setlocal set MONETDB=C:\Program Files\MonetDB\MonetDB5 set PATH=%MONETDB%\bin;%MONETDB%\lib;%MONETDB%\lib\MonetDB5;%PATH% "%MONETDB%\bin\mserver5.exe" --daemon=yes --set "prefix=%MONETDB%" --set "exec_prefix=%MONETDB%" "--dbpath=C:\Users\damico\AppData\Local\Temp\RtmpgBoQzw\mydb" %* --set mapi_port=50001 if ERRORLEVEL 1 pause endlocal thanks!

1 0

merovingian.log too verbose?
by Roberto Cornacchia 03 Mar '17

03 Mar '17

Is there a way to make monetdbd a bit quieter? In a setting with many connections, it quickly becomes several GB of this: 2016-03-01 10:25:03 MSG merovingian[94243]: proxying client localhost:52272 for database 'xxx' to mapi:monetdb:///var/lib/monetdb/dbfarms/yyy/xxx/.mapi.sock?database=xxx 2016-03-01 10:25:03 MSG merovingian[94243]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying

2 4

Announcement: New Dec2016-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 03 Mar '17

03 Mar '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP2 bugfix release Bug Fixes * 4034: argnames array in rapi.c has fixed length (that was too short) * 6080: mserver5: rel_bin.c:2391: rel2bin_project: Assertion `0' failed. * 6081: Segmentation fault (core dumped) * 6082: group.subgroup is undefined if group by is used on an expression involving only constants * 6111: Maximum number of digits for hge decimal is listed as 39 in sys.types. Should be 38 as DECIMAL(39) is not supported. * 6112: Crash upgrading Jul2015->Jun2016 * 6130: Query rewriter crashes on a NULL pointer when having a correlated subquery * 6133: A crash occurs when preparing an INSERT on joined tables during the query semantic phase * 6141: Getting an error message regarding a non-GROUP-BY column rather than an unknown identifier * 6177: Server crashes * 6186: Null casting causes no results (silent server crash?) * 6189: Removing a NOT NULL constraint from a PKey column should NOT be allowed * 6190: CASE query crashes database * 6191: MT_msync failed with "Cannot allocate memory" * 6192: Numeric column stores wrong values after adding large numbers * 6193: converting to a smaller precision (fewer or no decimals after decimal point) should round/truncate consistently * 6194: splitpart returns truncated last part if it contains non ascii caracters * 6195: Cast from huge decimal type to smaller returns wrong results * 6196: Database crashes after generate_series query * 6198: COALESCE could be more optimized * 6201: MonetDB completely giving up on certain queries - no error and no result * 6202: querying a table with an ordered index on string/varchar column crashes server and makes server unrestartable! * 6203: copy into: Failed to import table Leftover data 'False' * 6205: Integer addition overflow * 6206: casting strings with more than one trailing zero ('0') to decimal goes wrong * 6209: Aggregation over complex OR expressions produce wrong results * 6210: Upgrading a database from Jun2015 or older crashes the server * 6213: SQLsmith causes server to crash * 6217: Segfault in rel_optimizer (sqlsmith) * 6218: grouped quantiles with all null group causes following groups to return null * 6224: mal_parser: cannot refer to types containing an underscore

1 0

monetdb.org system maintenance
by Sjoerd Mullender 01 Mar '17

01 Mar '17

Tomorrow, March 2, we will be upgrading the hardware for the system on which all monetdb.org infrastructure runs. Expect some downtime. -- Sjoerd Mullender

1 0

monetdb cluster
by 张馨匀 01 Mar '17

01 Mar '17

Hi all, Do you have any idea about the query throughput regarding monetdb cluster, is it slow? I created the monetdb cluster using remote table, when I do query, it's really slow, about 3sec/query. Do you know how to improve this? Or MonetDB can actually do better than this? And How? Thank you

3 2

compilation src error with the latest version
by 张馨匀 28 Feb '17

28 Feb '17

Hi all, Do you know how to fix the src compilation error? Error is as following: *fatal error: *geos_c.h: No such file or directory #include <geos_c.h> * ^* compilation terminated. Thanks,

4 3

Running aggregates?
by Stefan Manegold 27 Feb '17

27 Feb '17

Hi, does MonetDB support any efficient/optimized functionality to compute running aggregates? A quick check/test suggest that aggregation functions with windows functions like below does not seem to be supported, is it? ======== SELECT somedate, somevalue, SUM(somevalue) OVER(ORDER BY somedate ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS RunningTotal FROM Table ======== Is their anything else that would provide running aggregates? Or am I (for now?) bound to calculating running aggregates using the obvious theta-self-join (expected to be "non-optimal" due to redundant work and large (huge) intermediate results), or "hijacking"/"mis-using" the bulk-version of a (to be implemented) scalar function? Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

3 3

MALException:opt.remap:Function 'calc.isnil' not defined
by Anthony Damico 27 Feb '17

27 Feb '17

hi, i upgraded monetdb and am hitting this error with code that previously worked. my (giant) query has lots of CAST and IS NULL commands in it.. trying to narrow it down now, but does anyone have ideas about what might've introduced this? it looks like something is calling IS NULL where it shouldn't be allowed to? thanks

4 13

single-user / locked mode
by Roberto Cornacchia 20 Feb '17

20 Feb '17

Hi, I'm confused about the single-user mode (working with latest Dec2016 from hg). *Question/suggestion 1* 'mserver5 --single-user' can also be set with 'monetdb set singleuser=yes'. However, 'monetdb get all' doesn't show 'singleuser' as one of the options, I had to guess it myself. It would be nice if monetdb could show an actual list of available options, instead a manually compiled one. *Question 2* is 'mserver5 --single-user' expected to put the database in a locked state, equivalent to 'monetdb lock' ? If the answer is yes, then it doesn't do it consistently: - it does allow to use COPY INTO.... LOCKED (from which you would think the database is locked) but: - it doesn't show the status of the database as locked - it indeed allows new client connections Could you clarify how 'single-user' and 'locked' relate? *Question 3 *(long-ish) I am pushing, via JDBC, the entire English DBpedia via COPY INTO (without 'LOCKED'), into a dbfarm on local SSD. It is about 200GB raw data and it takes ~9 hours. Data goes through a pipeline of light pre-processing (feeding thread) and then inserted into the stream of COPY INTO (output thread). What I observe is that, at the beginning, the feeding thread is mush active, and the output thread very little. After a couple of hours, the output thread is mostly active, and the feeding thread mostly idle. I told myself that MonetDB is taking more and more time to flush appends to disk. That's where I though using 'COPY INTO .. LOCKED' would help. Not only it didn't help, but it got the symptoms worse. The output thread is almost 100% active, and the feeding thread almost 100% idle. Indeed, it is already running for 12h and still going on. Was I wrong to expect that 'LOCKED' would help? Thanks, Roberto

2 2

Option to suppress status messages when running -f mode
by Anderson, David B 17 Feb '17

17 Feb '17

All, Is there a way to suppress the output of status messages like "3000 affected rows" when using mclient and "-f csv"? I have an sql script where I keep reducing a table and inserting the rows into another table. The final sql command extracts the final result. Would be nice if the status messages could be suppressed completely or sent to stderr (a --quiet option). I could also just run two scripts, or pipe the output through grep to remove the offending lines. Thanks, Dave

3 2

Best way to force table into hot set
by Anderson, David B 13 Feb '17

13 Feb '17

All, Is there a preferred method to move an entire table into the database hot-set? I guess I can have a query which computes across all of the columns in the table, but wondered if there was something better/cleaner. Thanks, Dave

1 0

prevent mitosis from partitioning (filter) function arguments
by Roberto Cornacchia 12 Feb '17

12 Feb '17

I have implemented a custom join as a filter function. By design, it cannot work correctly when the input tables are partitioned by mitosis. How can I prevent mitosis from partitioning its inputs? Roberto

2 6

Severe performance degradation
by Dennis Pallett 10 Feb '17

10 Feb '17

Hi all, We recently upgraded to the Dec2016 release but had to downgrade immediately after experiencing a severe degradation in performance. I have attached 4 graphs of our performance monitoring (of the last 3 days). It should be fairly obvious when we upgraded and when we downgraded again. We're still looking into the root cause but it appears that it has something to do with memory leaks when using aggregations on varchar colums with a high cardinality. Has anyone else experienced the same performance degradation? Is this a known (regression) bug? Best regards, Dennis Pallett

1 1

QueryTimeout Question
by Sebastián Acevedo 09 Feb '17

09 Feb '17

Hi, there is any way to chnage the default querytimeout for a user?... i need to use this to make a restriction for long querys, if i run call sys.settimeout(1) this change is only apllied to the current session and if i login again with the same user the timeout is not longer applied. +---------+-----------------+------+------------------+------+-------+ | user | login | sess | lastcommand | quer | activ | : : : iont : : ytim : e : : : : imeo : : eout : : : : : ut : : : : +=========+=================+======+==================+======+=======+ | monetdb | 2017-02-03 19:1 | 0 | 2017-02-03 19:13 | 1 | true | : : 2:37.000000 : : :28.000000 : : : | monetdb | 2017-02-03 18:0 | 0 | 2017-02-03 18:10 | 0 | false | : : 9:47.000000 : : :18.000000 : : : | monetdb | 2017-02-03 19:1 | 0 | 2017-02-03 19:13 | 0 | false | : : 3:10.000000 : : :12.000000 : : : +---------+-----------------+------+------------------+------+-------+ Thanks Sebastián Acevedo

3 4

Python Aggregate functions
by Anderson, David B 06 Feb '17

06 Feb '17

Mark, I think there is a bug in the implementation of the PYTHON_MAP aggregate functions. The return value of a PYTHON_MAP aggregate function can change if there are other functions in the query. The return value of my function weighted_percentile_0 (definition below) changes if I include the function median() in my query. The results are incorrect if median() isn't in the query, but are correct if median() is in the query. In this test case, the weights (v2) are all 1 and the values (v1) are uniformly sampled integers from [1,10]. Number of rows for each fctr is ~1000, and my function should reduce to min(v1) in this case. Is some information bleeding between threads? Should I submit a bug report? This is on the default branch. Thanks, Dave sql>select fctr,weighted_percentile_0(v1,v2),min(v1) from mini where fctr < 10 group by fctr order by fctr; +------+--------------------------+------+ | fctr | L1 | L2 | +======+==========================+======+ | 1 | 3 | 1 | | 2 | 2 | 1 | | 3 | 2 | 1 | | 4 | 1 | 1 | | 5 | 3 | 1 | | 6 | 3 | 1 | | 7 | 2 | 1 | | 8 | 1 | 1 | | 9 | 2 | 1 | +------+--------------------------+------+ 9 tuples (126.928ms) sql>select fctr,weighted_percentile_0(v1,v2),min(v1),median(v1) from mini where fctr < 10 group by fctr order by fctr; +------+--------------------------+------+------+ | fctr | L1 | L2 | L3 | +======+==========================+======+======+ | 1 | 1 | 1 | 5 | | 2 | 1 | 1 | 5 | | 3 | 1 | 1 | 5 | | 4 | 1 | 1 | 5 | | 5 | 1 | 1 | 5 | | 6 | 1 | 1 | 5 | | 7 | 1 | 1 | 5 | | 8 | 1 | 1 | 5 | | 9 | 1 | 1 | 5 | +------+--------------------------+------+------+ 9 tuples (519.195ms) sql> CREATE AGGREGATE weighted_percentile_0(a DOUBLE, w DOUBLE) RETURNS DOUBLE LANGUAGE PYTHON_MAP { import numpy as np # Standardize and sort based on values in a q = np.array([0]) / 100.0 idx = np.argsort(a) a_sort = a[idx] w_sort = w[idx] # Get the cumulative sum of weights ecdf = np.cumsum(w_sort) # Find the percentile index positions associated with the percentiles p = q * (w_sort.sum() - 1) # Find the bounding indices (both low and high) idx_low = np.searchsorted(ecdf, p, side='right') idx_high = np.searchsorted(ecdf, p + 1, side='right') idx_high[idx_high > ecdf.size - 1] = ecdf.size - 1 # Calculate the weights weights_high = p - np.floor(p) weights_low = 1.0 - weights_high # Extract the low/high indexes and multiply by the corresponding weights x1 = np.take(a_sort, idx_low) * weights_low x2 = np.take(a_sort, idx_high) * weights_high wp = np.add(x1,x2) return(wp[0]) };

3 4

Virtual Column
by imad hajj chahine 01 Feb '17

01 Feb '17

Hi, How far is the feature of supporting virtual column, with the possibility to persist it as a physical column, from the future release of monetdb? This feature is supported in mysql starting the 5.7 version. https://dev.mysql.com/doc/refman/5.7/en/create-table-generated-columns.html Thank you.

1 0

SQL grammar
by imad hajj chahine 01 Feb '17

01 Feb '17

Hi, I am interested to know how actually monetdb parse an SQL expression and generate the expression tree, are you using lex/yacc similar grammar tool. I am currently using antlr for some basic parsing tasks, but my objective is to be able to generate a complete expression tree from an SQL expression. Thank you.

1 0

Installing Monetdb in on Ubuntu 16.04.1 LTS via repo
by Yan Grange 01 Feb '17

01 Feb '17

Hi there, I'm trying to install monetdb via the instructions on the website inside a Docker container derived from the Ubuntu 16.04.1 LTS image. I get the following error message: root@ContainerHost:/local# sudo apt-get update Ign:1 http://dev.monetdb.org/downloads/deb xenial InRelease Get:2 http://dev.monetdb.org/downloads/deb xenial Release [3878 B] Get:3 http://dev.monetdb.org/downloads/deb xenial Release.gpg [163 B] Hit:4 http://archive.ubuntu.com/ubuntu xenial InRelease Hit:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease Hit:6 http://archive.ubuntu.com/ubuntu xenial-security InRelease Fetched 4041 B in 0s (6161 B/s) Reading package lists... Done W: http://dev.monetdb.org/downloads/deb/dists/xenial/Release.gpg: Signature by key 213E64DCD5DDC2C063CC39FD053C3ED40583366F uses weak digest algorithm (SHA1) W: Invalid 'Date' entry in Release file /var/lib/apt/lists/partial/dev.monetdb.org_downloads_deb_dists_xenial_Release E: Failed to fetch http://dev.monetdb.org/downloads/deb/dists/xenial/Release Invalid 'Valid-Until' entry in Release file /var/lib/apt/lists/partial/dev.monetdb.org_downloads_deb_dists_xenial_Release E: Some index files failed to download. They have been ignored, or old ones used instead. I did only find one reference to this in a previous discussion. The advice given was to create a bug report. Since for that I need to have an account and that I can't find an account creation link on the bugzilla page, I decided to use this mailing list. Regards and thanks a lot, Yan Grange

2 1

problem stopping/starting monetdbd
by Lynn Carol Johnson 30 Jan '17

30 Jan '17

Hi all - Yesterday our monetdb instance seemed to be running slowing, perhaps due to user queries that had been killed but maybe not cleaned up. We decided to stop the dbs, then start/stop monetdbd. I executed the following commands, but was unable to restart the daemon: sudo monetdbd stop /local/monetdb/dbfarmBL sudo monetdbd start /local/monetdb/dbfarmBL We had our admin reboot the machine, thinking this would clear up everything. This morning I tried again to start monetdbd. The command comes back with no messages, but I get a connection refused message when trying to access a database: could not connect to localhost:50000: Connection refused Get all shows this: [lcj34@cbsudc01 anp68_ije_Locus9Combined_Result]$ monetdbd get all /local/monetdb/dbfarmBL property value hostname cbsudc01 dbfarm /local/monetdb/dbfarmBL status no monetdbd is serving this dbfarm mserver unknown (monetdbd not running) logfile /local/monetdb/dbfarmBL/merovingian.log pidfile /local/monetdb/dbfarmBL/merovingian.pid sockdir /tmp listenaddr 128.84.3.206 port 50000 exittimeout 60 forward proxy discovery yes discoveryttl 600 control no passphrase <unknown> mapisock /tmp/.s.monetdb.50000 controlsock /tmp/.s.merovingian.50000 [lcj34@cbsudc01 anp68_ije_Locus9Combined_Result]$ The log shows the following message: 2017-01-27 07:53:26 MSG maizeFullGenomeDB[14286]: # Listening for UNIX domain connection requests on mapi:monetdb:///local/monetdb/dbfarmBL/maizeFullGenomeDB/.mapi.sock 2017-01-27 07:53:26 MSG maizeFullGenomeDB[14286]: # MonetDB/SQL module loaded 2017-01-27 07:53:29 ERR merovingian[13705]: client error: cannot connect: Connection refused 2017-01-27 07:55:41 MSG merovingian[13705]: caught SIGTERM, starting shutdown sequence 2017-01-27 07:55:50 MSG merovingian[15395]: Merovingian 1.7 (Jun2016-SP2) starting 2017-01-27 07:55:50 MSG merovingian[15395]: monitoring dbfarm /local/monetdb/dbfarmBL 2017-01-27 07:55:50 MSG merovingian[15395]: accepting connections on TCP socket cbsudc01:50000 2017-01-27 07:55:50 MSG merovingian[15395]: accepting connections on UNIX domain socket /tmp/.s.monetdb.50000 2017-01-27 07:55:50 ERR merovingian[15395]: binding to datagram socket port 50000 failed: no available address 2017-01-27 07:55:50 MSG merovingian[15395]: Merovingian 1.7 stopped 2017-01-27 07:55:50 ERR merovingian[15395]: fatal startup condition encountered, aborting startup Any ideas what to try? Thanks - Lynn

3 7

Re: running make command gives error on installation
by Pham Duc 29 Jan '17

29 Jan '17

Hi Miguel, I guess that you tried MonetDB 11.2.0 as it is mentioned in the "Heuristics-based Query Optimisation for SPARQL" paper (https://openproceedings.org/2012/conf/edbt/TsialiamanisSFCB12.pdf) Did you check the instruction provided by the ICS-FORTH researchers ( http://www.ics.forth.gr/isl/index_main.php?l=e&c=646 ) for installing MonetDB? If not, please follow that instruction. If it still does not work, you will need to contact ICS-FORTH researchers (please see the contact at the end of the instruction) as that RDF version of MonetDB 11.2.0 was specifically developed for their research with the support of our group members (e.g., Lefteris). Best regards, Minh Duc ----- Original Message ----- From: "Martin Kersten" <martin.kersten(a)cwi.nl> To: "Communication channel for MonetDB users" <users-list(a)monetdb.org>, "P Boncz" <P.Boncz(a)cwi.nl>, "Pham Minh Duc" <P.Minh.Duc(a)cwi.nl> Sent: Sunday, January 29, 2017 8:41:12 AM Subject: Re: running make command gives error on installation Hi Indeed, this is a branch used by Pham Duc during his PhD research. The code base has not been kept in sync with the main branches and you should contact him/his advisor for further support. regards, Martin On 28/01/2017 21:59, Miguel Mármol wrote: > Hi, > I am trying to install MonetDB5 11.2.0 (The version of MonetDB is not an official MonetDB release. This version includes an RDF frontend that is used to parse RDF datasets and load them in MonetDB.). > > I get the following error when I run the make command: > > > miguel@miguel-VirtualBox:~/Downloads/MonetDB$ sudo make > make all-recursive > make[1]: Entering directory '/home/miguel/Downloads/MonetDB' > Making all in buildtools > make[2]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' > make all-recursive > make[3]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' > Making all in Mx > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' > Making all in conf > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/conf' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/conf' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/conf' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/conf' > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' > make[4]: Nothing to be done for 'all-am'. > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' > make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' > make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' > Making all in common > make[2]: Entering directory '/home/miguel/Downloads/MonetDB/common' > make all-recursive > make[3]: Entering directory '/home/miguel/Downloads/MonetDB/common' > Making all in stream > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/stream' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/stream' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/stream' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/stream' > Making all in options > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/options' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/options' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/options' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/options' > Making all in utils > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/utils' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/utils' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/utils' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/utils' > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common' > make[4]: Nothing to be done for 'all-am'. > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common' > make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/common' > make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/common' > Making all in clients > make[2]: Entering directory '/home/miguel/Downloads/MonetDB/clients' > make all-recursive > make[3]: Entering directory '/home/miguel/Downloads/MonetDB/clients' > Making all in mapilib > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapilib' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapilib' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapilib' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapilib' > Making all in mapiclient > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' > /bin/bash ../../libtool --tag=CC --mode=link gcc -Wall -Wextra -g -pedantic -Wno-long-long -Wno-declaration-after-statement -D_REENTRANT -export-dynamic -o mclient mclient.o ReadlineTools.o libmcutil.la ../mapilib/libmapi.la > ../../common/stream/libstream.la -lcrypt -pthread > libtool: link: gcc -Wall -Wextra -g -pedantic -Wno-long-long -Wno-declaration-after-statement -D_REENTRANT -o .libs/mclient mclient.o ReadlineTools.o -pthread -Wl,--export-dynamic ./.libs/libmcutil.a ../mapilib/.libs/libmapi.so > ../../common/stream/.libs/libstream.so -lcrypt -pthread -Wl,-rpath -Wl,/usr/local/monet/lib > ../mapilib/.libs/libmapi.so: undefined reference to `SHA256' > ../mapilib/.libs/libmapi.so: undefined reference to `SHA384' > ../mapilib/.libs/libmapi.so: undefined reference to `RIPEMD160' > ../mapilib/.libs/libmapi.so: undefined reference to `MD5' > ../mapilib/.libs/libmapi.so: undefined reference to `SHA1' > ../mapilib/.libs/libmapi.so: undefined reference to `SHA512' > ../mapilib/.libs/libmapi.so: undefined reference to `SHA224' > collect2: error: ld returned 1 exit status > Makefile:666: recipe for target 'mclient' failed > make[5]: *** [mclient] Error 1 > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' > Makefile:568: recipe for target 'all' failed > make[4]: *** [all] Error 2 > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' > Makefile:764: recipe for target 'all-recursive' failed > make[3]: *** [all-recursive] Error 1 > make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/clients' > Makefile:709: recipe for target 'all' failed > make[2]: *** [all] Error 2 > make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/clients' > Makefile:600: recipe for target 'all-recursive' failed > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory '/home/miguel/Downloads/MonetDB' > Makefile:530: recipe for target 'all' failed > make: *** [all] Error 2 > > Please any help will be really appreciated, thanks. > > Miguel > > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list >

1 0

running make command gives error on installation
by Miguel Mármol 29 Jan '17

29 Jan '17

Hi, I am trying to install MonetDB5 11.2.0 (The version of MonetDB is not an official MonetDB release. This version includes an RDF frontend that is used to parse RDF datasets and load them in MonetDB.). I get the following error when I run the make command: miguel@miguel-VirtualBox:~/Downloads/MonetDB$ sudo make make all-recursive make[1]: Entering directory '/home/miguel/Downloads/MonetDB' Making all in buildtools make[2]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' make all-recursive make[3]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' Making all in Mx make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' Making all in conf make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/conf' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/conf' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/conf' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/conf' make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' make[4]: Nothing to be done for 'all-am'. make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' Making all in common make[2]: Entering directory '/home/miguel/Downloads/MonetDB/common' make all-recursive make[3]: Entering directory '/home/miguel/Downloads/MonetDB/common' Making all in stream make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/stream' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/stream' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/stream' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/stream' Making all in options make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/options' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/options' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/options' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/options' Making all in utils make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/utils' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/utils' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/utils' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/utils' make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common' make[4]: Nothing to be done for 'all-am'. make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common' make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/common' make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/common' Making all in clients make[2]: Entering directory '/home/miguel/Downloads/MonetDB/clients' make all-recursive make[3]: Entering directory '/home/miguel/Downloads/MonetDB/clients' Making all in mapilib make[4]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapilib' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapilib' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapilib' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapilib' Making all in mapiclient make[4]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' /bin/bash ../../libtool --tag=CC --mode=link gcc -Wall -Wextra -g -pedantic -Wno-long-long -Wno-declaration-after-statement -D_REENTRANT -export-dynamic -o mclient mclient.o ReadlineTools.o libmcutil.la ../mapilib/libmapi.la ../../common/stream/libstream.la -lcrypt -pthread libtool: link: gcc -Wall -Wextra -g -pedantic -Wno-long-long -Wno-declaration-after-statement -D_REENTRANT -o .libs/mclient mclient.o ReadlineTools.o -pthread -Wl,--export-dynamic ./.libs/libmcutil.a ../mapilib/.libs/libmapi.so ../../common/stream/.libs/libstream.so -lcrypt -pthread -Wl,-rpath -Wl,/usr/local/monet/lib ../mapilib/.libs/libmapi.so: undefined reference to `SHA256' ../mapilib/.libs/libmapi.so: undefined reference to `SHA384' ../mapilib/.libs/libmapi.so: undefined reference to `RIPEMD160' ../mapilib/.libs/libmapi.so: undefined reference to `MD5' ../mapilib/.libs/libmapi.so: undefined reference to `SHA1' ../mapilib/.libs/libmapi.so: undefined reference to `SHA512' ../mapilib/.libs/libmapi.so: undefined reference to `SHA224' collect2: error: ld returned 1 exit status Makefile:666: recipe for target 'mclient' failed make[5]: *** [mclient] Error 1 make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' Makefile:568: recipe for target 'all' failed make[4]: *** [all] Error 2 make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' Makefile:764: recipe for target 'all-recursive' failed make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/clients' Makefile:709: recipe for target 'all' failed make[2]: *** [all] Error 2 make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/clients' Makefile:600: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/home/miguel/Downloads/MonetDB' Makefile:530: recipe for target 'all' failed make: *** [all] Error 2 Please any help will be really appreciated, thanks. Miguel

2 1

apt having trouble installing deb packages
by Clayton A Davis 27 Jan '17

27 Jan '17

Ubuntu seems to be having an issue with the deb packages. On 16.04, it complains about the "Valid-Until" entry in the Release file, and on 16.10 it complains about the "Date" entry and the security. Here are the error messages from 16.04: W: http://dev.monetdb.org/downloads/deb/dists/xenial/Release.gpg: Signature by key 213E64DCD5DDC2C063CC39FD053C3ED40583366F uses weak digest algorithm (SHA1) W: Invalid 'Date' entry in Release file /var/lib/apt/lists/dev.monetdb.org_downloads_deb_dists_xenial_Release W: http://www.openprinting.org/download/printdriver/debian/dists/lsb3.2/Releas…: Signature by key F8897B6F00075648E248B7EC24CBF5474CFD1E2F uses weak digest algorithm (SHA1) E: Failed to fetch http://dev.monetdb.org/downloads/deb/dists/xenial/Release Invalid 'Valid-Until' entry in Release file /var/lib/apt/lists/dev.monetdb.org_downloads_deb_dists_xenial_Release And from 16.10: W: http://dev.monetdb.org/downloads/deb/dists/yakkety/Release.gpg: Signature by key 213E64DCD5DDC2C063CC39FD053C3ED40583366F uses weak digest algorithm (SHA1) W: Invalid 'Date' entry in Release file /var/lib/apt/lists/partial/dev.monetdb.org_downloads_deb_dists_yakkety_Release E: The repository 'http://dev.monetdb.org/downloads/deb yakkety Release' provides only weak security information. N: Updating from such a repository can't be done securely, and is therefore disabled by default. N: See apt-secure(8) manpage for repository creation and user configuration details. I know the workarounds to get it working on my machine, I just wanted to put this issue out there in case it started with the recent release. -CAD

1 0

Announcement: New Dec2016-SP1 Bugfix release of MonetDB suite
by Panagiotis Koutsourakis 26 Jan '17

26 Jan '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP1 bugfix release Bug Fixes * 4039: Slow mserver5 start after drop of tables (> 1 hour) * 4048: Segfault on vacuum with parallel updates * 6079: pushselect optimizer bug on MAL snippet * 6140: INNER JOIN gives the results of a CROSS JOIN * 6150: Query giving wrong results, extra records are appearing * 6175: The program can't start because python27.dll is missing from your computer. * 6178: AVG + GROUP BY returns NULL for some records that should have results * 6179: mergetable optimizer messes up sample * 6182: sys.shutdown triggers assertion in clients.c * 6184: Incorrect result set - Extra records in result set

1 0

Backup Strategies
by Vincent Sheffer 25 Jan '17

25 Jan '17

I have been attempting to create a viable period backup strategy for MonetDB and have had little success to date. Our database is way too big to use msqldump, so our only option is the one outlined here: https://www.monetdb.org/Documentation/UserGuide/FastDumpRestore. I follow those steps and use rsync to do an incremental backup of the dbfarm to another device (from fast SSD to RAID5 array). That works fine, until it doesn’t, in which case the back up gets corrupted somehow and MonetDB won’t start from the backup. That is the state I am currently in. This is the error that I am getting: !FATAL: logger_load: BBPrename to sql_snapshots_bid failed Any help on 1. Fixing the existing backup, and/or 2. Helping me understand a better way to do period, incremental backups. Thanks, Vince The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.

4 8

Database crash on executing a big insert
by Shrihari Shastry 25 Jan '17

25 Jan '17

Hi, I' trying to perform a big insert of about 4 mil rows of data: insert into clustertable1(keyword, p0,p1,p2,p3,p4,p5,p6,p7,p8,p9,p10,p11,p12,p13,p14,p15,p16,p17,p18,p19,p20,p21,p22,p23,p24,p25,p26,p27,p28,p29,p30,p31,p32,p33,p34,p35,p36,p37,p38,p39,p40,p41,p42,p43,p44,p45,p46,p47,p48,p49,p50) select keyword,p0,p1,p2,p3,p4,p5,p6,p7,p8,p9,p10,p11,p12,p13,p14,p15,p16,p17,p18,p19,p20,p21,p22,p23,p24,p25,p26,p27,p28,p29,p30,p31,p32,p33,p34,p35,p36,p37,p38,p39,p40,p41,p42,p43,p44,p45,p46,p47,p48,p49,p50 from cluster_submaster where cluster_no = '0'; But the moment I fire this query, the server crashes with the below printed in the logs: 2017-01-25 10:05:11 MSG kwresearch[3418]: # Visit http://www.monetdb.org/ for further information 2017-01-25 10:05:11 MSG kwresearch[3418]: # Listening for UNIX domain connection requests on mapi:monetdb:///monetDB/myfarm/kwresearch/.mapi.sock 2017-01-25 10:05:11 MSG kwresearch[3418]: # MonetDB/SQL module loaded 2017-01-25 10:07:06 MSG merovingian[3201]: proxying client (local) for database 'kwresearch' to mapi:monetdb:///monetDB/myfarm/kwresearch/.mapi.sock?database=kwresearch 2017-01-25 10:07:06 MSG merovingian[3201]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2017-01-25 10:07:06 MSG merovingian[3201]: proxying client (local) for database 'kwresearch' to mapi:monetdb:///monetDB/myfarm/kwresearch/.mapi.sock?database=kwresearch 2017-01-25 10:07:06 MSG merovingian[3201]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2017-01-25 10:07:12 MSG merovingian[3201]: database 'kwresearch' (3418) was killed by signal SIGSEGV Appreciate any help! Thanks, Shrihari ******************* iquanti.com E-Mail Domain LEGAL DISCLAIMER ******************* This E-Mail may contain confidential and/or legally privileged Information and is meant for the intended recipient(s) only. If you have received this e-mail in error and are not the intended recipient/s, kindly notify us at it(a)iquanti.com and then delete this e-mail immediately from your system. You are also hereby notified that any use, any form of reproduction, dissemination, copying, disclosure, modification, distribution and/or publication of this e-mail, its contents or its attachment/s other than by its intended recipient/s is strictly prohibited and may be unlawful.

2 1

bool from aggregate
by Stefano Fioravanzo 25 Jan '17

25 Jan '17

Hello, I would like to know if there is a way to replicate the behavior of the bool_and function of Postgresql. An example of its use is the following: SELECT bool_and(A) FROM mytable GROUP BY B; bool_and will return true if all values in the group are true, false otherwise. I would like to do the same thing in monetdb. Thanks Stefano

2 1

Case insensitive version of the IN operator?
by Kevork Vartanian 13 Jan '17

13 Jan '17

Hi, I was wondering if there is a case insensitive version of the IN operator, similar to ILIKE. Or is the IN itself case insensitive? It's mainly to avoid using nested OR's and ILIKE's if a case insensitive IN exists, to make the queries much simpler. Also, I don't want to use functions like lower as it's a big data set and the use of functions may make the queries even slower. Please advise. Thank you. Regards, Kevork Vartanian Technical Solutions Consultant kevork.vartanian(a)innotta.com.au | +61 406 776 789 | www.innotta.com.au

3 2

mclient COPY INFO Local files to a remote MonetDB Server Error
by ping gao 高平(0) 11 Jan '17

11 Jan '17

Hi ALL mclient COPY INFO Local files to a remote MonetDB Server Error. Mclient how to copy a local file to a remote MonetDB Server? Known to use: mclient COPY INFO - < file ; Will appear: syntax error, unexpected sqlINT in: Error Error Info :could not open file ‘xxxxx’ : No such file or directory

3 4

COPY INFO ERROR
by ping gao 高平(0) 10 Jan '17

10 Jan '17

Hi ALL I encountered a problem using a COPY INFO. Strange thing is my IP is 59, execute the following statement is not a problem on the machine.So freak! IP:192.168.110.59 (Database Server) [hotel-revenue-online@bjs-hotel-RevenueCompute-02 data_visual_monetdb]$ mclient -u monetdb -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_booking_order FROM '/home/hotel-revenue-online/ping.gao/data_visual_monetdb/hotel_revenue_octopus_booking_order.sql' USING DELIMITERS ',','\\n'<smb://n'>" password: 2 affected rows IP:192.168.110.179 Error 1: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM /home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data USING DELIMITERS '\\001','\\n'<smb://001','//n'>” syntax error, unexpected '/', expecting STRING or '(' or STDIN in: "copy 2 records into hotel_revenue_octopus_hotel_info from /“ Error 2: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM '/home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data' USING DELIMITERS '\\001<smb://001>','\\n'<smb://n'>" could not open file '/home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data': No such file or directory Error 3: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM STDIN DELIMITERS '\\001','\\n'<smb://001','//n'>" - < /home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data 2 affected rows syntax error, unexpected sqlINT in: "90564318"

2 2

RE: COPY INFO ERROR
by Stefan Manegold 10 Jan '17

10 Jan '17

Not strange at all. Quotes are missing around file path (string). Sent from my Samsung Galaxy smartphone. -------- Original message --------From: "ping gao 高平(0)" <ping.gao(a)corp.elong.com> Date: 1/10/17 08:07 (GMT+01:00) To: Communication channel for MonetDB users <users-list(a)monetdb.org> Subject: COPY INFO ERROR Hi ALL I encountered a problem using a COPY INFO. Strange thing is my IP is 59, execute the following statement is not a problem on the machine.So freak! IP:192.168.110.59 (Database Server) [hotel-revenue-online@bjs-hotel-RevenueCompute-02 data_visual_monetdb]$ mclient -u monetdb -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_booking_order FROM '/home/hotel-revenue-online/ping.gao/data_visual_monetdb/hotel_revenue_octopus_booking_order.sql' USING DELIMITERS ',','\\n'" password: 2 affected rows IP:192.168.110.179 Error 1: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM /home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data USING DELIMITERS '\\001','\\n'” syntax error, unexpected '/', expecting STRING or '(' or STDIN in: "copy 2 records into hotel_revenue_octopus_hotel_info from /“ Error 2: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM '/home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data' USING DELIMITERS '\\001','\\n'" could not open file '/home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data': No such file or directory Error 3: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM STDIN DELIMITERS '\\001','\\n'" - < /home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data 2 affected rows syntax error, unexpected sqlINT in: "90564318"

2 1

MAL function syntax
by Moritz Bruder 09 Jan '17

09 Jan '17

Hi, I'm having problems with the exact syntax of MAL functions. I used the syntax from the documentation, which is working for some functions, but my hand-written example doesn't work (neither when pasted in the REPL, nor as input file): "^<identifier> expected" at the end of every line. Any idea what could cause this? (I'm using a fairly recent development version.) Regards, Moritz

2 1

C UDF
by imad hajj chahine 06 Jan '17

06 Jan '17

Hi, After reviewing all the other alternatives like SQL and Python UDF, I was either stuck on performance with SQL UDF or on usability with Python UDF (unable to use with aggregation, and not such great performance with dates), so I decided to go the hard way with C functions, as a bonus it will give me the possibility to change the functionalities without worrying about dependencies, which was not the case in other languages. The purpose is to create a set of formatting functions for Year, Quarter, Month, Week and Day brackets, and of course i need to create the bulk version of each function for performance. Starting from the MTIMEdate_extract_year_bulk, now i have the simple function working, and successfully calling it from mclient: *str* *UDFyearbracket(str *ret, const date *v)* *{* * if (*v == date_nil) {* * *ret = GDKstrdup(str_nil);* * } else {* * int year;* * fromdate(*v, NULL, NULL, &year);* * *ret = (str) GDKmalloc(15);* * sprintf(*ret, "%d", year);* * }* * return MAL_SUCCEED;* *}* For the bulk version i get an error in the log: gdk_atoms.c:1345: strPut: Assertion `(v[i] & 0x80) == 0' failed. *str* *UDFBATyearbracket(bat *ret, const bat *bid)* *{* * BAT *b, *bn;* * BUN i,n;* * str *y;* * const date *t;* * if ((b = BATdescriptor(*bid)) == NULL)* * throw(MAL, "UDF.BATyearbracket", "Cannot access descriptor");* * n = BATcount(b);* * bn = COLnew(b->hseqbase, TYPE_str, BATcount(b), TRANSIENT);* * if (bn == NULL) {* * BBPunfix(b->batCacheid);* * throw(MAL, "UDF.BATyearbracket", "memory allocation failure");* * }* * bn->tnonil = 1;* * bn->tnil = 0;* * t = (const date *) Tloc(b, 0);* * y = (str *) Tloc(bn, 0);* * for (i = 0; i < n; i++) {* * if (*t == date_nil) {* * *y = GDKstrdup(str_nil);* * } else* * UDFyearbracket(y, t);* * if (strcmp(*y, str_nil) == 0) {* * bn->tnonil = 0;* * bn->tnil = 1;* * }* * y++;* * t++;* * }* * BATsetcount(bn, (BUN) (y - (str *) Tloc(bn, 0)));* * bn->tsorted = BATcount(bn)<2;* * bn->trevsorted = BATcount(bn)<2;* * BBPkeepref(*ret = bn->batCacheid);* * BBPunfix(b->batCacheid);* * return MAL_SUCCEED;* *}* PS: I am not a c expert but i can find my way with basic operations and pointers. Any help or suggestions is appreciated. Thank you.

4 17

Upgraded to lastest MonetDB, now get error
by Bryan Senseman 04 Jan '17

04 Jan '17

Ok, I upgraded my Windows version of MonetDB to Dec2016 from Jun2016-SP1 and I get this error. !FATAL: BBPinit: head column must be VOID (ID = 3). I was able to successfully downgrade (uninstall/reinstall) Jun2016-SP1; do I need to install SP2 first? -- ------------------------------ This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.

1 0

COPY INFO Exception
by ping gao 高平(0) 03 Jan '17

03 Jan '17

Hi ALL I encountered a problem using a COPY INFO The command to execute: [hotel@02 data_visual_monetdb]$ mclient -u monetdb -d hotel-revenue -s "COPY 1727300 RECORDS INTO hotel_revenue_octopus_hotel_info FROM '/home/ping.gao/data_visual_monetdbinfo.data' USING DELIMITERS ',','\\n'<smb://n'>,'\"'" password: Failed to import table Leftover data '"0”' Data Format(Line separator —> \n): “test”,”test”,”test”,”test”

2 2

File System Backups
by Vincent Sheffer 30 Dec '16

30 Dec '16

I’m trying to do automated incremental backups following these instructions: https://www.monetdb.org/Documentation/UserGuide/FastDumpRestore My question is whether why isn’t it enough to just stop the DB? Is the waiting for open sessions a courtesy or is it required that no statements are executing prior to shutdown? Is it similar for the shutdown command? It gives people in a session time to exit or something? Again, these are to be automated at a time of night that queries should not be being executed. Bottom line question: is it enough to stop and lock a database to ensure the binary backup will have no errors that prevent the backup from being restored or are the other steps required? Thanks, Vince The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.

2 1

table renaming
by Stefano Fioravanzo 30 Dec '16

30 Dec '16

Hello, In this blog post https://www.monetdb.org/blog/monetdb-autumn-2016-feature-release-outlook <https://www.monetdb.org/blog/monetdb-autumn-2016-feature-release-outlook> it was mentioned the possibility of renaming tables. It is possible with the new Dec2016 release? Thank you Stefano

2 1

UDF performance issue
by imad hajj chahine 30 Dec '16

30 Dec '16

Hi, I have a weird behavior when i package some sql code in a function, the performance deteriorate. The same code running directly in select statement is blinking fast. ex: *select date_to_str("Expiry_Date",'%Y-%m-%d') from table*: exec time is 0.72s if i create a function: *CREATE FUNCTION DayBracket(pdate date) returns char(10)* *BEGIN* * return date_to_str(pdate,'%Y-%m-%d');* *end;* *grant execute on function DayBracket to public;* and execute *select DayBracket("Expiry_Date") from table*: exec time is 24s even when i set the limit to 1 its taking the same time, so i guess it should be something related to loading the function. The following function perform much better: *CREATE FUNCTION DayBracketOpt(pdate date) returns char(10)* *BEGIN* * return extract(year from pdate) || '-' || lpad(extract(month from pdate),2,'0') || '-' || lpad(dayofmonth(pdate),2,'0');* *end;* *grant execute on function DayBracketOpt to public;* I also tried to create a Python function: *CREATE FUNCTION pyDayBracket(pdate date) returns string* *LANGUAGE PYTHON {* * from time import strftime, strptime* * return [strftime("%Y-%m-%d", strptime(pdt,"%Y-%m-%d")) for pdt in pdate]* *};* it is taking too long and throwing an exception:buffer size mismatch PS: the pdate array type is detected in pythons as string not as date; Also how to delete a function with dependency, I need to replace some functions which are used in others functions and views. Any flags to turn off dependency check before dropping? Thank you.

4 11

Merry Christmas and Happy 2017
by Ying Zhang 23 Dec '16

23 Dec '16

Dear MonetDB users, The MonetDB team wishes you a very Merry Christmas and Happy New Year! Thank you very much for helping us improving our work! Hope to see you all back in 2017, in which we’re of course going to continue our work, and come up with new MonetDB releases with more features, better performance and improved stability. Best wishes, Ying Zhang (on behalf of the MonetDB team)

2 1

Bug #3878
by Brian Hood 22 Dec '16

22 Dec '16

Hi Guys, I remember this somehow got fixed in the next release or 2 however i am not sure how it got fixed and possibly via a different issue. So it should be closed. https://www.monetdb.org/bugzilla/show_bug.cgi?id=3878 Regards, Brian Hood

1 0

Announcement: New Dec2016 Feature release of MonetDB suite
by Panagiotis Koutsourakis 21 Dec '16

21 Dec '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016 feature release MonetDB5 Server * Removed the zorder module with functions zorder.encode, zorder.decode_x and zorder.decode_y. * Removed command bbp.getHeat(). * Removed bat.setColumn with two arguments and bat.setRole. Use bat.setColumn with one argument instead. * Removed function BKCappend_reverse_val_wrap: it was unused. * The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. * Keep a collection of full traces. Each time the SQL user applies the TRACE option, the full json trace is retained within the <dbpath>/<dbname>/sql_traces Build Environment * New packages MonetDB-python2 (Fedora) and monetdb-python2 (Debian/Ubuntu) have been created for Python 2 integration into MonetDB. * With OID size equal to ABI/word size, mserver5 does not need to print the OID size, anymore. * Removed obsolete code associated with long gone static linking option. * Removed configure option --enable-oid32 to compile with 32 bit OIDs on a 64 bit architecture. . * The Perl, PHP, and Python clients, and the JDBC driver each now have their own repositories and release cycles. The Python client is maintained by Gijs Molenaar on Github (https://github.com/gijzelaerr/pymonetdb) the other clients are maintained by CWI/MonetDB on our own server (https://dev.monetdb.org/hg/monetdb-java, https://dev.monetdb.org/hg/monetdb-perl, https://dev.monetdb.org/hg/monetdb-php) MonetDB Common * The tnokey values must now be 0 if it is not known whether all values in a column are distinct. * The 2-bit tkey field in the bat descriptor has been split into two single bit fields: tkey and tunique. The old tkey&BOUND2BTRUE value is now stored in tunique. * Implemented conversion to str from any type (not just the internal types). * VALcopy and VALinit both return their first argument on success or (and that's new) NULL on (allocation) failure. * BATattach now can also create a str BAT from a file consisting of null-terminated strings. The input file must be encoded using UTF-8. * BATattach now copies the input file instead of "stealing" it. * Removed the lastused "timestamp" from the BBP. * Removed batStamp field from BAT descriptor, and removed the BBPcurstamp function. * Removed unused functions BBPhot and BBPcold. * Removed BATderiveTailProps and BATderiveProps. Just set the properties you know about, or use BATsettrivprop. * Removed the macro BUNfirst. It can be replaced by 0. * Changed BATroles by removing the argument to set the name of the head column. * The head column is now completely gone. MonetDB is completely "headless". * The format of the BBP.dir file was simplified. Since the head column is VOID, the only value that needs to be stored is the head seqbase. * BATs now only have a single (logical) name. * The function BATmirror is gone. The HEAD column is always VOID (with a non-nil seqbase) and the TAIL column carries the data. All functions that deal with data work on the TAIL column. * BATkey now works on the TAIL column instead of the HEAD column. * Replaced BATseqbase with BAThseqbase and BATtseqbase, the former for setting the seqbase on the HEAD, the latter for setting the seqbase on the TAIL. * Replaced function BATnew with COLnew with slightly different arguments: the first argument of COLnew is the SEQBASE of the head column (which is always VOID). * The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. SQL * Removed functions sys.zorder_encode, sys.zorder_decode_x, and sys.zorder_decode_y. * The experimental recycler code is moved to the attic. * The syntax of bat.new(:oid,:any) has been changed by dropping the superflous :oid. All BATs are now binary associations between a void column and a materialized value column. (except for the internal :bat[:void,:void] representation of simple oid ranged tails.) Bug Fixes * 3357: Implement setQueryTimeout() * 3445: Add support for database name to dotmonetdb file * 3597: SQL to MAL listing looses types * 3973: JDBC hangs * 3976: Performance enhancement to LIKE without wildcards * 4005: Correlated update causes incorrect null constraint violation * 4016: merge table only optimises for point query * 4040: sys.storage call can take a long time * 4047: Segfault when updating a dropped table * 4050: Database corruption when running low on inode * 4057: missing bulk operations between constant and bat * 4061: SIGSEGV in candscan_lng * 4066: Deadlocked monetdbd * 6068: Error message about incompatible BBP version should be clearer * 6069: query with union all silently crashes * 6070: setting negative session query timeout should not be possible/allowed * 6071: where clause with cast and floor fails to sigsegv * 6072: Bind to UPD delta column does not get/show type information in EXPLAIN * 6073: Missing type information for constants in MAL explain * 6074: SET ROLE command does not work * 6075: gdk_calc.c:13113: BATcalcifthenelse_intern: Assertion `col2 != NULL' failed. * 6076: rel_optimizer.c:5426: rel_push_project_up: Assertion `e' failed. * 6077: mserver5: rel_optimizer.c:5444: rel_push_project_up: Assertion `e' failed. * 6078: rel_bin.c:2402: rel2bin_project: Assertion `0' failed. * 6084: Merge table point to wrong columns if columns in partition tables are deleted * 6108: monetdb5-sql sysv init script does not wait for shutdown * 6114: segfault raised in the query rewriter due to a null pointer * 6115: Assertion hit in the codegen * 6116: Codegen does not support certain kind of selects on scalar subqueries * 6117: Assertion hit in the query rewriting stage during the push down phase * 6118: SIGSEGV in strPut due to shared heap * 6119: Assertion hit in the MAL optimiser on a complex query * 6120: QUANTILE() treats NULL as if it is zero * 6121: SELECT a.col IN ( b.col FROM b ) FROM a statements with no error but no result * 6123: Assertion hit in the codegen #2 * 6124: CASE <column> WHEN NULL THEN 0 ELSE 1 END returns wrong result * 6125: Stack overflow in the query rewriter with a query having an OR condition and a nested SELECT subexpression * 6126: batcalc.== can't handle void BATs * 6139: Debian libmonetdb13 conflicts with libmonetdb5-server-geom

1 0

Returning empty results from an embedded python table function.
by Joseph D'silva 20 Dec '16

20 Dec '16

Hi All, I am trying to return an "empty" result set from an embedded python function (as in some situations I will not have valid rows to produce in the output). However the very many attempts of trying to return an "empty" result from the function have failed (as is documented in the example below). How do you get it done ? Thanks. DROP FUNCTION p_results_exp01; CREATE FUNCTION p_results_exp01() RETURNS TABLE(val1 STRING, val2 INTEGER) LANGUAGE PYTHON { #Unsupported result object. Expected either a list, dictionary, a numpy array, a numpy masked array or a pandas data frame, but received an object of type "<class 'NoneType'>" #return None; #Errors #Expected a return value with name "val1", but this key was not present in the dictionary. #return {}; #Errors #Error converting dict return value "val1": MALException:pyapi.eval:Unsupported result object. Expected either a list, dictionary, a numpy array, a numpy masked array or a pandas data frame, but received an object of type "<class 'NoneType'>". #return {'val1':None, 'val2':None}; #Errors #Error converting dict return value "val1": MALException:pyapi.eval:An array of size 0 was returned, yet we expect a list of 1 columns. The result is invalid.. #return {'val1':[], 'val2':[]}; #return {'val1':['value_1'], 'val2':[1]}; #Works, as it has value. }; SELECT * FROM p_results_exp01(); Joseph

2 3

Python integration
by Imad Hajj Chahine 17 Dec '16

17 Dec '16

Hi, I need to calculate weighted quartile from a list of values and their counts, and all solutions that i found revolves around calculating the cumulative count. Since monetdb lacks the sum over window function, I want to create an aggregate function in python to return the weighted quartile after computing the cumulative count. As I understand there is 3 branches: release, candidate and dev. Currently the release version is june2016, candidate is dec 2016, dev is the default. Is the python integration only supported on default branch? Any plan to move it to stable branch? Thank you.

3 3

SUM() OVER (ORDER BY..)
by Anthony Damico 13 Dec '16

13 Dec '16

hi, monetdb does not support SUM() OVER commands.. does anyone have a smart alternative to implement this? thanks SELECT batch_id, job_count, SUM(job_count) OVER (ORDER BY duration) as cumjobs, SUM(job_count) over () as totjobs, duration FROM test_data ;

5 13

min() vs median() for string columns
by Eyal Rozenberg 13 Dec '16

13 Dec '16

When I try to get the min() or the max() of a string column (whose type is CHAR(2) ) - that works; but when I try to get the median() - I get an error about not being able to convert a value (say, 'AA') to type bte. Is this intended behavior? A bug? Perhaps something I might be doing wrong? PS - In case it matters: * I'm using v11.23.7 * I'm using the USDT on-time flight data, in a DB you can setup using the script here: https://github.com/eyalroz/usdt-ontime-tools

1 0

MonetDB unpredictable performance
by Kevork Vartanian 08 Dec '16

08 Dec '16

Hi, I have a few questions and hopefully someone will be able to answer me and put me in the right direction. - When we're having performance issues, the only way we can resolve them, is by dumping the data again, sorted by datetime, and after that the queries are fast. When we do this, the table enables sorting on the datetime column and that's why the queries are fast as there won't be a full table scan. As soon as we insert new data (always in the same sorted order), firstly the table is losing the sorting and secondly the queries become extremely slow - We find it very unpredictable, and somehow not very deterministic, especially when it comes to indexing, sorting and log file consumption. We can't control those, it seems - The read ahead log is growing uncontrollably and we don't really know when it's committing - When running the queries (after the table loses the sorting), we can see it's writing too much data (storage usage grows significantly when query is still running) and that's most probably because it is doing a full table scan - Even if we restart the engine, it takes a while for it to be back, as it recommits the read ahead log which takes time, and it doesn't make a difference with the sorting and therefore the queries are still slow even after a restart Is there a way we can further optimise so we don't have these issues? We have more than 600 million entries and when the datetime field is sorted, the queries are quick. The only way we can force the sorting is when we dump the data again, but we can't really do that all the time and we need to insert new data at least once every night. It's an Ubuntu 14.04.4 LTS instance on AWS with 2 CPU's and 16 GB of RAM (and SSD drives). Thank you. Regards, Kevork Vartanian Technical Solutions Consultant kevork.vartanian(a)innotta.com.au | +61 406 776 789 | www.innotta.com.au

2 1

null value from python function
by Stefano Fioravanzo 05 Dec '16

05 Dec '16

Hello, Some months ago I reported a problem when returning null values from an embedded py function (may they be inside arrays or data frames). Indeed, the null values cannot be queried using sql like: SELECT * FROM t WHERE v IS NULL the above query does not return any result, but selecting the row mclient presents a ‘null’ values in the cell. I am using a development build of a couple of months ago, I would like to know if this issue has been resolved or not. Thank you Stefano

2 1

Database dropping connections frequently with proxying.
by Joseph D'silva 01 Dec '16

01 Dec '16

Running on Ubuntu 16.04 LTS, database and clients run on the same system. I am using the Dec2016 branch, and noticed that the database connections get dropped randomly, but in high frequency. This error occurs irrespective for whether I use mclient, python package (pymonetdb) or the JDBC driver from the site. >From what I could figure if I use the forward=redirect option instead of the default proxy, the frequency reduces drastically, but is still present (if I leave the mclient connection idle for like 5 minutes it will drop - is there an idle time out somewhere ? Also is there a way to force a network connection locally instead of unix socket ? ). Sample of some messages For the default proxy option =========================== >From merovingian.log 2016-12-01 10:59:33 MSG merovingian[29445]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying Python error due to dropped connection. >>> rows=cursor.execute("SELECT 'HELLO';"); Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.5/dist-packages/pymonetdb/sql/cursors.py", line 164, in execute block = self.connection.execute(query) File "/usr/local/lib/python3.5/dist-packages/pymonetdb/sql/connections.py", line 133, in execute return self.command('s' + query + ';') File "/usr/local/lib/python3.5/dist-packages/pymonetdb/sql/connections.py", line 138, in command return self.mapi.cmd(command) File "/usr/local/lib/python3.5/dist-packages/pymonetdb/mapi.py", line 208, in cmd self._putblock(operation) File "/usr/local/lib/python3.5/dist-packages/pymonetdb/mapi.py", line 310, in _putblock self._putblock_inet(block) File "/usr/local/lib/python3.5/dist-packages/pymonetdb/mapi.py", line 321, in _putblock_inet self.socket.send(flag) BrokenPipeError: [Errno 32] Broken pipe JDBC Error (I am using PyCharm to execute some SQLs) [08000] Connection to server lost! (mserver still alive?) On the other hand, mclient does not show any error message, tried -X option, but no message when it dies. It just dies while executing a command (probably because the connection is dropped) A peek at netstat shows that the connections disappear from it before mclient dies. $ netstat -anp | grep 50000 tcp 0 0 127.0.0.1:50000 0.0.0.0:* LISTEN 29445/monetdbd udp 0 0 127.0.0.1:50000 0.0.0.0:* 29445/monetdbd unix 2 [ ACC ] STREAM LISTENING 3869978213 <(386)%20997-8213> 29445/monetdbd /tmp/.s.monetdb.50000 unix 2 [ ACC ] STREAM LISTENING 3869978215 <(386)%20997-8215> 29445/monetdbd /tmp/.s.merovingian.50000 unix 3 [ ] STREAM CONNECTED 3875263615 29486/mserver5 /tmp/.s.monetdb.50000 unix 2 [ ] STREAM CONNECTED 3871676942 29486/mserver5 /tmp/.s.monetdb.50000 $ netstat -anp | grep 50000 tcp 0 0 127.0.0.1:50000 0.0.0.0:* LISTEN 29445/monetdbd udp 0 0 127.0.0.1:50000 0.0.0.0:* 29445/monetdbd unix 2 [ ACC ] STREAM LISTENING 3869978213 <(386)%20997-8213> 29445/monetdbd /tmp/.s.monetdb.50000 unix 2 [ ACC ] STREAM LISTENING 3869978215 <(386)%20997-8215> 29445/monetdbd /tmp/.s.merovingian.50000 unix 2 [ ] STREAM CONNECTED 3871676942 29486/mserver5 /tmp/.s.monetdb.50000 After switching to the forward option (where this issue is less frequent to be bothersome) =========================== >From merovingian.log 2016-12-01 11:22:59 MSG merovingian[29824]: redirecting client (local) for database 'voc' to mapi:monetdb://cerberus:50001/voc Although for most part the connections are stable, if and when mclient dies, I do notice that the netstat entries are NOT being removed. (see the entries piling up at the bottom though there are no active client processes) $ netstat -anp | grep 50000 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 127.0.0.1:50000 0.0.0.0:* LISTEN 29824/monetdbd tcp 1 0 127.0.0.1:50000 127.0.0.1:35870 CLOSE_WAIT 29824/monetdbd udp 0 0 127.0.0.1:50000 0.0.0.0:* 29824/monetdbd unix 2 [ ACC ] STREAM LISTENING 3876967185 29824/monetdbd /tmp/.s.monetdb.50000 unix 2 [ ACC ] STREAM LISTENING 3876967187 29824/monetdbd /tmp/.s.merovingian.50000 unix 2 [ ] STREAM CONNECTED 3877116313 29824/monetdbd /tmp/.s.monetdb.50000 unix 2 [ ] STREAM CONNECTED 3880819958 29824/monetdbd /tmp/.s.monetdb.50000 unix 2 [ ] STREAM CONNECTED 3878012589 29824/monetdbd /tmp/.s.monetdb.50000 unix 2 [ ] STREAM CONNECTED 3877885158 29824/monetdbd /tmp/.s.monetdb.50000 Thanks, Joseph

1 0

Building monetdb with Python 3
by Joseph D'silva 01 Dec '16

01 Dec '16

Hi all, Has anyone had any luck building the sources with Python 3 ? I tried building the Dec2016 branch from git, passing --with-python3=/usr/bin/python3 --with-pyversion=/usr/bin/python3 --with-pyconfig=/usr/bin/python3-config to configure, but received many errors during make under the pyapi directory, such as. /usr/include/python3.5m/object.h:86:5: error: missing braces around initializer [-Werror=missing-braces] { _PyObject_EXTRA_INIT \ ^ connection.c:159:5: note: in expansion of macro ‘PyObject_HEAD_INIT’ PyObject_HEAD_INIT(NULL) ^ /usr/include/python3.5m/object.h:86:5: note: (near initialization for ‘Py_ConnectionType.ob_base’) { _PyObject_EXTRA_INIT \ ^ connection.c:159:5: note: in expansion of macro ‘PyObject_HEAD_INIT’ PyObject_HEAD_INIT(NULL) ^ connection.c:159:5: error: missing initializer for field ‘ob_size’ of ‘PyVarObject {aka struct <anonymous>}’ [-Werror=missing-field-initializers] ................. connection.c:246:5: error: ‘return’ with a value, in function returning void [-Werror] import_array(); ^ It looks like some of the macro definitions to make python3 header files work with the pyapi source might be missing. I tried fixing some of the sources like unicode.c, formatinput.c etc. But it looks like majority of the source files have issues. Thanks, Joseph

2 2

MonetDB and IN search
by blacky_2k2＠yahoo.com 29 Nov '16

29 Nov '16

Hello, We’ve been battling for some time with what is fundamentally a simple query. We have a databases with about 15 columns which store int and float values. The query searches in each column with an IN condition , where the possible test values in the IN condition may vary from a 1 to 200 - where these 200 conditions are all the possible values of that column. So normally a query would look something like: SELECT id FROM table WHERE a IN (1,2,3,4) AND b IN (4,5,6,) AND c IN (2,5,6) ...... AND i = 5 etc. The database has about 2 mil rows but will grow to potentially 10 mil rows. A query similar to the one above takes about 80-100 ms. With about 50 test values for each IN. So far, the best results we got with MariaDB and proper indexing using Engine Memory where the above query takes about 2ms. We suspect MonetDB is not optimised for such queries, but since we don’t know much about its workings can you please confirm this? Otherwise, is there a way to improve the above query? Say using multiple JOINs instead of IN ? By the way, we are experimenting with MonetDB out of curiosity, we heard a lot of good things about it. Thank you very much, Silviu

3 4

Improved concurrency version of MonetDB
by Puthick Hok 24 Nov '16

24 Nov '16

Hi, I'm using MonetDB v11.23.13 (Jun2016-SP2). We decided to use MonetDB to store DNA marker data for our plant breeding platform (www.kddart.org) 2 years ago based on MonetDB performance on data retrieval and data update is also good. The use of MonetDB is not purely OLAP. There are lot of aspects of OLTP because people need to load marker data from their genotyping services. We finish integrating MonetDB into the platform and developing more tools for the system. As the platform is getting more mature, we started to do concurrency testing against the system including the part that is relying on MonetDB. Naively, I ran into a concurrency problem described in a master thesis by Rene Tassy (Improving Transactional Scalability in MonetDB). The work done in the thesis is very promising and I would like to thank you all for that. My question in relation to how I would deal with my problem is that what is the time frame for the improved MVCC Optimistic Concurrency Strategy in the MonetDB release? Regards, Puthick

2 1

Functions in MOnetdb
by Ariel Abadi 22 Nov '16

22 Nov '16

Hi! We are trying to create a couple of functions in Monet. Every function we create has a terrible bad performance. Can someone help me to find out what Im doing wrong ? Here is the function CREATE FUNCTION sb_glob.camelcase(input VARCHAR(500)) RETURNS VARCHAR(500) begin DECLARE len INT; DECLARE i INT; SET len = CHAR_LENGTH(input); SET input = LOWER(input); SET i = 0; WHILE (i < len) DO SET input = CASE WHEN ((SUBSTR(input,i,1) = ' ' OR SUBSTR(input,i,1) = '-' OR i = 0) AND (i < len)) THEN (LEFT(input,i) || UPPER(SUBSTR(input,i + 1,1)) || RIGHT(input,len - i - 1)) ELSE input END; SET i = i + 1; END WHILE; RETURN input; END Thks in advance!

2 1

filter based on json value of key/value pair
by Lynn Carol Johnson 21 Nov '16

21 Nov '16

Hello all - I'm having trouble pulling rows based on the value of a json key/value pair. I have created a table with a json object that looks like this: sql>\d testjson CREATE TABLE "sys"."testjson" ( "chr" INTEGER, "pos" INTEGER, "value" JSON ); sql> I have entries in the table that look as below: sql>select * from testjson; +------+-----------+----------------------------------------------------------------+ | chr | pos | value | +======+===========+================================================================+ | 1 | 209890809 | {"df":1,"pval":4.556021744872574E-6,"r2":0.0995066009518681} | | 1 | 789383847 | {"df":1,"pval":2.50962115178055E-6,"r2":0.10462833261259474} | | 2 | 127893782 | {"df":1,"pval":4.2825829011938765E-6,"r2":0.10003907080878045} | | 2 | 234533212 | {"df":2,"pval":4.556021744872574E-6,"r2":0.0995066009518681} | +------+-----------+----------------------------------------------------------------+ 4 tuples (1.027ms) sql> I would like to filter on json values. Not the json key, the json key's value. I can pull json values based on a non-json field, e.g. Grab the pval where chr>1 as below: sql>select json.filter(value,'r2') as r2Value from testjson where chr=1; +-----------------------+ | r2value | +=======================+ | [0.0995066009518681] | | [0.10462833261259474] | +-----------------------+ 2 tuples (1.179ms) sql> But I want to pull chr/pos where pval < some amount, or df > some number. For example, how would I write the query to select all rows where df > 1? This query should return the last row of the table above. I've tried the following: sql>select * from testjson where (json.filter(value,'df') > 1); types json(0,0) and tinyint(8,0) are not equal sql> Thanks for your help - Lynn

2 2

how to structure queries like SELECT a.col IN ( b.col FROM b ) FROM a ?
by Anthony Damico 21 Nov '16

21 Nov '16

hi monetdb team, i've pasted a minimal reproducible example of a few SQL queries that i had expected to work -- only about half of them do work, and the other half just error with 'ParseException:SQLparser:' my goal is to get the very last line working, so if anybody could suggest a hack or a workaround that doesn't involve a JOIN, i would appreciate it. thanks all!! CREATE TABLE table_one (this_column INTEGER) ; INSERT INTO table_one VALUES (1) ; INSERT INTO table_one VALUES (2) ; INSERT INTO table_one VALUES (3) ; INSERT INTO table_one VALUES (4) ; INSERT INTO table_one VALUES (5) ; CREATE TABLE table_two (this_column INTEGER) ; INSERT INTO table_two VALUES (1) ; INSERT INTO table_two VALUES (2) ; # works as expected SELECT SUM( ( table_one.this_column IN ( SELECT this_column FROM table_two ) ) ) FROM table_one ; SELECT MAX( ( table_one.this_column IN ( SELECT this_column FROM table_two ) ) ) FROM table_one ; # fails SELECT ( table_one.this_column IN ( SELECT this_column FROM table_two ) ) AS new_column FROM table_one ; # Server says 'ParseException:SQLparser:'. # works SELECT AVG( CASE WHEN table_one.this_column IN ( SELECT this_column FROM table_two ) THEN 1 ELSE 0 END ) AS new_column FROM table_one ; # fails SELECT ( CASE WHEN ( table_one.this_column IN ( SELECT this_column FROM table_two ) ) THEN 1 ELSE 0 END ) AS new_column FROM table_one ; # Server says 'ParseException:SQLparser:'. # fails [[this query is the one i would like to implement]] SELECT COUNT(*) , ( CASE WHEN ( table_one.this_column IN ( SELECT this_column FROM table_two ) ) THEN 1 ELSE 0 END ) AS new_column FROM table_one GROUP BY new_column ; # Server says 'ParseException:SQLparser:'.

3 8

Unable to install//access external R Packages inside of MonetDB
by Cas Saternos 10 Nov '16

10 Nov '16

Hello All, Just wanted to report a bug, or at least unexpected behavior. I am new to monetdb, and could not submit this to your bugzilla tracker. *Background* I successfully have used MonetDB with Docker ( https://www.monetdb.org/blog/docker-container-monetdb-r) and was able to create FUNCTIONs and AGGREGATEs with R (as described here: https://www.monetdb.org/content/embedded-r-monetdb) This blog post also states: "As a final note on usage, you can use any R package from CRAN within your function without explicitly installing it. We overloaded the library function in R to automatically install missing packages." *Description* However, I am not able to use external R packages available from CRAN. The session below describes both Docker commands and SQL used. This session was run on OSX using Docker Machine. Thank you, -Cas *bash-3.2$* docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d6b944445237 monetdb/monetdb-r-docker:latest "/usr/bin/supervisord" 13 minutes ago Up 13 minutes 0.0.0.0:32768->50000/tcp monetdb-r-docker *bash-3.2$* docker exec -it monetdb-r-docker mclient db user:monetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://d6b944445237: 50000/db' Type \q to quit, \? for a list of available commands auto commit mode: on # Create and use a function to show that MonetDB with basic R expressions works: *sql>*CREATE FUNCTION test_add() RETURNS DOUBLE LANGUAGE R { 1 + 1 }; operation successful (1.254ms) *sql>*SELECT test_add(); +--------------------------+ | test_add | +==========================+ | 2 | +--------------------------+ # Add only the library function call to the function just shown (and give the function a new name). The function is created successfully, but an error occurs when the function is invoked in a query: *sql>*CREATE FUNCTION test_library() RETURNS DOUBLE LANGUAGE R { library("ggplot2"); 1 + 1 }; operation successful (1.378ms) *sql>*SELECT test_library(); Error running R expression. Error message: Error in .library.original(package, help, pos, lib.loc, character.only = T, : there is no package called 'ggplot2' Calls: as.data.frame -> <Anonymous> -> library -> .library.original *sql>*

1 0

NULL in COPY INTO FROM bulk
by Mathieu Raillard 10 Nov '16

10 Nov '16

Hi all, We are working with monetdb since 1 year and half, the performance are great. We are using in production the July 2015 release and we want to upgrade to the last release. We installed the new version and ran some tests to ensure the compatibility, unfortunatly we discovered a change in the way monetdb is handling NULL in bulk import. We are exporting datas from Mysql and by default we get a \N for every NULL value plus every field are enclosed by "" ( even number and timestamp). Exemple : 1,test,2016-01-01 00:00:00,null in database become "1","test","2016-01-01 00:00:00',"\N" in bulk. This bulk is working fine with the July 2015 realease but we cant import "\N" in number and timestamp with the june 2016 realase Have you an idea how can we make it works ? Here is some results from my tests On the July 2015 release : create schema test; create table test.test ( field1 int, field2 timestamp, field3 varchar(20) ); Bulk1 : "1","2016-01-01 00:00:00","test" COPY 10 RECORDS INTO test.test FROM '/tmp/bulk1' USING DELIMITERS ',','\n','"' NULL AS '\\N' ; 1 affected row (52.812ms) Bulk2 : "1","2016-01-01 00:00:00","\N" COPY 10 RECORDS INTO test.test FROM '/tmp/bulk2' USING DELIMITERS ',','\n','"' NULL AS '\\N' ; 1 affected row (47.852ms) Bulk3 : "1","\N","test" COPY 10 RECORDS INTO test.test FROM '/tmp/bulk3' USING DELIMITERS ',','\n','"' NULL AS '\\N' ; 1 affected row (58.811ms) Bulk4 : "\N","2016-01-01 00:00:00","test" COPY 10 RECORDS INTO test.test FROM '/tmp/bulk4' USING DELIMITERS ',','\n','"' NULL AS '\\N' ; 1 affected row (32.812ms) Bulk5 : "\N","\N","\N" COPY 10 RECORDS INTO test.test FROM '/tmp/bulk5' USING DELIMITERS ',','\n','"' NULL AS '\\N' ; 1 affected row (63.785ms) select * from test.test field1 field2 field3 1 2016-01-01 00:00:00.0 test 1 2016-01-01 00:00:00.0 <null> 1 <null> test <null> 2016-01-01 00:00:00.0 test <null> <null> <null> Tested on the Jun 2016 release : create schema test; create table test.test ( field1 int, field2 timestamp, field3 varchar(20) ); Bulk1 : "1","2016-01-01 00:00:00","test" COPY 10 RECORDS INTO test.test FROM '/tmp/bulk1' USING DELIMITERS ',','\n','"' NULL AS '\\N' ; 1 affected row (61.812ms) Bulk2 : "1","2016-01-01 00:00:00","\N" COPY 10 RECORDS INTO test.test FROM '/tmp/bulk2' USING DELIMITERS ',','\n','"' NULL AS '\\N' ; 1 affected row (47.314ms) Bulk3 : "1","\N","test" COPY 10 RECORDS INTO test.test FROM '/tmp/bulk3' USING DELIMITERS ',','\n','"' NULL AS '\\N' ; Failed to import table line 1 field 2 'timestamp(7)' expected in '\N' Bulk4 : "\N","2016-01-01 00:00:00","test" COPY 10 RECORDS INTO test.test FROM '/tmp/bulk4' USING DELIMITERS ',','\n','"' NULL AS '\\N' ; Failed to import table line 1 field 1 'int' expected in '\N' Bulk5 : "\N","\N","\N" COPY 10 RECORDS INTO test.test FROM '/tmp/bulk5' USING DELIMITERS ',','\n','"' NULL AS '\\N' ; Failed to import table line 1 field 1 'int' expected in '\N' select * from test.test +--------+----------------------------+--------+ | field1 | field2 | field3 | +========+============================+========+ | 1 | 2016-01-01 00:00:00.000000 | test | | 1 | 2016-01-01 00:00:00.000000 | N | +--------+----------------------------+--------+ Regards Mathieu

1 0

COPY subquery INTO file_name - INCLUDE field name
by Ariel Abadi 10 Nov '16

10 Nov '16

Hi! We are using the COPY *subquery* INTO *file_name *[ [USING] DELIMITERS *field_separator* [',' *record_separator* [ ',' *string_quot*e ]]] [ NULL AS *null_string* ] BUT we need to have in the file the name of the fields on the first row. Im looking on the documentation, but I cannot find anything related to this, can someone help me with this? Thks in advance!

3 3

Fwd: Performance tuning
by Rajendra Bhat 09 Nov '16

09 Nov '16

Hi team, I am running monetdb on 16 core processor and 30gb ram. I have table around 54 fields. gzip compressed csv size is around 6.5 GB. query will take around 6 min. please suggest tuning options. query: Select count(distinct userid),sum(mrp)/sum(noOfItems),sum(mrp)/sum( orderCount),sum(revenue),sum(orderCount),sumnoOfItems),sum( returnCount),sum(rtoCount),sum(0),sum(0),sum(tDiscountPer),sum(cDiscountPer) from historical where gender='men' and brand='puma' Thanks and Regards Rajendra Bhat -- Thanks and Regards Rajendra Bhat

2 3

How do I force start a new monetdbd instance?
by Anderson, David B 09 Nov '16

09 Nov '16

Or what files do I need to delete from the file system? When I try to start a new monetdbd instance, I am getting the error: another monetdbd is already running but there are no monetdb processes running. . .

2 1

TimeZone conversion functions
by Ariel Abadi 03 Nov '16

03 Nov '16

Hi! Can anyone help me understand how or where I can check the included functions on MonetDB to convert timestamp from one timezone to another? Thanks in advance! Rgds Ariel

2 2

MAL optimizer result inspection
by Moritz Bruder 02 Nov '16

02 Nov '16

Hi, is it possible to look at the output of a specific optimizer? I tried using *optimizer.showFlowGraph*, which just hangs, and *optimizer.showPlan*, which does not show some variables. Is there maybe something I missed? Regards, Moritz

2 1

Getting "NOT NULL constraint violated" for a column with a DEFAULT value
by Eyal Rozenberg 24 Oct '16

24 Oct '16

I have a column defined as follows: "div5tailnum" VARCHAR(10) NOT NULL DEFAULT '' (that's the output of \d mytable), or if you like: Div5TailNum VARCHAR(10) NOT NULL DEFAULT '' (that's from the CREATE TABLE command). I'm trying to load data from a CSV file using the following SQL: COPY OFFSET 2 INTO mytable FROM file.csv USING DELIMITERS ',','\n','"' NULL AS ''; but I get: INSERT INTO: NOT NULL constraint violated for column ontime.div5tailnum shouldn't that be impossible, seeing how I've provided a default?

3 4

Cannot install MonetDB on Ubuntu 16.04.1 LTS (Xenial Xerus)
by Peter Lee 24 Oct '16

24 Oct '16

Hi- I couldn't install MonetDB on Ubuntu 16.04.1 LTS (Xenial Xerus) hosted on EC2 yesterday. I kept getting the error below. It looks like the repository does not have a "source" folder. However, I noticed that the "Trusty" version had a "source" folder, and so I tried installing the Trusty version on my Xenial box, and it installed successfully. So far, I haven't encountered any problems, but was there anything I did wrong in the installation process? Will I encounter problems down the road with the Trusty version installed on a Xenial box? $ sudo apt-get update Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial InRelease Hit:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB] Ign:5 http://dev.monetdb.org/downloads/deb xenial InRelease Get:6 http://dev.monetdb.org/downloads/deb xenial Release [2,715 B] Get:7 http://dev.monetdb.org/downloads/deb xenial Release.gpg [163 B] Fetched 97.3 kB in 0s (185 kB/s) Reading package lists... Done W: http://dev.monetdb.org/downloads/deb/dists/xenial/Release.gpg: Signature by key 213E64DCD5DDC2C063CC39FD053C3ED40583366F uses weak digest algorithm (SHA1) E: Failed to fetch http://dev.monetdb.org/downloads/deb/dists/xenial/Release Unable to find expected entry 'monetdb/source/Sources' in Release file (Wrong sources.list entry or malformed file) E: Some index files failed to download. They have been ignored, or old ones used instead.

3 2

Fwd: query with group.subgroupdone' undefined in
by José Cavieres 22 Oct '16

22 Oct '16

Hello, Can anyone explain why adding "2*" in the following query results in an error: Original Query: sql>select (CAST(now()-'1970-01-01 00:00:00' as BIGINT)) as minuto,avg(CAST(N6_end - trStart as BIGINT)) as promedio from testquery where trStart>'2016-10-06 00:00:00.000000' GROUP BY minuto; +---------------+--------------------------+ | minuto | promedio | +===============+==========================+ | 1475770734000 | null | +---------------+--------------------------+ Query with "2*": sql>select 2*CAST(now()-'1970-01-01 00:00:00' as BIGINT) as minuto,avg(CAST(N6_end - trStart as BIGINT)) as promedio from testquery where trStart>'2016-10-06 00:00:00.000000' GROUP BY minuto; TypeException:user.s20_1[10]:'group.subgroupdone' undefined in: (X_15:any,r1_15:any,r2_15:any) := group.subgroupdone(X_14:hge); program contains errors Explain: ------------------------------------------------------------ ------------------------------------------------------------ ---------------------+ | mal | +=========================================================== ============================================================ ======================+ | function user.s22_1(A0:bte,A1:str,A2:str):void; | sql.mvc(); | | X_5:timestamp := mtime.current_timestamp(); | | A1; | | calc.timestamp(X_6,7,1); | | X_10:lng := mtime.diff(X_5,X_7); | | calc.lng(X_10); | | calc.hge(X_11); | | A0; | | X_14:hge := calc.*(X_12,X_13); | | (X_15,r1_15,r2_15) := group.subgroupdone(X_14); | | algebra.projection(r1_15,X_14); | | bat.setKey(X_18,true); | | X_21:bat[:oid] := sql.tid(X_4,"sys","testquery") ; | | X_24:bat[:timestamp] := sql.bind(X_4,"sys","testquery" ,"trstart",0); | | (X_27,r1_27) := sql.bind(X_4,"sys","testquery" ,"trstart",2); | | X_30:bat[:timestamp] := sql.bind(X_4,"sys","testquery" ,"trstart",1); | | sql.delta(X_24,X_27,r1_27,X_30); | | algebra.projection(X_21,X_31); | | X_33:bat[:timestamp] := mal.multiplex("calc","timestam p",X_32,7); | | A2; | | calc.timestamp(X_36,7); | | algebra.thetasubselect(X_33,X_37,">"); | | X_40:bat[:timestamp] := sql.bind(X_4,"sys","testquery" ,"n6_end",0); | | (X_42,r1_42) := sql.bind(X_4,"sys","testquery" ,"n6_end",2); | | X_44:bat[:timestamp] := sql.bind(X_4,"sys","testquery" ,"n6_end",1); | | sql.delta(X_40,X_42,r1_42,X_44); | | algebra.projection(X_21,X_45); | | algebra.projection(X_38,X_46); | | algebra.projection(X_38,X_32); | | X_49:bat[:lng] := mal.multiplex("mtime","diff",X _47,X_48); | | batcalc.lng(X_49); | | batcalc.dbl(X_52); | | X_54:bat[:dbl] := aggr.subavg(X_53,X_15,r1_15,tr ue,true); | | bat.new(nil:oid,nil:str); | | bat.new(nil:oid,nil:str); | | bat.new(nil:oid,nil:str); | | bat.new(nil:oid,nil:int); | | bat.new(nil:oid,nil:int); | | bat.append(X_58,".L1"); | | bat.append(X_61,"minuto"); | | bat.append(X_62,"hugeint"); | | bat.append(X_63,72); | | bat.append(X_65,0); | | bat.append(X_66,"sys.L3"); | | bat.append(X_68,"promedio"); | | bat.append(X_70,"double"); | | bat.append(X_72,53); | | bat.append(X_74,0); | | sql.resultSet(X_76,X_78,X_80,X_82,X_84,X_19,X_54); | | end user.s22_1; | | # querylog.define("explain select 2*cast(now()-\\'1970-01-01 00:00:00\\' as bigint) as minuto,avg(cast(n6_end - trstart as bigint)) as prom | : edio from testquery where trstart>\\'2016-10-06 00:00:00.000000\\' group by minuto;","default_pipe") : Environment: - MonetDB v11.23.7 (Jun2016-SP1) from release package - Centos and Ubuntu - Create table attached - The type of the columns in the query is timestamp - A previous versions of this email was rejected because the "create table" statement is too big for the 40Kbytes limit of this list. Thanks Jose Cavieres

4 7

system init script
by Heyns Emiliano 21 Oct '16

21 Oct '16

Does anyone have a working init script for MonetDB? -- Groet, Emiliano Heyns

1 0

how well does MonetDB scale to large servers (e.g. 4 CPU/64+ cores/2+TB RAM)?
by Igor Mendelev 21 Oct '16

21 Oct '16

[Sorry if this question was already asked and answered before - I'm new to this mailing list.] Recently servers with up to 6TB RAM became relatively affordable to be used with MonetDB (considering total cost of server hw + DB sw license) - but will MonetDB scale well to use all that hardware? I know it can be tried on the cloud using (for example) x1.32xlarge EC2 spot instances but before doing that I'd like to ask people with more knowledge/experience with MonetDB if it at least looks feasible, if there are any important tuning parameters for such hw etc. If you've tired similar hardware and/or know some relevant implementaton details maybe you can share your experience? Thanks. Igor

2 1

Data lost
by Luciano Sasso 17 Oct '16

17 Oct '16

Hi, I am delete rows from table, with "delete from mytable where id=x", and rows deleted no return in select "select * from mytable where id=x", its OK ! Version "Jul2015-SP1" But, when restart database, deleted rows return in select "select * from mytable where id=x". Its bug ?, or i have execute vacuum command, how to execute vacuum command. ? -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | 17 3202-4040 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1

new monetdb version, old farm
by Lynn Carol Johnson 17 Oct '16

17 Oct '16

Hello - My understanding is you should not run a new monetdb version with a dbfarm created from an older version. We have just updated our system from a Jul2015-SP3 version to the latest Jun2016-SP2 version. I have created a new db farm and plan to recreate the tables in the new farm. My question; How do you remove the old dbfarm? Our data is quite large (1.8 T) and I'd like to retrieve this space for other use. While I can start the db against the new farm, I can find no method of removing the old dbfarm to reclaim this space. Thanks - Lynn

5 7

Zero copy optimization in embedded python and loop back queries
by Joseph D'silva 16 Oct '16

16 Oct '16

Hi All, I was wondering if zero copy optimization in embedded Python is applicable even for results returned inside functions that use _conn.execute("select ...") ? Also is there something to look for in explain plan that will indicate if zero copy optimization is used or not ? Thanks -J

1 0

weird behavior when using views with calculated columns and double type
by imad hajj chahine 14 Oct '16

14 Oct '16

Hi, When paginating over a view that contains a calculated double columns. i got a TypeException error and sometimes the server crash. To replicate the error, create a table with 2 double columns col1 and col2, fill it with some test data; after create a view as select col1, col2, col1+col2 as col3 from table. when running select col1, col2, col3 from view limit 100 offset 100 i got the error. Also in related context, when i use 2 quantiles function on the calculated view column i get the same error. select quantile(col3, 0.25), quantile(col3, 0.75) from view. I am using jun2016-sp1 on windows. Thank you.

2 1

Announcement: New Jun2016-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 13 Oct '16

13 Oct '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2016-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jun 2016-SP2 bugfix release Build Environment * We now use the CommonCrypto library instead of the OpenSSL library on Darwin. Bug Fixes * 3939: Assert failure on concurrent queries when querying sys.queue * 4019: Casting a timestamp from a string results in NULL * 4025: expressions in the WHERE clause that evaluates incorrectly * 4038: After upgrade from 11.21.19, jdbc couldn't list tables for non sys users * 4044: Server crash when trying to delete a table has been added to a merge table with "cascade" at the end * 4049: Wrong results for queries with "OR" and "LEFT JOIN" * 4052: Infinite loop in rel_select * 4054: copy into file wrongly exports functions * 4058: Server crashes with a particular conditional query * 4059: Geom functions only visible by user monetdb * 4060: BAT leak in some aggregate queries * 4062: Error: SELECT: no such binary operator 1800 790 9186'like(varchar,varchar)' when used in query running in other schema than sys * 4064: Assertion: column not found * 4067: Relevant column name not printed when a CSV parsing error occurs * 4070: Extra condition in join predicate of explicit join produces wrong MAL code * 4074: Cannot use prepared statements when caching disabled * 6065: CTE with row number and union fails within MAL

1 0

EXPLAIN output *almost* re-parsable
by Roberto Cornacchia 10 Oct '16

10 Oct '16

Almost for fun, I did this little experiment: take the output of an EXPLAIN from sql, and execute it in MAL (mclient -l mal). I found only *1* detail which doesn't parse, here it is in a minimal example: * Got the explain of a trivial query: sql>\f raw sql>explain select name from sys.functions; % .explain # table_name % mal # name % clob # type % 94 # length function user.s4_1():void; X_32:void := querylog.define("explain select name from sys.functions;","default_pipe",21); X_16 := bat.new(nil:oid,nil:str); X_24 := bat.append(X_16,"sys.functions"); X_19 := bat.new(nil:oid,nil:str); X_26 := bat.append(X_19,"name"); X_20 := bat.new(nil:oid,nil:str); X_27 := bat.append(X_20,"varchar"); X_21 := bat.new(nil:oid,nil:int); X_29 := bat.append(X_21,256); X_23 := bat.new(nil:oid,nil:int); X_31 := bat.append(X_23,0); X_1 := sql.mvc(); C_2:bat[:oid] := sql.tid(X_1,"sys","functions"); X_5:bat[:str] := sql.bind(X_1,"sys","functions","name",0); (C_8,r1_8) := sql.bind(X_1,"sys","functions","name",2); X_11:bat[:str] := sql.bind(X_1,"sys","functions","name",1); X_13 := sql.delta(X_5,C_8,r1_8,X_11); X_14 := algebra.projection(C_2,X_13); sql.resultSet(X_24,X_26,X_27,X_29,X_31,X_14); end user.s4_1; # optimizer.mitosis() # optimizer.dataflow() * Opened a MAL session, initialized sql, and pasted the function above: mal>sql.init(); mal>function user.s4_1():void; mal> X_32:void := querylog.define("explain select name from sys.functions;","default_pipe",21); mal> X_16 := bat.new(nil:oid,nil:str); mal> X_24 := bat.append(X_16,"sys.functions"); mal> X_19 := bat.new(nil:oid,nil:str); mal> X_26 := bat.append(X_19,"name"); mal> X_20 := bat.new(nil:oid,nil:str); mal> X_27 := bat.append(X_20,"varchar"); mal> X_21 := bat.new(nil:oid,nil:int); mal> X_29 := bat.append(X_21,256); mal> X_23 := bat.new(nil:oid,nil:int); mal> X_31 := bat.append(X_23,0); mal> X_1 := sql.mvc(); mal> C_2:bat[:oid] := sql.tid(X_1,"sys","functions"); mal> X_5:bat[:str] := sql.bind(X_1,"sys","functions","name",0); mal> (C_8,*r1_8*) := sql.bind(X_1,"sys","functions","name",2); mal> X_11:bat[:str] := sql.bind(X_1,"sys","functions","name",1); mal> X_13 := sql.delta(X_5,C_8,r1_8,X_11); mal> X_14 := algebra.projection(C_2,X_13); mal> sql.resultSet(X_24,X_26,X_27,X_29,X_31,X_14); mal>end user.s4_1; MAPI = (monetdb) /tmp/.s.monetdb.54500 QUERY = end user.s4_1; ERROR = !TypeException:user.s4_1[17]:'sql.delta' undefined in: X_13:any := sql.delta(X_5:bat[:str],C_8:bat[:oid],*r1_8:bat[:any]*,X_11:bat[:str]); mal> The error is easy to find: sql.delta() expects r1_8 to be of type :bat[:str], but it gets :bat[:any] instead. * If I fix it manually: mal>function user.s4_1():void; mal> X_32:void := querylog.define("explain select name from sys.functions;","default_pipe",21); mal> X_16 := bat.new(nil:oid,nil:str); mal> X_24 := bat.append(X_16,"sys.functions"); mal> X_19 := bat.new(nil:oid,nil:str); mal> X_26 := bat.append(X_19,"name"); mal> X_20 := bat.new(nil:oid,nil:str); mal> X_27 := bat.append(X_20,"varchar"); mal> X_21 := bat.new(nil:oid,nil:int); mal> X_29 := bat.append(X_21,256); mal> X_23 := bat.new(nil:oid,nil:int); mal> X_31 := bat.append(X_23,0); mal> X_1 := sql.mvc(); mal> C_2:bat[:oid] := sql.tid(X_1,"sys","functions"); mal> X_5:bat[:str] := sql.bind(X_1,"sys","functions","name",0); mal> (C_8,*r1_8:bat[:str]*) := sql.bind(X_1,"sys","functions","name",2); mal> X_11:bat[:str] := sql.bind(X_1,"sys","functions","name",1); mal> X_13 := sql.delta(X_5,C_8,r1_8,X_11); mal> X_14 := algebra.projection(C_2,X_13); mal> sql.resultSet(X_24,X_26,X_27,X_29,X_31,X_14); mal>end user.s4_1; mal> Yay, it works. ** The reason I write this email:* - I suspect that the output of EXPLAIN is not meant to necessarily be re-parsable, but just to be informative - Nevertheless, it really is almost re-parsable. I tried a much larger query, (2000+ lines of MAL), and this was the only detail. It looks like fixing that small detail could be worth it. - Also, I wonder if this is, in fact, a bug. Binds to access types 0 and 1 do get the right type information. Why not access type 2?

2 2

More info about the MAL plan cache
by Roberto Cornacchia 07 Oct '16

07 Oct '16

Hi there, I have a couple of questions about the plan cache. 1) I know it caches MAL a user-functions for a given "SQL query". What does "SQL query" really means? The unparsed SQL script? The parse tree? The relational algebra tree? The normalized/optimized algebra tree? 2) What are the conditions for the cache to be used? Does it kick in only for some types of queries? I noticed some rather large queries (which take 300-400ms just to be translated from SQL to MAL) are not being cached (or so it seems - how can I check?) 3) Hypothetically speaking, would it be possible to interleave SQL and MAL queries? I am well aware that using MAL directly is not recommended - I only would like to know about feasibility. My typical use case is: START TRANSACTION; a few INSERT INTO "parameters"; SELECT * FROM my_big_view; -- my_big_view uses table "parameters" ROLLBACK; -- values inserted into "parameters" are flushed View "my_big_view" can be a rather large query. It takes an awful lot of time just to get a MAL plan for it. Suppose I have already a MAL plan for it, at client side. I could start a MAL session and run it. I guess though, the main problem is that it wouldn't be in the same transaction as the preceding (sql) INSERT INTO, so it wouldn't get the values inserted into "parameters". Would the only possibility be to move also the INSERT INTO into MAL? So skip SQL altogether? Thanks for you help, Roberto

1 1

slowdown after delete
by Stefano Fioravanzo 03 Oct '16

03 Oct '16

Hello, In my db I have a table with ~300million rows on which I have to perform ~1k aggregation queries periodically. I have noticed that if I delete even a small amount of rows from the table I have a greatly reduced performance compared to when no delete has been made. Specifically, with no delete the 1k queries complete in about 2 hours, if I delete a few records the task completes in about 10-12 hours. Inserting new rows in the table does not present the issue. Moreover if I dump and restore the table after the delete, everything’s back to normal and the queries execute fast. So the issue seems to be related to the fact that monet does not actually delete the records from the bat file, right? Are you aware of the issue? Is there some kind of vacuum operation available? If not, what is the fastest way to dump or copy and restore the table; or is there any other trick that may solve the issue? Any suggestion would be much appreciated, Thank you Stefano

2 1

Select with 5K+ columns kills server
by Pascal Heus 03 Oct '16

03 Oct '16

Hi: I have a MonetDB 5 server v11.23.7 "Jun2016-SP1 instance hosting a table with over 5600 columns (for the US General Social Survey) If we run a simple query like SELECT * FROM gss7214_v4 limit 2 we get the first two records as expected. However, the same query with all the column names explicitly stated like select "year","id","wrkstat","hrs1","hrs2","evwork","occ","prestige","wrkslf",<...thousands of additional columns - see attached...> from "gss7214_v4" WHERE "year"=1972 LIMIT 2 OFFSET 0; seem to kill the server. The process keeps running but become non-responsive. Now this is on a AWS EC2 t2.small instance with just ~2Gb or memory. Maybe more memory may solve this issue (?). But in any case concerned that the server hangs without warning or error message in the log. Any potential workaround? Suggestion most appreciated best *P

3 2

slowdown after delete
by Stefano Fioravanzo 30 Sep '16

30 Sep '16

Hello, In my db I have a table with ~300million rows on which I have to perform ~1k aggregation queries periodically. I have noticed that if I delete even a small amount of rows from the table I have a greatly reduced performance compared to when no delete has been made. Specifically, with no delete the 1k queries complete in about 2 hours, if I delete a few records the task completes in about 10-12 hours. Inserting new rows in the table does not present the issue. Moreover if I dump and restore the table after the delete, everything’s back to normal and the queries execute fast. So the issue seems to be related to the fact that monet does not actually delete the records from the bat file, right? Are you aware of the issue? Is there some kind of vacuum operation available? If not, what is the fastest way to dump or copy and restore the table; or is there any other trick that may solve the issue? Any suggestion would be much appreciated, Thank you Stefano

3 6

Running different MonetDB versions in parallel
by Moritz Bruder 30 Sep '16

30 Sep '16

Hi, so I read several times on the mailing-list, that it is indeed possible to run several MonetDB instances on the same server. I tried this myself with the following configuration: System: Linux ... 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux MonetDB version: mercurial revision 324165f1d055 from branch default (also tested with the topmost version) Configure script parameters: --disable-geom --disable-fits --disable-netcdf --disable-lidar --disable-shp --disable-gsl --disable-rintegration --disable-pyintegration --disable-jdbc --prefix=/home/bruder/monetdb-root --disable-assert --disable-testing --enable-optimize --program-suffix='-dsh' 1. Another version is running with a different user on the default port 50000 2. With my user, I set up a local db-farm, using port 49999. (A prefix is used on binaries, to prevent any conflicts with the existing version.) Creating a new database with *monetdbd* works, but once I try to start it with *monetdb*, it gives me the following error message: starting database 'test'... FAILED start: starting 'test' failed: database 'test' appears to shut itself down after starting, check monetdbd's logfile for possible hints Another thing I tried was starting mserver5 by hand: mserver5-dsh --dbpath=/home/bruder/monetdb-dbfarm/test --set merovingian_uri=mapi:monetdb://dbiron2:49999/test --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/bruder/monetdb-dbfarm/test/.mapi.sock --set monet_vault_key=/home/bruder/monetdb-dbfarm/test/.vaultkey --set gdk_nr_threads=16 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes This time it won't accept my password (I assume the default password is *monetdb*): mclient-dsh -p49999 -dtest -umonetdb InvalidCredentialsException:checkCredentials:invalid credentials for user 'monetdb' I added the log file and the properties of the db-farm. Regards, Moritz

3 5

suggestions
by Yves Dorfsman 30 Sep '16

30 Sep '16

Is there a place to make suggestions on improvement for monetdb? Three things that probably seem minor compared to the goals of monetdb, but which would be very helpful in my opinion: - better copy to csv, a la postgres, with keywords lile "csv" and "header" - ctrl-c to cancel a query but not kill the mclient sesion - history kept and re-used between mclient sessions -- http://yves.zioup.com gpg: 4096R/32B0F416

3 2

Binary bulk load null values
by Dennis Pallett 25 Sep '16

25 Sep '16

Hi all, I'm working on a utility to bulk load data into MonetDB using the binary bulk loading options. At the moment I'm running into the issue whereby I'm unable to insert real NULL values in varchar columns. Only a literal "null" is inserted, which obviously fails the 'WHERE col IS NULL' condition. Can anyone tell me what value is used to import a null value into a varchar column? Best regards, Dennis

3 3

subsort fails silently
by Moritz Bruder 20 Sep '16

20 Sep '16

Hi, I just noticed a bug in the current revision (accc14b26ef3) of the default branch. I added a short mal snippet to reproduce the issue. I guess it's somehow related to sorting a void BAT, but I couldn't reduce the bug to a simple BAT (without the TPCH table). Regards, Moritz

2 1

RAID 0
by Vincent Sheffer 20 Sep '16

20 Sep '16

Are there any MonetDB specific advice/instructions for using a RAID 0 SSD array with MonetDB beyond make sure you have backups? I’m not sure what I’m looking for, other than may be tips on how to configure a dbfarm (i.e. monetdbd/mserver5 properties). Thanks, Vince The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.

2 2

MonetDB Crystal client code in my article
by Brian Hood 18 Sep '16

18 Sep '16

Hi Guys, I've included a very basic MonetDB client code in the Crystal Language that actually connects to mserver5 performs a query and exits just to demonstrate how easy that is in Crystal to use a C binding but also for creating some more interest in MonetDB. https://github.com/puppetpies/Snippets/blob/master/Crystal%20-%20Ruby%20dyn… I will be publishing it on Medium / Slideshare next week. Regards, Brian Hood

1 0

COPY FROM support compressed files?
by Yves Dorfsman 16 Sep '16

16 Sep '16

Can I COPY FROM directly from compressed files? If so, what format (tried with zstd files, didn't work, but that's very new so not surprised)? Thanks. -- http://yves.zioup.com gpg: 4096R/32B0F416

3 2

Does R run in a sandbox in MonetDB?
by Pablo S 08 Sep '16

08 Sep '16

I'm refering to the MonetDB feature that allows to create R functions that run embedded in the database. For example, you can do: CREATE FUNCTION rapi00() RETURNS TABLE (d INTEGER) LANGUAGE R { seq(1,10); }; Now, I need my users to create dynamically these user defined functions by letting them enter these on the UI; the application will create the function in the database and a process will run them. Problem is that R allows to run operating system level commands, such as list.files(path=".") or even file.remove("some_file") Will MonetDB allow these external interactions with the operating system, or it runs in a sandbox?

2 1