- users-list - monetdb.org

update on tqgen
by Sahasranaman M S 07 Sep '16

07 Sep '16

Hi all, Just an update on *tqgen*, in case anyone is interested - The latest master in github.com/niocs/tqgen now runs around 8 times faster than the earlier version. Also, in the OutFilePat parameter, if there is no YYYYMMDD, then all data goes to a single file, unlike earlier when the file kept getting overwritten. The documentation on the gh page has also been updated. If there is anything that can be done to make it more useful, please let me know and I will try to add it. Thanks, Sahas On Thu, Jul 14, 2016 at 9:29 PM, Sahasranaman M S <sahasr(a)naman.ms> wrote: > Actually, I just remembered tqgen has a couple of other package > dependencies from github, so the best way to download and build it is to > use this command: > > $ go get github.com/niocs/tqgen > $ cd $GOPATH/bin > $ ./tqgen --OutFilePat /path/to/data/tq.YYYYMMDD.csv --NumStk 1000 > --Seed 234532 --DateBeg 20200101 --DateEnd 20210831 > > Thanks, > Sahas > > On Thu, Jul 14, 2016 at 9:18 PM, Sahasranaman M S <sahasr(a)naman.ms> wrote: > >> Hello, >> >> Just a suggestion, if you think the issue is related to the size of the >> data, then you could use a random data generator to test it. We were >> facing a similar issue with shareable data for testing. So I wrote a >> program called tqgen in Golang (https://github.com/niocs/tqgen) >> >> It can be used to generate large amounts of csv data for testing, and to >> share the data, we just need to share the command we used to generate it, >> and the other person can run it to generate the exact same data. >> >> It produces datewise fictional trade/quote data for fictional stocks. It >> puts each date's data into a separate file, and each file is around 128MB, >> so if we run it for a daterange of 1.5 years, it should generate around >> 70GB of data. It takes a while to finish because the data is not >> completely random and it is related to other rows. >> >> To generate, you could run: >> $ git clone https://github.com/niocs/tqgen >> $ cd tqgen >> $ go build tqgen.go >> $ ./tqgen --OutFilePat /path/to/data/tq.YYYYMMDD.csv --NumStk 1000 >> --Seed 234532 --DateBeg 20200101 --DateEnd 20210831 >> >> I usually load them one by one so that I could record load time stats, >> etc, but you could use this bash script (https://github.com/niocs/ >> hcsv/blob/master/hcat) to concatenate them without repeating the >> headers, and then load to MonetDB. (I will later add a parameter to tqgen >> to save to a single file) >> >> To do that, >> $ git clone https://github.com/niocs/hcsv >> $ hcsv/hcat /path/to/data/tq.20*.csv > /path/to/data/tq.big.csv >> >> If this causes the same issue, then others could regenerate the same >> data to try. Hope this helps. >> >> Thanks, >> Sahas >> >> >> >>> On Jul 14, 2016 10:20 AM, "Anthony Damico" <ajdamico(a)gmail.com> wrote: >>> >>>> can you winnow down the problem enough to construct a reproducible >>>> example? thanks >>>> >>>> On Thursday, July 14, 2016, Stefano Fioravanzo <fioravanzos(a)gmail.com> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> No the data is not public, and testing in on another machine is not an >>>>> easy option right now. >>>>> >>>>> Regards, >>>>> >>>>> Stefano >>>>> >>>>> On 14 Jul 2016, at 15:49, Anthony Damico <ajdamico(a)gmail.com> wrote: >>>>> >>>>> hi, any chance these data are public? if not, can you test on a >>>>> separate box and see if the exact same thing happens or if it's >>>>> hardware-specific? >>>>> >>>>> On Thursday, July 14, 2016, Stefano Fioravanzo <fioravanzos(a)gmail.com> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I currently have a table with 307524407 rows and 52 columns, it was >>>>>> loaded from a 78GB cvs file, but I do not know how much space the table >>>>>> actually takes on disk. >>>>>> If I query the table with a simple query like: >>>>>> >>>>>> select count(*) from big_table where field =1; >>>>>> >>>>>> mserver5 crashes with ‘segmentation fault' >>>>>> I tried also the following query (adding the LIMIT statement): >>>>>> select count(*) from farmap_movimento where farmacia_id =1 >>>>>> limit 10; >>>>>> >>>>>> and the output from mserver5 was: >>>>>> mserver5: gdk_select.c:869: fullscan_int: Assertion `cnt < >>>>>> (bn)->S->capacity' failed. >>>>>> Aborted >>>>>> >>>>>> Doing a more complex query (the one I am aiming for), with also a >>>>>> group by and a few aggregation functions (sum, min) on the fields, the >>>>>> outputs is: >>>>>> mserver5: gdk_bat.c:2066: BATsetcount: Assertion >>>>>> `b->S->capacity >= cnt' failed. >>>>>> Aborted >>>>>> >>>>>> This seems to be a table size problem, so…is there anything I can do? >>>>>> >>>>>> I am using MonetDB 5 server v11.22.0 >>>>>> The machine has 30GB of RAM and 60GB of swap. >>>>>> Available disk space: ~50GB >>>>>> >>>>>> Any help would be much appreciated, >>>>>> >>>>>> Regards, >>>>>> >>>>>> Stefano >>>>>> >>>>>> _______________________________________________ >>>>>> users-list mailing list >>>>>> users-list(a)monetdb.org >>>>>> https://www.monetdb.org/mailman/listinfo/users-list >>>>>> >>>>> _______________________________________________ >>>>> users-list mailing list >>>>> users-list(a)monetdb.org >>>>> https://www.monetdb.org/mailman/listinfo/users-list >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> users-list mailing list >>>> users-list(a)monetdb.org >>>> https://www.monetdb.org/mailman/listinfo/users-list >>>> >>>> >> >

1 0

Merovingian: allow binds to given ip/interface
by Robert Koch 06 Sep '16

06 Sep '16

Referring to comment #13 of this bug report, I would like to know where exactly I can config listening to 0.0.0.0 instead of 127.0.0.1. As of Jun2016 relaease I cannot connect to Port 50000 from foreign hosts any more. https://www.monetdb.org/bugzilla/show_bug.cgi?id=2407#c13 Thank you Robert

2 1

loopback queries in embedded R
by Stefano Fioravanzo 05 Sep '16

05 Sep '16

Hello, A few months ago I was told that loopback queries for embedded R were not ready yet, but would be after some time. Is the feature ready? If so, how does it work? (I am familiar with loopback queries with embedded python) I am currently using the development default branch, compiled a couple of weeks ago. Thank you, Stefano

2 2

error while entering a newly created database
by Xu,Wenjian 01 Sep '16

01 Sep '16

Hi all, I have downloaded the latest version MonetDB-11.23.7 and built it from source. But I have encountered errors while entering a newly created database: =========================================================== $ monetdbd start ~/monetdb_dbfarm2 $ monetdb create topk_synthetic $ monetdb release topk_synthetic $ mclient -dtopk_synthetic *CREATE FUNCTION: name 'md5' (clob) already in use* *ParseException:SQLparser:CREATE FUNCTION: name 'md5' (clob) already in use* =========================================================== Logs are as follows: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2016-09-01 09:45:21 MSG merovingian[30268]: starting database 'topk_synthetic', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) 2016-09-01 09:45:22 MSG topk_synthetic[30287]: arguments: /usr/local/bin/mserver5 --dbpath=/home/zeroxwj/monetdb_dbfarm2/topk_synthetic --set merovingian_uri=mapi:monetdb://SourceCode:50000/topk_synthetic --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/zeroxwj/monetdb_dbfarm2/topk_synthetic/.mapi.sock --set monet_vault_key=/home/zeroxwj/monetdb_dbfarm2/topk_synthetic/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # MonetDB 5 server v11.23.7 "Jun2016-SP1" 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # Serving database 'topk_synthetic', using 8 threads 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # Found 15.625 GiB available main-memory. 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # Copyright (c) 1993-July 2008 CWI. 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # Visit http://www.monetdb.org/ for further information 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # Listening for UNIX domain connection requests on mapi:monetdb:///home/zeroxwj/monetdb_dbfarm2/topk_synthetic/.mapi.sock 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # MonetDB/SQL module loaded 2016-09-01 09:45:22 MSG merovingian[30268]: proxying client (local) for database 'topk_synthetic' to mapi:monetdb:///home/zeroxwj/monetdb_dbfarm2/topk_synthetic/.mapi.sock?database=topk_synthetic 2016-09-01 09:45:22 MSG merovingian[30268]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # SQL catalog created, loading sql scripts once 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 09_like.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 10_math.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 11_times.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 12_url.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 13_date.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 14_inet.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 15_querylog.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 16_tracelog.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 17_temporal.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 19_cluster.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 20_vacuum.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 21_dependency_functions.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 22_clients.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 23_skyserver.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 24_zorder.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 25_debug.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 26_sysmon.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 27_rejects.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 39_analytics.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 39_analytics_hge.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 40_json.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 40_json_hge.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 41_jsonstore.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 41_md5sum.sql 2016-09-01 09:45:22 MSG merovingian[30268]: proxying client localhost:54853 for database 'topk_synthetic' to mapi:monetdb:///home/zeroxwj/monetdb_dbfarm2/topk_synthetic/.mapi.sock?database=topk_synthetic 2016-09-01 09:45:22 MSG merovingian[30268]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # SQL catalog created, loading sql scripts once 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 09_like.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 10_math.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 11_times.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 12_url.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 13_date.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 14_inet.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 15_querylog.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 16_tracelog.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 17_temporal.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 19_cluster.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 20_vacuum.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 21_dependency_functions.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 22_clients.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 23_skyserver.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 24_zorder.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 25_debug.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 26_sysmon.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 27_rejects.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 39_analytics.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 39_analytics_hge.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 40_json.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 40_json_hge.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 41_jsonstore.sql 2016-09-01 09:45:22 MSG topk_synthetic[30287]: # loading sql script: 41_md5sum.sql >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As far as I understand, those sql scripts (e.g., 09_like.sql) are for initialization purpose. What is the reason that it loads those scripts twice? BTW, I can start an existing database successfully. I have never encountered such errors before. Any suggestion would be helpful. Thanks. Best regards, Wenjian

1 0

cannot access db with mserver5
by Stefano Fioravanzo 31 Aug '16

31 Aug '16

Hello, I have been using monetdb starting my db farm with monetdbd and it has always worked fine. I have tried to start manually mserver5 with the command mserver5 --set embedded_py=true --dbpath=/path/to/db --daemon=yes —set mapi_open=true The process starts correctly, but when I try to launch mclient it says InvalidCredentialsException:checkCredentials:invalid credentials for user ‘monetdb’, what can I do? I want to start the database with mserver5 because I need remote access (—set mapi_open=true), is there a way to enable this flag using monetdbd? Thanks, Stefano

3 3

GEOM Module installed in Sys Schema
by Vincent Sheffer 30 Aug '16

30 Aug '16

It seems that the functions of the Geom module is only accessible from the sys schema. That further implies that only users with access to the sys schema can call Geom functions. What is the best way to ensure that all users can access the functions from the Geom module? Geom types seem to work fine. Thanks, Vince The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.

2 1

segfault remote table count: could not find L1.L1 NOT NULL as L1 L1.L1
by Anton Kravchenko 25 Aug '16

25 Aug '16

Hi there, when I run 'count' on remote table I get a segfault on worker server: 'could not find L1.L1 NOT NULL as L1 L1.L1 Segmentation fault' Here is master(server/client) and worker (server/client), all are living on localhost (as it was shown in example https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/DistributedQuery… ). master client [akravchenko@hnode1 ~]$ mclient -d mst Welcome to mclient, the MonetDB/SQL interactive terminal (Jun2016-SP1) Database: MonetDB v11.23.7 (Jun2016-SP1), 'mst' Type \q to quit, \? for a list of available commands auto commit mode: on sql>CREATE REMOTE TABLE t1 (i int) on 'mapi:monetdb://localhost:50001/rt1'; operation successful (1.357ms) sql>\d REMOTE TABLE sys.t1 sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (16.205ms) *sql>select count(*) from t1;* *an error occurred on connection: Connection lost* *sql>* worker client [akravchenko@hnode1 ~]$ mclient -d rt1 -p 50001 Welcome to mclient, the MonetDB/SQL interactive terminal (Jun2016-SP1) Database: MonetDB v11.23.7 (Jun2016-SP1), 'rt1' Type \q to quit, \? for a list of available commands auto commit mode: on sql>\d TABLE sys.t1 sql>select count(*) from t1; +------+ | L1 | +======+ | 2 | +------+ 1 tuple (0.594ms) worker server [akravchenko@hnode1 ~]$ mserver5 --dbpath=rt1 --set mapi_port=50001 # MonetDB 5 server v11.23.7 "Jun2016-SP1" # Serving database 'rt1', using 4 threads # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked # Found 31.357 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/ # MonetDB/SQL module loaded *>could not find L1.L1 NOT NULL as L1* *L1.L1* *Segmentation fault* master server [akravchenko@hnode1 ~]$ mserver5 --dbpath=mst # MonetDB 5 server v11.23.7 "Jun2016-SP1" # Serving database 'mst', using 4 threads # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked # Found 31.357 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module loaded Thank you, Anton

2 3

Is it possible to access two databases in a single select?
by Pablo S 25 Aug '16

25 Aug '16

Is this possible in MonetDB? select * from database1.table1 t1, database2.table2 t2 where t1.col1 = t2.col2 where database1 and database2 are two databases running simultaneously.

2 1

Concurrent query execution issue
by Stefano Fioravanzo 24 Aug '16

24 Aug '16

Hello, I am trying to execute in parallel a few queries in MonetDB. I have read in the documentation that Monet uses the OCC transaction model so read operations should not block the table. I have 3 CREATE TABLE queries that read from the same source table, I launch 3 different mclient processes and launch one query on each client. The queries execute in parallel as expected, I can see them running in sys.queue(). The first query that completes, creates the table and commits fine. When the other two queries finish their job I can see the message “operation successful” but right after the client prints "COMMIT: transaction is aborted because of concurrency conflicts, will ROLLBACK instead”. What am I doing wrong? Apart from the newly created table, I did non write anything to the db so it should work fine. Any suggestion would be much appreciated, Thanks, Stefano

2 2

Run two MonetDB databases simoultaneously in Windows
by Pablo S 21 Aug '16

21 Aug '16

I'm trying to run two MonetDB databases simoultaneously in Windows. I named the two databases mydb and mydb2. I changed the database name in M5server.bat to mydb, and added -database=mydb to mclient.bat Then I copied the M5server.bat file and changed the farm to mydb2. I also copied the mclient.bat file and added -database=mydb2. I run the two server bats to start the two databases and that seems to work fine (no errors). Then I run the first mclient.bat and it connects to the server with no issues. But when I run the copied mclient.bat and try to connect using user/password monetdb/monetdb, I get the following error and cannot connect: request for database 'mydb2', but this is database 'mydb', did you mean to connect to monetdbd instead? How to run two databases simoultaneously in Windows? How to connect to two databases at the same time?

3 2

Program contains errors
by Stefano Fioravanzo 19 Aug '16

19 Aug '16

Hello, I will briefly recap the problem I have already reported a few weeks ago: I am using monetdb default branch (I did not compile the stable version because I need embedded python) and I have a large table (~80 GB). A simple query like SELECT * FROM large_table WHERE field=x, fails. mserver5 reports this error: mserver5: gdk_select.c:867: fullscan_int: Assertion `cnt < (bn)->S.capacity' failed. It seems that monet thinks it doesn’t have enough space to do a full scan of the table, maybe? (The machine has 800GB of free disk space and 30GB of RAM - Ubuntu 14.04). I was using a default branch of december 2015, I have tried to use the latest default branch (4/08/16), recompiled, installed but nothing, always the same error. So, as suggested, I have compiled the latest stable release (Jun2016-SP1). Now even a query which does not require monet to read the whole table (like SELECT * FROM large_table LIMIT 10 - This query worked fine in the default branch version) fails with ‘Program contains error’ message by client. Selecting a single column of the table works just fine. Moreover, with the stable release, the same issue arises with the second biggest table of the database (~35 GB). [Here you can find the configuration of MonetDB: http://pastie.org/private/r8gtssg944aqwnigrysa] All queries on all the other tables in the database (they are all much smaller than these two) work fine. I do not understand if the issue is in the size of table or somewhere else. How can I know what ‘error’ the message ‘program contains errors’ is referring to? NOTE: At the beginning, using the default compiled branch everything worked just fine! Every query on my large_table worked as expected. Then I had to recreate the database from scratch so I deleted the dbfarm and created a new one. That’s when this headache started… At this point I don’t know what to do, any help would be MUCH appreciated. Thanks, Stefano

4 7

synchronize data from mysql performance
by wang.tong＠shomop.com 19 Aug '16

19 Aug '16

Hi, I try to synchronize data from mysql to monetdb. Steps: 1. Read the modified rows from mysql table periodically. 2. Delete the rows in monetdb by primary key. (To avoid 'PRIMARY KEY constraint violated' error.) The sql is like 'delete from a where id in (1, 2, 3)'. 3. Insert the rows into monetdb by 'copy into ... from stdin'. I find step 2 may be too slow. When there are 2 million rows, delete 10,000 rows takes about 1 minute. When there are 30 million rows, delete 10,000 rows takes about 20 minute. I have to do join on the 30 million rows table, so I cannot split it. I try to upgrade the server from 2 core 8 G memory to 8 core 32 G memory, but it has no effect. It seems the limitation is CPU, and the deleted operation can only use 1 core. How can I optimize the performance? Is there any better approach to synchronize data? Thanks in advance! OS version: centos-release-7-0.1406.el7.centos.2.5.x86_64 mserver5 --version MonetDB 5 server v11.23.7 "Jun2016-SP1" (64-bit, 64-bit oids, 128-bit integers) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 31.0GiB available memory, 8 available cpu cores Libraries: libpcre: 8.32 2012-11-30 (compiled with 8.32) openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL 1.0.1e-fips 11 Feb 2013) libxml2: 2.9.1 (compiled with 2.9.1) Compiled by: root@jstu87d9w6zj4u (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 wang.tong(a)shomop.com

2 2

database was killed by signal SIGSEGV
by 王彤 19 Aug '16

19 Aug '16

Hi, I try to synchronize data from mysql to monetdb. Total rows is about 15 million. I insert the rows in batch. Each batch has about 10000 rows. I use 'copy into from stdin' as in http://dev.monetdb.org/hg/MonetDB/file/tip/java/example/SQLcopyinto.java . The virtual memory continuously grows. When it grows to about 80G, the server is killed by signal SIGSEGV. The database size on disk is about 30GB. Below is the server infomation and error log. Any help would be greatly appreciated! OS version: centos-release-7-0.1406.el7.centos.2.5.x86_64 mserver5 --version MonetDB 5 server v11.23.7 "Jun2016-SP1" (64-bit, 64-bit oids, 128-bit integers) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 7.4GiB available memory, 2 available cpu cores Libraries: libpcre: 8.32 2012-11-30 (compiled with 8.32) openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL 1.0.1e-fips 11 Feb 2013) libxml2: 2.9.1 (compiled with 2.9.1) Compiled by: root@jstu87d9w6zj4u (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 in /data/dbfarm/merovingian.log 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) fails, try to free up space [memory in use=98280928,virtual memory in use=3779503584] 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) result [mem=98280928,vm=3779503584] 2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.173.87.201:46953 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm 2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.153.193.249:55915 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm 2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) fails, try to free up space [memory in use=185008352,virtual memory in use=3840344288] 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) result [mem=185008352,vm=3840344288] 2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.173.87.201:46963 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm 2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(51838976) fails, try to free up space [memory in use=99455712,virtual memory in use=3766391520] 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(51838976) result [mem=99455712,vm=3766391520] 2016-08-18 17:29:04 MSG merovingian[10194]: database 'shomopcrm' (10246) was killed by signal SIGSEGV 2016-08-18 17:29:04 MSG merovingian[10194]: starting database 'shomopcrm', up min/avg/max: 17s/15h/5d, crash average: 1.00 0.90 0.40 (23-11=12) 2016-08-18 17:29:05 MSG shomopcrm[984]: arguments: /usr/local/bin/mserver5 --dbpath=/data/dbfarm/shomopcrm --set merovingian_uri=mapi:monetdb://jstu87d9w6zj4u:30001/shomopcrm --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/dbfarm/shomopcrm/.mapi.sock --set monet_vault_key=/data/dbfarm/shomopcrm/.vaultkey --set gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-08-18 17:29:05 MSG shomopcrm[984]: # MonetDB 5 server v11.23.7 "Jun2016-SP1" 2016-08-18 17:29:05 MSG shomopcrm[984]: # Serving database 'shomopcrm', using 2 threads 2016-08-18 17:29:05 MSG shomopcrm[984]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-08-18 17:29:05 MSG shomopcrm[984]: # Found 7.390 GiB available main-memory. 2016-08-18 17:29:05 MSG shomopcrm[984]: # Copyright (c) 1993-July 2008 CWI. 2016-08-18 17:29:05 MSG shomopcrm[984]: # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved 2016-08-18 17:29:05 MSG shomopcrm[984]: # Visit http://www.monetdb.org/ for further information 2016-08-18 17:29:05 MSG shomopcrm[984]: # Listening for UNIX domain connection requests on mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock 2016-08-18 17:29:05 MSG shomopcrm[984]: # MonetDB/SQL module loaded in /var/log/messages, there is an error: kernel: mserver5[981]: segfault at 18 ip 00007fe10f845a89 sp 00007fcf38e79760 error 4 in lib_sql.so[7fe10f817000+175000] wang.tong(a)shomop.com

2 1

Distributed Query Processing questions
by Anton Kravchenko 18 Aug '16

18 Aug '16

Hi there, I have few questions on merge/remote tables. Are there known MonetDB users that use(d) merge/remote tables in production? Are there official MonetDB performance benchmarks for merge/remote vs local tables? Thank you, Anton

2 1

distributed query processing example
by Anton Kravchenko 16 Aug '16

16 Aug '16

Hi there, I am trying to follow a distributed query processing example from https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/DistributedQuery… and here is my experience: Welcome to mclient, the MonetDB/SQL interactive terminal (Jun2016-SP1) Database: MonetDB v11.23.7 (Jun2016-SP1), 'mapi:monetdb://hnode3.secdev.local:52018/db3' Type \q to quit, \? for a list of available commands auto commit mode: on sql>CREATE MERGE TABLE mt1 (t int); operation successful (2.367ms) sql>CREATE REMOTE TABLE t1 (i int) on 'mapi:monetdb://hnode1.secdev.local:52016/db1'; operation successful (1.287ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (4.694ms) *sql>select count(*) from t1;* *Connection lost* *sql>select count(*) from t1;* *illegal input, not a JSON header (got '')* *sql>select count(*) from t1;* *Connection lost* *sql>select count(*) from t1;* *Connection lost* *sql>select count(*) from t1;* *illegal input, not a JSON header (got '')* *sql>select count(*) from t1;* *illegal input, not a JSON header (got '')* *sql>select count(*) from t1;* *an error occurred on connection: Connection lost* *sql>select count(*) from t1;* *Connection lost* sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (508.682ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (6.227ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (5.065ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (4.178ms) sql> P.S. Is there something wrong with my monetdb cluster setup or is it a real bug? On hnode3 (CentOS 6.8 machine): [akravchenko@hnode3 ~]$ monetdbd -v MonetDB Database Server v1.7 (Jun2016-SP1) [akravchenko@hnode3 ~]$ monetdb discover location hnode3.secdev.local:52018* mapi:monetdb://hnode3.secdev.local:52018/db3 [akravchenko@hnode3 ~]$ monetdb status name state health remarks db3 R 20h 100% 11m mapi:monetdb://hnode3.secdev.local:52018/db3 [akravchenko@hnode3 ~]$ monetdbd get all hnode3-dbfarm property value hostname hnode3.secdev.local dbfarm hnode3-dbfarm status monetdbd[18052] 1.7 (Jun2016-SP1) is serving this dbfarm mserver /usr/bin/mserver5 logfile hnode3-dbfarm/merovingian.log pidfile hnode3-dbfarm/merovingian.pid sockdir /tmp listenaddr 0.0.0.0 port 52018 exittimeout 60 forward proxy discovery yes discoveryttl 600 control yes passphrase {SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35 mapisock /tmp/.s.monetdb.52018 controlsock /tmp/.s.merovingian.52018 and on hnode1 (CentOS 6.8 machine): [akravchenko@hnode1 ~]$ monetdb -v MonetDB Database Server Toolkit v1.1 (Jun2016-SP1) [akravchenko@hnode1 ~]$ monetdb discover location hnode1.secdev.local:52016* mapi:monetdb://hnode1.secdev.local:52016/db1 [akravchenko@hnode1 ~]$ monetdb status name state health remarks db1 R 4m 15% 32m mapi:monetdb://hnode1.secdev.local:52016/db1 [akravchenko@hnode1 ~]$ monetdbd get all hnode1-dbfarm/ property value hostname hnode1.secdev.local dbfarm hnode1-dbfarm/ status monetdbd[1355] 1.7 (Jun2016-SP1) is serving this dbfarm mserver /usr/bin/mserver5 logfile hnode1-dbfarm//merovingian.log pidfile hnode1-dbfarm//merovingian.pid sockdir /tmp listenaddr 0.0.0.0 port 52016 exittimeout 60 forward proxy discovery yes discoveryttl 600 control yes passphrase {SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35 mapisock /tmp/.s.monetdb.52016 controlsock /tmp/.s.merovingian.52016 [akravchenko@hnode1 ~]$ Thank you, Anton

1 0

Add a new optimizer pipeline.
by Xu,Wenjian 10 Aug '16

10 Aug '16

Hi all, MonetDB version: 11.23.7 Ubuntu version: 14.04 I created a new optimizer pipeline by adding an additional optimization step after 'default_pipe', i.e.: '*my_customized_pipe*'='optimizer.inline();optimizer.candidates(); optimizer.remap();optimizer.costModel();optimizer.coercions(); optimizer.evaluate();optimizer.pushselect();optimizer.aliases(); optimizer.mitosis();optimizer.mergetable();optimizer.deadcode(); optimizer.aliases();optimizer.constants();optimizer.commonTerms(); optimizer.projectionpath();optimizer.deadcode();optimizer.reorder(); optimizer.reduce();optimizer.matpack();optimizer.dataflow(); optimizer.querylog();optimizer.multiplex();optimizer.generator(); optimizer.profiler();optimizer.garbageCollector(); *optimizer.customizedstep()*' What I have done is: 1) Create an empty optimizer step: -- 1.1 add *opt_customizedstep.c/h* in package *monetdb/optimizer/**, leave function body *OPTcustomizedstepImplementation() *empty*, *i.e. with one statement **return 0** * -- *1.2 register the step by modifying *opt_wrapper.c *and* optimizer.mal* 2) Add the definition of the new pipeline in *opt_pipes.c* 3) Modify the configuration file to compile the added source files. After re-build the system, I can enable the new optimizer pipeline by *set optimizer='my_customized_pipe'*. However, I have two issues: i) using EXPLAIN statement, I found the MAL plan different from the one generated by 'default_pipe' (which should be the same since I did nothing in the added step?). Specifically, the plan seems to be not affected by *optimizer.reorder() *step, i.e., bat.new() and bat.append() instructions appear in the end of the plan. ii) then I entered mdb mode by DEBUG statement. But it seems that some command did not work as before. For example, *help* command does not display all commands available; *optimizer* command does not display actions performed by each optimizer step. What's wrong with my procedure to add a new optimizer pipeline? Any comment would be helpful. Best regards, Wenjian

3 3

[Low level] Understanding duplicate entries in delta BAT (delete BAT)
by Frédéric Jolliton 08 Aug '16

08 Aug '16

Hi, As per MonetDB Solution suggestion, we are asking here about this topic. We are studying how MonetDB store its database on disk. So far we were able to understand pretty much all the files (BAT, journal, BBP.dir, heap, hash, imprints, and so on). We have custom tools to decode and dump them (experimental at this point). There is one thing however that is unclear: we thought that "delta BAT", which are the BAT assigned to an SQL tables to list the OIDs of rows that were deleted (named "D_<schema>_<table>" internaly), would contains only *uniques* values (since you can never delete twice the same row). But on several databases that we are running, we found that some of theses BATs contain duplicates. Lot of duplicates actually. Especially on D_sys__columns and D_sys__tables BATs (respectively for the sys._columns and sys._tables system tables). Some databases do not have this "issue" (while they all have the same schema, and process the same kind of data as the other ones). Can someone explain if duplicates are expected in theses BATs ? Here is an excerpt of D_sys__tables: # hexdump -C 02/210.tail 00000000 2e 00 00 00 00 00 00 00 2f 00 00 00 00 00 00 00 |......../.......| 00000010 30 00 00 00 00 00 00 00 31 00 00 00 00 00 00 00 |0.......1.......| 00000020 32 00 00 00 00 00 00 00 33 00 00 00 00 00 00 00 |2.......3.......| 00000030 34 00 00 00 00 00 00 00 35 00 00 00 00 00 00 00 |4.......5.......| 00000040 36 00 00 00 00 00 00 00 37 00 00 00 00 00 00 00 |6.......7.......| 00000050 42 00 00 00 00 00 00 00 43 00 00 00 00 00 00 00 |B.......C.......| 00000060 44 00 00 00 00 00 00 00 45 00 00 00 00 00 00 00 |D.......E.......| 00000070 46 00 00 00 00 00 00 00 47 00 00 00 00 00 00 00 |F.......G.......| 00000080 48 00 00 00 00 00 00 00 49 00 00 00 00 00 00 00 |H.......I.......| 00000090 4a 00 00 00 00 00 00 00 4b 00 00 00 00 00 00 00 |J.......K.......| 000000a0 4c 00 00 00 00 00 00 00 4d 00 00 00 00 00 00 00 |L.......M.......| 000000b0 4e 00 00 00 00 00 00 00 4f 00 00 00 00 00 00 00 |N.......O.......| 000000c0 50 00 00 00 00 00 00 00 50 00 00 00 00 00 00 00 |P.......P.......| 000000d0 51 00 00 00 00 00 00 00 50 00 00 00 00 00 00 00 |Q.......P.......| 000000e0 51 00 00 00 00 00 00 00 52 00 00 00 00 00 00 00 |Q.......R.......| 000000f0 50 00 00 00 00 00 00 00 51 00 00 00 00 00 00 00 |P.......Q.......| 00000100 52 00 00 00 00 00 00 00 53 00 00 00 00 00 00 00 |R.......S.......| 00000110 50 00 00 00 00 00 00 00 51 00 00 00 00 00 00 00 |P.......Q.......| 00000120 52 00 00 00 00 00 00 00 53 00 00 00 00 00 00 00 |R.......S.......| ... You can clearly see duplicates OID (the first one being 0x50 at 0xc8). Its entry in BBP.dir (split into sections): 136 32 tmp_210 tmpr_210 02/210 610523782 2 171807 0 0 171807 172032 0 0 0 0 void 0 1 1793 0 0 0 0 0 1000651 0 0 0 oid 8 0 1024 24 25 27 1 46 773603235 1374456 1376256 1 We are using: MonetDB 5 server v11.21.14 (64-bit, 64-bit oids, 128-bit integers). -- Frédéric Jolliton Sécuractive

4 3

Does R run in a sandbox in MonetDB/R?
by Pablo S 07 Aug '16

07 Aug '16

MonetDB allows to create R functions that run embedded in the database. For example, you can do CREATE FUNCTION rapi00() RETURNS TABLE (d INTEGER) LANGUAGE R { seq(1,10); }; Now, I need my users to create dynamically these user defined functions by entering R statements on a screen; the application will create the functions in the database and run them. Problem is that R allows to run operating system level commands, such as list.files(path=".") or even file.remove("some_file") Will MonetDB allow these external interactions with the operating system, or it runs in a sandbox?

1 0

How could I obtain the running time information in the MAL interface of mclient?
by Xu,Wenjian 07 Aug '16

07 Aug '16

Hi all, MonetDB version: 11.23.7 Ubuntu version: 14.04 How could I obtain the running time information in the MAL interface of mclient? I tried to use the *--interactive* option but it did not work. Specifically, I do the following: =============================================== $ *mclient -dnba2009 -lmsql --interactive=ms* Welcome to mclient, the MonetDB interactive terminal (Jun2016-SP1) Type \q to quit, \? for a list of available commands msql>\<query.txt // I save the MAL instructions of a query (obtained from EXPLAIN statement) in this file msql>user.s1_3("1998"); // call the function of that query % sys.player, sys.player, sys.player # table_name % p_lastname, p_lastseason, p_weight # name % char, int, int # type % 100, 4, 3 # length [ "Wheeler", 1998, 165 ] [ "Baker", 1998, 175 ] [ "Garnett", 1998, 186 ] ...... ======================================================== I would like to obtain the running time information after getting the result of that query. How can I do it? Any help would be appreciated. Best, Wenjian

4 6

Strange behaviour with optimizer.pushselect and batalgebra.like
by Moritz Bruder 05 Aug '16

05 Aug '16

Hi, I get a really strange behaviour with the *pushselect* optimizer for this piece of MAL code: http://codepad.org/bqulPEtV . The problem I'm having is, that depending on whether the *pushselect* optimizer is used or not, the query will yield different results. Context info: 1. MonetDB version number: 11.23.3 2. Operating system platform: Linux 4.6.4-1-ARCH #1 SMP PREEMPT Mon Jul 11 19:12:32 CEST 2016 x86_64 GNU/Linux 3. Compiled from source tarball 4. http://codepad.org/bqulPEtV Regards, Moritz

1 0

Using MonetDB with Scala/Slick
by Pablo S 05 Aug '16

05 Aug '16

I develop in Scala, and need to access a MonetDB database. I use the Scala standard, Slick, however Slick doesn't have a driver to access MonetDB. I'm interested in developing a Slick/MonetDB driver (it's a JDBC wrapper) but I'd like to know first if someone already started the process (we could join forces). Please let me know. Thanks.

1 0

When would the function group.multicolumn() used?
by Xu,Wenjian 03 Aug '16

03 Aug '16

Hi all, I found the following new MAL function in the new release of version 11.23.3: ============================================================ pattern group.multicolumn(b:bat[:any]...) (ref:bat[:oid],grp:bat[:oid],hist:bat[:any]) address GROUPmulticolumngroup; comment Derivation of a group index over multiple columns. ============================================================ I thought I may be used by queries with multiple attributes in the GROUP BY clause. So I tried to EXPLAIN such a query (e.g., SELECT ... FROM... GROUP BY ColA, ColB), but it is still compiled with multiple group.subgroup() followed by a group.subgroupdone(). So, when would this function group.multicolumn() used? Thanks, Wenjian

1 0

a few good stackoverflow qs without answers..
by Anthony Damico 30 Jul '16

30 Jul '16

http://stackoverflow.com/questions/tagged/monetdb?sort=newest&pageSize=50

1 0

infinity value
by Stefano Fioravanzo 26 Jul '16

26 Jul '16

Hello, I am creating a table from the result of a python function. One column is composed of double values, some of them are np.inf and are translated in monet as inf I cannot find a way to query for these values, how does monetdb handle infinity? Moreover, if I later do some query on this table and apply an arithmetic operation to the inf values, monetdb throws ‘overflow in calculation’ error, I would expect any operation (except for special cases) to result in inf. Is this a known issue or an intended design? Regards, Stefano

1 0

mapi.class.php
by Alberto Ferrari 26 Jul '16

26 Jul '16

For those who are interested, here is (attached) the "php_mapi.inc" which I changed to a php class (OOP) Feel free to use or even upload to any repository. Regards.

2 1

Announcement: New Jun2016-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 22 Jul '16

22 Jul '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2016-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jun 2016-SP1 bugfix release . * Lots of memory leaks have been plugged across the whole system. Java Module * Corrected PROCEDURE_TYPE output value of method DatabaseMetaData.getProcedures(). It used to return procedureReturnsResult. Now it returns procedureNoResult. Corrected ORDINAL_POSITION output value of method DatabaseMetaData.getProcedureColumns(). It used to start with 0, but as procedures do not return a result value it now starts with 1 for all the procedure arguments, as defined by the JDBC API. * Improved output of method DatabaseMetaData.getProcedures(). The REMARKS column now contains the procedure definition as stored in sys.functions.func. The SPECIFIC_NAME column now contains the procedure unique identifier as stored in sys.functions.id. This allows the caller to retrieve the specific overloaded procedure which has the same name, but different arguments. Also improved output of method DatabaseMetaData.getProcedureColumns(). The SPECIFIC_NAME column now contains the procedure unique identifier as stored in sys.functions.id. This allows the caller to retrieve the proper arguments of the specific overloaded procedure by matching the SPECIFIC_NAME value. * Improved output of method DatabaseMetaData.getFunctions(). The REMARKS column now contains the function definition as stored in sys.functions.func. The SPECIFIC_NAME column now contains the function unique identifier as stored in sys.functions.id. This allows the caller to retrieve the specific overloaded function which has the same name, but different arguments. Also improved output of method DatabaseMetaData.getFunctionColumns(). The SPECIFIC_NAME column now contains the function unique identifier as stored in sys.functions.id. This allows the caller to retrieve the proper arguments of the specific overloaded function by matching the SPECIFIC_NAME value. Bug Fixes * 4014: KILL signal * 4021: Analyze query does not escape input [security] * 4026: JDBC driver incorrectly converts TINYINT fields to String instead of an integer type. * 4028: inputs not the same size * 4031: mclient doesn't accept - argument to refer to stdin * 4032: no decimal places after update. ODBC driver * 4035: SQL Function call bug * 4036: Possible sql_catalog corruption due to unclean backuped tail

1 0

Determining join order
by Moritz Bruder 20 Jul '16

20 Jul '16

Hi, in the process of writing a MAL backend for a query compiler, I'm trying to optimize the join order of arbitrary theta joins on whole relations (i.e. predicates may be arbitrary). To implement this, I use a simple heuristic for 2-way theta joins on relations: First the cheap predicates are identified, then the necessary joins are performed on the columns. After that all results of the cheap predicates are merged via joins on the OIDs. Finally, expensive predicates (i.e. with a lot of qualifying tuples) are implemented by applying a binary scalar operation and using a *subselect* and a *projection*/*leftfetchjoin*, since all columns are aligned now. To conclude: 2-way joins may not be atomic MAL joins (i.e. not column-wise), but they very well could be. So, what I'm looking for is some facility in MonetDB, with which I am able to break down a n-way join into 2-way joins, with the smallest possible intermediate results, without assuming too much about how those 2-way joins are implemented. The first thing I found was the *scheduler* module, which is also on the official website ( https://www.monetdb.org/Documentation/Manuals/MonetDB/Optimizers/Memoization ). But that seems no more up to date, since most of the functions are missing and the semantics of join operators in MAL has changed quite a bit. Is there anything else I can use for this task (aside from implementing some dynamic programming algorithm within MAL)? Kind regards, Moritz

1 0

Limits on loaded shared libraries
by Anderson, David B 20 Jul '16

20 Jul '16

All, I was planning on loading a shared library within either an R or Python function inside a MonetDB server. Is this supported? What kinds of constraints do I need to worry about? Guessing any memory problems will impact the server, but not sure about thread safety etc. Thanks, Dave

1 0

MERGE TABLE ERROR
by Ariel Abadi 20 Jul '16

20 Jul '16

Hi! We are facing a problem with MERGE Tables. We have a process in php which connects to the DB using the mapi ( https://www.monetdb.org/Documentation/Manuals/SQLreference/Programming/MAPI <https://www.google.com/url?q=https%3A%2F%2Fwww.monetdb.org%2FDocumentation%…> ) The process creates a new table everyday as follows: CREATE TABLE cub8_20160717 AS (SELECT * FROM cub8) WITH NO DATA; Then, attaches that new table to the MERGE table called cub8_cubo as follows: ALTER TABLE cub8_cubo ADD TABLE cub8_20160717 ; Once all this is created, it start inserting data. Everything works fine, with NO errors. BUT if I run a query against cub8_cubo I dont see cub8_20160717 data. So I run this query in order to check if the new table was attached to the merge: SELECT a.name, b.name, s.name FROM sys.tables as a JOIN sys.dependencies d ON a.id = d.depend_id JOIN sys.tables as b ON b.id= d.id JOIN sys.schemas as s ON a.schema_id = s.id WHERE a.type= 3 AND a.name ILIKE '%cub8%' ORDER BY 1,2; +-----------+---------------+ | name | name | +===========+===============+ | cub8_cubo | cub8_20160710 | | cub8_cubo | cub8_20160711 | | cub8_cubo | cub8_20160712 | | cub8_cubo | cub8_20160713 | | cub8_cubo | cub8_20160714 | | cub8_cubo | cub8_20160715 | | cub8_cubo | cub8_20160716 | | cub8_cubo | cub8_20160717 | So I try to detached from cub8_cubo, with the command: ALTER TABLE cub8_cubo DROP TABLE cub8_20160717; ALTER TABLE: table 'sb_traf.cub8_20160717' isn't part of the MERGE TABLE 'sb_traf.cub8_cubo' So I ADD it again: ALTER TABLE cub8_cubo ADD TABLE cub8_20160717; and now yes... I can see the data. If I need to detach that table, I need to run ALTER ... DROP twice or three times to get it done. It seems to be a bug?, are we doing something wrong ?? Thks in advance! *PS.:We are using * *MonetDB Database Server v1.7 (Jun2016)* *Welcome to mclient, the MonetDB/SQL interactive terminal (Jun2016)*

3 8

embedded py/R on stable release
by Stefano Fioravanzo 17 Jul '16

17 Jul '16

Hello, Are ambedded python/R included in the latest stable release of MonetDB or do I have to install the dev version from source? Thank you Regards, Stefano Fioravanzo

3 3

jdbc connection problems
by Anderson, David B 16 Jul '16

16 Jul '16

All, Do I need to start the server with any special options in order to connect to it via jdbc? I have installed the driver in DBVisualizer v9.1.7 and am using the URL jdbc:monetdb://mrsvm1d:50000/pytest but I keep getting connection refused errors from the driver (included below). When I connect locally via mclient, I have: Welcome to mclient, the MonetDB/SQL interactive terminal (unreleased) Database: MonetDB v11.24.0 (unreleased), 'mapi:monetdb://mrsvm1d:50000/pytest' Type \q to quit, \? for a list of available commands auto commit mode: on sql> Here is my server command line (do I need to change the mapi items?): /opt/mr/monetdb/bin/mserver5.bin --set gdk_nr_threads 3 --dbpath=/home/da64666/MonetDB/pytest --set merovingian_uri mapi:monetdb://mrsvm1d:50000/pytest --set mapi_open false --set mapi_port 0 --set mapi_usock /home/da64666/MonetDB/pytest/.mapi.sock --set monet_vault_key /home/da64666/MonetDB/pytest/.vaultkey --set gdk_nr_threads 2 --set max_clients 64 --set sql_optimizer default_pipe --set embedded_py true --set monet_daemon yes Thanks, Dave java.sql.SQLException: Unable to connect (mrsvm1d:50000): Connection refused: connect at nl.cwi.monetdb.jdbc.MonetConnection.<init>(MonetConnection.java:244) at nl.cwi.monetdb.jdbc.MonetDriver.connect(MonetDriver.java:166) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.onseven.dbvis.g.B.D.?(Z:1413) at com.onseven.dbvis.g.B.F$A.call(Z:1474) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) SQLException: SQLState(08006) registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@130b7ed registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@7e673e registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@115a9a7 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1057d25 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1abd8e3 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@63900c registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1ff6814 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@9a6101 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@74e0a3 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@a0148b registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1070f4 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@9701ad registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1b6da12 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1769a58 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@4ea676 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@2d0f93 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1c2163a registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1056ee7 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1570d7b registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@18b128d registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@fae160 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@17d9594 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@cb361 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@c1f8d registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@8fe97c registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1108377 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@faf9a3 registerDriver: nl.cwi.monetdb.jdbc.MonetDriver@1c77aba java.sql.SQLException: Unable to connect (mrsvm1d:50000): Connection refused: connect at nl.cwi.monetdb.jdbc.MonetConnection.<init>(MonetConnection.java:244) at nl.cwi.monetdb.jdbc.MonetDriver.connect(MonetDriver.java:166) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.onseven.dbvis.g.B.D.?(Z:1413) at com.onseven.dbvis.g.B.F$A.call(Z:1474) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)

3 5

Re: monet crashes with query to VERY large table
by Sahasranaman M S 15 Jul '16

15 Jul '16

Hello, Just a suggestion, if you think the issue is related to the size of the data, then you could use a random data generator to test it. We were facing a similar issue with shareable data for testing. So I wrote a program called tqgen in Golang (https://github.com/niocs/tqgen) It can be used to generate large amounts of csv data for testing, and to share the data, we just need to share the command we used to generate it, and the other person can run it to generate the exact same data. It produces datewise fictional trade/quote data for fictional stocks. It puts each date's data into a separate file, and each file is around 128MB, so if we run it for a daterange of 1.5 years, it should generate around 70GB of data. It takes a while to finish because the data is not completely random and it is related to other rows. To generate, you could run: $ git clone https://github.com/niocs/tqgen $ cd tqgen $ go build tqgen.go $ ./tqgen --OutFilePat /path/to/data/tq.YYYYMMDD.csv --NumStk 1000 --Seed 234532 --DateBeg 20200101 --DateEnd 20210831 I usually load them one by one so that I could record load time stats, etc, but you could use this bash script ( https://github.com/niocs/hcsv/blob/master/hcat) to concatenate them without repeating the headers, and then load to MonetDB. (I will later add a parameter to tqgen to save to a single file) To do that, $ git clone https://github.com/niocs/hcsv $ hcsv/hcat /path/to/data/tq.20*.csv > /path/to/data/tq.big.csv If this causes the same issue, then others could regenerate the same data to try. Hope this helps. Thanks, Sahas > On Jul 14, 2016 10:20 AM, "Anthony Damico" <ajdamico(a)gmail.com> wrote: > >> can you winnow down the problem enough to construct a reproducible >> example? thanks >> >> On Thursday, July 14, 2016, Stefano Fioravanzo <fioravanzos(a)gmail.com> >> wrote: >> >>> Hello, >>> >>> No the data is not public, and testing in on another machine is not an >>> easy option right now. >>> >>> Regards, >>> >>> Stefano >>> >>> On 14 Jul 2016, at 15:49, Anthony Damico <ajdamico(a)gmail.com> wrote: >>> >>> hi, any chance these data are public? if not, can you test on a >>> separate box and see if the exact same thing happens or if it's >>> hardware-specific? >>> >>> On Thursday, July 14, 2016, Stefano Fioravanzo <fioravanzos(a)gmail.com> >>> wrote: >>> >>>> Hello, >>>> >>>> I currently have a table with 307524407 rows and 52 columns, it was >>>> loaded from a 78GB cvs file, but I do not know how much space the table >>>> actually takes on disk. >>>> If I query the table with a simple query like: >>>> >>>> select count(*) from big_table where field =1; >>>> >>>> mserver5 crashes with ‘segmentation fault' >>>> I tried also the following query (adding the LIMIT statement): >>>> select count(*) from farmap_movimento where farmacia_id =1 >>>> limit 10; >>>> >>>> and the output from mserver5 was: >>>> mserver5: gdk_select.c:869: fullscan_int: Assertion `cnt < >>>> (bn)->S->capacity' failed. >>>> Aborted >>>> >>>> Doing a more complex query (the one I am aiming for), with also a group >>>> by and a few aggregation functions (sum, min) on the fields, the outputs is: >>>> mserver5: gdk_bat.c:2066: BATsetcount: Assertion >>>> `b->S->capacity >= cnt' failed. >>>> Aborted >>>> >>>> This seems to be a table size problem, so…is there anything I can do? >>>> >>>> I am using MonetDB 5 server v11.22.0 >>>> The machine has 30GB of RAM and 60GB of swap. >>>> Available disk space: ~50GB >>>> >>>> Any help would be much appreciated, >>>> >>>> Regards, >>>> >>>> Stefano >>>> >>>> _______________________________________________ >>>> users-list mailing list >>>> users-list(a)monetdb.org >>>> https://www.monetdb.org/mailman/listinfo/users-list >>>> >>> _______________________________________________ >>> users-list mailing list >>> users-list(a)monetdb.org >>> https://www.monetdb.org/mailman/listinfo/users-list >>> >>> >>> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list >> >>

3 5

monet crashes with query to VERY large table
by Stefano Fioravanzo 14 Jul '16

14 Jul '16

Hello, I currently have a table with 307524407 rows and 52 columns, it was loaded from a 78GB cvs file, but I do not know how much space the table actually takes on disk. If I query the table with a simple query like: select count(*) from big_table where field =1; mserver5 crashes with ‘segmentation fault' I tried also the following query (adding the LIMIT statement): select count(*) from farmap_movimento where farmacia_id =1 limit 10; and the output from mserver5 was: mserver5: gdk_select.c:869: fullscan_int: Assertion `cnt < (bn)->S->capacity' failed. Aborted Doing a more complex query (the one I am aiming for), with also a group by and a few aggregation functions (sum, min) on the fields, the outputs is: mserver5: gdk_bat.c:2066: BATsetcount: Assertion `b->S->capacity >= cnt' failed. Aborted This seems to be a table size problem, so…is there anything I can do? I am using MonetDB 5 server v11.22.0 The machine has 30GB of RAM and 60GB of swap. Available disk space: ~50GB Any help would be much appreciated, Regards, Stefano

2 3

ROLE copy_role
by Alberto Ferrari 14 Jul '16

14 Jul '16

Hi all: In the new version of Monet (jun/2016), when normal USER (not "monetdb" user) runs "COPY INTO" or "INSERT", it only works with "SET ROLE copy_role;" before the query. Is that really needed? Should I prepend "SET ROLE copy_role" before every INSERT / COPY INTO query?? Doesn't make sense for me. I supposed the user/permissions configuration is enough for that? Thanks in advance.

1 0

adding binary columns to existing table
by Lynn Carol Johnson 14 Jul '16

14 Jul '16

Hi all - I've been using Monetdb's binary bulk loads and love it. Very fast, very efficient. We have a large database (2.3 billion rows) to which we frequently add new columns as new data is curated. To date, I've been re-creating the table each time, adding the new columns, and performing a binary bulk load for data in all columns. My question: Does monetdb allow for altering a table and then loading a binary file just for that column? Or is this not efficient as it requires an update on every row to add the new column's data? As our table columns reach into the hundreds it would be nice to "append" new columns, but perhaps logistically this isn't reasonable. Thanks for your insight - Lynn

1 0

Server crashes when using a prepared statement with a function
by João Victor Risso 12 Jul '16

12 Jul '16

Hello, I have a SQL function (myFunc) that computes the sum of the records on a table (tbl1) and returns the sum (along with other attributes) as a table. tbl1 references tables tbl2 and tbl3 through fk_tbl2_id and fk_tbl3_id's, respectively. Since I want to use this function in a Node.js API and the parameters for the function will come from the users of the API, it is safer to use a prepared statement to bind the parameters. Whenever I try to execute the following prepared statement with the function: > PREPARE SELECT * FROM myFunc(?, ?, ?); > EXEC (2008, 1, 2); The server crashes, as shown in the server log and dmesg: > 2016-07-12 10:56:40 MSG merovingian[3110]: database 'testdb' (19875) was killed by signal SIGSEGV > mserver5[18201]: segfault at 30 ip 00007f2d87be51e4 sp 00007f2d86f63b80 error 4 in lib_sql.so[7f2d87b85000+199000] The SQL function for myFunc is the following: > CREATE FUNCTION myFunc(year int, table2_id int, table3_id int) > RETURNS TABLE(name text, total bigint) > BEGIN > RETURN ( > SELECT > 'Overall', > COALESCE(SUM(t1.num_enrollments), 0) > FROM > tbl1 AS t1 > WHERE > t1.census_year = year > AND > (CASE(table2_id) > WHEN 0 THEN > (t1.fk_table2_id > 0) > ELSE > (t1.fk_table2_id = table2_id) > END) > AND > (CASE(table3_id) > WHEN 0 THEN > (t1.fk_table3_id > 0) > ELSE > (t1.fk_table3_id = table3_id) > END) > ); > END; I'm running MonetDB v11.21.19 (Jul2015-SP4) on Ubuntu 14.04 LTS. I could not find any bug reports related to the issue. How can I debug it further? Any help is deeply appreciated. Best regards, João Victor Risso.

1 0

select * from sys.db_users() SIGSEGV
by Vincent Sheffer 12 Jul '16

12 Jul '16

We are running: [root@monetdb monetdb5]# mserver5 --version MonetDB 5 server v11.21.19 "Jul2015-SP4" (64-bit, 64-bit oids, 128-bit integers) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 503.6GiB available memory, 28 available cpu cores Libraries: libpcre: 8.32 2012-11-30 (compiled with 8.32) openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with ) libxml2: 2.9.1 (compiled with 2.9.1) Compiled by: @9bedafc83c11 (x86_64-unknown-linux-gnu) Compilation: gcc -gdwarf-4 -g3 Linking : /usr/bin/ld -m elf_x86_64 When select * from sys.db_users() is executed when logged in as the monetdb user there is no problem, but when run under any other user we get the following stack trace: [Switching to Thread 0x7f418e7f3700 (LWP 26599)] 0x00007f41bd5cf059 in db_users_wrap (cntxt=0x7f41be899630, mb=0x7f412416e620, stk=0x7f4124114bd0, pci=0x7f41241255d0) at sql_user.c:160 160sql_user.c: No such file or directory. (gdb) where #0 0x00007f41bd5cf059 in db_users_wrap (cntxt=0x7f41be899630, mb=0x7f412416e620, stk=0x7f4124114bd0, pci=0x7f41241255d0) at sql_user.c:160 #1 0x00007f41c519f1f7 in runMALsequence (cntxt=0x7f41be899630, mb=0x7f412416e620, startpc=1, stoppc=0, stk=0x7f4124114bd0, env=0x0, pcicaller=0x0) at mal_interpreter.c:631 #2 0x00007f41c519e819 in callMAL (cntxt=0x7f41be899630, mb=0x7f412416e620, env=0x7f418e7f2a68, argv=0x7f418e7f2af0, debug=0 '\000') at mal_interpreter.c:447 #3 0x00007f41bd5d50d6 in SQLexecutePrepared (c=0x7f41be899630, be=0x7f41242417f0, q=0x7f4124117740) at sql_execute.c:328 #4 0x00007f41bd5d54c9 in SQLengineIntern (c=0x7f41be899630, be=0x7f41242417f0) at sql_execute.c:390 #5 0x00007f41bd5d4257 in SQLengine (c=0x7f41be899630) at sql_scenario.c:1323 #6 0x00007f41c51bcaf7 in runPhase (c=0x7f41be899630, phase=4) at mal_scenario.c:515 #7 0x00007f41c51bccb4 in runScenarioBody (c=0x7f41be899630) at mal_scenario.c:559 #8 0x00007f41c51bcda1 in runScenario (c=0x7f41be899630) at mal_scenario.c:579 #9 0x00007f41c51bdcd8 in MSserveClient (dummy=0x7f41be899630) at mal_session.c:439 #10 0x00007f41c51bd8b2 in MSscheduleClient (command=0x7f4124279b90 "p\233'$A\177", challenge=0x7f418e7f2da0 "Up8bNTal6v", fin=0x7f4124117560, fout=0x7f4124241680) at mal_session.c:319 #11 0x00007f41c526d001 in doChallenge (data=0x7f41a0001690) at mal_mapi.c:184 #12 0x00007f41c4bae17f in thread_starter (arg=0x7f41a00008c0) at gdk_system.c:458 #13 0x00007f41c2814dc5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f41c2541ced in clone () from /lib64/libc.so.6 Thanks, Vince The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.

1 0

range request on timestamp column in prepared statement
by Frank Tetzel 11 Jul '16

11 Jul '16

Hello, i have trouble to get a parametrized range request on a timestamp column working. Small example in mclient: create table test(creationDate timestamp); insert into test values('2012-04-02 00:00:00'), ('2012-03-30 00:00:00'),('2012-04-20 00:00:00'); -- works prepare select * from test where creationDate < ?; exec 21(TIMESTAMP '2012-04-01 00:00:00'); -- works select * from test where creationDate between TIMESTAMP '2012-04-01 00:00:00' and TIMESTAMP '2012-04-01 00:00:00' + interval '5' day; -- syntax error, unexpected '?', expecting SCOLON prepare select * from test where creationDate between ? and ? + interval ? day; -- fails too: program contains errors prepare select * from test where creationDate between ? and ? + interval '5' day; exec 22(TIMESTAMP '2012-04-01 00:00:00', TIMESTAMP '2012-04-01 00:00:00'); Does anybody know of a workaround? Is there a builtin function for adding days to a timestamp i missed? I couldn't find anything in the documentation. Btw, i'm on v11.23.3. Regards, Frank

1 1

Is there any scan sharing technique in MonetDB?
by Xu,Wenjian 07 Jul '16

07 Jul '16

Hi all, I understand that MonetDB would recycle intermediate results to accelerate subsequent OLAP queries that involve common sub-queries as previous ones. I am wondering that, 1. Does MonetDB implement any scan sharing techniques (or generally, multi-query optimization techniques) in the context of concurrent workloads? For example, two concurrent queries Q1 and Q2 both require scanning column X. Is it possible for their query plans to share the scan operation? 2. If no, what is the reason? too complex to implement? or any reason related to MonetDB's execution paradigm that prevents such sharing techniques? Any comments would be helpful, thanks. Best regards, Wenjian

2 2

R UDFs
by Anderson, David B 06 Jul '16

06 Jul '16

All, Quick question, how many R interpreters are running within MonetDB if you are using R UDFs? Is there a single shared interpreter or are there multiple copies running? Thanks, Dave

2 16

DATABASE NOT STARTING
by Ariel Abadi 04 Jul '16

04 Jul '16

Guys! Im desperate... the DB is not starting anymore... Can someone help me? Im starting the DB but I receive the following msg: starting database 'sb_traf'... FAILED start: starting 'sb_traf' failed: database 'sb_traf' appears to shut itself down after starting, check monetdbd's logfile for possible hints This is part of the log... maybe somebody has a clue. 2016-07-02 22:15:30 MSG merovingian[28268]: proxying client static.10-10-10-225.clients.your-server.de:47462 for database 'sb_traf' to mapi:monetdb:///opt/monetdb/CUSTOMERnew/sb_traf/.mapi.sock?database=sb_traf 2016-07-02 22:15:30 MSG merovingian[28268]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-07-02 22:15:30 MSG merovingian[28268]: proxying client static.10-10-10-225.clients.your-server.de:47463 for database 'sb_traf' to mapi:monetdb:///opt/monetdb/CUSTOMERnew/sb_traf/.mapi.sock?database=sb_traf 2016-07-02 22:15:30 MSG merovingian[28268]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-07-02 22:20:29 MSG merovingian[28268]: proxying client static.10-10-10-225.clients.your-server.de:47726 for database 'sb_traf' to mapi:monetdb:///opt/monetdb/CUSTOMERnew/sb_traf/.mapi.sock?database=sb_traf 2016-07-02 22:20:29 MSG merovingian[28268]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-07-02 22:20:29 MSG merovingian[28268]: database 'sb_traf' (28351) was killed by signal SIGSEGV 2016-07-02 22:20:36 MSG merovingian[28268]: starting database 'sb_traf', up min/avg/max: 11s/1d/3w, crash average: 1.00 0.40 0.50 (69-25=44) 2016-07-02 22:20:36 MSG sb_traf[29578]: arguments: /usr/bin/mserver5 --dbpath=/opt/monetdb/CUSTOMERnew/sb_traf --set merovingian_uri=mapi:monetdb://CUSTOMER-monetbd:54321/sb_traf --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/monetdb/CUSTOMERnew/sb_traf/.mapi.sock --set monet_vault_key=/opt/monetdb/CUSTOMERnew/sb_traf/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-07-02 22:20:36 MSG sb_traf[29578]: # MonetDB 5 server v11.21.19 "Jul2015-SP4" 2016-07-02 22:20:36 MSG sb_traf[29578]: # Serving database 'sb_traf', using 8 threads 2016-07-02 22:20:36 MSG sb_traf[29578]: # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-07-02 22:20:36 MSG sb_traf[29578]: # Found 20.000 GiB available main-memory. 2016-07-02 22:20:36 MSG sb_traf[29578]: # Copyright (c) 1993-July 2008 CWI. 2016-07-02 22:20:36 MSG sb_traf[29578]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2016-07-02 22:20:36 MSG sb_traf[29578]: # Visit http://www.monetdb.org/ for further information 2016-07-02 22:20:36 MSG sb_traf[29578]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/monetdb/CUSTOMERnew/sb_traf/.mapi.sock 2016-07-02 22:20:36 MSG sb_traf[29578]: # Start processing logs sql/sql_logs version 52200 2016-07-02 22:20:36 MSG sb_traf[29578]: # Start reading the write-ahead log 'sql_logs/sql/log.81841' 2016-07-02 22:20:36 MSG sb_traf[29578]: # Finished reading the write-ahead log 'sql_logs/sql/log.81841' 2016-07-02 22:20:36 MSG sb_traf[29578]: # Finished processing logs sql/sql_logs 2016-07-02 22:20:36 MSG merovingian[28268]: database 'sb_traf' (29578) was killed by signal SIGSEGV 2016-07-02 22:20:36 ERR merovingian[28268]: client error: database 'sb_traf' appears to shut itself down after starting, check monetdbd's logfile for possible hints 2016-07-02 22:20:37 MSG merovingian[28268]: starting database 'sb_traf', up min/avg/max: 11s/1d/3w, crash average: 1.00 0.50 0.50 (70-25=45) 2016-07-02 22:20:37 MSG sb_traf[29582]: arguments: /usr/bin/mserver5 --dbpath=/opt/monetdb/CUSTOMERnew/sb_traf --set merovingian_uri=mapi:monetdb://CUSTOMER-monetbd:54321/sb_traf --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/monetdb/CUSTOMERnew/sb_traf/.mapi.sock --set monet_vault_key=/opt/monetdb/CUSTOMERnew/sb_traf/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-07-02 22:20:37 MSG sb_traf[29582]: # MonetDB 5 server v11.21.19 "Jul2015-SP4" 2016-07-02 22:20:37 MSG sb_traf[29582]: # Serving database 'sb_traf', using 8 threads 2016-07-02 22:20:37 MSG sb_traf[29582]: # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-07-02 22:20:37 MSG sb_traf[29582]: # Found 20.000 GiB available main-memory. 2016-07-02 22:20:37 MSG sb_traf[29582]: # Copyright (c) 1993-July 2008 CWI. 2016-07-02 22:20:37 MSG sb_traf[29582]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2016-07-02 22:20:37 MSG sb_traf[29582]: # Visit http://www.monetdb.org/ for further information 2016-07-02 22:20:37 MSG sb_traf[29582]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/monetdb/CUSTOMERnew/sb_traf/.mapi.sock 2016-07-02 22:20:37 MSG sb_traf[29582]: # Start processing logs sql/sql_logs version 52200 2016-07-02 22:20:37 MSG sb_traf[29582]: # Start reading the write-ahead log 'sql_logs/sql/log.81841' 2016-07-02 22:20:37 MSG sb_traf[29582]: # Finished reading the write-ahead log 'sql_logs/sql/log.81841' 2016-07-02 22:20:37 MSG sb_traf[29582]: # Finished processing logs sql/sql_logs 2016-07-02 22:20:37 MSG merovingian[28268]: database 'sb_traf' (29582) was killed by signal SIGSEGV 2016-07-02 22:20:37 ERR merovingian[28268]: client error: database 'sb_traf' appears to shut itself down after starting, check monetdbd's logfile for possible hints

4 5

python map
by Stefano Fioravanzo 04 Jul '16

04 Jul '16

Hello, I have a long and complex embedded python function. This function depends on the single rows of the input columns, so I would like to parallelize it. I’ve read about the post on embedded python and python_map. I understand that using python_map should automatically split (in a not specified way) the input columns and fire different (how many?) python processes executing on their input slice. Finally it should pack all the partial results. Well, I have tried to use python_map, but nothing different happened. Is there something more I have to do apart from setting LANGUAGE PYTHON_MAP? Regards, Stefano

2 9

Positional placeholders in prepared statement
by Frank Tetzel 29 Jun '16

29 Jun '16

Hi, does MonetDB support positional placeholders like '$1' instead of just '?' in prepared statements? I want to refer to a parameter more than once and possibly also out of order. Is there some alternative available? Regards, Frank

2 1

Resetting monetdb administrative password
by Roman S 29 Jun '16

29 Jun '16

Hello, Can somebody tell me how to reset monetdb administrative password? I’ve tried to stop mserver5, start it manually, oldp:=clients.getPasswordHash(“monetdb"); newp:=clients.backendsum("newpassword"); clients.changePassword(oldp, newp); Stoping it again and start in normal mode But it didn’t help: mclient --database=mydb --user=monetdb password:<usingnewpass> InvalidCredentialsException:checkCredentials:invalid credentials for user 'monetdb'

2 2

MERGE Tables works ?
by Ariel Abadi 28 Jun '16

28 Jun '16

Hi Everybody! Im facing a problem. We register telecomunications calls. As the volume is so big, we decided to store the data in different tables. The criteria we used for splitting the info was to create a table per each Switch and Day. So we have Sw1_20160601, Sw2_20160602...Sw3_20160620..etc etc Each table has aprox 15MM rows. In order to create some reports we have created a *MERGE TABLE *and added to this table the 90 tables (3 switches x 30 days). BUT, now Im trying to run a very simple query, like SELECT cdr_date FROM xxxx LIMIT 3; To get at least 3 rows (just to see that this schema works). It took 21 mins to give me those 3 rows.... Am I doing something wrong? Rgds Ariel

3 4

Announcement: New Jun2016 Feature release of MonetDB suite
by Sjoerd Mullender 23 Jun '16

23 Jun '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2016 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jun 2016 feature release MonetDB5 Server * Extended the storage() table producing function to also accept storage([schemaname [, tablename [, columnname]]]) * The :bat[:oid,:any] type descriptor has been turned into its columnar version :bat[:any] * Converted the *.mal scripts into *.malC versions to prepare for removal of the mserver console. * Removed functions mat.hasMoreElements, mat.info, mat.mergepack, mat. newIterator, mat.project, mat.pack2, mat.sortReverse, mat.sort, and mat.slice. * Removed grouped aggregate functions from the aggr module in which the groups were indicated by the head column of the bat to be aggregated. Use the interface with a separate group bat instead. * The server now stops executing a query when the client disconnects. * Removed algebra.join. Use algebra.subjoin instead. * Removed algebra.antijoin. Use algebra.subantijoin or algebra.subthetajoin instead. * The MAL function 'leftfetchjoin' is renamed to its relational algebra version 'projection' * The generic property handling scheme has been removed. It was used in just a few places, for widely different purposes and contained cruft. * Removed functions str.iconv and str.codeset. Internally, strings are always in UTF-8 encoding, so we can't allow code set conversions. * Removed algebra.like with a BAT argument. Use algebra.likesubselect instead. * Removed algebra.leftjoin. Use algebra.subleftjoin or algebra.leftfetchjoin instead. * Removed algebra.tdiff and algebra.tinter. * Removed algebra.sample. Use sample.subuniform instead. * Removed algebra.select. Use algebra.subselect instead. * Removed algebra.revert. * Removed bat.order and bat.orderReverse functions. * Changed client.getUsers to return two bats, one with the user id and one with the user name. * Implemented algebra.subdiff and algebra.subinter. * Removed algebra.tdifference and algebra.tintersect. * Removed algebra.tunion. Build Environment * A new package MonetDB-lidar (Fedora) or libmonetdb5-server-lidar (Debian/Ubuntu) has been created to work with LiDAR data. * Implemented a systemd configuration file for a monetdbd.service on systems that support it (Fedora, newer Ubuntu). Client Package * The Ruby client is now in a separate repository (http://dev.monetdb.org/hg/monetdb-ruby) and released independently. Geom Module * The geom module has been completely overhauled. Types are now specified as GEOMETRY(POINT) instead of POINT, old functions have been removed, new ones introduced. However, even with all the changes, a database upgrade should still be possible (as always, make a backup first). MonetDB Common * Removed BATconst. Use BATconstant instead. * Changed BATconstant. It now has a new first argument with the seqbase for the head column. * Removed BATmmap. It was no longer used. * BUNdelete and BATdel don't accept a foce argument and only allow deleting values that have not yet been committed. BUNdelete exchanges the deleted value with the last value (if the deleted value isn't the last value). BATdel compacts the BAT by shifting values after the deleted values up. The list of to-be-deleted values in BATdel must be sorted and unique. * Removed BUNreplace from list of exported functions. It wasn't used, and there is still BUNinplace and void_inplace that do more-or-less the same thing. * Changed BATderiveHeadProps to BATderiveTailProps (and it now works on the tail column). * Removed unused functions BATalpha, BATdelta, and BATprev. * Removed function VIEWcombine. Use BATdense instead. * Removed "left" parameter from BUNinplace. It wasn't used since the BAT it works on is dense headed. * Created function BATdense to easily create a [void,void] BAT with specified seqbases for head and tail and a count. * Removed function BATmark. When all head columns are dense, BATmark basically only created a new [void,void] BAT. * Renamed BATsubsort to BATsort. * Removed "sub" from the names of the function BATsubselect, BATthetasubselect, BATsubcross, BATsubleftjoin, BATsubouterjoin, BATsubthetajoin, BATsubsemijoin, BATsubdiff, BATsubjoin, BATsubbandjoin, BATsubrangejoin, and BATsubunique. * Removed BATsubleftfetchjoin: it was not used. * Removed BATcross1. Use BATsubcross instead. * Removed all versions of the SORTloop macro. * Removed Batkdiff. Use BATsubdiff instead. * Removed BATselect. Use BATsubselect instead. * Removed BATsemijoin. Use BATsubsemijoin instead. * Removed BATjoin. Use BATsubjoin instead. * Removed BATleftjoin. Use BATsubleftjoin or BATproject instead. * Removed BATleftfetchjoin. Use BATproject instead. * Removed BUNins from the list of exported functions. * Removed legacy functions BATuselect and BATuselect_. * Removed legacy functions BATantijoin, BATbandjoin, BATouterjoin, BATrangejoin, and BATthetajoin. * Removed function BATrevert. * BATordered now works on the TAIL column. * Removed obsolete functions BATorder() and BATorder_rev(). * Implemented BATsubdiff which returns a list of OIDs (sorted, i.e. usable as candidate list) of tuples in the left input whose value does not occur in the right input. * Removed function BATkintersect. It wasn't used anymore. It's functionality can be obtained by using BATsubsemijoin. * Removed the function BATkunion. Java Module * Fixed problem in DatabaseMetaData.getUDTs() when it was called with types parameter filled. It used to throw SQException with message: SELECT: identifier 'DATA_TYPE' unknown. Now it returns the UDTs which match the provided array of data types. * Implemented MonetDatabaseMetaData.supportsConvert() and MonetDatabaseMetaData.supportsConvert(int fromType, int toType) methods. It used to always return false. Now it returns true for the supported conversions. This fixes Bug 3460. * ResultSet.setFetchSize(): added a dummy implementation to get rid of the SQLFeatureNotSupportedException. In MonetDB, it does not make sense to set the fetch size of a result set. If one really wants to set the fetch size, one should use Statement.setFetchSize() instead. * Fixed resource leak in ResultSetMetaData. It created and cached a ResultSet object for each column but never closed the ResultSet objects. * Corrected DatabaseMetaData methods which accept a catalog filter argument. Those methods will now filter the results on the specified catalog name, whereas previously the catalog filter argument was ignored. * Corrected output of column KEY_SEQ of DatabaseMetaData methods: getPrimaryKeys(), getImportedKeys(), getExportedKeys() and getCrossReference(). It now starts at 1 instead of 0 previously. * Corrected DatabaseMetaData.getSchemas() by returning 2 instead of 3 columns. * Improved DatabaseMetaData.getColumns() by returning two additional columns: IS_AUTOINCREMENT and IS_GENERATEDCOLUMN. * Improved DatabaseMetaData.getTypeInfo(). It now returns better information on LITERAL_PREFIX, LITERAL_SUFFIX, CREATE_PARAMS, CASE_SENSITIVE, FIXED_PREC_SCALE and MAXIMUM_SCALE for some data types. Also the returned rows are now ordered by DATA_TYPE, TYPE_NAME, PRECISION as required by the specs. Also corrected output column names "searchable" into "SEARCHABLE" and "MAXIMUM SCALE" into "MAXIMUM_SCALE" to match the JDBC specification. * Corrected DatabaseMetaData.getPseudoColumns(). It used to return 12 empty rows. Now it returns no rows as MonetDB does not have pseudo columns. * Implemented method DatabaseMetaData.getClientProperties(). It used to always return a resultset with 4 completely empty rows. It now returns a resultset with the possible connection properties. * Implemented method DatabaseMetaData.getUDTs(). It used to return an empty resultset. Now it returns the User Defined Types such as inet, json, url and uuid. * Corrected the returned table types in DatabaseMetaData.getTableTypes(). It now returns all 10 table types (as stored in sys.table_types) instead of the previously 8 hardcoded table types. For old MonetDB servers which do not have the sys.table_types table, the old behavior is retained. * Implemented methods DatabaseMetadata.getProcedures() and DatabaseMetadata.getProcedureColumns(). They used to return an empty resultset. Now they return the expected Procedures and ProcedureColumns. Also getProcedureColumns() now returns a resultset with all 20 columns instead of 13 columns previously. * Method getFunctionColumns() in DatabaseMetadata used to throw an SQLException: getFunctionColumns(String, String, String, String) is not implemented. This method is now implemented and returns a resultset. * Method getFunctions() in DatabaseMetadata used to throw an SQLException: SELECT: no such column 'functions.sql' This has been corrected. It now returns a resultset as requested. * The resultsets of DatabaseMetadata methods now no longer return a value for the *_CAT columns as MonetDB does not support Catalogs. * Fixed a memory leak in MonetDatabaseMetaData.java for a static cache which kept references to closed Connection objects. * Improved JdbcClient program when presenting query data to console. It used to send an SQL catalog query for each query result column which slowed down the interactive response considerably. These additional SQL catalog queries have been eliminated. * Corrected MonetResultSet.getObject(String columnName). It no longer throws a NullPointerException in cases where internally a MonetVirtualResultSet is used. * Fixed Connection.isValid(): this method should never attempt to close the connection, even if an error has occurred. SQL * Disallow GRANT <some-user> TO <role-or-use>. Only explicitly created roles can be granted. * Extended grantable schema privileges: when a user is granted the "sysadmin" role, the user now hcan not only create schemas, but also drop schemas. * Introduced COPY INTO/ COPY FROM global privileges. These privileges allows other users than 'monetdb' to be granted the privileges to use the MonetDB bulk data loading/exporting features, i.e., COPY FROM <file> and COPY INTO <file>. Bug Fixes * 2407: Merovingian: allow binds to given ip/interface * 2815: No SRID support * 3361: constants as MAL function parameters prevent intermediate reuse * 3374: UNIQUE constraint does not set tkey property on the corresponding BAT * 3460: incomplete implementation of JDBC driver supportsConvert(), supportsConvert(int fromType, int toType) methods in MonetDatabaseMetaData.java * 3711: JDBC connection using jdbcclient.jar is very slow * 3877: MonetDBLite should allow close then re-open databases? without restarting R * 3911: Invalid connect() call in 'redirect' mode * 3920: query with DISTINCT and correlated scalar subquery in SELECT list causes Assertion failure and crash of mserver5 * 3927: COUNT( distinct my_clob ) , QUANTILE( my_double , number ) fails in dev build * 3956: MonetDBLite unable to execute LIMIT 1 statement * 3972: Drastic Memory leak of 600GBs while generating plan for Query with 25 joins * 3974: Prepared statement rel_bin.c:2378: rel2bin_project: Assertion `0' failed. * 3975: Suspicious code in store_manager() on exit path * 3978: SQL returns TypeException 'aggr.subcorr' undefined for sys.corr function * 3980: JOIN with references on both sides crashes mserver * 3981: Incorrect LEFT JOIN when FROM clause contains nested subqueries * 3983: Creation of a Foreign Key which partially maps to a primary key is accepted without a warning * 3984: Multiple paths in the .profile * 3985: ruby-monetdb-sql gem fails for negative timezone offset (USA, etc.) * 3987: Segfault on malformed csv import * 3991: MonetDBLite feature request: default monetdb.sequential to FALSE on windows * 3994: MonetDBLite dbDisconnect with shutdown=TRUE freezes the console on windows * 3995: NullPointerException when calling getObject() * 3997: calling scalar functions sys.isaUUID(str) or sys.isaUUID(uuid) fail * 3999: length() returns wrong length for strings which have spaces at the end of the string * 4010: RELEASE SAVEPOINT after ALTER TABLE crashes mserver5 * 4011: sys.sessions.user column always shows 'monetdb' * 4013: GDKextendf does not free up memory when it fails due to insufficient resources * 4015: Daemon crashes on database release command

1 0

Announcement: New Jun2016 Feature release of MonetDB suite
by Sjoerd Mullender 23 Jun '16

23 Jun '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2016 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jun 2016 feature release MonetDB5 Server * Extended the storage() table producing function to also accept storage([schemaname [, tablename [, columnname]]]) * The :bat[:oid,:any] type descriptor has been turned into its columnar version :bat[:any] * Converted the *.mal scripts into *.malC versions to prepare for removal of the mserver console. * Removed functions mat.hasMoreElements, mat.info, mat.mergepack, mat. newIterator, mat.project, mat.pack2, mat.sortReverse, mat.sort, and mat.slice. * Removed grouped aggregate functions from the aggr module in which the groups were indicated by the head column of the bat to be aggregated. Use the interface with a separate group bat instead. * The server now stops executing a query when the client disconnects. * Removed algebra.join. Use algebra.subjoin instead. * Removed algebra.antijoin. Use algebra.subantijoin or algebra.subthetajoin instead. * The MAL function 'leftfetchjoin' is renamed to its relational algebra version 'projection' * The generic property handling scheme has been removed. It was used in just a few places, for widely different purposes and contained cruft. * Removed functions str.iconv and str.codeset. Internally, strings are always in UTF-8 encoding, so we can't allow code set conversions. * Removed algebra.like with a BAT argument. Use algebra.likesubselect instead. * Removed algebra.leftjoin. Use algebra.subleftjoin or algebra.leftfetchjoin instead. * Removed algebra.tdiff and algebra.tinter. * Removed algebra.sample. Use sample.subuniform instead. * Removed algebra.select. Use algebra.subselect instead. * Removed algebra.revert. * Removed bat.order and bat.orderReverse functions. * Changed client.getUsers to return two bats, one with the user id and one with the user name. * Implemented algebra.subdiff and algebra.subinter. * Removed algebra.tdifference and algebra.tintersect. * Removed algebra.tunion. Build Environment * A new package MonetDB-lidar (Fedora) or libmonetdb5-server-lidar (Debian/Ubuntu) has been created to work with LiDAR data. * Implemented a systemd configuration file for a monetdbd.service on systems that support it (Fedora, newer Ubuntu). Client Package * The Ruby client is now in a separate repository (http://dev.monetdb.org/hg/monetdb-ruby) and released independently. Geom Module * The geom module has been completely overhauled. Types are now specified as GEOMETRY(POINT) instead of POINT, old functions have been removed, new ones introduced. However, even with all the changes, a database upgrade should still be possible (as always, make a backup first). MonetDB Common * Removed BATconst. Use BATconstant instead. * Changed BATconstant. It now has a new first argument with the seqbase for the head column. * Removed BATmmap. It was no longer used. * BUNdelete and BATdel don't accept a foce argument and only allow deleting values that have not yet been committed. BUNdelete exchanges the deleted value with the last value (if the deleted value isn't the last value). BATdel compacts the BAT by shifting values after the deleted values up. The list of to-be-deleted values in BATdel must be sorted and unique. * Removed BUNreplace from list of exported functions. It wasn't used, and there is still BUNinplace and void_inplace that do more-or-less the same thing. * Changed BATderiveHeadProps to BATderiveTailProps (and it now works on the tail column). * Removed unused functions BATalpha, BATdelta, and BATprev. * Removed function VIEWcombine. Use BATdense instead. * Removed "left" parameter from BUNinplace. It wasn't used since the BAT it works on is dense headed. * Created function BATdense to easily create a [void,void] BAT with specified seqbases for head and tail and a count. * Removed function BATmark. When all head columns are dense, BATmark basically only created a new [void,void] BAT. * Renamed BATsubsort to BATsort. * Removed "sub" from the names of the function BATsubselect, BATthetasubselect, BATsubcross, BATsubleftjoin, BATsubouterjoin, BATsubthetajoin, BATsubsemijoin, BATsubdiff, BATsubjoin, BATsubbandjoin, BATsubrangejoin, and BATsubunique. * Removed BATsubleftfetchjoin: it was not used. * Removed BATcross1. Use BATsubcross instead. * Removed all versions of the SORTloop macro. * Removed Batkdiff. Use BATsubdiff instead. * Removed BATselect. Use BATsubselect instead. * Removed BATsemijoin. Use BATsubsemijoin instead. * Removed BATjoin. Use BATsubjoin instead. * Removed BATleftjoin. Use BATsubleftjoin or BATproject instead. * Removed BATleftfetchjoin. Use BATproject instead. * Removed BUNins from the list of exported functions. * Removed legacy functions BATuselect and BATuselect_. * Removed legacy functions BATantijoin, BATbandjoin, BATouterjoin, BATrangejoin, and BATthetajoin. * Removed function BATrevert. * BATordered now works on the TAIL column. * Removed obsolete functions BATorder() and BATorder_rev(). * Implemented BATsubdiff which returns a list of OIDs (sorted, i.e. usable as candidate list) of tuples in the left input whose value does not occur in the right input. * Removed function BATkintersect. It wasn't used anymore. It's functionality can be obtained by using BATsubsemijoin. * Removed the function BATkunion. Java Module * Fixed problem in DatabaseMetaData.getUDTs() when it was called with types parameter filled. It used to throw SQException with message: SELECT: identifier 'DATA_TYPE' unknown. Now it returns the UDTs which match the provided array of data types. * Implemented MonetDatabaseMetaData.supportsConvert() and MonetDatabaseMetaData.supportsConvert(int fromType, int toType) methods. It used to always return false. Now it returns true for the supported conversions. This fixes Bug 3460. * ResultSet.setFetchSize(): added a dummy implementation to get rid of the SQLFeatureNotSupportedException. In MonetDB, it does not make sense to set the fetch size of a result set. If one really wants to set the fetch size, one should use Statement.setFetchSize() instead. * Fixed resource leak in ResultSetMetaData. It created and cached a ResultSet object for each column but never closed the ResultSet objects. * Corrected DatabaseMetaData methods which accept a catalog filter argument. Those methods will now filter the results on the specified catalog name, whereas previously the catalog filter argument was ignored. * Corrected output of column KEY_SEQ of DatabaseMetaData methods: getPrimaryKeys(), getImportedKeys(), getExportedKeys() and getCrossReference(). It now starts at 1 instead of 0 previously. * Corrected DatabaseMetaData.getSchemas() by returning 2 instead of 3 columns. * Improved DatabaseMetaData.getColumns() by returning two additional columns: IS_AUTOINCREMENT and IS_GENERATEDCOLUMN. * Improved DatabaseMetaData.getTypeInfo(). It now returns better information on LITERAL_PREFIX, LITERAL_SUFFIX, CREATE_PARAMS, CASE_SENSITIVE, FIXED_PREC_SCALE and MAXIMUM_SCALE for some data types. Also the returned rows are now ordered by DATA_TYPE, TYPE_NAME, PRECISION as required by the specs. Also corrected output column names "searchable" into "SEARCHABLE" and "MAXIMUM SCALE" into "MAXIMUM_SCALE" to match the JDBC specification. * Corrected DatabaseMetaData.getPseudoColumns(). It used to return 12 empty rows. Now it returns no rows as MonetDB does not have pseudo columns. * Implemented method DatabaseMetaData.getClientProperties(). It used to always return a resultset with 4 completely empty rows. It now returns a resultset with the possible connection properties. * Implemented method DatabaseMetaData.getUDTs(). It used to return an empty resultset. Now it returns the User Defined Types such as inet, json, url and uuid. * Corrected the returned table types in DatabaseMetaData.getTableTypes(). It now returns all 10 table types (as stored in sys.table_types) instead of the previously 8 hardcoded table types. For old MonetDB servers which do not have the sys.table_types table, the old behavior is retained. * Implemented methods DatabaseMetadata.getProcedures() and DatabaseMetadata.getProcedureColumns(). They used to return an empty resultset. Now they return the expected Procedures and ProcedureColumns. Also getProcedureColumns() now returns a resultset with all 20 columns instead of 13 columns previously. * Method getFunctionColumns() in DatabaseMetadata used to throw an SQLException: getFunctionColumns(String, String, String, String) is not implemented. This method is now implemented and returns a resultset. * Method getFunctions() in DatabaseMetadata used to throw an SQLException: SELECT: no such column 'functions.sql' This has been corrected. It now returns a resultset as requested. * The resultsets of DatabaseMetadata methods now no longer return a value for the *_CAT columns as MonetDB does not support Catalogs. * Fixed a memory leak in MonetDatabaseMetaData.java for a static cache which kept references to closed Connection objects. * Improved JdbcClient program when presenting query data to console. It used to send an SQL catalog query for each query result column which slowed down the interactive response considerably. These additional SQL catalog queries have been eliminated. * Corrected MonetResultSet.getObject(String columnName). It no longer throws a NullPointerException in cases where internally a MonetVirtualResultSet is used. * Fixed Connection.isValid(): this method should never attempt to close the connection, even if an error has occurred. SQL * Disallow GRANT <some-user> TO <role-or-use>. Only explicitly created roles can be granted. * Extended grantable schema privileges: when a user is granted the "sysadmin" role, the user now hcan not only create schemas, but also drop schemas. * Introduced COPY INTO/ COPY FROM global privileges. These privileges allows other users than 'monetdb' to be granted the privileges to use the MonetDB bulk data loading/exporting features, i.e., COPY FROM <file> and COPY INTO <file>. Bug Fixes * 2407: Merovingian: allow binds to given ip/interface * 2815: No SRID support * 3361: constants as MAL function parameters prevent intermediate reuse * 3374: UNIQUE constraint does not set tkey property on the corresponding BAT * 3460: incomplete implementation of JDBC driver supportsConvert(), supportsConvert(int fromType, int toType) methods in MonetDatabaseMetaData.java * 3711: JDBC connection using jdbcclient.jar is very slow * 3877: MonetDBLite should allow close then re-open databases? without restarting R * 3911: Invalid connect() call in 'redirect' mode * 3920: query with DISTINCT and correlated scalar subquery in SELECT list causes Assertion failure and crash of mserver5 * 3927: COUNT( distinct my_clob ) , QUANTILE( my_double , number ) fails in dev build * 3956: MonetDBLite unable to execute LIMIT 1 statement * 3972: Drastic Memory leak of 600GBs while generating plan for Query with 25 joins * 3974: Prepared statement rel_bin.c:2378: rel2bin_project: Assertion `0' failed. * 3975: Suspicious code in store_manager() on exit path * 3978: SQL returns TypeException 'aggr.subcorr' undefined for sys.corr function * 3980: JOIN with references on both sides crashes mserver * 3981: Incorrect LEFT JOIN when FROM clause contains nested subqueries * 3983: Creation of a Foreign Key which partially maps to a primary key is accepted without a warning * 3984: Multiple paths in the .profile * 3985: ruby-monetdb-sql gem fails for negative timezone offset (USA, etc.) * 3987: Segfault on malformed csv import * 3991: MonetDBLite feature request: default monetdb.sequential to FALSE on windows * 3994: MonetDBLite dbDisconnect with shutdown=TRUE freezes the console on windows * 3995: NullPointerException when calling getObject() * 3997: calling scalar functions sys.isaUUID(str) or sys.isaUUID(uuid) fail * 3999: length() returns wrong length for strings which have spaces at the end of the string * 4010: RELEASE SAVEPOINT after ALTER TABLE crashes mserver5 * 4011: sys.sessions.user column always shows 'monetdb' * 4013: GDKextendf does not free up memory when it fails due to insufficient resources * 4015: Daemon crashes on database release command

1 0

MonetDB vs. the new MariaDB ColumnStore
by Arno Mittelbach 20 Jun '16

20 Jun '16

Hi all, MariaDB has released a first alpha version of their new ColumnStore (which is essentially InfiniDB). We have done a first tentative performance test and did a comparison of MariaDB ColumnStore vs. MonetDB. For most of the test cases, MonetDB wins hands down. If you are interested in the numbers, have a look at https://reportserver.net/blog/2016/06/20/mariadb-columnstore-vs-innodb-vs-m… Best Regards, Arno

1 0

NULL value with embedded py
by Stefano Fioravanzo 20 Jun '16

20 Jun '16

Hello, I am creating a table using the result of an embedded py function. The result of the function is a dictionary composed of numpy arrays. The function and the CREATE TABLE work well. Inside the arrays of the dictionary, I have a few np.nan values, they seem to be recognized by monet during the creation of the table. In fact, when I query the whole table, I can see ‘null’ in the proper places. But if I try to compare against these null values, I get nothing (SELECT * FROM table where field is null). After a few trials, it seems to me that monet understands that np.nan is a NULL value, but then when it inserts the value into the table the internal representation (?) is somewhat different form a ‘normal’ NULL value. Any suggestion? Thanks, Stefano

3 2

Recover an inconsistent database?
by Vijay Krishna 20 Jun '16

20 Jun '16

Hi, I'm getting the following error while starting a database. *!FATAL: Logger_new: inconsistent database, catalog does not exist* Is there any means to recover this database with data and start it? Or, the only way is to create a new database? (P.S.: There is no backup of data) Thanks & Regards, Vijay.

1 0

aggregation with complex function
by Stefano Fioravanzo 16 Jun '16

16 Jun '16

Hello, I have an embedded python function which, given two integer parameters, after doing some stuff returns a table. I would like to group by a table by these two input parameter, and for each resulting group call the function. The result would be the concatenation of all the tables returned by the function. I understand that a thing like this would be very easy if the function returned just one value e.g.: select my_aggr_function(value1, value2) from table group by value1, value2 But how can I do this if my_aggr_function returns multiple rows?? Thank you, Stefano

4 15

Dynamic Queries
by Ariel Abadi 16 Jun '16

16 Jun '16

Hi Im trying to create a Procedure to create (or not) tables dynamically. Does Monet can do something like the following?? CREATE PROCEDURE(pdate date, tname varchar(10)) BEGIN SET TABLENAME = CONCAT(tname, '_', YEAR(pdate)*100+MONTH(pdate)); CREATE TABLE IF NOT EXIST TABLENAME; END Thanks in advance! Rgds

3 2

Issue with bugzilla
by Roberto Cornacchia 14 Jun '16

14 Jun '16

There seems to be an issue with Bugzilla. If I make a bug report and select Jul2015-SP4, after saving the version is Oct2014-SP2. The first few times that this happened I thought it had been my mistake in selecting the version, but now I'm sure it wasn't. Roberto

2 1

Update JOIN?
by Alberto Ferrari 14 Jun '16

14 Jun '16

Is it possible to Update Join in Monet? I need something like this... (just an example) UPDATE MY_TABLE AS a, ( SELECT (SUM( t.mins * t.acpm ) / t.rcpmax) AS calc1 (SUM( t.mins * t.acxh ) / t.acxmax) AS calc2 FROM MY_TABLE AS t GROUP BY t.source, t.name_id ) AS q SET a.awc = q.calc1, a.lcr = q.calc2 WHERE a.source = q.source AND a.name_id = q.name_id When I try to execute this type of query, I get this error... syntax error, unexpected AS, expecting SET in: "update sb_rmsv.rmsv_nts_nt_sum_report as"

4 5

Announcement: New release candidate of MonetDB suite
by Sjoerd Mullender 14 Jun '16

14 Jun '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 This is a heads up announcement. We at CWI and MonetDB have prepared a release candidate for the Jun2016 release. Usually we just foist a new release upon you, our unsuspecting users, but this time, because there are so many changes, we thought it better to give you a little warning. See http://dev.monetdb.org/downloads/testing/ for the release and the preliminary release notes [1]. Among the changes is a complete overhaul of the MonetDB-geom module. If you use that, make very sure you have your backups in order. Also note that since this new version depends on an up-to-date version of the geos library, there is no geom module available for older Linux releases (in particular, Debian 7 (wheezy), Ubuntu 12.04 LTS (Precise Pangolin) and Ubuntu 14.04 LTS (Trusty Tahr), and RedHat Enterprise Linux (CentOS and Scientific Linux) 6). The plan is to turn this release candidate into an actual release before the end of this month (June 2016). If you find a bug in this release candidate, please let us know on the bug tracker [2]. We will then determine whether to go ahead with the release (and hopefully fix the bug in the next bug-fix release) or postpone the release and fix the bug first. [1] https://dev.monetdb.org/downloads/testing/relnotes.html [2] http://bugs.monetdb.org (a.k.a. https://www.monetdb.org/bugzilla/) - -- Sjoerd Mullender -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJXX/EsAAoJEISMxT6LrWYg+AcIAIgIST/dUrL8fDjwp3/G41zc 4g9xWbSxIE/+nZLjlHM6Onr4sETceOVdDVuxUGoIbeu9+DZNQqEakUUrABrldvLx ie/kbNINtdntFvfF3W5Lyo/lWhel3IUZZhEXvsBT9JCSfg5kiLKYr/CPqw5QgOcU XgccIn7RAZYovfVoLo7jLxiMLeT9DJp74gy1OkjrIBwQ/GPVam/CzaO5Ungu0dRY cZiqoRhW1GEtUQHABq0O9J/gJcHfFbSlwDeQcnTouybO320gBRISYqgV8aTt0RdU /MVVkdOU3hLsShjAgWinRbQPNVwCLcZNAfxER9q4Z9tt15n/3E6WtPeKo2Bv030= =rnCK -----END PGP SIGNATURE-----

1 0

Announcement: New release candidate of MonetDB suite
by Sjoerd Mullender 14 Jun '16

14 Jun '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 This is a heads up announcement. We at CWI and MonetDB have prepared a release candidate for the Jun2016 release. Usually we just foist a new release upon you, our unsuspecting users, but this time, because there are so many changes, we thought it better to give you a little warning. See http://dev.monetdb.org/downloads/testing/ for the release and the preliminary release notes [1]. Among the changes is a complete overhaul of the MonetDB-geom module. If you use that, make very sure you have your backups in order. Also note that since this new version depends on an up-to-date version of the geos library, there is no geom module available for older Linux releases (in particular, Debian 7 (wheezy), Ubuntu 12.04 LTS (Precise Pangolin) and Ubuntu 14.04 LTS (Trusty Tahr), and RedHat Enterprise Linux (CentOS and Scientific Linux) 6). The plan is to turn this release candidate into an actual release before the end of this month (June 2016). If you find a bug in this release candidate, please let us know on the bug tracker [2]. We will then determine whether to go ahead with the release (and hopefully fix the bug in the next bug-fix release) or postpone the release and fix the bug first. [1] https://dev.monetdb.org/downloads/testing/relnotes.html [2] http://bugs.monetdb.org (a.k.a. https://www.monetdb.org/bugzilla/) - -- Sjoerd Mullender -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJXX/EsAAoJEISMxT6LrWYg+AcIAIgIST/dUrL8fDjwp3/G41zc 4g9xWbSxIE/+nZLjlHM6Onr4sETceOVdDVuxUGoIbeu9+DZNQqEakUUrABrldvLx ie/kbNINtdntFvfF3W5Lyo/lWhel3IUZZhEXvsBT9JCSfg5kiLKYr/CPqw5QgOcU XgccIn7RAZYovfVoLo7jLxiMLeT9DJp74gy1OkjrIBwQ/GPVam/CzaO5Ungu0dRY cZiqoRhW1GEtUQHABq0O9J/gJcHfFbSlwDeQcnTouybO320gBRISYqgV8aTt0RdU /MVVkdOU3hLsShjAgWinRbQPNVwCLcZNAfxER9q4Z9tt15n/3E6WtPeKo2Bv030= =rnCK -----END PGP SIGNATURE-----

1 0

copy into from csv with delimiters in quoted field
by Frank Tetzel 10 Jun '16

10 Jun '16

Hello, i have an issue with the "copy into" statement and a csv file. In the documentation[1] it says: "Field and record separators can be embedded in quoted fields." This does not work in v11.21.19. example.csv: 1,"\,\,",42,, 2,",,",23,, create table issue(id int, str varchar(32), val int, empty int); copy into issue from 'example.csv' using delimiters ',' NULL AS ''; The first line with the escaped delimiters works, the second does not: Failed to import table Leftover data '23,,' Is this a bug i should report or am i doing something wrong with the above statements? Regards, Frank [1] https://www.monetdb.org/Documentation/Manuals/SQLreference/CopyInto

2 2

Segfault when using current_date() in SELECT query
by Dennis Pallett 10 Jun '16

10 Jun '16

Hi all, We seem to be getting a segfault on our databases when we try to use the CURRENT_DATE() function in a simple select query. For example: SELECT CURRENT_DATE(), groep.* FROM groep; Is this a (known) bug in MonetDB? Best regards, Dennis Pallett

1 0

Why aren't hashes preserved by joins?
by Roberto Cornacchia 10 Jun '16

10 Jun '16

I noticed a difference between - a hash-based string selection from a persistent, read-only table - a hash-based join on the same table and same column They both build a hash on the same string column (verified with gdb), but the select can reuse the hash (second call is almost free), while the join keeps rebuilding the hash. Is this expected? Roberto

2 4

function IFNULL equivalent?
by Alberto Ferrari 09 Jun '16

09 Jun '16

Hello everybody, I know this question was alreaday made, but I can´t find an answer... Is there a function equivalent to mysql IFNULL() ? If no... has someone created any similar user function? Thanks in advanced

2 2

Constant segmentation faults
by Nikita Salnikov-Tarnovski 07 Jun '16

07 Jun '16

Hello. I am evaluating MonetDB to use in our project, but so far have some real problems with MonetDB constantly having segmentation faults. Here is my setup: I am using AWS EC2 instance with 16GB of RAM, monetdb farm is location on ESB volume. Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://ip-10-0-0-47:50000/mysql-copy' In there I have several tables, among them is one table I am most interested in, problemsview, with total columnsize of 120GB, largest column is around 11GB. The moment I execute “select count(1) from problemsview;” I get segmentation fault: mserver5[1300]: segfault at 0 ip 00007f86a0ca0712 sp 00007f869b6faa80 error 4 in lib_sql.so[7f86a0c72000+199000] Next select * from problemsview limit 10; wait for 20-30 seconds, and then segmentation fault: mserver5[1313]: segfault at 18 ip 00007f68cf2a0726 sp 00007f68ce14ea80 error 4 in lib_sql.so[7f68cf272000+199000] So essentially I cannot do anything, I always get segmentation fault. Can this be fixed somehow? Thank you in advance, Nikita Salnikov-Tarnovski Plumbr Founder and Master Developer @JavaPlumbr/@iNikem

3 5

segmentation fault with loopback query
by Stefano Fioravanzo 06 Jun '16

06 Jun '16

Hello, I am trying to execute a loopback query in embedded python: CREATE FUNCTION myfunc() RETURNS TABLE(n STRING) LANGUAGE PYTHON { res = _conn.execute("SELECT max(d) FROM mytable;") result = dict() result['n'] = str(res) return result }; When accessing the object ‘res’, mserver5 outputs ‘segmentation fault’ and exits. I think that the issue is in using the max() function in the query, if I leave that out everything works fine. The function: CREATE FUNCTION myfunc() RETURNS TABLE(n STRING) LANGUAGE PYTHON { res = _conn.execute("SELECT d FROM mytable;") result = dict() result['n'] = str(res) return result }; outputs a correct dictionary. Stefano

3 5

group_concat in MAL not working due to algebra.uselect
by Puthick Hok 04 Jun '16

04 Jun '16

Hi, Last year, I was looking for the MAL definition of group_concat and Nik Schuiling gave me the MAL definition which is below. We compile MonetDB from source and this definition was working on MonetDB-11.19.9 (Oct2014-SP2). Since this version, there has been a number of newer versions of MonetDB. So, I was trying to upgrade our system to the latest version of MonetDB (11.21.13). After the installation of the newer version, I ran: select "DataSetId", group_concat("FieldName") from datasetmarkermeta group by "DataSetId"; from mclient interface and got the following error message: TypeException:kddart.subgroup_concat[5]:'algebra.uselect' undefined in: TIDs:any := algebra.uselect(g:bat[:oid,:oid],grpid:oid); MALException:kddart.subgroup_concat[0]:Error in cloned function TypeException:user.s2_1[15]:'kddart.subgroup_concat' undefined in: X_27:bat[:oid,:str] := kddart.subgroup_concat(X_26:bat[:oid,:str],X_16:bat[:oid,:oid],r1_16:bat[:oid,:oid],true:bit); program contains errors It says algebra.uselect function is not defined. How do I fix this problem? group_concat MAL definition start ------------------------------------------- module kddart; command group_concat(b:bat[:oid,:str]):str address KDDARTgroup_concat comment "Mal kddart"; function group_concat(b:bat[:oid, :str]):str; value := ""; barrier (h, t) := iterator.new(b); value := value + t; value := value + ""; redo (h, t) := iterator.next(b); exit (h, t); return value; end group_concat; function subgroup_concat(b:bat[:oid,:any_1],g:bat[:oid,:oid],e:bat[:oid,:any_2],skip_nils:bit) :bat[:oid,:str]; nw := aggr.count(e); nl := calc.lng(nw); bn := bat.new(:oid, :str, nl); # check 'e' has some values - this is the list of group IDs in the head # 'g' is the group to the data BAT head ID mapping # 'b' is the data column BAT that we are aggregating over barrier (grpid, t) := iterator.new(e); # select GID from TID->GID map to get matching TIDs TIDs := algebra.uselect(g,grpid); # get DATA for matching TIDs b_data := algebra.kintersect(b,TIDs); # aggregate grpval := group_concat(b_data); # Store the result for this group bat.insert(bn, grpid, grpval); redo (grpid, t) := iterator.next(e); exit (grpid, t); return bn; end subgroup_concat;

2 3

MonetDB-11.21.19 Compilation failure on JDBC tests
by Ashish Kumar Singh 04 Jun '16

04 Jun '16

All, I am trying to compile monetdb MonetDB-11.21.19 from sources where its failing with error below: prepare: [echo] Debug is true, optimise is false compile_mcl: [echo] Compiling MCL driver: [copy] Copying 1 file to /opt/MonetDB-11.21.19/java/src/main/java/nl/cwi/monetdb/jdbc [echo] Configured build for the 2.19 (Liberica) edition driver compile_jdbc: [echo] Compiling JDBC driver [javac] Compiling 1 source file to /opt/MonetDB-11.21.19/java/tests/jar_jdbctests [javac] This version of java does not support the classic compiler; upgrading to modern jar_jdbc: BUILD FAILED /opt/MonetDB-11.21.19/java/tests/build.xml:75: The following error occurred while executing this line: /opt/MonetDB-11.21.19/java/build.xml:96: java.io.FileNotFoundException: /opt/MonetDB-11.21.19/java/tests/jar_jdbctests/META-INF/services/java.sql.Driver (No such file or directory) Below is my configure output : I am compiling with JDK 8 latest , and ant 1.7 MonetDB is configured as follows: * Compilation specifics: Host: x86_64-unknown-linux-gnu Compiler: gcc (gcc-4.4.7; gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17) Copyright (C) 2010 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.) CFLAGS: -g -O2 $(X_CFLAGS) X_CFLAGS: LDFLAGS: * Enabled/disabled build options: strict is disabled (by default) assert is disabled (by default) debug is disabled (by default) optimize is disabled (by default) developer is disabled (by default) instrument is disabled (by default) profile is disabled (by default) * Enabled/disabled components: gdk is enabled monetdb5 is enabled sql is enabled geom is disabled (geos-config not found) console is enabled gsl is disabled (gsl library not found) fits is disabled (cfitsio library not found) netcdf is disabled (netcdf library not found) microhttpd is disabled (by default) rintegration is disabled (libR library not found) odbc is disabled (unixODBC library required for ODBC driver) jdbc is enabled control is enabled testing is enabled developer is disabled (by default) strict is disabled (by default) debug is disabled (by default) assert is disabled (by default) optimize is disabled (by default) * Available features/extensions: atomic_ops = no (atomic_ops library not found) bz2 = no (bz2 library not found) curl = no (libcurl not found) getaddrinfo = yes gsl = no (gsl library not found) java = yes java_control = yes java_jdbc = yes libxml2 = yes openssl = yes pcre = yes perl = yes pthread = yes python2 = yes python3 = no (Python 3 executable not found) readline = no (readline header file not found) rubygem = no (no rubygem executable found) samtools = no (samtools/bam.h header not found) sphinxclient = no (sphinxclient.h header not found) unixodbc = no (odbcinst.h not found) valgrind = no (by default) zlib = yes * Important options: OID size: 64 bits largest integer type: 128 bits largest decimal type: 38 digits

1 0

query json from embedded py
by Stefano Fioravanzo 01 Jun '16

01 Jun '16

Hello, I am trying to query a table with a JSON field with a loopback query in embedded python. I get this error: SQL Query Failed: MALException:pyapi.eval:Unsupported SQL Type: son Is JSON not supported? Do I have to store the JSON object as a string in case I want to query it from python? Thank you, Stefano

2 1

Query database inside embedded R
by Stefano Fioravanzo 01 Jun '16

01 Jun '16

Hello, We are trying to transition from a Postgres database to MonetDB. We had a few R function calls, so we decided to try embedded R. One of our R functions is designed to query the postgres database to get some more data to process. The problem, now, is how to integrate this function in Monet. We are using embeddedR to reduce latency, so using MonetDB.R inside the embedded function (if it is even possible) to query the database would be a total waste. Clearly the ultimate solution would be to make all the queries outside the function, but it would be really really hard to achieve this result given how the function is designed. What I am asking is the is some solution to this, like having some way to query the database inside embedded R avoiding the overhead of MonetDB.R. Thank you, Stefano

2 5

Query database inside embedded R
by Stefano Fioravanzo 31 May '16

31 May '16

Hello, We are trying to transition from a Postgres database to MonetDB. We had a few R function calls, so we decided to try embedded R. One of our R functions is designed to query the postgres database to get some more data to process. The problem, now, is how to integrate this function in Monet. We are using embeddedR to reduce latency, so using MonetDB.R inside the embedded function (if it is even possible) to query the database would be a total waste. Clearly the ultimate solution would be to make all the queries outside the function, but it would be really really hard to achieve this result given how the function is designed. What I am asking is the is some solution to this, like having some way to query the database inside embedded R avoiding the overhead of MonetDB.R. Thank you, Stefano

1 0

embedded R function not working
by Stefano Fioravanzo 30 May '16

30 May '16

Hello, I have the following query: CREATE TABLE mytable AS ( SELECT <fields….> FROM table1 AS a, my_R_function(a.field1, ...) AS asd WHERE <some conditions> ) WITH DATA; MonetDB says: SELECT: no such column ‘a.field1’ The issue is in how I call the R function, since when I remove it everything’s just fine. My aim is to call the function for each row of table1. The new table will have a couple of fields from table1 and a bunch of other fields computed from the R function. How am I supposed to do that? Thank you, Stefano

2 1

MonetDB and KNIME compatibility
by Roman S 26 May '16

26 May '16

Hello. Maybe someone have experience with this kind of problem: I’ve downloaded drivers from http://dev.monetdb.org/downloads/Java/Jul2015-SP4/ <http://dev.monetdb.org/downloads/Java/Jul2015-SP4/> added them to KNIME and now, when i’m trying to make something more difficult than just counting rows in table, I see an error: ERROR Database Reader 0:3 SQL Exception reading Object of type "2005": Operation getCharacterStream() currently not supported - all further errors are suppressed and reported on debug level only ERROR Database Reader 0:3 Execute failed: Cell at index 1 is null!

1 0

storing -1 in a tinyint
by Lynn Carol Johnson 25 May '16

25 May '16

Hi all - I apologize for all the emails, but I am banging on monetdb a lot lately. Today I've run into a problem storing a -1 in a "tinyint" column. My java code writes bytes to a binary file. This file will be loaded via binary bulk load into a "tinyint" column. The values in this column range from -1 to 3. I am writing the negative value to the byte file as below: byte missingData = (byte)-1; writer.writeByte(missingData); The data loads correctly into monetdb, and I can print it: sql>\d minusonetest CREATE TABLE "sys"."minusonetest" ( "chr" INTEGER, "pos" INTEGER, "coelorachis" TINYINT ); sql> sql>select * from minusonetest where chr=2 limit 10; +------+------+-------------+ | chr | pos | coelorachis | +======+======+=============+ | 2 | 1 | -1 | | 2 | 2 | -1 | | 2 | 3 | -1 | | 2 | 4 | -1 | | 2 | 5 | -1 | | 2 | 6 | -1 | | 2 | 7 | -1 | | 2 | 8 | -1 | | 2 | 9 | -1 | | 2 | 10 | -1 | +------+------+-------------+ 10 tuples (10.7s) sql> BUT, when I run the select statement below, it tosses me out of the DB and prints errors to the log file: select * from minusone test where chr=2 and coelorachis > -1; The select fails for any value in the "coelorachis > x" clause above (i.e., > -1, > 0, > 1, etc) If I run the above select and have NOT written any negative numbers to the column, all is good. It only fails if there are negative numbers stored in the tinyint column. The merovingian.log file has these messages: 2016-05-25 12:47:23 ERR testDB[21730]: *** Error in `/usr/bin/mserver5': free(): invalid next size (fast): 0x00007fb0ac003920 *** 2016-05-25 12:47:23 ERR testDB[21730]: ======= Backtrace: ========= 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libc.so.6(+0x7d023)[0x7fb0df3bc023] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libbat.so.12(GDKfree+0x13)[0x7fb0e1b21233] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libbat.so.12(BATsubunique+0x18a9)[0x7fb0e1d23f69] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libbat.so.12(BATsubunique+0xbe7)[0x7fb0e1d232a7] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libbat.so.12(BATimprints+0x581)[0x7fb0e1cef791] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libbat.so.12(BATsubselect+0x4ecf)[0x7fb0e1ac532f] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libbat.so.12(BATthetasubselect+0x14b)[0x7fb0e1ac78eb] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libmonetdb5.so.19(ALGthetasubselect2+0x1c4)[0x7fb0e210db64] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libmonetdb5.so.19(runMALsequence+0xdeb)[0x7fb0e20ac55b] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libmonetdb5.so.19(+0x45e5b)[0x7fb0e20ade5b] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libpthread.so.0(+0x7dc5)[0x7fb0df707dc5] 2016-05-25 12:47:23 ERR testDB[21730]: /lib64/libc.so.6(clone+0x6d)[0x7fb0df43528d] 2016-05-25 12:47:23 ERR testDB[21730]: ======= Memory map: ======== 2016-05-25 12:47:23 ERR testDB[21730]: 00400000-00405000 r-xp 00000000 08:03 22036485 /usr/bin/mserver5 2016-05-25 12:47:23 ERR testDB[21730]: 00604000-00605000 r--p 00004000 08:03 22036485 /usr/bin/mserver5 2016-05-25 12:47:23 ERR testDB[21730]: 00605000-00606000 rw-p 00005000 08:03 22036485 /usr/bin/mserver5 2016-05-25 12:47:23 ERR testDB[21730]: 00606000-00608000 rw-p 00000000 00:00 0 2016-05-25 12:47:23 ERR testDB[21730]: 00892000-01e3e000 rw-p 00000000 00:00 0 [heap] 2016-05-25 12:47:23 ERR testDB[21730]: 7fabab28a000-7fabab29f000 r-xp 00000000 08:03 22020111 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 2016-05-25 12:47:23 ERR testDB[21730]: 7fabab29f000-7fabab49e000 ---p 00015000 08:03 22020111 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 2016-05-25 12:47:23 ERR testDB[21730]: 7fabab49e000-7fabab49f000 r--p 00014000 08:03 22020111 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 2016-05-25 12:47:23 ERR testDB[21730]: 7fabab49f000-7fabab4a0000 rw-p 00015000 08:03 22020111 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 Any ideas? There is no problem if I use Byte.MIN_VALUE instead of -1. This translates to NULL and is selectable. Lynn

1 1

I WAS WRONG ! Re: additional details on IS NULL problem
by Lynn Carol Johnson 24 May '16

24 May '16

After all this, it seems I was doing something stupid. I had originally used Float.MIN_VALUE, which is the smallest POSITIVE minimum value. I also tried hard coding -3.40282e+38, but this had issues as Stephan pointed out. What I neglected to try, and what works for float is public static float flt_nil = -Float.MAX_VALUE; Sorry for all the confusion, but I learned something. Null values for float types DO work. Thanks for your patience and help - Lynn On 5/24/16, 10:44 AM, "users-list on behalf of Lynn Carol Johnson" <users-list-bounces+lcj34=cornell.edu(a)monetdb.org on behalf of lcj34(a)cornell.edu> wrote: >Yes, we’re using float as 32 bit, storing into a real. > >Still no sunshine with your suggestions below. Java doesn’t allow me to >assign > public static float flt_nil = (float) >-340282346638528859811704183484516925440; >It complains “the literal -340282346638528859811704183484516925440 of type >int is out of range”. This isn’t surprising as 32 bits will only hold up >to -2^31 (-2,147,483,648) > >Using static float flt_nil = (float) 0xff7fffff; gives me the value >-8388609.0, which is incorrect. > >Does storing null work for you using “float” types? Is this just a >java-to-C issue? Still looks like we’ll need to pick a default other than >NULL for our “real” data. > >Thanks - Lynn > > >On 5/24/16, 9:53 AM, "users-list on behalf of Stefan Manegold" ><users-list-bounces+lcj34=cornell.edu(a)monetdb.org on behalf of >Stefan.Manegold(a)cwi.nl> wrote: > >>Hi Lynn, >> >>public static float flt_nil = (float) -3.40282e+38; >> >>does not necessarily create the exact bit pattern >>as C macro -FLT_MAX, mainly as "-3.40282e+38" is only an >>approximation of the exact value: >>"-340282346638528859811704183484516925440" >> >>You can either try >> >>public static float flt_nil = (float) >>-340282346638528859811704183484516925440; >> >>or use the actual bit pattern in hex notation, e.g., >> >>public static float flt_nil = (float) 0xff7fffff; >> >>Please take my Java(?) code snippets with a grain of salt; >>I'm everything but a Java expert --- or better, I'm a Java >>illiterate ... >> >>Be also aware that >>C type float, MAL typr :flt and SQL type REAL are 32-bit, >>while >>C type double, MAL type :dbl, and SQL types FLOAT & DOUBLE are 64-bit >> >>Hence, for 64-bit types you'd need to use one of >> >>public static double dbl_nil = (double) >>-179769313486231570814527423731704356798070567525844996598917476803157260 >>7 >>8002853876058955863276687817154045895351438246423432132688946418276846754 >>6 >>7035375169860499105765512820762454900903893289440758685084551339423045832 >>3 >>6903222948165808559332123348274797826204144723168738177180919299881250404 >>0 >>26184124858368; >> >>public static double dbl_nil = (double) 0xffefffffffffffff; >> >> >>Hope this helps ... >> >>Best, >>Stefan >> >>----- On May 24, 2016, at 3:42 PM, Lynn Carol Johnson lcj34(a)cornell.edu >>wrote: >> >>> Thanks, Stephan. >>> >>> Short response good news: using int_nil = ‹2147483648 (-2^31) is >>> translated properly to NULL for int values. The IS NULL and IS NOT >>>NULL >>> queries are fine. >>> >>> Short response bad news: I am unable to get NULL to work for Float >>> regardless of what value I used. Details below: >>> >>> I replaced my Float.MIN_VALUE with flt_nil defined as below : >>> >>> public static float flt_nil = (float) -3.40282e+38; >>> >>> Ran again with just 50 rows (created as binary files). At this point >>>my >>> belief is NULL does not work at all for float with binary files. There >>>are >>> 16 of the >>> 50 values for column amesbyd_weir that are null, but I can only select >>> those values if I check for the column < -3.40282e+38. Here are the >>>sql >>> results: >>> >>> sql>select count(amesbyd_weir) from weirtestnull; >>> +------+ >>>| L1 | >>> +======+ >>>| 50 | >>> +------+ >>> 1 tuple (1.933ms) >>> sql>select chr,pos,amesbyd_weir from weirtestnull where amesbyd_weir IS >>> NULL; >>> +-----+-----+--------------+ >>>| chr | pos | amesbyd_weir | >>> +=====+=====+==============+ >>> +-----+-----+--------------+ >>> 0 tuples (1.291ms) >>> sql>select chr,pos,amesbyd_weir from weirtestnull where amesbyd_weir = >>> -3.40282e+38; >>> +-----+-----+--------------+ >>>| chr | pos | amesbyd_weir | >>> +=====+=====+==============+ >>> +-----+-----+--------------+ >>> 0 tuples (0.528ms) >>> sql> >>> sql>select chr,pos,amesbyd_weir from weirtestnull where amesbyd_weir < >>> -3.40282e+38; >>> +------+--------+-----------------+ >>>| chr | pos | amesbyd_weir | >>> +======+========+=================+ >>>| 10 | 228864 | -3.40282e+38 | >>>| 10 | 228962 | -3.40282e+38 | >>>| 10 | 228972 | -3.40282e+38 | >>>| 10 | 229011 | -3.40282e+38 | >>>| 10 | 229025 | -3.40282e+38 | >>>| 10 | 229187 | -3.40282e+38 | >>>| 10 | 229367 | -3.40282e+38 | >>>| 10 | 229405 | -3.40282e+38 | >>>| 10 | 229444 | -3.40282e+38 | >>>| 10 | 229593 | -3.40282e+38 | >>>| 10 | 229615 | -3.40282e+38 | >>>| 10 | 229671 | -3.40282e+38 | >>>| 10 | 229829 | -3.40282e+38 | >>>| 10 | 230000 | -3.40282e+38 | >>>| 10 | 230052 | -3.40282e+38 | >>>| 10 | 230120 | -3.40282e+38 | >>> +------+--------+-----------------+ >>> 16 tuples (2.999ms) >>> sql> >>> >>> >>> NOTE: Dumping the 50 row database to a .txt file, dropping the table >>>and >>> reloading into monetdb did NOT fix it as it did when I used Float.NaN >>> values. That is actually irrelevant for us as our db tables are large >>> (2.1 Billion rows in 1 table, 80M in another) and as we add additional >>> columns, binary loading is the only way to go. We love the binary load >>> option, and have been very happy with the speed of monetdb processing. >>> >>> For now, we¹re going to follow PLINK convention and use -99 for our >>>Float >>> null values. >>> >>> Thanks - Lynn >>> >>> >>> On 5/23/16, 4:06 PM, "users-list on behalf of Stefan Manegold" >>> <users-list-bounces+lcj34=cornell.edu(a)monetdb.org on behalf of >>> Stefan.Manegold(a)cwi.nl> wrote: >>> >>>>Hi Lynn, >>>> >>>>thanks for diving into this! >>>> >>>>The information that you loaded your data using from binary files using >>>>copy *binary* into is crucial for us to understand and investigate the >>>>problem! >>>> >>>>In fact, I recently discovered and fixed a similar bug; cf., >>>>https://www.monetdb.org/bugzilla/show_bug.cgi?id=3937 >>>> >>>>This fix is in the Jul2013-SP3 bug-fix release. >>>>Hence, I'm wondering why you (still) experience this (or a similar) >>>>problem. >>>> >>>>Of what type are your data / columns? >>>>Integer? decimal? floating point? string? >>>> >>>>If more then one type, do you experience these problems with all types >>>>or just with some? >>>>If only some, which? >>>> >>>>If the problem occurs only with floating point numbers, >>>>are you sure you're using the exact NULL representation that MonetDB >>>>uses? >>>> >>>>Otherwise a potential source of the problem --- I'm just speculating >>>>--- >>>>could be the following: >>>> >>>>You might have "non-valid" floating point numbers like NaN or INF in >>>>your >>>>data >>>>that are not known to SQL and hence MonetDB. >>>>If so, >>>>When parsed from text, MonetDB does (seems to?) silently turn them into >>>>(MonetDB's >>>>internal representation of) NULL. >>>>However, when loding your binary data, MonetDB stores these values >>>>"as-is" >>>>not recognizing them as NULL; then processing does not recognize them >>>>as >>>>NULL, either; >>>>only rendering produces NULL from the "non-valid" internal >>>>representation. >>>> >>>>For the correct internal representation of NULL values in MonetDB per >>>>type see >>>>https://www.monetdb.org/wiki/MonetDB_type_system >>>> >>>>If my speculation is true, we'd need to consider making our copy binary >>>>into >>>>bulk-loading for floating point numbers (more) fail-save by actually >>>>scanning >>>>each column and checking for "invalid" floating point number --- >>>>whatever >>>>that >>>>might cost ... >>>> >>>>Best, >>>>Stefan >>>> >>>>----- On May 23, 2016, at 8:37 PM, Lynn Carol Johnson lcj34(a)cornell.edu >>>>wrote: >>>> >>>>> Hi Anthony - >>>>> >>>>> I¹ve narrowed down the problem. The bug isn¹t related to size, but >>>>>rather how >>>>> the table data was created/loaded. Loading from a .txt file gives >>>>>good >>>>> behavior, loading from binary does not. >>>>> >>>>> Normally I create binary files for each column and use the binary >>>>>bulk >>>>>loader to >>>>> load. That is what was failing. When I first tested with 50 columns, >>>>>I >>>>>created >>>>> the test file from sql via the command >>>>> copy (select * Š) into Œ/workdir/ŠkellyFirst50.txt using delimiters >>>>>Œ\t¹,¹\n¹; >>>>> >>>>> Then I created a new table called kellyFirst50 and used the ³COPY >>>>>INTO >>>>> kellyfirst50 from Œ/workdirŠkellyFirst50.txt¹ using delimiters >>>>>Œ\t,¹\n¹; ² >>>>> command. Checking for ³IS NULL² worked, so I thought it was a size >>>>>problem. >>>>> >>>>> But, when trying to figure out at what point the file size mattered, >>>>>I >>>>>ran java >>>>> scripts to create binary files of ever decreasing sizes, then loaded >>>>>them via >>>>> the COPY BINARY commands. Checking for IS NULL always failed here, >>>>>even >>>>>when I >>>>> only loaded 50 values. >>>>> >>>>> Here are the results when loading from binary files into monetdb. >>>>>Using >>>>>IS NOT >>>>> NULL works, but IS NULL does not: >>>>> >>>>> >>>>> >>>>> sql>\</workdir/lcj34/monetdbFiles/isNull_test/create_weirFSTTest.sql >>>>> >>>>> operation successful >>>>> >>>>> sql>\</workdir/lcj34/monetdbFiles/isNull_test/copy_weirFSTTest.sql >>>>> >>>>> 50 affected rows >>>>> >>>>> >>>>> >>>>> sql> >>>>> >>>>> >>>>> >>>>> >>>>> sql>select count(*) from weirtestnull; >>>>> >>>>> +------+ >>>>> >>>>>| L1 | >>>>> >>>>> +======+ >>>>> >>>>>| 50 | >>>>> >>>>> +------+ >>>>> >>>>> 1 tuple (2.014ms) >>>>> >>>>> sql>select count(amesbyd_weir) from weirtestnull; >>>>> >>>>> +------+ >>>>> >>>>>| L1 | >>>>> >>>>> +======+ >>>>> >>>>>| 50 | >>>>> >>>>> +------+ >>>>> >>>>> >>>>> >>>>> 1 tuple (0.865ms) >>>>> >>>>> sql>select count(amesbyd_weir) from weirtestnull where amesbyd_weir >>>>>IS >>>>>NOT NULL; >>>>> >>>>> +------+ >>>>> >>>>>| L1 | >>>>> >>>>> +======+ >>>>> >>>>>| 34 | >>>>> >>>>> +------+ >>>>> >>>>> 1 tuple (3.005ms) >>>>> >>>>> >>>>> >>>>> sql> >>>>> >>>>> sql>select * from weirtestnull where amesbyd_weir IS NULL; >>>>> >>>>> >>>>>+-----+-----+---------------+--------------+--------------+----------- >>>>>- >>>>>-- >>>>>--+--------------------+------------------+----------------+ >>>>> >>>>>| chr | pos | amesbycn_weir | amesbyd_weir | amesbyf_weir | >>>>>amesbynam_weir | >>>>>| amesbyturkeyp_weir | dentbyflint_weir | dentbynam_weir |> >>>>> >>>>> >>>>>+=====+=====+===============+==============+==============+=========== >>>>>= >>>>>== >>>>>==+====================+==================+================+ >>>>> >>>>> >>>>>+-----+-----+---------------+--------------+--------------+----------- >>>>>- >>>>>-- >>>>>--+--------------------+------------------+----------------+ >>>>> >>>>> 0 tuples (4.041ms) !4 columns dropped! >>>>> >>>>> >>>>> >>>>> sql> >>>>> >>>>> >>>>> Dump the data into a .txt file: >>>>> >>>>> >>>>> >>>>> sql>copy (select * from weirtestnull) into >>>>> '/workdir/db_outfiles/weirtestnull.txt' using delimiters '\t','\n'; >>>>> >>>>> 50 affected rows (3.809ms) >>>>> >>>>> sql> >>>>> >>>>> Now drop the table, recreate it, but this time load from the txt file >>>>>created >>>>> above: >>>>> >>>>> >>>>> >>>>> sql>drop table weirtestnull; >>>>> >>>>> operation successful (1.948ms) >>>>> >>>>> sql>\</workdir/lcj34/monetdbFiles/isNull_test/create_weirFSTTest.sql >>>>> >>>>> operation successful >>>>> >>>>> sql>copy into weirtestnull from >>>>>'/workdir/db_outfiles/weirtestnull.txt' >>>>>using >>>>> delimiters '\t','\n'; >>>>> >>>>> 50 affected rows (57.643ms) >>>>> >>>>> sql> >>>>> >>>>> >>>>> Re-run the commands I now get correct values for IS NULL: >>>>> >>>>> >>>>> >>>>> sql>select count(*) from weirtestnull; >>>>> >>>>> +------+ >>>>> >>>>>| L1 | >>>>> >>>>> +======+ >>>>> >>>>>| 50 | >>>>> >>>>> +------+ >>>>> >>>>> 1 tuple (1.851ms) >>>>> >>>>> sql>select count(amesbyd_weir) from weirtestnull; >>>>> >>>>> +------+ >>>>> >>>>>| L1 | >>>>> >>>>> +======+ >>>>> >>>>>| 34 | >>>>> >>>>> +------+ >>>>> >>>>> 1 tuple (0.983ms) >>>>> >>>>> sql> >>>>> >>>>> sql>select * from weirtestnull where amesbyd_weir IS NULL; >>>>> >>>>> >>>>>+------+--------+------------+------------+------------+------------+- >>>>>- >>>>>-- >>>>>---------+-------------+-------------+-------------+-------------+---- >>>>>- >>>>>-- >>>>>------+ >>>>> >>>>>| chr | pos | amesbycn_w | amesbyd_we | amesbyf_we | amesbynam_ | >>>>>amesbyturke | >>>>>| dentbyflint | dentbynam_w | dentbyturke | flintbynam_ | flintbyturk >>>>>|> >>>>> >>>>> : : : eir : ir : ir : weir : yp_weir : _weir : eir : ypen_weir : weir >>>>>: >>>>> eypen_weir :> >>>>> >>>>> >>>>>+======+========+============+============+============+============+= >>>>>= >>>>>== >>>>>=========+=============+=============+=============+=============+==== >>>>>= >>>>>== >>>>>======+ >>>>> >>>>>| 10 | 228864 | 2.95496e-0 | null | null | -0.0001263 | null | null | >>>>>-0.00028749 >>>>>| | null | -0.00029885 | null | >>>>> >>>>> : : : 5 : : : 25 : : : 4 : : 1 : : >>>>> >>>>>| 10 | 228962 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 228972 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 229011 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 229025 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 229187 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 229367 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 229405 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 229444 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 229593 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 229615 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 229671 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 229829 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 230000 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 230052 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>>| 10 | 230120 | null | null | null | null | null | null | null | null >>>>>| >>>>>null | >>>>>| null | >>>>> >>>>> >>>>>+------+--------+------------+------------+------------+------------+- >>>>>- >>>>>-- >>>>>---------+-------------+-------------+-------------+-------------+---- >>>>>- >>>>>-- >>>>>------+ >>>>> >>>>> >>>>> >>>>> 16 tuples (5.2 >>>>> >>>>> >>>>> My java program uses Float.NaN and Integer.MIN_VALUE, Byte.MIN_VALUE >>>>>to >>>>>get >>>>> ³null² stored in the db. >>>>> >>>>> When we load directly from binary, are the null values created >>>>>differently? Is >>>>> something not marked that gets marked when loading from text files? >>>>> >>>>> With this information, can you reproduce, or do you want me to give >>>>>you >>>>>my >>>>> binary files in a dropbox somewhere? >>>>> >>>>> Thanks - Lynn >>>>> >>>>> >>>>> _______________________________________________ >>>>> users-list mailing list >>>>> users-list(a)monetdb.org >>>>> https://www.monetdb.org/mailman/listinfo/users-list >>>> >>>>-- >>>>| Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | >>>>| www.CWI.nl/~manegold/ | Science Park 123 (L321) | >>>>| +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | >>>>_______________________________________________ >>>>users-list mailing list >>>>users-list(a)monetdb.org >>>>https://www.monetdb.org/mailman/listinfo/users-list >>> >>> _______________________________________________ >>> users-list mailing list >>> users-list(a)monetdb.org >>> https://www.monetdb.org/mailman/listinfo/users-list >> >>-- >>| Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | >>| www.CWI.nl/~manegold/ | Science Park 123 (L321) | >>| +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | >>_______________________________________________ >>users-list mailing list >>users-list(a)monetdb.org >>https://www.monetdb.org/mailman/listinfo/users-list > >_______________________________________________ >users-list mailing list >users-list(a)monetdb.org >https://www.monetdb.org/mailman/listinfo/users-list

1 0

additional details on IS NULL problem
by Lynn Carol Johnson 24 May '16

24 May '16

2 7

issue with IS NULL
by Lynn Carol Johnson 23 May '16

23 May '16

Good morning - I am having an issue identifying columns with a null value in the July2015-SP3 monetdb version. A sample of my table with null values is below: sql>select * from kelly limit 10; +------+-------+------------+-------------+-------------+-------------+-------------+ | chr | pos | bkn22vsbkn | bkn22vsbkn3 | bkn22vsbkn4 | bkn22vstemp | bkn22vstrop |> : : : 16_weir_an : 5_weir_and_ : 0_weir_and_ : erate_cinta : ical_cinta_ :> : : : d_cockerha : cockerham_f : cockerham_f : _weir_and_c : weir_and_co :> : : : m_fst : st : st : ockerham_fs : ckerham_fst :> : : : : : : t : :> +======+=======+============+=============+=============+=============+=============+ | 1 | 10004 | null | null | null | null | null | | 1 | 10045 | 0.050576 | 0 | 0.154324 | 0.130628 | 0.0398957 | | 1 | 10077 | null | null | null | null | null | | 1 | 10084 | null | null | null | null | null | | 1 | 10097 | null | null | 0.0823631 | 0 | null | | 1 | 10112 | null | null | null | null | null | | 1 | 10128 | 0.00185251 | 0 | 0 | 0.00272515 | 0.00594128 | | 1 | 10162 | null | null | null | null | null | | 1 | 10270 | null | null | null | null | null | | 1 | 10299 | null | null | null | null | null | +------+-------+------------+-------------+-------------+-------------+-------------+ 10 tuples (18.9s) !144 columns dropped! note: to disable dropping columns and/or truncating fields use \w-1 sql> Verify the number of values in the table: sql>select count(1) from kelly; +----------+ | L1 | +==========+ | 83153144 | +----------+ 1 tuple (479.905ms) sql> I would like to pull all values that are NOT null from a specific column. My understanding is that running a “count” on a column should return the count of the non-null values. When I run a count on bkn22vsbkn40_weir_and_cockerham_fst I get the following result, which is the same value as the total number of entries above: sql>select count(bkn22vsbkn40_weir_and_cockerham_fst) from kelly; +----------+ | L1 | +==========+ | 83153144 | +----------+ 1 tuple (860.533ms) sql> The number returned is the same as the total number of entries, but it should be less. However, if I run a query to count the values that are NOT NULL, it counts them: sql>select count(1) from kelly where bkn22vsbkn40_weir_and_cockerham_fst IS NOT NULL; +----------+ | L1 | +==========+ | 41754484 | +----------+ 1 tuple (1.0s) sql> When I try to select column values based on my selected column having a null value, no results are returned. sql>select * from kelly where bkn22vsbkn40_weir_and_cockerham_fst IS NULL limit 10; +-----+-----+-------------------------------------+-------------------------------------+-------------------------------------+ | chr | pos | bkn22vsbkn16_weir_and_cockerham_fst | bkn22vsbkn35_weir_and_cockerham_fst | bkn22vsbkn40_weir_and_cockerham_fst |> +=====+=====+=====================================+=====================================+=====================================+ +-----+-----+-------------------------------------+-------------------------------------+-------------------------------------+ 0 tuples (1.1s) !146 columns dropped! sql> When I select column values based on my selected column value NOT null, I get values. sql>select * from kelly where bkn22vsbkn40_weir_and_cockerham_fst IS NOT NULL limit 10; +------+-------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+--------------+--------------+ | chr | pos | bkn22vsbkn1 | bkn22vsbkn3 | bkn22vsbkn4 | bkn22vstemp | bkn22vstrop | bkn33vsbkn1 | bkn33vsbkn2 | bkn33vsbkn35 | bkn33vsbkn40 |> : : : 6_weir_and_ : 5_weir_and_ : 0_weir_and_ : erate_cinta : ical_cinta_ : 6_weir_and_ : 2_weir_and_ : _weir_and_co : _weir_and_co :> : : : cockerham_f : cockerham_f : cockerham_f : _weir_and_c : weir_and_co : cockerham_f : cockerham_f : ckerham_fst : ckerham_fst :> : : : st : st : st : ockerham_fs : ckerham_fst : st : st : : :> : : : : : : t : : : : : :> +======+=======+=============+=============+=============+=============+=============+=============+=============+==============+==============+ | 1 | 10045 | 0.050576 | 0 | 0.154324 | 0.130628 | 0.0398957 | 0.0289548 | 0 | 0 | 0.124971 | | 1 | 10097 | null | null | 0.0823631 | 0 | null | 0.0406325 | 0.0226116 | 0.0226116 | 0 | | 1 | 10128 | 0.00185251 | 0 | 0 | 0.00272515 | 0.00594128 | null | 0 | null | null | | 1 | 10707 | 0.380632 | 0 | 0 | 0 | 0 | 0.44799 | 0.0206532 | 0.0111431 | 0.0234156 | | 1 | 10751 | 0.0462002 | 0.0439648 | 0.0084101 | 0.0505068 | 0 | 0.0475206 | 0 | 0.0453142 | 0.00522015 | | 1 | 10755 | 0.00147929 | 0 | 0.00290363 | 0.00560938 | 0 | null | 0 | 0 | null | | 1 | 10760 | 0 | 0.011568 | 0 | 0 | 0.0882033 | 0.00590868 | 0 | 0 | 0 | | 1 | 10814 | null | null | 0 | null | null | null | null | null | 0 | | 1 | 10838 | 0.133482 | 0 | 0 | 0.00616907 | 0.146222 | 0.0587818 | 0 | 0 | 0.0621118 | | 1 | 10839 | 0.133482 | 0 | 0 | 0.00616907 | 0.146222 | 0.0639509 | 0 | 0 | 0.0558049 | +------+-------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+--------------+--------------+ 10 tuples (2m 36s) !140 columns dropped! sql> My question: AM I doing something wrong, or is this a bug? I am running the july2015 sp3: [lcj34@cbsudc01 monetdbFiles]$ monetdb -v MonetDB Database Server Toolkit v1.1 (Jul2015-SP3) [lcj34@cbsudc01 monetdbFiles]$ Thanks - Lynn

2 3

Column level 3 problems
by wilco draijer 23 May '16

23 May '16

Hello, When I try to convert my monetdb sql database to rdf using d2rq I get the following error:Problem accessing /directory/genelocation_genelocation. Reason: TODO: column names of level >= 3:SELECT genelocation"."genelocation"."entrez_id" F 50 (E0) This has to do with the bug described in https://www.monetdb.org/bugzilla/show_bug.cgi?id=2526 Is there a way to get around this? Best regards, Wilco

2 1

BLOB
by Shmagi Kavtaradze 22 May '16

22 May '16

Hello community, currently I am using HUGEINT to store data and using UDF I perform bitwise AND on them. I want to store my data in BLOB and perform same actions, but in data mapping table https://www.monetdb.org/wiki/MonetDB_type_system, I was not able to find BLOB mapping for C and MAL. So: 1) What are C and MAL mappings for UDF, if they exist? 2) Is it possible to perform Bitwise AND on blob?

1 0

MonetDB cannot find file to load
by João Victor Risso 19 May '16

19 May '16

Hi All, I am benchmarking MonetDB against other databases, in special the current version of CitusDB's Postgres, to find out which one is a good fit for our OLAP application. In order to do that, I've decided to use a standard benchmark, TPC-H. However, I did not manage to load the data generated by dbgen (one of the tools included in the benchmark) into MonetDB. Here is the SQL statement I've tried to run: > COPY INTO region FROM '/home/user/tpch/dbgen/region.tbl' USING DELIMITERS '|', '\n' LOCKED; and it generated an error which says the file could not be found: > could not open file '/home/user/tpch/dbgen/region.tbl': No such file or directory even though the file exists. I'm running the MonetDB provided in the official repository and the version reported by mclient is: > MonetDB v11.21.19 (Jul2015-SP4) Any help would be appreciated. -- João Victor Tozatti Risso C3SL - Centro de Computação Científica e Software Livre Universidade Federal do Paraná Please consider using PGP encryption to make our communications more secure. You can find my public key at https://pgp.mit.edu/ | Key ID: 395192B8 If you want to know more about PGP encryption, check the following link: https://emailselfdefense.fsf.org/en/

2 1

Monetdb Keep alive in Pentaho
by Ariel Abadi 18 May '16

18 May '16

Hi All! We are using Pentaho (BA Server) and have created a JNDI connection using Monetdb. We are facing some "errors". It seems that the connection to pentaho sometimes drops, therefore wanted to know which will be the correct sintax (if there is one) to send a "keep alive" in the JNDI connection. Thks in advance

1 0

RE: Passing mapi_open Param to mserver5 via monetdbd (Syed M Raza)
by Syed M Raza 17 May '16

17 May '16

Hi Everyone Any help or pointer on starting monetdb server to accept remote connections? Original message attached below. Regards Raza -----Original Message----- Message: 1 Date: Wed, 11 May 2016 13:28:51 +0000 From: Syed M Raza <smr337(a)cornell.edu> To: "users-list(a)monetdb.org" <users-list(a)monetdb.org> Subject: Passing mapi_open Param to mserver5 via monetdbd Message-ID: <BY1PR0401MB1429201171375B1BD9BB46258E720(a)BY1PR0401MB1429.namprd04.prod.outlook.com> Content-Type: text/plain; charset="us-ascii" Hi All Is there a way to pass config params to mserver5 via monetdbd daemon? I am trying to open database for remote connections. After started the dbfarm using monetdbd I see entries in log file that calls mserver5 are made with mapi_open=false. I couldn't figure out how to pass mapi_open=open instead, or for that matter, other params to mserver5. What I understand from documentation that mserver5 should be managed via monetdbd daemon. Or should I try to start mserver5 separately, for each database in the dbfarm? How would those instances then attach to the monetdbd daemon? For what it worth, I did try - creating a monetdb.conf file in /path/to/dbfarm directory - setting map_open using monetdbd set name=value MonetDB v11.21.19 ( Jul2015-SP4) OS - Ubuntu 14.04 LTS Thanks for your help in advance. Regards Raza

4 3

Data caching in MonetDB
by Vijay Krishna 16 May '16

16 May '16

Hi, We were studying on the parallel execution of MonetDB and memory usage of MonetDB. 1. If two queries access the same table in parallel, is the table loaded twice into memory or its loaded only once? 2. Is the hot dataset from a column of a query cached in memory, so that any further queries accessing that column may use it? 3. Are the interim results of a query cached in memory for further queries? Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

1 0

Crystal Language - MonetDB Guide
by Brian Hood 14 May '16

14 May '16

Hi All, I've created a guide for using MonetDB with the Crystal Programming Language. https://github.com/puppetpies/crystal-monetdb-libmapi/blob/master/examples/… Enjoy! Regards, Brian Hood

1 0

Error when trying to load GO.db
by Willem Ligtenberg 12 May '16

12 May '16

Hi, I am trying to run some R code in database. Most of it is going pretty well, but I seem to have stumbled on a bug. I cannot load the following package, which is a dependency for some of my code. WGCNA I have been chasing it down, and it seems to be due to an error when trying to load GO.db. I get the following error: Error in .local(conn, statement, ...) : Unable to execute statement 'SELECT * FROM tmp_test();'. Server says '!Error running R expression. Error message: Error in as.data.frame((function() { : '. Digging a bit further it seems to be due to the following statement: dbconn <- dbFileConnect(dbfile) Error can be reproduced using: functionDef <- paste( "CREATE FUNCTION tmp_test() ", "RETURNS TABLE(output STRING)", "LANGUAGE R ", "{", "library(AnnotationDbi)", "datacache <- new.env(hash=TRUE, parent=emptyenv())", "pkgname <- 'GO.db'", "libname <- .libPaths()[1]", "dbfile <- system.file('extdata', 'GO.sqlite', package=pkgname, lib.loc=libname)", "assign('dbfile', dbfile, envir=datacache)", "dbconn <- dbFileConnect(dbfile)", "};", sep = "\n") dbGetQuery(conn, functionDef) dbGetQuery(conn, "SELECT * FROM tmp_test();") By the way, installing GO.db from within MonetDB works just fine. And can be done using the following R code: source("https://bioconductor.org/biocLite.R") biocLite("GO.db") Hints on how to resolve this are greatly appreciated. Kind regards, Willem

2 1

creating a table with char[3]
by Lynn Carol Johnson 11 May '16

11 May '16

Hi All - I'm trying to create a table with specific size for text fields. In particular, I am storing a 3 byte character and an 8 byte character. I tried the following but get errors: CREATE TABLE te_LTRTest(chr int, pos int, te_gag_protein tinyint, te_ap_protein tinyint, te_integrase_protein tinyint, te_rt_protein tinyint, te_rnaseh_protein tinyint, te_chromodomain_protein tinyint, te_envelope_protein tinyint, te_ltr_age real, te_numb_ltrtes_inside integer, te_superfamily char'('3')', te_family char'('8')'); Gives the error: sql>\</workdir/lcj34/monetdbFiles/michelle_TE/create_teLTRTest.sql syntax error, unexpected STRING, expecting ')' or ',' in: "create table te_ltrtest(chr int, pos int, te_gag_protein tinyint, te_ap_protein" sql> Switching to this: CREATE TABLE te_LTRTest(chr int, pos int, te_gag_protein tinyint, te_ap_protein tinyint, te_integrase_protein tinyint, te_rt_protein tinyint, te_rnaseh_protein tinyint, te_chromodomain_protein tinyint, te_envelope_protein tinyint, te_ltr_age real, te_numb_ltrtes_inside integer, te_superfamily char[3], te_family char[8]); Gives the following error: sql>\</workdir/lcj34/monetdbFiles/michelle_TE/create_teLTRTest.sql syntax error, unexpected '[', expecting ')' or ',' in: "create table te_ltrtest(chr int, pos int, te_gag_protein tinyint, te_ap_protein" sql> I have 2.1 billion rows so am using the binary bulk loads. My non-numeric binary files were created to contain 3-byte values and 8-byte values. This is what I am trying to load, however I'm not getting past the create table stage. I love the binary loader. This is the first time I've needed to include text data in my tables. Is this supported? I'm hoping it is just my syntax. Any suggestions? Thanks - Lynn

4 6

Can not import 128 bit values
by Shmagi Kavtaradze 11 May '16

11 May '16

I create table with HUGEINT type. When I am trying to import data from text file, Monetdb fires error: Failed to import table line 508 field 3 'hugeint' expected in '170141183460469231731687303715888300032', but this value has exactly 128 bits. Any ideas how to fix the issue? Thanks.

2 2

Passing mapi_open Param to mserver5 via monetdbd
by Syed M Raza 11 May '16

11 May '16

Hi All Is there a way to pass config params to mserver5 via monetdbd daemon? I am trying to open database for remote connections. After started the dbfarm using monetdbd I see entries in log file that calls mserver5 are made with mapi_open=false. I couldn't figure out how to pass mapi_open=open instead, or for that matter, other params to mserver5. What I understand from documentation that mserver5 should be managed via monetdbd daemon. Or should I try to start mserver5 separately, for each database in the dbfarm? How would those instances then attach to the monetdbd daemon? For what it worth, I did try - creating a monetdb.conf file in /path/to/dbfarm directory - setting map_open using monetdbd set name=value MonetDB v11.21.19 ( Jul2015-SP4) OS - Ubuntu 14.04 LTS Thanks for your help in advance. Regards Raza Consultant, GOBII project Genomics & Open-source Breeding Informatics Initiative Boyce Thompson Institute # 224 Cornell University Ithaca, NY 14853

1 0

monetdb is great, thanks everyone who makes it so
by Anthony Damico 09 May '16

09 May '16

4 3

Crystal binding MonetDB / Example project
by Brian Hood 08 May '16

08 May '16

Hi Guys, If your interested in Programming Languages i've recently built the binding for MonetDB in the Crystal Language if you don't know that Crystal is its a Ruby inspired language that is statically typed and compiles in byte code via LLVM yes interesting. Here is the Driver as they used to call them. https://github.com/puppetpies/crystal-monetdb-libmapi Very small example project that i will expand upon. https://github.com/puppetpies/crystal-monetdb-examples Also yes i will write a blog as well Martin i'm not sure what want me to write but i am put together a reference guide for people that might come along in the future to use this. Regards, Brian Hood

1 0

Memory management - MonetDB on CentOS
by Vijay Krishna 07 May '16

07 May '16

Hi, We have been running tests on Memory management with the latest version of MonetDB on a CentOS installation. The machine has 1TB hard disk and 128 GB RAM space. The swap memory configured is 2GB. We were aware that MonetDB completely relies on the OS for Memory Management and is prone to OOM Killer crash and hence we wanted to simulate the crash. Hence we ran hundreds of threads executing various queries and noticed the following pattern of memory handling. 1. Initially, RAM was filled. 2. Upon RAM overflow, swap memory (2 GB) was used. 3. Upon swap overflow, the swap memory got extended and the 1TB memory that was free on the machine was used completely. 4. The OOM killer got invoked only when this 1TB memory got filled. 5. On consecutive executions, swap memory was started to be used even when RAM had 70GB of memory unused and free (not even used for cache files). 6. The /tmp partition was never used. Memory got overfilled only in the current partition. These observations led to the following questions about the memory management with CentOS. 1. Does MonetDB has no influence on the above behaviour and is it that only the OS is responsible for this? 2. Will the swap memory be expanded from the committed size by the OS if swap is running out of memory? In our case it expanded from 2GB to 1 TB. Our OS setting of overcommit is 0 (sysctl vm.overcommit_memory = 0) 3. When will the OOM Killer be invoked usually? Upon Swap fill or upon Hard disk space full? 4. Why was memory swapped even when there is 70GB free space in RAM? We tried reducing the swappiness to 10 (sysctl vm.swappiness=10), even then memory was swapped with 70 GB free space in RAM. 5. Does the OS writes intermediate results to /tmp directory? Or is it that MonetDB is bypassing this? 6. Is there any other OS configurations that we need to cross check regarding memory management? Any help much appreciated. Thanks in advance. Regards, Vijay.

1 0

index clustering
by Shmagi Kavtaradze 06 May '16

06 May '16

When searching for index clustering, I read that no user defined index clustering exists in Monetdb. Does it mean that I can not cluster table according to index? If not, how can I sort table to speed-up query? Thanks in advance.

2 1

Problem with RANK/DENSE_RANK in a subquery (dense_rank_grp undefined)
by Knezevic Nikola 04 May '16

04 May '16

Hi all, I have several tables which represent different sources (sensors) of data. I want to find out which sensors picked up the signal first, and thus I decided to join these tables together, rank on common criteria and then select the first ones. However, that fails with: TypeException:user.s22_3[110]:'calc.dense_rank_grp' undefined in: calc.dense_rank_grp(X_505:date,X_507:str,X_509:int,X_511:int,X_513:lng); program contains errors Any ideas how to fix this? Cheers, Nikola Data: CREATE TABLE taba ( aday DATE, key1 varchar(15), msid int, sid int, eid bigint, r bigint ); CREATE TABLE tabb ( aday DATE, key1 varchar(15), msid int, sid int, aid bigint, eid bigint, r bigint ); CREATE TABLE tabc ( aday DATE, key1 varchar(15), msid int, sid int, aid bigint, m bigint, r bigint ); COPY 3 RECORDS INTO taba FROM stdin USING DELIMITERS '|','\n'; 2016-05-02|inst1|675|10001|200|100 2016-05-02|inst1|675|10002|500|111 2016-05-02|inst1|789|10001|800|135 COPY 3 RECORDS INTO tabb FROM stdin USING DELIMITERS '|','\n'; 2016-05-02|inst1|675|10001|77|200|88 2016-05-02|inst1|675|10002|100|500|112 2016-05-02|inst1|789|10001|112|800|134 COPY 3 RECORDS INTO tabc FROM stdin USING DELIMITERS '|','\n'; 2016-05-02|inst1|675|10001|77|200|100 2016-05-02|inst1|675|10002|100|500|111 2016-05-02|inst1|789|10001|112|800|130 Query: SELECT aday, key1, msid, sid, aid, eid, src, r, rank FROM ( SELECT * ,sys.DENSE_RANK() OVER (PARTITION BY aday, key1, msid, sid, eid ORDER BY r) as rank FROM ( (SELECT aday, key1, msid, sid, NULL as aid, eid, 'TABA' as src, r FROM taba) UNION (SELECT aday, key1, msid, sid, aid, eid, 'TABB' as src, r FROM tabb) UNION (SELECT aday, key1, msid, sid, aid, m as eid, 'TABC' AS src, r FROM tabc) ) AS dummy1 ) AS dummy2 WHERE rank = 1 ; Results: sql>CREATE TABLE taba ( more>aday DATE, more>key1 varchar(15), more>msid int, more>sid int, more>eid bigint, more>r bigint more>); operation successful (4.102ms) sql>CREATE TABLE tabb ( more>aday DATE, more>key1 varchar(15), more>msid int, more>sid int, more>aid bigint, more>eid bigint, more>r bigint more>); operation successful (3.172ms) sql>CREATE TABLE tabc ( more>aday DATE, more>key1 varchar(15), more>msid int, more>sid int, more>aid bigint, more>m bigint, more>r bigint more>); operation successful (3.326ms) sql> sql>COPY 3 RECORDS INTO taba FROM stdin USING DELIMITERS '|','\n'; more>2016-05-02|inst1|675|10001|200|100 more>2016-05-02|inst1|675|10002|500|111 more>2016-05-02|inst1|789|10001|800|135 3 affected rows (88.249ms) sql> sql>COPY 3 RECORDS INTO tabb FROM stdin USING DELIMITERS '|','\n'; more>2016-05-02|inst1|675|10001|77|200|88 more>2016-05-02|inst1|675|10002|100|500|112 more>2016-05-02|inst1|789|10001|112|800|134 3 affected rows (59.917ms) sql> sql>COPY 3 RECORDS INTO tabc FROM stdin USING DELIMITERS '|','\n'; more>2016-05-02|inst1|675|10001|77|200|100 more>2016-05-02|inst1|675|10002|100|500|111 more>2016-05-02|inst1|789|10001|112|800|130 3 affected rows (89.804ms) sql> sql>select * from taba; +------------+-------+------+-------+------+------+ | aday | key1 | msid | sid | eid | r | +============+=======+======+=======+======+======+ | 2016-05-02 | inst1 | 675 | 10001 | 200 | 100 | | 2016-05-02 | inst1 | 675 | 10002 | 500 | 111 | | 2016-05-02 | inst1 | 789 | 10001 | 800 | 135 | +------------+-------+------+-------+------+------+ 3 tuples (4.340ms) sql>select * from tabb; +------------+-------+------+-------+------+------+------+ | aday | key1 | msid | sid | aid | eid | r | +============+=======+======+=======+======+======+======+ | 2016-05-02 | inst1 | 675 | 10001 | 77 | 200 | 88 | | 2016-05-02 | inst1 | 675 | 10002 | 100 | 500 | 112 | | 2016-05-02 | inst1 | 789 | 10001 | 112 | 800 | 134 | +------------+-------+------+-------+------+------+------+ 3 tuples (3.484ms) sql>select * from tabc; +------------+-------+------+-------+------+------+------+ | aday | key1 | msid | sid | aid | eid | m | +============+=======+======+=======+======+======+======+ | 2016-05-02 | inst1 | 675 | 10001 | 77 | 200 | 100 | | 2016-05-02 | inst1 | 675 | 10002 | 100 | 500 | 111 | | 2016-05-02 | inst1 | 789 | 10001 | 112 | 800 | 130 | +------------+-------+------+-------+------+------+------+ 3 tuples (3.215ms) sql>SELECT * more>FROM ( more>SELECT * more> ,sys.DENSE_RANK() OVER (PARTITION BY aday, key1, msid, sid, eid ORDER BY r) as rank more> FROM ( more> (SELECT aday, key1, msid, sid, NULL as aid, eid, 'TABA' as src, r FROM taba) more> UNION more> (SELECT aday, key1, msid, sid, aid, eid, 'TABB' as src, r FROM tabb) more> UNION more> (SELECT aday, key1, msid, sid, aid, m as eid, 'TABC' AS src, r FROM tabc) more> ) AS dummy1 more>) AS dummy2; +------------+-------+------+-------+------+------+------+------+------+ | aday | key1 | msid | sid | aid | eid | src | r | rank | +============+=======+======+=======+======+======+======+======+======+ | 2016-05-02 | inst1 | 675 | 10001 | null | 200 | TABA | 100 | 2 | | 2016-05-02 | inst1 | 675 | 10002 | null | 500 | TABA | 111 | 1 | | 2016-05-02 | inst1 | 789 | 10001 | null | 800 | TABA | 135 | 3 | | 2016-05-02 | inst1 | 675 | 10001 | 77 | 200 | TABB | 88 | 1 | | 2016-05-02 | inst1 | 675 | 10002 | 100 | 500 | TABB | 112 | 2 | | 2016-05-02 | inst1 | 789 | 10001 | 112 | 800 | TABB | 134 | 2 | | 2016-05-02 | inst1 | 675 | 10001 | 77 | 200 | TABC | 100 | 2 | | 2016-05-02 | inst1 | 675 | 10002 | 100 | 500 | TABC | 111 | 1 | | 2016-05-02 | inst1 | 789 | 10001 | 112 | 800 | TABC | 130 | 1 | +------------+-------+------+-------+------+------+------+------+------+ 9 tuples (13.527ms) sql>SELECT aday, key1, msid, sid, aid, eid, src, r, rank more>FROM ( more>SELECT * more> ,sys.DENSE_RANK() OVER (PARTITION BY aday, key1, msid, sid, eid ORDER BY r) as rank more> FROM ( more> (SELECT aday, key1, msid, sid, NULL as aid, eid, 'TABA' as src, r FROM taba) more> UNION more> (SELECT aday, key1, msid, sid, aid, eid, 'TABB' as src, r FROM tabb) more> UNION more> (SELECT aday, key1, msid, sid, aid, m as eid, 'TABC' AS src, r FROM tabc) more> ) AS dummy1 more>) AS dummy2 more>WHERE rank = 1 more>; TypeException:user.s22_3[110]:'calc.dense_rank_grp' undefined in: calc.dense_rank_grp(X_505:date,X_507:str,X_509:int,X_511:int,X_513:lng); program contains errors

2 2

PCRE: references to captures in replace string
by Roberto Cornacchia 04 May '16

04 May '16

After declaring a wrapper for pcre.replace: sql>create function pcre_replace(s string, pattern string, repl string, flags string) returns string external name pcre."replace"; operation successful (6.187ms) I cannot get replacements with references to captures work (but the code in pcre.c seems to support it): sql>select pcre_replace('Apple Pear', '^([a-zA-Z]+)\\s*([a-zA-Z]+)$', '\\1 \\2', ''); +---------------------------+ | pcre_replace_single_value | +===========================+ | \1 \2 | +---------------------------+ 1 tuple (0.309ms) The pattern is matched, but the back references are replaced literally. Am I not getting the syntax correct? >From http://www.pcre.org/original/doc/html/pcresyntax.html I see I could use any of \1, \g1, \g{1] for back references, but none seem to work. Roberto

3 7

Problem: 'ALTER TABLE x ADD TABLE y' does not merge the data in from y.
by Knezevic Nikola 04 May '16

04 May '16

Hi all, I'm splitting my data into per-day tables and importing them into monetdb. Before the import, I first create a per-day table and then use 'ALTER TABLE "table" ADD TABLE "table__2016-05-03";' command (passed to cursor.execute() in python. "table" is defined to be a MERGE TABLE. Now, I had only a single day that was populated on the initial run, when both MERGE TABLE and day table were created. Today, I added a new day table and ran this: sql> SELECT day, count(*) FROM "table" GROUP BY day; +------------+---------+ | day | L1 | +============+=========+ | 2016-04-26 | 2959601 | +------------+---------+ sql>select day, count(*) from "trades__2016-05-02" group by day; +------------+---------+ | day | L1 | +============+=========+ | 2016-05-02 | 2220143 | +------------+---------+ So, the output is missing the current day, and I thought my python code didn't properly tie these tables together. However, inspecting sys.* tables gives me this: sql>\d (snip...) MERGE TABLE my_namespace.trades TABLE my_namespace.trades__2016-04-26 TABLE my_namespace.trades__2016-05-02 (snip...) sql>select * from sys.tables where name like 'trades%'; +-------+--------------------+-----------+-------+------+--------+---------------+--------+-----------+ | id | name | schema_id | query | type | system | commit_action | access | temporary | +=======+====================+===========+=======+======+========+===============+========+===========+ | 13480 | trades | 6497 | null | 3 | false | 0 | 0 | 0 | | 13518 | trades__2016-04-26 | 6497 | null | 0 | false | 0 | 0 | 0 | | 16203 | trades__2016-05-02 | 6497 | null | 0 | false | 0 | 0 | 0 | +-------+--------------------+-----------+-------+------+--------+---------------+--------+-----------+ 3 tuples (3.323ms) sql>select * from sys.dependencies where depend_id=13480; +-------+-----------+-------------+ | id | depend_id | depend_type | +=======+===========+=============+ | 13518 | 13480 | 2 | | 16203 | 13480 | 2 | +-------+-----------+-------------+ Finally, if I manually add the table, it succeeds: sql>alter table "trades" add table "trades__2016-05-02"; operation successful (13.894ms) sql>select day, count(*) from "trades" group by day; +------------+---------+ | day | L1 | +============+=========+ | 2016-04-26 | 2959601 | | 2016-05-02 | 2220143 | +------------+---------+ 2 tuples (44.196ms) sql>select * from sys.tables where name like 'trades%'; +-------+--------------------+-----------+-------+------+--------+---------------+--------+-----------+ | id | name | schema_id | query | type | system | commit_action | access | temporary | +=======+====================+===========+=======+======+========+===============+========+===========+ | 13480 | trades | 6497 | null | 3 | false | 0 | 0 | 0 | | 13518 | trades__2016-04-26 | 6497 | null | 0 | false | 0 | 0 | 0 | | 16203 | trades__2016-05-02 | 6497 | null | 0 | false | 0 | 0 | 0 | +-------+--------------------+-----------+-------+------+--------+---------------+--------+-----------+ 3 tuples (7.445ms) sql>select * from sys.dependencies where depend_id=13480; +-------+-----------+-------------+ | id | depend_id | depend_type | +=======+===========+=============+ | 13518 | 13480 | 2 | | 16203 | 13480 | 2 | +-------+-----------+-------------+ 2 tuples (1.714ms) For completeness, python code looks like this (cleaned up, to present the case, keep in mind that on initial import, this worked): if args.create: if args.partition and has_day: dest_create_merge_table = re.sub('CREATE TABLE', 'CREATE MERGE TABLE', dest_create_table, count=1) print("Creating table with the following content:\n{ddl}".format(ddl=dest_create_merge_table), file=sys.stderr) try: cursor = monet_conn.cursor() cursor.execute(dest_create_merge_table) except monetdb.exceptions.OperationalError as e: if re.search('CREATE TABLE: name .* already in use', e.args[0]): print("Master table already present") else: raise(e) finally: cursor.close() if args.partition and has_day: cursor = monet_conn.cursor() for c in conditions: (text, day, sth) = c dest_create_partition = re.sub('CREATE TABLE "{table}"'.format(table=table), 'CREATE TABLE "{partition}"'.format(partition=partition_name(table, day)), dest_create_table, count=1) print("Creating partition with the following content:\n{ddl}".format(ddl=dest_create_partition), file=sys.stderr) cursor.execute(dest_create_partition) if args.debug: print('ALTER TABLE "{table}" ADD TABLE "{partition}"'.format(table=table, partition=partition_name(table, day))) cursor.execute('ALTER TABLE "{table}" ADD TABLE "{partition}"; COMMIT;'.format(table=table, partition=partition_name(table, day))) cursor.close() Can someone please help me debug this issue? Is this a bug? IIUC, metadata points that "trades__2016-05-02" belongs to "trades", but the queries do not reflect that. Thanks, Nikola

3 5

Crashed DB. cannot start DB
by Ariel Abadi 03 May '16

03 May '16

Hi! Can someone help me please with this issue? Im trying to start my db.. and Im getting the following erro from the log: 2016-05-03 15:08:38 MSG merovingian[14646]: starting database 'bm_traf', up min/avg/max: 3d/5d/1w, crash average: 1.00 0.40 0.13 (6-2=4) 2016-05-03 15:08:38 MSG bm_traf[28169]: arguments: /usr/bin/mserver5 --dbpath=/opt/monetdb/bm/bm_traf --set merovingian_uri=mapi:monetdb://monetdb:50000/bm_traf --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/monetdb/bm/bm_traf/.mapi.sock --set monet_vault_key=/opt/monetdb/bm/bm_traf/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-05-03 15:08:38 MSG merovingian[14646]: database 'bm_traf' (28169) has exited with exit status 1 2016-05-03 15:08:43 ERR merovingian[14646]: client error: database 'bm_traf' appears to shut itself down after starting, check monetdbd's logfile for possible hints Does anybody has a clue? Thks in advance!

2 2

Query Histroy & query parsing / translation / compilation / optimization time
by Stefan Manegold 03 May '16

03 May '16

Hi, looking for ways to measure the query parsing / translation / compilation / optimization time in MonetDB, I found these two documentations on the MonetDB website: https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/QueryHistory and https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/QueryHistoryNew While the documentation is not entirely clear on whether the latter did, will, or is supposed to replace the former (and if so, when), using the latest Jul2015-SP4 release, I get with the former: sql>set history=true; operation successful (137.545ms) sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (155.577ms) sql>select 2; +--------------+ | single_value | +==============+ | 2 | +--------------+ 1 tuple (193.633ms) sql>select * from sys.queryHistory; SELECT: no such table 'queryhistory' sql>select * from sys.callHistory; SELECT: no such table 'callhistory' sql>select * from sys.queryLog; SELECT: no such table 'querylog' sql>call sys.resetHistory(); SELECT: no such operator 'resethistory' I.e, this does not seem to work (anymore?). The latter seems to work fine: sql>call sys.querylog_enable(); sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (127.056ms) sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (124.303ms) sql>select 2; +--------------+ | single_value | +==============+ | 2 | +--------------+ 1 tuple (148.923ms) sql>select * from sys.querylog_catalog; +------+---------+----------------------------+-------------------------------------+--------------+-----------+------+----------+ | id | owner | defined | query | pipe | plan | mal | optimize | +======+=========+============================+=====================================+==============+===========+======+==========+ | 42@0 | monetdb | 2016-05-03 00:06:44.000000 | call sys.querylog_enable(); | default_pipe | user.s6_1 | 19 | 267 | | 43@0 | monetdb | 2016-05-03 00:06:47.000000 | select 1; | default_pipe | user.s7_1 | 21 | 273 | | 44@0 | monetdb | 2016-05-03 00:06:48.000000 | select 1; | default_pipe | user.s7_1 | 21 | 273 | | 45@0 | monetdb | 2016-05-03 00:06:50.000000 | select 1; | default_pipe | user.s7_1 | 21 | 273 | | 46@0 | monetdb | 2016-05-03 00:06:56.000000 | select * from sys.querylog_catalog; | default_pipe | user.s8_1 | 68 | 665 | +------+---------+----------------------------+-------------------------------------+--------------+-----------+------+----------+ 5 tuples (132.446ms) sql>select * from sys.querylog_calls; +------+----------------------------+----------------------------+------------------------------------------+--------+-------+------+------+------+ | id | start | stop | arguments | tuples | run | ship | cpu | io | +======+============================+============================+==========================================+========+=======+======+======+======+ | 41@0 | 2016-05-03 00:06:38.000000 | 2016-05-03 00:06:39.000000 | user.s5_1{autoCommit=true}():void; | 1 | 85485 | 1 | 3 | 24 | | 42@0 | 2016-05-03 00:06:44.000000 | 2016-05-03 00:06:44.000000 | user.s6_1{autoCommit=true}():void; | 1 | 71 | 0 | 0 | 0 | | 43@0 | 2016-05-03 00:06:47.000000 | 2016-05-03 00:06:47.000000 | user.s7_1{autoCommit=true}(A0:bte):void; | 1 | 120 | 0 | 0 | 0 | | 44@0 | 2016-05-03 00:06:48.000000 | 2016-05-03 00:06:48.000000 | user.s7_1{autoCommit=true}(A0:bte):void; | 1 | 104 | 0 | 0 | 0 | | 45@0 | 2016-05-03 00:06:50.000000 | 2016-05-03 00:06:50.000000 | user.s7_1{autoCommit=true}(A0:bte):void; | 1 | 66 | 1 | 0 | 0 | | 46@0 | 2016-05-03 00:06:56.000000 | 2016-05-03 00:06:56.000000 | user.s8_1{autoCommit=true}():void; | 8 | 886 | 0 | 50 | 0 | +------+----------------------------+----------------------------+------------------------------------------+--------+-------+------+------+------+ 6 tuples (100.344ms) sql>select * from sys.querylog_history; +------+---------+----------------------------+-------------------------------------+--------------+-----------+------+----------+----------------------------+----------------------------+------------------------------------------+--------+------+------+------+------+ | id | owner | defined | query | pipe | plan | mal | optimize | start | stop | arguments | tuples | run | ship | cpu | io | +======+=========+============================+=====================================+==============+===========+======+==========+============================+============================+==========================================+========+======+======+======+======+ | 42@0 | monetdb | 2016-05-03 00:06:44.000000 | call sys.querylog_enable(); | default_pipe | user.s6_1 | 19 | 267 | 2016-05-03 00:06:44.000000 | 2016-05-03 00:06:44.000000 | user.s6_1{autoCommit=true}():void; | 1 | 71 | 0 | 0 | 0 | | 43@0 | monetdb | 2016-05-03 00:06:47.000000 | select 1; | default_pipe | user.s7_1 | 21 | 273 | 2016-05-03 00:06:47.000000 | 2016-05-03 00:06:47.000000 | user.s7_1{autoCommit=true}(A0:bte):void; | 1 | 120 | 0 | 0 | 0 | | 44@0 | monetdb | 2016-05-03 00:06:48.000000 | select 1; | default_pipe | user.s7_1 | 21 | 273 | 2016-05-03 00:06:48.000000 | 2016-05-03 00:06:48.000000 | user.s7_1{autoCommit=true}(A0:bte):void; | 1 | 104 | 0 | 0 | 0 | | 45@0 | monetdb | 2016-05-03 00:06:50.000000 | select 1; | default_pipe | user.s7_1 | 21 | 273 | 2016-05-03 00:06:50.000000 | 2016-05-03 00:06:50.000000 | user.s7_1{autoCommit=true}(A0:bte):void; | 1 | 66 | 1 | 0 | 0 | | 46@0 | monetdb | 2016-05-03 00:06:56.000000 | select * from sys.querylog_catalog; | default_pipe | user.s8_1 | 68 | 665 | 2016-05-03 00:06:56.000000 | 2016-05-03 00:06:56.000000 | user.s8_1{autoCommit=true}():void; | 8 | 886 | 0 | 50 | 0 | | 47@0 | monetdb | 2016-05-03 00:07:33.000000 | select * from sys.querylog_calls; | default_pipe | user.s9_1 | 73 | 282 | 2016-05-03 00:07:33.000000 | 2016-05-03 00:07:34.000000 | user.s9_1{autoCommit=true}():void; | 9 | 2515 | 0 | 0 | 0 | +------+---------+----------------------------+-------------------------------------+--------------+-----------+------+----------+----------------------------+----------------------------+------------------------------------------+--------+------+------+------+------+ 6 tuples (152.709ms) Hence, maybe the documentation should be updated to remove the former? Also, two questions remain unanswered by the current documentation: 1) What is the unit of "mal integer, -- size of MAL plan"? bytes? lines? statements? 2) What exactly does "optimize bigint -- time in microseconds" measure? Only the time taken by MAL optimizers (i.e., after SQL parsing, relational optimization and relational-to-MAL translation, but before actual execution)? If so, is there still a way to measure the time(s) for SQL parsing, relational optimization, and (or) relational-to-MAL translation? Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Help with json.filters
by Alberto Ferrari 03 May '16

03 May '16

Can anyone help with json.filters? I have this example table ... CREATE TABLE tblJson ( "id" INT, "json_data" JSON, ); ... with these values... INSERT INTO tblJson (id, json_data) VALUES (1, '{"data":{"time":"2015-10-09 10:36:35","others":"xxx"}}'), (2, '{"data":{"time":"2015-10-09 12:18:20","others":"yyy"}}'), (3, '{"data":{"time":"2015-12-30 16:31:22","others":"zzz"}}'); The data is ok, I can select the rows, it's ok. Also I can select only the "time" field, like this... SELECT json.filter(json_data, 'data.time') FROM tblJson; But now I want to filter the content in "json_data" specific field, i.e. SELECT all the rows where "time like '2015-10-09%'" , or maybe "time = '2015-10-09 12:18:20'" I tried but can't do it. Any help? Thanks!

3 5

Problems with aggregate UDF (subaggregate undefined error)
by Anderson, David B 02 May '16

02 May '16

All, I have been able to create and install my own aggregate UDF (weightedpercentile), but when I try to execute it via mclient I am getting the following error: sql>select key, weightedpercentile(a,b) from t group by key; TypeException:user.s9_1[20]:'udf.subweightedpercentile' undefined in: X_34:bat[:any,:dbl] := udf.subweightedpercentile(X_26:bat[:oid,:dbl],X_33:bat[:oid,:dbl],X_16:bat[:oid,:oid],r1_16:bat[:oid,:oid],true:bit); program contains errors Does anyone know what I am doing incorrectly? I am at a total loss now. The error is almost identical to the error I get if I try to execute the corr() aggregate: sql>select key, corr(a,b) from t group by key; TypeException:user.s10_1[20]:'aggr.subcorr' undefined in: X_34:bat[:any,:dbl] := aggr.subcorr(X_26:bat[:oid,:dbl],X_33:bat[:oid,:dbl],X_16:bat[:oid,:oid],r1_16:bat[:oid,:oid],true:bit); program contains errors The corr() error is referenced in a recent bug submission (which I can't find in Bugzilla) If I look in the system function catalog, I see: sql>select * from sys.functions where name like '%weight%'; +------+--------------------+---------------------------------------------------+------+------+------+-------+-------+-------+------+ | id | name | func | mod | lang | type | side_ | varre | varar | sche | : : : : : uage : : effec : s : g : ma_i : : : : : : : : t : : : d : +======+====================+===================================================+======+======+======+=======+=======+=======+======+ | 9313 | weightedpercentile | -- weighted percentile | udf | 1 | 3 | false | false | false | 2000 | : : : create aggregate weightedpercentile(e1 real, e2 r : : : : : : : : : : : eal) returns real : : : : : : : : : : : external name udf.weightedpercentile; : : : : : : : : | 9318 | weightedpercentile | -- weighted percentile | udf | 1 | 3 | false | false | false | 2000 | : : : create aggregate weightedpercentile(e1 double, e2 : : : : : : : : : : : double) returns double : : : : : : : : : : : external name udf.weightedpercentile; : : : : : : : : +------+--------------------+---------------------------------------------------+------+------+------+-------+-------+-------+------+ 2 tuples (1.107ms) So my function appears to be installed. I have in (80_udf.sql): -- weighted percentile create aggregate weightedpercentile(e1 real, e2 real) returns real external name udf.weightedpercentile; -- weighted percentile create aggregate weightedpercentile(e1 double, e2 double) returns double external name udf.weightedpercentile; and in (udf.mal): # BAT MAL signatures module batudf; command weightedpercentile(one:bat[:oid,:bte],two:bat[:oid,:bte]):lng address UDFBATweightedpercentile comment "compute the weighted percentile (bte)" command weightedpercentile(one:bat[:oid,:int],two:bat[:oid,:int]):lng address UDFBATweightedpercentile comment "compute the weighted percentile (int)" command weightedpercentile(one:bat[:oid,:lng],two:bat[:oid,:lng]):lng address UDFBATweightedpercentile comment "compute the weighted percentile (lng)"

2 3

Problem make-install'ing MonetDB to $HOME/opt/monetdb
by Eyal Rozenberg 02 May '16

02 May '16

I'm trying to upgrade to the latest release of MonetDB (Jul2015-SP4) instead of an older one. I'm building and installing as a non-root user, into $HOME/opt/monetdb . Well, * ./bootstrap SUCCEEDED * ./configure --prefix=$HOME/opt/monetdb SUCCEEDED * ./make SUCCEEDED * ./make install FAILED Here's the error message: /usr/bin/install -c -m 644 mapi.h /home/eyalroz/opt/monetdb/include/monetdb/mapi.h /bin/sh ../../libtool --mode=install /usr/bin/install -c libmapi.la '/home/eyalroz/opt/monetdb/lib' libtool: install: error: cannot install `libmapi.la' to a directory not ending in /usr/local/lib I don't see why libtool is telling me that. What business is it of libtool what I want to call my directories? Anyway, it doesn't seem to me I tried to do something esoteric... how do you suggest I get around this? Eyal PS - It's a Fedora 22 machine in case that matters.

2 2

Explain analyze
by Shmagi Kavtaradze 02 May '16

02 May '16

In postgres I could use explain analyze to see actions performed by database and their cost. I was not able to run explain analyze. I ran EXPLAIN, but can not see any plan or cost. Any ideas how to get costs and query plan like in Postgres?

4 7

Crystal bindings for MonetDB
by Brian Hood 30 Apr '16

30 Apr '16

Hi Guys, I have got my Crystal Language bindings for MonetDB now working quite well. https://github.com/puppetpies/crystal-monetdb-libmapi I've seen options in the mapi code for a variable called mode that allow you format the response data in XML but it doesn't seem to be implemented fully in the mapi code. Are there any guide lines to processing this RAW query output. % threatmonitor.fruits, threatmonitor.fruits, threatmonitor.fruits, threatmonitor.fruits, threatmonitor.fruits # table_name % name, price, weight, comments, id # name % varchar, varchar, int, clob, int # type % 9, 4, 2, 11, 1 # length [ "Apple", "9.99", 50, NULL, 1 ] [ "Bananna", "3.99", 30, NULL, 2 ] [ "Orange", "7.99", 60, NULL, 3 ] [ "Peach", "5.00", 80, NULL, 4 ] [ "Kiwi", "9.00", 20, NULL, 5 ] [ "Tomato", "2.00", 20, "Yes a fruit", 6 ] [ "Pear", "4.00", 30, "Juicy", 7 ] [ "Nectarine", "6.00", 50, "Juicy", 8 ] Regards, Brian Hood

1 0

Case sensitive
by Ariel Abadi 30 Apr '16

30 Apr '16

Hi All! We are facing a problem, we used to work with mysql with the case insensitive. Now we have moved into MonetDB and we are facing a problem with this issue. Some of our queries are not bringing any data because of the sensitive.. SELECT * FROM Country = 'jamaica' ... 0 Records SELECT * FROM Country = 'Jamaica' ... 10 Records Some cases we solved it using the ILIKE, but we still have some code on our app that cannot be modified. Does anybody know if there is any flag or setup or config for the Database in Monet to make it insensitive ? Thanks in advance for the help!

6 6

Support for multi-column aggregate functions
by Anderson, David B 30 Apr '16

30 Apr '16

All, Does the current version of MonetDB support multi-column aggregate functions? I need to create a function which returns the "quantiles" of one column using another column as a weighting. It looks like PostgreSQL supports multi-column aggregates as of 8.2. I could write an embedded R function to compute this, but want to know if it is even possible to pass in multiple columns. Thanks, Dave

4 9

回复：Re: SUM() OVER (ORDER BY..)
by chenkaijiang＠bigknow.com.cn 29 Apr '16

29 Apr '16

An alternative way is to use temp tables to get the same result: select c1, sum(c2) over(partition by c3) from t1; ==> create temp table tmp1 (sum_c2 int, c3 int); insert into tmp1 select sum(c2), c3 from t1 group by c3; select c1, sum_c2, c3 from t1 join tmp1 on t1.c3=tmp1.c3; ----- 原始邮件 ----- 发件人：Roberto Cornacchia <roberto.cornacchia(a)gmail.com> 收件人：Communication channel for MonetDB users <users-list(a)monetdb.org> 主题：Re: SUM() OVER (ORDER BY..) 日期：2016-4-29 15:46:57 No, indeed, this syntax is not supported. On 29 April 2016 at 09:41, Anthony Damico <ajdamico(a)gmail.com> wrote: > hmm, that gives me the same error.. > > SELECT mpg, wt, SUM(wt) OVER (PARTITION BY hp) AS cum_wgt, SUM(wt) over () > AS tot_wgt, hp FROM mtcars; > > > Error in .local(conn, statement, ...) : > Unable to execute statement 'SELECT > mpg, > wt, > SUM(wt) OVER (PARTITION BY hp) AS cum_wgt, > SUM(wt) over () AS tot_wgt, > hp > FROM mtc...'. > Server says 'ParseException:SQLparser:SELECT: function 'sum' not found'. > > > sorry if i'm doing something silly.. > > > > > On Fri, Apr 29, 2016 at 3:37 AM, Roberto Cornacchia < > roberto.cornacchia(a)gmail.com> wrote: > >> I think what you are looking for is SUM(..) OVER(PARTITION BY ..) >> >> Otherwise where would the groups for the sum come from? >> >> Roberto >> >> On 29 April 2016 at 09:28, Anthony Damico <ajdamico(a)gmail.com> wrote: >> >>> hi, monetdb does not support SUM() OVER commands.. does anyone have a >>> smart alternative to implement this? thanks >>> >>> >>> SELECT >>> batch_id, >>> job_count, >>> SUM(job_count) OVER (ORDER BY duration) as cumjobs, >>> SUM(job_count) over () as totjobs, >>> duration >>> FROM >>> test_data ; >>> >>> >>> _______________________________________________ >>> users-list mailing list >>> users-list(a)monetdb.org >>> https://www.monetdb.org/mailman/listinfo/users-list >>> >>> >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list >> >> > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > > _______________________________________________ users-list mailing list users-list(a)monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

1 0

The Meaning of Constants in mapi.h
by Brian Hood 28 Apr '16

28 Apr '16

Hi Guys, #define MOK 0 #define MERROR (-1) #define MTIMEOUT (-2) #define MMORE (-3) #define MSERVER (-4) So far i've discovered MOK with value of 0 means connected to MServer and your query went well. MSERVER with value of -4 means that the server doesn't like the query and its bad SQL. I've not been able to discover what the others mean hope someone can help. I've also discovered that mapi_start_talking no longer seems to be implemented in the library guessing its deprecated. Regards, Brian Hood

1 0

Crystal bindings for MonetDB
by Brian Hood 26 Apr '16

26 Apr '16

Hi Guys, Well its been a long, long day putting this together but i actually have working Crystal Language binding for MonetDB. https://github.com/puppetpies/crystal-monetdb-libmapi/blob/4e55863d453be33b… This is the very first stages. I've managed to count 2 rows so far lol. Regards, Brian Hood

2 1

Merge/Remote table causes segfault
by Anderson, David B 26 Apr '16

26 Apr '16

All, I've been able to isolate the seg-fault in gdb, but am not sure how to proceed. Should I file a bug report? There is a NULL pointer access in q_enqueue_(). The q data pointer is NULL. If I look at the flow inside the flow event d, I see the error: "SQLException:sql.bind:unable to find sys.loan201508(purpose)", so it looks like the root cause is higher up/elsewhere. Dave (gdb) where #0 0x00002b39b92fbb2f in q_enqueue_ (q=0x1c0089e0, d=0x1c095b78) at mal_dataflow.c:156 #1 0x00002b39b92fbbbc in q_enqueue (q=0x1c0089e0, d=0x1c095b78) at mal_dataflow.c:164 #2 0x00002b39b92fdd88 in DFLOWscheduler (flow=0x1c094260, w=0x2b39b96a8e40) at mal_dataflow.c:709 #3 0x00002b39b92fe696 in runMALdataflow (cntxt=0x2b39be9b6328, mb=0x1bf02bd0, startpc=1, stoppc=399, stk=0x1c08b7d0) at mal_dataflow.c:892 #4 0x00002b39b93cad1c in MALstartDataflow (cntxt=0x2b39be9b6328, mb=0x1bf02bd0, stk=0x1c08b7d0, pci=0x1c056ac0) at language.c:147 #5 0x00002b39b92f9077 in runMALsequence (cntxt=0x2b39be9b6328, mb=0x1bf02bd0, startpc=1, stoppc=402, stk=0x1c08b7d0, env=0x1be28a30, pcicaller=0x1c08b710) at mal_interpreter.c:631 #6 0x00002b39b92f9418 in runMALsequence (cntxt=0x2b39be9b6328, mb=0x1bda6560, startpc=1, stoppc=0, stk=0x1be28a30, env=0x1be28a30, pcicaller=0x0) at mal_interpreter.c:743 #7 0x00002b39b92f824c in runMAL (cntxt=0x2b39be9b6328, mb=0x1bda6560, mbcaller=0x0, env=0x1be28a30) at mal_interpreter.c:365 #8 0x00002b39b9318b63 in MALengine (c=0x2b39be9b6328) at mal_session.c:637 #9 0x00002b39b9316ebd in runPhase (c=0x2b39be9b6328, phase=4) at mal_scenario.c:515 #10 0x00002b39b931707b in runScenarioBody (c=0x2b39be9b6328) at mal_scenario.c:559 #11 0x00002b39b9317164 in runScenario (c=0x2b39be9b6328) at mal_scenario.c:579 #12 0x00002b39b9317f74 in MSserveClient (dummy=0x2b39be9b6328) at mal_session.c:439 #13 0x00002b39b9317b8b in MSscheduleClient (command=0x1be20190 "p\001\342\033", challenge=0x2b39c017b0c0 "GJMKUZJ4", fin=0x1a3fdf90, fout=0x1bda6a50) at mal_session.c:319 #14 0x00002b39b93d0464 in doChallenge (data=0x1bd88880) at mal_mapi.c:184 #15 0x00002b39b9b4697b in thread_starter (arg=0x1a3fdf60) at gdk_system.c:458 #16 0x00002b39bb03683d in start_thread () from /lib64/libpthread.so.0 #17 0x000000330ded4fdd in clone () from /lib64/libc.so.6 ... (gdb) down #0 0x00002b39b92fbb2f in q_enqueue_ (q=0x1c0089e0, d=0x1c095b78) at mal_dataflow.c:156 156 q->data[q->last++] = d; (gdb) p *q $17 = { size = 0, last = 0, exitcount = 0, data = 0x0, l = { lock = 1 }, s = { sema = { __size = '\000' <repeats 31 times>, __align = 0 }, name = 0x0 } } (gdb) p d $18 = (FlowEvent) 0x1c095b78 (gdb) p *d $19 = { flow = 0x1c094260, pc = 55, blocks = 0, state = 1, clk = 0, cost = -1, hotclaim = 0, argclaim = 0 } (gdb) p *d->flow $20 = { cntxt = 0x2b39be9b6328, mb = 0x1bf02bd0, stk = 0x1c08b7d0, start = 2, stop = 399, status = 0x1c094fe0, error = 0x2b39cc00edc0 "SQLException:sql.bind:unable to find sys.loan201508(purpose)", nodes = 0x1c09a750, edges = 0x1c09c480, flowlock = { lock = 1 }, done = 0x1c0942d0 }

1 1

Fwd: JSON Failed to import
by Ariel Abadi 26 Apr '16

26 Apr '16

Now is working! Thank you guys! Rgds > > > ---------- Forwarded message ---------- > From: Sjoerd Mullender <sjoerd(a)monetdb.org> > Date: 2016-04-21 17:51 GMT-03:00 > Subject: Re: JSON Failed to import > To: users-list(a)monetdb.org > > > This works: > > 301|{"data":[{"cdr_dialnumber":"6596871820"}]} > > So the quoting of the json string doesn't help. > > > On 04/21/2016 09:26 PM, Ariel Abadi wrote: > > Hi! > > > > Can someone help me ? Im trying to upload a CSV file with json data, but > > Im receiving an error. Does anybody has an idea how to solve this? > > > > Thks in advance > > > > mclient -s"COPY 1 RECORDS INTO traf.jsontbl FROM STDIN USING > > DELIMITERS '|','\n','\"' NULL AS 'null'" - < j.csv > > > > *The data on j.csv is:* > > 301|"{\"data\":[{\"cdr_dialnumber\":\"6596871820\"}]}" > > > > *TABLE traf.xx > > * > > > sql>\d jsontbl > > CREATE TABLE "traf"."jsontbl" ( > > "a" INTEGER, > > "b" JSON > > ); > > > > > > > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > > > > -- > Sjoerd Mullender > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > >

1 0

Merge/Remote tables causing remote server to segfault
by Anderson, David B 25 Apr '16

25 Apr '16

All, I am seeing odd behavior with Remote/Merge tables. I have a pair of servers, one of which has a merge table that combines 12 remote & 12 local tables (24 total) into a single large table. Queries from the main server on the merge table cause the remote server to crash with a SIGSEGV. If I create a merge table which only contains the remote tables (loan2) the remote server doesn't crash, but I get different responses for the same query in the same session: /opt/flash1/gnmaloan$ mclient -d loan Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://malxcs1p:50000/loan' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select asofdate,sum(balance),count(loanseqnum) from loan2 group by asofdate; (mapi:monetdb://monetdb@malxcs2p/loan) unable to find sys.loan201402(loanseqnum) sql>select asofdate,sum(balance),count(loanseqnum) from loan2 group by asofdate; +------------+--------------------------+----------+ | asofdate | L1 | L2 | +============+==========================+==========+ | 2014-03-01 | 124075979043384 | 9177029 | | 2014-05-01 | 1.2634454179935e+14 | 9264814 | | 2014-07-01 | 126798740013672 | 9335352 | | 2014-09-01 | 126526301326951 | 9407070 | | 2014-11-01 | 1.2604434513281e+14 | 9478920 | | 2015-01-01 | 1.2644932715183e+14 | 9529639 | | 2015-03-01 | 127602153039423 | 9579604 | | 2015-05-01 | 126615476286119 | 9654842 | | 2015-07-01 | 125415108251273 | 9756258 | | 2015-09-01 | 124427385478338 | 9855118 | | 2015-11-01 | 126150962049943 | 9945919 | | 2016-01-01 | 129313947667802 | 10011556 | +------------+--------------------------+----------+ 12 tuples (0.9s) The only thing I see in the crashing server's log is: 2016-04-25 16:55:50 MSG merovingian[28226]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-04-25 16:55:50 MSG merovingian[28226]: database 'loan' (2706) was killed by signal SIGSEGV 2016-04-25 16:56:37 MSG merovingian[28226]: starting database 'loan', up min/avg/max: 2m/1h/2h, crash average: 1.00 0.90 0.33 (12-2=10) 2016-04-25 16:56:37 MSG loan[28022]: arguments: /opt/mr/monetdb/bin/mserver5 --dbpath=/opt/flash2/monetdb2/loan --set merovingian_uri=mapi:monetdb://malxcs2p:50000/loan --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/flash2/monetdb2/loan/.mapi.sock --set monet_vault_key=/opt/flash2/monetdb2/loan/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes Not sure where to start debugging this. I guess I will see if this still occurs with a small number of rows in the base tables. Thanks, Dave

1 0

MonetDB Remote / Merge bug ?
by Brian Hood 25 Apr '16

25 Apr '16

Hi All, It seems when i revert back the original query the first time it breaks and gives a MAL error then the second time it works fine its also very reproduce able. Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://mdb-master-01:50000/threatmonitor' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select * from ippacket a JOIN tcppacket b on (a.guid = b.guid) limit 1; +-----------+------------+------+------------+------+-------+------+------+------+------+------------+------------+------+------+------+------------+------------+------+-------+------+------+------+------+------+------+------+ | guid | recv_date | ip_d | ip_dst | ip_h | ip_id | ip_l | ip_m | ip_o | ip_p | ip_src | ip_sum | ip_t | ip_t | ip_v | guid | recv_date | tcp_ | tcp_d | tcp_ | tcp_ | tcp_ | tcp_ | tcp_ | tcp_ | tcp_ |> : : : f : : len : : en : f : ff : roto : : : os : tl : er : : : data : port : ack : fin : syn : rst : psh : urg : off :> : : : : : : : : : : : : : : : : : : _len : : : : : : : : :> +===========+============+======+============+======+=======+======+======+======+======+============+============+======+======+======+============+============+======+=======+======+======+======+======+======+======+======+ | 0413e543- | 2015-07-09 | Y | 172.17.0.1 | 5 | 28147 | 60 | N | 0 | 6 | 172.17.42. | 19108 | 0 | 64 | 4 | 0413e543-5 | 2015-07-09 | 0 | 15672 | N | N | Y | N | N | N | 10 | : 52b7-ee...> 20:50:32 : : : : : : : : : 1 : : : : : 2b7-eeda-0 : 20:50:32 : : : : : : : : : : : > +0000 : : : : : : : : : : : : : : a77-a2de...> +0000 : : : : : : : : : : +-----------+------------+------+------------+------+-------+------+------+------+------+------------+------------+------+------+------+------------+------------+------+-------+------+------+------+------+------+------+------+ 1 tuple (250.636ms) !6 columns dropped, 2 fields truncated! sql>select * from ippacket a JOIN tcppacket b on (b.guid = a.guid) limit 1; +-----------+------------+------+------------+------+-------+------+------+------+------+------------+------------+------+------+------+------------+------------+------+-------+------+------+------+------+------+------+------+ | guid | recv_date | ip_d | ip_dst | ip_h | ip_id | ip_l | ip_m | ip_o | ip_p | ip_src | ip_sum | ip_t | ip_t | ip_v | guid | recv_date | tcp_ | tcp_d | tcp_ | tcp_ | tcp_ | tcp_ | tcp_ | tcp_ | tcp_ |> : : : f : : len : : en : f : ff : roto : : : os : tl : er : : : data : port : ack : fin : syn : rst : psh : urg : off :> : : : : : : : : : : : : : : : : : : _len : : : : : : : : :> +===========+============+======+============+======+=======+======+======+======+======+============+============+======+======+======+============+============+======+=======+======+======+======+======+======+======+======+ | 0413e543- | 2015-07-09 | Y | 172.17.0.1 | 5 | 28147 | 60 | N | 0 | 6 | 172.17.42. | 19108 | 0 | 64 | 4 | 0413e543-5 | 2015-07-09 | 0 | 15672 | N | N | Y | N | N | N | 10 | : 52b7-ee...> 20:50:32 : : : : : : : : : 1 : : : : : 2b7-eeda-0 : 20:50:32 : : : : : : : : : : : > +0000 : : : : : : : : : : : : : : a77-a2de...> +0000 : : : : : : : : : : +-----------+------------+------+------------+------+-------+------+------+------+------+------------+------------+------+------+------+------------+------------+------+-------+------+------+------+------+------+------+------+ 1 tuple (248.468ms) !6 columns dropped, 2 fields truncated! sql>select * from ippacket a JOIN tcppacket b on (a.guid = b.guid) limit 1; (mapi:monetdb://monetdb@172.17.0.6/threatmonitor) 'user.l10' undefined in: (rmt2533_X_18_bat_oid_str:bat[:oid,:str],rmt2535_X_21_bat_oid_str:bat[:oid,:str],rmt2537_X_22_bat_oid_int:bat[:oid,:int],rmt2539_X_24_bat_oid_int:bat[:oid,:int],rmt2541_X_25_bat_oid_str:bat[:oid,:str],rmt2543_X_26_bat_oid_str:bat[:oid,:str],rmt2545_X_27_bat_oid_str:bat[:oid,:str],rmt2547_X_28_bat_oid_str:bat[:oid,:str],rmt2549_X_29_bat_oid_str:bat[:oid,:str],rmt2551_X_30_bat_oid_str:bat[:oid,:str],rmt2553_X_31_bat_oid_int:bat[:oid,:int],rmt2555_X_32_bat_oid_int:bat[:oid,:int],rmt2556_X_33_bat_oid_lng:bat[:oid,:lng],rmt2557_X_35_bat_oid_str:bat[:oid,:str],rmt2558_X_36_bat_oid_int:bat[:oid,:int],rmt2559_X_37_bat_oid_str:bat[:oid,:str],rmt2560_X_38_bat_oid_int:bat[:oid,:int],rmt2562_X_39_bat_oid_oid:bat[:oid,:oid]) := user.l10(); sql>select * from ippacket a JOIN tcppacket b on (a.guid = b.guid) limit 1; +-----------+------------+------+------------+------+-------+------+------+------+------+------------+------------+------+------+------+------------+------------+------+-------+------+------+------+------+------+------+------+ | guid | recv_date | ip_d | ip_dst | ip_h | ip_id | ip_l | ip_m | ip_o | ip_p | ip_src | ip_sum | ip_t | ip_t | ip_v | guid | recv_date | tcp_ | tcp_d | tcp_ | tcp_ | tcp_ | tcp_ | tcp_ | tcp_ | tcp_ |> : : : f : : len : : en : f : ff : roto : : : os : tl : er : : : data : port : ack : fin : syn : rst : psh : urg : off :> : : : : : : : : : : : : : : : : : : _len : : : : : : : : :> +===========+============+======+============+======+=======+======+======+======+======+============+============+======+======+======+============+============+======+=======+======+======+======+======+======+======+======+ | 0413e543- | 2015-07-09 | Y | 172.17.0.1 | 5 | 28147 | 60 | N | 0 | 6 | 172.17.42. | 19108 | 0 | 64 | 4 | 0413e543-5 | 2015-07-09 | 0 | 15672 | N | N | Y | N | N | N | 10 | : 52b7-ee...> 20:50:32 : : : : : : : : : 1 : : : : : 2b7-eeda-0 : 20:50:32 : : : : : : : : : : : > +0000 : : : : : : : : : : : : : : a77-a2de...> +0000 : : : : : : : : : : +-----------+------------+------+------------+------+-------+------+------+------+------+------------+------------+------+------+------+------------+------------+------+-------+------+------+------+------+------+------+------+ 1 tuple (213.998ms) !6 columns dropped, 2 fields truncated! sql> Regards, Brian Hood

1 0

A good strategy for loading bulk data into monetdb
by Knezevic Nikola 25 Apr '16

25 Apr '16

Hi all, I'm trying to load large amounts of data into monetdb (migrating from mysql). Since I'm using Jul2015-SP4 release, I can't create local files and I can't use external tools, my approach is as following: For each table, I partition the data based on the day, and then: 10 create one table in monetdb for each day, and a merge table that takes all these tables 20 fetch data from mysql (mysql -B -N -C ...) 30 _pipe_ data to mclient (that is invoked with mclient ... -c "COPY 2000000 RECORDS FROM STDIN INTO ..." -) 40 load data into corresponding table (the right day) 50 sleep some seconds 60 goto 20 Each daily table is around 6'000'000 rows, but I have some that have ~150'000'000 rows. However, I noticed that when I'm doing this, I quickly run out of disk space, as monetdb is creating really big logs (in /data/monetdb/some_schema/sql_logs). Size of that folder is usually 600KB, but once the process of migration starts, it goes to several GBs. Not only that, but /data/monetdb/some_schema/bat also grows a lot. The surprising part, and the motivation for this emails, is how well monetdb compresses the data. If I migrate only one day worth of data, and then restart monetdbd (once or twice) and run 'analyze some_schema', sql_logs size drops to normal levels, and I get a lot of reclaimed space (even in bat directory). I tried to put a 'sleep(120)' after each partition and running 'analyze some_schema' after each iteration, but couldn't get the same behaviour (and it is also quite slow). Is there a way to trigger the compression and reduce disk space overhead without restarting mserver, given the above constraints? Thanks, Nikola

3 4

MonetDB Driver / MAPI Implementation.
by Brian Hood 23 Apr '16

23 Apr '16

Hi All, I am have tried to convert the Ruby MonetDB driver into Crystal with little success as i want to MonetDB with the Crystal Programming Language for higher performance and to handle bigger datasets at more C level performance. I'm currently making a transition from Ruby to Crystal and want to know where the specification of MonetDB's MAPI interface lives. Any help appreciated. Regards, Brian Hood

2 1

Merovingian startup error: rapi.prelude missing
by Anderson, David B 23 Apr '16

23 Apr '16

All, I've built MonetDB from source using 11.21.19, but am having problems starting the Merovingian process. I am getting the following error even though I chose -disable-rintegration in the configure step. Does anyone know what I might have done incorrectly? Thanks, Dave 2016-04-22 23:36:11 MSG loan[7218]: # MonetDB 5 server v11.21.19 "Jul2015-SP4" 2016-04-22 23:36:11 MSG loan[7218]: # Serving database 'loan', using 24 threads 2016-04-22 23:36:11 MSG loan[7218]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-04-22 23:36:11 MSG loan[7218]: # Found 188.787 GiB available main-memory. 2016-04-22 23:36:11 MSG loan[7218]: # Copyright (c) 1993-July 2008 CWI. 2016-04-22 23:36:11 MSG loan[7218]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2016-04-22 23:36:11 MSG loan[7218]: # Visit http://www.monetdb.org/ for further information 2016-04-22 23:36:11 MSG loan[7218]: !TypeException:user.main[14]:object code for command rapi.prelude missing 2016-04-22 23:36:11 MSG loan[7218]: !TypeException:user.main[14]:'rapi.prelude' undefined in: rapi.prelude(); 2016-04-22 23:36:11 MSG merovingian[6082]: database 'loan' (7218) has exited with exit status 0 2016-04-22 23:36:11 MSG merovingian[6082]: database 'loan' has shut down 2016-04-22 23:36:15 ERR merovingian[6082]: client error: database 'loan' started up, but failed to open up a communication channel

3 4

force monetdb not to write intermediate results
by Shmagi Kavtaradze 22 Apr '16

22 Apr '16

I have 64gb ram on the server and want monetdb to use ram instead of writing intermediate results. Is there any possibility to do that? Thanks in advance.

1 0

hash join - why hashing on the smaller bat?
by Roberto Cornacchia 22 Apr '16

22 Apr '16

Related to my previous question about persisting hashes, I would like to throw another one. BATsubjoin has a series of heuristics to decide what type of join implementation to use. When using hash-join, the latest rule says: if nothing else applied, build a hash on the smaller bat. Could you tell me what is the rationale for this? >From what I could verify: - when sizes are comparable: it doesn't really make much difference which side is hashed - when sizes differ much: sure, building the hash table on that is much cheaper, but the join as a whole becomes 4-5 times slower then when hashing on the larger bat. In which case hashing on the larger bat is a good option? Cheers, Roberto

2 3

JSON Failed to import
by Ariel Abadi 21 Apr '16

21 Apr '16

Hi! Can someone help me ? Im trying to upload a CSV file with json data, but Im receiving an error. Does anybody has an idea how to solve this? Thks in advance mclient -s"COPY 1 RECORDS INTO traf.jsontbl FROM STDIN USING DELIMITERS '|','\n','\"' NULL AS 'null'" - < j.csv *The data on j.csv is:* 301|"{\"data\":[{\"cdr_dialnumber\":\"6596871820\"}]}" *TABLE traf.xx* sql>\d jsontbl CREATE TABLE "traf"."jsontbl" ( "a" INTEGER, "b" JSON );

3 3

The extra optimization option with configure
by Vijay Krishna 19 Apr '16

19 Apr '16

Hi, Today while compiling from MonetDB source code, I found the following option that could be given with the ./configure step of installation. --enable-optimize enable extra optimization (default=no) Will enabling this gives an extra optimization for the database's performance? Or its simply optimizing the installation process? Either cases, what does this optimize? Thanks, Vijay.

1 0

Some performance oddities in numerical comparisons on merge tables
by Marc Ballarin 18 Apr '16

18 Apr '16

Hi, I have a larger *merge table*, of which the relevant columns in this query should be: "timestamp" TIMESTAMP "abort_flag" TINYINT "timestamp" is roughly ascending, but far from monotonous. "abort_flag" has a low cardinality with the following possible values and number of occurrences: | 0 | 735660 | | 1 | 30119 | | 2 | 1486 | | 4 | 15 | Both columns are completely populated. Total row numbers are: merge table: 3,773,559,881 current partition: 30,754,353 I get the the following performance numbers: sql>select count(*) from mytable where timestamp between timestamp '2016-04-15 13:05:05' and timestamp '2016-04-15 13:20:05' and abort_flag in (0); +--------+ | L1 | +========+ | 735660 | +--------+ 1 tuple (1.0s) sql>select count(*) from mytable where timestamp between timestamp '2016-04-15 13:05:05' and timestamp '2016-04-15 13:20:05' and abort_flag = 0; +--------+ | L1 | +========+ | 735660 | +--------+ 1 tuple (34.0s) So there is a very significant, and IMHO surprising, performance difference between the "equals" and the "in" comparison. Doing the same query directly against the sub table has a much more consistent performance: sql>select count(*) from mytable_20160415 where timestamp between timestamp '2016-04-15 13:05:05' and timestamp '2016-04-15 13:20:05' and abort_flag IN (0); +--------+ | L1 | +========+ | 735660 | +--------+ 1 tuple (618.914ms) sql>select count(*) from mytable_20160415 where timestamp between timestamp '2016-04-15 13:05:05' and timestamp '2016-04-15 13:20:05' and abort_flag = 0; +--------+ | L1 | +========+ | 735660 | +--------+ 1 tuple (1.1s) The same can be observed with another column, where ">= 4" is much slower than the (in my case) equivalent "IN (4,5,6)". I guess, for my uses I can always switch to the "in" conditions, because all relevant columns should have very low cardinality, but in general, this seems odd. Is there an explanation for this behaviour, like optimizer problems with merge tables? Or maybe a way to avoid it? Regards, Marc

3 3

回复：Re: Some performance oddities in numerical comparisons on merge tables
by chenkaijiang＠bigknow.com.cn 17 Apr '16

17 Apr '16

I understand that if MonetDB uses the timestamp as a filter (more selective), then it is faster. But does the MonetDB get this speed by purely sequence scan on timestamp column with 3,773,559,881 rows, or does it use some technique such as binary search? > Total row numbers are: > merge table: 3,773,559,881 > current partition: 30,754,353 > > I get the the following performance numbers: > sql>select count(*) from mytable where timestamp between timestamp > '2016-04-15 13:05:05' and timestamp '2016-04-15 13:20:05' and abort_flag > in (0); > +--------+ > | L1 | > +========+ > | 735660 | > +--------+ > 1 tuple (1.0s) ----- 原始邮件 ----- 发件人：Niels Nes <Niels.Nes(a)cwi.nl> 收件人：Communication channel for MonetDB users <users-list(a)monetdb.org> 主题：Re: Some performance oddities in numerical comparisons on merge tables 日期：2016-4-17 18:31:11 On Fri, Apr 15, 2016 at 05:18:27PM +0200, Marc Ballarin wrote: > Hi, > > I have a larger *merge table*, of which the relevant columns in this > query should be: > "timestamp" TIMESTAMP > "abort_flag" TINYINT > > "timestamp" is roughly ascending, but far from monotonous. > "abort_flag" has a low cardinality with the following possible values > and number of occurrences: > | 0 | 735660 | > | 1 | 30119 | > | 2 | 1486 | > | 4 | 15 | > > Both columns are completely populated. > > Total row numbers are: > merge table: 3,773,559,881 > current partition: 30,754,353 > > I get the the following performance numbers: > sql>select count(*) from mytable where timestamp between timestamp > '2016-04-15 13:05:05' and timestamp '2016-04-15 13:20:05' and abort_flag > in (0); > +--------+ > | L1 | > +========+ > | 735660 | > +--------+ > 1 tuple (1.0s) > > sql>select count(*) from mytable where timestamp between timestamp > '2016-04-15 13:05:05' and timestamp '2016-04-15 13:20:05' and abort_flag > = 0; > +--------+ > | L1 | > +========+ > | 735660 | > +--------+ > 1 tuple (34.0s) > > So there is a very significant, and IMHO surprising, performance > difference between the "equals" and the "in" comparison. Well not that surprising. The choice on which operator in MonetDB isn't based on the content of the table but using relatively simpel rules. A equality is done before a range, and a range before an in. So in case of the in, the very selective range query is done first. In case of the equality, the equality is done first. Niels > > Doing the same query directly against the sub table has a much more > consistent performance: > sql>select count(*) from mytable_20160415 where timestamp between > timestamp '2016-04-15 13:05:05' and timestamp '2016-04-15 13:20:05' and > abort_flag IN (0); > +--------+ > | L1 | > +========+ > | 735660 | > +--------+ > 1 tuple (618.914ms) > > sql>select count(*) from mytable_20160415 where timestamp between > timestamp '2016-04-15 13:05:05' and timestamp '2016-04-15 13:20:05' and > abort_flag = 0; > +--------+ > | L1 | > +========+ > | 735660 | > +--------+ > 1 tuple (1.1s) > > The same can be observed with another column, where ">= 4" is much > slower than the (in my case) equivalent "IN (4,5,6)". > > I guess, for my uses I can always switch to the "in" conditions, because > all relevant columns should have very low cardinality, but in general, > this seems odd. > > Is there an explanation for this behaviour, like optimizer problems with > merge tables? Or maybe a way to avoid it? > > Regards, > Marc > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list -- Niels Nes, Manager ITF, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room L3.14, phone ++31 20 592-4098 sip:4098@sip.cwi.nl url: https://www.cwi.nl/people/niels e-mail: Niels.Nes(a)cwi.nl _______________________________________________ users-list mailing list users-list(a)monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

2 1

Use buffer instead of cache if possible
by Shmagi Kavtaradze 17 Apr '16

17 Apr '16

I am doing join and bitwise and on a huge data. Monetdb caches 61 GB of ram and writes 152GB on disk. I use dstat for resource monitoring. I see that buffer is not used at all, monetdb just caches memory and used reads/writes. I know monetdb uses mapped files, but is there a possibility to use buffer more than caching? I was not able to find any information how to force process use more buffer. P.S sorry for stupid question, I am not very strong in linux stuff. Thanks in advance.

1 0

Problem with COPY INTO
by Knezevic Nikola 15 Apr '16

15 Apr '16

Hi all, I've lost a significant time trying to get data from mysql database into monetdb. Since I have a lot of data, I wanted to make it as fast and automated as possible, but to no avail. There are several issues I encountered, and I'd appreciate any help on any of them. I'm running the latest monetDB on CentOS: % mclient --version mclient, the MonetDB interactive terminal (Jul2015-SP4) support for command-line editing compiled-in character encoding: UTF-8 1. I tried following the last example on https://www.monetdb.org/Documentation/SQLreference/Permissions in order to get another user to be able to perform COPY INTO, but it fails: % mclient -u monetdb -d some_db password: Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://internal:50000/some_db' Type \q to quit, \? for a list of available commands auto commit mode: on sql>GRANT COPY FROM TO copy_role; syntax error, unexpected COPY in: "grant copy" 2. Ok, since I can't run COPY INTO FROM FILE, I decided to go with FROM STDIN version. To do so, I dump the mysql database and import that in monetdb. That also fails: % mysql -N -B -u reader -pxxx some_db -h some_host -e "SELECT * FROM errortable;"|mclient -d some_db -s "COPY 15 RECORDS INTO errortable FROM STDIN USING DELIMITERS '\t','\n','\'';" - Failed to import table Column value 1 missing syntax error, unexpected sqlINT in: "100" % mclient -u writer -d some_db password: Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://internal:50000/some_db' Type \q to quit, \? for a list of available commands auto commit mode: on sql>\d errortable CREATE TABLE "some_db"."errortable" ( "rejectreason" INTEGER NOT NULL, "text" CHARACTER LARGE OBJECT ); Another try also fails (still using STDIN): % mysql -N -B -u reader -pxxx some_db -h some_host -e "SELECT * FROM errortable;" >test.out % mclient -d some_db -s "COPY 15 RECORDS INTO errortable FROM STDIN USING DELIMITERS '\t','\n','\'';" <test.out Failed to import table Column value 1 missing % mclient -d some_db -s "COPY 15 RECORDS INTO errortable FROM STDIN USING DELIMITERS '\t','\n','\'';" - <test.out Failed to import table Column value 1 missing syntax error, unexpected sqlINT in: "100" (note different error messages, and also note that 100 is a perfectly good sqlINT) If I try the following, this succeeds. However, this is something I'm trying to avoid, as I'd had to create huge files (100+GB), which is an absolute waste of space and time. % mysql -N -B -u reader -pxxx some_db -h some_host -e "SELECT * FROM errortable;" >test.out % mclient -d some_db -s "COPY 15 RECORDS INTO errortable FROM '/home/nikola/test.out' USING DELIMITERS '\t','\n','\'';" COPY INTO: insufficient privileges: COPY INTO from file(s) requires database administrator rights, use 'COPY INTO "errortable" FROM STDIN' instead # obviously, fails due to (1) % mclient -u monetdb -d some_db -s "COPY 15 RECORDS INTO some_db.errortable FROM '/home/nikola/test.out' USING DELIMITERS '\t','\n','\'';" password: 15 affected rows The file in question is super simple, a list of integers with textual description: 100 Some limit exceeded 101 Another limit exceeded 404 Service temporarily not available 405 Service not available 500 Server not available 700 End of world as we know it 701 User already logged in 10000 Book not found 10001 Price reasonability doubtfull 10002 Client is well known 20001 Activation in progress 20002 Pre-load limit exceeded 30000 Pre-load limit exceeded again 1 test 5 Invalid value 0 to divide with Any hints on how to solve this super annoying problem? Thanks, Nikola

6 12

Re: Can MonetDB be slowed down by a single query?
by Vijay Krishna 14 Apr '16

14 Apr '16

Hi Martin, Thanks for the reply. Is there any workaround to restrict the memory used by a single query or client connection - like some configuration variables? I see a lot of worker threads executing a single query in parallel. I studied that the number of worker threads can be controlled by the gdk_threads variable. But is there anyway to restrict the maximum memory that can be used by any particular worker thread? Or can this be controlled at the OS level? We thought of restricting the memory used by the entire mserver with ulimit, but that still doesn't solve the problem of the possibility of a single query causing a trouble. Is there any other way that this can be controlled as the OS level? @Martin and Others, If both are NO, then what are the safety measures to be followed in production to avoid such problems? (Or, I may sound little dumb - is this not a production problem at all?) Has anyone faced any such issues? Thanks in advance. Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305. On Thu, Apr 14, 2016 at 1:20 PM, Vijay Krishna <vijayakrishna55(a)gmail.com> wrote: > Hi Martin, > > Thanks for the reply. > > Is there any workaround to restrict the memory used by a single query or > client connection - like some configuration variables? > > I see a lot of worker threads executing a single query in parallel (please > refer screenshot attached). I studied that the number of worker threads can > be controlled by the gdk_threads variable. But is there anyway to restrict > the maximum memory that can be used by any particular worker thread? > > Or can this be controlled at the OS level? > We thought of restricting the memory used by the entire mserver with > ulimit, but that still doesn't solve the problem of the possibility of a > single query causing a trouble. Is there any other way that this can be > controlled as the OS level? > > @Martin and Others, > If both are NO, then what are the safety measures to be followed in > production to avoid such problems? > (Or, I may sound little dumb - is this not a production problem at all?) > Has anyone faced any such issues? > > Thanks in advance. > > > > > Thanks & Regards, > > Vijayakrishna.P. > Mobile : (+91) 9500402305. > > On Thu, Apr 14, 2016 at 12:15 PM, Martin Kersten <martin.kersten(a)cwi.nl> > wrote: > >> On 14/04/16 08:30, Vijay Krishna wrote: >> >>> Hi, >>> >>> As per MonetDB's architecture, the memory allocated to a single query >>> (thread) is left with the OS and is not controlled by the mserver. >>> >>> If so, I am wondering how MonetDB would react to the following scenario. >>> >>> I receive a client connection with a single costly query whose execution >>> would require all of the RAM's memory. >>> >> Hi >> >> Both queries will run concurrently and fight for the total memory. >> However, after some time, the oldest query is put on lower priority >> >> >>> Now, when I receive a second connection that runs a similar costly >>> operation, will MonetDB start processing this second query by using the >>> available meagre memory (RAM leftovers or from disk swap) or will MonetDB >>> wait till the first query releases some of >>> its memory? >>> >>> >>> >>> Thanks & Regards, >>> >>> Vijayakrishna.P. >>> Mobile : (+91) 9500402305. >>> >>> >>> _______________________________________________ >>> users-list mailing list >>> users-list(a)monetdb.org >>> https://www.monetdb.org/mailman/listinfo/users-list >>> >>> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list >> > >

1 0

Can MonetDB be slowed down by a single query?
by Vijay Krishna 14 Apr '16

14 Apr '16

Hi, As per MonetDB's architecture, the memory allocated to a single query (thread) is left with the OS and is not controlled by the mserver. If so, I am wondering how MonetDB would react to the following scenario. I receive a client connection with a single costly query whose execution would require all of the RAM's memory. Now, when I receive a second connection that runs a similar costly operation, will MonetDB start processing this second query by using the available meagre memory (RAM leftovers or from disk swap) or will MonetDB wait till the first query releases some of its memory? Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

2 1

Strange behaviour of epoch() when converting from Unix time
by Marc Ballarin 12 Apr '16

12 Apr '16

Hi, I am observing a surprising behaviour of the epoch function: The following works as expected: SELECT epoch(now()) AS "NOW", epoch(now() - INTERVAL '5' MINUTE) AS "5 MIN AGO"; +------------+------------+ | NOW | 5 MIN AGO | +============+============+ | 1460453860 | 1460453560 | +------------+------------+ The inverse does not: SELECT epoch(1460453860) AS "NOW", epoch(1460453860-300) AS "300 SECONDS AGO"; +----------------------------+----------------------------+ | NOW | 300 SECONDS AGO | +============================+============================+ | 2016-04-12 09:37:40.000000 | 1970-01-17 21:40:53.560000 | +----------------------------+----------------------------+ The "now" part is fine, the "300 seconds ago" part is obviously not what I expect. Am I missing something or is this a bug? Regards, Marc

2 1

Check reads per sec and CPU utilization
by Shmagi Kavtaradze 11 Apr '16

11 Apr '16

I was not able to find any information how to check CPU utilization and read per sec for a query. Any ideas how to get these results? Thanks in advance.

3 3

Re: How to do profiling in MAL interface?
by Constantine Feng 11 Apr '16

11 Apr '16

Hi Martin, Thanks. stethoscope does do some work. But since it is for online monitoring, it is inconvenient for offline analysis of trace profiles. After reading more docs on the web, I find Tachograph comes closest to what I want. A minor problem is that the schema of the generated files (.trace, .csv, .json) is not given. Somehow I have to guess. And seems that each file format covers a different subset of attributes. - Neo > > ------------------------------ > > Message: 2 > Date: Sat, 9 Apr 2016 08:17:15 +0200 > From: Martin Kersten <martin.kersten(a)cwi.nl> > To: users-list(a)monetdb.org > Subject: Re: How to do profiling in MAL interface? > Message-ID: <57089E6B.7050504(a)cwi.nl> > Content-Type: text/plain; charset=windows-1252; format=flowed > > You better consider using the stethoscope for doing this kind of analysis. > The documentation need some updates. > > > On 09/04/16 06:07, Constantine Feng wrote: > > Hi all, > > > > I know I can use the TRACE SQL command. But I want to do it in the > "-lmal" interface (because I massaged the MAL plans a bit to test my > research ideas). > > > > Particularly, I find the information given on this website ( > https://www.monetdb.org/Documentation/Manuals/MonetDB/Modules/profiler ) > doesn't match the source code. I can't find the profiler.activate() or > profiler.deactivate(), for example. > > > > > > mal>manual.help("profiler.*"); > > profiler.cpuload > > profiler.cpustats > > profiler.cleanup > > profiler.closeStream > > profiler.dumpTrace > > profiler.getSystemTime > > profiler.getUserTime > > profiler.getDiskWrites > > profiler.getDiskReads > > profiler.getTrace > > profiler.noop > > profiler.openStream > > profiler.reset > > profiler.setHeartbeat > > profiler.stop > > profiler.stethoscope > > profiler.tomograph > > > > > > Version: (Jul2015-SP3). Build from source. > > > > - Neo > > > > > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > > > >

1 0

Announcement: New Jul2015-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 11 Apr '16

11 Apr '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP4 bugfix release MonetDB Common * Fixed a bug that caused various instances where old data returned or where crashes occurred. The problem was that internally data wasn't always marked dirty when it was being changed, causing later processing to not deal with the changed data correctly. Bug Fixes * 3905: MonetDB doesn't handle ANY/SOME/ALL operator correctly * 3929: R aggregate not recognized when using 3 or more parameters * 3965: Not possible to quote/escape single quote character in the name of the file to load. * 3968: Missing double use of column names

1 0

Can not enable python
by Shmagi Kavtaradze 10 Apr '16

10 Apr '16

I created function in python, but when tried to call, I receive : Embedded Python has not been enabled. Start server with --set embedded_py=true I stopped farm. used mserver5 --set embedded_py=true and got in mserver5 console output: # MonetDB/SQL module loaded # MonetDB/Python module loaded Then I started farm and still getting same error. Then I did this: monetdb stop dbname monetdb set embedpy=true dbname monetdb start dbname Still getting same. Any ideas what I am doing wrong?

2 2

Fwd: Function does not complete
by Shmagi Kavtaradze 09 Apr '16

09 Apr '16

Thanks for answer. I have bits of size 50, so I need bigint/long long int. Problem is in while loop. When I use bitand(a,b) and numbers of digits of a and b both are 18-19, the system enters endless loop I think. Bigint maximum value is 19 digits long.I have mistakes in code when using unsigned int and I changed, but same problem. As I know from data types page, monetdb uses only signed integers, so I use only signed ones. Any ideas why does it hang on 18-19 digits and not on 1-17 digits long numbers?

1 0

How to do profiling in MAL interface?
by Constantine Feng 09 Apr '16

09 Apr '16

Hi all, I know I can use the TRACE SQL command. But I want to do it in the "-lmal" interface (because I massaged the MAL plans a bit to test my research ideas). Particularly, I find the information given on this website ( https://www.monetdb.org/Documentation/Manuals/MonetDB/Modules/profiler ) doesn't match the source code. I can't find the profiler.activate() or profiler.deactivate(), for example. mal>manual.help("profiler.*"); profiler.cpuload profiler.cpustats profiler.cleanup profiler.closeStream profiler.dumpTrace profiler.getSystemTime profiler.getUserTime profiler.getDiskWrites profiler.getDiskReads profiler.getTrace profiler.noop profiler.openStream profiler.reset profiler.setHeartbeat profiler.stop profiler.stethoscope profiler.tomograph Version: (Jul2015-SP3). Build from source. - Neo

2 1

Controlling --dbextras Using MonetDB(D)
by Alex Manaev 09 Apr '16

09 Apr '16

Hi, I use wrapper shell in such case. May be useful for you. $ mv /usr/bin/mserver5 /usr/bin/mserver5.bin $ vim /usr/bin/mserver5 #!/usr/bin/env bash exec /usr/bin/mserver5.bin --set gdk_vmtrim yes --set gdk_vm_maxsize 200000000000 --dbextra=/mnt/data/monetdb/extra $@ Alexey Manaev, Head of software development CJSC "RUTUBE"

3 4

monetdb crash as start : Logger_new: BBPrename to sql_snapshots_bid failed
by Pierre-Adrien Coustillas 08 Apr '16

08 Apr '16

Hello, after a server crash : # mclient comptage_cookie_telerama monetdbd: internal error while starting mserver, please refer to the logs # cat merovingian.log 2016-04-08 14:28:33 MSG merovingian[13582]: starting database 'comptage_cookie_telerama', up min/avg/max: 28m/4w/10w, crash average: 1.00 1.00 1.00 (246-6=240) 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: arguments: /usr/bin/mserver5 --dbpath=/var/lib/monetdb/comptage_cookie_telerama --set merovingian_uri=mapi:monetdb://groupelemonde-crm:50000/comptage_cookie_telerama --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/lib/monetdb/comptage_cookie_telerama/.mapi.sock --set monet_vault_key=/var/lib/monetdb/comptage_cookie_telerama/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # MonetDB 5 server v11.21.13 "Jul2015-SP2" 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Serving database 'comptage_cookie_telerama', using 8 threads 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Found 64.000 GiB available main-memory. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Copyright (c) 1993-July 2008 CWI. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Visit http://www.monetdb.org/ for further information 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Listening for UNIX domain connection requests on mapi:monetdb:///var/lib/monetdb/comptage_cookie_telerama/.mapi.sock 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Start processing logs sql/sql_logs version 52200 2016-04-08 14:28:34 MSG merovingian[13582]: database 'comptage_cookie_telerama' (16350) has exited with exit status 1 2016-04-08 14:28:34 ERR merovingian[13582]: client error: database 'comptage_cookie_telerama' appears to shut itself down after starting, check monetdbd's logfile for possible hints 2016-04-08 14:28:34 MSG merovingian[13582]: starting database 'comptage_cookie_telerama', up min/avg/max: 28m/4w/10w, crash average: 1.00 1.00 1.00 (247-6=241) 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: arguments: /usr/bin/mserver5 --dbpath=/var/lib/monetdb/comptage_cookie_telerama --set merovingian_uri=mapi:monetdb://groupelemonde-crm:50000/comptage_cookie_telerama --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/lib/monetdb/comptage_cookie_telerama/.mapi.sock --set monet_vault_key=/var/lib/monetdb/comptage_cookie_telerama/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # MonetDB 5 server v11.21.13 "Jul2015-SP2" 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Serving database 'comptage_cookie_telerama', using 8 threads 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Found 64.000 GiB available main-memory. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Copyright (c) 1993-July 2008 CWI. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Visit http://www.monetdb.org/ for further information 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Listening for UNIX domain connection requests on mapi:monetdb:///var/lib/monetdb/comptage_cookie_telerama/.mapi.sock 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Start processing logs sql/sql_logs version 52200 2016-04-08 14:28:34 MSG merovingian[13582]: database 'comptage_cookie_telerama' (16354) has exited with exit status 1 2016-04-08 14:28:39 ERR merovingian[13582]: client error: database 'comptage_cookie_telerama' appears to shut itself down after starting, check monetdbd's logfile for possible hints # cat .gdk_lock USR=107 PID=16354 TIME=Fri Apr 8 14:28:34 2016 @ LOGON USR=107 PID=16354 TIME=Fri Apr 8 14:28:34 2016 @ !FATAL: Logger_new: BBPrename to sql_snapshots_bid failed USR=107 PID=16354 TIME=Fri Apr 8 14:28:34 2016 @ LOGOFF do you have an idea ? Thank, Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35 // 09 54 38 70 90

1 0

any ideas why CPU usage would never go over 25%?
by Anthony Damico 08 Apr '16

08 Apr '16

hi, we have a powerful windows server with 200GB RAM and 40 cores. i've been using it for a few weeks now, and mserver.exe never seems to use more than 10 of the cores. mserver does say "using 40 threads" so they're all available is it just rare for any SQL query to use more than 10 cores at once, or is it possible i've misconfigured something? thanks

1 0

Function does not complete
by Shmagi Kavtaradze 07 Apr '16

07 Apr '16

When I run the own written function it runs and does no exit. I was waiting for 5-6 hours, checked dstats, no disk writes, only 1 core works and no memory usage. It is like stuck and can not exit. It happens when I use function separately for each variable, without callingm for example: while( *a ){ countA += *a & 1; *a >>= 1; } while( *b ){ countB += *b & 1; *b >>= 1; } while( c ){ countC += c & 1; c >>= 1; } but when I call functions like this monetdb works fine: int countSetBits1(unsigned int n){ unsigned int count = 0; while(n){ count += n & 1; n >>= 1; } return count; } countA = countSetBits1(*a); countB = countSetBits1(*b); countC = countSetBits1(c); I am not strong in C or monetdb internals, but in Netbeans both codes work well. The entire code : str UDFbitanddec(flt *ret, lng *a, lng *b) { int c = 0; int countA = 0; int countB = 0; int countC = 0; int maxCount = 0; unsigned int iA = *a; unsigned int iB = *b; c = iA & iB; while( iA ){ countA += iA & 1; iA >>= 1; } while( iB ){ countB += iB & 1; iB >>= 1; } while( c ){ countC += c & 1; c >>= 1; } if (countA >= countB) { maxCount = countA; } else { maxCount = countB; } *ret = (float)countC /maxCount; return MAL_SUCCEED; }

2 1

Exception Error
by Erie, Rob 07 Apr '16

07 Apr '16

I am in the process of evaluating databases for analysis at my organization. I have installed MonetDB and loaded some data. I was attempting to compare results both in performance and accuracy between MySQL and MonetDB. I have been receiving Overflow in Calculation errors from MonetDB. I am a total novice with MonetDB. Here is the information about my system: Red Hat Enterprise Linux Server release 7.2 (Maipo) (Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux) MySQL Server version: 5.7.11-log MySQL Community Server Java version "1.8.0_74"; Java(TM) SE Runtime Environment (build 1.8.0_74-b02); Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode). Apache Tomcat 7.0.54 Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP3) Database: MonetDB v11.21.17 (Jul2015-SP3), 'mapi:monetdb://ip-10-153-0-5.ec2.internal:50000/set_mktng' Issue Description: I would like to know the correlation between two variables group by a variable. SQL: select milageRange, sys.corr(numberOfTheseParts,avgHomeValue) FROM aPartsByZipRange INNER JOIN avgHomeValueZipPeriod ON zipCode = zip where numberOfTheseParts <> 0 and avgHomeValue <>0 group by milageRange; Error: TypeException:user.s78_2[44]:'aggr.subcorr' undefined in: X_75:bat[:any,:int] := aggr.subcorr(X_72:bat[:oid,:int],X_74:bat[:oid,:int],X_67:bat[:oid,:oid],r1_67:bat[:oid,:oid],true:bit); program contains errors -- -------------------------------------------------- This e-mail transmission contains information intended only for the use of the recipient(s) named above. Further, it contains information that may be privileged and confidential. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this message (including any attachments) is strictly prohibited. If you have received this e-mail in error, please notify the sender by reply e-mail and then delete this message from your mail system. Thank you for your compliance.

3 4

Question
by Сергей 07 Apr '16

07 Apr '16

Does anyone create replication with MonetDB? I've tried many times but still cannot run MonetDB in master-slave mode(replication). I have such errors when start slave server: !mvc_init: unable to create system tables !SQLException:SQLinit:Catalogue initialization failed I've done all that written in https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/TransactionRepli… but it does not work. Does anyone can help me? Is transaction-replication included to July2015SP3 release? Or I must build MonetDB from sources? Thanks.

1 0

Create R function that can be applied to an entire table
by Willem Ligtenberg 07 Apr '16

07 Apr '16

Hi, I guess I am pushing the user defined R functions to their limits. I know want to implement some batch correcting in R and then apply that to an entire table. I have written R code, that will generate a function definition that has matching parameters. And then I want to call the function as follows: CREATE TABLE data_bc AS SELECT * FROM batch_correction( (SELECT * FROM data) ) WITH DATA; I then get the following error: Error in .local(conn, statement, ...) : Unable to execute statement 'CREATE TABLE hcs_batch AS SELECT * FROM batch_correction( (SELECT * FROM hcs_norm) ) WITH DATA;'. Server says '!Command too large' Looking at the source, this error is raised in rapi.c on either line 467 or 473. After editing the source to figure out which of the lines was triggered, I figured out is was on line: 467 it is triggered, so the following code raises the issue that the command is too large: pos = 0; for (i = pci->retc + 2; i < pci->argc && pos < sizeof(argnames); i++) { pos += snprintf(argnames + pos, sizeof(argnames) - pos, "%s%s", args[i], i < pci->argc - 1 ? ", " : ""); } if (pos >= sizeof(argnames)) { msg = createException(MAL, "rapi.eval", "Command too large1"); goto wrapup; } (yes, I added the 1 to large, to make it unique) If this is not a bug, any help in getting my code to work is greatly appreciated. Kind regards, Willem

2 3

Disabling do_sort() does not work.
by Xu,Wenjian 06 Apr '16

06 Apr '16

Hi, I would like to replace MonetDB' sorting algorithms (timsort/quicksort) with my own. As the first step, I commented all content of function *do_sort()* in gdk_batop.c to disable the sorting functionality. Then I re-build the monetdb (with no error) and type a SQL statement with ORDER-BY clause in mclient interface: "select o_orderkey, o_custkey, o_orderstatus from orders order by o_custkey limit 10;" I was expecting to see the dis-ordered result. However, it turns out that the result is still ordered by o_custkey. I am confused about the result. Did I miss something? Any help would be great. Best regards, John

1 0

accessing intermediate optimizer results.
by Xu,Wenjian 01 Apr '16

01 Apr '16

Hi all, I would like to see the intermediate result (i.e., MAL instructions) after each optimizer step in a optimizer pipeline. So I searched Monetdb's documents and found the following sentence in https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/OptimizerPipelin… : "Alternatively, the SQL DEBUG statement modifier in combination with the 'o' command provides access to the intermediate optimizer results." I cannot figure out what this sentence means. What I know about SQL DEBUG statement is to use "debug <SQL statement>" to enter the debug mode and iterate through each (final) MAL instruction one by one. How could I use 'o' command to access intermediate optimizer results? I am using MonetDB 11.21.17 (JUL2015 SP3) and my environment is Ubuntu14.04. Thank you very much. Best regards, XU Wenjian

2 4

No previous prototype for function
by Shmagi Kavtaradze 31 Mar '16

31 Mar '16

I use two versions of Monetdb. 1 is PythonUDF and another Monetdb 11.21.13. Function "bitand" is compiled and used well in Monetdb, but when I try to compile PythonUDF it gives me errors: udf.c:14:1: error: no previous prototype for 'UDFbitand' [-Werror=missing-prototypes] UDFbitand(flt *ret, lng*a, lng *b) ^ cc1: all warnings being treated as errors Makefile:876: recipe for target 'lib_udf_la-udf.lo' failed any ideas? thanks in advance.

4 5

no sorted column?
by Roberto Cornacchia 31 Mar '16

31 Mar '16

Before filing a bug report, I'd like to ask here whether this is intentional/expected. sql>create table t(a int); operation successful (0.650ms) sql>insert into t values (1),(0),(3),(2); 4 affected rows (0.638ms) sql> sql>create table sorted_t as select a from t order by a with data; operation successful (0.807ms) sql> sql>select table,column,sorted from sys.storage() where "table"='sorted_t'; +----------+--------+--------+ | table | column | sorted | +==========+========+========+ | sorted_t | a | false | +----------+--------+--------+ I am pretty sure it use to be that the sorting used in a "CREATE TABLE AS ... WITH DATA" would result in the tsorted property to be true for the (first) sorted column. Now this isn't the case. Is that a bug or a feature? Roberto

4 6

Possible serious Memory leak?
by Vijay Krishna 31 Mar '16

31 Mar '16

Hi, Today, we encountered this serious memory issue with MonetDB. *Data : *10 tables - each with ~600 columns - all tables are EMPTY (No rows). *Query : *One query that involves with 25 joins across all the 10 tables. *MonetDB version :* July2015-SP3 The above query is ran on the same MonetDB version with the same table structure with all tables empty on two different machines as follows. *Scenario 1 - Query ran on Macbook Pro * - RAM - 8 GB (with other processes like Chrome/Eclipse running in parallel) - Free disk space - ~100 GB - MonetDB July2015-SP3 downloaded as installable The query is ran. *Expected result* - return an empty set in a few seconds. *Actual result* - returns an empty set in ~5 minutes RAM - 8 GB utilised completely. Swap space used from disk - 10 GB This swap space is cleared after sometime automatically (around 5 mins) *Scenario 2 - With a CentOS virtual machine* - RAM - 128 GB - Free disk space - ~650 GB - MonetDB July2015-SP3 compiled from source code The query is ran. *Expected result* - return an empty set in a few seconds. *Actual result* - the query rebooted this high end CentOS machine. RAM - 128 GB utilised completely. After reboot, the swap memory was freed. I am puzzled with the following questions. 1. Why should a query that runs on 10 EMPTY tables require memory in GBs? 2. How did this query manage to run on a Mac with just 8 GB RAM, but crashed a machine with 128 GB RAM? Is this a known memory leak issue? Any help much appreciated. Thanks in advance. Regards, Vijay.

3 3

Find all running processes?
by Vijay Krishna 31 Mar '16

31 Mar '16

How to find all running processes and their statuses in MonetDB?? I found this... https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/monitor But I couldn't get this work... I ran a query (which took some 20 seconds to complete) in a window and connected the same database in another terminal and ran "select * from sys.queue" (several times during those 20 seconds) but I could get only this... sql>select * from sys.queue; +-------+---------+----------------------------+----------------------------+----------+---------+------------+--------------------------+ | qtag | user | started | estimate | progress | status | tag | query | +=======+=========+============================+============================+==========+=========+============+==========================+ | 27339 | monetdb | 2015-08-13 05:35:40.000000 | null | null | running | 12532134@0 | select * from sys.queue; | +-------+---------+----------------------------+----------------------------+----------+---------+------------+--------------------------+ I am using Oct-2014 version of monetdb on CentOS... Is there a way to find all running processes and their statuses in MonetDB?? Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

2 3

Speeding SELECT for a small number of fields OVER large number of record in Perl DBD::monetdb
by Puthick Hok 31 Mar '16

31 Mar '16

Hi, I'm using Perl DBD::monetdb from MonetDB source for close to a year now. Recently, I discovered using SQL SELECT for one or two fields for a large number of records, for example, 100000 records is too slow compared to how faster MonetDB engine is. The same SQL SELECT is way faster in MySQL due to C implementation of DBD::mysql. MonetDB connector in Python is still way faster than Perl DBD::monetdb because it allows user to tell the connector the maximum number of records the user needs and the connector uses Xexport command straightaway and only once. In Perl, DBD::monetdb calls Mapi.pm via getReply which issues Xexport command many times depending on the number of records in the SELECT statement. Each time, it retrieves only 100 records. For 100000 records, it needs to call Xexport 1000 times. I did my own patch to make getReply call Xexport only once a the rest of the records after the initial request. With the patch, the time it takes to select 2 fields for 101024 records drops from more 24 seconds to under 4 seconds. My patch is: <pre> sub getReply { my ($self)= @_; if ($self->{active} == 0) { return $self->getBlock(); } elsif ($self->{next} < $self->{replysize} + $self->{skip}) { return $self->getRow(); } elsif (${self}->{offset} + $self->{replysize} < $self->{count}) { # get next slice my $rs = $self->{replysize}; my $offset = $self->{offset} + $rs; my $therest = $self->{count} - $self->{replysize}; $self->putblock("Xexport $self->{id} $offset $therest"); # FAST #$self->putblock("Xexport $self->{id} $offset $rs"); # SLOW return $self->getBlock(); } else { # close large results, but only send on next query if ($self->{id} > 0 && $self->{count} != $self->{replysize}) { push @{$self->{piggyback}}, "Xclose $self->{id}"; $self->{skip_in}++; } $self->{active} = 0; } return $self->{active}; } </pre> I hope this could help other people. Regards, Puthick

1 0

Re: Temporary duplicated rows
by Alex Manaev 30 Mar '16

30 Mar '16

Effect is reproduced with console on the fresh database. Here is real time console log. Can anybody explain what happens? sql>select id, status from morgoth_reportquery where status <> 4 and id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 4 | | 6880 | 4 | | 6880 | 4 | +------+--------+ 3 tuples (2.551ms) <--- result of previous experiments sql>select id, status from morgoth_reportquery where id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 4 | +------+--------+ 1 tuple (2.746ms) sql>update morgoth_reportquery set status = 2 where id = 6880; 1 affected row (3.306ms) sql>select id, status from morgoth_reportquery where id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 4 | +------+--------+ 1 tuple (2.216ms) sql>select id, status from morgoth_reportquery where status <> 4 and id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 4 | | 6880 | 4 | | 6880 | 4 | | 6880 | 4 | +------+--------+ 4 tuples (2.634ms) sql>select id, status from morgoth_reportquery where status = 4 and id = 6880; +----+--------+ | id | status | +====+========+ +----+--------+ 0 tuples (2.337ms) sql>select id, status from morgoth_reportquery where status = 1 and id = 6880; +----+--------+ | id | status | +====+========+ +----+--------+ 0 tuples (3.294ms) sql>select id, status from morgoth_reportquery where status = 4 and id = 6880; +----+--------+ | id | status | +====+========+ +----+--------+ 0 tuples (2.416ms) sql>select id, status from morgoth_reportquery where id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 2 | +------+--------+ 1 tuple (2.186ms) sql>select id, status from morgoth_reportquery where status <> 4 and id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 2 | +------+--------+ 1 tuple (4.145ms) sql> Alexey Manaev, Head of software development CJSC "RUTUBE" On 30.03.2016 17:43, users-list-request(a)monetdb.org wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Temporary duplicated rows (Alex Manaev) > 2. UDF returning 0 as a result (Shmagi Kavtaradze) > 3. Re: UDF returning 0 as a result (Mark Raasveldt) > 4. Re: UDF returning 0 as a result (Sjoerd Mullender) > 5. Re: Find all running processes? (Pierre-Adrien Coustillas) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 30 Mar 2016 14:18:16 +0300 > From: Alex Manaev <amanaev(a)rutube.ru> > To: <users-list(a)monetdb.org> > Subject: Temporary duplicated rows > Message-ID: <56FBB5F8.7060309(a)rutube.ru> > Content-Type: text/plain; charset="utf-8"; format=flowed > > Hi, > > I have been faced with strange behavior. > > Sometimes after update that changes field "status", a select query > returns multiple identical rows. > > > sql>select * from morgoth_reportquery where not status = 2 and id = > 9472 order by id desc limit 5; > +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ > | id | repo | name | created_date > | ready_date | stat | hash | > params | user | appr | temp | noti | > : : rt_i : : > : : us : > : : _id : oved : late : fica : > : : d : : > : : : > : : : : _id : tion : > : : : : > : : : > : : : : : _ema : > : : : : > : : : > : : : : : ils : > +======+======+=============================+============================+============================+======+=============================+=============================+======+======+======+======+ > | 9472 | 13 | ??????????? ??????: ??????? | 2015-12-01 12:28:59.000000 > | null | 1 | 5f34b8459f4834867a2621f7d6b | - > !!python/tuple [from_date | 1 | 0 | 8 | | > : : : ???????? 2016-02-01 - 2016- : > : : : 65c8a : , > 2016-02-01] : : : : : > : : : 02-11 : > : : : : - > !!python/tuple [to_date, : : : : : > : : : : > : : : : > 2016-02-11] : : : : : > : : : : > : : : : - > !!python/tuple [update_me : : : : : > : : : : > : : : : diainfo, > 0] : : : : : > : : : : > : : : : - > !!python/tuple : : : : : > : : : : > : : : : - > agreements... > : : : : > | 9472 | 13 | ??????????? ??????: ??????? | 2015-12-01 12:28:59.000000 > | null | 1 | 5f34b8459f4834867a2621f7d6b | - > !!python/tuple [from_date | 1 | 0 | 8 | | > : : : ???????? 2016-02-01 - 2016- : > : : : 65c8a : , > 2016-02-01] : : : : : > : : : 02-11 : > : : : : - > !!python/tuple [to_date, : : : : : > : : : : > : : : : > 2016-02-11] : : : : : > : : : : > : : : : - > !!python/tuple [update_me : : : : : > : : : : > : : : : diainfo, > 0] : : : : : > : : : : > : : : : - > !!python/tuple : : : : : > : : : : > : : : : - > agreements... > : : : : > +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ > 2 tuples (8.446ms) !2 fields truncated! > > > In same time another queries return only one row (normal state) > > sql>select * from morgoth_reportquery where status = 1 and id = 9472 > order by id desc limit 5; > +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ > | id | repo | name | created_date > | ready_date | stat | hash | > params | user | appr | temp | noti | > : : rt_i : : > : : us : > : : _id : oved : late : fica : > : : d : : > : : : > : : : : _id : tion : > : : : : > : : : > : : : : : _ema : > : : : : > : : : > : : : : : ils : > +======+======+=============================+============================+============================+======+=============================+=============================+======+======+======+======+ > | 9472 | 13 | ??????????? ??????: ??????? | 2015-12-01 12:28:59.000000 > | null | 1 | 5f34b8459f4834867a2621f7d6b | - > !!python/tuple [from_date | 1 | 0 | 8 | | > : : : ???????? 2016-02-01 - 2016- : > : : : 65c8a : , > 2016-02-01] : : : : : > : : : 02-11 : > : : : : - > !!python/tuple [to_date, : : : : : > : : : : > : : : : > 2016-02-11] : : : : : > : : : : > : : : : - > !!python/tuple [update_me : : : : : > : : : : > : : : : diainfo, > 0] : : : : : > : : : : > : : : : - > !!python/tuple : : : : : > : : : : > : : : : - > agreements... > : : : : > +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ > 1 tuple (6.474ms) !1 field truncated! > > > Another queries without filter by field `status` give a normal result > (one row). > > Effect is eliminated in several minutes after update by itself. > > > DBMS : MonetDB v11.21.17 (Jul2015-SP3) > OS: CentOS 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC > 2015 x86_64 x86_64 x86_64 GNU/Linux > > Issue was reproduced for the first time with version SP-2. > > Database was created on SP-1. Issue had not appeared with this version. > > > >

1 0

Can no connect to Monetdb database
by Shmagi Kavtaradze 30 Mar '16

30 Mar '16

I compiled Monetdb-python version over regular Monetdb source compilation and after it I receive error: 2016-03-30 20:14:11 MSG merovingian[23154]: starting database 'db', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) 2016-03-30 20:14:11 MSG db[23176]: arguments: /usr/local/bin/mserver5 --dbpath=/home/shmagi/db/db --set merovingian_uri=mapi:monetdb://infdbserv02:50000/db --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/shmagi/db/db/.mapi.sock --set monet_vault_key=/home/shmagi/db/db/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-03-30 20:14:11 MSG merovingian[23154]: database 'db' (23176) has crashed (dumped core) 2016-03-30 20:14:16 ERR merovingian[23154]: client error: database 'db' appears to shut itself down after starting, check monetdbd's logfile for possible hints I tried ro ./configure, make clean, make , make install from pure monetdb source (without python), but still receive these errors. I tried to use prefix to install in another directory, tried to delete manually all files I found, but still same error. Any ideas how to fix this problem?

1 0

UDF returning 0 as a result
by Shmagi Kavtaradze 30 Mar '16

30 Mar '16

I wrote a function bitand. When I do "select bitand(1111,1010);" it gives "0" instead of "0.5". My function: --UDF.c-- str UDFbitand(long long int a, long long b) { long long int weirdAnd(unsigned int a, unsigned int b) { long long int result = 0; int coef = 1; while (a || b) { result += ((a % 10) && (b % 10)) * coef; coef *= 10; a /= 10; b /= 10; } return result; } long long int temp = weirdAnd(a, b); long long int temp2 = weirdAnd(a, b); int length; int count; while (temp != 0) { temp/=10; length++; } while (temp2 != 0) { if (temp2 % 10 == 1) count++; temp2 /= 10; } float final = count/(float)length; return MAL_SUCCEED; } --UDF.mal-- command bitand(a:lng,b:lng):flt address UDFbitand comment "bitand"; --80_UDF.sql-- create function bitand(a BIGINT,b BIGINT) returns REAL external name udf.bitand; When I test my C code in netbeans it works as supposed. Any ideas? Thanks in advance.

3 3

Temporary duplicated rows
by Alex Manaev 30 Mar '16

30 Mar '16

Hi, I have been faced with strange behavior. Sometimes after update that changes field "status", a select query returns multiple identical rows. sql>select * from morgoth_reportquery where not status = 2 and id = 9472 order by id desc limit 5; +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ | id | repo | name | created_date | ready_date | stat | hash | params | user | appr | temp | noti | : : rt_i : : : : us : : : _id : oved : late : fica : : : d : : : : : : : : : _id : tion : : : : : : : : : : : : : _ema : : : : : : : : : : : : : ils : +======+======+=============================+============================+============================+======+=============================+=============================+======+======+======+======+ | 9472 | 13 | Ежемесячные отчеты: Медиате | 2015-12-01 12:28:59.000000 | null | 1 | 5f34b8459f4834867a2621f7d6b | - !!python/tuple [from_date | 1 | 0 | 8 | | : : : хнологии 2016-02-01 - 2016- : : : : 65c8a : , 2016-02-01] : : : : : : : : 02-11 : : : : : - !!python/tuple [to_date, : : : : : : : : : : : : : 2016-02-11] : : : : : : : : : : : : : - !!python/tuple [update_me : : : : : : : : : : : : : diainfo, 0] : : : : : : : : : : : : : - !!python/tuple : : : : : : : : : : : : : - agreements... > : : : : | 9472 | 13 | Ежемесячные отчеты: Медиате | 2015-12-01 12:28:59.000000 | null | 1 | 5f34b8459f4834867a2621f7d6b | - !!python/tuple [from_date | 1 | 0 | 8 | | : : : хнологии 2016-02-01 - 2016- : : : : 65c8a : , 2016-02-01] : : : : : : : : 02-11 : : : : : - !!python/tuple [to_date, : : : : : : : : : : : : : 2016-02-11] : : : : : : : : : : : : : - !!python/tuple [update_me : : : : : : : : : : : : : diainfo, 0] : : : : : : : : : : : : : - !!python/tuple : : : : : : : : : : : : : - agreements... > : : : : +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ 2 tuples (8.446ms) !2 fields truncated! In same time another queries return only one row (normal state) sql>select * from morgoth_reportquery where status = 1 and id = 9472 order by id desc limit 5; +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ | id | repo | name | created_date | ready_date | stat | hash | params | user | appr | temp | noti | : : rt_i : : : : us : : : _id : oved : late : fica : : : d : : : : : : : : : _id : tion : : : : : : : : : : : : : _ema : : : : : : : : : : : : : ils : +======+======+=============================+============================+============================+======+=============================+=============================+======+======+======+======+ | 9472 | 13 | Ежемесячные отчеты: Медиате | 2015-12-01 12:28:59.000000 | null | 1 | 5f34b8459f4834867a2621f7d6b | - !!python/tuple [from_date | 1 | 0 | 8 | | : : : хнологии 2016-02-01 - 2016- : : : : 65c8a : , 2016-02-01] : : : : : : : : 02-11 : : : : : - !!python/tuple [to_date, : : : : : : : : : : : : : 2016-02-11] : : : : : : : : : : : : : - !!python/tuple [update_me : : : : : : : : : : : : : diainfo, 0] : : : : : : : : : : : : : - !!python/tuple : : : : : : : : : : : : : - agreements... > : : : : +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ 1 tuple (6.474ms) !1 field truncated! Another queries without filter by field `status` give a normal result (one row). Effect is eliminated in several minutes after update by itself. DBMS : MonetDB v11.21.17 (Jul2015-SP3) OS: CentOS 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Issue was reproduced for the first time with version SP-2. Database was created on SP-1. Issue had not appeared with this version. -- Alexey Manaev, Head of software development CJSC "RUTUBE"

1 0

Controlling --dbextras Using MonetDB(D)
by Vincent Sheffer 30 Mar '16

30 Mar '16

mserver5 has an argument called —dbextras. I prefer to control mserver5 using monetdbd, but I don't see any way of specifying this option in either monetdb or monetdbd. Is there, say, a property that can be set in monetdbd or monetdb. The latter probably makes more sense, as I think I would want to control it at the DB level not the dbfarm level. Thanks, Vince The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.

1 0

simultaneously-created tables not generating
by Anthony Damico 29 Mar '16

29 Mar '16

i have three clients connected to one mserver. they are simultaneously running these three commands, which share table `z` but not the x or y tables-- CREATE TABLE y1 AS SELECT x1.* FROM x1 LEFT JOIN SELECT col FROM Z ON x1.id = z.id WITH DATA ; CREATE TABLE y2 AS SELECT x2.* FROM x2 LEFT JOIN SELECT col FROM Z ON x2.id = z.id WITH DATA ; CREATE TABLE y3 AS SELECT x3.* FROM x3 LEFT JOIN SELECT col FROM Z ON x3.id = z.id WITH DATA ; when all three commands complete, tables y2 and y3 do not exist if i run the commands sequentially, everything works. am i making a mistake in how sql transactions should behave, or is this potentially a bug that i should figure out how to reproduce for bugzilla? thanks!

2 4

Ways to avoid OOMKiller problem?
by Vijay Krishna 29 Mar '16

29 Mar '16

Hi, We came across this - https://www.monetdb.org/Documentation/OOMkiller and we were thinking on some ways to overcome this situation. 1. Is there any configuration in MonetDB to restrict the maximum RAM usage of the mserver process? 2. Is there any configuration in MonetDB to restrict the maximum RAM used by a single thread? 3. Is there any suggested solution for the OOMKiller problem? If you have explored anything else on this, please share your experience with MonetDB server Memory management. Any help much appreciated. Thanks in advance. Regards, Vijay.

3 3

speed cost of stacking tables that do not need to be stacked
by Anthony Damico 27 Mar '16

27 Mar '16

just some UNION ALL speed tests that others might find useful.. i have one big table for each year of data. the year-tables can be analyzed separately - because all of my desired results are going to be within-year averages and counts. since each table has a single year of data, these two statements give equivalent results: query #1: SELECT * FROM ( ( SELECT yr , AVG( value ) FROM y1 GROUP BY yr ) UNION ALL ( SELECT yr , AVG( value ) FROM y2 GROUP BY yr ) ) AS zzz ; query #2: SELECT yr , AVG( value ) FROM ( ( SELECT yr , value FROM y1 ) UNION ALL ( SELECT yr , value FROM y2 ) ) AS zzz GROUP BY yr ; query #1: 290 seconds query #2: 390 seconds query #1: 22 seconds query #2: 87 seconds fwiw- this would take about 24 hours in anything other than monetdb ;)

1 0

database has crashed (dumped core)
by Shmagi Kavtaradze 25 Mar '16

25 Mar '16

I recomplied monetdb several times (make & make install). Everything was fine. Then I downloaded monetdb_python. It was not able to install and I decided to recompile and install again regular Monetdb, but when I "mclient -u monetdb -d voc" I get error in the log file: 2016-03-25 17:26:18 MSG merovingian[49368]: starting database 'voc', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) 2016-03-25 17:26:18 MSG voc[49379]: arguments: /usr/local/bin/mserver5 --dbpath=/home/shmagi/db/voc --set merovingian_uri=mapi:monetdb://infdbserv02:50000/voc --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/shmagi/db/voc/.mapi.sock --set monet_vault_key=/home/shmagi/db/voc/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-03-25 17:26:18 MSG merovingian[49368]: database 'voc' (49379) has crashed (dumped core) 2016-03-25 17:26:18 ERR merovingian[49368]: client error: database 'voc' appears to shut itself down after starting, check monetdbd's logfile for possible hints 2016-03-25 17:26:18 MSG merovingian[49368]: starting database 'voc', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) 2016-03-25 17:26:19 MSG voc[49382]: arguments: /usr/local/bin/mserver5 --dbpath=/home/shmagi/db/voc --set merovingian_uri=mapi:monetdb://infdbserv02:50000/voc --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/shmagi/db/voc/.mapi.sock --set monet_vault_key=/home/shmagi/db/voc/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-03-25 17:26:19 MSG merovingian[49368]: database 'voc' (49382) has crashed (dumped core) 2016-03-25 17:26:23 ERR merovingian[49368]: client error: database 'voc' appears to shut itself down after starting, check monetdbd's logfile for possible hints. Any ideas how to fix or delete all files of Monetdb to install from the scratch?

1 0

Delete monetdb
by Shmagi Kavtaradze 25 Mar '16

25 Mar '16

Sorry for a stupid questions, but how can I completely delete Monetdb from ubuntu, if I compiled it myself?

4 4

could this be multithreaded? happens after any large import after server stop & restart
by Anthony Damico 25 Mar '16

25 Mar '16

it is dreadfully slow on windows. thanks all # MonetDB 5 server v11.21.13 "Jul2015-SP2" # Serving database 'marketscan', using 40 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 223.875 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50100/ # Start processing logs sql/sql_logs version 52200 # Start reading the write-ahead log 'sql_logs\sql\log.14' # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (0% done) # still reading write-ahead log "sql_logs\sql\log.14" (1% done) # still reading write-ahead log "sql_logs\sql\log.14" (1% done) # still reading write-ahead log "sql_logs\sql\log.14" (1% done) # still reading write-ahead log "sql_logs\sql\log.14" (1% done) # still reading write-ahead log "sql_logs\sql\log.14" (1% done) # still reading write-ahead log "sql_logs\sql\log.14" (1% done) # still reading write-ahead log "sql_logs\sql\log.14" (2% done) # still reading write-ahead log "sql_logs\sql\log.14" (2% done) # still reading write-ahead log "sql_logs\sql\log.14" (2% done) # still reading write-ahead log "sql_logs\sql\log.14" (2% done) # still reading write-ahead log "sql_logs\sql\log.14" (2% done) # still reading write-ahead log "sql_logs\sql\log.14" (3% done) # still reading write-ahead log "sql_logs\sql\log.14" (3% done) # still reading write-ahead log "sql_logs\sql\log.14" (3% done) # still reading write-ahead log "sql_logs\sql\log.14" (3% done) # still reading write-ahead log "sql_logs\sql\log.14" (3% done) # still reading write-ahead log "sql_logs\sql\log.14" (4% done) # still reading write-ahead log "sql_logs\sql\log.14" (4% done) # still reading write-ahead log "sql_logs\sql\log.14" (4% done) # still reading write-ahead log "sql_logs\sql\log.14" (4% done) # still reading write-ahead log "sql_logs\sql\log.14" (4% done) # still reading write-ahead log "sql_logs\sql\log.14" (4% done) # still reading write-ahead log "sql_logs\sql\log.14" (5% done) # still reading write-ahead log "sql_logs\sql\log.14" (5% done) # still reading write-ahead log "sql_logs\sql\log.14" (5% done) # still reading write-ahead log "sql_logs\sql\log.14" (5% done) # still reading write-ahead log "sql_logs\sql\log.14" (5% done) # still reading write-ahead log "sql_logs\sql\log.14" (6% done) # still reading write-ahead log "sql_logs\sql\log.14" (6% done) # still reading write-ahead log "sql_logs\sql\log.14" (6% done) # still reading write-ahead log "sql_logs\sql\log.14" (6% done) # still reading write-ahead log "sql_logs\sql\log.14" (6% done) # still reading write-ahead log "sql_logs\sql\log.14" (7% done) # still reading write-ahead log "sql_logs\sql\log.14" (7% done) # still reading write-ahead log "sql_logs\sql\log.14" (7% done) # still reading write-ahead log "sql_logs\sql\log.14" (7% done) # still reading write-ahead log "sql_logs\sql\log.14" (7% done) # still reading write-ahead log "sql_logs\sql\log.14" (7% done) # still reading write-ahead log "sql_logs\sql\log.14" (8% done) # still reading write-ahead log "sql_logs\sql\log.14" (8% done) # still reading write-ahead log "sql_logs\sql\log.14" (8% done) # still reading write-ahead log "sql_logs\sql\log.14" (8% done) # still reading write-ahead log "sql_logs\sql\log.14" (8% done) # still reading write-ahead log "sql_logs\sql\log.14" (8% done) # still reading write-ahead log "sql_logs\sql\log.14" (8% done) # still reading write-ahead log "sql_logs\sql\log.14" (8% done) # still reading write-ahead log "sql_logs\sql\log.14" (9% done) # still reading write-ahead log "sql_logs\sql\log.14" (9% done) # still reading write-ahead log "sql_logs\sql\log.14" (9% done) # still reading write-ahead log "sql_logs\sql\log.14" (9% done) # still reading write-ahead log "sql_logs\sql\log.14" (9% done) # still reading write-ahead log "sql_logs\sql\log.14" (10% done) # still reading write-ahead log "sql_logs\sql\log.14" (10% done) # still reading write-ahead log "sql_logs\sql\log.14" (10% done) # still reading write-ahead log "sql_logs\sql\log.14" (10% done) # still reading write-ahead log "sql_logs\sql\log.14" (10% done) # still reading write-ahead log "sql_logs\sql\log.14" (11% done) # still reading write-ahead log "sql_logs\sql\log.14" (11% done) # still reading write-ahead log "sql_logs\sql\log.14" (11% done) # still reading write-ahead log "sql_logs\sql\log.14" (11% done) # still reading write-ahead log "sql_logs\sql\log.14" (12% done) # still reading write-ahead log "sql_logs\sql\log.14" (12% done) # still reading write-ahead log "sql_logs\sql\log.14" (12% done) # still reading write-ahead log "sql_logs\sql\log.14" (12% done) # still reading write-ahead log "sql_logs\sql\log.14" (12% done) # still reading write-ahead log "sql_logs\sql\log.14" (13% done) # still reading write-ahead log "sql_logs\sql\log.14" (13% done) # still reading write-ahead log "sql_logs\sql\log.14" (13% done) # still reading write-ahead log "sql_logs\sql\log.14" (13% done) # still reading write-ahead log "sql_logs\sql\log.14" (13% done) # still reading write-ahead log "sql_logs\sql\log.14" (13% done) # still reading write-ahead log "sql_logs\sql\log.14" (14% done) # still reading write-ahead log "sql_logs\sql\log.14" (14% done) # still reading write-ahead log "sql_logs\sql\log.14" (14% done) # still reading write-ahead log "sql_logs\sql\log.14" (14% done) # still reading write-ahead log "sql_logs\sql\log.14" (15% done) # still reading write-ahead log "sql_logs\sql\log.14" (15% done) # still reading write-ahead log "sql_logs\sql\log.14" (15% done) # still reading write-ahead log "sql_logs\sql\log.14" (15% done) # still reading write-ahead log "sql_logs\sql\log.14" (15% done) # still reading write-ahead log "sql_logs\sql\log.14" (15% done) # still reading write-ahead log "sql_logs\sql\log.14" (16% done) # still reading write-ahead log "sql_logs\sql\log.14" (16% done) # still reading write-ahead log "sql_logs\sql\log.14" (16% done) # still reading write-ahead log "sql_logs\sql\log.14" (16% done) # still reading write-ahead log "sql_logs\sql\log.14" (16% done) # still reading write-ahead log "sql_logs\sql\log.14" (17% done) # still reading write-ahead log "sql_logs\sql\log.14" (17% done) # still reading write-ahead log "sql_logs\sql\log.14" (18% done) # still reading write-ahead log "sql_logs\sql\log.14" (18% done) # still reading write-ahead log "sql_logs\sql\log.14" (18% done) # still reading write-ahead log "sql_logs\sql\log.14" (19% done) # still reading write-ahead log "sql_logs\sql\log.14" (19% done) # still reading write-ahead log "sql_logs\sql\log.14" (20% done) # still reading write-ahead log "sql_logs\sql\log.14" (20% done) # still reading write-ahead log "sql_logs\sql\log.14" (20% done) # still reading write-ahead log "sql_logs\sql\log.14" (21% done) # still reading write-ahead log "sql_logs\sql\log.14" (21% done) # still reading write-ahead log "sql_logs\sql\log.14" (22% done) # still reading write-ahead log "sql_logs\sql\log.14" (22% done) # still reading write-ahead log "sql_logs\sql\log.14" (22% done) # still reading write-ahead log "sql_logs\sql\log.14" (23% done) # still reading write-ahead log "sql_logs\sql\log.14" (23% done) # still reading write-ahead log "sql_logs\sql\log.14" (23% done) # still reading write-ahead log "sql_logs\sql\log.14" (24% done) # still reading write-ahead log "sql_logs\sql\log.14" (24% done) # still reading write-ahead log "sql_logs\sql\log.14" (24% done) # still reading write-ahead log "sql_logs\sql\log.14" (24% done) # still reading write-ahead log "sql_logs\sql\log.14" (25% done) # still reading write-ahead log "sql_logs\sql\log.14" (25% done) # still reading write-ahead log "sql_logs\sql\log.14" (25% done) # still reading write-ahead log "sql_logs\sql\log.14" (26% done) # still reading write-ahead log "sql_logs\sql\log.14" (26% done) # still reading write-ahead log "sql_logs\sql\log.14" (27% done) # still reading write-ahead log "sql_logs\sql\log.14" (27% done) # still reading write-ahead log "sql_logs\sql\log.14" (27% done) # still reading write-ahead log "sql_logs\sql\log.14" (28% done) # still reading write-ahead log "sql_logs\sql\log.14" (28% done) # still reading write-ahead log "sql_logs\sql\log.14" (28% done) # still reading write-ahead log "sql_logs\sql\log.14" (28% done) # still reading write-ahead log "sql_logs\sql\log.14" (29% done) # still reading write-ahead log "sql_logs\sql\log.14" (30% done) # still reading write-ahead log "sql_logs\sql\log.14" (30% done) # still reading write-ahead log "sql_logs\sql\log.14" (31% done) # still reading write-ahead log "sql_logs\sql\log.14" (31% done) # still reading write-ahead log "sql_logs\sql\log.14" (31% done) # still reading write-ahead log "sql_logs\sql\log.14" (32% done) # still reading write-ahead log "sql_logs\sql\log.14" (32% done) # still reading write-ahead log "sql_logs\sql\log.14" (32% done) # still reading write-ahead log "sql_logs\sql\log.14" (33% done) # still reading write-ahead log "sql_logs\sql\log.14" (33% done) # still reading write-ahead log "sql_logs\sql\log.14" (34% done) # still reading write-ahead log "sql_logs\sql\log.14" (34% done) # still reading write-ahead log "sql_logs\sql\log.14" (34% done) # still reading write-ahead log "sql_logs\sql\log.14" (35% done) # still reading write-ahead log "sql_logs\sql\log.14" (35% done) # still reading write-ahead log "sql_logs\sql\log.14" (35% done) # still reading write-ahead log "sql_logs\sql\log.14" (35% done) # still reading write-ahead log "sql_logs\sql\log.14" (36% done) # still reading write-ahead log "sql_logs\sql\log.14" (36% done) # still reading write-ahead log "sql_logs\sql\log.14" (36% done) # still reading write-ahead log "sql_logs\sql\log.14" (36% done) # still reading write-ahead log "sql_logs\sql\log.14" (37% done) # still reading write-ahead log "sql_logs\sql\log.14" (37% done) # still reading write-ahead log "sql_logs\sql\log.14" (37% done) # still reading write-ahead log "sql_logs\sql\log.14" (38% done) # still reading write-ahead log "sql_logs\sql\log.14" (39% done) # still reading write-ahead log "sql_logs\sql\log.14" (40% done) # still reading write-ahead log "sql_logs\sql\log.14" (41% done) # still reading write-ahead log "sql_logs\sql\log.14" (42% done) # still reading write-ahead log "sql_logs\sql\log.14" (44% done) # still reading write-ahead log "sql_logs\sql\log.14" (45% done) # still reading write-ahead log "sql_logs\sql\log.14" (46% done) # still reading write-ahead log "sql_logs\sql\log.14" (47% done) # still reading write-ahead log "sql_logs\sql\log.14" (48% done) # still reading write-ahead log "sql_logs\sql\log.14" (49% done) # still reading write-ahead log "sql_logs\sql\log.14" (50% done) # still reading write-ahead log "sql_logs\sql\log.14" (50% done) # still reading write-ahead log "sql_logs\sql\log.14" (51% done) # still reading write-ahead log "sql_logs\sql\log.14" (52% done) # still reading write-ahead log "sql_logs\sql\log.14" (53% done) # still reading write-ahead log "sql_logs\sql\log.14" (54% done) # still reading write-ahead log "sql_logs\sql\log.14" (55% done) # still reading write-ahead log "sql_logs\sql\log.14" (55% done) # still reading write-ahead log "sql_logs\sql\log.14" (56% done) # still reading write-ahead log "sql_logs\sql\log.14" (57% done) # still reading write-ahead log "sql_logs\sql\log.14" (57% done) # still reading write-ahead log "sql_logs\sql\log.14" (58% done) # still reading write-ahead log "sql_logs\sql\log.14" (59% done) # still reading write-ahead log "sql_logs\sql\log.14" (59% done) # still reading write-ahead log "sql_logs\sql\log.14" (60% done) # still reading write-ahead log "sql_logs\sql\log.14" (60% done)

2 2

Announcement: New Jul2015-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 25 Mar '16

25 Mar '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP3 bugfix release MonetDB5 Server * Fixed potential crash in MAL debugger when accessing BATs by index. Functionality dropped as it is also a security leak. Bug Fixes * 2972: SQL URL functionality contains errors * 3549: bulk string operations very slow * 3881: Server crashes on bulk load * 3890: Window function + group by in subselect, rel2bin_project: Assertion `0' failed * 3891: MonetDB crashes when executing SQL with window function * 3900: null handling in some sql statements is incorrect * 3906: Multi-column 1-N table-function with mitosis produces different column counts * 3908: LEFT JOIN with OR conditions triggers assertion * 3909: Incorrect column name in OR condition of LEFT JOIN crashes mserver * 3910: COPY INTO table (column1, column2) got wrong result * 3912: When table/column names conflicts, data ends in multiple tables! * 3917: Date difference returns month_interval instead of day_interval * 3918: MonetDB.R version 1.0.1 incorrectly constructs the batfile script * 3919: Table conflict when the table name and fields are identical * 3921: Creating a table from a complex query crashes mserver or triggers assertion * 3922: AVG( column ) returns NaN rather than Inf when column contains Inf * 3928: When killing a virtual machine, sql_logs/sql/log is empty * 3930: Wrong typecast on character columns in prepared statements when using Umlaute * 3932: CASE expressions with constants are not evaluated correctly * 3933: replace "exit" by "throw new Exception" * 3937: bad BAT properties with binary copy into and NULL values * 3938: Wrong error message on violating foreign key constraint * 3940: Date calculation and comparison produce wrong result * 3941: Wrong coercion priority * 3948: SQL: select * from sys.sys.table_name; is accepted but should return an error * 3951: extern table_funcs not visible from Windows DLL for extensions like vaults (crashes) * 3952: Stream table gives segfault * 3953: MIN/MAX of a UUID column produces wrong results * 3954: Consolidate table assertion error * 3955: (incorrect) MAL loop instead of manifold triggered by simple change in target list

1 0

Fwd: Error after compiling monetdb
by Shmagi Kavtaradze 24 Mar '16

24 Mar '16

I am trying to write a C User Defined Function. It works in Neatbeans. What I want is, to take value from cells, insert them in text or char array and compare each corresponding array position to each other. As values I will have "1"s and "0s". E.g "101010100011". THe code I wrote in udf.c: /* Comparing bag of words */ char BitAnd(char Cell1[], char Cell2[]) { Cell1[10000]; Cell2[10000]; int counter; float result; int strlen = sizeof(Cell1)/sizeof(Cell1[0])-1; int i; for (i=0; i < strlen; i++) { if (Cell1[i] == Cell2[i]) { counter++; } } result = (float)counter / strlen; } /* MAL wrapper */ char UDFBitAnd(char Cell1[], char Cell2[]) { /* assert calling sanity */ assert(Cell1[] != NULL && Cell2[] != NULL); return MAL_SUCCEED; } In udf.mal I have: command BitAnd(Cell1:chr,Cell2:chr):flt address UDFBitAnd comment "Compare bag of words"; In 80_udf.sql: -- Comparing bag of words create function BitAnd(Cell1[] char, Cell2[] char) returns float external name udf.BitAnd; I compiled Monetdb, but when I try to connect with "monetdb" to db it gives me error. In the log file I found: MSG voc[18857]: !SyntaxException:parseError:command BitAnd(Cell1:chr,Cell2:chr):flt MSG voc[18857]: !SyntaxException:parseError: ^Type identifier expected MSG voc[18857]: !SyntaxException:parseError:command BitAnd(Cell1:chr,Cell2:chr):flt MSG voc[18857]: !SyntaxException:parseError: ^',' expected MSG merovingian[8780]: database 'voc' (18857) has exited with exit status 0 MSG merovingian[8780]: database 'voc' has shut down ERR merovingian[8780]: client error: database 'voc' started up, but failed to open up a communication channel

2 7

MonetDB schema for Yii 1.1 (PHP framework)?
by Kevork Vartanian 24 Mar '16

24 Mar '16

Hi, I am working on a mini project and using the Yii 1.1 PHP framework (OOP framework, using MVC) and trying to connect to a MonetDB database (changing from MySQL). I see that there is a MonetDB PHP client (https://dev.monetdb.org/hg/MonetDB/file/tip/clients/php/lib/php_monetdb.php) I believe it is MAPI based, but I don't know if that can be used in Yii by creating a wrapper or so. Yii has special classes to retrieve metadata information from database (the http://www.yiiframework.com/doc/api/1.1/CDbSchema) and it also has subclasses for several databases supported in Yii "out of the box" (cubrid, mssql, mysql, oci, pgsql and sqlite). It uses the CActiveDataProvider class which is based on ActiveRecord (http://www.yiiframework.com/doc/api/1.1/CActiveDataProvider) and has methods to retrieve the data from database along with different criteria. Knowing MonetDB also has an ODBC driver, I installed unixodbc along with the PHP PDO driver for ODBC, the MonetDB ODBC driver, and have done all the needed set up to make ODBC work. By the way, my test server is an Ubuntu VM. Now that I have the ODBC working, I am able to connect to the database, and when using SQL statements (building the sql query, command, datareader, etc) everything works in Yii, and I can retrieve the needed data, but when I want to use the widgets and make use of the framework and MVC, I start getting errors and I believe it's all because of not having the right schema class. I tried "forcing" the driver name and tested all the existing schemas (the ones I specified above, that come with Yii) but to no avail. I'm not a database expert or an advanced PHP (and Yii) developer but have some development experience and was able to work on this project and build a nice web application with MySQL but now that I'm planning to change the DB and use MonetDB, I believe I need expert advice and some help from real experts, and I'm hoping someone on this list will be able to shed some light. I couldn't find much on the Internet. Thank you. Regards, Kevork Vartanian Technical Solutions Consultant kevork.vartanian(a)innotta.com.au | +61 406 776 789 | www.innotta.com.au

2 1

How to quote/escape single quote character in the name of the file?
by Gennadiy 23 Mar '16

23 Mar '16

Hello, I am trying to load data from a text file that has single quote character in its name into a table: > "C:\Program Files\MonetDB\MonetDB5\mclient.bat" --port=50000 --user=monetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP2) Database: MonetDB v11.21.13 (Jul2015-SP2), 'soft10_20160318_172621' Type \q to quit, \? for a list of available commands auto commit mode: on sql> sql> sql>COPY 120 OFFSET 2 RECORDS INTO "observations_to_score1" FROM 'C:\\tmp\\sachin's_data_with_requested_column_03_03_2016.csv' USING DELIMITERS ',', '\n', '"' LOCKED; more> more> Everything works fine if there is no single quote character in the name of the file: sql>COPY 120 OFFSET 2 RECORDS INTO "observations_to_score1" FROM 'C:\\tmp\\sachins_data_with_requested_column_03_03_2016.csv' USING DELIMITERS ',', '\n', '"' LOCKED; 120 affected rows (12.3s) I was trying to precede the single quote character by a \ (backslash) or to double it, but it still did not work: sql>COPY 120 OFFSET 2 RECORDS INTO "observations_to_score1" FROM 'C:\\tmp\\sachin\'s_data_with_requested_column_03_03_2016.csv' USING DELIMITERS ',', '\n', '"' LOCKED; could not open file 'C:\\tmp\\sachin\'s_data_with_requested_column_03_03_2016.csv': No such file or directory sql>COPY 120 OFFSET 2 RECORDS INTO "observations_to_score1" FROM 'C:\\tmp\\sachin''s_data_with_requested_column_03_03_2016.csv' USING DELIMITERS ',', '\n', '"' LOCKED; could not open file 'C:\\tmp\\sachin\'s_data_with_requested_column_03_03_2016.csv': No such file or directory > dir c:\tmp\sa*.csv Volume in drive C is S3A9506D003 Volume Serial Number is 00B9-C775 Directory of c:\tmp 2016-03-18 17:12 144,202 sachin's_data_with_requested_column_03_03_2016.csv 2016-03-18 17:12 144,202 sachins_data_with_requested_column_03_03_2016.csv 2 File(s) 288,404 bytes 0 Dir(s) 377,532,665,856 bytes free I am using the following version of MonetDB: # MonetDB 5 server v11.21.13 "Jul2015-SP2" # Serving database 'soft10_20160318_172621', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 3.799 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # Start processing logs sql/sql_logs version 52200 # Start reading the write-ahead log 'sql_logs\sql\log.11' # Finished reading the write-ahead log 'sql_logs\sql\log.11' # Finished processing logs sql/sql_logs # MonetDB/SQL module loaded > Thank you, -- Gennadiy

3 5

Fwd: [Bug 3960] SELECT * with UNION ALL does not check column order
by Anthony Damico 23 Mar '16

23 Mar '16

hi team, should monetdb throw an informative error when SELECT * gets used with UNION? monetdb and postgres both allow users to make the mistake below. from my perspective it should simply fail instead of mis-matching silently. thanks ---------- Forwarded message ---------- From: <bugzilla-daemon(a)monetdb.org> Date: Wed, Mar 23, 2016 at 10:42 AM Subject: [Bug 3960] SELECT * with UNION ALL does not check column order To: ajdamico(a)gmail.com https://www.monetdb.org/bugzilla/show_bug.cgi?id=3960 Niels Nes <niels(a)cwi.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution|--- |WONTFIX --- Comment #4 from Niels Nes <niels(a)cwi.nl> --- postgres=# postgres=# CREATE TABLE x ( a integer , b integer ); CREATE TABLE postgres=# CREATE TABLE y ( b integer , a integer); CREATE TABLE postgres=# INSERT INTO x VALUES ( 1 , 2 ) ; INSERT 0 1 postgres=# INSERT INTO y VALUES ( 3 , 4 ) ; INSERT 0 1 postgres=# select * from x; a | b ---+--- 1 | 2 (1 row) postgres=# select * from y; b | a ---+--- 3 | 4 (1 row) postgres=# CREATE TABLE z AS ( SELECT * FROM x ) UNION ALL ( SELECT * FROM y ) WITH DATA; SELECT 2 postgres=# SELECT * FROM z; a | b ---+--- 1 | 2 3 | 4 (2 rows) ie what we do is normal behaviour.... -- You are receiving this mail because: You are on the CC list for the bug. You reported the bug.

1 0

Perl DBD::monetdb performance (slow) vs Python (fast)
by Puthick Hok 21 Mar '16

21 Mar '16

Hi, We are using MonetDB (MonetDB-11.19.9) compiled from source in openSUSE 12.2. We use the Perl client package from the source file. It works fine until we need to select 100000 plus records on a single field from a database in MonetDB. It works but it is lot slower than in python and in mclient. Selecting one field with a limit of 100,000 records takes under 1 second using mclient and python. However, it takes more than 23 seconds in Perl. Has anyone observed this performance issue with Perl DBD::monetdb? How Python connector manages to retrieve data very fast? and to make Perl connector work as fast as Python. Thank you in advance. Regards, Puthick

1 0

Re: MonetDB schema for Yii 1.1 (PHP framework)?
by Mark Raasveldt 17 Mar '16

17 Mar '16

Turns out I wasn't subscribed to the list on this email account, let's try that again. ----- Original Message ----- From: "Mark Raasveldt" <m.raasveldt(a)cwi.nl> To: "Kevork Vartanian" <kevork.vartanian(a)innotta.com.au> Cc: "users-list" <users-list(a)monetdb.org> Sent: Thursday, March 17, 2016 11:25:35 AM Subject: Re: MonetDB schema for Yii 1.1 (PHP framework)? Hey Kevork, I don't mind answering questions through email, but I'm afraid I have nothing to do with the PHP client (in fact, I didn't even know we had a PHP client for MonetDB). The author at the top of the page simply states who has performed the last commit to the MonetDB repository, not who created or authored that specific file. You must have clicked on the file after I performed a commit. I have CCed this message to the users' list, hopefully someone that at least knows about the existence of the MonetDB PHP client can help you out there. Regards, Mark ----- Original Message ----- From: "Kevork Vartanian" <kevork.vartanian(a)innotta.com.au> To: "m raasveldt" <m.raasveldt(a)cwi.nl> Sent: Thursday, March 17, 2016 4:31:54 AM Subject: MonetDB schema for Yii 1.1 (PHP framework)? Hi Mark, Hope this email finds you well. Apologies in advance in case you don't like to be contacted directly by email. I saw your name and email address as being the author of the MonetDB PHP client (https://dev.monetdb.org/hg/MonetDB/file/tip/clients/php/lib/php_monetdb.php) and wanted to contact you to ask a few questions, if you don't mind, of course. I am working on a mini project and using the Yii 1.1 PHP framework (OOP framework, using MVC) and trying to connect to a MonetDB database (changing from MySQL). I can see the PHP client is MAPI based, but I don't know if that can be used in Yii by creating a wrapper or so. Yii has special classes to retrieve metadata information from database (the http://www.yiiframework.com/doc/api/1.1/CDbSchema) and it also has subclasses for several databases supported in Yii "out of the box" (cubrid, mssql, mysql, oci, pgsql and sqlite). It uses the CActiveDataProvider class which is based on ActiveRecord (http://www.yiiframework.com/doc/api/1.1/CActiveDataProvider) and has methods to retrieve the data from database along with different criteria. Knowing MonetDB also has an ODBC driver, I installed unixodbc along with the PHP PDO driver for ODBC, the MonetDB ODBC driver, and have done all the needed set up to make ODBC work. By the way, my test server is an Ubuntu VM. Now that I have the ODBC working, I am able to connect to the database, and when using SQL statements (building the sql query, command, datareader, etc) everything works in Yii, and I can retrieve the needed data, but when I want to use the widgets and make use of the framework and MVC, I start getting errors and I believe it's all because of not having the right schema class. I tried "forcing" the driver name and tested all the existing schemas (the ones I specified above, that come with Yii) but to no avail. I'm not a database expert or an advanced PHP (and Yii) developer but have some development experience and was able to work on this project and build a nice web application with MySQL but now that I'm planning to change the DB and use MonetDB, I believe I need expert advice and some help from real experts. I don't know if you're familiar with Yii but do you think what I'm trying to achieve is doable? Also, MonetDB being columnar, I was thinking maybe I should look into the Yii classes for PostgreSQL (main one being http://www.yiiframework.com/doc/api/1.1/CPgsqlSchema with source code: https://github.com/yiisoft/yii/blob/1.1.17/framework/db/schema/pgsql/CPgsql…) and try to make similar ones for MonetDB but I doubt I will be able to do it without some help and guidance from experts. Your advice is greatly appreciated. Thank you. Regards, Kevork Vartanian Technical Solutions Consultant kevork.vartanian(a)innotta.com.au | +61 406 776 789 | www.innotta.com.au

1 0

User Defined Function
by Shmagi Kavtaradze 16 Mar '16

16 Mar '16

I have written a code in c, it works and now I want to implement it in Monetdb. 1) should I append the code in /sql/backends/monet5/UDF/udf.c file? 2) I have gone through udf.c, but still not sure how implement my c code there. I do not know C, just wanted to implement simple code. Can anyone help me adapting it to UDF? I want a function like BAnd(a.chunk, b.chunk), and instead of {"10101011"} in foo and bar, I want values of the columns. #include<stdio.h> main (){ const char foo[] = {"10101011"}; const char bar[] = {"01110111"}; int counter; float result; int strlen = sizeof(foo)/sizeof(foo[0])-1; for (int i=0; i < strlen; i++) { if (foo[i] == bar[i]) { counter++; } } result = (float)counter / strlen; printf("%d::%d::%f", counter, strlen, result); return 0; }

1 0

Re: Duplicate and reappearing tables.
by Nicolas Stalder 16 Mar '16

16 Mar '16

> > Message: 5 > Date: Tue, 15 Mar 2016 21:27:10 +0100 (CET) > From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Subject: Re: Duplicate and reappearing tables. > > Hi Nicolas, > > could you please share which version of MonetDB you're using on which OS? > > Also, did you initially create your database with the very same version of > MonetDB that you're using now, or did you create it with an earlier version > (which) and then upgrade it to the one you're using now (directly or via > intermetiate release steps)? > Stefan, It's Jul2015-SP2 running on Debian 8 (jessie). We loaded it from scratch, two daily files 100-250MB in size. Earlier we were running Jul2015-SP1 with the same issue. > Message: 7 > Date: Wed, 16 Mar 2016 08:02:47 +0000 > From: Marcel Blok <m.blok(a)chs.nl> > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Subject: RE: Duplicate and reappearing tables. > > Hi Nicolas, > > We have (and unfortunately sometimes still do) experience similar issues. > The issues appear most often at stopping and starting the database. In an > earlier thread (named "WAL read files") Pierre-Adrien found out that > underscores in table and/or column names could cause conflicts, and cause > similar issues. Maybe this is related to the issue you are experiencing? > > See also the bug report: > https://www.monetdb.org/bugzilla/show_bug.cgi?id=3912 I am aware of this report. However, I do not have any tables where <table_name>_<column_name> conflicts. The WAL files would be in db/sql_logs/sql, yes? I see a 12 byte file "log" and a 513K file "log.791". Can I inspect or understand what those files contain? What is in this numbered log file? I also tried loading a msqldump of the data into a new MonetDB instance, which eventually crashes with "Illegal input sequence". Are there any ways of making mserver5 more verbose? Clearly, something is happening at database startup. Cheers, Nicolas.

1 0

Shipping data / old voyages
by Brian Hood 16 Mar '16

16 Mar '16

Hi Guys, I have spent quite a bit of time with the voc database developing another project. I have where did you acquire this data as its very interesting. Regards Brian Hood

2 2

Duplicate and reappearing tables.
by Nicolas Stalder 16 Mar '16

16 Mar '16

Hi all I am running a currently roughly 400G-sized single Monet instance. We load daily CSV files into corresponding tables, and use merge tables to query the data (e.g., feed_20160301, feed_20160302, ... and feed_mt). We have a problem mainly with duplicate tables (same schema + table name, different table_id in sys.tables). My approach is to "DROP TABLE foo" for N - 1 times if there are N copies (is there even a way to SELECT or DROP by table_id?). This seemingly fixes the problem, however after restarting the database, the dropped tables reappear. Similarly, I have some old empty tables, which reappear after being dropped when the database is restarted. We start and stop the database to roll over connections during maintenance. Is this a known issue? Can it be fixed? Not sure how to make this reproducible. Thanks, Nicolas Stalder.

3 2

will monetdb work well if it needs to swap onto a SSD?
by Anthony Damico 15 Mar '16

15 Mar '16

pretty sure that monetdb hits a huge performance wall when you use up all your ram, but i'm curious if anyone has experience with swapping onto a SSD? is the speed-loss nearly equally bad, or is swapping onto a SSD a reasonable choice? thanks all

1 0

Backslash n interpreted as newline.
by Nicolas Stalder 15 Mar '16

15 Mar '16

Hi all I have noticed that when loading tables from CSV files via COPY INTO ... USING DELIMITERS '\t','\n', the ASCII characters '\' followed by 'n' are interpreted as a newline, leading to the loading being aborted due to an incorrect number of columns in this row. This seems to me to be a bug? Also, it would be most helpful if the error message contained a line number (as is the case if there is unicode in the CSV file). Thanks, Nicolas Stalder.

1 0

Debugging: !SELECT: no such operator 'normalize2'
by Willem Ligtenberg 15 Mar '16

15 Mar '16

I have created a function in R for MonetDB. My first function works fine: dbGetQuery(conn, "DROP FUNCTION normalize;") functionDef <- paste( "CREATE FUNCTION normalize(welltype STRING, data_column DOUBLE) RETURNS TABLE (i DOUBLE) LANGUAGE R {", "idx <- which(welltype == 'LC')", "100 * data_column / median(data_column[idx])", "};", sep = "\n") dbGetQuery(conn, functionDef) # example use stmt <- "SELECT barcode, normalize(welltype_code, data_column1) FROM hcs;" dbGetQuery(conn, stmt) # Works! Yay! So I wanted to create a more advanced version: dbGetQuery(conn, "DROP FUNCTION normalize2;") functionDef <- paste( "CREATE FUNCTION normalize2(barcode DOUBLE, welltype STRING, data_column DOUBLE) RETURNS TABLE (i DOUBLE) LANGUAGE R {", "idx <- which(welltype == 'LC')", "100 * data_column / median(data_column[idx])", "};", sep = "\n") dbGetQuery(conn, functionDef) As you see, the only thing that I have changed so far, is the addition of an extra argument to the function. However, this fails: # example use stmt <- "SELECT normalize2(barcode, welltype_code, data_column1) FROM hcs;" dbGetQuery(conn, stmt) I checked, and the types are all fine: barcode = DOUBLE welltype_code = CHARACTER LARGE OBJECT data_column1 = DOUBLE So what is going wrong? Kind regards, Willem

2 3

why "INSERT INTO ... SELECT" is concurrency conflicts with "CREATE TABLE"
by Pierre-Adrien Coustillas 15 Mar '16

15 Mar '16

Hi, Why is that the query INSERT INTO io__selection_top (id_navigateur) SELECT id_navigateur FROM bdm__index_client_cookies_id_navigateur; has a "COMMIT: transaction is aborted because of concurrency conflicts, will ROLLBACK instead" whith this query : DROP TABLE pierre; What is the table that a conflict ? SYS.TABLES ? Thanks Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35 // 09 54 38 70 90

3 2

Monetdb crashes.
by Souri Ratnaparkhi 14 Mar '16

14 Mar '16

Hello, We have a main table with 126 columns and approx. 94000 rows. We are trying to load approx 2500 records from temporary table into main table using query “insert into select” query. The database crashes upon query execution. We get error “*monetdbd: internal error while starting mserver, please refer to the logs*” when to try to connect again. No specific error is created in log file. 1500 records successfully loads from temp table to main table. We also created another main table with same DDL, that table is empty & inserted all the records from temp successfully without any issue. What could be the reason? Is 126 cols an issue? Please help. We are stuck. (btw: We have machine about 200GB of hard disk & 16 GB RAM (14 GB free)) OS* - Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-55-generic x86_64)* Monetdb version* - MonetDB v11.21.13 (Jul2015-SP2)* Thank you [image: Description: Description: Description: Description: logo] * Souri Ratnaparkhi United Software Associates Pvt. Ltd. Pune, India Mob: 8975677255* * Phone (India Lines): (91) 020-25445550 **www.usain.com* <http://www.usain.com/>

4 10

User Defined Function and regular Function
by Shmagi Kavtaradze 13 Mar '16

13 Mar '16

I read documentation, but have some questions: 1) What is the real difference between UDF and just creating function in sql? I mean the speed of execution. 2) If I create a function, is it implicitly C? I am asking because there is python binding also. 3) If I have already installed Monetdb and want to create UDF, should I recompile whole monetdb? I am not good enough yet to do something like this. I am asking, because I want to take a text type column full of 0s and 1s ("110101010"), represent it as a "int" or "bit" and do bitwise AND against other columns. When I am saying represent as int/bit, I mean not to change ("110101010") into bit, just let C know that it should interpret that number already as int or bit.

2 2

Function
by Shmagi Kavtaradze 12 Mar '16

12 Mar '16

I want to compare two integer columns (filled with 1s and 0s) to each other with bitwise and, then count number of 1s and divide by length of column entries. I wrote a function, but not sure if it is right or how to create in in Monetdb from console. create function BAnd(int a, int b) returns float begin declare count int; int length = floor(log10(abs(a))) + 1; temp = a & b; while (temp != 0) { if (n % 10 == 1) count++; temp /= 10; } return count/length; end;

1 0

Bitwise operations in Monetdb
by Shmagi Kavtaradze 11 Mar '16

11 Mar '16

I am new to Monetdb. I am using Postgresql mainly, but want to check Monetdb performace. In Postres I have column of type bit() filled with 0s and 1s. Then I am comparing each row to all other rows with bitwise AND on that column. Monetdb does not have bit() type so I used text. Any ideas how to do bitwise AND in monetdb and what type of column should I use for this? The query I tried: select a.sentenecid, b.sentenecid, a.sentence AND b.sentence from test a, test b;

4 11

Monitoring JDBC Connections
by Arno Mittelbach 11 Mar '16

11 Mar '16

Hi all, I am looking for a way to monitor open remote connections (i.e., JDBC connections) and possibly to force close them. As far as I can tell, MonetDB restricts the number of simultaneous connections to 64 by default. Via select * from sys.queue I can monitor running queries but I haven't yet figured out a way to list all open connections. Consider e.g. the case that a JDBC connection is for some reason not properly closed. Any hints would be much appreciated. Best regards, Arno

1 0

R session crashes every time with MonetDBLite package
by David Jankoski 08 Mar '16

08 Mar '16

Hi everone, I am new to this help list so before I post my problem I just want to ask how to provide as much information as possible ( in terms of log files and similar) The problem : Using the MonetDBLite package for R i pushed a relatively big chunk of data (~ 11 Mil rows x 14 cols) in a monetDB lite on disk. Now _every_ time i try connecting to it and running some commands, it crashes my R session. Happens with RStudio and Rgui. So, what additional information to this should i provide in order to best describe the problem : - Machine & OS info, R version, sessionInfo(), R script i'm trying to run. Are there any additional logs kept by monetDB which would be helpful ? Thanks in front, David -- [image: hellotrip] <http://www.hellotrip.fr/> *David Jankoski* <http://www.hellotrip.fr/> Teerketelsteeg 1 1012TB Amsterdam www.hellotrip.com [image: Twitter] <https://twitter.com/hellotrip_nl>[image: facebook] <http://facebook.com/HellotripNederland>[image: Linkedin] <https://www.linkedin.com/company/hellotrip>

2 1

Re: MonetDB: leftmart - trying to fix Windows compilation: do not "i...
by Lefteris 08 Mar '16

08 Mar '16

Dear Souri, please direct these kind of questions not at the developers-list but at the users-list (users-list(a)monetdb.org) Now for the issue at hand. Which version of monetdb are you using and which operating system? In order for us to reproduce the error, we need to know the schema of both the main and temp table (you can hide attribute names if it is sensitive information) and also the *exact* query you are using to copy rows from one table to the other. If you could also share the values of the records would also help. In general you have to provide us with more information on your setup, and a way to reproduce the problem:) Also, did you check the merovegnian.log ? no errors there? Maybe you can attach a debugger and see where monetdb fails? Thank you, Lefteris On Tue, Mar 8, 2016 at 6:08 AM, Souri Ratnaparkhi < souri.ratnaparkhi(a)usain.com> wrote: > Hello monetdb team, > > A gentle reminder. > We have a main table with 126 columns and approx. 94000 rows. > We are trying to load approx 2500 records from temporary table into main > table using query “insert into select” query. The database crashes upon > query execution. No specific error is created in log file. > When tried with 1500 records, it works. > We also created another main table with same DDL, that table is empty & > inserted all the records from temp successfully without any issue. > > What could be the reason? Is 126 cols an issue? Please help. We are stuck. > > (btw: We have machine about 200GB of hard disk & 16 GB RAM (14 GB free)) > > Thank you > > > Souri Ratnaparkhi > United Software Associates Pvt. Ltd. > Pune, India > Mob: 8975677255 > Phone (India Lines): (91) 020-25445550 > www.usain.com > > > -----Original Message----- > From: developers-list > [mailto:developers-list-bounces+souri.ratnaparkhi=usain.com@monetdb.org] > On Behalf Of Stefan Manegold > Sent: Tuesday, March 08, 2016 5:05 AM > To: Lefteris > Cc: Communication channel for developers of the MonetDB > Subject: Re: MonetDB: leftmart - trying to fix Windows compilation: do not > "i... > > > My mistake. Sorry! > > Stefan > > ----- On Mar 8, 2016, at 12:12 AM, Lefteris lsidir(a)gmail.com wrote: > > > Stefan, > > > > this needs to be inlined because it is used many times. I will deal > > with compilation issues later on. This is a dev branch and we are > > running different experiments every night. But with this change if > > measured times are different I would not know if it is because of > > other changes or because of the inline statement removal. > > > > I would appreciate if you did not make any changes in this branch at > > the moment. > > > > thank you, > > > > lefteris > > > > > > On Mon, Mar 7, 2016 at 8:34 PM, Stefan Manegold < > > commits+Stefan.Manegold=cwi.nl(a)monetdb.org> wrote: > > > >> Changeset: ea1aed8beff8 for MonetDB > >> URL: > >> http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ea1aed8beff8 > >> Modified Files: > >> gdk/gdk_bloom.c > >> Branch: leftmart > >> Log Message: > >> > >> trying to fix Windows compilation: do not "inline" BLOOMask() > >> > >> On Windows, compilation fails with > >> failed invocation: cl -GF -W3 -WX -MD -nologo -Ox -Zi -Oi -I. -I.\.. > >> -I.\..\..\gdk -I..\common\options -I.\..\..\gdk\..\common\options > >> -I..\common\stream -I.\..\..\gdk\..\common\stream -I..\common\utils > >> -I.\..\..\gdk\..\common\utils -I.\..\..\buildtools\conf -DHAVE_ICONV > >> -IC:\Libraries\iconv-1.11.1.win64-vs2010\include -DHAVE_LIBATOMIC_OPS > >> -IC:\Libraries\atomic_ops-7.4.2 -DNDEBUG -LD -Felibbat.dll > >> gdk_batop.obj gdk_select.obj gdk_search.obj gdk_hash.obj gdk_tm.obj > >> gdk_orderidx.obj gdk_align.obj gdk_bbp.obj gdk_heap.obj gdk_utils.obj > >> gdk_atoms.obj gdk_qsort.obj gdk_ssort.obj gdk_storage.obj gdk_bat.obj > >> gdk_delta.obj gdk_cross.obj gdk_system.obj gdk_value.obj > >> gdk_posix.obj gdk_logger.obj gdk_sample.obj gdk_calc.obj gdk_aggr.obj > >> gdk_group.obj gdk_imprints.obj gdk_bloom.obj gdk_join.obj > >> gdk_project.obj gdk_unique.obj gdk_firstn.obj libbat.res /link > >> ..\common\options\monet_options.obj > >> ..\common\stream\libstream.lib ..\common\utils\mutils.obj > >> ..\common\utils\prompt.obj wsock32.lib Ws2_32.l ib > >> C:\Libraries\zlib-1.2.8.win64-vs2010\lib\zdll.lib > >> C:\Libraries\bzip2-1.0.6.win64-vs2010\lib\libbz2.lib psapi.lib > >> Creating library libbat.lib and object libbat.exp gdk_join.obj : > >> error LNK2019: unresolved external symbol BLOOMask referenced in > >> function hashjoin libbat.dll : fatal error LNK1120: 1 unresolved > >> externals > >> > >> > >> Maybe, the fact that BLOOMask() is "inline"d triggers this problem ... > >> > >> > >> diffs (11 lines): > >> > >> diff --git a/gdk/gdk_bloom.c b/gdk/gdk_bloom.c > >> --- a/gdk/gdk_bloom.c > >> +++ b/gdk/gdk_bloom.c > >> @@ -240,7 +240,6 @@ BATbloom(BAT *b) > >> return GDK_SUCCEED; > >> } > >> > >> -inline > >> int BLOOMask(BUN v, Bloomfilter *bloom) { > >> BUN hv,mv,x,y,z; > >> _______________________________________________ > >> checkin-list mailing list > >> checkin-list(a)monetdb.org > >> https://www.monetdb.org/mailman/listinfo/checkin-list > > -- > | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | > | www.CWI.nl/~manegold/ | Science Park 123 (L321) | > | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | > _______________________________________________ > developers-list mailing list > developers-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/developers-list >

1 0

BATfetchjoin does not hit always => can't use fetchjoin
by John Burgess 07 Mar '16

07 Mar '16

Hi, I'm completely new to monetdb and have been dropped in at the deep end with an application support issue at a customer site. The release running is FEB2013 SP6 on Windows Server 2012 64 bit. There are customizations including aggregate functions written in MAL so the issue may not derive from monetdb source. I have searched bug lists but not found an obvious match for the error that is being reported, so I'm wondering whether anyone has experienced similar symptoms and found a solution. (Bug 3375 is the closest match and was fixed in the release in use) The reported problem is that the application works normally for a period until something causes monetdb to consistently report errors: Error: GDK reported error. SQLState: 22000 ErrorCode: 0 Error: BATfetchjoin(tmp_3662,tmp_4362) does not hit always (|bn|=0 != 2660745=|l|) => can't use fetchjoin. SQLState: 22000 ErrorCode: 0 It has been noted that the error appears to be in response to some SQL which up to that point had been working : select count(*) as "_m1_", avg("B"."C") as "_m2_" from "A" left outer join "B" on "A"."A_ID" = "B"."A_ID" Further, there is a foreign key relationship defined: ALTER TABLE "B" ADD CONSTRAINT fk_B FOREIGN KEY ("A_ID") REFERENCES "A"("A_ID") and dropping the index corresponding to the foreign key causes monetdb to recover. i.e. drop index fk_B All suggestions would be gratefully appreciated. In the meantime, I will try to reproduce the issue in house. Thanks, John Burgess ________________________________

3 5

should DROP TABLE work even after a table has been SET READ ONLY?
by Anthony Damico 07 Mar '16

07 Mar '16

hi, this bugzilla ticket is marked as resolved, but this behavior seems unintuitive to me.. is it ok? thanks team CREATE TABLE my_table AS SELECT 1 AS my_column WITH DATA; ALTER TABLE my_table SET READ ONLY; SELECT COUNT(*) FROM my_table; DROP TABLE my_table; from https://www.monetdb.org/bugzilla/show_bug.cgi?id=3895

3 3

extreme performance breakdown after update
by Andreas F. Hoffmann 06 Mar '16

06 Mar '16

Hello List, I'm running monetdb on a Debian server and the software is managed via Debian packages. A few days ago I installed an update for the monetdb5-server package. After that I noticed a problem with queries with the function lower() in the WHERE clause: >>WHERE LOWER(“fieldname”) LIKE '%lower_string%'<< without any change to the data, the requests became much, much slower then before (from some hundred microseconds to some minutes). Not only in my application but also in the monetdb client... I could solve the problem via a switch from using LIKE to ILIKE and no lower()-function, but I'm curious what could be the cause for this dramatic performance breakdown? I searched the release history but I found nothing... Has anybody experienced the same effect? Anny ideas on this phenomenon? Greetings from Germany, Andreas (theafh)

4 6

how to turn off double quotes
by Anton Kravchenko 03 Mar '16

03 Mar '16

Hi there, I wonder if there is a way to "copy select * from some_table into '\path\to\some.csv' delimiters ','; ", so that in output csv there is no double quotes for char vars? (for example, some_string instead of "some_string") P.S. I am using MonetDB v11.21.11 (Jul2015-SP1) on Windows. Thank you, Anton

1 1

(no subject)
by Anthony Damico 03 Mar '16

03 Mar '16

http://stackoverflow.com/questions/35730919/monetdb-select-query-slow-when-…

1 0

how to change the user "monetdb" password?
by Lynn Carol Johnson 02 Mar '16

02 Mar '16

Hi all - I've been googling to try and find a method of changing the default password for user monetdb. The links I've found are all 8-9 years old (e.g. This<https://www.monetdb.org/pipermail/users-list/2007-December/001345.html> one) My commands: 1. monetdbd stop <mydbfarm> 2. cd /Usr/local/monetdb/bin 3. ./mserver5 Then execute a change password command and get these results, which look similar to the email exchange noted in the 8 year old link above: rs-btlcj34lt:bin lcj34$ ./mserver5 # MonetDB 5 server v11.21.11 "Jul2015-SP1" # Serving database 'demo', using 8 threads # Compiled for x86_64-apple-darwin14/64bit with 64bit OIDs and 128bit integers dynamically linked # Found 16.000 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # Start processing logs sql/sql_logs version 52200 # Finished processing logs sql/sql_logs # MonetDB/SQL module loaded >changePassword("monetdb","bucklerLab"); !TypeException:user.main[1]:'user.changePassword' undefined in: user.changePassword("monetdb":str,"bucklerLab":str); > We are opening up our database to some outside folks. I've successfully created a read-only user/pwd for them, but want to protect the db by changing the default monetdb user password. I'm practicing on my own local copy before messing with our "real" db, but can't get this to work. Is there some other way I should be going about this? One additional question: What is the "passphrase" parameter I see when I run "monetdbd get all <mydbfarm>"? I tried playing with that but have not found what it effects. Thanks - Lynn

2 2

MySQL unix_timestamp() equivalent?
by Vijay Krishna 02 Mar '16

02 Mar '16

Hi, I am looking for an Epoch (bigint) to date/timestamp converter in MonetDB - something similar to unix_timestamp() in MySQL. I referred this page - https://www.monetdb.org/Documentation/SQLreference/Temporal and tried out a few functions, but it seems to work only with the existing Timestamp columns. Am I missing something here? I am using MonetDB Jan2014 release on a Mac OS-X. Any help much appreciated. Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

4 7

MAPI COPY INTO from stream
by Gustavo Brian 02 Mar '16

02 Mar '16

Hi, I'm trying to bulk load data into monetdb through COPY INTO statement on the fly. I've created an aggregator script that will create a session and send chunk of data using MAPI interface. Like this: "sCOPY INTO foo FROM STDIN USING DELIMITERS ',','\\n';" //+ "\n" + "100,10,0,'10'\n" + "\n" I've been following this example: http://dev.monetdb.org/hg/MonetDB/file/tip/java/example/SQLcopyinto.java (haven't tested) I planned to add as part of the monetdb nodejs module: https://github.com/MonetDB/monetdb-nodejs/issues/1 but seems to be out of the scope of that repository. Sadly I'm not getting any error just doesn't work. What's the correct way ? Thanks.

2 1

Typecasting in prepared statements
by Arno Mittelbach 24 Feb '16

24 Feb '16

Hi all, I am having a strange problem regarding prepared queries with Umlaute (e.g., ä). It seems that the rule for type casting strings correspond to the length of the string. The problem is that an Umlaut seems to count for two "normal" characters. Here is an example. Consider the following simple table: create table A (a varchar(4)); Now if we create a prepared statement (ultimately I want to do this from Java) such as prepare select * from A where a = ?; we can execute the statement and supply 4 characters as the argument: exec 3('1234'); This works fine. However, if I am using an Umlaut within the parameter, then things don't work out. For example exec 4('ä123'); fails with the following error message EXEC: wrong type for argument 1 of prepared statement: char, expected varchar However, if I now shorten the parameter by "1" character, it works again. That is, exec 5('ä12'); is ok. Can anybody explain this behavior? Is this a bug? And is there a workaround? The only thing I came up with so far is to double the size of every column. Best regards, Arno

3 3

Slow query - round 2
by Luciano Sasso 24 Feb '16

24 Feb '16

Hi, Do you think that many ( very many ) inserts / updates on this table , inserts that were aborted by network error could corrupt data and cause it? Remember , table have 100M rows. I have table with 30M rows, and run well work. Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1

File system level backup of running database
by Dennis Pallett 23 Feb '16

23 Feb '16

Hi all, Is it possible to create a file system level backup of a running MonetDB database? What are the downsides of doing this? For what it's worth, our databases are generally never updated outside of pre-fixed windows so only SELECT queries will be executed during any file system level backup. Best regards, Dennis Pallett

4 3

do these two commands create an error in the dev version of monetdb?
by Anthony Damico 23 Feb '16

23 Feb '16

they do work in the latest release but they crash monetdblite, making me wonder about the root culprit.. thanks all create table my_table (my_id clob, my_double double); SELECT COUNT( DISTINCT my_id ) AS unique_enrollees , QUANTILE( my_double , 0.25 ) AS some_quantile FROM my_table;

2 2

is there a downloadable nightly build exe installer for windows?
by Anthony Damico 23 Feb '16

23 Feb '16

or do i need to do something with these? http://dev.monetdb.org/hg/MonetDB/file/tip/NT/installer64 never done this before..curious how cumbersome it is i want to check whether this is a bug introduced within mserver or a bug specific to monetdblite https://www.monetdb.org/bugzilla/show_bug.cgi?id=3927 thanks!

2 2

Slow query
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi Stefan About "prepare statements calls/traces ", in paralell select count(*), this service that insert in 10 and 10 minutes. But even without such active service , the time is the same, 90 minutes. My mistake, my server have 6 cores. Hi, (as a side note, I'm afraid I'm too "old fashioned" and do no longer understand "modern" MonetDB traces; at least, I have no idea, why there are so many prepare statements calls/traces in your trace of a simple select count(*) query ... ) Having said that, the trace suggests that your machine either has 6 cores, or MonetDB has decided to split it in 6 slices for processing, although you machine has only 4 cores? Also, the trace reveals that the vast majority of the time is spent in 6 (one per slice) sql.tid() calls: ....... -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Slow query
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi, Martin Yes, before performance analysis, load is zero, the server is Linix console only. My RAM memory is low for this database size. what do you think ? Att, Hi, about slow query Yes, there is, but still left over resources. While the query (which takes 90 minutes) are running, to perform another in another table (9M rows), this latter returns usually in seconds. Hi Its all about concurrent processes while working with MonetDB. All those processes compete for RAM and disk bandwidth. Even running a browser already takes off a few GB from your RAM. At some point, you end up swapping intermediates to disk, which is a slow process (depending on your HDD). For performance analysis, you better shut down all the processes you don't need. Ideally, the CPU load = 0 before doing a performance analysis. > " > > A simple "select count(*) from table", run on 90 minutes. > > that is not normal. most likely there are a lot of other > processes competing for resources. > > > " -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Slow query
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi, send now trace for query in link: https://drive.google.com/folderview?id=0B0L6U0kZpUaFNUU3ak9PNUFVZW8&usp=sha… Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Slow Query
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi, send attached trace for query. Att, --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

3 2

Re: users-list Digest, Vol 42, Issue 17
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi, about slow query Yes, there is, but still left over resources. While the query (which takes 90 minutes) are running, to perform another in another table (9M rows), this latter returns usually in seconds. " A simple "select count(*) from table", run on 90 minutes. that is not normal. most likely there are a lot of other processes competing for resources. " Em 21/02/2016 08:00, users-list-request(a)monetdb.org escreveu: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: Slow Query (Martin Kersten) > 2. Re: Slow Query (Stefan Manegold) > 3. might be of interest (Anthony Damico) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 20 Feb 2016 21:34:06 +0100 > From: Martin Kersten <martin(a)monetdb.org> > To: users-list(a)monetdb.org > Subject: Re: Slow Query > Message-ID: <56C8CDBE.3060006(a)monetdb.org> > Content-Type: text/plain; charset=windows-1252; format=flowed > > On 20/02/16 19:39, Luciano Sasso wrote: >> Hi, >> >> I have MonetDB JUL2015-SP2, with table of 100M rows and 20 coluns, >> 4 of number type, others string type. >> >> A simple "select count(*) from table", run on 90 minutes. > that is not normal. most likely there are a lot of other > processes competing for resources. > >> Hardware and SO is: Debian 8, 4 cores, 11G RAM and 20G free on disc. >> >> While monet run the query, command top, show 42G of virtual memory >> >> Its normal ?, any idea ? >> >> thanks. >> >> >> -- >> >> Luciano Sasso Vieira >> Data Scientist & Solutions Architect >> >> <http://www.gsgroup.com.br>luciano(a)gsgroup.com.br | tel: 17 3353-0833 | cel: 17 99706-9335 >> <http://www.gsgroup.com.br>www.gsgroup.com.br >> >> >> Este e-mail foi enviado por um computador sem v?rus e protegido pelo Avast. >> www.avast.com <https://www.avast.com/sig-email> >> >> >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list >> > > > ------------------------------ > > Message: 2 > Date: Sat, 20 Feb 2016 22:16:01 +0100 (CET) > From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Subject: Re: Slow Query > Message-ID: <1956506938.11256849.1456002961125.JavaMail.zimbra(a)cwi.nl> > Content-Type: text/plain; charset="utf-8" > > Hi, > > this is indeed unexpected. > > To possibly better understand what's happening, > could you please run your query prefixed with keyword TRACE > and share the trace (profiling) result? > > Best, > Stefan > > ----- On Feb 20, 2016, at 7:39 PM, Luciano Sasso luciano(a)gsgroup.com.br wrote: > >> Hi, >> >> I have MonetDB JUL2015-SP2, with table of 100M rows and 20 coluns, >> 4 of number type, others string type. >> >> A simple "select count(*) from table", run on 90 minutes. >> >> Hardware and SO is: Debian 8, 4 cores, 11G RAM and 20G free on disc. >> >> While monet run the query, command top, show 42G of virtual memory. >> >> Its normal ?, any idea ? >> >> thanks. >> >> >> -- >> >> >> Luciano Sasso Vieira >> Data Scientist & Solutions Architect >> >> luciano(a)gsgroup.com.br | tel: 17 3353-0833 | cel: 17 99706-9335 >> www.gsgroup.com.br >> >> >> Este e-mail foi enviado por um computador sem v?rus e protegido pelo Avast. >> www.avast.com >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1

might be of interest
by Anthony Damico 21 Feb '16

21 Feb '16

http://stackoverflow.com/questions/32380663/improve-speed-of-queries-using-…

1 0

Slow Query
by Luciano Sasso 20 Feb '16

20 Feb '16

Hi, I have MonetDB JUL2015-SP2, with table of 100M rows and 20 coluns, 4 of number type, others string type. A simple "select count(*) from table", run on 90 minutes. Hardware and SO is: Debian 8, 4 cores, 11G RAM and 20G free on disc. While monet run the query, command top, show 42G of virtual memory. Its normal ?, any idea ? thanks. -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

3 2

Error in connecting monetdb server to localhost
by Debangana Sarkar 17 Feb '16

17 Feb '16

when I am connecting to MonetDB server it is showing an error : FATAL logger error there is no logger catalog but a log file is found. Are you sure you are using the same combination of database --dbpath and log directory Press any key to continue. Client error is: couldnot connect to localhost:50000 unknow error. Previously I installed the monetDB server and client and it was working.But for some reason it was uninstalled.Now after installing this error is coming. Please help

1 0

is it possible for a user to turn assertions on/off in mserver easily?
by Anthony Damico 16 Feb '16

16 Feb '16

might help with future testing..thanks all

3 3

Re: Re: Some time monetdbd can't start existing dbfarm
by Haiqing Ying 12 Feb '16

12 Feb '16

Thanks, Lefteris. You might be right and the log did suggest some file (.vaultkey) can't be found before it stopped working. I'll move the dbfarm out of /tmp and give it a try. Regards, HQ

1 0

Re: Re: Some time monetdbd can't start existing dbfarm
by Haiqing Ying 11 Feb '16

11 Feb '16

Hi Brian, Thanks for looking into this issue. Here is some more details in meroveringian.log - The DB was up for about a week. There're a lot of lines of 'proxying client ...', until the line 'client error: cannot start database 'EXM': no .vaultkey found...' 2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33861 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM 2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33862 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM 2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33863 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM 2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 08:36:20 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) 2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 08:36:20 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) 2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 08:36:21 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) And then, a lot of lines like below: 2016-02-07 14:44:18 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 14:44:23 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) Eventually, there're lot of errors like this: 2016-02-08 01:13:58 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-08 01:14:03 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) 2016-02-08 01:14:11 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file s 2016-02-08 01:14:18 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file s 2016-02-08 01:14:26 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file S Then neither mclient nor monetdbd stop cmd works any more. This is what I have in /tmp/exm folder after reboot: [centos@ip-10-72-0-98 exm]$ pwd /tmp/exm [centos@ip-10-72-0-98 exm]$ ls -al total 8088 drwxr-xr-x. 3 centos centos 62 Feb 9 17:33 . drwxrwxrwt. 11 root root 4096 Feb 11 13:36 .. drwx------. 4 centos centos 77 Feb 9 17:33 EXM -rw-------. 1 centos centos 0 Feb 9 17:33 .merovingian_lock -rw-------. 1 centos centos 8272151 Feb 9 17:33 merovingian.log [centos@ip-10-72-0-98 exm]$ ls -al EXM total 8 drwx------. 4 centos centos 77 Feb 9 17:33 . drwxr-xr-x. 3 centos centos 62 Feb 9 17:33 .. drwx------. 66 centos centos 4096 Feb 2 19:48 bat -rw-------. 1 centos centos 0 Feb 7 08:36 .gdk_lock srwx------. 1 centos centos 0 Jan 27 14:50 .mapi.sock drwx------. 3 centos centos 16 Jan 27 14:50 sql_logs -rw-------. 1 centos centos 22 Feb 9 17:33 .uplog If needed, I can send you the whole merovingian.log file (which is 8GB in size). Thanks, HQ

1 0

copy into stdout and redirect to a file
by Anton Kravchenko 11 Feb '16

11 Feb '16

Hi there, I wonder if one can copy a query result into stdout and redirect it to a file? I've tried: mclient.bat -u monetdb -d somedb -s "copy select * from sometable into stdout" - > .\somefile.csv But, only the table header is saved in somefile.csv... P.S. 'select * from sometable' shows all data rows as expected. I am using MonetDB v11.21.11 (Jul2015-SP1) on Windows. Thank you, Anton

2 2

Some time monetdbd can't start existing dbfarm
by Haiqing Ying 11 Feb '16

11 Feb '16

Hi, I created a dbfarm (/tmp/exm ), then created database (EXM), load data, do queries, etc, everything works fine for a week. Then I found mclient stopped working (can't connect to monetdb). The log file shows 'too many files opened' issue. I guess it might be some connection leak issue. I tried 'monetdbd stop /tmp/exm', but it's not responding. It looks I can't stop the dbfarm now. I then reboot the system and after reboot tried 'monetdbd start /tmp/exm' but I got an error saying ' cannot find file in /tmp/exm'. I verified the /tmp/exm folder did exist and there're still sub folder ( ./EXM) and log file still under /tmp/exm. The only option left seems to be creating a new dbfarm and recreating the whole database again. We're moving to use MonetDB in production and if this is a real bug then it would be too risky for production. The question is, is this a real issue? Or it's just some user error that I made? How do I make sure we won't run into this situation again in production mode. Many thanks, HQ

3 2