- users-list - monetdb.org

new gdk bug (on windows) that might've been introduced by Feb2013-SP4
by Anthony Damico 30 Sep '13

30 Sep '13

if you run this R code (with MonetDB.R), you will automatically load a series of brazilian census bureau data sets onto your computer. after all tables have been downloaded and imported, the R script will run a very simple CREATE TABLE command that will cause mserver to shut down when it should not. library(downloader) setwd( "C:/My Directory/CENSO/" ) source_url( " https://raw.github.com/ajdamico/usgsd/master/Censo%20Demografico/download%2…" , prompt = FALSE , echo = TRUE ) after mserver.exe crashes, even if i re-load mserver.exe manually and then connect to it with mclient.exe, and run this block of code: create table c10_dom_pre_fpc as (SELECT * FROM ac_dom) UNION ALL (SELECT * FROM al_dom) UNION ALL (SELECT * FROM am_dom) UNION ALL (SELECT * FROM ap_dom) UNION ALL (SELECT * FROM ba_dom) UNION ALL (SELECT * FROM ce_dom) UNION ALL (SELECT * FROM df_dom) UNION ALL (SELECT * FROM es_dom) UNION ALL (SELECT * FROM go_dom) UNION ALL (SELECT * FROM ma_dom) UNION ALL (SELECT * FROM mg_dom) UNION ALL (SELECT * FROM ms_dom) UNION ALL (SELECT * FROM mt_dom) UNION ALL (SELECT * FROM pa_dom) UNION ALL (SELECT * FROM pb_dom) UNION ALL (SELECT * FROM pe_dom) UNION ALL (SELECT * FROM pi_dom) UNION ALL (SELECT * FROM pr_dom) UNION ALL (SELECT * FROM rj_dom) UNION ALL (SELECT * FROM rn_dom) UNION ALL (SELECT * FROM ro_dom) UNION ALL (SELECT * FROM rr_dom) UNION ALL (SELECT * FROM rs_dom) UNION ALL (SELECT * FROM sc_dom) UNION ALL (SELECT * FROM se_dom) UNION ALL (SELECT * FROM sp1_dom) UNION ALL (SELECT * FROM sp2_rm_dom) UNION ALL (SELECT * FROM to_dom ) WITH DATA; the exact same crash occurs, so this is a new mserver.exe bug not a MonetDB.R bug. should i file a bug report? thanks!!! in mserver---- # MonetDB 5 server v11.15.15 "Feb2013-SP4" # Serving database 'censo_demografico', using 8 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 7.860 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50011/ # MonetDB/JAQL module loaded # MonetDB/SQL module loaded >= .\..\..\gdk\gdk_posix.c:800: MT_mremap(bat\02\65\26501.tail,468c50000,1889785 61024,755914244096): GDKextend() failed in mclient---- C:\Windows\System32>cd .. C:\Windows>cd .. C:\>cd "Program Files" C:\Program Files>cd MonetDB C:\Program Files\MonetDB>mclient censo_demografico -p 50011 'mclient' is not recognized as an internal or external command, operable program or batch file. C:\Program Files\MonetDB>mclient 'mclient' is not recognized as an internal or external command, operable program or batch file. C:\Program Files\MonetDB>ls 'ls' is not recognized as an internal or external command, operable program or batch file. C:\Program Files\MonetDB>dir Volume in drive C has no label. Volume Serial Number is 882E-FD8A Directory of C:\Program Files\MonetDB 09/06/2013 04:34 PM <DIR> . 09/06/2013 04:34 PM <DIR> .. 09/27/2013 12:27 PM <DIR> MonetDB5 0 File(s) 0 bytes 3 Dir(s) 451,568,218,112 bytes free C:\Program Files\MonetDB>cd MonetDB5 C:\Program Files\MonetDB\MonetDB5>mclient censo_demografico -p 50011 user(win32):monetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Feb2013-SP4) Database: MonetDB v11.15.15 (Feb2013-SP4), 'censo_demografico' Type \q to quit, \? for a list of available commands auto commit mode: on sql>create table c10_dom_pre_fpc as (SELECT * FROM ac_dom) UNION ALL (SELECT * F ROM al_dom) UNION ALL (SELECT * FROM am_dom) UNION ALL (SELECT * FROM ap_dom) UN ION ALL (SELECT * FROM ba_dom) UNION ALL (SELECT * FROM ce_dom) UNION ALL (SELEC T * FROM df_dom) UNION ALL (SELECT * FROM es_dom) UNION ALL (SELECT * FROM go_do m) UNION ALL (SELECT * FROM ma_dom) UNION ALL (SELECT * FROM mg_dom) UNION ALL ( SELECT * FROM ms_dom) UNION ALL (SELECT * FROM mt_dom) UNION ALL (SELECT * FROM pa_dom) UNION ALL (SELECT * FROM pb_dom) UNION ALL (SELECT * FROM pe_dom) UNION ALL (SELECT * FROM pi_dom) UNION ALL (SELECT * FROM pr_dom) UNION ALL (SELECT * FROM rj_dom) UNION ALL (SELECT * FROM rn_dom) UNION ALL (SELECT * FROM ro_dom) U NION ALL (SELECT * FROM rr_dom) UNION ALL (SELECT * FROM rs_dom) UNION ALL (SELE CT * FROM sc_dom) UNION ALL (SELECT * FROM se_dom) UNION ALL (SELECT * FROM sp1_ dom) UNION ALL (SELECT * FROM sp2_rm_dom) UNION ALL (SELECT * FROM to_dom ) WITH DATA; GDK reported error. HEAPextend: failed to extend to 755914244096 for 02\65\26501.tail: MT_mremap() f ailed sql> sql>

3 4

MonetDB compile error 11.15.15
by Brian Hood 28 Sep '13

28 Sep '13

Hi Guys, I just trying to update my Arch Linux package and when building came up with this compile error the package passed the sha1sum after download. Hope someone can help. In file included from /usr/include/string.h:25:0, from /var/abs/AUR/monetdb/src/MonetDB-11.15.15/monetdb5/extras/jaql/parser/jaql.y:23: /usr/include/features.h:327:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp] # warning _FORTIFY_SOURCE requires compiling with optimization (-O) ^ y.tab.c: In function 'jaqlparse': y.tab.c:1665:32: error: 'scanner' undeclared (first use in this function) y.tab.c:1665:32: note: each undeclared identifier is reported only once for each function it appears in make[9]: *** [libjaqlp_la-jaql.tab.lo] Error 1 Regards, Brian Hood

1 0

regular expression - substring
by Budulinku Dejmihrasku 27 Sep '13

27 Sep '13

Hi, i am looking for any idea/suggestion how to do something like Oracle has, i.e. REGEXP_SUBSTR(source, pattern[, position [, occurrence[, match_parameter]]]) which returns the desired substring. why i need this: i have vast amount of data to parse - i.e. i need to get all of the many possible [parameter=value] pairs from an url access log in order to feed the dimension tables w/parameter-value from the URL by parsing the string directly in MonetDB when i write a SQL function to do this (or any string parsing as a matter of fact), it is rather slow when parsing millions of records. i know that there is the PCRE.MAL and i already use with some success functions INDEX, MATCH or REPLACE but i was not able to find anything similar to the REGEXP_SUBSTR functionality to get the desired substring in one go would most likely help me to get better performance any suggestion? please? thanks milan

2 1

Stopping queries, deleting tuples and other general use questions
by Ruben Silva 27 Sep '13

27 Sep '13

Hi, after a while using MonetDB some general use questions pops up. I've collected some which answers I think will be of great help for any MonetDB user. * * *#1:* In the monetdb manpage we can read: *“The kill command immediately sends a SIGKILL and should only be used as last resort for a database that doesn't respond any more. Killing a database may result in (partial) data loss.”* * * 1.1 If kill is the last resort, what are the other options? * * *#2:* System monitor (Oct 2013)<http://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/monitor> 2.1 Will this be available in October or is there a new estimate? 2.2 Does the current development version include this feature yet so we can test it? I'm interested mainly in the methods sys.pause(qtag), sys.resume(qtag) and sys.stop(qtag), *specially sys.stop(qtag)*. 2.4 Is there some alternative method already available in the Feb2013 release to achieve this action? *#3:* In the monetdb manpage we can read about the *nclients* property: *“Sets the maximum amount of clients that can connect to this database at the same time. Setting this to a high value is discouraged.” * 3.1 What do you consider as a high value? Fabian Groffen stated in a similar question *"Concurrency is always a problem for MonetDB, since in the ideal case it means the processes are fighting for resources with each other." * 3.2 Does this mean (theoretically) that if I can run Query A in 10s in a system with resources R (Ram, Disk, Processor), with Rx2 I can run Query A and B (equivalent in effort to A) in the same 10s? *#4:* www.monetdb.org/Documentation/Manuals/SQLreference/Transactions: *"WARNING. The tuples being deleted are only marked as such. They do not reduce the table size. It calls for a vacuum cleaning algorithm."* * * I've found the /usr/lib/monetdb5/createdb/20_vacuum.sql script and after some googling I found the following post* * http://mail.monetdb.org/pipermail/users-list/2011-August/005051.html without an answer. 4.1 Does the following sequence frees the space of the deleted rows? sql>delete from sys.transfer_ip ; 651950 affected rows (117.620ms) sql>call vacuum('sys','transfer_ip'); Best regards, Ruben Silva

2 1

Create Sorted View in monetdb
by Khaled Dehdouh 27 Sep '13

27 Sep '13

Please can you help me. I want to create a view sorted according to attributes like Oracle, my query is as follows: *Create view V1 as Select Name, First_Name, Age from **People O**redr by Name**;* but Monetdb shows me the following error "CREATE VIEW: ORDER BY not supported." How should I do to create a sorted view ?.

2 1

Announcement: New Feb2013-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 27 Sep '13

27 Sep '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Feb 2013-SP4 bugfix release Merovingian * monetdb now no longer compresses output when not connected to a terminal, bug #3292 Bug Fixes * 2781: SQL localtime() returns GMT time * 2962: client timezone information needs to be handled also for modifications to timestamp values * 3232: The function weekofyear returns results not according to the iso 8601 standard * 3255: M5 tests inlineFunction & ifthencst fail since changeset 207ed7d7e03a * 3292: monetdb status should print full line * 3307: distinct multicolumn performance and implementation suggestion for count distinct * 3310: Cast value to integer gives wrong result * 3311: strLength assertion on join and concat * 3313: Assertion in rel2bin_project * 3315: Incorrect behavior with subqueries in DELETE * 3317: correlated subselect does not work * 3318: UPDATE does not work with DECLEAREd TABLE in FUNCTION * 3319: UNIQUE column in DECLAREd TABLE in FUNCTION results in segfault * 3329: Function AVG fails during runtime in a procedure * 3330: Corrupt mitosis plan with group-by query * 3332: MEDIAN: no such operator 'median(decimal)' * 3336: DB files not removed if all rows are deleted, even after restart * 3337: Monet ODBC gives wrong schema information * 3339: Segvault in complex function call * 3340: crash on seeming incorrect SQL query * 3341: segfault caused by select rank() on a newly-loaded table * 3342: incorrect automatic cast from decimal to tinyint * 3345: copy binary into fails with the wrong error message * 3346: Deadlock in mal_dataflow's q_dequeue() (SciQL-2 branch) * 3348: SQL: LIKE incorrect with _ * 3349: When selecting everything from a joined result, column contents are incorrect in one of my use cases. * 3351: gzip'ed files cannot be bulkloaded into DB * 3352: Median function with decimal type doesn't work * 3353: Readonly tables call for UNSET * 3354: Introduce query time-out * 3358: Remote client connection stays in CLOSE_WAIT state * 3360: uninitialized memory causing bad output in query plans * 3362: ALTER TABLE operations silently remove previously set READ ONLY property * 3363: a join causes SIGSEGV after creation of Unique Key * 3367: Fully qualified order by column gives "relational query without result" * 3369: \0 Byte appears in MAPI

1 0

Re: Contents of users-list Digest, Vol 13, Issue 20
by Edgar Mejia 25 Sep '13

25 Sep '13

1. I have MonetDB Database Server Toolkit v1.1 (Feb2013-SP3) (i install so: sudo apt-get install monetdb5-sql monetdb-client) shell> monetdbd create /path/to/mydbfarm shell> monetdbd start /path/to/mydbfarm shell> monetdb create voc shell> monetdb release voc 2. Ubuntu 12.04.3 LTS 3. Schema and data: Schema: CREATE TABLE timedimension ( id integer NOT NULL, yeart int DEFAULT NULL, quarter varchar(2) DEFAULT NULL, weekyear int DEFAULT NULL, dayt int DEFAULT NULL, namemonth varchar(10) DEFAULT NULL, nameday varchar(20) DEFAULT NULL, montht int DEFAULT NULL, day_of_week varchar(15) DEFAULT NULL, date date DEFAULT NULL, month2 char(2) DEFAULT NULL ) CREATE TABLE bi_con_cdr_diario ( ano INT , facmesfac VARCHAR(6) , comcodigo INT , succodigo VARCHAR(3) , procodigo INT , pronombre VARCHAR(20) , prycodigo VARCHAR(20) , fecllamada timestamp , clicodigo INT , clinombre VARCHAR(80) , vencodigo INT , vennombre VARCHAR(50) , sercodigo VARCHAR(3) , pryservicio VARCHAR(4) , sertipo VARCHAR(3) , serclase VARCHAR(4) , trdtipotelefono CHAR(10) , cfacodigo INT , cfanombre VARCHAR(30) , trddurllamada INT , minutosreales DOUBLE , minutosenteros INT , minutosfacturados DOUBLE , costo DOUBLE , costohit DOUBLE , valorbruto DOUBLE , valorneto DOUBLE , trdvalorlocalneto DOUBLE , trdvalorlocalimpu DOUBLE , valorlocal DOUBLE , codpaisori INT , nompaisori VARCHAR(80) , codpaisdes INT , nompaisdes VARCHAR(80) , comnombre2 VARCHAR(15) , linprnombre VARCHAR(40) , trdcodciuori BIGINT , trdnomciuori VARCHAR(80) , sernombre VARCHAR(50) , idtimedimension integer , linprcodigo INT , trdcodciudes INT , trdnomciudes VARCHAR(80) ); Timedimension data:20080101,2008,"1",1,1,"Ene","",1,"Martes","2008-01-01","01" bi_con_cdr_diario data:2013,"092013",4,"CTA",4,"EMBRATEL","HITBRAEMB","2013-09-04 00:00:00",3608,"Air Liquide do Brasil Ltda - Canoas",420,"Flavio Eugenio/Caroline Carneiro","E-N","E-N","ddd","PSTN","Fijo",89,"Consumo",459,7.65,10,10,0.66924,0.66924,1.43199992,1.1456000328063967,3.437,0.471,3.9080000000000004,55,"Brasil",55,"Brazil","HitBrasil","HIT VOICE",0,"0","HIT CALLSERVICE EMB NAL",20130904,1,0,"0",\N But the error is when i do group by as (select sum(minutosfacturados),succodigo,montht,comnombre2 from bi_con_cdr_diario as cdr inner join timedimension as td on cdr.idtimedimension=td.id where comcodigo=4 group by succodigo,comnombre2,montht;) and have 10.000.000 of rows. I appreciate very much your help. I believe this is one great product. Best Regards, Edgar Mejia

1 0

Re: Contents of users-list Digest, Vol 13, Issue 20
by Edgar Mejia 25 Sep '13

25 Sep '13

Thank you Jennie Zhang. 1. I have MonetDB Database Server Toolkit v1.1 (Feb2013-SP3) (i install so: sudo apt-get install monetdb5-sql monetdb-client) shell> monetdbd create /path/to/mydbfarm shell> monetdbd start /path/to/mydbfarm shell> monetdb create voc shell> monetdb release voc 2. Ubuntu 12.04.3 LTS 3. Schema view schemadatamonetdb.sql file 4. Data: timedimension view datatimediM.csv bi_con_cdr_diario: cdr.csv But the error is when i do group by as (select sum(minutosfacturados),succodigo,montht,comnombre2 from bi_con_cdr_diario as cdr inner join timedimension as td on cdr.idtimedimension=td.id where comcodigo=4 group by succodigo,comnombre2,montht;) and have 10.000.000 of rows. I appreciate very much your help. I believe this is one great product. Best Regards, Edgar Mejia

1 0

Monetdb crashed as group by
by Edgar Mejia 24 Sep '13

24 Sep '13

select sum(minutosfacturados),succodigo,montht,comnombre2 from bi_con_cdr_diario as cdr inner join timedimension as td on cdr.idtimedimension=td.id where comcodigo=4 group by succodigo,comnombre2,montht; but for example as this query work fine: select sum(minutosfacturados),succodigo,montht from bi_con_cdr_diario as cdr inner join timedimension as td on cdr.idtimedimension=td.id where comcodigo=4 group by succodigo,montht; What i need to do for to fix this? Thanks Edgar M

3 2

Whats coming in upcoming build monetdb release
by Ashish Kumar Singh 21 Sep '13

21 Sep '13

Hi , During my periodic search for "Is a new monetdb build is going to be released soon" , I found one is placed under testing recently. Is there a place where it can be seen and reviewed what kind of features/fixes are expected in upcoming release? That will be a great help for some of the folks like me to plan ahead! Regards, Ashish

1 0

Re: users-list Digest, Vol 13, Issue 16
by Rose Beck 21 Sep '13

21 Sep '13

Thanks a lot for replying :) But its really unclear to me as to how can I load RDF data into monetdb. As I have a 1000 property values and I can not manually create table for each of the property values. TSQL allows something like SELECT * INTO someTable FROM sourceTable. RDF query engines like RDF3X, BitMat (Medha Atre, et.al.) have all employed monetdb. So, if there is some way out can you please help me with it. On Sat, Sep 21, 2013 at 12:00 PM, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: How to automate creation of property tables in monetdb? > (Rose Beck) > 2. Re: How to automate creation of property tables in monetdb? > (Martin Kersten) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 21 Sep 2013 11:49:10 +0530 > From: Rose Beck <rosebeck.82(a)gmail.com> > To: users-list(a)monetdb.org > Subject: Re: How to automate creation of property tables in monetdb? > Message-ID: > <CAPx9ze9DR0bmy9joSZewwrSL_UNQGP=HefZLHax= > WZAAfDK3Cw(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > A gentle reminder. Can someone please help me with this? > > > > > On Fri, Sep 20, 2013 at 7:58 PM, Rose Beck <rosebeck.82(a)gmail.com> wrote: > > > I am trying to load data (from myfile, the format of myfile is explained > > below) into monetdb using itsSQL interface. mytable format: > > > > 1 2 3 > > 5 7 8 > > 8 2 3..... > > > > The commands which I am using for performing the same are given below: > > > > sql> create table mytable(subject integer, property integer, object > integer); > > > > sql> copy 10000000 records into mytable from myfile delimiter ' ','\n'; > > > > For each property value (i.e. middle column) I create a separate table: > > > > create table prop1(subject integer, object integer); > > create table prop2(subject integer, object integer); > > create table prop3(subject integer, object integer); ... > > > > and then fill these tables using the following commands: > > > > INSERT into prop1 SELECT subject, object FROM triples_table where > property=prop1 ORDER BY subject, object; > > > > Is there some way to automate this process. i.e. right now I am manually > > creating a separate table for all the distinct properties (using create > > table prop1(subject integer, object integer); ) and then filling this > > table using insert command (INSERT into prop1 SELECT subject, object FROM > > triples_table where property=prop1 ORDER BY subject, object; ). Is there > > some way by which I may automate this procedure. > > > > One of the methods which I know of is to create a C++/python program and > > then call all functions like "create table" by using shell script. But is > > there a faster method to achieve the same. > > > > Also can you please guide as to how can I create and integrate dictionary > > (RDF: id to string mapping) to sql queries? > > <users-list(a)monetdb.org> > > >

2 2

How to automate creation of property tables in monetdb?
by Rose Beck 21 Sep '13

21 Sep '13

I am trying to load data (from myfile, the format of myfile is explained below) into monetdb using itsSQL interface. mytable format: 1 2 3 5 7 8 8 2 3..... The commands which I am using for performing the same are given below: sql> create table mytable(subject integer, property integer, object integer); sql> copy 10000000 records into mytable from myfile delimiter ' ','\n'; For each property value (i.e. middle column) I create a separate table: create table prop1(subject integer, object integer); create table prop2(subject integer, object integer); create table prop3(subject integer, object integer); ... and then fill these tables using the following commands: INSERT into prop1 SELECT subject, object FROM triples_table where property=prop1 ORDER BY subject, object; Is there some way to automate this process. i.e. right now I am manually creating a separate table for all the distinct properties (using create table prop1(subject integer, object integer); ) and then filling this table using insert command (INSERT into prop1 SELECT subject, object FROM triples_table where property=prop1 ORDER BY subject, object; ). Is there some way by which I may automate this procedure. One of the methods which I know of is to create a C++/python program and then call all functions like "create table" by using shell script. But is there a faster method to achieve the same. Also can you please guide as to how can I create and integrate dictionary (RDF: id to string mapping) to sql queries? <users-list(a)monetdb.org>

2 2

Shouldn't multiple identical bind calls be impossible?
by Roberto Cornacchia 19 Sep '13

19 Sep '13

I'd like to ask a simple question before reporting what I suspect to be a bug. On a Feb2013, using default_pipe (same holds for no_mitosis and sequential pipes), no matter what the query and the data schema look like (really, that shouldn't matter here), is it for any reason possible to have a MAL plan with repeated, identical bind operations? Like this: function user.s4_1():void; ... X_40 := sql.bind(X_1,"spinque","dict","idstr",0); ... X_126 := sql.bind(X_1,"spinque","dict","idstr",0); ... X_298 := sql.bind(X_1,"spinque","dict","idstr",0); .. X_384 := sql.bind(X_1,"spinque","dict","idstr",0); ... ... and so on My understanding is: with commonTerms optimizer enabled (I'm just using the standard pipes), there should be no way to see those calls repeated: - they have no side-effects - they do not depend on any user data Why isn't the first bind reused in the rest of the plan? Of course my worries are not much about those calls per se, which are super fast, but about my suspicion that entire subqueries built on top of such binds are replicated unnecessarily. I don't want to focus on the actual query here, because if I am correct (please correct me otherwise), the situation above is already enough to submit a more complete bug report. By the way, another suspicious instruction that I see in my MAL plan is: X_25 := X_24; Shouldn't this be removed by the aliases optimizer? Best, Roberto

3 4

overflow in calculation
by Hassan, Ahmad 19 Sep '13

19 Sep '13

Hi, I am trying latest MonetDB-11.15.11 release on Ubuntu 64bit with TPCH dataset of size 120GB (scale factor 100). I am testing on a system with 512Gb RAM. Out of 22 TPCH queries, query #1 always fail and thrown the following error: overflow in calculation. Command exited with non-zero status 1 113.04 Rest of the queries 2-22 runs fine for 120GB data. If I try with 200GB of data then more queries fail. Any suggestions to work around this problem please? Thanks. Kind Regards, Ahmad

2 1

tomograph not working with remote function?
by Mengmeng Chen 18 Sep '13

18 Sep '13

Hi, We have a MAL function and we are trying to use tomograph to profile the execution of it when it is executed on a remote node. However, the tomograph fails right after the function is executed. We are just wondering if we did something wrong and if tomograph can work with a remote function call. Here's what we want to do: >From node A, we run a MAL function, which will execute a MAL function on both node A and node B using remote.exec() method. Here's what we did and what happened: 1. in A start mclient 2. in A and B, start tomograph "tomograph -d db1 -h localhost" 3. in A, start the driver function, which will call remote.exec for the MAL function on both A and B Then, in A and B, tomograpg exits promptly ubuntu@MC-MoDB1:~->tomograph -d db1 -h localhost -- opened UDP profile stream MC-MoDB1:50010 for localhost Created tomogram 'tomograph' Run: 'gnuplot tomograph.gpl' to create the 'tomograph.pdf' file The memory map is stored in 'tomograph.dat' The trace is saved in 'tomograph.trace' for use with --trace option Looking at the tomograph trace file generated on node B, we have something like: 9 1430 18111 17039 71 71 2 0 0 remote.connect "X_22 := remote.connect(\"mapi:monetdb:// 192.168.4.4:50000/db1\ <http://192.168.4.4:50000/db1%5C>",\"monetdb\",\"monetdb\",\"msql\");", ] 11 1007 10501 9496 70 71 2 0 0 remote.connect "X_24 := remote.connect(\"mapi:monetdb:// 192.168.5.4:50000/db1\ <http://192.168.5.4:50000/db1%5C>",\"monetdb\",\"monetdb\",\"msql\");", ] 69 23408 23605 198 72 72 2 0 0 remote.bintype "remote.bintype();", ] 11 10867 10867 0 71 71 5 0 0 profiler.wait 0.00 0.04 0.75 0.00 0.00 0.00 0.00 0.17 0.17 0.17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.25 0.01 0.25 0.00 0.00 0.00 0.00 0.00 9 18497 18497 0 71 71 5 0 0 profiler.wait 0.00 0.04 0.75 0.00 0.00 0.00 0.00 0.17 0.17 0.17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.25 0.01 0.25 0.00 0.00 0.00 0.00 0.00 69 23978 23978 0 72 72 4 0 0 profiler.ping 0.00 0.04 0.75 0.00 0.00 0.00 0.00 0.17 0.17 0.17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.25 0.01 0.25 0.00 0.00 0.00 0.00 0.00 ================================ on node A, the tomograph trace file looks like: 4 327 453 127 66 66 2 0 0 remote.bintype "remote.bintype();", ] 4 684 684 0 66 66 4 0 0 profiler.ping 0.15 0.15 0.15 0.15 0.15 0.15 0.15 0.15 0.15 0.17 0.15 0.15 0.15 0.15 0.15 0.17 0.15 0.25 0.15 0.15 0.15 0.15 0.15 0.15 0.15 0.20 0.17 0.15 0.15 0.15 0.15 0.15 ================================ Any help would be appreciated! Thanks, Mengmeng

2 1

Re: Need information about culstering monetdb
by Hannes Mühleisen 16 Sep '13

16 Sep '13

Hello Luis, first, please direct such general questions to the mailing list. On 09/04/2013 01:01 AM, Luis Moreno wrote: > I want to know which way works a cluster system MonetDB > > This works distributing information in each node or replicated the > information in each node and distributing query processing or duplicated > information in each node and distributes the connexions as a load balancer > > I hope you can help me > > Im testing monetdb with the Pentaho business intelligence suit. To answer your questions, there are multiple ways of running a clustered MonetDB setup. There is some support for clustered operations in monetdbd, see the section on "MULTIPLEX-FUNNELS" [1]. Apart from that, some virtual duct tape is required to set things up, but this nothing to be afraid of and the load balancing scenario is certainly possible. Best, Hannes [1] http://www.monetdb.org/Documentation/monetdbd-man-page

1 0

Bulk Import
by Thomas Johann 15 Sep '13

15 Sep '13

Hi everybody, I have a problem inserting a large amount of data into a monetdb database using bulk import. I'm running the following commands in a loop: connection.execute("COPY *19000000* OFFSET 2 RECORDS INTO XXX FROM '" + csv + "' USING DELIMITERS ';','\n' ;") connection.commit() , where csv is a different csv-file in each round of the loop, but always containing 18000001 rows of data. To be sure that enough memory is allocated I chose 19000000 in the execute command. I now have two questions: 1. Should the number of records (here 19000000) represent the number of lines per .csv-file or the number of lines of the final database (number of csv-files * 18Mio.)??? 2. Can you think of any reason why monetdb would stop reading one specific variable, while continuing to read the others? Let's say my csv has 8 columns and 18000000 rows with no missing values in the raw data. Until Row 16537472 the total data is read-in, but for the following lines variable 3 is missing until line 18000000 while variable 1 as well as 3-8 are perfectly fine. Can this be due to memory or harddisk speed constraints? Why is no error message raised? It would be great if someone could help me. Thanks, Thomas

4 4

Monetdb blocked
by Edgar Mejia 14 Sep '13

14 Sep '13

Hi, please, i have on server as 3.750 mb as ubuntu 12.04, one processor and 2.000 mhz. I install here monetdb and have 12.000,000 millions of rows. I had one problem as one query. For example this query block database: select "timedimension"."namemonth" as "c0", "bi_con_cdr_diario"."comnombre2" as "c1", "bi_con_cdr_diario"."succodigo" as "c2", sum("bi_con_cdr_diario"."minutosfacturados") as "m0" from "timedimension" as "timedimension", "bi_con_cdr_diario" as "bi_con_cdr_diario" where "bi_con_cdr_diario"."idtimedimension" = "timedimension"."id" and "bi_con_cdr_diario"."comcodigo" = '4' group by "timedimension"."namemonth", "bi_con_cdr_diario"."comnombre2", "bi_con_cdr_diario"."succodigo" if i delete (and "bi_con_cdr_diario"."comcodigo" = '4' ) this work fine. Too i im testing very others querys and work fine, but why this not work and block database? Thaks for your help. Edgar M

1 0

Query Problem with LIKE clause
by Hancel Gonzalez 11 Sep '13

11 Sep '13

Hi, finally i installed MonetDB. Now i get the next problem =S When i execute one query, i get error and i dont know how to solve. Please, help me. =( Hancel. PD: Error is: *MAPI = (monetdb) /tmp/.s.monetdb.50000* *ACTION= read_line* *QUERY = select s0.value,s1.value* * from (* * (select f0.subject as r0,f0.object as r1* * from p1 f0,p0 f1* * where f1.subject=f0.subject and f1.object=58758 and ((select value from strings where id = f0.object) like '%ambilateral%'))* * )facts, strings s0, strings s1* * where s0.id=facts.r0 and s1.id=facts.r1;* *ERROR = !Connection terminated*

4 6

Compiling problem
by Hancel Gonzalez 11 Sep '13

11 Sep '13

Hi again, i have a problem installing MonetDB 11.9.7 from source code. When i do the * make* , get the next error in the process =S Can i help me? Many thanks. Regards, Hancel. PD: Error here: *fits.c:54:20: error: fitsio.h: No such file or directory* *fits.c:55:21: error: fitsio2.h: No such file or directory* *fits.c:56:21: error: longnam.h: No such file or directory* *fits.c: In function 'fits2mtype':* *fits.c:126: error: 'TBIT' undeclared (first use in this function)* *fits.c:126: error: (Each undeclared identifier is reported only once* *fits.c:126: error: for each function it appears in.)* *fits.c:127: error: 'TLOGICAL' undeclared (first use in this function)* *fits.c:129: error: 'TBYTE' undeclared (first use in this function)* *fits.c:130: error: 'TSBYTE' undeclared (first use in this function)* *fits.c:132: error: 'TSTRING' undeclared (first use in this function)* *fits.c:134: error: 'TUSHORT' undeclared (first use in this function)* *fits.c:135: error: 'TSHORT' undeclared (first use in this function)* *fits.c:137: error: 'TUINT' undeclared (first use in this function)* *fits.c:138: error: 'TINT' undeclared (first use in this function)* *fits.c:140: error: 'TLONG' undeclared (first use in this function)* *fits.c:141: error: 'TULONG' undeclared (first use in this function)* *fits.c:142: error: 'TLONGLONG' undeclared (first use in this function)* *fits.c:144: error: 'TFLOAT' undeclared (first use in this function)* *fits.c:146: error: 'TDOUBLE' undeclared (first use in this function)* *fits.c:149: error: 'TCOMPLEX' undeclared (first use in this function)* *fits.c:150: error: 'TDBLCOMPLEX' undeclared (first use in this function)* *fits.c: In function 'fits2subtype':* *fits.c:161: error: 'TBIT' undeclared (first use in this function)* *fits.c:162: error: 'TLOGICAL' undeclared (first use in this function)* *fits.c:165: error: 'TBYTE' undeclared (first use in this function)* *fits.c:166: error: 'TSBYTE' undeclared (first use in this function)* *fits.c:169: error: 'TSTRING' undeclared (first use in this function)* *fits.c:172: error: 'TUSHORT' undeclared (first use in this function)* *fits.c:173: error: 'TSHORT' undeclared (first use in this function)* *fits.c:176: error: 'TUINT' undeclared (first use in this function)* *fits.c:177: error: 'TINT' undeclared (first use in this function)* *fits.c:180: error: 'TULONG' undeclared (first use in this function)* *fits.c:181: error: 'TLONG' undeclared (first use in this function)* *fits.c:182: error: 'TLONGLONG' undeclared (first use in this function)* *fits.c:185: error: 'TFLOAT' undeclared (first use in this function)* *fits.c:188: error: 'TDOUBLE' undeclared (first use in this function)* *fits.c:192: error: 'TCOMPLEX' undeclared (first use in this function)* *fits.c:193: error: 'TDBLCOMPLEX' undeclared (first use in this function)* *fits.c: In function 'FITSexportTable':* *fits.c:209: error: 'fitsfile' undeclared (first use in this function)* *fits.c:209: error: 'fptr' undeclared (first use in this function)* *fits.c:303: error: 'USHORT_IMG' undeclared (first use in this function)* *fits.c:305: error: 'READWRITE' undeclared (first use in this function)* *fits.c:308: error: 'BINARY_TBL' undeclared (first use in this function)* *fits.c:332: error: 'TLOGICAL' undeclared (first use in this function)* *fits.c:361: error: 'TBYTE' undeclared (first use in this function)* *fits.c:419: error: 'TSHORT' undeclared (first use in this function)* *fits.c:448: error: 'TINT' undeclared (first use in this function)* *fits.c:477: error: 'TLONG' undeclared (first use in this function)* *fits.c:506: error: 'TFLOAT' undeclared (first use in this function)* *fits.c:535: error: 'TDOUBLE' undeclared (first use in this function)* *fits.c: In function 'FITSdir':* *fits.c:573: error: 'fitsfile' undeclared (first use in this function)* *fits.c:573: error: 'fptr' undeclared (first use in this function)* *fits.c:588: error: 'READONLY' undeclared (first use in this function)* *fits.c: In function 'FITSdirpat':* *fits.c:607: error: 'fitsfile' undeclared (first use in this function)* *fits.c:607: error: 'fptr' undeclared (first use in this function)* *fits.c:631: error: 'READONLY' undeclared (first use in this function)* *fits.c: In function 'FITStest':* *fits.c:646: error: 'fitsfile' undeclared (first use in this function)* *fits.c:646: error: 'fptr' undeclared (first use in this function)* *fits.c:651: error: 'READONLY' undeclared (first use in this function)* *fits.c: In function 'FITSattach':* *fits.c:670: error: 'fitsfile' undeclared (first use in this function)* *fits.c:670: error: 'fptr' undeclared (first use in this function)* *fits.c:684: error: 'READONLY' undeclared (first use in this function)* *fits.c:731: error: 'ASCII_TBL' undeclared (first use in this function)* *fits.c:731: error: 'BINARY_TBL' undeclared (first use in this function)* *fits.c:735: error: 'TSTRING' undeclared (first use in this function)* *fits.c: In function 'FITSloadTable':* *fits.c:834: error: 'fitsfile' undeclared (first use in this function)* *fits.c:834: error: 'fptr' undeclared (first use in this function)* *fits.c:842: error: 'FLEN_VALUE' undeclared (first use in this function)* *fits.c:878: error: 'READONLY' undeclared (first use in this function)* *fits.c:886: error: 'ASCII_TBL' undeclared (first use in this function)* *fits.c:886: error: 'BINARY_TBL' undeclared (first use in this function)* *fits.c:906: error: 'TSTRING' undeclared (first use in this function)* *fits.c:970: error: 'FLEN_ERRMSG' undeclared (first use in this function)* *make[9]: *** [lib_fits_la-fits.lo] Error 1* *make[9]: Leaving directory `/home/hancel/MonetDB-11.9.7/sql/backends/monet5/vaults'* *make[8]: *** [all] Error 2* *make[8]: Leaving directory `/home/hancel/MonetDB-11.9.7/sql/backends/monet5/vaults'* *make[7]: *** [all-recursive] Error 1* *make[7]: Leaving directory `/home/hancel/MonetDB-11.9.7/sql/backends/monet5'* *make[6]: *** [all] Error 2* *make[6]: Leaving directory `/home/hancel/MonetDB-11.9.7/sql/backends/monet5'* *make[5]: *** [all-recursive] Error 1* *make[5]: Leaving directory `/home/hancel/MonetDB-11.9.7/sql/backends'* *make[4]: *** [all] Error 2* *make[4]: Leaving directory `/home/hancel/MonetDB-11.9.7/sql/backends'* *make[3]: *** [all-recursive] Error 1* *make[3]: Leaving directory `/home/hancel/MonetDB-11.9.7/sql'* *make[2]: *** [all] Error 2* *make[2]: Leaving directory `/home/hancel/MonetDB-11.9.7/sql'* *make[1]: *** [all-recursive] Error 1* *make[1]: Leaving directory `/home/hancel/MonetDB-11.9.7'* *make: *** [all] Error 2*

4 11

Setting connection time-out?
by Dennis Pallett 10 Sep '13

10 Sep '13

Hello, Is it possible to set a time-out in MonetDB for connections? On our test instance of MonetDB we are running into connection limits because our developers are starting/stopping our application all day long without properly closing the connections. On production this is not a problem because there the connections are closed properly. Best regards, Dennis Pallett

1 0

GeoSpatial
by Lee Hachadoorian 05 Sep '13

05 Sep '13

I found out about MonetDB a few months ago and am currently giving it a whirl. I am a geographer and am interested in the GeoSpatial extension. I have found limited documentation [1][2], which (I think) has changed since I first saw it a few months ago. I remember something about a shapefile import application that was based on shp2pgsql used by PostGIS, and also something about using OpenJump to visualize the data stored in the MonetDB database, but I cannot find this information any more. I have successfully created a spatial table by copying a CREATE TABLE statement from PostGIS, then using COPY ... TO in PostGIS and COPY ... FROM in MonetDB. But I would like to know if a shp2pgsql equivalent exists for MonetDB, or if there is any work on or plans for OGR functionality. I would also like to know how users visualize MonetDB spatial data in OpenJump or any other GIS, which incidentally also leads into the question of OGR functionality since so many FOSS GISes (e.g. QGIS) use OGR for data loading. Best, --Lee [1] http://www.monetdb.org/Documentation/Extensions/GIS [2] http://www.monetdb.org/Documentation/Extensions/GIS/SpatialTypes -- Lee Hachadoorian Assistant Professor in Geography, Dartmouth College http://freecity.commons.gc.cuny.edu

2 1

Database crash; won't start anymore
by Ralph Holz 30 Aug '13

30 Aug '13

Hi, I've tried to load a good chunk of the Internet census into our MonetDB - about 100M rows. Although the import scripts completed, it seems it shot down Monet: 2013-08-28 04:10:35 MSG merovingian[9857]: database 'census' (10773) was killed by signal SIGBUS 2013-08-28 10:47:59 MSG merovingian[9857]: database 'census' has crashed after start on 2013-08-28 01:14:07, attempting restart, up min/avg/max: 4m/4d/3w, crash average: 1.00 0.10 0.03 (8-7=1) This is then followed by 2013-08-28 10:48:32 MSG census[19161]: # still reading write-ahead log "/mnt/monetdb/census/sql_logs/sql/log.990" (0% done) [...] 2013-08-28 10:53:12 MSG census[19161]: # still reading write-ahead log "/mnt/monetdb/census/sql_logs/sql/log.990" (4% done) 2013-08-28 10:53:34 MSG merovingian[9857]: database 'census' (19161) has crashed (dumped core) Restarting it, yielded the same results for a few times. Once or twice it got to 35% in the WAL 2013-08-28 15:34:27 MSG census[14420]: # still reading write-ahead log "/mnt/monetdb/census/sql_logs/sql/log.990" (35% done) 2013-08-28 15:34:59 MSG merovingian[9857]: database 'census' (14420) has crashed (dumped core) 2013-08-28 15:34:59 ERR merovingian[9857]: client error: database 'census' has crashed after starting, manual intervention needed, check monetdbd's logfile for details 2013-08-28 15:34:59 ERR merovingian[9857]: client error: client (local) sent challenge in incomplete block: 2013-08-28 15:34:59 ERR merovingian[9857]: client error: client (local) sent challenge in incomplete block: 2013-08-28 15:34:59 MSG merovingian[9857]: database 'census' has crashed after start on 2013-08-28 14:43:34, attempting restart, up min/avg/max: 4m/4d/3w, crash average: 1.00 0.90 0.30 (16-7=9) Since then, it won't start (I don't know what caused the client error BTW). Data loss is not a concern, but we'd sure like to know if this can be fixed or if we should start looking at another DB... Thanks for any help, Ralph -- Ralph Holz I8 - Network Architectures and Services Technische Universität München http://www.net.in.tum.de/de/mitarbeiter/holz/ Phone +49.89.289.18043 PGP: A805 D19C E23E 6BBB E0C4 86DC 520E 0C83 69B0 03EF

3 7

Archlinux AUR Package
by Brian Hood 29 Aug '13

29 Aug '13

Hi All, I maintain a package build in the user repository which i update every new release. aur.archlinux.org - just search Regards, Brian Hood

1 0

Clustered Indexes with monetdb
by Ashish Kumar Singh 28 Aug '13

28 Aug '13

HI All, Can somebody point me to relevant information on how to do clustered indexes with monetdb? Regards, Ashish

3 7

Server - location of tmp database objects
by Radovan Bičiště 27 Aug '13

27 Aug '13

Hello, I was wondering if there is a way to specify for database where to put tmp files? For example when we use temporary tables etc. it is filling our data disk. I have another disk on the machine that I would like to use for temporary tables etc. Thank you for any hints. Best regards, Radovan -- __________________________ Radovan Bičiště ceos data s.r.o. třída SNP 402/48 500 03 Hradec Králové Czech Republic mobil CZ: +420 601 563 014 skype: rbiciste

4 3

Technical Problem connecting with database
by Hancel Gonzalez 27 Aug '13

27 Aug '13

Hi, i have a problem: i installed *monetdb 11.9.7 *from source code and created the database farm and the database with commands *monetdbd *and *monetdb, *respectively. The problem is when i try connect to the database (*mclient*) and dont work, see the log file but i dont how to solve. Please, anyone can help me to solve? This is the message in log file: *2013-08-26 21:48:41 MSG merovingian[7622]: starting database 'Berlin10MDB', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0)* * * *2013-08-26 21:48:41 MSG Berlin10MDB[7668]: arguments: /usr/local/bin/mserver5 --set gdk_dbfarm=/home/gdesimone/monet_data --dbname=Berlin10MDB --set merovingian_uri=mapi:monetdb://Athena:50000/Berlin10MDB --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/gdesimone/monet_data/Berlin10MDB/.mapi.sock --set monet_vault_key=/home/gdesimone/monet_data/Berlin10MDB/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes* * * *2013-08-26 21:48:41 MSG merovingian[7622]: database 'Berlin10MDB' (7668) has exited with exit status 127* * * *2013-08-26 21:48:41 ERR merovingian[7622]: client error: database 'Berlin10MDB' appears to shut itself down after starting, check monetdbd's logfile for possible hints* * * Many thanks! Regards, Hancel.

3 5

Database Problem
by Thomas Johann 23 Aug '13

23 Aug '13

Dear all, I’m new to monetdb database programming and have a problem setting up my first database which I can not explain by myself: I’m basically running the following Python code: tempdir = 'E:\\xetra\\tempdir\\monetdb' connection = monetdb.sql.connect(username="monetdb", password="monetdb", hostname="localhost", database="database") # expected table layout connection.execute("CREATE TABLE XXX(Name_1 TEXT, Name_2 TEXT, Name_3 TEXT, Name_4 TEXT, Num_1 INT, Num_2 REAL,Num_3 INT,Name_5 CHARACTER(1))") def convert_table(filename,path): print("Reading CSV") try: start = time.clock() csv_to_sql = path + "\\" + filename connection.execute("COPY 100000000 OFFSET 2 RECORDS INTO XXX FROM '" + csv_to_sql + "' USING DELIMITERS ';','\n' ;") elapsed = (time.clock() - start) connection.commit() print("Time to write to database: " + str(elapsed) + " seconds") except: #get wrong transaction out of DB with rollback connection.rollback() try: #DB conversion start = time.clock() infile = open(csv_to_sql, 'rb') csv_to_sql_converted=path+ '\\converted.csv' outfile = open(csv_to_sql_converted, 'wb') BLOCKSIZE = 100000000 # experiment with size while True: block = infile.read(BLOCKSIZE) if not block: break outfile.write(block.decode('cp1252','strict').encode('utf-8','strict')) infile.close() outfile.close() elapsed = (time.clock() - start) print("Time to convert file to UTF-8: " + str(elapsed) + " seconds") #write to DB start = time.clock() connection.execute("COPY 100000000 OFFSET 2 RECORDS INTO XXX FROM '" + csv_to_sql_converted + "' USING DELIMITERS ';','\n' ;") connection.commit() os.remove(csv_to_sql_converted) elapsed = (time.clock() - start) print("Time to write to database: " + str(elapsed) + " seconds") except Exception as inst: #get wrong transaction out of DB with rollback connection.rollback() print type(inst) # the exception instance print inst.args # arguments stored in .args print inst # __str__ allows args to printed directly print("Could not write file "+csv_to_sql+"to the database XXX") filename_list = [“XYZ.csv”, “ABC.csv”,…] path = “C:\\” for filename in filename_list: convert_table(filename, path) I had to alter the code as the actual csvs contain very sensitive data, which I am not allowed to publish. But the structure is the same. I do not get any error message and the program runs through properly. Usually it goes through the “try:” part of the function. When I look into the resulting database the number of lines is as expected (so the observations must be somewhere in the database) but there are some gaps in the output, meaning that some values from the .csvs are just not copied and I am unable to compare those gap-lines to the original data as the identifier is usually among the missing variables. Some variables from the original data are just missing. This happens (at least to what I can say) in an unsystematic way. I have the impression that MonetDB might just be overwhelmed by the amount of data. Is this possible? The .csvs have each a size of 1.3GB and I’m reading more than 1000 of them into the database which has a final size of about 2 TB. I’m running this under Windows Server 2008 R2, 32GB RAM, AMD Opteron 4234 6-Core 3.10 GHz, using Anaconda. When I manually read in one single file, that made problems beforehand, I will find it in the database without error. So, I’m pretty puzzled what is going on here. Might it happen that the connection.execute-command runs through without triggering an error message but still not reading the whole dataset? I would be very glad about any help from the community. Thanks, Thomas

2 2

Installation and configuration on Windows 8
by m- dev 22 Aug '13

22 Aug '13

I'm not familiar with database administration, but I guess I'm having similar problems to this Windows user: http://mail.monetdb.org/pipermail/users-list/2013-June/006078.html I installed DB5-SQL, Geom and ODBC 32 and 64bit, default options for all. After that it's not clear to me which steps are for Linux or Windows, or both. Or how to set up the configuration file. In Windows 8 I start 'MonetDB SQL Server'. I can also start 'MonetDB SQL Client'. i) I added PATH in Environment variable to the installation folder (that defaults to 32-bit Program files): ;C:\Program Files\MonetDB\MonetDB5 But I couldn't set up a configuration file, either from within the Server: > cat ~/.monetdb SyntaxException:parseError:cat ~/.monetdb SyntaxException:parseError: ^,';' expected Or the Client: sql>cat ~/.monetdb more>name=#### more>password=######### more>language=SQL more> Creating that as a text file renamed config.monetdb in the installation folder doesn't work either. ii) I've extracted the voc_dump.sql file, but the Guide doesn't specify the location of mclient.exe and when I found it in C:\Program Files\MonetDB\MonetDB5\bin but it won't open because: "libmapi.dll is missing...Try reinstalling the program to fix this problem" Clearly I haven't set it up MonetDB correctly, I can only guess, but what are the correct steps? Is there any other guide or tutorial on installation and configuration on Windows 8? Thanks, Mark

2 2

SELECT DISTINCT on datatype inet causes high load even after cancelling
by Ralph Holz 22 Aug '13

22 Aug '13

Good day, We're experiencing a phenomenon that I can't quite explain. We have a table of maybe 110M entries mapping IP addresses to some values. The data type of field ip_addr is net. No indices or constraints set. A SELECT COUNT(ip) is zippy - result in some ms. Any SELECT, in fact, that does not use WHERE is fast. E.g.: SELECT ip, result FROM iptable LIMIT 10000; +------------------ 10000 tuples (4.6s) However: A SELECT COUNT(DISTINCT ip) does not return even after waiting an hour or, just to try it out once, 24 hrs. Load remained at 27 during that time. I restarted the server after that, as I had done ctrl-c in mclient, and that seems to let the current query continue? What would be the right way to cancel a query? I then tried SELECT ip FROM iptable WHERE ip = inet '1.0.0.1' That IP is the second in the column. The query did not return even after 10 minutes. That seems a bit excessive. Does anyone have a clue why this is so? Should I avoid inet? Thanks, Ralph Disclaimer: I come from a psql background and these are my first steps with vertical-col data stores. Server: MonetDB 5 server v11.15.11 "Feb2013-SP3" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 142.0GiB available memory, 24 available cpu cores Libraries: libpcre: 8.30 2012-02-04 (compiled with 8.02) openssl: OpenSSL 0.9.8o 01 Jun 2010 (compiled with OpenSSL 0.9.8o 01 Jun 2010) libxml2: 2.7.8 (compiled with 2.7.8) Compiled by: root(a)dev.monetdb.org (x86_64-pc-linux-gnu) Compilation: gcc -O3 -fomit-frame-pointer -pipe -Wp,-D_FORTIFY_SOURCE=2 Linking : /usr/bin/ld -m elf_x86_64 -- Ralph Holz I8 - Network Architectures and Services Technische Universität München http://www.net.in.tum.de/de/mitarbeiter/holz/ Phone +49.89.289.18043 PGP: A805 D19C E23E 6BBB E0C4 86DC 520E 0C83 69B0 03EF

1 0

MonetDB OOM crash in CentOS
by Ravikumar Balasubramaniam 20 Aug '13

20 Aug '13

Hi, MonetDB is crashing because of OOM. I have tried following config update but in vain. sysctl -w vm.overcommit_memory=2 sysctl -w vm.overcommit_ratio=75 Is there any configuration to limit mserver memory usage? Database contains 300 million records and the select query expects 120,620 records as result. This is the log message: Out of memory: Kill process 30650 (mserver5) score 981 or sacrifice child Killed process 30650, UID 222, (mserver5) total-vm:35263192kB, anon-rss:14911384kB, file-rss:208kB *mserver5 version*: MonetDB 5 server v11.15.11 "Feb2013-SP3" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 14.5GiB available memory, 4 available cpu cores Libraries: libpcre: 7.8 2008-09-05 (compiled with 7.8) openssl: OpenSSL 1.0.0 29 Mar 2010 (compiled with OpenSSL 1.0.0-fips 29 Mar 2010) libxml2: 2.7.6 (compiled with 2.7.6) Compiled by: ec2-user@ip-10-252-152-159 (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 *Filesystem Size Used Avail Use% Mounted on* /dev/xvde1 5.0G 2.4G 2.4G 50% / tmpfs 7.3G 0 7.3G 0% /dev/shm /dev/xvdf 414G 209G 184G 54% /media/ephemeral0 (dbfarm) Regards, Ravi

1 0

MonetDB cluster management in EC2
by Ravikumar Balasubramaniam 20 Aug '13

20 Aug '13

I have followed http://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/ClusterManagement… to setup 3 node(in amazon EC2) cluster. I have considered the server from where I control remote databases as leader and others as remote servers. The 'monetdb discover' in the leader is not showing any other database(except self dbs). I have enabled discovery and control properties in the remote servers. What other configurations are needed for successful discovery in EC2 environment? MonetDB cluster enables mclient to combine all the (select)query results from remote servers and itself. The application should take care of shared write into nodes. Is this fair understanding? Regards, Ravi

2 2

MonetDB and Initial Memory requirements of database startup
by Ashish Kumar Singh 20 Aug '13

20 Aug '13

Hi All, Is there any check in monetdb at startup time which will ascertain that for a dataset of size x mb in hard disk how much minimum free memory should be available in system so that monetdb can run successfully. If so please point me to relevant section. Thanks! Regards, Ashish

2 2

change variables within select statement
by Christian Kleinke 19 Aug '13

19 Aug '13

Dear all, is there a possibility to change declared variables in monetdb within a select statement. For example in MySQL the variables can be declared with the @-symbol, and such variables easily could be changed in a case-when condition. Here a little example of what I want to do (in MySQL syntax): SET @id = ''; SET @pos = 1; drop table if exists test_table; create table test_table as select *, if (@id<> id, @pos:=1, @pos:=@pos+1) as position_number, @id:=id, from table_with_ids; Here the rows with identical ids should be counted, and if the id changes the position_number should start again as zero. If somebody has a hint for me, to execute such a query in monetdb, I would be very glad. Greetings, Christian

3 3

Fwd: MonetDB pip python3
by Martin Kersten 18 Aug '13

18 Aug '13

-------- Original Message -------- Subject: MonetDB pip python3 Date: Sat, 17 Aug 2013 20:21:51 +0200 From: Christian Tismer <tismer(a)stackless.com> To: Martin Kersten <Martin.Kersten(a)cwi.nl> Hi Martin, by chance I came along MonetDB today, and I think this can be the solution to my database problems. (used zodb and durus, but there are way too many columns and rows to handle with these). I am trying to use MonetDB with python 3.3. Unfortunately, the pip installer sees only the python 2 version. I had a look into the repository - the python 3 version is there. I saw you active right now, so I'm asking: Should I clone the repo and install the py3.3 version from there, or is it planned to add this to pip, soon? (I can work on this if it makes sense) cheers - Chris (that Stackless guy...) -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

2 1

b and g must be aligned
by Miguel Ping 16 Aug '13

16 Aug '13

Hi, I'm resurrecting this since I've been out of town and only today I got a chance to investigate further. I've recompiled with -O0 to prevent optimizations from "hiding" the values, and in the debugger I got this: b->U->count BUN 764314 g->U->count BUN 17 b->H->seq oid 0 g->H->seq oid 0 b->H->dense unsigned int 1 g->H->dense unsigned int 1 The stack call is as follows: BAThllaggr() at udf.c AGGRsubhllaggcand() at udf.c AGGRsubhllagg() at udf.c malCommandCall() at mal_interpreter.c runMALsequence() at mal_interpreter.c DFLOWworker() at mal_dataflow.c start_thread() at pthread_create.c clone() at clone.S 0x0 -------- Original Message -------- Subject: Re: b and g must be aligned Date: Fri, 26 Jul 2013 11:34:53 +0100 From: Sjoerd Mullender <sjoerd(a)acm.org> Reply-To: Communication channel for MonetDB users <users-list(a)monetdb.org> To: Communication channel for MonetDB users <users-list(a)monetdb.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2013-07-26 12:23, Miguel Ping wrote: > On 07/25/2013 01:51 PM, Sjoerd Mullender wrote: On 2013-07-25 > 14:04, Miguel Ping wrote: >>>> Hi all, >>>> >>>> We're hitting this error "b and g must be aligned". I tracked >>>> the src to a commit about some alignment code thing in >>>> gdk_calc: >>>>http://www.mail-archive.com/checkin-list@monetdb.org/msg09731.html >>>> >>>> (Fix alignment conversion in compatibility code for grouped >>>> aggregates.) >>>> >>>> Can you guys please explain what's the reason behind this >>>> error? I can't understand by just looking to the src of >>>> gdk_calc.c >>>> >>>> Thanks! > When using grouped aggregates, the grouping bat must be aligned > with the value bat. The value bat is b and contains the values you > want to aggregate. The group bat is g and contains for each value > in b the group (an oid) it belongs to. Equal group ids means the > same group. These bats must be aligned, because we need to know for > each value in b to which group it belongs. Aligned means: same > length, and same head column values. The head columns must be > dense (a sequence of numbers starting at some value, and each next > value exactly one larger than the previous). Dense sequences are > usually not stored explicitly in MonetDB. We only store the first > value in the hseqbase field. So the hseqbase fields of b and g > must be equal. The one exception to this is when the bats are both > empty. This last exception is the change to gdk_calc.c in that > changeset. > > -- Sjoerd Mullender >> _______________________________________________ users-list >> mailing listusers-list(a)monetdb.org >>http://mail.monetdb.org/mailman/listinfo/users-list >> >> > Thanks for your explanation. I still don't understand how can > there be a misalignment. I would expect MonetDB to feed my hll > aggregate functions with the correct values. Can it be that I may > have some NULL values and the validation is failing because of > that? If this happens when running a SQL query, it's a bug. I don't think NULLs have anything to do with it. NULL values are stored in-line. You might want to look at b->U->count, g->U->count, b->H->seq, g->H->seq, b->H->dense, g->H->dense when the misalignment happens (either in the debugger or by using printf--but realize that count and seq are not int, so %d is not going to work). Also things like the MAL plan (prepend SQL query with EXPLAIN) and the stack trace might be useful. - -- Sjoerd Mullender -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird -http://www.enigmail.net/ iQCVAwUBUfJQyT7g04AjvIQpAQK/JAP9HCp/aFaYWv0jodfPUnSRVgFSsdjTn/VL ttSsmAF+yomGMDIne2311f/D51F3/nte7Utx+01lgvArapWErhjGN1hzPSr5LQbs PZ6dUfNcH8Rt2AtT3uSxfkFZy9VRDCNNXPei43IgMS2HxVZ48pnAVkNcpBbW3Gms GwdvU7bSZtM= =2zSg -----END PGP SIGNATURE----- _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

5 20

monetdb lost column info when queryring
by integrity 16 Aug '13

16 Aug '13

some of the column "clientport" is NULL, but if change to another query method, it appeared again. 2. another bug: sql>select * from t_ivr_result limit 1; could not allocate space for HEAPextend: failed to extend to 15096348672 for 02/04/20413.theap another time: ERROR: HEAPextend: failed to extend to 23460839424 for 02/21/22122.theap it seems monetdb applied a heap of 15G ,the RAM is 64G, root@10.187.22.175:~# free -g total used free shared buffers cached Mem: 62 45 16 0 0 43 -/+ buffers/cache: 2 60 Swap: 2 0 2 so the left memory is 16G, but failed to applied 15G heap. and the table t_ivr_result is only 3G, there is no need to apply such a large heap. sql>select count(*) from t_ivr_result; +----------+ | L1 | +==========+ | 40299239 | +----------+ 1 tuple (137.023ms) sometimes the heap is 24G: ERROR: MALException:mat.pack:could not allocate space for ERROR: HEAPextend: failed to extend to 23460839424 for 02/21/22122.theap Thanks, Meng

2 2

Timestamp truncation in MonetDB
by Ravikumar Balasubramaniam 13 Aug '13

13 Aug '13

Hi, I'm trying to truncate timestamp value to hour and/or minute precision. I combined cast and extract functions as: Problem: Truncate '2013-08-12 23:59:59' to '2013-08-12 23:00:00' select cast(timestamp '2013-08-12 23:59:59' as date)||' '||extract(hour from timestamp '2013-08-12 23:59:59')||':00:00'; Output: 2013-08-12 23:00:00 But this is expensive because of repeated function calls. Is there any faster and built-in function? Regards, Ravi

2 1

Count number of records returned by a subquery in monetdb
by Ravikumar Balasubramaniam 13 Aug '13

13 Aug '13

Hi, I try to execute a query like following: When I execute, select table_name from system order by table_name; no syntax error. But when I execute, select count(*) from (select table_name from system order by table_name) as count; I get syntax error, unexpected ORDER, expecting INTERSECT or EXCEPT or UNION or ')' in: "select count(*) from (select table_name from system order" What is the correct syntax? Regards, Ravi

3 2

Integrating a C++ module in MonetDB
by Adnan Agbaria 13 Aug '13

13 Aug '13

Hi All, I've implemented a module implemented in C++ for MonetDB. You can think that this module has a user-defined function. What changes should be made in Makefile.ag (of this module) to compile and build it with C++ compiler. Regards!

2 1

potential data contention in _list_find_name() during BAT binding?
by Mengmeng Chen 13 Aug '13

13 Aug '13

Hi, I'm reading the source code of MonetDB and I have a question about multi-threaded BAT binding and would appreciate it if someone could help to answer. Here is my question: When a sql_bind() MAL instruction is called, I see we have a callstack something like: mvc_bind_wrap -> mvc_bind-> mvc_bind_column -> find_sql_column -> _cs_find_name->_list_find_name In _list_find_name, we have something like: if (l && !l->ht && list_length(l) > HASH_MIN_SIZE && l->sa) { l->ht = hash_new(l->sa, list_length(l), (fkeyvalue)&base_key); for (n = l->h; n; n = n->next ) { sql_base *b = n->data; int key = base_key(b); hash_add(l->ht, key, b); } } if (l && l->ht) { int key = hash_key(name); sql_hash_e *he = l->ht->buckets[key&(l->ht->size-1)]; for (; he; he = he->chain) { sql_base *b = he->value; if (b->name && strcmp(b->name, name) == 0) return b; } return NULL; } if (l) for (n = l->h; n; n = n->next) { sql_base *b = n->data; /* check if names match */ if (name[0] == b->name[0] && strcmp(name, b->name) == 0) { return b; } } Looks like the function is trying to create a hash table for the list if called the first time. However, there is no protection against data contention. Now suppose the mitosis optimizer is turned on and we could have something like: X_157:bat[:oid,:oid] := sql.tid(X_4,"sys","lineitem",0,8); X_159:bat[:oid,:oid] := sql.tid(X_4,"sys","lineitem",1,8); X_161:bat[:oid,:oid] := sql.tid(X_4,"sys","lineitem",2,8); X_163:bat[:oid,:oid] := sql.tid(X_4,"sys","lineitem",3,8); X_165:bat[:oid,:oid] := sql.tid(X_4,"sys","lineitem",4,8); X_167:bat[:oid,:oid] := sql.tid(X_4,"sys","lineitem",5,8); X_169:bat[:oid,:oid] := sql.tid(X_4,"sys","lineitem",6,8); X_171:bat[:oid,:oid] := sql.tid(X_4,"sys","lineitem",7,8); If my understanding is correct, since these MAL instructions has no dependency in each other, they could potentially be executed in different threads simultaneously. How would MonetDB prevent data contention in _list_find_name()? Particularly the case that if thread A has created a hash table and has not yet inserted the elements while thread B tries to search through the list and thinks the hash table is ready and starts to use it. It will fail to return the correct column since the hash table is empty, right? Thanks, Mengmeng

2 3

MonetDB and Tableau
by Dimitrios Gyalistras 12 Aug '13

12 Aug '13

Hello I am new to MonetDB and I want to use it together with the "Tableau Desktop" (Version 8) software running on a Mac, later possibly also together with a Tableau Server. However, the first attempts are discouraging: when attempting to connect to a MonetDB database, the Tableau Desktop software seems to get stuck in an endless loop. Some details: I run "MonetDB-11.15.11-i686-Darwin-9" on a Mac with OS X 10.6.8. I was able to create my first MonetDB database under OS X and define a simple 10-rows test table. The Tableau software runs under Windows 7 within Parallels Desktop 8. I installed the "2013-06-19 18:20 2.4M MonetDB ODBC driver, 32 bit" under Windows, and used the Administrative Tool "Data Sources (ODBC)" provided by Windows to configure the MonetDB DSN. In Tableau I finally selected Connect to Data/Other data bases (ODBC) and I was able to select a Table. Then I pressed OK – and Tableau got stuck in the process. I also tried using a custom SQL-query (SELECT * FROM "TestTable"). Here Tableau was able to establish a live connection to the MonetDB, but when working with the table, i.e. when dragging dimensions and measures into the rows and columns shelfs, Tableau announced connection errors. Does it make sense to proceed with this project? Any infos on the following configurations would be highly appreciated: MonetDB on a Mac <--> Tableau Desktop on a Mac MonetDB on a Linux server <--> Tableau Desktop on a Mac MonetDB on a Linux server <--> Tableau Server (Windows) Thanks Dimitrios –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– Dimitrios Gyalistras Synergy BTC AG Buristrasse 49 CH–3006 Bern Switzerland Tel: +41 78 602 54 09 Mail: gyalistras(a)synergy.ch Web: www.synergy.ch ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

1 0

COPY INTO with ordered column
by Sebastian Dorok 09 Aug '13

09 Aug '13

Hi all, I have some slow queries that I want to accelerate by utilizing information about sorting. My table definition: create table genomics.grch36 (site bigint, base char(1)); I use COPY INTO to populate the table. The source file size is 40GB and contains more than 3 billion rows. I can guarantee that the data in file is ordered by site. I query the data like this: sql>select base from genomics.grch36 where site between 10000 and 10010; +------+ | base | +======+ | N | | T | | A | | A | | C | | C | | C | | T | | A | | A | | C | +------+ 11 tuples (5m 23s) I think 5 minutes seem too much for this query. Primary key or an index on 'site' don't work or at least aren't recognized in query execution. I think MonetDB would benefit when knowing that the data is ordered by site. Wouldn't it? Because I know that the data is ordered by 'site', how can I tell MonetDB about this fact to speed up the query? Any suggestions? Regards, Sebastian

2 6

Monetdb performance improvement with more RAM
by Anshuman Kumar 08 Aug '13

08 Aug '13

Hello All, We are evaluating monetdb for use as a tick database. We are storing all the data into a single table. Our table has a column for timestamp and a few other fields. We wish to run queries against it, the most useful of which is to return all entries within a given timestamp range. This query takes a long time to execute for even a 15 minute interval (about 40 to 60 seconds). We have noted that the system RAM usage goes up to the maximum (almost 8GB) while we run such queries on monetdb. We wanted to understand whether installing more RAM (say, we upgrade to 16 GB) will result in a significant performance improvement for such queries on monetdb. Also, if this is the case, we would like to know, how much RAM is monetdb optimised for? The following is the specification of our server: Intel Xeon X3430 2.4Ghz 64Bit 8Gb Ram 500Gb HDD - Raid0 Running on Windows Server 2008 R2. We are using the 20130212 build of monetdb. Thank you, Anshuman

2 1

Local, searchable mail archive
by Brandon Jackson 08 Aug '13

08 Aug '13

Would anyone be able to recommend an approach or software on either Windows or Linux that would enable me to replicate the MonetDB mail archives to my local machine to make it easer to search for answers and study topics before posting questions asked before. Even if I could get an export of the whole thing in some format that be consumed by a mail client. I did read that mailman is capable of such archives, but it seems to require a real administrative user and password to create the archive. I wouldn't expect a frequent completely up to date copy of the archive, just a backup that is closer to now and stays reasonably up to date (few months) Thanks for any suggestions. Brandon Sent from my iPhone

1 0

Re:monetdb lost column info when queryring
by integrity 08 Aug '13

08 Aug '13

sql>select * from kbms_logserver_msg t where command = 'CmdWxInfo' and logdate = '2013-08-06' and functionret <> '0' and servertime > '1375757230' limit 10; +------------+-----------+--------------+-------------+----------+------------+---------------+------------+----------------+------------+------------+ | logdate | command | commandtype | functionret | costtime | servertime | serverip | serverport | clientip | clientport | clienttype | +============+===========+==============+=============+==========+============+===============+============+================+============+============+ | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3032 | 1375757231 | 10.187.143.80 | 7000 | 10.153.130.228 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3032 | 1375757244 | 10.187.143.80 | 7000 | 10.153.130.228 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3030 | 1375757257 | 10.187.143.80 | 7000 | 10.153.130.229 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3032 | 1375757962 | 10.187.143.80 | 7000 | 10.153.130.229 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3030 | 1375757970 | 10.187.143.80 | 7000 | 10.153.130.228 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 60 | 1375760693 | 10.187.143.80 | 7000 | 10.153.130.229 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 60 | 1375760693 | 10.187.143.80 | 7000 | 10.153.130.228 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 59 | 1375760693 | 10.187.143.80 | 7000 | 10.153.130.229 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 63 | 1375760872 | 10.187.143.80 | 7000 | 10.153.130.229 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 54 | 1375760872 | 10.187.143.80 | 7000 | 10.153.130.229 | | | +------------+-----------+--------------+-------------+----------+------------+---------------+------------+----------------+------------+------------+ 10 tuples (55.709ms) sql>select * from kbms_logserver_msg t where command = 'CmdWxInfo' and logdate = '2013-08-06' and functionret <> '0' and servertime = '1375757231' limit 10; +------------+-----------+--------------+-------------+----------+------------+---------------+------------+----------------+------------+------------+ | logdate | command | commandtype | functionret | costtime | servertime | serverip | serverport | clientip | clientport | clienttype | +============+===========+==============+=============+==========+============+===============+============+================+============+============+ | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3032 | 1375757231 | 10.187.143.80 | 7000 | 10.153.130.228 | 44328 | | +------------+-----------+--------------+-------------+----------+------------+---------------+------------+----------------+------------+------------+ 1 tuple (52.761ms) in the first query,when servertime > '1375757230' limit 10;clientport are all NULL. while in the second query, servertime = '1375757231' ,clientport appeard again. it proves that the clientport exist but just didn't appear in the query. Thanks, Meng

4 5

Determining Storage Consumption
by Sebastian Dorok 08 Aug '13

08 Aug '13

Hi all, I'm trying to figure out how much storage MonetDB (Feb2013-SP2) uses when populated with my data on a fresh database. Here is my table definition: create table genomics.grch36 (site bigint, base char(1)); I'm populating the table via COPY INTO. The source file size is 40GB and contains more than 3 billion rows. Following the suggestions on http://www.monetdb.org/Documentation/Userguide/diskspace, I get following result for the SQL query: sql>select * from storage() where "table" = 'grch36'; +----------+--------+--------+--------+----------+------------+-----------+-------------+----------+---------+--------+ | schema | table | column | type | location | count | typewidth | columnsize | heapsize | indices | sorted | +==========+========+========+========+==========+============+===========+=============+==========+=========+========+ | genomics | grch36 | site | bigint | 12/1252 | 3137459420 | 8 | 25099675360 | 0 | 0 | false | | genomics | grch36 | base | char | 12/1247 | 3137459420 | 1 | 6274918840 | 131072 | 0 | false | +----------+--------+--------+--------+----------+------------+-----------+-------------+----------+---------+--------+ I assume that 'columnsize' refers to the storage demand. I'm I right? Based on the numbers given, I assume a storage need of round about 31 GB (without any compression). Calling 'du -h' on my database directory confirms this assumption. 4,0K ./bat/14 44K ./bat/10 108K ./bat/07 476K ./bat/01 30G ./bat/12 256K ./bat/02 4,0K ./bat/03 4,0K ./bat/HC 4,0K ./bat/16 4,0K ./bat/17 12K ./bat/13 4,0K ./bat/LEFTOVERS 28K ./bat/BACKUP 4,0K ./bat/15 4,0K ./bat/11 12K ./bat/04 30G ./bat 8,0K ./sql_logs/sql 12K ./sql_logs 4,0K ./box 30G . But why is the 'columnsize' of the 'base' column something about 6GB whereas 3 billion times 1 Byte (for a 1 char) is 3GB? For the 'site' column this calculation works! Another question that comes into my mind: The number of distinct values in the 'base' column is 5. Thus, assuming a ideal dictionary encoding 3 Bit would be sufficient to encode all values. Since MonetDB uses dictionary compression, can it utilize this fact? Regards, Sebastian

1 0

SQL - syntax error in DELETE ALL FROM statement
by Radovan Bičiště 08 Aug '13

08 Aug '13

Hi, It is me again. :( I'm trying to "truncate" according to http://www.monetdb.org/Documentation/SQLreference/Updates using DELETE ALL FROM statement. But what I get is syntax error. :( ceos@lux-analytics:~$ mclient -h localhost -d zkdev1 -u drmaxtst -p 50000 password: Welcome to mclient, the MonetDB/SQL interactive terminal (Feb2013-SP3) Database: MonetDB v11.15.11 (Feb2013-SP3), 'mapi:monetdb://lux-analytics:50000/zkdev1' Type \q to quit, \? for a list of available commands auto commit mode: on sql>delete all from crmkarta; syntax error, unexpected ALL, expecting FROM in: "delete all" sql> Thank you for any hints. Best regards, Radovan -- __________________________ Radovan Bičiště

3 5

rpm for monetdb
by Sanjoy Bose 07 Aug '13

07 Aug '13

Hi, I am new to monetdb and considering using it for our Analytics need. Is there any rpm available for Centos 5 / Centos 6 RedHat 5 / Redhat 6 for its latest release or do I have to build from source ? Thanks, Sanjoy

4 6

SQL - temporary table data inside stored procedure
by Radovan Bičiště 07 Aug '13

07 Aug '13

Hello, I must be doing something horribly wrong. I have a stored procedure where I have temporary table declared, something like: DECLARE TABLE t1tmp (id bigint); Then in the same procedure I do INSERT INTO t1tmp SELECT .....; When I do Count(*) from temporary table in the procedure it returns 0. Fun starts after procedure called. When call is done temporary table is full of data as expected. But it seems the data is nowhere to be found when I look for them inside the procedure. Am I missing something again? Thanks for any hints, Radovan TEST CASE: CREATE PROCEDURE test1() BEGIN DECLARE TABLE t1tmp (bigint); INSERT INTO t1tmp SELECT id FROM t1; DECLARE v_cont BIGINT; -- returns 0 SELECT COUNT(*) INTO v_count FROM t1tmp; END; CALL test1; -- actually returns the data SELECT * FROM t1tmp; -- __________________________ Radovan Bičiště ceos data s.r.o. třída SNP 402/48 500 03 Hradec Králové Czech Republic

2 2

Data Vault:SyntaxException
by integrity 06 Aug '13

06 Aug '13

hi all, What's the correct way to use FIT DATA VAULT in mal? I got the following errors and still havent try out the right syntax ...... [svomhpc@localhost /home/data2]$>/home/data1/opt/MonetDB.fits/bin/mclient -lmal -dmydb Welcome to mclient, the MonetDB interactive terminal (unreleased) Type \q to quit, \? for a list of available commands mal>?fits.attach() pattern fits.attach(fname:str):void address FITSattach; comment Open a FITS file and return catalog of the table HDUs mal>call fits.attach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = call fits.attach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); ERROR = !SyntaxException:parseError:call fits.attach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); !SyntaxException:parseError: ^';' expected mal> mal> mal>call fitsattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = call fitsattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); ERROR = !SyntaxException:parseError:call fitsattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); !SyntaxException:parseError: ^';' expected mal> mal>call FITSattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = call FITSattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); ERROR = !SyntaxException:parseError:call FITSattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); !SyntaxException:parseError: ^';' expected Thanks! Meng

4 15

Re: How to use Data Vault with FITS files?
by integrity 05 Aug '13

05 Aug '13

Hi Ying, The thesis doesn't have instructions on how to use it....or it doesn't describe it very clearly maybe. Bests, Meng ------------------ Original ------------------ From: "Ying Zhang"<Y.Zhang(a)cwi.nl>; Date: Mon, Aug 5, 2013 04:18 PM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Subject: Re: How to use Data Vault with FITS files? Hai Meng, I already sent you the master thesis on FITS data vault. Didn't you receive it? Regards, Jennie On Mon, Aug 05, 2013 at 03:52:50PM +0800, integrity wrote: > Hi all, > > Could you give a tutorial of Data vault with FITS files, i searched the > moentdb.org and google, but didn't find the step-by-step tutorials. > > Thanks! > Meng > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

2 1

Re: How to use Data Vault with FITS files?
by integrity 05 Aug '13

05 Aug '13

Thanks very much Stefan! Does ' configure MonetDB with "--enable-fits" ' mean I have to re-install MonetDB? If I re-install it, can i still use my current db-farm? I have loaded a large catalog into it for a long time... Regards, Meng ------------------ Original ------------------ From: "Stefan Manegold"<Stefan.Manegold(a)cwi.nl>; Date: Mon, Aug 5, 2013 04:13 PM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Subject: Re: How to use Data Vault with FITS files? Hi Meng, to get the MonetDB FITS support, you need to have the cfitsio-dev[el] package installed and you need to configure MonetDB with "--enable-fits". However, while the respective code is more than 2 years old, I just realize that there is no (user-)documentation for it, and so far, support for fits files is only available in MAL, but not (yet?) in SQL. Since the original author of that code is no longer with out group, I currently cannot estimate, when we will have the resources to make this experimental code user-/production-ready. If you are familiar with MAL, please have a look at $ mclient -lmal Welcome to mclient, the MonetDB interactive terminal (unreleased) Type \q to quit, \? for a list of available commands mal>?fits.*() pattern fits.attach(fname:str):void address FITSattach; comment Open a FITS file and return catalog of the table HDUs pattern fits.export(tablename:str):void address FITSexportTable; comment Export a table to a FITS file command fits.fitstest(filename:str):int address FITStest; comment Returns the type of first extension in the FITS file filename pattern fits.load(tablename:str):void address FITSloadTable; comment Load a FITS table from an attached file pattern fits.listdirpattern(dirname:str,pattern:str):void address FITSdirpat; comment Attach all FITS file in the directory, giving a pattern pattern fits.listdir(dirname:str):void address FITSdir; comment Attach all FITS files in the directory mal> And if you're familiar with the MonetDB code base, you find the respective code in sql/backends/monet5/vaults/*fits* Best, Stefan ----- Original Message ----- > Hi all, > > > Could you give a tutorial of Data vault with FITS files, i searched the > moentdb.org and google, but didn't find the step-by-step tutorials. > > > Thanks! > Meng > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

2 1

How to use Data Vault with FITS files?
by integrity 05 Aug '13

05 Aug '13

Hi all, Could you give a tutorial of Data vault with FITS files, i searched the moentdb.org and google, but didn't find the step-by-step tutorials. Thanks! Meng

4 3

How to use Data Vault with FITS files?
by integrity 04 Aug '13

04 Aug '13

Hi all, Could you give a tutorial of Data vault with FITS files, i searched the moentdb.org and google, but didn't find the step-by-step tutorials. Thanks! Meng

1 0

Scale Factor 2000 TPCH Database?
by Stephen P. Morgan 02 Aug '13

02 Aug '13

Has anyone successfully created a scale factor 2000 TPCH database with MonetDB? We're having some problems trying to do so. We don't have enough disk space to hold both the data (created by dbgen) and the MonetDB database, so we have to create the database in two steps. (Create the first half of the data as dbgen table files, load it, delete the dbgen files, create the second half of the data as dbgen table files, load it, then delete those files.) The problem shows up when loading the second half of the data: the mserver crashes and won't restart. This is what the dbfarm/merovingian.log says: 2013-08-02 18:42:19 MSG merovingian[39857]: database 'q1_s2000_n2' (39920) has crashed (dumped core) 2013-08-02 18:42:19 ERR merovingian[39857]: client error: database 'q1_s2000_n2' has crashed after starting, manual intervention needed, check monetdbd's logfile for details 2013-08-02 18:42:19 MSG merovingian[39857]: database 'q1_s2000_n2' has crashed after start on 2013-08-02 18:42:19, attempting restart, up min/avg/max: 0s/0s/0s, crash average: 1.00 0.50 0.17 (5-0=5) Other databases on the same system are still running. We're running Feb13 SP1 with minor local modifications that are not coming into play at this point. We're running on a large Linux server. Is there anything we can look at to help diagnose the problem? Thanks, Steve Morgan

3 3

Copy From SQL server
by soumik ghosh 02 Aug '13

02 Aug '13

Hello All, Could someone point me to an example of copying a table from SQL Server to MonetDB over a JDBC or ODBC bridge? Is this possible at all? Thanks Soumik

5 7

Background processes in MonetDB
by DrRock 01 Aug '13

01 Aug '13

Dear fellows, we installed MonetDB a few weeks ago on a Linux Machine (8 cores, 32GB RAM) and our overall experience is very good. So far, we are testing Monet with different ammounts of data, roughly peaking at 250GB. After loading plenty of data into the database or after interrupting a query, Monet feels somehow slow for a certain period of time. Another observation we made is that diskspace remains occupied after the deletion of all tables in the database for a certain time. Further, indices can be created, although this seems not to be really essential as MonetDB provides some kind of own intelligence, optimizing queries/ tables by themself. All of mentioned observations point at some kind of background processes and optimizing routines. Is there a way to modify the behaviour of MonetDB (Such as providing diskspace instantly after deletion of tables)? Is it possible to switch these processes off? Best regards, DrRock

1 0

user defined directory
by Adnan Agbaria 31 Jul '13

31 Jul '13

I'd like to add a new directory named CAP at MonetDBRoot/sql/backends/monet5 The new directory contains cap.c, cap.h, cap.mal, 80_cap.mal, 80_cap.sql, and makefile.ag I modified MonetDBRoot/sql/backends/monet5/Makefile.ag at: SUBDIRS = NOT_WIN32?vaults UDF CAP LSST ENABLE_DATACELL?datacell However, when I run MonetDBRoot/bootstrap, i got the following error message: ... /home/adnan/MonetDB-11.15.11/sql/backends/monet5 /home/adnan/MonetDB-11.15.11/sql/backends/monet5/vaults /home/adnan/MonetDB-11.15.11/sql/backends/monet5/UDF /home/adnan/MonetDB-11.15.11/sql/backends/monet5/CAP Traceback (most recent call last): File "buildtools/autogen/autogen.py", line 157, in <module> (InstallList, DocList, OutList) = main(topdir, topdir, automake, []) File "buildtools/autogen/autogen.py", line 147, in main main(d, topdir, automake, incdirsmap, conditional + cond) File "buildtools/autogen/autogen.py", line 147, in main main(d, topdir, automake, incdirsmap, conditional + cond) File "buildtools/autogen/autogen.py", line 147, in main main(d, topdir, automake, incdirsmap, conditional + cond) File "buildtools/autogen/autogen.py", line 147, in main main(d, topdir, automake, incdirsmap, conditional + cond) File "buildtools/autogen/autogen.py", line 132, in main read_makefile(p, cwd) File "buildtools/autogen/autogen.py", line 96, in read_makefile p.parse(token, lineno, line) File "buildtools/autogen/autogen.py", line 71, in parse v = self.stack[last] IndexError: list index out of range Any idea please?

3 2

Re: Monetdb copy binary time varys very much!
by integrity 30 Jul '13

30 Jul '13

Hi Stefan, Just as you said ,we issue the next copy into as soon as the previous ended. What would be different if i mimic the real-world scenario and respect these gaps between each two consecutive copy into's? Thank! Meng ------------------ Original ------------------ From: "Stefan Manegold"<Stefan.Manegold(a)cwi.nl>; Date: Mon, Jul 29, 2013 02:45 PM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! Hi Meng, My analysis was mainly an (educated) guess of what (most probably) happens. To be sure, you need to profile your system in detail, e.g., monitor CPU and I/O activities. Having said that, with less RAM, you might force the system to write the loaded data to disk instantly with each copy into, making each copy into slower than merely loading the data into disk, but the worst case might become better since each time less data has to we written than with more RAM. Again, this is my guess what happening; the behaviour you observe might be caused by something else; only a detailed profiling analysis can tell. Also, if you eventually want to query your 300+ GB (or even more?) efficiently, you might want to have a suitable system, in particular sufficient RAM. (Would you mind sharing the hardware characteristics of your machine?). Moreover, what was the time gap between two consecutive copy into's in your experiment, i.e., did you issue the next copy into as soon as the previous ended? Does this mimic the your "real-world" scenario realistically? Or would there be a time gap between to copy into's in reality? I recall you mentioned some 15 seconds? If so, you should rerun your experiment respecting these gaps between each two consecutive copy into's. Best, Stefan ----- Original Message ----- > Hi Stefan, > > > Due to all the data in RAM having to be written to disk when it's full, if > the RAM grows larger and i don't change hard disk, the data volume become > larger, will it make the COPY BINARY INTO much slower compared to the small > RAM? > > > Contrarily, if the RAM get smaller, will it make the COPY BINARY INTO > quicker? > > > I just want to lower the peak value of COPY, the 10+ seconds every about a > hundred times. > > > For the most time COPY BINARY INTO took only less than 1 second, is it > because they are written to RAM not disk when the RAM is not full? if so, > can i write to disk more frequently before the disk is full so i could lower > down the peak value of COPY. > > > > > Regrads, > Meng > > > ------------------ Original ------------------ > From: "Stefan Manegold"<Stefan.Manegold(a)cwi.nl>; > Date: Fri, Jul 26, 2013 11:10 PM > To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; > > Subject: Re: Monetdb copy binary time varys very much! > > > > Hi, > > I'm not sure whether I understand correct what you are doing. > > If you repeat the test 1000 times, does that mean that (1) 10000 times you > re-create (or empty) the table and thus always copy into an empty table, or > (2) 10000 times you copy into the same (growing) table, i.e., resulting in a > table of 10,000 times 200,000 rows, i.e., 2,000,000,000 rows, i.e., ~16 GB > per column, i.e., ~336 GB in total? > > (Only) in case (1) the binary files to be imported are simply moved at zero > costs. > In case (2), only the first copy into (into the empty table) can simply move > the files at zero costs; all subsequent copy into (into a no longe empty > table) must copy the files (and delete them afterwards to mimic the same > behavior as the initial copy into), which is of cause not "for free". > > Also, as Martin explained, unless your machine has (significantly) more RAM > than the ~336 GB of data you copy, the data needs to be written to disk in > between, making some copy into's "slower" than others. There's not much to > do about that other than (a) getting more RAM, or (b) improving I/O > bandwidth by using either a high performance RAID or SSDs. > > Stefan > > . > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

2 1

SQL: AVG function works in SQL but it does not in procedure
by Radovan Bičiště 29 Jul '13

29 Jul '13

Hello, We are unable to solve the following problem. This SQL runs fine as standalone: INSERT INTO crmkartastitek ( cislo_karty, stitek_id, stitek_nazev, stitek_upresneni, stitek_hodnota_decimal ) select cislo_karty,170,'KEO','',round(avg(obrat),2) from (select ckp.cislo_karty, ckp.id_expedice, sum(ckp.pcena_sd) as obrat from crmkartapohyby ckp join crmkartaaktivita ck on ckp.cislo_karty = ck.cislo_karty where ckp.pohyb in ('VR', 'VV') and ck.aktivita between 1 and 3 -- omezení na LM a L3M group by ckp.cislo_karty, ckp.id_expedice ) as exp_pripady group by cislo_karty; But when wrapped in procedure raises an error as follows: CREATE PROCEDURE crm_segm_keo_rl(p_interval INT, p_datum timestamp) BEGIN -- hlavička procedury DECLARE v_proc VARCHAR(250); SET v_proc = 'crm_segm_kpn_rl'; -- nastav id pravidla DECLARE v_stitek_id int; DECLARE v_stitek_nazev,v_stitek_upresneni VARCHAR(200); -- nastav štítek SET v_stitek_id = 170; SELECT nazev,upresneni INTO v_stitek_nazev,v_stitek_upresneni FROM crmregstitek WHERE stitek_id = v_stitek_id; -- proveď výpočet pouze pokud spouštím měsíční přepočet IF p_interval = crm_segm_vrat_spusteni(v_stitek_id) THEN INSERT INTO crmkartastitek ( cislo_karty, stitek_id, stitek_nazev, stitek_upresneni, stitek_hodnota_decimal ) select cislo_karty,v_stitek_id,v_stitek_nazev,v_stitek_upresneni,round(avg(obrat),2) from (select ckp.cislo_karty, ckp.id_expedice, sum(ckp.pcena_sd) as obrat from crmkartapohyby ckp join crmkartaaktivita ck on ckp.cislo_karty = ck.cislo_karty where ckp.pohyb in ('VR', 'VV') and ck.aktivita between 1 and 3 -- omezení na LM a L3M group by ckp.cislo_karty, ckp.id_expedice ) as exp_pripady group by cislo_karty; END IF; END; Procedure is created fine. call crm_segm_keo_rl(3,timestamp '2013-07-01 00:00:00'); Causes: An error occurred when executing the SQL command: call crm_segm_keo_rl(3,timestamp '2013-07-01 00:00:00') TypeException:user.crm_segm_keo_rl[362]:'aggr.subavg' undefined in: _546:bat[:any,:dbl] := aggr.subavg(_543:bat[:oid,:dbl], _209:bat[:oid,:oid], r1_209:bat[:oid,:oid], _535:bit) [SQL State=22000] Next: TypeException:user.s4_24[5]:'user.crm_segm_keo_rl' undefined in: _9:void := user.crm_segm_keo_rl(_5:int, _7:timestamp) [SQL State=22000] Next: program contains errors [SQL State=39000] Are we missing something ? Thank you, Radovan -- Radovan Biciste Hradec Kralove Czech Republic

1 0

Re: Monetdb copy binary time varys very much!
by integrity 29 Jul '13

29 Jul '13

Yes, I use the server only myself only to run the MonetDB test and notify other users not to use the server in advance. and I reboot the server before tests . ------------------ Original ------------------ From: "Ying Zhang"<Y.Zhang(a)cwi.nl>; Date: Mon, Jul 29, 2013 04:28 PM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! On Jul 29, 2013, at 05:57, integrity wrote: > Hi Stefan, > > Due to all the data in RAM having to be written to disk when it's full, if the RAM grows larger and i don't change hard disk, the data volume become larger, will it make the COPY BINARY INTO much slower compared to the small RAM? > > Contrarily, if the RAM get smaller, will it make the COPY BINARY INTO quicker? > > I just want to lower the peak value of COPY, the 10+ seconds every about a hundred times. Hai Meng, Have you made sure that there are no processes running other than the ones strictly necessary? Have you closed all user applications, such as mail client, browser, and chat clients? Regards, Jennie > > For the most time COPY BINARY INTO took only less than 1 second, is it because they are written to RAM not disk when the RAM is not full? if so, can i write to disk more frequently before the disk is full so i could lower down the peak value of COPY. > > > Regrads, > Meng > > ------------------ Original ------------------ > From: "Stefan Manegold"<Stefan.Manegold(a)cwi.nl>; > Date: Fri, Jul 26, 2013 11:10 PM > To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; > Subject: Re: Monetdb copy binary time varys very much! > > Hi, > > I'm not sure whether I understand correct what you are doing. > > If you repeat the test 1000 times, does that mean that (1) 10000 times you > re-create (or empty) the table and thus always copy into an empty table, or > (2) 10000 times you copy into the same (growing) table, i.e., resulting in a > table of 10,000 times 200,000 rows, i.e., 2,000,000,000 rows, i.e., ~16 GB > per column, i.e., ~336 GB in total? > > (Only) in case (1) the binary files to be imported are simply moved at zero > costs. > In case (2), only the first copy into (into the empty table) can simply move > the files at zero costs; all subsequent copy into (into a no longe empty > table) must copy the files (and delete them afterwards to mimic the same > behavior as the initial copy into), which is of cause not "for free". > > Also, as Martin explained, unless your machine has (significantly) more RAM > than the ~336 GB of data you copy, the data needs to be written to disk in > between, making some copy into's "slower" than others. There's not much to > do about that other than (a) getting more RAM, or (b) improving I/O > bandwidth by using either a high performance RAID or SSDs. > > Stefan > > . > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

1 0

Re: Monetdb copy binary time varys very much!
by integrity 29 Jul '13

29 Jul '13

Hi Stefan, Due to all the data in RAM having to be written to disk when it's full, if the RAM grows larger and i don't change hard disk, the data volume become larger, will it make the COPY BINARY INTO much slower compared to the small RAM? Contrarily, if the RAM get smaller, will it make the COPY BINARY INTO quicker? I just want to lower the peak value of COPY, the 10+ seconds every about a hundred times. For the most time COPY BINARY INTO took only less than 1 second, is it because they are written to RAM not disk when the RAM is not full? if so, can i write to disk more frequently before the disk is full so i could lower down the peak value of COPY. Regrads, Meng ------------------ Original ------------------ From: "Stefan Manegold"<Stefan.Manegold(a)cwi.nl>; Date: Fri, Jul 26, 2013 11:10 PM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! Hi, I'm not sure whether I understand correct what you are doing. If you repeat the test 1000 times, does that mean that (1) 10000 times you re-create (or empty) the table and thus always copy into an empty table, or (2) 10000 times you copy into the same (growing) table, i.e., resulting in a table of 10,000 times 200,000 rows, i.e., 2,000,000,000 rows, i.e., ~16 GB per column, i.e., ~336 GB in total? (Only) in case (1) the binary files to be imported are simply moved at zero costs. In case (2), only the first copy into (into the empty table) can simply move the files at zero costs; all subsequent copy into (into a no longe empty table) must copy the files (and delete them afterwards to mimic the same behavior as the initial copy into), which is of cause not "for free". Also, as Martin explained, unless your machine has (significantly) more RAM than the ~336 GB of data you copy, the data needs to be written to disk in between, making some copy into's "slower" than others. There's not much to do about that other than (a) getting more RAM, or (b) improving I/O bandwidth by using either a high performance RAID or SSDs. Stefan .

3 2

Deleting millions of rows every night
by Dennis Pallett 29 Jul '13

29 Jul '13

Hello all, We are going to be using MonetDB as a data mart between our data warehouse and our front-end applications and are planning to insert and delete several millions of rows from MonetDB every night as part of our data update cycle. We've noticed that MonetDB doesn't support the TRUNCATE statement so we're deleting all the data with DELETE FROM table. But is this the best way to delete millions of rows every night? Won't performance or free disk space degrade over time? Best regards, Dennis Pallett

2 1

MonetDB and SQLWorkbench
by Satoshi Nagayasu 29 Jul '13

29 Jul '13

Hi, I have hacked SQLWorkbench to work with MonetDB. SQLWorkbench is a detabase-independent GUI query tool, written in Java. http://www.sql-workbench.net/ To support MonetDB in SQLWorkbench, a new entry needs to be added in DatabaseTemplates.xml, which is contained in sqlworkbench.jar. This modifed files are available: http://www.uptime.jp/downloads/monetdb/SQLWorkbench/ It seems working well so far, and this patch is supposed to be contributed to the SQLWorkbench upstream. If you are looking for a nice GUI tool to issue queries to MonetDB, take a look. Regards, -- Satoshi Nagayasu <snaga(a)uptime.jp> Uptime Technologies, LLC. http://www.uptime.jp

3 3

Re: Monetdb copy binary time varys very much!
by integrity 28 Jul '13

28 Jul '13

Hi Ying, As you mentioned the 2nd case, I just interchanged create 21 files and excute COPY INTO. The plot times include only the execution time of COPY INTO. In 15 seconds, the data volumn i have to process is just similar to storing 200000 rows,21 columns in this simulation program. Thanks! Meng ------------------ Original ------------------ From: "Ying Zhang"<Y.Zhang(a)cwi.nl>; Date: Sun, Jul 28, 2013 00:47 AM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! Hai Meng, Since you repeat the COPY INTO statement 10000 times, I wonder how exactly did you it and measure the time. For instance, did you first create _all_ necessary files (i.e., 10000 x 21 files) and execute 10000 COPY INTO consecutively? Or did you interchange create 21 files and execute COPY INTO? In the second case, are the times in your plots only the execution time of COPY INTO, or do they also include the time to create the files? Again, in the second case, have you also measured the time of creating the files? You have ever mentioned that the total available execution time in your application is 15 seconds. How many data do you need to process within this time? Regards, Jennie On Jul 26, 2013, at 10:40, integrity wrote: > Hi dear all, > > I did a test of insert 2,000,000,000 rows into MonetDB with "COPY BINARY INTO FROM binary_file", > in this test > 1. i generate 21 files, each file represents a table column and has 200000 rows. > I created them with rand() number and use fwrite() binary write method. > > the table creation sql command is : > create table tmatch(id bigint,a float,b float,c float, d float, e float, > f float,g float,h float, i float,j float, > k float,l float,m float, n float,o float, > p float,q float,r float, s float,t float); > the table has 21 columns,each column has 8 bytes, so each column file is c1=200000*21*8 Byte= 268800000 Byte=3.2MB > 2. I use "COPY BINARY INTO FROM above_binary" to load each binary file into tmatch. > The test was run 10000 times repeatedly. > > the average time of 10000 times is only 1.0635589558955727, > but when at 9043th time, it cost 227m36.136 ,some times later the time value increase to a large number, is it because of flush data from cache into database after the cache is full? > The problem is since we have to control the total process time within 15 seconds , I am wondering if you can help me reduce the maximum time to a lower point? > > > Thanks very much! > Meng > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

2 1

Monetdb copy binary time varys very much!
by integrity 27 Jul '13

27 Jul '13

Hi dear all, I did a test of insert 2,000,000,000 rows into MonetDB with "COPY BINARY INTO FROM binary_file", in this test 1. i generate 21 files, each file represents a table column and has 200000 rows. I created them with rand() number and use fwrite() binary write method. the table creation sql command is : create table tmatch(id bigint,a float,b float,c float, d float, e float, f float,g float,h float, i float,j float, k float,l float,m float, n float,o float, p float,q float,r float, s float,t float); the table has 21 columns,each column has 8 bytes, so each column file is c1=200000*21*8 Byte= 268800000 Byte=3.2MB 2. I use "COPY BINARY INTO FROM above_binary" to load each binary file into tmatch. The test was run 10000 times repeatedly. the average time of 10000 times is only 1.0635589558955727, but when at 9043th time, it cost 227m36.136 ,some times later the time value increase to a large number, is it because of flush data from cache into database after the cache is full? The problem is since we have to control the total process time within 15 seconds , I am wondering if you can help me reduce the maximum time to a lower point? Thanks very much! Meng

4 3

Re: Monetdb copy binary time varys very much!
by integrity 26 Jul '13

26 Jul '13

you mean change hardware?Is there any software resolution? ------------------ Original ------------------ From: "Martin Kersten"<martin(a)monetdb.org>; Date: Fri, Jul 26, 2013 10:52 PM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! see http://en.wikipedia.org/wiki/Solid-state_drive On 7/26/13 4:48 PM, Angelasweet wrote: Yes, time is the biggest issue to our project, I cannot find SSD, could you please explain the SSD concept? Thanks! _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

1 0

Re: Monetdb copy binary time varys very much!
by integrity 26 Jul '13

26 Jul '13

Is it possible to set the cache to low down the maximum copy time? ------------------ Original ------------------ From: "Martin Kersten"<Martin.Kersten(a)cwi.nl>; Date: Fri, Jul 26, 2013 10:13 PM To: "users-list"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! Oops, correct, for binary copies it does not recognize it On 7/26/13 4:11 PM, Angelasweet wrote: > Hi Martin, > I follow you advise, and there is error as follows: > > ERROR = !COPY INTO: cannot pass number of records when using binary COPY > INTO in: "copy 200000 records binary into tmatch > from('/data/writeBinaryWM/id','/data/writeBinaryWM/0','/data/writeBinaryWM/1','/data/writeBinaryWM/2','/data/writeBinaryWM/3','/data/writeBinaryWM/4', > '/data/writeBinaryWM/5','/data/writeBinaryWM/6','/data/writeBinaryWM/7','/data/writeBinaryWM/8','/data/writeBinaryWM/9','/data/writeBinaryWM/10', '/data/writeBinaryWM/11', '/data/writeBinaryWM/12', '/data/writeBinaryWM/13', '/data/writeBinaryWM/14', '/data/writeBinaryWM/15', '/data/writeBinaryWM/16', '/data/writeBinaryWM/17', '/data/writeBinaryWM/18', '/data/writeBinaryWM/19' )" > !current transaction is aborted (please ROLLBACK) > > > 2013/7/26 Martin Kersten <Martin.Kersten(a)cwi.nl > <mailto:Martin.Kersten@cwi.nl>> > > This runs shows expected flushing. > > You might get somewhat better performance when you follow > the advice in > http://www.monetdb.org/__Documentation/Cookbooks/__SQLrecipes/LoadingBulkDa… > <http://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/LoadingBulkData> > e.g. use COPY 44000000 RECORDS .... > > The COPY INTO forms mentioned so far share one important drawback. > When a lot of data is to be inserted, the server doesn't know how > much memory to allocate for the tables, and so will likely allocate > too little. This means that during the insertion process, the server > has to grow the allocated memory area. This can be an expensive > operation. Therefore, it is better to give the server a count of > how many records are to be inserted: > > COPY n RECORDS INTO table FROM 'file'; > > Here n should be a number that is at least as large as the actual > number of records to be inserted. If the file contains more than n > records, only n will be inserted, if the file contains fewer, all > values will be inserted. Giving a higher number is especially > useful if multiple COPY INTO queries are to be done on the same > table. The first COPY INTO, when the table is still empty, should > be the total count of to-be-inserted values so that the server will > allocate enough memory when the tables are first created (they are > only really created once data is inserted). > On 7/26/13 10:58 AM, integrity wrote: > > > > ------------------ Original ------------------ > *From: * "Martin Kersten"<Martin.Kersten(a)cwi.nl > <mailto:Martin.Kersten@cwi.nl>__>; > *Date: * Fri, Jul 26, 2013 04:46 PM > *To: * "users-list"<users-list(a)__monetdb.org > <mailto:users-list@monetdb.org>>; > *Subject: * Re: Monetdb copy binary time varys very much! > > > Hi, > You should plot the times of each of your runs to see the trend. > This single run may be an outlier, which could come from anything > in your system environment. Even a seemingly harmless concurrent > program using significant memory could compete with MonetDB. > > And indeed, at some point you will see disk IO. > > > regards, Martin > > > > > _________________________________________________ > users-list mailing list > users-list(a)monetdb.org <mailto:users-list@monetdb.org> > http://mail.monetdb.org/__mailman/listinfo/users-list > <http://mail.monetdb.org/mailman/listinfo/users-list> > > _________________________________________________ > users-list mailing list > users-list(a)monetdb.org <mailto:users-list@monetdb.org> > http://mail.monetdb.org/__mailman/listinfo/users-list > <http://mail.monetdb.org/mailman/listinfo/users-list> > > > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list > _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

3 3

Re: Monetdb copy binary time varys very much!
by integrity 26 Jul '13

26 Jul '13

------------------ Original ------------------ From: "Martin Kersten"<Martin.Kersten(a)cwi.nl>; Date: Fri, Jul 26, 2013 04:46 PM To: "users-list"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! Hi, You should plot the times of each of your runs to see the trend. This single run may be an outlier, which could come from anything in your system environment. Even a seemingly harmless concurrent program using significant memory could compete with MonetDB. And indeed, at some point you will see disk IO. regards, Martin

3 3

remote.exec()
by Stephen P. Morgan 26 Jul '13

26 Jul '13

I have a question about remote.exec(). Suppose multiple (say N) remote machines call remote.exec() to start a function on a given local machine. What happens if N is greater than the number of threads supported by the local machine? Are the remote.execs queued until there is a thread available to execute them, or are they simply not started? If the letter, how can one tell whether a remote.exec was run? I have another question abut remote.get() transferring a BAT from one machine to another. Is there a limit on the size of a BAT that can be transferred by a single call to remote.get()? If the BAT is very large, is it paged through disk, or is it alsways transferred directly from memory to memory? Thanks, Steve Morgan

4 5

Re: Monetdb copy binary time varys very much!
by integrity 26 Jul '13

26 Jul '13

1 0

Re: Monetdb copy binary time varys very much!
by integrity 26 Jul '13

26 Jul '13

1 0

Re: Monetdb copy binary time varys very much!
by integrity 26 Jul '13

26 Jul '13

------------------ Original ------------------ From: "Martin Kersten"<Martin.Kersten(a)cwi.nl>; Date: Fri, Jul 26, 2013 04:46 PM To: "users-list"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! Hi, You should plot the times of each of your runs to see the trend. This single run may be an outlier, which could come from anything in your system environment. Even a seemingly harmless concurrent program using significant memory could compete with MonetDB. And indeed, at some point you will see disk IO. regards, Martin

1 0

Re: Monetdb copy binary time varys very much!
by integrity 26 Jul '13

26 Jul '13

------------------ Original ------------------ From: "Martin Kersten"<Martin.Kersten(a)cwi.nl>; Date: Fri, Jul 26, 2013 04:46 PM To: "users-list"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! Hi, You should plot the times of each of your runs to see the trend. This single run may be an outlier, which could come from anything in your system environment. Even a seemingly harmless concurrent program using significant memory could compete with MonetDB. And indeed, at some point you will see disk IO. regards, Martin On 7/26/13 10:40 AM, integrity wrote: > Hi dear all, > > I did a test of insert 2,000,000,000 rows into MonetDB with "COPY BINARY > INTO FROM binary_file", > in this test > 1. i generate 21 files, each file represents a table column and has > 200000 rows. > I created them with rand() number and use fwrite() binary write method. > > the table creation sql command is : > *create table tmatch(id bigint,a float,b float,c float, d float, e float,* > *f float,g float,h float, i float,j float,* > *k float,l float,m float, n float,o float,* > *p float,q float,r float, s float,t float);* > the table has 21 columns,each column has 8 bytes, so each column file is > c1=200000*21*8 Byte= 268800000 Byte=3.2MB > 2. I use "COPY BINARY INTO FROM above_binary" to load each binary file > into tmatch. > The test was run 10000 times repeatedly. > > the average time of 10000 times is only 1.0635589558955727, > but when at 9043th time, it cost 227m36.136 ,some times later the time > value increase to a large number, is it because of flush data from > cache into database after the cache is full? > The problem is since we have to control the total process time within 15 > seconds , I am wondering if you can help me reduce the maximum time to a > lower point? > > > Thanks very much! > Meng > > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list > _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

1 0

ALTER in procedure
by Radovan Bičiště 23 Jul '13

23 Jul '13

Hello, This is my first post here. I tried to search the archive but did not find anything related. We build application and would like disable foreign keys in a procedure. Something like: CREATE PROCEDURE crmkartaaktivita_drop_cons() BEGIN -- hlavička procedury DECLARE v_proc VARCHAR(250); SET v_proc = 'crmkartaaktivita_drop_cons'; -- drop ALTER TABLE crmkartastitek DROP CONSTRAINT crmkartastitek_crmkartaaktivita_FK; ALTER TABLE crmkartastitekm DROP CONSTRAINT crmkartastitekm_crmkartaakitvita_FK; END; / I'm getting "Statement 'ALTER_TABLE' is not a valid flow control statement". Is there any way around please? Our algorithm should like this: 1. disable/drop foreign keys. 2. load data. 3. enable/create constraints. Thanks, Radovan -- Radovan Biciste Czech Republic

2 7

If MonetDB could use table partition ?
by integrity 22 Jul '13

22 Jul '13

I am using "COPY BINARY INTO ...FROM ..." to insert 1000 FITS files into a table and the table has 21 columns, every FITS file has 200000 records. After some FITS files, the loading becoming very slowly, so If MonetDB could use table partition to accelerating the copy? or if Data Vaults could be loading more quickly? Thanks! Meng

2 2

Re: Could i binary bulk load to a CERTAIN column?
by integrity 21 Jul '13

21 Jul '13

my table test4 is : create table test4(id bigint, a float, b float, c float); so i didnt selectively load to test4 but it crashed. my code is : #include <stdio.h> #include <stdlib.h> #include <time.h> #include <mapi.h> void itoa (int n,char s[]) { int i,j,sign,k; char tmp; if((sign=n)<0)//记录符号 n=-n;//使n成为正数 i=0; do{ s[i++]=n%10+'0';//取下一个数字 }while ((n/=10)>0);//删除该数字 if(sign<0) s[i++]='-'; s[i]='\0'; for(j=i-1,k=0;j>k;j--,k++) { tmp = s[k]; s[k] = s[j]; s[j] = tmp; } } int writetoFile() { unsigned int seed; int k,j,i=0; long al[200000]; double af[200000]; FILE *fp; char fname[5]; fp = fopen("id","wb"); //创建二进制文件 for(k=1;k<=200000;k++) { al[k] = rand(); } fwrite(al, sizeof(long), 200000,fp); //fwrite以二进制形式对文件操作 fclose(fp); for(j=1;j<=20;j++) { for(k=1;k<=200000;k++) { af[k] = (rand() /(double)(RAND_MAX)); } itoa(j,fname); fp=fopen(fname,"wb"); fwrite(af, sizeof(double), 200000,fp); fclose(fp); } } void die(Mapi dbh, MapiHdl hdl) { if (hdl != NULL) { mapi_explain_query(hdl, stderr); do { if (mapi_result_error(hdl) != NULL) mapi_explain_result(hdl, stderr); } while (mapi_next_result(hdl) == 1); mapi_close_handle(hdl); mapi_destroy(dbh); } else if (dbh != NULL) { mapi_explain(dbh, stderr); mapi_destroy(dbh); } else { fprintf(stderr, "command failed\n"); } exit(-1); } MapiHdl query(Mapi dbh, char *q) { MapiHdl ret = NULL; if ((ret = mapi_query(dbh, q)) == NULL || mapi_error(dbh) != MOK) die(dbh, ret); return(ret); } void update(Mapi dbh, char *q) { MapiHdl ret = query(dbh, q); if (mapi_close_handle(ret) != MOK) die(dbh, ret); } int filetoMonetDB() { Mapi dbh; MapiHdl hdl = NULL; dbh = mapi_connect("localhost",50000, "monetdb", "monetdb", "sql","mydb2" ); if(mapi_error(dbh)) die(dbh,hdl); char *sql = NULL; sql = (char *)malloc(100); memset(sql,0,100); //sprintf(sql, "copy binary into tmatch from('/home/data1/writeBinary/id','/home/data1/writeBinary/1','/home/data1/writeBinary/2','/home/data1/writeBinary/3','/home/data1/writeBinary/4')"); // '/home/data1/writeBinary/5','/home/data1/writeBinary/6','/home/data1/writeBinary/7','/home/data1/writeBinary/8','/home/data1/writeBinary/9','/home/data1/writeBinary/10', \ // '/home/data1/writeBinary/11', '/home/data1/writeBinary/12', '/home/data1/writeBinary/13', '/home/data1/writeBinary/14', '/home/data1/writeBinary/15', \ // '/home/data1/writeBinary/16', '/home/data1/writeBinary/17', '/home/data1/writeBinary/18', '/home/data1/writeBinary/19', '/home/data1/writeBinary/20' )"); // sprintf(sql,"insert into test2 values(2)"); sprintf(sql, "copy binary into test3 from('/home/data1/writeBinary/id','/home/data1/writeBinary/1')"); //sprintf(sql,"copy binary into test4 from('/home/data1/writeBinary/id','/home/data1/writeBinary/1','home/data1/writeBinary/2','home/data1/writeBinary/3')"); //sprintf(sql,"copy binary into test4(id) from('/home/data1/writeBinary/id')"); update(dbh, sql); mapi_destroy(dbh); free(sql); return 1; } int main() { struct timeval tpstart,tpend; double timeuse; gettimeofday(&tpstart,NULL); writetoFile(); gettimeofday(&tpend,NULL); timeuse=1000000*(tpend.tv_sec-tpstart.tv_sec)+tpend.tv_usec-tpstart.tv_usec;//unit:us timeuse /= 1000000;//unit:s printf("21 columns write to file used %f seconds.\n",timeuse) ; gettimeofday(&tpstart,NULL); filetoMonetDB(); gettimeofday(&tpend,NULL); timeuse=1000000*(tpend.tv_sec-tpstart.tv_sec)+tpend.tv_usec-tpstart.tv_usec;//unit:us timeuse /= 1000000;//unit:s printf("Files load to MonetDB used %f seconds.\n",timeuse) ; return 1; }

3 3

Could i binary bulk load to a CERTAIN column?
by integrity 21 Jul '13

21 Jul '13

Hi all, when i use : copy binary into test4 from('/home/data1/writeBinary/id','/home/data1/writeBinary/1','home/data1/writeBinary/2','home/data1/writeBinary/3'); it fails: 21 columns write to file used 0.195698 seconds. *** glibc detected *** ./fitscolumntodb: malloc(): memory corruption: 0x0000000001d743e0 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3efde7d982] /lib64/libc.so.6(__libc_malloc+0x63)[0x3efde7f313] /lib64/libc.so.6(__strdup+0x22)[0x3efde85c42] /home/data1/opt/MonetDB/lib/libmapi.so.6(+0xb24f)[0x7f77d4ca724f] /home/data1/opt/MonetDB/lib/libmapi.so.6(mapi_query+0xa0)[0x7f77d4cac410] ./fitscolumntodb[0x400fc5] ./fitscolumntodb[0x40101c] ./fitscolumntodb[0x401198] ./fitscolumntodb[0x40127d] /lib64/libc.so.6(__libc_start_main+0xf5)[0x3efde21a05] ./fitscolumntodb[0x400bc9] ======= Memory map: ======== 00400000-00402000 r-xp 00000000 08:01 18874373 /home/data1/writeBinary/fitscolumntodb 00601000-00602000 r--p 00001000 08:01 18874373 /home/data1/writeBinary/fitscolumntodb 00602000-00603000 rw-p 00002000 08:01 18874373 /home/data1/writeBinary/fitscolumntodb 01d74000-01d95000 rw-p 00000000 00:00 0 [heap] 3efd600000-3efd620000 r-xp 00000000 fd:01 700425 /usr/lib64/ld-2.16.so 3efd820000-3efd821000 r--p 00020000 fd:01 700425 /usr/lib64/ld-2.16.so 3efd821000-3efd822000 rw-p 00021000 fd:01 700425 /usr/lib64/ld-2.16.so 3efd822000-3efd823000 rw-p 00000000 00:00 0 3efde00000-3efdfad000 r-xp 00000000 fd:01 700426 /usr/lib64/libc-2.16.so 3efdfad000-3efe1ad000 ---p 001ad000 fd:01 700426 /usr/lib64/libc-2.16.so 3efe1ad000-3efe1b1000 r--p 001ad000 fd:01 700426 /usr/lib64/libc-2.16.so 3efe1b1000-3efe1b3000 rw-p 001b1000 fd:01 700426 /usr/lib64/libc-2.16.so 3efe1b3000-3efe1b8000 rw-p 00000000 00:00 0 3efe200000-3efe216000 r-xp 00000000 fd:01 700427 /usr/lib64/libpthread-2.16.so 3efe216000-3efe416000 ---p 00016000 fd:01 700427 /usr/lib64/libpthread-2.16.so 3efe416000-3efe417000 r--p 00016000 fd:01 700427 /usr/lib64/libpthread-2.16.so 3efe417000-3efe418000 rw-p 00017000 fd:01 700427 /usr/lib64/libpthread-2.16.so 3efe418000-3efe41c000 rw-p 00000000 00:00 0 3efe600000-3efe603000 r-xp 00000000 fd:01 700430 /usr/lib64/libdl-2.16.so 3efe603000-3efe802000 ---p 00003000 fd:01 700430 /usr/lib64/libdl-2.16.so 3efe802000-3efe803000 r--p 00002000 fd:01 700430 /usr/lib64/libdl-2.16.so 3efe803000-3efe804000 rw-p 00003000 fd:01 700430 /usr/lib64/libdl-2.16.so 3efea00000-3efea15000 r-xp 00000000 fd:01 672671 /usr/lib64/libz.so.1.2.7 3efea15000-3efec14000 ---p 00015000 fd:01 672671 /usr/lib64/libz.so.1.2.7 3efec14000-3efec15000 r--p 00014000 fd:01 672671 /usr/lib64/libz.so.1.2.7 3efec15000-3efec16000 rw-p 00015000 fd:01 672671 /usr/lib64/libz.so.1.2.7 3efee00000-3efee07000 r-xp 00000000 fd:01 700431 /usr/lib64/librt-2.16.so 3efee07000-3eff006000 ---p 00007000 fd:01 700431 /usr/lib64/librt-2.16.so 3eff006000-3eff007000 r--p 00006000 fd:01 700431 /usr/lib64/librt-2.16.so 3eff007000-3eff008000 rw-p 00007000 fd:01 700431 /usr/lib64/librt-2.16.so 3eff600000-3eff615000 r-xp 00000000 fd:01 700435 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 3eff615000-3eff814000 ---p 00015000 fd:01 700435 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 3eff814000-3eff815000 r--p 00014000 fd:01 700435 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 3eff815000-3eff816000 rw-p 00015000 fd:01 700435 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 3effa00000-3effa5d000 r-xp 00000000 fd:01 700439 /usr/lib64/libpcre.so.1.0.1 3effa5d000-3effc5c000 ---p 0005d000 fd:01 700439 /usr/lib64/libpcre.so.1.0.1 3effc5c000-3effc5d000 r--p 0005c000 fd:01 700439 /usr/lib64/libpcre.so.1.0.1 3effc5d000-3effc5e000 rw-p 0005d000 fd:01 700439 /usr/lib64/libpcre.so.1.0.1 3effe00000-3effe1f000 r-xp 00000000 fd:01 700440 /usr/lib64/libselinux.so.1 3effe1f000-3f0001e000 ---p 0001f000 fd:01 700440 /usr/lib64/libselinux.so.1 3f0001e000-3f0001f000 r--p 0001e000 fd:01 700440 /usr/lib64/libselinux.so.1 3f0001f000-3f00020000 rw-p 0001f000 fd:01 700440 /usr/lib64/libselinux.so.1 3f00020000-3f00022000 rw-p 00000000 00:00 0 3f00200000-3f00216000 r-xp 00000000 fd:01 700438 /usr/lib64/libresolv-2.16.so 3f00216000-3f00415000 ---p 00016000 fd:01 700438 /usr/lib64/libresolv-2.16.so 3f00415000-3f00416000 r--p 00015000 fd:01 700438 /usr/lib64/libresolv-2.16.so 3f00416000-3f00417000 rw-p 00016000 fd:01 700438 /usr/lib64/libresolv-2.16.so 3f00417000-3f00419000 rw-p 00000000 00:00 0 3f06200000-3f06203000 r-xp 00000000 fd:01 700443 /usr/lib64/libcom_err.so.2.1 3f06203000-3f06402000 ---p 00003000 fd:01 700443 /usr/lib64/libcom_err.so.2.1 3f06402000-3f06403000 r--p 00002000 fd:01 700443 /usr/lib64/libcom_err.so.2.1 3f06403000-3f06404000 rw-p 00003000 fd:01 700443 /usr/lib64/libcom_err.so.2.1 3f06600000-3f06603000 r-xp 00000000 fd:01 665783 /usr/lib64/libkeyutils.so.1.4 3f06603000-3f06802000 ---p 00003000 fd:01 665783 /usr/lib64/libkeyutils.so.1.4 3f06802000-3f06803000 r--p 00002000 fd:01 665783 /usr/lib64/libkeyutils.so.1.4 3f06803000-3f06804000 rw-p 00003000 fd:01 665783 /usr/lib64/libkeyutils.so.1.4 3f06a00000-3f06a0a000 r-xp 00000000 fd:01 700441 /usr/lib64/libkrb5support.so.0.1 3f06a0a000-3f06c09000 ---p 0000a000 fd:01 700441 /usr/lib64/libkrb5support.so.0.1 3f06c09000-3f06c0a000 r--p 00009000 fd:01 700441 /usr/lib64/libkrb5support.so.0.1 3f06c0a000-3f06c0b000 rw-p 0000a000 fd:01 700441 /usr/lib64/libkrb5support.so.0.1 3f06e00000-3f06e28000 r-xp 00000000 fd:01 700442 /usr/lib64/libk5crypto.so.3.1 3f06e28000-3f07028000 ---p 00028000 fd:01 700442 /usr/lib64/libk5crypto.so.3.1 3f07028000-3f07029000 r--p 00028000 fd:01 700442 /usr/lib64/libk5crypto.so.3.1 3f07029000-3f0702a000 rw-p 00029000 fd:01 700442 /usr/lib64/libk5crypto.so.3.1 3f0702a000-3f0702b000 rw-p 00000000 00:00 0 3f07200000-3f072d9000 r-xp 00000000 fd:01 700444 /usr/lib64/libkrb5.so.3.3 3f072d9000-3f074d9000 ---p 000d9000 fd:01 700444 /usr/lib64/libkrb5.so.3.3 3f074d9000-3f074e3000 r--p 000d9000 fd:01 700444 /usr/lib64/libkrb5.so.3.3 3f074e3000-3f074e5000 rw-p 000e3000 fd:01 700444 /usr/lib64/libkrb5.so.3.3 3f07600000-3f07641000 r-xp 00000000 fd:01 700445 /usr/lib64/libgssapi_krb5.so.2.2 3f07641000-3f07840000 ---p 00041000 fd:01 700445 /usr/lib64/libgssapi_krb5.so.2.2 3f07840000-3f07841000 r--p 00040000 fd:01 700445 /usr/lib64/libgssapi_krb5.so.2.2 3f07841000-3f07843000 rw-p 00041000 fd:01 700445 /usr/lib64/libgssapi_krb5.so.2.2 3f07e00000-3f07f9b000 r-xp 00000000 fd:01 700446 /usr/lib64/libcrypto.so.1.0.1c 3f07f9b000-3f0819a000 ---p 0019b000 fd:01 700446 /usr/lib64/libcrypto.so.1.0.1c 3f0819a000-3f081b4000 r--p 0019a000 fd:01 700446 /usr/lib64/libcrypto.so.1.0.1c 3f081b4000-3f081bf000 rw-p 001b4000 fd:01 700446 /usr/lib64/libcrypto.so.1.0.1c 3f081bf000-3f081c4000 rw-p 00000000 00:00 0 3f08e00000-3f08e5b000 r-xp 00000000 fd:01 700447 /usr/lib64/libssl.so.1.0.1c 3f08e5b000-3f0905a000 ---p 0005b000 fd:01 700447 /usr/lib64/libssl.so.1.0.1c 3f0905a000-3f0905e000 r--p 0005a000 fd:01 700447 /usr/lib64/libssl.so.1.0.1c 3f0905e000-3f09064000 rw-p 0005e000 fd:01 700447 /usr/lib64/libssl.so.1.0.1c 3f0fa00000-3f0fa0f000 r-xp 00000000 fd:01 665786 /usr/lib64/libbz2.so.1.0.6 3f0fa0f000-3f0fc0e000 ---p 0000f000 fd:01 665786 /usr/lib64/libbz2.so.1.0.6 3f0fc0e000-3f0fc0f000 r--p 0000e000 fd:01 665786 /usr/lib64/libbz2.so.1.0.6 3f0fc0f000-3f0fc10000 rw-p 0000f000 fd:01 665786 /usr/lib64/libbz2.so.1.0.6 3f11600000-3f11663000 r-xp 00000000 fd:01 700543 /usr/lib64/libfreebl3.soAborted (core dumped) but if i load fewer columns, it succeed: sprintf(sql, "copy binary into test3 from('/home/data1/writeBinary/id','/home/data1/writeBinary/1')"); so if i can load into table with a certain column ? Thanks! Meng

2 1

XQuery support in MonetDB
by Stephen Cameron 20 Jul '13

20 Jul '13

Hello List, Is there anyone making use of XQuery with MonetDB? Why was this XQuery work terminated/frozen, obviously its now a low priority, but why? People seem to resist saying that something is not as good an idea as first thought! Thanks for your insights. Steve Cameron collinta.com.au

2 1

Must the file be deleted after "copy binary into table from file"？
by integrity 19 Jul '13

19 Jul '13

can i have the file reserved? Thanks Meng

4 3

Re: Must the file be deleted after "copy binary into table from file"？
by integrity 19 Jul '13

19 Jul '13

Thanks very much for explaining these to me, but is it possible to provide two options of move, one do the move and reserve, one just move? That will be very convenient for uses.^_^ Sincerely, Meng PS,google mail in China is very unstable, often got disconnected from internet for some reason, so I have to applie another account here. ------------------ Original ------------------ From: "Stefan Manegold"<Stefan.Manegold(a)cwi.nl>; Date: Fri, Jul 19, 2013 02:15 PM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Cc: "Niels Nes"<Niels.Nes(a)cwi.nl>; "sjoerd"<Sjoerd.Mullender(a)cwi.nl>; Subject: Re: Must the file be deleted after "copy binary into table from file"？ Hi Meng, for efficiently, i.e., to avoid a file copy, MonetDB's binary copy into simply moves(^) the user-provided and to be loaded files into its dbfarm. Thus, if you need to preserve a copy of the files you provide, you need to do so yourself before executing a binary copy into. Best, Stefan (^) 1) The actual behavior (move) is actually different than the copy that the documentation at http://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/BinaryBulkLoad suggests. We need to fix the latter. 2) The actual behavior currently differs between data types; for numerical types, the user-provided files are moved as explained above; for stings, the files are read and processed, i.e., the used-provided file remains in place as-is. We need to consider unifying this behavior, e.g., by deleting also the user provided files for strings once they have been processed/loaded. 3) In case the copy into file, e.g., due to a failing sanity check after the file is moved, that data is factually lost, i.e., the files are gone, but no data is loaded into the database; we need to either document this clearly, or make the implementation behave like a transaction. ----- Original Message ----- > can i have the file reserved? > > > > Thanks > Meng > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

2 1

Join index
by Bouslah Ahmed 18 Jul '13

18 Jul '13

Hi, I want to optimize the runtime of this query Select sum(a.C4), b.C1, b.C2, b.C3 from v1 a, v2 b where (b.C1 = a.C1 and b.C2 = a.C2 and b.C3 = a.C3) or (b.C1 = a.C1 and b.C2 = a.C2 and b.C3 = 0) or (b.C1 = a.C1 and b.C2 = 0 and b.C3 = 0) or (b.C1 = 0 and b.C2 = 0 and b.C3 = 0) group by b.C1, b.C2, b.C3; why I thought of creating a BITMAP index join or create a bitmap index for v1 columns (c1, c2, c3) and v2 (c1, c2, c3). I can't do this with MonetDB can you help me.

2 1

failed to open file when 'copy binary into table from binaryformfile'
by integrity 17 Jul '13

17 Jul '13

hi all, when i use the command in sql: sql>copy binary into testtable from('file1'); failed to open file file1 the file1 contents : [svomhpc@localhost /home/data1]$>cat file1 0123[svomhpc@localhost /home/data1]$>hexdump file1 0000000 0000 3000 0000 3100 0000 3200 0000 3300 0000010 can anyone tell me what's wrong with file1 or how to use copy binary with? Meng

5 8

International characters in MonetDB
by aaaaaa ggggggg 15 Jul '13

15 Jul '13

Hello All, I have been trying to do some bulk uploads from CSV files into a MonetDB database. All the fields in the database are VARCHAR. The text in some of the fields contains international characters such as : ü, Ó, é etc. When I execute : copy into database from 'filename.csv' using delimiters ',','\n','"' NULL as ''; I get an error stating that the fields with words containing the international characters could not be imported, and the import process stops. How can I solve this issue? (I am running Monetdb on windows) Thanks, SG

3 5

Giving monetdb a fair chance
by Franck Routier 12 Jul '13

12 Jul '13

Hi, I am benchmarking different DB alternatives for our BI application. We are using Mondrian/JPivot on a dataset with ~300 millions row, and several parent/child dimensions. We are running some 50 real life scenario (ie 50 differents cubes using parts of this dataset) and measuring user response time on a server with several stripped Sata3 SSD, 32 GB ram and a i7-3820 CPU @ 3.60GHz (6 cores). During the tests, Monetdb performed by far the worst of Postgresql, VectorWise and (obvioulsy) MonetDB. Things that take 10 seconds with Postgresql take 450 second with monetdb... This was not expected at all, so looking at the server, I saw that MonetDB was consuming all memory (why not), including swap : bad idea I think. I was expecting MonetDB to memory mapped files when needed, real memory, but not use swap as it it was real memory... So here is my question : with a (quite) big dataset, 32GB ram, is there anything I can do to help MonetDB behave according to expectancies ? (maybe I should use ulimit ?). Thanks for your input, Franck

4 10

Problem with Mtest.py and monetdb command
by Bingyi Cao 11 Jul '13

11 Jul '13

Hey, I am new to MonetDB. I installed MonetDB yesterday. And I run Mtest.py. The results are as below: *!ERROR: Testing FAILED SIGNIFICANTLY (18 out of 1854 tests failed)* * * * 40 out of 1854 tests could not be executed* * 1 out of 1854 tests produced slightly different output* * sql/test/BugTracker-2012/predicate_select.Bug-3090* * 17 out of 1854 tests produced SIGNIFICANTLY different output* * monetdb5/optimizer/inlineFunction* * monetdb5/optimizer/ifthencst* * monetdb5/tests/BugTracker/kunion-and-nil.Bug-1667* * sql/jdbc/tests/Test_PSsomeamount* * sql/jdbc/tests/Test_PSlargebatchval* * sql/jdbc/tests/Test_PSmanycon* * sql/jdbc/tests/Test_PSmetadata* * sql/jdbc/tests/Test_PSsqldata* * sql/jdbc/tests/Test_Rsqldata* * sql/jdbc/tests/BugConcurrent_sequences* * sql/test/BugTracker-2010/incomplete-utf8-sequence.Bug-2575* * sql/test/BugTracker-2012/conditions_when_for_triggers_do_not_work.Bug-2073* * sql/test/BugTracker-2012/create_function.Bug-3172* * sql/test/BugTracker-2012/currenttime.Bug-2781* * sql/test/BugTracker-2012/null_except_null.Bug-3040* * sql/test/BugTracker-2012/day-of-month-localization.Bug-2962* * sql/test/Triggers/recursive_triggers* * * * * * First, check the testing results in /usr/local/mTests/index.html !* * * * Then, fix the problems by:* * - fixing sources and test scripts* * - fixing stable output by hand* * - approving test output by Mapprove.py (cf. Mapprove.py -?)* * * * After that, re-run Mtest.* I hasn't change any source code yet. I am not sure whether I installed it correctly. Also now I have no problem with command monetdbd, mserver5, mclient. However, I cannot use command with monetdb, the error information is always as below: monetdb: cannot find a control socket, use -h and/or -p I did a little bit research, to me it seems that it should automatically find a socket. Tell me if I am wrong. I also try to use the option -h -P, which doesn't work still. Do I actually install it correctly I tried to uninstall and reinstall it,but it hasn't solve the problem at all. Thanks very much!

2 3

Re: Fw:how are user defined function arguments passed ?
by integrity 10 Jul '13

10 Jul '13

Thanks, it's working, I am so excited, thank you all Best regards! Meng ------------------ Original ------------------ From: "Martin Kersten"<Martin.Kersten(a)cwi.nl>; Date: Wed, Jul 10, 2013 05:23 PM To: "users-list"<users-list(a)monetdb.org>; Subject: Re: Fw:how are user defined function arguments passed ? Yes, PATTERNs and COMMANDs have different interfaces. regards, Martin On 7/10/13 10:41 AM, Lefteris wrote: > try on your mal definition toinstead of the keyword pattern, to use > the keyword command. Patterns have a different signature if i remember > correctly. > > On Wed, Jul 10, 2013 at 10:28 AM, integrity <357416268(a)qq.com> wrote: >> Thanks very much for your instant reply,Lefteris! >> >> I wrote a simple test UDF, but it doesnt work. >> >> myudf.c >> #include "monetdb_config.h" >> #include "myudf.h" >> >> str >> addint(int *ret,int *one) >> { >> >> *ret=(*one)+10; >> >> return MAL_SUCCEED; >> >> } >> ------------------------------------------------------------------------------------------------------------ >> myudf.h >> /* In your own module, replace "UDF" & "udf" by your module's name */ >> >> #ifndef _SQL_MYUDF_H_ >> #define _SQL_MYUDF_H_ >> #include "sql.h" >> #include <string.h> >> >> /* This is required as-is (except from renaming "UDF" & "udf" as suggested >> * above) for all modules for correctly exporting function on Unix-like and >> * Windows systems. */ >> >> #ifdef WIN32 >> #ifndef LIBMYUDF >> #define myudf_export extern __declspec(dllimport) >> #else >> #define myudf_export extern __declspec(dllexport) >> #endif >> #else >> #define myudf_export extern >> #endif >> >> /* export MAL wrapper functions */ >> >> myudf_export str addint(int *ret, int *one); >> #endif /* _SQL_MYUDF_H_ */ >> ----------------------------------------------------------------------------------------------------- >> myudf.mal >> # scalar MAL signatures >> module myudf; >> >> pattern addint(one:int):int >> address addint >> comment "add one and two"; >> ----------------------------------------------------------------------------------------------------- >> 82_myudf.sql >> create function addint(one INT) >> returns int external name myudf.addint; >> ----------------------------------------------------------------------------------------------------- >> when i execute in sql: >> sql>select addint(76777); >> +---------------------+ >> | addint_single_value | >> +=====================+ >> | 0 | >> +---------------------+ >> 1 tuple (1.364ms) >> sql> >> sql>select addint(100); >> +---------------------+ >> | addint_single_value | >> +=====================+ >> | 0 | >> +---------------------+ >> 1 tuple (0.295ms) >> sql> >> sql> >> sql> >> sql>select addint(101); >> +---------------------+ >> | addint_single_value | >> +=====================+ >> | 0 | >> +---------------------+ >> 1 tuple (0.289ms) >> sql>select addint(1010000000000); >> overflow in conversion of 1010000000000 to int. >> sql> >> sql>select addint(1010000); >> +---------------------+ >> | addint_single_value | >> +=====================+ >> | 0 | >> +---------------------+ >> 1 tuple (1.087ms) >> sql> >> >> i checked every format of above files many times, i cannot find what's wrong >> with my program... >> it seems that the parameter didn't got passed to MAL.... >> >> Thanks again for your help! >> >> Sincerely, >> >> Meng >> ------------------ Original ------------------ >> From: "Lefteris"<lsidir(a)gmail.com>; >> Date: Wed, Jul 10, 2013 04:13 PM >> To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; >> Subject: Re: Fw:how are user defined function arguments passed ? >> >> The str *ret is you return value as described in the MAL signature >> reverse(ra1:str):str >> >> the ra1 is the name of a str variable in MAL, which is mapped to the >> str *src variable of the UDF. Naming here plays no role, just the >> types and the order of definition. Similarly, the :str end of the mal >> signature reverse(ra1:str):str means that it returns a str type, which >> in turn is mapped to the str *ret variable of the UDF signature. >> Naming plays no role again, just that the first variable of the >> signature in the udf is the return variable. Ofcourse it does not need >> to be always a str, it depends on the definition of the mal signature. >> >> On Wed, Jul 10, 2013 at 9:59 AM, integrity <357416268(a)qq.com> wrote: >>> Could anyone explain this question to me, please? >>> >>> I am still wondering on it.... >>> >>> Thanks! >>> >>> Meng >>> ------------------ Original ------------------ >>> From: "integrity"<357416268(a)qq.com>; >>> Date: Fri, Jul 5, 2013 03:42 PM >>> To: "users-list"<users-list(a)monetdb.org>; >>> Subject: how are user defined function arguments passed ? >>> >>> Hi everyone, >>> >>> In >>> >>> http://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/UserDefinedFuncti… >>> In thesection about how to extend SQL with a simple scalar function to >>> reverse a string >>> >>> step 3. Extension starts with a definitin of the MAL signatures. See the >>> example given, or browse through the files in monetdb5/modules/mal/*.mal >>> to >>> get a glimpse on how to write them. The MonetDB kernel documentation >>> provides more details. The file contains the MAL snippet: >>> command reverse(ra1:str):str >>> address UDFreverse >>> comment "Reverse a string"; >>> >>> step 4. The signature says that it expects a command body implementation >>> under the name UDFreverse, shown below. The C-signature is a direct >>> mapping, >>> where arguments are passed by reference and the return value(s) >>> references >>> are the first in the arguments list. The body should return a (malloced) >>> string to denote an exception being raised or MAL_SUCCEED upon access. >>> #include "udf.h" >>> >>> static str >>> reverse(const char *src) >>> { >>> size_t len; >>> str ret, new; >>> >>> /* The scalar function returns the new space */ >>> len = strlen(src); >>> ret = new = GDKmalloc(len + 1); >>> if (new == NULL) >>> return NULL; >>> new[len] = 0; >>> while (len > 0) >>> *new++ = src[--len]; >>> return ret; >>> } >>> >>> str >>> UDFreverse(str *ret, str *src) >>> { >>> if (*src == 0 || strcmp(*src, str_nil) == 0) >>> *ret = GDKstrdup(str_nil); >>> else >>> *ret = reverse(*src); >>> return MAL_SUCCEED; >>> } >>> >>> >>> I don't know why the argument name of "command reverse(ra1:str):str" >>> different from function definition UDFreverse(str *ret, str *src)? >>> >>> how does ret come from? is it according to the naming rules that the first >>> argument is the result, must it be a string type? >>> >>> Thanks! >>> >>> Meng >>> >>> >>> _______________________________________________ >>> users-list mailing list >>> users-list(a)monetdb.org >>> http://mail.monetdb.org/mailman/listinfo/users-list >>> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> http://mail.monetdb.org/mailman/listinfo/users-list >> . >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> http://mail.monetdb.org/mailman/listinfo/users-list >> > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list > _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

1 0

Re: Fw:how are user defined function arguments passed ?
by integrity 10 Jul '13

10 Jul '13

Thanks very much for your instant reply,Lefteris! I wrote a simple test UDF, but it doesnt work. myudf.c #include "monetdb_config.h" #include "myudf.h" str addint(int *ret,int *one) { *ret=(*one)+10; return MAL_SUCCEED; } ------------------------------------------------------------------------------------------------------------ myudf.h /* In your own module, replace "UDF" & "udf" by your module's name */ #ifndef _SQL_MYUDF_H_ #define _SQL_MYUDF_H_ #include "sql.h" #include <string.h> /* This is required as-is (except from renaming "UDF" & "udf" as suggested * above) for all modules for correctly exporting function on Unix-like and * Windows systems. */ #ifdef WIN32 #ifndef LIBMYUDF #define myudf_export extern __declspec(dllimport) #else #define myudf_export extern __declspec(dllexport) #endif #else #define myudf_export extern #endif /* export MAL wrapper functions */ myudf_export str addint(int *ret, int *one); #endif /* _SQL_MYUDF_H_ */ ----------------------------------------------------------------------------------------------------- myudf.mal # scalar MAL signatures module myudf; pattern addint(one:int):int address addint comment "add one and two"; ----------------------------------------------------------------------------------------------------- 82_myudf.sql create function addint(one INT) returns int external name myudf.addint; ----------------------------------------------------------------------------------------------------- when i execute in sql: sql>select addint(76777); +---------------------+ | addint_single_value | +=====================+ | 0 | +---------------------+ 1 tuple (1.364ms) sql> sql>select addint(100); +---------------------+ | addint_single_value | +=====================+ | 0 | +---------------------+ 1 tuple (0.295ms) sql> sql> sql> sql>select addint(101); +---------------------+ | addint_single_value | +=====================+ | 0 | +---------------------+ 1 tuple (0.289ms) sql>select addint(1010000000000); overflow in conversion of 1010000000000 to int. sql> sql>select addint(1010000); +---------------------+ | addint_single_value | +=====================+ | 0 | +---------------------+ 1 tuple (1.087ms) sql> i checked every format of above files many times, i cannot find what's wrong with my program... it seems that the parameter didn't got passed to MAL.... Thanks again for your help! Sincerely, Meng ------------------ Original ------------------ From: "Lefteris"<lsidir(a)gmail.com>; Date: Wed, Jul 10, 2013 04:13 PM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Subject: Re: Fw:how are user defined function arguments passed ? The str *ret is you return value as described in the MAL signature reverse(ra1:str):str the ra1 is the name of a str variable in MAL, which is mapped to the str *src variable of the UDF. Naming here plays no role, just the types and the order of definition. Similarly, the :str end of the mal signature reverse(ra1:str):str means that it returns a str type, which in turn is mapped to the str *ret variable of the UDF signature. Naming plays no role again, just that the first variable of the signature in the udf is the return variable. Ofcourse it does not need to be always a str, it depends on the definition of the mal signature. On Wed, Jul 10, 2013 at 9:59 AM, integrity <357416268(a)qq.com> wrote: > Could anyone explain this question to me, please? > > I am still wondering on it.... > > Thanks! > > Meng > ------------------ Original ------------------ > From: "integrity"<357416268(a)qq.com>; > Date: Fri, Jul 5, 2013 03:42 PM > To: "users-list"<users-list(a)monetdb.org>; > Subject: how are user defined function arguments passed ? > > Hi everyone, > > In > http://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/UserDefinedFuncti… > In thesection about how to extend SQL with a simple scalar function to > reverse a string > > step 3. Extension starts with a definitin of the MAL signatures. See the > example given, or browse through the files in monetdb5/modules/mal/*.mal to > get a glimpse on how to write them. The MonetDB kernel documentation > provides more details. The file contains the MAL snippet: > command reverse(ra1:str):str > address UDFreverse > comment "Reverse a string"; > > step 4. The signature says that it expects a command body implementation > under the name UDFreverse, shown below. The C-signature is a direct mapping, > where arguments are passed by reference and the return value(s) references > are the first in the arguments list. The body should return a (malloced) > string to denote an exception being raised or MAL_SUCCEED upon access. > #include "udf.h" > > static str > reverse(const char *src) > { > size_t len; > str ret, new; > > /* The scalar function returns the new space */ > len = strlen(src); > ret = new = GDKmalloc(len + 1); > if (new == NULL) > return NULL; > new[len] = 0; > while (len > 0) > *new++ = src[--len]; > return ret; > } > > str > UDFreverse(str *ret, str *src) > { > if (*src == 0 || strcmp(*src, str_nil) == 0) > *ret = GDKstrdup(str_nil); > else > *ret = reverse(*src); > return MAL_SUCCEED; > } > > > I don't know why the argument name of "command reverse(ra1:str):str" > different from function definition UDFreverse(str *ret, str *src)? > > how does ret come from? is it according to the naming rules that the first > argument is the result, must it be a string type? > > Thanks! > > Meng > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list > _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

3 2

Fw:how are user defined function arguments passed ?
by integrity 10 Jul '13

10 Jul '13

Could anyone explain this question to me, please? I am still wondering on it.... Thanks! Meng ------------------ Original ------------------ From: "integrity"<357416268(a)qq.com>; Date: Fri, Jul 5, 2013 03:42 PM To: "users-list"<users-list(a)monetdb.org>; Subject: how are user defined function arguments passed ? Hi everyone, In http://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/UserDefinedFuncti… In thesection about how to extend SQL with a simple scalar function to reverse a string step 3. Extension starts with a definitin of the MAL signatures. See the example given, or browse through the files in monetdb5/modules/mal/*.mal to get a glimpse on how to write them. The MonetDB kernel documentation provides more details. The file contains the MAL snippet: command reverse(ra1:str):str address UDFreverse comment "Reverse a string"; step 4. The signature says that it expects a command body implementation under the name UDFreverse, shown below. The C-signature is a direct mapping, where arguments are passed by reference and the return value(s) references are the first in the arguments list. The body should return a (malloced) string to denote an exception being raised or MAL_SUCCEED upon access. #include "udf.h" static str reverse(const char *src) { size_t len; str ret, new; /* The scalar function returns the new space */ len = strlen(src); ret = new = GDKmalloc(len + 1); if (new == NULL) return NULL; new[len] = 0; while (len > 0) *new++ = src[--len]; return ret; } str UDFreverse(str *ret, str *src) { if (*src == 0 || strcmp(*src, str_nil) == 0) *ret = GDKstrdup(str_nil); else *ret = reverse(*src); return MAL_SUCCEED; } I don't know why the argument name of "command reverse(ra1:str):str" different from function definition UDFreverse(str *ret, str *src)? how does ret come from? is it according to the naming rules that the first argument is the result, must it be a string type? Thanks! Meng

2 1

Bind variables and parameterising the expression in case when
by Dan 10 Jul '13

10 Jul '13

Hi, I seem to have come across a strange restriction in using bind variables in MonetDB. If i use bind variables in the normal way by saying where field=${var} then it works. But not in this case: Here is a sample: select case when ${Brand} = 'All' And here is the error: EXEC: wrong type for argument 1 of prepared statement: char, expected char This is using the latest version of the jdbc driver (2.9) via Pentaho. Not quite sure how to set up a test case of this via a normal SQL client though. Thanks for any suggestions! Dan

2 1

Connection monitoring
by Taka Yasuda 09 Jul '13

09 Jul '13

Hi, What is the best way to monitor the connections to the monetdb server? Regards, Taka

2 4

Problems executing custom aggregation function on large datasets
by Sebastian Dorok 09 Jul '13

09 Jul '13

Hi all, I'm developing a custom aggregation function. Everything works fine until I use it on large datasets (ca. 270 Mio. rows). Here is what my function signature looks like: str AGGSimpleGenotype(int *ret, int *bid, int *gid, int *eid, bit ignore_nils); I assume that the BATs identified by bid and gid have the same length. That's why my function checks this circumstance. On my test dataset (ca. 200 rows) everything works fine. When I use my production dataset my aggregation function returns an error complaining that the BATs have not the same length and stops aggregating (by design)! How can this happen? The SQL query is: select grpgenotype(base) from test.bases group by pos; I attached the MAL plan generated for the large dataset. When I execute it step by step I get an error in line 39 saying: mal>X_106 := sql.delta(X_56,X_70,X_71,X_8); MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = X_106 := sql.delta(X_56,X_70,X_71,X_8); ERROR = !TypeException:user.main[1]:'sql.delta' undefined in: X_106:any := sql.delta(X_56:bat[:oid,:str], X_70:bat[:oid,:oid], X_71:bat[:oid,:str], X_8:bat[:oid,:any]) This error occurs before my function is entered but I think it must be related with my problem. Is it an internal MonetDB problem? I'm using the Feb2013-SP2 release. Any ideas? Thx.

4 4

how to debug and use user defined functions?
by hlfwm hlfwm 09 Jul '13

09 Jul '13

i defined a user defined function "binarycopy",when i use my function, there is error: sql>select binarycopy('starid','1111',1); TypeException:user.s1_1[5]:'svom.binarycopy' undefined in: _9:any := svom.binarycopy(_6:str, _7:str, _8:int) program contains errors i want to debug the error with mal, there also some error: [svomhpc@localhost /home/data1]$>mclient -l mal -u monetdb -d mydb4 password: Welcome to mclient, the MonetDB interactive terminal (unreleased) Type \q to quit, \? for a list of available commands mal> mal> mal>function test(i:int):str; mal>io.print(i); mal>i:=i*2; mal>b:=bat.new(:int,:int); mal>bat.insert(b,1,i); mal>io.print(b); mal>return test:= "ok"; mal>end test; MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = end test; ERROR = !TypeException:user.test[3]:'bat.new' undefined in: b:any := bat.new(_5:int, _6:int) !TypeException:user.test[4]:'bat.insert' undefined in: _7:any := bat.insert(b:any, _8:int, i:int) mal> could anyone tell me how to debug my function ,please? Thanks!

8 28

create table as fails
by Franck Routier 08 Jul '13

08 Jul '13

Hi, I am trying to create a table as select, but it fails... Here is the statement: create table imagerie_ft as select dwhdim.*, dwhq135.q135 from dwhdim join dwhq135 on dwhdim.dwhcolcsu = dwhq135.dwhcolcsu; The select alone works fine. The same statement with create view instead of create table works fine. But the create table fails with: Error: syntax error, unexpected SCOLON, expecting WITH in: "create table imagerie_ft as SQLState: 42000 ErrorCode: 0 Error: select dwhdim.*, dwhq135.q135 SQLState: 22000 ErrorCode: 0 Error: from dwhdim join dwhq135 on dwhdim.dwhcolcsu = dwhq135.dwhcolcsu;" SQLState: 22000 ErrorCode: 0 Should I report a bug, or is using the * in the statement for a create table a non supported sql idiom? Franck

3 2

aggregate function in MAL
by Scott Mathieson 05 Jul '13

05 Jul '13

Hi, I am trying to create a simple aggregate function entirely in MAL and am now struggling at the final hurdle. I have created the aggregate function such that it aggregates across an entire table but in the sub aggregate function I can't work out how to link the group to data BAT (g) to the data. The aggregate is designed to simply collect a sample of the values from the named column and return them as a pipe delimited string value. Can anyone tell me how to use the 'g' bat to join to the 'b' bat in the sub... aggregate function to get only the data rows for a specific group - currently I only get 2 rows of data per group even though I know that there should be 100k+ rows in each group? Here's the code: function bin_sample(b:bat[:oid, :any]):str; value := ""; bsamp := sample.uniform(b, 1000:wrd); barrier (h, t) := iterator.new(bsamp); tmp := "" + t; type := bat.getTailType(bsamp); test := str.startsWith(type, "str"); # Force the string type to be cast to a string or you just end up with numeric values barrier xxxx := test; tmp := str.str(t); exit xxxx; value := value + tmp; value := value + "||"; redo (h, t) := iterator.next(bsamp); exit (h, t); return value; end bin_sample; function subbin_sample(b:bat[:oid,:any_1],g:bat[:oid,:oid],e:bat[:oid,:any_2],skip_nils:bit) :bat[:oid,:str]; bn:=bat.new(:oid, :str); # check 'e' has some values - this is the list of group IDs in the head # 'g' is the group to the data BAT head ID mapping # 'b' is the data column BAT that we're aggregating over barrier (grpid, t) := iterator.new(e); # create a single row BAT of the group id we're processing bn_gid := bat.new(:oid, :oid); bat.insert(bn_gid, grpid, grpid); # join this to the GID->DATA map BAT to get the Group DATA oids b_g2dr := bat.reverse(g); b_gid2data := algebra.leftjoin(bn_gid, b_g2dr); grpval := ""; # Get the BAT of the actual data to be processed # # !!! HOW TO GET THIS TO WORK???????? # b_data := algebra.join(b_gid2data, b); grpval := bin_sample(b_data); # Store the result for this group bat.insert(bn, grpid, grpval); redo (grpid, t) := iterator.next(e); exit (grpid, t); return bn; end subbin_sample; Scott Mathieson, ________________________________

2 3

Failing to load a table
by Franck Routier 05 Jul '13

05 Jul '13

Hi all, I am trying to copy a table from a postgresql database into monetdb, but it fails, with monetdb filling the disks. I am using latest version of Monetdb (Feb2013-SP3) on Ubuntu Linux, with 32 GB Ram and SSD disks. To copy into Monetdb, I have a java program that runs mclient as: mclient --language=sql --databas=database --host=host --user=user /tmp/command.ctl with command.ctl : COPY nnn RECORDS INTO table FROM STDIN USING DELIMITERS'|', '\n', '\' NULL AS '' Then the Postgresql resultset is read from jdbc, formatted and written into STDIN. So far so good, it work with my dimensions tables (small), and with a 295 millions rows fact table that has 109 columns. This fact table takes 86 GB on the disk for Postgresql storage (plus indices). I have another form of the fact table, with less rows (184 millions) but more columns (505). While the first form of the table has a measureId column and only one value column, the other form of the table has one value column per measure. (It is a "sum(case when measureId=X then value else null ... group by" version of the first table). Loading this "columnar" table fails, with monetdb filling the disks (something like 500 GB was left before the loading process) and quitting (not sure what happens exactly here on the server, but I get a broken pipe error in java, and the monetdb database is lost). The table size on disk for Postgresql is 37 GB. Using VectorWise, the table loads, and take only 7 GB on disks, but the loading process (very similar to what we do with MonetDB) takes a very long time( 25 hours), much longer than what happens with the 109 columns tables. I am trying to make sense of that and understand why this table is giving a (very) hard time to monetdb (and somehow VectorWise as well). Does anyone have an explanation ? thanks, Franck

4 12

how are user defined function arguments passed ?
by integrity 05 Jul '13

05 Jul '13

Hi everyone, In http://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/UserDefinedFuncti… In thesection about how to extend SQL with a simple scalar function to reverse a string step 3. Extension starts with a definitin of the MAL signatures. See the example given, or browse through the files in monetdb5/modules/mal/*.mal to get a glimpse on how to write them. The MonetDB kernel documentation provides more details. The file contains the MAL snippet: command reverse(ra1:str):str address UDFreverse comment "Reverse a string"; step 4. The signature says that it expects a command body implementation under the name UDFreverse, shown below. The C-signature is a direct mapping, where arguments are passed by reference and the return value(s) references are the first in the arguments list. The body should return a (malloced) string to denote an exception being raised or MAL_SUCCEED upon access. #include "udf.h" static str reverse(const char *src) { size_t len; str ret, new; /* The scalar function returns the new space */ len = strlen(src); ret = new = GDKmalloc(len + 1); if (new == NULL) return NULL; new[len] = 0; while (len > 0) *new++ = src[--len]; return ret; } str UDFreverse(str *ret, str *src) { if (*src == 0 || strcmp(*src, str_nil) == 0) *ret = GDKstrdup(str_nil); else *ret = reverse(*src); return MAL_SUCCEED; } I don't know why the argument name of "command reverse(ra1:str):str" different from function definition UDFreverse(str *ret, str *src)? how does ret come from? is it according to the naming rules that the first argument is the result, must it be a string type? Thanks! Meng

1 0

Announcement: New Feb2013-SP3 Bugfix release of MonetDB suite
by Hannes Muehleisen 02 Jul '13

02 Jul '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>.

2 1

SQL question
by Rémi Daniel 28 Jun '13

28 Jun '13

Hello, I'm a French user of MonetDB, and I have a SQL question, I want to execute on MonetDB equivalent of this mysql request, and i don't find any solution : "SELECT field1, field2 FROM table GROUP BY field3 LIMIT 5;" I want to do a GROUP BY of a field3 which is not in the SELECT to get a limit of request results by this field3, like the "first" function of some databases. Does anyone have any idea on how I can do that ? Thanks for your help, Remi -- <http://inbox.fr> Rémi Daniel Développeur Web rdaniel(a)inbox.fr 30, rue Vincent Moris 92240 Malakoff - France Fax : +33 (0)1 55 48 02 79

5 7

how to write binary data into BAT tail?
by hlfwm hlfwm 28 Jun '13

28 Jun '13

i want to create a monetdb function to help write my astronomical *binary*processing result into database,this function will load the binary result from memory directly into monetdb to save time, so i defined a UDF binaryCopy() , svom.c int binaryCopy(str columnname, str buffer, int count) { mvc *sql = NULL; sql_table *tmatch = NULL; /*BATs in which the data of the columns of the output table will be stored*/ BAT *tmatch_column = NULL; BUN nr_rows; /* pointer to the tail of above BAT, which is just a normal C-array */ lng *tmatch_column_lng_t = NULL; flt *tmatch_column_flt_t = NULL; /*_bind_table is just a wrapper that tries to see if it can find the table match_table under three default schemas*/ if(!(tmatch = _bind_table(sql, NULL, "testtable"))) //return sql_message("42S02!BINARYLOAD():no such table test"); return 0; //nr_match = 20000; nr_rows = count; /* create BATs for all columns of the output table 'tmatch' */ if(strcmp(columnname,"starid")==0 || strcmp(columnname,"crossid")==0 ||strcmp(columnname,"catid")==0 ) { tmatch_column = BATnew(TYPE_void, TYPE_lng,nr_rows); }else { tmatch_column = BATnew(TYPE_void,TYPE_flt,nr_rows); } /* now tmatch_starid_t points to the first element of the BAT's tails */ tmatch_column_lng_t = (lng*)Tloc(tmatch_column,BUNfirst(tmatch_column)); /* now we can write data into the BAT tail */ if((strcmp(columnname,"starid")==0) || (strcmp(columnname,"crossid")==0) ||(strcmp(columnname,"catid")==0) ) { *memcpy(tmatch_column_lng_t,buffer,nr_rows*sizeof(lng)); //is it right to write binary data into BAT???* }else { memcpy(tmatch_column_flt_t, buffer, nr_rows*sizeof(flt)); } /* Set proper properties for all BATs */ BATsetcount(tmatch_column,nr_rows); BATseqbase(tmatch_column,0); /* sorted:1, column is sorted in ascending order */ tmatch_column->tsorted = 0; /* if the tail is reverse sorted. */ tmatch_column->trevsorted = 0; /* if the tail contains NIL values */ tmatch_column->T->nil = 0; /* if the tail doesn't contain NIL values */ tmatch_column->T->nonil = 0; /* if the values in the tail are unique.Since BATkey only works with BAT heads, * BATmirror is used to switch the head and tail of the BATs */ if(strcmp(columnname,"starid")==0) { BATkey(BATmirror(tmatch_column),1); //only starid is 1 }else { BATkey(BATmirror(tmatch_column),0); } /* finally, append the result BATs to the result table * ignore errors, since we are finishing */ _append_bat(sql,tmatch,columnname,tmatch_column); /* The BATs now contain the data for the columns of the output tables, * append_bat registers in the system where to find the BAT for those columns. */ /* To avoid memory leak, BAT reference counter must be decreased after use. * That can be done using BBPunfix. */ if(tmatch_column) BBPunfix(tmatch_column->batCacheid); return 1; //success } Thanks!

2 2

Windows: Listen on addresses other than 127.0.0.1.
by Neil French 27 Jun '13

27 Jun '13

Hi, I am trying to get MonetDB Server for Windows x64 to listen on something other than 127.0.0.1. In the Getting started guide it mentions setting "mapi_open=yes" which I have added to M5Server.bat as: "%MONETDB%\bin\mserver5.exe" --set "mapi_open=yes" --set "prefix=%MONETDB%" --set "exec_prefix=%MONETDB%" %MONETDBFARM% %* However when I start the server I still see: # MonetDB 5 server v11.15.7 "Feb2013-SP2" # Serving database 'Data', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 11.989 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/JAQL module loaded # MonetDB/SQL module loaded When I do a netstat –a I see port 50000 is listening only on 127.0.0.1 with nothing else using port 50000 on any other address. When I perform a select * from environment(); I see: mapi_port | 50000 mapi_open | yes mapi_autosense | false I have Windows Firewall disabled and have tried allowing m5server.exe through the firewall. Could somebody please tell me how I get MonetDB to listen on anything other than 127.0.0.1 on Windows? Thanks, Neil. IMPORTANT The information contained in this e-mail and any attachments is intended only for the named recipient and may be privileged or confidential. If you are not the intended recipient, please notify us immediately on +44 (0)1908 425000 and do not disclose, copy, distribute or take any action based on the contents of this e-mail. You should understand and accept that, when communicating with us by e-mail, it is not a totally secure communication medium. We accept no liability for any direct, indirect or consequential loss arising from any action taken in reliance on the information contained in this e-mail and give no warranty or representation as to its accuracy or reliability. DIGITALK has the facility to monitor and read both incoming and outgoing communications by e-mail. In line with industry efforts to reduce the proliferation of un-solicited SPAM, DIGITALK uses methods and ban-lists to prevent malicious content reaching our users. This message and any attachments has been scanned for known viruses. However, we would advise you to ensure the content is indeed virus free. We do not, to the extent permitted by law, accept any liability (whether in contract, negligence or otherwise) for any virus infection and/or external compromise of security and/or breach of confidentiality in relation to transmissions sent by e-mail. VAT No: GB 876 3287 81. Reg No: 3080801 Place of Registration: England Registered Office Address: 2 Radian Court, Knowlhill, Milton Keynes.

2 1

Indexing in monetdb
by Anshuman Kumar 27 Jun '13

27 Jun '13

Hello everyone on the mailing list, Please find some time to go through my queries. I created a table with some fields out of which 1 is timestamp. I wanted to have indexing on this table. While going through the available documentation for monetdb, I found out that it creates its own indexing and might ignore user provided indexing. I went ahead and started pushing in data into the table with single inserts (10000 of them accumulated in a transaction). My table is pretty big now and it contains about 250 million records (over 2 months of data). Even simple queries to find records between specified timestamps takes a lot of time(12-13 mins in some cases) We really feel the need of indexing on the table(don't know if that can help). I happened to stumble upon an article on monetdb mailers: <http://mail.monetdb.org/pipermail/users-list/2008-March/001726.html> http://mail.monetdb.org/pipermail/users-list/2008-March/001726.html According to this article if I insert data into the table using - INSERT INTO table_name (...) ORDER BY X,Y. The order by here will enforce indexing on X, Y. I would like to confirm whether this really enforce indexing? I have a few other questions pertaining to my case: 1) Is there a predefined way of speeding up such queries in monetdb? 2) If the insert with an order by clause is a work around then what if going further, I start inserting with the insert query with the order by clause. a)Will that enforce indexing on the whole table or on only those records which were inserted using such inserts? b) Or should I just move the data from my table with 250 million records into another table using the insert with order by clause? 3)One more question regarding the setup of monetdb - I have initiated monetdb using the M5server.bat on my windows 2008 R2 64 bit server. This makes it run as a service and starts off at the server start-up. This is all nice and acceptable. The problem comes when I need to restart the monetdb server. The only way I was able to do this was to kill the process (mServer5.exe) since I was not able to figure out the service (in the windows services) which starts monetdb. So my simple question is how to restart monetdb server if it gets installed by M5server.bat? Can somebody please help me with the queries? Regards, Anshuman

2 1

RE: Indexing in monetdb
by Baskar Duraikannu 27 Jun '13

27 Jun '13

Anshuman I do not have answer to your queries. We are trying to move monetdb and would be extremely helpful if you could let me know 1. How big is your database? 2. Are you running in production? --- Original Message --- From: "Anshuman Kumar" <anshuman.kumar(a)flowedge.in> Sent: June 27, 2013 5:44 AM To: users-list(a)monetdb.org Subject: Indexing in monetdb Hello everyone on the mailing list, Please find some time to go through my queries. I created a table with some fields out of which 1 is timestamp. I wanted to have indexing on this table. While going through the available documentation for monetdb, I found out that it creates its own indexing and might ignore user provided indexing. I went ahead and started pushing in data into the table with single inserts (10000 of them accumulated in a transaction). My table is pretty big now and it contains about 250 million records (over 2 months of data). Even simple queries to find records between specified timestamps takes a lot of time(12-13 mins in some cases) We really feel the need of indexing on the table(don't know if that can help). I happened to stumble upon an article on monetdb mailers: <http://mail.monetdb.org/pipermail/users-list/2008-March/001726.html> http://mail.monetdb.org/pipermail/users-list/2008-March/001726.html According to this article if I insert data into the table using - INSERT INTO table_name (...) ORDER BY X,Y. The order by here will enforce indexing on X, Y. I would like to confirm whether this really enforce indexing? I have a few other questions pertaining to my case: 1) Is there a predefined way of speeding up such queries in monetdb? 2) If the insert with an order by clause is a work around then what if going further, I start inserting with the insert query with the order by clause. a)Will that enforce indexing on the whole table or on only those records which were inserted using such inserts? b) Or should I just move the data from my table with 250 million records into another table using the insert with order by clause? 3)One more question regarding the setup of monetdb - I have initiated monetdb using the M5server.bat on my windows 2008 R2 64 bit server. This makes it run as a service and starts off at the server start-up. This is all nice and acceptable. The problem comes when I need to restart the monetdb server. The only way I was able to do this was to kill the process (mServer5.exe) since I was not able to figure out the service (in the windows services) which starts monetdb. So my simple question is how to restart monetdb server if it gets installed by M5server.bat? Can somebody please help me with the queries? Regards, Anshuman _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

1 0

Indexing in monetdb
by Anshuman Kumar 26 Jun '13

26 Jun '13

Hello everyone on the mailing list, Please find some time to go through my queries. I created a table with some fields out of which 1 is timestamp. I wanted to have indexing on this table. While going through the available documentation for monetdb, I found out that it creates its own indexing and might ignore user provided indexing. I went ahead and started pushing in data into the table with single inserts (10000 of them accumulated in a transaction). My table is pretty big now and it contains about 250 million records (over 2 months of data). Even simple queries to find records between specified timestamps takes a lot of time(12-13 mins in some cases) We really feel the need of indexing on the table(don't know if that can help). I happened to stumble upon an article on monetdb mailers: http://mail.monetdb.org/pipermail/users-list/2008-March/001726.html According to this article if I insert data into the table using - INSERT INTO table_name (...) ORDER BY X,Y. The order by here will enforce indexing on X, Y. I would like to confirm whether this really enforce indexing? I have a few other questions pertaining to my case: 1) Is there a predefined way of speeding up such queries in monetdb? 2) If the insert with an order by clause is a work around then what if going further, I start inserting with the insert query with the order by clause. a)Will that enforce indexing on the whole table or on only those records which were inserted using such inserts? b) Or should I just move the data from my table with 250 million records into another table using the insert with order by clause? 3)One more question regarding the setup of monetdb - I have initiated monetdb using the M5server.bat on my windows 2008 R2 64 bit server. This makes it run as a service and starts off at the server start-up. This is all nice and acceptable. The problem comes when I need to restart the monetdb server. The only way I was able to do this was to kill the process (mServer5.exe) since I was not able to figure out the service (in the windows services) which starts monetdb. So my simple question is how to restart monetdb server if it gets installed by M5server.bat? Can somebody please help me with the queries? Regards, Anshuman

1 0

Strange behavior
by Victor Almeida 25 Jun '13

25 Jun '13

Hi all, I'm using monetdb on two servers and meet some strange behavior. The two servers are of the same configuration as the following: CPU: Intel(R) Xeon(R) CPU E5-2430L, 6 Cores, 12 Threads, 2 GHz, 64 bits. Memory: 64GB OS: Ubuntu Server 13.04 with 3.8.0-25-generic x86_64 kernel MonetDB: v11.15.7 And they are both bought for less than 4 months. The problem is that the performance of MonetDB is very unstable, even for very simple select * queries. The experiment setting is as the following: Data: Server1: Table1 loaded from a 19GB CSV file (CSV1). Server2: Table1 loaded from another 19GB CSV file (CSV2). CSV1 and CSV2 have the same schema but have different contents. But they both have size of about 19GB and both have 155000000 tuples. The schema is: Field1 VARCHAR(16), Field2 VARCHAR(100), Field3 DATE, Field4 FLOAT, Field5 VARCHAR(64), Field6 CHAR(3), Field7 CHAR(6), Field8 VARCHAR(32), Field9 INT In addition, both the tables have indices on Field1, Field2 and Field3. Experiment Query: echo -e '\\w-1 \n \\> /dev/null \n select * from Table1;' | mclient -d expdatabase --interactive The problem is that the performance varies a lot between the servers and also between different runs on the same server. Here is the result: Server1: 4m 12s Server2: 305m 29s The system conditions during the experiment: Disk speed: I run the command: cat CSV > /dev/null. The speed is normal, about 2m ~ 2m 30s . System load: The machine load during the query executions are both very low. The CPUs are almost completely idle and free memories are abundant. Each monetdb server took about 50G. To eliminate possible system configuration problems. I did two more experiments. 1) I move CSV2 to server1, stop the server1 monetdb and build and start a new monetdb database using CSV2. Then I run the same query again on the new database. The time elapse is 134m 8s. 2) After 1) is done I stop the new monetdb and restart the old monetdb on server1, and run the same query again. This time the time elapse becomes 388m 37s (Comparing 4m 12s in the first run). Further disk observations: During the above two experiments, I monitor the disk read speed of monetdb using iotop. The speed keeps about only 500 ~ 550 k/s. While the read speed of cat Table1.csv > /dev/null keeps around 135m/s. I did not do any configuration about monetdb. Is it possible that I misconfigured something? Thanks. Victor

2 1

broken link
by Anthony Damico 23 Jun '13

23 Jun '13

on http://www.monetdb.org/Home the text below links to a non-existent page :) *The Researcher's Guide to the Data Deluge: Querying a Scientific Database in Just a Few Seconds<http://repository.cwi.nl/search/searchrepository.php?id=18546> .* not sure if it should link to http://oai.cwi.nl/oai/asset/18546/18546B.pdf

2 1

User defined aggregation function (SQL)
by Sebastian Dorok 20 Jun '13

20 Jun '13

Hi all, I am evaluating MonetDB in the context of genomic data. Therefore I'm trying to implement a custom aggregation function. To get a feeling for the system I set up a UDF that just returns identical values. Now I'm trying to implement this simple function as aggregate function for groupings but always run into an error when executing this: select grpident(base) from test.bases group by pos; "TypeException:user.s1_1[18]:'udf.subident' undefined in: _30:bat[:any,:str] := udf.subident(_15:bat[:oid,:str], _27:bat[:oid,:oid], r1_27:bat[:oid,:oid], _1:bit) program contains errors" Here is the MAL plan: +-----------------------------------------------------------------+ | mal | +=================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_3:bat[:oid,:oid] := sql.tid(X_2,"test","bases"); | | X_6 := sql.bind(X_2,"test","bases","base",0); | | (X_9,r1_9) := sql.bind(X_2,"test","bases","base",2); | | X_12 := sql.bind(X_2,"test","bases","base",1); | | X_14 := sql.delta(X_6,X_9,r1_9,X_12); | | X_15 := algebra.leftfetchjoin(X_3,X_14); | | calc.str("bases"); | | calc.str("base"); | | X_18 := sql.bind(X_2,"test","bases","pos",0); | | (X_20,r1_20) := sql.bind(X_2,"test","bases","pos",2); | | X_22 := sql.bind(X_2,"test","bases","pos",1); | | X_23 := sql.delta(X_18,X_20,r1_20,X_22); | | X_24 := algebra.leftfetchjoin(X_3,X_23); | | calc.str("bases"); | | calc.str("pos"); | | (X_27,r1_27,r2_27) := group.subgroupdone(X_24); | | X_30:bat[:any,:str] := udf.subident(X_15,X_27,r1_27,true); | | calc.str("bases"); | | calc.str("L1"); | | calc.str("bases"); | | calc.str("L1"); | | X_36 := sql.resultSet(1,1,X_30); | | sql.rsColumn(X_36,"test.bases","L1","clob",1,0,X_30); | | X_40 := io.stdout(); | | sql.exportResult(X_40,X_36); | | end s1_1; | +-----------------------------------------------------------------+ I'm using following setup: MonetDB 5 server v11.15.7 "Feb2013-SP2" (64-bit, 64-bit oids) Found 1.9GiB available memory, 2 available cpu cores Libraries: libpcre: 8.31 2012-07-06 (compiled with 8.31) openssl: OpenSSL 1.0.1c 10 May 2012 (compiled with OpenSSL 1.0.1c 10 May 2012) libxml2: 2.9.0 (compiled with 2.9.0) Here's what my related code looks like: udf.c ##### str GRPident(int *ret, int *bid, int *eid){ ... } udf.h ##### udf_export str GRPident(int *ret, int *bid, int *eid); udf.mal ####### module udf ... command ident(b:bat[:oid,:str], e:bat[:oid,:str]) :bat[:oid,:str] address GRPident comment "identity of groups"; udf.sql ####### create aggregate grpident(src string) returns string external name udf.ident; I already renamed the MAL command to "subident" and changed the SQL expression accordingly without success. Furthermore I already had a look at "batxml.c" as suggested here: http://old.nabble.com/users-list-Digest,-Vol-3,-Issue-30-td34736774.html but don't see what I'm doing wrong. What do I have to do to implement a custom aggregate function? Thanks, Sebastian

2 3

Support for simultaneous queries / read locking
by Mark Hansen 13 Jun '13

13 Jun '13

We are looking to use MonetDb as the backend of a multi-user BI system. Before we get too far down the road with development, I'm wondering how MonetDb will perform with 10, 50, 100, 1000 simultaneous users? We are not concerned about writes - just reads, as we plan to do updates as a nightly batch process when we can tolerate a small window of performance downgrades on the queries. Is there any read-locking that limits concurrent queries from accessing the same data? If so, is it at the column level, or finer grained? Does anyone have experience using MonetDb for handling lots of concurrent queries? Thanks in advance for any info. -- Mark

2 1

Import data - MonetDB
by Raquel Ribeiro 13 Jun '13

13 Jun '13

Hi, I am trying do bulk load like I saw here http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/LoadingBulkData , but I am not having success when doing it. I am trying to bulk load more than 60k registers using the following command: > mclient -d adventurework -s "COPY INTO FactInternetSales FROM STDIN USING DELIMITERS ',', '\n', '\"'" - < C:Users\dir\text.txt After running this command I do not receive any error message but I cannot see the data in the database either. What am I missing here? Am I trying to this in the wrong way? PS: I also tried with a smaller file but in the end I have the same result. I attached my partial data file in case of being necessary and the create of this table. And the system is Windows. Thanks for your attention. Best regards, -- Raquel Ribeiro

3 4

question about concurrent read/write
by Roberto Cornacchia 13 Jun '13

13 Jun '13

Hi there, I'm trying to chase the cause of a problem with concurrent reads/writes, that shows up as either a "COMMIT FAILED" error or an assertion (delta_append_bat: Assertion `!c || ((c)->U->count) == bat->ibase' failed), probably depending timing issues. The scenario is roughly: two threads, one reading, one writing. Sometimes, they need to read/write from/to the same table. I'm aware that this may raise issues, but I'd like to get straight what the real cause is (and possibly what the cause of the assertion is). Imagine two JDBC connections, R (read only) and W (write only), in the following timeline: TS = transaction start TC = transaction commit TR = transaction trollback CO = cursor open C++ = increment cursor (read one row) IN = insert R | W _______________ 1| | TS 2| TS | 3| CO | IN 4| C++ | 5| | TC 6| | TS 6| C++ | IN 7| TR | 8| | TC The question is: is this supposed to fail or to succeed? There are no UPDATE in W, only INSERT INTO (actually COPY INTO). I tried a similar scenario with 2 mclient sessions, continuously reading from R (without cursors) and writing and committing from W. R kept nicely showing no updates, until the ROLLBACK. Then, it showed the updates from W. But I'm not sure whether the scenario above with cursors should behave the same. This is using very up to date Feb2013 branch. I guess the assertion is a bug, and I'd like to file it, but at the moment it's quite hard to reproduce consistently, perhaps it would be easier if I could get a better understanding of the scenario above. Thanks, Roberto

2 1

Building MonetDB on Windows (x64)
by Scott Mathieson 12 Jun '13

12 Jun '13

Hi, I am trying to build the latest sources on Windows (x64) and I'm having problems getting the compilation to work, specifically the code in Mx does not seem to be fully compliant with the Windows VC compiler (the use of __attribute__ in the MxFcnDef.h file to be a case in point although there are lots of other examples of gcc specific syntax in this code). I believe I have followed the instructions in windowsbuild.rst but to no avail. Am I missing something - is Mx.exe required for the build and if so, can it be built with VS2010? Thanks, Scott ________________________________

1 0

JDBC driver does not return stored procedure information.
by Thomas Kellerer 11 Jun '13

11 Jun '13

Hello list, while adding support for MonetDB to my tool "SQL Workbench/J" (http://www.sql-workbench.net) I noticed that the calls to DatabaseMetaData.getProcedures() DatabaseMetaData.getProcedureColumns() will always return an empty ResultSet even if there are stored procedures or functions in the database. Are there any plans to fix this bug? If yes, what will be the timeframe for that? Regards Thomas Kellerer

1 0

Inconsistency in select
by Flavio Coelho 10 Jun '13

10 Jun '13

Dear list, We have found this strange inconsistency when selecting from data. Consider the two following select clauses: in the first one with a "where" filter, we notice a number of missing data on the resulting table (hint: they are not really missing in the database). However if we look to any of the specific lines which appear to have missing data, as done in the second select, we confirm that the data is there (not missing). What is going on? Am I doing anything wrong? Is this a bug of the mclient? the version I am running is this: MonetDB Database Server Toolkit v1.1 (Feb2013-SP2) sql>select * from notas_c where instalacao=412307404; +-------------+-----------+------------+------------+------+--------------+------+------------------------------+------------+-----------+--------------------------------------------------+ | id | instalaca | data_de_cr | data_do_en | tipo | descricao_do | codi | descricao_da_codificacao_da_ | data_fim_d | status_do | descricao_do_status_do_usuario | : : o : iacao_da_n : cerramento : _de_ : _tipo_de_not : fica : nota : a_avaria : _usuario : : : : : ota : _da_nota : nota : a : cao_ : : : : : : : : : : _de_ : : da_n : : : : : : : : : : serv : : ota : : : : : : : : : : ico : : : : : : : +=============+===========+============+============+======+==============+======+==============================+============+===========+==================================================+ | 35004675 | 412307404 | 2007-10-22 | 2011-06-14 | AF | | AF01 | | 2007-10-30 | NPRO ARQU | Laudo Válido Improcedente//Enviado via Interface | | 10009362195 | 412307404 | 2009-10-28 | 2009-10-31 | AL | | AL06 | | 2009-10-31 | VIRE EXEC | Vistoria Reprovada//Executada no Campo | | 10009372721 | 412307404 | 2009-10-29 | 2009-11-17 | AG | | AG22 | | 2009-11-17 | PRO | Procedente | | 10009455607 | 412307404 | 2009-11-10 | 2009-11-17 | AD | | AD05 | | 2009-11-16 | EXEC | Executada no Campo | | 10010449356 | 412307404 | 2010-03-30 | 2010-08-11 | AJ | | AJ11 | | 2010-08-11 | EXEC | Executada no Campo | | 10010832683 | 412307404 | 2010-05-21 | 2010-05-24 | AD | | AD05 | | 2010-05-24 | REJE | Rejeitada | | 10011114747 | 412307404 | 2010-06-29 | 2010-06-30 | AG | | AG44 | Serviço fora do prazo | 2010-06-30 | IMP | Improcedente | | 10011308855 | 412307404 | 2010-07-22 | 2010-07-28 | AD | | AD05 | BO - Manutenção na lente | 2010-07-28 | EXEC | Executada no Campo | | 10011475191 | 412307404 | 2010-08-11 | 2010-08-16 | AB | | AB08 | AB08 Auc c/subst c ramal Tri | 2010-08-16 | EXEC | Executada no Campo | | 10012702114 | 412307404 | 2011-01-05 | 2011-01-10 | A3 | | A375 | Corte MCV exec na rede | 2011-01-07 | EXEC | Executada no Campo | | 10014234776 | 412307404 | 2011-06-14 | 2011-06-15 | A3 | | A375 | Corte MCV exec na rede | 2011-06-15 | EXEC | Executada no Campo | | 10014271409 | 412307404 | 2011-06-15 | 2011-06-16 | A5 | Religação BT | A513 | Religa Normal-Exec Rede | 2011-06-16 | EXEC | Executada no Campo | +-------------+-----------+------------+------------+------+--------------+------+------------------------------+------------+-----------+--------------------------------------------------+ 12 tuples (0.9s) sql>select * from notas_c where id=35004675; +----------+-----------+------------+------------+------+--------------------+------+--------------------------------+------------+-----------+---------------------------------------------+ | id | instalaca | data_de_cr | data_do_en | tipo | descricao_do_tipo_ | codi | descricao_da_codificacao_da_no | data_fim_d | status_do | descricao_do_status_do_usuario | : : o : iacao_da_n : cerramento : _de_ : de_nota : fica : ta : a_avaria : _usuario : : : : : ota : _da_nota : nota : : cao_ : : : : : : : : : : _de_ : : da_n : : : : : : : : : : serv : : ota : : : : : : : : : : ico : : : : : : : +==========+===========+============+============+======+====================+======+================================+============+===========+=============================================+ | 35004675 | 412307404 | 2007-10-22 | 2011-06-14 | AF | VerifTensão Fornec | AF01 | Verif Tensão Fornecimento - BT | 2007-10-30 | NPRO ARQU | Laudo Válido Improcedente//Enviado via Inte | : : : : : : : : : : : rface : +----------+-----------+------------+------------+------+--------------------+------+--------------------------------+------------+-----------+---------------------------------------------+ 1 tuple (325.965ms) -- Flávio Codeço Coelho ================ +55(21) 3799-5567 Professor Escola de Matemática Aplicada Fundação Getulio Vargas Praia de Botafogo, 190 sala 312 Rio de Janeiro - RJ 22250-900 Brasil

4 8

disable foreign key checks
by Gijs Molenaar 10 Jun '13

10 Jun '13

Hi monetdb people, I'm doing some work again on Djonet, implementing missing features. One of the features is to flush the content of a set tables e.g. remove all rows. The problem is that the order the tables are processed is important in the case of foreign key relations. They way this is implemented in other database drivers for django is to temporarily disable foreign key relations (mysql) or to TRUNCATE multiple tables in one statement (postgresql). Is there an (undocumented) way of accomplishing something similar? I prefer not to temporarily alter the table. Thanks, -- Gijs Molenaar http://www.astro.uva.nl/people/gijs-molenaar/

3 4

gdk_heap
by Hassan, Ahmad 06 Jun '13

06 Jun '13

Hi, I have read the gdk_heap code in detail to understand whether MonetDB reuses the same virtual address range over and over again for multiple allocations. From my analyses, I found out the following: 1) HEAPalloc is used for memory allocation. If requested size is under GDK_mmap_minsize then STORE_MEM (GDKmalloc|malloc) is used for allocation Otherwise STORE_MMAP (anonymous mmap) or STORE_PRIV (DISK mmap) is used 2) HEAPfree use HEAPcacheAdd to return the previously allocated virtual address space to the cache whose size is fixed at 5 elements. So this means only these 5 address ranges will be used over time and their size would be extended using HEAPextend 3) Apart from those 5 HEAPcache entries, MonetDB doesn't use any reallocation of same virtual address range for multiple objects but just relies on operating system to handle the allocation and de-allocation through malloc/mmap. Does this mean that MonetDB just keeps only 5 HEAPcache entries to reduce the overhead of mmap calls and manage those virtual addresses by itself for future allocations? [HEAP_CACHE_SIZE = 5] I understand that this is very simplistic analyses of what MonetDB actually does in gdk_heap but any information on the gdk_heap would be really helpful. Thanks. Kind Regards, Ahmad

1 0

which version of monetdb works with cygwin ?
by Kaniska_Mandal＠DELL.com 06 Jun '13

06 Jun '13

Hi, Which linux flavor of MonetDB (Fedora / FreeBSD / Debian) can I run in Cygwin ? If it's not possible to run a linux version of monetdb in Cygwin, then what are the steps to compile monetdb src into Cygwin ? Thanks, Kaniska

2 1

mserver5 asserts under valgrind (callgrind)
by Stephen P. Morgan 06 Jun '13

06 Jun '13

I ran valgrind –tool=callgrind on mserver5 while executing TPCH Q1 (in SQL) on a scale factor 100 database on a dual-processor, x86_64 linux server. The program failed due to an assertion in DELTAproject, namely assert(tres) on line 3614 in sql.mx. Without valgrind, the query completes correctly when mserver5 is run from a similar command line. What appears below is the valgrind program output. Steve Morgan ==1800== Callgrind, a call-graph generating cache profiler ==1800== Copyright (C) 2002-2011, and GNU GPL'd, by Josef Weidendorfer et al. ==1800== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info ==1800== Command: /usr/local/bin/mserver5 --dbpath=/home/ubuntu/monet/dbfarm/tpch --set merovingian_uri=mapi:monetdb://SM-MoDB3:50000/tpch --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/ubuntu/monet/dbfarm/tpch/.mapi.sock --set monet_vault_key=/home/ubuntu/monet/dbfarm/tpch/.vaultkey --set gdk_nr_threads=32 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes ==1800== ==1800== For interactive control, run 'callgrind_control -h'. # MonetDB 5 server v11.15.3 "Feb2013-SP1" # Serving database 'tpch', using 32 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked # Found 188.887 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for UNIX domain connection requests on mapi:monetdb:///home/ubuntu/monet/dbfarm/tpch/.mapi.sock # MonetDB/GIS module loaded # MonetDB/JAQL module loaded # MonetDB/SQL module loaded #GDKmalloc(116404584) fails, try to free up space [memory in use=4416764392,virtual memory in use=31869532648] #GDKmalloc(116404584) result [mem=3251134360,vm=15052332952] #GDKmallocmax: recovery ok. Continuing.. #GDKmmap(1600192512) fails, try to free up space [memory in use=8024487008,virtual memory in use=26927363168] #GDKmmap(1600192512) result [mem=5046231544,vm=20698587640] #GDKmmap: recovery ok. Continuing.. #GDKmmap(1600192512) fails, try to free up space [memory in use=5046232200,virtual memory in use=20898866312] #GDKmmap(1600192512) fails, try to free up space [memory in use=5046232200,virtual memory in use=22499058824] #GDKmmap(1600192512) result [mem=4897637384,vm=22350464008] mserver5: ../../../../MonetDB/sql/backends/monet5/sql.mx:3614: DELTAproject: Assertion `tres' failed. #GDKmmap(1600192512) result [mem=5097655904,vm=22550482528] #GDKmmap(1600192512) result [mem=5097655904,vm=22550482528] ==1800== ==1800== Events : Ir ==1800== Collected : 142458949886 ==1800== ==1800== I refs: 142,458,949,886 ./out: line 1: 1800 Killed valgrind --tool=callgrind /usr/local/bin/mserver5 --dbpath=/home/ubuntu/monet/dbfarm/tpch --set merovingian_uri=mapi:monetdb://SM-MoDB3:50000/tpch --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/ubuntu/monet/dbfarm/tpch/.mapi.sock --set monet_vault_key=/home/ubuntu/monet/dbfarm/tpch/.vaultkey --set gdk_nr_threads=32 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes

2 1

How to create database in windows
by Kaniska_Mandal＠DELL.com 06 Jun '13

06 Jun '13

Hi, I am a monetdb newbee Once I have installed monetdb on windows, I cannot find monetdbd or monetdb commands. Can you please let me know how do I create a new database and dbfarm from command line in windows ? Thanks, Kaniska

3 4

show processlist
by Christian Braun 05 Jun '13

05 Jun '13

Hello, is there a way to get a list of the running queries / processes from a MonetDB server? Something like "show processlist" command MySQL has. Thanks, Christian Braun.

2 1

Database sampling doesn't work in new release.
by Jun Liu 05 Jun '13

05 Jun '13

MonetDB version: Feb2013-SP2 Using mclient to execute a SQL like: SELECT * FROM TEST SAMPLE 10; It will return "Connection terminated" and quit.

3 3

Question on Quicksort
by Stephen P. Morgan 05 Jun '13

05 Jun '13

We have a question about MonetDB's quicksort routine. It seems that sometimes the head and tail values in a BAT remain paired and sometimes not after passing the BAT through quicksort. For example, a BAT might have the following values before and after quicksor. (This is an artificial example.) index H T ----------------- 0 1 0 1 3 0 3 7 1 4 7 0 5 7 2 6 7 0 7 32 3 8 34 0 9 35 4 10 38 0 After MonetDB qsort index H T ----------------- 0 1 0 1 3 0 3 7 1 4 7 0 5 7 5 <----- Why is this value 5? Why not 2? 6 7 0 7 32 4 <----- Shouldn't it still be 3? 8 34 0 9 35 3 <----- Shouldn't it still be 4? 10 38 0 With some queries, the values seem to move in pairs, but in others, they don't. With some queries, the head-tail pairing seems to be maintained, but with others, it isn't. For example, the first query below apparently preserves the pairing (when run against the TPCH database), but the second query does not (and this happens consistently). select l_suppkey, l_partkey from lineitem where l_partkey > 150000 order by l_orderkey; select o_orderkey, o_custkey, o_orderstatus from orders where o_custkey > 40000 and o_custkey < 10000000 order by o_orderkey; Any light you might throw on this would be very helpful. Thanks, Steve Morgan

2 3

BAT Buffer Pool functionality
by Hassan, Ahmad 04 Jun '13

04 Jun '13

Hi, I am reading Monetdb code to understand the functionality of BAT Buffer Pool (BBP). I understood that it doesn't allocate allocate memory for actual BAT but BATs are first instantiated on the HEAP memory using BATcreatedesc, BATnewstorage where BATcreatedesc also registers the entry of new BAT into BBP. BBP only allocates HEAP memory for maintaining the bookkeeping information for the BATs and keeping the directory structure information. I understood from the code that BBP keeps looking for the HOT/COLD state of BATs and performs unloading if memory is growing or descriptors are increasing using BBPTrim. Does this mean File descriptor here? Questions: 1) I want to know if MonetDB allocates a large heap space internally then reuse that space for allocating memory chunks to BATs on demand or every time MonetDB ask operating system to extend heap size when new BAT is created? 2) I want to understand how BBP manages the HEAP memory. When a BAT is no longer needed, then what BBP does with that memory. Does it return that memory to OS and shrinks the heap size used by MonetDB or it holds that space for future allocations? 3) A typical TPCH query generates 100's of intermediate BATs. Does MonetDB allocates memory for those BATs during query execution and returns it back to operating system when query finishes? 4) When we start MonetDB then how much initial heap memory is allocated by the MonetDB system. Does BBP allocates this memory on the HEAP? If not, then which part of the code knows which BATs to load at the start time and how much HEAP memory needs to be allocated for those BATs. I found the list of all BATs in Monetdb through bbp.get() mal call. I also understand that SQL catalog maintains the mapping of table to column names and column names to BAT_in_BBP mapping. But I couldn't able to find which BAT I should access through mal code to get the sql catalog mappings. I want to use MAL to see which table is composed of which column is linked to which BAT in BBP. Thanks for the help. Kind Regards, Ahmad

2 3

Possible to reset or flush out RAM usage in Windows without closing and re-opening mserver.exe?
by Anthony Damico 23 May '13

23 May '13

Hi, I'm running a two-day-long program on Windows x64 with MonetDB.R that breaks somewhere into the second day due to what I believe is overloaded RAM? I was under the impression that this should not really happen with MonetDB, since it's a database that doesn't load everything into RAM.. Here's what's happening: A big, complicated series of importations, merges, and recodes all work for the first two days on about 250,000,000 records of data.. but all the while my Windows Task Manager will slowly show the physical memory usage meter ticking up in the "Performance" tab. mserver.exe does NOT appear to be using RAM in the "Memory" column of the "Processes" tab, so it looks like there's a ghost using RAM. Then, after two days, the script will break (mserver output shown below) - returning a GDK error on a seemingly-benign line of code, just a simple recode: Error in .local(conn, statement, ..., async = async) : UPDATE ep10 SET rx_cso_cov = CVRD_D_PLAN_PD_AMT WHERE DRUG_CVRG_STUS_CD IN ( 'C','S','O' ) failed! Server says: !GDKerror:!ERROR: GDKmallocmax: failed for 589484032 bytes After that, if I close mserver.exe and open it up again, it takes about twenty minutes to get the ">" mserver prompt. But it does open up eventually.. and at that point, if I re-connect to the server with R and run the exact same line, it works without error. So it appears I just need to flush out that ghost/mystery RAM usage somehow? Does this error make sense? There is no way this is due to lack of hard disk space - this server's C drive has 15GB free and the attached drive actually hosting all of this data has 235GB free. I don't want to rule it out, but I also doubt this problem is related to MonetDB.R, since the scenario I described does not require that I restart my R console, just mserver.exe. I can work around the problem by manually killing and re-starting mserver.exe at a few logical breakpoints in my program, but I didn't think this was supposed to happen in the first place? Thanks!!! Anthony # MonetDB 5 server v11.15.7 "Feb2013-SP2" # Serving database 'medicare_sample', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 20.000 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:49800/ # MonetDB/JAQL module loaded # MonetDB/SQL module loaded ># SQL catalog created, loading sql scripts once # loading sql script: 09_like.sql # loading sql script: 10_math.sql # loading sql script: 11_times.sql # loading sql script: 12_url.sql # loading sql script: 13_date.sql # loading sql script: 14_inet.sql # loading sql script: 15_history.sql # loading sql script: 16_tracelog.sql # loading sql script: 17_compress.sql # loading sql script: 18_dictionary.sql # loading sql script: 19_cluster.sql # SQL catalog created, loading sql scripts once # loading sql script: 09_like.sql # loading sql script: 20_vacuum.sql # loading sql script: 10_math.sql # loading sql script: 11_times.sql # loading sql script: 12_url.sql # loading sql script: 21_dependency_functions.sql # loading sql script: 13_date.sql # loading sql script: 14_inet.sql # loading sql script: 15_history.sql # loading sql script: 16_tracelog.sql # loading sql script: 17_compress.sql # loading sql script: 18_dictionary.sql # loading sql script: 19_cluster.sql # loading sql script: 20_vacuum.sql # loading sql script: 21_dependency_functions.sql # loading sql script: 22_clients.sql # loading sql script: 23_skyserver.sql # loading sql script: 24_zorder.sql # loading sql script: 25_debug.sql # loading sql script: 22_clients.sql # loading sql script: 23_skyserver.sql # loading sql script: 39_analytics.sql # loading sql script: 24_zorder.sql # loading sql script: 25_debug.sql # loading sql script: 39_analytics.sql # loading sql script: 75_storagemodel.sql # loading sql script: 75_storagemodel.sql # loading sql script: 80_udf.sql # loading sql script: 80_udf.sql # loading sql script: 99_system.sql # loading sql script: 99_system.sql #GDKmalloc(589484032) fails, try to free up space [memory in use=71647756904,vir tual memory in use=72723595880] #GDKmalloc(589484032) result [mem=71615637640,vm=72198121608]

2 2

Fwd: Using the inet and url type functions
by Christopher Nelson 23 May '13

23 May '13

Hi, I noticed in the manual that there are many useful functions for the inet and url types. For example, I would like to convert an inet column to text for use in a Python script (since the python-monetdb module does not support the inet type.) I tried: sql>SELECT wlid, text(src), ts FROM web_log WHERE src IS NOT NULL ORDER BY src, ts LIMIT 5;syntax error, unexpected '(', expecting SCOLON in: "select wlid, text(" sql>SELECT wlid, inet:text(src), ts FROM web_log WHERE src IS NOT NULL ORDER BY src, ts LIMIT 5; unexpected symbol (:) sql>SELECT wlid, inet.text(src), ts FROM web_log WHERE src IS NOT NULL ORDER BY src, ts LIMIT 5; syntax error, unexpected '(', expecting SCOLON in: "select wlid, inet.text(" I have read through the manual sections on types and functions several times now, and I cannot see how to use the specified functions on these types. Any help would be greatly appreciated. Thanks! -={C}=-

2 8

Using something like "trace" within the MAL interpreter
by Masood Mortazavi 20 May '13

20 May '13

Hi, The "trace" commands works well when mclient is in the SQL interpreter mode. How about when mclient is in MAl interpreter mode? Is there a way to use the "trace" command in this mode? Best regards, Masood

3 6

support for non-median quantile functions?
by Anthony Damico 15 May '13

15 May '13

Hi, would it be possible to expand the median() function to be expanded work on any quantile? Martin's original request did say quantile, not just median :D http://osdir.com/ml/db.monetdb.bugs/2008-06/msg00206.html Given the MonetDB.R connection, I think a lot more statisticians are going to be using MonetDB -- and they'll find a lot of value in being able to calculate statistics like the 25th and 75th, percentiles quickly. I can submit a bugs.monetdb.org report if that's appropriate? Thanks for considering this!!

2 2

Re: Help for MonetDB source code (SQL Module)
by Stefan Manegold 14 May '13

14 May '13

Dear Khaled, for questions about MonetDB, I kindly ask you to use the respective mailing list(s); I Cc my answer to the users list. See also http://www.monetdb.org/Developers/Mailinglists On Mon, May 13, 2013 at 06:41:45PM +0200, Khaled Dehdouh wrote: > Dear sir, > > I am a student at university of LYON. Concerning MonetDB, I need your help, > please on some points. > > > - I want to do some optimizations in "GROUP BY" function of SQL module. > For this end, can you show me where is exactly the function code > responsible to perform the "Group By" operation on MonetDB source > codein order to modif > y it. I'm wondering how you know that "optimizations" are necessary/possible without knowing the actual code, but I guess you know. You find the respective code in gdk/gdk_group.c > - I would like to know also how to recompile the code after modification > of the "GROUP BY" function. See http://www.monetdb.org/Developers/SourceCompile http://dev.monetdb.org/hg/MonetDB/file/tip/HowToStart.rst http://dev.monetdb.org/hg/MonetDB/file/1272d67f24be/buildtools/doc/windowsb… > - How creating a new SQL command that performs function with parameters and > combines two existing functions in MONETDB. See http://www.monetdb.org/Documentation/SQLreference/Modules or your favorite SQL:2003 documentation. MonetDB is SQL standard compliant. Best, Stefan > Thanks in advance. > > Best regards, > Khaled DEHDOUH. -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

GROUP BY
by Bouslah Ahmed 14 May '13

14 May '13

Hello, I am a student at a university in France. I work on MonetDB, Windows platform. I have some questions: I want to optimize the "Group by" function of the SQL module in order to create a new function combining the "Group by" function and another function. For this I want to know: 1 - How to locate the class containing the code of the "Group by" function 2 - How to edit the code in question to change it 3 - Which editor should be used to compile the project. 4 - I tried to use "Visual Studio 2012" to open the project "MonetDB" but I have not managed to do. Regards Ahmed Bouslah.

3 2

Question on installing extensions
by Kaniska_Mandal＠DELL.com 12 May '13

12 May '13

Hi All, How to install extensions for Streams, XML and Json support ? Many thanks, Kaniska

2 1

TINYINT display problems in 3rd party query clients
by Diethard Steiner 10 May '13

10 May '13

Hi, I am wondering why TINYINT values are displayed with different values in 3rd party SQL clients like Squirrel (please see attached screenshot): So 1 is shown as 49 in example. I used following JDBC driver version: monetdb-jdbc-2.8.jar Any ideas? Thanks, Diethard

3 5

Feb2013-SP2 mserver loading time much longer than Feb2013 on Windows x86_64
by Anthony Damico 03 May '13

03 May '13

Hi, after upgrading to MonetDB Feb2013-SP2, I've noticed a huge increase in the loading time for the server to get the ">" shown in the last line below (indicating it's loaded and ready for mclient connections). It now takes between three and ten minutes. Has anything changed in the Windows version of the server between Feb2013 and Feb2013-SP2 that might cause this? Once loaded, the queries are still lightning-fast, but I'm debating whether I ought to re-create all of my databases from scratch using Feb2013-SP2, if there's any reason doing that might help? Any clues? :) Thanks!!! # MonetDB 5 server v11.15.7 "Feb2013-SP2" # Serving database 'medicare_sample', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 20.000 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:49800/ # MonetDB/JAQL module loaded >

1 0

question about arrays in SciQL
by Gabor Urbanics 03 May '13

03 May '13

Hi All, I have a question about MonetDB/SciQL, in particular regards to the unbounded arrays and their updates. I've created two arrays (one bounded and an unbounded) essentially with the same schemata in my database as follows: """ sql>\d ARRAY sys.facts ARRAY sys.facts_bounded sql>\d sys.source CREATE TABLE "sys"."source" ( "x" INTEGER, "y" INTEGER, "v" VARCHAR(32) ); sql>\d sys.facts CREATE ARRAY "sys"."facts" ( "x" INTEGER DIMENSION [0:1:1], "y" INTEGER DIMENSION [0:1:*], "v" VARCHAR(32) ); """ My goal at the moment is to use the unbounded array as it would fit very well the domain of my application: several coordinates-like columns and a couple of attributes, but the ranges of the coordinates are not known beforehand. Unfortunately, I am facing some problems when I try to insert values into the unbounded array: """ sql>insert into sys.facts values (0,0,'42'); 0 affected rows (0.938ms) """ On the other hand, the same insert operation works just fine with the bounded array: """ sql>insert into sys.facts_bounded values (0,0,'42'); 1 affected row (20.594ms) sql>select * from sys.facts_bounded; +------+------+------+ | x | y | v | +======+======+======+ | 0 | 0 | 42 | | 0 | 1 | null | ... (5 lines omitted for brevity) | 1 | 3 | null | +------+------+------+ 8 tuples (0.612ms) """ I would like to know if this is the supposed way one can insert values into unbounded arrays or I am doing something wrong. I compiled the source from the branch "sciql" from the mercurial repository 2 days ago, as my understanding was that the built versions do not support SciQL. Is there any other way I can get a MonetDB with SciQL support? If it helps anything, the beginning of the welcome message of the mserver5 is as follows: """ # MonetDB 5 server v11.16.0 # This is an unreleased version # Serving database 'monetdb5', using 4 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked """ Any help or suggestion is appreciated! Best regards, Gabor

2 3

column alias combined with ORDER BY returns wrong data
by Budulinku Dejmihrasku 03 May '13

03 May '13

Hi, please note the test cases below - it should be self-explanatory i am on MonetDB v11.15.7 (Feb2013-SP2), i run the test before the upgrade with the same results please advice thanks a lot milan *test case 1, no ORDER BY - correct** **----------------------------------* WITH SAWITH0 AS ( select 'a' as c1, 'b' as c2, 'c' as c3, 1 as c4 ) select 0 as c1, D1.c1 as c2, D1.c2 as c3, D1.c3 as c4, D1.c4 as c5 from SAWITH0 D1 +------+------+------+------+------+ | c1 | c2 | c3 | c4 | c5 | +======+======+======+======+======+ | 0 | a | b | c | 1 | +------+------+------+------+------+ *test case 2, add ORDER BY - wrong result (repeats the 1st value for all columns)* *--------------------------------------------------------------------------------* WITH SAWITH0 AS ( select 'a' as c1, 'b' as c2, 'c' as c3, 1 as c4 ) select 0 as c1, D1.c1 as c2, D1.c2 as c3, D1.c3 as c4, D1.c4 as c5 from SAWITH0 D1 order by 5, 4, 3, 2 --order by c5,c4,c3,c2 +------+------+------+------+------+ | c1 | c2 | c3 | c4 | c5 | +======+======+======+======+======+ | 0 | a | a | a | a | +------+------+------+------+------+ Note that the result is the same even if you use the column alias instead of the ord.num. * **test case 3, use completely different alias with order by - correct ** **-------------------------------------------------------------------* WITH SAWITH0 AS ( select 'a' as c1, 'b' as c2, 'c' as c3, 1 as c4 ) select 0 as c1, D1.c1 as a2, D1.c2 as a3, D1.c3 as a4, D1.c4 as a5 from SAWITH0 D1 order by 5, 4, 3, 2 +------+------+------+------+------+ | c1 | a2 | a3 | a4 | a5 | +======+======+======+======+======+ | 0 | a | b | c | 1 | +------+------+------+------+------+

4 6

Re: should I be able to CAST a BOOLEAN type to a DOUBLE ( zero-one ) variable?
by Stefan Manegold 02 May '13

02 May '13

cast syntax is "cast( <expression> as type )" Anthony Damico <ajdamico(a)gmail.com> wrote: I tried lots of different things (pasted below) and none of them worked. I ended up using a CASE statement which worked around the issue, but I would've thought something like the command.. SELECT ( ( COLNAME > 0 )*1.000 ) AS ANY_VALUE FROM TABLENAME ; ..should give a valid response instead of an error? But I'm not sure if MonetDB wants to support this type of CAST/CONVERT operation? (None of the things I tried worked, not just the on-the-fly multiplication by 1.000). Using Feb2013-SP2 on Windows x86_64 If MonetDB should be able to convert TRUE/FALSE values to numerics, I'll submit a bug report with some PLAN and EXPLAIN results? :) Thanks for the great software!! sql>select ( mpg > 10 ) from mtcars ; +-------+ | >_mpg | +=======+ | null | | null | | true | | null | | true | | null | | true | | true | | true | | null | | null | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | null | | true | | true | +-------+ 32 tuples (15.010ms) sql>select ( mpg > 10 )*1.000 from mtcars ; TypeException:user.s11_2[15]:'calc.sht' undefined in: _59:any := calc.sht(_57:bi t, _22:int, _23:int) TypeException:user.s11_2[17]:'bat.insert' undefined in: _62:any := bat.insert(_5 2:bat[:oid,:sht], _60:oid, _59:any) program contains errors sql>select ( ( mpg > 10 )*1.000 ) from mtcars ; TypeException:user.s12_2[15]:'calc.sht' undefined in: _59:any := calc.sht(_57:bi t, _22:int, _23:int) TypeException:user.s12_2[17]:'bat.insert' undefined in: _62:any := bat.insert(_5 2:bat[:oid,:sht], _60:oid, _59:any) program contains errors sql>select ( ( mpg >= 10 )*1.000 ) from mtcars ; TypeException:user.s13_2[15]:'calc.sht' undefined in: _59:any := calc.sht(_57:bi t, _22:int, _23:int) TypeException:user.s13_2[17]:'bat.insert' undefined in: _62:any := bat.insert(_5 2:bat[:oid,:sht], _60:oid, _59:any) program contains errors sql>select ( ( mpg >= 10 ) ) from mtcars ; +--------+ | >=_mpg | +========+ | null | | null | | true | | null | | true | | null | | true | | true | | true | | null | | null | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | null | | true | | true | +--------+ 32 tuples (103.019ms) sql>select CAST( ( mpg >= 10 ) ) as DOUBLE from mtcars ; syntax error, unexpected ')', expecting AS in: "select cast( ( mpg >= 10 ) )" sql>select CAST( ( mpg >= 10 ) ) AS DOUBLE from mtcars ; syntax error, unexpected ')', expecting AS in: "select cast( ( mpg >= 10 ) )" sql>select CAST( mpg >= 10 ) AS DOUBLE from mtcars ; syntax error, unexpected ')', expecting AS in: "select cast( mpg >= 10 )" sql>select ( CAST( mpg >= 10 ) AS DOUBLE ) from mtcars ; syntax error, unexpected ')', expecting AS in: "select ( cast( mpg >= 10 )" sql>select ( CAST( mpg >= 10 AS DOUBLE ) from mtcars ; syntax error, unexpected FROM, expecting ')' in: "select ( cast( mpg >= 10 as do uble ) from" sql>select ( CAST( mpg >= 10 AS DOUBLE ) as x from mtcars ; syntax error, unexpected AS, expecting ')' in: "select ( cast( mpg >= 10 as doub le ) as" sql>select ( CAST( mpg >= 10 ) AS DOUBLE as x from mtcars ; syntax error, unexpected ')', expecting AS in: "select ( cast( mpg >= 10 )" sql>select CONVERT( double , mpg > 10 ) as x from mtcars; syntax error, unexpected sqlDOUBLE in: "select convert( double" sql>select CAST( TRUE as double ) from mtcars ; types boolean(1,0) and double(53,0) are not equal sql>select CAST( TRUE*1.000 as double ) from mtcars ; TypeException:user.s15_2[11]:'calc.sht' undefined in: _21:any := calc.sht(_20:bi t, _22:int, _23:int) program contains errors sql>select CAST( 'TRUE' as bit ) as z from mtcars; type (bit) unknown in: "select cast( 'TRUE' as bit )" syntax error, unexpected AS in: "as" sql>select CAST( TRUE as bit ) as z from mtcars; type (bit) unknown in: "select cast( true as bit )" syntax error, unexpected AS in: "as" sql>select CAST( TRUE as double ) as z from mtcars; types boolean(1,0) and double(53,0) are not equal sql>

3 3

should I be able to CAST a BOOLEAN type to a DOUBLE ( zero-one ) variable?
by Anthony Damico 02 May '13

02 May '13

I tried lots of different things (pasted below) and none of them worked. I ended up using a CASE statement which worked around the issue, but I would've thought something like the command.. SELECT ( ( COLNAME > 0 )*1.000 ) AS ANY_VALUE FROM TABLENAME ; ..should give a valid response instead of an error? But I'm not sure if MonetDB wants to support this type of CAST/CONVERT operation? (None of the things I tried worked, not just the on-the-fly multiplication by 1.000). Using Feb2013-SP2 on Windows x86_64 If MonetDB should be able to convert TRUE/FALSE values to numerics, I'll submit a bug report with some PLAN and EXPLAIN results? :) Thanks for the great software!! sql>select ( mpg > 10 ) from mtcars ; +-------+ | >_mpg | +=======+ | null | | null | | true | | null | | true | | null | | true | | true | | true | | null | | null | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | null | | true | | true | +-------+ 32 tuples (15.010ms) sql>select ( mpg > 10 )*1.000 from mtcars ; TypeException:user.s11_2[15]:'calc.sht' undefined in: _59:any := calc.sht(_57:bi t, _22:int, _23:int) TypeException:user.s11_2[17]:'bat.insert' undefined in: _62:any := bat.insert(_5 2:bat[:oid,:sht], _60:oid, _59:any) program contains errors sql>select ( ( mpg > 10 )*1.000 ) from mtcars ; TypeException:user.s12_2[15]:'calc.sht' undefined in: _59:any := calc.sht(_57:bi t, _22:int, _23:int) TypeException:user.s12_2[17]:'bat.insert' undefined in: _62:any := bat.insert(_5 2:bat[:oid,:sht], _60:oid, _59:any) program contains errors sql>select ( ( mpg >= 10 )*1.000 ) from mtcars ; TypeException:user.s13_2[15]:'calc.sht' undefined in: _59:any := calc.sht(_57:bi t, _22:int, _23:int) TypeException:user.s13_2[17]:'bat.insert' undefined in: _62:any := bat.insert(_5 2:bat[:oid,:sht], _60:oid, _59:any) program contains errors sql>select ( ( mpg >= 10 ) ) from mtcars ; +--------+ | >=_mpg | +========+ | null | | null | | true | | null | | true | | null | | true | | true | | true | | null | | null | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | true | | null | | true | | true | +--------+ 32 tuples (103.019ms) sql>select CAST( ( mpg >= 10 ) ) as DOUBLE from mtcars ; syntax error, unexpected ')', expecting AS in: "select cast( ( mpg >= 10 ) )" sql>select CAST( ( mpg >= 10 ) ) AS DOUBLE from mtcars ; syntax error, unexpected ')', expecting AS in: "select cast( ( mpg >= 10 ) )" sql>select CAST( mpg >= 10 ) AS DOUBLE from mtcars ; syntax error, unexpected ')', expecting AS in: "select cast( mpg >= 10 )" sql>select ( CAST( mpg >= 10 ) AS DOUBLE ) from mtcars ; syntax error, unexpected ')', expecting AS in: "select ( cast( mpg >= 10 )" sql>select ( CAST( mpg >= 10 AS DOUBLE ) from mtcars ; syntax error, unexpected FROM, expecting ')' in: "select ( cast( mpg >= 10 as do uble ) from" sql>select ( CAST( mpg >= 10 AS DOUBLE ) as x from mtcars ; syntax error, unexpected AS, expecting ')' in: "select ( cast( mpg >= 10 as doub le ) as" sql>select ( CAST( mpg >= 10 ) AS DOUBLE as x from mtcars ; syntax error, unexpected ')', expecting AS in: "select ( cast( mpg >= 10 )" sql>select CONVERT( double , mpg > 10 ) as x from mtcars; syntax error, unexpected sqlDOUBLE in: "select convert( double" sql>select CAST( TRUE as double ) from mtcars ; types boolean(1,0) and double(53,0) are not equal sql>select CAST( TRUE*1.000 as double ) from mtcars ; TypeException:user.s15_2[11]:'calc.sht' undefined in: _21:any := calc.sht(_20:bi t, _22:int, _23:int) program contains errors sql>select CAST( 'TRUE' as bit ) as z from mtcars; type (bit) unknown in: "select cast( 'TRUE' as bit )" syntax error, unexpected AS in: "as" sql>select CAST( TRUE as bit ) as z from mtcars; type (bit) unknown in: "select cast( true as bit )" syntax error, unexpected AS in: "as" sql>select CAST( TRUE as double ) as z from mtcars; types boolean(1,0) and double(53,0) are not equal sql>

1 0

Understanding strides in memory accesses
by Hassan, Ahmad 01 May '13

01 May '13

Hi All, I am analysing the monetdb memory access pattern to identify various strides in LLC accesses. But most of the accesses per object are being observed as non-strided which is not what I expected. As monetdb involves BATs and these are arrays so I was thinking that there would be more strided pattern rather than random accesses unless BAT arrays are being treated as trees and random access is being used to access the items. To understand it further, I have now retrieved the execution plan of TPCH Q1 and the plan has various segments like follows: X_455:bat[:oid,:str] := sql.bind(X_7,"sys","lineitem","l_linestatus",2,5,16); X_1346 := algebra.kdifference(X_400,X_455); X_1379 := algebra.kunion(X_1346,X_455); X_1412 := algebra.leftjoin(X_1180,X_1379); (X_1452,X_1453) := group.multicolumns(X_1293,X_1412); X_1514 := bat.mirror(X_1452); X_1516 := algebra.leftjoin(X_1514,X_1412); I have two questions: 1) As there are many repeated segments like above in whole query plan, so what is the context behind multiple repeated segments please. 2) I want to understand what is the input to most common operators like leftjoin, kdifference, kunion and whether they process those inputs sequentially or randomly. What is the best way to get this understanding please? From the plan it is obvious that the input is the BAT(s), but how do the operators access the BAT is really a question. Thanks for the help. Kind Regards, Ahmad

3 3

Fwd: How to get deleted space back from monetdb
by Gaurav Meena 01 May '13

01 May '13

I have searched entire archive, and this question have been asked many times i.e. to get the space back from monetdb after deleting some rows in monetdb. The answer I found was not consistent one of them was COPY SELECT * FROM <table> INTO <file>; DELETE FROM <table>; COPY INTO <table> FROM <file>; Which suggest that deleting rows (or cleaning??) table will get back the space. One was from this question http://mail.monetdb.org/pipermail/users-list/2011-July/005036.html 1) copy all table contents to another table, drop old table, copy again I have tried both deleting rows and then droping the table but the result was inconsistant. May be monetdb cleans up after some interval Other Related Questions were 1. http://mail.monetdb.org/pipermail/users-list/2010-March/003952.html 2. FEB 23, 2010 Reeves, Matthew J. wrote: >* Hello,*>* *>* I have a MonetDB database that contains a table of rolling events.*>* There are 200 million records with 500,000 bulk loaded each day with the*>* oldest 500,000 deleted at midnight. I'm finding the database gets slower*>* as time goes on. It gets to the point of being almost unusable after a*>* week. Is there something I should be doing different?*Hello Matthew, Thanks for using MonetDB. Interesting application target. This is a known issue. The system keeps the deleted tuples and rebuilds the table. A vacuum-like feature is needed and on the wishlist. For now, taking a copy of the table would 'solve' your problem. regards, Martin >* *>* Thanks,*>* Matt*>* * Gaurav Meena

2 1

Inner join.
by Bouslah Ahmed 30 Apr '13

30 Apr '13

Hello, I'm working on MonetDB under win7, and I encountered some problems that hinder me 1 - I need to make an amendment on the inner join; 2 - To do this, I ask you to explain how the inner join works; 3 - In what part of the program (source code) is implemented the inner join. Cordially.

2 1

sql.clients function removed
by Pete Hollobon 29 Apr '13

29 Apr '13

Hi, It looks like the sql.clients() function was removed (changeset 45a4e2205363). I was wondering why, and whether there is a suitable alternative? Regards, Pete.

1 1

Announcement: New Feb2013-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 29 Apr '13

29 Apr '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>.

1 0

All tested on the same column, some functions work, some don't?
by Anthony Damico 24 Apr '13

24 Apr '13

Hi, I'm using MonetDB on Windows with the MonetDB.R connector. I'm getting some very strange errors on a particular column (called `rx_cso_num`) where *some* of the math functions work and others do not? I tested the median() function using mclient - I get the same error - so this has nothing to do with the fact that I'm connecting through R. I'm running these queries on a confidential data set, so unfortunately I cannot provide a reproducible example with this.. I'm sorry for the hassle, and hoping the error messages below might provide enough of a clue to resolve the issue? If this really isn't enough, I can send some snipped mclient -X output and/or try my best to create some fake data that also creates some of these problems? Thanks!! :) Here's a list of functions that *work* on the column without a problem: "count","distinct","min", "max", "sum","avg","abs","sqrt","floor","ceiling","exp","log","cos","sin","tan","acos","asin","atan","cosh","sinh","tanh" Here's the errors I get for each of the functions that do not: [1] "median" Error in dbGetQuery(db, paste0("select ", i, "(rx_cso_num) from x07")) : Unable to execute statement 'select median(rx_cso_num) from x07'. Server says '!22003!overflow in conversion of 133 to bte.'. [1] "sign" Error in dbGetQuery(db, paste0("select ", i, "(rx_cso_num) from x07")) : Unable to execute statement 'select sign(rx_cso_num) from x07'. Server says '!TypeException:user.s5_1[25]:'bat.insert' undefined in: _93:any := bat.insert(_83:bat[:oid,:int], _91:oid, _90:bte)'. [1] "stddev_pop" Error in dbGetQuery(db, paste0("select ", i, "(rx_cso_num) from x07")) : Unable to execute statement 'select stddev_pop(rx_cso_num) from x07'. Server says '!SELECT: no such unary operator 'stddev_pop(wrd)''. [1] "stddev" Error in dbGetQuery(db, paste0("select ", i, "(rx_cso_num) from x07")) : Unable to execute statement 'select stddev(rx_cso_num) from x07'. Server says '!SELECT: no such unary operator 'stddev(wrd)''. [1] "prod" Error in dbGetQuery(db, paste0("select ", i, "(rx_cso_num) from x07")) : Unable to execute statement 'select prod(rx_cso_num) from x07'. Server says '!22003!overflow in conversion of 133 to bte.'. Here's my mserver window: # MonetDB 5 server v11.15.3 "Feb2013-SP1" # Serving database 'medicare_sample', using 8 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 7.860 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:49800/ # MonetDB/JAQL module loaded # MonetDB/SQL module loaded > Thanks!!!

2 9

problem with foreign key constrain violation
by Gijs Molenaar 23 Apr '13

23 Apr '13

Hi MonetDB, I think I'm really confused or I found a bug. Please have a look below. I'm expecting a 'aggregation_book_contact_id_fkey' violation, since that row is clearly missing. But I'm getting a 'aggregation_book.aggregation_book_publisher_id_fkey' violation. Is it me or is this error message wrong? sql>\d sys.aggregation_book CREATE TABLE "sys"."aggregation_book" ( "id" INTEGER NOT NULL DEFAULT next value for "sys"."seq_6205", "isbn" VARCHAR(9) NOT NULL, "name" VARCHAR(255) NOT NULL, "pages" INTEGER NOT NULL, "rating" DOUBLE NOT NULL, "price" DECIMAL(6,2) NOT NULL, "contact_id" INTEGER NOT NULL, "publisher_id" INTEGER NOT NULL, "pubdate" DATE NOT NULL, CONSTRAINT "aggregation_book_id_pkey" PRIMARY KEY ("id"), CONSTRAINT "aggregation_book_contact_id_fkey" FOREIGN KEY ("contact_id") REFERENCES "sys"."aggregation_author" ("id"), CONSTRAINT "aggregation_book_publisher_id_fkey" FOREIGN KEY ("publisher_id") REFERENCES "sys"."aggregation_publisher" ("id") ); sql>select * from aggregation_publisher; +------+--------+------------+ | id | name | num_awards | +======+========+============+ | 1 | Apress | 3 | +------+--------+------------+ 1 tuple (0.588ms) sql>select * from aggregation_author; +----+------+-----+ | id | name | age | +====+======+=====+ +----+------+-----+ 0 tuples (0.570ms) sql>INSERT INTO "aggregation_book" ("id", "isbn", "name", "pages", more>"rating", "price", "contact_id", "publisher_id", "pubdate") VALUES (1, more>'159059725', 'The Definitive Guide to Django: Web Development Done more>Right', 447, 4.5, '30.00', 1, 1, '2007-12-06'); INSERT INTO: FOREIGN KEY constraint 'aggregation_book.aggregation_book_publisher_id_fkey' violated -- Gijs Molenaar http://www.astro.uva.nl/people/gijs-molenaar/

4 6

Re: Bug grouping on string column, only when results filtered
by Stefan Manegold 17 Apr '13

17 Apr '13

as string-related bug has recently been fixed (see MonetDB bug tracker; I don't recall the bug ID right now) and will be shipped with the upcoming Feb2013-SP2 release, a preview of which is avilable from the testing repository. Stefan Pete Hollobon <hollobon(a)gmail.com> wrote: Hi Hannes, Thanks for looking into this. I'm using a 64-bit build. Here's my uname and the versions from the MonetDB binaries. ~% uname -a Linux lx1 3.5.0-17-generic #28-Ubuntu SMP Tue Oct 9 19:31:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ~% /usr/bin/monetdbd version MonetDB Database Server v1.7 (Feb2013-SP1) ~% /usr/bin/monetdb version MonetDB Database Server Toolkit v1.1 (Feb2013-SP1) ~% /usr/bin/mclient --version mclient, the MonetDB interactive terminal (Feb2013-SP1) support for command-line editing compiled-in character encoding: UTF-8 However, I realised that my local Feb2013 build was using the shared libraries from the Ubuntu debs. I fixed this, and that build now seems to be working properly. I can still reproduce it with the distributed binaries though. Whatever bug it was must have been fixed already - apologies for wasting your time. Regards, Pete. On 17 April 2013 12:56, Hannes Mühleisen <Hannes.Muehleisen(a)cwi.nl> wrote: > Hello Pete, > > thank you for your bug report and the effort you have put into making this > reproducible for us. However, having followed your instructions with both > the Feb2013 version as well as the current development branch, I was not > able to also provoke this issue. > > I got the following results (Both Feb2013 and default branch): > > ➜ ~ mclient bug -s "select a, count(*) from test where b = 20227 group by > a" > +------+--------+ > | a | L1 | > +======+========+ > | 2T | 193058 | > | 1T | 202267 | > | 5T | 87038 | > | 4T | 100302 | > | 3T | 142326 | > | 15T | 95 | > | 20T | 95 | > | 30T | 14277 | > | 7T | 767 | > | 6T | 22863 | > > | 10T | 127 | > | 6V | 216594 | > | 9T | 104 | > | 8T | 88 | > +------+--------+ > 14 tuples > > ➜ ~ mclient bug -s "select a, count(*) from test group by a" > > +------+---------+ > | a | L1 | > +======+=========+ > | 2T | 5799418 | > | 1T | 6072825 | > | 5T | 2605374 | > | 4T | 2998157 | > | 3T | 4262607 | > | 15T | 2703 | > | 20T | 2818 | > | 30T | 497657 | > | 7T | 22460 | > | 6T | 686789 | > | 10T | 3496 | > | 6V | 6550540 | > | 9T | 1853 | > | 8T | 1639 | > +------+---------+ > 14 tuples > > > Can you perhaps give more information abuont your environment? For example, > this a 64 or 32 bit build? You could also try the development default branch > and see if the problem also occurs there. > > Best, > > Hannes > > > > > > On 17/04/13 10:33, Pete Hollobon wrote: >> >> Hi, >> >> I think I've found a bug in MonetDB. It only occurs when grouping by a >> text column and filtering the results; with no "where" clause, the >> results are fine. The table is fairly large (~30 million rows). >> >> ~% ./generate_test_csv.py test.csv >> ~% mclient bug -s "create table test (b integer, a varchar(5))" >> operation successful >> ~% mclient bug -s "copy into test FROM '/home/pete/test.csv' DELIMITERS >> ','" >> 29508336 affected rows >> >> There are no nulls or empty strings in column "a", however: >> >> ~% mclient bug -s "select a, count(*) from test where b = 20227 group by >> a" >> +------+--------+ >> | a | L1 | >> +======+========+ >> | | 763088 | >> | 10T | 127 | >> | 6V | 216594 | >> | 9T | 104 | >> | 8T | 88 | >> +------+--------+ >> >> When the "where" clause is removed, no nulls are returned: >> >> ~% mclient bug -s "select a, count(*) from test group by a" >> +------+---------+ >> | a | L1 | >> +======+=========+ >> | 2T | 5799418 | >> | 1T | 6072825 | >> | 5T | 2605374 | >> | 4T | 2998157 | >> | 3T | 4262607 | >> | 15T | 2703 | >> | 20T | 2818 | >> | 30T | 497657 | >> | 7T | 22460 | >> | 6T | 686789 | >> | 10T | 3496 | >> | 6V | 6550540 | >> | 9T | 1853 | >> | 8T | 1639 | >> +------+---------+ >> 14 tuples >> >> I have tested with the latest MonetDB Ubuntu debs and also with a >> local build of the Hg Feb2013 branch tip. Both exhibit the same >> problem. I'm running Linux (Ubuntu 12.10) on a 6 core Xeon with 16Gb >> RAM. I've tried with minimal_pipe and the results are the same. >> >> Please let me know if any other information would be useful. >> >> Regards, >> Pete. >> >> >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> http://mail.monetdb.org/mailman/listinfo/users-list >> > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

1 0

Bug grouping on string column, only when results filtered
by Pete Hollobon 17 Apr '13

17 Apr '13

Hi, I think I've found a bug in MonetDB. It only occurs when grouping by a text column and filtering the results; with no "where" clause, the results are fine. The table is fairly large (~30 million rows). ~% ./generate_test_csv.py test.csv ~% mclient bug -s "create table test (b integer, a varchar(5))" operation successful ~% mclient bug -s "copy into test FROM '/home/pete/test.csv' DELIMITERS ','" 29508336 affected rows There are no nulls or empty strings in column "a", however: ~% mclient bug -s "select a, count(*) from test where b = 20227 group by a" +------+--------+ | a | L1 | +======+========+ | | 763088 | | 10T | 127 | | 6V | 216594 | | 9T | 104 | | 8T | 88 | +------+--------+ When the "where" clause is removed, no nulls are returned: ~% mclient bug -s "select a, count(*) from test group by a" +------+---------+ | a | L1 | +======+=========+ | 2T | 5799418 | | 1T | 6072825 | | 5T | 2605374 | | 4T | 2998157 | | 3T | 4262607 | | 15T | 2703 | | 20T | 2818 | | 30T | 497657 | | 7T | 22460 | | 6T | 686789 | | 10T | 3496 | | 6V | 6550540 | | 9T | 1853 | | 8T | 1639 | +------+---------+ 14 tuples I have tested with the latest MonetDB Ubuntu debs and also with a local build of the Hg Feb2013 branch tip. Both exhibit the same problem. I'm running Linux (Ubuntu 12.10) on a 6 core Xeon with 16Gb RAM. I've tried with minimal_pipe and the results are the same. Please let me know if any other information would be useful. Regards, Pete.

2 2

beginning with monetdb on windows
by Satoshi Nagayasu 09 Apr '13

09 Apr '13

Hi, I'm new to the MonetDB, and trying MonetDB on Windows. At first, I have installed MonetDB on Windows with using MonetDB5-SQL-Installer-x86_64-20130227.msi. http://dev.monetdb.org/downloads/Windows/Feb2013-SP1/ Then, I attempted to follow the Getting Started guide. http://www.monetdb.org/Documentation/Tutorial However, I couldn't find monetdb.exe(bat?) and monetdbd.exe(bat?) in my monetdb installed directory. What can I do for the first step? Any comments? Regards, -- Satoshi Nagayasu <snaga(a)uptime.jp> Uptime Technologies, LLC. http://www.uptime.jp

4 6

MonetDB server crashing on Windows for seemingly simple SELECT statements
by Anthony Damico 01 Apr '13

01 Apr '13

Hi, I'm hitting some very strange errors with a few of my data tables. The data users agreement for these datasets does not allow me to share the actual output and I'm having trouble getting a similar error to occur on public data, so hopefully my description of what's going on will help the troubleshooting. I can send more information about the structure of these tables if that would be helpful? This command causes mserver.exe to crash every time: select state_cd , sum(sex) from x01 where (bene_age_at_end_ref_yr = 64) group by state_cd order by state_cd; This command causes mserver.exe to crash after running it three consecutive times. The first two times, it works. select state_cd , sum(sex) from x01 where (bene_age_at_end_ref_yr < 65) group by state_cd order by state_cd; ALL of these commands work without any trouble on the same data table: select state_cd from x01 where (bene_age_at_end_ref_yr > 64) group by state_cd order by state_cd; select state_cd , sum( metro ) as metro from x01 group by state_cd order by state_cd; select state_cd , sum( metro ) as metro from x01 where (bene_age_at_end_ref_yr > 64) group by state_cd order by state_cd; select distinct bene_age_at_end_ref_yr from x01 order by bene_age_at_end_ref_yr' ) select state_cd , count(sex) from x01 where (bene_age_at_end_ref_yr = 64) group by state_cd order by state_cd; select state_cd , sum(sex) from x01 group by state_cd order by state_cd; select state_cd , sum(sex) from x01 where (bene_age_at_end_ref_yr > 64) group by state_cd order by state_cd; select state_cd , sum(sex) from x01 where (bene_age_at_end_ref_yr >= 64) group by state_cd order by state_cd; I have no idea where to start troubleshooting what's going on here. Can anyone recommend additional tests to diagnose the root of the problem? I really don't understand how flipping the greater than sign on the otherwise same query could cause it to break.. I've pasted a bunch of diagnostics and the mclient in -X mode below, but it really doesn't seem helpful. Thanks!!!!! Here's my mserver.exe version: # MonetDB 5 server v11.15.1 "Feb2013" # Serving database 'medicare_sample', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 20.000 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:49800/ # MonetDB/JAQL module loaded # MonetDB/SQL module loaded > When running the first command that breaks inside mclient -X, here's what happens: user(win32):monetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Feb2013) mapi_query:128:SELECT "name", "value" FROM sys.env() AS env WHERE "name" IN ('gd k_dbname', 'monet_version', 'monet_release', 'merovingian_uri') fetch next block: start at:4 got next block: length:200 text:&1 0 3 2 3 % .env, .env # table_name % name, value # name % varchar, varchar # type % 13, 15 # length [ "gdk_dbname", "medicare_sample" ] [ "monet_version", "11.15.1" ] [ "monet_release", "Feb2013" ] got complete block: text:&1 0 3 2 3 % .env, .env # table_name % name, value # name % varchar, varchar # type % 13, 15 # length [ "gdk_dbname", "medicare_sample" ] [ "monet_version", "11.15.1" ] [ "monet_release", "Feb2013" ] read_line:&1 0 3 2 3 allocating new result set got complete block: text:% .env, .env # table_name % name, value # name % varchar, varchar # type % 13, 15 # length [ "gdk_dbname", "medicare_sample" ] [ "monet_version", "11.15.1" ] [ "monet_release", "Feb2013" ] read_line:% .env, .env # table_name got complete block: text:% name, value # name % varchar, varchar # type % 13, 15 # length [ "gdk_dbname", "medicare_sample" ] [ "monet_version", "11.15.1" ] [ "monet_release", "Feb2013" ] read_line:% name, value # name got complete block: text:% varchar, varchar # type % 13, 15 # length [ "gdk_dbname", "medicare_sample" ] [ "monet_version", "11.15.1" ] [ "monet_release", "Feb2013" ] read_line:% varchar, varchar # type got complete block: text:% 13, 15 # length [ "gdk_dbname", "medicare_sample" ] [ "monet_version", "11.15.1" ] [ "monet_release", "Feb2013" ] read_line:% 13, 15 # length got complete block: text:[ "gdk_dbname", "medicare_sample" ] [ "monet_version", "11.15.1" ] [ "monet_release", "Feb2013" ] read_line:[ "gdk_dbname", "medicare_sample" ] got complete block: text:[ "monet_version", "11.15.1" ] [ "monet_release", "Feb2013" ] read_line:[ "monet_version", "11.15.1" ] got complete block: text:[ "monet_release", "Feb2013" ] read_line:[ "monet_release", "Feb2013" ] fetch next block: start at:204 got next block: length:0 text: got complete block: text:? read_line:? Database: MonetDB v11.15.1 (Feb2013), 'medicare_sample' closing result set Type \q to quit, \? for a list of available commands auto commit mode: on mapi_query:46:SET TIME ZONE INTERVAL '-05:00' HOUR TO MINUTE fetch next block: start at:206 got next block: length:3 text:&3 got complete block: text:&3 read_line:&3 allocating new result set fetch next block: start at:209 got next block: length:0 text: got complete block: text:? read_line:? closing result set sql>select state_cd , sum(sex) from x01 where (bene_age_at_end_ref_yr = 64) grou p by state_cd order by state_cd; mapi_query_part:109:select state_cd , sum(sex) from x01 where (bene_age_at_end_r ef_yr = 64) group by state_cd order by state_cd; fetch next block: start at:211 Connection terminated Press any key to continue . . . When running the second command that breaks inside mclient -X, here's what happens: The first two times, the command runs fine.. Then the third time, it gives: read_line:? +----------+-------+ 67 tuples (241.424ms) closing result set sql>select state_cd , sum(sex) from x01 where (bene_age_at_end_ref_yr < 65) grou p by state_cd order by state_cd; mapi_query_part:109:select state_cd , sum(sex) from x01 where (bene_age_at_end_r ef_yr < 65) group by state_cd order by state_cd; fetch next block: start at:2403 Connection terminated Press any key to continue . . .

3 2

select + group by expression alias from a view breaks connection
by Budulinku Dejmihrasku 29 Mar '13

29 Mar '13

Hi, i hit a strange problem when using alias for an expression in group by i run into this with a complex view containing multiple UNION ALL queries but after a while everything boiled down to a fairly simple test case that will always produce the error Welcome to mclient, the MonetDB/SQL interactive terminal (Feb2013-SP1) Database: MonetDB v11.15.3 (Feb2013-SP1), 'mapi:monetdb://lux-analytics:50000/zkdev1' Type \q to quit, \? for a list of available commands auto commit mode: on sql>CREATE VIEW mmtest_v AS SELECT date'2012-01-01' as period, 1 as quantity; operation successful (213.074ms) sql>SELECT extract(month from period) as m, count(1) as cnt FROM mmtest_v GROUP BY m; *Connection terminated* I am on Ubuntu 12.04.2 LTS As the alias in GROUP BY works in other cases i expect this is a bug - pls confirm and i will be happy to file it. BTW it looks like MonetDB has generally issues with selects from more complex views - i.e. views based on another view(s). In such cases the results do not match expectations or there is even an error. I can reproduce the problem but i only on quite large set of data (typically 20+ mil. records) and after some data manipulation got involved, so i cannot give a simple test case as above. Any idea how to pass such problem (when large data-set is involved) to the development team? Thanks in advance milan

3 3

RAM consumption of MonetDB
by Hassan, Ahmad 27 Mar '13

27 Mar '13

Hi Team, Please may I ask what main memory management strategy has been adopted for MonetDB in terms of memory utilization. For example I loaded the TPCH benchmark dataset in MonetDB (v11.13.5) with scale factor 40. The data on disk is 43Gb in size. But when I run the TPCH queries then the main memory used by mserver stays in the range of 0.5GB to 3GB. I was thinking that the monetDB would load the tables into memory because TPCH queries are doing joins over various tables but it seems that monetdb loads only small subset of the complete dataset. Please can someone explain what is going in the background and why I always see 0.5GB to 3GB memory resident size of mserver process even for dataset of size 43G and if this is the normal expected behaviour of monetdb? The farm size on disk is: farms/f1/db1/bat# du . -hs 48G . Thanks. Kind Regards, Ahmad

3 5

Re: Quickstart Guide
by Stefan Manegold 20 Mar '13

20 Mar '13

Sterling, Thanks for notifing! Could you please file a bug report via http://bugs.monetdb.org/ ? Thanks! Stefan Sterling Paramore <gnilrets(a)gmail.com> wrote: The link at http://www.monetdb.org/Downloads/Tutorial is broken and I don't see any way to contact the creators of the web page. How can I get information about how to install MonetDB? Thanks, Sterling

4 4

Re: Quickstart Guide
by Stefan Manegold 20 Mar '13

20 Mar '13

Sterling, did you start the monetdb daemon (monetdbd), first? See also http://www.monetdb.org/Documentation/monetdbd Stefan Sterling Paramore <gnilrets(a)gmail.com> wrote: Thanks, but I'm not terribly interested in building from source. These installation instructions seem to end after it's compiled. I downloaded the binaries for OS X and put them in /usr/local/monetdb, and added /usr/local/monetdb/bin to my PATH. I found http://www.monetdb.org/Documentation/Tutorial (which is maybe supposed to be the same thing as the broken link?) But when I try the first step, I get the following error: [- opt -] monetdb create /opt/monetdb monetdb: cannot find a control socket, use -h and/or -p Google was not helpful. Any ideas? Thanks, Sterling On Tue, Mar 19, 2013 at 6:52 PM, Hassan, Ahmad <ahmad.hassan(a)sap.com> wrote: > HI Sterling,**** > > ** ** > > Installation from source can be seen at: > http://www.monetdb.org/wiki/MonetDB:Building_from_sources**** > > ** ** > > Cheers, Ahmad**** > > ** ** > > *From:* users-list-bounces+ahmad.hassan=sap.com(a)monetdb.org [mailto: > users-list-bounces+ahmad.hassan=sap.com(a)monetdb.org] *On Behalf Of *Sterling > Paramore > *Sent:* 20 March 2013 01:49 > *To:* users-list(a)monetdb.org > *Subject:* Quickstart Guide**** > > ** ** > > The link at http://www.monetdb.org/Downloads/Tutorial is broken and I > don't see any way to contact the creators of the web page. How can I get > information about how to install MonetDB?**** > > ** ** > > Thanks,**** > > Sterling**** > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list > >

1 0

Quickstart Guide
by Sterling Paramore 20 Mar '13

20 Mar '13

The link at http://www.monetdb.org/Downloads/Tutorial is broken and I don't see any way to contact the creators of the web page. How can I get information about how to install MonetDB? Thanks, Sterling

2 2

Performance of PHP client > fetching rows
by Richard Siebeling 18 Mar '13

18 Mar '13

Hi, I querying my database using a script in PHP and returning the result as a JSON (so that it can be retrieved in Javascript). However I notice that querying the database (sending the query to the database) takes about half the time if takes PHP to retrieve the rows (row by row using monetdb_fetch_object). In the documentation I see that the MAPI interface has a fetch_all_rows function, however I can't find this function in the PHP library. In the Python example I see that you are also using a cursor.fetchall() procedure. What is the best way to approach this problem > querying the database and returning all rows as a JSON object Is PHP the way to go (it is the easiest for me, since I know that), if it is, what is the best way or should I consider another language like Python or maybe Java to implement this function? Below I list the PHP code I see that send the query ($q_statement) to the database and formulating the JSON: // Measure time $time_end = microtime(true); $execution_time = ($time_end - $time_start); //dividing with 60 will give the execution time in minutes other wise seconds error_log($execution_time.'sec - Formulate query'); $time_start = microtime(true); $db = monetdb_connect($lang = "sql", $host = "127.0.0.1", $port = "50000" , $username = "monetdb", $password = "monetdb", $database = "social" ) or die(monetdb_last_error()); $res = monetdb_query($db, monetdb_escape_string($q_statement)); // Measure time $time_end = microtime(true); $execution_time = ($time_end - $time_start); //dividing with 60 will give the execution time in minutes other wise seconds error_log($execution_time.'sec - Query database'); $time_start = microtime(true); /* Iterate over the result set returning rows as objects */ while ( $row = monetdb_fetch_object($res) ) { $rows[] = $row; } echo json_encode($rows); // Measure time $time_end = microtime(true); $execution_time = ($time_end - $time_start); //dividing with 60 will give the execution time in minutes other wise seconds $total_execution_time = ($time_end - $initial_start); error_log($execution_time.'sec - Return records'); error_log($total_execution_time.'sec - Total time'); $time_start = microtime(true); thanks in advance, Richard

2 1

Single Quotes VS Double Quotes
by Diethard Steiner 17 Mar '13

17 Mar '13

Hi, I ran into a problem when setting up a user with single quotes: sql>CREATE USER 'tpch' WITH PASSWORD 'tpch' NAME 'TPC-H Admin' SCHEMA 'sys'; syntax error, unexpected STRING in: "create user 'tpch'" Then I followed exactly the quatation style of the voc example and it worked: sql>CREATE USER "tpch" WITH PASSWORD 'tpch' NAME 'TPC-H Admin' SCHEMA "sys"; operation successful (3.663ms) So now with user and schema enclosed in double quotes, password and name in single quotes, it worked. This inconsistent usage of single and double quotes seems a bit strange to me. Should I file a bug report for this? Or is there a specific reason for this? As a final test I enclosed all values in double quotes: sql>CREATE USER "test2" WITH PASSWORD "test2" NAME "Test User 2" SCHEMA "sys"; syntax error, unexpected IDENT, expecting STRING in: "create user "test2" with password "test2"" Thanks, Diethard

2 2

Extract quarter from date
by Diethard Steiner 17 Mar '13

17 Mar '13

Hi, When I run this: SELECT DISTINCT(EXTRACT(QUARTER FROM o_orderdate)) FROM orders; I get following error: Error: syntax error, unexpected IDENT in: "select distinct(extract(quarter" SQLState: 42000 ErrorCode: 0 Is Quarter not supported or is this a bug? Extract Year, month, day are all working fine. Environment: RHEL (Amazon Linux), Latest version of MonetDB compiled from source code. Thanks, Diethard

4 6

Connecting Tableau to MonetDB
by Pedro Salgueiro 15 Mar '13

15 Mar '13

Hi, I am trying to connect Tableau to a MonetDB instance which is running in a linux machine, using the ODBC MonetDB driver. Tableau manages to connect to MonetDB and is able to list the tables, but then it fails with the following error while performing any Analysis: [MonetDB][ODBC Driver 11.13.9] Invalid identifier 'none:product:nk' > I tried two versions of the ODBC, the latest and Oct2012-SP, but the problem is present in both versions. It's worth pointing that I am able to use the same MonetDB with the JDBC with no problems. <http://dev.monetdb.org/downloads/Windows/Oct2012-SP3/> While googling for this particular problem, I found the following mail in the mailing list: http://mail.monetdb.org/pipermail/users-list/2011-November/005248.html Has there been any developments in this field? Best regards, Pedro Salgueiro

2 1

standard deviation in SQL
by Anshuman Kumar 15 Mar '13

15 Mar '13

Hi, I'm trying to find an aggregate function to find standard deviation. Does monetDB support such a function? For example:- to find the average of the selected values we can use- Select AVG(age) from students where grade = 4; I was taking the help of google and found this link- http://old.nabble.com/users-list-Digest,-Vol-3,-Issue-8-td34660082.html Then I tried the solution given by Percy Wegmann in the above link. After that when I run this query- Select STDEV(age) from students where grade = 4; [Age being a column of type double AND I get similar kind of error for int] I get this error - TypeException:user.s1_1[21]:'aggr.stddev' undefined in: _33:any := aggr.stddev(_ 32:bat[:oid,:dbl]) program contains errors I need to have this function working in order to be able to use MonetDB. Kindly help. Regards, Anshuman

3 4

stddev function disappeared?
by Anthony Damico 14 Mar '13

14 Mar '13

Hello, I'm trying to run some old code with a recent upgrade of MonetDB and the standard deviation function appears to have disappeared. I'm running MonetDB on Windows. This issue was previously discussed and resolved in this thread: http://old.nabble.com/users-list-Digest%2C-Vol-3%2C-Issue-8-tt34660082.html… I could re-open this bug report if appropriate? http://bugs.monetdb.org/show_bug.cgi?id=2454#c18 My server specs are below, and the actual commands below that. Thank you!!! # MonetDB 5 server v11.15.1 "Feb2013" # Serving database 'bsapuf', using 8 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 7.860 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50003/ # MonetDB/JAQL module loaded # MonetDB/SQL module loaded > c:\Program Files\MonetDB\MonetDB5>mclient -p 50003 "bsapuf" user(win32):monetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Feb2013) Database: MonetDB v11.15.1 (Feb2013), 'bsapuf' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select sum( hha_pmt_amt ) from hha08; +-----------+ | L1 | +===========+ | 675676300 | +-----------+ 1 tuple (2.311ms) sql>select sum( hha_pmt_amt ), avg( hha_pmt_amt ) from hha08; +-----------+--------------------------+ | L1 | L2 | +===========+==========================+ | 675676300 | 5412.8151310993435 | +-----------+--------------------------+ 1 tuple (63.559ms) sql>select sum( hha_pmt_amt ), median( hha_pmt_amt ) from hha08; +-----------+------+ | L1 | L2 | +===========+======+ | 675676300 | 3500 | +-----------+------+ 1 tuple (9.601ms) sql>select sum( hha_pmt_amt ), stddev( hha_pmt_amt ) from hha08; SELECT: no such unary operator 'stddev(int)' sql>select sum( hha_pmt_amt ), sd( hha_pmt_amt ) from hha08; SELECT: no such unary operator 'sd(int)' sql>select sum( hha_pmt_amt ), sddev( hha_pmt_amt ) from hha08; SELECT: no such unary operator 'sddev(int)' sql>select sum( hha_pmt_amt ), var( hha_pmt_amt ) from hha08; SELECT: no such unary operator 'var(int)' sql>select sum( hha_pmt_amt ), variance( hha_pmt_amt ) from hha08; SELECT: no such unary operator 'variance(int)' sql>select sum( hha_pmt_amt ), stdevp( hha_pmt_amt ) from hha08; SELECT: no such unary operator 'stdevp(int)' sql>

3 4

SQL function for getting the last inserted ID
by Gijs Molenaar 12 Mar '13

12 Mar '13

Hi, A typical question that is probably asked before, but I can't find an answer in the archives or in the doc. Is there a SQL function for getting the last inserted ID for a auto increment or serial field? PostgreSQL has lastval(). I know it is returned in the query result, but I need to have the value inside a SQL function. Also I know that if you manually create a sequence for ID generation you can get the next value with: SELECT NEXT VALUE FOR seq_freq INTO freqid; But we are trying to keep our queries as generic as possible (close to impossible, but still) hence we want to use the SERIAL type. Or is there a way to find out what is the name of the sequence made when giving a SERIAL type to a column? thanks, -- Gijs Molenaar http://www.astro.uva.nl/people/gijs-molenaar/

3 3

Insertion problem for large number of columns
by Shajon 12 Mar '13

12 Mar '13

Dear Monet Team, Today I was trying to insert data in MonetDB using " insert into ." command through jdbc from my application but the server is crashing & no data is inserting. That table has 25 columns and I am passing all values. Thanks Shajon

5 6

Questions
by Bouslah Ahmed 10 Mar '13

10 Mar '13

Hi, I am a university student and I use MonetDB. I have two questions to ask you regarding loading data using mclient: 1 - Is it possible to load multiple csv files into multiple tables in one query? For exemple i have five csv files and five tables and I want to use a single command line instead of five mclient -u us1 -d demo -s " COPY INTO DWDATE FROM STDIN USING DELIMITERS '|','\n'" - <F:\2Go\DATE.tbl mclient -u us1 -d demo -s " COPY INTO CUSTOMER FROM STDIN USING DELIMITERS '|','\n'" - <F:\2Go\CUSTOMER.tbl mclient -u us1 -d demo -s " COPY INTO SUPPLIER FROM STDIN USING DELIMITERS '|','\n'" - <F:\2Go\SUPPLIER.tbl mclient -u us1 -d demo -s " COPY INTO PART FROM STDIN USING DELIMITERS '|','\n'" - <F:\2Go\PART.tbl mclient -u us1 -d demo -s " COPY INTO LINEORDER FROM STDIN USING DELIMITERS '|','\n'" -<F:\2Go\LINEORDER.tbl 2 - Is it possible to enter the password in the request as well as user and database name? This will save me introduce each time the password For exemple mclient -u us1 -d demo -s " COPY INTO DWDATE FROM STDIN USING DELIMITERS '|','\n'" - <F:\2Go\DATE.tbl <PassWord> thank you.

1 0

Parse dates in CSV file, CSV vault
by Aubrey Barnard 10 Mar '13

10 Mar '13

MonetDB Users, Is YYYY-MM-DD the only date format that can be parsed and read from a CSV with the COPY command? Or can the format be specified? On a related note, is the CSV vault feature ready for use (Feb2013 version)? The mailing list implies it is not ready and there is nothing in the release notes. If it is ready, how do I set it up? Thanks much for helping me get my data loaded. Aubrey Barnard

2 1

RE: Installation on Ubuntu 12.10 Quantal
by Duncan Garmonsway 10 Mar '13

10 Mar '13

Thanks, that worked. Didn't realise I was still such a noob... Cheers, Duncan On 10 March 2013 00:00, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. INSERT INTO ... SELECT * FROM ... OFFSET X seems to leave > destination table in an inconsistent state (Percy Wegmann) > 2. Re: INSERT INTO ... SELECT * FROM ... OFFSET X seems to leave > destination table in an inconsistent state (Martin Kersten) > 3. Installation on Ubuntu 12.10 Quantal (Duncan Garmonsway) > 4. Re: Installation on Ubuntu 12.10 Quantal (Brandon Jackson) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 8 Mar 2013 10:22:53 -0600 > From: Percy Wegmann <percy(a)clicksecurity.com> > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Subject: INSERT INTO ... SELECT * FROM ... OFFSET X seems to leave > destination table in an inconsistent state > Message-ID: > < > CADpZG+Scab3V8u_isKziJobUgCYO4BEmkCWMB8iOD-pjggTWQQ(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Dear Monet Team, > > Today I ran into an issue with copying data between tables. I've tried it > on 11.13.7 and 11.13.9 with the same results. What I find is that when > including an OFFSET clause in my source query, the destination table ends > up in a weird state. > > I've logged a bug for it - http://bugs.monetdb.org/show_bug.cgi?id=3249 > > I'm using the workaround of ordering my data and using a LIMIT instead of > an OFFSET, which seems to work. Just wanted to let you know. > > Thanks, > Percy >

1 0

Installation on Ubuntu 12.10 Quantal
by Duncan Garmonsway 09 Mar '13

09 Mar '13

Hello, After installation I am getting the error: $ monetdbd create ~/monetdb/ monetdbd: error while loading shared libraries: libstream.so.3: cannot open shared object file: No such file or directory But the library (and the symlink to it) are right here: $ ls -l /usr/local/lib | grep libstream -rwxr-xr-x 1 root root 943 Mar 9 14:23 libstream.la lrwxrwxrwx 1 root root 18 Mar 9 14:23 libstream.so -> libstream.so.3.1.5 lrwxrwxrwx 1 root root 18 Mar 9 14:23 libstream.so.3 -> libstream.so.3.1.5 -rwxr-xr-x 1 root root 107879 Mar 9 14:23 libstream.so.3.1.5 And there were no errors reported during any of ./configure sudo make sudo make install My installation script was: sudo apt-get install libxml2-dev bison ant flex unixODBC-dev # download and unpack source wget http://dev.monetdb.org/downloads/sources/Latest/MonetDB-11.15.3.tar.xz tar -xvf MonetDB-11.15.3.tar.xz # install cd MonetDB-11.15.3 ./configure sudo make sudo make install Help much appreciated, Duncan

2 1

INSERT INTO ... SELECT * FROM ... OFFSET X seems to leave destination table in an inconsistent state
by Percy Wegmann 08 Mar '13

08 Mar '13

Dear Monet Team, Today I ran into an issue with copying data between tables. I've tried it on 11.13.7 and 11.13.9 with the same results. What I find is that when including an OFFSET clause in my source query, the destination table ends up in a weird state. I've logged a bug for it - http://bugs.monetdb.org/show_bug.cgi?id=3249 I'm using the workaround of ordering my data and using a LIMIT instead of an OFFSET, which seems to work. Just wanted to let you know. Thanks, Percy

2 1

Announcement: New Feb2013-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 07 Mar '13

07 Mar '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Feb 2013-SP1 bugfix release Bug Fixes * 3228: Python client - mapi.Connection.__getbyes should throw exception when 0 bytes received from socket * 3229: GRANT does not work for UPDATE * 3230: GRANT SELECT priveleges not set in SQL FUNCTIONs * 3233: getting a crash of server with segmentation fault on SLES 11 64 SP2 * 3235: MapiPP.pm unquote '0' as 'undef' * 3236: mclient \d produces no output * 3237: Assertion on BATgroup_internal * 3241: SEGFAULT on string processing when using optimizer.reduce()

1 0

good new monetdb question on stackoverflow
by Anthony Damico 28 Feb '13

28 Feb '13

http://stackoverflow.com/questions/15136868/why-is-group-by-giving-only-one… I couldn't figure out the answer..

2 1

Investigating Segfault
by Percy Wegmann 25 Feb '13

25 Feb '13

Dear Monet Team, We're having an issue in which certain data is causing mserver5 to crash. After this condition is hit, mserver5 crashes at every startup and always dumps an identical core. We're running Monet 11.13.7 on Ubuntu Linux 64 bit. The error is happening on line 1142 of gdk_atoms.c: if (GDK_STRCMP(v, (str) (next + 1) + extralen) == 0) { Examining the core dump revealed that (next + 1) + extralen is referring to an out of bounds address. Here's the backtrace: #0 0x00007faf58414829 in strPut (h=0x1e2d180, dst=0x7fff592cf8f8, v=0x314dac0 "SAD014H1") at gdk_atoms.c:1142 #1 0x00007faf582dc935 in BATappend (b=0x1e2cf90, n=0x32dfdb0, force=1 '\001') at gdk_batop.c:578 #2 0x00007faf584c301e in la_bat_updates (lg=0x2d9b030, la=0x2c3ef48) at gdk_logger.c:429 #3 0x00007faf584c3cf9 in la_apply (lg=0x2d9b030, c=0x2c3ef48) at gdk_logger.c:645 #4 0x00007faf584c3f26 in tr_commit (lg=0x2d9b030, tr=0x2e247d0) at gdk_logger.c:705 #5 0x00007faf584c4533 in logger_readlog (lg=0x2d9b030, filename=0x7fff592d1e80 "/opt/clicksecurity/data/_monetdb/click/sql_logs/sql/log.56") at gdk_logger.c:823 #6 0x00007faf584c482a in logger_readlogs (lg=0x2d9b030, fp=0x2d9b160, filename=0x7fff592d3f90 "/opt/clicksecurity/data/_monetdb/click/sql_logs/sql/log") at gdk_logger.c:896 #7 0x00007faf584c6f3e in logger_new (debug=0, fn=0x7faf500adfa8 "sql", logdir=0x7faf50090a08 "sql_logs", dbname=0x1fa3da0 "click", version=52001, prefuncp=0x7faf500746a1 <bl_preversion>, postfuncp=0x7faf500747ed <bl_postversion>) at gdk_logger.c:1420 #8 0x00007faf584c704e in logger_create (debug=0, fn=0x7faf500adfa8 "sql", logdir=0x7faf50090a08 "sql_logs", dbname=0x1fa3da0 "click", version=52001, prefuncp=0x7faf500746a1 <bl_preversion>, postfuncp=0x7faf500747ed <bl_postversion>) at gdk_logger.c:1446 #9 0x00007faf50075b19 in bl_create (logdir=0x7faf50090a08 "sql_logs", dbname=0x1fa3da0 "click", cat_version=52001) at bat_logger.c:249 #10 0x00007faf50060ce4 in store_init (debug=0, store=store_bat, logdir=0x7faf50090a08 "sql_logs", dbname=0x1fa3da0 "click", stk=0) at store.c:1287 #11 0x00007faf4ffe3d3c in mvc_init (dbname=0x1fa3da0 "click", debug=0, store=store_bat, stk=0) at sql_mvc.c:51 #12 0x00007faf4ff66874 in SQLinit () at sql_scenario.c:230 #13 0x00007faf4ff6651f in SQLprelude () at sql_scenario.c:159 #14 0x00007faf58b3085d in malCommandCall (stk=0x2d36e80, pci=0x2ea5520) at mal_interpreter.c:137 #15 0x00007faf58b331b5 in runMALsequence (cntxt=0x7faf5988c020, mb=0x1e04310, startpc=1, stoppc=0, stk=0x2d36e80, env=0x0, pcicaller=0x0) at mal_interpreter.c:710 #16 0x00007faf58b323c1 in runMAL (cntxt=0x7faf5988c020, mb=0x1e04310, startpc=1, mbcaller=0x0, env=0x0, pcicaller=0x0) at mal_interpreter.c:454 #17 0x00007faf58b60a08 in MALengine (c=0x7faf5988c020) at mal_session.c:619 #18 0x00007faf58b5f21f in malBootstrap () at mal_session.c:64 #19 0x00007faf58b1313b in mal_init () at mal.c:244 #20 0x000000000040340e in main (argc=22, av=0x7fff592db568) at mserver5.c:582 I'm digging into it now, but I was hoping that it might ring some bells. Thanks, -- Percy Wegmann +1 512 637 8500 ext 148

2 3

MonetDB Feb2013 Release --dbpath option
by Gennadiy 22 Feb '13

22 Feb '13

Hello, Before MonetDB Feb2013 Release, I used to start MonetDB using the following Windows command: MSQLserver.bat --dbname=MyNewDatabase MSQLserver.bat called M5server.bat, M5server.bat set --dbfarm option. The result was that database with name specified with --dbname option was created in the directory specified with --dbfarm option. MonetDB Feb2013 Release replaced --dbname and --dbfarm options with single --dbpath option. This --dbpath option is set to some_path\dbfarm\demo in M5server.bat. So, now (MonetDB Feb2013 Release) when I start MonetDB with the following Windows command: MSQLserver.bat --dbpath=some_path\MyNewDatabase MonetDB ignores my --dbpath option, because it uses --dbpath option from M5server.bat! So, how can I start MonetDB and create my own database? Thank you very much, Gennadiy

1 1

Windows Concurrency Error when accessing two separate tables in two separate mclient instances connected to the same mserver?
by Anthony Damico 22 Feb '13

22 Feb '13

Hi, I am using a brand new mserver.exe instance of MonetDB5 on Windows. I am getting the error: "COMMIT: transaction is aborted because of concurency conflicts, will ROLLBACK instead" that I'm not sure I should? Shouldn't I be able to use the COPY command on two separate CSV files into two separate data tables at once? I don't think this behavior is correct, but maybe I'm wrong? :/ If this is incorrect behavior, should I file a bug report for this? The files you can use to re-create this error are: http://downloads.cms.gov/BSAPUF/2008_BSA_PartD_Events_PUF_1.zip http://downloads.cms.gov/BSAPUF/2008_BSA_PartD_Events_PUF_2.zip I create an mserver.exe instance with: # MonetDB 5 server v11.13.9 "Oct2012-SP3" # Serving database 'bsapuf', using 8 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 7.860 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50003/ # MonetDB/JAQL module loaded # MonetDB/SQL module loaded Then I ran this command in the first mclient: CREATE TABLE bsa1 (PDE_EVENT_ID VARCHAR(255), BENE_SEX_IDENT_CD DOUBLE PRECISION, BENE_AGE_CAT_CD DOUBLE PRECISION, PDE_DRUG_CD DOUBLE PRECISION, PDE_DRUG_STR_CD DOUBLE PRECISION, PDE_DRUG_STR_UNITS_CD DOUBLE PRECISION, PDE_DRUG_DOSE_CD DOUBLE PRECISION, PDE_DRUG_CLASS_CD DOUBLE PRECISION, PDE_DRUG_QTY_DIS DOUBLE PRECISION, PDE_DRUG_DAY_SPLY_CD DOUBLE PRECISION, PDE_DRUG_COST DOUBLE PRECISION, PDE_DRUG_PAT_PAY_CD DOUBLE PRECISION, PDE_DRUG_TYPE_CD DOUBLE PRECISION) ; copy 1000000 offset 2 records into bsa1 from 'c:\\temp\\2008_BSA_PartD_Events_PUF_1.csv' using delimiters ',' NULL AS '' ; And (at the same time) ran this command in the second mclient: CREATE TABLE bsa2 (PDE_EVENT_ID VARCHAR(255), BENE_SEX_IDENT_CD DOUBLE PRECISION, BENE_AGE_CAT_CD DOUBLE PRECISION, PDE_DRUG_CD DOUBLE PRECISION, PDE_DRUG_STR_CD DOUBLE PRECISION, PDE_DRUG_STR_UNITS_CD DOUBLE PRECISION, PDE_DRUG_DOSE_CD DOUBLE PRECISION, PDE_DRUG_CLASS_CD DOUBLE PRECISION, PDE_DRUG_QTY_DIS DOUBLE PRECISION, PDE_DRUG_DAY_SPLY_CD DOUBLE PRECISION, PDE_DRUG_COST DOUBLE PRECISION, PDE_DRUG_PAT_PAY_CD DOUBLE PRECISION, PDE_DRUG_TYPE_CD DOUBLE PRECISION) ; copy 1000000 offset 2 records into bsa2 from 'c:\\temp\\2008_BSA_PartD_Events_PUF_2.csv' using delimiters ',' NULL AS '' ; Here were the results. One worked, one didn't: Microsoft Windows [Version 6.1.7601] Copyright (c) 2009 Microsoft Corporation. All rights reserved. C:\Users\anthonyd.KFF>cd .. C:\Users>cd .. C:\>cd "Program Files\MonetDB\MonetDB5" C:\Program Files\MonetDB\MonetDB5>mclient -p 50003 "bsapuf" user(win32):monetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Oct2012-SP3) Database: MonetDB v11.13.9 (Oct2012-SP3), 'bsapuf' Type \q to quit, \? for a list of available commands auto commit mode: on sql> sql> sql>CREATE TABLE bsa2 (PDE_EVENT_ID VARCHAR(255), BENE_SEX_IDENT_CD DOUBLE PRECI SION, BENE_AGE_CAT_CD DOUBLE PRECISION, PDE_DRUG_CD DOUBLE PRECISION, PDE_DRUG_S TR_CD DOUBLE PRECISION, PDE_DRUG_STR_UNITS_CD DOUBLE PRECISION, PDE_DRUG_DOSE_CD DOUBLE PRECISION, PDE_DRUG_CLASS_CD DOUBLE PRECISION, PDE_DRUG_QTY_DIS DOUBLE P RECISION, PDE_DRUG_DAY_SPLY_CD DOUBLE PRECISION, PDE_DRUG_COST DOUBLE PRECISION, PDE_DRUG_PAT_PAY_CD DOUBLE PRECISION, PDE_DRUG_TYPE_CD DOUBLE PRECISION) ; operation successful (9.511ms) sql> sql>copy 1000000 offset 2 records into bsa2 from 'c:\\temp\\2008_BSA_PartD_Event s_PUF_2.csv' using delimiters ',' NULL AS '' ; 1000000 affected rows (42.4s) sql> XXXXXXXXXXXXXXXXXXXXXXXXXXXXX Microsoft Windows [Version 6.1.7601] Copyright (c) 2009 Microsoft Corporation. All rights reserved. C:\Users\anthonyd.KFF>cd .. C:\Users>cd .. C:\>cd "Program Files\MonetDB\MonetDB5" C:\Program Files\MonetDB\MonetDB5>mclient -p 50003 "bsapuf" user(win32):monetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Oct2012-SP3) Database: MonetDB v11.13.9 (Oct2012-SP3), 'bsapuf' Type \q to quit, \? for a list of available commands auto commit mode: on sql>\d sql>CREATE TABLE bsa1 (PDE_EVENT_ID VARCHAR(255), BENE_SEX_IDENT_CD DOUBLE PRECI SION, BENE_AGE_CAT_CD DOUBLE PRECISION, PDE_DRUG_CD DOUBLE PRECISION, PDE_DRUG_S TR_CD DOUBLE PRECISION, PDE_DRUG_STR_UNITS_CD DOUBLE PRECISION, PDE_DRUG_DOSE_CD DOUBLE PRECISION, PDE_DRUG_CLASS_CD DOUBLE PRECISION, PDE_DRUG_QTY_DIS DOUBLE P RECISION, PDE_DRUG_DAY_SPLY_CD DOUBLE PRECISION, PDE_DRUG_COST DOUBLE PRECISION, PDE_DRUG_PAT_PAY_CD DOUBLE PRECISION, PDE_DRUG_TYPE_CD DOUBLE PRECISION) ; operation successful (8.636ms) sql> sql>copy 1000000 offset 2 records into bsa1 from 'c:\\temp\\2008_BSA_PartD_Event s_PUF_1.csv' using delimiters ',' NULL AS '' ; 1000000 affected rows (41.6s) COMMIT: transaction is aborted because of concurency conflicts, will ROLLBACK in stead sql> sql> Thanks as always! Anthony

2 2

Announcement: New Feb2013 Feature release of MonetDB suite
by Sjoerd Mullender 22 Feb '13

22 Feb '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Feb 2013 feature release Testing Environment * enabled "top-level" Mtest.py So far, while Mtest.py could be called in any subdirectory of the MonetDB source tree (and could then run all tests in the entire sub-tree), it was not possible to call Mtest.py in the top-level MonetDB source directory to run all tests. Instead, to run all tests, Mtest.py had to be called at least 4 times, once in each of these directories: "clients", "monetdb5", "sql", "geom". Now, it is possible to call Mtest.py once in the top-level MonetDB source directory to run all tests in one go. The behaviour of calling Mtest.py in any subdirectory, including the four mentioned above, did not changed, other than that now obsolete command line options "-p / --package <package>" and "-5 / --monetdb5" have been removed. Java Module * merocontrol was changed to return server URIs, and lastStop time. Connections and dbpath were removed. * Mapi protocol v8 support was removed from MapiSocket. Protocol v8 has not been used by the servers any more since Apr2012 release Client Package * Mapi protocol v8 support was removed from all client drivers. Protocol v8 has not been used by the servers any more since Apr2012 release * The tool mnc was removed from installations * msqldump: Implmented an option (--table/-t) to dump a single table. * Changed msqdump's trace option to be in line with mclient. In both cases, the long option is --Xdebug and the short option is -X. MonetDB5 Server * mserver5: The --dbname and --dbfarm options have been replaced by the single --dbpath option. * The scheduler of mserver5 was changed to use a fixed set of workers to perform the work for all connected clients. Previously, each client connection had its own set of workers, easily causing resource problems upon multiple connections to the server. Merovingian * Upgrade support for dbfarms from Mar2011 and Aug2011 was dropped * monetdb status now uses a more condensed output, to cater for the uris being shown, and prints how long a database is stopped, or how long ago it crashed * monetdb status now prints the connection uri for each database, when available. The connections and database path properties have been dropped. * monetdb status now prints last crash date only if the database has not been started since. Bug Fixes * 2291: small doubles end up as NULL after arithmetic * 3215: Calculation Date function using interval and year * 3033: stethoscope needs better documentation * 3084: Timestamp arithmetic very slow (especially on Windows) * 3125: Python tests fail after recent Python API changes * 3172: assertion fails on table function with subselects as parameters * 3178: one scan is enough to implement ALGstdev_@1 in monetdb5/modules/kernel/algebra.mx * 3179: LIKE: batstr.like+algebra.uselect called instead of pcre.like_filter * 3193: Expressions not supported in the GROUP BY or ORDER BY clause. * 3216: "unknown property" error setting format and width in .monetdb file * 3217: gdk_posix fails to compile under NetBSD * 3221: can no execute large statements * 3227: MT_set_lock() call on an non-initialized lock

5 11

MonetDB documentation
by Brandon Jackson 15 Feb '13

15 Feb '13

Just curious, I follow the mailing lists closely very recently. Is there a way, or does it make sense for someone like me to have access to update the MonetDb documentation based on responses from the maintainers here? I would be happy to collect some of this information and send it in for review before posting as documentation. Brandon Sent from my iPhone

2 2

Windows Networked Drive Issue with COPY INTO
by Anthony Damico 15 Feb '13

15 Feb '13

I am running MonetDB5 Oct 2012 SP3 on my work Windows 7 desktop computer. My office has what I imagine is a pretty typical mix of networked machines, with a few more powerful servers and larger shared drives that everyone can connect to. (1) When I attempt to import a large CSV file into a MonetDB server on the _local_ disk (C:\My Directory\MonetDB\dbfarm), the importation command runs as expected. (2) When I attempt to import the same CSV file to a MonetDB server stored on a networked drive, the import command fails. (3) Finally, when I *remote desktop* into the server that the networked drive is actually attached to, the importation succeeds again! Here's the error seen in scenario (2) -- GDKerror:!ERROR: GDKunlink(bat\13\1327.tail) !OS: The process cannot access the file because it is being used by another process. GDKunlink(bat\13\1327.theap) !OS: The process cannot access the file because it is being used by another process. So it seems pretty clear that something about windows networked drives is the culprit. This error does not occur on every file (larger files seem more prone to the error), but if an import command does not work on a file, it _consistently_ does not work on that file. As an example, importing this large file works with methods (1) and (3) but not (2) -- http://downloads.cms.gov/BSAPUF/2008_BSA_Carrier_Line_Items_PUF_1.zip I tried disabling the windows firewall and using different server ports, but I still hit the above error. I spoke with Hannes about it briefly, he said: "The issue is probably related to the Windows mmap() implementation when talking to a SMB filesystem. It is far from certain that there is something we can do about this, but in either case, we'd like to know about it." Should I submit an official bug report to http://bugs.monetdb.org/ ? Perhaps we are overlooking something obvious about our windows network's configuration.. Thanks!!! Here is my MonetDB server info, followed by the COPY INTO commands run in -X mode for each of the three scenarios: # MonetDB 5 server v11.13.9 "Oct2012-SP3" # Serving database 'bsa', using 2 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 15.873 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/ # MonetDB/JAQL module loaded # MonetDB/SQL module loaded ************************** (1) - local disk - works ----- fetch next block: start at:196 got next block: length:0 text: got complete block: text:☺ read_line:☺ Database: MonetDB v11.13.9 (Oct2012-SP3), 'bsa' closing result set Type \q to quit, \? for a list of available commands auto commit mode: on mapi_query:46:SET TIME ZONE INTERVAL '-05:00' HOUR TO MINUTE fetch next block: start at:198 got next block: length:3 text:&3 got complete block: text:&3 read_line:&3 allocating new result set fetch next block: start at:201 got next block: length:0 text: got complete block: text:☺ read_line:☺ closing result set sql>create table mytable ( mapi_query_part:23:create table mytable ( fetch next block: start at:203 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:206 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_id VARCHAR(255), mapi_query_part:27: car_line_id VARCHAR(255), fetch next block: start at:208 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:211 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> bene_sex_ident_cd INTEGER, mapi_query_part:28: bene_sex_ident_cd INTEGER, fetch next block: start at:213 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:216 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> bene_age_cat_cd INTEGER, mapi_query_part:26: bene_age_cat_cd INTEGER, fetch next block: start at:218 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:221 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_icd9_dgns_cd VARCHAR(255), mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255), fetch next block: start at:223 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:226 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_hcpcs_cd VARCHAR(255), mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255), fetch next block: start at:228 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:231 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_betos_cd VARCHAR(255), mapi_query_part:33: car_line_betos_cd VARCHAR(255), fetch next block: start at:233 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:236 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_srvc_cnt INTEGER, mapi_query_part:28: car_line_srvc_cnt INTEGER, fetch next block: start at:238 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:241 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_prvdr_type_cd INTEGER, mapi_query_part:33: car_line_prvdr_type_cd INTEGER, fetch next block: start at:243 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:246 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_cms_type_srvc_cd VARCHAR(255), mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255), fetch next block: start at:248 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:251 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_place_of_srvc_cd INTEGER, mapi_query_part:36: car_line_place_of_srvc_cd INTEGER, fetch next block: start at:253 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:256 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_hcpcs_pmt_amt INTEGER mapi_query_part:27: car_hcpcs_pmt_amt INTEGER fetch next block: start at:258 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:261 got next block: length:0 text: got complete block: text:☺ read_line:☺ more>) ; mapi_query_part:4:) ; fetch next block: start at:263 got next block: length:3 text:&3 got complete block: text:&3 read_line:&3 allocating new result set fetch next block: start at:266 got next block: length:0 text: got complete block: text:☺ read_line:☺ operation successful (118.294ms) closing result set sql> fetch next block: start at:268 got next block: length:0 text: got complete block: text:☺ read_line:☺ sql>COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Directory\BSAPUF\2008 \2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','\"' NULL AS ' ' ; mapi_query_part:160:COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Direc tory\BSAPUF\2008\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n ','\"' NULL AS '' ; fetch next block: start at:270 got next block: length:14 text:&2 9676439 -1 got complete block: text:&2 9676439 -1 read_line:&2 9676439 -1 allocating new result set fetch next block: start at:284 got next block: length:0 text: got complete block: text:☺ read_line:☺ 9676439 affected rows (3m 26s) closing result set sql> ************************** (2) - networked disk - fails ----- read_line:☺ more> car_line_id VARCHAR(255), mapi_query_part:27: car_line_id VARCHAR(255), fetch next block: start at:208 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:211 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> bene_sex_ident_cd INTEGER, mapi_query_part:28: bene_sex_ident_cd INTEGER, fetch next block: start at:213 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:216 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> bene_age_cat_cd INTEGER, mapi_query_part:26: bene_age_cat_cd INTEGER, fetch next block: start at:218 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:221 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_icd9_dgns_cd VARCHAR(255), mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255), fetch next block: start at:223 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:226 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_hcpcs_cd VARCHAR(255), mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255), fetch next block: start at:228 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:231 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_betos_cd VARCHAR(255), mapi_query_part:33: car_line_betos_cd VARCHAR(255), fetch next block: start at:233 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:236 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_srvc_cnt INTEGER, mapi_query_part:28: car_line_srvc_cnt INTEGER, fetch next block: start at:238 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:241 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_prvdr_type_cd INTEGER, mapi_query_part:33: car_line_prvdr_type_cd INTEGER, fetch next block: start at:243 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:246 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_cms_type_srvc_cd VARCHAR(255), mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255), fetch next block: start at:248 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:251 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_place_of_srvc_cd INTEGER, mapi_query_part:36: car_line_place_of_srvc_cd INTEGER, fetch next block: start at:253 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:256 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_hcpcs_pmt_amt INTEGER mapi_query_part:27: car_hcpcs_pmt_amt INTEGER fetch next block: start at:258 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:261 got next block: length:0 text: got complete block: text:☺ read_line:☺ more>) ; mapi_query_part:4:) ; fetch next block: start at:263 got next block: length:3 text:&3 got complete block: text:&3 read_line:&3 allocating new result set fetch next block: start at:266 got next block: length:0 text: got complete block: text:☺ read_line:☺ operation successful (289.145ms) closing result set sql> fetch next block: start at:268 got next block: length:0 text: got complete block: text:☺ read_line:☺ sql>COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Directory\BSAPUF\2008 \2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','\"' NULL AS ' ' ; mapi_query_part:160:COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Direc tory\BSAPUF\2008\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n ','\"' NULL AS '' ; fetch next block: start at:270 got next block: length:252 text:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail) !!OS: The process cannot access the file because it is being used by another pro cess. !GDKunlink(bat\13\1327.theap) !!OS: The process cannot access the file because it is being used by another pro cess. ! got complete block: text:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail) !!OS: The process cannot access the file because it is being used by another pro cess. !GDKunlink(bat\13\1327.theap) !!OS: The process cannot access the file because it is being used by another pro cess. ! read_line:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail) allocating new result set got complete block: text:!!OS: The process cannot access the file because it is being used by anothe r process. !GDKunlink(bat\13\1327.theap) !!OS: The process cannot access the file because it is being used by another pro cess. ! read_line:!!OS: The process cannot access the file because it is being used by a nother process. got complete block: text:!GDKunlink(bat\13\1327.theap) !!OS: The process cannot access the file because it is being used by another pro cess. ! read_line:!GDKunlink(bat\13\1327.theap) got complete block: text:!!OS: The process cannot access the file because it is being used by anothe r process. ! read_line:!!OS: The process cannot access the file because it is being used by a nother process. got complete block: text:! read_line:! fetch next block: start at:522 got next block: length:0 text: got complete block: text:☺ read_line:☺ GDKerror:!ERROR: GDKunlink(bat\13\1327.tail) !OS: The process cannot access the file because it is being used by another proc ess. GDKunlink(bat\13\1327.theap) !OS: The process cannot access the file because it is being used by another proc ess. closing result set sql> ************************** (3) - remote desktop into server so networked disk becomes local disk - works ----- read_line:[ "monet_release", "Oct2012-SP3" ] fetch next block: start at:196 got next block: length:0 text: got complete block: text:☺ read_line:☺ Database: MonetDB v11.13.9 (Oct2012-SP3), 'bsa' closing result set Type \q to quit, \? for a list of available commands auto commit mode: on mapi_query:46:SET TIME ZONE INTERVAL '-05:00' HOUR TO MINUTE fetch next block: start at:198 got next block: length:3 text:&3 got complete block: text:&3 read_line:&3 allocating new result set fetch next block: start at:201 got next block: length:0 text: got complete block: text:☺ read_line:☺ closing result set sql>create table mytable3 ( mapi_query_part:24:create table mytable3 ( fetch next block: start at:203 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:206 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_id VARCHAR(255), mapi_query_part:27: car_line_id VARCHAR(255), fetch next block: start at:208 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:211 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> bene_sex_ident_cd INTEGER, mapi_query_part:28: bene_sex_ident_cd INTEGER, fetch next block: start at:213 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:216 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> bene_age_cat_cd INTEGER, mapi_query_part:26: bene_age_cat_cd INTEGER, fetch next block: start at:218 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:221 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_icd9_dgns_cd VARCHAR(255), mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255), fetch next block: start at:223 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:226 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_hcpcs_cd VARCHAR(255), mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255), fetch next block: start at:228 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:231 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_betos_cd VARCHAR(255), mapi_query_part:33: car_line_betos_cd VARCHAR(255), fetch next block: start at:233 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:236 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_srvc_cnt INTEGER, mapi_query_part:28: car_line_srvc_cnt INTEGER, fetch next block: start at:238 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:241 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_prvdr_type_cd INTEGER, mapi_query_part:33: car_line_prvdr_type_cd INTEGER, fetch next block: start at:243 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:246 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_cms_type_srvc_cd VARCHAR(255), mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255), fetch next block: start at:248 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:251 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_line_place_of_srvc_cd INTEGER, mapi_query_part:36: car_line_place_of_srvc_cd INTEGER, fetch next block: start at:253 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:256 got next block: length:0 text: got complete block: text:☺ read_line:☺ more> car_hcpcs_pmt_amt INTEGER mapi_query_part:27: car_hcpcs_pmt_amt INTEGER fetch next block: start at:258 got next block: length:3 text:☺☻ got complete block: text:☺☻ read_line:☺☻ fetch next block: start at:261 got next block: length:0 text: got complete block: text:☺ read_line:☺ more>) ; mapi_query_part:4:) ; fetch next block: start at:263 got next block: length:3 text:&3 got complete block: text:&3 read_line:&3 allocating new result set fetch next block: start at:266 got next block: length:0 text: got complete block: text:☺ read_line:☺ operation successful (276.500ms) closing result set sql> fetch next block: start at:268 got next block: length:0 text: got complete block: text:☺ read_line:☺ sql>COPY 9676440 offset 2 records INTO mytable3 FROM 'v:\temp\2008_BSA_Carrier_L ine_Items_PUF_1.csv' USING DELIMITERS ',','\n','\"' NULL AS '' ; mapi_query_part:141:COPY 9676440 offset 2 records INTO mytable3 FROM 'v:\temp\20 08_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','\"' NULL AS '' ; fetch next block: start at:270 got next block: length:14 text:&2 9676439 -1 got complete block: text:&2 9676439 -1 read_line:&2 9676439 -1 allocating new result set fetch next block: start at:284 got next block: length:0 text: got complete block: text:☺ read_line:☺ 9676439 affected rows (2m 14s) closing result set sql>

2 2

TPCH Throughput/latency for MonetDB
by Hassan, Ahmad 12 Feb '13

12 Feb '13

Hi, I intend to run TPCH benchmark on MonetDB. I have executed one iteration of benchmark through run.all script provided under sql/benchmarks/tpch. Please may I know if there is a script or framework available to run tpch benchmark for 'X' duration and get throughput and latency numbers for each query? Second, I want to generated data with big scale factor. I searched monetdb mailing list and found out that DBGEN should be used for that. As there is no specific DBGEN provided by monetdb so I have downloaded from http://www.tpc.org/tpch/ . But please let me know which of the following database shall I select while building DBGEN. Official DBGEN supports the following database options: INFORMIX, DB2, TDAT (Teradata), SQLSERVER, SYBASE, ORACLE, VECTORWISE Thanks for the help Kind Regards, Ahmad

1 0

monetdb status health
by Tapomay Dey 12 Feb '13

12 Feb '13

My questions are simple: what causes crashes? what is health? how do we stop health from degrading? Following is the status of my db- msearch_stats_db: database name: msearch_stats_db state: running locked: no scenarios: mal sql msql start count: 140 stop count: 1 crash count: 138 current uptime: 1m 49s average uptime: 15m 33s maximum uptime: 15m 33s minimum uptime: 15m 33s last start with crash: 2013-02-10 17:32:36 last start: 2013-02-10 17:32:47 last stop: 2013-02-06 11:27:15 average of crashes in the last start attempt: 0 average of crashes in the last 10 start attempts: 0.90 average of crashes in the last 30 start attempts: 0.97 Regards, Tapomay.

6 14

Suggested correction for the Java SQLcopyinto example
by Percy Wegmann 09 Feb '13

09 Feb '13

Dear Monet team, I've been getting quite a bit of use out of your example code here: http://dev.monetdb.org/hg/MonetDB/file/tip/java/example/SQLcopyinto.java Recently, though, I ran into a problem. If the "COPY INTO" statement that I issue is wrong, for example the referenced table doesn't exist, Monet comes back with an error. The example isn't checking for this error condition and proceeds to send data anyway. With a really large batch of data, this becomes a problem because the SQL parser ends up trying to parse all that data as if it were SQL and basically chokes on it. The solution is to add something like the following after the out.newLine() on line 86: error = in.waitForPrompt(); if (error != null) { throw new Exception(error); } Cheers, -- Percy Wegmann +1 512 637 8500 ext 148

3 5

Requirement gathering and questions about binary bulk loading.
by Brandon Jackson 08 Feb '13

08 Feb '13

Hello fellow MonetDB'ers, The Pentaho MonetDB Bulk Loader transformation step is now working pretty well in our tests. http://ci.pentaho.com/view/Data%20Integration/job/Kettle contains a continuous integration build of the software if anyone wanted to play with it. New Transformation -> Steps -> Bulk Loading (category)-> MonetDB Bulk Loader Thank you for the previous information about enabling log output from the MonetDB JDBC driver. After killing a stalled "monetdb start ~/sg-farm" thread and restarting it, MonetDB decided to work really well. It was suggested early on that I take a look at the binary bulk loading approach to get the fastest results. Now that I have a streaming ascii version of a bulk loader done for Pentaho Data Integration, I wanted to come back to the binary approach and see what that might look like and what to be aware of. I read http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/BinaryBulkLoad It looks like the requirements are: 1. Write a physical file per column to the disk the monetdb farm exists on. 2. Tables with DATES, TIMESTAMPS or UTF-8 characters ie: Simplified Chinese are verboten and must be handled with COPY INTO .... USING DELIMITERS Assuming the above is a correct understanding, given the command below: copy into Tmp from ('path_to_file_i', 'path_to_file_f', 'path_to_file_s'); 1. If run from a computer separate from the machine running the monetdb server, would the 'path_to_file_i' be a pathname relative to the monetdb farm root directory on the remote server machine? 2. This process should work for appending new data on to an existing table, right? 3. Am I out of luck if my table has any of the verboten data types mentioned above mixed in with supported binary bulkloadable types; aka C-array style data types? 4. If not out of luck for a mixed type table, what is the procedure to load the table? In practice I imagine I will not need to fret too much about DATES needing a COPY INTO .... USING DELIMITERS ascii streaming approach because it is likely part of a dimension table with integer keys matching integers on a fact table, which is far more likely to have the supported types MonetDB Binary Bulk-loading likes. Is it true that MonetDB does not handle joins well, but instead prefers single, denormalized tables with many columns? >From what you know of the internals of MonetDb, is the effort to make some kind of streaming binary bulk loader worth it in terms of execution time on the database converting to native binary types? The documentation used the words, 'slightly faster', but I do not have a feel for when 'slightly' makes a big difference. Thanks again for considering these questions. Brandon

2 1

High memory usage while loading data with many columns
by Andrew Duffey 08 Feb '13

08 Feb '13

Hi, I'm loading data into a table with >20,000 columns and several hundred thousand rows using "COPY INTO". It works, but I can't seem to make it take less than about 30GB of memory. I've tried specifying the number of records as well. Is there any way to reduce memory usage when loading? Is this much memory usage expected? It's a shame because our queries take very little memory. We have to size our machine simply to handle the load. It's impressive (and very convenient) that I can actually have a table with so many columns, but are there any other limitations I might come across? I've just been trying to keep the column count in the queries low. A "select *" seems to crash MonetDB. Thanks, Andrew

2 8

Data storage capacity in monetdb
by KHALED DEHDOUH 06 Feb '13

06 Feb '13

I installed the version of windows for monetdb http://www.monetdb.org/Downloads. I installed a database column, but when I went over 30 GB of stokage given. the query response time has become very slow. Please, I have two questions: - Are this version is limited or no storage ? - I have noticed that the name of the database is by default "demo". Can i create another one with this version with 1 TB?

4 3

Create stored procedure to add a list of ID's to a table
by Richard Siebeling 05 Feb '13

05 Feb '13

Hello, is it possible to create a stored procedure in MonetDB that will have one parameter containing a string with several thousands of comma-separated ID's and insert these comma-separated ID's as separate rows into a table? I am receiving thousands of ID's from a separate webservice and want to join these ID's to a table in monetDB, aggregate the results and return them back to the client for further processing in the GUI. I tried inserting the ID's into a temporary table using insert into instructions from PHP, but that didn't gave me the performance I wanted (about 6 seconds for 10.000 ID's). So the next approach would be to create a stored procedure receiving all the ID's in one parameter, splitting them and inserting them one-by-one into the temporary table... Or is there a better approach for this problem? thanks in advance, Richard

2 1

Memory allocations in MonetDB
by Hassan, Ahmad 04 Feb '13

04 Feb '13

Hi Team, I am running monetdb with default settings. After profiling the application instructions, I have noticed that all the memory allocation are currently being performed by the following functions: GDKmalloc, GDKmallocmax, GDKzalloc, HEAPalloc So mmap and malloc are being used with in all GDK* and HEAPalloc. I have three questions please: 1) Which memory allocations of monetdb are performed through mmap and which memory allocation are performed through malloc. I know this is very open ended question but little bit info about this would be helpful. Are the BAT structures allocated through mmap or malloc? 2) One thing I noticed that, when the DB is running and even if I am not running any queries, still the DB is constantly issuing load/store instructions to Main memory. I assume that some thread in monetdb is doing bookkeeping or persistence related stuff. Can you please point out which part of the code is responsible for this? I want to exclude that part of the code from profiling 3) When we start monetdb with multiple threads, whether multiple threads use the same common virtual address memory region to allocate memory? Or each thread has their own distinct memory region? Thanks. Kind Regards, Ahmad

2 1

Dynamic where clause in a stored procedure
by Vassilis Papakonstantinou 31 Jan '13

31 Jan '13

Hi all, Is it possible in monetdb to use a dynamic where clause in a selection query inside a stored procedure? More specifically, i want to pass the clause as parameter. Thank you, Vassilis

3 2

[help] I have a simple question about monetdb query plan.
by 이수호 31 Jan '13

31 Jan '13

p{margin:0;padding:0;} Hello, I'm a student of Korea Advanced Institute of Science and Technology in Korea. I want to compare late materialization with join and invisible join on monetdb. But when i typed SQL query, i couldn't choose which query plan to use. There was no choice to me. I think query optimizer choose some join method. Can I choose join method? or control query plan? I installed Monetdb on my desk-top and followed tutorials. Everything is done. There is no error. In this situation, i tried to control query plan. So I read reference on official homepage. But I couldn't find the information that how to control query plan. Can I control query plan? If I can, I really really want to know how do that. Where can i find some example? Please give me your favor.

2 1

caught SIGTERM, starting shutdown sequence
by Tapomay Dey 29 Jan '13

29 Jan '13

MSG merovingian[32005]: caught SIGTERM, starting shutdown sequence 2013-01-29 11:40:53 MSG control[32005]: control channel closed 2013-01-29 11:41:06 MSG merovingian[32005]: caught SIGTERM, starting shutdown sequence 2013-01-29 11:41:40 MSG merovingian[32005]: caught SIGTERM, starting shutdown sequence 2013-01-29 11:41:51 MSG merovingian[32005]: caught SIGTERM, starting shutdown sequence Every time I do a monetdbd stop {path_to_farm} I keep getting this in merovingian logs. But the monetdbd process never ends. This issue was not there on the same installation of monetdb initially. This problem started occurring annoyingly frequently after I renamed my last farm and created a fresh one.(with monetdbd not running) What leads me to stop the farm is that I am unable to connect to the db using mclient(There's another app accessing the DB using JDBC-c3p0). There's is no complaint in the merovingian log. The mclient command just hangs. I get back access once I force kill the monetdbd using kill -9. But it gets back into this state after some time. Running on Feb13 branch some older revision. Ubuntu 12.10 with lots of resources. Is this a known issue. Is there a workaround/config or something.

1 3

best way to get autogenerated primary key
by Mate Birkas 29 Jan '13

29 Jan '13

Hi, this is my first post to this list, so forgive me if this is not the right place to post this. I tried to do my research (google, monetdb.org, python api help) but didn't find an answer. How can I get the auto-generated primary key (integer) after executing an insert. I want to use it in a second insert to another table that references the first. (in PostgreSQL I used the RETURNING clause for this) thanks in advance Mate

3 5

MonetDB crash / termination issue.
by Brian Hood 29 Jan '13

29 Jan '13

Hi All, I'm not quite sure if i'm doing something wrong here but what is see if i performing a 2 table join on a primary key field is tablea and tableb work fine yet tablec which has less data crashes the daemon. SELECT a.ip_dst as ip_dst, b.tcp_dport as tcp_dport, count(*) as num from "myschema".tablea a LEFT JOIN "myschema".tableb b ON (a.guid = b.guid) WHERE b.tcp_dport IS NOT NULL GROUP BY a.ip_dst, b.tcp_dport ORDER BY num DESC LIMIT 20; This one works fine. However when i perform the join from tablea to tablec I get the following. sql>SELECT a.ip_dst as ip_dst, b.udp_dport as udp_dport, count(*) as num from "myschema".tablea a LEFT JOIN "myschema".tablec b ON (a.guid = b.guid) WHERE b.udp_dport IS NOT NULL GROUP BY a.ip_dst, b.udp_dport ORDER BY num DESC LIMIT 20 more>; Connection terminated I've added alot more detail to the pastebin link. I had the same issue with 11.13.5 after upgrading i have run into the same thing also. http://pastebin.com/m4XseSnH If you want to schema information to reproduce please just email me off list. Regards, Brian Hood

3 6

Strange concurrency issue
by Percy Wegmann 28 Jan '13

28 Jan '13

Dear MonetDB Team, We're starting to exercise our MonetDB installation more heavily now and I've run into a strange issue where I get an occasional "COMMIT: failed" message. It seems that the conditions required to produce this are: - Concurrently inserting into two different tables - At the same time, issuing a SELECT COUNT(*) against either of those tables I've logged a bug with more detailed information, including a Java test case: http://bugs.monetdb.org/show_bug.cgi?id=3210 I'm only just starting my investigation, but I suspect that COUNT(*) may be the culprit, since other selects like SELECT * don't seem to have this problem. Does this ring any bells? Are there any special optimizations for COUNT(*) that bypass the underlying BAT and/or have side effects on some shared table somewhere that could be causing a concurrency violation? Thanks, -- Percy Wegmann +1 512 637 8500 ext 148 _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

3 11

Bit functions (XOR and inverse)
by Eduardo Oliveira 28 Jan '13

28 Jan '13

Hi, I'm running MonetDB v11.13.9 (Oct2012-SP3), i have some doubts about bit functions, i was trying the bit functions looking at MySQL docs in MonetDB: http://dev.mysql.com/doc/refman/5.0/en/bit-functions.html And when i used ^, the pow was calculated instead of XOR, the ~ don't work as expected too. I google and I found this http://permalink.gmane.org/gmane.comp.db.monetdb.scm/28212 This patch will be applied on MonetDb, any date of release of MonetDb with ^ already as XOR? Right now what operator should i used to make a xor? I tested xor XOR and MonetDb didn't recognize. If i want to insert in binary form how i do? b'binary' works? Thanks in advance, -- *Eduardo Oliveira * *IT* ***Email:* eduardo.oliveira(a)adclick.pt *Web: *www.adclickint.com **<http://www.adclickint.com/> * * <http://www.adclickint.com/> <http://www.adclickint.com/>

3 6

Maximum number of redirects reached, aborting connection attempt. Sorry.
by Brandon Jackson 28 Jan '13

28 Jan '13

I am using a release version of MonetDB 11.13.9 "OCT2012-SP3" on Fedora 18, running as an unprivileged user with no firewall. The server quit allowing connections via JDBC and mclient this morning. mclient hangs indefinitely after asking for a password run from the same machine and account that started mserver5 via monetdbd. "monetdb status", reports a healthy database. JDBC connections initiated using monetdb-jdbc-2.8.jar through Pentaho Data Integration. "Maximum number of redirects reached, aborting connection attempt. Sorry." is the error message. Shutting down MonetDB seemed to work. It also seems to start up okay, but still does not allow any connections. I am happy to install MonetDB with debug symbols and provide some output from gdb if that would be helpful. Please let me know a few things I can try that may be helpful for filing a bug report and getting us back online. Thanks for your help, Brandon The stack trace that came back to java was this: Error connecting to database [monet_testing] : org.pentaho.di.core.exception.KettleDatabaseException: Error occured while trying to connect to the database Error connecting to database: (using class nl.cwi.monetdb.jdbc.MonetDriver) Maximum number of redirects reached, aborting connection attempt. Sorry. org.pentaho.di.core.exception.KettleDatabaseException: Error occured while trying to connect to the database Error connecting to database: (using class nl.cwi.monetdb.jdbc.MonetDriver) Maximum number of redirects reached, aborting connection attempt. Sorry. at org.pentaho.di.core.database.Database.normalConnect(Database.java:366) at org.pentaho.di.core.database.Database.connect(Database.java:315) at org.pentaho.di.core.database.Database.connect(Database.java:277) at org.pentaho.di.core.database.Database.connect(Database.java:267) at org.pentaho.di.core.database.DatabaseFactory.getConnectionTestReport(DatabaseFactory.java:86) at org.pentaho.di.core.database.DatabaseMeta.testConnection(DatabaseMeta.java:2464) at org.pentaho.ui.database.event.DataHandler.testDatabaseConnection(DataHandler.java:533) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:329) at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:139) at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:123) at org.pentaho.ui.xul.swt.tags.SwtButton.access$500(SwtButton.java:26) at org.pentaho.ui.xul.swt.tags.SwtButton$4.widgetSelected(SwtButton.java:119) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.eclipse.jface.window.Window.runEventLoop(Window.java:820) at org.eclipse.jface.window.Window.open(Window.java:796) at org.pentaho.ui.xul.swt.tags.SwtDialog.show(SwtDialog.java:378) at org.pentaho.ui.xul.swt.tags.SwtDialog.show(SwtDialog.java:304) at org.pentaho.di.ui.core.database.dialog.XulDatabaseDialog.open(XulDatabaseDialog.java:115) at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.open(DatabaseDialog.java:62) at org.pentaho.di.ui.trans.steps.monetdbbulkloader.MonetDBBulkLoaderDialog$20.widgetSelected(MonetDBBulkLoaderDialog.java:1350) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.trans.steps.monetdbbulkloader.MonetDBBulkLoaderDialog.open(MonetDBBulkLoaderDialog.java:905) at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:136) at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:7835) at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:2749) at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:705) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1221) at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7044) at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:8304) at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:580) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.pentaho.commons.launcher.Launcher.main(Launcher.java:134) Caused by: org.pentaho.di.core.exception.KettleDatabaseException: Error connecting to database: (using class nl.cwi.monetdb.jdbc.MonetDriver) Maximum number of redirects reached, aborting connection attempt. Sorry. at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:502) at org.pentaho.di.core.database.Database.normalConnect(Database.java:350) ... 51 more Caused by: java.sql.SQLException: Maximum number of redirects reached, aborting connection attempt. Sorry. at nl.cwi.monetdb.jdbc.MonetConnection.<init>(MonetConnection.java:254) at nl.cwi.monetdb.jdbc.MonetDriver.connect(MonetDriver.java:171) at java.sql.DriverManager.getConnection(DriverManager.java:582) at java.sql.DriverManager.getConnection(DriverManager.java:185) at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:482) ... 52 more Caused by: nl.cwi.monetdb.mcl.MCLException: Maximum number of redirects reached, aborting connection attempt. Sorry. at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:255) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:248) at nl.cwi.monetdb.jdbc.MonetConnection.<init>(MonetConnection.java:232) ... 56 more Hostname : 172.27.0.209 Port : 50000 Database name : testing

3 2

How to handle too much data ?
by mail_server＠lavabit.com 28 Jan '13

28 Jan '13

Hi, i am working on monetdb so as it can be used for handling huge streaming data, in the most efficient way. i am having a requirement of sustained inserts of rate around 50k-100k inserts per sec(maybe more).i have tested up the results on big enough data (upto 100Gb) and it's the fastest one, to be frank.Most data is 8 byte longs,1 column is time in secs and 1 column is string based.FYI,Data writer in single threaded. However,the issue is too much data and it will be going on for months( maybe years) and all queries have a range part.Most queries are on current data, while some will be for ages old data.Horizontal partitioning is the way to go as it seems. so what i thought was for efficiency, dump the arrays as per time in secs and then when a query comes, somehow form a new BAT (maybe using FUSE, since copying is unusable), join the data in temporary way and use it and discard.Few secs delay is okay even for the range query of 1 sec. as i have been digging around, http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/BinaryBulkLoad it's possible for full size columns. but since this is going to be a copy,it's inefficient. is there a way to get it done ( already exists) ?? any pointers might help such as BAT Partition Manager and so on. if NOT, i may try to get down and get it done but some pointers might be useful from the experts. Thanks for help. mike

1 0

Unsigned integers
by Eduardo Oliveira 23 Jan '13

23 Jan '13

Hello, I would like to know if MonetDb supports unsigned integer, i couldn't find anything on documentation. I tried to created a table with unsigned int and i get an error. Thank you, -- *Eduardo Oliveira * *IT* ***Email:* eduardo.oliveira(a)adclick.pt *Web: *www.adclickint.com **<http://www.adclickint.com/> * * <http://www.adclickint.com/> <http://www.adclickint.com/>

2 1

Announcement: New Oct2012-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 22 Jan '13

22 Jan '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2012-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2012-SP3 bugfix release Bug Fixes * 2291: small doubles end up as NULL after arithmetic * 2579: SQL: incorrect cast from double to int * 2659: tuples INTERSECT vs count (... INTERSECT ...) differs * 2805: row_number doesn't work in complex query * 2886: warning: Catalog parameter '' ignored * 2977: timestamp minus date results in program error * 2978: now() + 1 results in illegal argument * 2999: Up-cast from SQL drops order information? * 3048: algebra.join undefined when using multiple arithmetic operations in SQL where-clause * 3124: insert into table with two foreign keys crashes mserver5 * 3136: Memleak (in querycache?) while querying * 3143: Type resolution error in SQL procedural code * 3163: mserver5 crashes with large number of concurrent clients * 3168: crash when insert varchar/char/clob with default NULL * 3169: php driver uses call-time pass-by-reference * 3173: SQL queries with limit and offset clauses are unable to use bind parameters for either * 3176: Same query and different results * 3181: Join query returns zero results when there is an index on the tables * 3185: inconsistency between documentation and implementation of command line options * 3186: case evaluted to early * 3188: Storage of client history * 3189: Overflow in calculation * 3190: When subtracting two columns of type "timestamp", no results are returned * 3191: rel_optimizer.c:5596: _rel_optimizer: Assertion `0' failed. * 3192: "No such column with index" error if CASE in SELECT clause * 3199: Inconspicuous sequence of prepare, execute, commit and close produces error * 3205: Inet datatype handling inconsistent between int, cast( as inet) and string * 3208: Unable to execute prepared statements with a large number of arguments * 3209: exp_bin: Assertion `0' failed * 3212: Query cache does not work properly

1 0

SQL function as non-privileged user
by Tapomay Dey 21 Jan '13

21 Jan '13

I have created a separate schema and user as follows: echo "CREATE USER \"tapomay\" WITH PASSWORD '~!@#$' NAME 'Ext User' SCHEMA \"sys\";" | mclient -d my_db echo "CREATE SCHEMA \"test_schema\" AUTHORIZATION \"tapomay\";" | mclient -d my_db echo "ALTER USER \"tapomay\" SET SCHEMA \"test_schema\";" | mclient -d my_db When logged in as "tapomay", I am able to execute curdate() but can't execute date_to_str(). I can run date_to_str() when logged in as the super user monetdb. There could be many other such utility functions. How do I enable all of them for the new user "tapomay". Regards, Tapomay.

2 1

BAT Data structures of a Schema
by Hassan, Ahmad 21 Jan '13

21 Jan '13

Hi Martin, I have the following list of tables in database schema: sql>\d TABLE sys.customer TABLE sys.lineitem TABLE sys.nation TABLE sys.orders TABLE sys.part TABLE sys.partsupp TABLE sys.region TABLE sys.supplier Please can you point me to the C code file(s) that hold the BAT objects for each column of the above tables please? I want to debug and analyse specifically that part of the code region that declares, initialises and maintains the list of BAT structures for the tables. The following query is providing the information about the tables and I am assuming that this query also access this information from all the individual BAT structures of a database schema. select * from storage() where "table" = 'lineitem'; Secondly, Is it possible to list of get the list of all the BAT structures (Column BATS and Delta BATs) in the running database please? Thanks. Kind Regards, Ahmad

4 6

Inconsistency in data selection. count*
by Tapomay Dey 21 Jan '13

21 Jan '13

select count(*) from table; Results: 168466 rows select count(*) from table where column=0; Results: 168466 rows select count(*) from table where column<>0; Results: 168466 rows When I do a "select column from table where client_id=0 limit 5;" I see correct non-zero values in the result although the selection condition is that the column value must be 0. I put the data on another instance/machine - I don't see this issue. Plz find a screenshot@ https://picasaweb.google.com/115470548136322675830/January212013?authuser=0… Another case: sql>select count(*) from table2 where column2=0; | 391326 | sql>select count(*) from table2 where column2<>0; | 141108 | sql>select count(*) from table2; | 399326 | sql>select column2 from table2 where column2=0 limit 5; +-----------+ | column2 | +===========+ | 53 | | 53 | | 53 | | 53 | | 53 | +-----------+ 5 tuples (156.631ms) Regards, Tapomay

1 0

Resource limits in MonetDB
by Charles Milutinovic 21 Jan '13

21 Jan '13

I am experimenting with using MonetDB for a project where the most natural implementation would involve ~100K tables (with the huge majority being near empty, a few incredibly massive) with approx. 10 columns each. MonetDB choked on this, with the following error: >!FATAL: BBPextend: trying to extend BAT pool beyond the limit (16384000) I am assuming that I hit this limit because of the extreme amount of tables. I am wondering if there are any other hard limits to be aware of in MonetDB that you could hit before hitting system limits.

3 2

FW: Monetdb on Solaris/Sparc: high system time problem
by Javier Picorel 21 Jan '13

21 Jan '13

Dear Martin, Thanks for your prompt reply. We have verified that the order in which the queries are executed doesn't affect the behavior of the query. The same problem arises at the same queries in any arbitrary order, which implies that it's related to the query itself. In addition, the queries' execution time and utilization are pretty deterministic. We have also verified that there is plenty of memory available and there are no other processes competing for resources (i.e., disk bandwidth, memory). We have used dtrace to profile what functions are being executed by MonetDB while being in the two phases (normal and problematic). In the normal phase, which lasts for a few tens of seconds and the behavior is what we see on Linux, MonetDB mainly access the following functions: - BUNfastins - re_match_no_ignore - strstr When MonetDB enters into that second (problematic) phase, the User/System time is 4% to 21% for a couple of minutes w/o any disk activity or swapping, MonetDB mainly accesses the following functions: - MT_sleep_ms - memset - __pollsys - _pollsys - _save_nv_regs - pselect - select Do these function names help you to understand the problematic behavior? Thanks in advance, Javier ________________________________ From: Martin Kersten <martin(a)monetdb.org<mailto:martin@monetdb.org>> Subject: Re: Monetdb on Solaris/Sparc: high system time problem Date: January 17, 2013 6:56:00 PM GMT+01:00 To: <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Reply-To: Communication channel for MonetDB users <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Dear Javier, Thank you for your report. It is hard to assess the consequences/impact of your particular setup. Your report suggests that something else was also running on your platform during this sequence, at that particular time. It could mean the MonetDB was fighting with another process over resources, i.e. memory and disk bandwidth. It could also indicate a thrashing kernel on shared kernel resources. If the system persists during repetitive runs under controlled system settings and it is immune to query order, then we might look further into the issue. regards, Martin On 1/17/13 3:44 PM, Javier Picorel wrote: Dear all, I'm running MonetDB (Dec2011-SP2) on Sparc Solaris 10 u9. I'm testing the TPC-H queries using SF10 and SF100 as the scale factors. My machine has 4 cores and 32GB of RAM and my farm is on a disk array. Unfortunately, there are some queries (e.g., Qry 7, 8, 9) that enters into phase in which the User/System time is 4% to 21% for a couple of minutes. Interestingly, the same issue doesn't show up on a Linux machine with a similar setup and the queries' system time is pretty low (<5%). Interestingly, vmstat output doesn't show any disk activity during the phase where the process is just in system, only there are high values on "re" and "mf" fields. There is no swapping going on ("po" and "pi" values are 0 in vmstat). These are the parameters of the DB: type default database shared default yes nthreads default 4 optpipe default_pipe master default no slave default <unknown> readonly default no nclients default 64 The way that I test the query is by running just one at a time. How can I mitigate this OS time? Thanks in advance, Javier _______________________________________________ users-list mailing list users-list(a)monetdb.org<mailto:users-list@monetdb.org> http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org<mailto:users-list@monetdb.org> http://mail.monetdb.org/mailman/listinfo/users-list

2 1

MonetDB under LLVM
by Hassan, Ahmad 18 Jan '13

18 Jan '13

Hi Team, Has anyone tried compiling/running MonetDB under LLVM/CLANG compiler framework please? Normally the use case is: 1) Produce set of llvm bitcode files *.bc from C code 2) Apply LLVM Pass to instrument *.bc files through llvm 'OPT' compiler tool 3) Link all *.bc files and produce executables of monetdb I am going to try monetdb under llvm. If anyone hasn't done it before then I may ask for the initial pointers to compile monedb under llvm instrumentation. Thanks. Kind Regards, Ahmad _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

5 8

Email archives for the forums
by Tapomay Dey 18 Jan '13

18 Jan '13

I don't see any forum archives since you switched from sourceforge. It's helpful to refer to other email threads. Is it there somewhere? I see the older one here: http://mail.monetdb.org/pipermail/developers-list/ and till 8th Jan '13 on sourceforge. Regards, Tapomay. _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

3 2

Monetdb on Solaris/Sparc: high system time problem
by Javier Picorel 17 Jan '13

17 Jan '13

Dear all, I'm running MonetDB (Dec2011-SP2) on Sparc Solaris 10 u9. I'm testing the TPC-H queries using SF10 and SF100 as the scale factors. My machine has 4 cores and 32GB of RAM and my farm is on a disk array. Unfortunately, there are some queries (e.g., Qry 7, 8, 9) that enters into phase in which the User/System time is 4% to 21% for a couple of minutes. Interestingly, the same issue doesn't show up on a Linux machine with a similar setup and the queries' system time is pretty low (<5%). Interestingly, vmstat output doesn't show any disk activity during the phase where the process is just in system, only there are high values on "re" and "mf" fields. There is no swapping going on ("po" and "pi" values are 0 in vmstat). These are the parameters of the DB: type default database shared default yes nthreads default 4 optpipe default_pipe master default no slave default <unknown> readonly default no nclients default 64 The way that I test the query is by running just one at a time. How can I mitigate this OS time? Thanks in advance, Javier _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 1