- users-list - monetdb.org

Join memory overhead, COPY INTO csv, MT_mremap failure.
by Valentin Reis 09 Apr '15

09 Apr '15

Hello all, To the purpose of evaluating MonetDB, I am trying to perform the following: Load two csv files into two tables, from disk A. Perform a JOIN operation on two fields, using disk B for the database. Pipe the result to a csv file, on disk A. This results in a very high disk B usage, and subsequent MT_mremap failure, as seen in merovingian.log: 2015-04-08 12:50:14 ERR google-trace[2737]: = gdk_posix.c:428: MT_mremap(./bat/06/675.tail,7e8738e90000,91992817664,110391328768): GDKextendf() failed Does somebody have ideas or explanations? Details follow: I am loading csv data from hard drive A. The files have size: -rw-r--r-- 1 fre fre 2.6G Apr 5 16:25 task_events_cut.csv -rw-r--r-- 1 fre fre 34G Apr 5 16:04 task_usage_cut.csv I create a database in a separate hard drive B (2TB): rm -rf /mnt/diskB/mdb mkdir /mnt/diskB/mdb/ monetdbd create /mnt/diskB/mdb monetdbd start /mnt/diskB/mdb monetdb create google-trace monetdb release google-trace I then load the data into the database: mclient ct_trace_events_reduced.sql -d google-trace pv /mnt/diskA/task_events_cut.csv |mclient -d google-trace -s "COPY INTO task_events_reduced FROM STDIN USING DELIMITERS ',','\\n'" - mclient ct_trace_usage_reduced.sql -d google-trace pv /mnt/diskA/task_usage_cut.csv |mclient -d google-trace -s "COPY INTO task_usage_reduced FROM STDIN USING DELIMITERS ',','\\n'" - using very standard scripts: $ cat ct_trace_events_reduced.sql ct_trace_usage_reduced.sql DROP TABLE task_events_reduced; CREATE TABLE task_events_reduced ( "job_id" BIGINT , "task_id" BIGINT , "class" SMALLINT , "priority" SMALLINT); DROP TABLE task_usage_reduced; CREATE TABLE task_usage_reduced ( "job_id" BIGINT , "task_id" BIGINT , "cpu_mean" FLOAT , "cpu_sample" FLOAT); These two operations take about 50 minutes, which is very reasonable. I then use mclient to do my join: mclient join.sql -d google-trace using the script: $ cat join.sql COPY (SELECT te.job_id, te.task_id, te.class, te.priority, tu.cpu_mean, tu.cpu_sample FROM (SELECT * FROM task_events_reduced )AS te RIGHT JOIN (SELECT * FROM task_usage_reduced )AS tu ON(te.job_id=tu.job_id AND te.task_id=tu.task_id) ) INTO '/diskA/join.csv' USING DELIMITERS ',','\n'; This results in more than three hours of data crunching on a google compute engine machine (16 processors and 100GB RAM), where disk B is being increasingly used, until it is full (2TB HDD). Then, the aforementionned error happens. I am not hoping that MonetDB would perform streaming I/O on the right file from the join. However the disk usage seems quite high. Is there a way to force MonetDB to do a hash join? Thanks a lot, Valentin Reis University of Warsaw

4 7

MonetDB Real time analytics
by Luciano Sasso 09 Apr '15

09 Apr '15

Hello, I can work in real time with the MonetDB, for example, a job extracting and inserting data in MonetDB as stream and other job for reading data, aggregation and creation of indicators for dashboads in real or near real time ? -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

1 0

RE: Details about MonetDB, data loss in restart server (Ying Zhang)
by Luciano Sasso 08 Apr '15

08 Apr '15

Hi, - OS is Debian 7 64-bit, in a VM in VMware virtualization System. - Hard disk 100G with 26% usage. - 5 cores. - 6G RAM. - Version MonetDB: MonetDB 5 server v11.15.17 "Feb2013-SP5" - Table: create table (id int, dt date, product varchar (50) .... etc, and another 30 columns between int and varchar. - Query: select dt, count (id) from table where produtct in (....) group by dt - History: Long time MonetDB use with Pentaho (Saiku) for analysis of data cubes, never had problems, works great with millions of records. When the problem occurred, the steps taken were: 1- A couple of days before it occurred, I made a 32 million load records to the table. 2- I used this table for many analyzes via SQL client (without using cubes) and everything was fine. 3- I created a cube in Pentaho for this table and also used a lot and everything went well. 4- When I put the dt fields in the cube and the interface seems to have caught the Pentaho and mserver held all machine resources (memory and CPU), less disk. This lock may have been because the Pentaho, but specifically the mondrian, have had a problem with the date field 5- I tried to stop the Pentaho, but did not work, had to force a kill -9 pid. 6- In the case of MonetDB, the command MonetDB stop dw, worked properly and stopped the service. 7- When I run MonetDB start dw, it took a while longer than usual, but the service start usually without showing error on console. 8- With other tables there were no data loss problem, only that table lost data just the product column, which was used in the query, the remaining columns have not suffered problems. But but at no time update or delete command has been executed, only selects were executed 9- All though I forced Pentaho standing in the execution of the query, but MonetDB stopped normal, and a query should never change data, of course? Log 2015-03-31 15:58:38 MSG control[1114]: (local): served status list 2015-03-31 15:58:38 MSG merovingian[1114]: sending process 14173 (database 'dw-zoombox') the TERM signal 2015-03-31 15:58:39 MSG merovingian[1114]: database 'dw-zoombox' (14173) has exited with exit status 0 2015-03-31 15:58:39 MSG merovingian[1114]: database 'dw-zoombox' has shut down 2015-03-31 15:58:39 MSG control[1114]: (local): stopped database 'dw-zoombox' 2015-03-31 15:58:44 MSG control[1114]: (local): served status list 2015-03-31 15:58:44 MSG merovingian[1114]: starting database 'dw-zoombox', up min/avg/max: 2d/9w/18w, crash average: 0.00 0.10 0.03 (3-2=1) 2015-03-31 15:58:45 MSG dw-zoombox[19366]: arguments: /usr/bin/mserver5 --dbpath=/dados/dbfarm/dw-zoombox --set merovingian_uri=mapi:monetdb://SRVBIL01:50000/dw-zoombox --set mapi_open=false --set mapi_port=0 --set mapi_usock=/dados/dbfarm/dw-zoombox/.mapi.sock --set monet_vault_key=/dados/dbfarm/dw-zoombox/.vaultkey --set gdk_nr_threads=5 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # MonetDB 5 server v11.15.17 "Feb2013-SP5" 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Serving database 'dw-zoombox', using 5 threads 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Found 11.765 GiB available main-memory. 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Copyright (c) 1993-July 2008 CWI. 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Visit http://www.monetdb.org/ for further information 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Listening for UNIX domain connection requests on mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # MonetDB/SQL module loaded 2015-03-31 15:59:34 MSG control[1114]: (local): started 'dw-zoombox' 2015-03-31 15:59:34 MSG merovingian[1114]: proxying client 192.168.1.7:49105 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 15:59:34 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:05:00 MSG merovingian[1114]: proxying client 192.168.1.7:49117 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:05:00 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:05:20 MSG merovingian[1114]: proxying client 192.168.1.7:49119 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:05:20 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:06:05 ERR discovery[1114]: cannot retrieve name info: Temporary failure in name resolution 2015-03-31 16:16:05 ERR discovery[1114]: cannot retrieve name info: Temporary failure in name resolution 2015-03-31 16:16:16 MSG merovingian[1114]: proxying client 192.168.1.8:50220 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:16:16 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:17:04 MSG merovingian[1114]: proxying client 192.168.1.7:49159 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:17:04 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 2 Apr 2015 23:16:39 +0200 > From: Ying Zhang <Y.Zhang(a)cwi.nl> > To: "Communication channel for developers of the MonetDB suite." > <developers-list(a)monetdb.org> > Cc: users-list(a)monetdb.org > Subject: Re: MonetDB, data loss in restart server > Message-ID: <6C2903C0-4399-462D-9D3F-822351DE1979(a)cwi.nl> > Content-Type: text/plain; charset=windows-1252 > > Hello Luciano, > > Thanks for using monetdb, and sorry for the inconvenience. > Would you please give us some background information about your setting: > > - which OS? > - hardware spec. of your system > - which monetdb version? > - what does your query do? > - when and how did the database stop? If the monetdb server stopped, did it give any error message? > - what's at the end of the merovingian log? > > With kind regards, > > Jennie > >> On Apr 01, 2015, at 06:46, Luciano Sasso <luciano(a)gsgroup.com.br> wrote: >> >> Hello, >> >> I have used the MonetDB in production for some time, and I have nothing to complain about, >> are tables with more than 30 columns and 30 million records, do analysis through cubes in Pentaho / Saiku. >> Have restarted the server several times, and never had problems with loss of data.But today, >> after performing a query that was consuming a lot of machine, stopped the database, and when I started, >> data from some columns were blank. How can it, will the data were still uncommitted? >> when performing a query the MonetDB changes the data or reorganized? which may have occurred, I am concerned. >> >> >> -- >> <Slice_01.png> <Slice_02.jpg> >> Luciano Sasso Vieira >> Gestor - Intelig?ncia de Neg?cios >> >> luciano(a)gsgroup.com.br | tel: 17 3353-0833 | cel: 17 9706-9335 >> <Slice_04.png> www.gsgroup.com.br >> <Slice_06.png> >> >> >> Este email foi escaneado pelo Avast antiv?rus. >> www.avast.com >> >> >> _______________________________________________ >> developers-list mailing list >> developers-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/developers-list > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > > > ------------------------------ > > End of users-list Digest, Vol 32, Issue 6 > ***************************************** > > -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

2 1

mserver5 segfault at 0 ip error 4 in lib_sql.so
by Caroptions Caroptions 08 Apr '15

08 Apr '15

Hi User-list! I am experiencing segfault monetdb crash, platform: CentOS Linux release 7.1.1503 (Core) 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Table: CREATE TABLE "marketdata"."quotes" ( "id" INTEGER NOT NULL DEFAULT next value for "marketdata"."seq_6391", "qtime" TIMESTAMP WITH TIME ZONE NOT NULL, "sdate" TIMESTAMP WITH TIME ZONE, "sym" VARCHAR(10) NOT NULL, "cur" VARCHAR(10) NOT NULL, "open" DOUBLE NOT NULL, "high" DOUBLE NOT NULL, "low" DOUBLE NOT NULL, "close" DOUBLE NOT NULL, "volume" DOUBLE, CONSTRAINT "quotes_id_pkey" PRIMARY KEY ("id") ); SQL: WITH cte(qtime, open, close, sdate, id, rnk) as ( select qtime, sdate, id, row_number() over (partition by sdate order by qtime asc) as rnk from marketdata.quotes where sym='SPY') select * from cte where rnk=1; ~2M records loaded from csv. Always segfault crashes monetdb daemon and DB became not repairable. Reproducible on MonetDB-11.19.9 installed from yum repository, as well as compiled from sources, with-optimization and without. Also it crashed on latest development source branch MonetDB-11c0c6cf614f with optimization and without. >From mclient and JDBC client. Any help is very appreciated!! I am available to debug / provide additional information if needed. Thanks, John

2 2

RE: MonetDB, data loss in restart server
by Luciano Sasso 08 Apr '15

08 Apr '15

OK, to update, just use apt-get, I will have problems with structure of database? I can use the old dbfarm, will be compatible? On Wed, Apr 08, 2015 at 11:08:53AM -0300, Luciano Sasso wrote: >/ Anyone have any idea about ? />/ />/ />/ - OS is Debian 7 64-bit, in a VM in VMware virtualization System. />/ - Hard disk 100G with 26% usage. />/ - 5 cores. />/ - 6G RAM. />/ - Version MonetDB: MonetDB 5 server v11.15.17 "Feb2013-SP5" /Thats a very old version. I recall a fix which we did in the meantime related to possible data loss, ie we added extra f(data)sync calls. Which could very well be the problem on a virtual system. Niels >/ - Table: create table (id int, dt date, product varchar (50) .... etc, />/ and another 30 columns between int and varchar. />/ - Query: select dt, count (id) from table where produtct in (....) group />/ by dt />/ / -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

2 1

Incorrect results on joining with same table
by Vijay Krishna 08 Apr '15

08 Apr '15

Hi, I am using MonetDB Jan2014 SP1, running on MacOS. I have a table like this. sql>select * from two; +------+---------------------------------------------------------------------------------------------- | id | name | istrue | joindate | jointime | count | +======+==================================================== | 1 | null | false | 2017-12-08 | 2014-06-30 14:05:31.000000 | 2 | | 2 | vijay | false | 2017-12-08 | 2014-06-30 14:05:59.000000 | 3 | | 3 | krish | true | 2017-12-08 | 2014-06-30 14:06:17.000000 | 5 | | 4 | bat | true | null | null | null | | 5 | gotham | false | null | null | null | | 6 | wayne | false | null | null | 3 | +------+------------------------------------------------------------------------------------------------- 6 tuples (3.504ms) I tried to do a join on this table with the same table with the following query and got this result. sql>select two.id, two.name, two.count, twoAlias.id as id2, twoAlias.name as name2 from two left join two twoAlias on two.count=twoAlias.id; +------+------------------------------------------- -------------------------------------------------------+ | id | name | count | id2 | name2 +=============================================================+ | 1 | null | 2 | 2 | vijay | | 2 | vijay | 3 | 3 | krish | | 3 | krish | 5 | 5 | gotham | | 4 | bat | null | 3 | krish | | 5 | gotham | null | null | null | | 6 | wayne | 3 | null | null | +------+-------------------------------- --------------------------------------------------------------------+ 6 tuples (5.605ms) As it could be seen clearly, *this join is clearly resulting in a wrong output* at the 4th row. *The join criterion two.count=twoAlias.id is satisfied in the first 3 rows, but are clearly violated in the 4th row*, where in the values in the join columns (count and id2 here) are clearly found to be different. Hence the join rows are not matching. I expected that the output would be +------+------------------------------------------- -------------------------------------------------------+ | id | name | count | id2 | name2 +=============================================================+ | 1 | null | 2 | 2 | vijay | | 2 | vijay | 3 | 3 | krish | | 3 | krish | 5 | 5 | gotham | | 4 | bat | null | null | null | | 5 | gotham | null | null | null | | 6 | wayne | 3 | null | null | +------+-------------------------------- --------------------------------------------------------------------+ 6 tuples (5.605ms) Is this really a bug?? Else, what would be the logic behind this? Thanks in advance. With Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

2 2

RE: MonetDB, data loss in restart server
by Luciano Sasso 08 Apr '15

08 Apr '15

Anyone have any idea about ? - OS is Debian 7 64-bit, in a VM in VMware virtualization System. - Hard disk 100G with 26% usage. - 5 cores. - 6G RAM. - Version MonetDB: MonetDB 5 server v11.15.17 "Feb2013-SP5" - Table: create table (id int, dt date, product varchar (50) .... etc, and another 30 columns between int and varchar. - Query: select dt, count (id) from table where produtct in (....) group by dt Message: 1 Date: Thu, 2 Apr 2015 23:16:39 +0200 From: Ying Zhang<Y.Zhang at cwi.nl <https://www.monetdb.org/mailman/listinfo/users-list>> To: "Communication channel for developers of the MonetDB suite." <developers-list at monetdb.org <https://www.monetdb.org/mailman/listinfo/users-list>> Cc:users-list at monetdb.org <https://www.monetdb.org/mailman/listinfo/users-list> Subject: Re: MonetDB, data loss in restart server Message-ID:<6C2903C0-4399-462D-9D3F-822351DE1979 at cwi.nl <https://www.monetdb.org/mailman/listinfo/users-list>> Content-Type: text/plain; charset=windows-1252 Hello Luciano, Thanks for using monetdb, and sorry for the inconvenience. Would you please give us some background information about your setting: - which OS? - hardware spec. of your system - which monetdb version? - what does your query do? - when and how did the database stop? If the monetdb server stopped, did it give any error message? - what's at the end of the merovingian log? With kind regards, Jennie -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

2 1

MonetDB for DW
by Luciano Sasso 08 Apr '15

08 Apr '15

Which of you with experience MonetDB production with large volume of data? You use some NoSQL base for storage of DW, and makes loads for MonetDB, for analysis, or use the monetdDB with dw too? Att, -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

1 0

Cluster, How work ?
by Luciano Sasso 08 Apr '15

08 Apr '15

On a cluster using MonetDB, the documentation is not clear how it works. Data is automatically replicated between cluster nodes? There is an automatic load balancing? There sharding? Att, -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

1 0

Sorting cluster's level queries
by José Raúl Pérez Rodríguez 06 Apr '15

06 Apr '15

Hi, I am new in monetdb. I was quering a multiplex funnel database and obviously (after read the description of multiplex funnel), the return values don't show up sorted. My question is: if using multiplex funnel databases queries can't be sorted, then how are sorted the queries at cluster level, i.e queries that retrieve data from 2 or more nodes? and how to perform that kind of quieries (not using a multiplex funnel). Thanks,

1 0

About "Zero-copy integration between MonetDB and R"
by Aris Setyawan 06 Apr '15

06 Apr '15

Hi all, First, thanks for implementing multi-core, parallel execution engine, ini MonetDB. This make MonetDB, the only opensource dbms, which have this feature. I'm sure, since MonetDB use mmap file, it is already use Zero-copy memory technique when loading data into memory for further processing. But, why the "Zero-copy integration between MonetDB and R" is not implemented in MonetDB release? -Aris

2 1

Building MonetDB from RPM on ROSA Desktop R4
by Brian Hood 05 Apr '15

05 Apr '15

Hi Guys, I know this isn't a support distro at all i'm just trying to get MonetDB to work on my ROSA Desktop R4 machine which is Linux / KDE with a Twist distro. http://pastebin.com/wiH6RPzQ Its all about this at the bottom. 1. cd . && /bin/sh ./config.status rpm.mk 2. config.status: creating rpm.mk 3. cd . && /bin/sh /home/brian/Downloads/rpmbuild/BUILD/MonetDB-11.19.7/buildtools/conf/missing automake-1.13 --foreign 4. configure.ac:372: error: required file 'buildtools/conf/compile' not found 5. configure.ac:372: 'automake --add-missing' can install 'compile' 6. Makefile:521: recipe for target 'Makefile.in' failed 7. make: *** [Makefile.in] Error 1 8. error: Bad exit status from /var/tmp/rpm-tmp.25756 (%build) 9. Automake the wrong version i'm not sure it seems to be upto date. Anybody any ideas on the off chance somebody has seen this before ? Regards, Brian Hood

2 2

Re: MonetDB, data loss in restart server
by Luciano Sasso 05 Apr '15

05 Apr '15

- OS is Debian 7 64-bit, in a VM in VMware virtualization System. - Hard disk 100G with 26% usage. - 5 cores. - 6G RAM. - Version MonetDB: MonetDB 5 server v11.15.17 "Feb2013-SP5" - Table: create table (id int, dt date, product varchar (50) .... etc, and another 30 columns between int and varchar. - Query: select dt, count (id) from table where produtct in (....) group by dt Message: 1 Date: Thu, 2 Apr 2015 23:16:39 +0200 From: Ying Zhang<Y.Zhang(a)cwi.nl> To: "Communication channel for developers of the MonetDB suite." <developers-list(a)monetdb.org> Cc:users-list@monetdb.org Subject: Re: MonetDB, data loss in restart server Message-ID:<6C2903C0-4399-462D-9D3F-822351DE1979(a)cwi.nl> Content-Type: text/plain; charset=windows-1252 Hello Luciano, Thanks for using monetdb, and sorry for the inconvenience. Would you please give us some background information about your setting: - which OS? - hardware spec. of your system - which monetdb version? - what does your query do? - when and how did the database stop? If the monetdb server stopped, did it give any error message? - what's at the end of the merovingian log? With kind regards, Jennie -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

1 0

Re: MonetDB, data loss in restart server
by Ying Zhang 02 Apr '15

02 Apr '15

Hello Luciano, Thanks for using monetdb, and sorry for the inconvenience. Would you please give us some background information about your setting: - which OS? - hardware spec. of your system - which monetdb version? - what does your query do? - when and how did the database stop? If the monetdb server stopped, did it give any error message? - what's at the end of the merovingian log? With kind regards, Jennie > On Apr 01, 2015, at 06:46, Luciano Sasso <luciano(a)gsgroup.com.br> wrote: > > Hello, > > I have used the MonetDB in production for some time, and I have nothing to complain about, > are tables with more than 30 columns and 30 million records, do analysis through cubes in Pentaho / Saiku. > Have restarted the server several times, and never had problems with loss of data.But today, > after performing a query that was consuming a lot of machine, stopped the database, and when I started, > data from some columns were blank. How can it, will the data were still uncommitted? > when performing a query the MonetDB changes the data or reorganized? which may have occurred, I am concerned. > > > -- > <Slice_01.png> <Slice_02.jpg> > Luciano Sasso Vieira > Gestor - Inteligência de Negócios > > luciano(a)gsgroup.com.br | tel: 17 3353-0833 | cel: 17 9706-9335 > <Slice_04.png> www.gsgroup.com.br > <Slice_06.png> > > > Este email foi escaneado pelo Avast antivírus. > www.avast.com > > > _______________________________________________ > developers-list mailing list > developers-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/developers-list

1 0

MonetDB in Production
by Luciano Sasso 02 Apr '15

02 Apr '15

Hi, I use MonetDB production together with Pentaho BI Server and Saiku, works very well and never had problems. I am currently about 30 million entries in a table of about 40 columns, occupying 9G disk. I'll use it now on other projects that this volume will increase a lot. On what basis of size and number of records you have experience and have arrived? Use in production? I was worried because a few days ago I had a corrupt basis, that is, some columns were blank after restarte the bank, do not know if is related to the data base or was disk failure, to join if I do not have evidence of failure disk, but I can not say. Att, -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

1 0

Putting field description in SQL CREATE table
by Puthick Hok 02 Apr '15

02 Apr '15

Hi, I am working on developing a database web service layer for an open source plant/animal breeding platform. So far I rely on having field comment in CREATE TABLE SQL statement like CREATE TABLE "extract" ( "ExtractId" int NOT NULL auto_increment COMMENT 'Id of the DNA extract') ; for the online help. So far, we use MySQL for general storage, postgres for GIS and environment data storage. Both MySQL and Postgres support field comment in slightly different means. We are going to switch to MonetDB as our improved marker data storage. I have tested that the previous CREATE TABLE SQL would not work in MonetDB. Will MonetDB support this feature in the future? Regards, Puthick

3 2

Multiple nodes
by Jiří Nádvorník 02 Apr '15

02 Apr '15

Hi, Does MonetDB support some kind of installation over multiple nodes (servers)? How can I control data partitioning over these nodes? Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic <mailto:nadvornik.ji@gmail.com> nadvornik.ji(a)gmail.com

1 0

C++ user defined function
by Jiří Nádvorník 01 Apr '15

01 Apr '15

Hi, I'm trying to write a function in C++ and I thought I would just provide a makefile for it in my custom folder in MonetDB/sql/backends/monet5/CUSTOM. But it seems like all of the Makefiles in UDF sample are generated by the bootstrap. How do I get over this? I need to use external C++ libraries in the makefile.. Thank you for help. Best regards, Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic <mailto:nadvornik.ji@gmail.com> nadvornik.ji(a)gmail.com

4 9

UDF insert
by Jiří Nádvorník 01 Apr '15

01 Apr '15

Hi, I'm trying to find this in already defined C functions, but with no luck.. What is the set of commands I need to use when I want to insert a row into a table? The row is just array of <dbl, dbl, int, int>. Also, how do I pass a row as a parameter to the function? Do I need to pass all of the columns separately, or can I pass the whole row as an array? Thank you for pointing me the right direction. Cheers, Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic <mailto:nadvornik.ji@gmail.com> nadvornik.ji(a)gmail.com

1 0

MonetDB default character set
by Puthick Hok 01 Apr '15

01 Apr '15

Hi, MonetDB is very good. I'm happy to have found this amazing open source product on the Internet. We want to use MonetDB to store high density marker (genotyping or snp) data. Is the default character set is UTF8 in MonetDB? Regards, Puthick

3 2

Mal description for a BAT function
by Jiří Nádvorník 31 Mar '15

31 Mar '15

Hi, I'm sorry for spamming so much, but I have to get more into the idea of BATs before I will be a little more self-sufficient. I need a function which takes two table columns as parameters. I would come up with this MAL declaration: command build_chunks(ra:bat[:oid,:dbl], dec:bat[:oid,:dbl], nside_low:int, nside_high:int):bat[:oid,:oid] address OSPSbuild_chunks comment "Build HEALPix groups for further processing."; It should take ra, dec parameters (from my DB table) and then some integer parameters. I guess it should return an oid, because all it does is that it should write some computed result (multiple rows) to another table. I have still no idea how I will do that, but I feel I'm getting closer J. The function produces a c++ vector of some integers, which I want to write down to another table. Is that MAL declaration looking correct? How should I call that function? I understand that SELECT reverse('hello'), but I don't know when the parameter is passed from a query. Thank you very much for pointing me in the right direction. Best regards, Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic <mailto:nadvornik.ji@gmail.com> nadvornik.ji(a)gmail.com

1 0

UDF function example
by Jiří Nádvorník 31 Mar '15

31 Mar '15

Hi, When I download the source code from mercurial and run: 1. bootstrap, configure, make, make install 2. monetdbd start myFarm, mclient demo 3. sql>select 'hello',reverse('hello'); The output is: SELECT: no such unary operator 'reverse(char)' What can be wrong here? I had the original installation from packages, then I apt-get removed them and installed via the make install. Any ideas appreciated. Best regards, Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic <mailto:nadvornik.ji@gmail.com> nadvornik.ji(a)gmail.com

3 3

Table partitioning over nodes
by Jiří Nádvorník 31 Mar '15

31 Mar '15

Hi All, I have one table with data that can be grouped easily. I need to iterate over these groups, count their members and distribute them uniformly over the database nodes (physically). If I have three groups A, B, C of size (1, 1, 2) and two physical nodes - I want A,B to be on node no. 1 and C on the node no. 2. The question is quite simple. Is it possible? Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic <mailto:nadvornik.ji@gmail.com> nadvornik.ji(a)gmail.com

1 0

COPY INTO FROM STDIN and auto increment column
by fadila.mumbasic＠uniserv.com 30 Mar '15

30 Mar '15

Hi, We are facing problems with COPY INTO .. FROM STDIN and an auto increment column, which is the primary key column in the table. The client is our own java client. If we don't set any value for this column (which is our expected constellation) there is neither error nor exception, but no record is inserted into the table. A try with a null value for this column produces the same result (no exception, no record in the table) The only way to insert records is to set the proper value for the column, but it is not the supposed behavior of an auto increment column, isn't it? Best regards, Fadila

2 1

SQL string from PreparedStatement ??
by Vijay Krishna 30 Mar '15

30 Mar '15

Hi, Is there any way to get the SQL query as a string from the PreparedStatement of MonetDB?? (Like in some DBs, toString will give the SQL string) Any help much appreciated. Thanks in advance. Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

2 1