- users-list - monetdb.org

Monitoring MonetDB performance
by Alexander Gordt 07 Jan '16

07 Jan '16

Hello together, I’ve got a performance issue with my project, where the DB performance is going down for what ever reason. The activity monitor of OSX shows me the following numbers about the mserver5-process: The CPU usage is very low (15-25% of one core!), the IO traffic on my ssd is fair (about 10-40 MB/s), but the speed slows down. I’ve had a look in the merovingian.log, but there’s only the connection shown. For the first 5000 messages, the CPU usage of mserver5 was about 70-110% - why is it going down?… How could I analyze, why the database performance is going down? Is it possible, that there is an issue with the JDBC-driver, that the performance is decreasing? I attache a log file where you can see that the performance is quite good for some time, but then goes down for whatever reason. 18:40:46 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 15.711 seconds - totally inserted 5200 messages 18:40:46 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:41:03 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 17.189 seconds - totally inserted 5400 messages 18:41:03 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:41:27 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 23.834 seconds - totally inserted 5600 messages 18:41:27 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:42:19 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 52.13 seconds - totally inserted 5800 messages 18:42:19 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:43:09 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 49.892 seconds - totally inserted 6000 messages 18:43:09 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:43:59 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 49.786 seconds - totally inserted 6200 messages 18:43:59 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:45:31 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 92.389 seconds - totally inserted 6400 messages 18:45:31 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:47:26 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 114.412 seconds - totally inserted 6600 messages 18:47:26 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:49:24 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 118.324 seconds - totally inserted 6800 messages One message results in 10-50 insert statements in different tables and the 200 messages are processed in one transaction. The absolute performance isn’t nice, but that is another topic – I’ve read about "copy into“ and that kind of stuff – but that gonna be part in a later optimization. I tried to find out, if there is a long running process, but „select * from sys.queue;“ didn’t show me anything of interest – the result shows only this select. Is there a way to gather some statistics about executed queries and their runtime? Or about blocking resources… Thanks in advance for bringing any light in the black box :) Best regards, Alex

1 1

About JDBC Driver
by Luciano Sasso 07 Jan '16

07 Jan '16

Steps for error: 1- In testing with java client and JDBC 2:19, an error occurs when saving decimal field, for example the value 11:23 is recorded as 13. 2- When I run the same test with the 2.8 driver, saves normally and correct. Not only is the 2.19 driver, all above 2.8 is that same mistake. My Java client is Pentaho Data Integration, My MonetDB is JUL2015-SPI I go zip package test, and send fo tests. Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1

About database crack
by chenkaijiang＠bigknow.com.cn 06 Jan '16

06 Jan '16

Hi, all, I find that in Oct2010 branch (http://dev.monetdb.org/hg/MonetDB/shortlog/Oct2010) it looks like that the database cracking technology has been implemented. But looks like that it wasn't merged into the latest release (Jul2015), right? Why not merge this into the latest release? Is there any plan to merge it in the future?

3 3

MonetDBLite error in R
by Iain Wallace 06 Jan '16

06 Jan '16

hi all, I was wondering if anybody could suggest how to debug this error: Server says 'GDKerror:!ERROR: BATsetaccess: tmp_3505 has -1 views I am trying to do a simple join using dplyr sample%>%left_join(interaction_pairs) The tables appear to be working correctly as I can collect them and use them in R properly interaction_pairs<-collect(interaction_pairs) Thanks Iain

2 1

invalid credentials using mserver5
by Xu,Wenjian 06 Jan '16

06 Jan '16

Hi all, Normally I am using *monetdbd*+"mclient" combination to use MonetDB. Now, since I want to see the debug info in each gdk operator, I have to use *mserver5*. I do as follows (JULY2015-SP1+ubuntu14.04): Step1: $ mserver5 --dbpath=${HOME}/[my_dbpath] --daemon=yes --algorithms Step2: In another console: $ mclient -d [dbname] -u monetdb password: monetdb It replies: InvalidCredentialsException:checkCredentials:invalid credentials for user 'monetdb' However, user:monetdb, pwd:monetdb is the default one and I can use it successfully in *monetdbd*+"mclient" combination. Any idea? Thanks, XU Wenjian

2 1

Jdbc Driver
by Luciano Sasso 05 Jan '16

05 Jan '16

Hi, Its error "https://www.monetdb.org/bugzilla/show_bug.cgi?id=3561", persist in version 2.19, exists most new version with bug fix ? -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1

Dropping foreign key constraint
by Alexander Gordt 05 Jan '16

05 Jan '16

Hello together, I’m very new to MonetDB, so please be gracious :) I’m trying to drop a foreign key, what should be a very simple task – but I didn’t manage to do it. The following statement ran successfully – as the result says – but the foreign key is still there: alter table visit drop constraint visit_configuration_id_fkey; I also tried this – but this command is not supported?? alter table visit drop foreign key visit_configuration_id_fkey; An error occurred when executing the SQL command: alter table visit drop foreign key visit_configuration_id_fkey syntax error, unexpected FOREIGN in: "alter table visit drop foreign" [SQL State=42000] Execution time: 0.01s 1 statement failed. Best regards and thanks for any hints, Alex

2 4

could == be aliased to the = operator? or will that break things?
by Anthony Damico 04 Jan '16

04 Jan '16

hello, just an idea-- if monetdb gets a flood of new R users over the next year, they might get thrown off by things breaking when they use a double-equals sign (example below).. sqlite accepts the double-equals as a single equals. thanks > dbGetQuery(db,'select count(distinct mycolumn) from mytable where yr=2003') L1 1 334343 > dbGetQuery(db,'select count(distinct mycolumn) from mytable where yr==2003') Error in .local(conn, statement, ...) : Unable to execute statement 'select count(distinct mycolumn) from mytable where yr==2003'. Server says 'syntax error, unexpected '=' in: "select count(distinct mycolumn) from mytable where yr=="' [#42000]. >

1 0

Re: Update: Extract Table Statistics after Data import using Monet.Read.Csv
by Anthony Damico 31 Dec '15

31 Dec '15

https://www.monetdb.org/bugzilla/show_bug.cgi?id=3835 next version release should fix this.. or use monetdblite which already has the fix (the example scripts i sent use monetdblite) On Dec 30, 2015 11:53 PM, "Jigar Shah" <jigar.shah1984(a)hotmail.com> wrote: > Thanks a lot Dr. Hannes, Anthony. The trick worked perfectly. See the > stats below: > > > > > > However, there is one odd behavior I have noticed. Even after completion > of the query, the RAM usage is still the same as when running the query. > The RAM usage only reduces once I terminate the connection and stop the > server. > > Is this behavior normal for the MonetDB server ? Also, ‘Src_MonetDB’ > creates a separate connection than ‘dbConnect’ > > Can the two connection run at the same time ? I noticed that sometimes > when I use the Src_MonetDB connection, the dbConnect gets terminated > automatically. > > Thanks a lot for your help on these. > > Best Regards, > > Jigar > > Sent from Mail <http://go.microsoft.com/fwlink/?LinkId=550986> for > Windows 10 > > > > > *From: *Anthony Damico <ajdamico(a)gmail.com> > *Sent: *Tuesday, December 29, 2015 6:22 AM > *To: *Communication channel for MonetDB users <users-list(a)monetdb.org> > *Subject: *Re: Extract Table Statistics after Data import using > Monet.Read.Csv > > > > for an example of this, see > > > https://github.com/ajdamico/asdfree/tree/master/Basic%20Stand%20Alone%20Med… > > On Dec 29, 2015 2:50 AM, "Hannes Mühleisen" <hannes.muehleisen(a)cwi.nl> > wrote: > > Hi Jigar, > > > On 29.12.2015, at 01:21, Jigar Shah <jigar.shah1984(a)hotmail.com> wrote: > > > > The dataset is divided into have 12 tables of approx. 2 GB each. In > order to recreate the entire dataset, I want to use the MERGE function to > merge all the 12 tables into one dataset for further analysis. > > > You can pass a vector of _all_ your file names to monetdb.read.csv(), this > will load all of them into a new table and does not require the merge. The > different schemas come from the function guessing the column types from the > file. > > Hannes > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > > >

1 0

Performing a direct access to MonetDB's BAT tables via MAL
by Giacomo Bergami 31 Dec '15

31 Dec '15

Good Morning/Evening everyone. My aim is to direcly access to the BAT column store representation of MonetDB through the standard libraries (MAPI/JDBC) - if possible. I know that each SQL query gets rewrited into a BAT algebra expression, but in my case I want to access to a single value per time (e.g. I want to access to the value stored in the i-th row and in the column labelled "j" of a given table) and maybe in this case the direct access is quicker than performing the whole SQL query. I've already tried to use the mclient with the MAL language and read the documentation, and I tried to use the client with something like this: mdb.start(); bbp.open(); bbp.prelude(); a := bbp.bind("fx", "size"); where "fx" should be the name of the table and "size" one of its attributes. But unfortunately I do not know how to search for a specific row-key and to get the correspondant value, and AFAIK i have to get the id of the column "size". Thanks in advance, Giacomo Bergami ____________________________ Giacomo Bergami, Ph.D Student University of Bologna https://www.unibo.it/sitoweb/giacomo.bergami2

3 5