- users-list - monetdb.org

Slow Query
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi, send attached trace for query. Att, --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

3 2

Re: users-list Digest, Vol 42, Issue 17
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi, about slow query Yes, there is, but still left over resources. While the query (which takes 90 minutes) are running, to perform another in another table (9M rows), this latter returns usually in seconds. " A simple "select count(*) from table", run on 90 minutes. that is not normal. most likely there are a lot of other processes competing for resources. " Em 21/02/2016 08:00, users-list-request(a)monetdb.org escreveu: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: Slow Query (Martin Kersten) > 2. Re: Slow Query (Stefan Manegold) > 3. might be of interest (Anthony Damico) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 20 Feb 2016 21:34:06 +0100 > From: Martin Kersten <martin(a)monetdb.org> > To: users-list(a)monetdb.org > Subject: Re: Slow Query > Message-ID: <56C8CDBE.3060006(a)monetdb.org> > Content-Type: text/plain; charset=windows-1252; format=flowed > > On 20/02/16 19:39, Luciano Sasso wrote: >> Hi, >> >> I have MonetDB JUL2015-SP2, with table of 100M rows and 20 coluns, >> 4 of number type, others string type. >> >> A simple "select count(*) from table", run on 90 minutes. > that is not normal. most likely there are a lot of other > processes competing for resources. > >> Hardware and SO is: Debian 8, 4 cores, 11G RAM and 20G free on disc. >> >> While monet run the query, command top, show 42G of virtual memory >> >> Its normal ?, any idea ? >> >> thanks. >> >> >> -- >> >> Luciano Sasso Vieira >> Data Scientist & Solutions Architect >> >> <http://www.gsgroup.com.br>luciano(a)gsgroup.com.br | tel: 17 3353-0833 | cel: 17 99706-9335 >> <http://www.gsgroup.com.br>www.gsgroup.com.br >> >> >> Este e-mail foi enviado por um computador sem v?rus e protegido pelo Avast. >> www.avast.com <https://www.avast.com/sig-email> >> >> >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list >> > > > ------------------------------ > > Message: 2 > Date: Sat, 20 Feb 2016 22:16:01 +0100 (CET) > From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Subject: Re: Slow Query > Message-ID: <1956506938.11256849.1456002961125.JavaMail.zimbra(a)cwi.nl> > Content-Type: text/plain; charset="utf-8" > > Hi, > > this is indeed unexpected. > > To possibly better understand what's happening, > could you please run your query prefixed with keyword TRACE > and share the trace (profiling) result? > > Best, > Stefan > > ----- On Feb 20, 2016, at 7:39 PM, Luciano Sasso luciano(a)gsgroup.com.br wrote: > >> Hi, >> >> I have MonetDB JUL2015-SP2, with table of 100M rows and 20 coluns, >> 4 of number type, others string type. >> >> A simple "select count(*) from table", run on 90 minutes. >> >> Hardware and SO is: Debian 8, 4 cores, 11G RAM and 20G free on disc. >> >> While monet run the query, command top, show 42G of virtual memory. >> >> Its normal ?, any idea ? >> >> thanks. >> >> >> -- >> >> >> Luciano Sasso Vieira >> Data Scientist & Solutions Architect >> >> luciano(a)gsgroup.com.br | tel: 17 3353-0833 | cel: 17 99706-9335 >> www.gsgroup.com.br >> >> >> Este e-mail foi enviado por um computador sem v?rus e protegido pelo Avast. >> www.avast.com >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1

might be of interest
by Anthony Damico 21 Feb '16

21 Feb '16

http://stackoverflow.com/questions/32380663/improve-speed-of-queries-using-…

1 0

Slow Query
by Luciano Sasso 20 Feb '16

20 Feb '16

Hi, I have MonetDB JUL2015-SP2, with table of 100M rows and 20 coluns, 4 of number type, others string type. A simple "select count(*) from table", run on 90 minutes. Hardware and SO is: Debian 8, 4 cores, 11G RAM and 20G free on disc. While monet run the query, command top, show 42G of virtual memory. Its normal ?, any idea ? thanks. -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

3 2

Error in connecting monetdb server to localhost
by Debangana Sarkar 17 Feb '16

17 Feb '16

when I am connecting to MonetDB server it is showing an error : FATAL logger error there is no logger catalog but a log file is found. Are you sure you are using the same combination of database --dbpath and log directory Press any key to continue. Client error is: couldnot connect to localhost:50000 unknow error. Previously I installed the monetDB server and client and it was working.But for some reason it was uninstalled.Now after installing this error is coming. Please help

1 0

is it possible for a user to turn assertions on/off in mserver easily?
by Anthony Damico 16 Feb '16

16 Feb '16

might help with future testing..thanks all

3 3

Re: Re: Some time monetdbd can't start existing dbfarm
by Haiqing Ying 12 Feb '16

12 Feb '16

Thanks, Lefteris. You might be right and the log did suggest some file (.vaultkey) can't be found before it stopped working. I'll move the dbfarm out of /tmp and give it a try. Regards, HQ

1 0

Re: Re: Some time monetdbd can't start existing dbfarm
by Haiqing Ying 11 Feb '16

11 Feb '16

Hi Brian, Thanks for looking into this issue. Here is some more details in meroveringian.log - The DB was up for about a week. There're a lot of lines of 'proxying client ...', until the line 'client error: cannot start database 'EXM': no .vaultkey found...' 2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33861 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM 2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33862 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM 2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33863 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM 2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 08:36:20 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) 2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 08:36:20 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) 2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 08:36:21 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) And then, a lot of lines like below: 2016-02-07 14:44:18 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 14:44:23 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) Eventually, there're lot of errors like this: 2016-02-08 01:13:58 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-08 01:14:03 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) 2016-02-08 01:14:11 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file s 2016-02-08 01:14:18 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file s 2016-02-08 01:14:26 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file S Then neither mclient nor monetdbd stop cmd works any more. This is what I have in /tmp/exm folder after reboot: [centos@ip-10-72-0-98 exm]$ pwd /tmp/exm [centos@ip-10-72-0-98 exm]$ ls -al total 8088 drwxr-xr-x. 3 centos centos 62 Feb 9 17:33 . drwxrwxrwt. 11 root root 4096 Feb 11 13:36 .. drwx------. 4 centos centos 77 Feb 9 17:33 EXM -rw-------. 1 centos centos 0 Feb 9 17:33 .merovingian_lock -rw-------. 1 centos centos 8272151 Feb 9 17:33 merovingian.log [centos@ip-10-72-0-98 exm]$ ls -al EXM total 8 drwx------. 4 centos centos 77 Feb 9 17:33 . drwxr-xr-x. 3 centos centos 62 Feb 9 17:33 .. drwx------. 66 centos centos 4096 Feb 2 19:48 bat -rw-------. 1 centos centos 0 Feb 7 08:36 .gdk_lock srwx------. 1 centos centos 0 Jan 27 14:50 .mapi.sock drwx------. 3 centos centos 16 Jan 27 14:50 sql_logs -rw-------. 1 centos centos 22 Feb 9 17:33 .uplog If needed, I can send you the whole merovingian.log file (which is 8GB in size). Thanks, HQ

1 0

copy into stdout and redirect to a file
by Anton Kravchenko 11 Feb '16

11 Feb '16

Hi there, I wonder if one can copy a query result into stdout and redirect it to a file? I've tried: mclient.bat -u monetdb -d somedb -s "copy select * from sometable into stdout" - > .\somefile.csv But, only the table header is saved in somefile.csv... P.S. 'select * from sometable' shows all data rows as expected. I am using MonetDB v11.21.11 (Jul2015-SP1) on Windows. Thank you, Anton

2 2

Some time monetdbd can't start existing dbfarm
by Haiqing Ying 11 Feb '16

11 Feb '16

Hi, I created a dbfarm (/tmp/exm ), then created database (EXM), load data, do queries, etc, everything works fine for a week. Then I found mclient stopped working (can't connect to monetdb). The log file shows 'too many files opened' issue. I guess it might be some connection leak issue. I tried 'monetdbd stop /tmp/exm', but it's not responding. It looks I can't stop the dbfarm now. I then reboot the system and after reboot tried 'monetdbd start /tmp/exm' but I got an error saying ' cannot find file in /tmp/exm'. I verified the /tmp/exm folder did exist and there're still sub folder ( ./EXM) and log file still under /tmp/exm. The only option left seems to be creating a new dbfarm and recreating the whole database again. We're moving to use MonetDB in production and if this is a real bug then it would be too risky for production. The question is, is this a real issue? Or it's just some user error that I made? How do I make sure we won't run into this situation again in production mode. Many thanks, HQ

3 2