- users-list - monetdb.org

Fwd: Function does not complete
by Shmagi Kavtaradze 09 Apr '16

09 Apr '16

Thanks for answer. I have bits of size 50, so I need bigint/long long int. Problem is in while loop. When I use bitand(a,b) and numbers of digits of a and b both are 18-19, the system enters endless loop I think. Bigint maximum value is 19 digits long.I have mistakes in code when using unsigned int and I changed, but same problem. As I know from data types page, monetdb uses only signed integers, so I use only signed ones. Any ideas why does it hang on 18-19 digits and not on 1-17 digits long numbers?

1 0

How to do profiling in MAL interface?
by Constantine Feng 09 Apr '16

09 Apr '16

Hi all, I know I can use the TRACE SQL command. But I want to do it in the "-lmal" interface (because I massaged the MAL plans a bit to test my research ideas). Particularly, I find the information given on this website ( https://www.monetdb.org/Documentation/Manuals/MonetDB/Modules/profiler ) doesn't match the source code. I can't find the profiler.activate() or profiler.deactivate(), for example. mal>manual.help("profiler.*"); profiler.cpuload profiler.cpustats profiler.cleanup profiler.closeStream profiler.dumpTrace profiler.getSystemTime profiler.getUserTime profiler.getDiskWrites profiler.getDiskReads profiler.getTrace profiler.noop profiler.openStream profiler.reset profiler.setHeartbeat profiler.stop profiler.stethoscope profiler.tomograph Version: (Jul2015-SP3). Build from source. - Neo

2 1

Controlling --dbextras Using MonetDB(D)
by Alex Manaev 09 Apr '16

09 Apr '16

Hi, I use wrapper shell in such case. May be useful for you. $ mv /usr/bin/mserver5 /usr/bin/mserver5.bin $ vim /usr/bin/mserver5 #!/usr/bin/env bash exec /usr/bin/mserver5.bin --set gdk_vmtrim yes --set gdk_vm_maxsize 200000000000 --dbextra=/mnt/data/monetdb/extra $@ Alexey Manaev, Head of software development CJSC "RUTUBE"

3 4

monetdb crash as start : Logger_new: BBPrename to sql_snapshots_bid failed
by Pierre-Adrien Coustillas 08 Apr '16

08 Apr '16

Hello, after a server crash : # mclient comptage_cookie_telerama monetdbd: internal error while starting mserver, please refer to the logs # cat merovingian.log 2016-04-08 14:28:33 MSG merovingian[13582]: starting database 'comptage_cookie_telerama', up min/avg/max: 28m/4w/10w, crash average: 1.00 1.00 1.00 (246-6=240) 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: arguments: /usr/bin/mserver5 --dbpath=/var/lib/monetdb/comptage_cookie_telerama --set merovingian_uri=mapi:monetdb://groupelemonde-crm:50000/comptage_cookie_telerama --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/lib/monetdb/comptage_cookie_telerama/.mapi.sock --set monet_vault_key=/var/lib/monetdb/comptage_cookie_telerama/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # MonetDB 5 server v11.21.13 "Jul2015-SP2" 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Serving database 'comptage_cookie_telerama', using 8 threads 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Found 64.000 GiB available main-memory. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Copyright (c) 1993-July 2008 CWI. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Visit http://www.monetdb.org/ for further information 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Listening for UNIX domain connection requests on mapi:monetdb:///var/lib/monetdb/comptage_cookie_telerama/.mapi.sock 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Start processing logs sql/sql_logs version 52200 2016-04-08 14:28:34 MSG merovingian[13582]: database 'comptage_cookie_telerama' (16350) has exited with exit status 1 2016-04-08 14:28:34 ERR merovingian[13582]: client error: database 'comptage_cookie_telerama' appears to shut itself down after starting, check monetdbd's logfile for possible hints 2016-04-08 14:28:34 MSG merovingian[13582]: starting database 'comptage_cookie_telerama', up min/avg/max: 28m/4w/10w, crash average: 1.00 1.00 1.00 (247-6=241) 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: arguments: /usr/bin/mserver5 --dbpath=/var/lib/monetdb/comptage_cookie_telerama --set merovingian_uri=mapi:monetdb://groupelemonde-crm:50000/comptage_cookie_telerama --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/lib/monetdb/comptage_cookie_telerama/.mapi.sock --set monet_vault_key=/var/lib/monetdb/comptage_cookie_telerama/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # MonetDB 5 server v11.21.13 "Jul2015-SP2" 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Serving database 'comptage_cookie_telerama', using 8 threads 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Found 64.000 GiB available main-memory. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Copyright (c) 1993-July 2008 CWI. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Visit http://www.monetdb.org/ for further information 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Listening for UNIX domain connection requests on mapi:monetdb:///var/lib/monetdb/comptage_cookie_telerama/.mapi.sock 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Start processing logs sql/sql_logs version 52200 2016-04-08 14:28:34 MSG merovingian[13582]: database 'comptage_cookie_telerama' (16354) has exited with exit status 1 2016-04-08 14:28:39 ERR merovingian[13582]: client error: database 'comptage_cookie_telerama' appears to shut itself down after starting, check monetdbd's logfile for possible hints # cat .gdk_lock USR=107 PID=16354 TIME=Fri Apr 8 14:28:34 2016 @ LOGON USR=107 PID=16354 TIME=Fri Apr 8 14:28:34 2016 @ !FATAL: Logger_new: BBPrename to sql_snapshots_bid failed USR=107 PID=16354 TIME=Fri Apr 8 14:28:34 2016 @ LOGOFF do you have an idea ? Thank, Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35 // 09 54 38 70 90

1 0

any ideas why CPU usage would never go over 25%?
by Anthony Damico 08 Apr '16

08 Apr '16

hi, we have a powerful windows server with 200GB RAM and 40 cores. i've been using it for a few weeks now, and mserver.exe never seems to use more than 10 of the cores. mserver does say "using 40 threads" so they're all available is it just rare for any SQL query to use more than 10 cores at once, or is it possible i've misconfigured something? thanks

1 0

Function does not complete
by Shmagi Kavtaradze 07 Apr '16

07 Apr '16

When I run the own written function it runs and does no exit. I was waiting for 5-6 hours, checked dstats, no disk writes, only 1 core works and no memory usage. It is like stuck and can not exit. It happens when I use function separately for each variable, without callingm for example: while( *a ){ countA += *a & 1; *a >>= 1; } while( *b ){ countB += *b & 1; *b >>= 1; } while( c ){ countC += c & 1; c >>= 1; } but when I call functions like this monetdb works fine: int countSetBits1(unsigned int n){ unsigned int count = 0; while(n){ count += n & 1; n >>= 1; } return count; } countA = countSetBits1(*a); countB = countSetBits1(*b); countC = countSetBits1(c); I am not strong in C or monetdb internals, but in Netbeans both codes work well. The entire code : str UDFbitanddec(flt *ret, lng *a, lng *b) { int c = 0; int countA = 0; int countB = 0; int countC = 0; int maxCount = 0; unsigned int iA = *a; unsigned int iB = *b; c = iA & iB; while( iA ){ countA += iA & 1; iA >>= 1; } while( iB ){ countB += iB & 1; iB >>= 1; } while( c ){ countC += c & 1; c >>= 1; } if (countA >= countB) { maxCount = countA; } else { maxCount = countB; } *ret = (float)countC /maxCount; return MAL_SUCCEED; }

2 1

Exception Error
by Erie, Rob 07 Apr '16

07 Apr '16

I am in the process of evaluating databases for analysis at my organization. I have installed MonetDB and loaded some data. I was attempting to compare results both in performance and accuracy between MySQL and MonetDB. I have been receiving Overflow in Calculation errors from MonetDB. I am a total novice with MonetDB. Here is the information about my system: Red Hat Enterprise Linux Server release 7.2 (Maipo) (Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux) MySQL Server version: 5.7.11-log MySQL Community Server Java version "1.8.0_74"; Java(TM) SE Runtime Environment (build 1.8.0_74-b02); Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode). Apache Tomcat 7.0.54 Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP3) Database: MonetDB v11.21.17 (Jul2015-SP3), 'mapi:monetdb://ip-10-153-0-5.ec2.internal:50000/set_mktng' Issue Description: I would like to know the correlation between two variables group by a variable. SQL: select milageRange, sys.corr(numberOfTheseParts,avgHomeValue) FROM aPartsByZipRange INNER JOIN avgHomeValueZipPeriod ON zipCode = zip where numberOfTheseParts <> 0 and avgHomeValue <>0 group by milageRange; Error: TypeException:user.s78_2[44]:'aggr.subcorr' undefined in: X_75:bat[:any,:int] := aggr.subcorr(X_72:bat[:oid,:int],X_74:bat[:oid,:int],X_67:bat[:oid,:oid],r1_67:bat[:oid,:oid],true:bit); program contains errors -- -------------------------------------------------- This e-mail transmission contains information intended only for the use of the recipient(s) named above. Further, it contains information that may be privileged and confidential. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this message (including any attachments) is strictly prohibited. If you have received this e-mail in error, please notify the sender by reply e-mail and then delete this message from your mail system. Thank you for your compliance.

3 4

Question
by Сергей 07 Apr '16

07 Apr '16

Does anyone create replication with MonetDB? I've tried many times but still cannot run MonetDB in master-slave mode(replication). I have such errors when start slave server: !mvc_init: unable to create system tables !SQLException:SQLinit:Catalogue initialization failed I've done all that written in https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/TransactionRepli… but it does not work. Does anyone can help me? Is transaction-replication included to July2015SP3 release? Or I must build MonetDB from sources? Thanks.

1 0

Create R function that can be applied to an entire table
by Willem Ligtenberg 07 Apr '16

07 Apr '16

Hi, I guess I am pushing the user defined R functions to their limits. I know want to implement some batch correcting in R and then apply that to an entire table. I have written R code, that will generate a function definition that has matching parameters. And then I want to call the function as follows: CREATE TABLE data_bc AS SELECT * FROM batch_correction( (SELECT * FROM data) ) WITH DATA; I then get the following error: Error in .local(conn, statement, ...) : Unable to execute statement 'CREATE TABLE hcs_batch AS SELECT * FROM batch_correction( (SELECT * FROM hcs_norm) ) WITH DATA;'. Server says '!Command too large' Looking at the source, this error is raised in rapi.c on either line 467 or 473. After editing the source to figure out which of the lines was triggered, I figured out is was on line: 467 it is triggered, so the following code raises the issue that the command is too large: pos = 0; for (i = pci->retc + 2; i < pci->argc && pos < sizeof(argnames); i++) { pos += snprintf(argnames + pos, sizeof(argnames) - pos, "%s%s", args[i], i < pci->argc - 1 ? ", " : ""); } if (pos >= sizeof(argnames)) { msg = createException(MAL, "rapi.eval", "Command too large1"); goto wrapup; } (yes, I added the 1 to large, to make it unique) If this is not a bug, any help in getting my code to work is greatly appreciated. Kind regards, Willem

2 3

Disabling do_sort() does not work.
by Xu,Wenjian 06 Apr '16

06 Apr '16

Hi, I would like to replace MonetDB' sorting algorithms (timsort/quicksort) with my own. As the first step, I commented all content of function *do_sort()* in gdk_batop.c to disable the sorting functionality. Then I re-build the monetdb (with no error) and type a SQL statement with ORDER-BY clause in mclient interface: "select o_orderkey, o_custkey, o_orderstatus from orders order by o_custkey limit 10;" I was expecting to see the dis-ordered result. However, it turns out that the result is still ordered by o_custkey. I am confused about the result. Did I miss something? Any help would be great. Best regards, John

1 0

accessing intermediate optimizer results.
by Xu,Wenjian 01 Apr '16

01 Apr '16

Hi all, I would like to see the intermediate result (i.e., MAL instructions) after each optimizer step in a optimizer pipeline. So I searched Monetdb's documents and found the following sentence in https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/OptimizerPipelin… : "Alternatively, the SQL DEBUG statement modifier in combination with the 'o' command provides access to the intermediate optimizer results." I cannot figure out what this sentence means. What I know about SQL DEBUG statement is to use "debug <SQL statement>" to enter the debug mode and iterate through each (final) MAL instruction one by one. How could I use 'o' command to access intermediate optimizer results? I am using MonetDB 11.21.17 (JUL2015 SP3) and my environment is Ubuntu14.04. Thank you very much. Best regards, XU Wenjian

2 4

No previous prototype for function
by Shmagi Kavtaradze 31 Mar '16

31 Mar '16

I use two versions of Monetdb. 1 is PythonUDF and another Monetdb 11.21.13. Function "bitand" is compiled and used well in Monetdb, but when I try to compile PythonUDF it gives me errors: udf.c:14:1: error: no previous prototype for 'UDFbitand' [-Werror=missing-prototypes] UDFbitand(flt *ret, lng*a, lng *b) ^ cc1: all warnings being treated as errors Makefile:876: recipe for target 'lib_udf_la-udf.lo' failed any ideas? thanks in advance.

4 5

no sorted column?
by Roberto Cornacchia 31 Mar '16

31 Mar '16

Before filing a bug report, I'd like to ask here whether this is intentional/expected. sql>create table t(a int); operation successful (0.650ms) sql>insert into t values (1),(0),(3),(2); 4 affected rows (0.638ms) sql> sql>create table sorted_t as select a from t order by a with data; operation successful (0.807ms) sql> sql>select table,column,sorted from sys.storage() where "table"='sorted_t'; +----------+--------+--------+ | table | column | sorted | +==========+========+========+ | sorted_t | a | false | +----------+--------+--------+ I am pretty sure it use to be that the sorting used in a "CREATE TABLE AS ... WITH DATA" would result in the tsorted property to be true for the (first) sorted column. Now this isn't the case. Is that a bug or a feature? Roberto

4 6

Possible serious Memory leak?
by Vijay Krishna 31 Mar '16

31 Mar '16

Hi, Today, we encountered this serious memory issue with MonetDB. *Data : *10 tables - each with ~600 columns - all tables are EMPTY (No rows). *Query : *One query that involves with 25 joins across all the 10 tables. *MonetDB version :* July2015-SP3 The above query is ran on the same MonetDB version with the same table structure with all tables empty on two different machines as follows. *Scenario 1 - Query ran on Macbook Pro * - RAM - 8 GB (with other processes like Chrome/Eclipse running in parallel) - Free disk space - ~100 GB - MonetDB July2015-SP3 downloaded as installable The query is ran. *Expected result* - return an empty set in a few seconds. *Actual result* - returns an empty set in ~5 minutes RAM - 8 GB utilised completely. Swap space used from disk - 10 GB This swap space is cleared after sometime automatically (around 5 mins) *Scenario 2 - With a CentOS virtual machine* - RAM - 128 GB - Free disk space - ~650 GB - MonetDB July2015-SP3 compiled from source code The query is ran. *Expected result* - return an empty set in a few seconds. *Actual result* - the query rebooted this high end CentOS machine. RAM - 128 GB utilised completely. After reboot, the swap memory was freed. I am puzzled with the following questions. 1. Why should a query that runs on 10 EMPTY tables require memory in GBs? 2. How did this query manage to run on a Mac with just 8 GB RAM, but crashed a machine with 128 GB RAM? Is this a known memory leak issue? Any help much appreciated. Thanks in advance. Regards, Vijay.

3 3

Find all running processes?
by Vijay Krishna 31 Mar '16

31 Mar '16

How to find all running processes and their statuses in MonetDB?? I found this... https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/monitor But I couldn't get this work... I ran a query (which took some 20 seconds to complete) in a window and connected the same database in another terminal and ran "select * from sys.queue" (several times during those 20 seconds) but I could get only this... sql>select * from sys.queue; +-------+---------+----------------------------+----------------------------+----------+---------+------------+--------------------------+ | qtag | user | started | estimate | progress | status | tag | query | +=======+=========+============================+============================+==========+=========+============+==========================+ | 27339 | monetdb | 2015-08-13 05:35:40.000000 | null | null | running | 12532134@0 | select * from sys.queue; | +-------+---------+----------------------------+----------------------------+----------+---------+------------+--------------------------+ I am using Oct-2014 version of monetdb on CentOS... Is there a way to find all running processes and their statuses in MonetDB?? Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

2 3

Speeding SELECT for a small number of fields OVER large number of record in Perl DBD::monetdb
by Puthick Hok 31 Mar '16

31 Mar '16

Hi, I'm using Perl DBD::monetdb from MonetDB source for close to a year now. Recently, I discovered using SQL SELECT for one or two fields for a large number of records, for example, 100000 records is too slow compared to how faster MonetDB engine is. The same SQL SELECT is way faster in MySQL due to C implementation of DBD::mysql. MonetDB connector in Python is still way faster than Perl DBD::monetdb because it allows user to tell the connector the maximum number of records the user needs and the connector uses Xexport command straightaway and only once. In Perl, DBD::monetdb calls Mapi.pm via getReply which issues Xexport command many times depending on the number of records in the SELECT statement. Each time, it retrieves only 100 records. For 100000 records, it needs to call Xexport 1000 times. I did my own patch to make getReply call Xexport only once a the rest of the records after the initial request. With the patch, the time it takes to select 2 fields for 101024 records drops from more 24 seconds to under 4 seconds. My patch is: <pre> sub getReply { my ($self)= @_; if ($self->{active} == 0) { return $self->getBlock(); } elsif ($self->{next} < $self->{replysize} + $self->{skip}) { return $self->getRow(); } elsif (${self}->{offset} + $self->{replysize} < $self->{count}) { # get next slice my $rs = $self->{replysize}; my $offset = $self->{offset} + $rs; my $therest = $self->{count} - $self->{replysize}; $self->putblock("Xexport $self->{id} $offset $therest"); # FAST #$self->putblock("Xexport $self->{id} $offset $rs"); # SLOW return $self->getBlock(); } else { # close large results, but only send on next query if ($self->{id} > 0 && $self->{count} != $self->{replysize}) { push @{$self->{piggyback}}, "Xclose $self->{id}"; $self->{skip_in}++; } $self->{active} = 0; } return $self->{active}; } </pre> I hope this could help other people. Regards, Puthick

1 0

Re: Temporary duplicated rows
by Alex Manaev 30 Mar '16

30 Mar '16

Effect is reproduced with console on the fresh database. Here is real time console log. Can anybody explain what happens? sql>select id, status from morgoth_reportquery where status <> 4 and id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 4 | | 6880 | 4 | | 6880 | 4 | +------+--------+ 3 tuples (2.551ms) <--- result of previous experiments sql>select id, status from morgoth_reportquery where id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 4 | +------+--------+ 1 tuple (2.746ms) sql>update morgoth_reportquery set status = 2 where id = 6880; 1 affected row (3.306ms) sql>select id, status from morgoth_reportquery where id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 4 | +------+--------+ 1 tuple (2.216ms) sql>select id, status from morgoth_reportquery where status <> 4 and id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 4 | | 6880 | 4 | | 6880 | 4 | | 6880 | 4 | +------+--------+ 4 tuples (2.634ms) sql>select id, status from morgoth_reportquery where status = 4 and id = 6880; +----+--------+ | id | status | +====+========+ +----+--------+ 0 tuples (2.337ms) sql>select id, status from morgoth_reportquery where status = 1 and id = 6880; +----+--------+ | id | status | +====+========+ +----+--------+ 0 tuples (3.294ms) sql>select id, status from morgoth_reportquery where status = 4 and id = 6880; +----+--------+ | id | status | +====+========+ +----+--------+ 0 tuples (2.416ms) sql>select id, status from morgoth_reportquery where id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 2 | +------+--------+ 1 tuple (2.186ms) sql>select id, status from morgoth_reportquery where status <> 4 and id = 6880; +------+--------+ | id | status | +======+========+ | 6880 | 2 | +------+--------+ 1 tuple (4.145ms) sql> Alexey Manaev, Head of software development CJSC "RUTUBE" On 30.03.2016 17:43, users-list-request(a)monetdb.org wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Temporary duplicated rows (Alex Manaev) > 2. UDF returning 0 as a result (Shmagi Kavtaradze) > 3. Re: UDF returning 0 as a result (Mark Raasveldt) > 4. Re: UDF returning 0 as a result (Sjoerd Mullender) > 5. Re: Find all running processes? (Pierre-Adrien Coustillas) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 30 Mar 2016 14:18:16 +0300 > From: Alex Manaev <amanaev(a)rutube.ru> > To: <users-list(a)monetdb.org> > Subject: Temporary duplicated rows > Message-ID: <56FBB5F8.7060309(a)rutube.ru> > Content-Type: text/plain; charset="utf-8"; format=flowed > > Hi, > > I have been faced with strange behavior. > > Sometimes after update that changes field "status", a select query > returns multiple identical rows. > > > sql>select * from morgoth_reportquery where not status = 2 and id = > 9472 order by id desc limit 5; > +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ > | id | repo | name | created_date > | ready_date | stat | hash | > params | user | appr | temp | noti | > : : rt_i : : > : : us : > : : _id : oved : late : fica : > : : d : : > : : : > : : : : _id : tion : > : : : : > : : : > : : : : : _ema : > : : : : > : : : > : : : : : ils : > +======+======+=============================+============================+============================+======+=============================+=============================+======+======+======+======+ > | 9472 | 13 | ??????????? ??????: ??????? | 2015-12-01 12:28:59.000000 > | null | 1 | 5f34b8459f4834867a2621f7d6b | - > !!python/tuple [from_date | 1 | 0 | 8 | | > : : : ???????? 2016-02-01 - 2016- : > : : : 65c8a : , > 2016-02-01] : : : : : > : : : 02-11 : > : : : : - > !!python/tuple [to_date, : : : : : > : : : : > : : : : > 2016-02-11] : : : : : > : : : : > : : : : - > !!python/tuple [update_me : : : : : > : : : : > : : : : diainfo, > 0] : : : : : > : : : : > : : : : - > !!python/tuple : : : : : > : : : : > : : : : - > agreements... > : : : : > | 9472 | 13 | ??????????? ??????: ??????? | 2015-12-01 12:28:59.000000 > | null | 1 | 5f34b8459f4834867a2621f7d6b | - > !!python/tuple [from_date | 1 | 0 | 8 | | > : : : ???????? 2016-02-01 - 2016- : > : : : 65c8a : , > 2016-02-01] : : : : : > : : : 02-11 : > : : : : - > !!python/tuple [to_date, : : : : : > : : : : > : : : : > 2016-02-11] : : : : : > : : : : > : : : : - > !!python/tuple [update_me : : : : : > : : : : > : : : : diainfo, > 0] : : : : : > : : : : > : : : : - > !!python/tuple : : : : : > : : : : > : : : : - > agreements... > : : : : > +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ > 2 tuples (8.446ms) !2 fields truncated! > > > In same time another queries return only one row (normal state) > > sql>select * from morgoth_reportquery where status = 1 and id = 9472 > order by id desc limit 5; > +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ > | id | repo | name | created_date > | ready_date | stat | hash | > params | user | appr | temp | noti | > : : rt_i : : > : : us : > : : _id : oved : late : fica : > : : d : : > : : : > : : : : _id : tion : > : : : : > : : : > : : : : : _ema : > : : : : > : : : > : : : : : ils : > +======+======+=============================+============================+============================+======+=============================+=============================+======+======+======+======+ > | 9472 | 13 | ??????????? ??????: ??????? | 2015-12-01 12:28:59.000000 > | null | 1 | 5f34b8459f4834867a2621f7d6b | - > !!python/tuple [from_date | 1 | 0 | 8 | | > : : : ???????? 2016-02-01 - 2016- : > : : : 65c8a : , > 2016-02-01] : : : : : > : : : 02-11 : > : : : : - > !!python/tuple [to_date, : : : : : > : : : : > : : : : > 2016-02-11] : : : : : > : : : : > : : : : - > !!python/tuple [update_me : : : : : > : : : : > : : : : diainfo, > 0] : : : : : > : : : : > : : : : - > !!python/tuple : : : : : > : : : : > : : : : - > agreements... > : : : : > +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ > 1 tuple (6.474ms) !1 field truncated! > > > Another queries without filter by field `status` give a normal result > (one row). > > Effect is eliminated in several minutes after update by itself. > > > DBMS : MonetDB v11.21.17 (Jul2015-SP3) > OS: CentOS 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC > 2015 x86_64 x86_64 x86_64 GNU/Linux > > Issue was reproduced for the first time with version SP-2. > > Database was created on SP-1. Issue had not appeared with this version. > > > >

1 0

Can no connect to Monetdb database
by Shmagi Kavtaradze 30 Mar '16

30 Mar '16

I compiled Monetdb-python version over regular Monetdb source compilation and after it I receive error: 2016-03-30 20:14:11 MSG merovingian[23154]: starting database 'db', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) 2016-03-30 20:14:11 MSG db[23176]: arguments: /usr/local/bin/mserver5 --dbpath=/home/shmagi/db/db --set merovingian_uri=mapi:monetdb://infdbserv02:50000/db --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/shmagi/db/db/.mapi.sock --set monet_vault_key=/home/shmagi/db/db/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-03-30 20:14:11 MSG merovingian[23154]: database 'db' (23176) has crashed (dumped core) 2016-03-30 20:14:16 ERR merovingian[23154]: client error: database 'db' appears to shut itself down after starting, check monetdbd's logfile for possible hints I tried ro ./configure, make clean, make , make install from pure monetdb source (without python), but still receive these errors. I tried to use prefix to install in another directory, tried to delete manually all files I found, but still same error. Any ideas how to fix this problem?

1 0

UDF returning 0 as a result
by Shmagi Kavtaradze 30 Mar '16

30 Mar '16

I wrote a function bitand. When I do "select bitand(1111,1010);" it gives "0" instead of "0.5". My function: --UDF.c-- str UDFbitand(long long int a, long long b) { long long int weirdAnd(unsigned int a, unsigned int b) { long long int result = 0; int coef = 1; while (a || b) { result += ((a % 10) && (b % 10)) * coef; coef *= 10; a /= 10; b /= 10; } return result; } long long int temp = weirdAnd(a, b); long long int temp2 = weirdAnd(a, b); int length; int count; while (temp != 0) { temp/=10; length++; } while (temp2 != 0) { if (temp2 % 10 == 1) count++; temp2 /= 10; } float final = count/(float)length; return MAL_SUCCEED; } --UDF.mal-- command bitand(a:lng,b:lng):flt address UDFbitand comment "bitand"; --80_UDF.sql-- create function bitand(a BIGINT,b BIGINT) returns REAL external name udf.bitand; When I test my C code in netbeans it works as supposed. Any ideas? Thanks in advance.

3 3

Temporary duplicated rows
by Alex Manaev 30 Mar '16

30 Mar '16

Hi, I have been faced with strange behavior. Sometimes after update that changes field "status", a select query returns multiple identical rows. sql>select * from morgoth_reportquery where not status = 2 and id = 9472 order by id desc limit 5; +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ | id | repo | name | created_date | ready_date | stat | hash | params | user | appr | temp | noti | : : rt_i : : : : us : : : _id : oved : late : fica : : : d : : : : : : : : : _id : tion : : : : : : : : : : : : : _ema : : : : : : : : : : : : : ils : +======+======+=============================+============================+============================+======+=============================+=============================+======+======+======+======+ | 9472 | 13 | Ежемесячные отчеты: Медиате | 2015-12-01 12:28:59.000000 | null | 1 | 5f34b8459f4834867a2621f7d6b | - !!python/tuple [from_date | 1 | 0 | 8 | | : : : хнологии 2016-02-01 - 2016- : : : : 65c8a : , 2016-02-01] : : : : : : : : 02-11 : : : : : - !!python/tuple [to_date, : : : : : : : : : : : : : 2016-02-11] : : : : : : : : : : : : : - !!python/tuple [update_me : : : : : : : : : : : : : diainfo, 0] : : : : : : : : : : : : : - !!python/tuple : : : : : : : : : : : : : - agreements... > : : : : | 9472 | 13 | Ежемесячные отчеты: Медиате | 2015-12-01 12:28:59.000000 | null | 1 | 5f34b8459f4834867a2621f7d6b | - !!python/tuple [from_date | 1 | 0 | 8 | | : : : хнологии 2016-02-01 - 2016- : : : : 65c8a : , 2016-02-01] : : : : : : : : 02-11 : : : : : - !!python/tuple [to_date, : : : : : : : : : : : : : 2016-02-11] : : : : : : : : : : : : : - !!python/tuple [update_me : : : : : : : : : : : : : diainfo, 0] : : : : : : : : : : : : : - !!python/tuple : : : : : : : : : : : : : - agreements... > : : : : +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ 2 tuples (8.446ms) !2 fields truncated! In same time another queries return only one row (normal state) sql>select * from morgoth_reportquery where status = 1 and id = 9472 order by id desc limit 5; +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ | id | repo | name | created_date | ready_date | stat | hash | params | user | appr | temp | noti | : : rt_i : : : : us : : : _id : oved : late : fica : : : d : : : : : : : : : _id : tion : : : : : : : : : : : : : _ema : : : : : : : : : : : : : ils : +======+======+=============================+============================+============================+======+=============================+=============================+======+======+======+======+ | 9472 | 13 | Ежемесячные отчеты: Медиате | 2015-12-01 12:28:59.000000 | null | 1 | 5f34b8459f4834867a2621f7d6b | - !!python/tuple [from_date | 1 | 0 | 8 | | : : : хнологии 2016-02-01 - 2016- : : : : 65c8a : , 2016-02-01] : : : : : : : : 02-11 : : : : : - !!python/tuple [to_date, : : : : : : : : : : : : : 2016-02-11] : : : : : : : : : : : : : - !!python/tuple [update_me : : : : : : : : : : : : : diainfo, 0] : : : : : : : : : : : : : - !!python/tuple : : : : : : : : : : : : : - agreements... > : : : : +------+------+-----------------------------+----------------------------+----------------------------+------+-----------------------------+-----------------------------+------+------+------+------+ 1 tuple (6.474ms) !1 field truncated! Another queries without filter by field `status` give a normal result (one row). Effect is eliminated in several minutes after update by itself. DBMS : MonetDB v11.21.17 (Jul2015-SP3) OS: CentOS 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Issue was reproduced for the first time with version SP-2. Database was created on SP-1. Issue had not appeared with this version. -- Alexey Manaev, Head of software development CJSC "RUTUBE"

1 0

Controlling --dbextras Using MonetDB(D)
by Vincent Sheffer 30 Mar '16

30 Mar '16

mserver5 has an argument called —dbextras. I prefer to control mserver5 using monetdbd, but I don't see any way of specifying this option in either monetdb or monetdbd. Is there, say, a property that can be set in monetdbd or monetdb. The latter probably makes more sense, as I think I would want to control it at the DB level not the dbfarm level. Thanks, Vince The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.

1 0

simultaneously-created tables not generating
by Anthony Damico 29 Mar '16

29 Mar '16

i have three clients connected to one mserver. they are simultaneously running these three commands, which share table `z` but not the x or y tables-- CREATE TABLE y1 AS SELECT x1.* FROM x1 LEFT JOIN SELECT col FROM Z ON x1.id = z.id WITH DATA ; CREATE TABLE y2 AS SELECT x2.* FROM x2 LEFT JOIN SELECT col FROM Z ON x2.id = z.id WITH DATA ; CREATE TABLE y3 AS SELECT x3.* FROM x3 LEFT JOIN SELECT col FROM Z ON x3.id = z.id WITH DATA ; when all three commands complete, tables y2 and y3 do not exist if i run the commands sequentially, everything works. am i making a mistake in how sql transactions should behave, or is this potentially a bug that i should figure out how to reproduce for bugzilla? thanks!

2 4

Ways to avoid OOMKiller problem?
by Vijay Krishna 29 Mar '16

29 Mar '16

Hi, We came across this - https://www.monetdb.org/Documentation/OOMkiller and we were thinking on some ways to overcome this situation. 1. Is there any configuration in MonetDB to restrict the maximum RAM usage of the mserver process? 2. Is there any configuration in MonetDB to restrict the maximum RAM used by a single thread? 3. Is there any suggested solution for the OOMKiller problem? If you have explored anything else on this, please share your experience with MonetDB server Memory management. Any help much appreciated. Thanks in advance. Regards, Vijay.

3 3

speed cost of stacking tables that do not need to be stacked
by Anthony Damico 27 Mar '16

27 Mar '16

just some UNION ALL speed tests that others might find useful.. i have one big table for each year of data. the year-tables can be analyzed separately - because all of my desired results are going to be within-year averages and counts. since each table has a single year of data, these two statements give equivalent results: query #1: SELECT * FROM ( ( SELECT yr , AVG( value ) FROM y1 GROUP BY yr ) UNION ALL ( SELECT yr , AVG( value ) FROM y2 GROUP BY yr ) ) AS zzz ; query #2: SELECT yr , AVG( value ) FROM ( ( SELECT yr , value FROM y1 ) UNION ALL ( SELECT yr , value FROM y2 ) ) AS zzz GROUP BY yr ; query #1: 290 seconds query #2: 390 seconds query #1: 22 seconds query #2: 87 seconds fwiw- this would take about 24 hours in anything other than monetdb ;)

1 0

database has crashed (dumped core)
by Shmagi Kavtaradze 25 Mar '16

25 Mar '16

I recomplied monetdb several times (make & make install). Everything was fine. Then I downloaded monetdb_python. It was not able to install and I decided to recompile and install again regular Monetdb, but when I "mclient -u monetdb -d voc" I get error in the log file: 2016-03-25 17:26:18 MSG merovingian[49368]: starting database 'voc', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) 2016-03-25 17:26:18 MSG voc[49379]: arguments: /usr/local/bin/mserver5 --dbpath=/home/shmagi/db/voc --set merovingian_uri=mapi:monetdb://infdbserv02:50000/voc --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/shmagi/db/voc/.mapi.sock --set monet_vault_key=/home/shmagi/db/voc/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-03-25 17:26:18 MSG merovingian[49368]: database 'voc' (49379) has crashed (dumped core) 2016-03-25 17:26:18 ERR merovingian[49368]: client error: database 'voc' appears to shut itself down after starting, check monetdbd's logfile for possible hints 2016-03-25 17:26:18 MSG merovingian[49368]: starting database 'voc', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) 2016-03-25 17:26:19 MSG voc[49382]: arguments: /usr/local/bin/mserver5 --dbpath=/home/shmagi/db/voc --set merovingian_uri=mapi:monetdb://infdbserv02:50000/voc --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/shmagi/db/voc/.mapi.sock --set monet_vault_key=/home/shmagi/db/voc/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-03-25 17:26:19 MSG merovingian[49368]: database 'voc' (49382) has crashed (dumped core) 2016-03-25 17:26:23 ERR merovingian[49368]: client error: database 'voc' appears to shut itself down after starting, check monetdbd's logfile for possible hints. Any ideas how to fix or delete all files of Monetdb to install from the scratch?

1 0