- users-list - monetdb.org

[MonetDB-users] Slow range join performance / regression with Dec2011-SP1
by Viktor Rosenfeld 02 Mar '12

02 Mar '12

Hi, I have a few queries that join a table with itself using a ranged predicate on the primary key. The simplest query is: SELECT count(*) AS count FROM rank_copy rank1, rank_copy rank2 WHERE rank1.cat_s = true AND rank2.pos_vvfin = true AND rank1.pre < rank2.pre AND rank2.pre < rank1.post As you can see, the range lookup on rank2.pre is restricted on both sides by rank1.pre and rank1.post. The selectivity of the attributes and the complete query is: - row count : 2431307 - query : 95977 (0.039) - cat_s : 148799 (0.061) - pos_vvfin : 71965 (0.030) PostgreSQL is able to evaluate the query in 2 seconds using an index on pre. (This index is automatically generated because pre is the primary key.) MonetDB Dec2011 requires about 35 seconds. The MAL plan is posted below. I've also tried sorting the table on pre, and created an (advisory) index on pre but this makes no difference. +---------------------------------------------------------------------------+ | mal | +===========================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_9:bat[:oid,:bit] := sql.bind(X_2,"sys","rank_copy","pos_vvfin",0); | | X_10 := algebra.uselect(X_9,true:bit); | | X_11 := algebra.markT(X_10,0@0:oid); | | X_12 := bat.reverse(X_11); | | X_4:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_13 := algebra.leftjoin(X_12,X_4); | | X_17:bat[:oid,:bit] := sql.bind(X_2,"sys","rank_copy","cat_s",0); | | X_18 := algebra.uselect(X_17,true:bit); | | X_19 := algebra.markT(X_18,0@0:oid); | | X_20 := bat.reverse(X_19); | | X_23 := algebra.leftjoin(X_20,X_4); | | X_15:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_21 := algebra.leftjoin(X_20,X_15); | | X_24 := algebra.join(X_13,X_23,X_21,false,false); | | X_26 := algebra.markT(X_24,0@0:oid); | | X_27 := bat.reverse(X_26); | | X_28 := algebra.leftjoin(X_27,X_13); | | X_29 := aggr.count(X_28); | | sql.exportValue(1,"sys.rank2","count":str,"wrd",64,0,6,X_29,""); | | end s1_1; | +---------------------------------------------------------------------------+ 22 tuples If I use a slightly more complicated query that has the boolean attributes pos_vvfin and cat_s on another table that is joined by rank, the situation is even worse. SELECT count(*) AS count FROM node_copy node1 JOIN rank_copy rank1 ON (node1.id = rank1.node_ref), node_copy node2 JOIN rank_copy rank2 ON (node2.id = rank2.node_ref) WHERE node1.cat_s = true AND node2.pos_vvfin = true AND rank1.pre < rank2.pre AND rank2.pre < rank1.post PostgreSQL evaluates the query in 9 seconds, again using the primary key indexes on node.id and rank.pre. MonetDB Dec2011 requires 14 minutes. The MAL plan follows: +-----------------------------------------------------------------------------------------------+ | mal | +===============================================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_19:bat[:oid,:oid] := sql.bind_idxbat(X_2,"sys","rank_copy","FK_rank_copy_node_ref",0); | | X_13:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","pos_vvfin",0); | | X_14 := algebra.uselect(X_13,true:bit); | | X_15 := algebra.markT(X_14,0@0:oid); | | X_16 := bat.reverse(X_15); | | X_9:bat[:oid,:int] := sql.bind(X_2,"sys","node_copy","id",0); | | X_11 := bat.mirror(X_9); | | X_17 := algebra.leftjoin(X_16,X_11); | | X_18 := bat.reverse(X_17); | | X_37:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","cat_s",0); | | X_39 := algebra.uselect(X_37,true:bit); | | X_40 := algebra.markT(X_39,0@0:oid); | | X_41 := bat.reverse(X_40); | | X_42 := algebra.leftjoin(X_41,X_11); | | X_43 := bat.reverse(X_42); | | X_24 := algebra.join(X_19,X_18); | | X_25 := algebra.markT(X_24,0@0:oid); | | X_26 := bat.reverse(X_25); | | X_4:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_27 := algebra.leftjoin(X_26,X_4); | | X_29:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_30 := algebra.join(X_27,X_4,X_29,false,false); | | X_32 := algebra.markT(X_30,0@0:oid); | | X_33 := bat.reverse(X_32); | | X_44 := bat.reverse(X_30); | | X_45 := algebra.markT(X_44,0@0:oid); | | X_46 := bat.reverse(X_45); | | X_47 := algebra.leftjoin(X_46,X_19); | | X_48 := algebra.join(X_47,X_43); | | X_49 := algebra.markT(X_48,0@0:oid); | | X_50 := bat.reverse(X_49); | | X_51:bat[:oid,:int] := algebra.leftjoinPath(X_50,X_33,X_27); | | X_52 := aggr.count(X_51); | | sql.exportValue(1,"sys.rank2","count","wrd",64,0,6,X_52,""); | | end s1_1; | +-----------------------------------------------------------------------------------------------+ 37 tuples The Dec2011-SP1 release takes even longer, about 18 minutes. The MAL plan is unchanged. The culprit appears to be the ranged lookup because if I replace it with an equi-join on another attribute MonetDB evaluates it in about 100 ms. SELECT count(*) AS count FROM node_copy node1 JOIN rank_copy rank1 ON (node1.id = rank1.node_ref), node_copy node2 JOIN rank_copy rank2 ON (node2.id = rank2.node_ref) WHERE node1.cat_s = true AND node2.pos_vvfin = true AND rank1.pre = rank2.parent I've tried rewriting the query using WITH and had some success. The first query again takes 35 seconds: WITH rank1 AS (SELECT * FROM rank_copy WHERE cat_s = true), rank2 AS (SELECT * FROM rank_copy WHERE pos_vvfin = true) SELECT count(*) FROM rank1, rank2 WHERE rank1.pre < rank2.pre AND rank2.pre < rank1.post The second query also takes 35 seconds when rewritten: WITH rank1 AS (SELECT * FROM rank_copy JOIN node_copy ON (rank_copy.node_ref = node_copy.id) WHERE node_copy.cat_s = true), rank2 AS (SELECT * FROM rank_copy JOIN node_copy ON (rank_copy.node_ref = node_copy.id) WHERE node_copy.pos_vvfin = true) SELECT count(*) FROM rank1, rank2 WHERE rank1.pre < rank2.pre AND rank2.pre < rank1.post MAL plan: +-----------------------------------------------------------------------------------------------+ | mal | +===============================================================================================+ | function user.s1_2{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_20:bat[:oid,:oid] := sql.bind_idxbat(X_2,"sys","rank_copy","FK_rank_copy_node_ref",0); | | X_13:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","pos_vvfin",0); | | X_14 := algebra.uselect(X_13,true:bit); | | X_15 := algebra.markT(X_14,0@0:oid); | | X_16 := bat.reverse(X_15); | | X_9:bat[:oid,:int] := sql.bind(X_2,"sys","node_copy","id",0); | | X_11 := bat.mirror(X_9); | | X_17 := algebra.leftjoin(X_16,X_11); | | X_18 := bat.reverse(X_17); | | X_32:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","cat_s",0); | | X_33 := algebra.uselect(X_32,true:bit); | | X_34 := algebra.markT(X_33,0@0:oid); | | X_35 := bat.reverse(X_34); | | X_36 := algebra.leftjoin(X_35,X_11); | | X_37 := bat.reverse(X_36); | | X_22 := algebra.join(X_20,X_18); | | X_23 := algebra.markT(X_22,0@0:oid); | | X_24 := bat.reverse(X_23); | | X_5:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_25 := algebra.leftjoin(X_24,X_5); | | X_38 := algebra.join(X_20,X_37); | | X_39 := algebra.markT(X_38,0@0:oid); | | X_40 := bat.reverse(X_39); | | X_43 := algebra.leftjoin(X_40,X_5); | | X_27:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_41 := algebra.leftjoin(X_40,X_27); | | X_44 := algebra.join(X_25,X_43,X_41,false,false); | | X_46 := algebra.markT(X_44,0@0:oid); | | X_47 := bat.reverse(X_46); | | X_48 := algebra.leftjoin(X_47,X_25); | | X_49 := aggr.count(X_48); | | sql.exportValue(1,"sys.rank2","L1","wrd",64,0,6,X_49,""); | | end s1_2; | +-----------------------------------------------------------------------------------------------+ 35 tuples Is there anything I can do to further speed up the execution of these queries? Thanks, Viktor

1 0

[MonetDB-users] "ERROR = !Connection terminated" for some SQL queries
by Viktor Rosenfeld 01 Mar '12

01 Mar '12

Hi, the following query results in a "Connection terminated error": SELECT count(*) FROM ( SELECT DISTINCT node1.id AS id1, node2.id AS id2, node1.toplevel_corpus FROM node AS node1, rank AS rank1, component AS component1, node AS node2, rank AS rank2, component AS component2 WHERE component1.id = component2.id AND component1.name IS NULL AND component1.type = 'd' AND component2.name IS NULL AND component2.type = 'd' AND rank1.component_ref = component1.id AND rank1.node_ref = node1.id AND rank1.pre = rank2.parent AND rank2.component_ref = component2.id AND rank2.node_ref = node2.id ) AS solutions; The full error message is: MAPI = annis@moos.local:50000 ACTION= read_line QUERY = SELECT count(*) FROM ( SELECT DISTINCT node1.id AS id1, node2.id AS id2, node1.toplevel_corpus FROM node AS node1, rank AS rank1, component AS component1, node AS node2, rank AS rank2, component AS component2 WHERE component1.id = component2.id AND component1.name IS NULL AND component1.type = 'd' AND component2.name IS NULL AND component2.type = 'd' AND rank1.component_ref = component1.id AND rank1.node_ref = node1.id AND rank1.pre = rank2.parent AND rank2.component_ref = component2.id AND rank2.node_ref = node2.id ) AS solutions; ERROR = !Connection terminated I can get it to run if I remove the following line: component1.id = component2.id AND or if I remove the following 4 lines: component1.name IS NULL AND component1.type = 'd' AND component2.name IS NULL AND component2.type = 'd' AND However, if I include all constraints on component, the error appears. Let me know, if you need more information. Cheers, Viktor

2 3

[MonetDB-users] Database crash
by Stathis Gkotsis 29 Feb '12

29 Feb '12

Hi all, I am using MonetDB Dec 2011 version. Seeing that I could not connect to 'mydatabase' from mclient, I tailed the log: 2012-02-29 11:39:32 MSG mydatabase[7045]: # Serving database 'mydatabase', using 2 threads 2012-02-29 11:39:32 MSG mydatabase[7045]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2012-02-29 11:39:32 MSG mydatabase[7045]: # Found 999.238 MiB available main-memory. 2012-02-29 11:39:32 MSG mydatabase[7045]: # Copyright (c) 1993-July 2008 CWI. 2012-02-29 11:39:32 MSG mydatabase[7045]: # Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved 2012-02-29 11:39:32 MSG mydatabase[7045]: # Visit http://www.monetdb.org/ for further information 2012-02-29 11:39:32 MSG mydatabase[7045]: # Listening for UNIX domain connection requests on mapi:monetdb:///var/monetdb5/dbfarm/mydatabase/.mapi.sock 2012-02-29 11:39:32 MSG mydatabase[7045]: # MonetDB/SQL module loaded 2012-02-29 11:39:32 MSG merovingian[6978]: database 'mydatabase' (7045) was killed by signal SIGSEGV 2012-02-29 11:39:32 ERR merovingian[6978]: client error: database 'mydatabase' has crashed after starting, manual intervention needed, check monetdbd's logfile for details Any ideas? Thank you.

1 0

[MonetDB-users] Very Slow Selects
by Joseph Brower 29 Feb '12

29 Feb '12

We're using MonetDB and hitting it fairly heavily with read queries. We only use importing of text files to insert information, and that happens every 30 seconds or so. When we're only inserting information, we consume less than 1% of the cpu and things run just fine. We are also able to occasionally query the DB for various peices of information (counts, averages, etc) and it returns quite quickly. The problem is that after we've been running for 15-20 minutes of importing and querying heavily, performance drops to where queries take multiple minutes to finish. The queries generally finish in less than half a second. Does anyone have any ideas on how we can best track down this issue? Is there any more information that I need to provide? Thanks, Joseph Brower

6 7

[MonetDB-users] Announcement: New Dec2011-SP1 Bugfix release of MonetDB suite
by Fabian Groffen 29 Feb '12

29 Feb '12

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2011-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at http://www.monetdb.org/. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2011-SP1 bugfix release Build Environment * Fixed some of the package names for Debian/Ubuntu. Packages for libraries should contain the major number of the library version number. This was not always the case. * On Debian and Ubuntu, install Python modules in dist-packages instead of site-packages. This fixed bug 2997. SQL * Fixed a crash that happened at the end of a database upgrade to the Dec2011 database scheme. The crash happened during cleanup after the database was upgraded, so it was merely inconvenient. * Stripped off implementation-specific parts from error messages before they get presented to the user. Java Module * Fixed bug in DatabaseMetaData.getSchemas() method that caused an SQL error when called with catalog and schema argument. * Resolved a bug where JDBC and Control connections could terminate abruptly with 'Connection closed' messages MonetDB5 Server * Fixed potential crash by dealing better with non-standard types. Merovingian * Fixed a bug in the multiplex-funnel where certain clients would abort on responses for update queries. * Fixed problem where version and mserver properties for monetdbd were not always successfully retrieved. Bug #2982. * Fixed problem where shutdown of monetdbd would lead to shutting down database 'control' which does not exist. Bug #2983. * Fixed issue causing (harmless) 'error reading from control channel' messages. Bug #2984. * Resolved problem where remote start/stop/etc. commands with monetdb would report error 'OK'. Bug #2984. Bug Fixes * 2862: foreign key self references cause violation upon alter table add constraint... * 2933: "TypeException: algebra.join undefined" when evaluating EXISTS (SELECT ...) predicate * 2946: Invalid BAT if left join condition has only columns from right table * 2964: LEFT OUTER JOIN with constants returns errors * 2969: Result precision issue with Decimal data type * 2973: Date coercion functionality tests * 2979: misc merovingian control problems * 2982: monetdbd status and mserver properties have random values * 2983: monetdbd attempts to stop non-existing database 'control' * 2984: monetdb create/release/start/stop returns error "OK" for remote connections * 2995: Test geom/sql/Tests/example.sql crashes since changeset 6399077ff8a4 * 2997: monetdb-python packages installed in wrong dir on Ubuntu 10.04 LTS (lucid) * 3004: Incorrect evaluation of conditions in WHERE part of SQL statement * 3009: Segmentation fault on simple query * 3013: calc.and is not yet defined for dbl * 3014: dayofweek comment is wrong in mtime.mx:700 * 3017: rel_bin.c:611: exp_bin: Assertion `0' failed. with between on multicolumns * 3020: Assertion error in debugging mode * 3034: INNER JOIN with Subquery produces mal errors * 3041: WHERE condition ignored after 2 LEFT JOINs with subqueries

1 0

[MonetDB-users] max concurrent limit?
by Allen Zhang 28 Feb '12

28 Feb '12

Hi, When connecting to monetdb Aug 2011 SP2 version, we encountered error states : Maximum concurrent client connection (64) reached, any solutions? Best Regards Allen

4 4

[MonetDB-users] MonetDB JDBC 1.9 question
by Charalampos Nikolaou 23 Feb '12

23 Feb '12

Hi, I am getting some strange exceptions using MonetDB JDBC 1.8/1.9, and I decided to look into the code. While inspecting the code I came across these strange lines in MonetConnection.executeQuery(String[] templ, String query) method (http://dev.monetdb.org/hg/MonetDB/file/718da8ca0a1a/java/src/nl/cwi/monetdb…) In my case, these lines are executed for sure when doing a batch insert. So, suppose that the batch contains the following commands (as constructed by the MonetStatement.executeBatch() method): exec 1(1, 957339737330229055); exec 1(2, 278262503670654331); exec 1(805306369, 3763943296910752235) The lines in question take the above commands as a string, prepend a 's' character and append a ';' at the end. The resulting commands, which are written in the server's socket, are the following: sexec 1(1, 957339737330229055); exec 1(2, 278262503670654331); exec 1(805306369, 3763943296910752235); First of all, I am not familiar with the internals of JDBC drivers. Taking this into account, is this what it should be? From a symmetric point of view, I would assume that the correct would be the following: sexec 1(1, 957339737330229055); sexec 1(2, 278262503670654331); sexec 1(805306369, 3763943296910752235); That is, it should prepend a 's' character before an exec command. Last, are these exec commands (with or without a prepending 's') specific to MonetDB? In either case, is there any documentation to get familiar with their meaning? Thanks a lot, Babis

6 20

[MonetDB-users] Prepared statement not found (error: 07003)
by George Garbis 23 Feb '12

23 Feb '12

Hello all, I have a java project that creates a database using more than one connections and I get the following error: > 22000!07003:EXEC: no prepared statement with id: 1 > After digging in log files I created a simplified version of the problematic commands: Class.forName("nl.cwi.monetdb.jdbc.MonetDriver"); > Connection con1 = > DriverManager.getConnection("jdbc:monetdb://localhost/db1", "monetdb", > "monetdb"); > con1.setAutoCommit(true); > Connection con2 = > DriverManager.getConnection("jdbc:monetdb://localhost/db1", "monetdb", > "monetdb"); > con2.setAutoCommit(true); > > PreparedStatement ps2 = con2.prepareStatement("INSERT INTO table1 (a) > VALUES (CAST(? AS INTEGER))"); > ps2.setInt(1,1); ps2.addBatch(); > > Statement s1 = con1.createStatement(); > s1.execute("CREATE TABLE table2 ( a INTEGER )"); > s1.close(); > > ps2.executeBatch(); > During execution of ps2.executeBatch() the aforementioned error is occurred. The problem occurs because between creating the first prepared statement and executing it another statement is created and executed (this odd sequence of commands happens because the project is multi-threaded). The two queries change different tables so i wonder if there is something wrong in the commands above. It seems that MonetDB handles in a special way prepared statements and I have to execute each one before I define a new one. If yes can you give me some hints why this is happening or how i can work around this problem? Thank you in advance, George Garbis

3 3

[MonetDB-users] SQL:2003 support
by Gatis Ozoliņš 22 Feb '12

22 Feb '12

Are there any plans for SQL:2003 support? I am particularly interested in window functions (LAG, LEAD, DENSE_RANK ...) and MERGE statement.

1 0

[MonetDB-users] Transaction management in stored procedures
by Vassilis Papakonstantinou 20 Feb '12

20 Feb '12

Hi all, Does anyone knows how can I handle transactions in a stored procedure? In procedure definition on SQL Reference says that a procedure_statement may consists of transaction_statement etc. How transaction_statement implemented in a stored procedure? I tried (inside the BEGIN ... END block of stored procedure) the: START TRANSACTION; sql statements COMMIT; and: sql statements COMMIT; but both they do not work. Thanks, Vassilis

2 4