- users-list - monetdb.org

[MonetDB-users] Database crash
by Stathis Gkotsis 16 Mar '12

16 Mar '12

Hi all, yesterday my server lost power and when I rebooted my monetdb database would not start. This is what I see in the logs: gdk_dbfarm=/var/monetdb5/dbfarm --dbname=mydatabase --set merovingian_uri=mapi:monetdb://mydatabase:50000/mydatabase --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/monetdb5/dbfarm/mydatabase/.mapi.sock --set monet_vault_key=/var/monetdb5/dbfarm/mydatabase/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2012-03-16 01:15:34 MSG merovingian[897]: database 'mydatabase' (1793) has exited with exit status 1 2012-03-16 01:15:34 ERR merovingian[897]: client error: database 'mydatabase' has crashed after starting, manual intervention needed, check monetdbd's logfile for details 2012-03-16 01:15:36 MSG merovingian[897]: database 'mydatabase' has crashed after start on 2012-03-13 17:57:44, attempting restart, up min/avg/max: 32s/32s/32s, crash average: 1.00 0.10 0.03 (2-1=1) 20 Any ideas on how to fix this? Thank you, Stathis

1 0

[MonetDB-users] Announcement: New Dec2011-SP2 Bugfix release of MonetDB suite
by Fabian Groffen 16 Mar '12

16 Mar '12

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2011-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at http://www.monetdb.org/. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2011-SP2 bugfix release Java Module * Implemented missing Number types support in PreparedStatement.setObject() Bug Fixes * 2915: Query (mclient or jdbc) with 2 outer joins appears to crash connectionor server * 2947: Queries where a "is null" clause is combined with two equals clauses fail to produce the expected result. * 2968: Memory Leaks * 2974: thetajoin produces invalid/inconsistent bat * 2988: Subquery in select part goes wrong * 2998: Equivalent multi-way join queries give different results * 3002: String plus interval result in program errors upon comparison * 3011: count(*) in subquery results in 1 for query without results * 3012: bitwise comparison fails after a cast * 3029: test sql/test/Dependencies/Tests/Dependencies.sql fails after recent changes * 3031: Virtually all SQL tests fail on Windows * 3036: optimizers are not getting set correctly through set optimizer * 3038: ~190 SQL tests fail on Darwin, FreeBSD & Solaris (only) since introduction of "(extended) relational algebra" * 3043: WITH clause is not allowed in derived table * 3045: algebra.project(0@0,bat) fails at propcheck * 3052: Wrong Error Code on Rollback Exception

1 0

[MonetDB-users] rename table?
by Christian Braun 16 Mar '12

16 Mar '12

Hello! Is it possible to rename a table? Regards, Christian.

1 0

[MonetDB-users] Query ignored
by Mints, Alexey 15 Mar '12

15 Mar '12

Hello! I have an issue with MonetDB query. This one select count(*) from catalogedsources c1 left outer join catalogedsources c2 on ( c2.cat_id = c1.cat_id + 2 AND ((c1.decl - c2.decl) * (c1.decl - c2.decl))< 1e-06) where c1.cat_id = 4 AND c1.decl BETWEEN 77.0 AND 83.0 AND c1.ra BETWEEN 42.0 AND 48.0 ; works fine. But if I replace "c2.cat_id = c1.cat_id + 2" with "c2.cat_id = 6" - i receive nothing. Not zero, not empty dataset, but just nothing at all. "EXPLAIN" does not work as well. Also, with more complex join conditions I have no result as well. I am using: Welcome to mclient, the MonetDB/SQL interactive terminal (Dec2011-SP1) Database: MonetDB v11.7.7 (Dec2011-SP1), 'mapi:monetdb://linux-6d2s.site:50000/*****' What can this be? Regards, Alexey

1 0

[MonetDB-users] MonetDB out of memory errors
by Stathis Gkotsis 14 Mar '12

14 Mar '12

Hi all, I am using MonetDB v11.7.7 (Dec2011-SP1) in an application which constantly feeds data in tables (daily tables). The communication with the server is done through the mapi API. This is accomplished through the "COPY INTO" command. From time to time (every 1-2 days) I see errors in the application logs such as: exception at 00000000004C0109: Connect error (Error code -4): monetdbd: an internal error has occurred, refer to the logs for details, please try again later . exception at 00000000004C0109: Connect error (Error code -4): monetdbd: there are no available connections for 'mydatabase' . exception at 00000000004C0109: Connect error (Error code -4): monetdbd: there are no available connections for 'mydatabase' Then looking at dmesg log: [964932.242080] 251025 pages non-shared [964932.242083] Out of memory: kill process 11585 (monetdbd) score 787911 or a child [964932.302785] Killed process 11781 (mserver5) Sometimes, monetdb can recover from this error and then the loading continues normally, some others I have to manually restart monetdb. Is there anything I can do to troubleshoot this issue? Is there a possibility for me to fall back to an earlier version of MonetDB with potentially less features but more stability? Thanks for all the help, Stathis

2 2

[MonetDB-users] rdf support
by Jürgen Jakobitsch 14 Mar '12

14 Mar '12

hi, first of all congrats to this great project! i came to monetDB via a search for "column stores", since i know that one of the big players in the field of linked data and rdf storage switched to usage of a column store in one of their latest releases. i just built from source without any hassle on openSuse 11.4 with support for rdf since this is my primary interest at the moment. i didn't find much of an info regarding monetDB and rdf besides an old nabble-mail and a small-meeting wiki. is there some more info available? any pointer really appreciated. wkr jürgen -- | Jürgen Jakobitsch, | Software Developer | Semantic Web Company GmbH | Mariahilfer Straße 70 / Neubaugasse 1, Top 8 | A - 1070 Wien, Austria | Mob +43 676 62 12 710 | Fax +43.1.402 12 35 - 22 COMPANY INFORMATION | http://www.semantic-web.at/ PERSONAL INFORMATION | web : http://www.turnguard.com | foaf : http://www.turnguard.com/turnguard | skype : jakobitsch-punkt | xmlns:tg = "http://www.turnguard.com/turnguard#"

1 0

[MonetDB-users] Replication/Redundancy
by Joseph Brower 13 Mar '12

13 Mar '12

We're using MonetDB quite heavily and are really enjoying it. We're now wanting to add replication/redundancy to ensure we have as little downtime as possible. We're also thinking of balancing reads if that is possible. I've heard this is possible, but I'm unsure on the best way to approach it. I've not been able to find many docs that talk about this either. Could someone point me in the right direction? Thanks, Joseph Brower

4 10

[MonetDB-users] monetdbd: binding to datagram socket port 50000 failed: no available address
by Charalampos Nikolaou 09 Mar '12

09 Mar '12

When I start monetdbd I get the following error "monetdbd: binding to datagram socket port 50000 failed: no available address" Prior to this error I had properly terminated the server, so there is no server running. How can I overcome that problem? Also, netstat gives the following: $ netstat -p udp | grep 50000 udp4 1562 0 *.50000 *.* Thanks, Babis P.S.: I am on a Mac if that plays any role.

2 2

[MonetDB-users] some questions regarding queries in MonetDB
by Freddy Priyatna 06 Mar '12

06 Mar '12

Dear all, I've been using MonetDB version 5 on Ubuntu machine for evaluating BSBM queries. The dataset and queries can be found here[1]. It seems that there are some queries that are not supported on MonetDB : 1) SELECT p.nr, p.label FROM product p, producttypeproduct ptp WHERE p.nr=ptp.product AND "productType"=35 AND "propertyNum1">500 AND "propertyNum3"<2000 AND 168 IN (SELECT "productFeature" FROM productfeatureproduct WHERE productfeatureproduct.product=p.nr) AND 147 NOT IN (SELECT "productFeature" FROM productfeatureproduct WHERE product=p.nr) >From the error, the subquery seems can't refer to the table alias "p". 2) SELECT distinct p.nr, p.label FROM product p, product po, (Select distinct pfp1.product FROM productfeatureproduct pfp1, (SELECT "productFeature" FROM productfeatureproduct WHERE product=2) pfp2 WHERE pfp2."productFeature"=pfp1."productFeature") pfp WHERE p.nr=pfp.product AND po.nr=2 AND p.nr <> po.nr AND p."propertyNum1" < (po."propertyNum1"+120) AND p."propertyNum1" > (po."propertyNum1"-120) AND p."propertyNum2" < (po."propertyNum2"+170) AND p."propertyNum2" > (po."propertyNum2"-170) This is strange, because if I just comment one of the last 2 lines, the query works. But the last two lines can't be used together. 3) SELECT nr, label FROM product WHERE label like "%countries%"; It seems that MonetDB doesn't support regex. 4) SELECT * FROM (select label from product where nr=2) p left join ((select o.nr as onr, o.price, v.nr as vnr, v.label from offer o, vendor v where 2=o.product AND o.vendor=v.nr AND v.country='GB' AND o."validTo">'2008-07-01') ov right join (select r.nr as rnr, r.title, pn.nr as pnnr, pn.name, r.rating1, r.rating2 from review r, person pn where r.product=2 AND r.person=pn.nr) rpn on (1=1)) on (1=1); It seems that MonetDB can't do join with true condition because when I tried to run a simple query : SELECT * FROM product p RIGHT JOIN offer o ON (1=1) Then MonetDB gives an error. Could anyone please help me how to run the queries above? Many thanks in advance. Freddy [1] http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/ExploreUse…

5 11

[MonetDB-users] duplicate tables in schema
by Bas Kaptijn 05 Mar '12

05 Mar '12

Hi, repeatedly we get in a situation where tables end up duplicate in the sys.tables table: SELECT count(*) FROM tables WHERE name='mytable'; returns: 2 We think this should never be the case. It seems this gives problems with dumping the database and in other situations. This is an older version of MonetDB (Apr2011 SP1). We want to know whether someone knows this was/is a symptom of any known issue Sincerely, Bas

3 2

[MonetDB-users] Slow range join performance / regression with Dec2011-SP1
by Viktor Rosenfeld 02 Mar '12

02 Mar '12

Hi, I have a few queries that join a table with itself using a ranged predicate on the primary key. The simplest query is: SELECT count(*) AS count FROM rank_copy rank1, rank_copy rank2 WHERE rank1.cat_s = true AND rank2.pos_vvfin = true AND rank1.pre < rank2.pre AND rank2.pre < rank1.post As you can see, the range lookup on rank2.pre is restricted on both sides by rank1.pre and rank1.post. The selectivity of the attributes and the complete query is: - row count : 2431307 - query : 95977 (0.039) - cat_s : 148799 (0.061) - pos_vvfin : 71965 (0.030) PostgreSQL is able to evaluate the query in 2 seconds using an index on pre. (This index is automatically generated because pre is the primary key.) MonetDB Dec2011 requires about 35 seconds. The MAL plan is posted below. I've also tried sorting the table on pre, and created an (advisory) index on pre but this makes no difference. +---------------------------------------------------------------------------+ | mal | +===========================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_9:bat[:oid,:bit] := sql.bind(X_2,"sys","rank_copy","pos_vvfin",0); | | X_10 := algebra.uselect(X_9,true:bit); | | X_11 := algebra.markT(X_10,0@0:oid); | | X_12 := bat.reverse(X_11); | | X_4:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_13 := algebra.leftjoin(X_12,X_4); | | X_17:bat[:oid,:bit] := sql.bind(X_2,"sys","rank_copy","cat_s",0); | | X_18 := algebra.uselect(X_17,true:bit); | | X_19 := algebra.markT(X_18,0@0:oid); | | X_20 := bat.reverse(X_19); | | X_23 := algebra.leftjoin(X_20,X_4); | | X_15:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_21 := algebra.leftjoin(X_20,X_15); | | X_24 := algebra.join(X_13,X_23,X_21,false,false); | | X_26 := algebra.markT(X_24,0@0:oid); | | X_27 := bat.reverse(X_26); | | X_28 := algebra.leftjoin(X_27,X_13); | | X_29 := aggr.count(X_28); | | sql.exportValue(1,"sys.rank2","count":str,"wrd",64,0,6,X_29,""); | | end s1_1; | +---------------------------------------------------------------------------+ 22 tuples If I use a slightly more complicated query that has the boolean attributes pos_vvfin and cat_s on another table that is joined by rank, the situation is even worse. SELECT count(*) AS count FROM node_copy node1 JOIN rank_copy rank1 ON (node1.id = rank1.node_ref), node_copy node2 JOIN rank_copy rank2 ON (node2.id = rank2.node_ref) WHERE node1.cat_s = true AND node2.pos_vvfin = true AND rank1.pre < rank2.pre AND rank2.pre < rank1.post PostgreSQL evaluates the query in 9 seconds, again using the primary key indexes on node.id and rank.pre. MonetDB Dec2011 requires 14 minutes. The MAL plan follows: +-----------------------------------------------------------------------------------------------+ | mal | +===============================================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_19:bat[:oid,:oid] := sql.bind_idxbat(X_2,"sys","rank_copy","FK_rank_copy_node_ref",0); | | X_13:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","pos_vvfin",0); | | X_14 := algebra.uselect(X_13,true:bit); | | X_15 := algebra.markT(X_14,0@0:oid); | | X_16 := bat.reverse(X_15); | | X_9:bat[:oid,:int] := sql.bind(X_2,"sys","node_copy","id",0); | | X_11 := bat.mirror(X_9); | | X_17 := algebra.leftjoin(X_16,X_11); | | X_18 := bat.reverse(X_17); | | X_37:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","cat_s",0); | | X_39 := algebra.uselect(X_37,true:bit); | | X_40 := algebra.markT(X_39,0@0:oid); | | X_41 := bat.reverse(X_40); | | X_42 := algebra.leftjoin(X_41,X_11); | | X_43 := bat.reverse(X_42); | | X_24 := algebra.join(X_19,X_18); | | X_25 := algebra.markT(X_24,0@0:oid); | | X_26 := bat.reverse(X_25); | | X_4:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_27 := algebra.leftjoin(X_26,X_4); | | X_29:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_30 := algebra.join(X_27,X_4,X_29,false,false); | | X_32 := algebra.markT(X_30,0@0:oid); | | X_33 := bat.reverse(X_32); | | X_44 := bat.reverse(X_30); | | X_45 := algebra.markT(X_44,0@0:oid); | | X_46 := bat.reverse(X_45); | | X_47 := algebra.leftjoin(X_46,X_19); | | X_48 := algebra.join(X_47,X_43); | | X_49 := algebra.markT(X_48,0@0:oid); | | X_50 := bat.reverse(X_49); | | X_51:bat[:oid,:int] := algebra.leftjoinPath(X_50,X_33,X_27); | | X_52 := aggr.count(X_51); | | sql.exportValue(1,"sys.rank2","count","wrd",64,0,6,X_52,""); | | end s1_1; | +-----------------------------------------------------------------------------------------------+ 37 tuples The Dec2011-SP1 release takes even longer, about 18 minutes. The MAL plan is unchanged. The culprit appears to be the ranged lookup because if I replace it with an equi-join on another attribute MonetDB evaluates it in about 100 ms. SELECT count(*) AS count FROM node_copy node1 JOIN rank_copy rank1 ON (node1.id = rank1.node_ref), node_copy node2 JOIN rank_copy rank2 ON (node2.id = rank2.node_ref) WHERE node1.cat_s = true AND node2.pos_vvfin = true AND rank1.pre = rank2.parent I've tried rewriting the query using WITH and had some success. The first query again takes 35 seconds: WITH rank1 AS (SELECT * FROM rank_copy WHERE cat_s = true), rank2 AS (SELECT * FROM rank_copy WHERE pos_vvfin = true) SELECT count(*) FROM rank1, rank2 WHERE rank1.pre < rank2.pre AND rank2.pre < rank1.post The second query also takes 35 seconds when rewritten: WITH rank1 AS (SELECT * FROM rank_copy JOIN node_copy ON (rank_copy.node_ref = node_copy.id) WHERE node_copy.cat_s = true), rank2 AS (SELECT * FROM rank_copy JOIN node_copy ON (rank_copy.node_ref = node_copy.id) WHERE node_copy.pos_vvfin = true) SELECT count(*) FROM rank1, rank2 WHERE rank1.pre < rank2.pre AND rank2.pre < rank1.post MAL plan: +-----------------------------------------------------------------------------------------------+ | mal | +===============================================================================================+ | function user.s1_2{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_20:bat[:oid,:oid] := sql.bind_idxbat(X_2,"sys","rank_copy","FK_rank_copy_node_ref",0); | | X_13:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","pos_vvfin",0); | | X_14 := algebra.uselect(X_13,true:bit); | | X_15 := algebra.markT(X_14,0@0:oid); | | X_16 := bat.reverse(X_15); | | X_9:bat[:oid,:int] := sql.bind(X_2,"sys","node_copy","id",0); | | X_11 := bat.mirror(X_9); | | X_17 := algebra.leftjoin(X_16,X_11); | | X_18 := bat.reverse(X_17); | | X_32:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","cat_s",0); | | X_33 := algebra.uselect(X_32,true:bit); | | X_34 := algebra.markT(X_33,0@0:oid); | | X_35 := bat.reverse(X_34); | | X_36 := algebra.leftjoin(X_35,X_11); | | X_37 := bat.reverse(X_36); | | X_22 := algebra.join(X_20,X_18); | | X_23 := algebra.markT(X_22,0@0:oid); | | X_24 := bat.reverse(X_23); | | X_5:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_25 := algebra.leftjoin(X_24,X_5); | | X_38 := algebra.join(X_20,X_37); | | X_39 := algebra.markT(X_38,0@0:oid); | | X_40 := bat.reverse(X_39); | | X_43 := algebra.leftjoin(X_40,X_5); | | X_27:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_41 := algebra.leftjoin(X_40,X_27); | | X_44 := algebra.join(X_25,X_43,X_41,false,false); | | X_46 := algebra.markT(X_44,0@0:oid); | | X_47 := bat.reverse(X_46); | | X_48 := algebra.leftjoin(X_47,X_25); | | X_49 := aggr.count(X_48); | | sql.exportValue(1,"sys.rank2","L1","wrd",64,0,6,X_49,""); | | end s1_2; | +-----------------------------------------------------------------------------------------------+ 35 tuples Is there anything I can do to further speed up the execution of these queries? Thanks, Viktor

1 0

[MonetDB-users] "ERROR = !Connection terminated" for some SQL queries
by Viktor Rosenfeld 01 Mar '12

01 Mar '12

Hi, the following query results in a "Connection terminated error": SELECT count(*) FROM ( SELECT DISTINCT node1.id AS id1, node2.id AS id2, node1.toplevel_corpus FROM node AS node1, rank AS rank1, component AS component1, node AS node2, rank AS rank2, component AS component2 WHERE component1.id = component2.id AND component1.name IS NULL AND component1.type = 'd' AND component2.name IS NULL AND component2.type = 'd' AND rank1.component_ref = component1.id AND rank1.node_ref = node1.id AND rank1.pre = rank2.parent AND rank2.component_ref = component2.id AND rank2.node_ref = node2.id ) AS solutions; The full error message is: MAPI = annis@moos.local:50000 ACTION= read_line QUERY = SELECT count(*) FROM ( SELECT DISTINCT node1.id AS id1, node2.id AS id2, node1.toplevel_corpus FROM node AS node1, rank AS rank1, component AS component1, node AS node2, rank AS rank2, component AS component2 WHERE component1.id = component2.id AND component1.name IS NULL AND component1.type = 'd' AND component2.name IS NULL AND component2.type = 'd' AND rank1.component_ref = component1.id AND rank1.node_ref = node1.id AND rank1.pre = rank2.parent AND rank2.component_ref = component2.id AND rank2.node_ref = node2.id ) AS solutions; ERROR = !Connection terminated I can get it to run if I remove the following line: component1.id = component2.id AND or if I remove the following 4 lines: component1.name IS NULL AND component1.type = 'd' AND component2.name IS NULL AND component2.type = 'd' AND However, if I include all constraints on component, the error appears. Let me know, if you need more information. Cheers, Viktor

2 3

[MonetDB-users] Database crash
by Stathis Gkotsis 29 Feb '12

29 Feb '12

Hi all, I am using MonetDB Dec 2011 version. Seeing that I could not connect to 'mydatabase' from mclient, I tailed the log: 2012-02-29 11:39:32 MSG mydatabase[7045]: # Serving database 'mydatabase', using 2 threads 2012-02-29 11:39:32 MSG mydatabase[7045]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2012-02-29 11:39:32 MSG mydatabase[7045]: # Found 999.238 MiB available main-memory. 2012-02-29 11:39:32 MSG mydatabase[7045]: # Copyright (c) 1993-July 2008 CWI. 2012-02-29 11:39:32 MSG mydatabase[7045]: # Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved 2012-02-29 11:39:32 MSG mydatabase[7045]: # Visit http://www.monetdb.org/ for further information 2012-02-29 11:39:32 MSG mydatabase[7045]: # Listening for UNIX domain connection requests on mapi:monetdb:///var/monetdb5/dbfarm/mydatabase/.mapi.sock 2012-02-29 11:39:32 MSG mydatabase[7045]: # MonetDB/SQL module loaded 2012-02-29 11:39:32 MSG merovingian[6978]: database 'mydatabase' (7045) was killed by signal SIGSEGV 2012-02-29 11:39:32 ERR merovingian[6978]: client error: database 'mydatabase' has crashed after starting, manual intervention needed, check monetdbd's logfile for details Any ideas? Thank you.

1 0

[MonetDB-users] Very Slow Selects
by Joseph Brower 29 Feb '12

29 Feb '12

We're using MonetDB and hitting it fairly heavily with read queries. We only use importing of text files to insert information, and that happens every 30 seconds or so. When we're only inserting information, we consume less than 1% of the cpu and things run just fine. We are also able to occasionally query the DB for various peices of information (counts, averages, etc) and it returns quite quickly. The problem is that after we've been running for 15-20 minutes of importing and querying heavily, performance drops to where queries take multiple minutes to finish. The queries generally finish in less than half a second. Does anyone have any ideas on how we can best track down this issue? Is there any more information that I need to provide? Thanks, Joseph Brower

6 7

[MonetDB-users] Announcement: New Dec2011-SP1 Bugfix release of MonetDB suite
by Fabian Groffen 29 Feb '12

29 Feb '12

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2011-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at http://www.monetdb.org/. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2011-SP1 bugfix release Build Environment * Fixed some of the package names for Debian/Ubuntu. Packages for libraries should contain the major number of the library version number. This was not always the case. * On Debian and Ubuntu, install Python modules in dist-packages instead of site-packages. This fixed bug 2997. SQL * Fixed a crash that happened at the end of a database upgrade to the Dec2011 database scheme. The crash happened during cleanup after the database was upgraded, so it was merely inconvenient. * Stripped off implementation-specific parts from error messages before they get presented to the user. Java Module * Fixed bug in DatabaseMetaData.getSchemas() method that caused an SQL error when called with catalog and schema argument. * Resolved a bug where JDBC and Control connections could terminate abruptly with 'Connection closed' messages MonetDB5 Server * Fixed potential crash by dealing better with non-standard types. Merovingian * Fixed a bug in the multiplex-funnel where certain clients would abort on responses for update queries. * Fixed problem where version and mserver properties for monetdbd were not always successfully retrieved. Bug #2982. * Fixed problem where shutdown of monetdbd would lead to shutting down database 'control' which does not exist. Bug #2983. * Fixed issue causing (harmless) 'error reading from control channel' messages. Bug #2984. * Resolved problem where remote start/stop/etc. commands with monetdb would report error 'OK'. Bug #2984. Bug Fixes * 2862: foreign key self references cause violation upon alter table add constraint... * 2933: "TypeException: algebra.join undefined" when evaluating EXISTS (SELECT ...) predicate * 2946: Invalid BAT if left join condition has only columns from right table * 2964: LEFT OUTER JOIN with constants returns errors * 2969: Result precision issue with Decimal data type * 2973: Date coercion functionality tests * 2979: misc merovingian control problems * 2982: monetdbd status and mserver properties have random values * 2983: monetdbd attempts to stop non-existing database 'control' * 2984: monetdb create/release/start/stop returns error "OK" for remote connections * 2995: Test geom/sql/Tests/example.sql crashes since changeset 6399077ff8a4 * 2997: monetdb-python packages installed in wrong dir on Ubuntu 10.04 LTS (lucid) * 3004: Incorrect evaluation of conditions in WHERE part of SQL statement * 3009: Segmentation fault on simple query * 3013: calc.and is not yet defined for dbl * 3014: dayofweek comment is wrong in mtime.mx:700 * 3017: rel_bin.c:611: exp_bin: Assertion `0' failed. with between on multicolumns * 3020: Assertion error in debugging mode * 3034: INNER JOIN with Subquery produces mal errors * 3041: WHERE condition ignored after 2 LEFT JOINs with subqueries

1 0

[MonetDB-users] max concurrent limit?
by Allen Zhang 28 Feb '12

28 Feb '12

Hi, When connecting to monetdb Aug 2011 SP2 version, we encountered error states : Maximum concurrent client connection (64) reached, any solutions? Best Regards Allen

4 4

[MonetDB-users] MonetDB JDBC 1.9 question
by Charalampos Nikolaou 23 Feb '12

23 Feb '12

Hi, I am getting some strange exceptions using MonetDB JDBC 1.8/1.9, and I decided to look into the code. While inspecting the code I came across these strange lines in MonetConnection.executeQuery(String[] templ, String query) method (http://dev.monetdb.org/hg/MonetDB/file/718da8ca0a1a/java/src/nl/cwi/monetdb…) In my case, these lines are executed for sure when doing a batch insert. So, suppose that the batch contains the following commands (as constructed by the MonetStatement.executeBatch() method): exec 1(1, 957339737330229055); exec 1(2, 278262503670654331); exec 1(805306369, 3763943296910752235) The lines in question take the above commands as a string, prepend a 's' character and append a ';' at the end. The resulting commands, which are written in the server's socket, are the following: sexec 1(1, 957339737330229055); exec 1(2, 278262503670654331); exec 1(805306369, 3763943296910752235); First of all, I am not familiar with the internals of JDBC drivers. Taking this into account, is this what it should be? From a symmetric point of view, I would assume that the correct would be the following: sexec 1(1, 957339737330229055); sexec 1(2, 278262503670654331); sexec 1(805306369, 3763943296910752235); That is, it should prepend a 's' character before an exec command. Last, are these exec commands (with or without a prepending 's') specific to MonetDB? In either case, is there any documentation to get familiar with their meaning? Thanks a lot, Babis

6 20

[MonetDB-users] Prepared statement not found (error: 07003)
by George Garbis 23 Feb '12

23 Feb '12

Hello all, I have a java project that creates a database using more than one connections and I get the following error: > 22000!07003:EXEC: no prepared statement with id: 1 > After digging in log files I created a simplified version of the problematic commands: Class.forName("nl.cwi.monetdb.jdbc.MonetDriver"); > Connection con1 = > DriverManager.getConnection("jdbc:monetdb://localhost/db1", "monetdb", > "monetdb"); > con1.setAutoCommit(true); > Connection con2 = > DriverManager.getConnection("jdbc:monetdb://localhost/db1", "monetdb", > "monetdb"); > con2.setAutoCommit(true); > > PreparedStatement ps2 = con2.prepareStatement("INSERT INTO table1 (a) > VALUES (CAST(? AS INTEGER))"); > ps2.setInt(1,1); ps2.addBatch(); > > Statement s1 = con1.createStatement(); > s1.execute("CREATE TABLE table2 ( a INTEGER )"); > s1.close(); > > ps2.executeBatch(); > During execution of ps2.executeBatch() the aforementioned error is occurred. The problem occurs because between creating the first prepared statement and executing it another statement is created and executed (this odd sequence of commands happens because the project is multi-threaded). The two queries change different tables so i wonder if there is something wrong in the commands above. It seems that MonetDB handles in a special way prepared statements and I have to execute each one before I define a new one. If yes can you give me some hints why this is happening or how i can work around this problem? Thank you in advance, George Garbis

3 3

[MonetDB-users] SQL:2003 support
by Gatis Ozoliņš 22 Feb '12

22 Feb '12

Are there any plans for SQL:2003 support? I am particularly interested in window functions (LAG, LEAD, DENSE_RANK ...) and MERGE statement.

1 0

[MonetDB-users] Transaction management in stored procedures
by Vassilis Papakonstantinou 20 Feb '12

20 Feb '12

Hi all, Does anyone knows how can I handle transactions in a stored procedure? In procedure definition on SQL Reference says that a procedure_statement may consists of transaction_statement etc. How transaction_statement implemented in a stored procedure? I tried (inside the BEGIN ... END block of stored procedure) the: START TRANSACTION; sql statements COMMIT; and: sql statements COMMIT; but both they do not work. Thanks, Vassilis

2 4

[MonetDB-users] Subquery with INNER JOIN
by George Garbis 20 Feb '12

20 Feb '12

Hello to all, I have some problems trying to use subqueries with INNER JOIN. Assume the following schema: CREATE TABLE table1 (a INTEGER); > CREATE TABLE table2 (a INTEGER, b INTEGER); > > INSERT INTO table1 VALUES(1); > INSERT INTO table1 VALUES(2); > INSERT INTO table2 VALUES(2, 2); > I pose this query > SELECT * FROM (SELECT a, 2 AS b FROM table1) t1 INNER JOIN table2 t2 ON > (t1.b = t2.b); > and I get the following mal errors: > TypeException:user.s5_1[44]:'bat.mirror' undefined in: _62:any := > bat.mirror(_61:int) > TypeException:user.s5_1[46]:'algebra.join' undefined in: _64:any := > algebra.join(_62:any, _63:bat[:oid,:oid]) > TypeException:user.s5_1[47]:'algebra.markH' undefined in: _65:any := > algebra.markH(_64:any, _66:oid) > TypeException:user.s5_1[48]:'algebra.markT' undefined in: _67:any := > algebra.markT(_64:any, _66:oid) > TypeException:user.s5_1[49]:'bat.reverse' undefined in: _68:any := > bat.reverse(_67:any) > TypeException:user.s5_1[50]:'algebra.join' undefined in: _69:any := > algebra.join(_68:any, _61:int) > TypeException:user.s5_1[51]:'algebra.join' undefined in: _70:any := > algebra.join(_65:any, _59:bat[:oid,:int]) > TypeException:user.s5_1[55]:'algebra.join' undefined in: _77:any := > algebra.join(_76:bat[:void,:oid], _65:any) > TypeException:user.s5_1[56]:'bat.reverse' undefined in: _78:any := > bat.reverse(_77:any) > TypeException:user.s5_1[57]:'algebra.markT' undefined in: _79:any := > algebra.markT(_78:any, _43:oid) > TypeException:user.s5_1[58]:'bat.reverse' undefined in: _80:any := > bat.reverse(_79:any) > TypeException:user.s5_1[59]:'algebra.leftjoin' undefined in: _81:any := > algebra.leftjoin(_80:any, _42:bat[:oid,:int]) > TypeException:user.s5_1[64]:'algebra.project' undefined in: _86:any := > algebra.project(_81:any, _83:int) > TypeException:user.s5_1[65]:'algebra.join' undefined in: _87:any := > algebra.join(_86:any, _85:bat[:int,:oid]) > TypeException:user.s5_1[66]:'bat.reverse' undefined in: _88:any := > bat.reverse(_87:any) > TypeException:user.s5_1[67]:'algebra.markT' undefined in: _89:any := > algebra.markT(_88:any, _21:oid) > TypeException:user.s5_1[68]:'bat.reverse' undefined in: _90:any := > bat.reverse(_89:any) > TypeException:user.s5_1[69]:'algebra.leftjoin' undefined in: _91:any := > algebra.leftjoin(_90:any, _20:bat[:oid,:int]) > TypeException:user.s5_1[76]:'algebra.markT' undefined in: _103:any := > algebra.markT(_87:any, _101:oid) > TypeException:user.s5_1[77]:'bat.reverse' undefined in: _104:any := > bat.reverse(_103:any) > TypeException:user.s5_1[78]:'algebra.leftjoin' undefined in: _105:any := > algebra.leftjoin(_104:any, _81:any) > TypeException:user.s5_1[79]:'algebra.project' undefined in: _106:any := > algebra.project(_105:any, _96:bte) > TypeException:user.s5_1[86]:'algebra.leftjoin' undefined in: _113:any := > algebra.leftjoin(_80:any, _59:bat[:oid,:int]) > TypeException:user.s5_1[87]:'algebra.leftjoin' undefined in: _114:any := > algebra.leftjoin(_104:any, _113:any) > 39000!program contains errors > If i don't use a subquery and i first create the table i need, the query executes flawlessly and it returns the expected results. CREATE TABLE table3 (a INTEGER, b INTEGER); > INSERT INTO table3(a,b) (SELECT a, 2 AS b FROM table1); > SELECT * FROM (SELECT a, b FROM table3) t1 INNER JOIN table2 t2 ON (t1.b = > t2.b); > Results: > +------+------+------+----+ > | a | b | a | b | > +==+===+===+===+ > | 2 | 2 | 2 | 2 | > | 1 | 2 | 2 | 2 | > +------+------+------+----+ > Can anyone help? Is there something wrong in my first query or it is a bug? Thank you in advance, George Garbis

2 1

[MonetDB-users] mapi get field types
by Stathis Gkotsis 18 Feb '12

18 Feb '12

Hi, I am issuing a query using the mapi using mapi_query. Is there a way to get the returned field types through the api? Thank you, Stathis

2 1

[MonetDB-users] Ending up with a sql/log.XX of 4GB
by Stefan de Konink 17 Feb '12

17 Feb '12

Over the night my database was running. Doing both insert (constant) and select jobs (incidental). It seems that the combination is fatal. Doing each thing on independant databases I don't have problems at all, but doing them within the same database all kind of badness arrises. The next badness is the internal sql log of 4GB, MonetDB reports disk full. Which is actually the case. The actual database contents is only 49MB upon start (after inserting static data). I expect 200.000 extra rows of dynamic data, with a cyclic pattern, to be added over a week time, overwritting eachother. I wonder if it could be related to 'no-autocommit' behavior of select queries outside of transactions. I would assume it does not make sense at all, but that seems the most 'logical' reason... I could recover from disk full to free up space, wait a long time with a high load. SQL log was empty. Please advise, Stefan

1 0

[MonetDB-users] Table column order and type
by Pete Black 16 Feb '12

16 Feb '12

Hi, From what I understand indexes are indicative[1], sortedness is determined at runtime[2] and the storage model should not effect the general schema design[3]. Has anyone had any experience with large time series where the column order and types have effected performance? For example having data with less variance and "uniqueness" earlier in a table and splitting up timestamps and storing their date and daytime components separately. Thanks, Pete [1]http://www.monetdb.org/Documentation/Manuals/SQLreference/Indices [2]http://old.nabble.com/column-sortedness-to33053179.html#a33053179 [3]http://old.nabble.com/General-Questions-on-Columnar-DB-Design-and-Queryin…

2 1

[MonetDB-users] Microsecond timestamp precision?
by Pete Black 16 Feb '12

16 Feb '12

Hi, The documentation doesn't mention the resolution of the timestamp[1] directly, but given the examples for the daytime one can assume that the it holds millisecond resolution. Why then are the trailing zeros below printed out? And is there a way to get microsecond resolution for example using mal instead? sql>DECLARE t timestamp(9); sql>SET t='2012-02-16 01:15:25.123456'; sql>SELECT t; +----------------------------+ | single_value | +============================+ | 2012-02-16 01:15:25.123000 | +----------------------------+ 1 tuple (1.293ms) Thanks, Pete [1]http://www.monetdb.org/Documentation/SQLreference/Temporal

1 0

[MonetDB-users] Does Monetdb supports Date functions?
by Dongli 16 Feb '12

16 Feb '12

Hi guys, Does Monetdb supports "datediff" function? I didn't found this function while I run "select name from functions". Besides, Does Monetdb supports other Date relevant functions? Any advices will be highly appreciated. Thanks in advance, Tony

4 3

[MonetDB-users] Does Monetdb supports UTF-8 characters when I use 'as' clause?
by Dongli 15 Feb '12

15 Feb '12

Hi folks, Please take a look at following issue that I got today: The following queries I executed aim to test if Monetdb supports GBK characters(Chinese), since I have to use Chinese characters in my system to display the reports. I’m not sure if I missed some setting such as ‘set characters=’utf-8’? The following alias after the ‘as’ is a Chinese character ‘I’. sql>select source as '我' from session; 42000!syntax error, unexpected STRING in: "select source as '我'" sql>select source as 我 from session; unexpected character (U+6211) sql>select source as "我" from session; Invalid identifier '我' Thanks in advance, Tony

3 3

[MonetDB-users] database 'dw' has crashed after starting, manual intervention needed, check monetdbd's logfile for details
by Dongli 15 Feb '12

15 Feb '12

Hi folks, How can I recover from a crashed database? I got this error when I restart a running database 'dw', Do you have any ideas on this? 2012-02-15 13:59:28 MSG control[27007]: (local): served status list 2012-02-15 13:59:28 MSG merovingian[27007]: startup of database under maintenance 'dw' forced 2012-02-15 13:59:28 MSG merovingian[27007]: database 'dw' has crashed after start on 2012-02-15 13:58:47, attempting restart, up min/avg/max: 1d/1d/1d, crash average: 1.00 1.00 0.97 (45-1=44) 2012-02-15 13:59:28 MSG dw[27831]: arguments: /usr/lib/monetdb/bin/mserver5 --set gdk_dbfarm=/data/warehouse --dbname=dw --set merovingian_uri=mapi:monetdb://monet00:50000/dw --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/warehouse/dw/.mapi.sock --set monet_vault_key=/data/warehouse/dw/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2012-02-15 13:59:28 MSG dw[27831]: # MonetDB 5 server v11.7.5 "Dec2011" 2012-02-15 13:59:28 MSG dw[27831]: # Serving database 'dw', using 24 threads 2012-02-15 13:59:28 MSG dw[27831]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2012-02-15 13:59:28 MSG dw[27831]: # Found 23.533 GiB available main-memory. 2012-02-15 13:59:28 MSG dw[27831]: # Copyright (c) 1993-July 2008 CWI. 2012-02-15 13:59:28 MSG dw[27831]: # Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved 2012-02-15 13:59:28 MSG dw[27831]: # Visit http://www.monetdb.org/ for further information 2012-02-15 13:59:28 MSG dw[27831]: # Listening for UNIX domain connection requests on mapi:monetdb:///data/warehouse/dw/.mapi.sock 2012-02-15 13:59:28 MSG dw[27831]: # MonetDB/SQL module loaded 2012-02-15 13:59:28 MSG merovingian[27007]: database 'dw' (27831) was killed by signal SIGSEGV 2012-02-15 13:59:28 ERR control[27007]: (local): failed to fork mserver: database 'dw' has crashed after starting, manual intervention needed, check monetdbd's logfile for details >>The following is my platform information: MonetDB 5 server v11.7.5 "Dec2011" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 23.5GiB available memory, 24 available cpu cores Libraries: libpcre: 6.6 06-Feb-2006 (compiled with 6.6) openssl: OpenSSL 0.9.8e-rhel5 01 Jul 2008 (compiled with OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008) libxml2: 2.6.26 (compiled with 2.6.26) Compiled by: root@monet00 (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 Thanks in advance, Tony

1 0

[MonetDB-users] failed to fork mserver: database 'dw' has crashed after starting, manual intervention needed, check monetdbd's logfile for details
by Tony 15 Feb '12

15 Feb '12

Hi folks, How can I recover from a crashed database? I got this error when I restart a running database 'dw', Do you have any ideas on this? 2012-02-15 13:59:28 MSG control[27007]: (local): served status list 2012-02-15 13:59:28 MSG merovingian[27007]: startup of database under maintenance 'dw' forced 2012-02-15 13:59:28 MSG merovingian[27007]: database 'dw' has crashed after start on 2012-02-15 13:58:47, attempting restart, up min/avg/max: 1d/1d/1d, crash average: 1.00 1.00 0.97 (45-1=44) 2012-02-15 13:59:28 MSG dw[27831]: arguments: /usr/lib/monetdb/bin/mserver5 --set gdk_dbfarm=/data/warehouse --dbname=dw --set merovingian_uri=mapi:monetdb://monet00:50000/dw --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/warehouse/dw/.mapi.sock --set monet_vault_key=/data/warehouse/dw/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2012-02-15 13:59:28 MSG dw[27831]: # MonetDB 5 server v11.7.5 "Dec2011" 2012-02-15 13:59:28 MSG dw[27831]: # Serving database 'dw', using 24 threads 2012-02-15 13:59:28 MSG dw[27831]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2012-02-15 13:59:28 MSG dw[27831]: # Found 23.533 GiB available main-memory. 2012-02-15 13:59:28 MSG dw[27831]: # Copyright (c) 1993-July 2008 CWI. 2012-02-15 13:59:28 MSG dw[27831]: # Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved 2012-02-15 13:59:28 MSG dw[27831]: # Visit http://www.monetdb.org/ for further information 2012-02-15 13:59:28 MSG dw[27831]: # Listening for UNIX domain connection requests on mapi:monetdb:///data/warehouse/dw/.mapi.sock 2012-02-15 13:59:28 MSG dw[27831]: # MonetDB/SQL module loaded 2012-02-15 13:59:28 MSG merovingian[27007]: database 'dw' (27831) was killed by signal SIGSEGV 2012-02-15 13:59:28 ERR control[27007]: (local): failed to fork mserver: database 'dw' has crashed after starting, manual intervention needed, check monetdbd's logfile for details >>The following is my platform information: MonetDB 5 server v11.7.5 "Dec2011" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 23.5GiB available memory, 24 available cpu cores Libraries: libpcre: 6.6 06-Feb-2006 (compiled with 6.6) openssl: OpenSSL 0.9.8e-rhel5 01 Jul 2008 (compiled with OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008) libxml2: 2.6.26 (compiled with 2.6.26) Compiled by: root@monet00 (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 Thanks in advance, Tony

2 1

[MonetDB-users] Odd Single Thread Only Processing in Dec 2011 Release
by Henry Addington 15 Feb '12

15 Feb '12

Hi: We are testing Dec 2011 Release to upgrade from Oct 2010 SP1 release on CentOS 5.2 64-bit with 100+ GB memory. We have a BI tool that users use to generate various reports. We have access to the SQL statements that get generated by the BI tool and sent to the MonetDB. So, we know exactly the SQL statements that are being sent to MonetDB. One very odd thing that we are seeing the December release is that the reports from the BI tool are very, very slow (x10 - x15 from Oct releasee). It is obvious why looking at the server resource use. It looks like only 1 cpu is being used for all the queries! However, if I take the SQL statements generated by the BI tool and run them in DBVisualizer, all 20 cpu cores are used and the results return quickly. I am at a loss as to why we are seeing such different performance from identical SQL statements... Is there a configuration setting or something in the December release that we overlooked? Can you please help us troubleshoot? Thank you. HS

2 3

[MonetDB-users] select distinct ... where algorithm
by zack_pcd 13 Feb '12

13 Feb '12

I am seeing the following: select count(*) from f_sorted; +-----------+ | L7 | +===========+ | 102612097 | +-----------+ 1 tuple (0.988ms) select count(*) from f_sorted where yyyymm='201110'; +----------+ | L10 | +==========+ | 23505343 | +----------+ 1 tuple (1.0s) select count(distinct uuid) from f_sorted where yyyymm='201110'; +---------+ | L6 | +=========+ | 5941581 | +---------+ 1 tuple (23.8s) Very fast thus far! However: copy (select distinct(uuid) from f_sorted where yyyymm='201110') into '/home/user/filename' USING DELIMITERS ',','\n','"' NULL as 'null'; This has been running for hours and has yet to start writing to the file. I get this when I explain the query: +----------------------------------------------------------------------------------------+ | mal | +========================================================================================+ | function user.s5_3{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_22:bat[:oid,:str] := sql.bind(X_2,"sys","f_sorted","yyyymm",0); | | X_23 := algebra.uselect(X_22,"201110":str); | | X_20:bat[:oid,:str] := sql.bind(X_2,"sys","f_sorted","yyyymm",2); | | X_24 := algebra.kdifference(X_23,X_20); | | X_21 := algebra.uselect(X_20,"201110":str); | | X_25 := algebra.kunion(X_24,X_21); | | X_18:bat[:oid,:str] := sql.bind(X_2,"sys","f_sorted","yyyymm",1); | | X_19 := algebra.uselect(X_18,"201110":str); | | X_26 := algebra.kunion(X_25,X_19); | | X_16:bat[:oid,:oid] := sql.bind_dbat(X_2,"sys","f_sorted",1); | | X_17 := bat.reverse(X_16); | | X_27 := algebra.kdifference(X_26,X_17); | | X_28 := algebra.markT(X_27,0@0:oid); | | X_29 := bat.reverse(X_28); | | X_10:bat[:oid,:str] := sql.bind(X_2,"sys","f_sorted","uuid",0); | | X_8:bat[:oid,:str] := sql.bind(X_2,"sys","f_sorted","uuid",2); | | X_12 := algebra.kdifference(X_10,X_8); | | X_13 := algebra.kunion(X_12,X_8); | | X_5:bat[:oid,:str] := sql.bind(X_2,"sys","f_sorted","uuid",1); | | X_14 := algebra.kunion(X_13,X_5); | | X_30 := algebra.leftjoin(X_29,X_14); | | (ext44,grp42) := group.done(X_30); | | X_33 := bat.mirror(ext44); | | X_34 := algebra.leftjoin(X_33,X_30); | | X_35 := sql.resultSet(1,1,X_34); | | sql.rsColumn(X_35,"sys.f_sorted","uuid":str,"varchar",2048,0,X_34); | | X_43 := io.stdout(); | | sql.exportResult(X_43,X_35); | | end s5_3; | +----------------------------------------------------------------------------------------+ 31 tuples (1.350ms) Is it doing something wrong? I don't expect the query to take so long (hours) to produce the output, especially since the distinct count is also very fast. The copy-into *seems* to be irrelevant to the problem, but I decided to include it just in case, as that's how I am using the query. -- View this message in context: http://old.nabble.com/select-distinct-...-where-algorithm-tp33297578p332975… Sent from the monetdb-users mailing list archive at Nabble.com.

4 6

[MonetDB-users] False dependency
by Stathis Gkotsis 10 Feb '12

10 Feb '12

Hi all, trying to install: libmonetdb-client-dev_11.7.5-20120120_amd64.deb on Ubuntu 10.04, I see that there is a wrong dependency on libmonetdb-client1. Is there a way around this? Thank you.

1 0

[MonetDB-users] re cycler_pipe and cracker_pipe
by zack_pcd 08 Feb '12

08 Feb '12

Is it possible to have a mix of these pipe's optimizers so you get the benefit of both? Or are these two pipes incompatible? (recycler_pipe and cracker_pipe) -- View this message in context: http://old.nabble.com/recycler_pipe-and-cracker_pipe-tp33282420p33282420.ht… Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] LIMIT clause not recognized in nested query in FROM clause
by Viktor Rosenfeld 07 Feb '12

07 Feb '12

Hi, it appears that clauses like LIMIT or OFFSET are not allowed if the query is used inside a FROM clause. The following query throws an error: SELECT * FROM ( SELECT id FROM corpus LIMIT 10 ) AS inner_table; The error message is: syntax error, unexpected LIMIT, expecting INTERSECT or EXCEPT or UNION or ')' in: "select * Cheers, Viktor

1 0

[MonetDB-users] How can I execute SQL via mclient directly?
by bai dongli 02 Feb '12

02 Feb '12

Hi guys, I'm pretty new for Monetdb, As per the introduce of the online document, I have to type the password on the command line window when I want to execute SQL via the mclient program. It there a way like this which can be used in Mysql: "mysql --user --password databasename -e SQL" in monetdb? I didn't found the argument that can used for the "password" in mclient. Thanks, Tony

2 4

[MonetDB-users] Can't drop "global temporary table" in SCHEMA tmp
by bai dongli 02 Feb '12

02 Feb '12

Hi guys, I got a problem when I trying to drop a global temporary table in my database, I got following errors: sql>drop table tmp.test; 42S02!DROP TABLE: no such table 'test' I got the same error as above when I trying to execute a "select" query on that table, It's so weird, cause the table "test" was been there: sql>\d TABLE tmp.test The even more weird things happened when I execute following command: sql>create global temporary table tmp.test(id int); operation successful (1.888ms) sql>\d TABLE tmp.test sql>\d test table tmp.test is not unique, corrupt catalog? sql>\d TABLE tmp.test sql>drop table tmp.test more>; operation successful (1.470ms) sql>\d TABLE tmp.test Please pay attention on this line "table tmp.test is not unique, corrupt catalog?". Do anybody can help me out of here? Thanks, Tony

1 0

[MonetDB-users] Batch sql inserts in monetdb
by Federico Dayan 01 Feb '12

01 Feb '12

Hello all, I'm working in a project and we are planning to use monetdb. Our Data: We have around 15 millon users and 6,000 columns. Our Problem We need to be able to query specific log data. We use Hadoop to massage the data and create a sql file with sql inserts (around 15 millon insert with 3000 columns) Out Question Is it efficent to run a command like. By efficent I refe:. betweent 1 to 6 hours efficent, after that, a big problem. monetdb -u ....... < my_sql_insert.sql with 15 millon inserts and between 1000 to 3000 columns? (I'm not always going to create insert for the 6000 columns) could monetdb handle multiple of those command if a split the file. monetdb -u ....... < my_sql_insert_1.sql monetdb -u ....... < my_sql_insert_2.sql monetdb -u ....... < my_sql_insert_3.sql ... Is SQL COPY more efficient than a sql scrip file with inserts? BAT files is it significant faster ? I'm trying to avoid this approach since it seems a little bit obscue. any other suggestion? I'm not able to run all this test and get the info myself becuase I will not have a monetdb ready soon. Thank you very much and I would be glad to send back an email with the results I got and contribute in that way. Federico

4 3

[MonetDB-users] size of table
by Pavel Stehule 30 Jan '12

30 Jan '12

Hello I can't to search in doc a way to get size of tables. Is there some function or statement how to do it? Thank you Pavel Stehule

3 3

[MonetDB-users] Best backup strategy
by Sékine Coulibaly 27 Jan '12

27 Jan '12

Hi there, Thanks again for your last replies regarding the insertion into base. Using COPY INTO worked like a charm. A production deployment means : - Backup - High availability So my questions here, not addressed in the online docs are : - Regarding backup, is there a way to make a consistent backup (I mean putting the DB in a "backuping state", and save/archive the files) - To allow for high availability, can I use clusterization ? Can horizontal scalability be achieved through clusterization ? Thank you !

1 0

[MonetDB-users] Announcement: New Dec2011 Feature release of MonetDB suite
by Sjoerd Mullender 26 Jan '12

26 Jan '12

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2011 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at http://www.monetdb.org/. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2011 feature release SQL * mclient: The csv output format can now also be of the form csv=c and csv+c where c is the column separator. The form with csv+ produces a single header line with column names. * Changed a bug where the sign() function returned the same type as its argument instead of always an INTEGER. * Added a fix for bug #2834, which caused weird (failing) behaviour with PreparedStatements. * Fixed a bug when using default values for interval columns. Bug 2877. * A bug was fixed where deleted rows weren't properly accounted for in all operations. This was bug 2882. * A bug was fixed which caused an update to an internal table to happen too soon. The bug could be observed on a multicore system with a query INSERT INTO t (SELECT * FROM t) when the table t is "large enough". This was bug 2883. Java Module * Implemented getMetaData() method of PreparedStatement. * Fixed an AssertionError for special results from e.g. EXPLAIN queries. * Fixed overflow error when batching large statements, bug #2952 * Resolved a concurrency problem where ResultSet's date-related getters could cause odd stack traces when used by multiple threads at the same time, bug #2950. * JDBC now implements JDBCv4.1 which makes it possible to be built with Java 7 and up. JDBCv4.1 is a maintenance release of JDBCv4, and hence can be compiled with Java 6, although the added methods obviously are not part of the java.sql interfaces. * SQLExceptions thrown now carry a SQLSTATE. Until the server starts sending correct SQLSTATEs for all errors, server originated errors without SQLSTATE are considered generic data exceptions (22000). * Fixed a bug where closing ResultSets and PreparedStatements could lead to errors on concurrent running queries using the same Connection due to a race condition. * Changed version scheme of JDBC driver and MCL jar to be more standard, from monetdb-X.Y-<thing>.jar to monetdb-<thing>-X.Y.jar, bug #2943 * The embedded monet instance capability of MonetConnection was removed. * Bump JDBC version to 2.0 (Liberica). JDBC now implements JDBCv4 which makes it possible to be built with Java 6 and up. Java 5 and before are no longer supported, and can use the 1.x releases of the driver. * INTERVAL columns are now treated as decimals, since they can have sub-second precision. Client Package * Fix making connections with stethoscope to hosts without monetdbd. Bug #2944. * Fixed some bugs in the PHP interface affecting the mapi_ping(), monetdb_insert_id() and auto_commit() functions. Bugs #2936, #2937, #2938. * ODBC: Fixed SQLNumResultCols and SQLDescribeCol to return useful information after a call to SQLPrepare and before any SQLExecute. * mclient: The exact interpretation of the -i (--interactive) option and the - filename argument have changed. The - filename argument means read from standard input and no longer implies that no \ interpretation is done on the input. Instead, \ interpretation is done if either standard input is a terminal, or if the -i option is given. The -i option no longer causes mclient to read from standard input. It only means to do \ interpretation when reading from standard input. Use the - filename argument to read from standard input. Note that if no -s option is specified and no filename arguments are present, mclient still reads from standard input. * ODBC: Improved internal query for SQLSpecialColumns. Before, the query returned all columns taking part in a PRIMARY KEY *and* all columns taking part in a UNIQUE constraint. Now it returns only one or the other set. * ODBC: The database name is now used as the catalog name throughout. Functions that return a catalog name return the database name, and functions that match on catalog name match it with the database name. * ODBC: Implemented an easier way to create a log file of interactions with the ODBC driver. You can now add a connection attribute "LOGFILE=filename" to the connection string parameter of SQLBrowseConnect and SQLDriverConnect, and to the relevant part of the Windows registry or odbc.ini file. This value is only used if there is no environment variable ODBCDEBUG. * Quoting of object names for mclient's \d command is now more flexible and consistent with standard SQL quoting rules, bug #2846. * ODBC: Implemented SQLColumnPrivileges function. * Install new program, mnc, which provides netcat functionality, but based on MonetDB's communication libraries * Report full URI to database instead of just database when available in mclient. * The time format of the timer output can now be controlled with an optional argument to the -i option. ms, s and m force the time to be formatted as milliseconds, seconds or minutes + seconds respectively. * ODBC: Implemented a workaround in SQLTables for bug 2908. * Small improvement to mclient's table rendering for tables without any rows. Previously, the column names in the header could be squeezed to very small widths, degrading readability. * Python DB API connect() function now supports PEP 249-style arguments user and host, bug #2901 * mclient now checks the result of encoding conversions using the iconv library. * Fixed a source of crashes in mclient when a query on the command line using the -s option is combined with input on standard input (e.g. in the construct mclient -s 'COPY INTO t FROM STDIN ...' < file.csv). * Fixed bug 2897 where slow (network) reads could cause blocks to not be fully read in one go, causing errors in the subsequent use of those blocks. With thanks to Rémy Chibois. * Fixed a bug in mclient where processing queries from files could result in ghost empty results to be reported in the output * Fixed Perl DBD rowcount for larger results, bug #2889 * Perl: We now distinguish properly between TABLE and GLOBAL TEMPORARY (the latter are recognized by being in the "tmp" schema). * Perl: fixed a bunch of syntax errors. This fixes bug 2884. With thanks to Rémy Chibois. * Perl: Fixed DBD::monetdb table_info and tabletype_info. This fixes bug 2885. With thanks to Rémy Chibois. * mclient: fix display of varchar columns with only NULL values. * Fixed a bug in mclient/msqldump where an internal error occurred during dump when there are GLOBAL TEMPORARY tables. MonetDB5 Server * A couple of memory leaks have been plugged. * Introduced new variable max_clients that allows to define how many concurrent connections are allowed to be made against the database. * mal_interpreter.mx Protect against concurrent exceptions If multiple parallel blocks access the exception variables and perform GDKfree on old messages, then we may enter a case that one thread attempts a second free. Simple lock-based protection is the first line of defense. * bpm and partitions have been moved to the attic. It is replaced by the partition optimizer, still under construction. * The MAL debugger next/step operations semantics have been fixed. The profiler also now clearly shows entrance/exit of a MAL function. * The dataflow optimizer uses a less strict side-effect test for BAT new operations and better recognizes trivial plans. * Fixed a problem where MAL variables weren't properly cleared before reuse of the data strucutre. This problem could cause the data flow scheduler to generate dependencies between instructions that didn't actually exist, which in turn could cause circular dependencies among instructions with deadlock as a result. Bugs 2865 and 2888. Merovingian * Fixed crash in monetdb when an invalid property was retrieved using the get command, bug #2953. * monetdb get output is now grouped by database instead of by property * Unlike in previous releases, 'monetdbd get mserver' now returns the path to the mserver5 binary only for dbfarms that have a running monetdbd, instead of only for those that are not served by a monetdbd. * Add nclients property that controls the maximum number of concurrent clients allowed to access the database * Starting monetdbd without any arguments or without dbfarm is no longer supported. A dbfarm now must be provided for each command. * The control passphrase has been turned into a hash of the password, for a more stronger authorisation model as used by mclients. On upgrade, any existing passphrase is converted to the hashed version of the passphrase. * The monetdbd controlport option has been removed in favour of a boolean control option. On upgrade, when controlport was set to a non-zero value and a passphrase was set, control is set to true, or false otherwise. * The monetdbd discoveryport option has been removed in favour of a boolean discovery option. On upgrade, when discoveryport was set to a non-zero value, discovery is set to true, or false otherwise. * Fixed a bug where monetdbd's socket files from /tmp were removed when a second monetdbd was attempted to be started using the same port. * Resolved problem where monetdbd would terminate abnormally when databases named 'control', 'discovery' or 'merovingian' were stopped. * monetdbd get status now also reports the version of the running monetdbd * Improved response time of 'monetdb start' when the database fails to start. * Fixed a bug in monetdbd where starting a failing database could incorrectly be reported as a 'running but dead' database. * To avoid confusion, all occurrences of merovingian were changed into monetdbd for error messages sent to a client. MonetDB Common * Fixed memory detection on 32-bits Solaris systems with more memory than can be addressed in 32-bits mode, bug #2935 Bug Fixes

1 0

[MonetDB-users] measure performance time
by jfergar 26 Jan '12

26 Jan '12

Hi, I need to measure the sql query execution time of several queries. The mclient "-t" option does not exist in the newer version, and the "\< file_with_queries" SQL command does not seem to show the time when importing from a file. Any ideas?? -- View this message in context: http://old.nabble.com/measure-performance-time-tp33201681p33201681.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 3

[MonetDB-users] Strange disk I/O behavior
by Viktor Rosenfeld 26 Jan '12

26 Jan '12

Hi, I'm observing disk I/O when evaluating some queries which I am unable to explain. I have two datasets, one relatively small with the BATs taking up 1.6 MB on disk and one fairly large with 283 MB on disk. According to iostat, when I evaluate the attached query on the small dataset, 100 MB are written to disk, but none are read. This is on a Linux machine with 4 GB RAM running the Aug2011-SP3 release. I don't understand why data is written on disk, that (a) is never read again and (b) would easily fit into main memory. I am unable to evaluate the queries on the large dataset because MonetDB writes so much data that the disk fills up after a few minutes (more than 22GB). Please let me know if I can provide more information. Cheers, Viktor

3 12

[MonetDB-users] MonetDB to X100 relations
by Pavel Stehule 24 Jan '12

24 Jan '12

Hello I am interested about current state of MonetDB/X100 project? It is freely available? It is part of current MonetDB 5 project? Can we expect some progress and possible availability under some open source licence? Thank you for information Best regards Pavel Stehule

2 2

[MonetDB-users] ODBC Error
by Luiz Gustavo de Souza 18 Jan '12

18 Jan '12

Greetings MonetDB developers! I'm preparing MonetDB for an application, and I got the current error from the ODBC: [MonetDB][ODBC Driver 11.5.9]Cannot return the number of output columns. Query must be executed first. I'm using Windows 7 Ultimate 64bits with both ODBC x86 and x64, but the system and also the Microsoft Excel only find the x86 driver. How can I overcome this? Thank you very much.

2 1

[MonetDB-users] Slow import from a SQL (20 tps)
by Sékine Coulibaly 18 Jan '12

18 Jan '12

I'm evaluating MonetDB and importing an SQL file containing INSERT statements, so that I can populate the following table : CREATE TABLE session ( cmd character varying(25), date bigint, session_id character varying(64), score integer, src_ip character varying(25), dst_ip character varying(25), tx_bytes integer, rx_bytes integer, retx_bytes integer, scrambling character varying(16), url character varying(64), country character varying(32), orig character varying(3), status integer ) No index was created. >From what I can see, I'm somewhere between 20 and 40 insertions per second (Core i5, 4GB RAM, Windows7, MonetDb MonetDB5-SQL-Installer-i386-20111021.msi). The Windows task manager shows a disk write throughput of 300 kB/s which looks like very small, processor usage around 5%. I imported the file with mclient -u monetdb < myfile.sql I expected to import at 200 rows/second minimum. Am I doing something wrong ? What can be done to improve the loading speed ? Thank you SC

2 2

[MonetDB-users] How to find implementation function of a MAL function
by Charalampos Nikolaou 18 Jan '12

18 Jan '12

Hello, I would like to know whether there is any way to find out the C function implementing a MAL function through mclient (maybe with the ?pattern command). Let's say I search for batcalc.and(:bat, :bat) function. If there isn't any, then what is the C function implementing the MAL function I mentioned previously? Thanks, Babis

2 1

[MonetDB-users] C++ Driver (postgresql proxy)
by Sékine Coulibaly 18 Jan '12

18 Jan '12

Hi, I plan to use MonetDb in a C++ project and have the following questions/concerns : - Using an third party ODBC library (such as SOCI) to add ODBC support to my project looks like risky. Is MonetDB fully compliant with ODBC ? Does anyone know if it works well with SOCI (or any other open source ODBC library) ? - I really would prefer using an integrated library (Postgresql for example ships with a client library for embedding SQL in C/C++ code, and a libpq library). I guess this is not available for MonetDb. Confirmed ? - Previously, I came through a driver that allowed to proxy MonetDb to a virtual Postgresql. I mean that the proxy attacks MonetDb in ODBC (and hopefully it does it well), and then exposes the MonetDb database through the standard Postgesql interface. Hence you can use PgAdminIII, pgadmin... on port 5432 which is the Postgesql port. I can't find this proxy driver anymore. Does anyone ? Thank you !

1 0

[MonetDB-users] Stored Procedure code example.
by Miguel Chang 18 Jan '12

18 Jan '12

Hello there, before I start with my question I would like to thank all those developers who put so much effort on this great Open Source Database. My name is Miguel, I´m currently using the MonetDB database for a university project and so far it´s been great. By following the documentation on the webpage I´ve build a fully functional system but I´ve encoutered one problem: So far I´ve been unable to create an stored procedure, even after following all the instructions on the documentation page: http://www.monetdb.org/Documentation/Manuals/SQLreference/Procedures Here´s an example I´ve tried but that does not works: CREATE PROCEDURE SP_GET_ALL_EVENTS () BEGIN SELECT * FROM DEMO.EVENTS; END; >From the many variation I tried with the last query I always get these errors: "Unexpected From, Expected In" or "unexpected END" I´ve been trying many variations (INSERT, UPDATES, DELETES) and I get the same responses... So can someone please point me what´s the error in my code? and also show me some examples codes for stored procedure creation? Much Appreciated.

2 2

[MonetDB-users] MonetDB unexpected count/group by slowdown
by zack_pcd 17 Jan '12

17 Jan '12

I have a table f_trigger with 171,452,378 records. It has fields script_id and machine_uuid (and other fields not used here). I want to run select script_id,count(distinct machine_uuid) from f_trigger group by script_id; First, though, let's get the number of expected records only: select count(*) from (select script_id,count(distinct machine_uuid) from f_trigger group by script_id) as subq; +------+ | L7 | +======+ | 208 | +------+ 1 tuple (359.124ms) Very nice. Let's explain the query before I run it: explain select script_id,count(distinct machine_uuid) from f_trigger group by script_id; +-------------------------------------------------------------------------------+ | mal | +===============================================================================+ | function user.s0_1{autoCommit=true}():void; | | X_21 := nil:bat[:oid,:str]; | | X_39:bat[:oid,:wrd] := nil:bat[:oid,:wrd]; | | barrier X_596 := language.dataflow(); | | X_2 := sql.mvc(); | | X_12:bat[:oid,:str] := sql.bind(X_2,"sys","f_trigger","script_id",0); | | X_10:bat[:oid,:str] := sql.bind(X_2,"sys","f_trigger","script_id",2); | | X_14 := algebra.kdifference(X_12,X_10); | | X_15 := algebra.kunion(X_14,X_10); | | X_9:bat[:oid,:str] := sql.bind(X_2,"sys","f_trigger","script_id",1); | | X_16 := algebra.kunion(X_15,X_9); | | X_5:bat[:oid,:oid] := sql.bind_dbat(X_2,"sys","f_trigger",1); | | X_8 := bat.reverse(X_5); | | X_17 := algebra.kdifference(X_16,X_8); | | (ext28,grp26) := group.done(X_17); | | X_20 := bat.mirror(ext28); | | X_21 := algebra.join(X_20,X_17); | | X_29:bat[:oid,:str] := sql.bind(X_2,"sys","f_trigger","machine_uuid",0); | | X_30 := algebra.selectNotNil(X_29); | | X_26:bat[:oid,:str] := sql.bind(X_2,"sys","f_trigger","machine_uuid",2); | | X_31 := algebra.kdifference(X_30,X_26); | | X_28 := algebra.selectNotNil(X_26); | | X_32 := algebra.kunion(X_31,X_28); | | X_24:bat[:oid,:str] := sql.bind(X_2,"sys","f_trigger","machine_uuid",1); | | X_25 := algebra.selectNotNil(X_24); | | X_33 := algebra.kunion(X_32,X_25); | | X_34 := algebra.kdifference(X_33,X_8); | | (X_35,grp55) := group.derive(ext28,grp26,X_34); | | X_37 := bat.mirror(X_35); | | X_38 := algebra.join(X_37,X_34); | | X_39:bat[:oid,:wrd] := aggr.count(X_38,grp26,X_20); | | exit X_596; | | X_40 := sql.resultSet(2,1,X_21); | | sql.rsColumn(X_40,"sys.f_trigger","script_id","varchar",32768,0,X_21); | | sql.rsColumn(X_40,"sys.f_trigger","L10","wrd",64,0,X_39); | | X_53 := io.stdout(); | | sql.exportResult(X_53,X_40); | | end s0_1; | +-------------------------------------------------------------------------------+ 38 tuples (3.136ms) Running the query takes a very long time. While the query is running, I notice VERY heavy disk writes, and almost no reads and almost no CPU activity. Based on other information on the mailing list, I suspect the query generates intermediate data that gets writted to disk (instead of kept in memory). We are running monetdb on a 32 core linux machine with 256 GB of RAM and a fast raid array. Memory usage is under 20%, so I would not expect monetdb to be writing to disk so much (50-500MB/s writes)... Database settings: shared default yes nthreads local 32 optpipe local recycler_pipe master default no slave default <unknown> readonly local yes Stethoscope output: [1] # event, time, thread, usec, stmt, # name [1] [ 256, "14:02:38.525477", 2, 0, "calc.str(\"script_id\");" ] [1] [ 257, "14:02:38.525477", 2, 134, "calc.str(\"script_id\");" ] [1] [ 258, "14:02:38.525634", 2, 0, "calc.str(\"f_trigger\");" ] [1] [ 259, "14:02:38.525634", 2, 18, "calc.str(\"f_trigger\");" ] [1] [ 260, "14:02:38.525670", 2, 0, "calc.str(\"script_id\");" ] [1] [ 261, "14:02:38.525670", 2, 16, "calc.str(\"script_id\");" ] [1] [ 262, "14:02:38.525704", 2, 0, "calc.str(\"f_trigger\");" ] [1] [ 263, "14:02:38.525704", 2, 17, "calc.str(\"f_trigger\");" ] [1] [ 264, "14:02:38.525740", 2, 0, "calc.str(\"script_id\");" ] [1] [ 265, "14:02:38.525740", 2, 17, "calc.str(\"script_id\");" ] [1] [ 266, "14:02:38.525777", 2, 0, "calc.str(\"f_trigger\");" ] [1] [ 267, "14:02:38.525777", 2, 19, "calc.str(\"f_trigger\");" ] [1] [ 268, "14:02:38.525812", 2, 0, "calc.str(\"script_id\");" ] [1] [ 269, "14:02:38.525812", 2, 19, "calc.str(\"script_id\");" ] [1] [ 270, "14:02:38.525848", 2, 0, "calc.str(\"f_trigger\");" ] [1] [ 271, "14:02:38.525848", 2, 19, "calc.str(\"f_trigger\");" ] [1] [ 272, "14:02:38.525884", 2, 0, "calc.str(\"L11\");" ] [1] [ 273, "14:02:38.525884", 2, 19, "calc.str(\"L11\");" ] [1] [ 274, "14:02:38.525925", 2, 0, "calc.str(\"f_trigger\");" ] [1] [ 275, "14:02:38.525925", 2, 20, "calc.str(\"f_trigger\");" ] [1] [ 276, "14:02:38.525962", 2, 0, "calc.str(\"L11\");" ] [1] [ 277, "14:02:38.525962", 2, 18, "calc.str(\"L11\");" ] [1] [ 278, "14:02:38.526000", 2, 0, "calc.str(\"f_trigger\");" ] [1] [ 279, "14:02:38.526000", 2, 18, "calc.str(\"f_trigger\");" ] [1] [ 280, "14:02:38.526039", 2, 0, "calc.str(\"machine_uuid\");" ] [1] [ 281, "14:02:38.526039", 2, 19, "calc.str(\"machine_uuid\");" ] [1] [ 282, "14:02:38.526077", 2, 0, "calc.str(\"f_trigger\");" ] [1] [ 283, "14:02:38.526077", 2, 20, "calc.str(\"f_trigger\");" ] [1] [ 284, "14:02:38.526116", 2, 0, "calc.str(nil:str);" ] [1] [ 285, "14:02:38.526116", 2, 22, "calc.str(nil:str);" ] [1] [ 286, "14:02:38.528707", 2, 0, "X_21 := nil:bat[:oid,:str];" ] [1] [ 287, "14:02:38.528707", 2, 64, "X_21 := nil:bat[:oid,:str];" ] [1] [ 288, "14:02:38.528790", 2, 0, "X_39:bat[:oid,:wrd] := nil:bat[:oid,:wrd];" ] [1] [ 289, "14:02:38.528790", 2, 27, "X_39:bat[:oid,:wrd] := nil:bat[:oid,:wrd];" ] [1] [ 290, "14:02:38.528835", 2, 0, "barrier X_596 := language.dataflow();" ] [1] [ 291, "14:02:38.528928", 40, 0, "X_2 := sql.mvc();" ] [1] [ 292, "14:02:38.528928", 40, 141, "X_2 := sql.mvc();" ] [1] [ 293, "14:02:38.529100", 40, 0, "X_12:bat[:oid,:str] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"script_id\",0);" ] [1] [ 294, "14:02:38.529100", 40, 78, "X_12:bat[:oid,:str] =<tmp_7153>[171452378] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"script_id\",0);" ] [1] [ 295, "14:02:38.529255", 52, 0, "X_10:bat[:oid,:str] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"script_id\",2);" ] [1] [ 296, "14:02:38.529255", 52, 229, "X_10:bat[:oid,:str] =<tmp_11631>[0] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"script_id\",2);" ] [1] [ 297, "14:02:38.529491", 45, 0, "X_5:bat[:oid,:oid] := sql.bind_dbat(X_2=0,\"sys\",\"f_trigger\",1);" ] [1] [ 298, "14:02:38.529491", 45, 201, "X_5:bat[:oid,:oid] =<tmp_11616>[0] := sql.bind_dbat(X_2=0,\"sys\",\"f_trigger\",1);" ] [1] [ 299, "14:02:38.529699", 50, 0, "X_9:bat[:oid,:str] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"script_id\",1);" ] [1] [ 300, "14:02:38.529699", 50, 227, "X_9:bat[:oid,:str] =<tmp_11632>[0] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"script_id\",1);" ] [1] [ 301, "14:02:38.529951", 56, 0, "X_24:bat[:oid,:str] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"machine_uuid\",1);" ] [1] [ 302, "14:02:38.530203", 52, 0, "X_14 := algebra.kdifference(X_12=<tmp_7153>:bat[:oid,:str][171452378],X_10=<tmp_11631>:bat[:oid,:str][0]);" ] [1] [ 303, "14:02:38.529951", 56, 259, "X_24:bat[:oid,:str] =<tmp_11632>[0] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"machine_uuid\",1);" ] [1] [ 304, "14:02:38.530203", 52, 112, "X_14=<tmp_12321>[171452378] := algebra.kdifference(X_12=nil:bat[:oid,:str],X_10=<tmp_11631>:bat[:oid,:str][0]);" ] [1] [ 305, "14:02:38.530308", 45, 0, "X_8 := bat.reverse(X_5=<tmp_11616>:bat[:oid,:oid][0]);" ] [1] [ 306, "14:02:38.530308", 45, 143, "X_8=<tmpr_11616>[0] := bat.reverse(X_5=nil:bat[:oid,:oid]);" ] [1] [ 307, "14:02:38.530465", 57, 0, "X_29:bat[:oid,:str] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"machine_uuid\",0);" ] [1] [ 308, "14:02:38.530668", 56, 0, "X_25 := algebra.selectNotNil(X_24=<tmp_11632>:bat[:oid,:str][0]);" ] [1] [ 309, "14:02:38.530465", 57, 210, "X_29:bat[:oid,:str] =<tmp_10750>[171452378] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"machine_uuid\",0);" ] [1] [ 310, "14:02:38.530668", 38, 0, "X_26:bat[:oid,:str] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"machine_uuid\",2);" ] [1] [ 311, "14:02:38.530668", 56, 44, "X_25=<tmp_11632>[0] := algebra.selectNotNil(X_24=nil:bat[:oid,:str]);" ] [1] [ 312, "14:02:38.530668", 38, 1743, "X_26:bat[:oid,:str] =<tmp_11631>[0] := sql.bind(X_2=0,\"sys\",\"f_trigger\",\"machine_uuid\",2);" ] [1] [ 313, "14:02:38.530946", 61, 0, "X_15 := algebra.kunion(X_14=<tmp_12321>[171452378],X_10=<tmp_11631>:bat[:oid,:str][0]);" ] [1] [ 314, "14:02:38.531115", 57, 0, "X_30 := algebra.selectNotNil(X_29=<tmp_10750>:bat[:oid,:str][171452378]);" ] [1] [ 315, "14:02:38.531115", 57, 39, "X_30=<tmp_10750>[171452378] := algebra.selectNotNil(X_29=nil:bat[:oid,:str]);" ] [1] [ 316, "14:02:38.530946", 61, 210, "X_15=<tmp_12211>[171452378] := algebra.kunion(X_14=nil,X_10=nil:bat[:oid,:str]);" ] [1] [ 317, "14:02:38.531190", 57, 0, "X_31 := algebra.kdifference(X_30=<tmp_10750>[171452378],X_26=<tmp_11631>:bat[:oid,:str][0]);" ] [1] [ 318, "14:02:38.531271", 61, 0, "X_16 := algebra.kunion(X_15=<tmp_12211>[171452378],X_9=<tmp_11632>:bat[:oid,:str][0]);" ] [1] [ 319, "14:02:38.531190", 57, 99, "X_31=<tmp_12321>[171452378] := algebra.kdifference(X_30=nil,X_26=<tmp_11631>:bat[:oid,:str][0]);" ] [1] [ 320, "14:02:38.531271", 61, 42, "X_16=<tmp_12216>[171452378] := algebra.kunion(X_15=nil,X_9=nil:bat[:oid,:str]);" ] [1] [ 321, "14:02:38.531346", 57, 0, "X_28 := algebra.selectNotNil(X_26=<tmp_11631>:bat[:oid,:str][0]);" ] [1] [ 322, "14:02:38.531346", 57, 55, "X_28=<tmp_11631>[0] := algebra.selectNotNil(X_26=nil:bat[:oid,:str]);" ] [1] [ 323, "14:02:38.531408", 61, 0, "X_17 := algebra.kdifference(X_16=<tmp_12216>[171452378],X_8=<tmpr_11616>[0]);" ] [1] [ 324, "14:02:38.531466", 57, 0, "X_32 := algebra.kunion(X_31=<tmp_12321>[171452378],X_28=<tmp_11631>[0]);" ] [1] [ 325, "14:02:38.531408", 61, 59, "X_17=<tmp_12211>[171452378] := algebra.kdifference(X_16=nil,X_8=<tmpr_11616>[0]);" ] [1] [ 326, "14:02:38.531466", 57, 49, "X_32=<tmp_12216>[171452378] := algebra.kunion(X_31=nil,X_28=nil);" ] [1] [ 327, "14:02:38.531537", 61, 0, "(ext28,grp26) := group.done(X_17=<tmp_12211>[171452378]);" ] [1] [ 328, "14:02:38.531596", 57, 0, "X_33 := algebra.kunion(X_32=<tmp_12216>[171452378],X_25=<tmp_11632>[0]);" ] [1] [ 329, "14:02:38.531596", 57, 34, "X_33=<tmp_12261>[171452378] := algebra.kunion(X_32=nil,X_25=nil);" ] [1] [ 330, "14:02:38.531660", 57, 0, "X_34 := algebra.kdifference(X_33=<tmp_12261>[171452378],X_8=<tmpr_11616>[0]);" ] [1] [ 331, "14:02:38.531660", 57, 28, "X_34=<tmp_12216>[171452378] := algebra.kdifference(X_33=nil,X_8=nil);" ] [1] [ 332, "14:02:38.531537", 61, 4488161, "(ext28=<tmp_12207>[208],grp26=<tmp_12321>[171452378]) := group.done(X_17=<tmp_12211>[171452378]);" ] [1] [ 333, "14:02:43.019887", 61, 0, "X_20 := bat.mirror(ext28=<tmp_12207>[208]);" ] [1] [ 334, "14:02:43.019887", 61, 39, "X_20=<tmp_12261>[208] := bat.mirror(ext28=<tmp_12207>[208]);" ] [1] [ 335, "14:02:43.019962", 61, 0, "(X_35,grp55) := group.derive(ext28=<tmp_12207>[208],grp26=<tmp_12321>[171452378],X_34=<tmp_12216>[171452378]);" ] [1] [ 336, "14:02:43.020045", 60, 0, "X_21 := algebra.join(X_20=<tmp_12261>[208],X_17=<tmp_12211>[171452378]);" ] [1] [ 337, "14:02:43.020045", 60, 505, "X_21=<tmp_12332>[208] := algebra.join(X_20=<tmp_12261>[208],X_17=nil);" ] [1] [ 338, "13:50:05.243130", 21, 2365556004, "(X_41=<tmp_12243>[90737912],grp74) := group.derive(ext30=nil,grp28=<tmp_12111>[171452378],X_40=<tmp_12110>[171452378]);" ] [1] [ 339, "14:29:30.799335", 21, 0, "X_43 := bat.mirror(X_41=<tmp_12243>[90737912]);" ] [1] [ 340, "14:29:30.799335", 21, 44, "X_43=<tmp_12113>[90737912] := bat.mirror(X_41=nil);" ] [1] [ 341, "14:29:30.799408", 21, 0, "X_44 := algebra.join(X_43=<tmp_12113>[90737912],X_40=<tmp_12110>[171452378]);" ] [1] [ 342, "14:29:30.799408", 21, 1931164, "X_44=<tmp_12117>[90737912] := algebra.join(X_43=nil,X_40=nil);" ] [1] [ 343, "14:29:32.730825", 21, 0, "X_45:bat[:oid,:wrd] := aggr.count(X_44=<tmp_12117>[90737912],grp28=<tmp_12111>[171452378],X_20=<tmp_12107>[208]);" ] [1] [ 344, "14:29:32.730825", 21, 2686923, "X_45:bat[:oid,:wrd] =<tmp_12110>[208] := aggr.count(X_44=nil,grp28=nil,X_20=nil);" ] [1] [ 345, "13:49:55.404991", 35, 2380012994, "barrier X_602 := language.dataflow();" ] [1] [ 346, "14:29:35.418135", 35, 0, "X_47 := sql.resultSet(2,1,X_25=<tmp_12112>[10]);" ] [1] [ 347, "14:29:35.418135", 35, 53, "X_47 := sql.resultSet(2,1,X_25=<tmp_12112>[10]);" ] [1] [ 348, "14:29:35.418208", 35, 0, "X_46 := algebra.leftjoin(X_24=<tmp_12115>[10],X_45=<tmp_12110>:bat[:oid,:wrd][208]);" ] [1] [ 349, "14:29:35.418208", 35, 130, "X_46=<tmp_12172>[10] := algebra.leftjoin(X_24=nil,X_45=nil:bat[:oid,:wrd]);" ] [1] [ 350, "14:29:35.418370", 35, 0, "sql.rsColumn(X_47=1,\"sys.f_trigger\",\"script_id\",\"varchar\",32768,0,X_25=<tmp_12112>[10]);" ] [1] [ 351, "14:29:35.418370", 35, 36, "sql.rsColumn(X_47=1,\"sys.f_trigger\",\"script_id\",\"varchar\",32768,0,X_25=nil);" ] [1] [ 352, "14:29:35.418429", 35, 0, "sql.rsColumn(X_47=1,\"sys.f_trigger\",\"L6\",\"wrd\",64,0,X_46=<tmp_12172>[10]);" ] [1] [ 353, "14:29:35.418429", 35, 34, "sql.rsColumn(X_47=1,\"sys.f_trigger\",\"L6\",\"wrd\",64,0,X_46=nil);" ] [1] [ 354, "14:29:35.418483", 35, 0, "X_57 := io.stdout();" ] [1] [ 355, "14:29:35.418483", 35, 18, "X_57 := io.stdout();" ] [1] [ 356, "14:29:35.418515", 35, 0, "sql.exportResult(X_57==\"42083632\":streams,X_47=1);" ] [1] [ 357, "14:29:35.418515", 35, 73, "sql.exportResult(X_57==\"42083632\":streams,X_47=1);" ] [1] [ 358, "14:29:35.418611", 35, 0, "end s3_2;" ] [1] [ 359, "14:02:43.019962", 61, 2633497166, "(X_35=<tmp_12112>[90737912],grp55) := group.derive(ext28=nil,grp26=<tmp_12321>[171452378],X_34=<tmp_12216>[171452378]);" ] [1] [ 360, "14:46:36.517329", 61, 0, "X_37 := bat.mirror(X_35=<tmp_12112>[90737912]);" ] [1] [ 361, "14:46:36.517329", 61, 43, "X_37=<tmp_12206>[90737912] := bat.mirror(X_35=nil);" ] [1] [ 362, "14:46:36.517404", 61, 0, "X_38 := algebra.join(X_37=<tmp_12206>[90737912],X_34=<tmp_12216>[171452378]);" ] [1] [ 363, "14:46:36.517404", 61, 2374517, "X_38=<tmp_12211>[90737912] := algebra.join(X_37=nil,X_34=nil);" ] [1] [ 364, "14:46:38.892126", 61, 0, "X_39:bat[:oid,:wrd] := aggr.count(X_38=<tmp_12211>[90737912],grp26=<tmp_12321>[171452378],X_20=<tmp_12261>[208]);" ] [1] [ 365, "14:46:38.892126", 61, 2690276, "X_39:bat[:oid,:wrd] =<tmp_12216>[208] := aggr.count(X_38=nil,grp26=nil,X_20=nil);" ] [1] [ 366, "14:02:38.528835", 2, 2643053835, "barrier X_596 := language.dataflow();" ] [1] [ 367, "14:46:41.583009", 2, 0, "X_40 := sql.resultSet(2,1,X_21=<tmp_12332>[208]);" ] [1] [ 368, "14:46:41.583009", 2, 48, "X_40 := sql.resultSet(2,1,X_21=<tmp_12332>[208]);" ] [1] [ 369, "14:46:41.583076", 2, 0, "sql.rsColumn(X_40=1,\"sys.f_trigger\",\"script_id\",\"varchar\",32768,0,X_21=<tmp_12332>[208]);" ] [1] [ 370, "14:46:41.583076", 2, 40, "sql.rsColumn(X_40=1,\"sys.f_trigger\",\"script_id\",\"varchar\",32768,0,X_21=nil);" ] [1] [ 371, "14:46:41.583140", 2, 0, "sql.rsColumn(X_40=1,\"sys.f_trigger\",\"L11\",\"wrd\",64,0,X_39=<tmp_12216>:bat[:oid,:wrd][208]);" ] [1] [ 372, "14:46:41.583140", 2, 35, "sql.rsColumn(X_40=1,\"sys.f_trigger\",\"L11\",\"wrd\",64,0,X_39=nil:bat[:oid,:wrd]);" ] [1] [ 373, "14:46:41.583198", 2, 0, "X_53 := io.stdout();" ] [1] [ 374, "14:46:41.583198", 2, 18, "X_53 := io.stdout();" ] [1] [ 375, "14:46:41.583230", 2, 0, "sql.exportResult(X_53==\"35761888\":streams,X_40=1);" ] [1] [ 376, "14:46:41.583230", 2, 403, "sql.exportResult(X_53==\"35761888\":streams,X_40=1);" ] [1] [ 377, "14:46:41.583661", 2, 0, "end s1_1;" ] The query took 44 minutes to complete. Is this a bug (as I suspect)? Or is the group by/count distinct operation just slow? If it is a bug, is this fixed in dec-2011? If that is the case, I would like to verify this was fixed for us in dec-2011 - however the last "latest" build I tried was crashing for me. I am not sure which version of dec-2011 I should try for testing this. Thanks -- View this message in context: http://old.nabble.com/MonetDB-unexpected-count-group-by-slowdown-tp33137428… Sent from the monetdb-users mailing list archive at Nabble.com.

3 4

[MonetDB-users] column sortedness
by David Pennell 13 Jan '12

13 Jan '12

I'm running some experiments with a 1B row fact table (10 columns). - How can I tell if a column is sorted? - If I generate files in sorted order and use binary bulk loading, will monetdb consider the column sorted? If not, can I force it? I've been reading code, but understanding all the assignments to bn->tsorted is going to take a while. I get the general theme of setting tsorted during bat operations, but I've been unsuccessful in finding where it is initialized when a column is first accessed. TIA, -david

2 2

[MonetDB-users] Prepare statements as performance upgrade?
by Stefan de Konink 12 Jan '12

12 Jan '12

Hi, When I look into the executemany implementation of the Python DBAPI, I see that executemany is implemented as loop over the parameters given the same statement. In the mapi documentation there is written about using mapi_prepare_array() and mapi_execute_array(). Would such thing be beneficial in the general case? In my own program I have the following: while True: received = data_from_network() update_monetdb(received) It seems that the standard DBAPI doesn't expose prepared statements as some sort of transparent usage, so using SQL PREPARE sounds as the most logical way. What is the SQL statement to 'forget' the stored prepared statement? Stefan

1 0

[MonetDB-users] Disk space of a table
by Vassilis Papakonstantinou 11 Jan '12

11 Jan '12

Hello, Is there a way to get the disk space of a table? Vassilis

3 4

[MonetDB-users] how to load NaN as REAL
by bigu 10 Jan '12

10 Jan '12

I have a csv file and some columns are defined as REAL. The values some time have 'NaN' so when trying to load in the db it says cannot load NaN into REAL. COPY 1000000 OFFSET 2 RECORDS into mytable from 'C:\temp.csv' delimiters ',', '\n'; SQLException:importTable:value 'NaN' from line 11 field 10 not inserted, expecting type real failed to import table -- View this message in context: http://old.nabble.com/how-to-load-NaN-as-REAL-tp33107777p33107777.html Sent from the monetdb-users mailing list archive at Nabble.com.

3 2

[MonetDB-users] load CSV without header row
by bigu 09 Jan '12

09 Jan '12

Today installed monetdb for first time and wanted to play with some of the features but stuck with loading the file. Excuse me if this looks too basic. There is a 3GB csv file (1M rows and 400 columns). I am trying load in monetdb on my windows 7 machine. I first created a table mostly data is numeric exect for few that are metadata strings. How do i load this file without removing the header. The error says column_name not of type integer since the column_name is the actual header string but the values are integers. I can remove the header manually before loading but if i plan to scale then it is not a good option. Is there a way to say load "n" records from file starting at 'x' to 'y'? or ignore first line. COPY 1000000 RECORDS into mytable from 'C:\data.csv' delimiters ',', '\n'; Thanks. -- View this message in context: http://old.nabble.com/load-CSV-without-header-row-tp33096188p33096188.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 2

[MonetDB-users] [PATCH]: python-monetdb does not support RIPEMD160
by W-Mark Kubacki 09 Jan '12

09 Jan '12

Hello! Although MonetDB supports RIPEMD160, python-monetdb (version 11.5.9) does not. Patch [1] fixes that. Unfortunately Python's hashlib.algorithms and dir(hashlib) is "incomplete" in this regards and lists only algorithms which are guaranteed to be supported. Yet more are actually implemented. The patch moves the responsibility for checking whether RIPEMD160 (and others) is supported to hashlib, retaining the NotSupportedError. -- Mark Kubacki [1] https://raw.github.com/wmark/ossdl-overlay/master/dev-python/python-monetdb…

2 1

[MonetDB-users] Max Connection Limit
by Joseph Brower 06 Jan '12

06 Jan '12

Right now we are hitting the limit of 64 connections to MonetDB. We're working on some middleware that will consolidate those into only 2 or 3 connections, but until we finish it, we need to increase the limit. How do we go about doing that? Thanks, Joseph Brower

3 7

[MonetDB-users] Error in SELECT from a function via Mapi
by Iain Buchanan 02 Jan '12

02 Jan '12

Hi, I'm having problems calling a function via the Mapi interface (using Visual C++ under Windows and the 15th November download). It works fine when I call it directly, but fails when I call via Mapi. I create the function: create function tick.dummy(a integer) returns table (entry integer) begin return (select 1); end; I can then call this directly from mclient without any problems: sql>select entry from tick.dummy(1); +-------+ | entry | +=======+ | 1 | +-------+ 1 tuple (1.074ms) sql> However, if I call via Mapi I receive an error: SELECT: no such unary operator 'dummy(tinyint)' I'm using the following sequence of API calls: MapiHdl handle = mapi_prepare(connection, "SELECT entry FROM tick.dummy(?)"); int param(0); mapi_param_type(handle, 0, MAPI_INT, MAPI_AUTO, &param); if (mapi_execute(handle) != MOK) { // Error here } The above seems to work fine for a general SELECT statement, but fails for one with a function. Is there a different syntax for calling via Mapi? Iain

1 0

[MonetDB-users] ENUM style table with JOIN versus automatically deduplicated VARCHAR
by Stefan de Konink 02 Jan '12

02 Jan '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, MonetDB 'deduplicates' strings, is there any benchmark that outlines the performance difference of an ENUM with a low number of (string)items versus the use of a VARCHAR field? The technical question would be; Is it more efficient to do: select * from my_table, my_enum where mytable.my_enum_id = my_enum.id and my_enum.name = 'Something'; or select * from my_table where my_enum_varchar = 'something'; I can imagine that the following would out perform a string comparison, but implies hardcoded values. select * from my_table, my_enum where mytable.my_enum_id = hardcoded; Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk78fnkACgkQYH1+F2Rqwn3BUQCeOUtPIGzcHT1xpnsF4n78BQ/1 JWAAnjnzPeT8D7CtsjYgAiGg/WRIgNbQ =989f -----END PGP SIGNATURE-----

2 1

[MonetDB-users] MonetDB internals, does UPDATE update in place?
by Stefan de Konink 29 Dec '11

29 Dec '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, Just to be very sure; I know that the DELETE, INSERT strategy to update the database will guarantee a database that keeps on growing, because of the lack of VACUUM. Is the normal UPDATE statement for fixed width datatypes in place, or is it also a form of copy-on-write? Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAk78fPwACgkQYH1+F2Rqwn2fxACfWdXvwr4pWODNo4ZH9uq5oVS9 BmwAn16XGh8h/vYsGkvthKej3MA8Nyvi =KK5T -----END PGP SIGNATURE-----

1 0

[MonetDB-users] Maximum size for SQL command in mclient
by Charalampos Nikolaou 28 Dec '11

28 Dec '11

Hi, is there a maximum size for an SQL command (e.g., INSERT INTO statement) that is imposed in mclient? The connection is terminated when I execute several commands from a sql file one of which is quite long. The point where I get the error is at the 8192th byte (which is the last of 8KBs). Babis

2 2

[MonetDB-users] protocol violation: unexpected line in data block
by tangray 27 Dec '11

27 Dec '11

the whole log file : RD 1298855541531: read final block: 57 bytes RX 1298855541531: onxRWd3Yv:mserver:9:RIPEMD160,SHA256,SHA1,MD5:LIT:SHA512: RD 1298855541531: inserting prompt TD 1298855541531: write final block: 92 bytes TX 1298855541531: BIG:risk:{SHA256}01fa94fefcadb5918473e49aaa8e425538f8d40960b2d649348cb17c599031ae:sql:krisk: RD 1298855541546: read final block: 0 bytes RX 1298855541546: RD 1298855541546: inserting prompt TD 1298855541546: write final block: 48 bytes TX 1298855541546: sSET TIME ZONE INTERVAL '+08:00' HOUR TO MINUTE; RD 1298855541546: read final block: 0 bytes RX 1298855541546: RD 1298855541546: inserting prompt TD 1298855541546: write final block: 15 bytes TX 1298855541546: Xreply_size 250 RD 1298855541546: read final block: 0 bytes RX 1298855541546: RD 1298855541546: inserting prompt TD 1298855541546: write final block: 46 bytes TX 1298855541546: sSELECT "name", "value" FROM sys.env() as env; RD 1298855541546: read final block: 4246 bytes RX 1298855541546: &1 0 46 2 46 % .env, .env # table_name % name, value # name % varchar, varchar # type % 18, 203 # length [ "gdk_arch", "32biti686-pc-win32" ] [ "gdk_version", "1.40.3" ] [ "prefix", "D:\\Program Files\\MonetDB\\MonetDB5" ] [ "exec_prefix", "D:\\Program Files\\MonetDB\\MonetDB5" ] [ "gdk_dbname", "krisk" ] [ "gdk_dbfarm", "C:\\Documents and Settings\\swordtan\\Application Data\\MonetDB5\\dbfarm" ] [ "gdk_debug", "0" ] [ "gdk_alloc_map", "no" ] [ "gdk_vmtrim", "yes" ] [ "monet_admin", "adm" ] [ "monet_prompt", ">" ] [ "monet_welcome", "yes" ] [ "monet_mod_path", "D:\\Program Files\\MonetDB\\MonetDB5\\lib\\MonetDB5;D:\\Program Files\\MonetDB\\MonetDB5\\lib\\MonetDB5\\lib;D:\\Program Files\\MonetDB\\MonetDB5\\lib\\MonetDB5\\bin" ] [ "monet_daemon", "no" ] [ "host", "localhost" ] [ "mapi_port", "50000" ] [ "mapi_clients", "2" ] [ "mapi_open", "true" ] [ "mapi_autosense", "false" ] [ "default_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,mitosis,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,dataflow,history,multiplex,garbageCollector" ] [ "minimal_pipe", "inline,remap,deadcode,multiplex,garbageCollector" ] [ "sql_optimizer", "default_pipe" ] [ "sql_debug", "0" ] [ "standoff_ns", "" ] [ "standoff_start", "start" ] [ "standoff_end", "end" ] [ "config", "D:\\Program Files\\MonetDB\\MonetDB5\\etc\\monetdb5.conf" ] [ "mero_pidfile", "D:\\Program Files\\MonetDB\\MonetDB5\\var\\run\\MonetDB\\merovingian.pid" ] [ "mero_controlport", "50001" ] [ "nov2009_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,multiplex,garbageCollector" ] [ "replication_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,replication,multiplex,garbageCollector" ] [ "accumulator_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,accumulators,dataflow,history,multiplex,garbageCollector" ] [ "recycler_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,deadcode,constants,commonTerms,joinPath,deadcode,recycle,reduce,dataflow,history,multiplex,garbageCollector" ] [ "cracker_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,selcrack,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,multiplex,garbageCollector" ] [ "sidcrack_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,sidcrack,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,multiplex,garbageCollector" ] [ "datacell_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,deadcode,constants,commonTerms,joinPath,datacell,deadcode,reduce,dataflow,history,multiplex,garbageCollector" ] [ "octopus_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,mitosis,mergetable,deadcode,constants,commonTerms,joinPath,octopus,deadcode,reduce,dataflow,history,multiplex,garbageCollector" ] [ "mapreduce_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,mapreduce,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,dataflow,history,multiplex,garbageCollector" ] [ "datacyclotron_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,datacyclotron,mergetable,deadcode,constants,commonTerms,joinPath,reorder,deadcode,reduce,dataflow,history,replication,multiplex,garbageCollector" ] [ "derive_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,derivePath,joinPath,deadcode,reduce,dataflow,history,multiplex,garbageCollector" ] [ "dictionary_pipe", "inline,remap,dictionary,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,multiplex,garbageCollector" ] [ "compression_pipe", "inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,compression,dataflow,history,multiplex,garbageCollector" ] [ "gdk_mem_pagebits", "16" ] [ "gdk_mem_bigsize", "1048576" ] [ "monet_pid", "984" ] [ "monet_version", "5.22.3" ] RD 1298855541546: inserting prompt TD 1298855541562: write final block: 60 bytes TX 1298855541562: sselect * from risk.td_ind_security_held where id = 1560000; RD 1298855541578: read final block: 1600 bytes RX 1298855541578: &1 1 1 30 1 % risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held # table_name % id, bizdate, painnercode, currency, exchange, seccode, basickind, unrealizedgainchange, unrealizedgain, margingain, interestgain, othergain, yieldrate, accrualinterest, allgain, fullmv, marketvalue, lastfullmv, lastmarketvalue, lastcost, cost, buysum, sellsum, realizedgain, holdingqty, lastholdingqty, mvweight, lastmvweight, fullmvweight, lastfullmvweight # name % int, varchar, varchar, varchar, varchar, varchar, varchar, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal # type % 7, 8, 8, 3, 5, 6, 3, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 14, 14, 14, 14 # length RD 1298855541578: inserting prompt TD 1298855762109: write final block: 60 bytes TX 1298855762109: sselect * from risk.td_ind_security_held where id = 1560000; RD 1298855762125: read final block: 1600 bytes RX 1298855762125: &1 2 1 30 1 % risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held # table_name % id, bizdate, painnercode, currency, exchange, seccode, basickind, unrealizedgainchange, unrealizedgain, margingain, interestgain, othergain, yieldrate, accrualinterest, allgain, fullmv, marketvalue, lastfullmv, lastmarketvalue, lastcost, cost, buysum, sellsum, realizedgain, holdingqty, lastholdingqty, mvweight, lastmvweight, fullmvweight, lastfullmvweight # name % int, varchar, varchar, varchar, varchar, varchar, varchar, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal, decimal # type % 7, 8, 8, 3, 5, 6, 3, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 14, 14, 14, 14 # length RD 1298855762125: inserting prompt TD 1298856279453: write final block: 69 bytes TX 1298856279453: sselect id,bizdate from risk.td_ind_security_held where id = 1560000; RD 1298856279453: read final block: 162 bytes RX 1298856279453: &1 3 1 2 1 % risk.td_ind_security_held, risk.td_ind_security_held # table_name % id, bizdate # name % int, varchar # type % 7, 8 # length [ 1560000, "20090702" ] RD 1298856279453: inserting prompt TD 1298856279468: write final block: 1797 bytes TX 1298856279468: sSELECT 'krisk' AS "TABLE_CAT", "schemas"."name" AS "TABLE_SCHEM", "tables"."name" AS "TABLE_NAME", "columns"."name" AS "COLUMN_NAME", cast(CASE "columns"."type" WHEN 'wrd' THEN -5 WHEN 'sec_interval' THEN -5 WHEN 'bigint' THEN -5 WHEN 'timetz' THEN 92 WHEN 'char' THEN 1 WHEN 'int' THEN 4 WHEN 'clob' THEN 2005 WHEN 'month_interval' THEN 4 WHEN 'tinyint' THEN -6 WHEN 'table' THEN 2003 WHEN 'date' THEN 91 WHEN 'double' THEN 8 WHEN 'decimal' THEN 3 WHEN 'timestamptz' THEN 93 WHEN 'timestamp' THEN 93 WHEN 'blob' THEN 2004 WHEN 'real' THEN 7 WHEN 'time' THEN 92 WHEN 'oid' THEN 1111 WHEN 'boolean' THEN 16 WHEN 'smallint' THEN 5 WHEN 'varchar' THEN 12 ELSE 1111 END AS smallint) AS "DATA_TYPE", "columns"."type" AS "TYPE_NAME", "columns"."type_digits" AS "COLUMN_SIZE", "columns"."type_scale" AS "DECIMAL_DIGITS", 0 AS "BUFFER_LENGTH", 10 AS "NUM_PREC_RADIX", cast(CASE "null" WHEN true THEN 1 WHEN false THEN 0 END AS int) AS "NULLABLE", cast(null AS varchar(1)) AS "REMARKS", "columns"."default" AS "COLUMN_DEF", 0 AS "SQL_DATA_TYPE", 0 AS "SQL_DATETIME_SUB", 0 AS "CHAR_OCTET_LENGTH", "columns"."number" + 1 AS "ORDINAL_POSITION", cast(null AS varchar(1)) AS "SCOPE_CATALOG", cast(null AS varchar(1)) AS "SCOPE_SCHEMA", cast(null AS varchar(1)) AS "SCOPE_TABLE", cast(1111 AS smallint) AS "SOURCE_DATA_TYPE", CASE "null" WHEN true THEN CAST ('YES' AS varchar(3)) WHEN false THEN CAST ('NO' AS varchar(3)) END AS "IS_NULLABLE" FROM "sys"."columns" AS "columns", "sys"."tables" AS "tables", "sys"."schemas" AS "schemas" WHERE "columns"."table_id" = "tables"."id" AND "tables"."schema_id" = "schemas"."id" AND LOWER("schemas"."name") LIKE 'risk' AND LOWER("tables"."name") LIKE 'td_ind_security_held' AND LOWER("columns"."name") LIKE 'id' ORDER BY "TABLE_SCHEM", "TABLE_NAME", "ORDINAL_POSITION"; RD 1298856279484: read final block: 853 bytes RX 1298856279484: &1 4 1 22 1 % ., sys.schemas, .tables, .columns, ., .columns, .columns, .columns, ., ., ., ., .columns, ., ., ., ., ., ., ., ., . # table_name % TABLE_CAT, TABLE_SCHEM, TABLE_NAME, COLUMN_NAME, DATA_TYPE, TYPE_NAME, COLUMN_SIZE, DECIMAL_DIGITS, BUFFER_LENGTH, NUM_PREC_RADIX, NULLABLE, REMARKS, COLUMN_DEF, SQL_DATA_TYPE, SQL_DATETIME_SUB, CHAR_OCTET_LENGTH, ORDINAL_POSITION, SCOPE_CATALOG, SCOPE_SCHEMA, SCOPE_TABLE, SOURCE_DATA_TYPE, IS_NULLABLE # name % char, varchar, varchar, varchar, smallint, varchar, int, int, tinyint, tinyint, int, varchar, varchar, tinyint, tinyint, tinyint, int, varchar, varchar, varchar, smallint, varchar # type % 5, 4, 20, 2, 1, 3, 2, 1, 1, 2, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 4, 2 # length [ "krisk", "risk", "td_ind_security_held", "id", 4, "int", 32, 0, 0, 10, 0, NULL, NULL, 0, 0, 0, 1, NULL, NULL, NULL, 1111, "NO" ] RD 1298856279484: inserting prompt TD 1298856279500: write final block: 1802 bytes TX 1298856279500: sSELECT 'krisk' AS "TABLE_CAT", "schemas"."name" AS "TABLE_SCHEM", "tables"."name" AS "TABLE_NAME", "columns"."name" AS "COLUMN_NAME", cast(CASE "columns"."type" WHEN 'wrd' THEN -5 WHEN 'sec_interval' THEN -5 WHEN 'bigint' THEN -5 WHEN 'timetz' THEN 92 WHEN 'char' THEN 1 WHEN 'int' THEN 4 WHEN 'clob' THEN 2005 WHEN 'month_interval' THEN 4 WHEN 'tinyint' THEN -6 WHEN 'table' THEN 2003 WHEN 'date' THEN 91 WHEN 'double' THEN 8 WHEN 'decimal' THEN 3 WHEN 'timestamptz' THEN 93 WHEN 'timestamp' THEN 93 WHEN 'blob' THEN 2004 WHEN 'real' THEN 7 WHEN 'time' THEN 92 WHEN 'oid' THEN 1111 WHEN 'boolean' THEN 16 WHEN 'smallint' THEN 5 WHEN 'varchar' THEN 12 ELSE 1111 END AS smallint) AS "DATA_TYPE", "columns"."type" AS "TYPE_NAME", "columns"."type_digits" AS "COLUMN_SIZE", "columns"."type_scale" AS "DECIMAL_DIGITS", 0 AS "BUFFER_LENGTH", 10 AS "NUM_PREC_RADIX", cast(CASE "null" WHEN true THEN 1 WHEN false THEN 0 END AS int) AS "NULLABLE", cast(null AS varchar(1)) AS "REMARKS", "columns"."default" AS "COLUMN_DEF", 0 AS "SQL_DATA_TYPE", 0 AS "SQL_DATETIME_SUB", 0 AS "CHAR_OCTET_LENGTH", "columns"."number" + 1 AS "ORDINAL_POSITION", cast(null AS varchar(1)) AS "SCOPE_CATALOG", cast(null AS varchar(1)) AS "SCOPE_SCHEMA", cast(null AS varchar(1)) AS "SCOPE_TABLE", cast(1111 AS smallint) AS "SOURCE_DATA_TYPE", CASE "null" WHEN true THEN CAST ('YES' AS varchar(3)) WHEN false THEN CAST ('NO' AS varchar(3)) END AS "IS_NULLABLE" FROM "sys"."columns" AS "columns", "sys"."tables" AS "tables", "sys"."schemas" AS "schemas" WHERE "columns"."table_id" = "tables"."id" AND "tables"."schema_id" = "schemas"."id" AND LOWER("schemas"."name") LIKE 'risk' AND LOWER("tables"."name") LIKE 'td_ind_security_held' AND LOWER("columns"."name") LIKE 'bizdate' ORDER BY "TABLE_SCHEM", "TABLE_NAME", "ORDINAL_POSITION"; RD 1298856279515: read final block: 862 bytes RX 1298856279515: &1 5 1 22 1 % ., sys.schemas, .tables, .columns, ., .columns, .columns, .columns, ., ., ., ., .columns, ., ., ., ., ., ., ., ., . # table_name % TABLE_CAT, TABLE_SCHEM, TABLE_NAME, COLUMN_NAME, DATA_TYPE, TYPE_NAME, COLUMN_SIZE, DECIMAL_DIGITS, BUFFER_LENGTH, NUM_PREC_RADIX, NULLABLE, REMARKS, COLUMN_DEF, SQL_DATA_TYPE, SQL_DATETIME_SUB, CHAR_OCTET_LENGTH, ORDINAL_POSITION, SCOPE_CATALOG, SCOPE_SCHEMA, SCOPE_TABLE, SOURCE_DATA_TYPE, IS_NULLABLE # name % char, varchar, varchar, varchar, smallint, varchar, int, int, tinyint, tinyint, int, varchar, varchar, tinyint, tinyint, tinyint, int, varchar, varchar, varchar, smallint, varchar # type % 5, 4, 20, 7, 2, 7, 1, 1, 1, 2, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 4, 2 # length [ "krisk", "risk", "td_ind_security_held", "bizdate", 12, "varchar", 8, 0, 0, 10, 0, NULL, NULL, 0, 0, 0, 2, NULL, NULL, NULL, 1111, "NO" ] RD 1298856279515: inserting prompt TD 1298856508906: write final block: 77 bytes TX 1298856508906: sselect id,bizdate,allgain from risk.td_ind_security_held where id = 1560000; RD 1298856508906: read final block: 187 bytes RX 1298856508906: &1 6 1 3 1 % risk.td_ind_security_held, risk.td_ind_security_held, risk.td_ind_security_held # table_name % id, bizdate, allgain # name % int, varchar, decimal # type % 7, 8, 20 # length RD 1298856508906: inserting prompt

2 1

[MonetDB-users] java.lang.AssertionError: block 0 should have been fetched by now
by Viktor Rosenfeld 27 Dec '11

27 Dec '11

Hi, I get an error message when I retrieve a MAL plan over JDBC after the 250th call to ResultSet.next(). The relevant part of the stack trace is: java.lang.AssertionError: block 0 should have been fetched by now :( at nl.cwi.monetdb.jdbc.MonetConnection$ResultSetResponse.getLine(MonetConnection.java:1585) at nl.cwi.monetdb.jdbc.MonetResultSet.absolute(MonetResultSet.java:194) at nl.cwi.monetdb.jdbc.MonetResultSet.relative(MonetResultSet.java:2149) at nl.cwi.monetdb.jdbc.MonetResultSet.next(MonetResultSet.java:2116) The call to EXPLAIN works in mclient and returns a MAL plan that is 402 lines long. I can reproduce the error when I try to create the plan via jdbcclient, except that it says that block 1 should have been fetched and the exporter stops after line 160 or so. However, other SQL queries that also produce more than 250 results work over JDBC. Not sure what the difference is, but these queries only return column of integers. I'm using the Aug2011-SP3 release on OS X 10.6.8. Please let me know if I should provide other information like the SQL query or the MAL plan. Cheers, Viktor

2 3

[MonetDB-users] time series data scaling
by David Pennell 20 Dec '11

20 Dec '11

In a large scale time series row-store DB (Oracle), I'm accustomed to partitioning fact tables on time. This has has a couple of benefits: - since queries always include predicates on the time field, partition elimination can help a lot - when the time based partitions map one-to-one with tablespaces, removing old data is extremely fast. (when you've reached the maximum data retention period, you need to remove data as fast as you can load it). Are there any features in monetdb that would facilitate these needs (or ameliorate them)? TIA, -david

2 2

[MonetDB-users] Benefits of sorted columns
by Viktor Rosenfeld 19 Dec '11

19 Dec '11

Hi, I know that when a table is sorted on a column, value look-ups on that column will use a binary search instead of a linear search. Is there a similar benefit for joins? E.g. if I have 2 tables that contain a sorted column C and I do an equality join on this column (or a self-join using 2 aliases of the same table), I would expect a traditional DBMS to do a merge-join. Cheers, Viktor

2 2

[MonetDB-users] SQLException:SQLengine:Program contains errors
by Charalampos Nikolaou 17 Dec '11

17 Dec '11

Hi, I am getting the error "SQLException:SQLengine:Program contains errors" when I define a simple function that returns a table. The SQL reference on the site of MonetDB (http://www.monetdb.org/Documentation/SQLreference/Flowofcontrol) states that the accepted expression to a "TABLE" expression should be "select_no_parens_orderby" which is not defined in any place (or haven't found it) other than sql_parser.y. The function I define is the following and apart from the fact that is inserted successfully into the "functions" table, I checked that it can be produced by the grammar in sql_parser.y: CREATE FUNCTION refine () RETURNS TABLE (u varchar(32), v varchar(32), r varchar(5)) BEGIN RETURN TABLE ( ( SELECT R.u, R.v, R.r FROM R INTERSECT SELECT R.u, R.v, R.r FROM R ) ); END; Further, what is the proper way to deleting a function, in order to re-define it in the same session? I observed that is not sufficient to just delete it from sys.functions table. An entry #function_name is also present in sys._tables. Thanks, Babis

3 5

Re: [MonetDB-users] Information request
by Ying Zhang 16 Dec '11

16 Dec '11

On Fri, Dec 16, 2011 at 10:51:18PM +0800, dj zhang wrote: > Hello Jennie, > > Glad to hear from you! Thanks for your reply! I'm sorry that I didn't > realize that you guys are in a new year vacation. Hello Dongjie, I'd like to recommend you to subscribe to the MonetDB users mailling list: http://www.monetdb.org/Developers/Mailinglists , and sent your feature questions to the mailinglist. This way, your questions will reach the whole developer team at one, and you'll most probably get answers faster. I felt free to forward this e-mail to the mailing list. > Maybe I didn't express myself very clear. I've generated a table file in > which each line represents a record, and the fields of a record are > separated by "|", e.g. "abc|1.2.3.3|www.monetdb.org|3". I can bulk load > this file to MonetDB by SQL COPY INTO command. Unfortunately, you can't load this table using the binary bulk load feature of MonetDB, since this feature only handles numerical data, and doesn't not work with strings. How large (in GBs) is your total dataset? Maybe you don't have to use the binary bulk load feature... > Now I want to change the table file to column files, each file just > contains one column's values which also stored line by line. I want to know > if I can load this table by loading these column files. > > From your information, Can I say that I have to change these column files > to binary format that MonetDB recognise(BAT storage), then use binary bulk > load command to load the table to MonetDB. And if I change the files to > other binary format it will not work. Is that right? That's right. MonetDB just works with one binary format. Regards, Jennie > > Thank you so much! Once again happy new year! > > Best wishes, > > Dongjie > ------------------------------------- > > On Fri, Dec 16, 2011 at 8:55 PM, Ying Zhang <Y.Zhang(a)cwi.nl> wrote: > > > Hello Zhang Dongjie, > > > > First of all, thank you very much for your interest in MonetDB. > > > > > I have learn from the website document that I can load data use column > > > files when migrate between MonetDB instances, but the file has to be in > > the > > > format of binary version of the BAT storage if I didn't misunderstand. > > > > If each column of a table is already stored in binary format in a file, > > the table can be loaded using the "binary bulk load" feature of MonetDB, > > as described here: > > http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/BinaryBulkLoad > > > > If you are migrating your existing database to an exactly the same > > machine, you can just copy dbfarm sub-directory for your database. If > > you are migrating to a machine with different hardware, you need to dump > > the database and reload it on the destination machine, as described here: > > http://www.monetdb.org/Documentation/UserGuide/DumpRestore > > However, in this case, the dumped data can not be loaded using the > > binary bulk load feature, because the file containing dumpted data is an > > ASCII file. > > > > > So my question is, is there any way to load data to monetdb by column as > > > long as the record is store by column. Or do I have to first change the > > > column files to the binary version of BAT format then use the Binary Bulk > > > Load commands. > > > > You can only use the binary bulk load feature, iff the data of each > > column is stored in a separate file in binary format. Hence, yes, > > you'll have to generate those binary files, if you don't have them yet. > > However, if your data is already in, e.g., CSV format, you don't have to > > create binary files for the columns, since you can just load the CSV > > file. If your (CSV) data is not too large, the standard SQL COPY INTO > > feature should just work. > > > > > And if I have to change the format, is there any way to use google > > protobuf > > > to serialize the column file to the BAT format, or is there any solutions > > > like handlersocket for mysql? > > > > I'm unfortunately not familiar with protobuf and handlersocket. Maybe > > someone else in our group can answer these questions. > > > > Hope this helps a bit. > > > > > > With kind regads, > > > > Jennie Zhang > > > > > > > > > > By the way I think MonetDB is a great project and I wish it can find it's > > > way to commercialize very soon! > > > > > > Thank you very much, and Merry Christmas! > > > > > > Best regards, > > > > > > Dongjie > > > ------------------ > > > Dongjie Zhang > > > > > > Services Computing Technology and System Lab, > > > Cluster and Grid Computing Lab, > > > School of Computer Science and Technology, > > > Huazhong University of Science and Technology, > > > Wuhan, 430074, China > > > > > > 华中科技大学计算机科学与技术学院 > > > 服务计算技术与系统教育部重点实验室集群与网格计算湖北省重点实验室 > > > 湖北武汉 430074 > > > > > _______________________________________________ > > > Info mailing list > > > Info(a)monetdb.org > > > http://mail.monetdb.org/mailman/listinfo/info > > > > > > > -- > ------------------ > Dongjie Zhang > > Services Computing Technology and System Lab, > Cluster and Grid Computing Lab, > School of Computer Science and Technology, > Huazhong University of Science and Technology, > Wuhan, 430074, China > > 华中科技大学计算机科学与技术学院 > 服务计算技术与系统教育部重点实验室集群与网格计算湖北省重点实验室 > 湖北武汉 430074

1 0

[MonetDB-users] Invalid BAT if left join condition has only columns from right table
by George Garbis 15 Dec '11

15 Dec '11

Hello to all, a left join which have only right table columns in join condition seems to produce invalid bat. Assume we have the tables below: CREATE TABLE tableA(x integer, y integer); INSERT INTO tableA values(1,10); INSERT INTO tableA values(2,20); CREATE TABLE tableB(x integer, y integer); INSERT INTO tableB values(1,-10); INSERT INTO tableB values(2,-20); and pose the query: SELECT A.x, A.y, B.x, B.y FROM tableA A LEFT JOIN tableB B ON ( B.y < -10 ) ; the expected results are: +------+----- +------+------+ | x | y | x | y | +===+===+===+===+ | 1 | 10 | 2 | -20 | | 2 | 20 | 2 | -20 | +------+------+------+------+ but mclient prints this message: TypeException:user.s1_1[36]:'algebra.kdifference' undefined in: _55:any := algebra.kdifference(_54:bat[:oid,:int], _44:bat[:void,:oid]) TypeException:user.s1_1[37]:'algebra.markT' undefined in: _56:any := algebra.markT(_55:any, _21:oid) TypeException:user.s1_1[38]:'bat.reverse' undefined in: _57:any := bat.reverse(_56:any) TypeException:user.s1_1[39]:'algebra.join' undefined in: _58:any := algebra.join(_57:any, _20:bat[:oid,:int]) TypeException:user.s1_1[41]:'algebra.leftjoin' undefined in: _60:any := algebra.leftjoin(_59:bat[:oid,:void], _54:bat[:oid,:int]) TypeException:user.s1_1[43]:'bat.append' undefined in: _64:any := bat.append(_61:bat[:oid,:int], _60:any, _65:bit) TypeException:user.s1_1[44]:'bat.append' undefined in: _66:any := bat.append(_64:any, _58:any, _65:bit) TypeException:user.s1_1[55]:'algebra.join' undefined in: _80:any := algebra.join(_57:any, _79:bat[:oid,:int]) TypeException:user.s1_1[61]:'algebra.leftjoin' undefined in: _90:any := algebra.leftjoin(_59:bat[:oid,:void], _89:bat[:oid,:int]) TypeException:user.s1_1[63]:'bat.append' undefined in: _92:any := bat.append(_91:bat[:oid,:int], _90:any, _65:bit) TypeException:user.s1_1[64]:'bat.append' undefined in: _93:any := bat.append(_92:any, _80:any, _65:bit) TypeException:user.s1_1[70]:'algebra.project' undefined in: _102:any := algebra.project(_57:any, _100:int) TypeException:user.s1_1[83]:'bat.append' undefined in: _119:any := bat.append(_117:bat[:oid,:int], _102:any, _118:bit) TypeException:user.s1_1[89]:'algebra.project' undefined in: _126:any := algebra.project(_57:any, _125:int) TypeException:user.s1_1[96]:'bat.append' undefined in: _133:any := bat.append(_132:bat[:oid,:int], _126:any, _118:bit) SQLException:SQLengine:Program contains errors For ease I attach sql scripts for creating tables and posing the query Best regards, George Garbis

2 1

[MonetDB-users] ODBC Stops inserting
by Joseph Brower 13 Dec '11

13 Dec '11

Hello everyone! We're using the odbc driver to insert records every half second or so. These records are for analytics information about visitors to various websites that we host. (We found MonetDB to be very fast for fetching information like average request time, number of visitors, general analytics information like that.) We've run into a few issues though. The biggest concern is that we stop seeing new records entered after a while. The log shows 2011-12-13 17:00:00 MSG merovingian[86]: proxying client 10.60.5.13:37359 for database 'web_requests' to mapi:monetdb:///var/lib/monetdb/web_requests/web_requests/.mapi.sock?database=web_requests 2011-12-13 17:00:00 MSG merovingian[86]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying over and over for each machine that would be inserting these records. But no new records are ever added. Another concern is that in our installation, we don't see a MonetDB.conf file. I was hoping that we could specify how much RAM we'd like to allow MonetDB to consume. This is to ensure that our memory manager doesn't kill MonetDB, since that's the primary role of this server. Is this a file that we should create? I'm still fairly new to MonetDB and as a result, am still trying to find my way with it. The last concern I have is the log file size. Is there a way (once we've ironed out the before mentioned issues) to instruct MonetDB to only log things of a certain importance or higher? For example, to only log ERR rather than ERR and MSG messages? Our log files will become quite large and unmanageable if we don't disable or implement some sort of log rotation system. The version of MonetDB that we are running is MonetDB 5 server v11.5.7 "Aug2011-SP2". We are running under Ubuntu 10.04 64Bit. Thanks, Joseph Brower

2 3

[MonetDB-users] SpagoBI
by Jeremy OSullivan 09 Dec '11

09 Dec '11

Hello,We are trying to get SpagoBI to work with MonetDB. SpagoBI uses a JDBC interface. The problem is as follows: The import of the schema works fine. However, when you save the model in SpagoBI, the schema in MonetDB can no longer be viewed. We have tried this with MySQL and it works fine.Regards,Jeremy

1 0

[MonetDB-users] Can't create a csv vault?
by Mark Aufflick 08 Dec '11

08 Dec '11

Hi all, I'm very interested in the vault feature of monetdb, both for csv and for creating my own formats. Trouble is I can't make a simple example work. I've flattended the voc sample data into a csv file, and then try to load it so: create table test (type varchar(255), number integer, number_sup varchar(255), trip integer, trip_sup varchar(255), onboard_at_departure integer, death_at_cap integer, left_at_cape integer, onboard_at_cape integer, death_during_voyage integer, onboard_at_arrival integer) external csv '/tmp/all.csv'; But I get an error: syntax error, unexpected EXTERNAL, expecting SCOLON in: ... Do I need to enable vaults at configure time or create time or something? I didn't see an option for it. I built from the "Latest" tarball on OSX Lion. Cheers, Mark.

2 3

[MonetDB-users] stethoscope on OS X can't connect to database
by Viktor Rosenfeld 07 Dec '11

07 Dec '11

Hi, when I try to connect to a database with stethoscope I get an error: $ stethoscope -d annis_test -u annis_test -P ... +tis stethoscope: no databases found for 'annis_test' I'm using the Aug2011-SP3 release on OS X 10.6.8. The same command works under Linux. The instances are set up identically. Cheers, Viktor

2 2

[MonetDB-users] JDBC documentation inconsistencies
by Viktor Rosenfeld 29 Nov '11

29 Nov '11

Hi, the example at the top of the documentation of the JDBC client at http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/Clients/JDBC uses the -d switch to specify the table and the -b switch to specify the database. However the current version does not recognize the -b switch and the -d switch can be used to select the database as is documented further on the page. Also, the page says that TRACE is not possible via JDBC. However, as the jdbcclient.jar demonstrates it is indeed possible to use TRACE on a query by checking additional result sets. Cheers, Viktor

2 4

[MonetDB-users] Announcement: New Aug2011-SP3 Bugfix release of MonetDB suite
by Fabian Groffen 22 Nov '11

22 Nov '11

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2011-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at http://www.monetdb.org/. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Note that this is the last release of the Aug2011 branch. From now on, no further development will take place on this branch. Aug 2011-SP3 bugfix release SQL * Added a fix for bug #2834, which caused weird (failing) behaviour with PreparedStatements. Merovingian * Fixed a bug where monetdbd's socket files from /tmp were removed when a second monetdbd was attempted to be started using the same port. Bug Fixes * 2875: incomplete database initialization * 2912: Function body disappears after loading data - error message: SQLException:sql.resultSet:Cannot access descriptor * 2919: Add column and select in same transaction * 2920: Updated database from Aug2011 to Dec2011 make queries fail with "Subquery result missing". * 2921: complex where clause broken * 2923: Multiplication and cast from decimal to integer leads to 0 value * 2924: use LIKE with ESCAPE ignores OR operator * 2928: Cannot update a field with a count query based on an other table

1 0

[MonetDB-users] Workflow for independent, read only, discretely updated tables.
by Hedge Hog 20 Nov '11

20 Nov '11

Hi, I'm evaluating MonetDB, and the following is based on reading the docs and some of the monetdb-users email list archive. I wonder if the following setup+workflow is correct, and if so, is it monetdb best practice, given this use case: We will have read only tables, each distributed across several machines/servers, they will be updated daily. There will be no cross table queries, i.e. only one table touched by each quert. We would like to be able to update each table while not affecting the availability of any other table. The table_db's are not horizontally sharded, i.e. all the data for a query will always come from that one table. We cannot use udp multicast/broadcast so a monetdb cluster is not possible (unless a localhost cluster is possible/sensible?). The setup+workflow: - one table per database (this allows for independent table updates), let these be <table_db>. - update a master (writable) instance of the table on a 'special' monetdbd/machine. - copy the updated table to each machine. - To update the table on a machine: $> monetbd lock <mytable_db> $> monetbd stop <mytable_db> $> mclient -u monetdb <mytable_db> updatefile $> cat updatefile copy into MyTable from ('path_to_mytable_col_file_i', 'path_to_mytable_col_file_f', 'path_to_mytable_col_file_s'); $> monetbd release <mytable_db> $> monetbd start <mytable_db> Is the above the best pattern/architecture of monetdb for such a use case? Appreciate any insights people can offer TIA Hedge -- πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα [The fox knows many things, but the hedgehog knows one big thing.] Archilochus, Greek poet (c. 680 BC – c. 645 BC) http://hedgehogshiatus.com

3 4

[MonetDB-users] Unable to use Tableau
by Sékine Coulibaly 18 Nov '11

18 Nov '11

Hi there, I'm a new user to both Tableau and MonetDb. I run under Windows, and my MonetDb base is up and running, a table "matable" was created and populated. Although I can connect the MonetDb base to Tableau (although I get some warnings), as soon as Tableau sends a request to MonetDb, the ODBC driver returns the following : « Database error (0x80004005) : [MonetDB][ODBC Driver 11.5.7]Invalid identifier 'none:name:nk' » As can be seen here : http://puu.sh/8STw Has any of you already used Tableau and solved this issue ? Shall I consider using Pentaho ? Thank you ! Regards

2 1

[MonetDB-users] pcre not found when configure monetdb5
by xldelg＠sohu.com 18 Nov '11

18 Nov '11

hi, I tried to compile the latest monetdb(11.5.3) on centos6 x64, but the pcre library is not found when configure. I'm sure that the pcre library has been installed via compiling the source, the libpcre.la, libpcre.a, libpcre.so has been installed in /usr/local/lib. I also configure monetdb like: pcre_LIBS=/usr/local/lib pcre_CFLAGS=/usr/local/include ./configure, but it doesn't work. How to fix this problem? Any reply will be apprecitated.    

2 2

[MonetDB-users] "TypeException: algebra.join undefined" when evaluating EXISTS (SELECT ...) predicate
by Viktor Rosenfeld 16 Nov '11

16 Nov '11

Hi, I get an error message when executing the following query: SELECT count(*) FROM _rank AS _rank1, _rank AS _rank2 WHERE _rank1.pre = 23 AND _rank2.pre = 33 AND EXISTS (SELECT 1 FROM _rank AS ancestor WHERE ancestor.pre < _rank1.pre AND _rank1.pre < ancestor.post AND ancestor.pre < _rank2.pre AND _rank2.pre < ancestor.post); The error message is: TypeException:user.s0_1[99]:'algebra.join' undefined in: _129:any := algebra.join(_128:bat[:oid,:oid], _108:bat[:int,:oid]) SQLException:SQLengine:Program contains errors A minimal example of test data to run this query follows: CREATE TABLE _rank (pre INTEGER, post INTEGER); INSERT INTO _rank VALUES (22, 37); INSERT INTO _rank VALUES (23, 24); INSERT INTO _rank VALUES (33, 34); Cheers, Viktor

2 2

[MonetDB-users] General Questions on Columnar DB Design and Querying
by Sean McNamara 13 Nov '11

13 Nov '11

Hi all, I've managed to get MonetDB running and loaded with a chunk of data (thanks to some great help from the list) and am now trying to figure out the "right" way to make use of it. I don't have a whole lot of exposure to columnar stores, and am therefore not sure if I'm doing things in an idiomatically correct way. I have a couple general questions about columnar DBs, and hope its appropriate to ask them here even thought they aren't specific to MonetDB. If these questions aren't considered appropriate for this list I apologize in advance and will happily move them to a different forum (but would appreciate pointers to where such a forum might be.) Two questions are general, and one is fairly specific. First the general: 1. Does working with a columnar store fundamentally change the way one would structure the tables? I'm somewhat familiar with star-schema designs where a central fact table holds the entire attribute set in as rows of indexes with supporting lookup tables for each column type, but it seems like columnar stores achieve the same thing without actually requiring the developer to specifically build those structures. Tables are constructed in a de-normalized RDBMS manner and the underlying DB takes care of the vectorization. Is this a valid description, or am I fundamentally misunderstanding the basics? 2. If there are significant differences in the way columnar stores should be designed, can folks point me to any resources (web, book, article, etc.) to help educate me on the topic? Now the specific: I have several sets of time-series data modeled as follows: create table sensordata ( sensor varchar(32), epoch_ts bigint, reading double, CONSTRAINT SENSORDATA_PKEY PRIMARY KEY (sensor, epoch_ts) ); I've populated that data, and now want to query data for a subset of sensors and generate a table that shows the value of the sensors at each point in time. I'm not sure what the right way to do this is. My desired out:ut would be: epoch_ts sensor1 sensor2 delta -------------- ------------ ------------ ------ <time1> 23.5 24.1 0.6 <time2> 19.3 25.2 5.9 ... <timeN> 25.1 19.3 -5.8 I tried to build that query as follows but end up with NULL results (note: I'm in no way a SQL pro so it might be really ugly.) select a.epoch_ts as epoch, a.reading as sensor1_reading, b.reading as sensor2_reading, b.reading - a.reading as delta from (select epoch_ts, reading from sensordata where sensor='sensor1') a, (select epoch_ts, reading from sensordata where sensor='sensor2') b where b.epoch_ts = a.epoch_ts order by epoch asc; Is this a sane way to approach the problem, or am I completely off track here. Each subselect returns results, but they obviously aren't joining correctly and it feels like it might simply be the wrong way to try to build the result set. I appreciate any direction you might provide. Thanks!

2 1

[MonetDB-users] MonetDB Permission/COPY INTO Questions
by Sean McNamara 13 Nov '11

13 Nov '11

Hi All, I'm just staring to play with MonetDB, but am running into some trouble with a few basic tasks. I've managed to create a new DB and have added a table. I'd now like to bulk-load the table with a CSV file full of data. When I try to do so, I get the following: WS3:~ tank$ mclient -u testuser testdb Welcome to mclient, the MonetDB/SQL interactive terminal (Aug2011) Database: MonetDB v11.5.1, 'testdb' Type \q to quit, \? for a list of available commands auto commit mode: on sql>COPY 1800 records into testdata from 'testdata.csv'; COPY INTO: insufficient privileges: COPY INTO from file(s) requires administrator rights, use 'COPY INTO "testdata" FROM STDIN' instead I then try to copy into via STDIN: WS3:~ tank$ mclient -u testuser -s "COPY 1800 records into testdata from STDIN using delimiters ',';" testdb < testdata.csv WS3:~ tank$ mclient -u testuser testdb Welcome to mclient, the MonetDB/SQL interactive terminal (Aug2011) Database: MonetDB v11.5.1, 'testdb' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select * testdb; +-----+-----+------+ | foo | bar | baz | +===+==+===+ +-----+-----+------+ 0 tuples (2.307ms) sql> It seems clear I have some sort of permission/privilege issue, but I'm not clear how to resolve it. I've looked at all the tables and see the auths, users, user_role, and other tables, but it's not clear what actually needs to be tweaked there or if I should be using other DB commands to set things up. I've tried signing on as the monetdb user to do the import, but the monetdb user doesn't appear to be able to see the table: WS3:~ tank$ mclient -u monetdb testdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Aug2011) Database: MonetDB v11.5.1, 'testdb' Type \q to quit, \? for a list of available commands auto commit mode: on sql>COPY 1800 records into testdata from 'testdata.csv'; COPY INTO: no such table 'testdata' I realize this should all be quite simple, but am not finding my way forward. I've been searching the archives and hunting on google but am coming up empty. Can anyone point the way? Thanks!

6 11

[MonetDB-users] General questions on columnar DB design and querying
by Sean McNamara 13 Nov '11

13 Nov '11

Hi all, I've managed to get MonetDB running and loaded with a chunk of data (thanks to some great help from the list) and am now trying to figure out the "right" way to make use of it. I don't have a whole lot of exposure to columnar stores, and am therefore not sure if I'm doing things in an idiomatically correct way. I have a couple general questions about columnar DBs, and hope its appropriate to ask them here even thought they aren't specific to MonetDB. If these questions aren't considered appropriate for this list I apologize in advance and will happily move them to a different forum (but would appreciate pointers to where such a forum might be.) Two questions are general, and one is fairly specific. First the general: 1. Does working with a columnar store fundamentally change the way one would structure the tables? I'm somewhat familiar with star-schema designs where a central fact table holds the entire attribute set in as rows of indexes with supporting lookup tables for each column type, but it seems like columnar stores achieve the same thing without actually requiring the developer to specifically build those structures. Tables are constructed in a de-normalized RDBMS manner and the underlying DB takes care of the vectorization. Is this a valid description, or am I fundamentally misunderstanding the basics? 2. If there are significant differences in the way columnar stores should be designed, can folks point me to any resources (web, book, article, etc.) to help educate me on the topic? Now the specific: I have several sets of time-series data modeled as follows: create table sensordata ( sensor varchar(32), epoch_ts bigint, reading double, CONSTRAINT SENSORDATA_PKEY PRIMARY KEY (sensor, epoch_ts) ); I've populated that data, and now want to query data for a subset of sensors and generate a table that shows the value of the sensors at each point in time. I'm not sure what the right way to do this is. My desired out:ut would be: epoch_ts sensor1 sensor2 delta -------------- ------------ ------------ ------ <time1> 23.5 24.1 0.6 <time2> 19.3 25.2 5.9 ... <timeN> 25.1 19.3 -5.8 I tried to build that query as follows but end up with NULL results (note: I'm in no way a SQL pro so it might be really ugly.) select a.epoch_ts as epoch, a.reading as sensor1_reading, b.reading as sensor2_reading, b.reading - a.reading as delta from (select epoch_ts, reading from sensordata where sensor='sensor1') a, (select epoch_ts, reading from sensordata where sensor='sensor2') b where b.epoch_ts = a.epoch_ts order by epoch asc; Is this a sane way to approach the problem, or am I completely off track here. Each subselect returns results, but they obviously aren't joining correctly and it feels like it might simply be the wrong way to try to build the result set. I appreciate any direction you might provide. Thanks!

1 0

Re: [MonetDB-users] MonetDB Permission/COPY INTO Questions
by Allen Zhang 12 Nov '11

12 Nov '11

Sean, Maybe here is why you did not spot the duplicate: cat /tmp/testdata2.csv | cut -d, -f1,5 | sort | uniq -d Regards Allen On 11/12/11 12:39 PM, "Sean McNamara" <tank(a)thursday.org> wrote: >Looks like I had too much faith in the "uniq" command (or >misunderstand its constraints.) There are definitely duplicates in my >import set. :-( > >Back to data scrubbing... > >On Sat, Nov 12, 2011 at 12:08 PM, Allen Zhang <azhang(a)pmci-usa.com> wrote: >> Sean, >> >> Use sql select to find out duplicates. >> >> Regards >> Allen >> >> On 11/12/11 12:01 PM, "Sean McNamara" <tank(a)thursday.org> wrote: >> >>>I tried that (as both the normal dbuser and as admin). Both fail the >>>same way but don't give an indicator of where it feels the constraint >>>is violated: >>> >>>$ mclient -u monetdb -s "alter table mduser.testdata add constraint >>>testdata_pkey primary key (foo,epoch);" testdb >>>password: >>>SQLException:assert:UPDATE: PRIMARY KEY constraint >>>'testdata.testdata_pkey' violated >>> >>> >>>On Sat, Nov 12, 2011 at 11:42 AM, Allen Zhang <azhang(a)pmci-usa.com> >>>wrote: >>>> How about add constraint after you import? I think it might tell you >>>>which >>>> one had problem. >>>> >>>> Regards >>>> Allen >>>> >>>> On 11/12/11 11:19 AM, "Sean McNamara" <tank(a)thursday.org> wrote: >>>> >>>>>Brilliant suggestion, and I think it exposed the problem. >>>>> >>>>>$ mclient -u monetdb -s "copy into mduser.testdata from >>>>>'/tmp/testdata2.csv' using delimiters ',','\n','\'';" testdb >>>>>password: >>>>>SQLException:importTable:value 'ONE_SECOND' from line 12734 field 3 >>>>>not inserted, expecting type varchar(8) >>>>>failed to import table >>>>> >>>>>It looks like the field I created is too small to handle some of the >>>>>values in the file. After expanding all the varchar fields to 32 and >>>>>re-running, I get a different error: >>>>> >>>>>$ mclient -u monetdb -s "copy into mduser.testdata from >>>>>'/tmp/testdata2.csv' using delimiters ',','\n','\'';" testdb >>>>>password: >>>>>SQLException:assert:INSERT INTO: PRIMARY KEY constraint >>>>>'testdata.testdata_pkey' violated >>>>> >>>>>That makes me think I have duplicate data. The PKEY is comprised of >>>>>fields #1 and #5, so I check for that but it looks fine: >>>>> >>>>>$ cat /tmp/testdata2.csv | cut -d, -f1,5 |uniq -d >>>>>$ >>>>> >>>>>However if I drop and re-add the table without the constraint, it >>>>>works: >>>>> >>>>>sql>create table testdata ( >>>>>more> foo varchar(32), >>>>>more> bar varchar(32), >>>>>more> baz varchar(32), >>>>>more> woz varchar(32), >>>>>more> epoch bigint >>>>>more>); >>>>>operation successful (1.477ms) >>>>>sql>\q >>>>>$ mclient -u monetdb -s "copy into mduser.testdata from >>>>>'/tmp/testdata2.csv' using delimiters ',','\n','\'';" testdb >>>>>password: >>>>>2519931 affected rows >>>>> >>>>> >>>>>Thanks Allen! This gets me able to import. Now I'm wondering what >>>>>might be wrong with the constraint. Any insight there? >>>>> >>>>> >>>>> >>>>>On Sat, Nov 12, 2011 at 10:58 AM, Allen Zhang <azhang(a)pmci-usa.com> >>>>>wrote: >>>>>> What about just issue the following: >>>>>> >>>>>> mclient -s "copy into testdata from '/tmp/testdata2.csv' using >>>>>>delimiters >>>>>> ',','\n','\'';" >>>>>> >>>>>> >>>>>> Regards >>>>>> Allen >>>>>> >>>>>> >>>>>> On 11/12/11 10:48 AM, "Sean McNamara" <tank(a)thursday.org> wrote: >>>>>> >>>>>>>Thanks Allen. Red-herring there. My first file had "SPY" in the >>>>>>>first column, and the second had "testdata1", so the difference >>>>>>>there >>>>>>>is just because of the contents. If I convert the original SPY file >>>>>>>and re-run, I get the same behavior. >>>>>>> >>>>>>>$ mclient -s "copy into testdata from STDIN using delimiters >>>>>>>',','\n','\'';" testdb - < /tmp/testdata2.csv >>>>>>>syntax error, unexpected STRING in: "'SPY'" >>>>>>> >>>>>>> >>>>>>>On Sat, Nov 12, 2011 at 10:43 AM, Allen Zhang <azhang(a)pmci-usa.com> >>>>>>>wrote: >>>>>>>> Hi Sean, >>>>>>>> >>>>>>>> You should inspect your data quality, it seems in the first 3 >>>>>>>>imports, >>>>>>>>it >>>>>>>> stopped at "spy", I suspect it might be some \t in that supposed >>>>>>>>to >>>>>>>>be >>>>>>>> string, and last import you put single-quote around string, then >>>>>>>>it >>>>>>>> stopped at "testvalue1", search your import file, look for that >>>>>>>> "testvalue1", the problem should around there, maybe an extra >>>>>>>>field >>>>>>>> because wrong replace by sed. >>>>>>>> >>>>>>>> Best Regards >>>>>>>> Allen >>>>>>>> >>>>>>>> On 11/12/11 10:26 AM, "Sean McNamara" <tank(a)thursday.org> wrote: >>>>>>>> >>>>>>>>>Thanks Viktor, but no dice. I also tried converting from CSV to >>>>>>>>>TAB-delim to match what you were doing, but it's still not >>>>>>>>>working... >>>>>>>>> >>>>>>>>>$ mclient -s "copy into testdata from STDIN using delimiters >>>>>>>>>',','\n','';" testdb - < /tmp/testdata.csv >>>>>>>>>syntax error, unexpected IDENT in: "spy" >>>>>>>>> >>>>>>>>>$ sed 's/,/\'$'\t''/g' /tmp/testdata.csv > /tmp/testdata.dat >>>>>>>>> >>>>>>>>>$ mclient -s "copy into testdata from STDIN;" testdb - < >>>>>>>>>/tmp/testdata.dat >>>>>>>>>syntax error, unexpected IDENT in: "spy" >>>>>>>>> >>>>>>>>>$ mclient -s "copy into testdata from STDIN using delimiters >>>>>>>>>'\t','\n','';" testdb - < /tmp/testdata.dat >>>>>>>>>syntax error, unexpected IDENT in: "spy" >>>>>>>>> >>>>>>>>>I'm understanding the format of the documentation quite a bit >>>>>>>>>better >>>>>>>>>now after spending a few days with it, but I'm still not clear on >>>>>>>>>why >>>>>>>>>this insert is failing. >>>>>>>>>I went one step further and converted the input file to >>>>>>>>>single-quote >>>>>>>>>the string fields, but this fails as well: >>>>>>>>> >>>>>>>>>$ mclient -s "copy into testdata from STDIN using delimiters >>>>>>>>>',','\n','\'';" testdb - < /tmp/testdata2.csv >>>>>>>>>syntax error, unexpected STRING in: "'testvalue1'" >>>>>>>>> >>>>>>>>> >>>>>>>>>The problem has to be something trivial, but I have no idea what >>>>>>>>>it >>>>>>>>>is... >>>>>>>>> >>>>>>>>>On Sat, Nov 12, 2011 at 5:30 AM, Viktor Rosenfeld >>>>>>>>><listuser36(a)googlemail.com> wrote: >>>>>>>>>> Hi Sean, >>>>>>>>>> >>>>>>>>>> Sean McNamara wrote: >>>>>>>>>> >>>>>>>>>>> $ mclient -s "copy 2519931 records into testdata from STDIN >>>>>>>>>>>using >>>>>>>>>>> delimiters ',';" testdb - < /tmp/testdata.csv >>>>>>>>>>> syntax error, unexpected IDENT in: "testvalue1" >>>>>>>>>>> >>>>>>>>>>> where testvalue1 is one of the actual values in the FOO column >>>>>>>>>>> (actually the only value in this particular file.) >>>>>>>>>>> >>>>>>>>>>> I'm wondering if this is telling me it's expecting the field to >>>>>>>>>>>be >>>>>>>>>>> quoted, but that seems unlikely as it would make working with >>>>>>>>>>>CSVs >>>>>>>>>>> quite a bit painful. >>>>>>>>>>> >>>>>>>>>>> Can you guys kindly help me take another step forward. I >>>>>>>>>>>really >>>>>>>>>>> appreciate the guidance. >>>>>>>>>> >>>>>>>>>> I'm using >>>>>>>>>> >>>>>>>>>> COPY ... USING DELIMITERS '\t','\n',''; >>>>>>>>>> >>>>>>>>>> to bulk-load tab-delimited unquoted data. The '\t' is the >>>>>>>>>>default >>>>>>>>>>field >>>>>>>>>> separator, the '\n' is the default record separator. The final >>>>>>>>>>'' >>>>>>>>>>is >>>>>>>>>>the >>>>>>>>>> quote string -- empty in my case. >>>>>>>>>> >>>>>>>>>> See: >>>>>>>>>>http://www.monetdb.org/Documentation/Manuals/SQLreference/CopyInt >>>>>>>>>>o >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Viktor >>>>>>>>>> >>>>>>>>> >>>>>>>>>------------------------------------------------------------------ >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>---- >>>>>>>>>RSA(R) Conference 2012 >>>>>>>>>Save $700 by Nov 18 >>>>>>>>>Register now >>>>>>>>>http://p.sf.net/sfu/rsa-sfdev2dev1 >>>>>>>>>_______________________________________________ >>>>>>>>>MonetDB-users mailing list >>>>>>>>>MonetDB-users(a)lists.sourceforge.net >>>>>>>>>https://lists.sourceforge.net/lists/listinfo/monetdb-users >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>> >>>> >> >> >>

1 0

Re: [MonetDB-users] MonetDB Permission/COPY INTO Questions
by Allen Zhang 12 Nov '11

12 Nov '11

Sean, Use sql select to find out duplicates. Regards Allen On 11/12/11 12:01 PM, "Sean McNamara" <tank(a)thursday.org> wrote: >I tried that (as both the normal dbuser and as admin). Both fail the >same way but don't give an indicator of where it feels the constraint >is violated: > >$ mclient -u monetdb -s "alter table mduser.testdata add constraint >testdata_pkey primary key (foo,epoch);" testdb >password: >SQLException:assert:UPDATE: PRIMARY KEY constraint >'testdata.testdata_pkey' violated > > >On Sat, Nov 12, 2011 at 11:42 AM, Allen Zhang <azhang(a)pmci-usa.com> wrote: >> How about add constraint after you import? I think it might tell you >>which >> one had problem. >> >> Regards >> Allen >> >> On 11/12/11 11:19 AM, "Sean McNamara" <tank(a)thursday.org> wrote: >> >>>Brilliant suggestion, and I think it exposed the problem. >>> >>>$ mclient -u monetdb -s "copy into mduser.testdata from >>>'/tmp/testdata2.csv' using delimiters ',','\n','\'';" testdb >>>password: >>>SQLException:importTable:value 'ONE_SECOND' from line 12734 field 3 >>>not inserted, expecting type varchar(8) >>>failed to import table >>> >>>It looks like the field I created is too small to handle some of the >>>values in the file. After expanding all the varchar fields to 32 and >>>re-running, I get a different error: >>> >>>$ mclient -u monetdb -s "copy into mduser.testdata from >>>'/tmp/testdata2.csv' using delimiters ',','\n','\'';" testdb >>>password: >>>SQLException:assert:INSERT INTO: PRIMARY KEY constraint >>>'testdata.testdata_pkey' violated >>> >>>That makes me think I have duplicate data. The PKEY is comprised of >>>fields #1 and #5, so I check for that but it looks fine: >>> >>>$ cat /tmp/testdata2.csv | cut -d, -f1,5 |uniq -d >>>$ >>> >>>However if I drop and re-add the table without the constraint, it works: >>> >>>sql>create table testdata ( >>>more> foo varchar(32), >>>more> bar varchar(32), >>>more> baz varchar(32), >>>more> woz varchar(32), >>>more> epoch bigint >>>more>); >>>operation successful (1.477ms) >>>sql>\q >>>$ mclient -u monetdb -s "copy into mduser.testdata from >>>'/tmp/testdata2.csv' using delimiters ',','\n','\'';" testdb >>>password: >>>2519931 affected rows >>> >>> >>>Thanks Allen! This gets me able to import. Now I'm wondering what >>>might be wrong with the constraint. Any insight there? >>> >>> >>> >>>On Sat, Nov 12, 2011 at 10:58 AM, Allen Zhang <azhang(a)pmci-usa.com> >>>wrote: >>>> What about just issue the following: >>>> >>>> mclient -s "copy into testdata from '/tmp/testdata2.csv' using >>>>delimiters >>>> ',','\n','\'';" >>>> >>>> >>>> Regards >>>> Allen >>>> >>>> >>>> On 11/12/11 10:48 AM, "Sean McNamara" <tank(a)thursday.org> wrote: >>>> >>>>>Thanks Allen. Red-herring there. My first file had "SPY" in the >>>>>first column, and the second had "testdata1", so the difference there >>>>>is just because of the contents. If I convert the original SPY file >>>>>and re-run, I get the same behavior. >>>>> >>>>>$ mclient -s "copy into testdata from STDIN using delimiters >>>>>',','\n','\'';" testdb - < /tmp/testdata2.csv >>>>>syntax error, unexpected STRING in: "'SPY'" >>>>> >>>>> >>>>>On Sat, Nov 12, 2011 at 10:43 AM, Allen Zhang <azhang(a)pmci-usa.com> >>>>>wrote: >>>>>> Hi Sean, >>>>>> >>>>>> You should inspect your data quality, it seems in the first 3 >>>>>>imports, >>>>>>it >>>>>> stopped at "spy", I suspect it might be some \t in that supposed to >>>>>>be >>>>>> string, and last import you put single-quote around string, then it >>>>>> stopped at "testvalue1", search your import file, look for that >>>>>> "testvalue1", the problem should around there, maybe an extra field >>>>>> because wrong replace by sed. >>>>>> >>>>>> Best Regards >>>>>> Allen >>>>>> >>>>>> On 11/12/11 10:26 AM, "Sean McNamara" <tank(a)thursday.org> wrote: >>>>>> >>>>>>>Thanks Viktor, but no dice. I also tried converting from CSV to >>>>>>>TAB-delim to match what you were doing, but it's still not >>>>>>>working... >>>>>>> >>>>>>>$ mclient -s "copy into testdata from STDIN using delimiters >>>>>>>',','\n','';" testdb - < /tmp/testdata.csv >>>>>>>syntax error, unexpected IDENT in: "spy" >>>>>>> >>>>>>>$ sed 's/,/\'$'\t''/g' /tmp/testdata.csv > /tmp/testdata.dat >>>>>>> >>>>>>>$ mclient -s "copy into testdata from STDIN;" testdb - < >>>>>>>/tmp/testdata.dat >>>>>>>syntax error, unexpected IDENT in: "spy" >>>>>>> >>>>>>>$ mclient -s "copy into testdata from STDIN using delimiters >>>>>>>'\t','\n','';" testdb - < /tmp/testdata.dat >>>>>>>syntax error, unexpected IDENT in: "spy" >>>>>>> >>>>>>>I'm understanding the format of the documentation quite a bit better >>>>>>>now after spending a few days with it, but I'm still not clear on >>>>>>>why >>>>>>>this insert is failing. >>>>>>>I went one step further and converted the input file to single-quote >>>>>>>the string fields, but this fails as well: >>>>>>> >>>>>>>$ mclient -s "copy into testdata from STDIN using delimiters >>>>>>>',','\n','\'';" testdb - < /tmp/testdata2.csv >>>>>>>syntax error, unexpected STRING in: "'testvalue1'" >>>>>>> >>>>>>> >>>>>>>The problem has to be something trivial, but I have no idea what it >>>>>>>is... >>>>>>> >>>>>>>On Sat, Nov 12, 2011 at 5:30 AM, Viktor Rosenfeld >>>>>>><listuser36(a)googlemail.com> wrote: >>>>>>>> Hi Sean, >>>>>>>> >>>>>>>> Sean McNamara wrote: >>>>>>>> >>>>>>>>> $ mclient -s "copy 2519931 records into testdata from STDIN using >>>>>>>>> delimiters ',';" testdb - < /tmp/testdata.csv >>>>>>>>> syntax error, unexpected IDENT in: "testvalue1" >>>>>>>>> >>>>>>>>> where testvalue1 is one of the actual values in the FOO column >>>>>>>>> (actually the only value in this particular file.) >>>>>>>>> >>>>>>>>> I'm wondering if this is telling me it's expecting the field to >>>>>>>>>be >>>>>>>>> quoted, but that seems unlikely as it would make working with >>>>>>>>>CSVs >>>>>>>>> quite a bit painful. >>>>>>>>> >>>>>>>>> Can you guys kindly help me take another step forward. I really >>>>>>>>> appreciate the guidance. >>>>>>>> >>>>>>>> I'm using >>>>>>>> >>>>>>>> COPY ... USING DELIMITERS '\t','\n',''; >>>>>>>> >>>>>>>> to bulk-load tab-delimited unquoted data. The '\t' is the default >>>>>>>>field >>>>>>>> separator, the '\n' is the default record separator. The final '' >>>>>>>>is >>>>>>>>the >>>>>>>> quote string -- empty in my case. >>>>>>>> >>>>>>>> See: >>>>>>>>http://www.monetdb.org/Documentation/Manuals/SQLreference/CopyInto >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Viktor >>>>>>>> >>>>>>> >>>>>>>-------------------------------------------------------------------- >>>>>>>-- >>>>>>>-- >>>>>>>-- >>>>>>>---- >>>>>>>RSA(R) Conference 2012 >>>>>>>Save $700 by Nov 18 >>>>>>>Register now >>>>>>>http://p.sf.net/sfu/rsa-sfdev2dev1 >>>>>>>_______________________________________________ >>>>>>>MonetDB-users mailing list >>>>>>>MonetDB-users(a)lists.sourceforge.net >>>>>>>https://lists.sourceforge.net/lists/listinfo/monetdb-users >>>>>> >>>>>> >>>>>> >>>> >>>> >>>> >> >> >>

1 0

Re: [MonetDB-users] MonetDB Permission/COPY INTO Questions
by Allen Zhang 12 Nov '11

12 Nov '11

What about just issue the following: mclient -s "copy into testdata from '/tmp/testdata2.csv' using delimiters ',','\n','\'';" Regards Allen On 11/12/11 10:48 AM, "Sean McNamara" <tank(a)thursday.org> wrote: >Thanks Allen. Red-herring there. My first file had "SPY" in the >first column, and the second had "testdata1", so the difference there >is just because of the contents. If I convert the original SPY file >and re-run, I get the same behavior. > >$ mclient -s "copy into testdata from STDIN using delimiters >',','\n','\'';" testdb - < /tmp/testdata2.csv >syntax error, unexpected STRING in: "'SPY'" > > >On Sat, Nov 12, 2011 at 10:43 AM, Allen Zhang <azhang(a)pmci-usa.com> wrote: >> Hi Sean, >> >> You should inspect your data quality, it seems in the first 3 imports, >>it >> stopped at "spy", I suspect it might be some \t in that supposed to be >> string, and last import you put single-quote around string, then it >> stopped at "testvalue1", search your import file, look for that >> "testvalue1", the problem should around there, maybe an extra field >> because wrong replace by sed. >> >> Best Regards >> Allen >> >> On 11/12/11 10:26 AM, "Sean McNamara" <tank(a)thursday.org> wrote: >> >>>Thanks Viktor, but no dice. I also tried converting from CSV to >>>TAB-delim to match what you were doing, but it's still not working... >>> >>>$ mclient -s "copy into testdata from STDIN using delimiters >>>',','\n','';" testdb - < /tmp/testdata.csv >>>syntax error, unexpected IDENT in: "spy" >>> >>>$ sed 's/,/\'$'\t''/g' /tmp/testdata.csv > /tmp/testdata.dat >>> >>>$ mclient -s "copy into testdata from STDIN;" testdb - < >>>/tmp/testdata.dat >>>syntax error, unexpected IDENT in: "spy" >>> >>>$ mclient -s "copy into testdata from STDIN using delimiters >>>'\t','\n','';" testdb - < /tmp/testdata.dat >>>syntax error, unexpected IDENT in: "spy" >>> >>>I'm understanding the format of the documentation quite a bit better >>>now after spending a few days with it, but I'm still not clear on why >>>this insert is failing. >>>I went one step further and converted the input file to single-quote >>>the string fields, but this fails as well: >>> >>>$ mclient -s "copy into testdata from STDIN using delimiters >>>',','\n','\'';" testdb - < /tmp/testdata2.csv >>>syntax error, unexpected STRING in: "'testvalue1'" >>> >>> >>>The problem has to be something trivial, but I have no idea what it >>>is... >>> >>>On Sat, Nov 12, 2011 at 5:30 AM, Viktor Rosenfeld >>><listuser36(a)googlemail.com> wrote: >>>> Hi Sean, >>>> >>>> Sean McNamara wrote: >>>> >>>>> $ mclient -s "copy 2519931 records into testdata from STDIN using >>>>> delimiters ',';" testdb - < /tmp/testdata.csv >>>>> syntax error, unexpected IDENT in: "testvalue1" >>>>> >>>>> where testvalue1 is one of the actual values in the FOO column >>>>> (actually the only value in this particular file.) >>>>> >>>>> I'm wondering if this is telling me it's expecting the field to be >>>>> quoted, but that seems unlikely as it would make working with CSVs >>>>> quite a bit painful. >>>>> >>>>> Can you guys kindly help me take another step forward. I really >>>>> appreciate the guidance. >>>> >>>> I'm using >>>> >>>> COPY ... USING DELIMITERS '\t','\n',''; >>>> >>>> to bulk-load tab-delimited unquoted data. The '\t' is the default >>>>field >>>> separator, the '\n' is the default record separator. The final '' is >>>>the >>>> quote string -- empty in my case. >>>> >>>> See: >>>>http://www.monetdb.org/Documentation/Manuals/SQLreference/CopyInto >>>> >>>> Cheers, >>>> Viktor >>>> >>> >>>------------------------------------------------------------------------ >>>-- >>>---- >>>RSA(R) Conference 2012 >>>Save $700 by Nov 18 >>>Register now >>>http://p.sf.net/sfu/rsa-sfdev2dev1 >>>_______________________________________________ >>>MonetDB-users mailing list >>>MonetDB-users(a)lists.sourceforge.net >>>https://lists.sourceforge.net/lists/listinfo/monetdb-users >> >> >>

1 0

Re: [MonetDB-users] Extended Fetch ODBC Error
by Jeremy O'Sullivan 10 Nov '11

10 Nov '11

Hi Sjoerd, Would it be possible to test your thesis ? The OBI guys reckon that for odbc 2 connections, they use the function SQLExtendedFetch. This has bee replaced by SQLFetchScroll in odbc 3.5. Not sure if that helps, I know little or nothing about odbc, other than many BI tools only use this interface and not jdbc. Regards, Jeremy It is unclear from this message what exactly fails. My guess (since I saw something like that recently) is that Oracle's OBI EE tries to set the statement option (or attribute) SQL_ATTR_CONCURRENCY to a value that is not supported. We only support setting the value to SQL_CONCUR_READ_ONLY. Attempting to set the attribute to any other value will return the above error (not implemented). Perhaps what we should do instead here is accept the value and return a message (in the ODBC way) that instead of the value that the application asked for we set it to SQL_CONCUR_READ_ONLY. Other possible statement attributes that could return the feature not implemented error are SQL_ATTR_ASYNC_ENABLE, SQL_ATTR_CURSOR_SENSITIVITY, SQL_ATTR_ENABLE_AUTO_IPD, SQL_ATTR_FETCH_BOOKMARK_PTR, SQL_ATTR_KEYSET_SIZE, SQL_ATTR_MAX_LENGTH, SQL_ATTR_MAX_ROWS, SQL_ATTR_QUERY_TIMEOUT, SQL_ATTR_ROW_NUMBER, SQL_ATTR_SIMULATE_CURSOR, and SQL_ATTR_USE_BOOKMARKS. On 9 Nov 2011, at 12:26, Jeremy O'Sullivan wrote: > Hello, > > When trying to connect to MonetDB using Oracle's OBI EE, I get the error below. Has anyone seen this before ? > Regards, > Jeremy > > [MonetDB][ODBC Driver 11.5.7]Optional feature not implemented. > > [nQSError: 16006] Setting the statement option for extended fetch failed. > >

2 1

[MonetDB-users] Extended Fetch ODBC Error
by Jeremy O'Sullivan 09 Nov '11

09 Nov '11

Hello, When trying to connect to MonetDB using Oracle's OBI EE, I get the error below. Has anyone seen this before ? Regards, Jeremy [MonetDB][ODBC Driver 11.5.7]Optional feature not implemented. [nQSError: 16006] Setting the statement option for extended fetch failed.

2 1

[MonetDB-users] Tableau Alias error
by Jeremy O'Sullivan 09 Nov '11

09 Nov '11

Hello, When connecting from Tableau, the following syntax generates and error on the command line, is this correct ? (The :w triggers the error.) Regards, Jeremy select count(context_key) as "context_key:w" from context_dim;

2 1

[MonetDB-users] How to check wether a record exists or not in stored procedure
by dengxk 08 Nov '11

08 Nov '11

I am writing a stored procedure to insert an record into my table. If the record exists, just update it. it is easier to acheive in microsoft sql server with the sql: if exists( select column from table where condition ...). As monetdb doesn't support select statment and cursor in stored procedure, is there other walkround?

1 0

Re: [MonetDB-users] confirm e5d538cae9aefced13c1ef02f7f06476bbdd59e0
by charles 07 Nov '11

07 Nov '11

Hello Everyone, I have been starting to work with MonetDB for use with our large production database. It is a 'data warehouse' of sorts. Few comments: a) the php client was not included with built/binary distro b) user and password was not specified in http://www.monetdb.org/Documentation/monetdbd at "Next thing, making a connection to the database using mclient requires the -d argument to specify which database to connect to:" c) 'last crash' field is scary to sys admins/users :) I am going to investigate more, just wanted to pop my head in and say hello. As you can guess, our main dev platform is php, so I will need to be using the php extensions. Cheers, Charles

3 7

[MonetDB-users] alias error
by Jeremy O'Sullivan 07 Nov '11

07 Nov '11

Hello, The following syntax generates and error on the command line, is this correct ? The :w triggers the error. Regards, Jeremy select count(context_key) as "context_key:w" from context_dim;

1 0

[MonetDB-users] ODBC-Extended Fetch
by Jeremy O'Sullivan 07 Nov '11

07 Nov '11

Hello, When trying to connect to MonetDB using Oracle's OBI EE, I get the error below. Has anyone seen this before ? Regards, Jeremy [nQSError: 16001] ODBC error state: HYC00 code: 0 message: [MonetDB][ODBC Driver 11.5.7]Optional feature not implemented. [nQSError: 16006] Setting the statement option for extended fetch failed.

1 0

[MonetDB-users] Errors after executing "set history=true" for saving the query history
by Roberto Fernandez 05 Nov '11

05 Nov '11

Hi, I was trying to save my history based on this documentation http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/History http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/History And this is what is happening: $ mclient -dtest-db -umonetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Aug2011-SP2) Database: MonetDB v11.5.7 (Aug2011-SP2), 'test-db' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (0.623ms) sql>select count(*) from narvq.narvq where zip5 = '81657'; +------+ | L5 | +======+ | 2785 | +------+ 1 tuple (1.6s) sql>set history=true; sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (0.232ms) sql>select count(*) from narvq.narvq where zip5 = '81657'; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _23:any := user.keepquery(_24:wrd, _25:str, _26:lng, _27:lng) TypeException:user.s0_1[32]:'user.keepcall' undefined in: _73:any := user.keepcall(_24:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>select * from queryHistory; TypeException:user.s5_1[1]:'user.keepquery' undefined in: _43:any := user.keepquery(_44:wrd, _45:str, _46:lng, _47:lng) TypeException:user.s5_1[48]:'user.keepcall' undefined in: _111:any := user.keepcall(_44:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>select 1; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _12:any := user.keepquery(_13:wrd, _14:str, _15:lng, _16:lng) TypeException:user.s0_1[24]:'user.keepcall' undefined in: _55:any := user.keepcall(_13:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>set history=false; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _2:any := user.keepquery(_3:wrd, _4:str, _5:lng, _6:lng) TypeException:user.s0_1[19]:'user.keepcall' undefined in: _41:any := user.keepcall(_3:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql> SERVER INFORMATION: It is a Amazon Web Services Linux distribution (based on CentOS) $ mserver5 --version MonetDB 5 server v11.5.7 "Aug2011-SP2" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2011 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 7.3GiB available memory, 2 available cpu cores Libraries: libpcre: 7.8 2008-09-05 (compiled with 7.8) openssl: OpenSSL 1.0.0e 6 Sep 2011 (compiled with OpenSSL 1.0.0e-fips 6 Sep 2011) libxml2: 2.7.6 (compiled with 2.7.6) Compiled by: ec2-user@domU-12-31-39-0E-8D-F2 (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 I will really appreciate any help provided. Thanks, Roberto F -- View this message in context: http://old.nabble.com/Errors-after-executing-%22set-history%3Dtrue%22-for-s… Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] Errors saving query history
by Roberto Fernandez 05 Nov '11

05 Nov '11

Hi, I was trying to save my history based on this documentation http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/History http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/History And this is what is happening: $ mclient -dtest-db -umonetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Aug2011-SP2) Database: MonetDB v11.5.7 (Aug2011-SP2), 'test-db' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (0.623ms) sql>select count(*) from narvq.narvq where zip5 = '81657'; +------+ | L5 | +======+ | 2785 | +------+ 1 tuple (1.6s) sql>set history=true; sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (0.232ms) sql>select count(*) from narvq.narvq where zip5 = '81657'; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _23:any := user.keepquery(_24:wrd, _25:str, _26:lng, _27:lng) TypeException:user.s0_1[32]:'user.keepcall' undefined in: _73:any := user.keepcall(_24:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>select * from queryHistory; TypeException:user.s5_1[1]:'user.keepquery' undefined in: _43:any := user.keepquery(_44:wrd, _45:str, _46:lng, _47:lng) TypeException:user.s5_1[48]:'user.keepcall' undefined in: _111:any := user.keepcall(_44:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>select 1; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _12:any := user.keepquery(_13:wrd, _14:str, _15:lng, _16:lng) TypeException:user.s0_1[24]:'user.keepcall' undefined in: _55:any := user.keepcall(_13:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>set history=false; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _2:any := user.keepquery(_3:wrd, _4:str, _5:lng, _6:lng) TypeException:user.s0_1[19]:'user.keepcall' undefined in: _41:any := user.keepcall(_3:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql> SERVER INFORMATION: It is a Amazon Web Services Linux distribution (based on CentOS) $ mserver5 --version MonetDB 5 server v11.5.7 "Aug2011-SP2" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2011 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 7.3GiB available memory, 2 available cpu cores Libraries: libpcre: 7.8 2008-09-05 (compiled with 7.8) openssl: OpenSSL 1.0.0e 6 Sep 2011 (compiled with OpenSSL 1.0.0e-fips 6 Sep 2011) libxml2: 2.7.6 (compiled with 2.7.6) Compiled by: ec2-user@domU-12-31-39-0E-8D-F2 (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 I will really appreciate any help provided. Thanks, Roberto F -- View this message in context: http://old.nabble.com/Errors-saving-query-history-tp32786364p32786364.html Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] Errors saving query history
by Roberto Fernandez 05 Nov '11

05 Nov '11

Hi, I was trying to save my history based on this documentation http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/History http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/History And this is what is happening: $ mclient -dtest-db -umonetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Aug2011-SP2) Database: MonetDB v11.5.7 (Aug2011-SP2), 'test-db' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (0.623ms) sql>select count(*) from narvq.narvq where zip5 = '81657'; +------+ | L5 | +======+ | 2785 | +------+ 1 tuple (1.6s) sql>set history=true; sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (0.232ms) sql>select count(*) from narvq.narvq where zip5 = '81657'; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _23:any := user.keepquery(_24:wrd, _25:str, _26:lng, _27:lng) TypeException:user.s0_1[32]:'user.keepcall' undefined in: _73:any := user.keepcall(_24:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>select * from queryHistory; TypeException:user.s5_1[1]:'user.keepquery' undefined in: _43:any := user.keepquery(_44:wrd, _45:str, _46:lng, _47:lng) TypeException:user.s5_1[48]:'user.keepcall' undefined in: _111:any := user.keepcall(_44:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>select 1; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _12:any := user.keepquery(_13:wrd, _14:str, _15:lng, _16:lng) TypeException:user.s0_1[24]:'user.keepcall' undefined in: _55:any := user.keepcall(_13:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>set history=false; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _2:any := user.keepquery(_3:wrd, _4:str, _5:lng, _6:lng) TypeException:user.s0_1[19]:'user.keepcall' undefined in: _41:any := user.keepcall(_3:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql> SERVER INFORMATION: It is a Amazon Web Services Linux distribution (based on CentOS) $ mserver5 --version MonetDB 5 server v11.5.7 "Aug2011-SP2" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2011 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 7.3GiB available memory, 2 available cpu cores Libraries: libpcre: 7.8 2008-09-05 (compiled with 7.8) openssl: OpenSSL 1.0.0e 6 Sep 2011 (compiled with OpenSSL 1.0.0e-fips 6 Sep 2011) libxml2: 2.7.6 (compiled with 2.7.6) Compiled by: ec2-user@domU-12-31-39-0E-8D-F2 (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 I will really appreciate any help provided. Thanks, Roberto F -- View this message in context: http://old.nabble.com/Errors-saving-query-history-tp32786363p32786363.html Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] Errors saving query history
by Roberto Fernandez 05 Nov '11

05 Nov '11

Hi, I was trying to save my history based on this documentation http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/History http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/History And this is what is happening: $ mclient -dtest-db -umonetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Aug2011-SP2) Database: MonetDB v11.5.7 (Aug2011-SP2), 'test-db' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (0.623ms) sql>select count(*) from narvq.narvq where zip5 = '81657'; +------+ | L5 | +======+ | 2785 | +------+ 1 tuple (1.6s) sql>set history=true; sql>select 1; +--------------+ | single_value | +==============+ | 1 | +--------------+ 1 tuple (0.232ms) sql>select count(*) from narvq.narvq where zip5 = '81657'; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _23:any := user.keepquery(_24:wrd, _25:str, _26:lng, _27:lng) TypeException:user.s0_1[32]:'user.keepcall' undefined in: _73:any := user.keepcall(_24:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>select * from queryHistory; TypeException:user.s5_1[1]:'user.keepquery' undefined in: _43:any := user.keepquery(_44:wrd, _45:str, _46:lng, _47:lng) TypeException:user.s5_1[48]:'user.keepcall' undefined in: _111:any := user.keepcall(_44:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>select 1; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _12:any := user.keepquery(_13:wrd, _14:str, _15:lng, _16:lng) TypeException:user.s0_1[24]:'user.keepcall' undefined in: _55:any := user.keepcall(_13:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql>set history=false; TypeException:user.s0_1[1]:'user.keepquery' undefined in: _2:any := user.keepquery(_3:wrd, _4:str, _5:lng, _6:lng) TypeException:user.s0_1[19]:'user.keepcall' undefined in: _41:any := user.keepcall(_3:wrd, ctime:timestamp, args:str, xtime:lng, rtime:lng, foot:lng, memory:lng, tuples:wrd, inblock:lng, oublock:lng) SQLException:SQLengine:Program contains errors sql> SERVER INFORMATION: It is a Amazon Web Services Linux distribution (based on CentOS) $ mserver5 --version MonetDB 5 server v11.5.7 "Aug2011-SP2" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2011 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 7.3GiB available memory, 2 available cpu cores Libraries: libpcre: 7.8 2008-09-05 (compiled with 7.8) openssl: OpenSSL 1.0.0e 6 Sep 2011 (compiled with OpenSSL 1.0.0e-fips 6 Sep 2011) libxml2: 2.7.6 (compiled with 2.7.6) Compiled by: ec2-user@domU-12-31-39-0E-8D-F2 (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 I will really appreciate any help provided. Thanks, Roberto F -- View this message in context: http://old.nabble.com/Errors-saving-query-history-tp32786355p32786355.html Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] compile error when build MonetDB 11.5.7 on CentOS 5.5
by Allen Zhang 04 Nov '11

04 Nov '11

Hi, When compile on CentOS 5.5, following error pop up, can any one tell me which version of libcom_err.so is needed for this MonetDB version? Thanks Allen make[5]: Entering directory `/usr/src/MonetDB-11.5.7/common/stream' /bin/sh ../../libtool --tag=CC --mode=link gcc -DLIBSTREAM -g -O2 -D_REENTRANT -version-info 2:2:0 -o libstream.la -rpath /usr/local/MonetDBAug2011-SP2/lib libstream_la-stream.lo -lbz2 -L/usr/kerberos/lib64 -lssl -lcrypto -ldl -lz -L/usr/kerberos/lib64 -lcurl -ldl -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lidn -lssl -lcrypto -lz -pthread libtool: link: gcc -shared -fPIC -DPIC .libs/libstream_la-stream.o -lbz2 -L/usr/kerberos/lib64 -lcurl -ldl -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lidn -lssl -lcrypto -lz -O2 -pthread -pthread -Wl,-soname -Wl,libstream.so.2 -o .libs/libstream.so.2.0.2 /usr/bin/ld: skipping incompatible /usr/lib/libcom_err.so when searching for -lcom_err /usr/bin/ld: skipping incompatible /usr/lib/libcom_err.a when searching for -lcom_err /usr/bin/ld: cannot find -lcom_err collect2: ld returned 1 exit status

2 2

Re: [MonetDB-users] compile error when build MonetDB 11.5.7 on CentOS 5.5
by Stefan Manegold 04 Nov '11

04 Nov '11

Hi Allen, from your log, I guess your CentS is 64 bit, right? Is the installed libcom_err also 64 bit or 32 bit? The linker says the sitcom_err it finds is "incompatible" ... Stefan ----- Reply message ----- From: "Allen Zhang" <azhang(a)pmci-usa.com> To: <monetdb-users(a)lists.sourceforge.net> Subject: [MonetDB-users] compile error when build MonetDB 11.5.7 on CentOS 5.5 Date: Fri, Nov 4, 2011 01:07 Hi, When compile on CentOS 5.5, following error pop up, can any one tell me which version of libcom_err.so is needed for this MonetDB version? Thanks Allen make[5]: Entering directory `/usr/src/MonetDB-11.5.7/common/stream' /bin/sh ../../libtool --tag=CC --mode=link gcc -DLIBSTREAM -g -O2 -D_REENTRANT -version-info 2:2:0 -o libstream.la -rpath /usr/local/MonetDBAug2011-SP2/lib libstream_la-stream.lo -lbz2 -L/usr/kerberos/lib64 -lssl -lcrypto -ldl -lz -L/usr/kerberos/lib64 -lcurl -ldl -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lidn -lssl -lcrypto -lz -pthread libtool: link: gcc -shared -fPIC -DPIC .libs/libstream_la-stream.o -lbz2 -L/usr/kerberos/lib64 -lcurl -ldl -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lidn -lssl -lcrypto -lz -O2 -pthread -pthread -Wl,-soname -Wl,libstream.so.2 -o .libs/libstream.so.2.0.2 /usr/bin/ld: skipping incompatible /usr/lib/libcom_err.so when searching for -lcom_err /usr/bin/ld: skipping incompatible /usr/lib/libcom_err.a when searching for -lcom_err /usr/bin/ld: cannot find -lcom_err collect2: ld returned 1 exit status

1 0

[MonetDB-users] I want to join this group:)
by Tony Bai 02 Nov '11

02 Nov '11

Could you add me as a member of Monetdb please? Thanks, Tony

1 0

[MonetDB-users] Temporal coalescing query error
by Alexander S 29 Oct '11

29 Oct '11

Hello, I'm attempting to use monetdb for some temporal queries. Some queries require the output to be coalesced. For example, with schema emp(empno int, ts date, te date) the tuples (00001, '1/1/2001', '2/1/2001'), (00001, '2/1/2001', '3/1/2001') are output as (00001, '1/1/2001', '3/1/2001'). My coalescing query works in both MS SQL Server and DB2 but MonetDB gives me the error "SELECT: no such column 'm.ts'". 1) A leaner alternative query form using window clauses (ROWS UNBOUNDED PRECEDING) doesn't appear to be supported in MonetDB. So I'm stuck. Any help would be appreciated. Query: WITH Emps(empno, ts, te) AS (SELECT empno, ts, te FROM emp) SELECT DISTINCT F.empno, F.ts, L.te FROM Emps AS F, Emps AS L WHERE F.ts < L.te AND F.empno = L.empno AND NOT EXISTS (SELECT * FROM Emps AS M WHERE M.empno = F.empno AND F.ts < M.ts AND M.ts < L.te AND NOT EXISTS (SELECT * FROM Emps AS T1 WHERE T1.empno = F.empno AND T1.ts < M.ts AND M.ts <= T1.te) ) AND NOT EXISTS (SELECT * FROM Emps AS T2 WHERE T2.empno = F.empno AND ((T2.ts < F.ts AND F.ts <= T2.te) OR (T2.ts <= L.te AND L.te < T2.te)) ); Thanks, Alex -- View this message in context: http://old.nabble.com/Temporal-coalescing-query-error-tp32742108p32742108.h… Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] eEnd of Stream
by nat lu 28 Oct '11

28 Oct '11

I'm using Monet 11.5 on Ubuntu 11.04. Ive created a database and am attempting to run a series of queries against it with a test script. This is in attempt to use Monet with Apache Jena. I had earlier mailed about a problem with temp tables, but have dodged that by using the monetdb account for now. The first few succeed, but then I get the following error: java.sql.SQLException: Read from localhost:50000: End of stream reached (mserver still alive?) Why does this happen, under what circumstances does it arise ? Is there anything I can do to avoid it ? This is the query that it fails on: SELECT R_1.lex AS V_1_lex, R_1.datatype AS V_1_datatype, R_1.lang AS V_1_lang, R_1.type AS V_1_type, R_2.lex AS V_2_lex, R_2.datatype AS V_2_datatype, R_2.lang AS V_2_lang, R_2.type AS V_2_type FROM ( SELECT T_1.s AS X_1 FROM Triples AS T_1 WHERE ( T_1.p = 2004134117598721274 AND T_1.o = 435905340492217258 ) ) AS T_1 LEFT OUTER JOIN Triples AS T_2 ON ( T_2.p = 3816485599920428794 AND T_1.X_1 = T_2.s ) LEFT OUTER JOIN Nodes AS R_1 ON ( T_1.X_1 = R_1.hash ) LEFT OUTER JOIN Nodes AS R_2 ON ( T_2.o = R_2.hash )

3 7

[MonetDB-users] IsNumeric() clause in MonetDB
by Osmin Castillo 26 Oct '11

26 Oct '11

Hi everyone: How can I test if a VARCHAR field has numeric data?, at SQLServer I use IsNumeric() clause, what can i use in MonetDB? thanks in advance Osmin

1 0