- users-list - monetdb.org

[MonetDB-users] 64bit MonetDB, JDBC Insert via RJDBC, >300 million rows
by Yue Sheng 17 Mar '09

17 Mar '09

I'm been trying to insert from file for over 322 files, totally 12GB with over 600 million rows. The RJDBC connection hangs after 350 million row. I noticed the VSIZE in Activity Monitor is 30.85GB, closer to 32GB system memory. Is this the reason that I cannot connect to the database? What should I do? Error message I get from MonetDB server: # MonetDB server v5.8.4, based on kernel v1.26.4 # Serving database 'bigdata', using 8 threads # Compiled for i386-apple-darwin9.6.0/64bit with 64bit OIDs dynamically linked # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008- MonetDB B.V., all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /Users/someuser/MonetDB64/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.26.4 loaded >!ERROR: GDKsave: failed name=24/2460, ext=tail, mode 0 !ERROR: GDKsave: failed name=23/2317, ext=tail, mode 0 !ERROR: GDKsave: failed name=25/2503, ext=tail, mode 0 !ERROR: GDKsave: failed name=26/2657, ext=tail, mode 0 !ERROR: GDKsave: failed name=22/2274, ext=tail, mode 0 !ERROR: GDKsave: failed name=22/2224, ext=tail, mode 0 !ERROR: GDKsave: failed name=01/112, ext=head, mode 0 !ERROR: logger_exit: could not open /Users/someuser/MonetDB64/var/MonetDB5/sql_logs/bigdata/sql/log System:MacPro Dual Quad Xenon Leopard OS X 32GB RAM R 2.8.1 32Bit MonetDB v5.8.4 complied for i386-apple-darmin9.6.0/64bit Many thanks. Yuesheng

4 28

[MonetDB-users] Apples and oranges
by Nils Grimsmo 17 Mar '09

17 Mar '09

Hi! I have some questions as to how MonetDB/XQuery should be compared fairly to other systems. If I re-run a query multiple times in a single call to `mclient`, is any calculation re-used? How about if I run multiple similar queries in a single call? Example: $ cat www.xq count(doc("dblp")//www) $ cat www_s10.xq (count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www)) $ cat www_s100.xq (count(doc("dblp")//www), ... count(doc("dblp")//www)) $ mclient --language=xquery --time < www.xq 11760 Timer 22.552 msec (Assert we are running hot.) $ mclient --language=xquery --time < www.xq 11760 Timer 21.661 msec $ mclient --language=xquery --time < www_s10.xq 11760, [snip] 11760 Timer 33.063 msec $ mclient --language=xquery --time < www_s100.xq 11760, [snip] 11760 Timer 252.414 msec So the average execution times are 22, 3.3 and 2.5 milliseconds. Is the extra cost for the first query just starting up the client program, or is some calculation re-used? If we now look at more expensive queries: $ cat dblp_authors.xq count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]) Just repeating the same: $ cat dblp_authors_s10.xq (count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"])) Different but related queries: $ cat dblp_authors_x10.xq (count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Wen Gao"]), count(doc("dblp")/dblp//author[text()="Irith Pomeranz"]), count(doc("dblp")/dblp//author[text()="Hector Garcia-Molina"]), count(doc("dblp")/dblp//author[text()="Moshe Y. Vardi"]), count(doc("dblp")/dblp//author[text()="Joseph Y. Halpern"]), count(doc("dblp")/dblp//author[text()="Noga Alon"]), count(doc("dblp")/dblp//author[text()="Wei Li"]), count(doc("dblp")/dblp//author[text()="Ming Li"]), count(doc("dblp")/dblp//author[text()="Donald F. Towsley"]) ) $ mclient --language=xquery --time < dblp_authors.xq 351 Timer 1238.436 msec $ mclient --language=xquery --time < dblp_authors.xq 351 Timer 1253.927 msec $ mclient --language=xquery --time < dblp_authors_s10.xq 351, ... 351 Timer 1284.191 msec $ mclient --language=xquery --time < dblp_authors_x10.xq 351, 347, 346, 341, 334, 334, 330, 320, 320, 317 Timer 2610.589 msec Here the average times are 1238, 128 and 261 milliseconds. Here the difference is clearly not just startup of the client. If this was not a client-server architecture, I would guess the difference came from opening files, getting stuff into cache, etc.. Are there similar reasons here? What parts of the calculations are actually done inside the client, if any? If the answer is none, why is this behavior seen? In conclusion: When running multiple queries, what would be the most fair way to compare MonetDB/XQuery to other client/server architectures in your view? Concatenating the queries in a single call to `mclient`, or multiple calls? When timing a single query, can it be repeated multiple times in a single call, and the average taken, without being unfair? If I use for example MS SQL Server 2008, there is no gain from a single invocation of the client, whether I have multiple SQL statements SELECT x.query('$q) FROM t; ..., SELECT x.query('$q) FROM t; Or a single SQL statement with a list of XPath queries SELECT x.query('($q, ..., $q)') FROM t; Klem fra Nils -- http://www.idi.ntnu.no/~nilsgri/ Why is this thus? What is the reason of this thusness? - Artemus Ward

4 4

[MonetDB-users] owner of files in dbfarm changed
by Ross Bates 16 Mar '09

16 Mar '09

Hi All - I've been running an Feb2009 nightly install for a few weeks now with no issues. Tried to start the database this morning and was getting tons of errors about the BACKUP directory, SQL Catalog, etc... It turns out that somehow various files dbfarm directory were changed to root ownership. Checked the system logs, bash history, etc.... and root hasn't been used to start any monetdb services or anything else related to monetdb. I know it sounds far fetched but is there something internal to the db which could have done this? Anyone seen mysterious ownership changes like this before? As for a solution, should I be ok just doing a chown on everything below my monetdb home back to the installation user? Thanks, Ross

2 1

[MonetDB-users] [Fwd: making the server accessible from other machines]
by Martin Kersten 16 Mar '09

16 Mar '09

-------- Original Message -------- Subject: making the server accessible from other machines Date: Mon, 16 Mar 2009 11:59:29 +0100 From: francois guérin <replonk(a)gmail.com> To: monet(a)cwi.nl Hi, (i'm francois, this is not the first time i come to ask you some questions about monetdb...) i've tried to make the server accessible from another machine, but it dont work... when i launch the server, and enter the statement " xprc_trusted.delete(); ", everything's fine and i can access to my server with another machine. but, when i edit the MonetBD.conf file, as it's preconized in http://monetdb.cwi.nl/projects/monetdb/XQuery/Documentation/Security.html nothing change a quick look at my monetDB.conf file: # if not set port to address the HTTP server for pathfinder = mapi_port+1 xrpc_port="" # xrpc daemon is open for outside connections by default xrpc_open=yes # Additional options to be passed to the XRPC server, currently only # "timing". xrpc_options="" # IP addresses or hostname separated by ';' who can execute scripts in # <datadir>/MonetDB/xrpc/admin/ xrpc_admin="" # IP addresses or hostname separated by ';' who can retrieve XML # documents stored in the database using a URL of the form: # http://<host>[:port]/xrpc/<name>.xml xrpc_user="" # url prefixes of trusted modules separated by ';' xrpc_trusted="" As you can see, all slots are either open, either empty (that should mean any url/modules/user are allowed)

1 0

Re: [MonetDB-users] import error in python script
by Ranadeeran 16 Mar '09

16 Mar '09

Yes i did that but didn't work.. > You should set PYTHONPATH to something like > /prefix/lib/python2.5/site-packages > > Can you confirm? > > Thanx, > Lazaros. > > On Thu, Mar 12, 2009 at 08:36:08PM +0530, Ranadeeran wrote: >> hi all >> i use MonetDB Database Server Toolkit v0.4 and ubuntu 8.10. >> im unable to issue any of the monetdb commands without prefixing them >> with >> a ./ >> that is instead of mserver5,it works only when i give ./mserver5 >> >> And im able to issue these commands only when im in the bin directory >> though i ve permanantly set the PATH variable in .bash_profile >> >> when i try to connect to monetdb using the sample python script that is >> given in the docs,i get an import error telling unable to import >> MonetDbSQL,though i ve set the PYTHONPATH variable to >> /prefix/lib/python2.5/site-packages/MonetDb directory >> >> the script is >> import MonetSQLdb >> # Let the cursor return dictionaries instead of tuples: >> # x = MonetSQLdb.connect(cursorclass = MonetSQLdb.cursors.DictCursor) >> x = MonetSQLdb.connect(host = 'localhost', user = 'monetdb',password = >> 'monet\db', lang = 'sql') >> #x = MonetSQLdb.connect(lang='sql') >> c = x.cursor() >> c.execute('select * from test') >> #print c.fetchone() >> print c.fetchall() >> x.close() >> >> But the script works fine on Debian Etch. >> >> i need to overcome these two issues... >> >> Thanks in advance, >> Mahesh >> ----------------------------------------- This email was sent using TCEMail Service. Thiagarajar College of Engineering Madurai - 625015 (India)

1 0

[MonetDB-users] import error in python script
by Ranadeeran 13 Mar '09

13 Mar '09

hi all i use MonetDB Database Server Toolkit v0.4 and ubuntu 8.10. im unable to issue any of the monetdb commands without prefixing them with a ./ that is instead of mserver5,it works only when i give ./mserver5 And im able to issue these commands only when im in the bin directory though i ve permanantly set the PATH variable in .bash_profile when i try to connect to monetdb using the sample python script that is given in the docs,i get an import error telling unable to import MonetDbSQL,though i ve set the PYTHONPATH variable to /prefix/lib/python2.5/site-packages/MonetDb directory the script is import MonetSQLdb # Let the cursor return dictionaries instead of tuples: # x = MonetSQLdb.connect(cursorclass = MonetSQLdb.cursors.DictCursor) x = MonetSQLdb.connect(host = 'localhost', user = 'monetdb',password = 'monet\db', lang = 'sql') #x = MonetSQLdb.connect(lang='sql') c = x.cursor() c.execute('select * from test') #print c.fetchone() print c.fetchall() x.close() But the script works fine on Debian Etch. i need to overcome these two issues... Thanks in advance, Mahesh ----------------------------------------- This email was sent using TCEMail Service. Thiagarajar College of Engineering Madurai - 625015 (India)

4 3

[MonetDB-users] Random errors and segfaults executing concurrent queries
by Dominique Prunier 12 Mar '09

12 Mar '09

Hi, I'm experiencing a lot of instabilities using MonetDB-Feb2009 on a moderately big database (70GB) with 3 concurrent (read only) connections. In my test setup, i have 3 identical tables which i query in parallel (from 3 jdbc connections). Almost every time, i get some weird errors: TypeException:user.s213_7[16]:'io.s13_4' undefined in: _26:any := io.s13_4() TypeException:user.s244_2[18]:'io.s14_6' undefined in: _31:any := io.s14_6() TypeException:user.s236_8[31]:'io.s18_3' undefined in: _47:any := io.s18_3() .... Finally, at some point, the server often segfault. Is there anything i can do to improve this ? Thanks,

1 0

[MonetDB-users] Strange performance behavior on bulk import with / without sequences
by Cimballi 11 Mar '09

11 Mar '09

Hi, I have a strange behavior with MonetDB and the JDBC driver, maybe you have an explanation. Here is what I do : I import about 100 000 rows in MonetDB using the JDBC driver and simple SQL statements (not prepared) with Spring JDBC API. I made the test a first time using sequences, so before inserting the data I query for the next sequence number and then I insert the data. The second time, I generate the sequence number on the application side, using a simple increment, so I directly insert the data. And the strange result : In the first test, it took 7:30 minutes for the whole insert. In the second test, it took only 30 seconds for the whole insert. So, it seems there is a problem with the select performance, and in fact I saw this problem for another select operation. Do you have any idea about where it can come from ? Is there any configuration option or the way I use the JDBC API ? Thanks, Cimballi

2 3

[MonetDB-users] writing out all fields
by dariuszs 11 Mar '09

11 Mar '09

Hi, Selecting 50 fields out of 300 takes almost no time to write out to csv file, but selecting everything takes forever - is there any setting that can improve the speed to write out all fields from a table? Thanks. Dariusz.

2 1

[MonetDB-users] MonetDB-MAL usage questions
by Maria 10 Mar '09

10 Mar '09

Hi, I am having some problems using mal. firstly, I want to find bats of a table 'st' I loaded. I created a table 'st' using sql. > ./mclient -lsql -ddemo > \<simpletable.sql ==> it creates table 'st' with two columns 'id' and 'name' > \q then, I try to use mal. > ./mclient -lmal -ddemo > bbp.open(); > bbp.prelude(); > a := bbp.bind("st", "id"); then, I got following error msg > ERROR = !MALException:bbp.find:BAT not found I thought when a table is created using sql, it stored as number of bats and I can find it through bat buffer pool. but it seems that there is nothing in the bbp ( I tried bbp.getObjects(), and printed returns which contains no bats). Where can I get bats of table 'st'? What can be the logical name of each bat? (same as column name?? or tablename_columnname??) Second question) Although I've read all documents in MonetDB site, I'm still quite confused the way to use mal. For example, if I want to create table using mal, how could it be done? sql statement: "create table st ("id" integer not null, "name" varchar(30));" To understand mal, I think I need number of examples. Is there any documents I can study about mal? Is there any way to see system generated mal when I run sql statement? If I can, it can be good example code for me. I really hope that any one could help me. Thanks.

3 2

Re: [MonetDB-users] too deep recursion
by Nils Grimsmo 09 Mar '09

09 Mar '09

Hi, I just upgraded from the Augst to the Noveber super-ball, and the performance has worsened badly. Example queries on dblp.xml (441 MB): Q0: count(/dblp//author[text()="Michael Stonebraker"]) Q1: count(/dblp/*/author[text()="Michael Stonebraker"]) Query time in milliseconds: August November Q0 1100 4867 Q1 3993 17999 I have compiled with --enable-optimise both times. I query with: mclient --language=xquery --algebra --time < $QUERYFILE Is this performance degradation expected? If so, why? BTW: Is there any way of finding how much disk space a collection uses? Thank you for contributing free software! Klem fra Nils -- http://www.idi.ntnu.no/~nilsgri/ Why is this thus? What is the reason of this thusness? - Artemus Ward

3 4

[MonetDB-users] Kettle and MonetDB
by denvro 09 Mar '09

09 Mar '09

Hello, I have just installed Kettle 3.1 on my Archlinux machine. I wanted to use it with MonetDB but when i try to create a JDBC connection i get the following message: Error connecting to database [MonetDB] : org.pentaho.di.core.exception.KettleDatabaseException: Error occured while trying to connect to the database Error connecting to database: (using class nl.cwi.monetdb.jdbc.MonetDriver) Server challenge string unusable! Challenge contains too few tokens: !an internal error has occurred, please try again later I have merovingian running in the back and created a database "pentaho". See status below: bash-3.2# monetdb status -l pentaho pentaho: location: /usr/local/MonetDB/var/MonetDB5/dbfarm/pentaho database name: pentaho state: running locked: no scenarios: mal sql msql connections: mapi:monetdb://myhost:50001/ start count: 3 stop count: 2 crash count: 0 current uptime: 51m 10s average uptime: 13m 12s maximum uptime: 14m 42s minimum uptime: 11m 41s last crash: (unknown) last start: 2009-02-21 13:03:00 average of crashes in the last start attempt: 0 average of crashes in the last 10 start attempts: 0.00 average of crashes in the last 30 start attempts: 0.00 Can anybody help me with this. I already use the lates .jar (monetdb-1.9-jdbc). Kind Regards, Dennis -- View this message in context: http://www.nabble.com/Kettle-and-MonetDB-tp22141036p22141036.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 3

[MonetDB-users] Not working xquery
by Hans van Rijswijk (DT) 06 Mar '09

06 Mar '09

Hi all, Why does this xquery work: let $result := doc("document.xml")//*[properties] let $astypes := distinct-values(for $n in $result/ancestor-or-self::*[properties] return name($n)) return $astypes <> Whereas these xqueries don't: let $result := doc("document.xml")//properties/.. let $astypes := distinct-values(for $n in $result/ancestor-or-self::*[properties] return name($n)) return $astypes <> let $result := doc("document.xml")//properties/parent::* let $astypes := distinct-values(for $n in $result/ancestor-or-self::*[properties] return name($n)) return $astypes <> The latter two xqueries return an error: MAPI = monetdb@host:50000 QUERY = let $result := doc("document.xml")//properties/parent::* ERROR = !ERROR: ancestor-or-self: illegal data in size column or context list. !ERROR: PFll_ancestor_or_self: operation failed. Which MonetDB expert can explain? Regards, Hans v R

2 1

[MonetDB-users] vertically fragment a table manually
by Brightwin 05 Mar '09

05 Mar '09

hi all, Is it possible to vertically fragment a table manually in MonetDB,instead of the Binary Association Table which is automatically created by it. thanks in advance Brightwin. ----------------------------------------- This email was sent using TCEMail Service. Thiagarajar College of Engineering Madurai - 625015 (India)

3 2

[MonetDB-users] implementing high performance time series schemas in monetDB
by Tom H 05 Mar '09

05 Mar '09

Hi, I have recently been using the TSdbi/MySQL library in R, which allows mapping between time series matrices and data stored in a mysql backend - its actually pretty good, I've made loads of progress in a few days... However I now now looking at scaling it up to many parallel sources of time series data, and my limited understanding of implementing this type of schema suggests that column orientated databases like monetDB should perform better than MySQL for this type of data. (?) The schema suggested as the default for storing irregular time series is as follows; create table T ( id VARCHAR(40), date DATETIME, v double DEFAULT ); with an index on the ident and the DATETIME field; CREATE INDEX Tindex_id ON T (id); CREATE INDEX Tindex_date ON T (date); So I am looking to support 500+ writes per second, while entertaining a whole bunch of queries of time periods like; SELECT date, v FROM T WHERE id = 'series1' order by date select date, v from T WHERE date > NOW() - INTERVAL 1 WEEK AND date < NOW(); I was hoping for some advice on whether monetDB is suitable for this type of data, and performance suggestions to get it blisteringly fast!! Thanks, Tom

1 0

[MonetDB-users] too deep recursion
by Lazaros Koromilas 02 Mar '09

02 Mar '09

Hi all, We are trying to annotate some XML data with MonetDB/XQuery. For this I run some big queries, that contain unions of many Xpath expressions (say 100). I get the following error: ERROR = !fatal error: aborted too deep recursion Tried to use the option declare option pf:recursion-depth "<number>"; I see no difference in bahaviour. I also call mclient with -G, though I don't know if it's needed. MonetDB Server is v4.26.2 and default back-end is 'algebra'. Any help with this? Thank you, Lazaros. -- http://www.ics.forth.gr/~koromil

3 4

[MonetDB-users] SUM of NULLs is 0?
by kelas 01 Mar '09

01 Mar '09

Dear MonetDB-users, a complete newbie to MonetDB, I can't really tell whether it is a bug, feature or I'm just getting it all wrong. A script below illustrates some weird NULL handling behaviour found in Nov2008-SP2: -- years under audit CREATE TABLE BUDGET_PERIOD (BP smallint NOT NULL); INSERT INTO BUDGET_PERIOD VALUES (1998); INSERT INTO BUDGET_PERIOD VALUES (1999); INSERT INTO BUDGET_PERIOD VALUES (2000); INSERT INTO BUDGET_PERIOD VALUES (2001); INSERT INTO BUDGET_PERIOD VALUES (2002); INSERT INTO BUDGET_PERIOD VALUES (2003); INSERT INTO BUDGET_PERIOD VALUES (2004); INSERT INTO BUDGET_PERIOD VALUES (2005); -- annual expenditure data CREATE TABLE EXPENDITURE (BP smallint NOT NULL, AMOUNT DECIMAL(15,2)); INSERT INTO EXPENDITURE VALUES (1999, 0.00); INSERT INTO EXPENDITURE VALUES (2000, 1.00); INSERT INTO EXPENDITURE VALUES (2001, 2.00); INSERT INTO EXPENDITURE VALUES (2002, 3.00); INSERT INTO EXPENDITURE VALUES (2003, 4.00); INSERT INTO EXPENDITURE VALUES (2004, NULL); -- calculate past years expenditure as of year: expected to be NULL for 1998, returns 0 SELECT A.BP, SUM( B.AMOUNT ) AS TOTAL_EXPD_ASOF FROM BUDGET_PERIOD A LEFT JOIN EXPENDITURE B ON (A.BP >= B.BP) GROUP BY A.BP ORDER BY A.BP -- same as above using CASE: returns all nulls (?!) SELECT A.BP, SUM( CASE WHEN A.BP >= B.BP THEN B.AMOUNT ELSE NULL END ) FROM BUDGET_PERIOD A, EXPENDITURE B GROUP BY A.BP go Regards, Vladimir

2 1

[MonetDB-users] [Fwd: Re: monetdb, xquery and perl]
by Martin Kersten 01 Mar '09

01 Mar '09

-------- Original Message -------- Subject: Re: monetdb, xquery and perl Date: Fri, 27 Feb 2009 10:35:15 +0100 From: francois guérin <replonk(a)gmail.com> To: Martin.Kersten(a)cwi.nl References: <31b3364e0902230836l140c9d0cs932937c392e8e9d(a)mail.gmail.com> <49A2EBEC.4050306(a)cwi.nl> ok! I'll then try the xrpc requests... is there a way to delete a collection in the database? (deleting documents one by one in a given collection is quite fastidious) and a way to delete a database created by the command Mserver --dbname=<...> ? (the only way i found was to physically delete directories with that same name) Thanks, francois. 2009/2/23 Martin Kersten <Martin.Kersten(a)cwi.nl>: > francois guérin wrote: >> >> Hi, >> I am about to develop an app using monetdb and its xquery part. >> My app will essentially be made of perl code. >> But it seems that perl is only used in monetdb's sql part, isn't? like >> using sql statements to access arrays and rows, but there's no way to >> access xml collections. >> >> Am i right or haven't i read enough of monetdb's documentation? > > You are right. The PHP, Perl, JDBC application interfacing to > the XML/XQuery part is less developed. > > regards, Martin >> >> Thanks a lot, >> >> francois. > >

2 1

[MonetDB-users] [Fwd: Re: Monet DB for RDF databases, BLOB]
by Martin Kersten 26 Feb '09

26 Feb '09

1 0

[MonetDB-users] Set the user of MonetDB in Windows
by Xiaofang Tao 23 Feb '09

23 Feb '09

Hello, I intend to know how can I specify the user of one database on the Monet. Hoping for your reply. Thank you very much. TaoXiaofang

2 1

[MonetDB-users] external functions
by dariuszs 21 Feb '09

21 Feb '09

Hi, Does anybody have an example of how to call external functions written in C? Thank you. Dariusz.

2 1

[MonetDB-users] Corrupt DB - missing bat files
by David Svenson 19 Feb '09

19 Feb '09

Hi, When importing date with 'COPY INTO' I reached a database corruption state. When I open the server, it comes up fine, but immediately when I start a client the server crashes with this message: # Listening for connection requests on mapi:abacusdb://127.0.0.1:50000/ # AbacusDB/SQL module v2.27.0 loaded >!ERROR: GDKload: cannot open: name=01\106, ext=head !OS: The system cannot find the path specified. !ERROR: GDKload: failed name=01\106, ext=head The bat files for 01\106 that are mentioned in the error message are missing from the dbfarm. Does the fact that they are supposed to be in folder 01 mean that these are system tables? Can I recover from this state, or has my current database become unrecoverably corrupt? -- View this message in context: http://www.nabble.com/Corrupt-DB---missing-bat-files-tp22096097p22096097.ht… Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] Please, HELP me!
by Xiaofang Tao 18 Feb '09

18 Feb '09

Hello, my friends： I have just installed the latest version of MonetDB5 (SQL) on windows, 32bit system. When I start the client (start-programs-MonetDB5-MonetDB SQL Client), after I start the server (start-programs-MonetDB5-MonetDB SQL Server), the client window prompted start with sql, and I can not use the command like -d. -f, -H, -u and so on, moreover, I can not specify the user or the schema. If I want to use these commands and to specify the user and the schema, what I should do. Hoping for your reply! Best regards. Tao Xiaofang

1 0

[MonetDB-users] Please, HELP me!
by taoxiaofang 18 Feb '09

18 Feb '09

Hello, my friends： I have just installed the latest version of MonetDB5 (SQL) on windows, 32bit system. When I start the client (start-programs-MonetDB5-MonetDB SQL Client), after I start the server (start-programs-MonetDB5-MonetDB SQL Server), the client window prompted start with sql, and I can not use the command like -d. -f, -H, -u and so on, moreover, I can not specify the user or the schema. If I want to use these commands and to specify the user and the schema, what I should do. Hoping for your reply! Best regards. 2009-02-18 taoxiaofang

1 0

[MonetDB-users] Java 1.6 and MonetDB JDBC driver
by Fabian Groffen 17 Feb '09

17 Feb '09

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6795561 The MonetDB JDBC driver is affected by above bug, which manifests itself as an IndexOutOfBoundsException thrown somewhere deep from the heart of the driver in the mcl StartOfHeaderParser. The only workaround is to use JDK 1.4 or 1.5 if you hit this problem.

2 1

[MonetDB-users] sql - division
by dariuszs 13 Feb '09

13 Feb '09

Hello, How do I modify this sql to get correct answer? sql> select 10/100; I got 0 instead of 0.10 Thanks. Dariusz.

2 2

[MonetDB-users] Challenge string is not valid
by David Svenson 12 Feb '09

12 Feb '09

Hi, While working with MonetDB, I suddenly reached some sort of state in which I constantly get: "Challenge string is not valid" Whenever I try to connect to MonetDB. I am using the Mapi libraries, and get the same error if I use mclient.exe. I can't figure out how I reached this state and how I'm supposed to get out of it. Please help. Thanks, David. -- View this message in context: http://www.nabble.com/Challenge-string-is-not-valid-tp21976173p21976173.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] copy/select * from /into - very slow
by dariuszs 09 Feb '09

09 Feb '09

Hi, Could you please help me here. I've got 300 fields, 100 million records, 800 bytes per row in a table. Exporting 60 fields into a table takes about 1 minute and selecting * from table takes very long time, and I don't see lots of cpu usage when exporting all records. Thanks. Dariusz.

2 9

[MonetDB-users] Algebra Mode vs sql debug 16384
by Ross Bates 06 Feb '09

06 Feb '09

Hi All - can someone help me clarify the difference between Algebra "-G" mode and setting SQL debug equal to 16384 in the monetdb config file? With an older version of MonetDb I set the sql_debug=16384 in the config file so I could take advantage of Algebra mode when starting the server via mero. Today I compiled the Nov08-SP2 release and ran some tests using sql_debug=0 vs sql_debug=16384. I assumed they would do the exact same since Algebra mode is now the default. They returned very different results however. For SQL statements with a large number of predicates (100+), performance is about the same. But for anything smaller the query speed is much worse (10x avg) when I don't have sql_debug=16384 turned on. The query trace files are also different when I run the exact same query with and without the 16384 flag. I made sure to restart the server process in between. Also, below is the latest output from mserver --version: Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/ for further information Configured for prefix: /home/ross/monetNov08sp2 Libraries: openssl: OpenSSL 0.9.8g 19 Oct 2007 (compiled with OpenSSL 0.9.8g 19 Oct 2007) Compiled by: ross@home Compilation: gcc -O2 -std=c99 -fgnu89-inline -O6 -fomit-frame-pointer -finline-functions -falign-loops=4 -falign-jumps=4 -falign-functions=4 -fexpensive-optimizations -funroll-loops -frerun-cse-after-loop -frerun-loop-opt -ftree-vectorize Linking : /usr/bin/ld Any thoughts? Best Regards, Ross

5 4

[MonetDB-users] sql - string functions
by dariuszs 06 Feb '09

06 Feb '09

Hi, Just wondering why - strings functions seem to be very slow eg. 'select substring(field1,1,10) || substring(field2,1,20) as field3 from table1'. Thanks for help. Dariusz.

1 0

[MonetDB-users] Fedora 9 RPMs in Stable DailyBuild directory
by Hans van Rijswijk (DT) 06 Feb '09

06 Feb '09

Dear MonetDB users, In the release directories (http://monetdb.cwi.nl/downloads/Fedora/9/x86_64/) I can find RPMs for Fedora 9. Why can't I find Fedora 9 RPMs in the Stable DailyBuild directory (http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./200 90206/RPMs/) ? I needs to make a distribution including the latest Stable MonetDB and including Fedora 9 RPMs. Regards, Hans van Rijswijk

2 1

[MonetDB-users] Re naming a column via MAL/SQL
by Alfred Nordman 03 Feb '09

03 Feb '09

Hi guys, Is there a way to rename an existing column within a table via SQL? (the RENAME COLUMN syntax is not supported as it seems) and if not, can i do it via MAL? I tried to use MAL as follows, but after i do, the DB seems to get corrupt when i try to access the specific table. include sql; sql.init(); bat1 := sql.bind("sys","t1","Field1",0); bat.setName(bat1,"Field2"); bat.setPersistent(bat1,true); bot.commit; Thanks & Best Regards... -- View this message in context: http://www.nabble.com/Renaming-a-column-via-MAL-SQL-tp21807788p21807788.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] Logs & Corrupt databases
by David Svenson 02 Feb '09

02 Feb '09

Hi, Every once in a while I get this message in the server: !ERROR: Please move away <log folder> and its corresponding dbfarm. !mvc_init: unable to create system tables When I delete the logs from the log folder, the server starts normally and the database is ok. Any ideas on what the reason for that might be? Is it possible to disable logging altogether? Thanks, David. -- View this message in context: http://www.nabble.com/Logs---Corrupt-databases-tp21792046p21792046.html Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] Searching for possibilities to reduce the fluctuation in insertion times.
by John van Schie (DT) 02 Feb '09

02 Feb '09

Hi, We use MonetDB v4.26.4 (Nov 2008-sp2) for our application with the XQuery frontend. Often, queries are executed very fast, but sometimes, execution of the same query takes significantly more time. For example: We insert a XML document of 1.4Mb. with a maximum depth of 4. The document is structured as follows: <rootelement> <item> <child1> <sha1>sha1-value</sha1> <md5>md5-value</md5> </child1> </item> ... </rootelement> We have 8k 'item' elements. Often the insertion of this document is performed in ~0.5s, which is fine. But when the system is performing IO (e.g. copying a large file), the insertion can take up to 6 minutes. So I have two questions: 1) What is the cause of this large difference in execution times; 2) Is there a way to configure MonetDB (or the server) in such a way that the difference between the 'fast' and 'slow' is smaller. Regards, John van Schie

3 7

[MonetDB-users] sql performance - in vs between
by dariuszs 01 Feb '09

01 Feb '09

Hi, I know it's probably a trivial question but after looking at the following trace - is there a way to make sql 'in' behave like sql 'between', make *algebra.semijoin behave like **algebra.uselect*? Thanks for you help. Dariusz. :0[0]:W Strace select f25,count(*) from trw100 where f25 in ('F','M') group by f25; :13172[0]:R [ 0 usec mdb.setTimer(true); ] [ 15000 usec _3 := sql.bind("sys","trw100","f25",0); ] [ 16000 usec _8 := sql.bind("sys","trw100","f25",1); ] [ 16000 usec _10 := algebra.kunion(<tmp_5213>:bat[:oid,:str][96240405],<tmp_20266>:bat[:oid,:str][0]); ] [ 0 usec _3 := nil; ] [ 15000 usec _8 := nil; ] [ 16000 usec _11 := sql.bind("sys","trw100","f25",2); ] [ 15000 usec _13 := algebra.kdifference(<tmp_20616>[96240405],<tmp_61>:bat[:oid,:str][0]); ] [ 0 usec _10 := nil; ] [ 16000 usec _14 := algebra.kunion(<tmp_20617>[96240405],<tmp_61>:bat[:oid,:str][0]); ] [ 16000 usec _13 := nil; ] [ 0 usec _11 := nil; ] [ 15000 usec _15 := sql.bind_dbat("sys","trw100",1); ] [ 16000 usec _16 := bat.reverse(<tmp_20253>:bat[:oid,:oid][0]); ] [ 16000 usec _15 := nil; ] [ 0 usec _17 := algebra.kdifference(<tmp_20616>[96240405],<~tmp_20253>[0]); ] [ 15000 usec _16 := nil; ] [ 16000 usec _18 := bat.reverse(<tmp_20617>[96240405]); ] [ 0 usec _17 := nil; ] [ 15000 usec _19 := bat.new(nil,nil); ] [ 16000 usec bat.append(<tmp_20506>:bat[:oid,:str][1],"F",true); ] [ 0 usec bat.append(<tmp_20506>:bat[:oid,:str][2],"M",true); ] [ 16000 usec _25 := bat.reverse(<tmp_20506>:bat[:oid,:str][2]); ] [ 15000 usec _19 := nil; ] [ 10782000 usec _26 := *algebra.semijoin(<~tmp_20617>[96240405],<~tmp_20506>[2]); *] [ 15000 usec _18 := nil; ] [ 16000 usec _25 := nil; ] [ 15000 usec _27 := bat.reverse(<tmp_20620>[93838869]); ] [ 0 usec _26 := nil; ] [ 16000 usec _29 := algebra.markT(<~tmp_20620>[93838869],0@0); ] [ 16000 usec _27 := nil; ] [ 0 usec _30 := bat.reverse(<tmp_20617>[93838869]); ] [ 15000 usec _29 := nil; ] [ 453000 usec _31 := algebra.join(<~tmp_20617>[93838869],<tmp_20616>[96240405]); ] [ 63000 usec _30 := nil; ] [ 0 usec _14 := nil; ] [ 953000 usec (ext37,grp35) := group.new(<tmp_20621>[93838869]); ] [ 16000 usec _34 := bat.mirror(<tmp_20542>[2]); ] [ 15000 usec ext37 := nil; ] [ 0 usec _35 := algebra.join(<tmp_20620>[2],<tmp_20621>[93838869]); ] [ 47000 usec _31 := nil; ] [ 344000 usec _36 := aggr.count(<tmp_20616>[93838869],<tmp_20616>[93838869],<tmp_20620>[2]); ] [ 31000 usec grp35 := nil; ] [ 16000 usec _34 := nil; ] [ 15000 usec _37 := sql.resultSet(2,1,<tmp_20544>[2]); ] [ 0 usec sql.rsColumn(0,"sys.trw100","f25","varchar",1,0,<tmp_20544>[2]); ] [ 16000 usec _35 := nil; ] [ 16000 usec sql.rsColumn(0,"sys.trw100","count_f25","wrd",64,0,<tmp_20545>:bat[:oid,:wrd][2]); ] [ 15000 usec _36 := nil; ] [ 0 usec _45 := io.stdout(); ] &1 0 2 2 2 % sys.trw100, sys.trw100 # table_name % f25, count_f25 # name % varchar, wrd # type % 1, 8 # length [ "F", 29577383 ] [ "M", 64261486 ] [ 16000 usec sql.exportResult(137554336,0,""); ] [ 13172000 usec user.s0_2("F","M"); ] :13203[0]:R :39062[0]:W Strace select f25,count(*) from trw100 where f25 between 'F' and 'M' group by f25; :43156[0]:R [ 0 usec mdb.setTimer(true); ] [ 16000 usec _3 := sql.bind("sys","trw100","f25",0); ] [ 1750000 usec _8 := *algebra.uselect(<tmp_5213>:bat[:oid,:str][96240405],"F","M",true,true);* ] [ 15000 usec _10 := sql.bind("sys","trw100","f25",1); ] [ 0 usec _12 := algebra.uselect(<tmp_20266>:bat[:oid,:str][0],"F","M",true,true); ] [ 16000 usec _13 := algebra.kunion(<tmp_20620>[93838869],<tmp_20621>[0]); ] [ 16000 usec _8 := nil; ] [ 15000 usec _12 := nil; ] [ 0 usec _14 := sql.bind("sys","trw100","f25",2); ] [ 16000 usec _16 := algebra.kdifference(<tmp_20616>[93838869],<tmp_61>:bat[:oid,:str][0]); ] [ 16000 usec _13 := nil; ] [ 0 usec _17 := algebra.uselect(<tmp_61>:bat[:oid,:str][0],"F","M",true,true); ] [ 15000 usec _18 := algebra.kunion(<tmp_20621>[93838869],<tmp_20617>[0]); ] [ 16000 usec _16 := nil; ] [ 15000 usec _17 := nil; ] [ 0 usec _19 := sql.bind_dbat("sys","trw100",1); ] [ 16000 usec _20 := bat.reverse(<tmp_20253>:bat[:oid,:oid][0]); ] [ 16000 usec _19 := nil; ] [ 0 usec _21 := algebra.kdifference(<tmp_20616>[93838869],<~tmp_20253>[0]); ] [ 15000 usec _18 := nil; ] [ 16000 usec _20 := nil; ] [ 16000 usec _23 := algebra.markT(<tmp_20617>[93838869],0@0); ] [ 0 usec _21 := nil; ] [ 15000 usec _24 := bat.reverse(<tmp_20616>[93838869]); ] [ 16000 usec _23 := nil; ] [ 0 usec _25 := algebra.kunion(<tmp_5213>:bat[:oid,:str][96240405],<tmp_20266>:bat[:oid,:str][0]); ] [ 15000 usec _3 := nil; ] [ 16000 usec _10 := nil; ] [ 16000 usec _26 := algebra.kdifference(<tmp_20617>[96240405],<tmp_61>:bat[:oid,:str][0]); ] [ 0 usec _25 := nil; ] [ 15000 usec _27 := algebra.kunion(<tmp_20621>[96240405],<tmp_61>:bat[:oid,:str][0]); ] [ 16000 usec _26 := nil; ] [ 0 usec _14 := nil; ] [ 453000 usec _28 := algebra.join(<~tmp_20616>[93838869],<tmp_20617>[96240405]); ] [ 31000 usec _24 := nil; ] [ 16000 usec _27 := nil; ] [ 922000 usec (ext34,grp32) := group.new(<tmp_20621>[93838869]); ] [ 15000 usec _31 := bat.mirror(<tmp_20542>[2]); ] [ 16000 usec ext34 := nil; ] [ 0 usec _32 := algebra.join(<tmp_20620>[2],<tmp_20621>[93838869]); ] [ 47000 usec _28 := nil; ] [ 344000 usec _33 := aggr.count(<tmp_20617>[93838869],<tmp_20617>[93838869],<tmp_20620>[2]); ] [ 31000 usec grp32 := nil; ] [ 16000 usec _31 := nil; ] [ 15000 usec _34 := sql.resultSet(2,1,<tmp_20544>[2]); ] [ 16000 usec sql.rsColumn(1,"sys.trw100","f25","varchar",1,0,<tmp_20544>[2]); ] [ 0 usec _32 := nil; ] [ 15000 usec sql.rsColumn(1,"sys.trw100","count_f25","wrd",64,0,<tmp_20545>:bat[:oid,:wrd][2]); ] [ 16000 usec _33 := nil; ] [ 16000 usec _42 := io.stdout(); ] &1 1 2 2 2 % sys.trw100, sys.trw100 # table_name % f25, count_f25 # name % varchar, wrd # type % 1, 8 # length [ "F", 29577383 ] [ "M", 64261486 ] [ 15000 usec sql.exportResult(137554336,1,""); ] [ 4094000 usec user.s1_2("F","M"); ] :43187[0]:R :253953[0]:W Connection closed

3 3

[MonetDB-users] Building 64 bits
by David Svenson 01 Feb '09

01 Feb '09

Hi, I followed the instructions for building MonetDB in 64-bit, however, after finishing the build process and trying to run the server, I get this error message: !FATAL: BBPinit: database created with incompatible server: expected pointer size 8, got 4, expected OID size 4, got 4. I added "bits=64" to all my nmakes, but I guess something is still missing. Are there any other flags I need to raise in the build process that I might have missed? Thanks, David. -- View this message in context: http://www.nabble.com/Building-64-bits-tp21772757p21772757.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] mclient - copy into problem
by dariuszs 31 Jan '09

31 Jan '09

Hi, Please verify this - when you output a file from the table with 'copy... into...' and terminate that client - the output file is still being created even though the client was terminated. Thanks. Dariusz.

5 16

[MonetDB-users] Running into swap during large dataset loading
by Ukyo Virgden 30 Jan '09

30 Jan '09

Hi, I'm tring to build a database with around 1B rows. I'm testing on a 64bit AMD Opteron 2200 dual core x 2 CPU system with 16GB mem and 16G swap connected to an external fiber storage. I've got a set of datafiles around 8GB. each with 100M records. I'm running the following commands. copy 300000000 records into test_table from 'datafile-1.csv','datafile-2.csv','datafile-3.csv'; copy 300000000 records into test_table from 'datafile-4.csv','datafile-5.csv','datafile-6.csv'; copy 300000000 records into test_table from 'datafile-7.csv','datafile-8.csv','datafile-9.csv'; The first command completes around 15mins. The second also completes around 15min. However the the third command forces the system to swap and at some point cause the system to crash. I've also tried by NOT giving the record count i.e. copy into test_table from 'datafile-X.csv' which were not only slowe but also resulted in the same swap outage crash. I need to load around 3 billion records. Any thoughts on what might be the problem here? Thanks in advance, Regards, Ukyo

1 0

[MonetDB-users] sql_logs
by dariuszs 29 Jan '09

29 Jan '09

Hi, How do I read what's in the sql_logs directory files? Thanks. Dariusz.

3 5

[MonetDB-users] server crash/sql prompt
by dariuszs 29 Jan '09

29 Jan '09

Hi, Can somebody help me with this one. I was running a very long query, decided to terminate that job. Now when I try to get the client all I see is blinking prompt. How do I get back the sql prompt? Thanks. Dariusz.

5 26

[MonetDB-users] distinct values limit
by dariuszs 26 Jan '09

26 Jan '09

Hi, Is there a limit on how many distinct values you can have in a column. I've created a table with 100,000,000 records and one column (varchar (16)) has 2,600,000 distinct values so when I run the count on that column - it runs forever. When I watch memory usage it goes to few gigabytes and then it drops to almost nothing, then it goes to few gigs and then it drops. Thanks. Dariusz.

1 0

Re: [MonetDB-users] predictable crashes of MonetDB5 SQL server under Win32bit
by Jens Oehlschlägel 26 Jan '09

26 Jan '09

Thanks Stefan, > with loading ~1.5 GB on a 32 bit machine, there is a reasonable chance that > the loading process (that needs to be able to address all data concurrently) > comes close to or even hits the 32-bit (i.e., max. 4 GB, possibly only 2 GB) > address space limit. Does this limitation refers to the size of the data that I am adding per COPY INTO (was 0.2 GB) or the final size of the table (was at 1.9). Assuming the latter this would give a hard limit for table size under Win32. If this is the case, a workaround could be partioning the data into several tables. What is more efficient in MonetDB: chunks of rows at full columns width or chunks of columns at full row length? > Would it be possible that you could provide us with your data? > That would help us a lot to analyze, locate and hopefully fix the crash... I give you access by seperate mail, you can also create the test data file using the R script snippet in my previous mail. Have a nice weekend Jens

5 5

[MonetDB-users] memory release
by dariuszs 26 Jan '09

26 Jan '09

Hi, MonetDB-SQL server does not release the memory after loading data into table (loaded 40GB) terminating client session (closed the window waiting for count to complete) Also after restarting the server it takes very long time for the client to get the prompt Why is that? Thanks. Dariusz.

1 0

[MonetDB-users] Announcement: New bug fix release of MonetDB suite
by Sjoerd Mullender 23 Jan '09

23 Jan '09

The MonetDB team at CWI is pleased to announce the Nov2008-SP2 bug fix release of the MonetDB suite of programs. This is a bug fix release of the Nov2008 release. More information on this release can be found at <http://monetdb.cwi.nl/projects/monetdb/Development/Releases/Nov2008/#Nov200…>. The sources and binaries are available at <http://monetdb.cwi.nl/projects/monetdb/Download/>. -- Sjoerd Mullender

1 0

[MonetDB-users] predictable crashes of MonetDB5 SQL server under Win32bit
by Jens Oehlschlägel 23 Jan '09

23 Jan '09

Hi, I experience predictable crashes of MonetDB5 SQLserver under Win32bit when COPY INTO in batches of 1 Mio more than ~9Mio rows, which is ~1.5 GB DB size. Any settings to avoid that? Is this a bug? Is MonetDB size limited to available RAM? Or is this a known limit just under Win32? Logs of sql client error messages and details for replication below. Regards Jens Oehlschlägel Lenovo Thinkcentre with 3 GB RAM under Windows XP Professional SP2 RAM consumption of m5server.exe during load ~300MB according to task manager Free RAM 2.1 GB Size of dbfarm finally 1.9 GB DiskFreeSpace 1.8 TB [System Summary] Item Value OS Name Microsoft Windows XP Professional Version 5.1.2600 Service Pack 2 Build 2600 OS Manufacturer Microsoft Corporation System Name GEMUN-38396-10- System Manufacturer LENOVO System Model 6073AA7 System Type X86-based PC Processor x86 Family 6 Model 15 Stepping 11 GenuineIntel ~2992 Mhz Processor x86 Family 6 Model 15 Stepping 11 GenuineIntel ~2992 Mhz BIOS Version/Date LENOVO 2RKT41AUS, 3/20/2008 SMBIOS Version 2.5 Windows Directory C:\WINDOWS System Directory C:\WINDOWS\system32 Boot Device \Device\HarddiskVolume1 Locale Germany Hardware Abstraction Layer Version = "5.1.2600.3023 (xpsp_sp2_qfe.061030-0020)" Time Zone W. Europe Standard Time Total Physical Memory 3,072.00 MB Available Physical Memory 1.70 GB Total Virtual Memory 2.00 GB Available Virtual Memory 1.96 GB Page File Space 5.79 GB Page File D:\pagefile.sys sql>START TRANSACTION; 0 tuples Timer 0.000 msec 0 rows sql>COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/d atabasetest1mio.csv' USING DELIMITERS ','; Rows affected 1000000 Timer 67515.000 msec 0 rows sql>COMMIT; 0 tuples Timer 1255.000 msec 0 rows sql>START TRANSACTION; 0 tuples Timer 0.000 msec 0 rows sql>COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/d atabasetest1mio.csv' USING DELIMITERS ','; Rows affected 1000000 Timer 79141.000 msec 0 rows sql>COMMIT; 0 tuples Timer 6041.000 msec 0 rows sql>START TRANSACTION; 0 tuples Timer 0.000 msec 0 rows sql>COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/d atabasetest1mio.csv' USING DELIMITERS ','; Rows affected 1000000 Timer 74227.000 msec 0 rows sql>COMMIT; 0 tuples Timer 6943.000 msec 0 rows sql>START TRANSACTION; 0 tuples Timer 0.000 msec 0 rows sql>COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/d atabasetest1mio.csv' USING DELIMITERS ','; Rows affected 1000000 Timer 73084.000 msec 0 rows sql>COMMIT; 0 tuples Timer 6599.000 msec 0 rows sql>START TRANSACTION; 0 tuples Timer 0.000 msec 0 rows sql>COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/d atabasetest1mio.csv' USING DELIMITERS ','; Rows affected 1000000 Timer 67158.000 msec 0 rows sql>COMMIT; 0 tuples Timer 6520.000 msec 0 rows sql>START TRANSACTION; 0 tuples Timer 18529.000 msec 0 rows sql>COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/d atabasetest1mio.csv' USING DELIMITERS ','; Rows affected 1000000 Timer 65106.000 msec 0 rows sql>COMMIT; 0 tuples Timer 5184.000 msec 0 rows sql>START TRANSACTION; 0 tuples Timer 0.000 msec 0 rows sql>COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/d atabasetest1mio.csv' USING DELIMITERS ','; Rows affected 1000000 Timer 71136.000 msec 0 rows sql>COMMIT; 0 tuples Timer 9104.000 msec 0 rows sql>START TRANSACTION; 0 tuples Timer 21021.000 msec 0 rows sql>COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/d atabasetest1mio.csv' USING DELIMITERS ','; Rows affected 1000000 Timer 69761.000 msec 0 rows sql>COMMIT; 0 tuples !MALException: MT_mmap: MapViewOfFileEx(1558, 2, 0, 0, 14680064, 0) failed !OS: Not enough storage is available to process this command. !ERROR: MT_mmap: MapViewOfFileEx(1554, 2, 0, 0, 14680064, 0) failed !OS: Not enough storage is available to process this command. !ERROR: GDKload: cannot mmap(): name=15\1511, ext=tail !OS: Not enough storage is available to process this command. !ERROR: GDKload: failed name=15\1511, ext=tail !ERROR: MT_mmap: MapViewOfFileEx(1558, 2, 0, 0, 14680064, 0) failed !OS: Not enough storage is available to process this command. !ERROR: MT_mmap: MapViewOfFileEx(1554, 2, 0, 0, 14680064, 0) failed !OS: Not enough storage is available to process this command. !ERROR: GDKload: cannot mmap(): name=15\1511, ext=tail !OS: Not enough storage is available to process this command. !ERROR: GDKload: failed name=15\1511, ext=tail !ERROR: MT_mmap: MapViewOfFileEx(1558, 2, 0, 0, 14680064, 0) failed !OS: Not enough storage is available to process this command. !ERROR: MT_mmap: MapViewOfFileEx(1554, 2, 0, 0, 14680064, 0) failed !OS: Not enough storage is available to process this command. !ERROR: GDKload: cannot mmap(): name=15\1511, ext=tail !OS: Not enough storage is available to process this command. !ERROR: GDKload: failed name=15\1511, ext=tail !ERROR: MT_mmap: MapViewOfFileEx(1558, 2, 0, 0, 34603008, 0) failed !OS: Not enough storage is available to process this command. !ERROR: MT_mmap: MapViewOfFileEx(1554, 2, 0, 0, 34603008, 0) failed !OS: Not enough storage is available to process this command. !ERROR: GDKload: cannot mmap(): name=11\1117, ext=tail !OS: Not enough storage is available to process this command. !ERROR: GDKload: failed name=11\1117, ext=tail !ERROR: MT_mmap: MapViewOfFileEx(1558, 2, 0, 0, 69206016, 0) failed !OS: Not enough storage is available to process this command. !ERROR: MT_mmap: MapViewOfFileEx(1554, 2, 0, 0, 70254592, 0) failed !OS: Not enough storage is available to process this command. !ERROR: MT_mmap: MapViewOfFileEx(1558, 2, 0, 0, 70254592, 0) failed !OS: Not enough storage is available to process this command. !ERROR: MT_mmap: MapViewOfFileEx(1554, 2, 0, 0, 70254592, 0) failed !OS: Not enough storage is available to process this command. !ERROR: GDKload: cannot mmap(): name=15\1534, ext=tail !OS: Not enough storage is available to process this command. !ERROR: GDKload: failed name=15\1534, ext=tail 0 tuples Timer 9276.000 msec 0 rows sql>START TRANSACTION; 0 tuples Timer 0.000 msec 0 rows sql>COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/d atabasetest1mio.csv' USING DELIMITERS ','; Rows affected 1000000 Timer 96373.000 msec 0 rows sql>COMMIT; MAPI = voc@localhost:50000 ACTION= read_line QUERY = COMMIT; ERROR = Connection terminated Timer 44089.000 msec 0 rows -- restarting the crashed server looks good, but # MonetDB server v5.8.2, based on kernel v1.26.2 # Serving database 'demo', using 1 thread # Compiled for i686-pc-win32/32bit with 32bit OIDs dynamically linked # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008- MonetDB B.V., all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see D:\MonetDB\MonetDB5\etc\monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.26.2 loaded -- when starting mclient the server says >!ERROR: MT_mmap: MapViewOfFileEx(1528, 2, 0, 0, 69206016, 0) failed !OS: Not enough storage is available to process this command. #GDKmmap(69206016) fail => BBPtrim(enter) usage[mem=165285248,vm=1495793664] #GDKmmap(69206016) fail => BBPtrim(ready) usage[mem=165284840,vm=574619648] #GDKmmap: recovery ok. Continuing.. -- replication instructions -- log in to freshly installed MonetDB and create user CREATE USER "voc" WITH PASSWORD 'voc' NAME 'VOC Explorer' SCHEMA "sys"; CREATE SCHEMA "voc" AUTHORIZATION "voc"; ALTER USER "voc" SET SCHEMA "voc"; -- log in as voc and execute the following statements CREATE TABLE databasetest ( "id" INTEGER , "ubyte1" SMALLINT , "ubyte2" SMALLINT , "ubyte3" SMALLINT , "ubyte4" SMALLINT , "ubyte5" SMALLINT , "ubyte6" SMALLINT , "ubyte7" SMALLINT , "ubyte8" SMALLINT , "ubyte9" SMALLINT , "ubyte10" SMALLINT , "ubyte11" SMALLINT , "ubyte12" SMALLINT , "ubyte13" SMALLINT , "ubyte14" SMALLINT , "ubyte15" SMALLINT , "smallint1" SMALLINT , "smallint2" SMALLINT , "smallint3" SMALLINT , "smallint4" SMALLINT , "smallint5" SMALLINT , "smallint6" SMALLINT , "smallint7" SMALLINT , "smallint8" SMALLINT , "smallint9" SMALLINT , "smallint10" SMALLINT , "smallint11" SMALLINT , "smallint12" SMALLINT , "smallint13" SMALLINT , "smallint14" SMALLINT , "smallint15" SMALLINT , "float1" DOUBLE , "float2" DOUBLE , "float3" DOUBLE , "float4" DOUBLE , "float5" DOUBLE , "float6" DOUBLE , "float7" DOUBLE , "float8" DOUBLE , "float9" DOUBLE , "float10" DOUBLE , "float11" DOUBLE , "float12" DOUBLE , "float13" DOUBLE , "float14" DOUBLE , "float15" DOUBLE , "float16" DOUBLE , "float17" DOUBLE , "float18" DOUBLE , "float19" DOUBLE , "float20" DOUBLE ) ; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; START TRANSACTION; COPY 1000000 OFFSET 2 RECORDS INTO "voc"."databasetest" FROM 'F:/Periscope/databasetest1mio.csv' USING DELIMITERS ','; COMMIT; -- the file databasetest1mio.csv can be created with the following R script # this is an R comment and you find R at CRAN.r-project.org filename <- "F:/Periscope/databasetest1mio.csv" # create 1Mio rows in 100 batches of 10,000, total of 174MB csv file, takes ~2 minutes system.time({ b <- 1000L # number of batches n <- 1000L # batch size csvfile <- file(filename, open="w") for (i in 1:b){ x <- data.frame( id = ((i-1L)*n+1L):(i*n) # 4-byte integer record identifier , ubyte1 = sample(0:35, n, TRUE, prob=NULL) # unsigned byte columns (1 byte) , ubyte2 = sample(0:1, n, TRUE, prob=NULL) , ubyte3 = sample(0:3, n, TRUE, prob=NULL) , ubyte4 = sample(0:9, n, TRUE, prob=NULL) , ubyte5 = sample(0:24, n, TRUE, prob=NULL) , ubyte6 = sample(0:99, n, TRUE, prob=NULL) , ubyte7 = sample(-127:127, n, TRUE, prob=NULL) , ubyte8 = sample(0:1, n, TRUE, prob=c(0.75, 0.25)) , ubyte9 = sample(0:1, n, TRUE, prob=c(0.9,0.1)) , ubyte10 = sample(0:1, n, TRUE, prob=c(0.96,0.04)) , ubyte11 = sample(0:1, n, TRUE, prob=c(0.99,0.01)) , ubyte12 = sample(0:1, n, TRUE, prob=c(0.996,0.004)) , ubyte13 = sample(0:1, n, TRUE, prob=c(0.999,0.001)) , ubyte14 = sample(0:1, n, TRUE, prob=c(0.9996,0.0004)) , ubyte15 = sample(0:1, n, TRUE, prob=c(0.9999,0.0001)) , smallint1 = sample(1:4000, n, TRUE, prob=NULL) # smallint columns (2 bytes) , smallint2 = sample(1:2, n, TRUE, prob=NULL) , smallint3 = sample(1:4, n, TRUE, prob=NULL) , smallint4 = sample(1:10, n, TRUE, prob=NULL) , smallint5 = sample(1:25, n, TRUE, prob=NULL) , smallint6 = sample(1:100, n, TRUE, prob=NULL) , smallint7 = sample(1:256, n, TRUE, prob=NULL) , smallint8 = sample(1:1000, n, TRUE, prob=NULL) , smallint9 = sample(1:32000, n, TRUE, prob=NULL) , smallint10 = sample(0:1, n, TRUE, prob=c(0.75, 0.25)) , smallint11 = sample(0:1, n, TRUE, prob=c(0.9,0.1)) , smallint12 = sample(0:1, n, TRUE, prob=c(0.96,0.04)) , smallint13 = sample(0:1, n, TRUE, prob=c(0.99,0.01)) , smallint14 = sample(0:1, n, TRUE, prob=c(0.996,0.004)) , smallint15 = sample(0:1, n, TRUE, prob=c(0.999,0.001)) , float1 = round(runif(n, 0, 100), 1) # float columns (4 bytes) , float2 = round(runif(n, 0, 100), 1) , float3 = round(runif(n, 0, 100), 1) , float4 = round(runif(n, 0, 100), 1) , float5 = round(runif(n, 0, 100), 1) , float6 = round(runif(n, 0, 100), 1) , float7 = round(runif(n, 0, 100), 1) , float8 = round(runif(n, 0, 100), 1) , float9 = round(runif(n, 0, 100), 1) , float10 = round(runif(n, 0, 100), 1) , float11 = round(runif(n, 0, 100), 1) , float12 = round(runif(n, 0, 100), 1) , float13 = round(runif(n, 0, 100), 1) , float14 = round(runif(n, 0, 100), 1) , float15 = round(runif(n, 0, 100), 1) , float16 = round(runif(n, 0, 100), 1) , float17 = round(runif(n, 0, 100), 1) , float18 = round(runif(n, 0, 100), 1) , float19 = round(runif(n, 0, 100), 1) , float20 = round(runif(n, 0, 100), 1) ) write.table(x, file=csvfile, row.names=FALSE, col.names=(i==1L), append=i!=1L, dec=".", sep=",") } close(csvfile) })

3 2

Re: [MonetDB-users] copy into table
by Jens Oehlschlägel 22 Jan '09

22 Jan '09

A related feature for bulk import/export that would indeed make sense AND should be easy to implement is: exchange with flat binary files. This requires - the file to be set up with the correct byte-order - the file to contain only data types that have a fixed-width binary representation (like FLOAT, SINGLE, INTEGER, SMALLINT etc) but not variable widths (like VARCHAR) Such file would neither require record_seperator nor tuple_seperator. The only challenge is how to represent NULL, and for this you could turn to the definitions of NA (Not Available atomic value) used in the popular statistical interpreter R (www.r-project.org) for its native data types (double, integer) and for other data types you could use the NA conventions implemented in R's flat file package 'ff' (single, signed and unsigned integers, smallints, bytes, etc, see documentation for 'vmode'). Even dates could be represented unambiguously using the double representation of the POSIXct class in R. For the SQL interface you could simply define binary files by setting tuple_seperator and record_seperator to '' the empty string. If you use a trick and allow COPY INTO not only to accept a table_name but also a view_name - where the view represents a subset of the columns of one table - such an interface could even be used to populate a table that has columns which are not in the binary file, say the table has a varchar column or, exchange data column by column with systems that cannot work with mixed-type binary files (such as ff which currently can only handle homogeneous atomic files). Beside the obvious benefits - for example exporting large data to R for analysis purposes - such a file interface could also be used to create artifical data in R with certain statistical features for datawarehouse test loads (say exploring MonetDBs 'indexing' performance for distributions with certain entropy etc). Greetings Jens Oehlschlägel (one of the ff authors)

1 0

[MonetDB-users] copy into table
by dariuszs 21 Jan '09

21 Jan '09

Hello, Is there any way to copy flat file into the table not using delimiters? Thanks. Dariusz.

4 11

[MonetDB-users] SQL server crash
by dariuszs 20 Jan '09

20 Jan '09

Hi, The SQL server crashes when I try to join 3 tables using 27 bytes key? Can you help me? Thanks. Dariusz.

6 16

[MonetDB-users] Changing locale to support various Datetime formats
by David Svenson 20 Jan '09

20 Jan '09

Is there any recommended way to format dates according to different locales in Windows? I try to do COPY INTOs with dates, but get parsing errors, and I can't find any solution for it because I can't use mtime.str_to_date and such in Windows. Thanks, David. -- View this message in context: http://www.nabble.com/Changing-locale-to-support-various-Datetime-formats-t… Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] Python3.0
by Kees Varekamp 20 Jan '09

20 Jan '09

Hello, Does anyone know if the Python bindings are working with Python 3.0? Thanks

1 0

[MonetDB-users] MonetDb comparison Data Load
by Prometheus WillSurvive 19 Jan '09

19 Jan '09

Dear Developers-Users, First of all I would like to thanks all of your afforts to birng us these DB technologies and a platform for openly discussion. We are consulting big sized customers and deploying their data search technologies. Total size is around 1 petabytes daily . (total of our customers number ) . Their data size starts 1 TB (9.000.000.000 lines cvs) to 100 TB. so we have different size of customer base. We are looking opensource db solutions for a while due to lack of support and technological cumbersome of the current commercial solutions. We have 3 test customer now : 1-1 TB data dailiy read only purposes. 9.000.000.000 cvs file should be load to database . Data should be load to the database max 2 hours. after that comlex crm analitic queries. max 10 concurrent user/queries. database will be daily and wil not grow dailiy. so limit is 1 TB data load. 2-10 TB data daily. same as above .. 3-50 TB data stabil. will not grow. So we are looking for a solutions: *Fast data loading. How fast we load data to the MonetDB ? Is it could be multithreaded data loading possible ? *Is it possible to MPP ? can we add more server to share cpu resources and decrease the load time ? what is the best HW solution for above first customer ? Do we need still fat enterrprise servers or what ? We are also looking for InfoBright, Vertica and also Luciddb. VErtica is not opensource so out of the list. Infobright seems not giving its full power in the opensource.. Also we heard about the MonetDB X100 and interested. After 3 months succefully test we will support monetDB developer if its solved our problems. We can pay . Best Regards W.S

1 0

[MonetDB-users] Are there binary rpm install files for Red Hat Enterprise Linux?
by 小波顾 17 Jan '09

17 Jan '09

_________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

1 0

[MonetDB-users] Copy from file which contains Chinese characters failed.
by 小波顾 17 Jan '09

17 Jan '09

Copy from file which contains Chinese characters failed. My test table is create table t1(a int not null primary key, b varchar(1024)); My test file d:\data.txt contains two lines1,测试12,测试2 I have tried to save the data file using ANSI and UTF8 encoding using Notepad on windows,and tried to start mclient with -Ecp437, -Ecp936 and -Eutf-8 option, neither combination of thefile encoding and mclient encoding can load the file correctlly. When both using utf-8 the following error messages appears:sql>copy into t1 from 'd:\\data.txt' using delimiters ',', '\r\n';!SQLException:importTable:failed to import table!ERROR: value 'ï»?' while parsing 'ï»?,é¡¾å°æ³?!ERROR: 2,è¦ƒæ€? from line 0 field 0 not inserted, expecting type int!ERROR: TABLETload_bulk: read error (after loading 0 records)0 tuples When using other combinations, the following messages appears:sql>copy into t1 from 'd:\\data.txt' using delimiters ',', '\r\n';0 tupleswrite errorsql> The following steps works, but the exported data.txt isn't readable by human. mclient XXXXX -Ecp437 delete from t1;insert into t1 values(1,'²âÊÔ1');insert into t1 values(2,'²âÊÔ2'); copy select * from t1 into 'd:\\data.txt' using delimiters ',' ,'\r\n'; delete from t1;copy into t1 from 'd:\\data.txt' using delimiters ',' ,'\r\n'; _________________________________________________________________ News, entertainment and everything you care about at Live.com. Get it now! http://www.live.com/getstarted.aspx

3 8

[MonetDB-users] Can't connect to remote MonetDB5-SQL Server.
by 小波顾 17 Jan '09

17 Jan '09

I have installed all the rpms in my Fedora9 virtual machine: 2009-01-17 12:07 3,339,229 MonetDB-1.26.2-1.fc9.oid32.i386.rpm2009-01-17 12:07 84,115 MonetDB-client-1.26.2-1.fc9.oid32.i386.rpm2009-01-17 12:37 7,988 MonetDB-SQL-2.26.2-1.fc9.oid32.i386.rpm2009-01-17 12:36 873,636 MonetDB-SQL-server5-2.26.2-1.fc9.oid32.i386.rpm2009-01-14 22:07 5,054,522 MonetDB5-server-5.8.2-1.fc9.oid32.i386.rpmAnd using the following steps recreated the demo database: shell>merovingian ###Start the proxy damo shell>monetdb destroy demo shell>monetdb create demo shell>monetdb start demo Now I can connect to the demo database using mclient -lsql -d demo from the local machine, then I changed "mapi_open=true" in the file /etc/monetdb5.conf and restarted all the server processes, but I still can't connect to databases from remote machines. BTW, how to stop the merovingian process, I used kill:( _________________________________________________________________ Show them the way! Add maps and directions to your party invites. http://www.microsoft.com/windows/windowslive/events.aspx

1 1

[MonetDB-users] SIGBUS on mserver5 after large load.
by Schwarz, Omri 16 Jan '09

16 Jan '09

I have a large database that loads nicely on MonetDB from MySQL when it'd done with INSERTs with one row at a time. But when I group them in sets of 300 or so, for any reason, i.e. INSERT INTO TABLE VALUES (FOO), (BAR) ... ; it develops a memory leak which is not helped by setting gdk_mem_bigsize or gdk_vm_minsize. And when that makes me abort the load, the result is that backend data are corrupted. I can start the database server, but on the first connection attempt, this happens: [pid 18878] read(19, "\0\0", 2) = 2 [pid 18878] close(19) = 0 [pid 18878] stat("bat/02/224.tail", {st_mode=S_IFREG|0600, st_size=254, ...}) = 0 [pid 18878] stat("bat/02/224.tail.new", 0x416d9b00) = -1 ENOENT (No such file or directory) [pid 18878] open("bat/02/224.tail", O_RDONLY) = 19 [pid 18878] read(19, "\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1"..., 254) = 254 [pid 18878] close(19) = 0 [pid 18878] --- SIGBUS (Bus error) @ 0 (0) --- Process 18878 detached [pid 18877] +++ killed by SIGBUS +++ +++ killed by SIGBUS +++ So, a couple questions: 1. how hard would it be to write a backend integritiy checker for the binary association tables? 2. what more can I do to make sure all the changes in the backend are truly atomic ? 3. Are there good prospects for this memory leak being traced and addressed? 4. Can the BAT file format be changed to have more checks on its integrity? (checksums, sentinel variables, etc) 5. Might it make sense for the backend files to have a format relying on existing database libraries? BDB? Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

1 0

[MonetDB-users] Python client strangeness.
by Schwarz, Omri 15 Jan '09

15 Jan '09

when you set "gdk_debug" to 4 (for memory debugging), it causes the server to give empty lists to "select * from tables;" queries coming from Python DB API clients, but not when they come from mclient. Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

1 0

[MonetDB-users] Building clients - php and perl dependencies
by David Svenson 15 Jan '09

15 Jan '09

Hi, I am trying to build MonetDB on windows, and so far I have successfully built 'buildtoos' and 'MonetDB' from the CVS. Now I'm trying to build 'clients' without PHP or PERL, but I'm having difficulties. The documentation says that only Phyton is required, however, I still get build errors related to perl and php dependencies. I do not set HAVE_PERL=1 or HAVE_PHP=1, and therefor expect the build not to rely on them. Am I missing something? Is there something else I need to do in order to build this project? -- View this message in context: http://www.nabble.com/Building-clients---php-and-perl-dependencies-tp214802… Sent from the monetdb-users mailing list archive at Nabble.com.

2 3

[MonetDB-users] MALException:MAL.getAddress:address of 'bat.BKChasReadMode' not found (followup)
by Schwarz, Omri 15 Jan '09

15 Jan '09

I found the problem: the linker was silently balking at creating the .sos and not telling 'make' about it, so a full build was done without the shared libraries. It's a GNU linker/autoconf issue that the MonetDB team can't do much about besides posting this email where Google will find it, but thank you all for helping me diagnose the issue. Now I just have to work with the memory leak I'm seeing. I see I can log all memory operations, but where can I set memory usage to be limited altogether? Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

1 0

[MonetDB-users] MALException:MAL.getAddress:address of 'bat.BKChasReadMode' not found (followup)
by Schwarz, Omri 14 Jan '09

14 Jan '09

Making some progress. Starting with a fresh merovingian process, I did an "strace -fe file,process -p PID] on it. "monetdb create test" did nothing, but "start test" shows it opens the dbfarm directory, and mal_init.mal, but then fails to find any of the MonetDB5 .sos [excriciating scroll included below] because only the .a and .la files got installed there. The build tree is also stripped of .sos. [root@mork MonetDBFast]# ls lib/MonetDB5/lib lib_aggr.a lib_batifthen.la lib_clients.a lib_inet.la lib_mal_scheduler.a lib_pqueue.la lib_streams.la lib_aggr.la lib_batmath.a lib_clients.la lib_inspect.a lib_mal_scheduler.la lib_profiler.a lib_str.la [...] and then gets this for a segfault: [pid 9810] rename("bat/BBP.dir", "bat/BACKUP/BBP.dir") = 0 [pid 9810] open("/usr/local/MonetDBFast/lib/MonetDB5/mal_init.mal", O_RDONLY) = 15 [pid 9810] open("/usr/local/MonetDBFast/lib/MonetDB5/mal_init.mal", O_RDONLY) = 15 [pid 9810] open("/usr/local [...] [pid 9810] open("/usr/local/MonetDBFast/lib/MonetDB5/bin/lib_blob.so", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 9810] open("/usr/local/MonetDBFast/lib/MonetDB5/blob.mal", O_RDONLY) = 16 [pid 9810] open("/usr/local/MonetDBFast/lib/MonetDB5/blob.mal", O_RDONLY) = 16 [pid 9810] --- SIGSEGV (Segmentation fault) @ 0 (0) --- Process 9810 detached [pid 9811] +++ killed by SIGSEGV +++ [pid 9800] --- SIGCHLD (Child exited) @ 0 (0) --- [pid 9800] wait4(-1, Process 9800 suspended Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

1 0

Re: [MonetDB-users] PF: "Group By" work-around no longer working?
by Jan Rittinger 14 Jan '09

14 Jan '09

On Jan 14, 2009, at 11:53, jz@uva wrote: > Hi Jan, > > I was also using the GROUP BY construct. > This is my query: ... > let $qid := tijah:queryall-id($query_nexi, $options) > let $nodes := tijah:nodes($qid) > let $result := for $node in $nodes > return > typeswitch ($node) > case $elem-node as element(*) > return > <result> > <num> { (count($node/$elem-node/preceding::*) + 1) + This snippet of XQuery is to complex: 'count($node/$elem-node/...'. Both $node and $elem-node contain the tijah-result nodes (with type node() and element(*), respectively). What you want to write is 'count($elem-node/...'. > > > string-length(tijah:getINEXPath($node)) } </num> > <file> { data($node/$elem-node/ancestor-or-self::EAD/@FILE) } </ > file> > <title> { data($node/$elem-node/ancestor-or-self::EAD/@TITLE) } > </title> ... > > <score> { tijah:score($qid, $node/$elem-node) } </score> > </result> > default return error ("unexpected node") > > let $total := count($result) > return <results total="{$total}"> { > for $res at $rankGlobal in distinct-values($result/file) > let $cs-group := $result[file = $res] > for $cs-group2 at $rank in $cs-group > where exists($cs-group2) this line is unnecessary as there is always exactly one item bound to $cs-group2. > > and $rank >= 1 this line is unnecessary as the positional variable always starts from 1 (and can never be less). > > and $rank <= 8 > and $rankGlobal >= $start > and $rankGlobal <= $end > order by $cs-group2 cast as xs:integer what do you want here? I very much doubt that you are able to atomize an XML snippet, then cast the string-value into a number and still get something meaningful out. > > return > <out id="{$res}">{ $cs-group2 }</out> > } </results> > > With this query, I get different results in MPS and Alg. I guess MPS treats your order by differently---read incorrectly---by ignoring every character data after the first number which accidently is the content of element num. Perhaps your query already works the way you expect it by replacing the ORDER BY by the following snippet: 'order by number($cs-group2/ num)'. Jan -- Jan Rittinger Lehrstuhl Datenbanken und Informationssysteme Wilhelm-Schickard-Institut für Informatik Eberhard-Karls-Universität Tübingen http://www-db.informatik.uni-tuebingen.de/team/rittinger

3 3

[MonetDB-users] Monetdb source rpm install failed on Fedora 6.
by 小波顾 14 Jan '09

14 Jan '09

Now I am trying to install MonetDB5 on a Fedora 6 virtual machine, the source rpm I am using is MonetDB5-server-5.8.2-1.src.rpm, The following error messages was thrown when I enter rpm -i MonetDB5-server-5.8.2-1.src.rpm as root: [root@host01 monetdb5]# rpm -i MonetDB5-server-5.8.2-1.src.rpm warning: MonetDB5-server-5.8.2-1.src.rpm: Header V3 DSA signature: NOKEY, key ID 0583366fwarning: user sjoerd does not exist - using rootwarning: group sjoerd does not exist - using rootwarning: user sjoerd does not exist - using rootwarning: group sjoerd does not exist - using root Is there a step by step installation guide? _________________________________________________________________ News, entertainment and everything you care about at Live.com. Get it now! http://www.live.com/getstarted.aspx

3 2

[MonetDB-users] MALException:MAL.getAddress:address of 'bat.BKChasReadMode' not found
by Schwarz, Omri 14 Jan '09

14 Jan '09

I am trying to do a straight install of Monet's latest SuperBall (SP!) on Centos 5 (64 bits) and so here is the prologue to the build log: called with arguments: --enable-sql --enable-optimise --prefix=/var/tmp/TEST Copying /root/MonetDB-Nov2008-SuperBall-SP1/MonetDB to /var/tmp/MonetDB-1231879854 Copying /root/MonetDB-Nov2008-SuperBall-SP1/MonetDB-client to /var/tmp/MonetDB-1231879854 Copying /root/MonetDB-Nov2008-SuperBall-SP1/MonetDB5-server to /var/tmp/MonetDB-1231879854 Copying /root/MonetDB-Nov2008-SuperBall-SP1/MonetDB-SQL to /var/tmp/MonetDB-1231879854 * Processing MonetDB ./configure --prefix='/var/tmp/TEST' --enable-optimize --disable-debug --disable-assert --disable-strict --with-python-libdir=lib64/python2.5 --with-p ython-library=/usr/local/lib64/ --with-python-incdir=/usr/local/include/python2.5 The process completes and installs, but when merovingian tries to start a database, it logs this: MSG merovingian[17999]: refresh from mork.denhamcapital.corp for database TEST2 MSG merovingian[17999]: refresh from mork.denhamcapital.corp for database TEST MSG merovingian[17999]: starting database TEST due to control signal MSG TEST[18007]: executing '/usr/local/bin/mserver5' for database 'TEST' MSG merovingian[17999]: starting database 'TEST', up min/avg/max: 10/10/10, crash average: 0.00 0.00 0.00 (2-2=0) MSG TEST[18007]: # MonetDB server v5.8.0, based on kernel v1.26.0 MSG TEST[18007]: # Serving database 'TEST', using 8 threads MSG TEST[18007]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs statically linked MSG TEST[18007]: # Copyright (c) 1993-July 2008 CWI. MSG TEST[18007]: # Copyright (c) August 2008- MonetDB B.V., all rights reserved MSG TEST[18007]: # Visit http://monetdb.cwi.nl/ for further information MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:mdb: loader error lib_xml.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:bat5: loader error /usr/local/lib/MonetDB5/lib/lib_bat5.so: undefined symbol: BATconvert MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:batExtensions: loader error lib_bat5.so.0: cannot open shared object file: No such file or dire ctory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:algebraExtensions: loader error lib_xml.so.0: cannot open shared object file: No such file or d irectory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:pqueue: loader error lib_xml.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:factory: loader error lib_xml.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:const: loader error lib_xml.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:bbp: loader error lib_bat5.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:box: loader error lib_xml.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:mat: loader error lib_xml.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:bpm: loader error lib_bat5.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:inspect: loader error lib_xml.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:manual: loader error lib_xml.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:language: loader error lib_xml.so.0: cannot open shared object file: No such file or directory MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:constraints: loader error lib_xml.so.0: cannot open shared object file: No such file or directo ry MSG TEST[18007]: #WARNING: LoaderException:loadLibrary:mal_mapi: loader error /usr/local/lib/MonetDB5/lib/lib_mal_mapi.so: undefined symbol: ssl_rastr eam (and so on). I've tried both November and Novermber-SP1, with and without xquery, in misceleaneous and sundry combinations, and I wonder if it's a lib64 vesus lib issue, or whether mserver5/Mserver are failing to find some MAL files as well as thse so's. They have wound up in strange places like /usr/local/lib/MonetDB5/lib/lib_xml.so. What's the best way to tidy up this situation ? Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

2 1

[MonetDB-users] PF: "Group By" work-around no longer working?
by jz＠uva 14 Jan '09

14 Jan '09

Hi all, I am using MonetDB Nov2008-SP1 release with XQuery. I need to group the results that I retrieve by file. Since XQuery does not have a 'group by' function as in SQL, I was using the workaround of Borkar and Carey, see http://www.idealliance.org/proceedings/xml04/papers/229/XQueryExtensionsFin… This was working fine for the version of half year ago, but with the latest release the trick with 'distinct-values' is no longer working. The results are returned not grouped by file, e.g. {A,A,A}, {B,B}, {C,C,C,C}, etc. Is this true for other users as well? Am I going crazy? ;) Could someone please test whether this still works? Thanks and regards, junte

2 2

[MonetDB-users] Glusterfs db backend: MonetDB
by Mark V 13 Jan '09

13 Jan '09

Hi, It seems that glusterfs can be sensitive to the DB backend used in some cases. Has MonetDB been considered/contemplated as a back end? http://monetdb.cwi.nl/ http://monetdb.cwi.nl/projects/monetdb/SQL/Benchmark/TPCH/index.html Given MonetDB is still being developed this would be more of a 'roadmap' item. Mark

1 0

[MonetDB-users] Optimizer exceptions when working with Mapi Interface
by David Svenson 08 Jan '09

08 Jan '09

Hi all, I am encountering strange exceptions that are related to the optimizer when working with the MAPI interface. I seem to be getting them when I use a few parallel threads to query the database (Always with the same query). However, I don't get them in a consistent way - I can't reproduce an exact scenario. The exception messages I get also vary - but they are always related to the optimizer. Here are a few of the exception messages I got: 'optimizer.rsColumn' undefined in: _50:any := optimizer.rsColumn() 'calc.opt_remoteQueries' undefined in: _16:any := calc.opt_remoteQueries(_17:oid) 'optimizer.inc_parts' undefined in: _39:any := optimizer.inc_parts() Sometimes, once I get one of these, any following query will give me an exception too, unless I restart MonetDB. In these cases I tried to run the query through mclient, and got the exception too, but followed by the expected query result. Any idea where this is coming from and what I can do to prevent it? Thanks, David. -- View this message in context: http://www.nabble.com/Optimizer-exceptions-when-working-with-Mapi-Interface… Sent from the monetdb-users mailing list archive at Nabble.com.

2 3

[MonetDB-users] large COPY INTO jobs vs. INSERT
by David Svenson 08 Jan '09

08 Jan '09

Hi, I am working with MonetDB v 5.8.2 on a 32 bit machine with 2GB of RAM. I have a 20 CSV files, each containing 200,000 rows and 59 columns. I want to import all of them into one table. When I start running 'COPY INTO's to do that, everything goes well untill the 15th file (which should bring the table up from 2.8 million rows up to 3 million). The 15th file fails. When working with the MAPI interface, the database ends up being corrupt - but works again if I remove the logs. When trying the same directly through mclient.exe, it also fails at around the 15th file, and gives this error: !SQLException:sql.bind:Cannot access descriptor !ERROR: GDKload: cannot read: name=11\1100, ext=tail, 400000 bytes missing. !OS: The operation completed successfully. !ERROR: GDKload: failed name=11\1100, ext=tail 0 tuples I tried two things: 1. I ran 16 'COPY INTO', but this time into 16 different tables. Then I ran INSERT sqls on all of them into one target table. That worked fine, and I passed the 3 Million barrier I ran into with COPY INTO. 2. I tried the same on a 64 bit machine - Everything worked just fine with 'COPY INTO' So what does all of this mean? How do I interpret the error message? Can I / Should I expect from a 32 bit machine to be able to handle what I wanted it to? And if it's a memory issue - then how come everything worked with INSERTs but not with COPY INTOs. Thanks, David. -- View this message in context: http://www.nabble.com/large-COPY-INTO-jobs-vs.-INSERT-tp21351353p21351353.h… Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] copy data from postgresql to monetdb?
by Chris Curvey 08 Jan '09

08 Jan '09

Hi! Newbie here! Before I go about trying to re-invent the wheel, has anyone written scripts to reverse-engineer a PostgreSQL database, convert the SQL to MonetDB's dialect and copy in the data? (I've Google and I can't seem to find such an animal.) -Chris -- Corruptissima republica plurimae leges

2 1

[MonetDB-users] Function errors
by David Svenson 08 Jan '09

08 Jan '09

Hi there... I've been trying to export the mtime. str_to_date function to the sql interface by calling the following script from the sql client: create function str_to_date(s string, format string) returns date external name mtime."str_to_date"; After the function was created, I am trying to use it in the following way: select str_to_date('6 Dec 2001 12:33:45', '%d %b %Y %H:%M:%S'); But, I always get an error: "mtime.str_to_date:strptime support missing" Am I supposed to load some extra module into the server that is not loaded into the monet server on startup? Thanks for everthing… -- View this message in context: http://www.nabble.com/Function-errors-tp21336686p21336686.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

Re: [MonetDB-users] Hello. some questions about XRPC...
by Ying Zhang 07 Jan '09

07 Jan '09

Hello Nicolas, I need some more information about your problems to be able to help. First of all, some general questions: - which version of MonetDB/XQuery do you use? - which OS? - some hardware specifications On Jan 06, 2009, at 14:45 , Nicolas James wrote: > Hello Ying, > > Thanks for all these responses, they have been very helpful to me > (the real problem I had, and now resolved, was about the call to the > functions of pathfinder). > > I've coded a Java client for indexing my data (image descriptions > and annotations in XML files) in MonetDB/XQuery, 46,557 XML > documents / 181.1Mo and I use XRPC. Is 181.1MB the size of one document (then the total size is > 8TB!) or the size of all 46557 documents? Do I understand it correctly that you have a Java client which send XRPC request messages to MonetDB/XQuery server to add documents? How does your Java client exactly work, i.e.: - does it add one document at a time, or all 46557 documents in one request? - is your Java client and MonetDB/XQuery server on the same machine? - do you use pf:add-doc() or fn:put() to add the documents? > My client works well but I don't understand some MonetDB behaviors. > My first tests created collections like my dataset and his > directories structure, I don't understand what you mean here. As far as I know, it is not possible to create nested collections in MonetDB/XQuery, and it is not possible to create empty collections (without adding a document). So, how did you create the collections? > but I've some directories with over that 900 documents, and I think > that MonetDB doesn't like that (i.e. 900 documents in one > collections), because the indexation failed. I don't know the exact max nr. of documents a collection may have, I think it is limited by if you are using a 32bits system or a 64bits system. Using mlcient, adding 46,557 documents of 11k each on a Fedora machine was no problem; adding > 900 documents of 200MB each is still processing (disk usage is still growing) when I'm writing this e-mail :) Did you add your documents using your Java client? Have you tried to add your documents without using XRPC, e.g., using mclient? What do you mean with "indexation failed"? Is there any error message? > Then, I try to split my dataset, I create collections of 128 > documents, but my problem is that in MonetDB all my collections does > not contain 128 doc, some 128 others a number between 5 and 128... > It's a little strange because I don't think the reason is my client... For this problem, I have similar questions, i.e.: - how large are the documents? - how are they added _exactly_? - Did you get any error message? Kind regards, Jennie > > Have you ever heard about a similar problem in MonetDB? > > Thanks previous responses, > > best regards, > > Nicolas James > > > Ying Zhang wrote: >> Hello Nicolas, >> >> First of all, thanks for trying MonetDB. I will try to answer your >> XRPC >> questions. >> >> >> On Tue, Dec 16, 2008 at 04:35:33PM +0100, Nicolas James wrote: >> >>> Hello MonetDB, >>> >>> I have some questions about the writing of a client for MonetDB in >>> Java using the XRPC, and It could be very very helpful for my >>> research if you could answer to some questions. >>> >>> I'm a PhD student in the MAS laboratory, Ecole Centrale Paris, >>> France, and I work on image automatic annotation. >>> I think MonetDB is extremely powerful and I want to use it in my >>> research, I would to store some images annotations under the form >>> of XML file in MonetDB since my application (a Java application). >>> >>> So, with your documentation I coded a light client using the JDBC >>> driver for adding my documents in MonetDB, but this way don't >>> satisfy me, the XRPC method is very attractive, but I've some >>> problems. >>> >>> 1) how add a document using the XRPCWrapper (using the pf:add-doc, >>> it's considered like an updating function, I've read the document, >>> but I don't understand...). >>> >> >> XRPC has several components. >> The XRPC protocol is a way of communicating with MonetDB/XQuery to >> run >> XQuery queries. An XRPC server is integrated in MonetDB/XQuery to >> handle requests speaking the XRPC protocol. >> >> Hence, to communicate with MonetDB/XQuery using XRPC, you only need >> to >> send requests according to the XRPC protocol. For Java, there are >> several API functions for creating request messages, sending and >> receiving messages, and handling response messages. The Java doc of >> these APIs could be found here: >> http://homepages.cwi.nl/~fabian/MonetDB/Java%20Package.html/index.html?nl/c… >> The Java package of the APIs could be found under the directory where >> you have installed MonetDB: share/MonetDB/lib/xrpcapi.jar >> In the same directory there is an xrpcclient.jar package, which gives >> an example of how to use the APIs. It's source could found in: >> java/src/nl/cwi/monetdb/xquery/xrpc/client >> >> You don't need the XRPCWrapper, in your case. XRPCWrapper is >> needed if >> you want to handle (simple) XRPC requests using another XQuery engine >> than MonetDB/XQuery. >> >> >> >>> 2) Is-it possible to create collections using the XRPC (and add >>> files in this collection) ? >>> >> >> I'm not sure if I understand you correctly. >> >> With XRPC, you can execute the pf:add-doc() function (you need to >> put it >> into an user-defined-function). with pf:add-doc(), you could specify >> the name of the collection to which you want to add your document. >> If >> such collection doesn't exist, it is created. Otherwise, your >> document >> will be added to the existing collection. >> >> >>> 3) and with the JDBC driver ? >>> 4) using the JDBC driver, is it possible to specify the reference >>> id for the file (for instance indexing the file "foobar.xml" and >>> referenced "foobar.xml" in the database ? >>> >> >> What do you mean with 'indexing the file "foobar.xml"'? >> (Disclaimer: I don't know much about the JDBC driver) >> >> >> Hope this helps a bit. If you have any problem with using the XRPC >> API, >> please don't hesitate to ask. >> >> >> >> Regards, >> >> Jennie >> >> >>> These answers could help me, hope you could have time to answer! >>> >>> best regards, >>> >>> -- >>> Nicolas James >>> PhD student MAS / ECP >>> Phone: +33.1.41.13.17.37 >>> Office: Lab, C 441A >>> Laboratoire MAS - Ecole Centrale Paris >>> Grande Voie des Vignes >>> 92295 CHATENAY MALABRY Cedex >>> France >>> >>> >> >> > > > -- > Nicolas James > PhD student MAS / ECP > Phone: +33.1.41.13.17.37 > Office: Lab, C 441A > Laboratoire MAS - Ecole Centrale Paris > Grande Voie des Vignes > 92295 CHATENAY MALABRY Cedex > France >

1 0

[MonetDB-users] what is monet_vault_key used for?
by 小波顾 05 Jan '09

05 Jan '09

_________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

3 2

[MonetDB-users] Is there any way to run MonetDB as a Windows Service?
by 小波顾 05 Jan '09

05 Jan '09

Thanks. _________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

3 2

[MonetDB-users] COPY INTO on aggregate queries crashes MonetDB
by Peter Farkash 04 Jan '09

04 Jan '09

Hi, When I use COPY INTO with a count query, monetDB crashes. For example: copy select count (*) into 'C:\\copyinto.txt' using delimiters ','; Gives me the following result and kills the server: MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = copy select count (*) into 'C:\\copyinto.txt' using delimiters ','; ERROR = Connection terminated The same happens if I use sum, average, etc. On normal queries the copy into works just fine. I am using v5.8.2, and am getting the crash both when using mclient and the MAPI programming interface. Any ideas? -- View this message in context: http://www.nabble.com/COPY-INTO-on-aggregate-queries-crashes-MonetDB-tp2127… Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] What code page does Monetdb currentlly use?
by 小波顾 03 Jan '09

03 Jan '09

We are Chinese users, and hoping Monetdb will support multi-byte character encodings like UTF-8 and GBK. _________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

5 6

[MonetDB-users] Monetdb5 threading question.
by 小波顾 02 Jan '09

02 Jan '09

When I start the default demo database (or a new created one) process on Windows XP, the following messaged appears in the command window : # Serving databse 'demo', using 1 thread Does it mean the whole database process is single-threaded? The following section in the monetdb5.config file shows we can config multiple threads # Mserver 5 recognizes a series of command line names instead:# --threads ( 1 | PARMASK)# --memory (MEMMASK )# --properties (CHECKMASK | PROPMASK | BATMASK )# --io (IOMASK | PERFMASK )# --transactions ( TMMASK | DELTAMASK | TEMMASK)# --modules (DLMASK | LOADMASK)# --algorithms (ALGOMASK | ESTIMASK)# --xproperties (XPROPMASK )# --performance (JOINPROPMASK | DEADBEEFMASK) What does 'parmsk' mean, and how to config monetdb to use multiple threads? _________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

3 4

[MonetDB-users] Multiple database question.
by 小波顾 02 Jan '09

02 Jan '09

I have created a customed version of the MServer5.bat and MClient.bat to serve multiple databases. MServer5.bat : set MONETHOME=D:\Amber\Program\MonetDB\MonetDB5 set MONETDATAROOT=%1set MONETDATAROOT=%MONETDATAROOT:~0,-1%set MONETDBNAME=%2 if "%MONETDATAROOT%" == "" pauseif "%MONETDBNAME%" == "" pause set PATH=%MONETHOME%\bin;%MONETHOME%\lib;%MONETDB%\lib\MonetDB5;%PATH% set MONETDBFARM=--dbfarm="%MONETDATAROOT%\dbfarm"set SQLLOGDIR=--set "sql_logdir=%MONETDATAROOT%\sql_logs"set XQUERYLOGDIR=--set "xquery_logdir=%MONETDATAROOT%\xquery_logs" rem start the real server"%MONETHOME%\bin\mserver5.exe" --set "prefix=%MONETHOME%" --set "exec_prefix=%MONETHOME%" %MONETDBFARM% %SQLLOGDIR% --dbname=%MONETDBNAME% --dbinit="include sql;" MClient5.bat @set MONETHOME=D:\Amber\Program\MonetDB\MonetDB5 @set DBNAME=%1if "%DBNAME%" == "" pause @rem extend the search path with our EXE and DLL folders@rem we depend on pthreadVCE.dll having been copied to the lib folder@set PATH=%MONETHOME%\bin;%MONETHOME%\lib;%MONETHOME%\lib\bin;%PATH% @rem start the real client@"%MONETHOME%\bin\mclient.exe" -lsql -d %DBNAME% @if ERRORLEVEL 1 pause Using "MServer5.bat D:\MONETDB amber" I am able to start a new server process, ant it successfully created a new database in D:\MONETDB directory, but with the new server and default demo server process all running, I can't use mclient connect to the new database. MClient5.bat amber D:\>mclient.exe -E GB2312 -lsql -d amber!database 'amber' does not exist But the demo database can be connected to, and the new database can be connected to only when I close the demo database process. The netstat -a command shows there are two processes listening on port 50000. _________________________________________________________________ Invite your mail contacts to join your friends list with Windows Live Spaces. It's easy! http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx…

2 2

[MonetDB-users] Is there any load speed measurements of MonetDB?
by 小波顾 30 Dec '08

30 Dec '08

_________________________________________________________________ Drag n’ drop—Get easy photo sharing with Windows Live™ Photos. http://www.microsoft.com/windows/windowslive/photos.aspx

1 0

[MonetDB-users] Can someone point to me some resources about memory and storage model of MonenDB.
by 小波顾 30 Dec '08

30 Dec '08

Thanks. _________________________________________________________________ Invite your mail contacts to join your friends list with Windows Live Spaces. It's easy! http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx…

1 0

[MonetDB-users] Copy Into problem
by Alfred Nordman 30 Dec '08

30 Dec '08

Hi there, I'm trying to use the copy into with csv file (utf-8 encoded) and I encounter some problem. The create statement is: CREATE TABLE "sa3e7e9f4d666429ea2c5bb61a532f5b9" ("A" DATE ,"B" VARCHAR (22) ,"C" DECIMAL (12, 5) ,"D" DECIMAL (12, 5) ,"E" DECIMAL (12, 5) ,"F" DECIMAL (12, 5) ) The copy into statement is: COPY INTO "sa3e7e9f4d666429ea2c5bb61a532f5b9" FROM 'T:\\Prism\\Applications\\Desktop\\PrismDesktop\\bin\\Debug\\LocalRepository\\ImportedCSV\\ImportedCSV0.csv' USING DELIMITERS '|' NULL AS 'A56A9FC261A143a48F6019F602ABF409'; And csv is attached. Any advice?!?!?! tnx, Alfred. http://www.nabble.com/file/p21200441/ImportedCSV0.csv ImportedCSV0.csv -- View this message in context: http://www.nabble.com/Copy-Into-problem-tp21200441p21200441.html Sent from the monetdb-users mailing list archive at Nabble.com.

6 13

[MonetDB-users] COPY INTO problem
by Alfred Nordman 30 Dec '08

30 Dec '08

Hi there, I'm trying to use the copy into with csv file (utf-8 encoded) and I encounter some problem. The create statement is: CREATE TABLE "sa3e7e9f4d666429ea2c5bb61a532f5b9" ("A" DATE ,"B" VARCHAR (22) ,"C" DECIMAL (12, 5) ,"D" DECIMAL (12, 5) ,"E" DECIMAL (12, 5) ,"F" DECIMAL (12, 5) ) The copy into statement is: COPY INTO "sa3e7e9f4d666429ea2c5bb61a532f5b9" FROM 'T:\\Prism\\Applications\\Desktop\\PrismDesktop\\bin\\Debug\\LocalRepository\\ImportedCSV\\ImportedCSV0.csv' USING DELIMITERS '|' NULL AS 'A56A9FC261A143a48F6019F602ABF409'; And csv is attached. Any advice?!?!?! tnx, Alfred. http://www.nabble.com/file/p21193416/ImportedCSV0.csv ImportedCSV0.csv -- View this message in context: http://www.nabble.com/COPY-INTO-problem-tp21193416p21193416.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 2

[MonetDB-users] Announcement: New bug fix release of MonetDB suite
by Sjoerd Mullender 22 Dec '08

22 Dec '08

The MonetDB team at CWI is pleased to announce the Nov2008-SP1 bug fix release of the MonetDB suite of programs. This is a bug fix release of last month's Nov2008 release. More information on this release can be found at <http://monetdb.cwi.nl/projects/monetdb/Development/Releases/Nov2008/#Nov200…>. The sources and binaries are available at <http://monetdb.cwi.nl/projects/monetdb/Download/>. -- Sjoerd Mullender

3 2

[MonetDB-users] multi-users and memory problems
by Loredana Afanasiev 21 Dec '08

21 Dec '08

Dear developers, I'm currently running the CVS head of Dec 11, 2008. I apologies in advance for missing on details. If needed I'll provide more. Thank you, and lots of wonderful wishes for Xmas and for the New Year!!! (this is the last email you'll get from me this year, I promise :)) l. Observation 1: When trying to run big numbers of clients simultaneously, the server crushes. For me it crushed with 70 and 100 clients fired at the same time. Observation 2: After running one query in a multi-user scenario (N clients/threads simultaneously) the memory footprint of mserver (% of the memory that the server occupies) grew. After repeating the experiment M times the memory grew each time with a constant. Running the same query sequentially the same number of times leaves the footprint of mserver constant. Could it be a faulty memory cleaning? Query: q3.xq let $col := fn:collection("MotiesTweedeKamer") let $years := fn:distinct-values( for $date in $col//hiddendatum return fn:substring(fn:string($date),1,4)) for $y in $years order by $y ascending return <result year="{$y}" count="{ count($col//document[fn:substring(fn:string(.//hiddendatum),1,4) = $y]) }"/> N=50 M=1 $ perl runNclients.pl N=50 $ top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28563 lafanasi 20 0 6266m 5.1g 100m S 0 26.0 10:50.71 Mserver M=2 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28563 lafanasi 20 0 7546m 5.9g 100m S 0 30.1 21:42.67 Mserver ... N=60 M=10 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31182 lafanasi 20 0 17.3g 8.3g 100m S 0 42.3 263:45.37 Mserver ... M=30 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31182 lafanasi 20 0 22.7g 9.6g 100m S 0 48.8 395:08.87 Mserver Sequential run: N=1 M=50 times PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28372 lafanasi 20 0 839m 430m 100m S 1 2.1 6:26.65 Mserver M=100 times PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28372 lafanasi 20 0 839m 430m 100m S 1 2.1 6:26.65 Mserver Observation 3: The times it takes to run 10 Tijah search queries simultaneously is the same as the time it takes to run them sequentially. The percentage of CPU used in both cases by mserver is also similar. Does tijah support multi-users? Query: q2.xq let $opt := <TijahOptions ft-index="polietiekedata" ir-model="NLLR"/> let $c := collection("HAN") let $qid := tijah:query-id($c, "//spreker[about(.,%KEYWORD%)]", $opt) for $res in tijah:nodes($qid) return <pair>{( string($res/@naam), tijah:score($qid, $res))}</pair> 20 users/threads stopMserver,startMserver (to make sure that the server freed the memory) $time Run20Threads(mclient -lx q2.xq) 0.134u 0.118s 0:20.01 1.1% 0+0k 0+8io 0pf+0w sequential run, 20 times stopMserver, startMserver $ time for i in `seq 1 10`; do mclient -lx q2.xq; done 0.060u 0.067s 0:19.74 0.6% 0+0k 0+8io 0pf+0w Observation 4: The mserver memory footprint grows very fast when running Tijah queries. When the footprint reaches 98% the query processing time gets really slow or the server crushes. Query: the same as above Sequential run, M=90 times $stopMserver, startMserver $for i in `seq 1 10`; do mclient -lx q2.xq; done $ top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24527 lafanasi 20 0 26.2g 19g 80m S 66 98.0 4:24.57 Mserver

2 1

[MonetDB-users] mserver memory grows, system thrashes and hangs
by Bostr 18 Dec '08

18 Dec '08

Hello MonetDB Nov2008 release, built with all standard options on RedHat EL 5 (2.6.18-92.el5 #1 SMP), on HP Proliant DL380 8-way, 16GB RAM / 16GB swap, with dbfarm on a RAID0 internal 7-disk array. I'm doing bulk load tests (using mclient to run COPY INTO table FROM file), repeatedly loading 500,000 row (68M) CSV files into a fact table, while periodically running a suite of test queries in a separate process to test query times as the fact table grows. The problem is that as the fact table grows to around 80Million records, the database starts to consume all available system memory and swap, the system thrashes and comes to its knees. I also get database corruption causing foreign key violations. (More details below) Has anyone else seen this? Any ideas if it's a bug, memory leak? Or could this be caused by operator error? Suggestions? Cheers Bob PS More detailed account of the problem below: Everything starts off fine, files load in 8-10 seconds, queries quite fast. As the table starts to fill, the queries (not unexpectedly) start to slow. But (unexpectedly) the memory consumption of the mserver5 process steadily increases. At some point, around 80Million records, the system performance rapidly starts to degrade. Query times dramatically increase, load time also dramatically increases, and the entire system thrashes and becomes unusable. Running 'top' shows that mserver5 has consumed vurtually all physical memory, and most of the swap too. top - 16:33:34 up 28 days, 23:52, 8 users, load average: 1.68, 1.51, 1.96 Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 1.5%sy, 0.0%ni, 85.9%id, 12.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16439196k total, 16298404k used, 140792k free, 2300k buffers Swap: 18876364k total, 13315960k used, 5560404k free, 9824352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32380 monetdb 18 0 54.6g 14g 9.4g S 12 94.9 107:19.46 mserver5 29217 bostr 15 0 290m 5552 2264 S 1 0.0 12:01.37 gnome-terminal The first time this happened, I stopped the load scripts, restarted the database, and that helped things for a little while, but it wasn't long before the memory was all consumed again, and performance went to the dogs. Then loads started failing with foreign-key violation errors.. Inspection showed a corrupted dimension table that somehow ended up with a bunch of bad values in it.. So I blew away, and recreated the database. This time I tried adding the line 'gdk_nr_threads=1' to the monetdb5.conf file - having seen a defect report related to 'COPY INTO ' being buggy and this being the solution. But the same thing happened again.. it ran fine again until I had about 100Million rows, then memory problems/thrashing. Restarted the server again, and shortly after the data loads failed again with foreign-key violations and a corrupted dimension table.. -- View this message in context: http://www.nabble.com/mserver-memory-grows%2C-system-thrashes-and-hangs-tp2… Sent from the monetdb-users mailing list archive at Nabble.com.

4 5

[MonetDB-users] Again: Mserver fails on Fedora 9 - 32 bit
by Hans van Rijswijk (DT) 11 Dec '08

11 Dec '08

Hi MonetDB users, I downloaded Monet Fedora RPMs from: http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./2008 1209/RPMs/Fedora9.i386.oid32/ I installed these with yum on my Fedora fc9 installation, running within VMWare on my Windows XP 32 bit machine. When I startup Mserver everything seems to work well. I start with an empty dbfarm. Then I run the following scenario: - Add 3 XML documents with pf:add-doc() (some of them updatable) - Add 2 more XML documents with pf:add-doc() - Update one of these docs: do insert <something/> into doc("a.xml")/rootelement - Send a sync() command with mclient mil. - Check the content of the database with pf:documents() (is OK) - Knock down Mserver: kill <pid> - Startup Mserver again (don't ask me why I do it like this) After this scenario I cannot gain access to Mserver with mclient: !xchange_challenge: frontend xquery not found If I wait half an hour before killing Mserver this works fine. Questions: - Does sync() do what it is supposed to do? - Why is shutdown() not implemented? - What is the right way to shutdown Mserver when it is started as a daemon in the background? - How do I guarantee the database to remain intact with the abovementioned scenario? Regards, Hans van Rijswijk Netherlands Forensic Institute

4 7

[MonetDB-users] crash on Windows with local install
by Bryan Lewis 11 Dec '08

11 Dec '08

If I install MonetDB in the normal way on a Windows machine, it installs the C runtime library dlls in a location like C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT... However, we want to install MonetDB along with our product, in a single installation procedure that doesn't require the user to have administrative rights. That means we need to install the CRT directory as "private assemblies", copying it to the MonetDB application directories. I've almost gotten this to work by copying the CRT directory to MonetDB5/lib, bin and lib/MonetDB5. I can run MSQLServer.bat and mclient.bat in separate console windows. But when I try to run some SQL commands (my test case is a 'create table' and a 'copy into' from a CSV file), mclient dies with a Windows exception. The event in Event Viewer shows: Faulting application mclient.exe, version 0.0.0.0, faulting module ntdll.dll In theory the private assemblies are all that Windows needs, and in fact the programs run which implies that the libraries are being found. I don't understand why the crash happens later. I'm hoping someone else has tried this. The exception happens with the last official Windows release as well as with our built-from-cvs version. Thanks, Bryan Lewis

3 2

[MonetDB-users] Mserver fails on Fedora 9 - 32 bit
by Hans van Rijswijk (DT) 09 Dec '08

09 Dec '08

Hi MonetDB users, I downloaded Monet Fedora RPMs from: http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./2008 1209/RPMs/Fedora9.i386.oid32/ I installed these with yum on my Fedora fc9 installation, running within VMWare on my Windows XP 32 bit machine. I got the following messages when starting Mserver: # MonetDB Server v4.26.1 # based on GDK v1.26.1 # Copyright (c) 1993-July 2008, CWI. All rights reserved. # Copyright (c) August 2008-, MonetDB B.V.. All rights reserved. # Compiled for i686-redhat-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. # PF/Tijah module v0.9.0 loaded. http://dbappl.cs.utwente.nl/pftijah !ERROR: logger: could not use bat (209) for 1000000000_map_pid !ERROR: logger: could not use bat (210) for 1000000000_rid_size !ERROR: logger: could not use bat (442) for 1000000000_rid_level !ERROR: logger: could not use bat (392) for 1000000000_rid_prop !ERROR: logger: could not use bat (441) for 1000000000_rid_kind !ERROR: logger: could not use bat (412) for 1000000000_rid_nid !ERROR: logger: could not use bat (418) for 1000000000_nid_rid !ERROR: logger: could not use bat (211) for 1000000000_qn_histogram !ERROR: logger: could not use bat (451) for 1000000000_qn_prefix_uri_loc !ERROR: logger: could not use bat (452) for 1000000000_qn_uri_loc !ERROR: logger: could not use bat (453) for 1000000000_qn_prefix !ERROR: logger: could not use bat (454) for 1000000000_qn_uri !ERROR: logger: could not use bat (455) for 1000000000_qn_loc !ERROR: logger: could not use bat (456) for 1000000000_prop_text !ERROR: logger: could not use bat (457) for 1000000000_prop_com !ERROR: logger: could not use bat (458) for 1000000000_prop_ins !ERROR: logger: could not use bat (459) for 1000000000_prop_tgt !ERROR: logger: could not use bat (460) for 1000000000_prop_val !ERROR: logger: could not use bat (379) for 1000000000_attr_own !ERROR: logger: could not use bat (406) for 1000000000_attr_qn !ERROR: logger: could not use bat (427) for 1000000000_attr_prop !ERROR: logger: could not use bat (245) for 1000000001_map_pid !ERROR: logger: could not use bat (246) for 1000000001_rid_size !ERROR: logger: could not use bat (423) for 1000000001_rid_level !ERROR: logger: could not use bat (384) for 1000000001_rid_prop !ERROR: logger: could not use bat (436) for 1000000001_rid_kind !ERROR: logger: could not use bat (421) for 1000000001_rid_nid !ERROR: logger: could not use bat (100) for 1000000001_nid_rid !ERROR: logger: could not use bat (247) for 1000000001_qn_histogram !ERROR: logger: could not use bat (391) for 1000000001_qn_prefix_uri_loc !ERROR: logger: could not use bat (471) for 1000000001_qn_uri_loc !ERROR: logger: could not use bat (383) for 1000000001_qn_prefix !ERROR: logger: could not use bat (478) for 1000000001_qn_uri !ERROR: logger: could not use bat (479) for 1000000001_qn_loc !ERROR: logger: could not use bat (480) for 1000000001_prop_text !ERROR: logger: could not use bat (481) for 1000000001_prop_com !ERROR: logger: could not use bat (482) for 1000000001_prop_ins !ERROR: logger: could not use bat (483) for 1000000001_prop_tgt !ERROR: logger: could not use bat (484) for 1000000001_prop_val !ERROR: logger: could not use bat (414) for 1000000001_attr_own !ERROR: logger: could not use bat (103) for 1000000001_attr_qn !ERROR: logger: could not use bat (102) for 1000000001_attr_prop !ERROR: logger: could not use bat (380) for 1000000002_map_pid !ERROR: logger: could not use bat (425) for 1000000002_rid_size !ERROR: logger: could not use bat (438) for 1000000002_rid_level !ERROR: logger: could not use bat (378) for 1000000002_rid_prop !ERROR: logger: could not use bat (279) for 1000000002_rid_kind !ERROR: logger: could not use bat (399) for 1000000002_rid_nid !ERROR: logger: could not use bat (416) for 1000000002_nid_rid !ERROR: logger: could not use bat (405) for 1000000002_qn_histogram !ERROR: logger: could not use bat (431) for 1000000002_qn_prefix_uri_loc !ERROR: logger: could not use bat (443) for 1000000002_qn_uri_loc !ERROR: logger: could not use bat (229) for 1000000002_qn_prefix !ERROR: logger: could not use bat (472) for 1000000002_qn_uri !ERROR: logger: could not use bat (498) for 1000000002_qn_loc !ERROR: logger: could not use bat (499) for 1000000002_prop_text !ERROR: logger: could not use bat (500) for 1000000002_prop_com !ERROR: logger: could not use bat (501) for 1000000002_prop_ins !ERROR: logger: could not use bat (502) for 1000000002_prop_tgt !ERROR: logger: could not use bat (503) for 1000000002_prop_val !ERROR: logger: could not use bat (433) for 1000000002_attr_own !ERROR: logger: could not use bat (327) for 1000000002_attr_qn !ERROR: logger: could not use bat (468) for 1000000002_attr_prop !ERROR: logger: could not use bat (555) for 1000000003_map_pid !ERROR: logger: could not use bat (238) for 1000000003_rid_size !ERROR: logger: could not use bat (388) for 1000000003_rid_level !ERROR: logger: could not use bat (239) for 1000000003_rid_prop !ERROR: logger: could not use bat (386) for 1000000003_rid_kind !ERROR: logger: could not use bat (556) for 1000000003_rid_nid !ERROR: logger: could not use bat (557) for 1000000003_nid_rid !ERROR: logger: could not use bat (206) for 1000000003_qn_histogram !ERROR: logger: could not use bat (487) for 1000000003_qn_prefix_uri_loc !ERROR: logger: could not use bat (512) for 1000000003_qn_uri_loc !ERROR: logger: could not use bat (510) for 1000000003_qn_prefix !ERROR: logger: could not use bat (509) for 1000000003_qn_uri !ERROR: logger: could not use bat (508) for 1000000003_qn_loc !ERROR: logger: could not use bat (507) for 1000000003_prop_text !ERROR: logger: could not use bat (506) for 1000000003_prop_com !ERROR: logger: could not use bat (283) for 1000000003_prop_ins !ERROR: logger: could not use bat (409) for 1000000003_prop_tgt !ERROR: logger: could not use bat (393) for 1000000003_prop_val !ERROR: logger: could not use bat (465) for 1000000003_attr_own !ERROR: logger: could not use bat (204) for 1000000003_attr_qn !ERROR: logger: could not use bat (474) for 1000000003_attr_prop !ERROR: logger: could not use bat (489) for 1000000004_map_pid !ERROR: logger: could not use bat (370) for 1000000004_rid_size !ERROR: logger: could not use bat (428) for 1000000004_rid_level !ERROR: logger: could not use bat (546) for 1000000004_rid_prop !ERROR: logger: could not use bat (419) for 1000000004_rid_kind !ERROR: logger: could not use bat (196) for 1000000004_rid_nid !ERROR: logger: could not use bat (537) for 1000000004_nid_rid !ERROR: logger: could not use bat (470) for 1000000004_qn_histogram !ERROR: logger: could not use bat (511) for 1000000004_qn_prefix_uri_loc !ERROR: logger: could not use bat (396) for 1000000004_qn_uri_loc !ERROR: logger: could not use bat (390) for 1000000004_qn_prefix !ERROR: logger: could not use bat (382) for 1000000004_qn_uri !ERROR: logger: could not use bat (426) for 1000000004_qn_loc !ERROR: logger: could not use bat (387) for 1000000004_prop_text !ERROR: logger: could not use bat (518) for 1000000004_prop_com !ERROR: logger: could not use bat (532) for 1000000004_prop_ins !ERROR: logger: could not use bat (569) for 1000000004_prop_tgt !ERROR: logger: could not use bat (562) for 1000000004_prop_val !ERROR: logger: could not use bat (515) for 1000000004_attr_own !ERROR: logger: could not use bat (539) for 1000000004_attr_qn !ERROR: logger: could not use bat (224) for 1000000004_attr_prop # MonetDB/XQuery module v0.26.1 loaded (default back-end is 'algebra') # XRPC administrative console at http://127.0.0.1:51241/admin This is not all: also mclient could not communicate with Mserver: $ mclient -p51240 -lx !xchange_challenge: frontend xquery not found The same applies when running this distribution on Fedora 10. Did anyone encounter a similar problem; Does anyone know a workaround? Hope to hear from one of you soon.. Regards, Hans van Rijswijk Netherlands Forensic Institute

2 1

[MonetDB-users] Announcement: New feature release of MonetDB suite
by Sjoerd Mullender 08 Dec '08

08 Dec '08

The MonetDB team at CWI is pleased to announce the new "Nov2008" release of the MonetDB suite of programs. This is a feature release, involving, among other things, a change in the database format, so be sure to dump your databases before attempting an upgrade. More information on the new release is available at <https://sourceforge.net/forum/forum.php?forum_id=892621>. The sources and binaries are available at <http://monetdb.cwi.nl/projects/monetdb/Download/>. -- Sjoerd Mullender

1 0

[MonetDB-users] Mclient exit value
by Floris Ouwendijk (DT) 04 Dec '08

04 Dec '08

Dear users, While investigating a problem where a collection-query returns invalid data, I ran into problems with mclient seemingly not generating a proper exit value. If I run: $ mclient -lxq -p49980 -s"element { \"log\" } { pf:collection(\"cons02_log.xml\")//run }" -removed a lot of xml- </stdout> <fault!ERROR: XML Generation: tmp_3265 BAT does not have a 24 head. MAPI = monetdb@botswana.xiraf:49980 QUERY = element { "log" } { pf:collection("cons02_log.xml")//run } ERROR = !ERROR: !ERROR: XML Generation: NULL BAT has a 24 head, but tail is NULL. !ERROR: !ERROR: xquery_print_result_main: operation failed. $ echo $? 0 The above returns a lot of correct XML, then dumps an error to stderr, but returns error code 0. In contrast, the following works as expected: $ mclient -lxq -p49980 -s"eelement { \"log\" } { pf:collection(\"cons02_log.xml\")//run }" >/dev/null MAPI = monetdb@server:49980 QUERY = eelement { "log" } { pf:collection("cons02_log.xml")//run } ERROR = !parse error: syntax error, unexpected {, expecting $end on line 1, column 10 (next token is `{') !parse error: XQuery parsing failed !# halted in parser.tab.c (PFparse), line 5716 $ echo $? 1 Which is the expected result for a flawed query. Do you agree this is a problem that should be reported, or is there a better way of finding out whether there were problems running the query? Kind regards, Floris

2 1

[MonetDB-users] Building the nightly build on Centos 5
by Schwarz, Omri 01 Dec '08

01 Dec '08

Hi, all. I am trying to do a build on Centos 5 and am getting this bizarre issue which a web search shows has propped up with many other autoconf-based builds, but which does not have an easy resolution: /bin/sh ../../libtool --tag=CC --mode=link gcc -DLIBGDK -O2 -std=c99 -O6 -fomit-frame-pointer -finline-functions -falign-loops=4 -falign-jumps=4 -falign-functions=4 -fexpensive-optimizations -fu nroll-loops -frerun-cse-after-loop -frerun-loop-opt -ftree-vectorize -D_REENTRANT -o libbat.la -rpath /root/MonetDB/l ib libbat_la-gdk_scanselect.lo libbat_la-gdk_scanselect_defs.lo libbat_la-gdk_batop.lo libbat_la-gdk_search.lo libbat_la -gdk_tm.lo libbat_la-gdk_align.lo libbat_la-gdk_bbp.lo libbat_la-gdk_heap.lo libbat_la-gdk_setop.lo libbat_la-gdk_utils.l o libbat_la-gdk_atoms.lo libbat_la-gdk_qsort.lo libbat_la-gdk_ssort.lo libbat_la-gdk_storage.lo libbat_la-gdk_bat.lo libb at_la-gdk_delta.lo libbat_la-gdk_relop.lo libbat_la-gdk_system.lo libbat_la-gdk_value.lo libbat_la-gdk_rangejoin.lo libba t_la-gdk_posix.lo libbat_la-gdk_logger.lo ../common/libmutils.la ../common/libstream.la -lz -lpthread -ldl gcc -shared .libs/libbat_la-gdk_scanselect.o .libs/libbat_la-gdk_scanselect_defs.o .libs/libbat_la-gdk_batop.o .libs/lib bat_la-gdk_search.o .libs/libbat_la-gdk_tm.o .libs/libbat_la-gdk_align.o .libs/libbat_la-gdk_bbp.o .libs/libbat_la-gdk_he ap.o .libs/libbat_la-gdk_setop.o .libs/libbat_la-gdk_utils.o .libs/libbat_la-gdk_atoms.o .libs/libbat_la-gdk_qsort.o .lib s/libbat_la-gdk_ssort.o .libs/libbat_la-gdk_storage.o .libs/libbat_la-gdk_bat.o .libs/libbat_la-gdk_delta.o .libs/libbat_ la-gdk_relop.o .libs/libbat_la-gdk_system.o .libs/libbat_la-gdk_value.o .libs/libbat_la-gdk_rangejoin.o .libs/libbat_la-g dk_posix.o .libs/libbat_la-gdk_logger.o -Wl,--rpath -Wl,/var/tmp/test/MonetDB/MonetDB/src/common/.libs -Wl,--rpath -Wl,/ root/MonetDB/lib -L/var/tmp/test/MonetDB/MonetDB/src/common/.libs ../common/.libs/libmutils.so ../common/.libs/libstream. so -lz -lpthread -ldl -Wl,-soname -Wl,libbat.so.0 -o .libs/libbat.so.0.0.0 .libs/libbat_la-gdk_scanselect_defs.o: In function `__strcspn_c1': gdk_scanselect_defs.c:(.text+0x0): multiple definition of `__strcspn_c1' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0x0): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strcspn_c2': gdk_scanselect_defs.c:(.text+0x34): multiple definition of `__strcspn_c2' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0x34): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strcspn_c3': gdk_scanselect_defs.c:(.text+0x78): multiple definition of `__strcspn_c3' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0x78): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strspn_c1': gdk_scanselect_defs.c:(.text+0xd4): multiple definition of `__strspn_c1' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0xd4): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strspn_c2': gdk_scanselect_defs.c:(.text+0xf0): multiple definition of `__strspn_c2' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0xf0): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strspn_c3': gdk_scanselect_defs.c:(.text+0x108): multiple definition of `__strspn_c3' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0x108): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strpbrk_c2': gdk_scanselect_defs.c:(.text+0x128): multiple definition of `__strpbrk_c2' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0x128): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strpbrk_c3': [... and so ad nauseam, containing redefinitions of amy low level LibC functions like toupper, tolower, fstat, et cetera. ] What do I do to dodge this? Thanks, all Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

1 0

[MonetDB-users] Records disappear in prepared statement insterts via JDCB and ODBC if using autocommit or batch commit
by Jens Oehlschlägel 01 Dec '08

01 Dec '08

Dear monetdb community, I am new to monetdb and I am very impressed about its speed and nice features like its csv bulk load command. I have a question on prepared statements: I noticed that when using prepared statements via ODBC or JDBC strange things happened: when inserting 1000 rows, not only insert statements became very slow, also many inserts silently never reached MonetDB, i.e. a random number of records passes and suddenly - observed with ODBC debugging activated - execs are responded with errors. I tried via ODBC (writing from R's RODBC package) and via JDBC (writing from DBVizualizer). I found out that deactivating autocommit will help (verified for ODBC and JDBC), but the problem reappears - again randomly - if the client tool sends batch commits each x records (verified only in DBVizualizer / JDBC, now with explicit errors). Is that a known limitation or a bug or am I doing wrong? Further down you find information for replication. Thanks for any help Kind regards Dr. Jens Oehlschlägel Munich P.S. Details I run MonetDB under WindowsProfessionalXP with 3 GB RAM. I installed: MonetDB5-SQL-Installer-i386-20080624.msi MonetDB-ODBC-Installer-i386-20080812.msi monetdb-1.8-jdbc.jar Then - as recommended - I created CREATE USER "voc" WITH PASSWORD 'voc' NAME 'VOC Explorer' SCHEMA "sys"; CREATE SCHEMA "voc" AUTHORIZATION "voc"; ALTER USER "voc" SET SCHEMA "voc"; and added my test table CREATE TABLE databasetest ( "id" INTEGER , "ubyte1" SMALLINT , "ubyte2" SMALLINT , "ubyte3" SMALLINT , "ubyte4" SMALLINT , "ubyte5" SMALLINT , "ubyte6" SMALLINT , "ubyte7" SMALLINT , "ubyte8" SMALLINT , "ubyte9" SMALLINT , "ubyte10" SMALLINT , "ubyte11" SMALLINT , "ubyte12" SMALLINT , "ubyte13" SMALLINT , "ubyte14" SMALLINT , "ubyte15" SMALLINT , "smallint1" SMALLINT , "smallint2" SMALLINT , "smallint3" SMALLINT , "smallint4" SMALLINT , "smallint5" SMALLINT , "smallint6" SMALLINT , "smallint7" SMALLINT , "smallint8" SMALLINT , "smallint9" SMALLINT , "smallint10" SMALLINT , "smallint11" SMALLINT , "smallint12" SMALLINT , "smallint13" SMALLINT , "smallint14" SMALLINT , "smallint15" SMALLINT , "float1" DOUBLE , "float2" DOUBLE , "float3" DOUBLE , "float4" DOUBLE , "float5" DOUBLE , "float6" DOUBLE , "float7" DOUBLE , "float8" DOUBLE , "float9" DOUBLE , "float10" DOUBLE , "float11" DOUBLE , "float12" DOUBLE , "float13" DOUBLE , "float14" DOUBLE , "float15" DOUBLE , "float16" DOUBLE , "float17" DOUBLE , "float18" DOUBLE , "float19" DOUBLE , "float20" DOUBLE ) ; The test data contains positive and negative numbers, and the following R script (to be executed with R from www.r-project.org) allows to replicate the problem # R-script commands for replication of problem with prepared statement inserts in MonetDB # You find the installer for the R interpreter under http://cran.at.r-project.org (windows version 2.8.0) # (linux and sourcecode versions also available) # just install and start R # then install the RODBC package (version 1.2-4) install.packages("RODBC") # then paste the following lines # activate ODBC package library(RODBC) # only if we deactivate autocommit the prepared statement inserts work correctly (and much faster) # this variable switches autocommit of and sends standard commit at end of prepared statements. noAutoCommit <- TRUE channel <- odbcDriverConnect(connection = "DSN=MonetDB;UID=voc;PWD=voc;DATABASE=demo;host=localhost;port=50000;" , case="tolower" , believeNRows = TRUE , colQuote='"' , tabQuote = '"' , DBMSencoding = "" , rows_at_time = 1000 , bulk_add = "yes" ) if (noAutoCommit) odbcSetAutoCommit(channel, autoCommit=FALSE) # test connection #sqlTables(channel) #sqlColumns(channel, "databasetest") b <- 1L # number of batches n <- 10000L # batch size system.time({ for (i in 1:b){ cat("i=", i, "\n") x <- data.frame( id = ((i-1L)*n+1L):(i*n) # 4-byte integer record identifier , ubyte1 = sample(0:35, n, TRUE, prob=NULL) # signed byte columns (1 byte) , ubyte2 = sample(0:1, n, TRUE, prob=NULL) , ubyte3 = sample(0:3, n, TRUE, prob=NULL) , ubyte4 = sample(0:9, n, TRUE, prob=NULL) , ubyte5 = sample(0:24, n, TRUE, prob=NULL) , ubyte6 = sample(0:99, n, TRUE, prob=NULL) , ubyte7 = sample(0:127, n, TRUE, prob=NULL) , ubyte8 = sample(0:1, n, TRUE, prob=c(0.75, 0.25)) , ubyte9 = sample(0:1, n, TRUE, prob=c(0.9,0.1)) , ubyte10 = sample(0:1, n, TRUE, prob=c(0.96,0.04)) , ubyte11 = sample(0:1, n, TRUE, prob=c(0.99,0.01)) , ubyte12 = sample(0:1, n, TRUE, prob=c(0.996,0.004)) , ubyte13 = sample(0:1, n, TRUE, prob=c(0.999,0.001)) , ubyte14 = sample(0:1, n, TRUE, prob=c(0.9996,0.0004)) , ubyte15 = sample(0:1, n, TRUE, prob=c(0.9999,0.0001)) , smallint1 = sample(1:4000, n, TRUE, prob=NULL) # smallint columns (2 bytes) , smallint2 = sample(1:2, n, TRUE, prob=NULL) , smallint3 = sample(1:4, n, TRUE, prob=NULL) , smallint4 = sample(1:10, n, TRUE, prob=NULL) , smallint5 = sample(1:25, n, TRUE, prob=NULL) , smallint6 = sample(1:100, n, TRUE, prob=NULL) , smallint7 = sample(1:256, n, TRUE, prob=NULL) , smallint8 = sample(1:1000, n, TRUE, prob=NULL) , smallint9 = sample(1:32000, n, TRUE, prob=NULL) , smallint10 = sample(0:1, n, TRUE, prob=c(0.75, 0.25)) , smallint11 = sample(0:1, n, TRUE, prob=c(0.9,0.1)) , smallint12 = sample(0:1, n, TRUE, prob=c(0.96,0.04)) , smallint13 = sample(0:1, n, TRUE, prob=c(0.99,0.01)) , smallint14 = sample(0:1, n, TRUE, prob=c(0.996,0.004)) , smallint15 = sample(0:1, n, TRUE, prob=c(0.999,0.001)) , float1 = round(runif(n, 0, 100), 1) # float columns (4 bytes) , float2 = round(runif(n, 0, 100), 1) , float3 = round(runif(n, 0, 100), 1) , float4 = round(runif(n, 0, 100), 1) , float5 = round(runif(n, 0, 100), 1) , float6 = round(runif(n, 0, 100), 1) , float7 = round(runif(n, 0, 100), 1) , float8 = round(runif(n, 0, 100), 1) , float9 = round(runif(n, 0, 100), 1) , float10 = round(runif(n, 0, 100), 1) , float11 = round(runif(n, 0, 100), 1) , float12 = round(runif(n, 0, 100), 1) , float13 = round(runif(n, 0, 100), 1) , float14 = round(runif(n, 0, 100), 1) , float15 = round(runif(n, 0, 100), 1) , float16 = round(runif(n, 0, 100), 1) , float17 = round(runif(n, 0, 100), 1) , float18 = round(runif(n, 0, 100), 1) , float19 = round(runif(n, 0, 100), 1) , float20 = round(runif(n, 0, 100), 1) ) sqlSave(channel, x, tablename="databasetest", rownames=FALSE, append=TRUE, fast=TRUE) if (noAutoCommit) odbcEndTran(channel, commit=TRUE) print(sqlQuery(channel, 'SELECT count(*) as cc, min(id) as minid, max(id) as maxid from databasetest;')) } }) sqlQuery(channel, 'SELECT count(*) as cc, min(id) as minid, max(id) as maxid from databasetest;') close(channel)

1 0

[MonetDB-users] a couple of questions
by Loredana Afanasiev 20 Nov '08

20 Nov '08

(I might have send this first to the wrong email address. Here's a repetition.) Hi team, I have a couple of small questions and remarks: * mclient timer doesn't work when running mclient with -s option. [lafanasi@zookst4 stats]$ mclient -t -lx -s "count(doc('xmark01.xml'))" 1 [lafanasi@zookst4 stats]$ mclient -lx -t q1.xq 1 Timer 5.606 msec * does mclient option -I (capital i) work? For example, the following command doesn't execute in 1 min. [lafanasi@zookst4 stats]$ mclient -lx -I xmark01.xml -s "count(.)" * Recently I couldn't access my db created with the latest release after recompilation from HEAD. Do I need to back-up my db (by dumping the tables) when I recompile a new version from HEAD? A new release? * Mserver crushes on XMark of 11GB (-f 100). The query is: "find the maximum length of the root to leaf paths, i.e., the tree depth". My specs: Linux zookst4 2.6.24.3-50.fc8 #1 SMP Thu Mar 20 13:39:08 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux, Mem 19GB. Do you have any tips on how to make it run? Thanks for your help! l.

2 1

[MonetDB-users] MonetDB binary distribution question
by Hans van Rijswijk (DT) 14 Nov '08

14 Nov '08

Dear MonetDB users, Our install procedure for MonetDB currently consists of unpacking MonetDB-Aug2008-SuperBall.tar.bz2 and running the monetdb-install.sh script. This takes a long time to complete. We would like to get or create a binary distribution for a Linux system (Ubuntu 8.04 / Fedora). For this, we like to know the following: - Is the Monet binary dependent on the system where it is built? - Can we pack the generated binaries and install them on a different Linux (Ubuntu/Fedora) computer? - Can we even install this binary in a different directory? - What external dependencies are used? - Which paths are used and which paths settings are assumed in the binary? In other words: what is necessary for a copied distribution to work correcly on a different computer and/or a different directory as for which it was built. Thanks in advance for your explanatory answer. Regards, Hans van Rijswijk Netherlands Forensic Institute Tel: 070-8886638

3 2

[MonetDB-users] Table trace in file system
by Alfred Nordman 13 Nov '08

13 Nov '08

Hi there, Is there any way to export\import tables between machines by only coping files and\or folders Thanks, Alfred -- View this message in context: http://www.nabble.com/Table-trace-in-file-system-tp20478519p20478519.html Sent from the monetdb-users mailing list archive at Nabble.com.

3 2

[MonetDB-users] MonetDB install script problem.
by Hans van Rijswijk (DT) 13 Nov '08

13 Nov '08

Dear monetdb-users, When installing MonetDB (compile from MonetDB-Aug2008-SuperBall.tar.bz2) we encountered a problem. We had Java JDK installed, but no Internet connection. The monetdb-install.sh script stopped with an error message: fetching MonetDB-java failed! After some investigation we found what the problem might be: In line 209 and following lines the monetdb-install.sh script tests whether a java compiler and ant are present. If present the fetch() procedure tries to get the Java sources from the tarball. Since the Java code is not included in the tarball, fetch() tries to get the Java sources from the Internet, with wget. Since we have no Internet connection this action fails too, and makes it fatal. In other runs with only JRE installed the monetdb-install.sh script did not complain, and just skipped the Java code. Our questions: - Is the Java code deliberately omitted in the source tarball. - Can someone at the CWI fix it: either include the Java code or make the script less dependent on Internet connection. Regards, Hans van Rijswijk Netherlands Forensic Institute Tel: 070-8886638

2 1

[MonetDB-users] More timing information in MonetDB/XQuery
by pitro55 13 Nov '08

13 Nov '08

Hi, I am running some XQuery queries on MonetDB/XQuery (june release) and I cannot get all timing information like trans, fetch, shred, print time. All I can get is one time ofomration Timer, which I guess is overall information about how much time did the query take. I tried to run queries both in session and from shell: 1. xquery>\t xquery>\<test.xq 2. mclient -lx -t test.xq My OS is MAC OS 10.5 (Leopard). When I tried it on Windows, I will get all the timing information. Thank you for your time and help. Peter -- View this message in context: http://www.nabble.com/More-timing-information-in-MonetDB-XQuery-tp20475840p… Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] JDBC Support
by nori 11 Nov '08

11 Nov '08

Hi, Is there a plan to support JDK(JRE)6 in JDBC driver? Regards. Nori.

2 2

[MonetDB-users] MonetDB server thread architecture
by xxiao 28 Oct '08

28 Oct '08

Hi, does anyone know whether there is any doc on how the server is implemented in Pthreads, such as how many threads are forked and what job each different thread is responsible for? Thanks a lot. -- View this message in context: http://www.nabble.com/MonetDB-server-thread-architecture-tp20134440p2013444… Sent from the monetdb-users mailing list archive at Nabble.com.

2 9

[MonetDB-users] How to print a BAT with MIL?
by Chaojing Yang 26 Oct '08

26 Oct '08

Dear all, I'm using MonetDB/XQuery. After i have added a few XML-files into MonetDB, i want to check out how their BATs looks like. /MonetDB>ls(); #---------------------------------------------------------------------------------------------------------------------------------# # name htype ttype count heat dirty status kind refcnt lrefcnt # # str str str lng int str str str int int # #---------------------------------------------------------------------------------------------------------------------------------# [ "1000000001_attr_own", "void", "oid", 5, 0, "clean", "disk", "pers", 0, 2 ] [ "1000000001_attr_prop", "void", "oid", 5, 0, "clean", "disk", "pers", 0, 2 ] [ "1000000001_attr_qn", "void", "oid", 5, 0, "clean", "disk", "pers", 0, 2 ] [ "1000000001_frag_root", "oid", "oid", 1, 0, "clean", "disk", "pers", 0, 1 ] [ "1000000001_map_pid", "void", "oid", 1, 0, "clean", "disk", "pers", 0, 2 ] [ "1000000001_nid_rid", "void", "oid", 25, 0, "clean", "disk", "pers", 0, 2 ] [ "1000000001_prop_com", "void", "str", 0, 0, "clean", "disk", "pers", 0, 2 ] [ "1000000001_prop_ins", "void", "str", 0, 0, "clean", "disk", "pers", 0, 2 ]/ *I get a list of BAT-data summary. But i cann't figure out how to use print functions from buildin module as listed:* / / 231 .COMMAND print(ANY value) = CMDprint_val; 232 "Print a MIL value between square brackets." 233 234 .COMMAND print(BAT[ANY::1,ANY], ...BAT[ANY::1,ANY]...) = CMDprint_table; 235 "BATs are printed with '#' for legend lines, and the BUNs on seperate 236 lines between brackets, containing each to comma separated values 237 (head and tail). 238 If multiple BATs are passed for printing, print() performs an implicit 239 natural join, producing a multiattribute table." 240 241 .COMMAND print(INT order, BAT[ANY::1,ANY], ...BAT[ANY::1,ANY]...) 242 = CMDprint_otable; 243 "The same as normal table print, but enforces to use the order of 244 BAT number [1..argc] to do the printing." 245 246 .COMMAND table(BAT[any::1,any], ...BAT[any::1,any]...) = print_table_default; 247 "Print an n-ary table. Like print(n-ary) but does not print oid column" 248 249 .COMMAND table(INT order, BAT[any::1,any], ...BAT[any::1,any]...) = print_table; 250 "Print an n-ary table. Use the ordering of param [1..argc]" 251 252 .COMMAND multitable(BAT[any,bat]) = CMDprint_multitable; 253 "Print an n-ary table contained within a BAT of BATs" 254 255 .COMMAND printf(str format, ...) = print_format; 256 "Print a formatted string on the standard client output." 257 258 .COMMAND sprintf(str format, ...) : str = print_str; 259 "Print a formatted string on the standard client output." 260 261 262 .END builtin; One single example may help me out of this trap. Best regards, Chaojing ------------------------------------------------------------------------------------------------------------------------------------------ Email: chaojing.yang(a)gmail.com Blog: http://goethe-ist-taoist.blogspot.com/ §1 Ich widerspreche der Nutzung oder Übermittlung meiner Daten für Werbezwecke oder sowie andere unbefugte Verwendungszwecke. (§ 28 Abs. 4 BDSG=Bundesdatenschutzgesetz). §2 Durch die Datenübertragung bzw. Kommunikation per E-Mail liegt ein Verstoß gegen das BDSG vor, da ihre Daten insoweit nicht ausreichend geschützt sind. Ich bitte Sie daher, mir schriftlich zu bestätigen, dass Sie mit dem Verstoß gegen das BDSG einverstanden sind und keine Schadensersatzanspräche bzw. Unterlassungsanspräche mir gegenüber geltend machen. §3 Der Inhalt dieser Nachricht ist vertraulich und nur für den oben angegebenen Empfänger bestimmt.

3 3

[MonetDB-users] Using MonetDB on Windows X64
by Daniel Murray 24 Oct '08

24 Oct '08

Hi all, I am trying to use MonetDB on a 32bit Windows XP SP2, but am having a lot of trouble given the size of my tables. Is there any experience in regards to using MOnetDB on a "virtual" server set up on Windows 2003 Server and configuring it with "X64" inlieu of a true "64bit". Thanks you in advance for your support. Daniel Murray G.

1 0

[MonetDB-users] Use cases for monetdb
by steve 24 Oct '08

24 Oct '08

What are the use cases for monetdb?

3 3

[MonetDB-users] MonetDB5 Server crashes and shuts-down
by Daniel Murray 22 Oct '08

22 Oct '08

Hi All, I am working with Windows XP SP2 (32-bit), with 2Gb RAM; am using Java 1.5. I downloaded MonetDB5-SQL-Installer-i386-20080624.msi; Squirell has been update to use monetdb-1.8-jdbc.jar. I am trying to upload data from MySQL to MonetDB5, using the MonetDB Bulk Loader that comes with Pentaho Kettle 3.1. I am using a MySQL table with a little over 9 millon rows & 7 columns (2 BIGINT, 2 INT and 3 FLOAT, consequently I understand no encoding issues are involved). After a number of failed transformation attempts to load my data to MonetDB, on all attempts the Monetdb Server is suddenly shutting down after some 20,000 records are processed. When trying to review the data -that I understand to be in MonetDB- using Squirell, the following server console screen out is shown (I had to modify the .config file to "gdk_debug=1") and as likewise, the MonetDB Server shuts down: ***console output start*** C:\MONETDB\CWI\MonetDB5>MSQLServer.bat # MonetDB server v5.6.0, based on kernel v1.24.0 # Serving database 'demo' # Compiled for i686-pc-win32/32bit with 32bit OIDs dynamically linked # Copyright (c) 1993-2008 CWI, all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see C:\MONETDB\CWI\MonetDB5\etc\monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.24.0 loaded >!01 ERROR: MT_mmap: MapViewOfFileEx(150c, 1, 0, 0, 196885248, 0) failed !01 OS: Espacio de almacenamiento insuficiente para procesar este comando. #01 #GDKmmap(196885248) fail => BBPtrim(enter) usage[mem=4583848,vm=1158283264] #01 #mallinfo.arena = 5927701 #01 #mallinfo.ordblks = 61996 #01 #mallinfo.smblks = 383 #01 #mallinfo.hblkhd = 0 #01 #mallinfo.hblks = 0 #01 #mallinfo.usmblks = 1768421 #01 #mallinfo.fsmblks = 345928 #01 #mallinfo.uordblks = 3811784 #01 #mallinfo.fordblks = 1568 #01 #BBPTRIM_ENTER: memsize=4583848,vmsize=1158283264 #01 #BBPTRIM: memtarget=0 vmtarget=1073741824 #01 #TRIMSCAN: mem=0 vm=1, start=1, limit=1 #01 #TRIMSCAN: 92596* 0=tmp_2143 (#20500000) #01 #TRIMSCAN: 92598* 1=tmp_2144 (#20510000) #01 #TRIMSCAN: 92599* 2=tmp_2145 (#20510000) #01 #TRIMSCAN: 92600* 3=tmp_2146 (#20510000) #01 #TRIMSCAN: 92601* 4=tmp_2147 (#20510000) #01 #TRIMSCAN: 92602* 5=tmp_2151 (#20500000) #01 #TRIMSCAN: end at 1 (size=1197) #01 #TRIMSELECT: dirty = 0 #01 #TRIMSELECT: candidate=tmp_2143 BAT*=454e700 #01 # (cnt=20500000, mode=4096, refs=0, wait=0, parent=0,0, lastused= 92598,92596,-2147391052) #01 #TRIMSELECT: delete tmp_2143 from trimlist (has been touched since last scan ) #01 #TRIMSELECT: candidate=tmp_2144 BAT*=454e7f8 #01 # (cnt=20510000, mode=4096, refs=0, wait=0, parent=0,0, lastused= 92599,92598,-2147391050) #01 #TRIMSELECT: delete tmp_2144 from trimlist (has been touched since last scan ) #01 #TRIMSELECT: candidate=tmp_2145 BAT*=454e8f0 #01 # (cnt=20510000, mode=4096, refs=0, wait=0, parent=0,0, lastused= 92600,92599,-2147391049) #01 #TRIMSELECT: delete tmp_2145 from trimlist (has been touched since last scan ) #01 #TRIMSELECT: candidate=tmp_2146 BAT*=454e9e8 #01 # (cnt=20510000, mode=4096, refs=0, wait=0, parent=0,0, lastused= 92601,92600,-2147391048) #01 #TRIMSELECT: delete tmp_2146 from trimlist (has been touched since last scan ) #01 #TRIMSELECT: candidate=tmp_2147 BAT*=454eae0 #01 # (cnt=20510000, mode=4096, refs=0, wait=0, parent=0,0, lastused= 92602,92601,-2147391047) #01 #TRIMSELECT: delete tmp_2147 from trimlist (has been touched since last scan ) #01 #TRIMSELECT: candidate=tmp_2151 BAT*=454ecd0 #01 # (cnt=20500000, mode=4096, refs=0, wait=0, parent=0,0, lastused= 92596,92602,-2147391046) #01 #TRIMSELECT: keep tmp_2151 [82035748,82035520] bytes [82035748,82035520] dir ty target(mem=0 vm=1073741824) #01 #TRIMSELECT: end #01 #TRIMSELECT: dirty = 1 #01 #TRIMSELECT: candidate=tmp_2151 BAT*=454ecd0 #01 # (cnt=20500000, mode=4096, refs=0, wait=0, parent=0,0, lastused= 92596,92602,-2147391046) #01 #TRIMSELECT: unload tmp_2151 [82035748,82035520] bytes [82035748,82035520] d irty #01 #TRIMSELECT: end #01 #BBPTRIM: lastused batid name #01 #BBPTRIM: -2147391046 1129 tmp_2151 #01 #BBPTRIM_EXIT: memsize=114688,vmsize=1076232192 #01 #GDKmmap(196885248) fail => BBPtrim(ready) usage[mem=4583792,vm=1076232192] #01 #mallinfo.arena = 5927717 #01 #mallinfo.ordblks = 61993 #01 #mallinfo.smblks = 384 #01 #mallinfo.hblkhd = 0 #01 #mallinfo.hblks = 0 #01 #mallinfo.usmblks = 1768421 #01 #mallinfo.fsmblks = 346232 #01 #mallinfo.uordblks = 3811728 #01 #mallinfo.fordblks = 1336 !01 ERROR: MT_mmap: MapViewOfFileEx(1510, 1, 0, 0, 196885248, 0) failed !01 OS: Espacio de almacenamiento insuficiente para procesar este comando. !01 ERROR: GDKload: cannot mmap(): name=21\2150, ext=tail !01 OS: Espacio de almacenamiento insuficiente para procesar este comando. !01 ERROR: GDKload failed: name=21\2150, ext=tail ***console output ends*** Furthermore, I have tried to find the folders holding \var\MonetDB5\dbfarm with no luck. Ie, they are not in the MonetDB folders that are installed when using the Windows Installer referenced above. I would very much appreciate any asisstance you can provide, as my interest is to actively use MonetDb in my DW / BI projects. Kind Regards, Daniel Murray

1 0