- users-list - monetdb.org

Re: [MonetDB-users] about the loadfile size
by Stefan Manegold 20 Jul '10

20 Jul '10

Dear Kun Ren, first of all, I would highly appreciate if you could stick to using the MonetDB mailing lists for such questions about MonetDB instead of sending me private email. On the one hand, this increases the chances for you to get quick and suitable answers, as more people read the mailing lists. On the other hand, other users might have similar questions and benefit from the answers sent to the mailing lists. Having said that, given that you mention neither the version of MonetDB you're using or any specification of your Linux & Windows machine(s) (in particular whether MonetDB, the OS, and the hardware architecture are 32- or 64-bit), nor whether you get any error or not (What exactly does "auto stop mean"? What exactly happens?) I can only guess that your Linux system is 64-bit while your Windows system (or at least the MonetDB version you installed on it) is 32-bit. For a detailed discussion of the impact of 32-bit vs. 64-bit on the scalability of MonetDB, please see http://sourceforge.net/mailarchive/message.php?msg_name=20100319145945.GA91… Hope this helps. Stefan On Tue, Jul 20, 2010 at 05:19:40PM +0800, kun ren wrote: > Dear professer Stefan, > Recently,I use TPC-H test the Monetdb, On windows ,so I use "copy > ,, records,,,into,,from" to load data to Monetdb,when the size of the file > is about 700M,It is OK.But when the size is 1.4G,the Monetdb server is auto > stop when loading. But when I am in Linux, it is no problem. So I want to > ask why it is wrong on windows when the file is so large? > Best regards, -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 1

[MonetDB-users] Announcement: New Jun2010-SP1 bugfix release of MonetDB suite
by Sjoerd Mullender 16 Jul '10

16 Jul '10

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2010-SP1 bug fix release of the MonetDB suite of programs. Lots of problems have been fixed, the most important one being the fix in the handling of database upgrades of databases created with the Feb2010 release to the current version. More information (including release notes) on this release is available at <http://monetdb.cwi.nl/Development/Releases/Jun2010/>. The download location has changed to <http://dev.monetdb.org/downloads/>. Please fix any bookmarks you may have. -- Sjoerd Mullender

1 0

[MonetDB-users] php mapi
by dariuszs 15 Jul '10

15 Jul '10

Hi, PHP does not return rows from the query when the result set reaches the size about 30KB, returns fine when it's less. Thanks.

4 8

[MonetDB-users] SQL text normalisation principles in relation to MonetDB
by Stefan de Konink 15 Jul '10

15 Jul '10

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, I rather have a general question and I would like to know something about normalisation in MonetDB in relation to the underlying storage model. Since my first courses in SQL, that was PostgreSQL at the time. It was a strong advise always try to normalise duplications of data, because it would always increase your search preformance to look up numerical keys over string comparisons. Given that in MonetDB, text-strings are deduplicated in a best effort way, factually become numbers, how does this compare to the additional costs of: - enums - foreign key relations For example, some database schema's use enums (or worse: varchars) for two values. For example 'accessible', 'inaccessible'. It is clear that the storage size for this field BOOLEAN NOT NULL, would be sufficient and of the length of one bit (best case). Now within MonetDB the string data is currently optimised, would it therefore cost additional time to create a secondary table to join to vs using a string field. Thus, does normalisation for 'manual' deduplication, hurt or not? Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.15 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkw9uQAACgkQYH1+F2Rqwn2sjACeNnvDQS+cXjzs1USpomkL6rz8 7wkAn2rF2ZRvEJbQ1cX+oxJsWPcMoTk6 =RzDh -----END PGP SIGNATURE-----

3 2

[MonetDB-users] how to write function like oracle nvl()?
by Mike Jason 13 Jul '10

13 Jul '10

Hi, I need some help on how to write some special functions that monetdb currently doesn't have. E.g., I want to use the oralce funtion nvl(), which is described as follows: http://download.oracle.com/docs/cd/B19306_01/server.102/b14200/functions105… Then, how can I write a function that has the some ability as the oralce's? By the way, now I use MonetDB-Feb2010.

2 5

[MonetDB-users] MonetDB - Function Sequence Error
by r8rdarklord 13 Jul '10

13 Jul '10

I have written a multithreaded ODBC program which tries to insert data to 11 tables in MonetDB. Each thread has a connection handle, a statment handle (comprising of insert into blah!....). Each thread on instantiation creates the handles (connection, statement etc.), connects to the db, binds meta-data, prepares query and executes the insert in a loop after reading the bind values from a file. The program runs fine to a point following which: SQLExecute: Error: SQLstate HY010, Errnr 0, Message [MonetDB][ODBC Driver 1.0]Function sequence error On closer examination it's the statement handle that is not in Prepared state. I thought that this maybe because the asynchronous execution of the previous execution of the commit isn't complete yet. I recoded my commit as follows: ret_sts = SQLEndTran(SQL_HANDLE_DBC, dbch, SQL_COMMIT); while (ret_sts == SQL_STILL_EXECUTING) ret_sts = SQLEndTran(SQL_HANDLE_DBC, dbch, SQL_COMMIT); check(ret_sts, SQL_HANDLE_DBC, dbch, "SQLEndTran"); However, the ret_sts returns -1 which means a SQL_ERROR This has me perplexed. Clearly the SQL is good else it wouldn't have executed a no. of times in the first place. The stmth also look good, so there is no stepping by threads on each others memory. I can post the whole program if needed. -- View this message in context: http://old.nabble.com/MonetDB---Function-Sequence-Error-tp29148194p29148194… Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

Re: [MonetDB-users] [Monetdb-developers] about import *.csv
by Stefan de Konink 12 Jul '10

12 Jul '10

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 You must reply to the mailinglist, and not to me directly. In order to help you with your segmentation fault you also must provide what version you are using. Op 12-07-10 06:57, kun ren schreef: > Thank you, when I am trying,something is wrong, > when i use command," mclient -l sql -d demo" > and then use command "create table t1(name char(100),number int);" to > create a table; > and then I create a csv file, the content of the file is: > renkun,1 > shaolong,2 > laoluo,3 > and then i use command as you said,"copy 3 records into t1 from 1.csv > delimiters ',','\n';" but the server said:"Segmentation fault > ",what wrong with me?thanks > Best regards; > > > > > > 2010/7/12 Stefan de Konink <stefan(a)konink.de <mailto:stefan@konink.de>> > > On Mon, 12 Jul 2010, kun ren wrote: > > Dear all, > I want to import *.csv into MonetDB,what interface do i will > use？ > > > Just mclient, create your table. And use copy into so for example > > copy i<<numofrows>> records into <<justcreatedtable>> from > 'filename' delimiters ',', '\n', null as ''; > > > Stefan > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.15 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkw6oSYACgkQYH1+F2Rqwn1OzwCglSIGOl+tKxMk+bbDPPaLUX+i plwAn0aeeJKs3TLiNk4cHPURovJqh8T7 =+Tr/ -----END PGP SIGNATURE-----

3 4

[MonetDB-users] hash accelerator
by Miguel Ángel Águila Lorente 09 Jul '10

09 Jul '10

Hello, I'm working with mal and I've seen in the web documentation, in the part of the inner core that there are built-in accelerator functions. As I can see, there is one search accelerator: hashing. ¿Could you tell me how to use this accelerator using mal language? Looking forward to hearing from you. Thank you. Mike

2 2

[MonetDB-users] MonetDB scalability
by Chris 09 Jul '10

09 Jul '10

Hi, I'm currently evaluating MonetDB for business reporting, and looking for information on scalability, and I can't seem to find any information on clustering options. Is there any support in MonetDB for clustering across multiple physical servers to improve performance? Thanks, Chris

6 10

[MonetDB-users] sql_debug magic number?
by Mark Bucciarelli 06 Jul '10

06 Jul '10

What's a good value for sql_debug in monetdb5.conf? I tried 1 and 17, but got nothing in merovingian.log. Looks like perhaps something was written to dbfarm/<db>/sql_logs/sql/log, but I couldn't make any sense of it. Thanks, m

1 0

Re: [MonetDB-users] SQL: count() counts distinct?
by Ying Zhang 04 Jul '10

04 Jul '10

Hi Niels, I have just updated my stable branch with your backport. It's fixed. Thanks! Jennie On Jul 04, 2010, at 18:42 , Niels Nes wrote: > On Sun, Jul 04, 2010 at 02:49:11PM +0200, Ying Zhang wrote: >> Dear all, >> >> I have a table "a" as the following: >> >> sql>select * from a; >> +------+------+ >> | x | y | >> +======+======+ >> | 1 | 1 | >> | 1 | 2 | >> | 1 | 3 | >> | 2 | 1 | >> | 2 | 2 | >> | 2 | 3 | >> | 2 | 1 | >> | 1 | 2 | >> | 2 | 3 | >> +------+------+ >> 9 tuples >> >> I want to count the number of distinct tuples in the table, so I did the following: >> >> sql>select count(*) from (select distinct x,y from a) as b; >> +------+ >> | L17 | >> +======+ >> | 2 | >> +------+ >> 1 tuple >> sql>select count(x) from (select distinct x,y from a) as b; >> +------+ >> | L20 | >> +======+ >> | 2 | >> +------+ >> 1 tuple >> sql>select count(y) from (select distinct x,y from a) as b; >> +------+ >> | L21 | >> +======+ >> | 3 | >> +------+ >> 1 tuple >> >> It looks like that count() automatically counts the distinct values of x and y, while "select distinct x,y from b" does give me what I want: >> >> sql>select distinct x, y from b; >> +------+------+ >> | x | y | >> +======+======+ >> | 1 | 1 | >> | 1 | 2 | >> | 1 | 3 | >> | 2 | 1 | >> | 2 | 2 | >> | 2 | 3 | >> +------+------+ >> 6 tuples >> >> Is this an expected behaviour of count()? If yes, how can I get the #tuples of "select distinct x,y from b"? >> >> Thanks in advance! >> >> Jennie >> >> PS> info about mserver5: > Jenny it seems one of the m5 optimizers breaks the count (select distinct) case. > In current this seems fixed again. > > Niels >> >> $ mserver5 >> # MonetDB server v5.18.6, based on kernel v1.36.6 >> # Serving database 'demo', using 4 threads >> # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked >> # Found 7.751 GiB available main-memory. >> # Copyright (c) 1993-July 2008 CWI. >> # Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved >> # Visit http://monetdb.cwi.nl/ for further information >> #warning: please don't forget to set your vault key! >> #(see [prefix]/stable/debug/etc/monetdb5.conf) >> # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ >>> include sql; >> # MonetDB/SQL module v2.36.6 loaded >> >> >> ------------------------------------------------------------------------------ >> This SF.net email is sponsored by Sprint >> What will you do first with EVO, the first 4G phone? >> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first >> _______________________________________________ >> MonetDB-users mailing list >> MonetDB-users(a)lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-users > > -- > Niels Nes, Centrum Wiskunde & Informatica (CWI) > Science Park 123, 1098 XG Amsterdam, The Netherlands > room L3.14, phone ++31 20 592-4098 sip:4098@sip.cwi.nl > url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl

1 0

[MonetDB-users] SQL: count() counts distinct?
by Ying Zhang 04 Jul '10

04 Jul '10

Dear all, I have a table "a" as the following: sql>select * from a; +------+------+ | x | y | +======+======+ | 1 | 1 | | 1 | 2 | | 1 | 3 | | 2 | 1 | | 2 | 2 | | 2 | 3 | | 2 | 1 | | 1 | 2 | | 2 | 3 | +------+------+ 9 tuples I want to count the number of distinct tuples in the table, so I did the following: sql>select count(*) from (select distinct x,y from a) as b; +------+ | L17 | +======+ | 2 | +------+ 1 tuple sql>select count(x) from (select distinct x,y from a) as b; +------+ | L20 | +======+ | 2 | +------+ 1 tuple sql>select count(y) from (select distinct x,y from a) as b; +------+ | L21 | +======+ | 3 | +------+ 1 tuple It looks like that count() automatically counts the distinct values of x and y, while "select distinct x,y from b" does give me what I want: sql>select distinct x, y from b; +------+------+ | x | y | +======+======+ | 1 | 1 | | 1 | 2 | | 1 | 3 | | 2 | 1 | | 2 | 2 | | 2 | 3 | +------+------+ 6 tuples Is this an expected behaviour of count()? If yes, how can I get the #tuples of "select distinct x,y from b"? Thanks in advance! Jennie PS> info about mserver5: $ mserver5 # MonetDB server v5.18.6, based on kernel v1.36.6 # Serving database 'demo', using 4 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked # Found 7.751 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see [prefix]/stable/debug/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ >include sql; # MonetDB/SQL module v2.36.6 loaded

1 0

[MonetDB-users] Announcement: New Jun2010 feature release of MonetDB suite
by Sjoerd Mullender 02 Jul '10

02 Jul '10

It is not June anymore, but nevertheless, the MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2010 feature release of the MonetDB suite of programs. More information (including release notes) on this release is available at <http://monetdb.cwi.nl/Development/Releases/Jun2010/>. The download location has changed to <http://dev.monetdb.org/downloads/>. Please fix any bookmarks you may have. -- Sjoerd Mullender

1 2

[MonetDB-users] mclient --password
by Matthieu Guamis 01 Jul '10

01 Jul '10

Hello, Wow! the "--password" option is no more accepted by mclient utility :) Is there any alternative for automatic login? Thanks a lot -- *Matthieu Guamis* *Logo Axège <http://www.axege.com/>* /Axège// 23,rue Saint-Simon 63000 Clermont-Ferrand/ Tél: +33 (0)4 63 05 95 40 Fax: +33 (0)4.73.70.65.29 Email: matthieu.guamis(a)axege.com <mailto:matthieu.guamis@axege.com>

3 2

[MonetDB-users] Server requires 'SHA512' hash, but client support was not compiled in
by Matthieu Guamis 01 Jul '10

01 Jul '10

Hello, I get this message "Server requires 'SHA512' hash, but client support was not compiled in" when trying to connect to database. I installed MonetDB on RHEL5 system (more precisely Oracle Entreprise Linux with SHA256 not available in Openssl) from the script "monetdb-install.sh". In a previous post <http://sourceforge.net/mailarchive/forum.php?thread_name=20090731161927.GC2…> Fabian Groffen wrote: > Adding --with-password-backend=XXX to the configure call should be > enough. > But I'm not sure what does it mean (sorry) so my question is, how do you do that? Thanks a lot for your help PS: mserver5 --version MonetDB server v5.18.6 (64-bit), based on kernel v1.36.6 (64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/ for further information Found 2.9GiB available memory, 8 available cpu cores Configured for prefix: /home/admaxg/MonetDB Libraries: libpcre: 6.6 06-Feb-2006 (compiled with 6.6) openssl: OpenSSL 0.9.8e-rhel5 01 Jul 2008 (compiled with OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008) Compiled by: admaxg(a)localhost.localdomain (x86_64-unknown-linux-gnu) Compilation: gcc -O2 Linking : ld -IPA -m elf_x86_64 -- *Matthieu Guamis* *Logo Axège <http://www.axege.com/>* /Axège// 23,rue Saint-Simon 63000 Clermont-Ferrand/ Tél: +33 (0)4 63 05 95 40 Fax: +33 (0)4.73.70.65.29 Email: matthieu.guamis(a)axege.com <mailto:matthieu.guamis@axege.com>

1 0

[MonetDB-users] sql: getting (or specifying) sequence name
by Mark Bucciarelli 01 Jul '10

01 Jul '10

Hi, Given a table with a sequence that controls the primary id, is there a SQL command I can run to get the name of that sequence? Alternatively, is there a way to specify the sequence name when creating the table? Thanks, m

3 5

[MonetDB-users] full text index?
by Patrick Schäfer 30 Jun '10

30 Jun '10

Hi, is there any support for a full text index? In the documentation it says: "port PF/Tijah to support the XQuery Full-Text specification." Thank you Patrick

1 0

[MonetDB-users] How to recover data from a crash?
by 李坤 30 Jun '10

30 Jun '10

Hi all. I plan to deploy monetdb into production environment, but i don't know how to recover from a crash. I find some sql log files in var\sql_logs\$dbname\sql\, can i use these log files to recover, and how? Or are there any other methods to recover from crashes?

3 3

[MonetDB-users] problem with monetdb
by Andrea Panetta 28 Jun '10

28 Jun '10

Hi at All. I've a problem with monetdb... I need to use it with jdbcDriver eith iSQL viewer but, when I user mi driver, and i load it with my credential, I've the following error Error: ResultSet is closed. It's strange because, when I try the driver with eclipse, my programs work well but, when I use this driver with an external software, like isql viewer, it don't work. How I can do to fix this error? Thank you so much. (PS: I've the monetdb jdbc driver version 1.15, and it should be the laste updated driver) -- Panetta Andrea Via Pertini 22 87055 - San Giovanni in Fiore (CS) tf - 329 / 3554684

3 3

[MonetDB-users] MonetDB aborts when importing over 10 million rows
by Luis Fernando Kauer 26 Jun '10

26 Jun '10

Hi, I'm still trying to evaluate MonetDB but I keep getting errors when trying to import over 10 million rows to a table. The server simply aborts and closes. I'm using MonetDB on Windows XP. Whenener I restart after the server had aborted it shows messages like this: #GDKmmap(71827456) fails, try to free up space [memory in use=18552560,virtual m emory in use=1837957120] #GDKmmap(71827456) result [mem=18552560,vm=1837957120] #GDKmmap(36175872) fails, try to free up space [memory in use=16499872,virtual m emory in use=1844903936] #GDKmmap(36175872) result [mem=16499872,vm=1844903936] #GDKmmap(143130624) fails, try to free up space [memory in use=15697728,virtual memory in use=1744764928] #GDKmmap(143130624) result [mem=15697728,vm=1744764928] #GDKmmap(143130624) fails, try to free up space [memory in use=13554656,virtual memory in use=1654456320] #GDKmmap(143130624) result [mem=13554656,vm=1654456320] #GDKmmap(143130624) fails, try to free up space [memory in use=8747960,virtual m emory in use=1650917376] #GDKmmap(143130624) result [mem=8747960,vm=1650917376] #GDKmmap(143130624) fails, try to free up space [memory in use=7946840,virtual m emory in use=1531052032] #GDKmmap(143130624) result [mem=7946840,vm=1531052032] !mvc_init: unable to create system tables !SQLException:SQLinit:Catalogue initialization failed !ERROR: HEAPextend: failed to extend to 71722596 for 14\1475tail !ERROR: HEAPextend: failed to extend to 35861298 for 15\1537tail !ERROR: HEAPextend: failed to extend to 142816048 for 15\1543tail !ERROR: HEAPextend: failed to extend to 142816048 for 15\1550tail !ERROR: HEAPextend: failed to extend to 142816048 for 15\1556tail !ERROR: HEAPextend: failed to extend to 142816048 for 15\1557tail #function user.main():void; # sql.prelude(); #end main; First I was trying to import files with about 3 million rows each but it started giving errors after the 3rd file. Then I tried to import through files with about 1 million rows each and I was able to import about 14 million rows, but after about 10 million rows I was only able to import one or two files in a row without MonetDB resetting. Is this a bug or a size limit for the available memory. Anything I could try? With regards, Luis Fernando

2 5

[MonetDB-users] "Parent Directory" links in download area give 404.
by Mark Bucciarelli 25 Jun '10

25 Jun '10

(1) open http://monetdb.cwi.nl/downloads/Ubuntu/ (2) click "Parent Directory" link (3) 404 I saw same behavior after opening http://monetdb.cwi.nl/downloads/sources/Feb2010-SP2 so it may be a global issue. Thanks, m

2 1

[MonetDB-users] Copy Into and .new files
by Alex Bo. 23 Jun '10

23 Jun '10

Hi, I am doing a lot of COPY INTOs, and I have noticed that sometimes while doing this MonetDB seems to hang while its process memory is slowly rising. I then noticed that while this is happening, files with a '.new' extension are added to the dbfarm folders. Now suddenly I reached a state in which no '.new' files are created at all, and the COPY INTOs work much faster because no 'hanging' occurs in which something is loaded into memory. What's the explanation for this? What can I do in order to always have COPY INTOs write directly to the final files without the '.new' intermediates? Thanks. -- View this message in context: http://old.nabble.com/Copy-Into-and-.new-files-tp28973215p28973215.html Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

Re: [MonetDB-users] Embedded server in Java with JDBC
by Luis Fernando Kauer 23 Jun '10

23 Jun '10

Hi, Thanks for the quick reply. When I say embedded I mean like in HSQLDB, which allows to embed the db server in the application instead of starting the db server in a separete process accepting connections from all clients. My requirements are that the end user should not have to deal with the database directly, like start and stop it, not even need to install/configure it separetely. But I would like to access the data through JDBC driver because the libraries I'm working with require it. I also would like a small footprint. If that's not possible, then I would like to be able to start and stop the server through the application, even if it runs in a separete process. The application allows the user to analyse and audit a large volume of data on different tables and MonetDB seems to be really fast for this. Each user project will have its own set of tables so it would be nice to be able to set the database files where the user chooses for each project. If that's not possible I'll have to stick with different schemas for each project. I'm still having some problems importing large tables (> 10 million rows divided in 10 files with 1 million rows each) when MonetDB is reporting the following error SQLException:sql:Failed to extend the BAT, perhaps disk full SQLException:importTable:failed to import table and on the console appears many times the follwing: #GDKmmap(111673344) fails, try to free up space [memory in use=56663400,virtual memory in use=1521352704] #GDKmmap(111673344) result [mem=56663160,vm=1127612416] #GDKmmap: recovery ok. Continuing.. Is there any configuration or a way to make MonetDB work with large tables without having more memory available? Thanks for your help! Regards, Luis Fernando

2 1

[MonetDB-users] Embedded server in Java with JDBC
by Luis Fernando Kauer 22 Jun '10

22 Jun '10

Hi, I would like to use the embedded server within a Java application connecting through JDBC. Is it possible? I would probably need to start the embedded server before trying to connect through JDBC, right? Or is there a connection string that makes the jdbc driver start and use an embedded server? That would be great! If not, what's the best way to start the embedded server in Java? Thanks for this great and fast dabatase. Regards, Luis Fernando

2 1

[MonetDB-users] Mapiclient password prompt
by jz＠uva 21 Jun '10

21 Jun '10

I use the Mclient in conjunction with XQuery. When I do a CVS checkout, I get the version of Mapiclient where the -P option has disappeared, yet it still asks me for a password. My question is: how do I disable the password prompt? It is a bit too much to enter the password for thousands of queries, although I can type quite quickly. ;-) junte

3 4