- users-list - monetdb.org

[MonetDB-users] Add primary key performance
by paulr135 13 Jan '11

13 Jan '11

I've loaded a 27GB (~165M rows) file into a table in MonetDB and now I am trying to add a primary key which consists of 4 fields - 2 varchars and 2 ints. It has been running for a few days (since Friday of last week). I've followed the instructions laid out on this post: http://old.nabble.com/MonetDB-SQL:-bulk-loading-large-data-sets---adding-ke… which involves setting the optimizer variable to "nov2009_pipe". I am unable to log in via mclient - I am able to put in my credentials but it just hangs at the "Welcome to mclient, the MonetDB/SQL interactive terminal (Jun2010-SP2)" line. So I have a few questions: 1. Is there any way I can see the status of the alter table ... add primary key command? 2. Is it possible to stop the alter table gracefully? Or will I have to shut the db down? 3. Is there anything I should do in future to speed up adding of primary keys? Thanks. -- View this message in context: http://old.nabble.com/Add-primary-key-performance-tp30397515p30397515.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 3

[MonetDB-users] Consultation about TPCH throughput test errors with MonetDB
by 金炳娥 12 Jan '11

12 Jan '11

Hi, Recently I has a problem when running TPCH with MonetDB. Follows are the configuration. Hardware:amd64 OS:Linux MonetDB Version:Oct2010 I loaded 100G' data to the MonetDB.Using TPCH Tester tools,I runned power test.After power test successed,running throughput test. There is an error. ERR TPCH[17170]: *** glibc detected *** /mnt/sdb1/tpch/MonetDB/install/bin/mserver5: free(): invalid pointer: 0x00002ac5381b5008 *** 2011-01-04 14:00:41 ERR TPCH[17170]: ======= Backtrace: ========= 2011-01-04 14:00:41 ERR TPCH[17170]: /lib64/libc.so.6[0x3aa5c71ce2] 2011-01-04 14:00:41 ERR TPCH[17170]: /lib64/libc.so.6(cfree+0x8c)[0x3aa5c7590c] 2011-01-04 14:00:41 ERR TPCH[17170]: /mnt/sdb1/tpch/MonetDB/install/lib/libbat.so.1[0x2b67e8c062f6] 2011-01-04 14:00:41 ERR TPCH[17170]: /mnt/sdb1/tpch/MonetDB/install/lib/libbat.so.1(HEAPfree+0xb9)[0x2b67e8b36511] 2011-01-04 14:00:41 ERR TPCH[17170]: /mnt/sdb1/tpch/MonetDB/install/lib/libbat.so.1(BATdelete+0x196)[0x2b67e8e1769a] 2011-01-04 14:00:41 ERR TPCH[17170]: /mnt/sdb1/tpch/MonetDB/install/lib/libbat.so.1[0x2b67e8b33697] 2011-01-04 14:00:41 ERR TPCH[17170]: /mnt/sdb1/tpch/MonetDB/install/lib/libbat.so.1[0x2b67e8b32fa8] 2011-01-04 14:00:41 ERR TPCH[17170]: /mnt/sdb1/tpch/MonetDB/install/lib/libmonetdb5.so.5[0x2b67e7f1da68] 2011-01-04 14:00:41 ERR TPCH[17170]: /mnt/sdb1/tpch/MonetDB/install/lib/libmonetdb5.so.5[0x2b67e7f1f6f1] 2011-01-04 14:00:41 ERR TPCH[17170]: /lib64/libpthread.so.0[0x3aa6806367] 2011-01-04 14:00:41 ERR TPCH[17170]: /lib64/libc.so.6(clone+0x6d)[0x3aa5cd30ad] 2011-01-04 14:00:41 ERR TPCH[17170]: ======= Memory map: ======== 2011-01-04 14:00:41 ERR TPCH[17170]: 00400000-00404000 r-xp 00000000 08:11 179159655 /mnt/sdb1/tpch/MonetDB/install/bin/mserver5 2011-01-04 14:00:41 ERR TPCH[17170]: 00604000-00605000 rw-p 00004000 08:11 179159655 /mnt/sdb1/tpch/MonetDB/install/bin/mserver5 2011-01-04 14:00:41 ERR TPCH[17170]: 00605000-00612000 rw-p 00605000 00:00 0 2011-01-04 14:00:41 ERR TPCH[17170]: 0a03f000-0e488000 rw-p 0a03f000 00:00 0 [heap] 2011-01-04 14:00:41 ERR TPCH[17170]: 4001c000-4001d000 ---p 4001c000 00:00 0 2011-01-04 14:00:41 ERR TPCH[17170]: 4001d000-4021d000 rw-p 4001d000 00:00 0 2011-01-04 14:00:41 ERR TPCH[17170]: 40287000-40288000 ---p 40287000 00:00 0 … 2011-01-04 14:00:41 ERR TPCH[17170]: 4bc66000-4be66000 rw-p 4bc66000 00:00 0 2011-01-04 14:00:41 ERR TPCH[17170]: 3aa5800000-3aa581c000 r-xp 00000000 fd:00 95158276 /lib64/ld-2.5.so 2011-01-04 14:00:41 ERR TPCH[17170]: 3aa5a1b000-3aa5a1c000 r--p 0001b000 fd:00 95158276 /lib64/ld-2.5.so 2011-01-04 14:00:41 ERR TPCH[17170]: 3aa5a1c000-3aa5a1d000 rw-p 0001c000 fd:00 95158276 /lib64/ld-2.5.so 2011-01-04 14:00:41 ERR TPCH[17170]: 3aa5c00000-3aa5d4c000 r-xp 00000000 fd:00 95158283 /lib64/libc-2.5.so 2011-01-04 14:00:41 ERR TPCH[17170]: 3aa5d4c000-3aa5f4c000 ---p 0014c000 fd:00 95158283 /lib64/libc-2.5.so The test data are correct. Do you know the reason above errors? I hope to have a letter from you before long.With every good wish. Yours sincerely， Binge Jin

2 1

[MonetDB-users] Concurrent aggregation crashes MonetDB
by Dávid Karnok 06 Jan '11

06 Jan '11

Hi! I'm running an evaluation test on the latest MonetDB/SQL stable version and always crashes possibly due an out-of-memory problem. My system: Windows 7 x86, 4GB memory, Core i7 920, data files are on an SSD disk MonetDB (default install): # MonetDB server v5.22.1, based on kernel v1.40.1 # Release Oct2010 # Serving database 'demo', using 8 threads # Compiled for i686-pc-win32/32bit with 32bit OIDs dynamically linked # Found 3.186 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see C:\Program Files\MonetDB\MonetDB5\etc\monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.40.1 loaded Using monetdb-1.17-jdbc driver. The test involves a simple, two column table: CREATE TABLE a (text VARCHAR(50), value INT) Which is then filled with 10.000.000 records of random values. The data files take up to 150MB disk space. The test routine creates multiple threads which connect to the local demo database, and simultaneously execute the same aggregation query. The test simulates a concurrent analysis-data mining scenario running on the same data. void parallelTest(int threadCount) { ExecutorService exec = Executors.newFixedThreadPool(threadCount); final CountDownLatch cnt = new CountDownLatch(threadCount); for (int i = 0; i < threadCount; i++) { final int j = i; exec.submit(new Runnable() { @Override public void run() { try { Class.forName("nl.cwi.monetdb.jdbc.MonetDriver"); Connection conn = DriverManager.getConnection("jdbc:monetdb://localhost/demo", "monetdb", "monetdb"); try { conn.setAutoCommit(false); long t1 = System.currentTimeMillis(); Statement stmt = conn.createStatement(); try { cnt.countDown(); cnt.await(); ResultSet rs = stmt.executeQuery("SELECT text, AVG(value) FROM a GROUP BY text "); while (rs.next()) { rs.getString(1); rs.getDouble(2); } System.out.printf("[%02d] Select time: %.2fs%n", j, (System.currentTimeMillis() - t1) / 1000.0); } finally { stmt.close(); } } finally { conn.close(); } } catch (Exception ex) { ex.printStackTrace(); } } }); } exec.shutdown(); try { exec.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS); } catch (InterruptedException ex) { } } I noticed that if the threadCount >= 4 the MonetDB console starts to print VirtualAlloc failed messages and sooner or later terminates. The memory usage seems to jump by 600MB, e.g. with 3 threads it consumes 1.7 GB memory, but with 4 threads, it can't consume 2.4GB. Replacing the query with a single result "SELECT MAX(VALUE) FROM a" causes the same crash after 5 threads. The same test running with the latest MySQL works perfectly, although slowly. No crashes or memory issues. Is this a memory leak in MonetDB or the result of its architecture, meaning I should use a 64bit system with lots of RAM? Regards, David Karnok

1 0

[MonetDB-users] MonetDB LIMIT usage
by Brett Spurrier 05 Jan '11

05 Jan '11

Hi all, I am trying to use the LIMIT clause in my query, and while this should be fairly straight froward, I am having trouble learning the differences between Monet's LIMIT and other T-sql LIMITS. I have the simple query: SELECT * FROM signatures ORDER BY scale ASC LIMIT 1; which works great and as expected. However, I really want the second record, so in other databases I can use: SELECT * FROM signatures ORDER BY scale ASC LIMIT 1, 1; I have also tried: SELECT * FROM signatures ORDER BY scale ASC LIMIT 1 1; However, these both fail in MonetDB. According to: http://monetdb.cwi.nl/SQL/Documentation/Queries.html#Queries it looks like it should be supported. Does anyone have any insight as to how to grab the second record in MonetDB (I.e., limiting the record results to one with an offset of one)??? I am at a loss here. Cheers, Brett

3 4

[MonetDB-users] Issues with commandline copy into and opening files
by Lex Slaghuis 03 Jan '11

03 Jan '11

Hi All Eversince I updated my system; The following blockbased import script doesn't work anymore: #!/bin/bash BLOCKSIZE=250000 #RECORDCOUNT L=$(wc -l $1 | awk '{ print $1}'); #MERGE BLOCKS INTO TABLE for I in $(seq 0 $BLOCKSIZE $L); do echo "$BLOCKSIZE $I $1/$L loading... "; mclient --time -d test --statement="COPY $BLOCKSIZE OFFSET $I records into \ wptool.wpstats from '$1' using delimiters ' ','\n','' null as '';"; echo "$BLOCKSIZE $I $1/$L loading...finished "; done Returns the following error (it used to work): IOException:streams.open:Could not open file ERROR: mnstr_open: could not open file '/var/www/loadstats/pagecounts-20110102-160000' The script is started through: sh importintodb.sh /var/www/loadstats/pagecounts-20110102-160000 Does anyone has some thoughts on this, or should I report a bug? I would really like to use commandline, as the hourly imports are 6M records big... The .monetdb file is placed locally and automatically authenticates with monetdb (admin). Lang=SQL, history=yes Regards Lex Slaghuis

2 1

[MonetDB-users] Monnetdb/Xquery
by Ramachandran, Prakash 30 Dec '10

30 Dec '10

Hi, I am currently evaluating software's for metadata storage that can be used in our product. I came across your product and found it very interesting and we might consider using it for our metadata storage. I just wanted to know if Monnetdb provides commercial support, if so can you please provide the detail on sales contacts. Thank in advance. Regards, Prakash R

1 0

[MonetDB-users] MAPI interface error
by Brett Spurrier 24 Dec '10

24 Dec '10

Hi all, I am having a bit of trouble and I am hoping a seasoned MonetDB expert can lend some advice. I am using the MAPI interface in my C/C++ application to talk to my MonetDB instance. I have a recursive INSERT INTO statement which populates the database, and it initially works pretty well. However, as I populate more (past 200 inserts in my case), I get a MAPI error (exampled by: mapi_explain_result()): MAPI = monetdb@localhost:50000 ACTION= mapi_start_talking ERROR = !Connection terminated No real reason as to why. Has anyone seen this before? My code: MapiHdl query(Mapi dbh, char *q) { MapiHdl ret = NULL; if ((ret = mapi_query(dbh, q)) == NULL || mapi_error(dbh) != MOK) die(dbh, ret); return(ret); } void update(Mapi dbh, char *q) { MapiHdl ret = query(dbh, q); if (mapi_close_handle(ret) != MOK) die(dbh, ret); } void main() { dbh = mapi_connect("localhost", 50000, "monetdb", "monetdb", "sql", "dbname"); if (mapi_error(dbh)) { die(dbh, hdl); } for (int i=0; i< 2500; i++) { char *p = INSERT_INTO_STATEMENT; update(dbh, p); } } Does anyone see any reason that the MAPI would choke mid way through the For loop? Any help would be majorly appreciated. I am at a loss. Cheers, Brett

3 14

[MonetDB-users] Announcement: New Jun2010-SP1 bugfix release of MonetDB suite
by Sjoerd Mullender 21 Dec '10

21 Dec '10

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2010-SP1 bug fix release of the MonetDB suite of programs. The most important and visible change is a security fix in the SQL COPY INTO command. This is detailed in the release notes. More information (including release notes) on this release is available at <http://monetdb.cwi.nl/Development/Releases/Oct2010/>. The download location is <http://dev.monetdb.org/downloads/>. -- Sjoerd Mullender

1 0

[MonetDB-users] How to limit monetdb memory consumption on Windows
by Bianco Prieto-Ilarion 18 Dec '10

18 Dec '10

Hi all! I have a problem. I'm using Oct2010 64-bit MonetDB build. I have a table with 8M records and when I'm trying to execute LEFT JOIN (from JDBC) MonetDB is crashed. I can see that it crashed due to memory (my laptop has 8Gb) consumption is unlimited. What can you suggest (just use Linux is a final step:)) Thanks, Bina Prieto

1 0

[MonetDB-users] MonetDB Installation
by Rob Fynes 16 Dec '10

16 Dec '10

Hello, Regarding the pre-compiled binaries directories for deb distributions "karmic" and "lucid" in the downloads section, there doesn't appear to be anything for x-query. Is this intentional? Regards, -Rob

2 2