- users-list - monetdb.org

Suggested correction for the Java SQLcopyinto example
by Percy Wegmann 09 Feb '13

09 Feb '13

Dear Monet team, I've been getting quite a bit of use out of your example code here: http://dev.monetdb.org/hg/MonetDB/file/tip/java/example/SQLcopyinto.java Recently, though, I ran into a problem. If the "COPY INTO" statement that I issue is wrong, for example the referenced table doesn't exist, Monet comes back with an error. The example isn't checking for this error condition and proceeds to send data anyway. With a really large batch of data, this becomes a problem because the SQL parser ends up trying to parse all that data as if it were SQL and basically chokes on it. The solution is to add something like the following after the out.newLine() on line 86: error = in.waitForPrompt(); if (error != null) { throw new Exception(error); } Cheers, -- Percy Wegmann +1 512 637 8500 ext 148

3 5

Requirement gathering and questions about binary bulk loading.
by Brandon Jackson 08 Feb '13

08 Feb '13

Hello fellow MonetDB'ers, The Pentaho MonetDB Bulk Loader transformation step is now working pretty well in our tests. http://ci.pentaho.com/view/Data%20Integration/job/Kettle contains a continuous integration build of the software if anyone wanted to play with it. New Transformation -> Steps -> Bulk Loading (category)-> MonetDB Bulk Loader Thank you for the previous information about enabling log output from the MonetDB JDBC driver. After killing a stalled "monetdb start ~/sg-farm" thread and restarting it, MonetDB decided to work really well. It was suggested early on that I take a look at the binary bulk loading approach to get the fastest results. Now that I have a streaming ascii version of a bulk loader done for Pentaho Data Integration, I wanted to come back to the binary approach and see what that might look like and what to be aware of. I read http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/BinaryBulkLoad It looks like the requirements are: 1. Write a physical file per column to the disk the monetdb farm exists on. 2. Tables with DATES, TIMESTAMPS or UTF-8 characters ie: Simplified Chinese are verboten and must be handled with COPY INTO .... USING DELIMITERS Assuming the above is a correct understanding, given the command below: copy into Tmp from ('path_to_file_i', 'path_to_file_f', 'path_to_file_s'); 1. If run from a computer separate from the machine running the monetdb server, would the 'path_to_file_i' be a pathname relative to the monetdb farm root directory on the remote server machine? 2. This process should work for appending new data on to an existing table, right? 3. Am I out of luck if my table has any of the verboten data types mentioned above mixed in with supported binary bulkloadable types; aka C-array style data types? 4. If not out of luck for a mixed type table, what is the procedure to load the table? In practice I imagine I will not need to fret too much about DATES needing a COPY INTO .... USING DELIMITERS ascii streaming approach because it is likely part of a dimension table with integer keys matching integers on a fact table, which is far more likely to have the supported types MonetDB Binary Bulk-loading likes. Is it true that MonetDB does not handle joins well, but instead prefers single, denormalized tables with many columns? >From what you know of the internals of MonetDb, is the effort to make some kind of streaming binary bulk loader worth it in terms of execution time on the database converting to native binary types? The documentation used the words, 'slightly faster', but I do not have a feel for when 'slightly' makes a big difference. Thanks again for considering these questions. Brandon

2 1

High memory usage while loading data with many columns
by Andrew Duffey 08 Feb '13

08 Feb '13

Hi, I'm loading data into a table with >20,000 columns and several hundred thousand rows using "COPY INTO". It works, but I can't seem to make it take less than about 30GB of memory. I've tried specifying the number of records as well. Is there any way to reduce memory usage when loading? Is this much memory usage expected? It's a shame because our queries take very little memory. We have to size our machine simply to handle the load. It's impressive (and very convenient) that I can actually have a table with so many columns, but are there any other limitations I might come across? I've just been trying to keep the column count in the queries low. A "select *" seems to crash MonetDB. Thanks, Andrew

2 8

Data storage capacity in monetdb
by KHALED DEHDOUH 06 Feb '13

06 Feb '13

I installed the version of windows for monetdb http://www.monetdb.org/Downloads. I installed a database column, but when I went over 30 GB of stokage given. the query response time has become very slow. Please, I have two questions: - Are this version is limited or no storage ? - I have noticed that the name of the database is by default "demo". Can i create another one with this version with 1 TB?

4 3

Create stored procedure to add a list of ID's to a table
by Richard Siebeling 05 Feb '13

05 Feb '13

Hello, is it possible to create a stored procedure in MonetDB that will have one parameter containing a string with several thousands of comma-separated ID's and insert these comma-separated ID's as separate rows into a table? I am receiving thousands of ID's from a separate webservice and want to join these ID's to a table in monetDB, aggregate the results and return them back to the client for further processing in the GUI. I tried inserting the ID's into a temporary table using insert into instructions from PHP, but that didn't gave me the performance I wanted (about 6 seconds for 10.000 ID's). So the next approach would be to create a stored procedure receiving all the ID's in one parameter, splitting them and inserting them one-by-one into the temporary table... Or is there a better approach for this problem? thanks in advance, Richard

2 1

Memory allocations in MonetDB
by Hassan, Ahmad 04 Feb '13

04 Feb '13

Hi Team, I am running monetdb with default settings. After profiling the application instructions, I have noticed that all the memory allocation are currently being performed by the following functions: GDKmalloc, GDKmallocmax, GDKzalloc, HEAPalloc So mmap and malloc are being used with in all GDK* and HEAPalloc. I have three questions please: 1) Which memory allocations of monetdb are performed through mmap and which memory allocation are performed through malloc. I know this is very open ended question but little bit info about this would be helpful. Are the BAT structures allocated through mmap or malloc? 2) One thing I noticed that, when the DB is running and even if I am not running any queries, still the DB is constantly issuing load/store instructions to Main memory. I assume that some thread in monetdb is doing bookkeeping or persistence related stuff. Can you please point out which part of the code is responsible for this? I want to exclude that part of the code from profiling 3) When we start monetdb with multiple threads, whether multiple threads use the same common virtual address memory region to allocate memory? Or each thread has their own distinct memory region? Thanks. Kind Regards, Ahmad

2 1

Dynamic where clause in a stored procedure
by Vassilis Papakonstantinou 31 Jan '13

31 Jan '13

Hi all, Is it possible in monetdb to use a dynamic where clause in a selection query inside a stored procedure? More specifically, i want to pass the clause as parameter. Thank you, Vassilis

3 2

[help] I have a simple question about monetdb query plan.
by 이수호 31 Jan '13

31 Jan '13

p{margin:0;padding:0;} Hello, I'm a student of Korea Advanced Institute of Science and Technology in Korea. I want to compare late materialization with join and invisible join on monetdb. But when i typed SQL query, i couldn't choose which query plan to use. There was no choice to me. I think query optimizer choose some join method. Can I choose join method? or control query plan? I installed Monetdb on my desk-top and followed tutorials. Everything is done. There is no error. In this situation, i tried to control query plan. So I read reference on official homepage. But I couldn't find the information that how to control query plan. Can I control query plan? If I can, I really really want to know how do that. Where can i find some example? Please give me your favor.

2 1

caught SIGTERM, starting shutdown sequence
by Tapomay Dey 29 Jan '13

29 Jan '13

MSG merovingian[32005]: caught SIGTERM, starting shutdown sequence 2013-01-29 11:40:53 MSG control[32005]: control channel closed 2013-01-29 11:41:06 MSG merovingian[32005]: caught SIGTERM, starting shutdown sequence 2013-01-29 11:41:40 MSG merovingian[32005]: caught SIGTERM, starting shutdown sequence 2013-01-29 11:41:51 MSG merovingian[32005]: caught SIGTERM, starting shutdown sequence Every time I do a monetdbd stop {path_to_farm} I keep getting this in merovingian logs. But the monetdbd process never ends. This issue was not there on the same installation of monetdb initially. This problem started occurring annoyingly frequently after I renamed my last farm and created a fresh one.(with monetdbd not running) What leads me to stop the farm is that I am unable to connect to the db using mclient(There's another app accessing the DB using JDBC-c3p0). There's is no complaint in the merovingian log. The mclient command just hangs. I get back access once I force kill the monetdbd using kill -9. But it gets back into this state after some time. Running on Feb13 branch some older revision. Ubuntu 12.10 with lots of resources. Is this a known issue. Is there a workaround/config or something.

1 3

best way to get autogenerated primary key
by Mate Birkas 29 Jan '13

29 Jan '13

Hi, this is my first post to this list, so forgive me if this is not the right place to post this. I tried to do my research (google, monetdb.org, python api help) but didn't find an answer. How can I get the auto-generated primary key (integer) after executing an insert. I want to use it in a second insert to another table that references the first. (in PostgreSQL I used the RETURNING clause for this) thanks in advance Mate

3 5

MonetDB crash / termination issue.
by Brian Hood 29 Jan '13

29 Jan '13

Hi All, I'm not quite sure if i'm doing something wrong here but what is see if i performing a 2 table join on a primary key field is tablea and tableb work fine yet tablec which has less data crashes the daemon. SELECT a.ip_dst as ip_dst, b.tcp_dport as tcp_dport, count(*) as num from "myschema".tablea a LEFT JOIN "myschema".tableb b ON (a.guid = b.guid) WHERE b.tcp_dport IS NOT NULL GROUP BY a.ip_dst, b.tcp_dport ORDER BY num DESC LIMIT 20; This one works fine. However when i perform the join from tablea to tablec I get the following. sql>SELECT a.ip_dst as ip_dst, b.udp_dport as udp_dport, count(*) as num from "myschema".tablea a LEFT JOIN "myschema".tablec b ON (a.guid = b.guid) WHERE b.udp_dport IS NOT NULL GROUP BY a.ip_dst, b.udp_dport ORDER BY num DESC LIMIT 20 more>; Connection terminated I've added alot more detail to the pastebin link. I had the same issue with 11.13.5 after upgrading i have run into the same thing also. http://pastebin.com/m4XseSnH If you want to schema information to reproduce please just email me off list. Regards, Brian Hood

3 6

Strange concurrency issue
by Percy Wegmann 28 Jan '13

28 Jan '13

Dear MonetDB Team, We're starting to exercise our MonetDB installation more heavily now and I've run into a strange issue where I get an occasional "COMMIT: failed" message. It seems that the conditions required to produce this are: - Concurrently inserting into two different tables - At the same time, issuing a SELECT COUNT(*) against either of those tables I've logged a bug with more detailed information, including a Java test case: http://bugs.monetdb.org/show_bug.cgi?id=3210 I'm only just starting my investigation, but I suspect that COUNT(*) may be the culprit, since other selects like SELECT * don't seem to have this problem. Does this ring any bells? Are there any special optimizations for COUNT(*) that bypass the underlying BAT and/or have side effects on some shared table somewhere that could be causing a concurrency violation? Thanks, -- Percy Wegmann +1 512 637 8500 ext 148 _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

3 11

Bit functions (XOR and inverse)
by Eduardo Oliveira 28 Jan '13

28 Jan '13

Hi, I'm running MonetDB v11.13.9 (Oct2012-SP3), i have some doubts about bit functions, i was trying the bit functions looking at MySQL docs in MonetDB: http://dev.mysql.com/doc/refman/5.0/en/bit-functions.html And when i used ^, the pow was calculated instead of XOR, the ~ don't work as expected too. I google and I found this http://permalink.gmane.org/gmane.comp.db.monetdb.scm/28212 This patch will be applied on MonetDb, any date of release of MonetDb with ^ already as XOR? Right now what operator should i used to make a xor? I tested xor XOR and MonetDb didn't recognize. If i want to insert in binary form how i do? b'binary' works? Thanks in advance, -- *Eduardo Oliveira * *IT* ***Email:* eduardo.oliveira(a)adclick.pt *Web: *www.adclickint.com **<http://www.adclickint.com/> * * <http://www.adclickint.com/> <http://www.adclickint.com/>

3 6

Maximum number of redirects reached, aborting connection attempt. Sorry.
by Brandon Jackson 28 Jan '13

28 Jan '13

I am using a release version of MonetDB 11.13.9 "OCT2012-SP3" on Fedora 18, running as an unprivileged user with no firewall. The server quit allowing connections via JDBC and mclient this morning. mclient hangs indefinitely after asking for a password run from the same machine and account that started mserver5 via monetdbd. "monetdb status", reports a healthy database. JDBC connections initiated using monetdb-jdbc-2.8.jar through Pentaho Data Integration. "Maximum number of redirects reached, aborting connection attempt. Sorry." is the error message. Shutting down MonetDB seemed to work. It also seems to start up okay, but still does not allow any connections. I am happy to install MonetDB with debug symbols and provide some output from gdb if that would be helpful. Please let me know a few things I can try that may be helpful for filing a bug report and getting us back online. Thanks for your help, Brandon The stack trace that came back to java was this: Error connecting to database [monet_testing] : org.pentaho.di.core.exception.KettleDatabaseException: Error occured while trying to connect to the database Error connecting to database: (using class nl.cwi.monetdb.jdbc.MonetDriver) Maximum number of redirects reached, aborting connection attempt. Sorry. org.pentaho.di.core.exception.KettleDatabaseException: Error occured while trying to connect to the database Error connecting to database: (using class nl.cwi.monetdb.jdbc.MonetDriver) Maximum number of redirects reached, aborting connection attempt. Sorry. at org.pentaho.di.core.database.Database.normalConnect(Database.java:366) at org.pentaho.di.core.database.Database.connect(Database.java:315) at org.pentaho.di.core.database.Database.connect(Database.java:277) at org.pentaho.di.core.database.Database.connect(Database.java:267) at org.pentaho.di.core.database.DatabaseFactory.getConnectionTestReport(DatabaseFactory.java:86) at org.pentaho.di.core.database.DatabaseMeta.testConnection(DatabaseMeta.java:2464) at org.pentaho.ui.database.event.DataHandler.testDatabaseConnection(DataHandler.java:533) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:329) at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:139) at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:123) at org.pentaho.ui.xul.swt.tags.SwtButton.access$500(SwtButton.java:26) at org.pentaho.ui.xul.swt.tags.SwtButton$4.widgetSelected(SwtButton.java:119) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.eclipse.jface.window.Window.runEventLoop(Window.java:820) at org.eclipse.jface.window.Window.open(Window.java:796) at org.pentaho.ui.xul.swt.tags.SwtDialog.show(SwtDialog.java:378) at org.pentaho.ui.xul.swt.tags.SwtDialog.show(SwtDialog.java:304) at org.pentaho.di.ui.core.database.dialog.XulDatabaseDialog.open(XulDatabaseDialog.java:115) at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.open(DatabaseDialog.java:62) at org.pentaho.di.ui.trans.steps.monetdbbulkloader.MonetDBBulkLoaderDialog$20.widgetSelected(MonetDBBulkLoaderDialog.java:1350) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.trans.steps.monetdbbulkloader.MonetDBBulkLoaderDialog.open(MonetDBBulkLoaderDialog.java:905) at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:136) at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:7835) at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:2749) at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:705) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1221) at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7044) at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:8304) at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:580) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.pentaho.commons.launcher.Launcher.main(Launcher.java:134) Caused by: org.pentaho.di.core.exception.KettleDatabaseException: Error connecting to database: (using class nl.cwi.monetdb.jdbc.MonetDriver) Maximum number of redirects reached, aborting connection attempt. Sorry. at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:502) at org.pentaho.di.core.database.Database.normalConnect(Database.java:350) ... 51 more Caused by: java.sql.SQLException: Maximum number of redirects reached, aborting connection attempt. Sorry. at nl.cwi.monetdb.jdbc.MonetConnection.<init>(MonetConnection.java:254) at nl.cwi.monetdb.jdbc.MonetDriver.connect(MonetDriver.java:171) at java.sql.DriverManager.getConnection(DriverManager.java:582) at java.sql.DriverManager.getConnection(DriverManager.java:185) at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:482) ... 52 more Caused by: nl.cwi.monetdb.mcl.MCLException: Maximum number of redirects reached, aborting connection attempt. Sorry. at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:255) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:395) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:248) at nl.cwi.monetdb.jdbc.MonetConnection.<init>(MonetConnection.java:232) ... 56 more Hostname : 172.27.0.209 Port : 50000 Database name : testing

3 2

How to handle too much data ?
by mail_server＠lavabit.com 28 Jan '13

28 Jan '13

Hi, i am working on monetdb so as it can be used for handling huge streaming data, in the most efficient way. i am having a requirement of sustained inserts of rate around 50k-100k inserts per sec(maybe more).i have tested up the results on big enough data (upto 100Gb) and it's the fastest one, to be frank.Most data is 8 byte longs,1 column is time in secs and 1 column is string based.FYI,Data writer in single threaded. However,the issue is too much data and it will be going on for months( maybe years) and all queries have a range part.Most queries are on current data, while some will be for ages old data.Horizontal partitioning is the way to go as it seems. so what i thought was for efficiency, dump the arrays as per time in secs and then when a query comes, somehow form a new BAT (maybe using FUSE, since copying is unusable), join the data in temporary way and use it and discard.Few secs delay is okay even for the range query of 1 sec. as i have been digging around, http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/BinaryBulkLoad it's possible for full size columns. but since this is going to be a copy,it's inefficient. is there a way to get it done ( already exists) ?? any pointers might help such as BAT Partition Manager and so on. if NOT, i may try to get down and get it done but some pointers might be useful from the experts. Thanks for help. mike

1 0

Unsigned integers
by Eduardo Oliveira 23 Jan '13

23 Jan '13

Hello, I would like to know if MonetDb supports unsigned integer, i couldn't find anything on documentation. I tried to created a table with unsigned int and i get an error. Thank you, -- *Eduardo Oliveira * *IT* ***Email:* eduardo.oliveira(a)adclick.pt *Web: *www.adclickint.com **<http://www.adclickint.com/> * * <http://www.adclickint.com/> <http://www.adclickint.com/>

2 1

Announcement: New Oct2012-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 22 Jan '13

22 Jan '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2012-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2012-SP3 bugfix release Bug Fixes * 2291: small doubles end up as NULL after arithmetic * 2579: SQL: incorrect cast from double to int * 2659: tuples INTERSECT vs count (... INTERSECT ...) differs * 2805: row_number doesn't work in complex query * 2886: warning: Catalog parameter '' ignored * 2977: timestamp minus date results in program error * 2978: now() + 1 results in illegal argument * 2999: Up-cast from SQL drops order information? * 3048: algebra.join undefined when using multiple arithmetic operations in SQL where-clause * 3124: insert into table with two foreign keys crashes mserver5 * 3136: Memleak (in querycache?) while querying * 3143: Type resolution error in SQL procedural code * 3163: mserver5 crashes with large number of concurrent clients * 3168: crash when insert varchar/char/clob with default NULL * 3169: php driver uses call-time pass-by-reference * 3173: SQL queries with limit and offset clauses are unable to use bind parameters for either * 3176: Same query and different results * 3181: Join query returns zero results when there is an index on the tables * 3185: inconsistency between documentation and implementation of command line options * 3186: case evaluted to early * 3188: Storage of client history * 3189: Overflow in calculation * 3190: When subtracting two columns of type "timestamp", no results are returned * 3191: rel_optimizer.c:5596: _rel_optimizer: Assertion `0' failed. * 3192: "No such column with index" error if CASE in SELECT clause * 3199: Inconspicuous sequence of prepare, execute, commit and close produces error * 3205: Inet datatype handling inconsistent between int, cast( as inet) and string * 3208: Unable to execute prepared statements with a large number of arguments * 3209: exp_bin: Assertion `0' failed * 3212: Query cache does not work properly

1 0

SQL function as non-privileged user
by Tapomay Dey 21 Jan '13

21 Jan '13

I have created a separate schema and user as follows: echo "CREATE USER \"tapomay\" WITH PASSWORD '~!@#$' NAME 'Ext User' SCHEMA \"sys\";" | mclient -d my_db echo "CREATE SCHEMA \"test_schema\" AUTHORIZATION \"tapomay\";" | mclient -d my_db echo "ALTER USER \"tapomay\" SET SCHEMA \"test_schema\";" | mclient -d my_db When logged in as "tapomay", I am able to execute curdate() but can't execute date_to_str(). I can run date_to_str() when logged in as the super user monetdb. There could be many other such utility functions. How do I enable all of them for the new user "tapomay". Regards, Tapomay.

2 1

BAT Data structures of a Schema
by Hassan, Ahmad 21 Jan '13

21 Jan '13

Hi Martin, I have the following list of tables in database schema: sql>\d TABLE sys.customer TABLE sys.lineitem TABLE sys.nation TABLE sys.orders TABLE sys.part TABLE sys.partsupp TABLE sys.region TABLE sys.supplier Please can you point me to the C code file(s) that hold the BAT objects for each column of the above tables please? I want to debug and analyse specifically that part of the code region that declares, initialises and maintains the list of BAT structures for the tables. The following query is providing the information about the tables and I am assuming that this query also access this information from all the individual BAT structures of a database schema. select * from storage() where "table" = 'lineitem'; Secondly, Is it possible to list of get the list of all the BAT structures (Column BATS and Delta BATs) in the running database please? Thanks. Kind Regards, Ahmad

4 6

Inconsistency in data selection. count*
by Tapomay Dey 21 Jan '13

21 Jan '13

select count(*) from table; Results: 168466 rows select count(*) from table where column=0; Results: 168466 rows select count(*) from table where column<>0; Results: 168466 rows When I do a "select column from table where client_id=0 limit 5;" I see correct non-zero values in the result although the selection condition is that the column value must be 0. I put the data on another instance/machine - I don't see this issue. Plz find a screenshot@ https://picasaweb.google.com/115470548136322675830/January212013?authuser=0… Another case: sql>select count(*) from table2 where column2=0; | 391326 | sql>select count(*) from table2 where column2<>0; | 141108 | sql>select count(*) from table2; | 399326 | sql>select column2 from table2 where column2=0 limit 5; +-----------+ | column2 | +===========+ | 53 | | 53 | | 53 | | 53 | | 53 | +-----------+ 5 tuples (156.631ms) Regards, Tapomay

1 0

Resource limits in MonetDB
by Charles Milutinovic 21 Jan '13

21 Jan '13

I am experimenting with using MonetDB for a project where the most natural implementation would involve ~100K tables (with the huge majority being near empty, a few incredibly massive) with approx. 10 columns each. MonetDB choked on this, with the following error: >!FATAL: BBPextend: trying to extend BAT pool beyond the limit (16384000) I am assuming that I hit this limit because of the extreme amount of tables. I am wondering if there are any other hard limits to be aware of in MonetDB that you could hit before hitting system limits.

3 2

FW: Monetdb on Solaris/Sparc: high system time problem
by Javier Picorel 21 Jan '13

21 Jan '13

Dear Martin, Thanks for your prompt reply. We have verified that the order in which the queries are executed doesn't affect the behavior of the query. The same problem arises at the same queries in any arbitrary order, which implies that it's related to the query itself. In addition, the queries' execution time and utilization are pretty deterministic. We have also verified that there is plenty of memory available and there are no other processes competing for resources (i.e., disk bandwidth, memory). We have used dtrace to profile what functions are being executed by MonetDB while being in the two phases (normal and problematic). In the normal phase, which lasts for a few tens of seconds and the behavior is what we see on Linux, MonetDB mainly access the following functions: - BUNfastins - re_match_no_ignore - strstr When MonetDB enters into that second (problematic) phase, the User/System time is 4% to 21% for a couple of minutes w/o any disk activity or swapping, MonetDB mainly accesses the following functions: - MT_sleep_ms - memset - __pollsys - _pollsys - _save_nv_regs - pselect - select Do these function names help you to understand the problematic behavior? Thanks in advance, Javier ________________________________ From: Martin Kersten <martin(a)monetdb.org<mailto:martin@monetdb.org>> Subject: Re: Monetdb on Solaris/Sparc: high system time problem Date: January 17, 2013 6:56:00 PM GMT+01:00 To: <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Reply-To: Communication channel for MonetDB users <users-list(a)monetdb.org<mailto:users-list@monetdb.org>> Dear Javier, Thank you for your report. It is hard to assess the consequences/impact of your particular setup. Your report suggests that something else was also running on your platform during this sequence, at that particular time. It could mean the MonetDB was fighting with another process over resources, i.e. memory and disk bandwidth. It could also indicate a thrashing kernel on shared kernel resources. If the system persists during repetitive runs under controlled system settings and it is immune to query order, then we might look further into the issue. regards, Martin On 1/17/13 3:44 PM, Javier Picorel wrote: Dear all, I'm running MonetDB (Dec2011-SP2) on Sparc Solaris 10 u9. I'm testing the TPC-H queries using SF10 and SF100 as the scale factors. My machine has 4 cores and 32GB of RAM and my farm is on a disk array. Unfortunately, there are some queries (e.g., Qry 7, 8, 9) that enters into phase in which the User/System time is 4% to 21% for a couple of minutes. Interestingly, the same issue doesn't show up on a Linux machine with a similar setup and the queries' system time is pretty low (<5%). Interestingly, vmstat output doesn't show any disk activity during the phase where the process is just in system, only there are high values on "re" and "mf" fields. There is no swapping going on ("po" and "pi" values are 0 in vmstat). These are the parameters of the DB: type default database shared default yes nthreads default 4 optpipe default_pipe master default no slave default <unknown> readonly default no nclients default 64 The way that I test the query is by running just one at a time. How can I mitigate this OS time? Thanks in advance, Javier _______________________________________________ users-list mailing list users-list(a)monetdb.org<mailto:users-list@monetdb.org> http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org<mailto:users-list@monetdb.org> http://mail.monetdb.org/mailman/listinfo/users-list

2 1

MonetDB under LLVM
by Hassan, Ahmad 18 Jan '13

18 Jan '13

Hi Team, Has anyone tried compiling/running MonetDB under LLVM/CLANG compiler framework please? Normally the use case is: 1) Produce set of llvm bitcode files *.bc from C code 2) Apply LLVM Pass to instrument *.bc files through llvm 'OPT' compiler tool 3) Link all *.bc files and produce executables of monetdb I am going to try monetdb under llvm. If anyone hasn't done it before then I may ask for the initial pointers to compile monedb under llvm instrumentation. Thanks. Kind Regards, Ahmad _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

5 8

Email archives for the forums
by Tapomay Dey 18 Jan '13

18 Jan '13

I don't see any forum archives since you switched from sourceforge. It's helpful to refer to other email threads. Is it there somewhere? I see the older one here: http://mail.monetdb.org/pipermail/developers-list/ and till 8th Jan '13 on sourceforge. Regards, Tapomay. _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

3 2

Monetdb on Solaris/Sparc: high system time problem
by Javier Picorel 17 Jan '13

17 Jan '13

Dear all, I'm running MonetDB (Dec2011-SP2) on Sparc Solaris 10 u9. I'm testing the TPC-H queries using SF10 and SF100 as the scale factors. My machine has 4 cores and 32GB of RAM and my farm is on a disk array. Unfortunately, there are some queries (e.g., Qry 7, 8, 9) that enters into phase in which the User/System time is 4% to 21% for a couple of minutes. Interestingly, the same issue doesn't show up on a Linux machine with a similar setup and the queries' system time is pretty low (<5%). Interestingly, vmstat output doesn't show any disk activity during the phase where the process is just in system, only there are high values on "re" and "mf" fields. There is no swapping going on ("po" and "pi" values are 0 in vmstat). These are the parameters of the DB: type default database shared default yes nthreads default 4 optpipe default_pipe master default no slave default <unknown> readonly default no nclients default 64 The way that I test the query is by running just one at a time. How can I mitigate this OS time? Thanks in advance, Javier _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 1