- users-list - monetdb.org

Should MonetDB really be used in production?
by Frédéric Jolliton 17 Jul '15

17 Jul '15

Greetings, We filled a bug few days ago about simple SAVEPOINT usage corrupting the database (both in Oct2014-SP4 and tip of hg). This is worrying us a lot, because that's yet another bug easy to trigger. We're studying MonetDB extensively for our future needs. We've written optimizers, custom aggregate functions in C, and we've studied many internal aspects. But yet, from time to time, we are hit by bugs when performing simple, trivial queries. You can see the history of bugs we filled on the bugtracker (regarding UNION, SELECT, ..). This become time consuming. This make us nervous because we might have hard time trusting our complexes queries run by MonetDB. The performance are stunning, but the quality seems not high enough for production. Projects like PostgreSQL shows that even Open Source projects can attains incredible stability and robustness. (Admittedly there are many more contributors to PostgreSQL.) Note that we understand that it is a complex software, and that it is hard to cover all possible corner cases in tests. But we're wondering how stable MonetDB is expected to be. Is this really used in production? Regarding the MonetDB website, the primary source of information for the project, we've not found any reference documentation regarding all the supported SQL features. Sure, there is a page listing what is *not* supported, there are pages on various topics, but it is unclear if SAVEPOINT is really supported. (If I can say, I think that the website probably needs some refresh and better navigation.) Here is the latest bug I was talking about: $ bin/mclient -p 50005 test Welcome to mclient, the MonetDB/SQL interactive terminal (Oct2014-SP4) Database: MonetDB v11.19.13 (Oct2014-SP4), 'mapi:monetdb://fjolliton:50005/test' Type \q to quit, \? for a list of available commands auto commit mode: on sql>start transaction; auto commit mode: off sql>create table foo (a int); operation successful (0.528ms) sql>savepoint sp1; auto commit mode: off sql>release savepoint sp1; auto commit mode: off sql>commit; auto commit mode: on sql>select * from foo; +---+ | a | +===+ +---+ 0 tuples (0.9s) So far, so good. Now try to connect again: $ bin/mclient -p 50005 test Welcome to mclient, the MonetDB/SQL interactive terminal (Oct2014-SP4) Database: MonetDB v11.19.13 (Oct2014-SP4), 'mapi:monetdb://fjolliton:50005/test' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select * from foo; SELECT: no such table 'foo' Whoops ! Regards, -- Frédéric Jolliton Sécuractive

9 10

Re: Re: Provide --password option for msqldump tool
by GMail 16 Jul '15

16 Jul '15

1 0

Fwd: Re: Provide --password option for msqldump tool
by Sherzod Mutalov 16 Jul '15

16 Jul '15

I added "password" option to mclient.c and msqldump.c, you can get changed sources from: https://gist.github.com/shmutalov/d3695ad67ad5195eeee6 I know, it is dangerous, but we need this option in our application. If you want I can provide mclient binaryies for Win7 x64.

1 0

Provide --password option for msqldump tool
by Aravind S Reddy 16 Jul '15

16 Jul '15

Hello Team, We were trying to dump multiple tables through 'msqldump' tool, and restoring it in different monetdb instance in a programatic way. But the difficulty here is, I do not see an option to pass the --password to 'msqldump' tool, where as it prompts for the password and accepts input from STDIN. Due to this, we are not able achieve our requirement programatically. And it would be good if -t(--table) option accepts multiple table names, through which we can dump the selected tables in a single shot. Regards, Aravind.S

4 5

Cluster advantages MonetDB
by Luciano Sasso 16 Jul '15

16 Jul '15

Hi, What the cluster Advantages MonetDB, If a query in shard tables in a cluster can not be ordered, what the advantage of using a MonetDB cluster? besides the replication in case of failure. Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

5 5

Job openings at MonetDB Solutions
by Ying Zhang 14 Jul '15

14 Jul '15

Dear all, MonetDB Solutions, the supporting company of MonetDB, is looking for core database developers. If you’re interested, please see the job description here: https://www.monetdbsolutions.com/company/careers Regards, Jennie

1 0

Re: User for multiple schemas
by Ying Zhang 14 Jul '15

14 Jul '15

Hai Brian, Glad to know it helps. We’ve found the cause of the problem I mentioned. It’s not random. It has something to do with changing the ROLE of a USER, and then the privileges of that ROLE seem not properly handled. We’re working on improving the documentation, extending the test suite, and fixing bugs concerning privileges. Regards, Jennie > On Jul 14, 2015, at 12:10, Brian Hood <brianh6854(a)googlemail.com> wrote: > > Hi Jennie, > > That was actually what i was looking for thanks for that i didn't realize that you could have more than one create schema schema_name AUTHORIZATION per_user > > I'll also see if i encounter the bug and if i can find anything reproducible. > > Regards, > > Brian Hood > > > On Tue, Jul 14, 2015 at 10:03 AM, Ying Zhang <Y.Zhang(a)cwi.nl> wrote: > Hello Brian, > > What authorisation are you looking for? MonetDB does support authorise one user as the owner of multiple schema’s, e.g.: > > create user brian with password ’brian123' name ’brian' schema sys; > create schema schema1 AUTHORIZATION brian; > create schema schema2 AUTHORIZATION brian; > > Not sure if this is what you’re looking for. > > However, we have noticed a problem (in the current default branch, probably also exists in Jul2015) in handling the privileges. After a user has been authorised as the owner of a schema, the user can CREATE, INSERT, UPDATE, … tables, however, *sometimes* the user can’t SELECT from his/her own tables. This seems to happen at random, which makes it extremely difficult to debug. However, restarting the MonetDB server seems to temporarily resolve the problem. > > We’re working on improving handling privileges in MonetDB. If you notice any errors and missing features, please let us know. > > Regards, > > Jennie > > > On Jun 23, 2015, at 08:05, Stefan Manegold <Stefan.Manegold(a)cwi.nl> wrote: > > > > Hi Brian, > > > > If i'm not mistaken and recall correctly, access rights / authorization are per table, not per schema - at least in MonetDB; hence, you'd have to grant access for each table in other than the user's schema separately. > > > > Best, > > Stefan > > > > > > On June 22, 2015 8:45:08 PM CEST, Brian Hood <brianh6854(a)googlemail.com> wrote: > > Hi All, > > > > There is something I have pondered on is there a way to have a username that can be authorized for multiple schemas under your dbfarm. > > > > If not what is the reason ? > > > > I haven't seen a way to do this so far. > > > > Regards, > > > > Brian Hood > > > > > > > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > > > > -- > > | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | > > | www.CWI.nl/~manegold/ | Science Park 123 (L321) | > > | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > >

1 0

User for multiple schemas
by Brian Hood 14 Jul '15

14 Jul '15

Hi All, There is something I have pondered on is there a way to have a username that can be authorized for multiple schemas under your dbfarm. If not what is the reason ? I haven't seen a way to do this so far. Regards, Brian Hood

3 2

How do you generate documentation from MonetDB source files?
by Jon Mulieri 12 Jul '15

12 Jul '15

I have long appreciated the wealth of information in the comments of source files throughout the code base. It is apparently formatted for some documentation extraction. Can someone provide instructions for generating the documentation from the source? Thanks! Jon

3 2

Reproducing EmbeddedR results from MonetDB.org site
by George, Glover E ERDC-RDE-ITL-MS 10 Jul '15

10 Jul '15

Hi all, I’ve recently been profiling several techniques for a workflow that we’ve been trying to improve here at USACE. Originally we used python scripts with sqlite, but we ran into scalability problems on large data sets. This led us to MonetDB, with the promise of columnar-based analysis and the hope of both parallel query’s “under-the-hood” and possibly a distributed workflow across an HPC system. Looking at my profiling results, this has led me to a number of questions that hopefully you all can help us with. I have a single table from the TPC-H benchmark – lineitem – populated with 360million entries: "l_orderkey" INTEGER NOT NULL, "l_partkey" INTEGER NOT NULL, "l_suppkey" INTEGER NOT NULL, "l_linenumber" INTEGER NOT NULL, "l_quantity" DECIMAL(15,2) NOT NULL, "l_extendedprice" DECIMAL(15,2) NOT NULL, "l_discount" DECIMAL(15,2) NOT NULL, "l_tax" DECIMAL(15,2) NOT NULL, "l_returnflag" CHAR(1) NOT NULL, "l_linestatus" CHAR(1) NOT NULL, "l_shipdate" DATE NOT NULL, "l_commitdate" DATE NOT NULL, "l_receiptdate" DATE NOT NULL, "l_shipinstruct" CHAR(25) NOT NULL, "l_shipmode" CHAR(10) NOT NULL, "l_comment" VARCHAR(44) NOT NULL, System Setup: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz - 16 cores (HT off) / 256G RAM MonetDB version 11.19.15 R version 3.2.0 1. I have followed the instructions here: https://www.monetdb.org/content/embedded-r-monetdb and tried to reproduce these results. I only attempt to reproduce the results from R (reading the data from CSV since – as noted in a previous email to this list – R fails to read in the 360 million rows using MonetDB.R but works fine reading the same amount from CSV), MonetDB, as well as MonetDB with Embedded R. - I compile both R and MonetDB from source. When building R, I include the option to build the R shared library (libR.so) and when compiling MonetDB I include the option for embedded R. - When creating the MonetDB database, I set the option for embedr=true. - The R function from the above URL is used, as well as the same SQL query – and it works…. However, the performance is much worse than using R alone. I know I must be overlooking something. Please see the image at https://goo.gl/wUvB2J (png on Google Drive). The X-axis is the number of rows, and the Y axis is the time in seconds. As you can see, embedded R is unexpectedly much worse than the other two. 2. While running the quantile function, only one core is active – whether embeddedR or just MonetDB’s. 3. When running the following query (after altering the table to include a new column z, of course): sql>update lineitem set z=l_extendedprice / l_quantity; 360011594 affected rows (3m 49s) … Multiple cores are active for the first 10 seconds, then it resorts to single core. A. What am I not understanding about MonetDB’s ability to use multiple cores? The only time I really see it use multiple cores seems to be when doing “copy into”. B. Can someone provide assistance in trying to replicate the Embedded R results? Should it be running in parallel? Or might something in my setup be configured incorrectly. Cheers!

3 3