- users-list - monetdb.org

Type of integers used to calculate a SUM
by Jan, Di-An 13 Aug '19

13 Aug '19

With hugeint enabled, it seems that the SUM of INT (32-bit) is done using HUGEINT (128-bit) while the SUM of BIGINT (64-bit) is done using BIGINT (64-bit). That seems weird. Is SUM supposed to use the largest type or use the same type as the input or something else? Database: MonetDB v11.33.3 (Apr2019) sql>CREATE TABLE test (bi BIGINT, i INT); sql>INSERT INTO test VALUES (0x7000000000000000, 0x70000000); sql>INSERT INTO test VALUES (0x7000000000000000, 0x70000000); sql>INSERT INTO test VALUES (1, 3); sql>INSERT INTO test VALUES (2, 4); sql>SELECT SUM(i) FROM test; -- larger than an INT -- +------------+ | L3 | +============+ | 3758096391 | +------------+ 1 tuple sql>SELECT SUM(bi) FROM test; -- larger than a BIGINT -- overflow in calculation. sql>EXPLAIN SELECT SUM(i) FROM test; +------------------------------------------------------------------------------------------------------+ | function user.s10_22():void; | | X_1:void := querylog.define("explain select sum(i) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:int] := sql.bind(X_4:int, "sys":str, "test":str, "i":str, 0:int); | | X_17:bat[:int] := algebra.projection(C_5:bat[:oid], X_8:bat[:int]); | | X_18:hge := aggr.sum(X_17:bat[:int]); | | exit X_74:bit; | | sql.resultSet("sys.L3":str, "L3":str, "hugeint":str, 128:int, 0:int, 7:int, X_18:hge); | | end user.s10_22; | +------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(bi) FROM test; +-------------------------------------------------------------------------------------------------------+ | function user.s12_22():void; | | X_1:void := querylog.define("explain select sum(bi) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:lng] := sql.bind(X_4:int, "sys":str, "test":str, "bi":str, 0:int); | | X_17:bat[:lng] := algebra.projection(C_5:bat[:oid], X_8:bat[:lng]); | | X_18:lng := aggr.sum(X_17:bat[:lng]); | | exit X_74:bit; | | sql.resultSet("sys.L3":str, "L3":str, "bigint":str, 64:int, 0:int, 7:int, X_18:lng); | | end user.s12_22; | +-------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(CAST(i AS HUGEINT)) FROM test; +-----------------------------------------------------------------------------------------------------------------------+ | function user.s20_22():void; | | X_1:void := querylog.define("explain select sum(cast(i as hugeint)) from test;":str, "default_pipe":str, 13:int); | | barrier X_75:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:int] := sql.bind(X_4:int, "sys":str, "test":str, "i":str, 0:int); | | X_17:bat[:int] := algebra.projection(C_5:bat[:oid], X_8:bat[:int]); | | X_18:bat[:hge] := batcalc.hge(X_17:bat[:int]); | | X_19:hge := aggr.sum(X_18:bat[:hge]); | | exit X_75:bit; | | sql.resultSet("sys.L4":str, "L4":str, "hugeint":str, 128:int, 0:int, 7:int, X_19:hge); | | end user.s20_22; | +-----------------------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(CAST(bi AS BIGINT)) FROM test; +-----------------------------------------------------------------------------------------------------------------------+ | function user.s8_22():void; | | X_1:void := querylog.define("explain select sum(cast(bi as bigint)) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:lng] := sql.bind(X_4:int, "sys":str, "test":str, "bi":str, 0:int); | | X_17:bat[:lng] := algebra.projection(C_5:bat[:oid], X_8:bat[:lng]); | | X_18:lng := aggr.sum(X_17:bat[:lng]); | | exit X_74:bit; | | sql.resultSet("sys.L4":str, "L4":str, "bigint":str, 64:int, 0:int, 7:int, X_18:lng); | +-----------------------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(CAST(i AS INT)) FROM test; +-----------------------------------------------------------------------------------------------------------+ | function user.s10_22():void; | | X_1:void := querylog.define("select sum(cast(i as int)) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:int] := sql.bind(X_4:int, "sys":str, "test":str, "i":str, 0:int); | | X_17:bat[:int] := algebra.projection(C_5:bat[:oid], X_8:bat[:int]); | | X_18:hge := aggr.sum(X_17:bat[:int]); | | exit X_74:bit; | | sql.resultSet("sys.L4":str, "L4":str, "hugeint":str, 128:int, 0:int, 7:int, X_18:hge); | | end user.s10_22; | +-----------------------------------------------------------------------------------------------------------+ sql>SELECT SUM(CAST(i AS INT)) FROM test; +------------+ | L4 | +============+ | 3758096391 | +------------+ 1 tuple Also, the cast in SUM(CAST(bi AS BIGINT)) seems to be optimized out while SUM(CAST(i AS HUGEINT)) has one more BAT operation than SUM(i) even though they seem to do the same thing, and the cast in SUM(CAST(i AS INT)) seems to have no effect. Di-an Jan

1 0

Books about Monetdb?
by Nobody Noone 09 Aug '19

09 Aug '19

Hi, Are there any books published about Monetdb? This listserv, monetdb.org and stackoverflow.com seem to be all I can find. Thanks!

2 1

understand how monetdb choose the best query plan
by DEMBELE Simon Pierre 08 Aug '19

08 Aug '19

Hello, J'm new in the use of monetdb, i want to understand how monetdb choose the best query plan. Is the model cost is used for that or what. Please if someone can clarify me. Thank yours time. -- Simon

2 2

Questions on upgrading tables based upon the system tables.
by Gotwisner, Dave 08 Aug '19

08 Aug '19

I am writing code to migrate from one version of our schema to another (such as if we add tables or add columns). The from version is whatever is physically being used, and the new version is compiled into data structures in our code. I can successfully create or remove tables, and add and drop columns. We have multiple tables that we then bind to a merge table, so the schemas must be exact. Our tables have a primary key. For the table and column based changes, querying sys.schemas, sys.tables, and sys.columns gives us everything we need. I can also figure out how to add the primary key on the new table (via alter table commands). However, I can't figure out how to find the set of columns in the primary key, to be able to build up the primary key. Here is the table definition for our tables that go into the merge table (%04d goes from 0000 to the number of threads we send bulk data with): CREATE TABLE if not exists data_availability__%04d ( interval_start BIGINT, source INT, PRIMARY KEY(interval_start, source) ); alter table data_availability add table data_availability__%04d; Looking at system tables gives the following information (before adding this table): sql>select * from sys.tables where name = 'data_availability'; +------+-------------------+------+------+------+-------+------+-------+-------+ | id | name | sche | quer | type | syste | comm | acces | tempo | : : : ma_i : y : : m : it_a : s : rary : : : : d : : : : ctio : : : : : : : : : : n : : : +======+===================+======+======+======+=======+======+=======+=======+ | 8889 | data_availability | 8800 | null | 3 | false | 0 | 0 | 0 | +------+-------------------+------+------+------+-------+------+-------+-------+ 1 tuple sql>select * from sys.keys where table_id = 8889; +------+----------+------+----------------------------------------------+------+--------+ | id | table_id | type | name | rkey | action | +======+==========+======+==============================================+======+========+ | 8887 | 8889 | 0 | data_availability_interval_start_source_pkey | -1 | -1 | +------+----------+------+----------------------------------------------+------+--------+ 1 tuple sql>select * from sys.key_types; +-------------+---------------+ | key_type_id | key_type_name | +=============+===============+ | 0 | Primary Key | | 1 | Unique Key | | 2 | Foreign Key | +-------------+---------------+ 3 tuples sql>select * from sys.idxs where table_id = 8889; +------+----------+------+----------------------------------------------+ | id | table_id | type | name | +======+==========+======+==============================================+ | 8888 | 8889 | 0 | data_availability_interval_start_source_pkey | +------+----------+------+----------------------------------------------+ 1 tuple sql>select * from sys.index_types; +---------------+-----------------------+ | index_type_id | index_type_name | +===============+=======================+ | 0 | Hash | | 1 | Join | | 2 | Order preserving hash | | 3 | No-index | | 4 | Imprint | | 5 | Ordered | +---------------+-----------------------+ 6 tuples sql>select * from sys.columns where table_id = 8889; +------+----------------+--------+-------------+------------+----------+---------+-------+--------+---------+ | id | name | type | type_digits | type_scale | table_id | default | null | number | storage | +======+================+========+=============+============+==========+=========+=======+========+=========+ | 8885 | interval_start | bigint | 64 | 0 | 8889 | null | false | 0 | null | | 8886 | source | int | 32 | 0 | 8889 | null | false | 1 | null | +------+----------------+--------+-------------+------------+----------+---------+-------+--------+---------+ 2 tuples The above data is for the merge table that we will bind the (new) table to. I can get the name and id of the primary key (which is also an index). I could not find a table that gives a column -> key binding. Is there such a table? If not, how do I get this information from MonetDB? Thanks, Dave

2 1

Question on querying two specific columns from sys.columns
by Gotwisner, Dave 07 Aug '19

07 Aug '19

I am trying to query the default and null columns from sys.columns. Both of these are sql key words, so queries either don't work or return the wrong value. Doing a "select default from ss.columns;" returns "syntax error, unexpected DEFAULT in: "select default"" error. Doing a "select null from sys.columns;" returns null for all records, even though it is a Boolean column with true/false values. I am currently working around this by doing "select *" and pulling out the relevant fields, but, I want to protect against a future release possibly changing the column ordering or changing the number of columns in this table (without also requiring an update to our code). Is there a way to escape the column names in a way that gives me what I want? Thanks, Dave

2 1

Re: Alter question about MonetDB
by Ying Zhang 06 Aug '19

06 Aug '19

Hai, This question should have been sent to the users mailing list, so I added that e-mail address. I can confirm this problem with an Apr2019 refreshed yesterday. Please see the 1st attachment. Also, after having restarted both mclients, I can still only query “tbl5” using the mclient on the left side. Please see the 2nd attachment. Can you please fire a bug report for this? Thanks! Jennie > On 6 Aug 2019, at 08:55, 角 <3089454366(a)qq.com> wrote: > > Hi,All: > I am a MonetDB user from China. I have some troubles when I was using MonetDB. I use mclient and enter this sql : "alter table t1 rename to t2；", while this mclient is alive,I use anthor mclient to connect the same database as the former and I enter this sql : "select * from t2" then I get an error : no such table. So how can I use alter command to make all mclients take effect? > _______________________________________________ > developers-list mailing list > developers-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/developers-list

1 0

Question on missing BAT file
by Sharma, Sreejith 06 Aug '19

06 Aug '19

All, Is there a way to figure out which BAT files constitutes a specific table in MonetDB? I know the location in sys.storage represents it, but I see one of the tail file is showing missing which is not available neither in sys.storage not in DB folder. We get this error during migration from older version (MonetDB 11.27.3) to MonetDB 11.33.3 version. 2019-07-31 10:43:54 MSG TSV_PROD_DB_A[161617]: !ERROR: BBPcheckbats: cannot stat file /monet_data01/DBFARM_TSV/TSV_PROD_DB_A/bat/10/45/104502.tail Regards, Sreejith

2 2

Fwd: Re: Managing Access to Monetdb Schemas
by ferreira 02 Aug '19

02 Aug '19

-------- Forwarded Message -------- Subject: Re: Managing Access to Monetdb Schemas Date: Fri, 2 Aug 2019 11:06:21 +0200 From: ferreira <pedro.ferreira(a)monetdbsolutions.com> To: Nobody Noone <nobody.noone.nobody.me(a)gmail.com> Hello! I found it strange why we don't support GRANT/REVOKE privileges statements on schemas. I looked into the SQL standard, and there is no mention of it. You can use roles to limit schema access for each user. Here you have an example: CREATE ROLE my_role; CREATE SCHEMA my_schema AUTHORIZATION my_role; CREATE USER my_user with password 'p1' name 'User with role' schema "my_schema"; GRANT my_role to my_user; You can add more fine grained access to each table with table privileges as you mentioned. As for specific rights such as DDL, we don't support it as of now. You can create a feature request on Bugzilla, but as it is not paid, it will have low priority. Best regards, Pedro On 7/31/19 4:40 PM, Nobody Noone wrote: > New to Monetdb, From a DBA perspective I’m having trouble > understanding how to manage access rights to schemas and the tables > therein. My department has many senior analysts who should have admin > rights to the tables within the schema (DML - select, insert, update, > delete, trunc, etc; DDL create table, drop table, create seq, etc). > There are also some users that need select only in a given schema. How > do I manage this without having to grant access per table? Is that > where roles come into play? Say super users get privilege_code_id= 79 > (SELECT,INSERT,UPDATE,DELETE,TRUNCATE) whereas some users would be > assigned privilege_code_id= 1 (SELECT). What about DDL rights? I am > not seeing those in the roles? I do see the ability when the schema is > created with CREATE SCHEMA "ABC" AUTHORIZATION "xyz_user". What I > don’t want to do is have to run a query that searches all tables in a > schema, grant admin DDL to them. > Thanks! > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list

1 0

Cast a string column with empty strings to int
by Ying Zhang 01 Aug '19

01 Aug '19

Hai, I have e.g. the following string column containing either numbers or empty strings: CREATE TABLE tst (s string); INSERT INTO tst VALUES ('1234'), (''), (‘4567'); I’d like to cast ’s' to an INT column. The following query works: sql>SELECT CAST(s AS INT) FROM (SELECT CASE WHEN s='' THEN NULL ELSE s END AS s FROM tst) AS tmp; +------+ | L4 | +======+ | 1234 | | null | | 4567 | +------+ 3 tuples But the following query doesn’t: sql>SELECT CASE WHEN s='' THEN NULL ELSE CAST(s AS INT) END FROM tst; conversion of string '' to type int failed. sql>SELECT CASE WHEN s='' THEN CAST(NULL as INT) ELSE CAST(s AS INT) END FROM tst; conversion of string '' to type int failed. This is probably because batcalc.int is applied on the whole “s” column as shown by the query plan: sql>EXPLAIN SELECT CASE WHEN s='' THEN NULL ELSE CAST(s AS INT) END FROM tst; +--------------------------------------------------------------------------------------------------------------------+ | mal | +====================================================================================================================+ … | C_7:bat[:oid] := sql.tid(X_6:int, "sys":str, "tst":str); | | X_10:bat[:str] := sql.bind(X_6:int, "sys":str, "tst":str, "s":str, 0:int); | | X_19:bat[:str] := algebra.projection(C_7:bat[:oid], X_10:bat[:str]); | | X_21:bat[:bit] := batcalc.==(X_19:bat[:str], "":str); | | X_24:bat[:bit] := batcalc.isnil(X_21:bat[:bit]); | | X_30:bat[:bit] := batcalc.ifthenelse(X_24:bat[:bit], false:bit, X_21:bat[:bit]); | | X_35:bat[:int] := batcalc.int(X_19:bat[:str]); | | X_36:bat[:int] := batcalc.ifthenelse(X_30:bat[:bit], nil:int, X_35:bat[:int]); | … Is my query wrong or the plan? Thanks, Jennie

2 2

Question with wildcarded expressions for LIKE clause
by Gotwisner, Dave 01 Aug '19

01 Aug '19

I am trying to do some constraints off the system tables to select a unique pattern of tables. We have tables like the following (I can't give the actual table names): * Config (a single record exists here, no merge tables use it) * Data_0000 (a small number of records exist in each of these) * Data_0001 * Data_0002 * Data_0003 * Data (this is a merge table over the last 4) * Test_table (a small number of records exist here, no merge tables use it) * Main__yyyy_mm_dd__hh__0000 (millions of records, yyyy, mm, dd, hh are all numbers, we have one set of tables per hour, and keep significantly over 1 weeks' worth of data) * Main__yyyy_mm_dd__hh__0001 * Main__yyyy_mm_dd__hh__0002 * Main__yyyy_mm_dd__hh__0003 * Main__yyyy_mm_dd__hh (this is a merge table over the last 4) If I do a select * from sys.tables where schema_id in (select id from schemas where name = 'frodo') and type in (select table_type_id from table_types where table_type_name = 'TABLE') I get the right thing - all non merge tables. I then want to further break it down into the tables that end in __[0-9][0-9][0-9][0-9] and those that don't (two separate queries for two separate purposes). Consider frodo the name of our schema. Adding and name like 'data\_\_____' appears to work. This should give me the 4 data_* tables. Adding and name like '%\_\_____' gives me all non-merge tables, including config and test_table. I'm assuming % is the wild card for 0 or more, and _ is the wild card for 1. This matches some documentation I have found. The \ are required because I need to explicitly look for 2 of the underscore character. Am I doing this correctly? Are these the right wildcarding characters? MS SQLServer gives a # as a match for a single digit, is there an equivalent? Or, is this just a bug in MonetDB? Thanks, Dave

3 5

Managing Access to Monetdb Schemas
by Nobody Noone 31 Jul '19

31 Jul '19

New to Monetdb, From a DBA perspective I’m having trouble understanding how to manage access rights to schemas and the tables therein. My department has many senior analysts who should have admin rights to the tables within the schema (DML - select, insert, update, delete, trunc, etc; DDL create table, drop table, create seq, etc). There are also some users that need select only in a given schema. How do I manage this without having to grant access per table? Is that where roles come into play? Say super users get privilege_code_id= 79 (SELECT,INSERT,UPDATE,DELETE,TRUNCATE) whereas some users would be assigned privilege_code_id= 1 (SELECT). What about DDL rights? I am not seeing those in the roles? I do see the ability when the schema is created with CREATE SCHEMA "ABC" AUTHORIZATION "xyz_user". What I don’t want to do is have to run a query that searches all tables in a schema, grant admin DDL to them. Thanks!

1 0

How to manage access rights
by Nobody Noone 30 Jul '19

30 Jul '19

New to Monetdb, From a DBA perspective I’m having trouble understanding how to manage access rights to schemas and the tables therein. My department has many senior analysts who should have admin rights to the tables within the schema (DML - select, insert, update, delete, trunc, etc; DDL create table, drop table, create seq, etc). There are also some users that need select only in a given schema. How do I manage this without having to grant access per table? Is that where roles come into play? Say super users get privilege_code_id= 79 (SELECT,INSERT,UPDATE,DELETE,TRUNCATE) whereas some users would be assigned privilege_code_id= 1 (SELECT). What about DDL rights? I am not seeing those in the roles? I do see the ability when the schema is created with CREATE SCHEMA "ABC" AUTHORIZATION "xyz_user". What I don’t want to do is have to run a query that searches all tables in a schema, grant admin DDL to them. Thanks!

1 0

Issue while migrating to MonetDB v11.33.3 (Apr 2019)
by Sharma, Sreejith 25 Jul '19

25 Jul '19

All, We are migrating our MonetDB database from MonetDB v11.27.13 (Jul2017-SP4) to MonetDB v11.33.3 (Apr 2019). We tried to bring up the DB instance with MonetDB v11.33.3 (Apr 2019), but getting below error though the same when we tried with previous version MonetDB v11.31.13 (Aug2018-SP2) it worked. Could you help us to identify the issue? 2019-07-24 04:00:21 ERR TSV_PROD_DB_T[88767]: #main thread:!ERROR: BBPcheckbats: cannot stat file /monet_data01/DBFARM_TSV_TEST/TSV_PROD_DB_T/bat/10/45/104502.tail 2019-07-24 04:00:21 ERR TSV_PROD_DB_T[88767]: #main thread:!OS: No such file or directory 2019-07-24 04:00:21 ERR TSV_PROD_DB_T[88767]: !FATAL: BBPinit: BBPcheckbats failed 2019-07-24 04:00:21 MSG TSV_PROD_DB_T[88767]: !ERROR: BBPcheckbats: cannot stat file /monet_data01/DBFARM_TSV_TEST/TSV_PROD_DB_T/bat/10/45/104502.tail 2019-07-24 04:00:21 MSG TSV_PROD_DB_T[88767]: !OS: No such file or directory 2019-07-24 04:00:23 MSG merovingian[88722]: database 'TSV_PROD_DB_T' (88767) has exited with exit status 1 2019-07-24 04:00:24 ERR control[88722]: (local): failed to fork mserver: database 'TSV_PROD_DB_T' appears to shut itself down after starting, check monetdbd's logfile (merovingian.log) for possible hints 2019-07-24 04:02:18 MSG control[88722]: (local): served status list Regards, Sreejith

2 1

(Без темы)
by Илья Хайбуллин 25 Jul '19

25 Jul '19

How can I write a workable join by null-safe eqauls? I have two tables: create table monet1_test ("Id" bigserial, key int, value varchar(2000)); create table monet2_test ("Id" bigserial, key int, value varchar(2000)); insert into monet1_test select value "Id", value "key", value "value" from generate_series(1,60000); insert into monet2_test select value "Id", value "key", value "value" from generate_series(1,200000); I tried select count(1) from monet2_test t1 inner join monet1_test t2 on ((t1.key = t2.key) or ((t1.key is null) and (t2.key is null))); and got errors GDK reported error: MT_mmap: CreateFileMapping(0000000000001200, &sa, 4, 22, 1508704256, NULL) failed OS: The printer is out of paper. HEAPalloc: Insufficient space for HEAP of 95997921280 bytes. MT_mmap: CreateFileMapping(00000000000011B4, &sa, 4, 22, 1508704256, NULL) failed OS: The printer is out of paper. HEAPalloc: Insufficient space for HEAP of 95997921280 bytes. I create bug-report: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6737

3 7

Indexing support on Json DataType
by Chatchawan Triperm 22 Jul '19

22 Jul '19

Hi All, Does anyone know whether there is built-in indexing on Json datatype or not. I would like to keep my dynamic columns in Json but worry about indexing. Regards, Chatchawan

3 4

Unexpected query result with merge tables and view after upgrade
by Enrico Maria Carmona 22 Jul '19

22 Jul '19

2 2

Questions on embedding R in MonetDB and client pacakge in R
by Andor 18 Jul '19

18 Jul '19

Hello everyone there, I am a beginner user of MonetDB , mainly working with database from R, new to this email list. Is there a seach function somewhere in this list? I can only see the archieve webpage. I have some general user questions, and would greatly appreciate your comments and suggestions. the embedding R in MonetDB.The very latest materials I could find on this topic on MonetDB websites, dated backed in 10/31/2014, I am wondering where I can find more embedding R in MonetDB related information? tutorial, documents etc the client package working with MonetDB from R I only see monetDB.R but not MonetDBLite in the CRAN packages index, yet in MonetDB site , both packages are introduced. any comments on the MonetDBLite, will it be there in the future? Thanks! Andor

1 0

yum upgrade doesn't allow existing db use new window functions
by Andy Allaway 09 Jul '19

09 Jul '19

Hi, I ran: yum upgrade MonetDB-SQL-server5 MonetDB-client When I create a new db I can call: select first_value(col1) over (partition by col2) from analytics; as per the blog monetdb.org/blog/extended_sql_window_functions However a db that existed before the update returns: Function first_value not found. I can see first_value in sys.functions. Any suggestions? Thanks!

2 2

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory
by RanchoYuan 04 Jul '19

04 Jul '19

Hi Stefan, Thanks for your reply! We have run the query a few times with different size of data. There we used 16G RAM(actually 13.5G was used), and find the size of 10G's data is the critical point that can run the query. All of the data files' size are listed below, each file name is a table name(there are only a few tables are refered -- store_sales, date_dim, item, customer, catalog_sales, web_sales): 7.4K call_center.dat 1.6M catalog_page.dat 212M catalog_returns.dat 2.9G catalog_sales.dat 27M customer_address.dat 64M customer.dat 77M customer_demographics.dat 9.9M date_dim.dat 77B dbgen_version.dat 149K household_demographics.dat 328B income_band.dat 2.6G inventory.dat 28M item.dat 61K promotion.dat 1.7K reason.dat 1.1K ship_mode.dat 27K store.dat 323M store_returns.dat 3.8G store_sales.dat 4.9M time_dim.dat 1.2K warehouse.dat 19K web_page.dat 98M web_returns.dat 1.5G web_sales.dat 12K web_site.dat So we guess that the monetdb has no memory management? For the output of `mserver5 --version` is: MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 17.0GiB available memory, 40 available cpu cores Libraries: libpcre: 8.38 2015-11-23 (compiled with 8.38) openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1 Mar 2016) libxml2: 2.9.3 (compiled with 2.9.3) Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 And the size of processes is not limited. To let you reproduce the problem conveniently, I'll provide more details here: you can get tpc-ds from its website(we use version 2.6.0). Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen -scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data. When data has been generated, using the script /expe.sh/ to create tables and load the data. The query script is 123.tpcds.23.sql.(The syntaxs of other queries that tpc-ds generates is not suitable for monetdb all, we don't modify them all when the problem occurred). One more question, I can't get your reply email, so I don't know how to reply you, for this case, I could only send a new mail echo time. Thanks! Regards, Rancho

4 3

Question on communications path between mserver5 and the MonetDB API (libmapi)
by Gotwisner, Dave 04 Jul '19

04 Jul '19

We are transferring a lot of data as the result of some of our queries (20-30 columns of integers and floats) for about 50,000 records, aggregating these from multiple MonetDB instances (on different machines). For this pattern of query, the largest amount of time spent on the query (from the network onwards, ignoring Mserver5 lookup time) is on the data conversion. When running "perf top" on our application, we notice we are spending a significant amount of time (the second most samples per perf top item at 14%) converting the data returned by libmapi from "const char *" to int64_t, int32_t, double, etc. We know the type of the data, and we have very few string columns in the database. Is there an option to MonetDB that allows it to pass binary data rather than textual data back and forth on the wire? Is there an API that will give the data back in the native type rather than in the on-the-wire string type? Both client and server are always running on 64-bit X86 boxes running Linux (meaning, little endian is guaranteed)? We use serialized protobufs whenever we send data on the wire when communicating between our various programs, and the data stream is significantly smaller than sending integers back and forth in ascii. If not, would switching to ODBC / JDBC give us better performance in this area? Thanks, Dave

2 1

Q: Logical plan
by Hanfeng Chen 04 Jul '19

04 Jul '19

Hello, A quick question. Is the logical plan generated by the keyword "plan" is the plan before it is sent to the SQL optimizer *OR* the output plan of the optimizer? >From the online documentation ( https://www.monetdb.org/Documentation/Manuals/SQLreference/PlanSQL) this sentence "*The relational plan used in the SQL optimizer* can be obtained by prepending the query with the keyword PLAN." is not clear to me. Thanks, Hanfeng Chen

2 1

Using the Monet API to connect to MonetDB if MonetDB comes up AFTER our application
by Gotwisner, Dave 24 Jun '19

24 Jun '19

I am currently trying to update my code that uses the Mapi API to retry connections if we start an application when MonetDB is down. Our programs do not run interactively, so detecting the failure, terminating the program, and restarting the application is problematic. There are a couple of issues here that I can use help with. What we are doing is calling mapi_connect(). Mapi_connect returns what appears to be a pointer (it's non-zero). So we call mapi_error with this return value. If we are not connected, we get an MERROR, not an MTIMEOUT, which is what I would expect. With an MERROR failure, we call mapi_error_str() to see what kind of error. We then must do a string compare (actually, a strstr() call looking for a "Connection refused" pattern inside the string. Is there a way to get an enum or #define value instead, with a different function call? Sort of like looking at errno rather than calling strerror(errno) and doing a string compare on the result. This would be way more efficient if we could do this. Yes, errors should be rare, but determining the type of error (or protecting against a string text change on an RPM update) shouldn't be expensive. We then call mapi_explain() to stderr. Is there a routine that returns the value of the output of explain() to a std::string or a "const char *"? We have a logger that does stuff with the strings before logging (timestamping, __line__ __file__, putting error codes, etc. Having the multiline output of the explain() information would be very useful for us. The above process is executed multiple times until we connect. Since the failed connects return what appears to be a pointer, do we need to do anything to free the pointers up? It doesn't appear that we have a memory leak (I ran valgrind). Note that if the connect fails, calling mapi_destroy() crashes. From the documentation, "mapi_destroy()" does "Free handle resources", so it seems reasonable to call it after a failed (and non-null) mapi_connect(). It should probably do nothing, it should not crash. Also, it appears that if I connect multiple times without disconnecting (I had a bug in my code), I can't connect more than 65 times (simultaneous connections). Is this an API issue or a server issue? The reason I ask, is that if it a client issue, we will be having a single client that potentially does queries against hundreds of servers, and aggregates the result. If it is a server issue, is there a parameter that we can tweak to increase this? We do multiple queries in parallel, each on their own connections. Thanks, Dave

2 2

Guidelines for MonetDB in production environments
by Roberto Cornacchia 20 Jun '19

20 Jun '19

Hi all, I'm struggling with optimizing resource sharing of MonetDB in production environments (see also: https://www.monetdb.org/pipermail/users-list/2018-June/010276.html) We run MonetDB instances for several projects / customers on each of our servers. Each MonetDB instance is a docker container (used mainly because of ease of deployment and environment isolation). It is not unusual to have 5-10 MonetDB containers on the same server. In principle, Docker does not even have much to do with this, but it puts everything in a realistic context. ** Memory mserver5 checks the system memory and calibrates on that. When 10 instances are running, they all assume they have the whole memory for themselves. Docker allows to set limits on the container memory. It does that by using cgroups (so Docker just makes things easier, but it's really about cgroups). However, memory limits set by cgroups are not namespaced ( https://ops.tips/blog/why-top-inside-container-wrong-memory/#memory-limits-… ). This means that each container will still see the whole memory and will simply get killed when the container limit has been reached (definitely not a solution). So far, the only mechanism I know to obtain the correect behavior is to run actual VMs for MonetDB. But this is very cumbersome and I want to avoid that as much as possible. Should we let 10 instances believe they each have the whole memory, and let them fight for it? (well, that's what's happening now, and I know for sure it's bad). Perhaps the solution can be as easy as allowing an explicit max memory setting, together with some documentation on the consequences of using low / high values. ** CPU Again, Docker allows to set quotas per container. I think cgroups CPU limits are namespaced, so perhaps this would just work well, I haven't really tried yet. ** I/O Same issue. It would be ideal to be able to set priorities, so that mserver5 instances that do background work get a lower I/O priority than instances serving online queries. Also, recommendations on swap settings would be interesting. How much swap? How to tune swappiness kernel settings? I am very aware that there is no simple answer to most of these questions. Many variables are in the picture. Still, some general thoughts from the developers would be appreciated. I think I have read pretty much everything has ever been written about MonetDB, but when it comes to resource utilization I have always bumped into the very unrealistic assumption that each MonetDB instance has a whole server for itself. As I mentioned above, things could get already much better with simple improvements, like allowing to set the maximum memory usable by each instance. But more in general, I feel there is much need for some guidelines for production environments. Or at least, to start the discussion. Best regards, Roberto

3 5

Error Upgrading to 11.33.3 (Apr2019)
by Ariel Abadi 19 Jun '19

19 Jun '19

Hi! I have a database that It started on version Dec2016-SP5. I have been upgrading from version to version without any problem. Now, Im trying to do the last upgrade (this means from Aug2018-SP2 to Apr2019. But Im facing an error. This is what I see on the log: 2019-06-06 15:17:55 ERR cli_gm[2188]: #main thread:!ERROR: BBPcheckbats: file /opt/monetdb/cli_gm/bat/50/65/506571.tail too small (expected 127992, actual 125580) 2019-06-06 15:17:55 MSG cli_gm[2188]: !ERROR: BBPcheckbats: file /opt/monetdb/cli_gm/bat/50/65/506571.tail too small (expected 127992, actual 125580) 2019-06-06 15:17:55 ERR cli_gm[2188]: !FATAL: BBPinit: BBPcheckbats failed 2019-06-06 15:17:58 MSG merovingian[2180]: database 'cli_gm' (2188) has exited with exit status 1 2019-06-06 15:17:58 ERR control[2180]: (local): failed to fork mserver: database 'cli_gm' appears to shut itself down after starting, check monetdbd's logfile (merovingian.log) for possible hints Is there anything I can do to solve this problem? Thks in advance Ariel PD.If I go back to version Aug2018-SP2 the database keeps working with no problem.

2 2

Questions about using COPY INTO when using GRANTS on tables
by Gotwisner, Dave 14 Jun '19

14 Jun '19

Our application was working quite well when using COPY INTO when inserting data into our tables as the default MonetDB user "monetdb" into the system schema (sys). We have subsequently added users and granted table access to the various users. At this point, INSERT works successfully, but COPY INTO does not. The error message for the COPY INTO statement indicated that we needed to use "ON CLIENT" in our load. That makes no difference. Here are the relevant commands to the system (note the user names, passwords, schema names, and tables names were modified for this example): Program that sets up our basic database does this: START TRANSACTION; CREATE USER admin WITH UNENCRYPTED PASSWORD '***' NAME 'Admin User' SCHEMA sys; CREATE SCHEMA test AUTHORIZATION admin; ALTER USER admin SET SCHEMA test; CREATE USER update WITH UNENCRYPTED PASSWORD '***' NAME 'Update User' SCHEMA test; COMMIT; Then, the program that creates the tables does the following: START TRANSACTION; CREATE MERGE TABLE x (...); GRANT ALL ON x TO "update"; CREATE TABLE x0 (...); GRANT ALL ON x0 TO "update"; CREATE TABLE x1 (...); GRANT ALL ON x1 TO "update"; CREATE TABLE x2 (...); GRANT ALL ON x2 TO "update"; CREATE TABLE x3 (...); GRANT ALL ON x3 TO "update"; COMMIT; And we insert (or try to) (4 threads ingest into their own table, the merge table is used to combine the result set on queries): START TRANSACTION; COPY 100000 OFFSET 2 RECORDS INTO x0 FROM '/dev/shm/MonetDB/foo.csv' (columnlist) ON CLIENT DELIMETERS ',','\n'; This returns an error: !/dev/shm/MonetDB/foo.csv': cannot retrieve files. Because of company policy, I can't provide the schema, but the columns in the csv file and the schema differ, which is why I am using the (...) and (columnlist) in the example. Again, when I omit the CREATE USER, CREATE SCHEMA, and GRANT clauses, everything works correctly. Any ideas on what is needed to make it work? Right now, the ingest application and MonetDB are on the same server. In the future, that might not be true. We are running 11.33.3 on CentOS 7. Thanks, Dave

2 1