- users-list - monetdb.org

TimeZone conversion functions
by Ariel Abadi 03 Nov '16

03 Nov '16

Hi! Can anyone help me understand how or where I can check the included functions on MonetDB to convert timestamp from one timezone to another? Thanks in advance! Rgds Ariel

2 2

MAL optimizer result inspection
by Moritz Bruder 02 Nov '16

02 Nov '16

Hi, is it possible to look at the output of a specific optimizer? I tried using *optimizer.showFlowGraph*, which just hangs, and *optimizer.showPlan*, which does not show some variables. Is there maybe something I missed? Regards, Moritz

2 1

Getting "NOT NULL constraint violated" for a column with a DEFAULT value
by Eyal Rozenberg 24 Oct '16

24 Oct '16

I have a column defined as follows: "div5tailnum" VARCHAR(10) NOT NULL DEFAULT '' (that's the output of \d mytable), or if you like: Div5TailNum VARCHAR(10) NOT NULL DEFAULT '' (that's from the CREATE TABLE command). I'm trying to load data from a CSV file using the following SQL: COPY OFFSET 2 INTO mytable FROM file.csv USING DELIMITERS ',','\n','"' NULL AS ''; but I get: INSERT INTO: NOT NULL constraint violated for column ontime.div5tailnum shouldn't that be … [View More]

3 4

Cannot install MonetDB on Ubuntu 16.04.1 LTS (Xenial Xerus)
by Peter Lee 24 Oct '16

24 Oct '16

Hi- I couldn't install MonetDB on Ubuntu 16.04.1 LTS (Xenial Xerus) hosted on EC2 yesterday. I kept getting the error below. It looks like the repository does not have a "source" folder. However, I noticed that the "Trusty" version had a "source" folder, and so I tried installing the Trusty version on my Xenial box, and it installed successfully. So far, I haven't encountered any problems, but was there anything I did wrong in the installation process? Will I encounter problems down the … [View More]

3 2

Fwd: query with group.subgroupdone' undefined in
by José Cavieres 22 Oct '16

22 Oct '16

Hello, Can anyone explain why adding "2*" in the following query results in an error: Original Query: sql>select (CAST(now()-'1970-01-01 00:00:00' as BIGINT)) as minuto,avg(CAST(N6_end - trStart as BIGINT)) as promedio from testquery where trStart>'2016-10-06 00:00:00.000000' GROUP BY minuto; +---------------+--------------------------+ | minuto | promedio | +===============+==========================+ | 1475770734000 | null | +---------------+-… [View More]-------------------------+ Query with "2*": sql>select 2*CAST(now()-'1970-01-01 00:00:00' as BIGINT) as minuto,avg(CAST(N6_end - trStart as BIGINT)) as promedio from testquery where trStart>'2016-10-06 00:00:00.000000' GROUP BY minuto; TypeException:user.s20_1[10]:'group.subgroupdone' undefined in: (X_15:any,r1_15:any,r2_15:any) := group.subgroupdone(X_14:hge); program contains errors Explain: ------------------------------------------------------------ ------------------------------------------------------------ ---------------------+ | mal | +=========================================================== ============================================================ ======================+ | function user.s22_1(A0:bte,A1:str,A2:str):void; | sql.mvc(); | | X_5:timestamp := mtime.current_timestamp(); | | A1; | | calc.timestamp(X_6,7,1); | | X_10:lng := mtime.diff(X_5,X_7); | | calc.lng(X_10); | | calc.hge(X_11); | | A0; | | X_14:hge := calc.*(X_12,X_13); | | (X_15,r1_15,r2_15) := group.subgroupdone(X_14); | | algebra.projection(r1_15,X_14); | | bat.setKey(X_18,true); | | X_21:bat[:oid] := sql.tid(X_4,"sys","testquery") ; | | X_24:bat[:timestamp] := sql.bind(X_4,"sys","testquery" ,"trstart",0); | | (X_27,r1_27) := sql.bind(X_4,"sys","testquery" ,"trstart",2); | | X_30:bat[:timestamp] := sql.bind(X_4,"sys","testquery" ,"trstart",1); | | sql.delta(X_24,X_27,r1_27,X_30); | | algebra.projection(X_21,X_31); | | X_33:bat[:timestamp] := mal.multiplex("calc","timestam p",X_32,7); | | A2; | | calc.timestamp(X_36,7); | | algebra.thetasubselect(X_33,X_37,">"); | | X_40:bat[:timestamp] := sql.bind(X_4,"sys","testquery" ,"n6_end",0); | | (X_42,r1_42) := sql.bind(X_4,"sys","testquery" ,"n6_end",2); | | X_44:bat[:timestamp] := sql.bind(X_4,"sys","testquery" ,"n6_end",1); | | sql.delta(X_40,X_42,r1_42,X_44); | | algebra.projection(X_21,X_45); | | algebra.projection(X_38,X_46); | | algebra.projection(X_38,X_32); | | X_49:bat[:lng] := mal.multiplex("mtime","diff",X _47,X_48); | | batcalc.lng(X_49); | | batcalc.dbl(X_52); | | X_54:bat[:dbl] := aggr.subavg(X_53,X_15,r1_15,tr ue,true); | | bat.new(nil:oid,nil:str); | | bat.new(nil:oid,nil:str); | | bat.new(nil:oid,nil:str); | | bat.new(nil:oid,nil:int); | | bat.new(nil:oid,nil:int); | | bat.append(X_58,".L1"); | | bat.append(X_61,"minuto"); | | bat.append(X_62,"hugeint"); | | bat.append(X_63,72); | | bat.append(X_65,0); | | bat.append(X_66,"sys.L3"); | | bat.append(X_68,"promedio"); | | bat.append(X_70,"double"); | | bat.append(X_72,53); | | bat.append(X_74,0); | | sql.resultSet(X_76,X_78,X_80,X_82,X_84,X_19,X_54); | | end user.s22_1; | | # querylog.define("explain select 2*cast(now()-\\'1970-01-01 00:00:00\\' as bigint) as minuto,avg(cast(n6_end - trstart as bigint)) as prom | : edio from testquery where trstart>\\'2016-10-06 00:00:00.000000\\' group by minuto;","default_pipe") : Environment: - MonetDB v11.23.7 (Jun2016-SP1) from release package - Centos and Ubuntu - Create table attached - The type of the columns in the query is timestamp - A previous versions of this email was rejected because the "create table" statement is too big for the 40Kbytes limit of this list. Thanks Jose Cavieres [View Less]

4 7

system init script
by Heyns Emiliano 21 Oct '16

21 Oct '16

Does anyone have a working init script for MonetDB? -- Groet, Emiliano Heyns

1 0

how well does MonetDB scale to large servers (e.g. 4 CPU/64+ cores/2+TB RAM)?
by Igor Mendelev 21 Oct '16

21 Oct '16

[Sorry if this question was already asked and answered before - I'm new to this mailing list.] Recently servers with up to 6TB RAM became relatively affordable to be used with MonetDB (considering total cost of server hw + DB sw license) - but will MonetDB scale well to use all that hardware? I know it can be tried on the cloud using (for example) x1.32xlarge EC2 spot instances but before doing that I'd like to ask people with more knowledge/experience with MonetDB if it at least looks … [View More]

2 1

Data lost
by Luciano Sasso 17 Oct '16

17 Oct '16

Hi, I am delete rows from table, with "delete from mytable where id=x", and rows deleted no return in select "select * from mytable where id=x", its OK ! Version "Jul2015-SP1" But, when restart database, deleted rows return in select "select * from mytable where id=x". Its bug ?, or i have execute vacuum command, how to execute vacuum command. ? -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 … [View More]

2 1

new monetdb version, old farm
by Lynn Carol Johnson 17 Oct '16

17 Oct '16

Hello - My understanding is you should not run a new monetdb version with a dbfarm created from an older version. We have just updated our system from a Jul2015-SP3 version to the latest Jun2016-SP2 version. I have created a new db farm and plan to recreate the tables in the new farm. My question; How do you remove the old dbfarm? Our data is quite large (1.8 T) and I'd like to retrieve this space for other use. While I can start the db against the new farm, I can find no method of … [View More]

5 7

Zero copy optimization in embedded python and loop back queries
by Joseph D'silva 16 Oct '16

16 Oct '16

Hi All, I was wondering if zero copy optimization in embedded Python is applicable even for results returned inside functions that use _conn.execute("select ...") ? Also is there something to look for in explain plan that will indicate if zero copy optimization is used or not ? Thanks -J

1 0

weird behavior when using views with calculated columns and double type
by imad hajj chahine 14 Oct '16

14 Oct '16

Hi, When paginating over a view that contains a calculated double columns. i got a TypeException error and sometimes the server crash. To replicate the error, create a table with 2 double columns col1 and col2, fill it with some test data; after create a view as select col1, col2, col1+col2 as col3 from table. when running select col1, col2, col3 from view limit 100 offset 100 i got the error. Also in related context, when i use 2 quantiles function on the calculated view column i get the … [View More]

2 1

Announcement: New Jun2016-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 13 Oct '16

13 Oct '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2016-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jun 2016-SP2 bugfix release Build Environment * We now use the … [View More]

1 0

EXPLAIN output *almost* re-parsable
by Roberto Cornacchia 10 Oct '16

10 Oct '16

Almost for fun, I did this little experiment: take the output of an EXPLAIN from sql, and execute it in MAL (mclient -l mal). I found only *1* detail which doesn't parse, here it is in a minimal example: * Got the explain of a trivial query: sql>\f raw sql>explain select name from sys.functions; % .explain # table_name % mal # name % clob # type % 94 # length function user.s4_1():void; X_32:void := querylog.define("explain select name from sys.functions;","default_pipe",21); … [View More]X_16 := bat.new(nil:oid,nil:str); X_24 := bat.append(X_16,"sys.functions"); X_19 := bat.new(nil:oid,nil:str); X_26 := bat.append(X_19,"name"); X_20 := bat.new(nil:oid,nil:str); X_27 := bat.append(X_20,"varchar"); X_21 := bat.new(nil:oid,nil:int); X_29 := bat.append(X_21,256); X_23 := bat.new(nil:oid,nil:int); X_31 := bat.append(X_23,0); X_1 := sql.mvc(); C_2:bat[:oid] := sql.tid(X_1,"sys","functions"); X_5:bat[:str] := sql.bind(X_1,"sys","functions","name",0); (C_8,r1_8) := sql.bind(X_1,"sys","functions","name",2); X_11:bat[:str] := sql.bind(X_1,"sys","functions","name",1); X_13 := sql.delta(X_5,C_8,r1_8,X_11); X_14 := algebra.projection(C_2,X_13); sql.resultSet(X_24,X_26,X_27,X_29,X_31,X_14); end user.s4_1; # optimizer.mitosis() # optimizer.dataflow() * Opened a MAL session, initialized sql, and pasted the function above: mal>sql.init(); mal>function user.s4_1():void; mal> X_32:void := querylog.define("explain select name from sys.functions;","default_pipe",21); mal> X_16 := bat.new(nil:oid,nil:str); mal> X_24 := bat.append(X_16,"sys.functions"); mal> X_19 := bat.new(nil:oid,nil:str); mal> X_26 := bat.append(X_19,"name"); mal> X_20 := bat.new(nil:oid,nil:str); mal> X_27 := bat.append(X_20,"varchar"); mal> X_21 := bat.new(nil:oid,nil:int); mal> X_29 := bat.append(X_21,256); mal> X_23 := bat.new(nil:oid,nil:int); mal> X_31 := bat.append(X_23,0); mal> X_1 := sql.mvc(); mal> C_2:bat[:oid] := sql.tid(X_1,"sys","functions"); mal> X_5:bat[:str] := sql.bind(X_1,"sys","functions","name",0); mal> (C_8,*r1_8*) := sql.bind(X_1,"sys","functions","name",2); mal> X_11:bat[:str] := sql.bind(X_1,"sys","functions","name",1); mal> X_13 := sql.delta(X_5,C_8,r1_8,X_11); mal> X_14 := algebra.projection(C_2,X_13); mal> sql.resultSet(X_24,X_26,X_27,X_29,X_31,X_14); mal>end user.s4_1; MAPI = (monetdb) /tmp/.s.monetdb.54500 QUERY = end user.s4_1; ERROR = !TypeException:user.s4_1[17]:'sql.delta' undefined in: X_13:any := sql.delta(X_5:bat[:str],C_8:bat[:oid],*r1_8:bat[:any]*,X_11:bat[:str]); mal> The error is easy to find: sql.delta() expects r1_8 to be of type :bat[:str], but it gets :bat[:any] instead. * If I fix it manually: mal>function user.s4_1():void; mal> X_32:void := querylog.define("explain select name from sys.functions;","default_pipe",21); mal> X_16 := bat.new(nil:oid,nil:str); mal> X_24 := bat.append(X_16,"sys.functions"); mal> X_19 := bat.new(nil:oid,nil:str); mal> X_26 := bat.append(X_19,"name"); mal> X_20 := bat.new(nil:oid,nil:str); mal> X_27 := bat.append(X_20,"varchar"); mal> X_21 := bat.new(nil:oid,nil:int); mal> X_29 := bat.append(X_21,256); mal> X_23 := bat.new(nil:oid,nil:int); mal> X_31 := bat.append(X_23,0); mal> X_1 := sql.mvc(); mal> C_2:bat[:oid] := sql.tid(X_1,"sys","functions"); mal> X_5:bat[:str] := sql.bind(X_1,"sys","functions","name",0); mal> (C_8,*r1_8:bat[:str]*) := sql.bind(X_1,"sys","functions","name",2); mal> X_11:bat[:str] := sql.bind(X_1,"sys","functions","name",1); mal> X_13 := sql.delta(X_5,C_8,r1_8,X_11); mal> X_14 := algebra.projection(C_2,X_13); mal> sql.resultSet(X_24,X_26,X_27,X_29,X_31,X_14); mal>end user.s4_1; mal> Yay, it works. ** The reason I write this email:* - I suspect that the output of EXPLAIN is not meant to necessarily be re-parsable, but just to be informative - Nevertheless, it really is almost re-parsable. I tried a much larger query, (2000+ lines of MAL), and this was the only detail. It looks like fixing that small detail could be worth it. - Also, I wonder if this is, in fact, a bug. Binds to access types 0 and 1 do get the right type information. Why not access type 2? [View Less]

2 2

More info about the MAL plan cache
by Roberto Cornacchia 07 Oct '16

07 Oct '16

Hi there, I have a couple of questions about the plan cache. 1) I know it caches MAL a user-functions for a given "SQL query". What does "SQL query" really means? The unparsed SQL script? The parse tree? The relational algebra tree? The normalized/optimized algebra tree? 2) What are the conditions for the cache to be used? Does it kick in only for some types of queries? I noticed some rather large queries (which take 300-400ms just to be translated from SQL to MAL) are not being cached (or … [View More]

1 1

slowdown after delete
by Stefano Fioravanzo 03 Oct '16

03 Oct '16

Hello, In my db I have a table with ~300million rows on which I have to perform ~1k aggregation queries periodically. I have noticed that if I delete even a small amount of rows from the table I have a greatly reduced performance compared to when no delete has been made. Specifically, with no delete the 1k queries complete in about 2 hours, if I delete a few records the task completes in about 10-12 hours. Inserting new rows in the table does not present the issue. Moreover if I dump and … [View More]

2 1

Select with 5K+ columns kills server
by Pascal Heus 03 Oct '16

03 Oct '16

Hi: I have a MonetDB 5 server v11.23.7 "Jun2016-SP1 instance hosting a table with over 5600 columns (for the US General Social Survey) If we run a simple query like SELECT * FROM gss7214_v4 limit 2 we get the first two records as expected. However, the same query with all the column names explicitly stated like select "year","id","wrkstat","hrs1","hrs2","evwork","occ","prestige","wrkslf",<...thousands of additional columns - see attached...> from "gss7214_v4" WHERE "year"=1972 … [View More]

3 2

slowdown after delete
by Stefano Fioravanzo 30 Sep '16

30 Sep '16

Hello, In my db I have a table with ~300million rows on which I have to perform ~1k aggregation queries periodically. I have noticed that if I delete even a small amount of rows from the table I have a greatly reduced performance compared to when no delete has been made. Specifically, with no delete the 1k queries complete in about 2 hours, if I delete a few records the task completes in about 10-12 hours. Inserting new rows in the table does not present the issue. Moreover if I dump and … [View More]

3 6

Running different MonetDB versions in parallel
by Moritz Bruder 30 Sep '16

30 Sep '16

Hi, so I read several times on the mailing-list, that it is indeed possible to run several MonetDB instances on the same server. I tried this myself with the following configuration: System: Linux ... 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux MonetDB version: mercurial revision 324165f1d055 from branch default (also tested with the topmost version) Configure script parameters: --disable-geom --disable-fits --disable-netcdf --disable-… [View More]

3 5

suggestions
by Yves Dorfsman 30 Sep '16

30 Sep '16

Is there a place to make suggestions on improvement for monetdb? Three things that probably seem minor compared to the goals of monetdb, but which would be very helpful in my opinion: - better copy to csv, a la postgres, with keywords lile "csv" and "header" - ctrl-c to cancel a query but not kill the mclient sesion - history kept and re-used between mclient sessions -- http://yves.zioup.com gpg: 4096R/32B0F416

3 2

Binary bulk load null values
by Dennis Pallett 25 Sep '16

25 Sep '16

Hi all, I'm working on a utility to bulk load data into MonetDB using the binary bulk loading options. At the moment I'm running into the issue whereby I'm unable to insert real NULL values in varchar columns. Only a literal "null" is inserted, which obviously fails the 'WHERE col IS NULL' condition. Can anyone tell me what value is used to import a null value into a varchar column? Best regards, Dennis

3 3

subsort fails silently
by Moritz Bruder 20 Sep '16

20 Sep '16

Hi, I just noticed a bug in the current revision (accc14b26ef3) of the default branch. I added a short mal snippet to reproduce the issue. I guess it's somehow related to sorting a void BAT, but I couldn't reduce the bug to a simple BAT (without the TPCH table). Regards, Moritz

2 1

RAID 0
by Vincent Sheffer 20 Sep '16

20 Sep '16

Are there any MonetDB specific advice/instructions for using a RAID 0 SSD array with MonetDB beyond make sure you have backups? I’m not sure what I’m looking for, other than may be tips on how to configure a dbfarm (i.e. monetdbd/mserver5 properties). Thanks, Vince The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination … [View More]

2 2

MonetDB Crystal client code in my article
by Brian Hood 18 Sep '16

18 Sep '16

Hi Guys, I've included a very basic MonetDB client code in the Crystal Language that actually connects to mserver5 performs a query and exits just to demonstrate how easy that is in Crystal to use a C binding but also for creating some more interest in MonetDB. https://github.com/puppetpies/Snippets/blob/master/Crystal%20-%20Ruby%20dyn… I will be publishing it on Medium / Slideshare next week. Regards, Brian Hood

1 0

COPY FROM support compressed files?
by Yves Dorfsman 16 Sep '16

16 Sep '16

Can I COPY FROM directly from compressed files? If so, what format (tried with zstd files, didn't work, but that's very new so not surprised)? Thanks. -- http://yves.zioup.com gpg: 4096R/32B0F416

3 2

Does R run in a sandbox in MonetDB?
by Pablo S 08 Sep '16

08 Sep '16

I'm refering to the MonetDB feature that allows to create R functions that run embedded in the database. For example, you can do: CREATE FUNCTION rapi00() RETURNS TABLE (d INTEGER) LANGUAGE R { seq(1,10); }; Now, I need my users to create dynamically these user defined functions by letting them enter these on the UI; the application will create the function in the database and a process will run them. Problem is that R allows to run operating system level commands, such as list.files(… [View More]

2 1