- users-list - monetdb.org

[MonetDB-users] Monetdb compilation from source on 32 bit machine
by Tapomay Dey 01 Aug '12

01 Aug '12

I checked the code at location and found that "value" is of type long and its bit precision is 61. I am trying to perform make on an Ubuntu 10.04 i686 architecture which is 32-bit In file included from xid.c:23: xid.h:42: error: width of ‘value’ exceeds its type xid.c: In function ‘XIDcompress’: xid.c:138: warning: format ‘%ld’ expects type ‘long int’, but argument 4 has type ‘BUN’ xid.c:161: warning: format ‘%ld’ expects type ‘long int’, but argument 4 has type ‘BUN’ xid.c: In function ‘XIDcolumndump’: xid.c:298: warning: format ‘%lo’ expects type ‘long unsigned int’, but argument 4 has type ‘oid’ make[7]: *** [libmal_la-xid.lo] Error 1

2 1

[MonetDB-users] group by cube
by DADOU 01 Aug '12

01 Aug '12

hello, Have monetdb operator OLAP like group by cube as oracle have i don't find any operator from CUBE or ROLLUP. Thunks -- View this message in context: http://old.nabble.com/group-by-cube-tp34241097p34241097.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] Slowdown in Jul2012 release
by Viktor Rosenfeld 01 Aug '12

01 Aug '12

Hi, I've noticed a slowdown in the Jul2012 which appears to be due to disk I/O. Printed below is the output of TRACE on the Jul2012 (11.11.5) release and the Apr2012-SP2 release (11.9.7) of the following query: #+BEGIN_EXAMPLE WITH span1 AS ( SELECT node.id FROM node ), solutions AS ( SELECT DISTINCT span1.id AS id1 FROM span1 ) SELECT count(*) FROM solutions #+END_EXAMPLE Jul2012: #+BEGIN_EXAMPLE +---------+-----------------------------------------------------------------------------------+ | ticks | stmt | +---------+-----------------------------------------------------------------------------------+ | 3 | X_1 := sql.mvc(); | | 11 | X_2:bat[:oid,:int] =<tmp_1230>[1262014] := sql.bind(X_1=0,"annis","node","id",0); | | 5 | X_7:bat[:oid,:oid] =<tmp_1366>[0] := sql.bind_dbat(X_1=0,"annis","node",1); | | 6 | X_9=<tmpr_1366>[0] := bat.reverse(X_7=nil:bat[:oid,:oid]); | | 13 | X_10=<tmp_2126>[1262014] := algebra.kdifference(X_2=nil:bat[:oid,:int],X_9=nil); | | 3543662 | (ext22=<tmp_2127>[1262014],grp20) := group.done(X_10=<tmp_2126>[1262014]); | | 18 | X_13=<tmp_2130>[1262014] := bat.mirror(ext22=nil); | | 82193 | X_14=<tmp_2131>[1262014] := algebra.leftjoin(X_13=nil,X_10=nil); | | 2185 | X_15 := aggr.count(X_14=nil); | | 15 | sql.exportValue(1,"annis.solutions","L1","wrd",64,0,6,X_15=1262014:wrd,""); | | 8 | end s0_1; | | 18 | function user.s0_1(); | | 3629118 | X_5:void := user.s0_1(); | +---------+-----------------------------------------------------------------------------------+ #+END_EXAMPLE This query is accompanied by a lot of disk I/O (both reading and writing) Apr2012-SP2: #+BEGIN_EXAMPLE +-------+-----------------------------------------------------------------------------------+ | ticks | stmt | +-------+-----------------------------------------------------------------------------------+ | 4 | X_1 := sql.mvc(); | | 12 | X_2:bat[:oid,:int] =<tmp_1471>[1262014] := sql.bind(X_1=0,"annis","node","id",0); | | 6 | X_7:bat[:oid,:oid] =<tmp_1365>[0] := sql.bind_dbat(X_1=0,"annis","node",1); | | 6 | X_9=<tmpr_1365>[0] := bat.reverse(X_7=nil:bat[:oid,:oid]); | | 14 | X_10=<tmp_2074>[1262014] := algebra.kdifference(X_2=nil:bat[:oid,:int],X_9=nil); | | 12116 | (ext22=<tmp_2107>[1262014],grp20) := group.done(X_10=<tmp_2074>[1262014]); | | 2427 | X_13=<tmp_2106>[1262014] := bat.mirror(ext22=nil); | | 46 | X_14=<tmp_2111>[1262014] := algebra.leftjoin(X_13=nil,X_10=nil); | | 6 | X_15 := aggr.count(X_14=nil); | | 11 | sql.exportValue(1,"annis.solutions","L1","wrd",64,0,6,X_15=1262014:wrd,""); | | 6 | end s0_1; | | 16 | function user.s0_1(); | | 15566 | X_5:void := user.s0_1(); | +-------+-----------------------------------------------------------------------------------+ #+END_EXAMPLE There is no disk I/O during this query. Notice the difference in runtime of the group.done statement in line 5. This statement is introduced by the DISTINCT in the query. Cheers, Viktor

1 0

[MonetDB-users] Multiplex-funnel cluster failover: master-master replication
by Tapomay Dey 31 Jul '12

31 Jul '12

Hi, I have setup a cluster of 3 machines with following tags: mydb/1/master, mydb/2/master, mydb/3/master On a fourth machine I have created a multiplex funnel as follows: monetdb create -m monetdb+monetdb@*/mydb/1/master,monetdb+monetdb@*/mydb/2/master,monetdb+monetdb@*/mydb/3/master mydbmaster My objective here is that I use this multiplex for only DDL/INSERTS/UPDATES. This would ensure replication on all the 3 masters thus achieving a hacked form of master-master replication. Then I could use any one of the masters for analytics querying. However I am stuck with the following issues while handling fail-over. 1. I expect that if one of the boxes die, others keep getting the updates until I realize the failure. The I could lock the multiplex, sync the dbfarms to catch up the failing db and then release the multiplex. 2. If the box with tag mydb/3/master dies, then the updates go to 1 and 2. But if mydb/1/master dies the none of the dbs get any more updates. Looks like the multiplex returns on the first failure without trying rest of the dbs. Any pointers as to where do I change this sequential failure logic? Also what do you think about my replication approach? Thanks and Regards, Tapomay

1 0

[MonetDB-users] Incorrect Results from Very Simple Queries ---- Updated
by Henry Addington 31 Jul '12

31 Jul '12

Please ignore my earlier email. I accidently sent the email before it was completed! We are seeing a very strange behavior from MonetDB from simple queries on a single table. In monthly_tou_avg_price table, we have monthly average aggregate data that were inserted from a very large table (> 2 billion) with hourly data. We have created the aggregate table to speed up certain queries. Strangely, we are getting incorrect results from very simple queries on the aggregate table. Q1 below returns 9,890 records with yyyy_mm column having empty value for the 2012-07 records. All of the 2012-07 records are missing the yyyy_mm value. We ran the SQL statement that inserted those records and it does not show any empty yyyy_mm values. If we check for empty string or null value in yyyy_mm column (Q2 and Q3) against the aggregate table, we do not get any hits. If we query for yyyy_mm= '2012-07' as in Q4, we get exactly 9890 records showing yyyy_mm populated with value 2012-07. How could this be possible, Q1 showing empty yyyy_mm values for the same 9,890 records and Q4 showing 2012-07 yyyy_mm values? Q1: select * from monthly_tou_avg_price where header_id = 6 Q2: select * from monthly_tou_avg_price where header_id = 6 and yyyy_mm = '' Q3: select * from monthly_tou_avg_price where header_id = 6 and yyyy_mm is null Q4: select * from monthly_tou_avg_price where header_id = 6 and yyyy_mm = '2012-07' Q5 below returns empty value for yyyy_mm even though yyyy_mm column does not have any empty or null values. Q6 does not return any empty or null yyyy_mm values and contradicts Q5 result by returning 2012-07. Furthermore, Q7 contradicts Q5 with 9,890 records with 2012-07 value in yyyy_mm column. Q5: select distinct yyyy_mm from monthly_tou_avg_price where header_id = 6 Q6: select distinct yyyy_mm from monthly_tou_avg_price Q7: select * from monthly_tou_avg_price where header_id = 6 and years = 2012 and months = 7 We also ran an UPDATE statement to set yyyy_mm to 2012-07 for rows with header_id = 6 and years = 2012 and months = 7. And, 9,890 records were successfully updated. However, Q1 through Q8 returned same results as above with no improvement. We have not seen this kind of issues in other tables. This problem seems to be limited to a table that is populated with an INSERT INTO statement. We checked the result returned by the select portion of Q8 and yyyy_mm is populated with 2012-07. But, the aggregate table seems to end up in a weird state where it returns incorrect results. Q8: insert into "monthly_tou_avg_price" select a.header_id, extract(year from a.opr_date) as y, extract(month from a.opr_date) as m, b.yyyymm as yyyy_mm, b.svalue as TOU, a.resource_name, round(avg(a.lmp_cong_prc),5), round(avg(a.lmp_ene_prc),5), round(avg(a.lmp_loss_prc),5), round(avg(a.lmp_prc),5), round(avg(a.ns_clr_prc),5), round(avg(a.rd_clr_prc),5), round(avg(a.ru_clr_prc),5), round(avg(a.sp_clr_prc),5), round(avg(a.shadow_prc),5), round(avg(a.fuel_prc),5), count(lmp_cong_prc) as actual_lmpcong_tou_count from oasislive.price a join oasislive.time b on a.yyyymmddhh = b.yyyymmddhh where a.opr_date between '2012-07-01' and '2012-07-31' group by a.header_id, y, m, yyyy_mm, TOU, a.resource_name order by y, m, a.header_id, TOU, a.resource_name; We can see this INSERT INTO related issue with other queries. Q9 shows that we do not have any values for 2011-03 ON period which is incorrect. In fact, if we specify '2011-03' as in Q10, it shows that we do in fact have values for the 2011-03 ON period! Q9: select yyyy_mm, tou, avg(lmp_prc) from monthly_tou_avg_price group by yyyy_mm, tou order by yyyy_mm, tou yyyy_mm tou L6 . . 2011-03 OFF 16.721532797520894 2011-03 ON (null) 2011-04 OFF 15.427080074671187 . . Q10: select yyyy_mm, tou, avg(lmp_prc) from monthly_tou_avg_price where yyyy_mm = '2011-03' group by yyyy_mm, tou order by yyyy_mm, tou yyyy_mm tou L11 2011-03 OFF 16.72153279752101 2011-03 ON 30.276486575893575 I apologize for the long-winded email. I just wasn't sure how to describe what I was seeing.... Please let me know if there is any additional information that I can provide you with. By the way, we are currently using August 2011 -SP2 release, but saw this issue in older releases as well. Best regards, Henry We are currently running MonetDB August 2011 SP2 release on CentOS

1 0

[MonetDB-users] Repeatedly writing to MonetDB
by Jerry Ramey 29 Jul '12

29 Jul '12

We are encountering a frequent error when writing to MonetDb repeatedly using the Java MAPI socket. We are running the April 2012 release. The error is: [7/29/12 12:10:26 PM] Cory Isaacson: 000!TypeException:user.s0_2[2]:'calc.str' undefined in: _3:any := calc.str(_4:str) 22000!TypeException:user.s0_2[3]:'calc.str' undefined in: _5:any := calc.str(_6:str) 22000!TypeException:user.s0_2[4]:'calc.str' undefined in: _7:any := calc.str(_8:str) 22000!TypeException:user.s0_2[5]:'calc.str' undefined in: _9:any := calc.str(_10:str) 22000!TypeException:user.s0_2[6]:'calc.str' undefined in: _11:any := calc.str(_12:str) The only solution is to stop the farm and restart the farm/database. We have a single Java process writing to the database using MAPI, in batches of 1000 rows at a time (or less), using the COPY INTO syntax based on the example Java loader.We have a single Java process writing to the database using MAPI, in batches of 1000 rows at a time (or less), using the COPY INTO syntax based on the example Java loader. Here are the important snippets of the Java code. You can see that we are opening/closing a connection for each batch, and we have only a single thread performing this function, no other processes are writing to the database. We do have other connections reading from the database (but in the most recent case we only had idle connections). This is running on CentOS 5.4, with Java 1.6. There is no swap space configured on this server (we read that this may be a requirement). The data volumes are very small at this point, only a few million rows. String query = "COPY " + (dataRowList.size() + 10) + " RECORDS INTO " + tableName + " FROM STDIN USING DELIMITERS '\\t','\\n' NULL AS '\\\\N';"; if(TRACE) logger.trace("commit: query : " + query); // the leading 's' is essential, since it is a protocol // marker that should not be omitted, likewise the // trailing semicolon out.write('s'); out.write(query); out.newLine(); // Write all data rows. for(String dataRow : dataRowList) { if(TRACE) logger.trace("commit: Writing data row: " + dataRow + " dataRow ends in newline: " + dataRow.endsWith("\n")); // The dataRow is \n terminated, so no need to output newLine to output stream. out.write(dataRow); } out.writeLine(""); // need this one for synchronisation over flush() error = in.waitForPrompt(); if (error != null) throw new Exception(error); out.writeLine(""); // server wants more, we're going to tell it, this is it error = in.waitForPrompt(); if (error != null) throw new Exception(error); // // disconnect from server // server.close(); } catch (IOException e) { logger.error("Unable to connect", e); throw new RuntimeException("Unable to connect", e); } catch (Throwable th) { logger.error("Commit error", th); ... throw new RuntimeException("Commit error: tableName: " + tableName, th); } finally { try { if(server != null) { server.close(); server = null; } } catch (Throwable th) { logger.error("Error closing MonetDb server connection", th); } } Thanks, Jerry

1 0

[MonetDB-users] Repeatedly writing to MonetDB
by Jerry Ramey 29 Jul '12

29 Jul '12

We are encountering a frequent error when writing to MonetDb repeatedly using the Java MAPI socket. We are running the April 2012 release. The error is: [7/29/12 12:10:26 PM] Cory Isaacson: 000!TypeException:user.s0_2[2]:'calc.str' undefined in: _3:any := calc.str(_4:str) 22000!TypeException:user.s0_2[3]:'calc.str' undefined in: _5:any := calc.str(_6:str) 22000!TypeException:user.s0_2[4]:'calc.str' undefined in: _7:any := calc.str(_8:str) 22000!TypeException:user.s0_2[5]:'calc.str' undefined in: _9:any := calc.str(_10:str) 22000!TypeException:user.s0_2[6]:'calc.str' undefined in: _11:any := calc.str(_12:str) The only solution is to stop the farm and restart the farm/database. We have a single Java process writing to the database using MAPI, in batches of 1000 rows at a time (or less), using the COPY INTO syntax based on the example Java loader.We have a single Java process writing to the database using MAPI, in batches of 1000 rows at a time (or less), using the COPY INTO syntax based on the example Java loader. Here are the important snippets of the Java code. You can see that we are opening/closing a connection for each batch, and we have only a single thread performing this function, no other processes are writing to the database. We do have other connections reading from the database (but in the most recent case we only had idle connections). This is running on CentOS 5.4, with Java 1.6. There is no swap space configured on this server (we read that this may be a requirement). The data volumes are very small at this point, only a few million rows. String query = "COPY " + (dataRowList.size() + 10) + " RECORDS INTO " + tableName + " FROM STDIN USING DELIMITERS '\\t','\\n' NULL AS '\\\\N';"; if(TRACE) logger.trace("commit: query : " + query); // the leading 's' is essential, since it is a protocol // marker that should not be omitted, likewise the // trailing semicolon out.write('s'); out.write(query); out.newLine(); // Write all data rows. for(String dataRow : dataRowList) { if(TRACE) logger.trace("commit: Writing data row: " + dataRow + " dataRow ends in newline: " + dataRow.endsWith("\n")); // The dataRow is \n terminated, so no need to output newLine to output stream. out.write(dataRow); } out.writeLine(""); // need this one for synchronisation over flush() error = in.waitForPrompt(); if (error != null) throw new Exception(error); out.writeLine(""); // server wants more, we're going to tell it, this is it error = in.waitForPrompt(); if (error != null) throw new Exception(error); // // disconnect from server // server.close(); } catch (IOException e) { logger.error("Unable to connect", e); throw new RuntimeException("Unable to connect", e); } catch (Throwable th) { logger.error("Commit error", th); ... throw new RuntimeException("Commit error: tableName: " + tableName, th); } finally { try { if(server != null) { server.close(); server = null; } } catch (Throwable th) { logger.error("Error closing MonetDb server connection", th); } } Thanks, Jerry

1 0

[MonetDB-users] substing made monetdb very very slow
by li, xiaohua 27 Jul '12

27 Jul '12

Hi all: we are testing monetdb on a sql select substring(fact_bk,1,1) as aa,sum(1) as recordcount from f_activity group by aa the table f_activity have about 5 million rows. when i run this sql, monetdb need about 30 seconds . (if we use another no calculated column, it need only 1-3 seconds) when I run this sql on sql server, it needs only 7 seconds. is monetdb not good at calculated column? or we have made some mistake? please tell me the reason, thank you. lixiaohua Shanghai,China

2 4

[MonetDB-users] how to access remote server in windows
by li, xiaohua 26 Jul '12

26 Jul '12

Hi all: we want to copy a text file into a remote server, we don't know how to set the mapi_open = yes on windows. we try add --set "mapi_open= yes" in the M5server.bat . but we don't know whether it take effect. how can we know? when we use jdbc or odbc to access a remote monetdb database. we try to use jdbc:monetdb://dvs001:5000/demo. or change the odbc host to an ip address: 192.168.1.100. we can't connect to the database. only we use localhost can we connect to the database. does monetdb support remote access in windows? lixiaohua Shanghai,China

3 7

[MonetDB-users] can't copy into a remote server in windows
by li, xiaohua 26 Jul '12

26 Jul '12

Hi all: we want to copy a txt file into a remote monetdb (windows). if we copy into local monetdb via ipaddress, it's OK. but if we copy the same txt file into a remote monetdb, it can't. Error: could not open file 'd:\\share\\book1.csv': Operation not permitted SQLState: 22000 ErrorCode: 0 what's the problem? is it a bug? or something we don't know? lixiaohua Shanghai,China

3 3