- developers-list - monetdb.org

Left/Right joins performance - Need help!!
by Vijay Krishna 08 Oct '14

08 Oct '14

Hi all, I have been comparing MySQL with MonetDB. Obviously, queries that took minutes in MySQL got executed in a matter of few seconds in Monet. However, I found a real blockade with joins. I have 2 tables - each with 150 columns. Among these (150+150) columns, around 60 are CHARACTER LARGE OBJECT type. Both the tables are populated with around 50,000 rows - with data in all the 150 columns. The average length of data in a CLOB type column is 9,000 (varying from 2 characters to 20,000 characters). The primary key of both the tables have same values and the join is always based on the primary key. The rows are by default inserted in ascending order on the primary key. When I ran *an inner join query on these two tables with about 5 criteria and with limit 1000, Monet processed this query in 5 seconds*, which is completely impressive compared to MySQL's (19 seconds). But when I ran *the same query with same criteria and limits using a left or right joins, Monet took around 5 minutes, which is clearly way behind MySQL's (just 22 seconds)*. Having read a lot about Monet's blinding speed compared to traditional relation row based DBs, I could feel that I am missing something, but couldn't figure out what. Can anyone please tell me why there is such a huge difference in such query execution time and how I can prevent it? Much grateful to have any help. Thanks a lot in advance. P.S.: I am running Monet on Macbook Pro - 2.3 GHz Core i7 - Quad core processor with 8 GB RAM. With Regards, Vijayakrishna.P. Mobile : +91-9500402305.

1 0

Re: MonetDB: default - Disable -pthread for Clang 6.*
by Sjoerd Mullender 19 Sep '14

19 Sep '14

Shouldn't this go to the Oct2014 branch? On 19/09/14 11:45, Dimitar Nedev wrote: > Changeset: 10ae6a3308bf for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=10ae6a3308bf > Modified Files: > configure.ag > Branch: default > Log Message: > > Disable -pthread for Clang 6.* > This is in line OS X Yosemite and more specifically Xcode 6.0.1 shipping with Clang verion 6 > > > diffs (19 lines): > > diff --git a/configure.ag b/configure.ag > --- a/configure.ag > +++ b/configure.ag > @@ -1885,11 +1885,11 @@ if test "x$have_pthread" != xno; then > save_LIBS="$LIBS" > save_CPPFLAGS="$CPPFLAGS" > case $GCC-$have_pthread-$CC_ver in > - yes-auto-clang-5.*|yes-yes-clang-5.*) > - # clang 5.* (at least on Mac OSX Mavericks) does not > + yes-auto-clang-5.*|yes-yes-clang-5.*|yes-auto-clang-6.*|yes-yes-clang-6.*) > + # clang 5.*/6.* (Xcode 6.0) does not > # seem to have / require -pthread as compiler > - # option; on Mac OSX Mavericks, "Apple LLVM version > - # 5.0 (clang-500.2.79) (based on LLVM 3.3svn)" > + # option; on Mac OS X Yosamite, "Apple LLVM > + # version 6.0 (clang-600.0.51) (based on LLVM 3.5svn)" > # complains about "error: argument unused during > # compilation: '-pthread'". > # However, disabling -pthread completely might be > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 2

How varchar/text column works ?
by Mike 16 Sep '14

16 Sep '14

Hi, i have been working on monetdb for quite some time and am trying to understand how string column works so as i can (try to ) make it import string columns as fast as possible.The tool is external and generates monetdb binary compatible columns( and may be used to optimize tables in offline mode). As far as reading code and understanding it. here is my part String columns is kept in 2 parts, x.tail and x.theap ( where x is number ) x.tail contains pointers into x.theap x.theap has 2 regions/parts first part contains a 8KB hash table, to detect collisions and second part contains 8 zeroes,8 Byte hash, variable length String which is null terminated and aligned on 8 byte boundary (i am talking about x64 bit arch) and next string follows and so on However, the problem i am facing is , the tools works and generates columns till x.tail contains pointers addressing x.theap positions less than 255. The moment pointers are 2 byte tail entry, monetdb is confused. The reason is tail pointers are varsized for efficiency, and if there are few small string, pointers will fit in small memory and not waste 8 bytes per entry. However, i am unable to understand how monetdb figures out, how big entry is currently being used ? in case of small strings, it's 1 byte (tested) in case of some bigger string, it's 2 bytes (tested) and even in large string tables, it 4 bytes (tested) and large (>4 GB) sting tables, it can be 8 bytes ??? ( this case has not been tested) i have understood, it has something to with bat->T->shift , bat->T->width but still it doesn't work.So, i am definitely missing something. So, my questions are 1) is it an efficient method ( not talking about space efficiency) to have var-sized tail pointers. because it might involve lots of casting and so on during calculations.. 2) how to disable var-sized tail pointers (if it's possible) so as all pointers are 4byte or 8 byte long ( any compile time option or runtime option ) 3) if 2) not possible , then how does monetdb calculate tail pointer width. for example, i am using following code, to initialize new bat with my tail and theap pointers cap=num_of_tail_entries; bs = BATcreatedesc(TYPE_void, TYPE_str, 1); bn = &bs->B; BATsetdims(bn); BATkey(bn, TRUE); BATsetcapacity(bn, cap); BATsetcount(bn, cap); bn->tsorted = 0; bn->trevsorted = 0; bn->tdense = 0; bn->tkey = 0; bn->tvarsized=1; // try to set shift to 1 and thus width to 2 bytes //bn->T->shift=1; // this property seems to be used to calculate tail ptr size //bn->T->width=2; // this property seems to be used to calculate tail ptr size bn->T->varsized=1; then i load both heap and theap. The above code works and i am able to query the table as long as tail pointers are 1 byte long. However, as soon as pointers size is 2 bytes,it fails, so i must set something in BAT init to tell tail ptr size. Any clues are welcome... ( if more information is required, please ask.) Thanks for any help . Rgds mike

1 1

Re: MonetDB: Oct2014 - split update bat within the catalog interface...
by Sjoerd Mullender 09 Sep '14

09 Sep '14

This causes an assertion failed during upgrade from a Jan2014 database. On 2014-09-09 08:24, Niels Nes wrote: > Changeset: 1c3d356ffdc2 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=1c3d356ffdc2 > Modified Files: > sql/backends/monet5/rel_bin.c > sql/backends/monet5/sql.c > sql/backends/monet5/sql_gencode.c > sql/backends/monet5/sql_optimizer.c > sql/backends/monet5/sql_statistics.c > sql/include/sql_catalog.h > sql/storage/bat/bat_storage.c > Branch: Oct2014 > Log Message: > > split update bat within the catalog interface to remove dependency on > BATselect. > > > diffs (truncated from 395 to 300 lines): > > diff --git a/sql/backends/monet5/rel_bin.c b/sql/backends/monet5/rel_bin.c > --- a/sql/backends/monet5/rel_bin.c > +++ b/sql/backends/monet5/rel_bin.c > @@ -733,7 +733,7 @@ stmt_col( mvc *sql, sql_column *c, stmt > (c->base.flag != TR_NEW || c->t->base.flag != TR_NEW /* alter */) && > (c->t->persistence == SQL_PERSIST || c->t->persistence == SQL_DECLARED_TABLE) && !c->t->commit_action) { > stmt *i = stmt_bat(sql->sa, c, RD_INS); > - stmt *u = stmt_bat(sql->sa, c, RD_UPD); > + stmt *u = stmt_bat(sql->sa, c, RD_UPD_ID); > sc = stmt_project_delta(sql->sa, sc, u, i); > sc = stmt_project(sql->sa, del, sc); > } else if (del) { /* always handle the deletes */ > @@ -751,7 +751,7 @@ stmt_idx( mvc *sql, sql_idx *i, stmt *de > (i->base.flag != TR_NEW || i->t->base.flag != TR_NEW /* alter */) && > (i->t->persistence == SQL_PERSIST || i->t->persistence == SQL_DECLARED_TABLE) && !i->t->commit_action) { > stmt *ic = stmt_idxbat(sql->sa, i, RD_INS); > - stmt *u = stmt_idxbat(sql->sa, i, RD_UPD); > + stmt *u = stmt_idxbat(sql->sa, i, RD_UPD_ID); > sc = stmt_project_delta(sql->sa, sc, u, ic); > sc = stmt_project(sql->sa, del, sc); > } else if (del) { /* always handle the deletes */ > diff --git a/sql/backends/monet5/sql.c b/sql/backends/monet5/sql.c > --- a/sql/backends/monet5/sql.c > +++ b/sql/backends/monet5/sql.c > @@ -489,7 +489,7 @@ table_has_updates(sql_trans *tr, sql_tab > > for ( n = t->columns.set->h; !cnt && n; n = n->next) { > sql_column *c = n->data; > - BAT *b = store_funcs.bind_col(tr, c, RD_UPD); > + BAT *b = store_funcs.bind_col(tr, c, RD_UPD_ID); > cnt |= BATcount(b) > 0; > BBPunfix(b->batCacheid); > } > @@ -1644,6 +1644,7 @@ mvc_bind_wrap(Client cntxt, MalBlkPtr mb > bn = BATslice(b, part_nr * psz, (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz)); > BATseqbase(bn, part_nr * psz); > } else { > + /* BAT b holds the UPD_ID bat */ > oid l, h; > BAT *c = mvc_bind(m, *sname, *tname, *cname, 0); > cnt = BATcount(c); > @@ -1651,23 +1652,39 @@ mvc_bind_wrap(Client cntxt, MalBlkPtr mb > l = part_nr * psz; > h = (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz); > h--; > - bn = BATmirror(BATselect(BATmirror(b), &l, &h)); > + bn = BATsubselect(b, NULL, &l, &h, 1, 0, 0); > BBPreleaseref(c->batCacheid); > } > BBPreleaseref(b->batCacheid); > b = bn; > + } else if (upd) { > + BAT *uv = mvc_bind(m, *sname, *tname, *cname, RD_UPD_VAL); > + int *uvl = (int *) getArgReference(stk, pci, 1); > + > + BBPkeepref(*bid = b->batCacheid); > + BBPkeepref(*uvl = uv->batCacheid); > + return MAL_SUCCEED; > } > if (upd) { > int *uvl = (int *) getArgReference(stk, pci, 1); > > if (BATcount(b)) { > - BAT *id = BATmirror(BATmark(b, 0)); > - BAT *vl = BATmirror(BATmark(BATmirror(b), 0)); > + BAT *uv = mvc_bind(m, *sname, *tname, *cname, RD_UPD_VAL); > + BAT *ui = mvc_bind(m, *sname, *tname, *cname, RD_UPD_ID); > + BAT *id = BATproject(b, ui); > + BAT *vl = BATproject(b, uv); > + bat_destroy(b); > + bat_destroy(ui); > + bat_destroy(uv); > BBPkeepref(*bid = id->batCacheid); > BBPkeepref(*uvl = vl->batCacheid); > } else { > + sql_schema *s = mvc_bind_schema(m, *sname); > + sql_table *t = mvc_bind_table(m, s, *tname); > + sql_column *c = mvc_bind_column(m, t, *cname); > + > *bid = e_bat(TYPE_oid); > - *uvl = e_bat(b->T->type); > + *uvl = e_bat(c->type.type->localtype); > } > BBPreleaseref(b->batCacheid); > } else { > @@ -1711,6 +1728,7 @@ mvc_bind_idxbat_wrap(Client cntxt, MalBl > bn = BATslice(b, part_nr * psz, (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz)); > BATseqbase(bn, part_nr * psz); > } else { > + /* BAT b holds the UPD_ID bat */ > oid l, h; > BAT *c = mvc_bind_idxbat(m, *sname, *tname, *iname, 0); > cnt = BATcount(c); > @@ -1718,23 +1736,37 @@ mvc_bind_idxbat_wrap(Client cntxt, MalBl > l = part_nr * psz; > h = (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz); > h--; > - bn = BATmirror(BATselect(BATmirror(b), &l, &h)); > + bn = BATsubselect(b, NULL, &l, &h, 1, 0, 0); > BBPreleaseref(c->batCacheid); > } > BBPreleaseref(b->batCacheid); > b = bn; > + } else if (upd) { > + BAT *uv = mvc_bind_idxbat(m, *sname, *tname, *iname, RD_UPD_VAL); > + int *uvl = (int *) getArgReference(stk, pci, 1); > + BBPkeepref(*bid = b->batCacheid); > + BBPkeepref(*uvl = uv->batCacheid); > + return MAL_SUCCEED; > } > if (upd) { > int *uvl = (int *) getArgReference(stk, pci, 1); > > if (BATcount(b)) { > - BAT *id = BATmirror(BATmark(b, 0)); > - BAT *vl = BATmirror(BATmark(BATmirror(b), 0)); > + BAT *uv = mvc_bind_idxbat(m, *sname, *tname, *iname, RD_UPD_VAL); > + BAT *ui = mvc_bind_idxbat(m, *sname, *tname, *iname, RD_UPD_ID); > + BAT *id = BATproject(b, ui); > + BAT *vl = BATproject(b, uv); > + bat_destroy(b); > + bat_destroy(ui); > + bat_destroy(uv); > BBPkeepref(*bid = id->batCacheid); > BBPkeepref(*uvl = vl->batCacheid); > } else { > + sql_schema *s = mvc_bind_schema(m, *sname); > + sql_idx *i = mvc_bind_idx(m, s, *iname); > + > *bid = e_bat(TYPE_oid); > - *uvl = e_bat(b->T->type); > + *uvl = e_bat((i->type==join_idx)?TYPE_oid:TYPE_wrd); > } > BBPreleaseref(b->batCacheid); > } else { > @@ -3959,7 +3991,7 @@ SQLcluster1(Client cntxt, MalBlkPtr mb, > c = o->data; > if (first) { > first = 0; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > msg = CLUSTER_key(&hid, &b->batCacheid); > BBPreleaseref(b->batCacheid); > if (msg) > @@ -3973,7 +4005,7 @@ SQLcluster1(Client cntxt, MalBlkPtr mb, > throw(SQL, "sql.cluster", "Can not access descriptor"); > } > > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL) > throw(SQL, "sql.cluster", "Can not access descriptor"); > msg = CLUSTER_apply(&bid, b, map); > @@ -4039,7 +4071,7 @@ SQLcluster2(Client cntxt, MalBlkPtr mb, > bat psum; > unsigned int bits = 10, off = 0; > first = 0; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > msg = MKEYbathash(&hid, &b->batCacheid); > BBPreleaseref(b->batCacheid); > if (msg) > @@ -4051,7 +4083,7 @@ SQLcluster2(Client cntxt, MalBlkPtr mb, > return msg; > } > > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL) > throw(SQL, "sql.cluster", "Can not access descriptor"); > msg = CLS_map(&bid, &mid, &b->batCacheid); > @@ -4132,7 +4164,7 @@ vacuum(Client cntxt, MalBlkPtr mb, MalSt > bids[i] = 0; > for (o = t->columns.set->h; o; o = o->next, i++) { > c = o->data; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL || (msg = (*func) (&bid, &(b->batCacheid), &(del->batCacheid))) != NULL) { > for (i--; i >= 0; i--) > BBPdecref(bids[i], TRUE); > @@ -4226,7 +4258,7 @@ SQLvacuum(Client cntxt, MalBlkPtr mb, Ma > > for (o = t->columns.set->h; o && ordered == 0; o = o->next) { > c = o->data; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL) > throw(SQL, "sql.vacuum", "Can not access descriptor"); > ordered |= BATtordered(b); > @@ -4409,7 +4441,7 @@ sql_storage(Client cntxt, MalBlkPtr mb, > for (ncol = (t)->columns.set->h; ncol; ncol = ncol->next) { > sql_base *bc = ncol->data; > sql_column *c = (sql_column *) ncol->data; > - BAT *bn = store_funcs.bind_col(tr, c, 0); > + BAT *bn = store_funcs.bind_col(tr, c, RDONLY); > lng sz; > > /*printf("schema %s.%s.%s" , b->name, bt->name, bc->name); */ > @@ -4474,7 +4506,7 @@ sql_storage(Client cntxt, MalBlkPtr mb, > sql_base *bc = ncol->data; > sql_idx *c = (sql_idx *) ncol->data; > if (c->type != no_idx) { > - BAT *bn = store_funcs.bind_idx(tr, c, 0); > + BAT *bn = store_funcs.bind_idx(tr, c, RDONLY); > lng sz; > > /*printf("schema %s.%s.%s" , b->name, bt->name, bc->name); */ > diff --git a/sql/backends/monet5/sql_gencode.c b/sql/backends/monet5/sql_gencode.c > --- a/sql/backends/monet5/sql_gencode.c > +++ b/sql/backends/monet5/sql_gencode.c > @@ -897,7 +897,7 @@ static int > q = newStmt2(mb, sqlRef, bindRef); > if (q == NULL) > return -1; > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > q = pushReturn(mb, q, newTmpVariable(mb, newBatType(ht, tt))); > } else > setVarType(mb, getArg(q, 0), newBatType(ht, tt)); > @@ -910,7 +910,7 @@ static int > return -1; > s->nr = getDestVar(q); > > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > /* rename second result */ > renameVariable(mb, getArg(q, 1), "r1_%d", s->nr); > } > @@ -924,7 +924,7 @@ static int > q = newStmt2(mb, sqlRef, bindidxRef); > if (q == NULL) > return -1; > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > q = pushReturn(mb, q, newTmpVariable(mb, newBatType(ht, tt))); > } else > setVarType(mb, getArg(q, 0), newBatType(ht, tt)); > @@ -937,7 +937,7 @@ static int > return -1; > s->nr = getDestVar(q); > > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > /* rename second result */ > renameVariable(mb, getArg(q, 1), "r1_%d", s->nr); > } > diff --git a/sql/backends/monet5/sql_optimizer.c b/sql/backends/monet5/sql_optimizer.c > --- a/sql/backends/monet5/sql_optimizer.c > +++ b/sql/backends/monet5/sql_optimizer.c > @@ -138,7 +138,7 @@ SQLgetStatistics(Client cntxt, mvc *m, M > if (i && (!isRemote(i->t) && !isMergeTable(i->t))) { > cnt = store_funcs.count_idx(tr, i, 1); > assert(cnt <= (size_t) GDK_oid_max); > - b = store_funcs.bind_idx(m->session->tr, i, 0); > + b = store_funcs.bind_idx(m->session->tr, i, RDONLY); > if (b) { > str loc; > if (b->batPersistence == PERSISTENT && BATlocation(&loc, &b->batCacheid) && loc) > @@ -158,7 +158,7 @@ SQLgetStatistics(Client cntxt, mvc *m, M > > cnt = store_funcs.count_col(tr, c, 1); > assert(cnt <= (size_t) GDK_oid_max); > - b = store_funcs.bind_col(m->session->tr, c, 0); > + b = store_funcs.bind_col(m->session->tr, c, RDONLY); > if (b) { > str loc; > if (b->batPersistence == PERSISTENT && BATlocation(&loc, &b->batCacheid) && loc) > diff --git a/sql/backends/monet5/sql_statistics.c b/sql/backends/monet5/sql_statistics.c > --- a/sql/backends/monet5/sql_statistics.c > +++ b/sql/backends/monet5/sql_statistics.c > @@ -98,7 +98,7 @@ sql_analyze(Client cntxt, MalBlkPtr mb, > for (ncol = (t)->columns.set->h; ncol; ncol = ncol->next) { > sql_base *bc = ncol->data; > sql_column *c = (sql_column *) ncol->data; > - BAT *bn = store_funcs.bind_col(tr, c, 0), *br; > + BAT *bn = store_funcs.bind_col(tr, c, RDONLY), *br; > BAT *bsample; > lng sz = BATcount(bn); > > diff --git a/sql/include/sql_catalog.h b/sql/include/sql_catalog.h > --- a/sql/include/sql_catalog.h > +++ b/sql/include/sql_catalog.h > @@ -92,8 +92,9 @@ > > #define RDONLY 0 > #define RD_INS 1 > -#define RD_UPD 2 > -#define QUICK 3 > +#define RD_UPD_ID 2 > +#define RD_UPD_VAL 3 > +#define QUICK 4 > > #define FRAME_ROWS 0 > #define FRAME_RANGE 1 > diff --git a/sql/storage/bat/bat_storage.c b/sql/storage/bat/bat_storage.c > --- a/sql/storage/bat/bat_storage.c > +++ b/sql/storage/bat/bat_storage.c > @@ -52,7 +52,7 @@ delta_bind_del(sql_dbat *bat, int access > (void) access; /* satisfy compiler */ > #endif > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

AND / OR evaluation inconsistency
by Scott Mathieson 03 Sep '14

03 Sep '14

Hi, I've come across something that is either a bug or is just my misunderstanding of how MonetDB deals with NULL results from boolean expressions. The issue I'm having is the OR and AND expressions behave differently in when evaluating a bollean expression where one of the variable is NULL (This is using version 11.15.19) If I run a query: SELECT ([some expression that evaluates to NULL] OR [some expression that evaluates to True] ) as "Result" - the returned value is <null> If I try SELECT ([some expression that evaluates to NULL] AND [some expression that evaluates to True] ) as "Result" - the returned value is True This would seem to suggest the AND evaluation is ignoring NULL or Treating NULL as meaning True whereas OR is propagating it. Which is correct and is this a bug? Thanks, Scott Mathieson ________________________________

3 2

Re: MonetDB: Oct2014 - fixed reuse
by Stefan Manegold 26 Aug '14

26 Aug '14

Niels, this checkin adds "div_by_zero_in_return.Bug-3555" (in addition to "orderby_on_constant.Bug-3554") to sql/test/BugTracker-2014/Tests/All but does not add the respective files (div_by_zero_in_return.Bug-3555.*). Do you have these files ready, or should we remove "div_by_zero_in_return.Bug-3555" from sql/test/BugTracker-2014/Tests/All ? The respective bug report is still open: https://www.monetdb.org/bugzilla/show_bug.cgi?id=3555 Thanks! Stefan ----- Original Message ----- > Changeset: 6c4080d30b3e for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=6c4080d30b3e > Added Files: > sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.sql > sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.err > sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.out > Modified Files: > monetdb5/modules/kernel/bat5.c > sql/backends/monet5/rel_bin.c > sql/backends/monet5/sql.c > sql/backends/monet5/sql_statement.c > sql/test/BugTracker-2014/Tests/All > Branch: Oct2014 > Log Message: > > fixed reuse > > fixed bug 3554, ie make sure single value order by /limit work > > > diffs (254 lines): > > diff --git a/monetdb5/modules/kernel/bat5.c b/monetdb5/modules/kernel/bat5.c > --- a/monetdb5/modules/kernel/bat5.c > +++ b/monetdb5/modules/kernel/bat5.c > @@ -2198,9 +2198,9 @@ BKCshrinkBATmap(int *ret, int *bid, int > Type *r = (Type*)Tloc(bn, BUNfirst(bn)); \ > for (;p<q; oidx++, p++) { \ > if ( *o == oidx ){ \ > - while ( *ol == bidx && ol>o) { \ > + while ( ol>o && *--ol == bidx) { \ > bidx--; \ > - ol--;q--; \ > + q--; \ > } \ > *r++ = *(--q); \ > o += (o < ol); \ > @@ -2241,9 +2241,10 @@ BKCreuseBAT(int *ret, int *bid, int *did > throw(MAL, "bat.reuse", MAL_MALLOC_FAIL ); > } > > - bidx= BUNlast(b)-1; > + oidx = b->hseqbase; > + bidx = oidx + BUNlast(b)-1; > o = (oid*)Tloc(bs, BUNfirst(bs)); > - ol= (oid*)Tloc(bs, BUNlast(bs))-1; > + ol= (oid*)Tloc(bs, BUNlast(bs)); > > switch(ATOMstorage(b->ttype) ){ > case TYPE_bte: reuseloop(bte); break; > @@ -2262,15 +2263,16 @@ BKCreuseBAT(int *ret, int *bid, int *did > > for (;p<q; oidx++, p++) { > if ( *o == oidx ){ > - while ( *ol == bidx && ol>o) { > + while ( ol > o && *--ol == bidx) { > bidx--; > - ol--;q--; > + q--; > } > BUNappend(bn, BUNtail(bi, --q), FALSE); > o += (o < ol); > bidx--; > - } else > + } else { > BUNappend(bn, BUNtail(bi, p), FALSE); > + } > } > } else { > switch( b->T->width){ > @@ -2298,6 +2300,7 @@ BKCreuseBAT(int *ret, int *bid, int *did > BBPkeepref(*ret= bn->batCacheid); > return MAL_SUCCEED; > } > + > str > BKCreuseBATmap(int *ret, int *bid, int *did) > { > @@ -2330,10 +2333,11 @@ BKCreuseBATmap(int *ret, int *bid, int * > throw(MAL, "bat.shrinkMap", MAL_MALLOC_FAIL ); > } > > - bidx= BUNlast(b)-1; > - o = (oid*)Tloc(bs, BUNfirst(bs)); > - ol= (oid*)Tloc(bs, BUNlast(bs)); > - r = (oid*)Tloc(bn, BUNfirst(bn)); > + oidx = b->hseqbase; > + bidx = oidx + BUNlast(b)-1; > + o = (oid*)Tloc(bs, BUNfirst(bs)); > + ol = (oid*)Tloc(bs, BUNlast(bs)); > + r = (oid*)Tloc(bn, BUNfirst(bn)); > > for (;oidx<bidx; oidx++) { > if ( *o == oidx ){ > @@ -2343,8 +2347,9 @@ BKCreuseBATmap(int *ret, int *bid, int * > *r++ = bidx; > o += (o < ol); > bidx--; > - } else > + } else { > *r++ = oidx; > + } > } > > BATsetcount(bn, BATcount(b)-BATcount(bs)); > diff --git a/sql/backends/monet5/rel_bin.c b/sql/backends/monet5/rel_bin.c > --- a/sql/backends/monet5/rel_bin.c > +++ b/sql/backends/monet5/rel_bin.c > @@ -2306,6 +2306,8 @@ rel2bin_project( mvc *sql, sql_rel *rel, > if (!orderbycolstmt) > return NULL; > > + /* handle constants */ > + orderbycolstmt = column(sql->sa, orderbycolstmt); > if (!limit) { /* topn based on a single column */ > limit = stmt_limit(sql->sa, orderbycolstmt, stmt_atom_wrd(sql->sa, 0), > l, LIMIT_FLAG(distinct, is_ascending(orderbycole), last, 1)); > } else { /* topn based on 2 columns */ > diff --git a/sql/backends/monet5/sql.c b/sql/backends/monet5/sql.c > --- a/sql/backends/monet5/sql.c > +++ b/sql/backends/monet5/sql.c > @@ -4136,6 +4136,11 @@ vacuum(Client cntxt, MalBlkPtr mb, MalSt > > /* get the deletions BAT */ > del = mvc_bind_dbat(m, *sch, *tbl, RD_INS); > + if (BATcount(del) == 0) { > + BBPreleaseref(del->batCacheid); > + return MAL_SUCCEED; > + } > + > > i = 0; > bids[i] = 0; > @@ -4163,6 +4168,7 @@ vacuum(Client cntxt, MalBlkPtr mb, MalSt > BBPdecref(bids[i], TRUE); > throw(SQL, name, "Too many columns to handle, use copy instead"); > } > + BBPreleaseref(del->batCacheid); > > mvc_clear_table(m, t); > for (o = t->columns.set->h, i = 0; o; o = o->next, i++) { > @@ -4245,13 +4251,15 @@ SQLvacuum(Client cntxt, MalBlkPtr mb, Ma > /* get the deletions BAT */ > del = mvc_bind_dbat(m, *sch, *tbl, RD_INS); > > - /* now decide on the algorithm */ > - if (ordered) { > - if (BATcount(del) > cnt / 20) > - SQLshrink(cntxt, mb, stk, pci); > - } else > - SQLreuse(cntxt, mb, stk, pci); > - > + if (BATcount(del) > 0) { > + /* now decide on the algorithm */ > + if (ordered) { > + if (BATcount(del) > cnt / 20) > + SQLshrink(cntxt, mb, stk, pci); > + } else { > + SQLreuse(cntxt, mb, stk, pci); > + } > + } > BBPreleaseref(del->batCacheid); > return MAL_SUCCEED; > } > diff --git a/sql/backends/monet5/sql_statement.c > b/sql/backends/monet5/sql_statement.c > --- a/sql/backends/monet5/sql_statement.c > +++ b/sql/backends/monet5/sql_statement.c > @@ -1719,6 +1719,9 @@ const_column(sql_allocator *sa, stmt *va > s->op1 = val; > s->op4.typeval = *ct; > s->nrcols = 1; > + > + s->tname = val->tname; > + s->cname = val->cname; > return s; > } > > diff --git a/sql/test/BugTracker-2014/Tests/All > b/sql/test/BugTracker-2014/Tests/All > --- a/sql/test/BugTracker-2014/Tests/All > +++ b/sql/test/BugTracker-2014/Tests/All > @@ -36,3 +36,5 @@ round.Bug-3542 > number_4_4.Bug-3543 > ifthenelse.Bug-3546 > subselect-reverse.Bug-3552 > +orderby_on_constant.Bug-3554 > +div_by_zero_in_return.Bug-3555 > diff --git a/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.sql > b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.sql > new file mode 100644 > --- /dev/null > +++ b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.sql > @@ -0,0 +1,1 @@ > +select a+b from (select 1 as a,1 as b) as q order by a limit 1; > diff --git > a/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.err > b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.err > new file mode 100644 > --- /dev/null > +++ b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.err > @@ -0,0 +1,37 @@ > +stderr of test 'orderby_on_constant.Bug-3554` in directory > 'sql/test/BugTracker-2014` itself: > + > + > +# 19:40:06 > > +# 19:40:06 > "mserver5" "--debug=10" "--set" "gdk_nr_threads=0" "--set" > "mapi_open=true" "--set" "mapi_port=31986" "--set" > "mapi_usock=/var/tmp/mtest-21225/.s.monetdb.31986" "--set" "monet_prompt=" > "--forcemito" "--set" "mal_listing=2" > "--dbpath=/home/niels/scratch/rc-monetdb/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2014" > "--set" "mal_listing=0" "--set" "embedded_r=yes" > +# 19:40:06 > > + > +# builtin opt gdk_dbpath = > /home/niels/scratch/rc-monetdb/Linux-x86_64/var/monetdb5/dbfarm/demo > +# builtin opt gdk_debug = 0 > +# builtin opt gdk_vmtrim = no > +# builtin opt monet_prompt = > > +# builtin opt monet_daemon = no > +# builtin opt mapi_port = 50000 > +# builtin opt mapi_open = false > +# builtin opt mapi_autosense = false > +# builtin opt sql_optimizer = default_pipe > +# builtin opt sql_debug = 0 > +# cmdline opt gdk_nr_threads = 0 > +# cmdline opt mapi_open = true > +# cmdline opt mapi_port = 31986 > +# cmdline opt mapi_usock = /var/tmp/mtest-21225/.s.monetdb.31986 > +# cmdline opt monet_prompt = > +# cmdline opt mal_listing = 2 > +# cmdline opt gdk_dbpath = > /home/niels/scratch/rc-monetdb/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2014 > +# cmdline opt mal_listing = 0 > +# cmdline opt embedded_r = yes > +# cmdline opt gdk_debug = 536870922 > + > +# 19:40:07 > > +# 19:40:07 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" > "--host=/var/tmp/mtest-21225" "--port=31986" > +# 19:40:07 > > + > + > +# 19:40:07 > > +# 19:40:07 > "Done." > +# 19:40:07 > > + > diff --git > a/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.out > b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.out > new file mode 100644 > --- /dev/null > +++ b/sql/test/BugTracker-2014/Tests/orderby_on_constant.Bug-3554.stable.out > @@ -0,0 +1,38 @@ > +stdout of test 'orderby_on_constant.Bug-3554` in directory > 'sql/test/BugTracker-2014` itself: > + > + > +# 19:40:06 > > +# 19:40:06 > "mserver5" "--debug=10" "--set" "gdk_nr_threads=0" "--set" > "mapi_open=true" "--set" "mapi_port=31986" "--set" > "mapi_usock=/var/tmp/mtest-21225/.s.monetdb.31986" "--set" "monet_prompt=" > "--forcemito" "--set" "mal_listing=2" > "--dbpath=/home/niels/scratch/rc-monetdb/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2014" > "--set" "mal_listing=0" "--set" "embedded_r=yes" > +# 19:40:06 > > + > +# MonetDB 5 server v11.19.0 > +# This is an unreleased version > +# Serving database 'mTests_sql_test_BugTracker-2014', using 4 threads > +# Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically > linked > +# Found 7.334 GiB available main-memory. > +# Copyright (c) 1993-July 2008 CWI. > +# Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved > +# Visit http://www.monetdb.org/ for further information > +# Listening for connection requests on > mapi:monetdb://localhost.nes.nl:31986/ > +# Listening for UNIX domain connection requests on > mapi:monetdb:///var/tmp/mtest-21225/.s.monetdb.31986 > +# MonetDB/GIS module loaded > +# MonetDB/SQL module loaded > +# MonetDB/R module loaded > + > +Ready. > + > +# 19:40:07 > > +# 19:40:07 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" > "--host=/var/tmp/mtest-21225" "--port=31986" > +# 19:40:07 > > + > +#select a+b from (select 1 as a,1 as b) as q order by a limit 1; > +% .L1 # table_name > +% L1 # name > +% smallint # type > +% 1 # length > +[ 2 ] > + > +# 19:40:07 > > +# 19:40:07 > "Done." > +# 19:40:07 > > + > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: Protobuf - Protocol buffer prototype for the record
by Stefan Manegold 26 Aug '14

26 Aug '14

Thanks, Hannes! Just for the records, new (and required) files pbstream.c pbstream.h mapi_pb-c.c mapi_pb-c.h are missing. Stefan ----- Original Message ----- > Changeset: af2757009ce3 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=af2757009ce3 > Modified Files: > clients/mapiclient/Makefile.ag > clients/mapiclient/mclient.c > clients/mapilib/Makefile.ag > clients/mapilib/mapi.c > clients/mapilib/mapi.h > common/stream/Makefile.ag > common/stream/stream.c > common/utils/mcrypt.c > common/utils/mcrypt.h > configure.ag > monetdb5/extras/jaql/Makefile.ag > monetdb5/extras/mal_optimizer_template/Makefile.ag > monetdb5/mal/Makefile.ag > monetdb5/mal/mal_authorize.c > monetdb5/mal/mal_client.c > monetdb5/mal/mal_client.h > monetdb5/mal/mal_debugger.c > monetdb5/mal/mal_readline.c > monetdb5/mal/mal_scenario.c > monetdb5/mal/mal_session.c > monetdb5/mal/mal_session.h > monetdb5/modules/atoms/Makefile.ag > monetdb5/modules/kernel/Makefile.ag > monetdb5/modules/mal/Makefile.ag > monetdb5/modules/mal/mal_mapi.c > monetdb5/modules/mal/mal_mapi.h > monetdb5/optimizer/Makefile.ag > monetdb5/scheduler/Makefile.ag > sql/backends/monet5/LSST/Makefile.ag > sql/backends/monet5/Makefile.ag > sql/backends/monet5/UDF/Makefile.ag > sql/backends/monet5/sql_result.c > sql/backends/monet5/sql_scenario.c > sql/backends/monet5/sql_scenario.h > sql/backends/monet5/vaults/Makefile.ag > sql/storage/bat/Makefile.ag > tools/merovingian/daemon/client.c > tools/mserver/Makefile.ag > tools/mserver/mserver5.c > Branch: Protobuf > Log Message: > > Protocol buffer prototype for the record > > > diffs (truncated from 2665 to 300 lines): > > diff --git a/clients/mapiclient/Makefile.ag b/clients/mapiclient/Makefile.ag > --- a/clients/mapiclient/Makefile.ag > +++ b/clients/mapiclient/Makefile.ag > @@ -16,7 +16,7 @@ > # All Rights Reserved. > > MTSAFE > -INCLUDES = ../mapilib ../../common/options ../../common/stream > $(READLINE_INCS) > +INCLUDES = ../mapilib ../../common/options ../../common/stream > $(READLINE_INCS) $(PROTOBUF_C_CFLAGS) > > lib_mcutil = { > NOINST > @@ -35,8 +35,8 @@ bin_msqldump = { > SOURCES = msqldump.c > LIBS = libmcutil ../mapilib/libmapi \ > ../../common/stream/libstream \ > - $(curl_LIBS) > -} > + $(curl_LIBS) > + } > > bin_stethoscope = { > SOURCES = stethoscope.c > diff --git a/clients/mapiclient/mclient.c b/clients/mapiclient/mclient.c > --- a/clients/mapiclient/mclient.c > +++ b/clients/mapiclient/mclient.c > @@ -347,13 +347,9 @@ fetch_line(MapiHdl hdl) > static int > fetch_row(MapiHdl hdl) > { > - char *reply; > > - do { > - if ((reply = fetch_line(hdl)) == NULL) > - return 0; > - } while (*reply != '[' && *reply != '='); > - return mapi_split_line(hdl); > + // TODO: this is almost a NOP, because we already have everything from the > PB. fetch_line() just has to pay attention to the export size in the future. > + return mapi_fetch_row(hdl); > } > > static void > @@ -668,11 +664,14 @@ CSVrenderer(MapiHdl hdl) > s = mapi_get_name(hdl, i); > if (s == NULL) > s = ""; > - mnstr_printf(toConsole, "%s%s", > + printf( "%s%s", > i == 0 ? "" : sep, s ? s : ""); > } > - mnstr_printf(toConsole, "\n"); > + printf( "\n"); > } > + > + // TODO: switch on type! > + > while (!mnstr_errnr(toConsole) && (fields = fetch_row(hdl)) != 0) { > for (i = 0; i < fields; i++) { > s = mapi_fetch_field(hdl, i); > @@ -681,37 +680,38 @@ CSVrenderer(MapiHdl hdl) > if (strchr(s, *sep) != NULL || > strchr(s, '\n') != NULL || > strchr(s, '"') != NULL) { > - mnstr_printf(toConsole, "%s\"", > + printf( "%s\"", > i == 0 ? "" : sep); > + printf("%s",s); > while (*s) { > switch (*s) { > case '\n': > - mnstr_write(toConsole, "\\n", 1, 2); > + printf( "\\n"); > break; > case '\t': > - mnstr_write(toConsole, "\\t", 1, 2); > + printf( "\\t"); > break; > case '\r': > - mnstr_write(toConsole, "\\r", 1, 2); > + printf( "\\r"); > break; > case '\\': > - mnstr_write(toConsole, "\\\\", 1, 2); > + printf( "\\\\"); > break; > case '"': > - mnstr_write(toConsole, "\"\"", 1, 2); > + printf( "\"\""); > break; > default: > - mnstr_write(toConsole, s, 1, 1); > + > break; > } > s++; > } > - mnstr_write(toConsole, "\"", 1, 1); > + printf("\""); > } else > - mnstr_printf(toConsole, "%s%s", > + printf( "%s%s", > i == 0 ? "" : sep, s ? s : ""); > } > - mnstr_printf(toConsole, "\n"); > + printf( "\n"); > } > } > > diff --git a/clients/mapilib/Makefile.ag b/clients/mapilib/Makefile.ag > --- a/clients/mapilib/Makefile.ag > +++ b/clients/mapilib/Makefile.ag > @@ -18,11 +18,11 @@ > MTSAFE > > INCLUDES = ../../common/options ../../common/stream ../../common/utils \ > - $(MSGCONTROL_FLAGS) > + $(MSGCONTROL_FLAGS) $(PROTOBUF_C_CFLAGS) > > lib_mapi = { > VERSION = $(MAPI_VERSION) > - SOURCES = mapi.c mapi.rc > + SOURCES = mapi.c mapi.rc > LIBS = $(SOCKET_LIBS) ../../common/stream/libstream \ > ../../common/options/libmoptions \ > ../../common/utils/libmcrypt $(openssl_LIBS) > diff --git a/clients/mapilib/mapi.c b/clients/mapilib/mapi.c > --- a/clients/mapilib/mapi.c > +++ b/clients/mapilib/mapi.c > @@ -772,10 +772,13 @@ > > #include "monetdb_config.h" > #include <stream.h> /* include before mapi.h */ > + > #include <stream_socket.h> > #include "mapi.h" > #include "mcrypt.h" > > +#include "pbstream.h" > + > #ifdef HAVE_UNISTD_H > # include <unistd.h> > #endif > @@ -946,6 +949,7 @@ struct MapiResultSet { > struct MapiColumn *fields; > struct MapiRowBuf cache; > int commentonly; /* only comments seen so far */ > + MapiReply__QueryResult *pb; > }; > > struct MapiStatement { > @@ -1429,7 +1433,6 @@ close_result(MapiHdl hdl) > { > struct MapiResultSet *result; > Mapi mid; > - int i; > > result = hdl->result; > if (result == NULL) > @@ -1438,125 +1441,13 @@ close_result(MapiHdl hdl) > assert(mid != NULL); > if (mid->trace == MAPI_TRACE) > printf("closing result set\n"); > - if (result->tableid >= 0 && result->querytype != Q_PREPARE) { > - if (mid->active && > - result->next == NULL && > - !mid->active->needmore && > - read_into_cache(mid->active, -1) != MOK) > - return MERROR; > - assert(hdl->npending_close == 0 || > - (hdl->npending_close > 0 && hdl->pending_close != NULL)); > - if (mid->active && > - (mid->active->active != result || > - result->cache.tuplecount < result->row_count)) > - { > - /* results for which we got all tuples at the initial > - * response, need not to be closed as the server already > - * did that immediately */ > - if (result->row_count > result->tuple_count) { > - /* can't write "X" commands now, so save for later */ > - REALLOC(hdl->pending_close, hdl->npending_close + 1); > - hdl->pending_close[hdl->npending_close] = result->tableid; > - hdl->npending_close++; > - } > - } else if (mid->to != NULL) { > - /* first close saved up to-be-closed tables */ > - for (i = 0; i < hdl->npending_close; i++) { > - char msg[256]; > - > - snprintf(msg, sizeof(msg), "Xclose %d\n", hdl->pending_close[i]); > - mapi_log_record(mid, msg); > - mid->active = hdl; > - if (mnstr_printf(mid->to, "%s", msg) < 0 || > - mnstr_flush(mid->to)) { > - close_connection(mid); > - mapi_setError(mid, mnstr_error(mid->to), "mapi_close_handle", > MTIMEOUT); > - break; > - } > - read_into_cache(hdl, 0); > - } > - hdl->npending_close = 0; > - if (hdl->pending_close) > - free(hdl->pending_close); > - hdl->pending_close = NULL; > - if (mid->to != NULL && result->tuple_count < result->row_count) { > - char msg[256]; > - > - snprintf(msg, sizeof(msg), "Xclose %d\n", result->tableid); > - mapi_log_record(mid, msg); > - mid->active = hdl; > - if (mnstr_printf(mid->to, "%s", msg) < 0 || > - mnstr_flush(mid->to)) { > - close_connection(mid); > - mapi_setError(mid, mnstr_error(mid->to), "mapi_close_handle", > MTIMEOUT); > - } else > - read_into_cache(hdl, 0); > - } > - } > - result->tableid = -1; > + if (hdl->active != NULL) { > + // TODO: cleanup > } > - if (mid->active == hdl && > - hdl->active == result && > - read_into_cache(hdl, -1) != MOK) > - return MERROR; > - assert(hdl->active != result); > - if (result->fields) { > - for (i = 0; i < result->maxfields; i++) { > - if (result->fields[i].tablename) > - free(result->fields[i].tablename); > - if (result->fields[i].columnname) > - free(result->fields[i].columnname); > - if (result->fields[i].columntype) > - free(result->fields[i].columntype); > - } > - free(result->fields); > - } > - result->fields = NULL; > - result->maxfields = result->fieldcnt = 0; > - if (result->cache.line) { > - for (i = 0; i < result->cache.writer; i++) { > - if (result->cache.line[i].rows) > - free(result->cache.line[i].rows); > - if (result->cache.line[i].anchors) { > - int j; > - > - for (j = 0; j < result->cache.line[i].fldcnt; j++) > - if (result->cache.line[i].anchors[j]) { > - free(result->cache.line[i].anchors[j]); > - result->cache.line[i].anchors[j] = NULL; > - } > - free(result->cache.line[i].anchors); > - } > - if (result->cache.line[i].lens) > - free(result->cache.line[i].lens); > - } > - free(result->cache.line); > - result->cache.line = NULL; > - result->cache.tuplecount = 0; > - } > - if (result->errorstr) > - free(result->errorstr); > - result->errorstr = NULL; > - result->hdl = NULL; > - hdl->result = result->next; > - if (hdl->result == NULL) > - hdl->lastresult = NULL; > - result->next = NULL; > - free(result); > + > return MOK; > } > > -static void > -add_error(struct MapiResultSet *result, char *error) > -{ > - /* concatenate the error messages */ > - size_t size = result->errorstr ? strlen(result->errorstr) : 0; > - > - REALLOC(result->errorstr, size + strlen(error) + 2); > - strcpy(result->errorstr + size, error); > - strcat(result->errorstr + size, "\n"); > -} > - > char * > mapi_result_error(MapiHdl hdl) > { > @@ -1571,17 +1462,11 @@ MapiMsg > mapi_next_result(MapiHdl hdl) > { > mapi_hdl_check(hdl, "mapi_next_result"); > - > - while (hdl->result != NULL) { > +// TODO: we only like to export more > + while (hdl->active != NULL) { > if (close_result(hdl) != MOK) > return MERROR; > - if (hdl->result && > - (hdl->result->querytype == -1 || > - /* basically exclude Q_PARSE and Q_BLOCK */ > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: Contents of developers-list Digest, Vol 24, Issue 3
by Foteini Alvanaki 14 Aug '14

14 Aug '14

The changes to the geos are neither documented nor send somewhere for future use. The implemented functions force users to create polygons without holes (there is no argument for the holes). In case the default geos library is used, an error informs the user that the geometry type she is using is not the correct one. My change in geos added this geometry type. Best, Foteini On 08/14/2014 12:00 PM, developers-list-request(a)monetdb.org wrote: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. Re: MonetDB: geo - MakePolygon/Polygon implmentation. Works > only for... (Stefan Manegold) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 13 Aug 2014 14:00:47 +0200 (CEST) > From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> > To: developers-list(a)monetdb.org > Subject: Re: MonetDB: geo - MakePolygon/Polygon implmentation. Works > only for... > Message-ID: <1961125276.5600498.1407931247727.JavaMail.zimbra(a)cwi.nl> > Content-Type: text/plain; charset=utf-8 > > > Thanks! > > purely out of curiosity: > > - are there checks & warnings/ error messages in place > for not (trying) to use it for polygons with holes, > as well as a (configure) check whether the geos library > provides the required functionality? > > - are the changes to the geos module documented (at least > as "diff" with reference to which code version it applies), > or have these changes even been send "up-stream" to have them > by default included in a future version of the library? > > Thanks! > Stefan > > ----- Original Message ----- >> Changeset: 4d4c6ce1a817 for MonetDB >> URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4d4c6ce1a817 >> Modified Files: >> geom/monetdb5/geom.c >> geom/monetdb5/geom.mal >> geom/sql/40_geom.sql >> Branch: geo >> Log Message: >> >> MakePolygon/Polygon implmentation. Works only for polygons without holes. >> Changes to geos library were necessary so, it will not work with standard >> geos library. >> >> >> diffs (80 lines): >> >> diff --git a/geom/monetdb5/geom.c b/geom/monetdb5/geom.c >> --- a/geom/monetdb5/geom.c >> +++ b/geom/monetdb5/geom.c >> @@ -600,7 +600,7 @@ str geom_2_geom(wkb** resWKB, wkb **valu >> int valueType = 0; >> >> int valueSRID = (*valueWKB)->srid; >> -//fprintf(stderr, "in geom_2_geom\n"); >> + >> /* get the geosGeometry from the wkb */ >> geosGeometry = wkb2geos(*valueWKB); >> /* get the number of coordinates the geometry has */ >> @@ -784,7 +784,7 @@ wkb* geos2wkb(const GEOSGeometry* geosGe >> geomWKB->srid = GEOSGetSRID(geosGeometry); >> memcpy(&geomWKB->data, w, wkbLen); >> GEOSFree(w); >> - >> + >> return geomWKB; >> } >> >> @@ -1944,13 +1944,13 @@ str wkbMakePolygon(wkb** out, wkb** exte >> } >> >> externalGeometry = wkb2geos(*external); >> - if ((GEOSGeomTypeId(externalGeometry)+1) != wkbLineString) { >> + if ((GEOSGeomTypeId(externalGeometry)+1) != wkbLinearRing) { >> *out = wkb_nil; >> GEOSGeom_destroy(externalGeometry); >> - throw(MAL, "geom.Polygon", "Geometries should be LineStrings"); >> + throw(MAL, "geom.Polygon", "Geometries should be LinearRings"); >> } >> >> - if(internalBAT_id == NULL) { >> + if(*internalBAT_id == 0) { >> geosGeometry = GEOSGeom_createPolygon(externalGeometry, NULL, 0); >> if(geosGeometry == NULL) { >> GEOSGeom_destroy(externalGeometry); >> diff --git a/geom/monetdb5/geom.mal b/geom/monetdb5/geom.mal >> --- a/geom/monetdb5/geom.mal >> +++ b/geom/monetdb5/geom.mal >> @@ -289,14 +289,14 @@ function MakeEnvelope(xmin:dbl, ymin:dbl >> return x; >> end MakeEnvelope; >> >> -#function MakePolygon(external:wkb) :wkb; >> -# x := Polygon(external, NULL, 0); >> -# return x; >> -#end MakePolygon; >> -#function MakePolygon(external:wkb, srid:int) :wkb; >> -# x := Polygon(external, NULL, srid); >> -# return x; >> -#end MakePolygon; >> +function MakePolygon(external:wkb) :wkb; >> + x := Polygon(external, nil:bat, 0); >> + return x; >> +end MakePolygon; >> +function MakePolygon(external:wkb, srid:int) :wkb; >> + x := Polygon(external, nil:bat, srid); >> + return x; >> +end MakePolygon; >> #function MakePolygon(external:wkb, internal:bat[:oid,:wkb]) :wkb; >> # x := Polygon(external, internal, 0); >> # return x; >> diff --git a/geom/sql/40_geom.sql b/geom/sql/40_geom.sql >> --- a/geom/sql/40_geom.sql >> +++ b/geom/sql/40_geom.sql >> @@ -4070,9 +4070,9 @@ CREATE FUNCTION ST_MakePointM(x double, >> --CREATE FUNCTION ST_LineFromMultiPoint(pointGeom Geometry) RETURNS Geometry >> external name geom."LineFromMultiPoint"; --gets mutlipoint returns >> linestring >> CREATE FUNCTION ST_MakeEnvelope(xmin double, ymin double, xmax double, ymax >> double, srid integer) RETURNS Geometry external name geom."MakeEnvelope"; >> CREATE FUNCTION ST_MakeEnvelope(xmin double, ymin double, xmax double, ymax >> double) RETURNS Geometry external name geom."MakeEnvelope"; >> ---CREATE FUNCTION ST_MakePolygon(geom Geometry) RETURNS Geometry external >> name geom."MakePolygon"; --gets linestring >> ---CREATE FUNCTION ST_MakePolygon(outerGeom Geometry, interiorGeoms >> Geometry[]) RETURNS Geometry external name geom."MakePolygon"; --gets >> linestrings >> ---CREATE FUNCTION ST_Polygon(geom Geometry, srid integer) RETURNS Geometry >> external name geom."MakePolygon"; --gets linestring >> +CREATE FUNCTION ST_MakePolygon(geom Geometry) RETURNS Geometry external name >> geom."MakePolygon"; --gets linestring >> +--CREATE FUNCTION ST_MakePolygon(outerGeom Geometry, interiorGeoms table(g >> Geometry)) RETURNS Geometry external name geom."MakePolygon"; --gets >> linestrings >> +CREATE FUNCTION ST_Polygon(geom Geometry, srid integer) RETURNS Geometry >> external name geom."MakePolygon"; --gets linestring >> CREATE FUNCTION ST_MakeBox2D(lowLeftPointGeom Geometry, upRightPointGeom >> Geometry) RETURNS mbr external name geom."MakeBox2D"; --gets 2d points >> --CREATE FUNCTION ST_3DMakeBox(lowLeftPointGeom Geometry, upRightPointGeom >> Geometry) RETURNS mbr external name geom."MakeBox3D"; --gets 3d points >> >> _______________________________________________ >> checkin-list mailing list >> checkin-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/checkin-list >>

1 0

Re: MonetDB: geo - MakePolygon/Polygon implmentation. Works only for...
by Stefan Manegold 13 Aug '14

13 Aug '14

Thanks! purely out of curiosity: - are there checks & warnings/ error messages in place for not (trying) to use it for polygons with holes, as well as a (configure) check whether the geos library provides the required functionality? - are the changes to the geos module documented (at least as "diff" with reference to which code version it applies), or have these changes even been send "up-stream" to have them by default included in a future version of the library? Thanks! Stefan ----- Original Message ----- > Changeset: 4d4c6ce1a817 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4d4c6ce1a817 > Modified Files: > geom/monetdb5/geom.c > geom/monetdb5/geom.mal > geom/sql/40_geom.sql > Branch: geo > Log Message: > > MakePolygon/Polygon implmentation. Works only for polygons without holes. > Changes to geos library were necessary so, it will not work with standard > geos library. > > > diffs (80 lines): > > diff --git a/geom/monetdb5/geom.c b/geom/monetdb5/geom.c > --- a/geom/monetdb5/geom.c > +++ b/geom/monetdb5/geom.c > @@ -600,7 +600,7 @@ str geom_2_geom(wkb** resWKB, wkb **valu > int valueType = 0; > > int valueSRID = (*valueWKB)->srid; > -//fprintf(stderr, "in geom_2_geom\n"); > + > /* get the geosGeometry from the wkb */ > geosGeometry = wkb2geos(*valueWKB); > /* get the number of coordinates the geometry has */ > @@ -784,7 +784,7 @@ wkb* geos2wkb(const GEOSGeometry* geosGe > geomWKB->srid = GEOSGetSRID(geosGeometry); > memcpy(&geomWKB->data, w, wkbLen); > GEOSFree(w); > - > + > return geomWKB; > } > > @@ -1944,13 +1944,13 @@ str wkbMakePolygon(wkb** out, wkb** exte > } > > externalGeometry = wkb2geos(*external); > - if ((GEOSGeomTypeId(externalGeometry)+1) != wkbLineString) { > + if ((GEOSGeomTypeId(externalGeometry)+1) != wkbLinearRing) { > *out = wkb_nil; > GEOSGeom_destroy(externalGeometry); > - throw(MAL, "geom.Polygon", "Geometries should be LineStrings"); > + throw(MAL, "geom.Polygon", "Geometries should be LinearRings"); > } > > - if(internalBAT_id == NULL) { > + if(*internalBAT_id == 0) { > geosGeometry = GEOSGeom_createPolygon(externalGeometry, NULL, 0); > if(geosGeometry == NULL) { > GEOSGeom_destroy(externalGeometry); > diff --git a/geom/monetdb5/geom.mal b/geom/monetdb5/geom.mal > --- a/geom/monetdb5/geom.mal > +++ b/geom/monetdb5/geom.mal > @@ -289,14 +289,14 @@ function MakeEnvelope(xmin:dbl, ymin:dbl > return x; > end MakeEnvelope; > > -#function MakePolygon(external:wkb) :wkb; > -# x := Polygon(external, NULL, 0); > -# return x; > -#end MakePolygon; > -#function MakePolygon(external:wkb, srid:int) :wkb; > -# x := Polygon(external, NULL, srid); > -# return x; > -#end MakePolygon; > +function MakePolygon(external:wkb) :wkb; > + x := Polygon(external, nil:bat, 0); > + return x; > +end MakePolygon; > +function MakePolygon(external:wkb, srid:int) :wkb; > + x := Polygon(external, nil:bat, srid); > + return x; > +end MakePolygon; > #function MakePolygon(external:wkb, internal:bat[:oid,:wkb]) :wkb; > # x := Polygon(external, internal, 0); > # return x; > diff --git a/geom/sql/40_geom.sql b/geom/sql/40_geom.sql > --- a/geom/sql/40_geom.sql > +++ b/geom/sql/40_geom.sql > @@ -4070,9 +4070,9 @@ CREATE FUNCTION ST_MakePointM(x double, > --CREATE FUNCTION ST_LineFromMultiPoint(pointGeom Geometry) RETURNS Geometry > external name geom."LineFromMultiPoint"; --gets mutlipoint returns > linestring > CREATE FUNCTION ST_MakeEnvelope(xmin double, ymin double, xmax double, ymax > double, srid integer) RETURNS Geometry external name geom."MakeEnvelope"; > CREATE FUNCTION ST_MakeEnvelope(xmin double, ymin double, xmax double, ymax > double) RETURNS Geometry external name geom."MakeEnvelope"; > ---CREATE FUNCTION ST_MakePolygon(geom Geometry) RETURNS Geometry external > name geom."MakePolygon"; --gets linestring > ---CREATE FUNCTION ST_MakePolygon(outerGeom Geometry, interiorGeoms > Geometry[]) RETURNS Geometry external name geom."MakePolygon"; --gets > linestrings > ---CREATE FUNCTION ST_Polygon(geom Geometry, srid integer) RETURNS Geometry > external name geom."MakePolygon"; --gets linestring > +CREATE FUNCTION ST_MakePolygon(geom Geometry) RETURNS Geometry external name > geom."MakePolygon"; --gets linestring > +--CREATE FUNCTION ST_MakePolygon(outerGeom Geometry, interiorGeoms table(g > Geometry)) RETURNS Geometry external name geom."MakePolygon"; --gets > linestrings > +CREATE FUNCTION ST_Polygon(geom Geometry, srid integer) RETURNS Geometry > external name geom."MakePolygon"; --gets linestring > CREATE FUNCTION ST_MakeBox2D(lowLeftPointGeom Geometry, upRightPointGeom > Geometry) RETURNS mbr external name geom."MakeBox2D"; --gets 2d points > --CREATE FUNCTION ST_3DMakeBox(lowLeftPointGeom Geometry, upRightPointGeom > Geometry) RETURNS mbr external name geom."MakeBox3D"; --gets 3d points > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

editing monetDB source code
by Hina Khan 12 Aug '14

12 Aug '14

Hi All, I am a PHD student and my research is related to data exploration. In my recent work I have proposed a new query processing technique using columnar data storage layer. For experiments I need to implement a new operator that works in between two "Select" operators. I wanted to know if there is any way that I can implement this new operator and add it to the source code of MonetDB and change its "Select" operator so that it calls the new operator I have inserted. So in short my question is " Is there any way to edit the source code of MonetDB". I would highly appreciate your help in this regard. Cheers, Hina

2 1