- developers-list - monetdb.org

Re: MonetDB: default - remove jsonstore code we do not need for the ...
by Stefan Manegold 21 Apr '15

21 Apr '15

Arjen, are these left-overs occurrences of "jsonstore" in the default branch still used / required, or can they also be removed? ======== .../default/source/MonetDB $ hg locate | grep --color -i jsonstore sql/scripts/41_jsonstore.sql .../default/source/MonetDB $ hg locate -X \**/Tests/*.stable.* | xargs grep -l -i jsonstore MonetDB.spec NT/installer32/MonetDB5-SQL-Installer.vdproj NT/installer64/MonetDB5-SQL-Installer.vdproj NT/monetdb_config.h.in NT/rules.msc clients/Tests/All monetdb5/mal/mal.c sql/backends/monet5/sql_upgrades.c sql/scripts/41_jsonstore.sql sql/scripts/Makefile.ag testing/Mtest.py.in ======== Thanks! Stefan ----- On Apr 21, 2015, at 12:37 PM, Sjoerd Mullender commits(a)monetdb.org wrote: > Changeset: e6f13746b75b for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e6f13746b75b > Removed Files: > sql/backends/monet5/rest/73_jsonstore.mal > sql/backends/monet5/rest/Makefile.ag > sql/backends/monet5/rest/Tests/All > sql/backends/monet5/rest/Tests/jsonstore00.sql > sql/backends/monet5/rest/Tests/jsonstore00.stable.err > sql/backends/monet5/rest/Tests/jsonstore00.stable.out > sql/backends/monet5/rest/Tests/jsonstore01.sql > sql/backends/monet5/rest/Tests/jsonstore01.stable.err > sql/backends/monet5/rest/Tests/jsonstore01.stable.out > sql/backends/monet5/rest/jsonstore.mal > sql/backends/monet5/rest/rest_jsonstore.c > sql/backends/monet5/rest/rest_jsonstore.h > sql/backends/monet5/rest/rest_jsonstore_handle_get.c > sql/backends/monet5/rest/rest_jsonstore_handle_get.h > Modified Files: > configure.ag > monetdb5/mal/Makefile.ag > monetdb5/mal/mal_http_daemon.c > sql/backends/monet5/Makefile.ag > Branch: default > Log Message: > > remove jsonstore code we do not need for the next feature release > > > diffs (truncated from 2238 to 300 lines): > > diff --git a/configure.ag b/configure.ag > --- a/configure.ag > +++ b/configure.ag > @@ -188,30 +188,13 @@ no-auto) > ;; > esac > > -dft_jsonstore=no > -AC_ARG_ENABLE(jsonstore, > - AS_HELP_STRING([--enable-jsonstore], > - [enable support for MonetDB/jsonstore (default=no)]), > - enable_jsonstore=$enableval, > - enable_jsonstore=$dft_jsonstore) > - > -dft_microhttpd=$enable_jsonstore > +dft_microhttpd=no > AC_ARG_ENABLE(microhttpd, > AS_HELP_STRING([--enable-microhttpd], > - [enable support for the builtin MircoHTTPD daemon (default: implied by > --enable-jsonstore)]), > + [enable support for the builtin MircoHTTPD daemon]), > enable_microhttpd=$enableval, > enable_microhttpd=$dft_microhttpd) > > -case "$enable_jsonstore-$enable_microhttpd" in > -yes-no) > - AC_MSG_ERROR([jsonstore require microhttpd]) > - ;; > -auto-no) > - enable_jsonstore=no > - disable_jsonstore="(jsonstore requires microhttpd)" > - ;; > -esac > - > dft_gsl=auto > AC_ARG_ENABLE(gsl, > AS_HELP_STRING([--enable-gsl], > @@ -2508,7 +2491,7 @@ if test x"$have_atomic_ops" = x"yes" ; t > fi > > dnl MicroHTTPD library > -dnl required for MonetDB5 SQL jsonstore, optional otherwise > +dnl optional > org_have_libmicrohttpd=auto > have_libmicrohttpd=$org_have_libmicrohttpd > if test "x$enable_microhttpd" != xno; then > @@ -2526,24 +2509,6 @@ if test x$enable_microhttpd != xno; then > AC_DEFINE(HAVE_MICROHTTPD, 1, [Define if the microhttpd server is to be > enabled]) > fi > > -org_have_liburiparser=auto > -have_liburiparser=$org_have_liburiparser > -if test "x$enable_jsonstore" != xno; then > - dnl Uriparser library > - dnl required for MonetDB5 SQL jsonstore, optional otherwise > - PKG_CHECK_MODULES([liburiparser], [liburiparser], > - [have_liburiparser=yes], > - [have_liburiparser=no; disable_jsonstore="(liburiparser not found)"; if test > "x$enable_jsonstore" = xyes; then AC_MSG_ERROR([LibUriParser library not found > but required for jsonstore]); fi]) > - > - if test "$have_libmicrohttpd-$have_liburiparser" = yes-yes; then > - AC_DEFINE(HAVE_JSONSTORE, 1, [Define that you want to build the jsonstore > library]) > - elif test "x$enable_jsonstore" = xyes; then > - AC_MSG_ERROR([cannot enable jsonstore > $why_have_libmicrohttpd$why_have_liburiparser]) > - else > - enable_jsonstore=no > - fi > -fi > - > dnl Check for gsl library for statistical functions > org_have_gsl=no > have_gsl=$org_have_gsl > @@ -2920,7 +2885,6 @@ AM_CONDITIONAL(HAVE_GDK, test "x$enable_ > AM_CONDITIONAL(HAVE_MONETDB5, test "x$enable_monetdb5" = xyes -o > "x$enable_monetdb5" = xauto) > AM_CONDITIONAL(HAVE_SQL, test "x$enable_sql" = xyes -o "x$enable_sql" = xauto) > AM_CONDITIONAL(HAVE_GEOM, test "x$enable_geom" = xyes -o "x$enable_geom" = > xauto) > -AM_CONDITIONAL(HAVE_JSONSTORE, test "x$enable_jsonstore" = xyes -o > "x$enable_jsonstore" = xauto) > AM_CONDITIONAL(HAVE_TESTING, test "x$enable_testing" = xyes -o > "x$enable_testing" = xauto) > AM_CONDITIONAL(HAVE_DEVELOPER, test "x$enable_developer" = xyes) > > @@ -3277,7 +3241,6 @@ for comp in \ > 'console ' \ > 'gsl ' \ > 'fits ' \ > - 'jsonstore ' \ > 'microhttpd ' \ > 'rintegration' \ > 'odbc ' \ > diff --git a/monetdb5/mal/Makefile.ag b/monetdb5/mal/Makefile.ag > --- a/monetdb5/mal/Makefile.ag > +++ b/monetdb5/mal/Makefile.ag > @@ -9,7 +9,6 @@ INCLUDES = ../../common/options \ > ../../common/utils \ > ../../clients/mapilib \ > ../../gdk \ > - #../../sql/backends/monet5/rest \ > $(READLINE_INCS) \ > $(libmicrohttpd_CFLAGS) > MTSAFE > diff --git a/monetdb5/mal/mal_http_daemon.c b/monetdb5/mal/mal_http_daemon.c > --- a/monetdb5/mal/mal_http_daemon.c > +++ b/monetdb5/mal/mal_http_daemon.c > @@ -12,262 +12,6 @@ > > #include "monetdb_config.h" > > -#ifdef HAVE_MICROHTTPD > -#include <sys/types.h> > -#include <sys/select.h> > -#include <sys/socket.h> > -#include <stdint.h> > -#include <stdarg.h> > -#include <stdio.h> > - > -#include "mal_client.h" > -#include "mal_http_daemon.h" > -#include "mal_private.h" > -#include <microhttpd.h> > - > -static MT_Id hdthread; > -static int volatile hdrunning; > -struct MHD_Daemon *http_daemon; > -#define PORT 8998 > -#define POSTBUFFERSIZE 512 > -#define MAXNAMESIZE 512 > -#define MAXANSWERSIZE 512 > - > -#define GET 0 > -#define POST 1 > - > -static > -http_request_handler http_handler; > - > -void register_http_handler(http_request_handler handler) { > - http_handler = handler; > -} > - > -struct connection_info_struct > -{ > - int connectiontype; > - char *answerstring; > - struct MHD_PostProcessor *postprocessor; > -}; > - > -static int > -send_page (struct MHD_Connection *connection, const char * url, > - const char * method, char *page, char * postdata) > -{ > - int ret; > - int rest; > - struct MHD_Response *response; > - > - rest = (*http_handler)(url, method, &page, postdata); > - (void)rest; > - response = > - MHD_create_response_from_buffer (strlen (page), > - (void *) page, > - MHD_RESPMEM_MUST_COPY); > - if (!response) > - return MHD_NO; > - > - ret = MHD_queue_response (connection, MHD_HTTP_OK, response); > - MHD_destroy_response (response); > - > - return ret; > -} > - > -static int > -iterate_post (void *coninfo_cls, enum MHD_ValueKind kind, const char *key, > - const char *filename, const char *content_type, > - const char *transfer_encoding, const char *data, uint64_t off, > - size_t size) > -{ > - struct connection_info_struct *con_info = coninfo_cls; > - char *answerstring; > - > - (void)key; > - (void)kind; > - (void)filename; > - (void)content_type; > - (void)transfer_encoding; > - (void)off; > - > - if (strcmp (key, "json") == 0) > - { > - if ((size > 0) && (size <= MAXNAMESIZE)) > - { > - answerstring = malloc (MAXANSWERSIZE); > - if (!answerstring) > - return MHD_NO; > - > - snprintf (answerstring, MAXANSWERSIZE, "%s", data); > - con_info->answerstring = answerstring; > - } else > - con_info->answerstring = NULL; > - > - return MHD_NO; > - } > - > - if (strcmp (key, "file") == 0) > - { > - if ((size > 0) && (size <= MAXNAMESIZE)) > - { > - answerstring = malloc (MAXANSWERSIZE); > - if (!answerstring) > - return MHD_NO; > - > - snprintf (answerstring, MAXANSWERSIZE, "%s", data); > - con_info->answerstring = answerstring; > - } else > - con_info->answerstring = NULL; > - > - return MHD_NO; > - } > - > - return MHD_YES; > -} > - > -static void > -request_completed (void *cls, struct MHD_Connection *connection, > - void **con_cls, enum MHD_RequestTerminationCode toe) > -{ > - struct connection_info_struct *con_info = *con_cls; > - > - (void)cls; > - (void)connection; > - (void)toe; > - > - if (con_info == NULL) > - return; > - > - if (con_info->connectiontype == POST) > - { > - MHD_destroy_post_processor (con_info->postprocessor); > - if (con_info->answerstring) > - free (con_info->answerstring); > - } > - > - free (con_info); > - *con_cls = NULL; > -} > - > -static int > -answer_to_connection (void *cls, struct MHD_Connection *connection, > - const char *url, const char *method, > - const char *version, const char *upload_data, > - size_t *upload_data_size, void **con_cls) > -{ > - char * page = NULL; > - struct connection_info_struct *con_info = NULL; > - int *done = cls; > - char *answerstring = NULL; > - char *errorpage = > - "<html><body>" > - "Failed to handle error in http request." > - "</body></html>"; > - > - (void)version; > - > - if (*con_cls == NULL) { > - con_info = malloc (sizeof (struct connection_info_struct)); > - if (con_info == NULL) > - return MHD_NO; > - con_info->answerstring = NULL; > - > - if (strcmp (method, "POST") == 0) { > - con_info->postprocessor = > - MHD_create_post_processor (connection, > - POSTBUFFERSIZE, > - iterate_post, > - (void *) con_info); > - > - if (con_info->postprocessor == NULL) { > - free (con_info); > - return MHD_NO; > - } > - con_info->connectiontype = POST; > - } else { > - con_info->connectiontype = GET; > - } > - *con_cls = (void *) con_info; > - return MHD_YES; > - } > - > - if (strcmp (method, "POST") == 0) { > - con_info = *con_cls; > - if (*upload_data_size != 0) { > - MHD_post_process (con_info->postprocessor, upload_data, > - *upload_data_size); > - *upload_data_size = 0; > - return MHD_YES; > - } else { > - if (con_info->answerstring != NULL) { > - return send_page(connection, url, method, page, > - con_info->answerstring); > - } > - } > - } > - > - if (strcmp (method, "PUT") == 0) { > - if (*upload_data_size != 0) { > - // TODO: check free answerstring > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

Re: developers-list Digest, Raúl
by José Raúl Pérez Rodríguez 13 Apr '15

13 Apr '15

Thank you very much Hannes, Then I can assume that queries to a distributed table can' t be sorted. It is a serious limitation in MonetDb scaling, I would like to work to help in adding this capability. Raúl > Today's Topics: > > 1. Re: Querying a monetdb cluster (Hannes M?hleisen) > > > Hello Jos? Ra?l, > >> On 09.04.2015, at 12:45, Jos? Ra?l P?rez Rodr?guez >> <joseraul.perezrodriguez(a)gmail.com> wrote: >> >> I maked a test inserting data into each particular node of a two node >> cluster, but I don't see the way of making queries to the cluster, I >> known that multiplex funnel can return data from several nodes, >> joining data obtained from each node, but with multiplex funnel, I >> can't sort the data. > This is not supported yet. > > Best, > > Hannes

2 1

Querying a monetdb cluster
by José Raúl Pérez Rodríguez 10 Apr '15

10 Apr '15

Hi, I am new in monetdb. How do I make a query to a cluster? I maked a test inserting data into each particular node of a two node cluster, but I don't see the way of making queries to the cluster, I known that multiplex funnel can return data from several nodes, joining data obtained from each node, but with multiplex funnel, I can't sort the data. There is not doc that explains this problem?

2 1

MonetDB, data loss in restart server
by Luciano Sasso 02 Apr '15

02 Apr '15

Hello, I have used the MonetDB in production for some time, and I have nothing to complain about, are tables with more than 30 columns and 30 million records, do analysis through cubes in Pentaho / Saiku. Have restarted the server several times, and never had problems with loss of data.But today, after performing a query that was consuming a lot of machine, stopped the database, and when I started, data from some columns were blank. How can it, will the data were still uncommitted? when performing a query the MonetDB changes the data or reorganized? which may have occurred, I am concerned. -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

2 1

Re: MonetDB: default - Defense against non-existing MAL primitives
by Stefan Manegold 20 Mar '15

20 Mar '15

FYI, test clients/R/Tests/survey.R still fails with "Server says '!TypeException:user.s94_2[40]:'batcalc.*' undefined in: X_168:bat[:oid,:hge] := batcalc.*(X_146,X_166);'." i.e., the check whether the requested new signature exists does not seem to work, yet. cf., http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=55075:ca15cab457b8&or… Stefan ----- Original Message ----- > Changeset: 4165dc34b2bb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4165dc34b2bb > Modified Files: > monetdb5/mal/mal_resolve.c > monetdb5/mal/mal_resolve.h > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > Defense against non-existing MAL primitives > and further limit the rewrite to the simpliest > and most expensive case encountered. > > > diffs (244 lines): > > diff --git a/monetdb5/mal/mal_resolve.c b/monetdb5/mal/mal_resolve.c > --- a/monetdb5/mal/mal_resolve.c > +++ b/monetdb5/mal/mal_resolve.c > @@ -115,7 +115,6 @@ findFunctionType(stream *out, Module sco > Symbol s; > InstrPtr sig; > int i, k, unmatched = 0, s1; > - /* int foundbutwrong=0; */ > int polytype[MAXTYPEVAR]; > int returns[256]; > int *returntype = NULL; > @@ -396,7 +395,6 @@ findFunctionType(stream *out, Module sco > } > } > if (s1 < 0) { > - /* if(getSignature(s)->token !=PATTERNsymbol) foundbutwrong++; */ > s = s->peer; > continue; > } > @@ -445,7 +443,6 @@ findFunctionType(stream *out, Module sco > * typed. This should be reflected in the symbol table. > */ > s1 = returntype[0]; /* for those interested */ > - /* foundbutwrong = 0; */ > /* > * If the call refers to a polymorphic function, we clone it > * to arrive at a bounded instance. Polymorphic patterns and > @@ -485,12 +482,6 @@ findFunctionType(stream *out, Module sco > * arguments, but that clashes with one of the target variables. > */ > wrapup: > - /* foundbutwrong has not been changed, commented out code above > - if (foundbutwrong && !silent) { > - showScriptException(out, mb, getPC(mb, p), TYPE, > - "type conflict in assignment"); > - } > - */ > if (returntype && returntype != returns) > GDKfree(returntype); > return -3; > @@ -819,13 +810,17 @@ chkTypes(stream *out, Module s, MalBlkPt > > /* > * Type checking an individual instruction is dangerous, > - * because it ignores data flow and declarations issues. > - * It is only to be used in isolated cases. > + * because it ignores data flow and variable declarations. > */ > -void > +int > chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr p) > { > - typeChecker(out, s, mb, p, FALSE); > + int olderrors= mb->errors; > + int error; > + typeChecker(out, s, mb, p, TRUE); > + error = mb->errors; > + mb->errors = olderrors; > + return error; > } > > void > diff --git a/monetdb5/mal/mal_resolve.h b/monetdb5/mal/mal_resolve.h > --- a/monetdb5/mal/mal_resolve.h > +++ b/monetdb5/mal/mal_resolve.h > @@ -20,7 +20,7 @@ > #define MAXTYPEVAR 10 > > mal_export void chkProgram(stream *out, Module s, MalBlkPtr mb); > -mal_export void chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr > p); > +mal_export int chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr > p); > mal_export void chkTypes(stream *out, Module s, MalBlkPtr mb, int silent); > mal_export void typeChecker(stream *out, Module scope, MalBlkPtr mb, > InstrPtr p, int silent); > mal_export int fcnBinder(stream *out, Module scope, MalBlkPtr mb, InstrPtr > p); > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -18,8 +18,9 @@ typedef struct{ > int fromtype; > int totype; > int src; > -/* not used, yet !?? > +/* not used, yet !?? Indeed > int digits; > + int fromscale; > int scale; > */ > } Coercion; > @@ -47,58 +48,60 @@ coercionOptimizerStep(MalBlkPtr mb, int > return 0; > } > > -/* Check coercions for numeric types that can be handled with smaller ones. > +/* Check coercions for numeric types towards :hge that can be handled with > smaller ones. > * For now, limit to +,-,/,*,% hge expressions > - * To be extended to deal with math calls as well. > + * Not every combination may be available in the MAL layer, which calls > + * for a separate type check before fixing it. > + * Superflous coercion statements will be garbagecollected later on in the > pipeline > */ > static void > -coercionOptimizerCalcStep(MalBlkPtr mb, int i, Coercion *coerce) > +coercionOptimizerCalcStep(Client cntxt, MalBlkPtr mb, int i, Coercion > *coerce) > { > InstrPtr p = getInstrPtr(mb,i); > - int r, a, b; > + int r, a, b, varid; > > + r = getColumnType(getVarType(mb, getArg(p,0))); > +#ifdef HAVE_HGE > + if ( r != TYPE_hge) > + return; > +#endif > if( getModuleId(p) != batcalcRef || getFunctionId(p) == 0) return; > if( ! (getFunctionId(p) == plusRef || getFunctionId(p) == minusRef || > getFunctionId(p) == mulRef || getFunctionId(p) == divRef || > *getFunctionId(p) =='%') || p->argc !=3) > return; > > - r = getColumnType(getVarType(mb, getArg(p,0))); > - switch(r){ > - case TYPE_bte: > - case TYPE_sht: > - case TYPE_int: > - case TYPE_lng: > -#ifdef HAVE_HGE > - case TYPE_hge: > -#endif > - break; > - case TYPE_dbl: > - case TYPE_flt: > - /* to be determined */ > - default: > - return; > - } > a = getColumnType(getVarType(mb, getArg(p,1))); > b = getColumnType(getVarType(mb, getArg(p,2))); > - if ( a == r && coerce[getArg(p,1)].src && coerce[getArg(p,1)].fromtype < r > ) /*digit/scale test as well*/ > + if ( a == r && coerce[getArg(p,1)].src && coerce[getArg(p,1)].fromtype < r > ) > + { > + varid = getArg(p,1); > getArg(p,1) = coerce[getArg(p,1)].src; > - if ( b == r && coerce[getArg(p,2)].src && coerce[getArg(p,2)].fromtype < r > ) /*digit/scale test as well*/ > + if ( chkInstruction(NULL, cntxt->nspace, mb, p)) > + p->argv[1] = varid; > + } > + if ( b == r && coerce[getArg(p,2)].src && coerce[getArg(p,2)].fromtype < r > ) > + { > + varid = getArg(p,2); > getArg(p,2) = coerce[getArg(p,2)].src; > + if ( chkInstruction(NULL, cntxt->nspace, mb, p)) > + p->argv[2] = varid; > + } > return; > } > > static void > -coercionOptimizerAggrStep(MalBlkPtr mb, int i, Coercion *coerce) > +coercionOptimizerAggrStep(Client cntxt, MalBlkPtr mb, int i, Coercion > *coerce) > { > InstrPtr p = getInstrPtr(mb,i); > int r, k; > > + (void) cntxt; > + > if( getModuleId(p) != aggrRef || getFunctionId(p) == 0) return; > if( ! (getFunctionId(p) == subavgRef ) || p->argc !=6) > return; > > r = getColumnType(getVarType(mb, getArg(p,0))); > k = getArg(p,1); > - // check the digits/scale > if( r == TYPE_dbl && coerce[k].src ) > getArg(p,1) = coerce[getArg(p,1)].src; > return; > @@ -124,35 +127,33 @@ OPTcoercionImplementation(Client cntxt,M > if (getModuleId(p) == NULL) > continue; > /* Downscale the type, avoiding hge storage when lng would be sufficient. > - * The code template can be extended to handle other downscale options as > well > */ > #ifdef HAVE_HGE > if ( getModuleId(p) == batcalcRef > && getFunctionId(p) == hgeRef > && p->retc == 1 > - && ( p->argc == 2 > - || ( p->argc == 5 > - && isVarConstant(mb,getArg(p,1)) > - && getArgType(mb,p,1) == TYPE_int > - && isVarConstant(mb,getArg(p,3)) > - && getArgType(mb,p,3) == TYPE_int > - && isVarConstant(mb,getArg(p,4)) > - && getArgType(mb,p,4) == TYPE_int > - /* from-scale == to-scale, i.e., no scale change */ > - && *(int*) getVarValue(mb, getArg(p,1)) == *(int*) > getVarValue(mb, getArg(p,4)) ) ) ){ > + && ( p->argc == 5 > + && isVarConstant(mb,getArg(p,1)) > + && getArgType(mb,p,1) == TYPE_int > + && isVarConstant(mb,getArg(p,3)) > + && getArgType(mb,p,3) == TYPE_int > + && isVarConstant(mb,getArg(p,4)) > + && getArgType(mb,p,4) == TYPE_int > + /* from-scale == to-scale, i.e., no scale change */ > + && *(int*) getVarValue(mb, getArg(p,1)) == *(int*) getVarValue(mb, > getArg(p,4)) ) ){ > k = getArg(p,0); > coerce[k].pc= i; > coerce[k].totype= TYPE_hge; > coerce[k].src= getArg(p,2); > coerce[k].fromtype= getColumnType(getArgType(mb,p,2)); > -/* not used, yet !?? > - if (p->argc == 5) { > - coerce[k].digits= getVarConstant(mb,getArg(p,3)).val.ival; > - coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > - } > +/* not used, yet !?? indeed > + coerce[k].fromscale= getVarConstant(mb,getArg(p,1)).val.ival; > + coerce[k].digits= getVarConstant(mb,getArg(p,3)).val.ival; > + coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > */ > } > #endif > +/* > if ( getModuleId(p) == batcalcRef > && getFunctionId(p) == dblRef > && p->retc == 1 > @@ -160,7 +161,7 @@ OPTcoercionImplementation(Client cntxt,M > || ( p->argc == 3 > && isVarConstant(mb,getArg(p,1)) > && getArgType(mb,p,1) == TYPE_int > - /* to-scale == 0, i.e., no scale change */ > + //to-scale == 0, i.e., no scale change > && *(int*) getVarValue(mb, getArg(p,1)) == 0 ) ) ) { > k = getArg(p,0); > coerce[k].pc= i; > @@ -168,8 +169,9 @@ OPTcoercionImplementation(Client cntxt,M > coerce[k].src= getArg(p,1 + (p->argc ==3)); > coerce[k].fromtype= getColumnType(getArgType(mb,p,1 + (p->argc ==3))); > } > - coercionOptimizerAggrStep(mb, i, coerce); > - coercionOptimizerCalcStep(mb, i, coerce); > +*/ > + coercionOptimizerAggrStep(cntxt,mb, i, coerce); > + coercionOptimizerCalcStep(cntxt,mb, i, coerce); > if (getModuleId(p)==calcRef && p->argc == 2) { > k= coercionOptimizerStep(mb, i, p); > actions += k; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 6

Re: MonetDB: default - Minor textual changes to remote tests
by Stefan Manegold 20 Mar '15

20 Mar '15

> Changeset: ca15cab457b8 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ca15cab457b8 > Modified Files: > monetdb5/modules/mal/Tests/All > monetdb5/modules/mal/Tests/remote07.mal > monetdb5/modules/mal/Tests/remote88.mal > Branch: default > Log Message: > > Minor textual changes to remote tests > > > diffs (104 lines): > > diff --git a/monetdb5/modules/mal/Tests/All b/monetdb5/modules/mal/Tests/All > --- a/monetdb5/modules/mal/Tests/All > +++ b/monetdb5/modules/mal/Tests/All > @@ -36,6 +36,7 @@ remote02 > HAVE_GEOM?remote03 > remote04 > remote06 > +# should be run as 'mclient -d demo -l mal remote07.mal For what it's worth: MAL tests with extension ".malC" (instead of ".mal") are automatically run trough "mclient -lmal ..." (instead of "mserver5 ...") by Mtest.py; cf., dev.monetdb.org/hg/MonetDB/file/tip/testing/README i.e., `hg rename monetdb5/modules/mal/Tests/remote07.mal monetdb5/modules/mal/Tests/remote07.malC` might be considered ... Stefan > remote08 > remote09 > remote11 [...] -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - Avoid coercion in aggr.subavg
by Stefan Manegold 16 Mar '15

16 Mar '15

According to Mtest, this makes TPCh Q1 return a wrong result. I suspect because this now indeed removes a scale-changing cast. Stefan ----- Original Message ----- > Changeset: 9779801a1482 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9779801a1482 > Modified Files: > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > Avoid coercion in aggr.subavg > Another example where we can avoid coercions in most cases > > > diffs (43 lines): > > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -84,6 +84,24 @@ coercionOptimizerCalcStep(MalBlkPtr mb, > return; > } > > +static void > +coercionOptimizerAggrStep(MalBlkPtr mb, int i, Coercion *coerce) > +{ > + InstrPtr p = getInstrPtr(mb,i); > + int r, k; > + > + if( getModuleId(p) != aggrRef || getFunctionId(p) == 0) return; > + if( ! (getFunctionId(p) == subavgRef ) || p->argc !=6) > + return; > + > + r = getColumnType(getVarType(mb, getArg(p,0))); > + k = getArg(p,1); > + // check the digits/scale > + if( r == TYPE_dbl && coerce[k].src ) > + getArg(p,1) = coerce[getArg(p,1)].src; > + return; > +} > + > int > OPTcoercionImplementation(Client cntxt,MalBlkPtr mb, MalStkPtr stk, InstrPtr > pci) > { > @@ -117,6 +135,14 @@ OPTcoercionImplementation(Client cntxt,M > coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > } > #endif > + if ( getModuleId(p) == batcalcRef && getFunctionId(p) == dblRef && p->retc > == 1 && ((p->argc == 3 && isVarConstant(mb,getArg(p,1))) || p->argc ==2) ){ > + k = getArg(p,0); > + coerce[k].pc= i; > + coerce[k].totype= TYPE_dbl; > + coerce[k].src= getArg(p,1 + (p->argc ==3)); > + coerce[k].fromtype= getColumnType(getArgType(mb,p,1 + (p->argc ==3))); > + } > + coercionOptimizerAggrStep(mb, i, coerce); > coercionOptimizerCalcStep(mb, i, coerce); > if (getModuleId(p)==calcRef && p->argc == 2) { > k= coercionOptimizerStep(mb, i, p); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - coercionOptimizerCalcStep(): fixing changeset...
by Stefan Manegold 16 Mar '15

16 Mar '15

IMHO this new coersion optimization as introduced in changeset 5786f1be12bb only works correctly for decimal casts (coersions) that do not change the scale (number of decimals), i.e., when the first and last argument (i.e., input and output number of decimals, respectively) are identical. This sanity check is currently missing in the code. Stefan ----- Original Message ----- > Changeset: 289a5a038774 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=289a5a038774 > Modified Files: > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > coercionOptimizerCalcStep(): fixing changeset 71bfaf7e9841: also handle > TYPE_hge > to re-activate the intended behavior of changeset 5786f1be12bb > > OPEN ISSUE / QUESTION: > > This optimizer now (only) handles (i.e., removes) decimal (up-)casts, > but no plain/pure integer casts at all. > > (a) Shouldn't we also handle (remove) plain/pure integer casts? > > (b) Can we always safely remove decimal (up-)casts without altering > semantics? > > > diffs (13 lines): > > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -65,6 +65,9 @@ coercionOptimizerCalcStep(MalBlkPtr mb, > case TYPE_sht: > case TYPE_int: > case TYPE_lng: > +#ifdef HAVE_HGE > + case TYPE_hge: > +#endif > break; > case TYPE_dbl: > case TYPE_flt: > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

About the Join Algorithm in MonetDB
by Zhiwei Fan 13 Mar '15

13 Mar '15

Dear Developers of MonetDB, ?I really appreciate your excellent work on MonetDB. I am a student in University of Wisconsin Madison and now I am doing research on processing machine learning in the column-oriented Database since we have found a way that could possibly improve the performance of learning process if we use data stored in column rather than data stored in row. For the research purpose, we need to know the detail of the join algorithm in column-store database in order to analyze the cost of the process. And monetDB, as we know, has done a very good job in this part, so I hope you could tell us about the details of the join algorithms implemented in the monetDB, that would be really helpful and we would really appreciate it! Thanks. Best Regards Zhiwei Fan ?

1 0

MAL property for thread-unsafe function
by Roberto Cornacchia 05 Mar '15

05 Mar '15

Hello, I have a function definition command batstemmer.stem(terms:bat[:oid,:str], stemmer_name:str):bat[:oid,:str] address CMDbatstem comment "Wrapper for snowball stemmer"; which internally uses the snowball stemmer (http://snowball.tartarus.org/) When the bat to be stemmed is large enough, mitosis will split it into chunks and call the function "stem" on each chunk, possibly in parallel. Problem is, the snowball stemmer implementation appears to be thread-unsafe, which causes a SIGSEGV. Indeed, using the no_mitosis_pipe solves the issue. However, this solution is suboptimal. Another solution I found is to mark the mal signature as {unsafe}. This works, although it does something a bit silly: it splits the table into chunks, then repacks everything, and finally runs my function on the re-packed bat (basically wasting effort on a useless split + repack). Now, my question is: is there a more focussed property to use? {unsafe} implies thread-unsafe, but it is actually stronger than that. For example, it also implies that there might be side-effects. Therefore, the result cannot be recycled. In my case, instead, the result is perfectly safe to be reused. Thanks for any tip. Roberto

2 2