- developers-list - monetdb.org

Announcement: New Jul2015-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 11 Apr '16

11 Apr '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP4 bugfix release MonetDB Common * Fixed a bug that caused various instances where old data returned or where crashes occurred. The problem was that internally data wasn't always marked dirty when it was being changed, causing later processing to not deal with the changed data correctly. Bug Fixes * 3905: MonetDB doesn't handle ANY/SOME/ALL operator correctly * 3929: R aggregate not recognized when using 3 or more parameters * 3965: Not possible to quote/escape single quote character in the name of the file to load. * 3968: Missing double use of column names

1 0

undefined reference to `pcre_config'
by John Krulick 06 Apr '16

06 Apr '16

Hi, I'm trying to build MonetDB from source and running into an error during configure. Running ./configure gives: ... ... checking for SHA256... yes checking for SHA384... yes checking for SHA512... yes checking for pcre... yes checking whether pcre comes with UTF-8 support... no configure: error: PCRE library not found but required for MonetDB5 At the bottom of the email is the error in the config.log. pcretest -C gives me: PCRE version 8.38 2015-11-23 Compiled with:q 8-bit support UTF-8 support Unicode properties support No just-in-time compiler support Newline sequence is LF \R matches all Unicode newlines Internal link size = 2 POSIX malloc threshold = 10 Parentheses nest limit = 250 Default match limit = 10000000 Default recursion depth limit = 10000000 Match recursion uses stack The issue, I think, is I have PCRE version 8.38 built in a non-standard location, /home/john/monetdb_prereq/pcre export pcre_LIBS=-L/home/john/monetdb_prereq/pcre/lib export pcre_CFLAGS=-I/home/john/monetdb_prereq/pcre/include Any idea what I'm missing? Thanks, John configure:20861: checking for SHA512 configure:20861: gcc -o conftest -g -O2 conftest.c -lssl -lcrypto -ldl -lz >&5 configure:20861: $? = 0 configure:20861: result: yes configure:20901: checking for pcre configure:20966: result: yes configure:20971: checking whether pcre comes with UTF-8 support configure:20992: gcc -o conftest -g -O2 -I/home/john/monetdb_prereq/pcre/include conftest.c -L/home/john/monetdb_prereq/pcre/lib >&5 /tmp/ccRjfKcQ.o: In function `main': /home/john/MonetDB-11.21.17/conftest.c:66: undefined reference to `pcre_config' collect2: ld returned 1 exit status configure:20992: $? = 1 configure: program exited with status 1 configure: failed program was: | /* confdefs.h */ | #define PACKAGE_NAME "MonetDB" | #define PACKAGE_TARNAME "MonetDB" | #define PACKAGE_VERSION "11.21.17" | #define PACKAGE_STRING "MonetDB 11.21.17" | #define PACKAGE_BUGREPORT "info(a)monetdb.org" | #define PACKAGE_URL "http://www.monetdb.org/" | #define PACKAGE "MonetDB" | #define VERSION "11.21.17" | #define HOST "x86_64-unknown-linux-gnu" | #define MONETDB_RELEASE "Jul2015-SP3" | #define HAVE_CONSOLE 1 | #define MONETDB5_PASSWDHASH "SHA512" | #define STDC_HEADERS 1 | #define HAVE_SYS_TYPES_H 1 | #define HAVE_SYS_STAT_H 1 | #define HAVE_STDLIB_H 1 | #define HAVE_STRING_H 1 | #define HAVE_MEMORY_H 1 | #define HAVE_STRINGS_H 1 | #define HAVE_INTTYPES_H 1 | #define HAVE_STDINT_H 1 | #define HAVE_UNISTD_H 1 | #define __EXTENSIONS__ 1 | #define _ALL_SOURCE 1 | #define _GNU_SOURCE 1 | #define _POSIX_PTHREAD_SEMANTICS 1 | #define _TANDEM_SOURCE 1 | #define SIZEOF_LONG 8 | #define HAVE_DLFCN_H 1 | #define LT_OBJDIR ".libs/" | #define NDEBUG 1 | #define DIFF "diff" | #define DIR_SEP '/' | #define DIR_SEP_STR "/" | #define PATH_SEP ':' | #define PATH_SEP_STR ":" | #define SO_PREFIX "lib" | #define SO_EXT ".so" | #define HAVE_LIBBZ2 1 | #define HAVE_MALLOPT 1 | #define HAVE_PTHREAD_H 1 | #define HAVE_SEMAPHORE_H 1 | #define HAVE_SCHED_H 1 | #define HAVE_PTHREAD_KILL 1 | #define HAVE_PTHREAD_SIGMASK 1 | #define HAVE_PTHREAD_SETSCHEDPRIO 1 | #define HAVE_LIBPTHREAD 1 | #define HAVE_LIBREADLINE 1 | #define HAVE_ICONV 1 | #define ICONV_CONST | #define HAVE_UUID 1 | #define HAVE_MD5 1 | #define HAVE_RIPEMD160 1 | #define HAVE_SHA1 1 | #define HAVE_SHA224 1 | #define HAVE_SHA256 1 | #define HAVE_SHA384 1 | #define HAVE_SHA512 1 | #define HAVE_OPENSSL 1 | /* end confdefs.h. */ | #include <pcre.h> | int | main () | { | int rc; pcre_config(PCRE_CONFIG_UTF8, &rc); exit(!rc); | ; | return 0; | } configure:20996: result: no configure:21021: error: PCRE library not found but required for MonetDB5

2 2

Announcement: New Jul2015-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 25 Mar '16

25 Mar '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP3 bugfix release MonetDB5 Server * Fixed potential crash in MAL debugger when accessing BATs by index. Functionality dropped as it is also a security leak. Bug Fixes * 2972: SQL URL functionality contains errors * 3549: bulk string operations very slow * 3881: Server crashes on bulk load * 3890: Window function + group by in subselect, rel2bin_project: Assertion `0' failed * 3891: MonetDB crashes when executing SQL with window function * 3900: null handling in some sql statements is incorrect * 3906: Multi-column 1-N table-function with mitosis produces different column counts * 3908: LEFT JOIN with OR conditions triggers assertion * 3909: Incorrect column name in OR condition of LEFT JOIN crashes mserver * 3910: COPY INTO table (column1, column2) got wrong result * 3912: When table/column names conflicts, data ends in multiple tables! * 3917: Date difference returns month_interval instead of day_interval * 3918: MonetDB.R version 1.0.1 incorrectly constructs the batfile script * 3919: Table conflict when the table name and fields are identical * 3921: Creating a table from a complex query crashes mserver or triggers assertion * 3922: AVG( column ) returns NaN rather than Inf when column contains Inf * 3928: When killing a virtual machine, sql_logs/sql/log is empty * 3930: Wrong typecast on character columns in prepared statements when using Umlaute * 3932: CASE expressions with constants are not evaluated correctly * 3933: replace "exit" by "throw new Exception" * 3937: bad BAT properties with binary copy into and NULL values * 3938: Wrong error message on violating foreign key constraint * 3940: Date calculation and comparison produce wrong result * 3941: Wrong coercion priority * 3948: SQL: select * from sys.sys.table_name; is accepted but should return an error * 3951: extern table_funcs not visible from Windows DLL for extensions like vaults (crashes) * 3952: Stream table gives segfault * 3953: MIN/MAX of a UUID column produces wrong results * 3954: Consolidate table assertion error * 3955: (incorrect) MAL loop instead of manifold triggered by simple change in target list

1 0

histogram
by Roberto Cornacchia 21 Mar '16

21 Mar '16

In a C UDF, looping on a [:oid,:str:] bat, I'm tokenizing each string tail into a str array: BATloop(..) { str *token_array = /* create a str array by tokenizing the str tail of this BUN */ /* append a histogram of token_array to result */ } The token array is expected to be in the range of 10 - 10K short strings (1 to 10 bytes). Question: To get the histogram, I'd turn the token_array into a BAT b, then use BATgroup: BATgroup(&gn, NULL, &hn, b, NULL, NULL, NULL); Do you see a more efficient way? Is there a group/histogram primitive implemented that works directly on arrays rather than BATs? Thanks, Roberto

1 0

Error after compiling monetdb
by Shmagi Kavtaradze 19 Mar '16

19 Mar '16

I am trying to write a C User Defined Function. It works in Neatbeans. What I want is, to take value from cells, insert them in text or char array and compare each corresponding array position to each other. As values I will have "1"s and "0s". E.g "101010100011". THe code I wrote in udf.c: /* Comparing bag of words */ char BitAnd(char Cell1[], char Cell2[]) { Cell1[10000]; Cell2[10000]; int counter; float result; int strlen = sizeof(Cell1)/sizeof(Cell1[0])-1; int i; for (i=0; i < strlen; i++) { if (Cell1[i] == Cell2[i]) { counter++; } } result = (float)counter / strlen; } /* MAL wrapper */ char UDFBitAnd(char Cell1[], char Cell2[]) { /* assert calling sanity */ assert(Cell1[] != NULL && Cell2[] != NULL); return MAL_SUCCEED; } In udf.mal I have: command BitAnd(Cell1:chr,Cell2:chr):flt address UDFBitAnd comment "Compare bag of words"; In 80_udf.sql: -- Comparing bag of words create function BitAnd(Cell1[] char, Cell2[] char) returns float external name udf.BitAnd; I compiled Monetdb, but when I try to connect with "monetdb" to db it gives me error. In the log file I found: MSG voc[18857]: !SyntaxException:parseError:command BitAnd(Cell1:chr,Cell2:chr):flt MSG voc[18857]: !SyntaxException:parseError: ^Type identifier expected MSG voc[18857]: !SyntaxException:parseError:command BitAnd(Cell1:chr,Cell2:chr):flt MSG voc[18857]: !SyntaxException:parseError: ^',' expected MSG merovingian[8780]: database 'voc' (18857) has exited with exit status 0 MSG merovingian[8780]: database 'voc' has shut down ERR merovingian[8780]: client error: database 'voc' started up, but failed to open up a communication channel

1 0

Adapting BATs
by Till Kolditz 18 Mar '16

18 Mar '16

Hi there, I'm just new to the list, so hello to everyone :-) I'm currently trying to find out how to adapt BATs in MonetDB for a scientific project. What I want to do is to add an encoding to data as some kind of a native DB feature. I.e. that a BAT stores data in some encoded fashion to detect certain kinds of errors that alter data in memory. By that, talking e.g. about an int BAT (column), the data width would increase e.g. from 4 to 5 Bytes (just as an example) where the additional byte stores the coding information. As I see it there are for a first shot two possibilities to do that: 1) Take one atom (e.g. int) and adapt all BAT / atom access code which touches int BATs. 2) Add some kind of new atom which is handled in a specific way. Since I have just started looking into MonetDB I now hope for a few hints to decide whether one of these approaches is sensible and whether there is maybe a simpler way. Thanks a lot in advance for any helpful comments :-) Best Till Kolditz -- Dipl.-Inf. Till Kolditz wiss. Mitarbeiter Technische Universität Dresden Fakultät Informatik Lehrstuhl Datenbanken 01062 Dresden Tel.: +49 (351) 463-38077 Fax: +49 (351) 463-38259 E-Mail: Till.Kolditz(a)tu-dresden.de jExam Group www: http://www.jexam.de Mail: till.kolditz(a)jexam.de

1 0

New Jun2016 stable branch created
by Sjoerd Mullender 14 Mar '16

14 Mar '16

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jun2016. So in each of your stable checkout directories you can do hg pull hg update -rJun2016 to update to this new branch. -- Sjoerd Mullender

1 0

Re: MonetDB: leftmart - trying to fix Windows compilation: do not "i...
by Lefteris 08 Mar '16

08 Mar '16

Stefan, this needs to be inlined because it is used many times. I will deal with compilation issues later on. This is a dev branch and we are running different experiments every night. But with this change if measured times are different I would not know if it is because of other changes or because of the inline statement removal. I would appreciate if you did not make any changes in this branch at the moment. thank you, lefteris On Mon, Mar 7, 2016 at 8:34 PM, Stefan Manegold < commits+Stefan.Manegold=cwi.nl(a)monetdb.org> wrote: > Changeset: ea1aed8beff8 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ea1aed8beff8 > Modified Files: > gdk/gdk_bloom.c > Branch: leftmart > Log Message: > > trying to fix Windows compilation: do not "inline" BLOOMask() > > On Windows, compilation fails with > failed invocation: cl -GF -W3 -WX -MD -nologo -Ox -Zi -Oi -I. -I.\.. > -I.\..\..\gdk -I..\common\options -I.\..\..\gdk\..\common\options > -I..\common\stream -I.\..\..\gdk\..\common\stream -I..\common\utils > -I.\..\..\gdk\..\common\utils -I.\..\..\buildtools\conf -DHAVE_ICONV > -IC:\Libraries\iconv-1.11.1.win64-vs2010\include -DHAVE_LIBATOMIC_OPS > -IC:\Libraries\atomic_ops-7.4.2 -DNDEBUG -LD -Felibbat.dll gdk_batop.obj > gdk_select.obj gdk_search.obj gdk_hash.obj gdk_tm.obj gdk_orderidx.obj > gdk_align.obj gdk_bbp.obj gdk_heap.obj gdk_utils.obj gdk_atoms.obj > gdk_qsort.obj gdk_ssort.obj gdk_storage.obj gdk_bat.obj gdk_delta.obj > gdk_cross.obj gdk_system.obj gdk_value.obj gdk_posix.obj gdk_logger.obj > gdk_sample.obj gdk_calc.obj gdk_aggr.obj gdk_group.obj gdk_imprints.obj > gdk_bloom.obj gdk_join.obj gdk_project.obj gdk_unique.obj gdk_firstn.obj > libbat.res /link ..\common\options\monet_options.obj > ..\common\stream\libstream.lib ..\common\utils\mutils.obj > ..\common\utils\prompt.obj wsock32.lib Ws2_32.l > ib C:\Libraries\zlib-1.2.8.win64-vs2010\lib\zdll.lib > C:\Libraries\bzip2-1.0.6.win64-vs2010\lib\libbz2.lib psapi.lib > Creating library libbat.lib and object libbat.exp > gdk_join.obj : error LNK2019: unresolved external symbol BLOOMask > referenced in function hashjoin > libbat.dll : fatal error LNK1120: 1 unresolved externals > > > Maybe, the fact that BLOOMask() is "inline"d triggers this problem ... > > > diffs (11 lines): > > diff --git a/gdk/gdk_bloom.c b/gdk/gdk_bloom.c > --- a/gdk/gdk_bloom.c > +++ b/gdk/gdk_bloom.c > @@ -240,7 +240,6 @@ BATbloom(BAT *b) > return GDK_SUCCEED; > } > > -inline > int BLOOMask(BUN v, Bloomfilter *bloom) > { > BUN hv,mv,x,y,z; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

3 2

Re: MonetDB: leftmart - silence some compilers compaining about prin...
by Stefan Manegold 07 Mar '16

07 Mar '16

Hi Lefteris, the portable format string that we ussually use for size_t in MonetDB is "SZFMT" Best, Stefan ----- On Mar 7, 2016, at 2:38 PM, lsidir commits+lsidir=cwi.nl(a)monetdb.org wrote: > Changeset: a76605fb0b33 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a76605fb0b33 > Modified Files: > gdk/gdk_bloom.c > Branch: leftmart > Log Message: > > silence some compilers compaining about printing a size_t as unsigned int or > long > > > diffs (12 lines): > > diff --git a/gdk/gdk_bloom.c b/gdk/gdk_bloom.c > --- a/gdk/gdk_bloom.c > +++ b/gdk/gdk_bloom.c > @@ -183,7 +183,7 @@ BATbloom(BAT *b) > > ALGODEBUG fprintf(stderr, "#BATbloom(b=%s#" BUNFMT ") %s: " > "create bloom filter: mbits = " BUNFMT ", ratio = " BUNFMT > - ", kfunc = %d, bytes = " BUNFMT "\n", BATgetId(b), > + ", kfunc = %d, bytes = %lu\n", BATgetId(b), > BATcount(b), b->T->heap.filename, > bloom->mbits, bloom->mbits/BATcount(b), > bloom->kfunc, bloom->bytes); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Monetdb database crashes
by Souri Ratnaparkhi 04 Mar '16

04 Mar '16

Hello Team, Hello monetdb team, We have a table with 126 columns and approx. 94000 rows. We are trying to load approx 2500 records from temporary table into main table using query “insert into select” query. The database crashes upon query execution. No specific error is created in log file. When tried with 1500 records, it works. We also created another table with same DDL, that table is empty & inserted all the records from temp successfully without any issue. What could be the reason? Is 126 cols an issue? Please help. We are stuck. (btw: We have machine about 200GB of hard disk & 16 GB RAM (14 GB free)) Thank you [image: Description: Description: Description: Description: logo] * Souri Ratnaparkhi United Software Associates Pvt. Ltd. Pune, India Mob: 8975677255* * Phone (India Lines): (91) 020-25445550 **www.usain.com* <http://www.usain.com/>

2 1