- users-list - monetdb.org

Error: cannot bind delete column
by Doherty, Adam 15 Nov '18

15 Nov '18

Hello all: I have run across a show-stopping error, "cannot bind delete column" which renders Monet unusable until I restart the service. A batch import of events runs every 10 minutes and has been unable to complete for the last 28 hours because the service is not accepting commands. I'm not sure where to look for the source of this error. Restarting the service correct the issue, but this is a problem as the Monet instance drives live Tableau dashboards using this imported data. Where do I need to start looking to track down the source of this error? I am currently running 11.29.3 on CentOS 7. Adam

2 1

Adding columns to tables via binary bulk loading
by Anderson, David B 12 Nov '18

12 Nov '18

All, Is it possible to add a column to an existing table via binary bulk loading? We have a table with ~1 billion rows and we would like to add columns to the table, but would prefer bulk loads for speed. Also, does anyone have a snippet of code to create the binary file format that bulk loading expects. I've tried a couple versions (simple int and double), but the server wouldn't recognize the files. Thanks, Dave

2 1

Unable to get sys.queue for all the users
by Murthy, Gautham 07 Nov '18

07 Nov '18

Hello, We are unable to get the information for all active users sessions when we use sys.queue as monetdb user. Also, the tried as user other than monetdb, it returned no access, We had provided the user select privilege on the sys.queue view. [monetadmin@lnx1487 scripts]$ /monet_binaries/MonetDB-11.27.13_PY/bin/mclient mapi:monetdb://lnx1487:50053/DB_BWS_S Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2017-SP4) Database: MonetDB v11.27.13 (Jul2017-SP4), 'mapi:monetdb://lnx1487:50053/DB_BWS_S' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select "user","active",count(*) as "No. Of Connections" from sys.sessions() group by "user","active"; +-------------------+--------+--------------------+ | user | active | No. Of Connections | +===================+========+====================+ | DM_POS_BWS_STG_P6 | true | 1 | | DM_POS_BWS_STG_P6 | false | 3 | | monetdb | true | 1 | +-------------------+--------+--------------------+ 3 tuples (6.027ms) sql>select * from sys.queue; +------+---------+----------------------------+----------------------------+----------+---------+--------+--------------------------+ | qtag | user | started | estimate | progress | status | tag | query | +======+=========+============================+============================+==========+=========+========+==========================+ | 3630 | monetdb | 2018-11-07 17:07:34.000000 | null | null | running | 3630@0 | select * from sys.queue; | +------+---------+----------------------------+----------------------------+----------+---------+--------+--------------------------+ 1 tuple (1.610ms) sql>\q [monetadmin@lnx1487 scripts]$ Thank You, Gautham

4 3

Cannot setup monetdb on Centos 7
by GAUQUELIN, Philippe 05 Nov '18

05 Nov '18

Hi Need help on setup. Thanks for your answer Trying to install I got this message. The GPG keys listed for the "MonetDB 7 - x86_64" repository are already installed but they are not correct for this package. More details below: []# yum reinstall https://dev.monetdb.org/downloads/epel/MonetDB-release-epel.noarch.rpm Loaded plugins: fastestmirror, langpacks MonetDB-release-epel.noarch.rpm | 6.4 kB 00:00:00 Examining /var/tmp/yum-root-x4i3eb/MonetDB-release-epel.noarch.rpm: MonetDB-release-epel-1.3-1.monetdb.noarch Resolving Dependencies --> Running transaction check ---> Package MonetDB-release-epel.noarch 0:1.3-1.monetdb will be reinstalled --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================================= Reinstalling: MonetDB-release-epel noarch 1.3-1.monetdb /MonetDB-release-epel.noarch 4.0 k Transaction Summary ======================================================================================================================================================================= Reinstall 1 Package Total size: 4.0 k Installed size: 4.0 k Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : MonetDB-release-epel-1.3-1.monetdb.noarch 1/1 Verifying : MonetDB-release-epel-1.3-1.monetdb.noarch 1/1 Installed: MonetDB-release-epel.noarch 0:1.3-1.monetdb Complete! [root@franalytic02 tmp]# []# rpm --import https://www.monetdb.org/downloads/MonetDB-GPG-KEY []# yum install MonetDB-SQL-server5 MonetDB-client Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: centos.crazyfrogs.org * epel: mirrors.ircam.fr * extras: centos.crazyfrogs.org * updates: centos.mirror.fr.planethoster.net Resolving Dependencies --> Running transaction check ---> Package MonetDB-SQL-server5.x86_64 0:11.31.11-20181011.el7 will be installed --> Processing Dependency: MonetDB5-server(x86-64) = 11.31.11-20181011.el7 for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 --> Processing Dependency: libstream.so.11()(64bit) for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 --> Processing Dependency: libmonetdb5.so.25()(64bit) for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 --> Processing Dependency: libbat.so.17()(64bit) for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 ---> Package MonetDB-client.x86_64 0:11.31.11-20181011.el7 will be installed --> Running transaction check ---> Package MonetDB.x86_64 0:11.31.11-20181011.el7 will be installed ---> Package MonetDB-stream.x86_64 0:11.31.11-20181011.el7 will be installed ---> Package MonetDB5-server.x86_64 0:11.31.11-20181011.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================================= Installing: MonetDB-SQL-server5 x86_64 11.31.11-20181011.el7 monetdb 854 k MonetDB-client x86_64 11.31.11-20181011.el7 monetdb 275 k Installing for dependencies: MonetDB x86_64 11.31.11-20181011.el7 monetdb 1.0 M MonetDB-stream x86_64 11.31.11-20181011.el7 monetdb 197 k MonetDB5-server x86_64 11.31.11-20181011.el7 monetdb 853 k Transaction Summary ======================================================================================================================================================================= Install 2 Packages (+3 Dependent packages) Total size: 3.1 M Installed size: 9.1 M Is this ok [y/d/N]: y Downloading packages: warning: /var/cache/yum/x86_64/7/monetdb/packages/MonetDB-stream-11.31.11-20181011.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID df0e54f3: NOKEY Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Importing GPG key 0x0583366F: Userid : "MonetDB Database System Packager <monet(a)cwi.nl>" Fingerprint: 213e 64dc d5dd c2c0 63cc 39fd 053c 3ed4 0583 366f Package : MonetDB-release-epel-1.3-1.monetdb.noarch (installed) >From : /etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Is this ok [y/N]: y Importing GPG key 0x0583366F: Userid : "MonetDB Database System Packager <monet(a)cwi.nl>" Fingerprint: 213e 64dc d5dd c2c0 63cc 39fd 053c 3ed4 0583 366f Package : MonetDB-release-epel-1.3-1.monetdb.noarch (installed) >From : /etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Is this ok [y/N]: y Public key for MonetDB-stream-11.31.11-20181011.el7.x86_64.rpm is not installed Failing package is: MonetDB-stream-11.31.11-20181011.el7.x86_64 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Philippe

2 1

Using capi.eval from MAL
by Renato Marroquín Mogrovejo 04 Nov '18

04 Nov '18

Hi all, I am trying to use the following patten from MAL[1] pattern eval(fptr:ptr,cpp:bit,expr:str,arg:any...):any... However, I am not sure how I should create the first parameter <fptr:ptr> from MAL. I am just trying to be able to call a C-UDF from MAL. From capi.c, it would seem that it is the client's context but I don't know how to get a handle of that. Any pointers are highly appreciated. Thanks! Renato M. [1] https://github.com/MonetDB/MonetDB/blob/2310e49a349ebcb8ff5b5c585928c8f4bd1…

2 4

Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full
by Murthy, Gautham 04 Nov '18

04 Nov '18

Hello, We are hitting the below error while loading data into table, There is no space crunch or memory (resource crunch). Please let us know if there are any restrictions in loading data. sql>COPY INTO "DM_POS_PROSCORES_MRT_P1".tier3_score_201807_part53 FROM '/backup8/schema_backups/tier3_score_201807_part53'; Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd version MonetDB Database Server v1.7 (Jul2017-SP4) You have mail in /var/spool/mail/monetadmin [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb version MonetDB Database Server Toolkit v1.1 (Jul2017-SP4) [monetadmin@lnx1538 monet_data01]$ mclient -version mclient, the MonetDB interactive terminal (Jul2017-SP4) support for command-line editing compiled-in character encoding: UTF-8 [monetadmin@lnx1538 monet_data01]$ df -hP /monet_data01 Filesystem Size Used Avail Use% Mounted on /dev/mapper/encrmonet_data01lv 12T 7.0T 4.4T 62% /monet_data01 [monetadmin@lnx1538 monet_data01]$ ps -ef|grep -i monetdbd monetad+ 331928 1 0 Nov02 ? 00:00:14 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_TEST01 monetad+ 333365 332892 0 10:57 pts/9 00:00:00 grep --color=auto -i monetdbd monetad+ 405353 1 0 Oct31 ? 00:00:30 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_PROSCORE_P [monetadmin@lnx1538 monet_data01]$ [monetadmin@lnx1538 monet_data01]$ free -g total used free shared buff/cache available Mem: 1007 18 302 7 685 978 Swap: 31 0 31 [monetadmin@lnx1538 monet_data01]$ Thank You, Gautham

3 3

Append an array of values to a BUN
by Anton Kravchenko 02 Nov '18

02 Nov '18

Hi there, Is there a way to append an array of values to a BUN, instead one value at a time? sql_table *tbl = mvc_bind_table(m, sch, "atable"); sql_column *col = mvc_bind_column(m, tbl, "acolumn"); BAT *tmp1 = COLnew(0, TYPE_lng, (BUN)nrows, TRANSIENT); long long v_long[3] = {100000000009,100000000010,100000000011}; for (irow=0; irow<nrows; irow++) BUNappend(tmp1, &v_long[irow], FALSE); mvc_append_column(m->session->tr, col1, tmp1); mvc_commit(m, 0, tr->name); bat_destroy(tmp1); Thanks, Anton

2 5

Circular Memory Only Database
by Doherty, Adam 26 Oct '18

26 Oct '18

Hello all: I am wondering if anyone has a remote table strategy such that "hot data" is stored in a table that is backed by RAM. I have a table with approximately 1 billion events that grows by approximately 1,000 events a minute. Table is a merge table of merge tables, the parent merge table has a child merge table for every year of data under those child tables are source tables that make up data for areas. So Area1_2017, Area2_2017 -> Events_2017 Area1_2018, Area2_2018 -> Events_2018 Events_2017, Events_2018 -> Events I am finding that as I add more data the system seems to slow down. I am wanting to keep a cyclical table of say the last 6 months from the Events table in a table backed solely by RAM So a query to Recent_Events would remain very fast or at the very least constant. This should be achievable with tmpfs. I am currently running 11.29.3 on CentOS 7 on VMware 6.7 Server has 128GB of RAM and 16 cores. Adam

2 1

SQLSTATE Error not showing up in mclient Aug2018 version (Aug2018 (11.31.7))
by Sharma, Sreejith 22 Oct '18

22 Oct '18

All, Do we have a way to compile mclient along with server source code? We are not able to see SQLSTATE error code on concurrency with Aug2018 (11.31.7) server version along with mclient Aug2018. Would like to see if we need to specifically compile client too? ERROR = !COMMIT: transaction is aborted because of concurrency conflicts, will ROLLBACK instead CODE: 40000 Regards, Sreejith

2 3

Re: users-list Digest, Vol 74, Issue 3
by Oxana Dolmatova 17 Oct '18

17 Oct '18

Hi Stefan, Thank you for the information, now I see the point. >Did you by any chance try any other DBMSs with your "non-standard" workload? >If so, how do those experiences and limitations compare to MonetDB? I did not run tests over conventional DBMSs because they have stricter limits compared to MonetDB (e.g. Postgres is limited to 1600 columns), but only over array DBMSs or math packages. Obviously, their column scalability is better, but they can not handle RA that efficiently. Best regards, Oksana On Tue, 16 Oct 2018 at 12:16, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: Column limit (Oxana Dolmatova) > 2. Re: Column limit (Stefan Manegold) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 15 Oct 2018 17:14:14 +0200 > From: Oxana Dolmatova <oxana.dolmatova(a)gmail.com> > To: E.Rozenberg(a)cwi.nl > Cc: users-list(a)monetdb.org > Subject: Re: Column limit > Message-ID: > <CAJJ2wAQi+4S+= > e9BfbStuJxD7izNh8z-mcBF8FYKxagjj5oRkA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Eyal, > > Sorry for the late reply. Yes, there is no problem with large number of > columns in sparse representation of arrays (i.e., dimension, dimension, > value). > However, in order to perform relational operations, this structure should > be pivoted first. When matrices are represented by tables, both algebras > can be applied to the same object, > so I'm specifically looking for the main bottleneck of processing hundreds > of thousands of columns in MonetDB. > > Best regards, > Oksana > > > On Wed, 10 Oct 2018 at 13:40, Eyal Rozenberg <E.Rozenberg(a)cwi.nl> wrote: > > > If you want to model an NxM matrix of elements of type MYTYPE in a DBMS > > (not just MonetDB), you would typically have a table with 3 columns: > > > > CREATE TABLE my_matrix ( > > "row" INTEGER NOT NULL, > > "col" INTEGER NOT NULL, > > "cell_value" MYTYPE NOT NULL, > > PRIMARY KEY ("row", "col") > > ); > > > > of length N x M. > > > > Anyways, I'm not the expert on Arrays in our group so I'll let others > > chime in. > > > > Eyal > > > > > > On 10/10/2018 13:01, Oxana Dolmatova wrote: > > > Hello, > > > > > > Thank you for all the replies and information! > > > The version is 11.23, plus I'm using the cloud platform > > with 4cpu-16ram-hpc. > > > The number of rows was just 1K for 15K attributes in a table. > > > I understand that MonetDB was made with no intention to create and > > > process tables with that many columns. But I'm interested in matrix > > > operations integration, so I would like to know what is the main > > > bottleneck to process such data (matrix-like sized) in the system. > > > > > > Best regards, > > > Oksana > > > > > > > > > On Tue, 9 Oct 2018 at 21:02, Eyal Rozenberg <eyalroz(a)technion.ac.il > > > <mailto:eyalroz@technion.ac.il>> wrote: > > > > > > I'll add two observations to what Zhang Ying said. > > > > > > 1. Internally in MonetDB, the index of a column (or a BAT) in > > MonetDB's > > > BBP (BAT Buffer Pool) is a signed 32-bit integer (on most > > platforms), so > > > there isn't a theoretical problem having more than 15,000 columns; > > but > > > MonetDB was not designed with that many tables in mind. The "long > > > dimension" is expected to be the length of columns. > > > > > > 2. When you have as many as 15,000 columns, you're not simply > finding > > > more attributes of the same items. So, for example, a person has a > > first > > > name, last name, year of birth, etc. etc. - but after several > dozens > > of > > > these you're probably adding columns such as "data relevant to this > > > person at point X", "data relevant to this person at point Y" and > so > > on. > > > > > > In your case, I'm guessing you have a lot of points and hence a lot > > of > > > columns with data for your different items. What you should > consider > > > doing instead is having an extra "coordinate" column. Thus, in the > > > example I gave, I would drop most of the columns, in favor of a > > second > > > table named "person_data". This table's columns would be > "person_id" > > (or > > > some other key columns), "sample_point" and "value". Thus instead > of > > M > > > columns and N records you would have N x M records with 3 columns. > > > > > > Eyal > > > > > > > > > > > > On 09/10/2018 16:50, Ying Zhang wrote: > > > > Hai Oksana, > > > > > > > > A.f.a.I.k., there is no explicit limit on the #columns you can > put > > > in a relation (did you mean ?table??). The reason that mserver5 > > > stopped responding is probably because it is busy with I/O. > > > > > > > > A long time ago, I once tried to create 1M tables, each with 3 > > > columns. After about a day or something, MonetDB has created ~400K > > > tables, and I could hear my HDD being constantly busy. MonetDB was > > > still working, but that was already a sufficient evidence that that > > > idea wasn?t going to work. > > > > > > > > So the question is actually what do you want to achieve? > > > > > > > > Regards, > > > > Jennie > > > > > > > > > > > >> On 9 Oct 2018, at 13:43, Oxana Dolmatova > > > <oxana.dolmatova(a)gmail.com <mailto:oxana.dolmatova@gmail.com>> > > wrote: > > > >> > > > >> Hello, > > > >> > > > >> Could you tell the limitation for number of columns in a > relation > > > and the main reason for that limitation. So far I could create > > > relations with up to 15000 columns. When the number of columns is > > > bigger mserver5 stops responding. > > > >> > > > >> Thank you and best regards, > > > >> Oksana Dolmatova > > > >> _______________________________________________ > > > >> users-list mailing list > > > >> users-list(a)monetdb.org <mailto:users-list@monetdb.org> > > > >> https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > _______________________________________________ > > > > users-list mailing list > > > > users-list(a)monetdb.org <mailto:users-list@monetdb.org> > > > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Oksana Dolmatova > > > > > > _______________________________________________ > > > users-list mailing list > > > users-list(a)monetdb.org > > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > -- > Best regards, > Oksana Dolmatova >

1 0