Dear Masood,

The BAT pool has an arbitrary limit set. Be aware that each BAT descriptor takes about 0.5K
and the structure lives in your malloc arena. Repeatedly extending it may lead to memory
fragmentation.

Overall, I would re-consider the reason for having >100K tables of which most are empty.
Aside from BAT storage overhead, the database catalog table structures are even worse
and syntax/semantic checks could become a severe performance issue. We haven't tested
that extreme case (aside from a single table with 4 M attributes).

regards, Martin

On 1/21/13 8:08 AM, Masood Mortazavi wrote:

Examining the source code (gdk_bbp.c and gdk.h), one finds a compile-time limit on the number of BATs (binary association tables), i.e. the basic internal unit of operation within MonetDB.

I've not tried its practical limits myself and haven't done the analysis for your case below (partly because exact numbers are not provided) but you may be able to extend this limit by rebuilding the dbms from source with a larger value for N_BBPINIT, for instance.

Regards,

Masood

P.S.

Here's a relevant fragment from gdk.h

#define N_BBPINIT 1000

#if SIZEOF_VOID_P == 4

#define BBPINITLOG 11

#else

#define BBPINITLOG 14

#endif

#define BBPINIT (1 << BBPINITLOG)

/* absolute maximum number of BATs is N_BBPINIT * BBPINIT */

On Fri, Jan 18, 2013 at 8:19 AM, Charles Milutinovic <milutino@gmail.com> wrote:

I am experimenting with using MonetDB for a project where the most
natural implementation would involve ~100K tables (with the huge
majority being near empty, a few incredibly massive) with approx. 10
columns each. MonetDB choked on this, with the following error:

>!FATAL: BBPextend: trying to extend BAT pool beyond the limit (16384000)

I am assuming that I hit this limit because of the extreme amount of
tables. I am wondering if there are any other hard limits to be aware
of in MonetDB that you could hit before hitting system limits.
_______________________________________________
users-list mailing list
users-list@monetdb.org
http://mail.monetdb.org/mailman/listinfo/users-list
_______________________________________________
users-list mailing list
users-list@monetdb.org
http://mail.monetdb.org/mailman/listinfo/users-list