- users-list - monetdb.org

[MonetDB-users] MonetDB database size restrictions
by Sam Mason 19 Jun '08

19 Jun '08

Hi, I hope you don't get asked this question too often, I've had a look for FAQs but couldn't find anything obvious. I'm trying to load some data into MonetDB (built on a i386 Debian "etch" box from MonetDB-Feb2008-SuperBall.tar.bz2) and I'm getting strange problems. I've got a table with 11 columns (2 int, 5 varchar, 4 date) and about 38 million rows (the text file I want to pull in is 3.3GB). If I try COPYing the data in, Monet alternately stops making progress (the computer is reading from the disk slowly, so maybe it's just going to take a *long* time to finish) or crashes with the following message: !SQLException:importTable:failed to import table !ERROR: GDKload: cannot mmap(): name=24/2470, ext=tail.priv !OS: Cannot allocate memory !ERROR: GDKload failed: name=24/2470, ext=tail.priv If I try splitting the file into smaller pieces (about a gig each), things work and I end up with the data in the database. It would be nice to know why I needed to split the file up first. Now I've got the data in I tried running a few queries to make sure everything went OK. I'm getting some very strange results when doing aggregations. The following work: SELECT count(*) FROM source_livestock; +----------+ | count_ | +==========+ | 37898310 | +----------+ SELECT sex_code, count(*) FROM source_livestock GROUP BY sex_code; +----------+---------------+ | sex_code |count_sex_code | +==========+===============+ | M | 17505304 | | F | 20392997 | | null | 9 | +----------+---------------+ This last query came back amazingly quickly, about ten times faster than Postgres. Monet looks like a fairly amazing database if I can routinely expect queries to be this much faster. SELECT count(*); +--------------+ | single_value | +==============+ | 1 | +--------------+ but the following don't do what I'd expect: SELECT count(1); MAPI = monetdb@endemic:50001 ACTION= read_line QUERY = select count(1); ERROR = Connection terminated I'd expect to get a single row with a single value containing the number 1 from this query. And for the following query I'd expect to get the same, just with the result being zero. SELECT count(null); !syntax error, unexpected sqlNULL in: "select count(null" This is how I found the bug: SELECT sex_code, count(1) FROM source_livestock GROUP BY sex_code; !TypeException:user.s3_1[26]:'aggr.count_no_nil' undefined in: _36:bat[:any,:int] := aggr.count_no_nil(_35:bte, _31:bat[:oid,:oid], _32:bat[:oid,:oid]) !SQLException:sql.rsColumn:Cannot access descriptor !WARNING: BATdescriptor: range error 1 I've come to expect COUNT(*) and COUNT(1) as having the same semantics. But I may be missing something. SELECT sex_code, count(1+1) FROM source_livestock GROUP BY sex_code; !TypeException:user.s4_1[28]:'aggr.count_no_nil' undefined in: _39:bat[:any,:int] := aggr.count_no_nil(_38:bte, _32:bat[:oid,:oid], _33:bat[:oid,:oid]) +----------+---------------------+ | sex_code |sql_add_single_value | +==========+=====================+ | M | monetdb | +----------+---------------------+ This second one is particularly bad because I get an incorrect answer back. Any idea what I'm doing wrong? Thanks, Sam

2 13

[MonetDB-users] MonetDB Server dies on connect
by Robert Schumacher 17 Jun '08

17 Jun '08

Hello, i was evaluating MonetDB. Got the Server running finr on Windows XP x64. I created about 12 Tabled by using a JDBC Client, then the Server died instantly. Since then when I start the Server it dies asap when I either start the Windows Client or perform a JDBC connection. I get no error log, no message, nothing. Has anybody an idea how to get MonetDB to stop crashing on connect? Thanks, Robert Robert Schumacher, Senior Software Developer prevero Systems GmbH Landsberger Straße 154 80339 München Tel +49 89 455 77 - 0 Fax +49 89 455 77 - 200 Registergericht: Amtsgericht München, HRB 143809 -- Geschäftsführer: Matthias Thurner Bitte beachten Sie unsere Veranstaltungshinweise unter www.prevero.de

4 14

Re: [MonetDB-users] [Monetdb-developers] arrow keys not working in mclient in stable nightly
by Martin Kersten 16 Jun '08

16 Jun '08

Rt Ibmer wrote: > Hi - I use a SSH client to access my Fedora 8 core box. When I was using the RELEASE built I was able to use left/right/up/down/home/end keys within mclient to edit the line. For instance I could push up arrow to have it automatically cycle through the previous lines, and then use left arrow to move over to a part of the line and process to change it. > There have not been changes in that area. Running against the current also does not show the problem > That all worked fine under the RELEASE build. > > However now that I have built from source the nightly stable (as of a couple nights ago) I can no longer use these keys. For instance when I press UP arrow it simply adds "^[[A" to the line. Other arrow keys / home /end do something similar. > Nothing has changed in this area. I mostly use a Putty ssh client to access my machine and work with mclient. It works as expected. You may want to check the preference setting of the ssh client. > Not sure but I think something broke regarding this at some point between the released build and the stable nightly. Thanks. > > > > > > ------------------------------------------------------------------------- > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services for > just about anything Open Source. > http://sourceforge.net/services/buy/index.php > _______________________________________________ > Monetdb-developers mailing list > Monetdb-developers(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-developers >

2 1

Re: [MonetDB-users] replication in MonetDB
by Martin Kersten 15 Jun '08

15 Jun '08

Alexandru Toth wrote: > Hello, > > I'm the developer of Cubulus OLAP > (http://sourceforge.net/projects/cubulus/ ) . Hello Alexandru Welcome to the MonetDB user community. > > On the roadmap there is a hint about replication, coming in SQL 2.2 > (February 2008) release. Is it implemented ? If not, is it possible to use We have been experimenting with replication techniques. Some of the hooks have been included in the distribution. However, we are not yet satisfied and with the result. It remains on our R&D list to come up with a novel and efficient solution. > some existing software ex: http://dbreplicator.org/ Middleware solutions could potentially provide a short term solution. However, we don't know if they are powerful enough or require functionality currently not available in our JDBC driver. > > Or, is it possible to load data into "master", stop it, sync the > dbfarm to several slaves (into new directories), and re-start the > slaves with the new dbfarm path ? Of course, slaves are used as > read-only copies. ofcourse, provided you don't forget to copy the sql_logs as well. You can even instruct the server that the database is readonly. See the documentation : http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/Storage-Modes.html regards, Martin > > -Alex > >

1 0

Re: [MonetDB-users] [Monetdb-developers] MonetDB Server dies on connect
by Martin Kersten 13 Jun '08

13 Jun '08

Sjoerd Mullender wrote: > Martin Kersten wrote: > >> Sjoerd, >> >> I was planning to try out the installers on my Vista machine. >> The previous version is v5.4.0 installed >> >> Uninstalling it gives the following error: >> There is a problem with this Windows Installer >> package. A program required for this install >> to complete could not be run. Contact your >> support personnel or package vendor. >> >> You want specific information, or should i manually >> remove the C:\Program Files\CWI/MonetDB5 >> > > Manually removing won't help unless you also edit the registry. > > My guess is that it complains about the uninstall of the ODBC driver. > Can you run > "C:\Program Files\CWI\MonetDB5\odbcinstall" /Uninstall > (I think that is the exact name) manually from a cmd window? > ran this program , which worked. However, uninstalling MonetDB still gives the same error.

2 1

[MonetDB-users] New default configure options for MonetDB
by Stefan Manegold 09 Jun '08

09 Jun '08

Dear MonetDB developer and users, For convenience of both developers and users as well as to comply even more with open source standards, we now set/use the following defaults for the configure options --enable-strict, --enable-assert, --enable-debug, --enable-optimize (Unless specified differently, the same defaults are used for both the "Stable" release branches and the "Current" development trunk.) When compiling from CVS sources (as mainly done by developers): strict=yes assert=yes debug=yes optimize=no (*) When compiling from packaged/distributed sources (i.e., tarballs) (as mainly done by users): strict=no assert=no debug=no optimize=no (*) For building binary distributions (RPMs, MSIs): strict=no assert=no debug=no optimize=yes With nightly testing for the "Stable" release branches: strict=yes assert=no debug=no optimize=yes With nightly testing for the "Current" development trunk: strict=yes assert=yes debug=no optimize=yes (*) IMPORTANT NOTE: Since "--enable-optimize=yes" is no longer the default for any case except binary packages, it is *strongly recommended* to (re)compile everything from scratch, *explicitly configured* with "--enable-debug=no --enable-optimize=yes", in case you want/need to run any performance experiments with MonetDB! (Please note: "--enable-X=yes" is equivalent to "--enable-X", and "--enable-X=no" is equivalent to "--disable-X".) Don't hesitate to ask in case you have any questions! Enjoy! Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [MonetDB-users] [Monetdb-developers] Fastest way to calculate median?
by Martin Kersten 08 Jun '08

08 Jun '08

Rt Ibmer wrote: > Hi - I have an int column with several million rows of data. What is the fastest way to write a query to get the median value of the column? Apparently there is no median() function in SQL. A web search revealed lots of various talk about using stored procedures and fancy functions. None of this looked like it would be very efficient. I searched these archives but didn't see a single mention of median. Thank you! > True, the complexity of the median() in essence requires a partial sort of the complete list. It is a member of a group of statistical functions. A feature request has been added to the list. > > > > > > ------------------------------------------------------------------------- > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services for > just about anything Open Source. > http://sourceforge.net/services/buy/index.php > _______________________________________________ > Monetdb-developers mailing list > Monetdb-developers(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-developers >

1 0

[MonetDB-users] Moving records from SQL Server to MonetDB
by Rt Ibmer 07 Jun '08

07 Jun '08

Hi All - I'm new to monetdb and it looks fantastic! I have about 2 millions rows of data that I'd like to export from SQL Server and import them into Monetdb so I can do some preliminary benchmarking. What is the best way to do this? The schemas are the same for SQL Server and MonetDB. So basically I'm looking for the best command to run on SQL Server to export the data in a format best suited for importing into MonetDB, and what is the command to then load the data into MonetDB? Thanks!

1 0

[MonetDB-users] [Fwd: RE: PROBLEM WITH MONETDB 5.0]
by Martin Kersten 07 Jun '08

07 Jun '08

-------- Original Message -------- Subject: RE: PROBLEM WITH MONETDB 5.0 Date: Sat, 7 Jun 2008 23:16:13 +0800 From: Tran Quoc Trung <tqtrung(a)comp.nus.edu.sg> To: 'Martin Kersten' <Martin.Kersten(a)cwi.nl> References: <38563.137.132.80.35.1212469981.squirrel(a)mysoc.nus.edu.sg> <4844E431.2090403(a)cwi.nl> <3363.137.132.80.35.1212475247.squirrel(a)mysoc.nus.edu.sg> <4844F364.70601(a)cwi.nl> Hi Mr. Martin, I have installed MonetDB both by using monetdb-install.h and following the document for expert in the website on Ubuntu 2.6 64-bits. I can use the MonetDB server and client. However, when writing a program (in C) to connect to MonetDB, I encounter the error that libMapi.so require libc-2.7.so (The libc in my system is libc-2.5.so). Are there any solution to handle this situation (b/c I restrict not to upgrade the libc due to using the common server with other project). Could you please help me to take a look at this. Thank you very much Trung. -----Original Message----- From: Martin Kersten [mailto:Martin.Kersten@cwi.nl] Sent: Tuesday, 3 June, 2008 3:32 PM To: tqtrung(a)comp.nus.edu.sg; Communication channel for MonetDB users Subject: Re: PROBLEM WITH MONETDB 5.0 tqtrung(a)comp.nus.edu.sg wrote: > Hi, > Thank you very much. I also guess the reason due to the size of the > operands. Currently, I am running on Linux Fedora Core 8, 32 bit. The > cardinality of the opearand s in the range 100,000 to 1,500,000 records. > the sizes themselves are not a problem, but the expected size of the join result could be. you could take a sample to see how it explodes (algebra.sample()) > Are there any other solutions if my machine could not play with those > queries. > Please help me to check it again. > > Thanks, > Trung > > > >> tqtrung(a)comp.nus.edu.sg wrote: >> >>> Dear all, >>> I am a PhD student at School of Computing, NUS, Singapore. Currently, I >>> used MonetDB as the DBMS to do some experiments. I issue a query with >>> about 10 joins, but the system responds with the following errors: >>> >>> >> Please inform us about the platform and distribution you are useing. >> >>> !MALException:algebra.join:GDKerror >>> !ERROR: GDKload: cannot mmap(): name=45/4542, ext=head.priv >>> !OS: Cannot allocate memory >>> >>> >> this typically is an error when the join algorithm can not allocated >> enough memory >> to hold the intermediate result. For example, on a 32-bit system this >> might cause >> a problem quickly. >> If the join result should fit in memory, then there might be a problem >> in the >> estimation of the intermediate size needed. For this we need more >> information, >> such as the cardinality of the operands and properties obtained by >> bat.info() >> >>> !ERROR: GDKload failed: name=45/4542, ext=head.priv >>> !ERROR: GDKload: cannot mmap(): name=45/4542, ext=tail.priv >>> !OS: Cannot allocate memory >>> !ERROR: GDKload failed: name=45/4542, ext=tail.priv >>> !ERROR: GDKload: cannot mmap(): name=45/4542, ext=head.priv >>> !OS: Cannot allocate memory >>> !ERROR: GDKload failed: name=45/4542, ext=head.priv >>> !ERROR: GDKload: cannot mmap(): name=45/4542, ext=head.priv >>> !OS: Cannot allocate memory >>> !ERROR: GDKload failed: name=45/4542, ext=head.priv >>> !ERROR: GDKload: cannot mmap(): name=44/4401, ext=tail.priv >>> !OS: Cannot allocate memory >>> !ERROR: GDKload failed: name=44/4401, ext=tail.priv >>> >>> I also try to change the parameter in monetdb5.conf about the memory: >>> gdk_maxsize and gdk_vm_minsize but it still does NOT work. >>> >>> >> please don't play with these settings. They only control a thread for >> flushing dirty pages to disk. >> >>> Could you please help me to figure out which parameters I could change >>> to >>> make it work. >>> >>> Thank you very much >>> Trung >>> >>> >>> >> > >

1 0

[MonetDB-users] [Fwd: Re: php extension]
by Martin Kersten 06 Jun '08

06 Jun '08

-------- Original Message -------- Subject: Re: php extension Date: Fri, 6 Jun 2008 11:20:39 +1200 From: Kees Varekamp <kees.varekamp(a)gmail.com> To: Sjoerd Mullender <sjoerd(a)acm.org> CC: Niels Nes <Niels.Nes(a)cwi.nl>, monet(a)cwi.nl, Sjoerd Mullender <Sjoerd.Mullender(a)cwi.nl> References: <d8981ee0806042218l13609c8an47065c5b9a8354e9(a)mail.gmail.com> <20080605054117.GA350(a)cwi.nl> <4847D4D4.8020608(a)acm.org> <d8981ee0806050543w19dd0028gbb9327c5378549b1(a)mail.gmail.com> <4847EEFD.8080102(a)acm.org> Tnx. Na een herstart doet ie het! Groeten, Kees 2008/6/6 Sjoerd Mullender <sjoerd(a)acm.org <mailto:sjoerd@acm.org>>: Kees Varekamp wrote: > Bedankt! Hij start nu, ik ben weer een stap verder. > > Krijg nu een *Warning*: monetdb_connect() [function.monetdb-connect > <http://localhost/monet/function.monetdb-connect>]: Unable to connect to > MonetDB server: gethostbyname failed > > Het voorbeeldje op de website > (http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/The-PHP-Library.ht…) > connect als volgt: > $db = monetdb_connect('sql', 'localhost', 50000, 'monetdb','monetdb') > > De synopsis eronder vermeldt echter: > monetdb_connect([string host [, string port [, string username [, string > password [, string language]]]]] > > Beide returnen bovenstaande foutmelding (zowel met localhost als met > 127.0.0.1 <http://127.0.0.1> <http://127.0.0.1>). Welke is juist? En klopt het dat ik in > principe de php_pgsql.dll functies kan vertalen naar Monet functies? > > Het is niet kritiek trouwens, if all else fails zet ik een Linux machine op. Het ziet ernaar uit dat de synopsis niet correct is en de volgorde van de argumenten inderdaad is zoals in dat voorbeeld. Ik heb dit ook net geprobeerd en het werkt. De hostnaam wordt met de C functie gethostbyname vertaald. Zowel de string 'localhost' als '127.0.0.1 <http://127.0.0.1>' zou moeten werken, maar het ligt aan je Windows installatie. Je kunt experimenteren met php -a (in een cmd window) en dan: php > print gethostbyname('localhost'); Dat zou 127.0.0.1 <http://127.0.0.1> moeten printen. Je kunt ook de file c:\windows\system32\drivers\etc\hosts bekijken om te zien of er een entry in staat voor localhost. -- Sjoerd Mullender

1 0