- users-list - monetdb.org

[MonetDB-users] optimizer.exportChunk exception
by Alex Bo. 11 Aug '09

11 Aug '09

Hi, I got this error on running "Select * from tables": !TypeException:user.s6_2[154]:'optimizer.exportChunk' undefined in: _237:any := optimizer.exportChunk() !TypeException:user.s6_2[155]:'optimizer.affectedRows' undefined in: _241:any := optimizer.affectedRows() Consequent queries were successful. Seems like a bug. -- View this message in context: http://www.nabble.com/optimizer.exportChunk-exception-tp24915518p24915518.h… Sent from the monetdb-users mailing list archive at Nabble.com.

2 2

[MonetDB-users] Query speed up
by Allen Zhang 11 Aug '09

11 Aug '09

Hi, All, I have a client program generates the following query against monetdb, can I create index on fact columns to speed thing up? What¹s the best approach to resolve this kind performance issue? SELECT DISTINCT t1.column1 FROM (fact t0 LEFT JOIN dimension1 t1 ON t0.header_id = t1.header_id) WHERE t0.column3 = 'xxxxxx' And t0.column5 IN ('item1¹, OEitem2¹, OEitem3¹, OEitem4¹); Thanks in advance. Allen

2 4

[MonetDB-users] MonetDB/XQuery Scaleability
by listanand＠gmail.com 10 Aug '09

10 Aug '09

Dear all, I am new to MonetDB/XQuery and am considering using it for storing and processing a large number of webpages (~1million) in XML format. I am hoping to know if this is indeed the right tool, and how well it will scale to handle tasks of this magnitude. Thanks in advance

2 3

[MonetDB-users] COPY...INTO from remote
by Matthew Jones 08 Aug '09

08 Aug '09

I noticed that COPY...INTO from a file, when executed from a remote MonetDB mclient instance tries to load the file from the system that the database is actually running on rather than the system running the remote client. Is this the expected behavior? Has anyone dealt with this or found another way to batch-load from a remote client? 73, Matthew W. Jones (KI4ZIB) http://matburt.net

3 3

[MonetDB-users] MonetDB Scaleability
by Calum Miller 06 Aug '09

06 Aug '09

Hi, I've been doing performance tests on MondetDB to see how it scales. I've been running tests on Amazon EC2 (7GB of memory, 64-Bit, CentOS 5.2) The table I am using has 60 columns with most being just simple characters of length 2, here is the query: select CollectionDEPOTID as c0, CollectionCountryID as c1, sum(ParcelCount) as m0 from parcel_stage_short as parcel_stage_short group by c0, c1; Here are the load time in batches of 8,000,000 using the following command: COPY 8000000 RECORDS INTO parcel_stage_short FROM '/mnt/ parcel_stage_short.txt' USING DELIMITERS '\t','\n'; Rows Load Time Increase % Increase 8000000 240434.17 8000000 424937.54 184503.37 76.74% 8000000 599621.98 359187.81 149.39% 8000000 769276.70 528842.53 219.95% 8000000 889600.24 889600.24 209.35% msecs msecs As can be seen the load time increases by around 200,000 msec for each additional 8Million rows added. Any tips on how to reduce the load speed? Would it be faster to load the table in one go (tried loading in bigger batches but loader fails)? Here are the performance times for each additional batch of 8,000,000 rows: Rows 1st Query 1st Increase % 1st Increase 2nd Query 2nd Increase % Increase 08000000 1176.53 16000000 4838.52 3661.99 311.25% 2239.61 1168.30 109.05% 24000000 33947.84 32771.31 2785.43% 9323.29 8251.98 770.27% 32000000 837114.13 835937.61 71051.29% 5456.96 4385.65 409.37% 40000000 668380.26 667203.73 56709.60% 8538.02 7466.71 696.97% msecs msecs msecs msecs The performance of the 1st query is unsurprisingly poor just after the data has been copied in The performance of the same query a second time yields much faster results The performance of the monetDb with 1st/2nd query degrades as load increases. Are these results typical? Is there anything I can do to increase performance on the 1st or 2nd queries? Thanks in advance Calum

3 5

[MonetDB-users] tijah exact matching
by Roy Walter 06 Aug '09

06 Aug '09

Is there a way to force a tijah query to return only exact matches? For example, if I search for "wind farm", as in: ("tijah:queryall[about(., wind farm)]") I get matches for "wind farm", "wind" and "farm" but say I only want matches for "wind farm"? I can of course loop through the result with contains(., "wind farm") to exclude unwanted results but that has performance implications. -- Roy

2 1

[MonetDB-users] Get attribute value as text how?
by Roy Walter 06 Aug '09

06 Aug '09

The following XQuery fragment will add the speakername attribute as an attribute of the tag: {$res/../@speakername}: {$res} becomes: Some text... What I want is: Roy Walter: Some text... How does MonetDB/XQuery handle this? I tried: {xs:string($res/../@speakername)}: {$res} but it errors. -- Roy

2 4

[MonetDB-users] how to read/write blob data ?
by 李坤 04 Aug '09

04 Aug '09

I want to store image files into monetdb sql server. I think there are two approaches to do it: the first one is to store the file's full path in a column with varchar type, the second one is to store the image data in a column with blob type. I don't want to see a lot of small image files in the directory, so I choose the second approach. After reading the monetdb manual, I find some examples like this: sql> insert into t1 values (1, blob '012'); I think this is not enough. I also try to use odbc api to store binary data into monetdb, but failed. So please help me to write the blob data into monetdb, as well as the read operation. Maybe there are other ways I don't know, please tell me, thanks!

2 1

[MonetDB-users] problem in extract second
by fatar fatar 03 Aug '09

03 Aug '09

query: select extract(second from '2009-07-24 16:03:35.0000'); result : 0.00 could you please tell me why the result is zero??? and how can i get more spesific result??? Akses email lebih cepat. Yahoo! menyarankan Anda meng-upgrade browser ke Internet Explorer 8 baru yang dioptimalkan untuk Yahoo! Dapatkan di sini! http://downloads.yahoo.com/id/internetexplorer

1 0

[MonetDB-users] underlying storage
by Mag Gam 01 Aug '09

01 Aug '09

What is the recommended storage for MonetDB? We are planning to buy a Sun storage appliance and were wondering if it would work with MonetDB

4 7

Re: [MonetDB-users] server requires 'SHA512' hash, but client support was not compiled in
by Calum Miller 31 Jul '09

31 Jul '09

Hi, Managed to work around this issue by changing the default hashing algorithm in the configure file from SHA512 to MD. Needed a server rebuild. Calum

2 1

[MonetDB-users] server requires 'SHA512' hash, but client support was not compiled in
by Calum Miller 30 Jul '09

30 Jul '09

Hi, I built monetDb from source on Centos 5.0 but get the following error when I try to connect to the server via "mclient -lsql --time -d demo" server requires 'SHA512' hash, but client support was not compiled in I can't find a reference anywhere on how to resolve this, help much appreciated! Thanks Calum Miller

2 1

[MonetDB-users] Executing tijah query from module fails
by Roy Walter 29 Jul '09

29 Jul '09

[This is a new thread following on from "XRPC issues"] Executing a tijah query in a module from both old and new front ends fails. Using the old front end (\g) produces the following error: xquery>import module namespace test="tj" at "http://localhost:50001/export/tj.xq "; more>execute at {"localhost"} {test:tijahSearch("Heathrow")} more><> doLoopLiftedRPC: error occurred at localhost: MAPI = monetdb@localhost:50000 QUERY = import module namespace test="tj" at "http://localhost:50001/export/tj.x q"; ERROR = !ERROR: HTTP Error Code : 404 Bad Request !ERROR: SOAP Fault Code : env:Sender !ERROR: SOAP Fault Reason: !ERROR: Error occurred during execution. !ERROR: interpret: no matching MIL operator to 'tj_query_handler(bit, BA T[void,BAT], BAT[oid,BAT], BAT[void,BAT], BAT[oid,oid], BAT[oid,BAT], BAT[oid,ln g], BAT[oid,dbl], BAT[oid,str], void)'. !MAYBE YOU MEAN: ! tj_query_handler(bit, BAT[void,BAT], BAT[void,BAT], BAT[void,BAT ], BAT[oid,any], BAT[oid,BAT], BAT[void,lng], BAT[void,dbl], BAT[void,str], BAT[ void,BAT]) : BAT[void,BAT] ! ! !ERROR: CMDhttp_post: operation failed. MAPI = monetdb@localhost:50000 QUERY = import module namespace test="tj" at "http://localhost:50001/export/tj.x q"; ERROR = !ERROR: doLoopLiftedRPC: execution failed at all destinations: !ERROR: localhost: !ERROR: HTTP Error Code : 404 Bad Request !ERROR: SOAP Fault Code : env:Sender !ERROR: SOAP Fault Reason: !ERROR: Error occurred during execution. !ERROR: interpret: no matching MIL operator to 'tj_query_handler(bit, BA T[void,BAT], BAT[oid,BAT], BAT[void,BAT], BAT[oid,oid], BAT[oid,BAT], BAT[oid,ln g], BAT[oid,dbl], BAT[oid,str], void)'. !MAYBE YOU MEAN: ! tj_query_handler(bit, BAT[void,BAT], BAT[void,BAT], BAT[void,BAT ], BAT[oid,any], BAT[oid,BAT], BAT[void,lng], BAT[void,dbl], BAT[void,str], BAT[ void,BAT]) : BAT[void,BAT] ! ! !ERROR: CMDhttp_post: operation failed. !ERROR: !ERROR: xquery> Has anyone succeeded in executing a tijah query from within a module? -- Roy

3 8

[MonetDB-users] massive data loss on stop, start
by Matthew Jones 29 Jul '09

29 Jul '09

I added this information as a comment to bug *2825243* but I'm not sure if it's the same so I thought I'd post it here also to see about getting some feedback: I am experiencing widespread data corruption when restarting the May2009-SP1 MonetDB release also, but mine is much simpler. On 64-bit ubuntu: Linux dsvm01 2.6.24-23-server #1 SMP Wed Apr 1 22:14:30 UTC 2009 x86_64 GNU/Linux With a table defined as follows: CREATE TABLE "tablespace"."test_table" ( "col1" varchar(50), "col2" varchar(50), "col3" varchar(50), "col4" varchar(50), "col5" varchar(50), "col6" int, "col7" timestamp(7), "col8" int, "col9" int, "col10" int ); I imported 20998687 records into this table with all columns populated and ran a couple of queries to check the data: select count(*) from test_table ; [ 20998687 ] select count(*) from test_table where col1 = '' or col2 = '' or col3 = ''; [ 0 ] select max(col7), min(col7) from test_table ; [ 2009-07-25 14:17:15.538000, 2009-07-24 15:29:12.524000 ] During the time I loaded the data and ran these queries I did not restart the database, it had been up for about 3 days I then issued: monetdb stop db and monetdb start db and re-ran the queries: select count(*) from raw_plays ; [ 20998687 ] select count(*) from raw_plays where col1 = '' or col2 = '' or col3 = ''; [ 20852061 ] select max(col7), min(col7) from test_table ; [ 2009-07-25 14:17:15.538000, -1-01-01 00:00:00.000000 ] This is some serious data corruption, I've lost data in the vast majority of these records! 73, Matthew W. Jones (KI4ZIB) http://matburt.net

2 3

[MonetDB-users] Announcement: May2009-SP2 bug fix release
by Sjoerd Mullender 29 Jul '09

29 Jul '09

The MonetDB team at CWI/MonetDB BV is pleased to announce the May 2009-SP2 bug fix release of the MonetDB suite of programs. This is the second bug fix release for the May2009 release. More information on this release can be found at <http://monetdb.cwi.nl/projects/monetdb/Development/Releases/May2009/#May200…>. The sources and binaries are available at <http://monetdb.cwi.nl/projects/monetdb/Download/>. -- Sjoerd Mullender

1 0

[MonetDB-users] can not delete data
by Allen Zhang 28 Jul '09

28 Jul '09

Hi, We had such a weird issue here, we are using Feb2009-SuperBall package, we managed get our data into monetdb, but some how when we try to delete data, it reappears under single user or readonly mode. Here is what we did, we put database in normal mode without gdk_single_user and gdk_readonly in monetdb5.conf. Issue a delete sql statement in mclient, now we query the data, seem the data is gone, and then we stop the database server, and start it in readonly mode. Then query the data again, the data come back!!! Seems to us the monetdb did not sync the memory with disk. Any ideas? Allen

1 0

[MonetDB-users] XRPC issues
by Roy Walter 28 Jul '09

28 Jul '09

Hi A couple of tijah related issues with query modules. 1. The following query when run from xrpc.html appears to ignore the $term argument and returns a very large result! module namespace tj = "tj"; declare function tj:tijahSearch($term as xs:string) as node()* { tijah:queryall("//speech[about(., $term)]") }; 2. Running the above query sometimes produces the following error: !ERROR: interpret: no matching MIL operator to 'tj_query_handler(bit BAT[void BAT], ... Thanks, Roy

2 10

[MonetDB-users] string coercion error
by Alex Bo. 27 Jul '09

27 Jul '09

Hi, What does the following exception mean: !SyntaxException: convertConstant: no string coercion for 'any' !SyntaxException:convertConstant:conversion error !SyntaxException efConstant:constant coercion error from void to any I can't reproduce it - I am getting this from time to time on simple queries that usually work. Thanks. -- View this message in context: http://www.nabble.com/string-coercion-error-tp24684760p24684760.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

Re: [MonetDB-users] result cache?
by Allen Zhang 27 Jul '09

27 Jul '09

Thanks Milena, Is that means monetdb does not provide final result cache? We have sql client will query same query to the database multiple times, imaging each query took 2 minutes to run, that¹s will be really painful when the client ask 10 times, since our database is readonly, so a real result cache will really help us. Regards Allen

3 6

[MonetDB-users] Segfault interpreting sql query
by Franck Routier 27 Jul '09

27 Jul '09

Hi, Monetdb is segfaulting, probably while interpreting a query or planning it. I first thought it had to do with data, but after emptying the tables, the query still makes monetdb segfault. Error in syslog says: [ 1230.812708] mserver5[5369]: segfault at 7fab4c2b5ff8 ip 00007fab4f894285 sp 00007fab4c2b6000 error 6 in lib_sql.so.2.30.2[7fab4f7e3000+116000] Following is the faulty ddl / query. Is there anything I can do to help debug this ? Franck The ddl is the following: CREATE TABLE cabact ( cabactcsu integer NOT NULL, cabact___rforefide character varying(32) NOT NULL, cabact___rteprcide character varying(32) NOT NULL, cabact___rtestdide character varying(32) NOT NULL, cabact___rfovsnide character varying(32) NOT NULL, cabactlil character varying(100) NOT NULL, cabactlic character varying(32) NOT NULL, cabacttxt character varying(1500), cabact___rfontride character varying(32), cabact_f1rfodstide character varying(32), cabact_f2rfodstide character varying(32), cabact_f3rfodstide character varying(32), cabact_f4rfodstide character varying(32), cabact_f5rfodstide character varying(32), cabact_f6rfodstide character varying(32), cabact_f7rfodstide character varying(32), cabact_f8rfodstide character varying(32), cabact_f9rfodstide character varying(32), cabact_c1rfodstide character varying(32), cabact_c2rfodstide character varying(32), cabact_c3rfodstide character varying(32), cabact_c4rfodstide character varying(32), cabact_c5rfodstide character varying(32), cabact_c6rfodstide character varying(32), cabact_c7rfodstide character varying(32), cabact_c8rfodstide character varying(32), cabact_c9rfodstide character varying(32), cabactqte double, cabact___rfountide character varying(32), cabactdtd timestamp with time zone NOT NULL, cabactdtf timestamp with time zone NOT NULL, cabactax1 integer NOT NULL DEFAULT 1, cabactax2 integer NOT NULL DEFAULT 2, cabactax3 integer NOT NULL DEFAULT 3, cabactax4 integer NOT NULL DEFAULT 4, cabactax5 integer NOT NULL DEFAULT 5, cabactax6 integer NOT NULL DEFAULT 6, cabactax7 integer NOT NULL DEFAULT 7, cabactax8 integer NOT NULL DEFAULT 8, cabactax9 integer NOT NULL DEFAULT 9 ); CREATE TABLE rfoade ( rfoade___rforefide character varying(50) NOT NULL, rfoade___rfovdeide character varying(50) NOT NULL, rfoade_i_rfodstide character varying(50) NOT NULL, rfoadeaxe integer DEFAULT 0 NOT NULL, rfoadervs integer NOT NULL, rfoadenpm integer DEFAULT 1, rfoade_s_rfodstide character varying(32) NOT NULL, rfoadegch character varying(120) DEFAULT 'AAAAA' NOT NULL, rfoadedrt character varying(120) DEFAULT 'ZZZZZ' NOT NULL, rfoadeniv integer DEFAULT 0 NOT NULL, rfoadetxt character varying(1800), rfoadenum integer DEFAULT 99999 NOT NULL, rfoadeden integer DEFAULT 999 NOT NULL, rfoadechm character varying(5500) DEFAULT 'INVALID' NOT NULL, rfoadeord integer DEFAULT 999999 NOT NULL ); And the query is: select * from cabact where cabact___rforefide = 'FHSJ' and cabact___rteprcide = 'CPTANA' and cabact___rtestdide = '100' and cabact___rfovsnide = '200805_001' and (cabact_f1rfodstide IS NULL or cabact_f1rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_f2rfodstide IS NULL or cabact_f2rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_f3rfodstide IS NULL or cabact_f3rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_f4rfodstide IS NULL or cabact_f4rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_f5rfodstide IS NULL or cabact_f5rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_f6rfodstide IS NULL or cabact_f6rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_f7rfodstide IS NULL or cabact_f7rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_f8rfodstide IS NULL or cabact_f8rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_f9rfodstide IS NULL or cabact_f9rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_c1rfodstide IS NULL or cabact_c1rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_c2rfodstide IS NULL or cabact_c2rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_c3rfodstide IS NULL or cabact_c3rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_c4rfodstide IS NULL or cabact_c4rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_c5rfodstide IS NULL or cabact_c5rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_c6rfodstide IS NULL or cabact_c6rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_c7rfodstide IS NULL or cabact_c7rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1')) and (cabact_c8rfodstide IS NULL or cabact_c8rfodstide IN (select rfoade_i_rfodstide from rfoade where rfoade___rfovdeide='SECA' and rfoade___rforefide = 'FHSJ' and rfoadervs='1'))

5 13

[MonetDB-users] JDBC InvalidCredentialsException
by McKennirey.Matthew 27 Jul '09

27 Jul '09

2 3

[MonetDB-users] result cache?
by Allen Zhang 24 Jul '09

24 Jul '09

Hi, Does monetDB provide result cache? How to enable it? Best Regards Allen

2 1

[MonetDB-users] Segfaulting Embedded MonetDB/SQL
by Charles Samuels 23 Jul '09

23 Jul '09

Greetings, I ran the attached program. I compiled it like so: c++ -g3 monetembedded.cpp `monetdb-sql-config --cflags --libs` `monetdb- clients-config --cflags --libs` `monetdb-config --cflags --libs` `monetdb5-config --cflags --libs` -lMapi -lembeddedsql5 -lmutils MonetDB is compiled from MonetDB-May2009-SuperBall-SP1 And it segfaults before getting out of embedded_sql(). Is my program wrong, or is there a known (or unknown) problem in MonetDB with Embedded SQL? Here're the stack traces: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb169bb90 (LWP 20796)] 0xb7f039fc in sql_find_subtype () from /home/charles/.local/lib/libembeddedsql5.so.2 (gdb) bt #0 0xb7f039fc in sql_find_subtype () from /home/charles/.local/lib/libembeddedsql5.so.2 #1 0xb1c1af1e in SQLinitClient () from /usr/lib/MonetDB5/lib/lib_sql.so #2 0x09a8df68 in ?? () (gdb) info thr * 3 Thread 0xb169bb90 (LWP 20796) 0xb7f039fc in sql_find_subtype () from /home/charles/.local/lib/libembeddedsql5.so.2 1 Thread 0xb6ec78e0 (LWP 20792) 0xb7f68424 in __kernel_vsyscall () (gdb) thr 1 [Switching to thread 1 (Thread 0xb6ec78e0 (LWP 20792))]#0 0xb7f68424 in __kernel_vsyscall () (gdb) bt #0 0xb7f68424 in __kernel_vsyscall () #1 0xb7c9e48b in read () from /lib/i686/cmov/libc.so.6 #2 0xb7a24fdd in pipe_read () from /home/charles/.local/lib/libstream.so.1 #3 0xb7a238ac in stream_readSht () from /home/charles/.local/lib/libstream.so.1 #4 0xb7a23950 in bs_read () from /home/charles/.local/lib/libstream.so.1 Thanks for your help, Charles -- Charles Samuels

1 0

[MonetDB-users] put()
by Roy Walter 23 Jul '09

23 Jul '09

Hi What is the argument list for the fn:put() function? -- Roy

2 3

Re: [MonetDB-users] Database size
by Charles Samuels 23 Jul '09

23 Jul '09

On Wednesday 2009 July 22 12:25:23 pm Martin Kersten wrote: > Charles Samuels wrote: > > Hi, > > > > I have read previously that monetdb is intended to be used only when the > > data can fit in memory. I'm not sure how to interpret this statement - > > does this mean that the entire database must fit in memory, or does it > > mean that only the data being operated on when selected from the database > > must fit in memory? > > MonetDB does *not* limit your database to fit in main memory. > > However, it uses main memory techniques to address the elements in > tables loaded for processing. This means that the hot-set accessed > during query execution is limited when you run on 32-bit machines. Can you elaborate a little bit? Does hot-set mean that a select on the database can only span the chunk that can fit in a 32-bit process's address space? (This seems like a perfectly reasonable limitation, I'm just making sure). > > > We are wanting to store *huge* amounts of data, ultimately several TBs. > > The actual data being select at any given time will only be several > > hundred MBs, however. Is this a reasonable usage of MonetDB? > > The largest database we use internally to experiment with is a 6TB > astronomical database. It contains two large tables, one with 400M rows of > 500 columns and another of 20 B rows and 6 columns. Both tables are larger > then 1TB each. That's really great to hear :) > > This runs on a Linux dual quadcore with 64G Ram and lots of disk space. > Ofcourse, at some point you well notice IO behavior ;) For what kind of operations do you need that amount of memory? If you were to do a select on just a few thousand rows on that same mega-database, would it be happy with more "mundane" amounts of memory? > > > But most importantly, I have noticed that all the row ID indexes are > > "int" - why the 32 bit limitation? This just seems like a symptom of the > > above problem; that the entire database must fit in memory. > > You can use the system with both 32- and 64- bit oids. Ah, I previously had a bit of a brain failure. When reading Mapi.h, I noticed that "mapi_get_row_count" returned an int - obviously you wouldn't want select to return 2^32 rows, which is what this function counts the rows of. Thanks Martin (and also Lefteris), Charles -- Charles Samuels

2 1