Hi all, I am trying to copy a table from a postgresql database into monetdb, but it fails, with monetdb filling the disks. I am using latest version of Monetdb (Feb2013-SP3) on Ubuntu Linux, with 32 GB Ram and SSD disks. To copy into Monetdb, I have a java program that runs mclient as: mclient --language=sql --databas=database --host=host --user=user /tmp/command.ctl with command.ctl : COPY nnn RECORDS INTO table FROM STDIN USING DELIMITERS'|', '\n', '\' NULL AS '' Then the Postgresql resultset is read from jdbc, formatted and written into STDIN. So far so good, it work with my dimensions tables (small), and with a 295 millions rows fact table that has 109 columns. This fact table takes 86 GB on the disk for Postgresql storage (plus indices). I have another form of the fact table, with less rows (184 millions) but more columns (505). While the first form of the table has a measureId column and only one value column, the other form of the table has one value column per measure. (It is a "sum(case when measureId=X then value else null ... group by" version of the first table). Loading this "columnar" table fails, with monetdb filling the disks (something like 500 GB was left before the loading process) and quitting (not sure what happens exactly here on the server, but I get a broken pipe error in java, and the monetdb database is lost). The table size on disk for Postgresql is 37 GB. Using VectorWise, the table loads, and take only 7 GB on disks, but the loading process (very similar to what we do with MonetDB) takes a very long time( 25 hours), much longer than what happens with the 109 columns tables. I am trying to make sense of that and understand why this table is giving a (very) hard time to monetdb (and somehow VectorWise as well). Does anyone have an explanation ? thanks, Franck
On Thu, Jul 04, 2013 at 02:41:38PM +0200, Franck Routier wrote:
Hi all,
I am trying to copy a table from a postgresql database into monetdb, but it fails, with monetdb filling the disks. I am using latest version of Monetdb (Feb2013-SP3) on Ubuntu Linux, with 32 GB Ram and SSD disks.
To copy into Monetdb, I have a java program that runs mclient as:
mclient --language=sql --databas=database --host=host --user=user /tmp/command.ctl
with command.ctl :
COPY nnn RECORDS INTO table FROM STDIN USING DELIMITERS'|', '\n', '\' NULL AS ''
Then the Postgresql resultset is read from jdbc, formatted and written into STDIN. So far so good, it work with my dimensions tables (small), and with a 295 millions rows fact table that has 109 columns. This fact table takes 86 GB on the disk for Postgresql storage (plus indices).
I have another form of the fact table, with less rows (184 millions) but more columns (505). While the first form of the table has a measureId column and only one value column, the other form of the table has one value column per measure. (It is a "sum(case when measureId=X then value else null ... group by" version of the first table).
Loading this "columnar" table fails, with monetdb filling the disks (something like 500 GB was left before the loading process) and quitting (not sure what happens exactly here on the server, but I get a broken pipe error in java, and the monetdb database is lost).
Please check what the monetdbd / merovingian log file says.
The table size on disk for Postgresql is 37 GB. Using VectorWise, the table loads, and take only 7 GB on disks, but the loading process (very similar to what we do with MonetDB) takes a very long time( 25 hours), much longer than what happens with the 109 columns tables.
I am trying to make sense of that and understand why this table is giving a (very) hard time to monetdb (and somehow VectorWise as well). Does anyone have an explanation ?
Can you share the schemas (column types) of both tables? For each type used the number of column with that type might be sufficient. If there are no string columns at all, that would be sufficient info for us to estimate the expected storage size. E.g., if all columns were 4 byt integers, your first fact table would take 295 M * 109 * 4 ~= 128 GB, while your second table would take 184 M * 505 * 4 ~= 372 GB (all withou constraints & indexes). Other than duplicate elimination (dictionary encoding) for string columns, MonetDB does not yet use compressed storage. We do have good experiences with compressed file systems, though. Do you have constraints and/or keys defined on your tables prior to bulk loading? If so, try bulk loading without constraints/keys defined, and add them afterwards via alter table statements. Best, Stefan
thanks, Franck
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
Le 04/07/2013 15:12, Stefan Manegold a écrit :
Please check what the monetdbd / merovingian log file says. Hum... /var/log/monetdb is empty. And I can't locate any monet* log file.
The table size on disk for Postgresql is 37 GB. Using VectorWise, the table loads, and take only 7 GB on disks, but the loading process (very similar to what we do with MonetDB) takes a very long time( 25 hours), much longer than what happens with the 109 columns tables.
I am trying to make sense of that and understand why this table is giving a (very) hard time to monetdb (and somehow VectorWise as well). Does anyone have an explanation ? Can you share the schemas (column types) of both tables? Yes, sure.
The first table is called dwhinv and has : 89 columns of type varchar(32). 2 timestamps (not null) 2 decimal (at least one of them is set on each line) 19 integers The second table is called dwhcol and has : 83 columns of type varchar(32) 2 timestamps 420 columns of type decimal
For each type used the number of column with that type might be sufficient. If there are no string columns at all, that would be sufficient info for us to estimate the expected storage size.
Other than duplicate elimination (dictionary encoding) for string columns, MonetDB does not yet use compressed storage. We do have good experiences with compressed file systems, though. What about null values ? I was under the impression that null values where not stored in monetdb (just non existant entries in the BATs) ? Because each decimal column has at most 1 million non null values, and many have a few thousands... Doesn't it matter ?
Do you have constraints and/or keys defined on your tables prior to bulk loading? No, there are no foreign keys, nor constraints of any sort.
Franck
Le 04/07/2013 15:12, Stefan Manegold a écrit :
Please check what the monetdbd / merovingian log file says. Hum... /var/log/monetdb is empty. And I can't locate any monet* log file.
It's called merovingian.log and is located in your dbfarm.
The table size on disk for Postgresql is 37 GB. Using VectorWise, the table loads, and take only 7 GB on disks, but the loading process (very similar to what we do with MonetDB) takes a very long time( 25 hours), much longer than what happens with the 109 columns tables.
I am trying to make sense of that and understand why this table is giving a (very) hard time to monetdb (and somehow VectorWise as well). Does anyone have an explanation ? Can you share the schemas (column types) of both tables? Yes, sure.
The first table is called dwhinv and has :
89 columns of type varchar(32). 2 timestamps (not null) 2 decimal (at least one of them is set on each line) 19 integers
The second table is called dwhcol and has :
83 columns of type varchar(32) 2 timestamps 420 columns of type decimal
integers are 4 byte. timestamps are IMHO also 4 byte I'm not sure what we use for decimals without digit specification; I assume we then use the largest, i.e., 8 byte. strings highly depend on the actual data (due to (best effort) duplicate elimination and dictionary encoding).
For each type used the number of column with that type might be sufficient. If there are no string columns at all, that would be sufficient info for us to estimate the expected storage size.
Other than duplicate elimination (dictionary encoding) for string columns, MonetDB does not yet use compressed storage. We do have good experiences with compressed file systems, though. What about null values ? I was under the impression that null values where not stored in monetdb (just non existant entries in the BATs) ?
NULL values are stored. For fixed width columns (e.g., all numbers), they take as much storage as any value of that type. For variable length types (e.g., strings, (both char & varchar)), they take 1 to 8 byte, depending on the number of distinct values in that column. Stefan
Because each decimal column has at most 1 million non null values, and many have a few thousands... Doesn't it matter ?
Do you have constraints and/or keys defined on your tables prior to bulk loading?
No, there are no foreign keys, nor constraints of any sort.
Franck
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
NULL values are stored. For fixed width columns (e.g., all numbers), they take as much storage as any value of that type. For variable length types (e.g., strings, (both char & varchar)), they take 1 to 8 byte, depending on the number of distinct values in that column. Ouch. That kind of defeats my plan to use the "columnar" version of my
Thanks Stefan for your time and input, table with MonetDB, as the trick with this huge table is that it is mostly filled with nulls and finding the non-null lines for a column is quick. At least it is with postgresql. Ok, I've got to rethink my approach. That said, would'nt avoiding to store nulls be an interesting evolution of MonetDB ? Best regards, Franck
You might consider using a large number of columns using your own key, glued together with a view. regards, Martin On 7/4/13 5:15 PM, Franck Routier wrote:
NULL values are stored. For fixed width columns (e.g., all numbers), they take as much storage as any value of that type. For variable length types (e.g., strings, (both char & varchar)), they take 1 to 8 byte, depending on the number of distinct values in that column. Ouch. That kind of defeats my plan to use the "columnar" version of my
Thanks Stefan for your time and input, table with MonetDB, as the trick with this huge table is that it is mostly filled with nulls and finding the non-null lines for a column is quick. At least it is with postgresql. Ok, I've got to rethink my approach.
That said, would'nt avoiding to store nulls be an interesting evolution of MonetDB ?
Best regards, Franck
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Use vertical decomposition schema design. create table large( i integer, j integer) can be represented by create table largeI( surrogate integer, i integer ); create table largeJ( surrogate integer, j integer ); create view large as select * from largeI join largeJ union select * from largeI left outerjoin largeJ union select * from largeI right outerjoin largeJ On 7/4/13 8:08 PM, Franck Routier wrote:
Hi Martin,
Le 04/07/2013 19:04, Martin Kersten a écrit :
You might consider using a large number of columns using your own key, glued together with a view.
What do you mean by 'using your own key' ?
Best regards, Franck
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Thanks a lot Martin for your input. I will experiment this idea and I'll come back to share the results I get iwth MonetDB. Best regards, Franck Le 04/07/2013 20:12, Martin Kersten a écrit :
Use vertical decomposition schema design.
create table large( i integer, j integer)
can be represented by create table largeI( surrogate integer, i integer ); create table largeJ( surrogate integer, j integer );
create view large as select * from largeI join largeJ union select * from largeI left outerjoin largeJ union select * from largeI right outerjoin largeJ
On 7/4/13 8:08 PM, Franck Routier wrote:
Hi Martin,
Le 04/07/2013 19:04, Martin Kersten a écrit :
You might consider using a large number of columns using your own key, glued together with a view.
What do you mean by 'using your own key' ?
Best regards, Franck
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Le 04/07/2013 16:48, Stefan Manegold a écrit :
Le 04/07/2013 15:12, Stefan Manegold a écrit :
Please check what the monetdbd / merovingian log file says. Hum... /var/log/monetdb is empty. And I can't locate any monet* log file. It's called merovingian.log and is located in your dbfarm.
Here is the log. Crashed at 13h04 with SIGBUS. 2013-07-04 09:45:49 MSG merovingian[10480]: starting database 'axabas', up min/avg/max: 19h/23h/1d, crash average: 0.00 0.00 0.00 (3-3=0) 2013-07-04 09:45:50 MSG axabas[10607]: arguments: /usr/bin/mserver5 --dbpath=/data/monetdb_farm/axabas --set merovingian_uri=mapi:monetdb://dome:50000/axabas --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/monetdb_farm/axabas/.mapi.sock --set monet_vault_key=/data/monetdb_farm/axabas/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2013-07-04 09:45:50 MSG axabas[10607]: # MonetDB 5 server v11.15.11 "Feb2013-SP3" 2013-07-04 09:45:50 MSG axabas[10607]: # Serving database 'axabas', using 8 threads 2013-07-04 09:45:50 MSG axabas[10607]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2013-07-04 09:45:50 MSG axabas[10607]: # Found 31.399 GiB available main-memory. 2013-07-04 09:45:50 MSG axabas[10607]: # Copyright (c) 1993-July 2008 CWI. 2013-07-04 09:45:50 MSG axabas[10607]: # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved 2013-07-04 09:45:50 MSG axabas[10607]: # Visit http://www.monetdb.org/ for further information 2013-07-04 09:45:50 MSG axabas[10607]: # Listening for UNIX domain connection requests on mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock 2013-07-04 09:45:50 MSG axabas[10607]: # MonetDB/SQL module loaded 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35333 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35334 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35335 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35336 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35337 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:46:33 MSG merovingian[10480]: proxying client localhost:35343 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:46:33 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 13:04:55 MSG merovingian[10480]: database 'axabas' (10607) was killed by signal SIGBUS 2013-07-04 13:55:46 MSG control[10480]: (local): served status list 2013-07-04 13:55:46 MSG merovingian[10480]: database 'axabas' has crashed after start on 2013-07-04 09:45:50, attempting restart, up min/avg/max: 19h/23h/1d, crash average: 1.00 0.10 0.03 (4-3=1) 2013-07-04 13:55:47 MSG control[10480]: (local): started 'axabas' 2013-07-04 13:55:47 MSG axabas[29366]: arguments: /usr/bin/mserver5 --dbpath=/data/monetdb_farm/axabas --set merovingian_uri=mapi:monetdb://dome:50000/axabas --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/monetdb_farm/axabas/.mapi.sock --set monet_vault_key=/data/monetdb_farm/axabas/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2013-07-04 13:56:31 ERR merovingian[10480]: client error: unknown or impossible state: 4 2013-07-04 13:56:31 ERR merovingian[10480]: client error: unknown or impossible state: 4 2013-07-04 13:56:45 MSG control[10480]: (local): served status list 2013-07-04 13:56:52 ERR control[10480]: (local): failed to destroy database 'axabas': failed to destroy 'axabas': unable to remove directory /data/monetdb_farm/axabas/bat/01: Directory not empty
Most probably due to full disk. Stefan ----- Original Message -----
Le 04/07/2013 16:48, Stefan Manegold a écrit :
Le 04/07/2013 15:12, Stefan Manegold a écrit :
Please check what the monetdbd / merovingian log file says. Hum... /var/log/monetdb is empty. And I can't locate any monet* log file. It's called merovingian.log and is located in your dbfarm.
Here is the log. Crashed at 13h04 with SIGBUS.
2013-07-04 09:45:49 MSG merovingian[10480]: starting database 'axabas', up min/avg/max: 19h/23h/1d, crash average: 0.00 0.00 0.00 (3-3=0) 2013-07-04 09:45:50 MSG axabas[10607]: arguments: /usr/bin/mserver5 --dbpath=/data/monetdb_farm/axabas --set merovingian_uri=mapi:monetdb://dome:50000/axabas --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/monetdb_farm/axabas/.mapi.sock --set monet_vault_key=/data/monetdb_farm/axabas/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2013-07-04 09:45:50 MSG axabas[10607]: # MonetDB 5 server v11.15.11 "Feb2013-SP3" 2013-07-04 09:45:50 MSG axabas[10607]: # Serving database 'axabas', using 8 threads 2013-07-04 09:45:50 MSG axabas[10607]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2013-07-04 09:45:50 MSG axabas[10607]: # Found 31.399 GiB available main-memory. 2013-07-04 09:45:50 MSG axabas[10607]: # Copyright (c) 1993-July 2008 CWI. 2013-07-04 09:45:50 MSG axabas[10607]: # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved 2013-07-04 09:45:50 MSG axabas[10607]: # Visit http://www.monetdb.org/ for further information 2013-07-04 09:45:50 MSG axabas[10607]: # Listening for UNIX domain connection requests on mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock 2013-07-04 09:45:50 MSG axabas[10607]: # MonetDB/SQL module loaded 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35333 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35334 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35335 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35336 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35337 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:46:33 MSG merovingian[10480]: proxying client localhost:35343 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:46:33 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 13:04:55 MSG merovingian[10480]: database 'axabas' (10607) was killed by signal SIGBUS 2013-07-04 13:55:46 MSG control[10480]: (local): served status list 2013-07-04 13:55:46 MSG merovingian[10480]: database 'axabas' has crashed after start on 2013-07-04 09:45:50, attempting restart, up min/avg/max: 19h/23h/1d, crash average: 1.00 0.10 0.03 (4-3=1) 2013-07-04 13:55:47 MSG control[10480]: (local): started 'axabas' 2013-07-04 13:55:47 MSG axabas[29366]: arguments: /usr/bin/mserver5 --dbpath=/data/monetdb_farm/axabas --set merovingian_uri=mapi:monetdb://dome:50000/axabas --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/monetdb_farm/axabas/.mapi.sock --set monet_vault_key=/data/monetdb_farm/axabas/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2013-07-04 13:56:31 ERR merovingian[10480]: client error: unknown or impossible state: 4 2013-07-04 13:56:31 ERR merovingian[10480]: client error: unknown or impossible state: 4 2013-07-04 13:56:45 MSG control[10480]: (local): served status list 2013-07-04 13:56:52 ERR control[10480]: (local): failed to destroy database 'axabas': failed to destroy 'axabas': unable to remove directory /data/monetdb_farm/axabas/bat/01: Directory not empty
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
And perhaps not properly detecting end-of-input from the client side? On 7/5/13 11:16 AM, Stefan Manegold wrote:
Most probably due to full disk.
Stefan
----- Original Message -----
Le 04/07/2013 16:48, Stefan Manegold a écrit :
Le 04/07/2013 15:12, Stefan Manegold a écrit :
Please check what the monetdbd / merovingian log file says. Hum... /var/log/monetdb is empty. And I can't locate any monet* log file. It's called merovingian.log and is located in your dbfarm.
Here is the log. Crashed at 13h04 with SIGBUS.
2013-07-04 09:45:49 MSG merovingian[10480]: starting database 'axabas', up min/avg/max: 19h/23h/1d, crash average: 0.00 0.00 0.00 (3-3=0) 2013-07-04 09:45:50 MSG axabas[10607]: arguments: /usr/bin/mserver5 --dbpath=/data/monetdb_farm/axabas --set merovingian_uri=mapi:monetdb://dome:50000/axabas --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/monetdb_farm/axabas/.mapi.sock --set monet_vault_key=/data/monetdb_farm/axabas/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2013-07-04 09:45:50 MSG axabas[10607]: # MonetDB 5 server v11.15.11 "Feb2013-SP3" 2013-07-04 09:45:50 MSG axabas[10607]: # Serving database 'axabas', using 8 threads 2013-07-04 09:45:50 MSG axabas[10607]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2013-07-04 09:45:50 MSG axabas[10607]: # Found 31.399 GiB available main-memory. 2013-07-04 09:45:50 MSG axabas[10607]: # Copyright (c) 1993-July 2008 CWI. 2013-07-04 09:45:50 MSG axabas[10607]: # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved 2013-07-04 09:45:50 MSG axabas[10607]: # Visit http://www.monetdb.org/ for further information 2013-07-04 09:45:50 MSG axabas[10607]: # Listening for UNIX domain connection requests on mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock 2013-07-04 09:45:50 MSG axabas[10607]: # MonetDB/SQL module loaded 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35333 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35334 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35335 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35336 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:45:51 MSG merovingian[10480]: proxying client localhost:35337 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:45:51 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 09:46:33 MSG merovingian[10480]: proxying client localhost:35343 for database 'axabas' to mapi:monetdb:///data/monetdb_farm/axabas/.mapi.sock?database=axabas 2013-07-04 09:46:33 MSG merovingian[10480]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2013-07-04 13:04:55 MSG merovingian[10480]: database 'axabas' (10607) was killed by signal SIGBUS 2013-07-04 13:55:46 MSG control[10480]: (local): served status list 2013-07-04 13:55:46 MSG merovingian[10480]: database 'axabas' has crashed after start on 2013-07-04 09:45:50, attempting restart, up min/avg/max: 19h/23h/1d, crash average: 1.00 0.10 0.03 (4-3=1) 2013-07-04 13:55:47 MSG control[10480]: (local): started 'axabas' 2013-07-04 13:55:47 MSG axabas[29366]: arguments: /usr/bin/mserver5 --dbpath=/data/monetdb_farm/axabas --set merovingian_uri=mapi:monetdb://dome:50000/axabas --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/monetdb_farm/axabas/.mapi.sock --set monet_vault_key=/data/monetdb_farm/axabas/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2013-07-04 13:56:31 ERR merovingian[10480]: client error: unknown or impossible state: 4 2013-07-04 13:56:31 ERR merovingian[10480]: client error: unknown or impossible state: 4 2013-07-04 13:56:45 MSG control[10480]: (local): served status list 2013-07-04 13:56:52 ERR control[10480]: (local): failed to destroy database 'axabas': failed to destroy 'axabas': unable to remove directory /data/monetdb_farm/axabas/bat/01: Directory not empty
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Le 04/07/2013 15:12, Stefan Manegold a écrit :
ld be sufficient info for us to estimate the expected storage size. E.g., if all columns were 4 byt integers, your first fact table would take 295 M * 109 * 4 ~= 128 GB, while your second table would take In fact, this table, along with a bunch of smaller dimensions tables, takes around 70 GB in monetdb. This is somewhat less than the 86 GB taken in Postgresql (and somewhat more than the 17 GB it takes in VectorWise).
Franck
participants (4)
-
Franck Routier -
Martin Kersten -
Martin Kersten -
Stefan Manegold