With interest i follow the discussion of these threads. Hardware is just one of the many ingredients to consider. The loading method (staged/single shot), the correctness of the input, the load on the machine,....and many other issues play a role. Furthermore, specifics of the reports can not trigger an action here. They lack the details a system administrator and database administrator need to isolate a problem in the system or the way in which it is used. What does "getting stuck" mean? System crashed? Operating system shows a cpu load of <3%? iostat shows no or excessive IO? What does 'top' tell? is the process priority punished by your OS? Can you connect to the serve with mclient -lsql still? Does it work with single threaded server instance (--set gdk-nr-threads=1)? All standard questions for the DBA to isolate the issue. The report of Dariuz is a step into the direction. In that case i would ask for the predicted load time based on a multi-step experiment, e.g. load 10M, 20M 40M, first. Second, the size in relationship to main memory and swapspace would be investigated. And, finally, i would run here with single threaded to isolate/ detect a possible internal concurrency conflict. regards, Martin Yue Sheng wrote:
I have 32gb, 8core, macpro, 2tb harddisk. So hardware wise pretty top-end I guess. In my case, we thought it might be a Mac issue. Having said that, we tried the same insert on a Linux box, and it worked. So it might be a Mac issue. Interesting to hear you have very similar problem on non-Mac box...
On Wed, Apr 8, 2009 at 11:19 AM, dariuszs <dariuszs@svnp.com <mailto:dariuszs@svnp.com>> wrote:
Hi, How much memory do you have? Dariusz.
Yue Sheng wrote: > I have similar problem when loading large dataset. I'm trying to load > 650million by 10. It gets stuck after ~350million. Interestingly not > always at the same point... > > On Wed, Apr 8, 2009 at 6:10 AM, dariuszs <dariuszs@svnp.com <mailto:dariuszs@svnp.com> > <mailto:dariuszs@svnp.com <mailto:dariuszs@svnp.com>>> wrote: > > Hi, > Can anybody help me with this issue? Thanks. Dariusz. > > > > dariuszs wrote: > > Hi, > > Specs: > > MonetDB Server 5.10.0, based on kernel v1.28.0 > > Windows x64 Server 2003 Enterprise > > Intel Dual Xenon 5470 server, SAS drives, 48GB of memory / 220GB > swap > > space > > > > CREATE TABLE xxx( > > C1 VARCHAR(16), > > C2 VARCHAR(15), > > C3 VARCHAR(1), > > C4 VARCHAR(4), > > C5 VARCHAR(1), > > C6 VARCHAR(10), > > C7 VARCHAR(2), > > C8 VARCHAR(1), > > C9 VARCHAR(1), > > C10 VARCHAR(1), > > C11 VARCHAR(1), > > C12 VARCHAR(1), > > C13 VARCHAR(2), > > C14 VARCHAR(1), > > C15 VARCHAR(2), > > C16 VARCHAR(30), > > C17 VARCHAR(16), > > C18 VARCHAR(2), > > C19 VARCHAR(5), > > C20 VARCHAR(4), > > C21 VARCHAR(3), > > C22 VARCHAR(2), > > C23 VARCHAR(1), > > C24 VARCHAR(2), > > C25 VARCHAR(1), > > C26 VARCHAR(2), > > C27 VARCHAR(1), > > C28 VARCHAR(2), > > C29 VARCHAR(3), > > C30 VARCHAR(4), > > C31 VARCHAR(10), > > C32 VARCHAR(1), > > C33 VARCHAR(2), > > C34 VARCHAR(4), > > C35 VARCHAR(1), > > C36 VARCHAR(1), > > C37 VARCHAR(6), > > C38 VARCHAR(6), > > C39 VARCHAR(2), > > C40 VARCHAR(6), > > C41 VARCHAR(1), > > C42 VARCHAR(6), > > C43 VARCHAR(2), > > C44 VARCHAR(1), > > C45 VARCHAR(1), > > C46 VARCHAR(1), > > C47 VARCHAR(1), > > C48 VARCHAR(1), > > C49 VARCHAR(1), > > C50 VARCHAR(1), > > C51 VARCHAR(2), > > C52 VARCHAR(6), > > C53 VARCHAR(2), > > C54 VARCHAR(2), > > C55 VARCHAR(6), > > C56 VARCHAR(2), > > C57 VARCHAR(1), > > C58 VARCHAR(1), > > C59 VARCHAR(1), > > C60 VARCHAR(1), > > C61 VARCHAR(1), > > C62 VARCHAR(1), > > C63 VARCHAR(1), > > C64 VARCHAR(4), > > C65 VARCHAR(1), > > C66 VARCHAR(1), > > C67 VARCHAR(1), > > C68 VARCHAR(3), > > C69 VARCHAR(2), > > C70 VARCHAR(2), > > C71 VARCHAR(1), > > C72 VARCHAR(1), > > C73 VARCHAR(1), > > C74 VARCHAR(1), > > C75 VARCHAR(2), > > C76 VARCHAR(8), > > C77 VARCHAR(8), > > C78 VARCHAR(8), > > C79 VARCHAR(1), > > C80 VARCHAR(1), > > C81 VARCHAR(1), > > C82 VARCHAR(1), > > C83 VARCHAR(6), > > C84 VARCHAR(6), > > C85 VARCHAR(6), > > C86 VARCHAR(1), > > C87 VARCHAR(1), > > C88 VARCHAR(1), > > C89 VARCHAR(1), > > C90 VARCHAR(1), > > C91 VARCHAR(1), > > C92 VARCHAR(1), > > C93 VARCHAR(1), > > C94 VARCHAR(1), > > C95 VARCHAR(1), > > C96 VARCHAR(1), > > C97 VARCHAR(1), > > C98 VARCHAR(1), > > C99 VARCHAR(1), > > C100 VARCHAR(1), > > C101 VARCHAR(1), > > C102 VARCHAR(1), > > C103 VARCHAR(1), > > C104 VARCHAR(4), > > C105 VARCHAR(4), > > C106 VARCHAR(6), > > C107 VARCHAR(8), > > C108 VARCHAR(8), > > C109 VARCHAR(1), > > C110 VARCHAR(1), > > C111 VARCHAR(8), > > C112 VARCHAR(6), > > C113 VARCHAR(1), > > C114 VARCHAR(1), > > C115 VARCHAR(1), > > C116 VARCHAR(1), > > C117 VARCHAR(1), > > C118 VARCHAR(1), > > C119 VARCHAR(1), > > C120 VARCHAR(3), > > C121 VARCHAR(1), > > C122 VARCHAR(1), > > C123 VARCHAR(1), > > C124 VARCHAR(6), > > C125 VARCHAR(6), > > C126 VARCHAR(1), > > C127 VARCHAR(1), > > C128 VARCHAR(1), > > C129 VARCHAR(22), > > C130 VARCHAR(27)); > > copy 160000000 records into xxx from 'm:/monetdb_data/xxx.txt'; > > commit; > > > > OK, so the server loads the file for about 4 hours and I see CPU > > activity, I see memory usage going up (47GB) and down several times, > > but then after 4 hours everything sits - also I see that server > > creates a files in the sql_logs directory 'log.179' which grew up to > > 47GB and then it stopped. > > > > I was able to successfully load 130 million records using the same > > file and the same system in about 3 hours. Thanks. Dariusz. > > > > > > > > > > Stefan Manegold wrote: > >> Which version of MonetDB? > >> Which OS? > >> Which HW arcitecture? > >> Which table schema? > >> What happens during "sits" (CPU bound, I/O bound, ...)? > >> How do you "load"? > >> > >> Stefan > >> > >> On Fri, Apr 03, 2009 at 02:40:10PM -0400, dariuszs wrote: > >> > >>> Hi, > >>> I've got a table with 130 fields. When I load 130 million records > >>> into that table (about 50G) - it's not a problem, but when I > try to > >>> load 160 million records (about 60G of data) into it this > thing just > >>> sits for hours with no error messages. Can somebody help me? > >>> Thanks. Dariusz. > >>> > >>> > >>> > >>> > ------------------------------------------------------------------------------ > >>> > >>> _______________________________________________ > >>> MonetDB-users mailing list > >>> MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net> > <mailto:MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net>> > >>> https://lists.sourceforge.net/lists/listinfo/monetdb-users > >>> > >>> > >> > >> > > > > > > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > http://p.sf.net/sfu/www-ibm-com > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net> > <mailto:MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net>> > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > http://p.sf.net/sfu/www-ibm-com > ------------------------------------------------------------------------ > > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/monetdb-users >
------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------
------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com
------------------------------------------------------------------------
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users