On Tue, Mar 24, 2009 at 11:04:29AM -0700, Yue Sheng wrote:
Thanks for the that. Recreating would be the way to go to replicate the issues. The problem is that I'm on Mac Box and the data set is (which itself is quite simple; maybe we can use a few line as guide then replicate 600million times?) too big to send around (not to mention data licensing issues). I have generated data and I'm busy loading it. Ie later more
Niels
On Tue, Mar 24, 2009 at 10:46 AM, <[1]Milena.Ivanova@cwi.nl> wrote:
Melina, great to get confirmation that it is indeed possible to create large single table. I presume the DB6 is hosted on Linux?
Hi, I just wanted to point out that there are examples of MonetDB supporting large tables. I have to clarify that the DB was created in October 2008 using the current version of MonetDB at that time on 64bit Linux. Arjen de Rijke, who made the load, can give you the other details if needed. However, a lot of important features have been changed since then. As Niels suggested, the best would be to re-create your situation as close as possible to identify the cause of the problem. Milena
May I ask a few questions: (1) did you create the table from multiple files? If so, what is the average and maximum size of each of these files? (2) did you try to create the table in one session? (i.e. loop loading all necessary files over several hour for example) (3) did you use the multi-thread version of bulk load COPY... INTO? (4) what version of DB and Kernel did you use? Is it compiled in 64bit? (5) how much memory does the server have? (6) finally, did you run into any problem in the process?
Many thanks!
On Tue, Mar 24, 2009 at 3:38 AM, <[2]Milena.Ivanova@cwi.nl> wrote:
Hi, then I'm totally lost. That was my last resort in creating a big table.
So: (1) I can't create a 600-million table using COPY (because it hangs) (2) I can't create two 300-million tables and combined them into one (bug)
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support 600-million-by-10 records in a SINGLE table?
Hi, In SkyServer DR6 database of 2.5TB MonetDB supports tables longer than 600M rows, although with smaller number of columns. For instance: sql>select count(*) from photoprofile; +-------------+ | count_ | +=============+ | 13282921742 | +-------------+ 1 tuple sql>select * from photoprofile limit 1;
+-----+------+---------------+---------------+----------------- ---+
| bin | band | profmean | proferr | objid |
+=====+======+===============+===============+================= ===+
| 0 | 0 | 1.49587098e-0 | 6.95472624e-1 | 587722951693303809 | : : : 8 : 0 : :
+-----+------+---------------+---------------+----------------- ---+
where 2 columns are tinyint, 2 - real and 1 -bigint. The
loaded by 'copy .. into ..' statement. Milena
Thanks.
On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes <[3]Niels.Nes@cwi.nl> wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined. NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both
The way you explain it, it seems a bug.
Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng <[1][4]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the
(2) The combined table is the size of table_1 +
only table_1 shows up in query On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2][5]lsidir@gmail.com> wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from
table_2? If not, then I would guess that this is a bug and it
needs
further investigation. lefteris
On Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng <[3][6]yuesheng8@gmail.com> wrote: > two tables: table_1 and table_2 both have exactly
schema. table_1 > has 2.15MM rows of data, table_2 has 3.35MM row of data. No indexing > created a new table: table_combined using same schema as table_1 > insert into table_combined select * from table_1; > (30 minutes later) > insert into table_combined select * from table_2; > > (40 minutes later) > select count(*) * 1 from table_combined; > this give ONLY 2.15MM rows with same rows as
> What happened to table_2?!? > Yes, I've check dfarm size. It equals 2*(table_1 +
table was table_1 and table. table_2. But table_1 and the same table_1 table_2 )
> So it's there but "just not seeing it" > > > > >
>
------------------ > Apps built with the Adobe(R) Flex(R) framework
and Flex
Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and
step-through
debugging. > Download the free 60 day trial. [4][7]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [5][8]MonetDB-users@lists.sourceforge.net >
[6][9]https://lists.sourceforge.net/lists/listinfo/monetdb-user
s > >
------------------ Apps built with the Adobe(R) Flex(R) framework and
Flex
Builder(TM) are powering Web 2.0 with engaging, cross-platform
capabilities.
Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and
step-through
debugging. Download the free 60 day trial. [7][10]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [8][11]MonetDB-users@lists.sourceforge.net
[9][12]https://lists.sourceforge.net/lists/listinfo/monetdb-use r
s
References
1. mailto:[13]yuesheng8@gmail.com 2. mailto:[14]lsidir@gmail.com 3. mailto:[15]yuesheng8@gmail.com 4. [16]http://p.sf.net/sfu/www-adobe-com 5. mailto:[17]MonetDB-users@lists.sourceforge.net 6.
[18]https://lists.sourceforge.net/lists/listinfo/monetdb-users
7. [19]http://p.sf.net/sfu/www-adobe-com 8. mailto:[20]MonetDB-users@lists.sourceforge.net 9. [21]https://lists.sourceforge.net/lists/listinfo/monetdb-users
---------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [22]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [23]MonetDB-users@lists.sourceforge.net
[24]https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: [25]http://www.cwi.nl/~niels e-mail: [26]Niels.Nes@cwi.nl
---------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [27]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [28]MonetDB-users@lists.sourceforge.net
[29]https://lists.sourceforge.net/lists/listinfo/monetdb-users
---------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial.
[30]http://p.sf.net/sfu/www-adobe-com__________________________
MonetDB-users mailing list [31]MonetDB-users@lists.sourceforge.net
[32]https://lists.sourceforge.net/lists/listinfo/monetdb-users
---------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [33]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [34]MonetDB-users@lists.sourceforge.net
[35]https://lists.sourceforge.net/lists/listinfo/monetdb-users
--------------------------------------------------------------- ---------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial.
[36]http://p.sf.net/sfu/www-adobe-com__________________________ _____________________
MonetDB-users mailing list [37]MonetDB-users@lists.sourceforge.net
[38]https://lists.sourceforge.net/lists/listinfo/monetdb-users
--------------------------------------------------------------- --------------- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [39]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [40]MonetDB-users@lists.sourceforge.net [41]https://lists.sourceforge.net/lists/listinfo/monetdb-users
References
1. mailto:Milena.Ivanova@cwi.nl 2. mailto:Milena.Ivanova@cwi.nl 3. mailto:Niels.Nes@cwi.nl 4. mailto:yuesheng8@gmail.com 5. mailto:lsidir@gmail.com 6. mailto:yuesheng8@gmail.com 7. http://p.sf.net/sfu/www-adobe-com 8. mailto:MonetDB-users@lists.sourceforge.net 9. https://lists.sourceforge.net/lists/listinfo/monetdb-user 10. http://p.sf.net/sfu/www-adobe-com 11. mailto:MonetDB-users@lists.sourceforge.net 12. https://lists.sourceforge.net/lists/listinfo/monetdb-user 13. mailto:yuesheng8@gmail.com 14. mailto:lsidir@gmail.com 15. mailto:yuesheng8@gmail.com 16. http://p.sf.net/sfu/www-adobe-com 17. mailto:MonetDB-users@lists.sourceforge.net 18. https://lists.sourceforge.net/lists/listinfo/monetdb-users 19. http://p.sf.net/sfu/www-adobe-com 20. mailto:MonetDB-users@lists.sourceforge.net 21. https://lists.sourceforge.net/lists/listinfo/monetdb-users 22. http://p.sf.net/sfu/www-adobe-com 23. mailto:MonetDB-users@lists.sourceforge.net 24. https://lists.sourceforge.net/lists/listinfo/monetdb-users 25. http://www.cwi.nl/~niels 26. mailto:Niels.Nes@cwi.nl 27. http://p.sf.net/sfu/www-adobe-com 28. mailto:MonetDB-users@lists.sourceforge.net 29. https://lists.sourceforge.net/lists/listinfo/monetdb-users 30. http://p.sf.net/sfu/www-adobe-com___________________________________________... 31. mailto:MonetDB-users@lists.sourceforge.net 32. https://lists.sourceforge.net/lists/listinfo/monetdb-users 33. http://p.sf.net/sfu/www-adobe-com 34. mailto:MonetDB-users@lists.sourceforge.net 35. https://lists.sourceforge.net/lists/listinfo/monetdb-users 36. http://p.sf.net/sfu/www-adobe-com___________________________________________... 37. mailto:MonetDB-users@lists.sourceforge.net 38. https://lists.sourceforge.net/lists/listinfo/monetdb-users 39. http://p.sf.net/sfu/www-adobe-com 40. mailto:MonetDB-users@lists.sourceforge.net 41. https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl