- users-list - monetdb.org

users-list post from ajdamico@gmail.com requires approval
by users-list-owner＠monetdb.org 28 Sep '12

28 Sep '12

As list administrator, your authorization is requested for the following mailing list posting: List: users-list(a)monetdb.org From: ajdamico(a)gmail.com Subject: invalid tuple received from server, got 138 columns, expected 476, ignoring Reason: Post by non-member to a members-only list At your convenience, visit: http://mail.monetdb.org/mailman/admindb/users-list to approve or deny the request.

1 0

problem starting database
by Peter 28 Sep '12

28 Sep '12

Hello, I'm seeing this error when starting a database. Not sure if anyone has had the same issue and whether anyone had a fix? The following is what I did % monetdbd start ~/dbfarm % monetdbd start ~/dbfarm % monetdb create voc % monetdb start voc starting database 'voc'... FAILED start: starting 'voc' failed: database 'voc' appears to shut itself down after starting, check monetdbd's logfile for possible hints The log says the following 2012-09-27 16:56:04 MSG merovingian[14761]: database 'voc' (14801) has exited with exit status 1 2012-09-27 16:56:04 ERR control[14761]: (local): failed to fork mserver: database 'voc' appears to shut itself down after starting, check monetdbd's logfile for possible hints Any comments / suggestions? Thank you, Peter _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

3 7

[MonetDB-users] Array database for large matrix
by Joshua Shuai Yuan 27 Sep '12

27 Sep '12

Hi All, I just tried monetdb with large tables (600,000 rows by 50,000 columns). There are two issues: 1. performance was low, it took about 4 seconds to insert a row of record when the number of columns was large (50,000). However it was pretty fast (about 100 rows per second) when the column number was small (for example, 10). 2. serious errors occurred very often. For example, after inserting random number of records (50, for example), monetdb started to report errors, like: monetdbd: internal error while starting mserver, please refer to the logs So I opened the log file, and the error was: 2012-09-14 08:26:36 ERR merovingian[13730]: client error: unknown or impossible state: 4 The Makefile (it is also contained in the attachments) listed below will start database server, create a database farm, create a database, create a big table and then insert records to the big table. Here it is: #=====start of Makefile===================== ROW_NUM=600000 COLUMN_NUM=50000 COLUMN_NUM_MINUS_ONE=$(shell echo $(COLUMN_NUM)-1 | bc) all: clean monetdbd create ./mydbfarm monetdbd start ./mydbfarm monetdb create test_big_db monetdb release test_big_db make create_table_sql -s > create_big_table.sql make insert_data_sql -s > insert_data.sql mclient -d test_big_db create_big_table.sql for i in $$(seq 1 $(ROW_NUM)); do \ mclient -d test_big_db insert_data.sql ;\ echo 'done inserting ' $$i ' rows'; \ done; mclient -d test_big_db -s "select * from test_big_table" > test_big_table.txt create_table_sql: @echo 'START TRANSACTION;' @echo 'CREATE TABLE "test_big_table" (' @for i in $$(seq 1 $(COLUMN_NUM_MINUS_ONE)); do \ echo ' "gen'$$i'" DOUBLE,'; \ done @echo ' "gen$(COLUMN_NUM)" DOUBLE' @echo ');' @echo 'COMMIT;' insert_data_sql: @echo -n 'INSERT INTO "test_big_table" VALUES (' ; @for j in $$(seq 1 $(COLUMN_NUM_MINUS_ONE)) ; do \ echo -n '0.323, '; \ done; @echo '0.324);'; insert_all_data_sql: @echo 'START TRANSACTION;' @for i in $$(seq 1 $(ROW_NUM)); do\ echo -n 'INSERT INTO "test_big_table" VALUES (' ;\ for j in $$(seq 1 $(COLUMN_NUM_MINUS_ONE)) ; do \ echo -n '0.323, '; \ done; \ echo '0.324);'; \ done @echo 'COMMIT;' clean: -killall mserver5 -killall monetdbd rm -rf ./mydbfarm rm -rf create_big_table.sql test_big_table.txt insert_data.sql #=====end of Makefile====================== Thank you very much ! Best wishes, Joshua Shuai Yuan On Thu, Sep 13, 2012 at 12:43 PM, Joshua Shuai Yuan < shuaiyuan.emory(a)gmail.com> wrote: > Sure, I will see what I can do. > > Best wishes, > Joshua Shuai Yuan > > > On Wed, Sep 12, 2012 at 11:40 PM, Wang, Fusheng <fusheng.wang(a)emory.edu>wrote: > >> Hi Joshua,**** >> >> ** ** >> >> The table we are managing is in the scale of 30K x 5K: 150M cells. If >> each table cell (double type) needs 8 bytes to represent (ideally), the >> space needed will be a couple of GB, or at most, in the scale of tens of >> GB. If that is the case, distributed setup may not be needed, as the data >> can be mostly cached by the database. MonetDB does support multi-cores >> and multi-disks, but setup across multiple machines is not supported based >> on my knowledge. **** >> >> ** ** >> >> Even though MonetDB claims unlimited number of columns, we should still >> be cautious on what performance we can achieve for the queries we want to >> provide. A pilot study on this could provide us some guideline. Do you >> think you can setup MonetDB, and create a benchmark table we can do some >> performance study?**** >> >> ** ** >> >> Interestingly, the large column issue is also discussed in a famous >> database blog:**** >> >> >> http://www.dbms2.com/2011/03/13/so-how-many-columns-can-a-single-table-have… >> **** >> >> ** ** >> >> One guy commented:**** >> >> ** ** >> >> “Genomics models were the primary driver. these folks typically have >> short but wide datasets of the order of 300,000 columns.”**** >> >> ** ** >> >> So we are not alone. And we have only a subset of the columns (30K >> versus 300K)? Then we are lucky!**** >> >> ** ** >> >> ** ** >> >> Fusheng**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> *From:* Joshua Shuai Yuan [mailto:shuaiyuan.emory@gmail.com] >> *Sent:* Wednesday, September 12, 2012 11:14 PM >> *To:* Wang, Fusheng >> *Cc:* Qin, Zhaohui >> *Subject:* Re: Array database for large matrix**** >> >> ** ** >> >> Hi Dr. Wang,**** >> >> That's really a good news. Does it support distributed database? Or do >> we need distributed one?**** >> >> ** ** >> >> >> Best wishes, >> Joshua Shuai Yuan >> >> **** >> >> On Wed, Sep 12, 2012 at 3:54 PM, Wang, Fusheng <fusheng.wang(a)emory.edu> >> wrote:**** >> >> Hi guys,**** >> >> **** >> >> It looks like the matrix structure can be nicely supported by array >> databases, like MonetDB. It supports unlimited number of columns for a >> table. It’s also open source. **** >> >> http://www.monetdb.org/Home/Features**** >> >> **** >> >> I know the group quite well, and I will chat with them on the use case to >> see if it’s a good fit. **** >> >> **** >> >> Thanks,**** >> >> **** >> >> Fusheng**** >> >> **** >> >> **** >> >> ** ** >> ------------------------------ >> >> >> This e-mail message (including any attachments) is for the sole use of >> the intended recipient(s) and may contain confidential and privileged >> information. If the reader of this message is not the intended >> recipient, you are hereby notified that any dissemination, distribution >> or copying of this message (including any attachments) is strictly >> prohibited. >> >> If you have received this message in error, please contact >> the sender by reply e-mail message and destroy all copies of the >> original message (including attachments).**** >> >> ** ** >> > >

2 2

[MonetDB-users] MonetDB project will move to monetdb.org
by Fabian Groffen 26 Sep '12

26 Sep '12

We would like to inform you that as of Januari 2013, the monetdb project on SourceForge will be removed. All services that SourceForge has kindly provided us with in the past will be migrated to our monetdb.org infrastructure. Currently, the only remaining services hosted on SourceForge are the three mailing lists monetdb-announce, monetdb-users and monetdb-developers. As of this message, these lists are migrated to their equivalents on monetdb.org. The new lists are as follows: monetdb-announce(a)lists.sourceforge.net -> announce-list(a)monetdb.org monetdb-users(a)lists.sourceforge.net -> users-list(a)monetdb.org monetdb-developers(a)lists.sourceforge.net -> developers-list(a)monetdb.org We invite you to subscribe to the new lists on monetdb.org at your earliest convenience. Any posts to the old lists are rejected from now on. Posts to the new lists at monetdb.org are forwarded in batches to the old lists until the end of this year. The MonetDB team would like to take this opportunity to thank SourceForge for the services provided over the past years. We hope to welcome all current subscribers on our new lists at monetdb.org, see: http://mail.monetdb.org/mailman/listinfo The MonetDB team

1 0

[MonetDB-users] invalid tuple received from server, got 138 columns, expected 476, ignoring
by Anthony Damico 26 Sep '12

26 Sep '12

Hello, I'm pretty new to MonetDB and I'm trying to perform what seems like it should be a relatively straightforward inner join between two tables. I would like to merge the two Alaska files from the American Community Survey -- http://www2.census.gov/acs2010_1yr/pums/csv_hak.zip contains a CSV file with one record per household http://www2.census.gov/acs2010_1yr/pums/csv_pak.zip contains a CSV file with one record per personThese two files should be merged on the field "serialno" I am working on Windows 7 x64. Here's the contents of my mserver5.exe window: # MonetDB 5 server v11.11.11 "Jul2012-SP2" # Serving database 'demo', using 2 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 3.905 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module loaded > The following commands.. 1) create a person-level table 2) create a household-level table 3) read the person-level CSV into the person table 4) read the household-level CSV into the household table 5) attempt to print the first five records with "select * from tablename limit 5" -- sometimes these work, sometimes they don't. 6) merge the two tables together 7) attempt to print the first five records of the merged table Unfortunately, I'm getting an error that I can't find with web searches.. The only place I can even find this error is http://bugs.monetdb.org/show_bug.cgi?id=2963 -- and that seems both time-related and fixed long ago.. I doubt it's a bug, I'm probably just making a beginner's mistake. invalid tuple received from server, got 138 columns, expected 476, ignoring +------+------+------+------+-------+------+---------+---------+------+------+ 1 tuple (417.984ms) !466 columns dropped! note: to disable dropping columns and/or truncating fields use \w-1 Even after setting \w to -1, the number of columns returned gets limited, crashing my program. I am ultimately trying to work with the new R/MonetDB package (which uses Java and is available from http://sqlsurvey.r-forge.r-project.org/) but none of the table interactivity will work until I'm able to return all of the columns properly. Any advice would be appreciated! Thanks!!! Anthony Damico Kaiser Family Foundation Here are the SQL commands: CREATE TABLE person (RT VARCHAR(255), SERIALNO INT, SPORDER INT, PUMA INT, ST INT, ADJINC INT, PWGTP INT, AGEP INT, CIT INT, CITWP INT, COW INT, DDRS INT, DEAR INT, DEYE INT, DOUT INT, DPHY INT, DRAT INT, DRATX INT, DREM INT, ENG INT, FER INT, GCL INT, GCM INT, GCR INT, HINS1 INT, HINS2 INT, HINS3 INT, HINS4 INT, HINS5 INT, HINS6 INT, HINS7 INT, INTP INT, JWMNP INT, JWRIP INT, JWTR INT, LANX INT, MAR INT, MARHD INT, MARHM INT, MARHT INT, MARHW INT, MARHYP INT, MIG INT, MIL INT, MLPA INT, MLPB INT, MLPC INT, MLPD INT, MLPE INT, MLPF INT, MLPG INT, MLPH INT, MLPI INT, MLPJ INT, MLPK INT, NWAB INT, NWAV INT, NWLA INT, NWLK INT, NWRE INT, OIP INT, PAP INT, RELP INT, RETP INT, SCH INT, SCHG INT, SCHL INT, SEMP INT, SEX INT, SSIP INT, SSP INT, WAGP INT, WKHP INT, WKL INT, WKW INT, WRK INT, YOEP INT, ANC INT, ANC1P INT, ANC2P INT, DECADE INT, DIS INT, DRIVESP INT, ESP INT, ESR INT, FOD1P INT, FOD2P INT, HICOV INT, HISP INT, INDP INT, JWAP INT, JWDP INT, LANP INT, MIGPUMA INT, MIGSP INT, MSP INT, NAICSP VARCHAR(255), NATIVITY INT, NOP INT, OC INT, OCCP INT, PAOC INT, PERNP INT, PINCP INT, POBP INT, POVPIP INT, POWPUMA INT, POWSP INT, PRIVCOV INT, PUBCOV INT, QTRBIR INT, RAC1P INT, RAC2P INT, RAC3P INT, RACAIAN INT, RACASN INT, RACBLK INT, RACNHPI INT, RACNUM INT, RACSOR INT, RACWHT INT, RC INT, SCIENGP INT, SCIENGRLP INT, SFN INT, SFR INT, SOCP VARCHAR(255), VPS INT, WAOB INT, FAGEP INT, FANCP INT, FCITP INT, FCITWP INT, FCOWP INT, FDDRSP INT, FDEARP INT, FDEYEP INT, FDOUTP INT, FDPHYP INT, FDRATP INT, FDRATXP INT, FDREMP INT, FENGP INT, FESRP INT, FFERP INT, FFODP INT, FGCLP INT, FGCMP INT, FGCRP INT, FHINS1P INT, FHINS2P INT, FHINS3C INT, FHINS3P INT, FHINS4C INT, FHINS4P INT, FHINS5C INT, FHINS5P INT, FHINS6P INT, FHINS7P INT, FHISP INT, FINDP INT, FINTP INT, FJWDP INT, FJWMNP INT, FJWRIP INT, FJWTRP INT, FLANP INT, FLANXP INT, FMARHDP INT, FMARHMP INT, FMARHTP INT, FMARHWP INT, FMARHYP INT, FMARP INT, FMIGP INT, FMIGSP INT, FMILPP INT, FMILSP INT, FOCCP INT, FOIP INT, FPAP INT, FPOBP INT, FPOWSP INT, FRACP INT, FRELP INT, FRETP INT, FSCHGP INT, FSCHLP INT, FSCHP INT, FSEMP INT, FSEXP INT, FSSIP INT, FSSP INT, FWAGP INT, FWKHP INT, FWKLP INT, FWKWP INT, FWRKP INT, FYOEP INT, pwgtp1 INT, pwgtp2 INT, pwgtp3 INT, pwgtp4 INT, pwgtp5 INT, pwgtp6 INT, pwgtp7 INT, pwgtp8 INT, pwgtp9 INT, pwgtp10 INT, pwgtp11 INT, pwgtp12 INT, pwgtp13 INT, pwgtp14 INT, pwgtp15 INT, pwgtp16 INT, pwgtp17 INT, pwgtp18 INT, pwgtp19 INT, pwgtp20 INT, pwgtp21 INT, pwgtp22 INT, pwgtp23 INT, pwgtp24 INT, pwgtp25 INT, pwgtp26 INT, pwgtp27 INT, pwgtp28 INT, pwgtp29 INT, pwgtp30 INT, pwgtp31 INT, pwgtp32 INT, pwgtp33 INT, pwgtp34 INT, pwgtp35 INT, pwgtp36 INT, pwgtp37 INT, pwgtp38 INT, pwgtp39 INT, pwgtp40 INT, pwgtp41 INT, pwgtp42 INT, pwgtp43 INT, pwgtp44 INT, pwgtp45 INT, pwgtp46 INT, pwgtp47 INT, pwgtp48 INT, pwgtp49 INT, pwgtp50 INT, pwgtp51 INT, pwgtp52 INT, pwgtp53 INT, pwgtp54 INT, pwgtp55 INT, pwgtp56 INT, pwgtp57 INT, pwgtp58 INT, pwgtp59 INT, pwgtp60 INT, pwgtp61 INT, pwgtp62 INT, pwgtp63 INT, pwgtp64 INT, pwgtp65 INT, pwgtp66 INT, pwgtp67 INT, pwgtp68 INT, pwgtp69 INT, pwgtp70 INT, pwgtp71 INT, pwgtp72 INT, pwgtp73 INT, pwgtp74 INT, pwgtp75 INT, pwgtp76 INT, pwgtp77 INT, pwgtp78 INT, pwgtp79 INT, pwgtp80 INT); CREATE TABLE household (RT VARCHAR(255), SERIALNO INT, DIVISION INT, PUMA INT, REGION INT, ST INT, ADJHSG INT, ADJINC INT, WGTP INT, NP INT, TYPE INT, ACR INT, AGS INT, BATH INT, BDSP INT, BLD INT, BUS INT, CONP INT, ELEP INT, FS INT, FULP INT, GASP INT, HFL INT, INSP INT, MHP INT, MRGI INT, MRGP INT, MRGT INT, MRGX INT, REFR INT, RMSP INT, RNTM INT, RNTP INT, RWAT INT, SINK INT, SMP INT, STOV INT, TEL INT, TEN INT, TOIL INT, VACS INT, VALP INT, VEH INT, WATP INT, YBL INT, FES INT, FINCP INT, FPARC INT, GRNTP INT, GRPIP INT, HHL INT, HHT INT, HINCP INT, HUGCL INT, HUPAC INT, HUPAOC INT, HUPARC INT, KIT INT, LNGI INT, MULTG INT, MV INT, NOC INT, NPF INT, NPP INT, NR INT, NRC INT, OCPIP INT, PARTNER INT, PLM INT, PSF INT, R18 INT, R60 INT, R65 INT, RESMODE INT, SMOCP INT, SMX INT, SRNT INT, SVAL INT, TAXP INT, WIF INT, WKEXREL INT, WORKSTAT INT, FACRP INT, FAGSP INT, FBATHP INT, FBDSP INT, FBLDP INT, FBUSP INT, FCONP INT, FELEP INT, FFSP INT, FFULP INT, FGASP INT, FHFLP INT, FINSP INT, FKITP INT, FMHP INT, FMRGIP INT, FMRGP INT, FMRGTP INT, FMRGXP INT, FMVP INT, FPLMP INT, FREFRP INT, FRMSP INT, FRNTMP INT, FRNTP INT, FRWATP INT, FSINKP INT, FSMP INT, FSMXHP INT, FSMXSP INT, FSTOVP INT, FTAXP INT, FTELP INT, FTENP INT, FTOILP INT, FVACSP INT, FVALP INT, FVEHP INT, FWATP INT, FYBLP INT, wgtp1 INT, wgtp2 INT, wgtp3 INT, wgtp4 INT, wgtp5 INT, wgtp6 INT, wgtp7 INT, wgtp8 INT, wgtp9 INT, wgtp10 INT, wgtp11 INT, wgtp12 INT, wgtp13 INT, wgtp14 INT, wgtp15 INT, wgtp16 INT, wgtp17 INT, wgtp18 INT, wgtp19 INT, wgtp20 INT, wgtp21 INT, wgtp22 INT, wgtp23 INT, wgtp24 INT, wgtp25 INT, wgtp26 INT, wgtp27 INT, wgtp28 INT, wgtp29 INT, wgtp30 INT, wgtp31 INT, wgtp32 INT, wgtp33 INT, wgtp34 INT, wgtp35 INT, wgtp36 INT, wgtp37 INT, wgtp38 INT, wgtp39 INT, wgtp40 INT, wgtp41 INT, wgtp42 INT, wgtp43 INT, wgtp44 INT, wgtp45 INT, wgtp46 INT, wgtp47 INT, wgtp48 INT, wgtp49 INT, wgtp50 INT, wgtp51 INT, wgtp52 INT, wgtp53 INT, wgtp54 INT, wgtp55 INT, wgtp56 INT, wgtp57 INT, wgtp58 INT, wgtp59 INT, wgtp60 INT, wgtp61 INT, wgtp62 INT, wgtp63 INT, wgtp64 INT, wgtp65 INT, wgtp66 INT, wgtp67 INT, wgtp68 INT, wgtp69 INT, wgtp70 INT, wgtp71 INT, wgtp72 INT, wgtp73 INT, wgtp74 INT, wgtp75 INT, wgtp76 INT, wgtp77 INT, wgtp78 INT, wgtp79 INT, wgtp80 INT); copy 50000 offset 2 records into person from 'C:\Users\user\desktop\ss10pak.csv' using delimiters ',','\n','"' NULL as '' ; copy 25000 offset 2 records into household from 'C:\Users\user\desktop\ss10hak.csv' using delimiters ',','\n','"' NULL as '' ; select * from person limit 5; select * from household limit 5; create table merged as select 'M' as rt, a.serialno, a.division, a.puma, a.region, a.st, a.adjhsg, a.adjinc, a.wgtp, a.np, a.type, a.acr, a.ags, a.bath, a.bdsp, a.bld, a.bus, a.conp, a.elep, a.fs, a.fulp, a.gasp, a.hfl, a.insp, a.mhp, a.mrgi, a.mrgp, a.mrgt, a.mrgx, a.refr, a.rmsp, a.rntm, a.rntp, a.rwat, a.sink, a.smp, a.stov, a.tel, a.ten, a.toil, a.vacs, a.valp, a.veh, a.watp, a.ybl, a.fes, a.fincp, a.fparc, a.grntp, a.grpip, a.hhl, a.hht, a.hincp, a.hugcl, a.hupac, a.hupaoc, a.huparc, a.kit, a.lngi, a.multg, a.mv, a.noc, a.npf, a.npp, a.nr, a.nrc, a.ocpip, a.partner, a.plm, a.psf, a.r18, a.r60, a.r65, a.resmode, a.smocp, a.smx, a.srnt, a.sval, a.taxp, a.wif, a.wkexrel, a.workstat, a.facrp, a.fagsp, a.fbathp, a.fbdsp, a.fbldp, a.fbusp, a.fconp, a.felep, a.ffsp, a.ffulp, a.fgasp, a.fhflp, a.finsp, a.fkitp, a.fmhp, a.fmrgip, a.fmrgp, a.fmrgtp, a.fmrgxp, a.fmvp, a.fplmp, a.frefrp, a.frmsp, a.frntmp, a.frntp, a.frwatp, a.fsinkp, a.fsmp, a.fsmxhp, a.fsmxsp, a.fstovp, a.ftaxp, a.ftelp, a.ftenp, a.ftoilp, a.fvacsp, a.fvalp, a.fvehp, a.fwatp, a.fyblp, a.wgtp1, a.wgtp2, a.wgtp3, a.wgtp4, a.wgtp5, a.wgtp6, a.wgtp7, a.wgtp8, a.wgtp9, a.wgtp10, a.wgtp11, a.wgtp12, a.wgtp13, a.wgtp14, a.wgtp15, a.wgtp16, a.wgtp17, a.wgtp18, a.wgtp19, a.wgtp20, a.wgtp21, a.wgtp22, a.wgtp23, a.wgtp24, a.wgtp25, a.wgtp26, a.wgtp27, a.wgtp28, a.wgtp29, a.wgtp30, a.wgtp31, a.wgtp32, a.wgtp33, a.wgtp34, a.wgtp35, a.wgtp36, a.wgtp37, a.wgtp38, a.wgtp39, a.wgtp40, a.wgtp41, a.wgtp42, a.wgtp43, a.wgtp44, a.wgtp45, a.wgtp46, a.wgtp47, a.wgtp48, a.wgtp49, a.wgtp50, a.wgtp51, a.wgtp52, a.wgtp53, a.wgtp54, a.wgtp55, a.wgtp56, a.wgtp57, a.wgtp58, a.wgtp59, a.wgtp60, a.wgtp61, a.wgtp62, a.wgtp63, a.wgtp64, a.wgtp65, a.wgtp66, a.wgtp67, a.wgtp68, a.wgtp69, a.wgtp70, a.wgtp71, a.wgtp72, a.wgtp73, a.wgtp74, a.wgtp75, a.wgtp76, a.wgtp77, a.wgtp78, a.wgtp79, a.wgtp80, sporder, pwgtp, agep, cit, citwp, cow, ddrs, dear, deye, dout, dphy, drat, dratx, drem, eng, fer, gcl, gcm, gcr, hins1, hins2, hins3, hins4, hins5, hins6, hins7, intp, jwmnp, jwrip, jwtr, lanx, mar, marhd, marhm, marht, marhw, marhyp, mig, mil, mlpa, mlpb, mlpc, mlpd, mlpe, mlpf, mlpg, mlph, mlpi, mlpj, mlpk, nwab, nwav, nwla, nwlk, nwre, oip, pap, relp, retp, sch, schg, schl, semp, sex, ssip, ssp, wagp, wkhp, wkl, wkw, wrk, yoep, anc, anc1p, anc2p, decade, dis, drivesp, esp, esr, fod1p, fod2p, hicov, hisp, indp, jwap, jwdp, lanp, migpuma, migsp, msp, naicsp, nativity, nop, oc, occp, paoc, pernp, pincp, pobp, povpip, powpuma, powsp, privcov, pubcov, qtrbir, rac1p, rac2p, rac3p, racaian, racasn, racblk, racnhpi, racnum, racsor, racwht, rc, sciengp, sciengrlp, sfn, sfr, socp, vps, waob, fagep, fancp, fcitp, fcitwp, fcowp, fddrsp, fdearp, fdeyep, fdoutp, fdphyp, fdratp, fdratxp, fdremp, fengp, fesrp, fferp, ffodp, fgclp, fgcmp, fgcrp, fhins1p, fhins2p, fhins3c, fhins3p, fhins4c, fhins4p, fhins5c, fhins5p, fhins6p, fhins7p, fhisp, findp, fintp, fjwdp, fjwmnp, fjwrip, fjwtrp, flanp, flanxp, fmarhdp, fmarhmp, fmarhtp, fmarhwp, fmarhyp, fmarp, fmigp, fmigsp, fmilpp, fmilsp, foccp, foip, fpap, fpobp, fpowsp, fracp, frelp, fretp, fschgp, fschlp, fschp, fsemp, fsexp, fssip, fssp, fwagp, fwkhp, fwklp, fwkwp, fwrkp, fyoep, pwgtp1, pwgtp2, pwgtp3, pwgtp4, pwgtp5, pwgtp6, pwgtp7, pwgtp8, pwgtp9, pwgtp10, pwgtp11, pwgtp12, pwgtp13, pwgtp14, pwgtp15, pwgtp16, pwgtp17, pwgtp18, pwgtp19, pwgtp20, pwgtp21, pwgtp22, pwgtp23, pwgtp24, pwgtp25, pwgtp26, pwgtp27, pwgtp28, pwgtp29, pwgtp30, pwgtp31, pwgtp32, pwgtp33, pwgtp34, pwgtp35, pwgtp36, pwgtp37, pwgtp38, pwgtp39, pwgtp40, pwgtp41, pwgtp42, pwgtp43, pwgtp44, pwgtp45, pwgtp46, pwgtp47, pwgtp48, pwgtp49, pwgtp50, pwgtp51, pwgtp52, pwgtp53, pwgtp54, pwgtp55, pwgtp56, pwgtp57, pwgtp58, pwgtp59, pwgtp60, pwgtp61, pwgtp62, pwgtp63, pwgtp64, pwgtp65, pwgtp66, pwgtp67, pwgtp68, pwgtp69, pwgtp70, pwgtp71, pwgtp72, pwgtp73, pwgtp74, pwgtp75, pwgtp76, pwgtp77, pwgtp78, pwgtp79, pwgtp80 from household as a inner join person as b on a.serialno = b.serialno with data; select * from merged limit 5; invalid tuple received from server, got 138 columns, expected 476, ignoring

2 1

[MonetDB-users] Debug sqoop-monetdb
by Jeremy O'Sullivan 25 Sep '12

25 Sep '12

Hi, I am trying to insert data into monetdb using sqoop, via the monet jdbc interface. It keeps crashing 20 odd rows into the import, with a monetdb ROLLBACK. The debug info from Sqoop is poor. Does anyone know if the monetdb ROLLBACK can be disabled, so that I can see the last row imported. Has anyone tried Sqoop to Monetdb, are there any debugging tips that you can give ? Regards, Jeremy sqoop export --connect jdbc:monetdb://cluster-03/ngpm -driver nl.cwi.monetdb.jdbc.MonetDriver --num-mappers 1 --password monetdb --username monetdb --table query_start --export-dir /test/dash_test/QueryStart.Log --fields-terminated-by ',' 12/09/24 17:27:49 INFO mapreduce.Job: Task Id : attempt_1347971665109_0247_m_000000_1, Status : FAILED Error: java.io.IOException: java.sql.SQLException: EXEC: wrong number of arguments for prepared statement: 100, expected 102 at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.j ava:192) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java :610) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:730) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja va:1232) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) Caused by: java.sql.SQLException: EXEC: wrong number of arguments for prepared statement: 100, expected 102 at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnectio n.java:2524) at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(MonetConnectio n.java:2274) at nl.cwi.monetdb.jdbc.MonetStatement.internalExecute(MonetStatement.java:508) at nl.cwi.monetdb.jdbc.MonetStatement.execute(MonetStatement.java:349) at nl.cwi.monetdb.jdbc.MonetPreparedStatement.execute(MonetPreparedStatement.ja va:247) at org.apache.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(Async SqlOutputFormat.java:230)

1 0

[MonetDB-users] Announcement: New Jul2012-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 21 Sep '12

21 Sep '12

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2012-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2012-SP2 bugfix release Java Module * Fixed a bug where DatabaseMetaData.getURL() did return null:0 for hostname:port. MonetDB5 Server * Changed the way nclients maximum was calculated to avoid 'out of client slots' errors way before the maximum was reached. Merovingian * Resolved a problem where monetdb could fail to start a database with the message "database 'X' started up, but failed to open up a communication channel". Bug #3134, comment #7. MonetDB Common * Fixed a bug in BATantijoin when either side is a singleton BAT. This fixes bug 3139. Bug Fixes * 3117: test sql/test/BugTracker-2011/Tests/history.Bug-2909.sql produces extra profiling output since changeset 817328b74e2f * 3134: Database gets killed by timeout during startup * 3139: Not equal evaluation does not work * 3144: Memleak when connecting to a database (mclient and mapi) * 3145: mclient error when enabling history * 3150: Multiple string equality tests fail to match (Test_JdbcClient)

1 0

[MonetDB-users] time for ssbm
by guangliang peng 18 Sep '12

18 Sep '12

hi, my query schema is SSBM.My database size is 65G.There are five tables : lineorder 600038145 rows part 1400000 date 2556 supplier 1000000 customer 3000000 and my host have enough memory space up to 500G when i run query select sum(lo_extendedprice*lo_discount) as revenue from voc.lineorder, voc.dates where lo_orderdate = d_datekey and d_weeknuminyear = 6 and d_year = 1994 and lo_discount between 5 and 7 and lo_quantity between 26 and 35; the result is below | revenue | +===============+ | 2605756481323 | +---------------+ 1 tuple (6.2s) but if i delete the condition of " lo_discount between 5 and 7" the run result is more faster than the original one. query select sum(lo_extendedprice*lo_discount) as revenue from voc.lineorder, voc.dates where lo_orderdate = d_datekey and d_weeknuminyear = 6 and d_year = 1994 and lo_quantity between 26 and 35; +---------------+ | revenue | +===============+ | 7990711777299 | +---------------+ 1 tuple (410.451ms) at the same time ,if i delete the condition of "lo_quantity between 26 and 35" the run result is more faster than the original one too. query select sum(lo_extendedprice*lo_discount) as revenue from voc.lineorder, voc.dates where lo_orderdate = d_datekey and d_weeknuminyear = 6 and d_year = 1994 and lo_quantity between 26 and 35; +---------------+ | revenue | +===============+ | 7990711777299 | +---------------+ 1 tuple (463.617ms) i am confused and if the semijoin costs a lot of execution time ? Thank you for your help. best reguards, guangliang peng

1 0

[MonetDB-users] Disk space cleanup policy
by Jukka 17 Sep '12

17 Sep '12

Hi, What is the disk space cleanup policy of MonetDb? If I remove all rows of table with delete command, the disk utilization of dbfarm is still the same. The storage() function is still reporting the same size for the table as before the delete command only count of rows was decreased to zero. Also executing drop table sql statement does not seem to release the disk space. -- Jukka

1 0

[MonetDB-users] MonetDb and too many open files
by Jukka 17 Sep '12

17 Sep '12

Hi, How many files MonetDb is keeping open during runtime? I have running some test on MonetDb on RedHat 6 environment and faced the problem with the too many open files error during the loading phase after having several tables created to the database. I managed to avoid the current error with increasing the file handle limit: vi /etc/security/limits.conf #add for file handle #see lsof -p pid monetdb soft nofile 65536 monetdb hard nofile 65536 Now the system is running, but after 500 tables in MonetDb, I see about 17000 file handle listed with lsof -p command. The most of those are pointing to files in dbfarm folder. Is the files of all columns of all tables kept all the time? Is there formula to estimate the number file handles needed? Is there any limit how many tables or columns we can have in one MonetDb instance? The version used is "MonetDB Database Server v1.6 (Jul2012-SP1)" compiled from sources for RedHat 6. -- Jukka jukkakar(a)iki.fi

1 0