Hi Stefan: Thanks for the quick reply. I ran the following on the first CSV file. awk -F"|" 'NF != 18 {print NR,$0}' mycsvfile.csv However none of the lines come back, and thus all lines are 18 fields. When I run COPY INTO with BEST EFFORT, I am able import all but 5,428 records from that file. Unfortunately, this is event data for an analytics dashboard, I can't have any records missing. Is there anything else I can dig into to find out why Monet is rejecting the records? Thanks, Adam -----Original Message----- From: users-list [mailto:users-list-bounces+adam.doherty=esso.ca@monetdb.org] On Behalf Of Stefan Manegold Sent: Tuesday, May 22, 2018 10:12 To: Communication channel for MonetDB users <users-list@monetdb.org> Subject: Re: Error: Failed to import - Leftover Data Hi Adam, the error suggest that (at least) one line in your CSV file contains more columns than the table you try to load the data into. The "..." after "Leftover data" should tell you what that "excess content" of that line is. Hence, searching for that "..." in your CVS file should help you to locate the problem in the CSV file. Best, Stefan ----- On May 22, 2018, at 5:06 PM, Doherty, Adam adam.doherty@esso.ca wrote:
Hi all:
I am receiving an error "Failed to import table ... Leftover data ..."
I am using a simple BASH script and the mclient tool to import my data from hundreds of CSV files. The fields are pipe | separated, text values are surrounded with double-quotes, and rows end with LF (\n).
This is the command I am using to process the files -
for csvfile in `ls | grep -i 2018.*\.csv$`; do mclient -d mydatabase -s "COPY INTO table FROM '/path/to/csv/files/$csvfile' USING DELIMITERS '|', '\n', '\"'"; done
I have a .monetdb file setup in my home directory so as not to need the username and password.
Thanks kindly,
Adam Doherty _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list