Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to specify the path of the error file #195

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

RatserX
Copy link
Contributor

@RatserX RatserX commented May 31, 2024

This change enables the use of the -e option for bcp (https://learn.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-ver16#-e-err_file). This allows the creation of error files that contain detailed information about the rows that the bcp command couldn't process for whatever reason when executed.

Also included is a change to the bcp_path param typing to add the pathlib.Path type just because the bcp() function already allows it.

@RatserX
Copy link
Contributor Author

RatserX commented Jun 2, 2024

Actually, let me try a different approach. I think it would be better to store the error file in a temp file every time there's an error, read the tempfile and append the full text to the BCPandasException.

Nevermind, that idea didn't work that well. The error file can actually get pretty big when there are a lot of errors in the input file. If we try to read and print that those errors, they can actually slow down the to_csv function. Not only that, but this adds too much clutter to the original BCPandasException; even if we log print it, it is still hard to understand without proper formatting.

So, to keep the changes simple, I think my current approach to use the BCP parameter -e and generate an error file if a path is provided works just fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant