Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Writer to ingest array values #28

Open
takuti opened this issue Jul 22, 2019 · 3 comments
Open

Enable Writer to ingest array values #28

takuti opened this issue Jul 22, 2019 · 3 comments

Comments

@takuti
Copy link
Contributor

takuti commented Jul 22, 2019

Currently all array values in DataFrame will be converted into string, but it's not ideal.

@takuti
Copy link
Contributor Author

takuti commented Nov 8, 2019

The most challenging part of this topic is in InsertIntoWriter since it requires carefully escaping quotes of array elements.

To make the situation simpler, it might be okay to support array column only in BulkImportWriter and SparkWriter first of all, because these writers directly load DataFrame / CSV file and automatically cares the type matters within the data format. This makes the behavior of BulkImportWriter / SparkWriter vs. InsertIntoWriter inconsistent though.

cc: @chezou

@chezou
Copy link
Member

chezou commented Nov 8, 2019

Or, we could have a way to update schema after uploading the dataframe.

@chezou
Copy link
Member

chezou commented Jan 29, 2020

Start implementation for BulkImportWriter.

Concerns for other writers are:

  • InsertIntoWriter requires tons of SQL escape
  • Introducing list handling in SparkWriter requires handling ArrayType which can be confusing for spark unfamiliar users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants