Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark not-nullable column converts to EXASOL column failed while saving #60

Closed
3cham opened this issue Jan 29, 2020 · 1 comment · Fixed by #63
Closed

Spark not-nullable column converts to EXASOL column failed while saving #60

3cham opened this issue Jan 29, 2020 · 1 comment · Fixed by #63

Comments

@3cham
Copy link
Contributor

3cham commented Jan 29, 2020 •
edited
Loading

While creating table in EXASOL we infer column information from spark schema.
Spark says if a column does not contain any null value then this column is not nullable, despite it can contain empty string. So our connector will set the column in the EXASOL DDL to NOT NULL. This leads to problem when writing the data to EXASOL if empty strings occur.

I suggest that we remove this nullable checking for world's peace since it does take a lot of time to find the cause of this problem :)

Let's me know about your opinions @morazow @jpizagno
@morazow
Copy link
Contributor

morazow commented Jan 30, 2020

Hey @3cham ,

Good to hear from you!

Yes, it is a good point! From my side, I would prefer to remove the NOT NULL for only string types. But, I guess we can remove it for all types. It puts the validation on the database side, but if users want that, they can filter the dataframe before insert or run ALTER command to add that constraint later inside the database. So, it should be safe to remove it from connector.

Please feel free to send a pull request!

morazow added a commit that referenced this issue Mar 30, 2020
@morazow
Remove NOT NULL constraint from String types.
5a7a604 
We create an Exasol table, if it did not exist, before saving the Spark
dataframe. The `NOT NULL` constraint was added to the create table DDL,
if the Spark schema field type is not nullable.

However, this can be problem in Exasol side. Because, Exasol puts `null`
if the string is empty for `VARCHAR` or `CLOB` column types. Therefore,
putting not null constraints fails when inserting empty strings.

This commit removes the `NOT NULL` constraints from string types even if
they are not nullable.

Fixes #60.
@morazow morazow mentioned this issue Mar 30, 2020
Merged
morazow added a commit that referenced this issue Apr 9, 2020
@morazow
Remove NOT NULL constraint from String types. (#63)
ac6bf9c 
We create an Exasol table, if it did not exist, before saving the Spark
dataframe. The `NOT NULL` constraint was added to the create table DDL,
if the Spark schema field type is not nullable.

However, this can be problem in Exasol side. Because, Exasol puts `null`
if the string is empty for `VARCHAR` or `CLOB` column types. Therefore,
putting not null constraints fails when inserting empty strings.

This commit removes the `NOT NULL` constraints from string types even if
they are not nullable.

Fixes #60.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove NOT NULL constraint from String types. exasol/spark-connector
2 participants
@morazow @3cham