You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the s3 loader writes files to a single s3 directory. This makes it somewhat hard to use with Athena or other hive query engines because we cannot set up time-based partitions by use of the directory structure. This also makes it harder to maintain and handle the data on s3.
Please allow optional configuration to load data onto s3 with a directory structure that can be filled with values from s3 loader's server timestamp.
eg. -
configure to store in monthly chunks:
s3_path = "base_dir/enriched/good/yr={YYYY}/mo={MM}"
store in base directory:
s3_path = "some_dir/raw"
store in hourly chunks:
s3_path = "base_dir/enriched/good/date={YYYY-MM-dd}/hour={HH}"
The text was updated successfully, but these errors were encountered:
Currently, the s3 loader writes files to a single s3 directory. This makes it somewhat hard to use with Athena or other hive query engines because we cannot set up time-based partitions by use of the directory structure. This also makes it harder to maintain and handle the data on s3.
Please allow optional configuration to load data onto s3 with a directory structure that can be filled with values from s3 loader's server timestamp.
eg. -
The text was updated successfully, but these errors were encountered: