Generate data based on estimated Delta table size

[tfayyaz]

As I have been using the data generator I have had to use trial and error to get a table size I require. Not sure if this is feasible but it would be great to generate data based on the final table size required instead of no. of rows.

Alternatively it might be useful to easily get back stats about the generated table size and use that to iteratively generate more data to reach the desired table size.

Currently I am doing the following to get back the table size which works well but needs to be run manually each time.

dfTestData.write.format("delta").mode("overwrite").saveAsTable("tfayyaz_db.test_data")
detail = spark.sql("DESCRIBE DETAIL tfayyaz_db.test_data")
print(detail.first()["sizeInBytes"]/1024/1024, "mb")

Thanks
Tahir

[ronanstokes-db]

i'm thinking that we could do something like the following:

add method `withTargetSize(sizeMb,
dataFormat,
tmpPath (optional tmp path override))

we would then generate 100,000 rows and write them to tmp location (default dbfs:/tmp) in format (default parquet)

Then we could use the size of 100k rows to compute how many rows will make up the target size.

The row count used for the calculations could be configurable also.

As size of result set will be a guestimate (due to possible random data), this will produce an approximately sized data output.

Note if size of data per row is not very variable (i.e limited arbitrary text generation), this should produce a dataset close to the target size.

Key question: Do you need exact target size or approximate sized outputs?

[ronanstokes-db]

For accurate target sizes, you would need to produce a data set larger than the target size and sample / write repeatedly until you get close to the target size.

For large data sets, this could be costly in terms of performance