Multiple error thresholds for each compressor

[treigerm]

Right now we only run each compressor only once on each data source. It would be nice to run it multiple times (probably 3 times is enough) with different absolute/relative error bounds. This then also allows us to plot some rate-distortion curves when comparing compressors.

[juntyr]

This is where the 100%, 99%, 95% error bounds will come in

[treigerm]

I am just starting to think about how we would change the structure of the library/scripts to account for the fact that for the same dataset and compressor we will have multiple results.

On the side of the compressors, we could have an argument that passes in absolute_error and/or relative_error argument to the build function of the compressor.

The results are currently stored in a directory structure of the form compressed-datasets/{dataset}/{compressor}/. We could simply add another level to this structure, i.e. compressed-datasets/{dataset}/{compressor}/{error_bound}.

Writing it out like this it actually seems not too much of an adjustment. What do you think of this approach @juntyr ? Once we settle on an overall strategy i'll start working on a PR for it. While we don't have the rigorously derived error bounds yet, it will be good to get the functionality implemented.

[juntyr]

I am just starting to think about how we would change the structure of the library/scripts to account for the fact that for the same dataset and compressor we will have multiple results.

On the side of the compressors, we could have an argument that passes in absolute_error and/or relative_error argument to the build function of the compressor.

The results are currently stored in a directory structure of the form compressed-datasets/{dataset}/{compressor}/. We could simply add another level to this structure, i.e. compressed-datasets/{dataset}/{compressor}/{error_bound}.

Writing it out like this it actually seems not too much of an adjustment. What do you think of this approach @juntyr ? Once we settle on an overall strategy i'll start working on a PR for it. While we don't have the rigorously derived error bounds yet, it will be good to get the functionality implemented.

One problem is that not all compressors natively support all kind of error bounds, e.g. tthresh doesn't support absolute error bounds, and most compressors don't really support relative error bounds. So in any case, we'll likely need to use some transformations to make them work. What I'm envisioning right now is separate Compressor.build methods for the different kinds of error bounds we'll use. Then the folder structure would be compressed-datasets/{dataset}/{compressor}/{error_kind}/{error_bound}

[juntyr]

Implemented by #15