Issues creating neuroglancer dataset from schema

[LarsKoeppel]

While creating a new dataset using the neuroglancer_precomputed driver, I encountered three specific issues regarding schema parameters and serialization. The code I used is at the end.

1. missing sharding configuration options
I am unable to set certain sharding parameters via the schema, specifically minishard_index_encoding. It appears to default to gzip with no way to override it. So it would be nice to have the option to set these parameters like it is possible with shard_data_encoding.

2. data_encoding default for sharding
When encoding is set to png, the data_encoding within shards defaults to gzip. The documentation states that jpeg and compressed_segmentation default to raw. I think png encoding should also default to raw.

3. png_level missing from dataset.spec()
Even when png_level is explicitly set in the initial open request, it disappears when calling dataset.spec().to_json(). It is correctly set in the info file but the TensorStore spec object does not show it.

This was the dataset i endet up creating:

dataset = ts.open(
    {
        "driver": "neuroglancer_precomputed",
        "schema": {
            "rank": 4,
            "dtype": "uint8",
            "chunk_layout": {
                "write_chunk": {"shape_soft_constraint": [1024, 1024, 128, 1]},
                "read_chunk": {"shape": [128, 128, 128, 1]},
            },
            "codec": {
                "driver": "neuroglancer_precomputed",
                "encoding": "png",
                "png_level": 6,
                "shard_data_encoding": "raw",
            },
            "domain": {
                "shape": [100000, 100000, 8, 1],
            },
            "dimension_units": [[10, "nm"], [10, "nm"], [100, "nm"], None],
        },
        "kvstore": {
            "driver": "memory",
        },
        "create": True,
    }
).result()

json_spec = dataset.spec().to_json()

I hope these findings are usefull.

[jbms]

The schema, being driver-independently, cannot fully represent all possible driver-specific options. However, you can specify both schema and scale_metadata and they will be merged together.

I assumed that it is normally always advantageous to use gzip encoding for the minishard index. Why do you want to disable that?

We will have to look into the png_level issue.

[LarsKoeppel]

Thank you for the tip with setting both schema and scale_metadata. But when i want to define some parameters in sharding, I was forced to also set some of the bit parameter, even when they were calculated by the schema. And the values in the scale_metadata will override the ones comming from the schema.

We have an aplication that expects that the minishard index is not encoded. I guess the developer of it did not want to bother with decoding it. Do you see any performance differences between gzip encoding and raw?

I also noticed another problem regarding the png_level.
The documentation states here:

When unset, the library default compression level is used.

This results in a -1 writen to the info file. But when I try to open the info file again. I get this error message:

ValueError: FAILED_PRECONDITION: Error opening "neuroglancer_precomputed" driver: Error reading local file ".../info": Error parsing object member "scales": Error parsing value at position 0: Error parsing object member "png_level": Expected integer in the range [0, 9], but received: -1