Feature Request: Persist comparison tolerances with test metadata

[michaeltryby]

Hey @davidchall nrtest is wonderful! We are putting it to great use for our projects. We are thinking about using it in our CI tool chain and had a thought on how to better facilitate that.

It would be great if default comparison tolerances could be persisted with the test metadata. The defaults could be overridden with values passed from the command line. This would preserve the current functionality and make it more straight forward to automate the regression test. Make sense?

Would you be open to including this feature? Thanks.

[davidchall]

Hi Michael,

Just to confirm, you are asking for comparison tolerance fields in the test configuration files? This would be nice because tests can then have personalized tolerances.

If we do this, I'd probably prefer to remove the ability to set the comparison tolerance from the command line. This would need to override the tolerance in all tests, which is just asking for trouble.

[michaeltryby]

@davidchall Yes like you say, comparison tolerance fields in the test configuration file. I also agree that setting the comparison tolerance from the command line could become problematic in that scenario.

How were you thinking of implementing this? I was thinking maybe add another optional field to the test configuration file. Perhaps, a dictionary were the key was the file type and the value was a list of tolerance values.

It would be nice to enclose tolerance values in a data structure. That way test functions with different input parameters besides rtol and atol could also be supported. Make sense?

[davidchall]

I've been thinking about this some more. The only truly comprehensive way to do this would be to change the Test config files in a backwards-incompatible way, so we can add metadata to each result file. Then (rtol, atol) values can be specified independently for each comparison. This can also allow custom arguments to be sent to the comparison routine. I really like this flexibility. The downside is that it would require an awful lot more effort to create a Test config file.

Something like:

{
    "name": "Scoring_01",
    "version": "1.0",
    "description": "Basic test shooting a 6cm diameter proton beam into a water phantom.",
    "args": [
        "Scoring_01.txt"
    ],
    "input_files": [
        "Scoring_01.txt",
        "GlobalParameters.txt"
    ],
    "output_files": [
        {
            "filename": "Dose.csv",
            "compare_routine": "topas binned",
            "compare_args": {
                "rtol": 0.01,
                "atol": 0
            }
        }
    ]
}