Question: Uncertainties for Metrics

[evancurtin]

Hello folks, thanks for putting together this nice library! I was wondering if there is a way to get uncertainties/confidence intervals for the metrics built-in, or have a recommended way to do so?

[cmacdonald]

When used via pt.Experiment, using the baseline flag will give you the t-test result; you can also add multiple testing correction. Not quite what you are asking for, but its along the same lines.

[seanmacavaney]

@evancurtin if you're referring to measure residuals, one of the providers (cwl-eval) supports calculating residuals, but this functionality isn't directly exposed by ir-measures. (We could add it if there's interest, though!)

[Zendelo]

+1 for interest in residuals for cwl-eval
It's one of the benefits of using RBP, and would be great to have it implemented

[seanmacavaney]

I'm thinking the most natural way to incorporate this would be as a parameter of the measure, e.g., RBP(residual=True). Does that sound reasonable @Zendelo?

[Zendelo]

Yeah, sounds intuitive to me.

…

________________________________ From: Sean MacAvaney ***@***.***> Sent: Thursday, May 22, 2025 2:40:22 AM To: terrierteam/ir_measures ***@***.***> Cc: Oleg Zendel ***@***.***>; Mention ***@***.***> Subject: Re: [terrierteam/ir_measures] Question: Uncertainties for Metrics (Issue #69) [https://avatars.githubusercontent.com/u/2607830?s=20&v=4]seanmacavaney left a comment (terrierteam/ir_measures#69)<#69 (comment)> I'm thinking the most natural way to incorporate this would be as a parameter of the measure, e.g., RBP(residual=True). Does that sound reasonable @Zendelo<https://github.com/Zendelo>? — Reply to this email directly, view it on GitHub<#69 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADGVETEQ4BSC6N4WUKPN5JD27ST7NAVCNFSM6AAAAABU4FEWMOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQOJYGU4DSNRXGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>