Objective: to assess the inter-rater reliability (IRR) of the revised Cochrane risk-of-bias tool for randomised trials (RoB2). Methods: Four raters independently applied RoB2 on critical and important outcomes of individually randomized parallel-group trials (RCTs) included in the Cochrane Review “Cannabis and cannabinoids for people with multiple sclerosis.” We calculated Fleiss’ Kappa for multiple raters and time to complete the tool; we performed a calibration exercise on five studies, then we developed an implementation document (ID) specific for the condition, and the intervention addressed by the review with instructions on how to answer the signalling questions of RoB2 tool. We measured IRR before and after the ID adoption Results: Eighty results related to seven outcomes from 16 RCTs were assessed. During calibration exercise we reached no agreement for overall judgment (IRR -0.15); IRR for individual domains ranged from no agreement to fair. Mean time to apply the tool was 168.5 minutes per study. Time to complete the calibration exercise and develop the ID was about 40 hours. After the ID adoption ID, overall agreement increased to slightly (IRR 0.11) for the first five studies and moderate (IRR 0.42) for the remaining 11. IRR for individual domains ranged from no agreement to almost perfect. Mean time to apply the tool decreased to 41 minutes. Conclusion: RoB2 tool is comprehensive but complex even for high experienced raters. The development of an ID specific for the review may improve reliability substantially.

Reliability of the revised Cochrane risk-of-bias tool for randomised trials (RoB2) improved with the use of implementation instruction / Minozzi, S.; Dwan, K.; Borrelli, F.; Filippini, G.. - In: JOURNAL OF CLINICAL EPIDEMIOLOGY. - ISSN 0895-4356. - 141:(2022), pp. 99-105. [10.1016/j.jclinepi.2021.09.021]

Reliability of the revised Cochrane risk-of-bias tool for randomised trials (RoB2) improved with the use of implementation instruction

Borrelli F.;
2022

Abstract

Objective: to assess the inter-rater reliability (IRR) of the revised Cochrane risk-of-bias tool for randomised trials (RoB2). Methods: Four raters independently applied RoB2 on critical and important outcomes of individually randomized parallel-group trials (RCTs) included in the Cochrane Review “Cannabis and cannabinoids for people with multiple sclerosis.” We calculated Fleiss’ Kappa for multiple raters and time to complete the tool; we performed a calibration exercise on five studies, then we developed an implementation document (ID) specific for the condition, and the intervention addressed by the review with instructions on how to answer the signalling questions of RoB2 tool. We measured IRR before and after the ID adoption Results: Eighty results related to seven outcomes from 16 RCTs were assessed. During calibration exercise we reached no agreement for overall judgment (IRR -0.15); IRR for individual domains ranged from no agreement to fair. Mean time to apply the tool was 168.5 minutes per study. Time to complete the calibration exercise and develop the ID was about 40 hours. After the ID adoption ID, overall agreement increased to slightly (IRR 0.11) for the first five studies and moderate (IRR 0.42) for the remaining 11. IRR for individual domains ranged from no agreement to almost perfect. Mean time to apply the tool decreased to 41 minutes. Conclusion: RoB2 tool is comprehensive but complex even for high experienced raters. The development of an ID specific for the review may improve reliability substantially.
2022
Reliability of the revised Cochrane risk-of-bias tool for randomised trials (RoB2) improved with the use of implementation instruction / Minozzi, S.; Dwan, K.; Borrelli, F.; Filippini, G.. - In: JOURNAL OF CLINICAL EPIDEMIOLOGY. - ISSN 0895-4356. - 141:(2022), pp. 99-105. [10.1016/j.jclinepi.2021.09.021]
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11588/884939
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 45
  • ???jsp.display-item.citation.isi??? 44
social impact