Reliability assessment of service-based software under operational profile uncertainty

We address the problem of operational reliability assessment through testing of software services delivered on-demand such as Web Services. Software reliability assessment is typically done for a specific operational profile: the profile is needed in testing to select or generate test cases (operational testing) in a way statistically similar to the anticipated use of software in operation; the observations of success/failure of test executions are used to predict software reliability in actual operation. It is well known that unless the profile is accurate, software reliability predictions obtained via operational testing cannot be trusted. We present a new way of dealing with the uncertainty in the operational profile adopting a two-stage Bayesian inference for reliability assessment. The technique relies on the availability of information about partitions of the input space. The approach is demonstrated on contrived examples and on a case study of real Web Services. We discuss the usefulness of the approach in dealing with two important practical problems: i) the true profile in operation differs from the one used in testing, ii) the profile in operation is changing continuously.