"A Good Bot Always Knows Its Limitations": Assessing Autonomous System Decision-making Competencies through Factorized Machine Self-confidence
arxiv(2024)
Abstract
How can intelligent machines assess their competencies in completing tasks?
This question has come into focus for autonomous systems that algorithmically
reason and make decisions under uncertainty. It is argued here that machine
self-confidence – a form of meta-reasoning based on self-assessments of an
agent's knowledge about the state of the world and itself, as well as its
ability to reason about and execute tasks – leads to many eminently computable
and useful competency indicators for such agents. This paper presents a
culmination of work on this concept in the form of a computational framework
called Factorized Machine Self-confidence (FaMSeC), which provides an
engineering-focused holistic description of factors driving an algorithmic
decision-making process, including outcome assessment, solver quality, model
quality, alignment quality, and past experience. In FaMSeC, self-confidence
indicators are derived from hierarchical `problem-solving statistics' embedded
within broad classes of probabilistic decision-making algorithms such as Markov
decision processes. The problem-solving statistics are obtained by evaluating
and grading probabilistic exceedance margins with respect to given competency
standards, which are specified for each decision-making competency factor by
the informee (e.g. a non-expert user or an expert system designer). This
approach allows `algorithmic goodness of fit' evaluations to be easily
incorporated into the design of many kinds of autonomous agents via
human-interpretable competency self-assessment reports. Detailed descriptions
and running application examples for a Markov decision process agent show how
two FaMSeC factors (outcome assessment and solver quality) can be practically
computed and reported for a range of possible tasking contexts through novel
use of meta-utility functions, behavior simulations, and surrogate prediction
models.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined