A Human-in-the-Loop Method for Developing Machine Learning Applications

An intelligent system is usually built based on a machine-learning model trained over offline datasets. However, such a system is difficult to adapt to new patterns or new data in the online environment, i.e. offline model has relatively poor online generalization. Moreover, understanding and solving errors taking place in the online system is also hard because errors arise in offline training pipelines and could propagate. We propose a novel systematic method, including design, architecture, and implementation to mix people's experience and intelligence with a relatively low cost. The method includes three negative-feedback loops and one data loop, supporting iterative and incremental developing procedure. Based on the method, we implemented a general system architecture and conducted two case studies. The results illustrate the effectiveness of the method.