Human-in-the-loop Classification for Multi-page Administrative Documents

Abstract

Real estate administrative documents present a unique challenge where each condo corporation's documents come from multiple sources and in varying formats. Moreover, each record has varying sizes, making it very time-consuming to read and recognize these documents systematically. A condo buyer must understand these documents before proceeding with the purchase. Our industry partner is a company that gathers and summarizes the condo corporation's documents and highlights essential information that the buyer should consider. In this process, the most laborious task is categorizing these segmented documents where the number of categories is unknown and changes based on the domain expert's knowledge. It is only possible to create an automatic classification system for such challenging data without the involvement of domain experts. So we propose a user-centric, scalable, and reliable multi-page document classification solution for our industry partner using a Human-in-a-loop (HITL) framework. It is a web application with an interactive annotation system for the user to provide feedback at the page and document level to train a two-step classification system. The classification pipeline is a modular system that incrementally learns from the domain expert's page and document-level annotation independently. We evaluated the level of feedback required by the HITL system by comparing our proposed system quantitatively and qualitatively with document-level input via a user study. In the same survey, we also assessed the need for the HITL system in real-world classification problems by comparing the automatic system with our proposed page-level annotation system. Unlike previous research, we explicitly used textual features and state-of-the-art (SoTA) language models to determine the predictive model suitable for real estate administrative documents that can be used in an online setting. In our work, we compare NLP models for multi-class classification for multi-page scanned documents and found that frequency-driven models performed better than deep language models. We demonstrated that the HITL system with page-level annotation ability works better than fully automated classification and document-level annotation for a real-world multi-page document classification problem.