PARLAY: A METHOD FOR CONSTRUCTING A PARAGRAPH-LEVEL NLI DATASET BASED ON MULTI-CATEGORYSCENARIOS

This paper presents a novel methodology for constructing a diverse andcomprehensive paragraph-level dataset tailored for Natural LanguageInference tasks. By incorporating five scenario categories – historical events,scientific explanations, everyday situations, news reports, and fictional stories– our approach ensures broad coverage of various reasoning patterns, includingtransitive reasoning. The methodology employs iterative and Chain-of-Thoughtprompting to elicit detailed, context-rich outcomes by generating intermediatereasoning steps. Our dataset is notable for adding detailed explanations foreach hypothesis, providing a valuable overview of the thought process behindeach categorization. To assess the quality of these outputs, we implementedthe OQACM metrics, a comprehensive framework for evaluating large languagemodel outputs across fluency, coherence, and inference quality. This approachimproves the interpretability of language models and offers detailed insightsinto their inferential processes, advancing NLI dataset development.