Design and Evaluation of a Route-Aware, Evidence-Grounded RAG Chatbot for Accessing Village Monographs and Census Data within Data Desa Presisi

Ikhlasul Amal; Irman Hermadi; Yani Nurhadryani

doi:10.63158/journalisi.v8i5.1856

Authors

Ikhlasul Amal IPB University
Irman Hermadi IPB University
Yani Nurhadryani IPB University

DOI:

https://doi.org/10.63158/journalisi.v8i5.1856

Keywords:

Data Desa Presisi, evidence-grounded RAG, privacy-aware routing, source provenance, structured-data QA

Abstract

Data Desa Presisi integrates narrative village monographs and structured census records, creating challenges for contextual, numerical, and cross-source information access. This study designed and evaluated a locally deployed, route-aware, evidence-grounded retrieval-augmented generation (RAG) chatbot that combines deterministic DuckDB queries, document retrieval, provenance, and privacy-aware routing. Evaluation used 100 ground-truth questions from Setu and Gunung Putri. Eighty-four database, monograph, and mixed-evidence questions were assessed using RAGAS and ROUGE-L, whereas 16 conversational and policy questions were evaluated through deterministic checks. Elasticsearch at a temperature of 0.1 produced the highest composite score (0.592018), although post hoc Wilcoxon tests found no statistically significant superiority over alternative configurations. The selected configuration achieved a 98.0% functional pass rate and a 97.92% numeric-match rate. Public-mode usability evaluation involving 21 village officials and staff yielded 96.03% effectiveness. Scalability testing covered pipeline construction and retrieval across 30 census inputs and 28 monographs, but did not assess concurrent users or full end-to-end production workloads. These findings provide preliminary evidence that route-aware RAG can improve transparent, evidence-grounded access to heterogeneous village data while preserving deterministic handling of numerical queries and privacy-sensitive information.

Downloads

Download data is not yet available.

References

[1] S. Sjaf et al., “Data Desa Presisi: A new method of rural data collection,” MethodsX, vol. 9, Art. no. 101868, 2022, doi: 10.1016/j.mex.2022.101868.

[2] K. F. Siman and J. Wiratama, “A web-based village administrative information system for improving service quality toward the smart village concept,” G-Tech: J. Teknol. Terap., vol. 7, no. 4, pp. 1519–1528, 2023, doi: 10.33379/gtech.v7i4.3196.

[3] N. Saidah, L. Khasanah, Asriyatuzzahra, and S. Ridloah, “Analisis strategi kesuksesan Kampung Digital Krandegan dalam mendukung program smart village,” J. Reg. Rural Dev. Plan., vol. 6, no. 2, pp. 123–135, 2022, doi: 10.29244/jp2wd.2022.6.2.123-135.

[4] Pemerintah Desa Krandegan, “Pemerintah Desa Krandegan maksimalkan pelayanan warga dengan chatbot artificial intelligence (AI),” 2025.

[5] A. Arafat, R. Rasyid, S. Hestiana, R. Rendi, S. Bantun, and J. Y. Sari, “Designing an AI-based village information system using research and development approach for public governance modernization in Popalia Village,” J. Tek. Inform. (JUTIF), vol. 6, no. 5, pp. 5251–5269, 2025, doi: 10.52436/1.jutif.2025.6.5.5257.

[6] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” Adv. Neural Inf. Process. Syst., vol. 33, pp. 9459–9474, 2020.

[7] V. Karpukhin et al., “Dense passage retrieval for open-domain question answering,” in Proc. 2020 Conf. Empirical Methods Nat. Lang. Process. (EMNLP), 2020, pp. 6769–6781, doi: 10.18653/v1/2020.emnlp-main.550.

[8] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, “M3-Embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,” in Findings Assoc. Comput. Linguistics: ACL 2024, 2024, pp. 2318–2335, doi: 10.18653/v1/2024.findings-acl.137.

[9] J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity search with GPUs,” IEEE Trans. Big Data, vol. 7, no. 3, pp. 535–547, 2021, doi: 10.1109/TBDATA.2019.2921572.

[10] G. V. Cormack, C. L. A. Clarke, and S. Büttcher, “Reciprocal rank fusion outperforms Condorcet and individual rank learning methods,” in Proc. 32nd Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2009, pp. 758–759, doi: 10.1145/1571941.1572114.

[11] T. Scholak, N. Schucher, and D. Bahdanau, “PICARD: Parsing incrementally for constrained auto-regressive decoding from language models,” in Proc. 2021 Conf. Empirical Methods Nat. Lang. Process. (EMNLP), 2021, pp. 9895–9901, doi: 10.18653/v1/2021.emnlp-main.779.

[12] S. Zeng et al., “The good and the bad: Exploring privacy issues in retrieval-augmented generation,” in Findings Assoc. Comput. Linguistics: ACL 2024, 2024, pp. 4505–4524, doi: 10.18653/v1/2024.findings-acl.267.

[13] Z. Tan et al., “Glue pizza and eat rocks—Exploiting vulnerabilities in retrieval-augmented generative models,” in Proc. 2024 Conf. Empirical Methods Nat. Lang. Process. (EMNLP), 2024, pp. 1610–1626, doi: 10.18653/v1/2024.emnlp-main.96.

[14] J. Qi, G. Sarti, R. Fernández, and A. Bisazza, “Model internals-based answer attribution for trustworthy retrieval-augmented generation,” in Proc. 2024 Conf. Empirical Methods Nat. Lang. Process. (EMNLP), 2024, pp. 6037–6053, doi: 10.18653/v1/2024.emnlp-main.347.

[15] K. Peffers, T. Tuunanen, M. A. Rothenberger, and S. Chatterjee, “A design science research methodology for information systems research,” J. Manage. Inf. Syst., vol. 24, no. 3, pp. 45–77, 2007, doi: 10.2753/MIS0742-1222240302.

[16] M. Raasveldt and H. Mühleisen, “DuckDB: An embeddable analytical database,” in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2019, pp. 1981–1984, doi: 10.1145/3299869.3320212.

[17] S. Robertson and H. Zaragoza, “The probabilistic relevance framework: BM25 and beyond,” Found. Trends Inf. Retrieval, vol. 3, no. 4, pp. 333–389, 2009, doi: 10.1561/1500000019.

[18] S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “RAGAs: Automated evaluation of retrieval-augmented generation,” in Proc. 18th Conf. Eur. Chapter Assoc. Comput. Linguistics: Syst. Demonstrations, 2024, pp. 150–158, doi: 10.18653/v1/2024.eacl-demo.16.

[19] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out, 2004, pp. 74–81.

[20] International Organization for Standardization, Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts, ISO 9241-11:2018, 2018.

[21] Y. Nurhadryani, S. K. Sianturi, I. Hermadi, and H. Khotimah, “Pengujian usability untuk meningkatkan antarmuka aplikasi mobile,” J. Ilmu Komput. Agri-Inform., vol. 2, no. 2, pp. 83–93, 2013, doi: 10.29244/jika.2.2.83-93.

[22] J. Saad-Falcon, O. Khattab, C. Potts, and M. Zaharia, “ARES: An automated evaluation framework for retrieval-augmented generation systems,” in Proc. 2024 Conf. North Amer. Chapter Assoc. Comput. Linguistics: Hum. Lang. Technol., 2024, pp. 338–354, doi: 10.18653/v1/2024.naacl-long.20.

[23] D. Ru et al., “RAGChecker: A fine-grained framework for diagnosing retrieval-augmented generation,” Adv. Neural Inf. Process. Syst., vol. 37, 2024, doi: 10.52202/079017-0692.

[24] C. Niu et al., “RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models,” in Proc. 62nd Annu. Meeting Assoc. Comput. Linguistics (ACL), 2024, pp. 10862–10878, doi: 10.18653/v1/2024.acl-long.585.

[25] T. Gao, H. Yen, J. Yu, and D. Chen, “Enabling large language models to generate text with citations,” in Proc. 2023 Conf. Empirical Methods Nat. Lang. Process. (EMNLP), 2023, pp. 6465–6488, doi: 10.18653/v1/2023.emnlp-main.398.