Optimizing YOLO-Based Algorithms for Real-Time BISINDO Alphabet Detection Under Varied Lighting and Background Conditions in Computer Vision Systems

Lilis Nur Hayati, Anik Nur Handayani, Wahyu Sakti Gunawan Irianto, Rosa Andrie Asmara, Dolly Indra, Nor Salwa Damanhuri

Abstract


This research explores the optimization of YOLO-based computer vision algorithms for real-time recognition of Indonesian Sign Language (BISINDO) letters under diverse environmental conditions. Motivated by the communication barriers faced by the deaf and hearing communities due to limited sign language literacy, the study aims to enhance inclusivity through advanced visual detection technologies. By implementing the YOLOv5s model, the system is trained to detect and classify correct and incorrect BISINDO hand signs across 52 classes (26 correct and 26 incorrect letters), utilizing a dataset of 3,900 images augmented to 10,920 samples. Performance evaluation employs k-fold cross-validation (k=10) and confusion matrix analysis across varied lighting and background scenarios, both indoor and outdoor. The model achieves a high average precision of 0.9901 and recall of 0.9999, with robust results in indoor settings and slight degradation observed under certain outdoor conditions. These findings demonstrate the potential of YOLOv5 in facilitating real-time, accurate sign language recognition, contributing toward more accessible human-computer interaction systems for the deaf community.


Keywords


YOLOv5, BISINDO, Sign Language Recognition, Object Detection, Real-Time Detection

Full Text:

PDF

References


M. F. Nur Hayati Lilis, Nur Handayani Anik, Wahyu Sakti Gunawan Iriantoa, Rosa Andrie Asmara, Dolly Indra, “Classifying BISINDO Alphabet using Tensorflow Object Detection API,” Ilk. J. Ilm., vol. 15, no. 2, pp. 358–364, 2023.

S. Anugerah, S. Ulfa, and A. Husna, “Pengembangan Video Pembelajaran Bahasa Isyarat Indonesia (BISINDO) Untuk Siswa Tunarungu Di Sekolah Dasar,” JINOTEP (Jurnal Inov. dan Teknol. Pembelajaran) Kaji. dan Ris. Dalam Teknol. Pembelajaran, vol. 7, no. 2, pp. 76–85, 2020, doi: 10.17977/um031v7i22020p076.

S. Daniels, N. Suciati, and C. Fathichah, “Indonesian Sign Language Recognition using YOLO Method,” IOP Conf. Ser. Mater. Sci. Eng., vol. 1077, no. 1, p. 012029, 2021, doi: 10.1088/1757-899x/1077/1/012029.

W. C. K. Sholikhatul Amaliya, Anik Nur Handayani, Muhammad Iqbal Akbar, Heru Wahyu Herwanto, Osamu Fukuda, “Study on Hand Keypoint Framework for Sign Language Recognition,” 10.1109/ICEEIE52663.2021.9616851, 2021, doi: 10.1109/ICEEIE52663.2021.9616851.

G. B. Rosa Andrie Asmara, Muhammad Ridwan, “Haar Cascade and Convolutional Neural Network Face Detection in Client-Side for Cloud Computing Face Recognition,” 2021 Int. Conf. Electr. Inf. Technol., 2021.

M. C. Bagaskoro, F. Prasojo, A. N. Handayani, E. Hitipeuw, A. P. Wibawa, and Y. W. Liang, “Hand image reading approach method to Indonesian Language Signing System (SIBI) using neural network and multi layer perseptron,” Sci. Inf. Technol. Lett., vol. 4, no. 2, pp. 97–108, 2023, doi: 10.31763/sitech.v4i2.1362.

E. Rahayu et al., “LUMINA : Linguistic unified multimodal Indonesian natural audio-visual dataset,” Data Br., vol. 54, p. 110279, 2024, doi: 10.1016/j.dib.2024.110279.

R. A. Asmara et al., “YOLO-based object detection performance evaluation for automatic target aimbot in first-person shooter games,” Bull. Electr. Eng. Informatics, vol. 13, no. 4, pp. 2456–2470, 2024, doi: 10.11591/eei.v13i4.6895.

R. Wulanningrum, A. N. Handayani, and A. P. Wibawa, “Perbandingan Instance Segmentation Image Pada YOLO8,” J. Teknol. Inf. dan Ilmu Komput., vol. 11, no. 4, pp. 753–760, 2024, doi: 10.25126/jtiik.1148288.

I. Zaeni, K. Kirana, Y. D. Mahandi, A. N. Handayani, and R. Fauzi, “Pelatihan SIBI (Sistem Isyarat Bahasa Indonesia) Berbasis Citra pada Siswa SLB Tunarungu Kota Malang,” J. Inov. Teknol. dan Edukasi Tek., vol. 1, no. 6, pp. 428–431, 2021, doi: 10.17977/um068v1i62021p428-431.

Tazyeen Fathima, Ashif Alam, Ashish Gangwar, Dev Kumar Khetan, and Prof. Ramya K, “Real-Time Sign Language Recognition and Translation Using Deep Learning Techniques,” Int. Res. J. Adv. Eng. Hub, vol. 2, no. 02, pp. 93–97, 2024, doi: 10.47392/irjaeh.2024.0018.

M. Rivera-Acosta, J. M. Ruiz-Varela, S. Ortega-Cisneros, J. Rivera, R. Parra-Michel, and P. Mejia-Alvarez, “Spelling Correction Real-Time American Sign Language Alphabet Translation System Based on Yolo Network and LSTM,” Electron., vol. 10, no. 9, 2021, doi: 10.3390/electronics10091035.

Z. S. Jannah and F. A. Sutanto, “Implementasi Algoritma YOLO (You Only Look Once) Untuk Deteksi Rias Adat Nusantara,” J. Ilm. Univ. Batanghari Jambi, vol. 22, no. 3, p. 1490, 2022, doi: 10.33087/jiubj.v22i3.2421.

R. A. Asmara, B. Syahputro, D. Supriyanto, and A. N. Handayani, “Prediction of Traffic Density using YOLO Object Detection and Implemented in Raspberry Pi 3b + and Intel NCS 2,” 4th Int. Conf. Vocat. Educ. Training, ICOVET 2020, pp. 391–395, 2020, doi: 10.1109/ICOVET50258.2020.9230145.

D. O. Pratama, U. N. Malang, A. N. Handayani, and U. N. Malang, “Development of Embedded System Learning Module Using Project-based Learning Method for Industrial Electronics Department,” Lect. J. Pendidik., vol. 16, pp. 225–238, 2025.

S. Tyagi, P. Upadhyay, I. Hoor Fatima, and A. Kumar Sharma, “American Sign Language Detection using YOLOv5 and YOLOv8,” Res. Sq., pp. 1–16, 2023, [Online]. Available: https://doi.org/10.21203/rs.3.rs-3126918/v1.

D. Indra, R. Satra, H. Azis, A. R. Manga, and H. L, “Detection System of Strawberry Ripeness Using K-Means,” Ilk. J. Ilm., vol. 14, no. 1, pp. 25–31, 2022, doi: 10.33096/ilkom.v14i1.1054.25-31.

M. A. A. K. Sanket Bankar, Tushar Kadam, Vedant Korhale, “Real Time Sign Language Recognition Using Deep Learning,” Int. J. Res. Appl. Sci. Eng. Technol., vol. 11, no. 9, pp. 193–199, 2023, doi: 10.22214/ijraset.2023.55621.

M. Zaki, “Hand Keypoint-Based CNN for SIBI Sign Language Recognition,” Int. J. Robot. Control Syst., vol. 5, no. 2, pp. 813–829, 2025.

D. Indra, S. Madenda, and E. P. Wibowo, “Feature Extraction of Bisindo Alphabets Using Chain Code Contour,” Int. J. Eng. Technol., vol. 9, no. 4, pp. 3415–3419, 2017, doi: 10.21817/ijet/2017/v9i4/170904142.

P. Choirina and R. A. Asmara, “Deteksi Jenis Kelamin Berdasarkan Citra Wajah Jarak Jauh Dengan Metode Haar Cascade Classifier,” J. Inform. Polinema, vol. 2, no. 4, p. 164, 2016, doi: 10.33795/jip.v2i4.77.

N. H. Amir, C. Kusuma, and A. Luthfi, “Refining the Performance of Neural Networks with Simple Architectures for Indonesian Sign Language System ( SIBI ) Letter Recognition Using Keypoint Detection,” Ilk. J. Ilm., vol. 17, no. 1, pp. 64–73, 2025.

R. A. Asmara, U. D. Rosiani, M. Mentari, A. R. Syulistyo, M. N. Shoumi, and M. Astiningrum, “An Experimental Study on Deep Learning Technique Implemented on Low Specification OpenMV Cam H7 Device,” Int. J. Informatics Vis., vol. 8, no. 2, pp. 1017–1029, 2024, doi: 10.62527/joiv.8.2.2299.

Y. N. Faudah, I. D. Ubaidah, F. F. Ibrahim, Nur Taliningsih, N. K. Sy, and M. A. Pramuditho, “Optimasi Convolutional Neural Network dan K-Fold Cross Validation pada Sistem Klasifikasi Glaukoma,” ELKOMIKA J. Tek. Energi Elektr. Tek. Telekomun. Tek. Elektron., vol. 10, no. 3, p. 728, 2022, doi: 10.26760/elkomika.v10i3.728.

A. T. Hermawan, I. A. E. Zaeni, A. P. Wibawa, Gunawan, W. H. Hendrawan, and Y. Kristian, “A Multi Representation Deep Learning Approach for Epileptic Seizure Detection,” J. Robot. Control, vol. 5, no. 1, pp. 187–204, 2024, doi: 10.18196/jrc.v5i1.20870.

C. Suardi, A. N. Handayani, R. A. Asmara, A. P. Wibawa, L. N. Hayati, and H. Azis, “Design of Sign Language Recognition Using E-CNN,” 3rd 2021 East Indones. Conf. Comput. Inf. Technol. EIConCIT 2021, pp. 166–170, 2021, doi: 10.1109/EIConCIT50028.2021.9431877.

A. P. Wibawa and F. Kurniawan, “Enhancing Teks Summarization of Humorous Texts with Attention-Augmented LSTM and Discourse-Aware Decoding,” Int. J. Eng. Sci. Inf. Technol., vol. 5, no. 3, pp. 156–168, 2025.

R. Sutjiadi, S. Sendari, H. W. Herwanto, and Y. Kristian, “Generating High-quality Synthetic Mammogram Images Using Denoising Diffusion Probabilistic Models: A Novel Approach for Augmenting Deep Learning Datasets,” 2024 Int. Conf. Inf. Technol. Syst. Innov. ICITSI 2024 - Proc., pp. 386–392, 2024, doi: 10.1109/ICITSI65188.2024.10929446.

A. M. Sarosa, A. Badriyah, R. A. Asmara, M. K. Wardani, D. F. Al Riza, and Y. Mulyani, “Performance Analysis of MobileNET on Grape Leaf Disease Detection,” 2024 Int. Conf. Adv. Inf. Sci. Dev. (ICAISD). IEEE, vol. 64–68, 2024.

M. O. Syahputra and L. Rosnita, “Analysis of Public Sentiment Toward Celebrity Endorsement On Media Social Using Support Vector Machine,” Int. J. Eng. Sci. Inf. Technol., vol. 4, no. 3, pp. 118–127, 2024.




DOI: https://doi.org/10.52088/ijesty.v5i3.948

Article Metrics

Abstract view : 0 times
PDF - 0 times

Refbacks

  • There are currently no refbacks.


Copyright (c) 2025 Lilis Nur Hayati, Anik Nur Handayani, Wahyu Sakti Gunawan Irianto, Rosa Andrie Asmara, Dolly Indra, Nor Salwa Damanhuri

International Journal of Engineering, Science, and Information Technology (IJESTY) eISSN 2775-2674