Article Open Access

Big Data and Data Mining for Efficient Energy Storage and Management

Mustafa Nazar, Zaid Ghanim Ali, Kahtan Mohammed Adnan, Ibraheem Mohammed Khalil, Waleed Nassar, Siti Sarah Maidin

Abstract


The rapid expansion of decentralized and renewable energy systems necessitates intelligent strategies for energy storage and management. This paper presents a comprehensive framework that leverages big data analytics and data mining to optimize energy storage systems within smart grid architectures. By integrating high-frequency data from IoT-enabled Li-Ion batteries, flow batteries, supercapacitor arrays, and hybrid systems, our methodology enhances storage efficiency, predictive accuracy, and fault detection. The approach uniquely combines an ensemble forecasting model (Random Forest and XGBoost), which achieved a 97% R² score in predicting energy demand, with Gaussian Mixture Models for consumer pattern clustering and canonical correlation analysis to model the impact of environmental variables. Validation on real-world datasets demonstrates significant performance gains without additional hardware. For instance, algorithmic optimization improved the round-trip efficiency of a Hybrid Battery Energy Storage System from 86.7% to 93.3% and a Li-Ion battery by 7%. The study underscores the critical influence of contextual variables like temperature and humidity on state-of-charge stability. Furthermore, the analytical framework demonstrated a 50% increase in system throughput (from 34 to 51 tasks/sec) after optimization. This research provides a replicable, data-driven model for deploying intelligent analytics in both microgrid and industrial-scale settings, paving the way for more adaptive and resilient energy infrastructures. Future work will explore edge computing and reinforcement learning to further enhance scalability and autonomy.


Keywords


Big Data Analytics, Energy Storage Optimization, Machine Learning, Smart Grids, Predictive Modelling

References


Liao, H., E. Michalenko, and S.C. Vegunta Review of Big Data Analytics for Smart Electrical Energy Systems. Energies, 2023. 16, DOI: 10.3390/en16083581.

Liu, H., Ni, Y., Wang, M., & Chang, L., Research and Application of Big Data Analysis in Energy Storage of Distributed Energy System. Journal of Physics: Conference Series, 2024.

Xiang, Z., et al., Operation Strategy Optimization of Energy Storage Virtual Synchronous Machine Using Big Data Analysis Technology. Journal of Physics: Conference Series, 2023. 2665(1): p. 012004.

Ponnusamy, V.K., et al. A Comprehensive Review on Sustainable Aspects of Big Data Analytics for the Smart Grid. Sustainability, 2021. 13, DOI: 10.3390/su132313322.

Dhanalakshmi, J. and N. Ayyanathan, A systematic review of big data in energy analytics using energy computing techniques. Concurrency and Computation: Practice and Experience, 2022. 34(4): p. e6647.

Zainab, A., et al., Big Data Management in Smart Grids: Technologies and Challenges. IEEE Access, 2021. 9: p. 73046-73059.

Liu, J., et al., Editorial: Advanced data-driven methods and applications for smart power and energy systems. Frontiers in Energy Research, 2023. Volume 10 - 2022.

Olanrewaju, O., Oduro, P., & Simpa, P., Engineering solutions for clean energy: Optimizing renewable energy systems with advanced data analytics Engineering Science & Technology Journal, 2024. 5(6).

Lu, G., et al. Research on Distributed Control of Energy Storage Based on Big Data Algorithm. in 2023 2nd International Conference on Artificial Intelligence and Autonomous Robot Systems (AIARS). 2023.

Liu, W., J. Zhao, and D. Wang, Data mining for energy systems: Review and prospect. WIREs Data Mining and Knowledge Discovery, 2021. 11(4): p. e1406.

D. A. Dewi and T. B. Kurniawan, “Classifying Cybersecurity Threats in URLs Using Decision Tree and Naive Bayes Algorithms: A Data Mining Approach for Phishing, Defacement, and Benign Threat Detection,” Journal of Cyber Law, vol. 1, no. 2, pp. 175–189, 2025, doi: 10.63913/jcl.v1i2.10.

J. P. B. Saputra, and A. Kumar, “Emotion Detection in Railway Complaints Using Deep Learning and Transformer Models : A Data Mining Approach to Analyzing Public Sentiment on Twitter,” Journal of Digital Society, vol. 1, no. 2, pp. 1–14, 2025, doi: 10.63913/jds.v1i2.6.

Y. Durachman and A. W. Bin Abdul Rahman, “Clustering Student Behavioral Patterns: A Data Mining Approach Using K-Means for Analyzing Study Hours, Attendance, and Tutoring Sessions in Educational Achievement,” Artificial Intelligence in Learning, vol. 1, no. 1, pp. 35–53, 2025, doi: 10.63913/ail.v1i1.5.

Ardabili, S., et al., Systematic Review of Deep Learning and Machine Learning for Building Energy. Frontiers in Energy Research, 2022. Volume 10 - 2022.

Vu, T.T., et al., Energy-Based Proportional Fairness in Cooperative Edge Computing. IEEE Transactions on Mobile Computing, 2024. 23(12): p. 12229-12246.

Z. Tian, Z. Lu, and Y. Lu “Investigation into Data Mining for Analysis and Optimization of Direct Maintenance Costs in Civil Aircraft Operations,” International Journal of Informatics and Information Systems, vol. 7, no. 1, pp. 35–43, 2024, doi: 10.47738/ijiis.v7i1.190.

I. G. A. K. Warmayana, Y. Yamashita, and N. Oka, “Decentralized Materials Data Management using Blockchain, Non-Fungible Tokens, and Interplanetary File System in Web3,” Journal of Applied Data Sciences, vol. 6, no. 1, pp. 742–752, 2025, doi: 10.47738/jads.v6i1.380.

Alpízar-Castillo, J., et al. Open-Access Model of a PV–BESS System: Quantifying Power and Energy Exchange for Peak-Shaving and Self Consumption Applications. Energies, 2023. 16, DOI: 10.3390/en16145480.

Zhang, J., et al., Research on monitoring and energy management systems for energy storage stations on the power generation side. Journal of Physics: Conference Series, 2024. 2846(1): p. 012040.

S. F. Pratama and P. A. Prastyo, “Evaluating Blockchain Adoption in Indonesia’s Supply Chain Management Sector,” Journal of Current Research in Blockchain, vol. 1, no. 3, pp. 190–213, 2024, doi: 10.47738/jcrb.v1i3.21.

A. Wang and Z. Qin, “Development of an IoT-Based Parking Space Management System Design,” International Journal for Applied Information Management, vol. 3, no. 2, pp. 91–100, 2023, doi: 10.47738/ijaim.v3i2.54.

P. Vinoth Kumar, S. Priya, D. Gunapriya, and M. Batumalay, “Novel Battery Management with Fuzzy Tuned Low Voltage Chopper and Machine Learning Controlled Drive for Electric Vehicle Battery Management: A Pathway Towards SDG,” Journal of Applied Data Sciences, vol. 5, no. 3, pp. 936–947, 2024, doi: 10.47738/jads.v5i3.236.

Dong, S., et al. Research on Architecture of Power Big Data High-Speed Storage System for Energy Interconnection. in 2021 IEEE 4th International Conference on Automation, Electronics and Electrical Engineering (AUTEEE). 2021.

Nesca, M., et al., A scoping review of preprocessing methods for unstructured text data to assess data quality. International Journal of Population Data Science, 2022. 7(1).

S. N. Z. H. Dzulkarnain, M. K. M. Nawawi, and R. Kashim, “Developing a Parallel Network Slack-Based Measure Model in the Occurrence of Hybrid Integer-Valued Data and Uncontrollable Factors,” Journal of Applied Data Sciences, vol. 5, no. 4, pp. 1790–1801, 2024, doi: 10.47738/jads.v5i4.407.

A. D. Buchdadi and A. S. M. Al-Rawahna, “Anomaly Detection in Open Metaverse Blockchain Transactions Using Isolation Forest and Autoencoder Neural Networks,” International Journal Research on Metaverse, vol. 2, no. 1, pp. 24–51, 2025, doi: 10.47738/ijrm.v2i1.20.

Marangu, D., Njenga, S., & Ndung'u, R. , Systematic Review of Models Used to Handle Class Imbalance in Anomaly Detection for Energy Consumption. International Journal of Artificial Intelligence and Applications (IJAIA), 2024. 15(3).

Verwiebe, P.A., et al. Modeling Energy Demand—A Systematic Literature Review. Energies, 2021. 14, DOI: 10.3390/en14237859.

A. R. Hananto and B. Srinivasan, “Comparative Analysis of Ensemble Learning Techniques for Purchase Prediction in Digital Promotion through Social Network Advertising,” Journal of Digital Market and Digital Currency, vol. 1, no. 2, pp. 125–143, 2024, doi: 10.47738/jdmdc.v1i2.7.

Wang, L., et al., Energy Management Strategy and Optimal Sizing for Hybrid Energy Storage Systems Using an Evolutionary Algorithm. IEEE Transactions on Intelligent Transportation Systems, 2022. 23(9): p. 14283-14293.




DOI: https://doi.org/10.52088/ijesty.v5i4.1759

Refbacks

  • There are currently no refbacks.


Copyright (c) 2025 Mustafa Nazar, Zaid Ghanim Ali, Kahtan Mohammed Adnan, Ibraheem Mohammed Khalil, Waleed Nassar, Siti Sarah Maidin

International Journal of Engineering, Science, and Information Technology (IJESTY) eISSN 2775-2674