The role of machine learning algorithms in improving the efficiency of statistical analysis for detecting nonlinear patterns in financial data: A comparative study between traditional and intelligent methods

Abstract

This research aims to study the transformative role of machine learning algorithms in improving the efficiency of statistical analysis of financial data, focusing on their ability to detect non-linear patterns that traditional statistical methods fail to capture. The study adopted a mixed-methods approach, combining a survey of a purposive sample of 72 experts and specialists in quantitative financial analysis with an experimental application comparing the performance of two machine learning models (Artificial Neural Networks and XGBoost) against a Multiple Linear Regression model, using real data from the S&P 500 index. The survey results revealed a high consensus among experts regarding the superiority of intelligent models in prediction accuracy and uncovering non-linear patterns, with this superiority being amplified with larger data volumes. The experimental results quantitatively confirmed these findings, demonstrating a clear outperformance of the machine learning models over the traditional one. The Artificial Neural Network achieved a 37% reduction in Root Mean Square Error (RMSE). Furthermore, these models showed a superior capability to identify anomalous and volatile market periods using techniques like clustering (K-Means) and anomaly detection (Isolation Forest). Based on these findings, the research recommends that financial institutions gradually adopt machine learning methodologies, invest in big data infrastructure, and calls for researchers to develop hybrid models and deepen comparative applied studies.