AI-Driven Auto-Architecting of Scalable Cloud Pipelines Using Workflow Analytics and Meta-Learning

Harish Kumar Krishnamurthy Sukumar; Navneet Dalipkumar Magotra; Shrutika Prakash Mokashi; Surya Narayana Reddy Chintacunta; Rohit Laheri

doi:10.37965/jait.2026.0982

AI-Driven Auto-Architecting of Scalable Cloud Pipelines Using Workflow Analytics and Meta-Learning

Authors

Harish Kumar Krishnamurthy Sukumar Department of Information Science, Shri Dharmasthala Manjunatheshwara College of Engineering and Technology, Dharwad, Karnataka, 580002, India https://orcid.org/0009-0004-5876-4182
Navneet Dalipkumar Magotra Department of Computer Science, Binghamton University, Binghamton 13902, New York, USA
Shrutika Prakash Mokashi Department of Business Intelligence, Northeastern University, New Haven, CT, USA
Surya Narayana Reddy Chintacunta Department of Data Science, University of Maryland Baltimore County, Jersey City, NJ, USA
Rohit Laheri Department of Computer Science & Engineering, Lovely Professional University Phagwara, Jalandhar, India

DOI:

https://doi.org/10.37965/jait.2026.0982

Keywords:

adaptive modular intelligence, cloud pipeline automation, Evolutionary Particle Genetic Optimizer, Flexible Batch Processing Scheduling, meta-orchestration system

Abstract

With the rapid expansion of artificial intelligence (AI) across industries, there is a growing demand for intelligent cloud systems capable of adapting to dynamic workloads and scaling autonomously. Traditional cloud pipeline architectures remain static and manually configured, resulting in performance bottlenecks, inefficient resource utilization, and poor scalability. To overcome these limitations, this study proposes an AI-driven cloud pipeline architecture that leverages historical workflow analytics for the autonomous generation and optimization of cloud pipeline architectures. At its core, the proposed AI-driven cloud pipeline architecture introduces adaptive modular intelligence (AMI), which modularizes tasks based on historical patterns through a dynamic Infrastructure-as-Code (IaC) configuration strategy. It further integrates Flexible Batch Processing Scheduling (FBPS) to intelligently switch between batch and stream processing modes based on workload urgency and system load, enabling real-time scheduling flexibility. To enhance optimization, the Evolutionary Particle Genetic Optimizer (EPGO) refines scheduling configurations through a multi-objective fitness function balancing throughput, latency, and resource efficiency. In parallel, meta-orchestration system (MOS) manages encrypted communication between modules adopting symmetric key encryption, ensuring secure data exchange and protection against unauthorized access or tampering. MOS also tracks configuration changes and dynamically adapts pipeline behaviors without manual intervention, providing a robust governance layer for secure execution, real-time monitoring, and version-controlled reproducibility across the entire cloud infrastructure. Experimental results confirm that the proposed framework surpasses existing methods such as Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Hybrid Genetic Algorithm (HGA), and Hybrid Parallel Enhanced Genetic Algorithm (HPEGA), achieving a minimum MAPE of 4.1%, faster convergence, improved scalability, and higher overall performance in dynamic AI-driven cloud environments.