Effective camouflage is needed for defense personnel and assets to blend seamlessly with the complex and dynamic environments. The camouflage technique must maintain the concealment capability across various environments and geographic regions. However, existing approaches, including manual design, computer-aided techniques, and deep learning methods, face significant challenges in achieving automation, scalability, and generalization across diverse and uncalibrated scenes. To address these limitations, we propose a novel diffusion-based method with few-shot learning to generate environment-guided camouflage patterns. Our method, called CamoX, consists of two major stages: meta learning and few-shot learning. In the meta learning stage, our method introduces a latent diffusion-based architecture that automatically generates camouflage patterns from noise, eliminating manual intervention and enabling scalable production. In the few-shot learning stage, our approach enforces similarity between the latent features of the camouflage patterns and target scenes by optimizing the guided mean absolute error loss. This innovation allows the generated camouflage patterns to adapt seamlessly to multiple environments with minimal retraining. Furthermore, this paper introduces a comprehensive camouflage dataset, called Camo-Meta, comprising 144,750 realistic camouflage patterns and associated metadata to support research in camouflage generation. Experimental results on multiple datasets demonstrate that CamoX outperforms existing state-of-the-art methods in key metrics.