Enhancing Steganography System Using Deep Learning and Hyperparameter Optimization

Abstract

     Steganography plays a vital role in secure message transmission, embedding information imperceptibly within cover images. This research addresses the challenge of creating stego images visually indistinguishable from originals, while enhancing resistance to steganalysis. We propose a novel steganographic system based on Generative Adversarial Networks (GANs) with hyperparameter tuning to enhance imperceptibility and security. Multi-scale feature extraction via Inception modules within the encoder facilitates efficient embedding with minimal distortion. A new combined metric, incorporating MSE, PAD, and MAE alongside PSNR and SSIM, provides a more comprehensive performance evaluation, addressing limitations of relying solely on PSNR/SSIM. For instance, while RMSprop achieved a PSNR of 57.32 dB, its combined metric score was 16.37, reflecting its weaknesses in error metrics. Conversely, SGD, with a PSNR of 59.55 dB, achieved a combined metric score of 34.73, demonstrating its balanced performance. Multiple optimization algorithms, including Adadelta, Adam, Nosadam, SparseAdam, Adamp, L-BFGS, RAdamplus, Tadam, RMSprop, Nadam, Ftrl, AdaGrad, AdamW, Adamwt, SGD, RAdam, and Naturalgrad, were evaluated. Nadam and Adadelta achieved the highest combined metric scores (36.71 and 34.61, respectively), correlating with superior PSNR and SSIM values, thus ensuring higher image fidelity and imperceptibility. The system effectively impedes steganalysis by reducing statistical discrepancies and utilizes tanh activation functions for enhanced security. This system achieves robust steganographic performance with minimal distortion and improved security, offering significant advancements in secure image communication. The new combined metric provides a nuanced performance assessment, guiding optimal algorithm selection.