Probabilistic Top-k Feature Attention with Transformer for Generative Models

This paper presents a novel method for enhancing generative models by leveraging **probabilistic top-k feature extraction** combined with a **Transformer** applied to the most informative features. Inspired by the concept of **Markov blankets**, our approach identifies and boosts the features that most influence output generation, allowing improved sample quality in Autoencoder-based frameworks. We provide an information-theoretic proof demonstrating that our method maximizes mutual information between selected latent features and the target output, reducing redundancy compared to standard Autoencoders (AEs) and Variational Autoencoders (VAEs). Experiments on the MNIST dataset show improved reconstruction quality while maintaining computational efficiency.