We previously proposed a decoding method utilizing hypothesis scores weighted by VAD-measures. This method uses two GMMs to obtain confidence measures. To achieve good search performance, we need to adapt the GMMs for environmental noise. We describe a new unsupervised on-line GMM adaptation method based on MAP estimation. The robustness of our method is further improved by weighting updating parameters of GMMs according to the confidence measure for the adaptation data. We also describe an approach to accelerate the adaptation by caching statistical values to adapt GMMs. Experimental results show that our adaptive decoding method significantly improves the word accuracy in a noisy environment with only a minor increase in the computational cost.