In a previous study, we developed an interface using semisynchronous speech and pen input. In this interface, a user speaks while writing, and the pen input complements the speech, enabling a higher recognition performance than with speech alone. When a user inputs speech and pen, there is a time lag between the two modes, and the lag differs among users. We propose a method for adapting to the different time lags of individual users. This method was evaluated in a Japanese continuous speech recognition task with three different pen-input interfaces including a QWERTY keyboard interface. The time-lag adaptation improved recognition accuracies by up to 0.5 point.