My personal opinion? It seems moderately unlikely that a player as strong as Kasparov, who could have played the entire game against you blindfolded, would have been adversely affected by this. Most strong players' personal chessboards have no coordinates written on them at all. (Mine certainly don't, and I'm not even strong. Identifying squares is just trivial.)
He probably just wasn't putting much mental energy into your game, as he had other simultaneous games to worry about, and then a move you made perhaps caught him by surprise in some way.
(It also seems unlikely coming from Kasparov specifically - he has historically raised quite a fuss, even when on camera, when simul exhibitions were not conducted to his standards. If the board bothered him, I think he would have had it flipped.)
He probably just wasn't putting much mental energy into your game, as he had other simultaneous games to worry about, and then a move you made perhaps caught him by surprise in some way.
(It also seems unlikely coming from Kasparov specifically - he has historically raised quite a fuss, even when on camera, when simul exhibitions were not conducted to his standards. If the board bothered him, I think he would have had it flipped.)