The "guest mode" allows for users not on the WiFi network to see and interact with the Chromecast using WiFi Direct. I do not believe there are any options on the Chromecast itself to limit access to users already on the local WiFi.
FWIW, Chromecast Audios are one step further than this, as they do not support this "guest mode" as they do not have a way to display the PIN. Anyone on the WiFi network can freely connect to the device.
That said, I have never seen a device automatically connect to a Chromecast or Chromecast Audio. Normally you have to explicitly connect to a Chromecast.
FWIW, Chromecast Audios are one step further than this, as they do not support this "guest mode" as they do not have a way to display the PIN. Anyone on the WiFi network can freely connect to the device.
That said, I have never seen a device automatically connect to a Chromecast or Chromecast Audio. Normally you have to explicitly connect to a Chromecast.