What can distract
Nothing, and everything, on a spectral scale. It really depends on how closely you mask the world (also see audio, below). There’s nothing wrong in designing something that operates as a heritage tour: directing the reader’s attention to things, explaining them and focusing them exactly where you want them to be. But it’s like being inside an Occulus Rift, suffocating, with an absence of peripheral vision, and no awareness of the real stuff around you. If you want to write VR, then write VR.
The value in digital’s relationship to reality is that it can abstract it away as needed. It can draw the eye (or the ear, or the hand) in as required, and bring everything down to the detail of a chalk mark on a wall, and it can zoom back out to the whole panorama. Weather is going to distract, but since that’s a given, we’d suggest you ignore it. The time of day is your friend though—the same piece experienced in daylight and at dusk might as well be in two completely different locales.