Optical coherence tomography (OCT) offers clinicians detailed in vivo structural imaging of individual retinal layers. This has lead to novel observations which were impossible by direct ophthalmoscopy. The technique also helps to re-introduce the concept of retinotopy to clinical practise. This review illustrates the anatomical foundation for retinotopy through detailed sketches of the pre- and post-laminar axons forming the optic nerve and their post-synaptic path from the dorsal lateral geniculate nucleus to the primary visual cortex in humans. With the mapped neuroanatomy in mind we use OCT-MRI pairs to discuss the patterns of neurodegeneration in eye and brain that are a consequence of the hard wired retinotopy: antero- and retrograde axonal degeneration which can, within the visual system, propagate trans-synaptically. The technical advances of OCT and MRI for the first time enable us to trace the signals of axonal degeneration through the visual system entirely at spectacular resolution. In addition to atrophic processes, we also summarize novel insights on tissue thickening mainly at the level of the optic disc, in particular peripaphillary hyperreflecitive ovoid mass like structures (PHOMS). In conclusion, the neuroanatomical insights provided by the combination of OCT and MRI allows us to separate incidental findings from sinister pathology and gives us new opportunities to tailor and monitor novel neuroprotective strategies.