In colorectal cancer (CRC) precursor lesions can be missed during screening due to ambiguity, limited depth sensitivity, or obscured by colonic folds. Optical coherence tomography (OCT) with automated detection may enhance accuracy. OCT imaging was performed on polyps and polyp fragments (300 patients). En face projections were then annotated. In processing, depth was then encoded in color to generate en face OCT projections. The projections were used to train an ensemble network based on malignant potential. The area under the curve (AUC) for the detection of malignant potential of all polyps was 0.90, for diminutive (<5 mm) the AUC was 0.88. Indicating a high degree of accuracy for classification of malignant potential ex vivo. Should results hold in vivo, this algorithm would meet the ASGE’s NPV PIVI criteria, which could allow clinical utilization of OCT for lower colon ‘diagnose and leave’ and/or ‘resect and discard’ strategies for diminutive colon polyps.