The bonding across the lattice and ordered structures endow crystals with unique symmetry and determine their macroscopic properties. Crystals with unique properties such as low-dimensional materials, metal-organic frameworks, and defected crystals, in particular, exhibit different structures from bulk crystals and possess exotic physical properties, making them intriguing subjects for investigation. To accurately predict the physical and chemical properties of crystals, it is crucial to consider long-range orders. While GNN excels at capturing the local environment of atoms in crystals, they often face challenges in effectively capturing longer-ranged interactions due to their limited depth. In this paper, we propose CrysToGraph ( Crystals with Transformers on Graph), a novel and robust transformer-based geometric graph network designed for unconventional crystalline systems, and UnconvBench, a comprehensive benchmark to evaluate models’ predictive performance on multiple categories of crystal materials. CrysToGraph effectively captures short-range interactions with transformer-based graph convolution blocks as well as long-range interactions with graph-wise transformer blocks. CrysToGraph proofs its effectiveness in modelling all types of crystal materials in multiple tasks, and moreover, it outperforms most existing methods, achieving new state-of-the-art results on two benchmarks. This work enhances the development of novel crystal materials in various fields, including the anodes, cathodes, and solid-state electrolytes.