Performance of real-time adaptive optics compensation in a turbulent channel with high-dimensional spatial-mode encoding