You could disable the display manager service with systemctl for example if your display manager is lightdm the run sudo systemctl disable lightdm.service. This will prevent the service from starting at boot.
I forgot to mention how to start the GUI, Its as simple as starting the service sudo systemctl start lightdm.service