You could disable the display manager service with
systemctl for example if your display manager is
lightdm the run
sudo systemctl disable lightdm.service. This will prevent the service from starting at boot.
I forgot to mention how to start the GUI, Its as simple as starting the service
sudo systemctl start lightdm.service