Not really through Caddy but for my setup I have it so the ssh port for Forgejo is only accessible through tailscale. So for push/pulling updated my ssh config file to something like

Host git.mysite.com HostName tailscaleMachineName User git Port 1234

Then doing git pull git@git.mysite.com:user/project.git works just fine as long as I am connected to tailscale

Otherwise you could open the port for Forgejo’s ssh so that you can access it without any vpn