mike_wooskey

@[email protected], @[email protected], and @[email protected],

THanks for your help. My main issue ended up being that I was trying to use Let's Encrypt's staging mode, but since staging certs are self-signed, Traefik was not accepting the requests. Also, though I had to switch Traefik's logging level to Info instead of error to see that.

By "server log", do you mean traefik's log? If so, this is the only thing I could find (and I don't know what it means): https://lemmy.d.thewooskeys.com/comment/514711

From traefik's access.log:

{"ClientAddr":"192.168.1.17:45930","ClientHost":"192.168.1.17","ClientPort":"45930","ClientUsername":"-","DownstreamContentSize":21,"DownstreamStatus":500,"Duration":13526669,"OriginContentSize":21,"OriginDuration":13462593,"OriginStatus":500,"Overhead":64076,"RequestAddr":"whoami.mydomain.com","RequestContentSize":0,"RequestCount":16032,"RequestHost":"whoami.mydomain.com","RequestMethod":"GET","RequestPath":"/","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"websecure-whoami-vpn@file","ServiceAddr":"10.13.16.1","ServiceName":"whoami-vpn@file","ServiceURL":{"Scheme":"https","Opaque":"","User":null,"Host":"10.13.16.1","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2024-04-30T00:21:51.533176765Z","StartUTC":"2024-04-30T00:21:51.533176765Z","TLSCipher":"TLS_CHACHA20_POLY1305_SHA256","TLSVersion":"1.3","entryPointName":"websecure","level":"info","msg":"","time":"2024-04-30T00:21:51Z"}
{"ClientAddr":"192.168.1.17:45930","ClientHost":"192.168.1.17","ClientPort":"45930","ClientUsername":"-","DownstreamContentSize":21,"DownstreamStatus":500,"Duration":13754666,"OriginContentSize":21,"OriginDuration":13696179,"OriginStatus":500,"Overhead":58487,"RequestAddr":"whoami.mydomain.com","RequestContentSize":0,"RequestCount":16033,"RequestHost":"whoami.mydomain.com","RequestMethod":"GET","RequestPath":"/favicon.ico","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"websecure-whoami-vpn@file","ServiceAddr":"10.13.16.1","ServiceName":"whoami-vpn@file","ServiceURL":{"Scheme":"https","Opaque":"","User":null,"Host":"10.13.16.1","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2024-04-30T00:21:51.74274202Z","StartUTC":"2024-04-30T00:21:51.74274202Z","TLSCipher":"TLS_CHACHA20_POLY1305_SHA256","TLSVersion":"1.3","entryPointName":"websecure","level":"info","msg":"","time":"2024-04-30T00:21:51Z"}

All I can tell from this is that there is a DownstreatStatus of 500. I don't know what that means.

Thanks for helping, @[email protected].

Both traefik containers (on the "server" and "client" VMs) and the wireguard server container were built with TRAEFIK_NETWORK_MODE=host. The VMs can ping each other and the Wireguard containers can ping each other.

Both traefik containers were built with TRAEFIK_LOG_LEVEL=warn but I changed them both to TRAEFIK_LOG_LEVEL=info just now. There's a tad more info in the logs, but nothing that seems pertinent.

Also, just to make sure the app is indeed running, I curled it from it's own container (I'm using myapp here instead of whoami, because whoami doesn't have a shell):

$ curl -L -k --header 'Host: myapp.mydomain.com localhost:8080

I can't seem to display html tags in this comment, but the results are the html tags for the web page for the app - so the app is up and running

Thanks so much for helping me troubleshoot this, @[email protected]!

Is the browser also using the LAN router for DNS? Some browsers are set to use DoT or DoH for DNS, which would mean they’d bypass your router DNS.

My browser was using DoH, but I turned it off and still have the same issue.

Do you also get “Internal Server Error” if you make the request with curl on the CLI on the laptop?

Yes, running curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51 on the laptop results in "Internal Server Error".

How did you check that mydomain is being resolved correctly on the laptop?

ping whoami.mydomain.com hits 192.168.1.51.

What do you get with curl from the other VM, or from the router, or from the host machine of the VM?

From the router:

Shell Output - curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0-
100    17  100    17    0     0   8200      0 --:--:-- --:--:-- --:--:-- 17000

100    21  100    21    0     0    649      0 --:--:-- --:--:-- --:--:--   649
Internal Server Error

From the wireguard client container on the "client" VM:

curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
Internal Server Error

From the traefik container on the "client" VM:

$ curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
Internal Server Error

From the "client" VM itself:

# curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
Internal Server Error

From the wireguard container on the "server" VM:

# curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
Internal Server Error

From the traefik container on the "server" VM (This is interesting. Why can't I ping from this traefik installation but a can from the other? But even though it won't ping, it did resolve to the correct IP):

$ ping whoami.mydomain.com
PING whoami.mydomain.com (192.168.1.51): 56 data bytes
ping: permission denied (are you root?)

From the "server" VM itself:

# curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
Internal Server Error

Thanks for helping, @[email protected].

I'm browsing from my laptop on the same network as promox: 192.168.1.0/24

The tunnel is relevant in that my ultimate goal will be to have "client" in the cloud so I can access my apps from the world while having all traffic into my house be through a VPN.

The VM's IPs are 192.168.1.50 ("server") and 192.168.1.51 ("client"). They can see everything on their subnet and everything on their subnet can see them.

Everything is using my router for DNS, and my router points myapp.mydomain.com and whoami.mydomain.com to “client”. And by "everything" I mean all computers on the subnet and all containers in this project.

Both VMs and my laptop resolve myapp.mydomain.com and whoami.mydomain.com to 192.168.1.51, which is "client", and can ping it.

Thanks for helping, @[email protected].

Both wireguard containers are using my router for DNS, and my router points myapp.mydomain.com and whoami.mydomain.com to "client".

I should add that I'm running Traefik 2.11.2 and wireguard from the Linuxserver image lscr.io/linuxserver/wireguard version v1.0.20210914-ls22.

I don't know if your problem is the same as mine was, but the symptom sounds the same.

The docker-compose.yaml file shown in the Forgejo documentation for docker installation shows this mount:

    volumes:
      - ./forgejo:/data

For me, Forgejo installed and created new resource files in /data and ignored the resource files gitea alread made.

I changed the volume to:

    volumes:
      - data:/var/lib/gitea

Forgejo then recognized the gitea resources.

Thanks for that info. I did combine an upgrade (1.20 to 1.21) with the migrations, but I guess I lucked into it working. My problem was that the container's path to the migrated gitea volume was incorrect.

Can you see the data you copied inside the container?

That led me to my problem! I did have the volume mounted, but the container's path was incorrect: Forgejo was recreating it's resource files as a new install because where it was looking for them, they didn't exist.

Thanks!