For me worked following configuration for jwilder/nginx-proxy container.

web:
  image: 'gitlab/gitlab-ce:latest'
  hostname: 'gitlab.it-expert.com.ua'
  environment:
    GITLAB_OMNIBUS_CONFIG: |
      external_url 'https://gitlab.it-expert.com.ua'
      registry_external_url 'https://registry.it-expert.com.ua'
    VIRTUAL_HOST: gitlab.it-expert.com.ua,registry.it-expert.com.ua
    VIRTUAL_PORT: 443
    VIRTUAL_PROTO: https
  volumes:
    - './data/config:/etc/gitlab'
    - './data/logs:/var/log/gitlab'
    - './data/data:/var/opt/gitlab'

Tricky part was to figure out how containers is connected and which and who should process SSL.

For this configuration you should supply SSL certificates both for nginx-proxy and gitlab-ce containers, because communications between them is also using SSL. For gitlab-ce use ./data/config/ssl folder.