Troubleshooting Around Cert-Manager

What happened?

There is an issue with Cert-Manager. It might be due to a previous upgrade of Cert-Manager.

• Certificates are not being automatically renewed.
• Even after the challenge, Let’s Encrypt does not issue certificates.

I thought that the problem is somewhat similar to the issue here.

https://github.com/cert-manager/cert-manager/issues/2517

and someone said that

If this can help someone. I setup http-redirect on my gateway. That’s was redirecting all HTTP request to https including the request for the challenge. Works well after removing the redirect.

What I did

My manifest for ingress was like that,

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - host: subdomain.domain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-service
            port:
              number: 80

for avoiding redirect and adding path to acme-challenge, I added below settings.

metadata:
  name: myapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"

spec:
  rules:
  - host: subdomain.domain.com
    http:
      paths:
      - path: /.well-known/acme-challenge
        pathType: ImplementationSpecific
        backend:
          service:
            name: acme-challenge-service # A service that cert-manager creates for challenge
            port:
              number: 8089

When I did

microk8s.kubectl get certificates

It was still FALSE and status was pending.

Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending


TLS:
  eye-ingress-tls terminates subdomain.domain.com
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  subdomain.domain.com  
                       /.well-known/acme-challenge   acme-challenge-service:8089 (<error: services "acme-challenge-service" not found>)
                       /                             wordpress:80 (10.1.129.66:80)
Annotations:           cert-manager.io/cluster-issuer: letsencrypt-prod
                       kubernetes.io/ingress.class: nginx
                       nginx.ingress.kubernetes.io/force-ssl-redirect: false
                       nginx.ingress.kubernetes.io/proxy-body-size: 50m
                       nginx.ingress.kubernetes.io/ssl-redirect: false
                       nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
Events:                <none>

I changed service name on manifest correctly based on actual service name, and deployed it.

rules:
- host: subdomain.domain.com
  http:
    paths:
    - path: /.well-known/acme-challenge
      pathType: ImplementationSpecific
      backend:
        service:
          name: cm-acme-http-solver-8th5z # This line
          port:
            number: 8089
    - path: /
      pathType: Prefix
      backend:
        service:
          name: wordpress
          port:
            number: 80

The status gets READY. But, the IP adress is not set to certificate.

NAME CLASS HOSTS ADDRESS PORTS AGE
wordpress-ingress subdomain.domain.com 127.0.0.1 80, 443 25h

Umm,,, IP is still not there.

microk8s.kubectl edit deployment nginx-ingress-microk8s-controller -n ingress

and changing parameters like this.

--publish-status-address=157.7.85.160

spec:
  template:
    spec:
      hostNetwork: true

and in the new specification of k8s, “nginx” should be set as like below:

metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
spec:
  ingressClassName: nginx

The next problem

The certificate itself is gotten, and DNS resolution seems working fine. But unfortunatelly, we cannot access to the site with name and do nslookup.

I finally found it, the ports were closed for some reason.

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

Haaaaa—, I spent hours for this.