GitLab support request

[filipproch]

Expected Behaviour

Installation should complete successfully.

Current Behaviour

Installations ends with a lot of

Failed to connect to 127.0.0.1 port 31111: Connection refused

And the OpenFaaS Cloud doesn't work (tried multiple times, used ./scripts/reset.sh every time)

Steps to Reproduce (for bugs)

1. ofc-bootstrap apply -f init.yaml
2. wait...

Context

Trying to deploy OpenFaaS cloud on my cluster

Your Environment

• faas-cli version ( full output ):

CLI:
 commit:  73004c23e5a4d3fdb7352f953247473477477a64
 version: 0.11.3

• Docker version docker version (e.g. Docker 17.0.05 ):

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

• Kubernetes version kubectl version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:42:50Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

• Are you using faas-netes or the Operator?

• Operating System and version (e.g. Linux, Windows, MacOS):
  MacOS

• Link to your project or a code example to reproduce issue:
  I just cloned the ofc-bootstrap repo and edited init.yaml

• Please also follow the troubleshooting guide and paste in any other diagnostic information you have:

• Cluster
  Digitalocean, freshly created cluster

• Registry/SCM
  GitLab EE, and GitLab Container Registry deployed on DO Droplet

The errors starts right here

Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Checking if OpenFaaS GW is up.
Calling the OpenFaaS server to validate the credentials...
Cannot connect to OpenFaaS on URL: http://127.0.0.1:31111. Get http://127.0.0.1:31111/system/functions: dial tcp 127.0.0.1:31111: connect: connection refused
Deploying: function-logs.
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.

Is OpenFaaS deployed? Do you need to specify the --gateway flag?
Put http://127.0.0.1:31111/system/functions: dial tcp 127.0.0.1:31111: connect: connection refused

Deploying: git-tar.
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.

Is OpenFaaS deployed? Do you need to specify the --gateway flag?
Put http://127.0.0.1:31111/system/functions: dial tcp 127.0.0.1:31111: connect: connection refused

Before everything seems okay.

[alexellis]

Please put the full logs into a gist and link it here.

What GitLab version is this? What you mean mean by "on DigitalOcean"? Are you using DOKS or your own kubernetes installation?

[alexellis]

/set title: GitLab support request

[alexellis]

Cc @martindekov

[filipproch]

GitLab version: GitLab Enterprise Edition 12.6.0-ee
Gist: https://gist.github.com/filipproch/737f44708c6ae5e733d31925a5deeaf5

[filipproch]

DigitalOcean - using DOKS, created new Cluster from the Web UI (same as the "Video" example shows)

[alexellis]

Given the error messages it looks like you've run the reset or run the tool more than once. Why don't you try running it on a fresh cluster?

[filipproch]

yeah I run it again now (with a reset before), to generate the log file, the first time (still with error), it was on a fresh cluster
will try again then

[alexellis]

Did you run through the whole troubleshooting guide already to see what might have gone wrong? Did you ensure that your cluster than meets the minimum requirements for RAM and nodes?

https://docs.openfaas.com/deployment/troubleshooting/

https://docs.openfaas.com/openfaas-cloud/self-hosted/troubleshoot/

[filipproch]

The same run on a fresh cluster
https://gist.github.com/filipproch/a8026157bd54d642468a31cbfca092ea

Parameters are exactly the same as in demo video (3 nodes, 4GB 2vCPU)

I looked at the troubleshooting but I am not Kubernetes expert, just starting with it and interested in OpenFaaS and this script does a lot of stuff. From what I see, for some reason, OpenFaaS doesn't start.

Also, this error is right in the beginning but I checked that the credentials are alright.

Validating registry credentials fileerror with registry credentials file. Please ensure it has been created correctly

[alexellis]

/msg: slack

[derek]

--
Join Slack to connect with the community
https://docs.openfaas.com/community

[alexellis]

The guides are so to don't need to be a Kubernetes expert. Perhaps try the community cluster which we host or see if someone on Slack can help you.

[blaisep]

@filipproch I have to say, I have made almost every mistake possible with ofc-bootstrap and I am happy to help you out....as a newcomer to k8s, I found that the path to a working ofc-boostrap might appear long and twisty.
Feel free to ping me in the Slack channel.

[alexellis]

Most people tend to get ofc-bootstrap working when following a tutorial or the instructions provided in a short period of time. I know a few folks have struggled, but that's the exception to the rule.

It's the Christmas holidays, but I tried OFC bootstrap today and it worked as expected with a GitHub configuration.

@viveksyngh @martindekov @Waterdrips can any of you take a look at GitLab for OFC? You can install GitLab on a VM or via helm.

[viveksyngh]

From the provided logs .. looks like OpenFaaS installation has failed.

Release "openfaas" does not exist. Installing it now.
 Error: Could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: webhook.cert-manager.io/v1beta1: the server is currently unable to handle the request

and that is the main cause of errors.

[filipproch]

Hi there, thanks to everyone for offering help and looking into this, it seems its because of new Kubernetes 1.16, after going back to 1.15 it works as expected.

[alexellis]

I've tested OpenFaaS Cloud with Kubernetes 1.16 and it worked for me.

@viveksyngh could you reproduce the error? Try with a fake set of GitLab URLs if you are short on time.

[filipproch]

Ok sorry seems false alarm, I now tested with fresh cluster and changing just Kubernetes version and still the same error. It seems before it started to work only because as part of trying to get it to work, I did many many changes (out of pure desperation) and hard to tell, which one solved it.
Unfortunately don't have more time to debug it now, since I already wasted a ton, but if I get to it later, will try to better diagnose the problem.

[alexellis]

I installed with GitHub yesterday and it worked first time. Unfortunately you've chosen a bad time to need help since mostly everyone is on holiday. I've asked a few people to help you, just wait please, no need to "waste a tone of time"

[filipproch]

I installed with GitHub yesterday and it worked first time. Unfortunately you've chosen a bad time to need help since mostly everyone is on holiday. I've asked a few people to help you, just wait please, no need to "waste a tone of time"

Thanks, understand the holidays, will watch this thread, but for me, it's the exact opposite, since I used the free time to play with/try out new stuff like openfaas.

[martindekov]

I am extremely tight on time, I am off from the project till 6th. Currently working on mandatory personal projects not counting the family affairs. I apologize, but I won't be able to take a look/test the Issue until the end of the holidays.

[viveksyngh]

I tried installing OpenFaas Cloud with ofc-bootstrap on a kubernetes cluster with version 1.16 with some dummy gitlab configuration. All ofc services got deployed correctly on running. I could not test the end to end flow but the installation was smooth.

[jyksnw]

I have been working through getting OpenFaaS cloud running on Civo’s managed Kubernetes and I am also running into the issue with getting the error: Could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: webhook.cert-manager.io/v1beta1: the server is currently unable to handle the request Other than deploying to Civo (k3s with traefik not added) I am using GitHub as my scm and Cloudflare for DNS. I found the following bug/issue on cert-manager that looks to be similar in nature to this one cert-manager/cert-manager#2273 some reported that helm 2.14.0 worked but newer versions did not. I have yet to try this. For those that have tested and had it work what version of helm are you running?

[Waterdrips]

I have been working through getting OpenFaaS cloud running on Civo’s managed Kubernetes and I am also running into the issue with getting the error: Could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: webhook.cert-manager.io/v1beta1: the server is currently unable to handle the request Other than deploying to Civo (k3s with trafiek not added) I am using GitHub as my scm and Cloudflare for DNS. I found the following bug/issue on cert-manager that looks to be similar in nature to this one jetstack/cert-manager#2273 some reported that helm 2.14.0 worked but newer versions did not. I have yet to try this. For those that have tested and had it work what version of helm are you running?

I'm using 2.16 - had to install it manually as the package manager wasn't that up to date

[Waterdrips]

I think this issue is related to this Helm issue: helm/helm#6361

I have raised a PR to wait for cert-manager to be ready before we move on, previously this caused the openfaas components not to be installed at all