How to operator Kubernetes cluster in LocalStack EKS?

seikyo-cho-lvgs · May 22, 2024, 4:20pm

When I create an EKS cluster with LocalStack:

awslocal eks create-cluster \
  --name cluster1 \
  --role-arn "arn:aws:iam::000000000000:role/eks-role" \
  --resources-vpc-config "{}"

Confirm:

awslocal eks describe-cluster --name cluster1
{
    "cluster": {
        "name": "cluster1",
        "arn": "arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1",
        "createdAt": "2024-05-22T13:52:00.327000+09:00",
        "version": "1.22",
        "endpoint": "https://localhost.localstack.cloud:4511",
        "roleArn": "arn:aws:iam::000000000000:role/eks-role",
        "resourcesVpcConfig": {
            "securityGroupIds": [],
            "endpointPublicAccess": true,
            "endpointPrivateAccess": false,
            "publicAccessCidrs": [
                "0.0.0.0/0"
            ]
        },
        "identity": {
            "oidc": {
                "issuer": "https://localhost.localstack.cloud/eks-oidc"
            }
        },
        "status": "ACTIVE",
....
}

Set kubernetes context:

awslocal eks update-kubeconfig --name cluster1 && \
    kubectl config use-context arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1

Why got error when run kubectl command?

kubectl get po
I0522 14:04:44.688006   27587 versioner.go:58] Get "https://localhost.localstack.cloud:4511/version?timeout=5s": EOF
E0522 14:04:55.119358   27587 memcache.go:265] couldn't get current server API group list: Get "https://localhost.localstack.cloud:4511/api?timeout=32s": EOF

The EKS’ endpoint is using 4511. The LocalStack’s default endpoint is 4566.
Is the port a reason?

Marcel · May 23, 2024, 12:06pm

Hi @seikyo-cho-lvgs,

For specific services, ports are assigned from the 4510-4559 range. You can see which ports are available in the docker-compose.yml.

Please check your local configuration of the .kube\config file. For a quick check, create a backup and then remove it and try again, creating the EKS in LocalStack.

seikyo-cho-lvgs · May 24, 2024, 1:43am

Hi @Marcel ,

Thank you for your reply.
I didn’t use docker-compose but created the EKS cluster by this guide:

About the port in my .kube/config, it’s 4511.

apiVersion: v1
kind: Config
clusters:
  - name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
    cluster:
      server: https://localhost.localstack.cloud:4511
      certificate-authority-data: very-long-string
      insecure-skip-tls-verify: false
users:
  - name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1beta1
        args:
          - --region
          - ap-northeast-1
          - eks
          - get-token
          - --cluster-name
          - cluster1
          - --output
          - json
        command: aws
contexts:
  - name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
    context:
      cluster: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
      name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
      user: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
preferences: {}
current-context: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1

The port is in the range, but why can’t conenct?

Marcel · May 24, 2024, 5:48am

I am unable to reproduce your issue with the shared details.

The EKS and the .kube/config file is correctly created,

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: very-long-string
    server: https://localhost.localstack.cloud:4510
  name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
contexts:
- context:
    cluster: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
    user: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
  name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
current-context: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
kind: Config
preferences: {}
users:
- name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - ap-northeast-1
      - eks
      - get-token
      - --cluster-name
      - cluster1
      - --output
      - json
      command: aws

I am able to invoke the kubectl afterwards

kubectl cluster-info
Kubernetes control plane is running at https://localhost.localstack.cloud:4510
CoreDNS is running at https://localhost.localstack.cloud:4510/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://localhost.localstack.cloud:4510/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy

Can you please confirm in what type of environment are you running LocalStack?
Are you changing the default configuration, as shown in docker-compose.yml?

seikyo-cho-lvgs · May 24, 2024, 7:12am

Hi @Marcel ,

I’m using Kubernetes in Rancher Desktop. I started LocalStack by CLI:

localstack start

Maybe clear the .kube/config can fix the issue:

bug: Not able to connect to an eks cluster on localstack via kubectl · Issue #10456 · localstack/localstack · GitHub

But I can’t confirm it now since when I create a new cluster this time, got status FAILED:

awslocal eks create-cluster \
  --name cluster1 \
  --role-arn "arn:aws:iam::000000000000:role/eks-role" \
  --resources-vpc-config "{}"
{
    "cluster": {
        "name": "cluster1",
        "arn": "arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1",
        "createdAt": "2024-05-24T16:04:50.704000+09:00",
        "version": "1.22",
        "roleArn": "arn:aws:iam::000000000000:role/eks-role",
        "resourcesVpcConfig": {
            "securityGroupIds": [],
            "endpointPublicAccess": true,
            "endpointPrivateAccess": false,
            "publicAccessCidrs": [
                "0.0.0.0/0"
            ]
        },
        "identity": {
            "oidc": {
                "issuer": "https://localhost.localstack.cloud/eks-oidc"
            }
        },
        "status": "CREATING",
        "platformVersion": "eks.5",
        "tags": {}
    }
}

awslocal eks describe-cluster --name cluster1
{
    "cluster": {
        "name": "cluster1",
        "arn": "arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1",
        "createdAt": "2024-05-24T16:04:50.704000+09:00",
        "version": "1.22",
        "roleArn": "arn:aws:iam::000000000000:role/eks-role",
        "resourcesVpcConfig": {
            "securityGroupIds": [],
            "endpointPublicAccess": true,
            "endpointPrivateAccess": false,
            "publicAccessCidrs": [
                "0.0.0.0/0"
            ]
        },
        "identity": {
            "oidc": {
                "issuer": "https://localhost.localstack.cloud/eks-oidc"
            }
        },
        "status": "FAILED",
        "platformVersion": "eks.5",
        "tags": {}
    }
}

I don’t know why.

seikyo-cho-lvgs · May 27, 2024, 2:17am

After I delete arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1 context in .kube/config, I can create the EKS cluster again.

localstack start

awslocal eks create-cluster \
  --name cluster1 \
  --role-arn "arn:aws:iam::000000000000:role/eks-role" \
  --resources-vpc-config "{}"

But when I run the kubectl command:

kubectl get nodes

I got this error:

E0527 11:08:27.358638   30722 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0527 11:08:27.642766   30722 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0527 11:08:27.932002   30722 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0527 11:08:28.215254   30722 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0527 11:08:28.509132   30722 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

When I run the command with -v

kubectl get nodes -v

The details is

kubectl get nodes -v=10
I0527 11:07:01.291100   30213 loader.go:395] Config loaded from file:  /Users/test/.kube/config
I0527 11:07:01.291405   30213 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.30.1 (darwin/arm64) kubernetes/6911225" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" 'https://localhost.localstack.cloud:4510/api?timeout=32s'
I0527 11:07:01.589791   30213 round_trippers.go:495] HTTP Trace: DNS Lookup for localhost.localstack.cloud resolved to [{127.0.0.1 }]
I0527 11:07:01.590093   30213 round_trippers.go:510] HTTP Trace: Dial to tcp:127.0.0.1:4510 succeed
I0527 11:07:01.888528   30213 round_trippers.go:553] GET https://localhost.localstack.cloud:4510/api?timeout=32s 401 Unauthorized in 597 milliseconds
I0527 11:07:01.888560   30213 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 0 ms TLSHandshake 3 ms ServerProcessing 5 ms Duration 597 ms
I0527 11:07:01.888564   30213 round_trippers.go:577] Response Headers:
I0527 11:07:01.888569   30213 round_trippers.go:580]     Cache-Control: no-cache, private
I0527 11:07:01.888574   30213 round_trippers.go:580]     Content-Type: application/json
I0527 11:07:01.888576   30213 round_trippers.go:580]     Date: Mon, 27 May 2024 02:07:00 GMT
I0527 11:07:01.888578   30213 round_trippers.go:580]     Content-Length: 129
I0527 11:07:01.888580   30213 round_trippers.go:580]     Audit-Id: d7d94a11-dab3-438c-87e3-a45ebff604a7
I0527 11:07:01.889035   30213 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0527 11:07:01.889291   30213 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0527 11:07:01.889304   30213 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0527 11:07:01.889391   30213 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.30.1 (darwin/arm64) kubernetes/6911225" 'https://localhost.localstack.cloud:4510/api?timeout=32s'
I0527 11:07:02.179045   30213 round_trippers.go:553] GET https://localhost.localstack.cloud:4510/api?timeout=32s 401 Unauthorized in 289 milliseconds
I0527 11:07:02.179075   30213 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 3 ms Duration 289 ms
I0527 11:07:02.179079   30213 round_trippers.go:577] Response Headers:
I0527 11:07:02.179085   30213 round_trippers.go:580]     Cache-Control: no-cache, private
I0527 11:07:02.179087   30213 round_trippers.go:580]     Content-Type: application/json
I0527 11:07:02.179089   30213 round_trippers.go:580]     Date: Mon, 27 May 2024 02:07:00 GMT
I0527 11:07:02.179091   30213 round_trippers.go:580]     Content-Length: 129
I0527 11:07:02.179093   30213 round_trippers.go:580]     Audit-Id: 4db51446-b5be-4109-977e-140d55557414
I0527 11:07:02.180338   30213 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0527 11:07:02.180593   30213 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0527 11:07:02.180604   30213 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0527 11:07:02.180609   30213 shortcut.go:103] Error loading discovery information: the server has asked for the client to provide credentials
I0527 11:07:02.180675   30213 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.30.1 (darwin/arm64) kubernetes/6911225" 'https://localhost.localstack.cloud:4510/api?timeout=32s'
I0527 11:07:02.470294   30213 round_trippers.go:553] GET https://localhost.localstack.cloud:4510/api?timeout=32s 401 Unauthorized in 289 milliseconds
I0527 11:07:02.470318   30213 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 3 ms Duration 289 ms
I0527 11:07:02.470324   30213 round_trippers.go:577] Response Headers:
I0527 11:07:02.470328   30213 round_trippers.go:580]     Audit-Id: 7733bff9-2e86-4a35-8b40-82adfa2c0531
I0527 11:07:02.470331   30213 round_trippers.go:580]     Cache-Control: no-cache, private
I0527 11:07:02.470334   30213 round_trippers.go:580]     Content-Type: application/json
I0527 11:07:02.470336   30213 round_trippers.go:580]     Date: Mon, 27 May 2024 02:07:01 GMT
I0527 11:07:02.470340   30213 round_trippers.go:580]     Content-Length: 129
I0527 11:07:02.470807   30213 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0527 11:07:02.471059   30213 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0527 11:07:02.471072   30213 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0527 11:07:02.471153   30213 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.30.1 (darwin/arm64) kubernetes/6911225" 'https://localhost.localstack.cloud:4510/api?timeout=32s'
I0527 11:07:02.756941   30213 round_trippers.go:553] GET https://localhost.localstack.cloud:4510/api?timeout=32s 401 Unauthorized in 285 milliseconds
I0527 11:07:02.756975   30213 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 3 ms Duration 285 ms
I0527 11:07:02.756978   30213 round_trippers.go:577] Response Headers:
I0527 11:07:02.756984   30213 round_trippers.go:580]     Content-Type: application/json
I0527 11:07:02.756986   30213 round_trippers.go:580]     Date: Mon, 27 May 2024 02:07:01 GMT
I0527 11:07:02.756989   30213 round_trippers.go:580]     Content-Length: 129
I0527 11:07:02.756991   30213 round_trippers.go:580]     Audit-Id: a6462cbd-e99d-4214-9111-157462643de9
I0527 11:07:02.756994   30213 round_trippers.go:580]     Cache-Control: no-cache, private
I0527 11:07:02.757417   30213 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0527 11:07:02.757670   30213 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0527 11:07:02.757680   30213 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0527 11:07:02.757746   30213 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.30.1 (darwin/arm64) kubernetes/6911225" 'https://localhost.localstack.cloud:4510/api?timeout=32s'
I0527 11:07:03.053843   30213 round_trippers.go:553] GET https://localhost.localstack.cloud:4510/api?timeout=32s 401 Unauthorized in 296 milliseconds
I0527 11:07:03.053871   30213 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 3 ms Duration 296 ms
I0527 11:07:03.053875   30213 round_trippers.go:577] Response Headers:
I0527 11:07:03.053880   30213 round_trippers.go:580]     Audit-Id: ec721429-86b8-47ad-acc0-79f39a3431cc
I0527 11:07:03.053897   30213 round_trippers.go:580]     Cache-Control: no-cache, private
I0527 11:07:03.053903   30213 round_trippers.go:580]     Content-Type: application/json
I0527 11:07:03.053905   30213 round_trippers.go:580]     Date: Mon, 27 May 2024 02:07:01 GMT
I0527 11:07:03.053906   30213 round_trippers.go:580]     Content-Length: 129
I0527 11:07:03.054261   30213 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E0527 11:07:03.054550   30213 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0527 11:07:03.054565   30213 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
I0527 11:07:03.054716   30213 helpers.go:246] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server has asked for the client to provide credentials",
  "reason": "Unauthorized",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "unknown"
      }
    ]
  },
  "code": 401
}]
error: You must be logged in to the server (the server has asked for the client to provide credentials)

Why the server asked for the client to provide credentials?
Doesn’t it provide by default? How to provide?

Marcel · May 27, 2024, 4:44am

Hi @seikyo-cho-lvgs,

The new error corresponds with the one from the GitHub issue bug: Not able to connect to an eks cluster on localstack via kubectl · Issue #10456 · localstack/localstack · GitHub 1.

Have you set the access? Can you please confirm that it’s set in the kube config file?

awslocal eks update-kubeconfig --name cluster1 && \
    kubectl config use-context arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1

seikyo-cho-lvgs · May 27, 2024, 4:58am

Hi @Marcel ,

Thank you for you reply.

Something in the .kube/config is:

users:
- name: rancher-desktop
  user:
    client-certificate-data: very-long-data
    client-key-data: very-long-data
- name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - ap-northeast-1
      - eks
      - get-token
      - --cluster-name
      - cluster1
      - --output
      - json
      command: aws

It seems the arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1 user doesn’t set a credential. And its command is aws. So maybe it’s the issue.
If I don’t connect to AWS, can’t I use LocalStack EKS?

Marcel · May 27, 2024, 6:50am

Please try to set up the AWS configuration similar to the one in the documentation.

seikyo-cho-lvgs · May 27, 2024, 10:14am

Hi @Marcel ,

Thank you. I set localstack profile and rewrite the config for kubernetes:

.aws/config

[profile localstack]
region=ap-northeast-1
output=json
endpoint_url=http://localhost:4566

.aws/credentials

[localstack]
aws_access_key_id=test
aws_secret_access_key=test

.kube/config

users:
- name: arn:aws:eks:ap-northeast-1:000000000000:cluster/cluster1
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - ap-northeast-1
      - eks
      - get-token
      - --cluster-name
      - cluster1
      - --output
      - json
      - --profile
      - localstack
      command: aws

Still can’t connect to the cluster:

E0527 19:01:21.076243   22178 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials

How ever, I can get token by the command directly:

eks get-token --region ap-northeast-1 --cluster-name cluster1 --output json --profile localstack
{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2024-05-27T10:25:47Z",
        "token": "k8s-aws-v1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    }
}

Marcel · May 27, 2024, 2:00pm

I’ve installed and tried to reproduce your issue with Rancher Desktop, but it’s working as expected.

Could you please try to clear the full .kube/ folder even with the cache folder?

I would also suggest doing a factory reset in the Rancher Desktop.

seikyo-cho-lvgs · May 28, 2024, 2:57am

Hi @Marcel ,

I tried your suggestion. Finally, it works:

kubectl get nodes
NAME                    STATUS   ROLES                  AGE   VERSION
k3d-cluster1-server-0   Ready    control-plane,master   25m   v1.22.6+k3s1

Really thank you very much for your help on this topic.

Topic		Replies	Views
EKS cluster not coming up Platform localstack	1	430	May 4, 2023
EKS Cluster in PODMAN/Localstack Feature Requests localstack , discussion-pages	20	1142	July 14, 2023
EKS Cluster creation status "Failed" every time on creation Platform	5	487	November 21, 2023
Localstack deploy on EKS Cluster - s3 NoSuchBucket: The Specified bucket does not exist	1	233	May 14, 2024
EKS and Helm Charts Platform	3	332	December 6, 2023

How to operator Kubernetes cluster in LocalStack EKS?

Related Topics